How Close Built a New AI Support Tool in a Week—and Made Our Support Team Faster Than Ever

Most people would love for AI to solve their problems instantly, seamlessly, and with zero human input.

But it doesn’t always work like that. 

Recently Joseph Sterner, our head of support, asked our co-founders to purchase a particular AI tool to help streamline his team’s work. They challenged him to see if he could vibe code one instead.

We wanted human, thoughtful, nuanced Support replies—the kinds our team is used to creating. We just wanted AI to help accelerate the “getting started” step. 

The result is a custom-built tool of our own invention. The outcome has been incredible. What Joseph built has helped make our team faster, more focused, and better equipped to offer timely customer support.

Along the way, we learned a ton about how AI tools could support … err, our Support in the future. We also learned how not to approach AI. 

Ultimately, this approach works for us better than an off-the-shelf tool. Here’s why.

What We Learned When Integrating AI Tools in Customer Support

We wanted to build a customer support tool that could do a few things for us:

• Process incoming queries so we can “triage” them and direct them to the right reps
• Create first-draft responses that our team can customize, saving time but not losing the human element
• Reference our existing help desk to provide accurate, customer-friendly information based on Close CRM’s knowledge base

There were a few obstacles in our way. 

First, AI support tools can lack the full context of our knowledge base. This causes AI to “bluff” its responses without working knowledge of Close.

Second, our support team isn’t highly “transactional,” but takes a more human approach. So each draft would still need a once-over from a real human.

No real off-the-shelf AI solution worked for us, so we turned to AI coding platforms like Replit to build a working prototype.

Because we were building an AI tool from “scratch,” it became our Play-Doh: we could mold it to fit into our existing workflow. Joseph wanted a process that could hit three primary targets:

• Intake new queries from Zendesk
• Read through our knowledge base to generate a helpful response
• Then drop that information into a draft for our support team

A rep could log into Zendesk, review the ticket for accuracy, and personalize it before sending it out.

And it worked. It took only a few days before we had a working model about ~80% of the way there.

(AI + human-in-the-loop FTW.)

The Art of Refining AI and Creating a Positive Feedback Loop

A word of caution: if you expect AI to get everything right from the get-go, you’re going to have a bad time.

Yeah, AI is revolutionary, but in our opinion, asking it to fix everything for you is the wrong way to think about it. AI is more useful for getting a working tool up and running. If it’s 80% of the way there, you’re doing well. You just need two things: consistent tweaking and a positive feedback loop.

A positive feedback loop is one where you feed more good information into your AI tool as it goes. Your AI tool might highlight some areas where that information is lacking, which helps you build your knowledge base. As that knowledge base becomes more accurate, your AI tool becomes more accurate.

The more you use the tool, the better it gets.

Once Joseph and the team had their “80%” solution, their primary challenge was refining AI’s output. One thing they noticed: if there was a gap in the knowledge base, AI tended to make up its answers.

So Joseph and the team created a new part of the tool. They’d have their internal team rate the AI’s efforts a thumbs-up or thumbs-down based on whether it had drafted an outstanding response or not.

To be clear, this tool isn’t life-changing. Our representatives are still providing real feedback, not answering 1,000 customer queries in microseconds.

But that shouldn’t be your expectation. AI is great at getting you 80% of the way there while your existing human team refines, improves, and personalizes.

Right now, we’re building “V2” of the tool. We have a few improvements we’re planning:

• Building an app that lives directly in Zendesk, rather than relying on third-party workflows
• Connecting to a Pinecone database to make the AI more of an “expert” so our reps don’t have to dig into the research themselves
• Adjusting verbosity, improving the quality of the drafts, and reducing AI “hallucinations” by more clearly limiting its knowledge base scope

Our V1 has already made it easier to rank urgent and non-urgent tickets. This has helped us build a “low-hanging fruit queue” for new reps who need more experience. They can choose from the easier, low-urgency tickets and build some customer support skills.

And maybe that doesn’t sound like a lot. But keep in mind that Joseph had been manually assigning tasks like this for several years. With a few days of AI coding under his belt, he automated a thankless task that used to eat into his time.

Understanding AI's strengths and weaknesses is the key here. 

“The way we see AI is that right now, it’s not at a point where we’d ever replace any support team members. But it can definitely help with the efficiency and the speed at which we can get some of these mundane tasks out of the support team’s hands.”

-Joseph Sterner, Head of Support

What We Learned About Creating Our Own AI Tools

As long as you realize AI isn’t as simple as pushing a button and expecting perfection, you can do a lot with it. You may not replace anyone yet, but you can lighten their loads and scale your operations. That’s particularly true if you adopt some of the lessons we learned:

Learn How to Craft and Refine Your AI Prompts

AI is often only as good as the quality of your prompts. In building these tools for Close, we realized each prompt needed to be clear with two main keys:

• Order: In using AI to develop tools, you have to be clear about the order of steps involved. So define your ideal workflow first. If you get the order wrong, the tool might not be as usable when you plug it in with apps like Zendesk.
• Specificity: Be crystal clear about the output you need. Sterner said he would explicitly tell AI when it needs to clean out “relic” features from the code that he no longer needs. The simpler your code is, the fewer opportunities there are for errors. 

Let Your Team Members Thrive Where Humans Need to Step In

Our AI tool is far from being a chatbot. It’s still too much of a people pleaser, sometimes creating “ghost” information to fill in gaps where it doesn’t have a good response.

But we could still overcome these challenges by embracing AI's strengths. We’d have AI sort through the content of support tickets and “triage” them by ranking them from most to least urgent. We’d have it create, but not complete, initial drafts for query replies. 

This didn’t replace our support team. It made them faster without lowering the quality of the support they offered. 

We also noticed other AI limitations. For instance, AI couldn’t log into customer accounts and manually help them. So our customer support-specific roles still need to be guided and supervised by humans dedicated to solving a customer’s problems.

Create a Positive Feedback Loop to Refine Your AI Solution

If you don’t create a positive feedback loop for your AI solution, it might disappoint you.

If you remember the case of the lawyers who submitted “bogus” case law that ChatGPT created, you know how important this is. Your AI needs to pull from authentic, accurate, proprietary information.

We found that in our earlier iterations, AI tried to please customers no matter the limits of our knowledge base. For example, if a customer asked about a feature like “bulk create tasks” that didn’t actually exist, AI would assume the feature existed and start offering support on how to use it.

Whoops.

With our feedback loop, each team member gives the AI drafts a rating so the AI can “learn” how well it’s doing. Over just a month, we’ve already noticed how much tighter it’s getting. We recommend a trust-but-verify approach: let humans evaluate its performance and train the AI over time.

Teach the AI Its Boundaries and When to Say “I Don’t Know”

We found that the more we limit our AI tools to our existing knowledge base, which itself can always expand and evolve, the better they perform. AI creates less “ghost” material if it knows what it’s not allowed to answer.

We’ve built a set of rules to tell our AI tool: hey, it’s okay if you tell customers when we don’t know something.

The hope is that this increased sophistication and accuracy will lead to a working chatbot one day. Even if AI can’t always provide the correct answer, it should at least admit that before it feeds the customers something that’s obviously incorrect.

Don’t “Rent” Your AI—Start Building It Yourself

If you consider AI a tool you can “rent” or a button you can push, we think you’re looking at it wrong.

Our initial plan was similar. We hoped for a prebuilt software solution that would make it easy to handle all of the features we listed above. But it was less expensive—and far easier—to learn how to build it ourselves. After all, AI could help with that, too.

Our takeaways?

• Let AI do the things AI is good at. You can outsource some things with AI. If you got into customer support, was it really your dream to dig through massive spreadsheets to glean customer insights? Or is that the exact kind of task that AI is good at, allowing you to function as the human whose passion is pleasing customers?
• Hand off the creative thinking to a human. Humans should review AI drafts, double-check AI responses, and manage the creative side of things. If you can pair a knowledgeable rep with the scaling of AI, you have the best of both worlds: a human touch, even when that human is supported by new AI functionality.
• Turn the AI into an assistant. Let AI tackle grunt work: analyzing incoming customer queries, organizing them, and prioritizing them by their urgency. This will give you an “immediate” win because it will take manual tasks off your plate. (Just ask Joseph.)

AI Isn’t Coming For Your Job—It’s Making Your Job Easier

AI isn’t replacing anyone at Close right now. 

It’s making the people who offer support more efficient, but we only use it to aid our team in improving the overall customer experience. It’s creating positive feedback loops where we can answer more customer queries or expand the quality of our knowledge database.

The key? Getting good at AI is more important than finding the “perfect” AI solution, which doesn’t exist.

Since using AI to create AI solutions, we’ve expanded our capacity to tweak, refine, and scale our processes. We expect that to pay dividends down the line. Our AI is getting better, acting more like a high-quality chatbot, and requiring less human intervention over time. 

When tools like that are better than the expensive, generic tools that your competition is using? That’s the sweet spot.

Start leveraging the power of AI by building within a CRM platform that makes your company better. Sign up for a free trial of Close today.