A closed support chat does not mean a solved customer problem

The dashboard can improve while the experience gets worse

Imagine a support team reviewing its first month with an AI agent. Fewer conversations reached staff, response times fell, and more chats were marked complete. Then someone notices customers opening new tickets about the same orders. The agent was ending conversations efficiently, but the underlying problems were still there.

This is an illustrative situation, not a claim about a particular vendor. It shows why support automation needs a definition of resolution that survives beyond the chat window. A customer may stop replying because the issue is fixed, because they are busy, or because they have given up. Those outcomes should not be treated as equivalent.

A starting point is to connect the conversation to an observable result. A cancellation should be reflected in the account. A replacement should have an order record. A troubleshooting case should include evidence that the service works again, or a clear handover when that cannot be established.

Some enquiries genuinely end with an explanation. In those cases, the team can assess whether the answer was correct, relevant, and sufficient, then examine repeat contact. The key is to choose the evidence appropriate to the job instead of applying the same closure metric to every conversation.

That decision belongs jointly to the client and the solution provider. If procurement rewards containment while the customer-success team cares about lasting resolution, the agent will be pulled toward incompatible goals.

Support often requires the customer to do something

Many support workflows are cooperative. An agent can reset an account setting, but the customer may need to restart a device, confirm an address, or check whether a payment appeared. The system has to coordinate both sides.

Sierra’s tau2-bench, introduced in June 2025, tests this through an environment where both the user and the agent can act. Its telecom scenarios make user guidance and coordination part of the problem, rather than evaluating only the agent’s own tool calls. This is a simulated benchmark, but the design exposes what tests miss when they ignore the customer's actions. [1]

For example, telling a customer to restart a router is not evidence that they restarted it. If the agent immediately runs the next diagnostic and interprets the unchanged result as a new fault, it may send the conversation down the wrong path. The product needs to represent what is waiting on the customer.

Instructions should also match what the person can reasonably do. A customer on a mobile connection may be unable to open a second browser window. Someone using a shared account may not have administrator access. An agent that repeats the same steps without adjusting can appear helpful in a transcript while making no progress.

Test this with users who misunderstand a step, perform only part of it, or report an unexpected result. The agent should recover without assuming dishonesty or asking them to repeat the entire journey. These cases matter because the support experience is partly determined by how well the system handles incomplete cooperation.

Policy compliance needs to follow the whole journey

A correct answer can still lead to an incorrect process. Suppose a replacement requires checking warranty status and confirming that a previous replacement has not already been sent. The agent may quote the warranty accurately while skipping the duplicate check.

JourneyBench, described in a January 2026 preprint, evaluates support agents against multi-step business policies. Across its tested conversations, an agent design with explicit policy control improved adherence compared with a static-prompt design. That finding does not establish the best architecture for every support desk, but it gives teams a reason to test how required steps are represented. [2]

For a real deployment, essential prerequisites should be visible to the execution layer. The tool that issues a replacement can require a verified order, an eligibility result, and a check for an existing shipment. The language model can still decide how to explain the process, while the application enforces the required evidence.

Exceptions need their own route. If staff sometimes waive a fee for a documented service failure, the system needs to know who can approve that decision and what information they require. A policy that says “use judgment” without describing the available authority leaves both developers and customers exposed to inconsistent outcomes.

The brand experience also depends on how the agent explains a limit. A refusal that accurately quotes a rule may still fail to tell the customer what can happen next. Evaluate whether the response offers a valid alternative, identifies the missing information, or transfers the case to someone who can make the decision.

The aim is to make the process dependable without turning the conversation back into a rigid phone menu.

Measure customer value alongside automation

There is credible evidence that disciplined development can improve both experience and self-service. A June 2026 paper from Nubank describes an evaluation-driven approach across several customer-support deployments. In its card-delivery deployment, the authors report a 37-percentage-point gain in AI transactional Net Promoter Score and a 29-point gain in self-service over earlier agent variants. These are company-reported results for that setting, not a forecast for other businesses. [3]

The example matters because customer experience and automation were measured together. A team can pursue fewer handovers while checking that customers are actually better served.

A support scorecard should follow the same logic. Track verified resolution, repeat contact within a suitable window, policy errors, and time spent by staff repairing or completing the work. Review these by issue type. A strong result on order-status questions can hide weak performance on missing deliveries.

Include the quality of escalation. Some cases should reach a person, and a good agent can still save time by collecting the right evidence first. Measure whether staff can continue from the handover, whether the customer repeats information, and whether the transfer reaches the right queue.

When running a controlled rollout, keep the comparison fair. If the AI receives simple enquiries while people receive complex disputes, their raw resolution rates answer different questions. Compare similar case types and account for changes in traffic or policy.

Support leaders should also review a sample of apparently successful conversations. Complaints and low ratings expose some failures, but silent abandonment can be harder to see. Look for unsupported completion claims and repeated instructions even when no customer explicitly complained.

The commercial agreement should reflect this broader view. A provider paid only for preventing human contact has little incentive to make escalation easy. A better acceptance process rewards successful handling, including appropriate transfers, and makes cleanup work visible.

Give support teams a place to rehearse difficult cases

The tooling opportunity is a support rehearsal environment that joins conversation, policy, and business state. It should let teams replay a case with a delayed shipment, an uncertain customer, and a failed lookup, then inspect whether the final outcome matches the promise made in chat.

Human specialists would define acceptable resolutions and realistic behaviour. Coding agents could build or modify the workflow, run those scenarios, and receive evidence about the exact missing step. That is more actionable than a general score for helpfulness.

A future product could connect repeat contacts to their earlier conversations and suggest new regression cases for review. Another could test handovers by measuring how much work a specialist still needs to do after receiving them. Both would focus development on the customer’s unfinished job.

The goal is a support agent that earns its automation rate through resolved work. Better testing tools should make that relationship visible before a team celebrates a quieter queue.