Why your agent works in the demo and struggles on Monday morning

The demo quietly removes half the job

Imagine an operations agent that handles an overdue delivery. During the sales call, someone gives it an order number, it finds the shipment, and it drafts a helpful update. Everyone can see the value. On Monday morning, a customer asks about “the second order,” the shipment record is stale, and the person who can approve a replacement is away. The agent is now doing a different job from the one everyone watched.

That difference belongs in the project scope. A demonstration establishes that the workflow can succeed. A production system has to choose the right path when information is missing, several paths look plausible, and an action can leave the business in an awkward state. The model is only one part of that system.

LangChain’s State of Agent Engineering report puts quality at the top of respondents’ production barriers. Its June 2026 publication draws on 1,340 responses collected in late 2025, with a strong technology-sector presence. It is a practitioner survey, not a census of businesses, but it highlights a recurring concern: getting an agent deployed and getting it to behave consistently are separate achievements. [1]

For a consultant, that changes what a proposal should promise. “Connect the CRM and automate follow-up” leaves the difficult decisions unspecified. A clearer scope explains which records qualify, what evidence permits a message, which exceptions require a person, and how the client will recognize a completed task.

The environment is part of the capability

Consider a booking workflow with several dependencies. The agent needs the right account, current availability, valid permissions, and a successful write to the booking system. A convincing response cannot compensate for a failure in any of those steps. Testing the language alone leaves the actual service largely untested.

Anthropic demonstrated a narrower version of this problem in February 2026. In its Terminal-Bench 2.0 experiments, changing infrastructure resources produced a six-percentage-point performance gap while the model, task set, and harness stayed fixed. This was a coding benchmark, not a customer-service trial. The result shows that runtime conditions can change measured agent performance enough to affect conclusions about the model. [2]

The same reasoning suggests what to examine in a business deployment. Does the agent run under the same permissions used in testing? Can the database return partial results? Are requests slower during business hours? Does the integration expose enough information to distinguish an empty account from a failed lookup? These are testable properties of the application.

Teams should also resist translating a broad capability score into a service guarantee. METR’s time-horizon measurements describe task difficulty using human completion time at a specified success probability. METR explicitly explains that this is not a measure of how long an agent can operate autonomously, and its tasks are cleaner than much real work. A result on that suite does not establish that an agent can run a client’s operations unattended. [3]

Define the operating conditions before polishing the pitch

A good first deliverable is a short description of the conditions under which the agent is expected to work. For a lead-response agent, that might include inbound enquiries in two languages, contacts with an identifiable company, and product questions covered by an approved knowledge base. Negotiated pricing and contract changes might stay with the sales team.

This gives both sides something concrete to discuss. The client can point out that a large share of enquiries arrives through forwarded email, or that the same buyer appears under several addresses. The developer can then include those cases before the workflow becomes a fixed-price commitment.

Examples should come from actual work, with permission and appropriate protection of private information. Ask staff to walk through recent difficult cases and show what they checked outside the main application. A warehouse employee might rely on a carrier portal. A support specialist might know that one customer has a contractual exception. If the agent cannot access that evidence, the team needs a different workflow.

The test set should include ordinary cases as well as awkward ones. An evaluation made entirely of disasters can push an agent to escalate everything. An evaluation made entirely of clean examples can encourage reckless confidence. Keep separate views of everyday performance and important exceptions so neither disappears inside an average.

The client also needs to agree on what counts as success. A drafted message, a sent message, and a resolved delivery problem are different outcomes. Scope and pricing become more defensible when the acceptance criteria name the outcome the business is actually buying.

Launch a small service with a visible recovery path

A sensible rollout starts with a bounded queue of work. The agent can prepare recommendations while people continue making the final change, or handle a subset where mistakes are easy to reverse. The point is to collect evidence under real conditions without giving every request the same level of autonomy.

Record what happened after the conversation ended. Did the replacement order exist? Did the customer return with the same problem? Did someone correct the CRM record? A team that reviews only transcripts may praise the response while missing the cleanup.

Failures should be assigned to the part of the system that needs work. A wrong account match calls for different changes from a misunderstood request. A timeout calls for different changes from an unsupported policy exception. Adding another sentence to the prompt is a poor default when the underlying issue is an unavailable field.

The handover should name an owner for unresolved work and specify what that owner receives. “Escalate to a human” is incomplete if the human sees an empty ticket. Include the customer’s request, verified facts, attempted actions, and the remaining decision. Then measure whether this handover saves work or creates another investigation.

A project becomes easier to expand when its limitations are visible. The next deployment decision can be based on a known category of cases, with a known recovery process, rather than another successful demonstration.

Give builders a business environment they can actually test

This creates room for development tools that reproduce the conditions around an agent. Such a workbench would collect approved examples, preserve relationships between records, and replay cases against a versioned copy of the relevant business systems. Engineers and coding agents could inspect the same failed run and propose changes against the same acceptance criteria.

ActOut’s focus on simulated business environments fits this direction, without making simulation a substitute for live validation. The harder product opportunity is measuring where the simulation differs from reality: missing exception types, unrealistic user behaviour, or tools that respond more cleanly than their production counterparts.

A future readiness report could show which workflows have been exercised, which assumptions remain untested, and what happened when dependencies failed. That would give consultants and clients a handover they can inspect and challenge. It would also make the next investment decision clearer: improve the model, repair an integration, collect better examples, or narrow the job.