Your company data makes sense to your staff but not to your agent

The difficult part starts after the connection works

Imagine asking a sales agent which customers are likely to delay renewal. It can access the CRM, support tickets, and meeting notes. The integrations work, the search results are relevant, and the answer contains citations. Yet it misses the largest account at risk because the CRM uses a parent-company name while support records use a subsidiary.

Connecting data does not establish what the records mean together. Staff often resolve these differences through familiarity. They know that a trading name belongs to the same company, that a field has not been maintained since a migration, or that the latest agreement sits in an email attachment. An agent needs accessible evidence or an explicit rule for the same decisions.

The Data Agent Benchmark examines this broader problem. Its October 2026 revision includes 104 queries across 17 datasets and four database management systems, with the best tested agent achieving 57% first-attempt success. The benchmark covers fragmented and inconsistent information rather than treating the task as SQL generation over one clean database. The score belongs to that evaluation, not to enterprise data agents in general. [1]

For a project team, the question is where meaning gets lost between systems. A successful connector test only proves that information can move. It does not prove that an agent can use it to make the right business decision.

Search relevance and business authority are different

Suppose an agent finds three documents describing a customer’s price: an old proposal, a signed agreement, and a recent internal discussion. All three may be highly relevant to the question. Only some may authorize the price the business should charge.

The retrieval layer should expose enough context to distinguish those roles. Useful fields include the document’s status, effective date, owner, and relationship to the customer or transaction. A search result without this context can force the agent to guess which version matters.

A September 2026 preprint, Era by Eon, explores enterprise questions that depend on facts implied across records rather than stated directly. Its generated-company benchmark becomes substantially harder when agents must distinguish similar records and uncover hidden relationships. The study is small and synthetic, but it illustrates why finding a plausible document is not equivalent to establishing the answer. [2]

The system should define authority separately for each important fact. The billing system might be authoritative for an issued invoice, while a signed contract establishes negotiated terms. A support note may explain an exception without having the authority to create one.

When sources conflict, preserve the conflict. An agent should be able to say that an agreement and an account field disagree, identify which decision depends on that difference, and route it for resolution. Silently merging incompatible records produces a clean-looking answer that nobody can audit.

This is also a client responsibility. If the business cannot explain which source governs a decision, the implementation team cannot reliably encode it. Resolving that ambiguity belongs in the project.

Clean test data can hide the real task

Synthetic examples are convenient because they are easy to share and easy to grade. They become misleading when every customer has a unique name, every field is complete, and every policy document is current. In that environment, the agent never has to resolve the problems that staff encounter daily.

Salesforce’s CRMArena-Pro, published in May 2025, builds a broader evaluation around sales, service, and quoting tasks, including multi-turn interactions and confidentiality checks. It is relevant because it treats enterprise work as a combination of business skills rather than a single retrieval task. [3]

For a client-specific evaluation, sample the irregularities that affect decisions. Include duplicate contacts, forwarded messages, missing timestamps, renamed accounts, and records that reference another system. These should come from a documented understanding of the workflow, with expected outcomes reviewed by someone who knows the business.

Data protection can accidentally remove the same structure you need to test. Replacing every name with the same placeholder makes account matching trivial or impossible. Replacing an identifier differently in each file breaks the relationship between an order and its messages. Removing all dates may erase the reason a policy applies.

A protected dataset should preserve the relationships relevant to the task while removing information that the evaluation does not need. Consistent substitutes, controlled access, and checks for residual sensitive content need to be part of dataset preparation. Pseudonymized records may still be identifiable, so changing names alone should not be described as guaranteed anonymization.

The data team should verify the transformed dataset against the intended questions. Can a reviewer still determine the correct customer, applicable version, and final state? If those answers changed during preparation, the test is measuring the transformation as much as the agent.

Turn unclear meaning into explicit engineering work

Start with one business question and trace the evidence needed to answer it. For an overdue invoice, that might include the invoice record, payment status, dispute messages, and the responsible account owner. Write down how the records are joined and how recent each must be.

Then inspect actual failures before changing the model. Did the agent miss a relevant source? Did it select the wrong customer? Did it use the right document but apply the wrong date? Each diagnosis points to a different fix.

Some improvements belong outside the model. Stable account identifiers, visible document versions, and structured permission filters reduce ambiguity before the agent begins reasoning. Other improvements belong in the workflow, such as asking for an order number when two plausible records remain.

Measure these stages separately. A final-answer score is useful, but it can hide whether the problem was retrieval, record matching, interpretation, or action. A small set of labelled cases can tell the team where additional work will have the greatest effect.

The client should also decide what happens when the data cannot support an answer. An internal agent can present a qualified finding with links to conflicting evidence. A customer-facing agent may need to pause an action until a person resolves the discrepancy. The response should match the consequence of guessing.

Over time, these cases become a map of the organization’s information problems. That map can improve ordinary operations as well as AI performance. Fixing an unreliable customer identifier helps every system that uses it, rather than compensating for the same issue in every prompt.

Build data preparation around the work being tested

There is a product opportunity between connectors and agent evaluation: a tool that turns authorized business records into coherent, protected test environments. It would preserve joins, versions, and access rules, then show which business questions remain answerable after transformation.

Local collection and obfuscation, the area addressed by ActOut Collect Client, can contribute to that workflow. Their value should be assessed through both protection quality and preservation of the context needed for each task. Neither should be assumed from the presence of a local model.

Future tools could flag broken relationships, detect competing sources of authority, and generate reviewed test cases around those conflicts. Human engineers and coding agents would then work from the same evidence package instead of inventing clean examples. That would make it easier to improve the underlying system and determine whether a model change actually helped with the company’s real information.