Agent wrong numbers: five failure modes

This is a composite war story from several teams that put agents on metrics before the rails were ready. Names and products are anonymized. The numbers looked confident. The decisions were not. Here are five failure modes that kept showing up, and how a governed path with datahashi changes the ending.

1. The confident wrong metric

An agent was asked for “retention.” It found a table column named retention_rate, averaged it, and reported a crisp weekly figure in chat. Nobody noticed the column was a partial experiment flag, not the company retention metric. Leadership reused the number in a planning doc.

Mode: name collision without definition. Fix: force discovery through list_catalog and definition checks through describe_metric before any run_semantic_query. If the metric is not in the catalog, the agent should stop, not improvise from columns.

2. The infinite lookback

A support-facing agent answered “how is refund volume trending?” by scanning years of events because the prompt omitted a window. The query was slow, the sample was dominated by early noisy data, and the “trend” was a mirage. Cost alerts fired after the meeting, not before.

Mode: unbounded time. Fix: agent-safe defaults on run_semantic_query so missing ranges become short defaults or hard errors. Teach prompts to state windows, but do not rely on prompts alone.

3. The grain smear

An agent mixed daily active users with a monthly subscription count, divided them, and called the result “engagement intensity.” The arithmetic was valid. The semantics were nonsense. The chart still shipped in a slide because it was pretty.

Mode: incompatible grains composed in the agent. Fix: describe_metric returns grain and allowed joins; agents must keep comparisons inside compatible definitions or ask a human. Do not let free-form math invent metrics that never existed in the semantic layer.

4. The filter that was not a filter

A sales agent “segmented by enterprise” by matching a free-text account note field containing the word enterprise. Real enterprise accounts that used a different label disappeared. Pipeline coverage looked healthier than finance believed.

Mode: ad hoc predicates outside governed attributes. Fix: only allow filters describe_metric lists. Notes and tags stay out of metric filters unless they are promoted to governed dimensions with owners and tests.

5. The silent partial result

An agent hit a timeout, received a partial payload from a flaky wrapper, and summarized it as the full answer. The wrapper did not mark truncation. The summary sounded complete. A later human query disagreed by double digits.

Mode: incomplete results treated as complete. Fix: tools must fail closed on timeout, cap, or truncation. run_semantic_query should return an error or an explicit truncated flag the agent is instructed to surface, never a quiet subset.

What changed after the incidents

Teams that recovered did the same few things. They cut agents off from raw warehouse SQL for metrics. They standardized on list_catalog, describe_metric, and run_semantic_query. They logged every agent metric call with metric name, window, filters, and user identity. They reviewed wrong-number incidents as tool-design bugs, not only model bugs.

Wrong numbers from agents are rarely random. They cluster into these modes: wrong object, wrong window, wrong grain, wrong filter, incomplete result. Design the MCP path so each mode fails loudly. Then the war stories get shorter, and the numbers in chat match the numbers in the rest of the company.