Eval harness: agent vs dashboard same-number tests

A confident chat transcript is not proof. Proof is an eval that runs the same metric request through an agent and through a dashboard (or console/REST) and asserts equal numbers. If those diverge, you do not have certified metrics for agents - you have two pipelines that sometimes agree.

datahashi makes that comparison honest: humans and agents share one published model version and the same Semantic Query shape. MCP tools never accept or return SQL, so the agent cannot “almost” match by inventing a nearby join.

What you are asserting

For each golden case, fix:

• Published model version
• Metric names, dimensions, filters, order, and limit
• Access mode and principal class (governed agent credential vs human console/REST)
• Expected numeric rows (or a tolerance policy for known float cases)

Then run the request two ways: agent tool calls only (list_catalog / describe_metric / run_semantic_query as needed, then run_semantic_query for the answer), and the identical Semantic Query in the console or over REST against a dashboard-backed definition of the same metric.

Assert row equality within tolerance. Fail if the agent answered from memory, paraphrased a prior turn, or produced a number without a tool result.

Minimal harness shape

Keep the harness boring and versioned in your repo:

1. A fixtures file of golden questions with structured expected queries and expected values.
2. An agent runner that forces tool use and records every MCP call.
3. A baseline runner that hits REST or the console API with the same payload and pinned version.
4. A comparator that checks metrics, dimensions, and numeric cells.
5. A gate in CI that fails the build on mismatch or on any attempt to call a SQL-shaped tool (there should be none to call).

Start with 5-10 cases drawn from metrics you already trust in a dashboard. Expand after those stay green across model publishes.

What to include in every case

• Pin version - Mid-flight YAML edits must not silently change either side.
• Refuse out-of-catalog - Include negative cases where the agent must say it cannot answer.
• Grain and filters - Ambiguous time grain and required metric filters are common drift sources; encode them explicitly.
• Principal - Run the agent with its governed credential, not a human explore-mode key.
• Metering check - Confirm both paths appear in metering so cost and scan volume stay visible.

Same number means same contract

Matching Excel exports from two handwritten SQL scripts is not the same bar. Those scripts can drift independently. With datahashi, both consumers ask by metric name. Joins are resolved by the model. Filters that define the metric cannot be dropped by the caller. Governance is the request shape, not a post-filter on agent SQL.

The thin layer stays thin: transforms live in the warehouse. The eval does not need a second metrics store. It needs one immutable published artifact and two doors into it.

Fail loudly, ship calmly

Treat same-number failures as release blockers for the agent, not as “prompt tuning” tickets. If the dashboard and agent disagree, fix the model, the credential, or the harness - do not paper over it with a longer system prompt.

Rollups have shipped, but they promise no latency numbers or p99 SLO. No benchmark exists. Do not bake latency SLOs into the eval. The trust bar you can claim today is certified answers: same metric, same request, same number.

Build the harness once. Run it on every model publish and every agent prompt change. That is how “agent vs dashboard” stops being a demo and becomes a regression suite.