Prove two consumers return the same number
Run a practical same-number test between two consumers (dashboard and agent) against one metric request.
The datahashi team3 min read
Governance talk is cheap until two consumers disagree. The useful test is practical: pick one certified metric, fix the dimensions and filters, call it from a dashboard-style consumer and from an agent-style consumer, and compare the numbers.
Why this test matters
Humans assume “we all use the warehouse” means “we all get the same answer.” Agents make the gap visible. A dashboard might use a BI measure. An agent might generate SQL from a schema. Both can look authoritative. Only a shared request shape makes equality something you can prove.
datahashi is built for that proof. Metrics come from a YAML model published as an immutable artifact. REST, console, and MCP share the same request shape. Humans and agents consume the same metric without hallucinated SQL. MCP tools list_catalog, describe_metric, and run_semantic_query do not accept or return SQL - so the agent path cannot quietly fork into creative joins.
Set up one metric request
Choose a metric that already matters operationally - something your dashboard already shows. Write down the exact request:
- Metric name (certified)
- Dimensions (if any)
- Filters (time window and any other predicates)
Freeze that trio. The test fails if either consumer freelances an extra filter or a different time zone assumption.
Describe the metric first so both consumers operate inside allowed dimensions and filters. If the dashboard was using a field the metric does not expose, you found a governance gap before you even compared numbers.
Consumer A: dashboard or console
From your dashboard config or the datahashi console, run the semantic query with the frozen metric, dimensions, and filters. Record the result and the request payload. Prefer the console or an API call that uses the same shape the agent will use - not a legacy BI expression that only looks similar in English.
Consumer B: agent via MCP
Point the agent at MCP. Have it list_catalog if needed, describe_metric for your chosen metric, then run_semantic_query with the same metric name, dimensions, and filters. Record the result and the request.
Do not ask the agent to “write SQL that matches the dashboard.” That reintroduces invention. The whole point is that neither consumer writes SQL in this path.
Compare and interpret
If the numbers match, you have evidence that two channels share one definition. Save the payloads. That pair is a regression fixture for later model changes.
If the numbers differ, inspect the requests before blaming the warehouse. Common causes:
- Different filters (inclusive vs exclusive dates, status sets)
- Different dimensions that change grain
- One consumer still on a non-semantic path
- One consumer pinned to an older published artifact
Because the model publishes as an immutable artifact, check that both callers hit the same publication. Metering helps here: you can see the requests that ran and align them side by side.
Keep the layer thin while you test
The warehouse holds transforms. datahashi serves the contract. It does not need a rollup for this proof to work. Rollups have shipped, and a p99 claim is still not part of the story. Do not postpone the test waiting for one.
Make it a habit
Run the two-consumer test when you certify a metric, when you change a published model, and when you onboard a new agent workflow. The ritual is short: freeze the triad, call twice, compare once. Teams that do this stop arguing about whose SQL was nicer and start arguing about whether the metric definition is right - which is the argument worth having.
Same metric. Same request shape. Two consumers. One number. That is the proof.
Product notes
An occasional email when we ship something or write something worth reading.