Metric contracts: the same-number test
A metric contract is a shared definition you can test. If two consumers disagree on the number, the contract failed.
The datahashi team3 min read
Governance talk often stops at policies and ownership charts. Metric contracts are sharper. A contract says how a number is defined, at what grain it is valid, and which filters are allowed. The same-number test is the acceptance criteria: two independent consumers asking for the same metric in the same window should get the same result.
What belongs in the contract
A useful metric contract includes a clear name, a human description, the aggregation rule, the time grain, and the entity it measures. It should say what is in scope and what is not. “Active users” without a definition of active is not a contract. It is a slogan.
Contracts also need ownership. Someone must be allowed to change the definition, and everyone else must consume it. Without that, agents and analysts will fork the logic in prompts and notebooks.
The same-number test
Pick a metric, a date range, and a filter set. Run it from a dashboard. Run it from an API client. Run it from an agent tool call. Compare the outputs. If they diverge, you do not have a semantic layer. You have multiple informal paths to “close enough.”
This test is boring on purpose. It does not require fancy percentiles or warehouse magic. It requires that the definition is registered once and resolved the same way for every caller.
How agents interact with contracts
Agents should not reverse-engineer warehouse tables to invent metrics. They should discover contracts. With datahashi, list_catalog shows what is available. describe_metric returns the contract details an agent needs before querying. run_semantic_query executes against that contract with a structured request.
That flow keeps the agent from guessing joins. It also makes failures useful. If a filter is invalid for a metric, the agent can adjust. If the metric is missing, a human can add a contract instead of patching a prompt.
Thin layer, real metering
datahashi stays thin. Metering exists so teams can see which metrics are called and how often. That helps prioritize which contracts to harden first. Rollups have shipped for hot questions. A p99 latency target has not. Governance starts with definitions and consistent resolution, not with speed claims.
Change management without chaos
When a contract changes, versioning and communication matter. Publish the change. Keep the metric id stable when the meaning is intended to update in place. If the meaning is genuinely new, create a new metric rather than silently altering an old one. Agents that cache descriptions should refresh from describe_metric rather than trusting stale prompt text.
Practical first step
Choose three headline metrics that already appear in executive reports. Write contracts for them. Point the dashboard, the API, and one agent workflow at those contracts. Run the same-number test weekly. Expand only after those three stay aligned.
Governance that cannot pass a same-number test is theater. Metric contracts turn governance into something you can verify every day, including when the consumer is an agent.
Product notes
An occasional email when we ship something or write something worth reading.