Audit trails for agent answers

When an agent quotes a number in Slack, a ticket, or an exec brief, compliance eventually asks: who asked, which metric, which filters, which model version, and was that principal allowed to see those rows? If the only record is the chat transcript, you cannot replay the answer.

datahashi treats every agent answer as the result of a metric request. Audit trails start from that request, not from prose the model generated afterward.

What to record (minimum viable audit)

For each MCP or REST call that touches data, retain:

• Principal (user or service) and credential id
• Access mode (governed vs explore) fixed on the credential
• Published model version pinned or resolved at request time
• Tool name (list_catalog, describe_metric, or run_semantic_query)
• Structured Semantic Query payload: metrics, dimensions, filters, order, limit, view
• Decision: success, forbidden, not_found, over_budget, or validation error
• Metering: bytes scanned and row count against that principal
• Timestamps and request id

You do not need to store warehouse SQL in the agent-facing log to make this useful - and agents never receive SQL from the MCP tools anyway. The structured request is the durable contract. If you also retain compiled SQL in an admin-only engine log for incident response, keep that separate from what the agent can see.

Tie chat answers to request ids

Agents paraphrase. Your audit trail should not. Pattern:

1. Every run_semantic_query response includes a request id (or your gateway attaches one).
2. The agent must cite that request id when it states a number in the product UI or outbound message.
3. Downstream stores (Slack archives, ticket systems) keep the citation next to the claim.

Compliance then replays: look up request id → see principal, metric names, filters, version, metering. If the agent answered without a citation, treat it as ungoverened content and reject it in review.

Governance is the request shape

Audit is weaker if agents can run arbitrary SQL beside the semantic path. With datahashi, MCP exposes six tools: list_catalog, describe_metric, run_semantic_query, drill_down, refine_semantic_query, and list_dimension_values. None accept or return SQL. Field grants and row-level rules apply on the server; there is no request shape that skips them. A “forbidden” is itself an auditable event - proof the agent asked for something outside policy.

Metering already exists as a usage signal per principal. Use the same identity keys in your audit index so cost, access, and answer replay share one notion of “who.”

How to implement without boiling the ocean

1. Export or stream query logs from the control plane into your SIEM or warehouse audit table, keyed by request id and principal.
2. Require citations in the agent system prompt and in any UI that posts numbers.
3. Pin model versions in production agent configs so replay is stable; a new publish is a new version, not an in-place mutate.
4. Retain catalog describe calls too - knowing which definitions an agent inspected helps explain why it chose a metric.
5. Redact secrets, never log raw warehouse credentials; OAuth tokens should appear as metadata only.

Do not invent customer-specific retention periods here - align retention with your existing compliance policy.

Replay drill

Pick a past agent message that contains a number. Resolve its request id. Re-run the same Semantic Query against the same published version in the console (or from a privileged replay tool). Confirm the number matches and that the principal’s grants still explain access. If you cannot complete that drill, you do not yet have an audit trail - you have chat logs.

Ship agents on certified metrics only when you can replay who asked what, on which version, under which policy.