An answer is useful when its meaning, source, permission and limits can be checked.
Consider a simple question: “What was net revenue last quarter?” The answer depends on which revenue definition is approved, which calendar is used, how returns are treated and which business units the person may access. A fluent sentence does not settle any of those choices.
Agree the meaning before the interface
For each priority question, document the metric definition, grain, time period, owner and authoritative source. Make disagreements visible. If finance and sales use different definitions, the assistant should not silently choose one. It should use an agreed definition or explain the ambiguity and ask for a choice.
Start with a small set of high-value questions and known-good answers. This is more useful than indexing every dataset at once. Google Cloud’s data governance guidance treats quality, access and lifecycle management as part of using data for analytics and AI.
Carry permissions into every answer
Authentication answers who the user is; authorisation answers what they may retrieve. Apply access rules when fetching documents or querying tables, not only when displaying the final response. Tag retrieved content with tenant, role and sensitivity where appropriate, and test with users who have different permissions. Do not rely on a prompt saying “ignore restricted data.”
OWASP’s RAG Security Cheat Sheet recommends access metadata on retrieved chunks and controls around retrieval pipelines. It also highlights that external or user-controlled content can carry malicious instructions. Treat retrieved content as evidence to evaluate, not as a new authority over the system.
Give every answer an evidence contract
A useful answer should show the source or report, the period and filters applied, when the underlying data was refreshed, and any important limitation. For a calculated metric, show the definition or a link to it. If the source is missing, stale or contradictory, say so. An answer that cannot be traced should not be presented as a verified business fact.
Answer: the approved net revenue value. Definition: finance metric version 3. Scope: Australian business units, Q3, excluding returns. Source: governed finance view. Freshness: last refresh time. Limit: two late records under review.
Test questions people actually ask
Build an evaluation set with straightforward queries, vague terms, conflicting definitions, missing data and permission edge cases. Compare numeric answers with trusted reports and have domain owners review explanations. Measure unsupported claims, wrong filters, access failures and useful abstentions separately. Re-test after source, schema, retrieval or model changes.
| Test case | Expected behaviour |
|---|---|
| Ambiguous metric | Ask which definition or state the chosen one |
| Restricted source | Refuse access without exposing its contents |
| Stale source | Show freshness and qualify the answer |
| No supporting record | Say the answer cannot be verified |
Build a governed first slice
Pick one domain, one owner, a small source set and a defined user group. Deliver a catalogue of approved questions, metric definitions, access rules, answer format and evaluation results. Expand only when the first slice is trustworthy and supportable.
Sources & further reading
External guidance informs this article; the examples and recommendations above are AI Dive’s interpretation for practical planning.