Agents Don't Lack Capability. They Lack What You Never Wrote Down.
Why enterprise data agents guess instead of asking: the definitional, boundary and exemplar context you never wrote down, how to sequence it, and how to test it.
Introduction
The agent in your production environment is not less capable than the one in the demo. It is working from a thinner written record of what your organization means and permits. Capability comes from the model; precision and permission come from context you either wrote down or did not. When a data agent returns a revenue figure that is internally consistent and wrong, or drops a table nobody meant to drop, the missing input is usually a definition, a boundary, or an example that existed only in someone's head.
This is a research-informed argument, not a report of deployments we have run. Its premises come from published work on how agents behave under underspecified instructions and from vendor and industry guidance on context engineering. The transfer to data and analytics agents is reasoning from those premises, and it is labelled as such where the evidence does not reach.
The path: why a demo and a production run diverge even though the model is identical, why the failure mode is guessing rather than refusal, what artifacts a data agent actually needs written down, how the agent consumes them, the order to write them in, how to test whether they are good enough, and what the absence costs.
The Demo Passes Because the Demo Is Specified
A demo is a fully specified task. Someone chose the question, the dataset, the date range, and the acceptable answer, often without noticing they had done it. Production is the same task with those choices removed and the agent left to supply them.
The clearest measurement of that gap comes from UnderSpecBench, a benchmark for action-boundary violations in coding agents on DevOps tasks (arXiv:2607.02294). The design isolates specification from difficulty: each of 69 task families keeps the same environment and the same ground-truth safe action while the instruction is varied along three axes — intent clarity, target certainty, and blast radius — producing 2,208 prompt variants. Across five agent×model configurations built on OpenCode, Claude Code, and Codex, 55.8–67.8% of runs violated at least one boundary. The authors' summary of the result is the sentence this article is built on: underspecification "does not mainly make agents fail; it makes them guess."
Scope matters here. This is DevOps instruction-following with shell commands, repositories, and operational APIs. It is not an analytics benchmark, and nothing in it measures a text-to-SQL agent or a semantic layer. The claim that the same mechanism operates in data work is an argument this piece makes from the mechanism, not a finding the study reports.
A second premise comes from AlphaEval, which argues that production requirements contain implicit constraints and undeclared domain expertise that retrospectively curated benchmarks strip out. Its authors describe tasks where agents optimize explicitly stated objectives while violating implicit constraints, and where a majority of responses fabricate a "best effort" answer rather than declaring a problem infeasible. Benchmarks built after the fact tend to contain the specification the production request omitted.
Put together: the demo passes because the demo is specified. The agent's competence did not change between the demo and the incident review. The specification did.
Guessing Is the Failure Mode, Not Refusal
The intuition most teams carry is that an uncertain agent will stop and ask. The benchmark found the opposite pattern. When the target is unclear, the agent picks a plausible object from local context and acts on it. The run completes. The transcript reads like progress.
The same study is specific about which cues help. Target underspecification sharply degrades action quality, while blast-radius cues barely reduce action propensity. In plain terms: telling an agent that an action is dangerous does not stop it from taking the action; telling it exactly which object the action applies to changes what it does. Warnings about consequences are not a substitute for naming the thing.
Consider an illustrative inference, not an observed case: an agent asked to "clean up the stale tables" in a warehouse. Nothing in that instruction names a schema, a definition of stale, or an exclusion list. The agent will select tables by last-modified metadata, decide what counts as stale, and drop them. Every step is defensible in isolation and the result is a deletion nobody authorized. The same shape applies to "pull last quarter's revenue": the agent chooses a revenue definition, a table, a fiscal calendar, and a date range, then returns a number that reconciles to itself.
This is why completion-centric evaluation overstates safe autonomy, as the benchmark's authors put it. A metric that records whether the task finished cannot distinguish a correct action from a wrong one taken confidently. For a data team, the practical consequence is that a passing demo and a passing eval suite can both be silent about the failure that matters.
What "Written Down" Actually Means for a Data Agent
For a data or analytics agent, the written record falls into three kinds of context, and they fail differently.
Definitional context — what things mean. Metric and entity definitions: what revenue counts, whether it is gross or net, recognized or booked, which currency conversion date applies, and at what grain the number is valid. The same for active customer, churn, and every entity the agent will join across. A definition without a calculation and a grain is a label, not a specification.
The machine-readable carrier for these is a semantic layer or metadata registry. ISO/IEC 11179-1:2023 is useful here as a framework for describing data elements — it treats metadata as descriptions of data and provides basic attributes for defining and specifying the contents of data dictionaries. It is a structural reference, not a compliance requirement, and adopting its vocabulary does not by itself tell an agent what your finance team means by margin.
The industry view is converging on the same point. The dbt 2026 State of Analytics Engineering report frames context for analytics as metadata — the descriptions and names that give meaning to data for humans and AI, along with documented standards for how work is done. That is a survey's framing, attributed to it, and it corroborates the category rather than proving any specific artifact works.
Boundary context — what the agent may do. Which tables and schemas are in scope, which rows are excluded by policy, which environments the agent may write to, and what it may never touch regardless of instruction. This layer is usually absent from a catalog because catalogs describe existence, not permission. It is also the layer whose absence produces the least reversible errors.
Exemplar context — what good looks like. Canonical question–answer pairs with the correct query attached. These do double duty: they show the agent the expected shape of an answer, and they become the seed of an evaluation set. A definition states a rule; an example demonstrates the rule applied to a real question, which is often what the agent needs to resolve an ambiguity the definition left open.
A data catalog typically covers part of the first category and almost none of the second and third. That is not a criticism of catalogs; it is a statement about what an agent needs that a catalog was never designed to carry.
How the Agent Consumes What You Wrote
Content is not enough. The same definition, written badly, can make an agent worse. Anthropic's context engineering guidance describes system-level instructions at the "right altitude," a zone between two failure modes: hardcoded, brittle logic that breaks on the first unanticipated case, and vague guidance that assumes shared context the model does not have. A metric definition written as a paragraph of business prose sits at the second extreme; a definition written as a fixed SQL string with no stated grain sits at the first.
Tool design carries boundary context as well. Writing tools for agents recommends namespacing related tools under common prefixes so an agent can select the right capability at the right time, and notes that well-chosen tool boundaries reduce the number of tools and descriptions loaded into context. A tool named for the operation and scoped to the permitted objects enforces part of the boundary document mechanically rather than relying on the agent to remember it.
Examples are the highest-leverage context per token. The same guidance strongly advises few-shot examples but warns against stuffing a laundry list of edge cases into a prompt; the recommendation is a curated set of diverse, canonical examples that portray expected behavior.
Finally, the constraint that makes curation mandatory rather than stylistic: context rot. As the number of tokens in the context window grows, a model's ability to accurately recall information from that context decreases, and the guidance treats context as a finite resource with diminishing marginal returns. The stated design goal is the smallest possible set of high-signal tokens that maximizes the likelihood of the desired outcome.
The practical form is progressive disclosure: some context loaded up front for speed, the rest retrieved just in time as the agent explores. The guidance points to Claude Code as an example of the hybrid — a file dropped into context up front, with search primitives used to retrieve files on demand. For a data agent, the analogue is a compact up-front block of core definitions and hard boundaries, with the long tail of table documentation retrieved when the agent reaches for it. More documentation is not automatically better documentation; past a point it degrades recall of the parts that matter.
Write These First, in This Order
The following order is a recommendation for a team starting from little written context. It is judgment about sequencing, not a standard anyone discovered, and a team with an established semantic layer will start elsewhere.
Metric and entity definitions for the questions the agent will actually be asked. Not the whole warehouse — the handful of questions that arrive most often. This unblocks the most common runs and forces the argument about what the numbers mean while the scope is still small.
The boundary document. Permitted objects, forbidden actions, escalation triggers. Write this second even though it is the cheapest artifact, because its absence produces the most damaging guesses. A wrong definition wastes analysis time; a wrong deletion does not come back. If your team can only write one thing this quarter, write this one.
Canonical question–answer pairs with the correct query. These serve as few-shot examples in production and as the first test cases, so the effort is spent once and used twice.
The evaluation set, drawn from real failures. Once the agent is running, its mistakes are the highest-value test cases available, because they are drawn from actual usage rather than imagined edge cases.
The broader semantic layer and metadata registry. The long-horizon investment. It scales coverage, but it is also the step most likely to stall, which is why it comes after the artifacts that produce immediate safety and signal.
The logic runs from unblocking, to protecting, to scaling: definitions first because they make the most common questions answerable and force the scope argument while it is still small; the boundary document second because it is cheap to write and must be in place before the agent performs an action that does not reverse; then examples and evals, where missing context costs wasted effort; and the registry last, where its absence costs coverage pace. Reversing the order — starting with a registry program before any boundary is written — is the common failure, because the registry is the most expensive artifact and the least urgent one.
Testing Whether the Context Is Good Enough
"Is our documentation sufficient?" becomes measurable once you stop asking it about the documentation and start asking it about the agent's behavior on tasks you wrote down.
Anthropic's eval guidance gives a workable starting size: 20–50 simple tasks drawn from real failures is a legitimate beginning, because early changes to an agent system have large, noticeable effects and small samples suffice. Two conditions make the set valid. Each task must be unambiguous with a reference solution, because everything the grader checks should be clear from the task description — an agent that fails for reasons of spec ambiguity is a broken task, not an incapable agent. And a 0% pass rate across many trials is usually a signal of a broken task rather than a weak model, which is a reason to re-read the specification before concluding anything about capability.
OpenAI's evaluation best practices add the operational point: evaluation is continuous, and the test data should mix production data, expert-authored correct answers, and historical logs. A one-time documentation audit is not an eval; it is a snapshot that goes stale as the business changes.
The diagnostic that matters most for this argument is failure classification. When an agent gets something wrong, sort the failure into three buckets: capability (the model could not do it), boundary (the model did something it was not permitted to do), and specification (the model did something plausible because the instruction or the written context did not determine the answer). Only the first is a model problem. The second is a permissions and tooling problem. The third is the documentation problem this article is about, and it is the bucket most teams never separate out, which is why they keep buying capability to fix a specification gap.
The closing test ties back to the thesis: if two competent humans reading only your written context would produce the same answer to a question, the context is probably sufficient for that question. If they would diverge, the agent will too — and it will not tell you it guessed.
What Undocumented Context Costs
Undocumented context does not disappear. It converts into costs that land on different budgets, which is why it is easy to miss in a platform comparison.
It becomes repeated human review: every agent output needs a knowledgeable person to check it, because the reviewer cannot rely on the agent having applied a definition that was never written. It becomes rework when a wrong definition propagates into a report and downstream decisions inherit the error. It becomes slower onboarding, because the knowledge lives in people rather than artifacts and each new analyst or engineer rebuilds it by asking around. And it becomes an evaluation that cannot be built, because success criteria were never written down to test against.
The dbt Labs analysis of the 2025 State of Analytics Engineering Report offers a relevant data point: 57% of analytics and data professionals said they spent most of their time maintaining or organizing datasets — the same level as the prior year, despite 70% using AI to help write code and documentation. Read carefully, this is a correlation the survey reports, not proof that documentation investment would reduce the maintenance burden. It is consistent with the argument that tooling alone has not removed the work, and it does not establish causation in either direction.
For procurement and finance readers, the implication is a change in what to ask. When comparing agent platforms, the question is not only what the underlying model scores on a benchmark. It is what context artifacts the platform can ingest, version, and enforce — whether metric definitions and boundary rules can be loaded as first-class inputs, whether permissions are enforced at the tool layer rather than requested in a prompt, and whether the platform can run your evaluation set against your definitions. A platform that cannot consume your written context leaves the specification work to be re-done in prompts on every deployment.
The Strongest Counterargument: Some Context Cannot Be Written Down
The strongest objection is not that documentation is unnecessary. It is that much operational knowledge is tacit, judgment-based, and changes faster than any document, so a documentation-first approach produces stale artifacts that mislead an agent worse than no documentation at all.
The concession is real. Stale context is a failure mode, and context rot means volume is not virtue — a large, outdated registry can bury the two definitions that are still correct. Any team that has maintained a data dictionary through a reorg knows the shape of this problem.
The response is that the choice is not "write everything" versus "write nothing." It is to write the high-consequence definitions and boundaries, keep them current, and give the agent retrieval tools for the volatile rest. That is the hybrid pattern the context engineering guidance describes: some context retrieved up front for speed, further autonomous exploration at the agent's discretion. Stable facts — what revenue means, which schema is off limits — belong in the up-front block. Volatile facts — which table currently holds the freshest copy — are better retrieved at runtime than frozen in a document.
The benchmark's finding on target underspecification supports this split. What degraded action quality was uncertainty about which object the action targeted. Naming the object precisely is exactly the kind of fact that can be written down and kept current, and it is the fact whose absence causes the wrong-target failures the study measured.
The honest weakness: there is no published evidence that documentation investment reduces agent boundary violations at a measured rate. The case here rests on the mechanism the research identifies — underspecification produces guessing, and naming targets reduces it — not on a controlled trial of documentation programs. A team should treat the first eval set as the beginning of that evidence for its own environment, not as confirmation of a general result.
Conclusion
The agent is not the variable you are missing. The written record of what your organization means and permits is. A stronger model may guess better, but it will still guess, and the guess will still arrive with the confidence of a completed task.
The shift this calls for is in how teams evaluate and procure agents: from capability benchmarks alone to context-readiness audits, and from "can it do the task" to "have we told it what the task is." That means asking vendors what artifacts their platform can ingest and enforce, and asking internally which definitions and boundaries exist only in people's heads.
The first move is small enough to start this week. Pick the three questions your agent gets asked most often. Write down the definitions and boundaries behind each one — the calculation, the grain, the permitted tables, the forbidden actions. Run those three as your first eval set and classify every failure as capability, boundary, or specification. If you have already written one of these artifacts, the useful question for the rest of us is which one you wrote first and what it changed.
Frequently Asked Questions
But doesn't a more capable model solve the underspecification problem?
Capability and specification are different variables. The benchmark held the environment and the ground-truth safe action constant and varied only the instruction, and boundary violations persisted across five agent×model configurations at 55.8–67.8% of runs. A stronger model may guess better, but it still guesses. The fix sits at the harness and system layer, which is where written context lives.
Isn't this just prompt engineering with a new name?
Prompt engineering optimizes instructions for a task. Context engineering curates the whole state the agent operates in — system instructions, tools, retrieved data, message history, and the written organizational context — across many turns. The artifacts described here (definitions, boundaries, examples, evals) outlive any single prompt and are maintained like code, which is a different lifecycle from a prompt edit.
We already have a data catalog. Isn't that the written context?
A catalog is necessary but not sufficient. It typically describes what exists; an agent also needs what things mean, which objects it may act on, and examples of correct output. A metadata registry following a framework like ISO/IEC 11179 gives structure to descriptions of data elements, but the boundary and exemplar layers usually have to be authored separately, and the catalog rarely carries permission scope.
How do we know when we have written enough?
Use the operational test rather than a volume target. Enough is when your eval set — drawn from real failures — passes at a rate you would accept in production, and when a new team member reading only the written context can answer the agent's common questions correctly. More documentation beyond that point can degrade performance through context rot.
Does this apply outside data and analytics agents?
The strongest direct evidence comes from DevOps coding agents, where the benchmark measured boundary violations under underspecified instructions. The mechanism — underspecification produces guessing — is general to agents that act on systems, but the specific artifacts differ by domain. The data and analytics artifact list here is reasoning from that mechanism, not a tested result.