Anthropic published a post about how they run self-service data analytics internally with Claude. It gives two numbers: 95% of business analytics queries are automated by Claude, at roughly 95% accuracy overall.

We’ve spent the past year building the same thing. This post first walks through how they did it, then compares it with our implementation: where we converged, and what’s worth copying. At the end, some advice for teams about to start.

How Anthropic does it

It starts with a judgment: the accuracy of an analytics agent is a context and verification problem, not a SQL generation problem. They contrast coding and analytics directly: coding is an open-ended solution space where docs and tests naturally backstop hallucination; analytics usually has a single correct answer from a single correct source, with no deterministic way to prove it right.

From that judgment, they attribute most errors to three failure modes.

First, concept-to-entity ambiguity: the data model has hundreds of plausible candidates and the agent can’t pick the right one. Take counting active users: which actions count as “active”? Do fraudulent users count? What lookback window?

Second, staleness: sources, definitions, and schemas change every day; the agent’s knowledge quietly rots and starts returning normal-looking wrong answers.

Third, retrieval failure: the right information is in the data model, properly annotated, but the search space is too big and the agent can’t find it.

Below is Anthropic’s four-layer stack. Each layer targets one or more of the failure modes.

┌────────────────────────────────────────────────────────────────────────────┐
│  04 Validation        evals / online checks                     all three  │
├────────────────────────────────────────────────────────────────────────────┤
│  03 Skills            thin router + runbook           staleness retrieval  │
├────────────────────────────────────────────────────────────────────────────┤
│  02 Sources of truth  semantics > lineage > context             ambiguity  │
├────────────────────────────────────────────────────────────────────────────┤
│  01 Data foundations  canonical data + metadata       ambiguity staleness  │
└────────────────────────────────────────────────────────────────────────────┘

From the bottom up.

The data foundations layer is the warehouse itself: models, transforms, tests, tables, plus the metadata describing them. The idea is to make ambiguity disappear before the agent ever searches: if “revenue” resolves to one governed dataset instead of forty plausible candidates, there is no ambiguity left. The core move is consolidation: collapse near-duplicate tables into a small set of single sources of truth, aggressively deprecate the rest, then enforce it three ways, through tooling, CI, and mandate, so everyone actually uses them. The first gate against staleness also lives here: all data code sits in one repo, and a model change that would break a downstream dashboard gets blocked by CI on the spot. Finally, metadata is maintained as a first-class product: table descriptions, grain, and lineage written to the point that an agent can read them, making the warehouse as legible as a codebase.

If the foundations layer is the warehouse itself, the sources of truth layer is the reference material the agent consults when querying, turning “weekly actives” in a question into one specific entity in the data model. Four references, in descending order of trust: the semantic layer is highest; when a question maps to a defined metric, the agent calls a function and gets the same number as every other surface in the company, and skill instructions force the agent to try this path first. Lineage is next: for questions the semantic layer doesn’t cover, the agent reasons from lineage about which upstream table to aggregate from. Then the query corpus, thousands of historical SQL files, intuitively valuable; the experiment says otherwise, more in the negative results below. Lowest is business context: an agent that doesn’t understand your business answers what the user asked, not what they meant, so they piped in company docs, roadmaps, and the org structure, letting the agent understand which product “the Q2 launch” refers to.

The skills layer. If sources of truth are knowledge (what a metric means), a skill is the working procedure (what to consult in what order, and what to do with it). A skill is just a folder of markdown the agent reads on demand, but this is the highest-leverage layer: without skills, Claude scored no better than 21% on their eval set; with them, consistently above 95%. Skills come in pairs: the knowledge skill handles routing, roughly “try the semantic layer first; if there’s no coverage, this domain has thirty reference docs covering tables, columns, joins, and gotchas”, shrinking a million-field search space to a few dozen documents. That is their answer to retrieval failure. The runbook skill handles process, writing down how a senior analyst works: clarify the question, find the sources, run the query, hand the result to adversarial review.

Last, the validation layer. However well the three layers below are built, errors still slip through, and this layer’s job is to catch them. Three kinds of checks, by stage. Before launch: a few dozen Q&A pairs per domain as offline evals; a domain that doesn’t clear the bar (initially 90%) can’t open to business users. On change: a fixed eval set, one variable at a time, and every meaningful skill change ships with the before-and-after pass rate in the PR. After launch: adversarial review plus provenance labels, and a scheduled agent that sweeps business channels for corrections, drafts doc-fix PRs, and feeds the same corrections back into the eval set.

More valuable than the wins are three negative results:

One: using an LLM to auto-generate semantic layer definitions from raw tables and query logs. The output looked plausible and was net negative on evals, because the generated definitions encoded the very ambiguity they were trying to eliminate. Conclusion: Claude can draft the docs; a human has to make the call on definitions.

Two: giving the agent retrieval access to the full history of SQL (thousands of dashboard and notebook queries) moved accuracy by less than a point. They chased down why: for 80% of the misses, the answer was right there in the corpus, and the agent did read it. It just didn’t use it. The question-to-entity mapping wasn’t there, and no amount of precedent helps without it. That one experiment rewrote months of their roadmap.

Three: the cost of slack maintenance. Offline accuracy fell from 95% to 65% within a month. The fix was putting skill docs in the same repo as the data models; a PR that changes a model must change the corresponding doc, enforced by a hook. Today 90% of their data-model PRs carry a skill change.

Where we converged

Two systems, built with no knowledge of each other, grew nearly the same skeleton.

Same call on the bottleneck. In “Six-Layer Context Stack for Data Agents”, I wrote that the bottleneck of enterprise data Q&A is context quality, not model capability. Their phrasing: accuracy is a context and verification problem, not a code generation issue.

Traditional data engineering isn’t obsolete. They put this line right under the architecture diagram:

Standard data engineering practices like dimensional modeling are just as important as they ever were.

Dimensional modeling, data quality checks, the standard practices matter as much as ever. My version of this, in “Data Architecture in the Agent Era”, was that the warehouse isn’t dead, it moved down a layer: lakehouse, metadata, scheduling, the old infrastructure turned from the data team’s deliverable into the foundation agents consume.

Human-curated semantic layer. Ours has been expert-annotated from day one: source tables, filters, and cross-day rules for every metric, all written down as structure. They verified the other road for everyone: LLM-generated definitions are net negative.

Routing structure. Their knowledge skill is a thin router that narrows the search space to a few dozen docs per domain; ours is a two-tier manifest-plus-detail structure, a lightweight index loaded in full, details fetched on hit. Different names, same problem: retrieval failure. Better to shrink the space to dozens of candidates than to let the agent search a million fields.

ETL code as the most authoritative semantics. They rank lineage as the most trusted reference after the semantic layer; our L3 pulls each table’s producing SQL from the scheduler and reverse-engineers what the fields actually mean. Annotations go stale; the code that produces the data doesn’t lie.

Business context. They call it the layer they underrated the longest; our L4 domain knowledge layer holds exactly this: analysis frameworks, known patterns (weekend push volume dips are normal), definition traps (two DAU metrics overlap and can’t be added).

Same take on evals: an LLM judge checks semantic equivalence rather than verbatim SQL, “knowing it can’t answer” counts as a correct answer, and user corrections automatically become eval candidates.

Homework worth copying

Four things to learn.

First, maintenance treated as an engineering problem. 95 to 65 in one month is the number I remember best from the whole post. Our update path is change notifications, human confirmation, agent edits the docs, and the loop runs on people pushing it. They weld docs and data models into the same PR, enforced by a hook. One is a reminder; the other is discipline. Our context is rotting too; we just haven’t measured the rate.

Second, the discipline of regression. Our eval system can diff, change one version of context and compare runs, but we never institutionalized “every skill-change PR carries an eval delta.” That rule caught a subtle kind of regression for them: well-intentioned doc additions that make things worse. They ran three consecutive rounds of doc polishing, all net negative; the docs got longer, not better. Without the rule, you would never see it.

Third, adversarial review. Six points of accuracy for 32% more tokens and 72% more latency is a good trade in self-service analytics: a wrong answer walking into a business decision costs far more. They also tried a cheaper model as the reviewer; most of the accuracy gain disappeared and the latency barely improved. We don’t have this step. For high-stakes questions at least, it’s worth adding.

Fourth, provenance labels. Every answer ends with the source tier (semantic layer, governed table, or on-the-fly exploration), data freshness, and an owner. It doesn’t make the answer more correct; it tells the user how much to trust it. The most dangerous failure mode is the silent one: a wrong answer that looks plausible. Anthropic admits they have no full solution either, and provenance is one of the few mitigations available. We already do most of this: every answer ends with a definition note covering the metric definition, source tables, filters, and data partitions, and it will even state which dimensions can’t be split and why. What’s missing is the last step. The sources carry no trust tiers, and a governed metric should not look the same to the user as a result stitched together from exploration; and there is no owner field to tell a skeptical user who to ask. Not a from-scratch build, an upgrade of the existing note into tiered trust labels.

A few places are just different routes, no ranking implied. Their post covers analytics Q&A only; the data-to-execution path, in-company access control, and multi-engine support don’t come up, all covered in “Data Architecture in the Agent Era” above. Memory works differently too: their corrections go straight into the source docs; our memory recalls at runtime and distills high-frequency entries into source files later. Cleaner versus faster to respond; each has its trade-offs.

For teams about to build this

The must-dos: two teams took different routes to nearly the same list.

Consolidate definitions. Turn “revenue” from forty candidates into one governed answer, and most of the ambiguity problem disappears. I called it entropy reduction; they call it canonical datasets. Same thing.

Humans write the semantic layer definitions. The model can draft docs and flag gaps; the owner of a definition must be a person.

Evals before scale. A few dozen Q&A pairs per domain, a pass-rate gate per domain, no launch below the bar. Plenty of teams build a gorgeous analytics environment with no way of knowing their agent’s accuracy.

Design maintenance the same day you design the system. 95% on launch day, 65% a month later if you let it sit. Docs in the same repo and same PR as the data models is the most effective practice I’ve seen so far.

The traps, ones they hit or steered around: dumping historical SQL on the agent and hoping it learns (accuracy moved less than a point); LLM-generated metric definitions (net negative); piling on docs without limit (longer, worse); overbuilding infrastructure around current model weaknesses (one model upgrade writes off the investment, so decide first whether you can afford to wait).

What happens to data practitioners

There’s a line in the post that’s easy to slide past: the end user of your data model is no longer a data expert, but an agent working on behalf of people at every skill level. Which means correctness can no longer count on the user: an analyst who saw an odd number would go check; a business user who gets a plausible-looking number just uses it. The gatekeeping didn’t disappear; it moved from the consumption side to the production side, into the data team’s hands.

That explains where humans appear in their post. Read it end to end and every human touchpoint is a gate: deciding semantic layer definitions, reviewing agent-drafted doc PRs, labeling ground truth for evals, owning launch quality per domain, signing off on anything bound for leadership. In the execution steps, writing SQL, finding tables, running queries, humans barely appear. The translation work (business question into SQL) got automated; the gatekeeping work (definitions, ground truth, accountability) is appreciating.

Anthropic’s data science team put the freed-up time into causal modeling, forecasting, and machine learning. That’s the official blog’s version. The question it doesn’t answer: gatekeeping seats are far fewer than translation seats, so where do the rest go?