Why We Still Need Data Agents
My earlier article, “Data Agent or Data Skills?”, discussed the organizational reasons: analysis needs to be independent, and business goals should not shape the interpretation. This article adds the technical reasons.
In “Can Coding Agents Do Data Mining?”, I wrote about two projects: subscriber churn diagnosis and analysis of a children’s content business. The agent used capabilities we had packaged for data retrieval, user profiles, behavior analysis, and experiment analysis.
But if we simply connect a general-purpose agent or coding agent to a database and let it read documents, write SQL, and run Python, can it consistently handle complex enterprise analysis?
Some errors will not cause code to fail. They can even pass tests the agent writes itself. Yet the agent may have selected the wrong population or presented an unsupported explanation as a conclusion, while still delivering a report that looks complete.
The Analysis Can Be Wrong Even When the Code Runs
A general-purpose agent can plan tasks, call tools, and manage context. A coding agent has a working environment built around code repositories, builds, tests, and debugging. Both can read documents, call data tools, and perform analysis.
When fixing a bug, it can change the code, run tests, and use the failures to keep working. The same feature can have different implementations, and tests and runtime results help check whether they meet the requirements. Software requirements can be ambiguous, and passing tests does not mean there are no bugs. But tools can directly expose at least some errors.
Analysis can use different methods, but once the population, metric definition, and time range have been established, the calculation has to respect them. We cannot switch to another definition just because it is easier to calculate. As for why something changed, there may be several explanations that need further investigation.
The difficulty is that taking the wrong analytical direction may not produce an error either.
SQL can execute and the numbers can reconcile, even though the two months being compared use different definitions. The SQL engine knows whether a field exists. It does not know whether that field represents the revenue the business is asking about. Python will not raise an exception because a report presents correlation as causation.
Whether grouped totals reconcile with the overall total, primary keys are duplicated, or a denominator is zero can all be checked automatically. But what a test can catch depends on which requirements were written into it.
If the test and the query share the same mistaken assumption, they may agree.
Why Did Revenue per User Fall?
Consider a hypothetical question: “Analyze why revenue per valid user has fallen.”
First, Establish the Business Meaning
The ambiguity starts immediately. Does “valid user” mean someone who registered, logged in, or paid? Does revenue mean payments received, net revenue after refunds, or revenue recognized over the service period? Each definition can map to a table and produce a number.
Suppose the business wants net revenue for paying subscribers, but the agent selects logged-in users and payment amounts. The tables and fields exist, the query runs, and grouped totals reconcile. It can also write a test confirming that revenue per user equals total revenue divided by the number of users.
That test will pass. Choosing the wrong numerator and denominator does not stop the division from being correct.
Having the coding agent read documents first helps. But an enterprise may have several historical definitions, different definitions across departments, and transformation logic that keeps changing. After finding a description, the agent still needs to establish which product and period it applies to, and whether the current data was produced under that definition. Running more queries will not reveal those rules from field names.
A data agent’s semantic layer needs to connect “valid user” and “net revenue” to the actual data. It needs to specify which definition to use, which tables and fields, and how to filter and join them. The definition’s source, effective date, and scope also need to be retained. Otherwise, an outdated definition may remain in use.
The agent retrieves data according to these definitions, then checks whether the generated query follows them. Writing “exclude trial users” in the analysis plan is not enough; the query has to exclude them too.
If definitions conflict or the data’s coverage is unclear, the agent needs to clarify that first. It should not pick whichever definition looks reasonable and keep calculating.
In “Six Layers of Context That Help Data Agents Understand Business”, I wrote about similar metrics being confused with one another, and about data coverage that was not documented in the schema and had to be confirmed through ETL logic.
After the Numbers Are Correct
Suppose there are two groups, with revenue per user of 100 yuan and 10 yuan respectively. Last month, each group accounted for half the population, so overall revenue per user was 55 yuan. This month, the higher-revenue group’s share falls to 20%, bringing the overall figure to 28 yuan. Revenue per user within each group has not changed, yet the overall figure has fallen substantially.
If the agent looks at the overall trend and a campaign running at the same time, then writes “users are less willing to pay,” the data does not support that explanation. It first needs to distinguish a change in population composition from lower spending within a group. After identifying which group’s share declined, it still needs to investigate why that share changed.
A coding agent can write this decomposition code, and it may decide to do so on its own. But if it does not think of it, the execution results will not remind it.
Analytical methods therefore need to specify when they apply and which checks cannot be skipped. Whether the data has fully arrived and whether the two months’ populations are directly comparable need to be established first. Allocating the revenue difference across groups completes a contribution decomposition; it may not identify the cause. A concurrent campaign can be a lead, but cannot directly explain the decline.
Asking the model to check its work again may reveal omissions. But if it has consistently accepted the premise that “an overall decline means individuals are spending less,” its review may simply confirm that the charts and numbers agree, then endorse the original explanation.
A data agent needs to organize these methods into Skills and use tools and scripts to check the analysis as it runs. If population decomposition is missing, add it. If data has not fully arrived, wait or narrow the observation range. If the analysis has located the change but lacks evidence to explain it, explicitly report “cause not yet established.” Business interpretations still need human judgment.
Agents make it inexpensive to explore hypotheses. Examining more dimensions also makes it easier to find coincidental correlations. Finding a pattern in one dataset and displaying it again using that same dataset does not constitute independent validation. Depending on the question, validation may require another time window, held-out samples, or further experiments.
A coding agent can perform all of these checks. The analytical method needs to specify when they are required and which conclusions to withdraw if validation fails.
When the Definition Changes, What Happens to Earlier Results?
After seeing an intermediate result, the user says, “Only count paying subscribers as valid users. Exclude trial users.” The population, denominator, grouped results, and explanations may all need to change. Or refund data arrives a day late, and yesterday’s net revenue needs updating today.
A coding agent can run long tasks, execute work in parallel, save files, and resume from checkpoints. But those capabilities do not tell us whether an earlier result is still applicable. The old CSV is intact, the charting script runs, and the chart files are still there, yet they may use the definition from before the change.
For example, the agent may rerun total revenue but reuse the denominator from the old population. Or it may update a table while leaving charts and explanations from the previous run in the report. Every step executes successfully, but the report mixes two sets of conditions.
A data agent needs to record the definition version, query, and data used to produce each result, as well as how subsequent calculations, charts, and conclusions depend on it. When the definition changes, results that depend on the old definition must be recalculated. When data changes, the relevant charts and explanations also need to be checked again. Before producing the report, it needs to confirm that they all use the same analytical conditions.
These dependencies need updating throughout the analysis. A file directory and execution log alone do not reveal which explanations are affected by a change in business definition. If it is unclear whether an old result still applies, a successful past execution is not enough to justify reusing it.
How a Data Agent Adds These Checks
A data agent needs to bring these business requirements into execution. Data Context holds metadata, metric definitions, transformation logic, business rules, and confirmed experience. The semantic layer connects business concepts to actual data. Skills organize analytical methods, while the runtime handles checks, saves results, and manages failures.
During data retrieval, definitions in the semantic layer need to be applied to both queries and checks. The numerator and denominator of revenue per user, user exclusions, and the grain to preserve after joins all need to be checked against the SQL and its results. Queries and checks use business-confirmed definitions, rather than merely validating whichever definition the model selected on the spot.
When investigating causes, campaign records, business rules, and past experience in Data Context help the agent form hypotheses. A revenue-diagnosis Skill requires it to separate composition changes from changes within groups, then check which users a campaign affected, when it happened, and whether it is consistent with the observed change. Tools handle the calculations. If evidence is missing, the agent keeps investigating or stops at “cause not yet established.” More documents cannot replace these steps.
The definition version and partitions used in the current analysis, along with which calculations and charts used that data, also need to be recorded. When a definition or dataset changes, the runtime can use these dependencies to flag affected results and arrange recalculation and review. The semantic layer records definition changes; execution records show where those definitions were used.
Whether these checks work needs to be tested on real tasks. Give the system conflicting definitions and see whether it asks for clarification. Introduce a join that double-counts revenue and see whether it detects the problem. Change a definition and see whether it withdraws the old results. Whether explanations go beyond the evidence also needs checking. Semantic definitions can be wrong, and methods have limits. Anything unconfirmed needs to remain explicit in the report.
Confirmed definitions and corrections go back into Data Context. Effective analytical procedures become Skills for future use. Unvalidated guesses remain hypotheses; they must not be passed on as business facts.
These capabilities can be built on a general-purpose agent or coding agent. Data semantics, analytical methods, and result dependencies require ongoing maintenance. When a check finds a problem, clarify, investigate further, or recalculate before proceeding.