What a dbt project is worth to an AI agent

I asked 2 AI agents the same question against the same 4 demo retail tables: What is total revenue? The one running Haiku summed the line items. The one running Opus looked at the orders first, saw that some were canceled, left them out, and said so in a note. An analyst could have asked whether canceled orders count before answering, but neither agent did. That is conversational analytics in 1 exchange: a plain-language question, 2 confident answers, and nothing in the interface to say which business rule applies.

I wanted to know why the 2 agents split on the same question and whether any of the dbt interfaces would have closed the gap.

So I measured it, and along the way I got a much clearer picture of what a dbt project is worth once the consumer of the data is an agent rather than a person. This post walks through 5 interfaces into the same project with the numbers attached: what each one did for accuracy, trust, and cost, and when a smaller model can stand in for a larger one.

The short version: The interface mattered significantly. Requiring Haiku to read the dbt documentation before querying took its wrong answers on the join questions from 4 to 0. The Semantic Layer was the only interface that brought both models to 0 wrong answers, and it did so at the lowest cost per trusted answer for both. For conversational analytics, the takeaway is clear: Better-governed access to data can matter as much as the model itself.

[CTA_MODULE]

Why I measured it

This work was partly inspired by MotherDuck's agent benchmark, which I learned about from a great webinar Jacob Matson and Alex Monahan delivered, Do AI Agents Need a Semantic Layer? Their methodology included tool-restricted agents, checks for agents bypassing the governed layer, and efficiency measured alongside accuracy.

The second motivation was my own interest in how agents work. Every agent I build against a dbt project starts with a decision about the interface: raw SQL over the tables, the Semantic Layer, or Agents Schema and context via the manifest and its documentation. I didn't really have a clear answer for what to use, when to use it, or which type of model to put behind it.

I built a test harness that held the models and questions constant while changing the interface. Each agent received a business question and limited tools, then worked until it could return an answer, flag ambiguity, or say the available data could not answer the question.

Every question had a verified answer. An answer was wrong if the agent responded but missed that result. An answer was trusted if it got the answer right or correctly said the data could not answer the question. A correct answer accompanied by uncertainty counted as luck rather than trust.

5 interfaces from 1 dbt project

Here are the 5 interfaces; each is a dbt artifact:

1. Raw SQL: Table names, column names, and data types, with no documentation.
2. Agents Schema, available: The same interface, plus tables generated using dbt Labs' open Agents Schema standard, containing model descriptions, column descriptions, and lineage. That documentation is the context everyone is talking about giving agents, so I use both words below. The agent is told the tables exist, but nothing makes it read them.
3. Agents Schema, required: The same tables, but the agent had to read the documentation before querying the data.
4. Manifest tools: Every dbt parse writes a manifest, 1 file holding every model, every column and its description, and the lineage between them. Here, the agent gets 3 tools over that file: keyword search, column lookup, and plain-language model definitions. It also gets SQL, but the harness rejects any query until the agent has consulted the manifest at least once.
5. Semantic Layer: Named metrics queried by named dimensions, with no direct SQL access. I tested both local MetricFlow and the hosted dbt Semantic Layer API.

I ran 2 types of questions:

1. 25 "join" questions across orders, line items, customers, and products, including grain and fan-out traps and 2 questions the data can't answer at all
2. 10 "why" questions asking why a metric moved between 2 periods, using data where the cause was planted ahead of time

Claude Haiku 4.5 and Claude Opus 5 ran every question 3 times on DuckDB. I also tested the Semantic Layer on Snowflake.

An earlier benchmark of 60 single-table questions across 3 industries gave me additional data for comparing 4 Claude models — Haiku 4.5, Sonnet 5, Opus 5, and Fable 5 — and DuckDB, Snowflake, and Databricks. Together, that round produced 6,480 runs.

Documentation fixed Haiku's grain error

Here are the "join" questions, 75 runs per cell. Cost per trusted answer is the model spend in the cell divided by the answers you could act on.

The Semantic Layer is the only row where both models reach 0, and it is the cheapest row for both. 6 of each cell's 75 runs are the 2 unanswerable questions, which every interface declined correctly.

Haiku's 4 raw-SQL misses were the same error repeated. When asked for average order total, it averaged individual line items and returned $255.33 instead of the correct order-level average of $512.03 in 3 runs out of 3, with no flag of doubt. The order total was not stored directly. It had to be calculated from line items, with the column documentation explaining that requirement.

Once Haiku read that documentation, through either the required Agents Schema or the manifest tools, the error disappeared. It returned 0 wrong answers across 150 runs on those 2 interfaces.

Simply making the Agents Schema available wasn't enough. Haiku consulted the documentation only 0.09 times per run and caught the error in just 1 of 3 runs.

Column documentation fixes a grain mistake like this one, as long as the agent is made to read it.

The Semantic Layer fixed Opus's revenue policy

Opus failed for a different reason. Its 11 misses while reading raw SQL came from a business-policy decision: It excluded canceled orders when calculating total revenue, electronics revenue, and average order total. It disclosed that choice, but the answers still didn't match the defined metric.

Documentation didn't fully solve the problem. Even when required to read the Agents Schema, Opus still excluded cancellations in 2 of 3 runs for both total revenue and electronics revenue.

The documentation described the columns, and the columns were not what Opus disagreed about.

The Semantic Layer resolved that ambiguity by defining total_revenue once. Behind that governed metric, Opus returned 0 wrong answers in 75 runs.

Tests and contracts on those definitions were not exercised in these runs; they are what keep the definitions true after the benchmark ends. The context and the semantic models came out of the same dbt project: The tables were dbt models over dbt sources, the documentation was model and column descriptions, 1 dbt parse produced the manifest that both the tools and the Agents Schema tables consume, the semantic models sit beside them, and the project compiled to DuckDB, Snowflake, Databricks, and a dbt platform environment without a line changed. I did not use a tool outside dbt to build any interface.

The “why” questions are a different job

The "why" questions asked which factor drove a metric between period 1 and period 2, from a closed list: category mix, subcategory mix, pricing, order volume, status mix, or nothing material. Each question had 1 planted cause.

A few were set up to be tricky on purpose: one where a visible factor moved the metric in the wrong direction, 2 where nothing had really moved, and one where 2 causes were an even split, so the right answer was to name one and mention the other.

Opus got 148 of 150 runs right across every interface. The other 2 were declines rather than incorrect answers.

Haiku missed questions at every interface. Its errors followed a consistent pattern: It selected the factor that changed most visibly instead of the one that actually moved the metric.

The Semantic Layer couldn't fix that reasoning error. The agent still had to interpret the numbers and determine causality.

What the Semantic Layer did change was the cost of getting the inputs. Haiku on the Semantic Layer reached its answer in 3.4 turns at about $0.02 per run. On every other interface, it used 13 to 17 turns, 1 query per turn, at up to $0.20. Opus, which plans its queries, needed 4 turns on raw SQL and 6.6 on the documentation interfaces.

Context must be required and scoped

The Agents Schema available interface taught me the most about how context actually gets used.

When documentation was optional, Haiku read it just 0.09 times per join run and never on "why" questions. Requiring a documentation read increased usage to 2.4 reads per run and reduced its wrong answers to 0, using 2-thirds of the tokens required by the manifest tools because reading a table is cheaper than a search-and-lookup conversation.

The earlier single-table study showed the other half of this. The most capable model in that round, Fable 5, was the worst manifest-tools performer at 7.0% wrong because its tools let it browse documentation across 3 industries. It answered a supply-chain forecasting question from a retail column of the same name in 9 runs out of 9 on that question. Scoping the tools to the question's domain took it to 3.3%, its own raw-SQL baseline, in a controlled A/B where the other model did not move.

Opus showed a similar pattern when it could access every model in the project, including duplicate retail data. Once access was limited to the relevant tables, cost per run fell by more than half.

I consider this a form of the model overthinking, and it overthinks in 2 directions: It searches wider than the question needs, which is what happened here, and it reads more into the question than was asked, which is what happened with the revenue policy. dbt has a different feature for each: Scoping the interface handles the 1st, and a governed metric handles the 2nd.

dbt produces the context and the metrics, and the agent builder still decides how much of the project an agent can access and whether it must read that context before querying. In these runs, those 2 decisions were what turned the dbt artifacts into the 0-wrong rows.

A smaller model can be competitive behind a governed interface

The earlier benchmark showed raw-SQL error rates generally falling as model cost increased, from 7.8% for Haiku to 3.3% for Fable. Behind the Semantic Layer, however, every model landed between 3% and 3.5%.

Haiku behind the Semantic Layer matched the accuracy of Fable writing its own SQL at 1/13 of the cost per answer: $0.0054 compared with $0.0707.

On the "join" questions, the 2 models I ran both went to 0 behind it. That is what makes Haiku viable for the governed questions in the routing below.

The dbt project also produced the same verified results across DuckDB, Snowflake, and Databricks, including a Databricks test that read Fivetran Managed Data Lake Service in place. Results matched on all 52 single-table checks and all 23 join checks for each engine.

dbt's hosted Semantic Layer reproduced the local MetricFlow results exactly on Snowflake: 0 wrong answers for both models. The Semantic Layer column was flat across engines in the single-table round while raw SQL varied by dialect, and the variation fell on the smallest model. In these runs, once the metric definition was fixed, the engine stopped changing the answer.

Choosing a practical dbt foundation for AI agents

1 dbt project supported every interface in this benchmark, so you don't have to choose one of them. The routing below is just a decision about which interface a given question goes through.

Here's the routing I am considering for future agents based on this benchmark:

Based on these runs, the governed questions — the ones a metrics bot in Slack or a dashboard Q&A box gets asked — are where I can use Haiku instead of a larger model. Questions that repeatedly require raw SQL are also candidates for a metric definition.

I'm already testing part of this approach in an agentic application. Haiku reads the question and a 1-line index of connected sources, then selects the few that appear relevant. Sonnet 5 reasons using only those tools, preventing the stronger model from wandering through the full catalog. If source selection fails, the application loads the full set.

The next step is to route questions covered by governed metrics directly to Haiku on the Semantic Layer without invoking Sonnet.

Haiku is only cheap when the interface is short. On the "why" questions, Haiku cost $0.110 per raw-SQL run compared with $0.135 for Opus because it needed 13 turns and 80,000 tokens while Opus finished in 4. Behind the Semantic Layer, Haiku's cost fell to $0.022 per run, compared with $0.138 for Opus.

The 2 models also differ in what they write down. Opus records its doubts — 16 to 19 right-by-luck answers with ambiguity notes in its raw-SQL cells — while Haiku mostly does not, with just 3 to 4. That matters a great deal if nobody reads the output before it gets used.

Here are 3 questions I can now ask about any agent before I point it at a dbt project, each tied to something that went wrong in these runs:

1. How much of the project can the agent see, and is it more than the question needs? A stronger model will explore all of it. Opus summed revenue across 2 datasets when both were visible; scoped to the right tables, it didn't.
2. Does the question have a governed definition, or will the model supply its own? Opus answered revenue with its own canceled-orders rule on every interface except the Semantic Layer.
3. Does the agent have to read the context before it queries, or is the context just there? Haiku read the Agents Schema tables 0.09 times per run when they were just there and 2.4 times when the harness insisted.

When nobody is going to read the answer before it gets used, I start the agent on the Semantic Layer or on context it is required to read. Those were the 2 places where Haiku's mistakes went away, and it rarely flagged those mistakes itself.

That also translates into a simple dbt project checklist for agent development:

1. Document every column: Column context eliminated Haiku's grain errors.
2. Create a semantic model for every governed KPI: The governed revenue definition eliminated Opus's policy errors.
3. Test those definitions: Tests and contracts were not exercised in these runs, and they are what keep the definitions true after the agent ships.
4. Publish the dbt project where an agent can reach it at runtime: A repo isn't callable. The metrics and docs have to be served through the hosted Semantic Layer, the Agents Schema tables in the warehouse, or the dbt MCP Server.
5. Start with a low-cost model on the Semantic Layer: If it comes back saying the metrics can't answer the question, hand the question to a higher-cost model with the docs. For a "why did this move" agent, start with the higher-cost model.

The join round came from 2 models and demo data, so I treat these findings as a starting point. The models, the interfaces, and dbt itself are moving quickly enough that I expect to be rewriting this list within a few months.

Across these runs, the interface changed the answer as often as the model did. The same 2 models went from 4 and 11 wrong answers on raw SQL to 0 behind the Semantic Layer, and nothing about the models changed in between.

See how dbt can help put that foundation into practice for your own AI workflows.

[CTA_MODULE]