Open Data Infrastructure needs context engineering to unify data use cases

The Open Data Infrastructure promises the ability to support every data use case through interoperability and a single source of truth. At Fivetran + dbt Labs, our preferred solution to the problem of moving and storing data a single time in a unified data architecture is the Managed Data Lake Service (MDLS), which delivers data into open table formats, keeps a data lake continuously synced from source systems, manages schema evolution automatically, and publishes metadata to a catalog so that every engine can access a consistent, governed view of the data. A unified data architecture can be combined with any compute or query engine, and support all use cases.

While the MDLS supports technical formats that can be read by a wide range of tools, you also need to turn raw data into a context that communicates a single source of truth to human and AI users of the data alike. Agents and humans cannot operate based on conflicting versions of the truth, especially given the continuous, high-scale nature of intelligent automation.

Context engineering is the practice of making information an AI system needs explicit, structured, trustworthy, and retrievable at usage. Unlike humans, AI systems lack tacit and tribal knowledge gained from experience, on-the-job training, self-directed learning, and common sense. They must be explicitly instructed with all of the relevant context, making context engineering key to realizing the potential of an Open Data Infrastructure.

Context engineering can be divided into 4 broad needs:

1. Semantics and provenance: What does the data represent, how is it calculated, and where did it come from?
2. Quality and state: Can this data be trusted right now, based on quality, freshness, and other execution state characteristics?
3. Governance and lifecycle management: How do you ensure that data assets are delivered and owned by the appropriate use cases and audiences?
4. Discovery and delivery: How will you tailor context for every agent or AI system?

[CTA_MODULE]

1. Semantics and provenance

Semantics and provenance concern the meaning and origin of every data asset. It can be boiled down into the following needs:

• Semantics, structure, and grain: The mapping between every data model and field to real-world business concepts, down to what each individual row represents.
• Business logic: The rules and transformations that produce each real-world business concept
• Lineage and provenance: Exposing and understanding the full process that produces every field and model, including all intermediate steps

Semantics are defined in dbt Semantic Layer, where metrics, dimensions, entities, and valid joins are authored, tested, and governed in the same workflow that builds the data. Open Semantic Interchange, now known as Apache Ossie (Incubating), a vendor-neutral spec dbt Labs helped found, keeps those definitions portable across query engines and tools rather than locked into whichever one authored them. Open standards such as Agents Schema give agents one place to look for that context: models, columns, lineage, and business meaning. Business logic is directly encoded through transformations, with models showing explicitly how source records become downstream business entities such as customers, orders, or revenue. dbt Catalog establishes the provenance and lineage of a data asset and every business entity by exposing every intermediate model and field that produces it.

2. Quality and state

Quality and state affirm to AI systems and other users that a model is usable and trustworthy based on freshness, quality, and other execution state characteristics (i.e., the status of recent or ongoing syncs).

With a clear assessment of quality and state, AI can qualify an answer, seek alternative sources, or decline to answer instead of assuming all data is equally reliable. The trustworthiness of data is based on a progression of evidence:

• Tests: Does the data satisfy expected rules?
• Contracts: Does the model still conform to expected schemas and constraints?
• Freshness: Is the data current enough for this use?
• Run status: Was the latest transformation completed successfully?

dbt Catalog, powered by the dbt Discovery API, generates and stores all the relevant information about a project run, including models, sources, nodes, and execution results.

3. Governance and lifecycle management

Beyond knowing the data assets in play, governance encompasses controlling access and permissions. More specifically, it concerns the hierarchies, classifications, dependencies, and boundaries that ensure that data assets are only sent to intended workflows and audiences. Lifecycle management includes ensuring context stays abreast of changing definitions, deprecated data assets, evolving schemas, and ownership changes.

dbt Mesh and its governance features are tailor-made to solve these needs, ensuring both control and responsible change over time.

4. Discovery and delivery

Discovery and delivery make sure every AI system has exactly the context it needs, and a minimum of noise.

The foreseeable future of agentic AI looks less like monolithic “do-it-all” agents and more like networks of specialized agents with specific and discrete instructions and use cases. It is usually better to augment such AI systems with precisely the handful of well-defined models, metrics, and dependencies they need than an entire warehouse’s schema.

This means identifying the relevant context, selecting the minimum viable subset of your data, and making it machine-readable. 

dbt Catalog, as previously described, surfaces models, metrics, owners, relationships, and status through search, resource pages, lineage, and ERD views. This same metadata is also programmatically available through the Discovery API. To connect an AI system to the relevant data, dbt MCP server offers a standardized protocol for retrieving models, metrics, lineage, and other context, as well as performing other permitted actions.

Context engineering helps human consumers of data, too

All of the context engineering required to make data legible and usable for AI also makes data usable for everyone who might embed it into products or rely on it to support decisions. No entity can reliably infer business meaning from table and column names alone.

Externalizing institutional knowledge through context engineering also makes work reproducible, enables meaningful self-service, facilitates collaboration, and makes onboarding and organizational changes easier and safer.

Context engineering converts data from something an organization merely possesses into something people and machines alike can reliably understand, use, verify, and maintain.

[CTA_MODULE]