How Open Data Infrastructure preserves choice in the age of AI

“Open” represents one of the most important promises in enterprise technology: the freedom to choose the right tools, adapt as needs change, and innovate without rebuilding the data foundation. Open formats, APIs, and source projects all help advance that vision. But genuine openness is ultimately measured by what customers can do — how easily they can move between technologies, combine platforms, and evolve their architecture over time.

At dbt Summit, George Fraser, CEO of Fivetran + dbt Labs, Russell Spitzer, Principal Software Engineer at Snowflake, Dipti Borkar, Vice President of Microsoft IQ and Microsoft One Lake at Microsoft, and Jordan Tigani, the Co-founder and CEO of MotherDuck, explored what it takes to fulfill that promise. Their conversation examined a defining question for the next generation of enterprise data architecture: What must remain portable for organizations to realize the full value of an open ecosystem?

The answer extends far beyond file formats. Organizations need to control where their data lives, choose the right compute engine for each workload, apply consistent security policies, preserve business meaning across tools, and change components without rebuilding the system around them. If any one of those layers becomes proprietary, lock-in simply moves somewhere else.

[CTA_MODULE]

Open formats are the beginning, not the finish line

Open table formats such as Apache Iceberg have changed the architecture of enterprise data. By separating data storage from the engine used to process it, they allow multiple platforms to work with the same underlying tables. That gives organizations more control over their data and more freedom to evolve the tools around it.

But an open table is not enough if every engine interprets access differently. Governance can become the next point of lock-in: policies are defined in one catalog, implemented differently across platforms, and recreated whenever a new engine enters the architecture.

The panel highlighted recent work on read restrictions in the Iceberg REST protocol as an important step forward. The goal is for a catalog to return not only table information, but also the rules that apply to the specific user requesting it. Instead of redefining access controls engine by engine, organizations could maintain one policy definition and carry it across an interoperable environment. As Snowflake’s Spitzer explained, “In the near future, you’ll be able to have one definition of your policies, one access point where you can talk to your catalog and get all that information into your engine at the same time.”

That distinction matters because flexibility without consistent governance is not a viable enterprise architecture. Data teams should not have to choose between openness and control. As Microsoft’s Borkar put it, “Customers have paid enough of the lock-in tax.” Moving data into an open format only to trap its security model in another proprietary layer would recreate the same problem under a different name.

True openness, therefore, requires policies to travel with the data. The format, catalog, engines, and security model must work together so that adding a new compute option does not create a new governance project.

“Customers have paid enough of the lock-in tax as we call it. We now are at open formats, which is great…we have to ensure that we don't get them locked in again.”
— Dipti Borkar, Vice President of Microsoft IQ and Microsoft One Lake at Microsoft

The best engine should win each workload

Once storage and governance become interoperable, compute can become a competitive layer rather than a permanent commitment.

The panel offered a practical example. dbt Lake Compute, built on DuckDB, is designed for non-interactive transformation workloads. MotherDuck also builds on DuckDB but focuses on low-latency, interactive analysis where a person is waiting for the result. Those engines can make different architectural choices because they serve different needs. With data stored in an open format, customers do not have to select one engine for every workload.

That is the deeper promise of decoupling storage and compute. As Microsoft’s Borkar explained, “The OneLake principle is open and unified, and we need to make sure that we have Fabric working on it, Snowflake working on it, Databricks working on it. Everyone has to access the same data. It truly needs to be open.” With that shared data foundation, a team can choose one engine for large transformations, another for interactive analysis, and another for AI agents. As MotherDuck’s Tigani summarized it, “May the best engine win.”

“And that I think shows the power of these open architectures because if you have your data in Iceberg, Delta, Duck Lake, you can use any engine you want. You can use the best engine for transformation, you can use the best engine for low latency, you can use the best engine for agents and may the best engine win.”
— Jordan Tigani, the Co-founder and CEO of MotherDuck

This model also changes how vendors create durable customer relationships. In a closed architecture, retention can depend partly on the cost and difficulty of leaving. In an open one, vendors must continue earning the workload through performance, economics, usability, and innovation. Snowflake’s Spitzer captured the principle directly: “We have to let all the doors be open and prove out on the value of the product that you can earn customers’ dollars.”

Open Data Infrastructure does not eliminate platforms or competition. It makes competition healthier. Customers gain the freedom to use the best tool for the job, while vendors gain a stronger incentive to keep making their products better.

The transition will not happen overnight. As Fraser noted, many enterprises are still migrating from data warehouses that have existed for decades. The most important contest is not only between the newest cloud platforms. It is between architectures designed around long-term dependency and those designed around customer choice.

Business meaning is the next portability problem

Making tables portable solves only part of the problem — especially as AI becomes a primary consumer of enterprise data. An agent does not simply need access to rows and columns. It needs to understand what “revenue,” “customer,” or “active account” means inside a particular business.

That makes the semantic layer a critical part of Open Data Infrastructure. Fraser stated that, as models have improved, a semantic model increasingly appears to be both a necessary and potentially sufficient foundation for connecting AI to enterprise data. Fivetran’s experience has shown that adding more complexity does not always improve the answer. Often, a clear model of business entities, metrics, and relationships gives an agent what it needs to reason effectively.

Others on the panel saw the semantic model as the starting point rather than the final answer. Richer ontologies and graphs may become necessary for some domains. More importantly, teams need evaluations that test whether an agent actually uses the available context and returns the correct result. If a CFO asks for last quarter’s revenue, there is a right answer. The sophistication of the context system matters less if the agent ignores it or applies it incorrectly.

This points to a new responsibility for data teams: maintaining evaluations for natural-language questions may become as important as maintaining tests for pipelines and transformations. Trusted conversational analytics will require teams to define not only business meaning, but also how they verify that an agent has understood it.

Semantic portability matters, too. Enterprises already have years of business logic encoded in dbt, Looker, Power BI, and other systems. They should not have to discard that work to adopt a new AI interface. Fivetran Context Layer is designed to connect to existing semantic models and publish that context to agents. Meanwhile, Apache Ossie is working toward a vendor-neutral interchange format for metrics, dimensions, and their relationships.

The ambition mirrors what Iceberg did for tables: make business meaning portable across platforms without losing its definition. If the industry succeeds, organizations will be able to change analytics, BI, or AI tools without translating the business from scratch each time.

Open infrastructure wins when adoption becomes invisible

Standards matter only when customers can use them without becoming experts in the machinery underneath.

That is why the final challenge is adoption. Many organizations already have functioning data stacks built around proprietary tables, tightly coupled compute, or long-established semantic models. A vision that requires them to replace everything at once will remain confined to early adopters.

Fraser described Fivetran’s role as making the transition easier for those existing environments. As platforms such as Snowflake and Microsoft add deeper Iceberg interoperability, Fivetran helps customers introduce open tables beneath their current data stack while preserving the workflows built above them. His stated goal was strikingly simple: make the change so easy that customers barely notice they have introduced Iceberg.

That is what mature Open Data Infrastructure looks like. Data lands in formats the customer controls, compute engines are matched to each workload, governance remains consistent across engines, and semantic definitions move between tools. Fivetran reliably delivers data into this foundation, while dbt transforms it and preserves the business logic people and agents need.

The value of that openness is practical, not ideological. It preserves strategic flexibility, lowers the cost of adopting better technology, protects existing investments in data and semantic work, and reduces the risk that today’s AI decision becomes tomorrow’s migration project.

The organizations best prepared for the next generation of analytics and AI will not be those that predict every winning technology. They will be those that build an architecture capable of changing when the winners change.

[CTA_MODULE]