Fully managed data integration is finally here

With our new dbt Cloud™ orchestration integration, automated end-to-end data integration is now at your fingertips
February 1, 2024

We are excited to announce the release of our newest integration with dbt Cloud, allowing you to orchestrate dbt Cloud jobs right from Fivetran’s platform. This integration means that all stages of data integration – extraction, loading and transformation – can now be managed through one platform with minimal engineering effort.

This isn’t our first rodeo

In 2022, Fivetran launched integrated scheduling for dbt Core™, our solution for orchestrating your open-source, self-hosted dbt data transformations. Integrated scheduling enables transformations to be triggered by the completion of Fivetran connector data syncs, ensuring that downstream tables – and the metrics and insights derived from them – remain up to date as new data becomes available. By tying transformations to the completion of data syncs, integrated scheduling also prevents redundant transformations from eating up compute costs.

Our team also began building our robust library of open-source, free dbt data models. These data models have helped data teams reduce engineering work by standardizing the base use cases for common SaaS data sources. This library continues to grow – including our new SAP data model – and the packages have been utilized by analytics engineers using both dbt Core and the fully managed dbt Cloud.

New capabilities provided by dbt Cloud orchestration

For data teams looking for a fully managed experience, similar to Fivetran, dbt Labs created dbt Cloud. dbt Cloud’s fully managed approach to data transformations means teams do not have to provision their own infrastructure, while also taking advantage of collaborative web-based development and navigation through a web UI. Many organizations have rapidly adopted dbt Cloud for these capabilities.

“dbt Cloud is designed to help teams manage the complexity of their modern data environments so they can ship trusted data faster,” says Luis Maldonado, VP Product of dbt Labs. “By automatically triggering dbt Cloud jobs to run when upstream data loads are complete, our over 1,000 shared customers can avoid tedious manual orchestration and have confidence that decisions are being made from the freshest data possible.”

The new Fivetran orchestration integration with dbt Cloud brings our integrated scheduling to dbt Cloud. Analysts and engineers can now schedule dbt Cloud jobs to run upon the successful load of new data from associated upstream connectors. You can set up, monitor and QA your pipelines directly from the Fivetran GUI with our robust run logs. Fully managed end-to-end data integration is now accessible to teams in an automated and accessible fashion.

“We know this will help data teams operate more efficiently and cost-effectively thanks to Fivetran’s data orchestration,” said Mac Noland, Co-Founder and Chief Data Officer at phData. “We even use this internally and find that our dbt Cloud and Fivetran instances are now fully synchronized and automated, reducing the time our team spends on pipeline build and maintenance to near null. This means our team - and our clients' teams - can now focus on more strategic projects while saving money since compute isn’t engaged until there’s new data available.”

The new dbt Cloud integration continues the Fivetran trend of simplifying data pipeline setup and maintenance, helping data teams reprioritize their attention while democratizing access to data and making it simpler and more reliable to use. Combined with Fivetran pre-built data models, the new integration offers a single environment from which it is possible to set up reports and dashboards in a matter of minutes.

“Our dbt Cloud integration delivers one platform to manage and automate end-to-end ELT pipelines with built-in choice. It allows for an organization’s entire pipeline to be fully managed and integrated, letting data teams focus more on data work versus pipeline management,” says Helge Scheil, EVP of Engineering at Fivetran. “This integration further underscores our commitment to providing our customers with a central platform that acts as a data foundation, regardless of the other tools they choose to build with.”

Getting started is dead simple

As with all other interactions involving Fivetran, getting started with dbt Cloud orchestration consists largely of navigating through a series of menus. The “Transformations” section of your Fivetran dashboard will present you with options for Quickstart data models, dbt Core and dbt Cloud. 

Choose the dbt Cloud option, authenticate your account with a Service token, choose your project, and select “Save & Test” per usual. 

From there you can use the “Add transformation” button to select jobs from your dbt Cloud project.

Set a schedule, either integrated - meaning dbt Cloud jobs run upon completion of upstream connector(s) - or custom if you want transformations to execute on fixed intervals or at fixed times. If you choose integrated, you then select which connectors’ successful data load you want to trigger the job run.

Once the transformation is live, you can monitor it for success using our run logs

Don’t just take our word for it. Experience the power of automated end-to-end data integration for yourself with a free trial.

Kostenlos starten

Schließen auch Sie sich den Tausenden von Unternehmen an, die ihre Daten mithilfe von Fivetran zentralisieren und transformieren.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Product
Product

Fully managed data integration is finally here

Fully managed data integration is finally here

February 1, 2024
February 1, 2024
Fully managed data integration is finally here
With our new dbt Cloud™ orchestration integration, automated end-to-end data integration is now at your fingertips

We are excited to announce the release of our newest integration with dbt Cloud, allowing you to orchestrate dbt Cloud jobs right from Fivetran’s platform. This integration means that all stages of data integration – extraction, loading and transformation – can now be managed through one platform with minimal engineering effort.

This isn’t our first rodeo

In 2022, Fivetran launched integrated scheduling for dbt Core™, our solution for orchestrating your open-source, self-hosted dbt data transformations. Integrated scheduling enables transformations to be triggered by the completion of Fivetran connector data syncs, ensuring that downstream tables – and the metrics and insights derived from them – remain up to date as new data becomes available. By tying transformations to the completion of data syncs, integrated scheduling also prevents redundant transformations from eating up compute costs.

Our team also began building our robust library of open-source, free dbt data models. These data models have helped data teams reduce engineering work by standardizing the base use cases for common SaaS data sources. This library continues to grow – including our new SAP data model – and the packages have been utilized by analytics engineers using both dbt Core and the fully managed dbt Cloud.

New capabilities provided by dbt Cloud orchestration

For data teams looking for a fully managed experience, similar to Fivetran, dbt Labs created dbt Cloud. dbt Cloud’s fully managed approach to data transformations means teams do not have to provision their own infrastructure, while also taking advantage of collaborative web-based development and navigation through a web UI. Many organizations have rapidly adopted dbt Cloud for these capabilities.

“dbt Cloud is designed to help teams manage the complexity of their modern data environments so they can ship trusted data faster,” says Luis Maldonado, VP Product of dbt Labs. “By automatically triggering dbt Cloud jobs to run when upstream data loads are complete, our over 1,000 shared customers can avoid tedious manual orchestration and have confidence that decisions are being made from the freshest data possible.”

The new Fivetran orchestration integration with dbt Cloud brings our integrated scheduling to dbt Cloud. Analysts and engineers can now schedule dbt Cloud jobs to run upon the successful load of new data from associated upstream connectors. You can set up, monitor and QA your pipelines directly from the Fivetran GUI with our robust run logs. Fully managed end-to-end data integration is now accessible to teams in an automated and accessible fashion.

“We know this will help data teams operate more efficiently and cost-effectively thanks to Fivetran’s data orchestration,” said Mac Noland, Co-Founder and Chief Data Officer at phData. “We even use this internally and find that our dbt Cloud and Fivetran instances are now fully synchronized and automated, reducing the time our team spends on pipeline build and maintenance to near null. This means our team - and our clients' teams - can now focus on more strategic projects while saving money since compute isn’t engaged until there’s new data available.”

The new dbt Cloud integration continues the Fivetran trend of simplifying data pipeline setup and maintenance, helping data teams reprioritize their attention while democratizing access to data and making it simpler and more reliable to use. Combined with Fivetran pre-built data models, the new integration offers a single environment from which it is possible to set up reports and dashboards in a matter of minutes.

“Our dbt Cloud integration delivers one platform to manage and automate end-to-end ELT pipelines with built-in choice. It allows for an organization’s entire pipeline to be fully managed and integrated, letting data teams focus more on data work versus pipeline management,” says Helge Scheil, EVP of Engineering at Fivetran. “This integration further underscores our commitment to providing our customers with a central platform that acts as a data foundation, regardless of the other tools they choose to build with.”

Getting started is dead simple

As with all other interactions involving Fivetran, getting started with dbt Cloud orchestration consists largely of navigating through a series of menus. The “Transformations” section of your Fivetran dashboard will present you with options for Quickstart data models, dbt Core and dbt Cloud. 

Choose the dbt Cloud option, authenticate your account with a Service token, choose your project, and select “Save & Test” per usual. 

From there you can use the “Add transformation” button to select jobs from your dbt Cloud project.

Set a schedule, either integrated - meaning dbt Cloud jobs run upon completion of upstream connector(s) - or custom if you want transformations to execute on fixed intervals or at fixed times. If you choose integrated, you then select which connectors’ successful data load you want to trigger the job run.

Once the transformation is live, you can monitor it for success using our run logs

Don’t just take our word for it. Experience the power of automated end-to-end data integration for yourself with a free trial.

Verwandte Beiträge

Why we were named dbt Labs™  Technology Partner of the Year
Company news

Why we were named dbt Labs™ Technology Partner of the Year

Beitrag lesen
Fivetran & dbt: The essential duo for modern analytics
Data insights

Fivetran & dbt: The essential duo for modern analytics

Beitrag lesen
Introducing Fivetran Transformations for dbt Core
Product

Introducing Fivetran Transformations for dbt Core

Beitrag lesen
No items found.
Insights from BBC Studios, Monzo Bank at Fivetran's London event
Blog

Insights from BBC Studios, Monzo Bank at Fivetran's London event

Beitrag lesen
Fivetran Product Update: July 2024
Blog

Fivetran Product Update: July 2024

Beitrag lesen
How to use Fivetran and Snowflake to move data and innovate
Blog

How to use Fivetran and Snowflake to move data and innovate

Beitrag lesen

Kostenlos starten

Schließen auch Sie sich den Tausenden von Unternehmen an, die ihre Daten mithilfe von Fivetran zentralisieren und transformieren.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.