I’m thrilled to announce Fivetran has been officially inducted into Prefect’s Premier Partnership program last month, with Prefect’s Head of Partnerships Andrew Black commenting “Our joint users love Fivetran’s simplicity and dependability where predefined app-to-app ELT is what’s called for” in this blog post. 

Prefect is a dataflow automation tool that allows a community of over 11,000 data engineers to orchestrate their modern data stacks. It uses Python to define Tasks that operate on data. Tasks can then be constructed into Flows where dependencies, parallelization and scheduling between the Tasks can be automated. Prefect can run on Prefect Cloud, or it can be installed in any Python environment with pip install Prefect.

Automated data ingestion in Prefect with FivetranSyncTask‍

Once Prefect is up and running, it’s functionality can be extended via integrations. The FivetranSyncTask, developed by Fivetran and Prefect, allows you to bring Fivetran connectors into the dataflow automation tool by starting a Fivetran connector through Prefect and waiting on the completion of its execution before Prefect can run another Task, like dbt. 

The FivetranSyncTask automates the extracting and loading processes that often kick off a dataflow. Instead of manual implementation, data engineers using the FivetranSyncTask can set up their modern data stack faster and scale quicker with the knowledge of exactly what data Fivetran is placing in their data destination, when it happens and what to do next. 

“With Fivetran and Prefect, we are able to manage our entire ELT pipeline with a single workflow,” Austen Bouza, data engineer at Carrot Fertility, told us. “This capability has allowed us to scale our analytics function quickly while being able to maintain guarantees around the recency and accuracy of metrics that are derived from multiple data sources.” 

For organizations running hundreds of dataflows, it can be difficult to manage all the code that's moving data around. When you’re managing many data sources, a simple change in a data type of a source schema can bring your dataflows offline if those sources are managed by brittle, manually written code. By offloading data integration code to Fivetran, these types of changes can be handled automatically, allowing for more time to be spent building new dataflows with the FivetranSyncTask instead of constantly fixing broken ones. 

To get started, download Prefect and check out the FivetranSyncTask.