It has been just over a year since the addition of the FivetranSyncTask to Prefect’s task library, which allowed Prefect developers to bring Fivetran connectors into their Prefect 1.x dataflows. Prefect 2 now has collections, which group Tasks and Flows that interact with a specific service or resource in the modern data stack such as dbt or Snowflake. Today, I’m excited to share the Fivetran Prefect Collection.
With Fivetran and Prefect, developers can now build more complex and automated responses to Fivetran data syncs and unify schedules across tools that make up an organization’s modern data stack.
The collection offers much more than what’s been previously possible with the FivetranSyncTask. Everything in the Fivetran Prefect Collection runs asynchronously. Many data flows in Prefect wait until Fivetran is finished placing data in its destination before continuing downstream with more work. This means more flows can run in parallel and Prefect can execute other tasks while it is waiting for Fivetran to finish. The Fivetran Prefect Collection is currently the only way to monitor Fivetran connectors in a data orchestration tool asynchronously. Unit tests for the collection also provide coverage that was not previously available for the FivetranSyncTask.
The new collection provides a variety of tasks and flows that improve on the flexibility of Fivetran’s integration with Prefect (you can read more on the components of the Fivetran Prefect Collection in our deep dive here). This allows new tasks and flows to be added if there is something that you need that is not in the collection already. Each new Task and Flow available in the Fivetran Prefect Collection also contains an example of its use, which makes the path to your first Prefect flow using Fivetran seamless.
With the collection, just one pre-written line of code for each data source is all that is needed to orchestrate automated data ingestion effectively. This allows more time to be spent on the other tasks that may make up a Prefect flow. Emerson Franks, Data Engineering Lead at RecRoom, tells us:,
“With the Fivetran Prefect Collection, I was able to have a flow up triggering our syncs in about 15 minutes. Having Prefect and Fivetran working together has made my development process much faster and has led to a great experience in the early goings of this project. I can’t emphasize enough how nice it is to not have to write my own provider for my Fivetran interactions. It’s even better knowing that I don’t have to spend extraneous amounts of time digging through the code to ensure that there aren’t things in there that shouldn’t be.”
You can read more on our conversation about the collection with Emerson here.
The Fivetran Prefect Collection was made in collaboration with Alexander Streed and the Prefect integrations team. It is maintained by Fivetran under a Apache License 2.0.
If you would like to contribute to the collection, check it out here. The Fivetran Prefect Collection can be added to your Prefect environment by running pip install prefect-fivetran.
%20(1).png)
.png)
.svg)
.svg)
.svg)