Three ways Fivetran improves development efficiency

Fivetran launches Terraform, Webhooks and Transformations APIs.

The Fivetran API enables developers and engineers to leverage Fivetran’s automation and platform extensibility within their own devops workflows. With our newest releases, it’s that much easier to optimize.  

We’re excited to announce the release of our Terraform provider into general availability, which saves hours of work for developers throughout their infrastructure development and management processes. 

Next, our Webhooks API is also entering general availability, enabling downstream trigger-based automation and communication without the need for constant API polling. 

And lastly, our new Transformations API is now in beta and unlocks the ability to programmatically set up full end-to-end pipelines. 

We’re working hard to build and improve our API services, so let’s jump in to discuss each release, their impact and what comes next.

Terraform provider

Our Terraform provider allows mature devops users who are all in on infrastructure-as-a-code to take advantage of Terrafom’s benefits like version control, reproducibility, automation and collaboration. Many users also utilize Terraform to move connectors between data environments – leveraging staging destination groups to prototype new connectors before pushing them to production. 

Release stage: General Availability

Recent improvements:

  • We now handle connector creating and scheduling as two separate parts of the config, so you can create and customize the schema in a single apply
  • We can handle Fivetran’s 300+ connector types more intuitively 
  • We automated adding new connector types based on our OpenAPI definition - eliminating the lag between a new connector launch and the new connector’s support in Terraform  

Future roadmap: While we feel confident in our data model and how we’ve built Terraform, we still don’t have support for all of our API resources yet - but we’re continuing work here. Keep an eye on our changelog as we expand our support for the rest of our API in Terraform. 

[CTA_MODULE]

Webhooks API 

Our Webhooks API is extremely popular with our customers looking to create trigger-based automations. We have more than 200 users taking advantage of our webhooks and millions of webhooks sent per week. 

From triggering dbt Transformations to automated Slack messages, our webhooks allow you to set off downstream changes right when they happen. We’ve invested a ton of time into making sure our Webhooks are resilient, scaleable and have the behavior you expect. With all the work we’ve done, users are able to avoid polling our API entirely and focus on a fully automated, event-driven pipeline. 

Release stage: General Availability

Recent improvements:

  • We added exponential backoff retries
  • We added multi-threaded processing to ensure low latency
  • We automated the disabling of dead events
  • We added sample code
  • We added support for many more events including edit_connector, allowing users to trigger changes based on a connector being authed and having captured its schema

Future roadmap: We’ll support even more events in the future, so bring us your trigger-based use cases.

[CTA_MODULE]

Transformations API

For customers managing high volumes of connectors and consistently adding new connectors with lots of dependent transformations, programmatically creating full pipelines is a requirement. After all, no one wants to do ‘clickops’ to add hundreds or thousands of transformations via the UI. Our Transformations API solves for this. 

Our Transformations API enables users to set up full end-to-end data pipelines programmatically. Simply set up a new connector, create an integrated dbt Transformation linked to your Github repo and complete your entire pipeline without breaking a sweat. 

For customers with highly scaled internal use cases or productized use cases like Powered by Fivetran, the Transformations API also allows them to deliver a fully integrated product to their end users. 

Release stage: Beta

Recent improvements:

  • We’ve refined the pathing, focusing on flatter, more intuitive pathing and full manual orchestration – run, pause and resume transformations when you need to

Future roadmap: 

  • Let us know your feedback as work towards our General Availability release! 
  • Questions on our release phase qualification? Read more about it in our docs.

[CTA_MODULE]

The bottom line 

We continue to improve our API services to add more functionality and greater efficiency for power users. While these releases optimize even more of the devops workflow, they’re only a portion of what’s possible with the Fivetran API. 

To learn more about leveraging the Fivetran API, just schedule a private demo. 

Start for free

Join the thousands of companies using Fivetran to centralize and transform their data.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Product
Product

Three ways Fivetran improves development efficiency

Three ways Fivetran improves development efficiency

August 16, 2023
August 16, 2023
Three ways Fivetran improves development efficiency
Fivetran launches Terraform, Webhooks and Transformations APIs.

The Fivetran API enables developers and engineers to leverage Fivetran’s automation and platform extensibility within their own devops workflows. With our newest releases, it’s that much easier to optimize.  

We’re excited to announce the release of our Terraform provider into general availability, which saves hours of work for developers throughout their infrastructure development and management processes. 

Next, our Webhooks API is also entering general availability, enabling downstream trigger-based automation and communication without the need for constant API polling. 

And lastly, our new Transformations API is now in beta and unlocks the ability to programmatically set up full end-to-end pipelines. 

We’re working hard to build and improve our API services, so let’s jump in to discuss each release, their impact and what comes next.

Terraform provider

Our Terraform provider allows mature devops users who are all in on infrastructure-as-a-code to take advantage of Terrafom’s benefits like version control, reproducibility, automation and collaboration. Many users also utilize Terraform to move connectors between data environments – leveraging staging destination groups to prototype new connectors before pushing them to production. 

Release stage: General Availability

Recent improvements:

  • We now handle connector creating and scheduling as two separate parts of the config, so you can create and customize the schema in a single apply
  • We can handle Fivetran’s 300+ connector types more intuitively 
  • We automated adding new connector types based on our OpenAPI definition - eliminating the lag between a new connector launch and the new connector’s support in Terraform  

Future roadmap: While we feel confident in our data model and how we’ve built Terraform, we still don’t have support for all of our API resources yet - but we’re continuing work here. Keep an eye on our changelog as we expand our support for the rest of our API in Terraform. 

[CTA_MODULE]

Webhooks API 

Our Webhooks API is extremely popular with our customers looking to create trigger-based automations. We have more than 200 users taking advantage of our webhooks and millions of webhooks sent per week. 

From triggering dbt Transformations to automated Slack messages, our webhooks allow you to set off downstream changes right when they happen. We’ve invested a ton of time into making sure our Webhooks are resilient, scaleable and have the behavior you expect. With all the work we’ve done, users are able to avoid polling our API entirely and focus on a fully automated, event-driven pipeline. 

Release stage: General Availability

Recent improvements:

  • We added exponential backoff retries
  • We added multi-threaded processing to ensure low latency
  • We automated the disabling of dead events
  • We added sample code
  • We added support for many more events including edit_connector, allowing users to trigger changes based on a connector being authed and having captured its schema

Future roadmap: We’ll support even more events in the future, so bring us your trigger-based use cases.

[CTA_MODULE]

Transformations API

For customers managing high volumes of connectors and consistently adding new connectors with lots of dependent transformations, programmatically creating full pipelines is a requirement. After all, no one wants to do ‘clickops’ to add hundreds or thousands of transformations via the UI. Our Transformations API solves for this. 

Our Transformations API enables users to set up full end-to-end data pipelines programmatically. Simply set up a new connector, create an integrated dbt Transformation linked to your Github repo and complete your entire pipeline without breaking a sweat. 

For customers with highly scaled internal use cases or productized use cases like Powered by Fivetran, the Transformations API also allows them to deliver a fully integrated product to their end users. 

Release stage: Beta

Recent improvements:

  • We’ve refined the pathing, focusing on flatter, more intuitive pathing and full manual orchestration – run, pause and resume transformations when you need to

Future roadmap: 

  • Let us know your feedback as work towards our General Availability release! 
  • Questions on our release phase qualification? Read more about it in our docs.

[CTA_MODULE]

The bottom line 

We continue to improve our API services to add more functionality and greater efficiency for power users. While these releases optimize even more of the devops workflow, they’re only a portion of what’s possible with the Fivetran API. 

To learn more about leveraging the Fivetran API, just schedule a private demo. 

Ready to learn more about the Terraform provider?
Click here
Ready to learn more about the Webhooks API?
Click here
Want to learn more about the Transformations API?
Click here
Topics
Share

Related blog posts

Terraform and the modern data stack as code
Product

Terraform and the modern data stack as code

Read post
Introducing the new Fivetran API reference
Product

Introducing the new Fivetran API reference

Read post
Modern data architecture allows you to have your cake and eat it, too
Data insights

Modern data architecture allows you to have your cake and eat it, too

Read post
No items found.
Build a data app with Streamlit, Fivetran and Snowflake
Blog

Build a data app with Streamlit, Fivetran and Snowflake

Read post
Everything you need to know about the Fivetran REST API
Blog

Everything you need to know about the Fivetran REST API

Read post
What to know about the Fivetran async provider in Airflow
Blog

What to know about the Fivetran async provider in Airflow

Read post

Start for free

Join the thousands of companies using Fivetran to centralize and transform their data.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.