We're excited to announce that the Fivetran Connector SDK is now in beta and publicly available to customers. This powerful tool allows data teams to seamlessly build data pipelines for sources not yet currently supported in the Fivetran connector catalog, like internal trading platforms or patient record systems. 

This capability makes Fivetran your platform for unlocking valuable insights from any data source to any destination. 

[CTA_MODULE]

What is the Fivetran Connector SDK?

The Connector SDK empowers data teams to create custom data pipelines by writing Python code within our secure cloud environment. Using familiar coding patterns, libraries and tools, developers can efficiently send data to their destination of choice — all while benefiting from our robust infrastructure.

Here are key use cases for the Fivetran Connector SDK:

• Connecting niche or unique proprietary sources
• Customizing data structures, filters or handling sensitive data 
• Integrating sources without a native Fivetran connector, such as unsupported APIs, files and databases
• Preprocessing data, including tasks like decryption
• Extending or augmenting existing Fivetran connectors 

Why is this a big deal?

Until now, data teams lacked a way to combine custom, DIY data pipelines with critical Fivetran features, such as idempotence, re-syncs and failure recovery. The Connector SDK eliminates these pain points, allowing teams to focus on extracting insights rather than creating the data writer or handling retries when the network blips. Furthermore, our service eliminates the need to set up, deploy and manage cloud infrastructure to execute your custom code.

The Connector SDK offers flexibility and control over historical and incremental syncs, delete capture, state management, multithreading data extraction, the ability to build a schema optimized for your needs and more. Streamline the process of obtaining data from unique sources into just a few steps, all while making maintaining data integrity easy. 

Key advantages of the Connector SDK:

• Get to production faster: Eliminate dependencies on internal DevOps or DIY infrastructure. Build and deploy faster.
• Fully managed platform: Harness sophisticated capabilities such as fully coded and maintained destination writers, connector-level resyncs, orchestration, automatic network retry and more. Fivetran can even automatically infer the schema. 
• Enterprise-grade security and compliance features: Leverage granular access controls, automated user provisioning, sensitive column-level data blocking, hashing and Metadata APIs. Fivetran has built in encryption for data in rest and transit for all connectors. In addition, Fivetran is certified for SOC 1 & SOC 2, ISO 27001, HITRUST and more.

How it works

Getting started with the Connector SDK is simple:

1. Install the SDK using the Python standard pip install command
2. Build your connector
   1. As much or as little Python as your data connection needs 
   2. Fast local testing using SQL queries to validate your schema 
3. Deploy your connector to Fivetran from the command line and sync to load data to your destination. 

The simplified process not only significantly reduces engineering time and expedites the process but also empowers more team members to build connections, thanks to Python’s ease of use and accessibility. Once deployed, your custom connectors function like any native Fivetran connector — complete with sync management, scheduling and compute resource management. You’ll also get full visibility into sync status, usage, logs and error handling through the Fivetran dashboard.

Sync status, usage, errors, warnings and more are all visible in the dashboard, while logs are available in the Fivetran Platform Connector. 

The Connector SDK calculates usage just like any other Fivetran connector.

‍“We chose the Fivetran Connector SDK because there wasn’t a native connector for Accelo, our CRM and project management system. As an agency managing a variety of data sources, we needed a way to extract, store and archive Accelo data to support advanced reporting and ensure a smooth migration. The Fivetran Connector SDK provided a flexible, easy-to-use solution, allowing us to build the connector ourselves and customize it to our exact needs.” 

— Ahmed Zedan, VP Technology Solutions, Izba Consulting

Fivetran Connector SDK vs Functions

While Functions remains a flexible option for developers needing non-Python languages, the Connector SDK offers a simpler and more efficient approach.

Instead of writing code that is centered around constructing and delivering data as a JSON object, developers can use a simplified coding approach, sending data row-by-row with UPSERT, UPDATE and DELETE operations. This is most clearly seen in the approach to paginating through large data sets, which in Functions had to be coordinated with our infrastructure and can now be cleanly handled directly in your connector.py code.

Furthermore, the Connector SDK comes with a test tool allowing you to easily and locally run your code and receive feedback quickly. You can see exactly what data will be delivered to your warehouse and its schema and use SQL to test your queries and use of the data.

How to get started 

Customers can access the Connector SDK through our documentation. No need for setup forms — deploy connections directly from your Python IDE using the Fivetran deploy command.

Our team of Professional Services experts is available to advise you as you get your first Connector SDK connection set up for your business, free of charge. File a support ticket to get started. 

Some Python libraries may require external drivers. If you need additional packages, work with your Account Manager to install them.

What’s next?

We plan to introduce even more features, including:

• Log messages: Add custom logs viewable in the dashboard.
• Dashboard configuration edits: Update your connection settings directly from the UI.

[CTA_MODULE]

‍