Data integration is deceptively complex. Considerations include scoping and scaling infrastructure, ensuring availability, recovering from failures and maintaining the system in response to changing data sources and business needs. Many common data integration tools offer frameworks for solving these tasks but still demand a considerable degree of engineering work from end users.
The Fivetran automated data movement platform automatically, reliably and securely centralizes data from hundreds of SaaS applications, files, events and databases into the Databricks Data Intelligence Platform. By combining Fivetran and Databricks, you gain the following capabilities:
[CTA_MODULE]
How Fivetran and Databricks work together
Fivetran seamlessly integrates with the Databricks ecosystem, performing a critical role in moving data into a data lake or warehouse. Databricks provides a destination for the data and tools to assist with governance (Unity Catalog), development (Asset Bundles) and AI capabilities (AutoML and Mosaic AI).
Using Fivetran and several tools in the Databricks ecosystem, you can follow a straightforward workflow for data applications of all kinds.
For conventional analytics and business intelligence, you can:
- Use Fivetran to integrate data into the Databricks Data Intelligence Platform.
- Use Fivetran, dbt and/or Databricks SQL to transform data into analytics-ready data models.
- Use third-party business intelligence platforms and/or Databricks SQL or Databricks Notebooks to produce reports and dashboards.
For AI/ML applications, you can:
- Use Fivetran to integrate data into the Databricks Data Intelligence Platform.
- Use Fivetran, dbt and/or Databricks SQL to transform data into analytics-ready data models.
- Use AutoML or MosaicML to set up ML or RAG (retrieval-augmented generation) models.some text
- AutoML natively supports off-the-shelf machine learning models for classification, regression and forecasting, and automates tuning and testing.
- You can also embed data as vectors into the vector database of your choice with MosaicML.
- Use Databricks Asset Bundles to organize the projects for your application using the languages and libraries of your choice, reducing development overhead.
This combination of platforms and tools enables you to build the full range of analytics and AI/ML applications:
- Curated, white-glove, real-time analytics – Data science is a difficult profession and talent is scarce. You can provide "insights as a service" for customers who choose to forgo building analytics capabilities in-house.
- Data aggregation and sharing services – You may gain valuable insights by collecting data from across a market, industry, supply chain, etc.
- Personalization and recommendation – You can use behavioral, psychographic and other data for decision support.
- Automation – You can use data to trigger all kinds of automatic behavior within your organization’s operations and the goods and services you bring to market.
- Generative AI of all kinds – Generative AI has the potential to become a phenomenally powerful productivity aide as well as the backbone for a wide range of innovative products, including:
- Chatbots for customer assistance and internal helpdesks
- Copilots for accelerating the production of code, documentation and other assets
- Diagnostic services and other tools for identifying obscure patterns
- Rapid brainstorming, design and prototyping tools
In short, with Fivetran and Databricks, the world of data is your oyster. The types of data applications you can build are limited only by your imagination.
For functionality not directly supported by the Databricks ecosystem, many third-party providers partner with and seamlessly integrate with the Databricks Data Intelligence Platform.
Getting started with Fivetran and Databricks
There are two ways to get started with Fivetran:
- Through Databricks Partner Connect on your Databricks dashboard (recommended!), or
- Through fivetran.com.
Here are step-by-step instructions for getting started with Fivetran through Databricks Partner Connect:
Step one: Log in to your Databricks account.
Step two: From the Dashboard, select Partner Connect and search for Fivetran.
Step three: Select Fivetran and connect. You can choose the specifications for your Databricks destination. Click Next.
Step four: For Email, enter the email address that you want Fivetran to use to create a 14-day trial Fivetran account for you, or enter the email address for your existing Fivetran account. Click the button with the label Connect to Fivetran or Sign in. If needed, you can also review some connection details. Click Connect to Fivetran.
Step five: After activation, you will be prompted to create a password before selecting Start Free Trial. No credit card is required to sign up for the free trial and use Fivetran for 14 days, and the free trial only begins once you have successfully completed an initial sync.
Step six: After creating an account, Partner Connect brings you to the Fivetran UI, where you can add connectors and have all of your data at your fingertips. After choosing a connector, follow the on-screen instructions to set it up.
Once you have created and synced your first connector, your free 14-day trial starts. Fivetran will deliver the data you need in a timely manner and in an easy-to-query format.
To see how you can further combine Fivetran with Databricks, consider attending or watching a recording of a hands-on lab. It can take as little as a few lines of SQL and Python to start spinning up your organization’s next breakthrough.
[CTA_MODULE]