Many enterprises rely on self-hosted data pipelines to effectively and securely move data from various sources into their cloud data warehouses and lakes. However, doing this at scale for existing use cases can be tough, let alone planning for future needs. The more data sources an organization utilizes, the more complicated and burdensome it becomes to set up, maintain, and update pipelines as changes are made.

That’s why organizations turn to containers to help them scale. Containers offer data engineering teams a lightweight, portable, and reproducible way to implement and scale their data pipelines. They allow teams to scale while dramatically improving the performance, reliability, and maintainability of data pipelines. Those advantages include:

• Managing environment dependencies, configurations, and runtimes
• Scaling computing resources by isolating environments on their machines
• Simplifying deployment by replicating data pipeline stacks
• And more

Popular containerization software, such as Kubernetes, integrates with some of the most popular data pipeline tools and provides automation, scalability, and high availability. Kubernetes simplifies the management of complex data pipeline workflows, optimizes compute usage, and enhances fault tolerance, making it easier to scale data pipelines effectively as data volumes grow. For these reasons, we are excited to announce that Fivetran’s Hybrid Deployment can now be implemented using Kubernetes. 

[CTA_MODULE]

Hybrid Deployment is a free feature in Fivetran’s Enterprise and Business Critical plans. It lets security-conscious enterprises run chosen data pipelines in their own environment or VPC from our easy-to-use UI. By running the data plane in your environment, you ensure no data ever leaves control. Unlike other on-premise data integration solutions, our pipelines are set up and maintained from Fivetran’s UI, updated, and supported by our robust team of engineers and support agents. With Kubernetes, scaling your Hybrid Deployment pipelines becomes easy. Kubernetes lets you run as many connectors on your Hybrid Deployment agent as your disc space and memory permit. This means you can initialize one agent and run all of the pipelines you need today while giving you the flexibility to add more resources as needed to meet future plans.With Hybrid Deploy and Kubernetes, you can:

• Automatically manage and scale your connector syncs
• Keep sensitive data secure in your environment
• Use Fivetran’s robust security features including data hashing, RBAC, and more
• Move data from 100s of sources to your most desired destinations.

Our Kubernetes support is available for AWS, Google Cloud, Azure, and local Kubernetes deployments. 

[CTA_MODULE]