High-volume replication for Oracle is here

Data teams are under mounting pressure to support high-volume data throughputs from busy databases. These databases, which store data ranging from ERP financial transactions and transportation logistics to inventory tracking and SaaS product backends, serve as one of the key sources for business analytics.

As enterprises track more user actions to tailor their digital experience or deploy more IoT devices, which collect data on millions of events, the large stores of data that already exist today will only continue to increase. Additionally, workloads related to AI/ML initiatives also work best with multiple inputs from different sources, with larger datasets making more effective projections and decisions in these workflows.

Based on these use cases, there will only be a greater need in the future to quickly throughput high volumes of data. But databases with high volumes are typically harder to replicate with an intraday frequency, hampering the ability of data teams to meet business users' and stakeholders' requests for fast access to data.

Introducing High Volume Agent connectors

To solve these challenges, Fivetran is launching a new line of High Volume Agent (HVA) connectors – beginning with Oracle as a source. These HVA connectors are able to minimize source load so data teams can realize faster setup times, limit maintenance and handle higher volumes and data structure changes.

With the new HVA connectors, enterprise data teams can deal with the already-high volumes they're seeing now and be assured that their pipelines can deal with broader use cases and more data in the future.

HVA connectors are most beneficial in cases of databases, like Oracle, that generate high data volumes or have a large repository of historical data. Common use cases include software product databases that track every imaginable event or databases used as ERP backends, where there might be thousands of SKUs or order fulfillments to track and monitor.

Data sheet: High volume agent connector for Oracle and SQL Server

DOWNLOAD

Built with ease-of-use and low-impact functionality

Following our acquisition of HVR, we've incorporated its support for high-volume replication with the use of an agent. Fivetran uses change data capture (CDC) methodology to only read the changes made to a database source directly from the log files.

Using an agent allows us direct read access to those log files without requiring queries to be run against the database for incremental loads. We can compress the data prior to sending to our processing servers to minimize replication latency overall and better support high transactional volumes.

Fivetran has long supported database replication from the most popular database sources to the most popular cloud destinations. Our product principles include the idea of ease of use, from implementation to maintenance, which is reflected in our database replication features.

We've reduced implementation times from months to days with features such as schema mapping and creation based on schema information available in the source database and by working with our partners — such as Snowflake, Google BigQuery and Databricks — to ensure compatibility with destination data types and formats. Our connectors incorporate best practices such as log-based CDC to capture incremental updates, including schema drift events, which we automatically apply to the destination to ensure you're always making decisions based on the most up-to-date information.

Will HVA connectors be separate from the existing Fivetran connectors?

Yes, these will be brand new connectors. The existing database connectors (e.g., Fivetran's Oracle connector) will still be a great option for customers who want to use a "remote capture" approach rather than an agent-based approach.

The key focus of Fivetran has always been getting you access to the key data you need to make decisions for your business without heavy personnel investment typically associated with building and maintaining your data pipelines. Our new HVA connectors for Oracle databases are another important step in our journey to be the best fully-managed CDC replication data solution.