Fivetran enabled the Enterprise Information Management team to replicate data in real-time from multiple sources into Snowflake to power business intelligence dashboards, giving them the ability to perform analyses across different teams from a single source of truth without impacting database performance. Replication through Fivetran reduced ETL batch loads in one instance from 31 hours to under 2 hours, another from days to under 1 hour.
- Database sources: SAP, Oracle
- Destination: Snowflake on AWS
- Cloud Platform: AWS
- Use Cases: Real-time data warehousing, Enterprise data integration, Data warehouse / data lake
Pitney Bowes Inc., a technology company founded in 1920 and headquartered in Stamford, Connecticut, provides global commerce solutions. Through its global eCommerce, Presort Services, and SendTech Solutions segments, the company provides eCommerce, shipping, mailing, and software solutions that help its clients succeed by simplifying the complexities of commerce and enabling billions of transactions around the world.
Challenge
The Enterprise Information Management team supports the back office with analytical reporting solutions that run in ETL batch mode. The team’s legacy ETL system extracted data from an Oracle Leasing application into Snowflake for analytical sales reporting. This solution no longer met the needs of Pitney Bowes as batch ETL processes for end-of-month reporting were taking up to 31 hours to reconcile. The team faced challenges identifying changed data due to the lack of tracking dates in the data, resulting in needless full extracts.
Also, to accommodate new reporting requirements, additional new very large tables needed to be incorporated into the integration processes. The current system was not scalable to meet this growing demand, so the team began to search for a Change Data Capture (CDC) tool.
In addition, the Sales Reporting organization required nightly extracts of SAP data to provide a timely start of business reporting. However, the high volume of daily data integration from SAP to Snowflake required a more powerful tool. More and more business organizations were requesting SAP data for dashboards and analytical reporting, which were impossible to satisfy using legacy ETL processes.
Solution
The Enterprise Information Management team uses Fivetran to sync, validate, and replicate data from Oracle and SAP sources into Snowflake’s cloud based data warehouse four times daily to power end-of-month analytical reporting for finance and sales.
With Fivetran and Snowflake, business users have access to timely SAP and Oracle Leasing data as a single source of truth in an enterprise database without impacting the source applications.
Fivetran’s real-time replication into Snowflake has provided the Information Management team significant time savings. The team replicates data through Snowflake and applies changes throughout the day, therefore the monthly ETL reconciliation that previously took 31 hours to complete is now completed in under 2 hours.
Fivetran and Snowflake also provide the team significantly faster processing times. Replication occurs in near real time so a separate ETL job that previously took days now occurs in less than an hour, and allows the EIM team to meet the increasing requests for Oracle Leasing and SAP data.
Why Fivetran and Snowflake
When reviewing replication and warehousing solutions, Pitney Bowes valued the flexibility, speed, scalability, and support that Fivetran offers — in addition to the seamless integration with the Snowflake data warehouse. Further, Pitney Bowes had difficulties finding a solution that was compatible with SAP, where Fivetran has a feature set that allows users to easily replicate from SAP.
Another key differentiator was the use of low-latency log-based CDC rather than trigger-based CDC replication, which enables the movement of large volumes of data without affecting the core systems. In addition, most of the replication management is handled by the Fivetran interface itself, in contrast to the manual oversight required for batch processing.
Additionally, Pitney Bowes found the sales, solutions engineering, and customer support teams to be extremely helpful, responsive, and knowledgeable.
.png)
.svg)
.svg)
.svg)