Original article published on SAP Insider here.

Organizations need data freedom and the ability to access and gain insights from their data whenever needed. Extracting data from SAP systems can uncover valuable insights into production processes, purchase processes, maintenance of equipment, stock optimization and for improving processes and tracking financial performance. However, pulling SAP data, often stored on-premises, is challenging due to complex configurations, strict security protocols and unique data types. Additionally, the extensive number of tables makes it difficult to identify valuable data and convert large datasets into actionable insights.

Refactoring SAP logic for new destinations

One of the frequent challenges encountered by organizations is the handling of Z transactions and the refactoring of business logic coupled with additional or different logic for existing business processes. This issue becomes particularly pressing when there is a need to replicate data from SAP to other systems. Z transactions, which are custom-developed functions or packages within SAP, play a crucial role in organizing and visualizing underlying raw data at the application layer. While Z transactions can populate custom objects like columns and/or tables, accessing and transferring this data can be complex and demanding.

Organizations often find that the logic developed within SAP is tightly coupled with its proprietary ABAP programming language. This presents a significant obstacle for use in different environments, such as data warehouses or data lakes, which typically rely on SQL. SAP specialists need to either create a new CDS view or an SAP extractor to retrieve the data. This introduces complexities, such as determining the delta. Fivetran, however, natively handles delta determination without requiring any effort or knowledge of the underlying SAP system. CDS views can be replaced by extracting the underlying tables and building the view in the data lake or cloud data warehouse.

Fivetran addresses the challenges of integrating SAP systems by leveraging a log-based change data capture, ensuring low overhead on SAP systems. Unlike traditional methods that query the database or application layer, Fivetran’s approach incurs minimal CPU usage (around 3.5% to 5%). This low overhead ensures that data capture is both efficient and reliable. While trigger-based solutions work by selecting data and capturing a snapshot at the moment of replication, log-based solutions capture every transaction and ensure transactional consistency. This method enables consistent and automated data capture with minimal impact on system performance. By facilitating the replication of both transactional data and application layer views (CDS), Fivetran ensures that organizations can continue their operations without disruption while planning for a more sustainable long-term solution.

Addressing SAP scalability challenges

Custom logic developed within an SAP system often faces scalability issues, causing significant challenges for organizations. Scalability issues often arise from the lack of available SAP data engineers needed to build custom extractors and CDS views. Creating custom objects typically requires significant effort from SAP data engineers. These challenges arise from the transactional nature of SAP systems, where extensive local logic imposes an additional computational load, leading to latencies and limited transformation frequencies.

While the SAP skill set can be leveraged to develop this on the new platform, complemented by training SQL developers and data engineers, organizations can also opt for additional scalable solutions for handling their SAP logic.

With Fivetran, extracting custom objects becomes effortless. The logic can be developed in cloud data warehouses and data lakes, where a larger pool of skilled data engineers is accessible. Fivetran’s solution addresses these scalability issues, guiding customers to understand that while their current approach may work temporarily, it is not viable in the long run. Fivetran enables them to transition to a more sustainable, scalable system, ensuring they can manage their data efficiently and effectively.

Fivetran is a cloud-based ELT platform that simplifies replicating SAP data to a chosen cloud destination and automates data replication, making it easier to extract data from SAP systems and load it efficiently. From handling large volumes of data with pre-built connectors for SAP, managing its multiple data formats and complex data models, Fivetran’s solutions reduce the time and resources needed for data extraction and replication and eliminates the need for custom integrations. By using Fivetran, organizations can quickly transfer their data to the cloud, accelerating the process of deriving insights and making informed business decisions.

To experience Fivetran for yourself, consider a demo or a free 14-day trial.