Programmatic control over data operations is an essential capability for data teams. As an organization’s data needs grow in scale and complexity over time and data teams pursue a wider variety of analytics and data-driven products, manual construction, maintenance and administration become increasingly untenable. Teams must also be able to integrate their data operations with additional tools and technologies. 

Sorenson Communications is a leading provider of captioning and interpretation services for the hard-of-hearing and deaf, with the mission to make communication accessible and clear regardless of signed or spoken language. Automated, real-time translation of all kinds depends heavily on natural language processing and the data used to train it.

Travis Smith, Senior Manager of Data Engineering at Sorenson, led the company’s efforts to better offer its customers real-time, data-dependent services. Sorenson needed to improve the speed and responsiveness of its data integration. Fivetran was integral to this effort, which included migrating to the cloud and modernizing its infrastructure. According to Smith:

“Fivetran came and saved the day for us. We had some changes in our data structure and a really fast go-to-market planned with a new CRM, and without Fivetran there was no real way to grab the data without a lot of custom development.” 

The data team faced challenges such as the time required to build and maintain data connectors as well as difficult, obscure bugs resulting from corner cases. The team also had to cope with the sheer diversity of its data sources. Ultimately, the data team connected over 50 database sources through Fivetran.

The Fivetran REST API, with its support for automated scripting using Python and the multitude of robust off-the-shelf scripts from the open-source framework, offered many important capabilities to Sorenson. Some of the most vexing problems are the simplest; their SQL Server certification rotation previously required periodic manual revalidation, typically prompted by a sudden loss of access. The data team replaced this process with a script that monitors and, if necessary, rotates certificates every ten minutes.

Smith used the Fivetran REST API jumpstarted efforts to systematize his team’s entire data operation from front to back, making procedures visible and repeatable. This was further accelerated with the help of off-the-shelf software packages such as the Python framework for the REST API as well as off-the-shelf, analytics-ready data models. The team was able to develop their infrastructure using CI/CD and programmatically create and edit connectors to a wide range of sources, including test and development environments. Moreover, using the Fivetran Platform Connector, the team was able to extract log data. By combining log data with the average time in minutes it would have taken to manually manage each event, the team was able to quantify the savings in engineering hours with a simple formula: SUM(events * avg_time/60).

By leveraging automation along with better observability and monitoring, Sorenson’s data team was able to considerably improve its SLA. As Smith remarked:

“Good data engineering is when I push a button to make something happen. Great data engineering is when you schedule automations that run whenever they need to and I don’t even need to click the button. That’s Fivetran.” 

All the Sorenson data team needed to contribute was a configuration file. The most time- and labor-intensive part of the process involved going into the portals for data sources to retrieve the necessary credentials.

Freed from day-to-day pipeline management by the API framework, sample scripts and automation, Smith’s team was able to set and forget their data integration and direct resources to new AI initiatives.

Fivetran REST API brings programmatic control to data integration

Programmatic control is central to performing administrative tasks at scale, integrating other tools and technologies into a data operations ecosystem and building a sustainable foundation for the future expansion of analytics and data products.

The Fivetran REST API and its accompanying Python framework enable programmatic control over important aspects of the Fivetran application, including:

• Certificate management – such as validating transport layer security (TLS) certificates
• Connector management – viewing, creating, editing, removing and running connectors and their schema configuration files
• Transformation orchestration – viewing, creating, editing, removing and running sequences of transformations 
• Destination management –  creating, editing and removing destinations
• Team management – viewing, editing and deleting teams 
• User management – viewing, inviting, editing and deleting users
• Webhook management – viewing, creating, editing, removing and testing webhooks

In addition, the REST API supports monitoring of lists and groups. The following diagram illustrates how all of these entities relate to each other: 

To experience the power of the Fivetran REST API and see how it can accelerate your data operations for yourself, consider a free 14-day trial or demo.