For a demo of using a cloud function to sync Justin Bieber's Twitter feed into a data warehouse, check out this webinar.
At Fivetran, our core product is a set of automated connectors that sync all your company's data into your data warehouse with zero configuration and minimal setup. This strategy works well for standard data sources: everything from Salesforce to MySQL to S3 can be synced to your data warehouse in one click. We've spent years building great connectors that capture incremental changes accurately from each source and deliver great schemas to your data warehouse.
But there are some data sources where our strategy of building standardized connectors doesn't work:
- Custom APIs
- Obscure APIs that have only a few users
- Data formats that are not self-describing, like Protobuf
How can Fivetran support custom data sources like these? Our answer is our cloud-functions connector. Here's how it works:
- You write a tiny function that fetches data from the custom source
- Host this function on a serverless computing platform: Google Cloud Functions, AWS Lambda or Azure Functions
- Fivetran calls your function every 15 minutes to fetch new data
- Fivetran deduplicates this data and merges it into your warehouse
To see how this works, let's walk through a simple example. We're going to pull data from Justin Bieber's twitter feed; this example is a little silly, but it contains just enough complexity to make a good demo.
Every Fivetran request follows this format: