5 more things Fivetran does with your files

Part 1 in this series covered the structural intelligence Fivetran brings to file ingestion. But enterprise file-based data is also operationally complex in ways that go beyond schema and type management. Files don't only live in cloud storage buckets; they live on internal file servers and departmental shared drives. They arrive in a wide variety of formats and contain unstructured content as well as tabular data. They are also produced by legacy systems with non-standard formats that predate the CSV era by decades.

These 5 capabilities address the operational realities that make file ingestion genuinely complete, not just for clean, cloud-native data, but for the messy data landscape enterprises actually inhabit.

[CTA_MODULE]

1. Filtering

Not all data in a file should reach the destination. A vendor export might include data for all customers globally, but your analytics environment only needs data for one region. A financial file might include test transactions alongside live ones, but loading test data into a production analytics environment causes real reporting errors.

Fivetran's filtering capability lets you define declarative rules that restrict which rows are loaded, based on column values. Filters are evaluated before data reaches the destination, so filtered rows consume no destination compute or storage. This is architecturally cleaner than filtering in transformation because when filtering happens at ingestion, the cost of loading unwanted data never materializes.

2. Hybrid deployment

The modern data stack narrative assumes data lives in cloud storage. In many organizations, significant file volumes still live on-premises: Windows file servers, SFTP servers in the data center, network-attached storage shared by departmental teams, or local machines that receive file drops from external partners. These aren't legacy problems that will disappear; they reflect real operational and compliance constraints.

Fivetran's Hybrid Deployment uses a local read agent that runs within the organization's network, reads files from local or network-accessible locations, and forwards data securely to Fivetran for processing. Once the agent is deployed, the connector appears in the Fivetran dashboard alongside all cloud connectors — same scheduling, same monitoring, same schema management. The fact that the file lives on a network share is an implementation detail invisible to downstream consumers.

3. Multiple structured file formats

'Files' is not a monolithic category. CSV, JSON Lines, Parquet, Avro, ORC, and XML each represent data differently, compress differently, and are suited to different use cases. A file ingestion platform that handles only 1 or 2 forces upstream format conversion, adding cost, complexity, and latency to every pipeline it supports.

Fivetran supports the full range: CSV with configurable delimiters and encoding options (compressed with gzip, zip, bz2, or zstd), JSON Lines for event logs and API dumps, Parquet and Avro with their embedded schemas for schema-on-read lake workflows, ORC for Hadoop-ecosystem sources, and XML for ERP and financial system exports. For binary columnar formats, the embedded schema means Fivetran skips the inference step entirely because the types are already declared in the file.

4. Unstructured Data

Traditional data integration handled structured data — rows and columns. AI has changed the picture. LLMs consume documents and text. Recommendation systems ingest large volumes of unstructured content. Compliance programs require that contracts, invoices, and regulatory submissions be retained and searchable alongside the structured data that references them. None of this fits in a relational table.

Fivetran's unstructured data capability ingests binary and document files into the destination alongside structured data, with metadata columns capturing file type, size, source path, and ingestion timestamp. This makes unstructured files first-class citizens in the data platform — with the same lineage, access controls, and observability as structured data, rather than living in a separate, ungoverned file store.

5. Easily extensible

Fivetran's file connectors are primarily designed for rectangular tabular data — files where every row has the same columns and a consistent delimiter. But enterprise environments contain a long tail of non-standard formats that remain stubbornly in use: EDI transaction sets in retail and logistics, HL7 messages in healthcare, fixed-width records from mainframe financial systems. These formats predate the CSV era and aren't going away.

The Connector SDK and custom parsing logic within Fivetran's file ingestion framework let teams handle these formats without a separate pipeline. EDI files can be parsed into normalized tables by segment type with parent-child keys linking hierarchy levels. HL7 messages can be parsed by segment into destination columns. Fixed-width records can be mapped by character position. In every case, the result runs on Fivetran's managed infrastructure — not as a separately maintained custom script.

10 capabilities, one platform

Across both parts in this series, these 10 capabilities cover the full spectrum of what enterprise file ingestion actually requires — from schema inference and type detection to hybrid deployment, multi-format support, and extensibility for the legacy formats that aren't going away. Fivetran handles the complexity so your team can handle the analysis.

[CTA_MODULE]