As a solution architect, I am always looking for faster ways to build connectors. I previously wrote about my experiences leveraging Cursor AI to accelerate the development of a Fivetran Connector SDK solution. Compared to Cursor AI, a full-fledged IDE, Claude Code is CLI-first and editor agnostic. Claude Code also offers a larger context window, enabling it to better handle more complex projects.
[CTA_MODULE]
Together, Fivetran Connector SDK and Claude Code bring critical attributes to the table.
Fivetran Connector SDK offers:
- Standardized framework: Consistent patterns for building enterprise-grade connectors
- Built-in Fivetran capabilities: State management, error handling, and in-flight transformations
- Production-readiness: Designed for enterprise-scale data pipelines with monitoring and observability
- Flexibility: Supports authenticated APIs, databases, webhooks, and any Python-accessible data source
Claude Code amplifies these capabilities through:
- Repository-aware AI: Full understanding of SDK examples, patterns, and best practices
- Claude 4.5 Sonnet: Advanced reasoning for complex connector logic and edge case handling
- Contextual awareness: Understanding of entire codebases, not just individual files
- Iterative refinement: Intelligent code generation that learns from your feedback
- Pair programming experience: Natural language conversations that guide implementation
This combination transforms connector development from a multi-day engineering task into a collaborative 30-minute session. This isn’t just automation; it’s intelligent collaboration between developers and AI, where Claude Code serves as an expert pair-programming partner that understands enterprise best practices, derived from SDK examples and system instructions.
The development workflow for Claude Code and the Fivetran Connector SDK consists of four simple steps:
- Build a foundation by cloning and opening a repository
- Give an initial, comprehensive prompt to the AI
- Test, iterate, and validate
- Deploy the connector
Step 1: Build a foundation by cloning and opening a repository
First, I cloned the Fivetran Connector SDK repository and opened it in Claude Code. The repository structure provides essential template files, including the base connector.py, configuration.json, and requirements.txt, along with embedded SDK documentation showcasing best practices and patterns. Most importantly, the sample connectors offer reference implementations for common use cases that Claude Code could learn from.
By opening the workspace in Claude Code, Claude 4.5 Sonnet gained immediate access to:
- Real-world connector examples from the
/examplesdirectory
- Best practice patterns embedded in SDK documentation
- Modern SDK patterns using direct operation calls (no yield required)
- Configuration schemas and validation patterns
- Error handling strategies and retry logic implementations
Step 2: Give an initial, comprehensive prompt to the AI
With the SDK repository open in Claude Code, I crafted a comprehensive prompt that leveraged the available context. The key was to be specific about requirements while allowing Claude Code to reference the SDK examples for implementation patterns.
My prompt strategy consists of the following:
- Describe the data source: API endpoint structure, authentication method, data format
- Specify sync requirements: Incremental syncing, pagination strategy, rate limiting needs
- Reference SDK patterns: "Follow the modern SDK patterns from the examples directory"
- Define success criteria: What the connector should accomplish, edge cases to handle
- Provide source context: What should Claude know about the data source to effectively replicate data. (notes.md)
Here's the original prompt that Claude used to create this connector:
"Create a Fivetran Connector SDK solution for the FDA Drug API that includes:
- Dynamic endpoint discovery (automatically finds available endpoints)
- Incremental syncing using date fields
- Configurable data flattening (flattened tables vs child tables)
- Rate limiting and quota management
- Support for both authenticated and unauthenticated access
- Robust state management for reliable incremental syncs
- Logical processing of data based on context in @notes.md, which contains pertinent information about this API
- Proper upserts based off of @fields.yaml, which contains response data structure
- Follow the best practices outlined in @CLAUDE.md
The connector must work with the FDA Drug API endpoints: NDC (National Drug Code), adverse events, drug labeling, and enforcement reports."
Claude Code’s ability to read and understand the contents of the repository meant that it automatically knew:
- How to structure the connector.py file following SDK conventions
- Modern operation patterns using direct calls (no yield required)
- Proper configuration.json schema with string-only values
- Error handling and retry logic patterns from SDK examples
- State management and checkpointing best practices
Start iterating with Claude 4.5 Sonnet
My initial prompt leveraged the cloned repository context and specified requirements for a production-ready connector, including incremental syncing, configurable batch processing with pagination, rate-limiting management, automatic JSON flattening for nested structures, ISO 8601 date normalization, retry logic with exponential backoff, and comprehensive state management for reliable incremental syncs. This context was stored in the @notes.md file. The prompt emphasized support for both authenticated and unauthenticated access with configurable batch limits for development and production environments.
The cloned repository provided Claude Code with concrete examples, enabling it to generate code that matched established SDK patterns rather than generic implementations. This included the modern approach of using direct operation calls without yield statements, making the code easier to understand and adopt.
The end result was an AI-generated connector code ready for testing, further iteration, and validation.
Step 3: Testing, iterating, and validating
The connector Claude generated wasn't just functional code, it was an implementation following Fivetran Connector SDK best practices. Claude Code's understanding of the repository examples meant it automatically applied patterns that would typically require deep knowledge (see the appendix at the end for more details). This simplified testing, iteration, and validation of the connector.
Testing the AI-generated connector
I ran the following locally via the CLI:
Install dependencies
pip install -r requirements.txt
Debug locally with sample data
fivetran debug --configuration configuration.jsonThe results:
- Successfully fetched data from API
- Flattened complex nested JSON structures
- Handled pagination across X pages
- Implemented proper rate limiting (verified with API logs)
- State management preserved across interruptions
- Data types correctly mapped to destination schema
Iterating, validating, and refining
The validation process was iterative and dialectical. At each step, I spoke with Claude Code to complete the connector. I did not treat the output as a black box; I still reviewed things manually, specifically:
- Error handling paths: Do all exceptions get caught appropriately?
- State management: Are checkpoints atomic and recoverable?
- Data transformations: Does flattening preserve data integrity?
- Performance implications: Are there N+1 query patterns?
This is a critical exercise in legibility anyway, because code must be maintainable by teams, not just the original developer.
The initial prompt produced a roughly 80% complete solution based on SDK patterns. I reviewed fivetran debug and discovered a pagination edge case, to which I asked Claude Code: "The connector stops when the API returns an empty cursor. How should we handle this?" Claude Code updated the pagination logic based on SDK patterns.
This resulted in a 95% complete connector. Further tests revealed, through the logs, a rate-limiting issue. I shared the log output with Claude Code, which led Claude to update the retry logic with exponential backoff and retry-after-header support.
Finally, I reviewed the generated DuckDB output and confirmed data types matched destination requirements. I determined that the connector was valid and ready for production deployment.
Step 4: Deploy the connector
Claude Code deployed the connector using the Fivetran deploy command. From start to finish, this process took about 25 minutes to produce a live connector actively syncing and processing data.
AI-assisted development is production-ready
This experience gave further evidence that AI-assisted development with Claude Code isn't just about rapid prototyping—it's about delivering production-grade solutions faster through intelligent pattern matching and repository understanding. Instead of building connectors from scratch, we can now collaborate with AI to compose proven patterns into new solutions. The developer's role evolves from writing every line to architecting solutions and validating AI-generated implementations.
What's next?
I'm currently exploring the following topics:
- Automated test generation: Using Claude Code to generate comprehensive test suites
- Performance optimization: AI-driven batch size and rate limit tuning
- Multi-connector deployments: Template-based connector generation for similar APIs
- Transformation pipelines: Generating dbt models for post-load transformations
[CTA_MODULE]
Start building production-ready connectors in under 30 minutes. Clone the Connector SDK repository, open it in Claude Code, and experience AI-assisted programming for data engineering today.
Resources:
