Does paying more for a coding model deliver better performance?

At Fivetran, we compared 3 OpenAI coding models in Claude Code through a name-masking proxy on our own codebase. Over two weeks, 39 engineers tested GPT 5.6 Luna, Terra, and Sol. We compared engineer preference, recorded spending, and pull request (PR) activity in their everyday workflow.

Public benchmarks measure capability on isolated tasks, but we wanted to test these models in our own repositories and tools and see whether higher-priced tiers actually deliver a better day-to-day experience than lower-cost options.

By masking model names, we could blind the study, evaluating how each tier performed in regular work without the results colored by brand familiarity or price expectations.

[CTA_MODULE]

How the study worked

From September 14–25, 2026, 39 voluntarily participating engineers were randomly assigned two models, one per week in random order, with equal representation across models. They used the models through Claude Code — their usual coding agent — and selected a preference afterward.

Engineers saw only model aliases. A LiteLLM proxy instructed models not to reveal their identities and removed model and provider names from responses. No participant reported seeing or identifying a model.

Of the 39 engineers, 35 used at least one assigned model, and 31 tested both. Four stopped early: one because Luna could not complete their tasks, one replaced Sol with Claude Sonnet, and two stopped using Terra because of setup and time constraints or repeated freezes and off-task actions. A few others missed testing time for personal reasons.

What engineers preferred

27 engineers answered the final survey, all after using both of their models. Sol came out ahead in both of its matchups: 6 of 9 engineers preferred it to Luna, and 7 of 10 preferred it to Terra. Terra edged Luna, 5 to 3.

The margins were thin, though. The survey asked engineers whether each preference was strong or weak, and almost half (12 of 27) were weak, including 4 of Sol’s 7 wins over Terra. With this few votes, none of the 3 splits is clearly more than chance.

What it cost

The models’ prices differed widely: Terra’s input rate was ten times Luna’s, and Sol’s twenty times. The study used these prices, in dollars per million tokens:

Spending rose with the prices. A typical engineer spent about $5 during their Luna week, $41 during their Terra week, and $168 during their Sol week. Averages were higher than medians because some engineers used their models much more heavily than others:

Pull requests

We also counted the PRs engineers opened during each assigned week. A typical engineer opened 5 PRs during their Luna week, 4 during their Sol week, and 2 or 3 during their Terra week. Because PR counts mostly reflect whether an engineer spent the week on a large refactor or a series of smaller changes, they don’t establish a productivity ranking on their own. Rather, they show engineers continued shipping steady work across all 3 tiers.

Price tier is a poor proxy for day-to-day value

We expected the higher-tier models, particularly Sol, to be clear winners. Instead, stripping away model names in everyday repository work showed that a 20x jump in token price, and a 30x+ jump in typical weekly spend from $5 on Luna to $168 on Sol, bought a modest edge in preference rather than a runaway victory. Sol led both of its matchups, but nearly half of the preferences were weak, and neither the survey votes nor PR activity established a decisive ranking. The practical lesson is that price tier is a poor proxy for day-to-day value: an inconclusive vote does not mean the models are equally capable, but throwing a heavyweight model at a straightforward task can actually yield a worse experience, trading speed and focus for extra steps, overbroad edits, and unnecessary cost.

In our setup, each model also handled the entire agent loop, including planning, background calls, and subagent work, at its own rates without delegating routine steps to a faster, cheaper model. When an agent took more steps before returning control to the engineer, that uniform routing compounded both cost and end-to-end execution time on Terra and Sol. We did not measure the savings or speedups that a mixed-model workflow might achieve.

The feedback we received suggests that responsiveness, planning depth, scope control, and tooling friction all shaped preference. Thorough planning paid off on large refactors, whereas fast responses and tight scope control mattered more on focused changes, where heavier models sometimes overreached. And while keeping Claude Code let engineers work in their familiar environment, coding agents are tuned around their native models, which introduced friction when running non-Anthropic models through the proxy.

Because a single model does not need to do both the heavy reasoning and the routine execution, a follow-up study should separate those roles. We plan to run a blinded comparison of planners and orchestrators in a non-proprietary harness, holding the execution models, tools, and budgets constant across predefined task categories to measure task completion, reviewed code quality, human effort, latency, and cost.

Model choice is a cornerstone of how we provide AI tokens to engineers: access to models from different providers, at different capability levels and price points, with the freedom to choose what fits each task. By studying where each model helps and where even the most capable models overcomplicate simple work, we aim to help engineers make better choices, build better software, and deliver greater value to our customers.

[CTA_MODULE]