Before you switch AI models, put the bake-off on trial

The following release-room scene is illustrative. The measurements and engineering failures that follow come from Ploy's published migration report, but Ploy does not describe this meeting.

A challenger model has just lost the bake-off.

Its failure count is too high. Its first cost run is roughly 50% worse. It burns through the tool-call budget, reads files that appear to be empty, and occasionally loses the reasoning state from a prior turn. The incumbent is slower, but familiar. Someone moves the migration ticket back to the backlog.

Then an engineer asks to cross-examine the test.

That question sits underneath Ploy's July 9 account of moving its production website-building agent to GPT-5.6 Sol. The challenger did not merely improve after prompt tuning. Several apparently decisive measurements changed meaning when Ploy repaired the machinery around the comparison.

The case matters now because OpenAI made the GPT-5.6 family generally available on July 9, with Sol, Terra, and Luna tiers across ChatGPT, Codex, and the API. OpenAI said the global rollout would continue over the following 24 hours. It also introduced explicit cache breakpoints, new cache economics, and beta multi-agent execution.

Those features make the release interesting. Ploy's experience makes it useful. A fair model bake-off has to evaluate the model and the assumptions the incumbent taught the surrounding system.

Exhibit one: the failure count accused the wrong defendant

Ploy's production agent plans and edits marketing websites. It reads a codebase, writes components, generates images, captures screenshots, and decides when the work is complete. Its evaluation suite runs that behavior against fixture workspaces, visual checks, content checks, tool trajectories, and file assertions.

In the first cross-model run, GPT-5.6 accumulated failures that looked like evidence against the model. Ploy reports that roughly one-third of those raw failures came from the harness instead.

The tool-call budgets had been sized around Claude Opus's sequential execution style. GPT-5.6 made parallel calls and crossed those limits while solving cases correctly. The executor also lacked support for batched file reads, a behavior GPT-5.6 used more often.

One visual dataset carried another hidden assumption. When a minScore value was absent, the harness defaulted to a perfect 1.0. A GPT-5.6 result scoring 0.98 failed. An Opus result that passed every individual check also failed under the same implicit threshold.

None of this proves that every challenger deserves a repaired score. It shows why raw failure counts are allegations, not verdicts. Each failure needs a trace that identifies whether the model, adapter, budget, evaluator, or fixture actually broke.

This is the production version of the problem in why better benchmarks can produce worse production outcomes: model capability and release stability are different measurements. Ploy's case adds a sharper point. The stability test can inherit the incumbent's habits and mistake unfamiliar execution for failure.

Ask one question before comparing pass rates: would this case still fail if the harness understood the challenger's legitimate execution pattern?

Exhibit two: success: true concealed an empty read

The second piece of evidence looked like bad model persistence. GPT-5.6 kept asking for files it had supposedly read. Tool calls multiplied. The agent appeared wasteful and confused.

Ploy's traces pointed somewhere more specific.

Its code tool exposed 25 top-level properties. One property, action, was required; the rest were optional. Claude typically sent the two or three properties it needed. Across three days of production code(read) traces, Ploy says GPT-5.6 sent all 25 properties on all 6,635 calls. Claude Opus 4.8 did so on 4 of 2,898 calls, and Claude Sonnet 5 did so on none of 1,933 calls.

The unused GPT-5.6 properties were not blank. They arrived with plausible values such as offset: 0. Ploy's file reader interpreted those values as intentional arguments. Between 52% and 64% of GPT-5.6 file reads returned empty.

The tool still answered success: true.

That response turned an adapter defect into model behavior. The model believed the tool had completed a valid read, received no file content, and tried again.

Ploy says prompt instructions to omit unused fields did not solve the problem. The working change happened at the provider boundary: optional properties became required but nullable for OpenAI-family models, then null values were removed before shared validation. Empty reads fell to zero, and the agent used roughly 30% fewer tool calls for the same work.

A model swap therefore changes more than the model identifier. A tool schema is a dialect shared by the model, provider, SDK, validator, and implementation. If one layer treats a plausible default as intent while another treats it as absence, a clean tool response can certify a failed action.

Cross-examine every suspicious loop with three facts: the exact arguments the model sent, the effect the tool performed, and the result object the model received. If those disagree, more prompting may only hide the faulty interpreter.

Exhibit three: the price comparison started with a cold cache

Ploy's first cost comparison made GPT-5.6 appear roughly 50% more expensive than Opus. That number could have ended the evaluation on its own.

The measured gap came from applying one provider's cache design to another.

Ploy's agent begins with a static prefix of about 29,000 tokens, including tool schemas and its core system prompt. The team reports 92% to 96% cache hit rates with its Anthropic configuration, where marked cache breakpoints can support an organization-wide shared prefix.

GPT-5.6's cache behavior required a different design. OpenAI's launch details describe explicit cache breakpoints, a 30-minute minimum cache life, cache writes billed at 1.25 times the uncached input rate, and cache reads discounted by 90%.

Ploy found that a conversation-scoped cache key gave a new conversation no shared first-call benefit, while one global key concentrated too much traffic. It moved to workspace-scoped keys and split the prompt into breakpointed layers.

In Ploy's system, first-call cache hits rose from roughly zero to 83.7%, uncached input tokens fell 28%, and GPT-5.6's per-suite cost moved below Opus. Ploy concluded that cache misconfiguration explained its initial measured cost gap.

That is a case result, not a universal savings claim. Ploy's traffic shape, prefix length, workspace boundaries, and evaluation tasks determine those numbers. Another workload can reach a different answer.

The transferable lesson is narrower: a price comparison is invalid when one candidate starts warm and the other starts cold. Record cache writes, cache reads, key scope, idle windows, and first-turn behavior beside token totals. Otherwise the model price is carrying an infrastructure penalty that belongs to the test setup.

A current model facts register can track list price and lifecycle status. A bake-off still has to measure the cost produced by the cache and orchestration design surrounding one real workload.

Exhibit four: yesterday's reasoning was a remote dependency

The final failure appeared intermittently in production conversations. Ploy says GPT-5.6's Responses API replayed earlier reasoning through server-side item references by default. Some conversations failed because a referenced reasoning item could not be found.

Ploy changed its implementation to use store: false, requesting encrypted reasoning content that could be replayed as self-contained data rather than relying on server-side pointers.

The important distinction is state location. Two requests can look append-only inside an application while still depending on provider-managed state that the application did not send. A migration test that covers only fresh, single-turn tasks will never expose that boundary.

Put a resumed conversation under examination. Pause it long enough to cross an idle window. Retry it after a deployment. Confirm which prior state lives in your database, which state lives with the provider, and how the workflow fails when either reference disappears.

That test belongs beside the budget sweep described in the agent evaluation budget curve. More time and more calls can reveal capability, but they can also introduce state dependencies that a short benchmark never exercises.

The verdict came after the exhibits changed

After repairing the harness, Ploy reran its redesign suite. The reported means cover 11 completed builds for Claude Opus 4.8 and 10 for GPT-5.6.

Scroll sideways to see all 3 columns.

Mean per completed build	Claude Opus 4.8	GPT-5.6 Sol
Cost	$3.06	$2.22
Wall-clock time	8m 00s	3m 42s
Input tokens	2.60M	1.70M
Output tokens	33.0K	17.1K
Visual score	0.936	0.970

Ploy summarizes the corrected result as 2.2 times faster, 27% less expensive, and about half the output tokens for its workload. It then made GPT-5.6 Sol the default model for every Ploy workspace.

The sample is small, first-party, and specific to a production website-building agent. It is not an independent benchmark or a promise that another agent will reproduce the result. Its value is the engineering sequence: every major conclusion became more credible after the team tried to disprove the measurement that produced it.

Give one bake-off 90 minutes of cross-examination

Use a production-shaped workflow whose current traces, cost, latency, and accepted output are already available. Do not begin with a synthetic task that neither candidate has to survive after launch.

Spend the first 20 minutes classifying failures. For every miss, identify the failing layer and inspect the full trace. Separate model errors from unsupported parallelism, batched operations, evaluator defaults, and tool-result ambiguity.

Spend the next 20 minutes on tool semantics. Compare argument presence, nulls, invented defaults, implementation effects, and returned success states. Force one empty, partial, or stale result and see whether the agent can recognize it.

Spend 20 minutes on cache economics. Run a cold first turn, a warm continuation, a new conversation in the same business scope, and a new scope. Measure writes and reads separately. Compare costs only after both candidates use an intentional cache design.

Use the final 30 minutes for continuity and judgment. Resume an older conversation. Replay a deployment boundary. Review the output against the acceptance criteria that matter to the business, not only the style the incumbent tends to produce.

Then rerun the exact same cases.

The useful output is not a prettier leaderboard. It is a list of findings whose meaning survived an adversarial review of the test. A challenger may still lose. If it does, the team should be able to say that the model lost after both systems received a fair interpreter, a warm cache, a valid state path, and the same business acceptance bar.

That is enough to support a production decision.

BaristaLabs helps teams cross-examine one model bake-off before a provider change reaches customer work. Bring one workflow, its incumbent traces, and the result your team currently calls acceptable.