1 · Work before benchmark
Begin with the production decision
Name the workflow, accountable owner, decision under review, current path, and consequence of a wrong or late result. Define what accepted, edited, rejected, escalated, and manual outcomes mean before choosing cases or metrics.
The minimum evidence should match the next decision. A draft-only internal workflow may need a smaller review than a tool-using path that changes customer records. Neither is served by a generic leaderboard detached from the work.
2 · Reproducible candidates
Name the complete incumbent and challenger paths
A model path is the complete route from the application request through the provider or runtime and back through tools and validation. Record the exact model version or digest, provider, region, endpoint, API surface, adapter, SDK, agent host, runtime, resolved dependencies, prompt, schemas, tools, instruction packages, state, cache, and relevant data-routing conditions.
Self-hosted paths also need hardware and quantization. The production-shaped Ollama comparison shows why a loaded model is not yet a usable application path. The open-model ownership guide separates task acceptance from the operating responsibility a deployment choice creates.
Provider routes can hide additional policy and configuration. Treat the endpoint and provider route as part of the candidate, not a footnote to its model name.
3 · Fair implementation
Make tool and adapter semantics equivalent before scoring
Comparable paths do not require identical provider-specific JSON. They require the same intended operation and business acceptance bar. Compare argument presence, null handling, correlation IDs, batched or parallel calls, implementation effects, returned results, and error semantics.
A final answer cannot prove that a tool path worked. Require a trace that joins the model request, adapter translation, tool effect, tool result, and final workflow state. Ploy’s first-party GPT-5.6 migration case and the CrewAI call-correlation example show how integration assumptions can look like model failures.
4 · Controlled conditions
Control cache, state, concurrency, retries, and budgets
Run a cold first turn, warm continuation, new conversation in the same business scope, resumed task after an idle period, and a deployment-boundary replay. Record provider-managed and application-managed state, cache keys and breakpoints, realistic concurrency, serial and parallel tool execution, and token, tool-call, retry, attempt, and wall-clock ceilings.
Report the stop rule and whether performance was still improving when the budget ended. The agent evaluation budget curve explains why reliability and cost must be read against the budget that produced them.
5 · Cases and holdout
Build a representative set and define the acceptance bar
Use sanitized production examples where policy allows. Include common work, difficult variants, known failures, missing or conflicting inputs, policy exceptions, tool errors, state resumes, and cases that should remain manual. Keep a holdout untouched until prompts, adapters, and thresholds are fixed.
Acceptance should cover reviewer judgment, required sources, structured output, required and prohibited tool effects, final workflow state, recovery, latency, cost, and review effort. When guardrails matter, preserve language and turn order and measure false positives, false negatives, and unavailable behavior. The multilingual, multi-turn guardrail guide provides the deeper test design.
There is no universal sample count or pass threshold. State the sample, traffic coverage, reviewer depth, and limits so the decision does not claim more than its evidence supports.
6 · Failure attribution
Classify each failure where the evidence first supports it
Keep the input ID, exact configuration, trace, layer assignment, reviewer note, and disposition for every case. Use unknown when the evidence cannot support attribution.
- Request validation or unsupported API shape
- Adapter or SDK translation
- Malformed structured output
- Model task or reasoning error
- Retrieval or source error
- Tool argument, execution, or result error
- State, cache, or resume error
- Timeout, budget, retry, or dependency failure
- Guardrail or policy error
- Evaluator, fixture, label, or threshold error
- Human rejection against the business acceptance bar
- Unknown until stronger evidence exists
The Gemini request-contract case is a concrete example of separating HTTP, schema, tool, timeout, retry, and reviewer rejection boundaries.
7 · Decision evidence
Measure accepted work, latency, cost, and review effort together
Report business acceptance by case type; edits, rejections, escalations, and reviewer disagreement; schema and tool-path completion; attributed failures; cold, warm, resumed, and realistic-load behavior; and end-to-end p50, p95, and p99 latency only where the sample supports those percentiles.
Keep input, output, cache-write, cache-read, tool, and infrastructure costs visible. Add human review time when it can be collected consistently. A cheaper token can produce a more expensive accepted case after retries, tool loops, and correction. Do not hide the underlying measures inside one weighted score.
For tool-using workflows, add source, approval, state-change, side-effect, and recovery receipts. The workflow-receipt evaluation guide explains the evidence a text-only score misses.