Production AI evaluation

How to evaluate an AI model for production

A production AI model evaluation should tell you whether a model-dependent workflow is ready to change, why cases passed or failed, and how the team will recover if the promoted path regresses. That requires testing more than the model name. The provider endpoint, adapter, tools, prompt and schema, cache, conversation state, retry budget, guardrails, and evaluator all shape the work the reviewer sees.

This guide shows how to compare one incumbent and one challenger against the same business acceptance bar, trace each failure to the layer that caused it, and turn the evidence into a promotion, restriction, retest, or rejection decision.

Copy the model evaluation worksheet See the GPT-5.6 bake-off case

This guide provides implementation-planning guidance. It is not a benchmark ranking, security certification, compliance opinion, or guarantee of production performance.

1Define the work
2Name both paths
3Fix comparison defects
4Run representative cases
5Read quality, cost, and failure evidence
6Promote with rollback or stop

Diagram of one production workload followed by an incumbent provider path, a challenger provider path, and one shared production acceptance review. Both paths record model and provider, adapter, tools, cache and state, and guardrail and policy; illustrative markers name possible model, adapter, tool, state, policy, and evaluator attribution layers, not measured failures or results. — A fair comparison preserves the work and acceptance bar while making each provider path, failure location, and operating condition visible.

1 · Work before benchmark

Begin with the production decision

Name the workflow, accountable owner, decision under review, current path, and consequence of a wrong or late result. Define what accepted, edited, rejected, escalated, and manual outcomes mean before choosing cases or metrics.

The minimum evidence should match the next decision. A draft-only internal workflow may need a smaller review than a tool-using path that changes customer records. Neither is served by a generic leaderboard detached from the work.

2 · Reproducible candidates

Name the complete incumbent and challenger paths

A model path is the complete route from the application request through the provider or runtime and back through tools and validation. Record the exact model version or digest, provider, region, endpoint, API surface, adapter, SDK, agent host, runtime, resolved dependencies, prompt, schemas, tools, instruction packages, state, cache, and relevant data-routing conditions.

Self-hosted paths also need hardware and quantization. The production-shaped Ollama comparison shows why a loaded model is not yet a usable application path. The open-model ownership guide separates task acceptance from the operating responsibility a deployment choice creates.

Provider routes can hide additional policy and configuration. Treat the endpoint and provider route as part of the candidate, not a footnote to its model name.

3 · Fair implementation

Make tool and adapter semantics equivalent before scoring

Comparable paths do not require identical provider-specific JSON. They require the same intended operation and business acceptance bar. Compare argument presence, null handling, correlation IDs, batched or parallel calls, implementation effects, returned results, and error semantics.

A final answer cannot prove that a tool path worked. Require a trace that joins the model request, adapter translation, tool effect, tool result, and final workflow state. Ploy’s first-party GPT-5.6 migration case and the CrewAI call-correlation example show how integration assumptions can look like model failures.

4 · Controlled conditions

Control cache, state, concurrency, retries, and budgets

Run a cold first turn, warm continuation, new conversation in the same business scope, resumed task after an idle period, and a deployment-boundary replay. Record provider-managed and application-managed state, cache keys and breakpoints, realistic concurrency, serial and parallel tool execution, and token, tool-call, retry, attempt, and wall-clock ceilings.

Report the stop rule and whether performance was still improving when the budget ended. The agent evaluation budget curve explains why reliability and cost must be read against the budget that produced them.

5 · Cases and holdout

Build a representative set and define the acceptance bar

Use sanitized production examples where policy allows. Include common work, difficult variants, known failures, missing or conflicting inputs, policy exceptions, tool errors, state resumes, and cases that should remain manual. Keep a holdout untouched until prompts, adapters, and thresholds are fixed.

Acceptance should cover reviewer judgment, required sources, structured output, required and prohibited tool effects, final workflow state, recovery, latency, cost, and review effort. When guardrails matter, preserve language and turn order and measure false positives, false negatives, and unavailable behavior. The multilingual, multi-turn guardrail guide provides the deeper test design.

There is no universal sample count or pass threshold. State the sample, traffic coverage, reviewer depth, and limits so the decision does not claim more than its evidence supports.

6 · Failure attribution

Classify each failure where the evidence first supports it

Keep the input ID, exact configuration, trace, layer assignment, reviewer note, and disposition for every case. Use unknown when the evidence cannot support attribution.

Request validation or unsupported API shape
Adapter or SDK translation
Malformed structured output
Model task or reasoning error
Retrieval or source error
Tool argument, execution, or result error
State, cache, or resume error
Timeout, budget, retry, or dependency failure
Guardrail or policy error
Evaluator, fixture, label, or threshold error
Human rejection against the business acceptance bar
Unknown until stronger evidence exists

The Gemini request-contract case is a concrete example of separating HTTP, schema, tool, timeout, retry, and reviewer rejection boundaries.

7 · Decision evidence

Measure accepted work, latency, cost, and review effort together

Report business acceptance by case type; edits, rejections, escalations, and reviewer disagreement; schema and tool-path completion; attributed failures; cold, warm, resumed, and realistic-load behavior; and end-to-end p50, p95, and p99 latency only where the sample supports those percentiles.

Keep input, output, cache-write, cache-read, tool, and infrastructure costs visible. Add human review time when it can be collected consistently. A cheaper token can produce a more expensive accepted case after retries, tool loops, and correction. Do not hide the underlying measures inside one weighted score.

For tool-using workflows, add source, approval, state-change, side-effect, and recovery receipts. The workflow-receipt evaluation guide explains the evidence a text-only score misses.

Copyable evaluation aid

Production AI model evaluation worksheet

Keep the worksheet blank until your team supplies real cases, configurations, measurements, and reviewer decisions. It is a portable comparison record, not a benchmark result.

Production AI model evaluation worksheet

Workflow:
Business owner:
Technical owner:
Decision being considered:
Current production path:
Incumbent candidate:
Challenger candidate:
Evaluation date and evidence location:

1. Work and acceptance
Business outcome this workflow must support:
Representative cases and source period:
Difficult, failure, and manual-only cases:
Untouched holdout:
Qualified reviewer / label owner:
Must-pass business criteria:
Required workflow or tool effects:
Prohibited effects:
Latency, cost, and review-effort limits:
Stop or pause condition:

2. Exact compared paths

| Configuration | Incumbent | Challenger |
| --- | --- | --- |
| Model, version, snapshot, or digest |  |  |
| Provider, region, endpoint, and API surface |  |  |
| Adapter, SDK, agent host, and runtime versions |  |  |
| Prompt, system instruction, schema, tools, and skills |  |  |
| Quantization and hardware, if self-hosted |  |  |
| State and storage behavior |  |  |
| Cache key, breakpoints, warm-up, and retention |  |  |
| Token, tool-call, retry, attempt, and time budgets |  |  |
| Concurrency and load condition |  |  |
| Guardrail, network, data, and permission boundary |  |  |

3. Evidence

| Evidence | Incumbent | Challenger | Decision note |
| --- | --- | --- | --- |
| Business acceptance by case type |  |  |  |
| Reviewer edits, rejections, and escalations |  |  |  |
| Structured-output and tool-path completion |  |  |  |
| Failures by model / adapter / tool / state / policy / evaluator / unknown |  |  |  |
| Cold, warm, resumed, and load latency |  |  |  |
| Input, output, cache, tool, and infrastructure cost |  |  |  |
| Human review effort |  |  |  |
| Guardrail, containment, and trajectory evidence where required |  |  |  |
| Known gaps or untested conditions |  |  |  |

4. Decision
Decision: promote / promote with restrictions / retest / reject
Reason:
First rollout scope:
Required monitoring:
Rollback trigger:
Fallback or manual path:
Decision owner:
Next review date or event:
Facts to add or refresh in the Model Facts Register:
Evidence that must remain linked from the register:

Routing policy extension

Extend the evaluation record for dynamic model routing

One conversation can cross several model targets when a routing policy can choose a different path for each eligible turn. A target is one configured provider and model path. An eligible turn is a model request that the policy is allowed to route. The fixed-model worksheet above remains the baseline; this extension records the additional decisions created by routing.

Keep the workflow, cases, acceptance bar, and fixed-model path unchanged. Treat the routing policy as the challenger. Compare completed work only after the experiment includes target changes, judge calls, retries, and routing overhead.

Record the policy name and version, whether it keeps session state, the rule that keeps later turns on one target, and the exact provider and model behind each eligible target name. For every eligible turn, connect the selected target and route signals to the context that target received. The routing decision can be reasonable while the next action is wrong if the selected model did not receive a required tool result, file location, constraint, or earlier commitment.

Join this operational record to the final workflow outcome. Selected targets, tokens, latency, and routing overhead show how the route ran. The owner still needs to record whether the completed work was accepted, edited, rejected, escalated, kept manual, failed, or stopped. Use the same acceptance bar as the fixed-model baseline.

Use privacy-safe case, session, and turn IDs. Keep sensitive prompts, customer records, credentials, secrets, and full conversation text in the approved evidence system. The worksheet should point to that evidence without copying it into a document that can be shared more widely.

Run ordinary cases and force six difficult conditions: a difficult request, a tool error, a handoff between targets, a resumed session, an unavailable target, and a request near the policy boundary. Record whether the policy escalates, stays on one target, retries, uses a fallback, or stops. Confirm that the operator can see what happened and that the receiving model received the required state.

Compare completed-task quality, cost, latency, retries, and reviewer effort with the fixed-model baseline. Continue the experiment only when the evidence supports the defined acceptance bar. Otherwise restrict the policy, return to the fixed target, or stop. Keep the fixed target available and verify the disable control before wider use.

The NVIDIA Switchyard field note is a current example of the architecture problem. NVIDIA documents routing strategies, protocol translation, session state, and operational telemetry. The current Switchyard repository also identifies the implementation as pre-alpha and not for production use. The worksheet fields below are a BaristaLabs evaluation aid, not NVIDIA production guidance.

Copy this extension after you complete the main production AI model evaluation worksheet. Leave a field blank until the team has evidence. Do not replace the fixed-model baseline with the routed result.

1. Routing policy and targets

Routing policy name:
Policy version or configuration digest:
Policy type or routing strategy:
State behavior: stateful / stateless / mixed:
State retained by the router:
Affinity rule and release condition:
Eligible turns:
Policy inputs or decision signals:
What the router can see:
What the selected target receives:
Context or data excluded from the selected target:
Default target:
Fallback target:
Expected behavior when a target is unavailable:
Disable or stop control:
Disable or stop owner:
Policy boundary:
Known assumptions or untested conditions:

Eligible target map
Target name	Provider	Exact model version, snapshot, or digest	Region and endpoint	Adapter, host, and runtime	Known limit

Router or judge mapping, if separate from the target map
Role	Provider	Exact model version, snapshot, or digest	Trigger	Cost and latency evidence location

2. Per eligible turn

Repeat one row for each turn that the policy can route.
Final workflow outcome: accepted / edited / rejected / escalated / manual / failed / stopped

Per eligible turn
Privacy-safe case ID	Privacy-safe session ID	Turn ID	Selected target	Route reason or signals	Context or state delivered	Model latency	Input, output, and cache tokens	Router or judge calls, latency, tokens, and cost	Errors and retries	Turn result	Final workflow outcome	Evidence link

3. Continuity and failure rehearsal

Required conditions:

Difficult request
Tool error
Handoff between targets
Resumed session
Target unavailable
Request near the policy boundary

Route action: escalate / stay on one target / retry / use fallback / stop

Continuity and failure rehearsal
Condition	Expected policy behavior	Targets observed	Required state	State observed at the receiving target	Route action	Final workflow outcome	Evidence link

Session continuity notes across handoff:
Session continuity notes after resume:
Observed behavior when a target was unavailable:
Observed behavior when the disable control was used:

4. Completed-task comparison

Completed-task comparison
Evidence	Fixed-model baseline	Routing policy	Conditions and limits
Completed-task acceptance and quality by case type
Completed-task cost, including model, router, judge, tool, and infrastructure cost
End-to-end latency
Routing and judging overhead
Model calls, tool calls, retries, and loops
Reviewer edits, rejections, escalations, and effort
Errors and attributed failures
Session continuity across handoff and resume
Unavailable-target and fallback behavior
Known untested conditions

5. Decision

Decision: continue experiment / continue with restrictions / return to fixed target / stop:
Reason:
Decision owner:
Restrictions:
Fixed fallback path:
Rollback trigger:
Disable or stop control verified:
Required monitoring:
Known untested conditions:
Next review date or event:
Evidence that must remain linked to the decision:

Dynamic model-routing evaluation extension

Use with the Production AI model evaluation worksheet.
Do not replace the fixed-model baseline.

0. Link to the main evaluation
Workflow:
Business owner:
Technical owner:
Decision being considered:
Fixed-model baseline path:
Baseline evidence location:
Acceptance bar reference:
Routing-policy evidence location:
Experiment window and build:
Privacy-safe case, session, and turn ID rule:

1. Routing policy and targets
Routing policy name:
Policy version or configuration digest:
Policy type or routing strategy:
State behavior: stateful / stateless / mixed
State retained by the router:
Affinity rule and release condition:
Eligible turns:
Policy inputs or decision signals:
What the router can see:
What the selected target receives:
Context or data excluded from the selected target:
Default target:
Fallback target:
Expected behavior when a target is unavailable:
Disable or stop control:
Disable or stop owner:
Policy boundary:
Known assumptions or untested conditions:

Eligible target map

| Target name | Provider | Exact model version, snapshot, or digest | Region and endpoint | Adapter, host, and runtime | Known limit |
| --- | --- | --- | --- | --- | --- |
|  |  |  |  |  |  |

Router or judge mapping, if separate from the target map:

| Role | Provider | Exact model version, snapshot, or digest | Trigger | Cost and latency evidence location |
| --- | --- | --- | --- | --- |
|  |  |  |  |  |

2. Per eligible turn

Repeat one row for each turn that the policy can route.
Final workflow outcome: accepted / edited / rejected / escalated / manual / failed / stopped

| Privacy-safe case ID | Privacy-safe session ID | Turn ID | Selected target | Route reason or signals | Context or state delivered | Model latency | Input, output, and cache tokens | Router or judge calls, latency, tokens, and cost | Errors and retries | Turn result | Final workflow outcome | Evidence link |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  |  |  |  |  |  |  |  |  |

3. Continuity and failure rehearsal

Required conditions:
- Difficult request
- Tool error
- Handoff between targets
- Resumed session
- Target unavailable
- Request near the policy boundary

Route action: escalate / stay on one target / retry / use fallback / stop

| Condition | Expected policy behavior | Targets observed | Required state | State observed at the receiving target | Route action | Final workflow outcome | Evidence link |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  |  |  |  |

Session continuity notes across handoff:
Session continuity notes after resume:
Observed behavior when a target was unavailable:
Observed behavior when the disable control was used:

4. Completed-task comparison

| Evidence | Fixed-model baseline | Routing policy | Conditions and limits |
| --- | --- | --- | --- |
| Completed-task acceptance and quality by case type |  |  |  |
| Completed-task cost, including model, router, judge, tool, and infrastructure cost |  |  |  |
| End-to-end latency |  |  |  |
| Routing and judging overhead |  |  |  |
| Model calls, tool calls, retries, and loops |  |  |  |
| Reviewer edits, rejections, escalations, and effort |  |  |  |
| Errors and attributed failures |  |  |  |
| Session continuity across handoff and resume |  |  |  |
| Unavailable-target and fallback behavior |  |  |  |
| Known untested conditions |  |  |  |

5. Decision
Decision: continue experiment / continue with restrictions / return to fixed target / stop
Reason:
Decision owner:
Restrictions:
Fixed fallback path:
Rollback trigger:
Disable or stop control verified:
Required monitoring:
Known untested conditions:
Next review date or event:
Evidence that must remain linked to the decision:

8 · Proportional safeguards

Add guardrail, containment, and trajectory tests where needed

Select tests according to the data, tools, permissions, duration, and consequences of the workload. A text-only internal draft does not need the same containment as a long-running tool-using agent.

Inspect ordered trajectories, pause conditions, reviewer evidence, and resume or stop decisions for long-running work. When an evaluation relaxes model refusals or exposes powerful tools, require infrastructure containment outside the model and assume allowed dependencies can fail. The trajectory-monitoring guide covers sequence-level review.

9 · Controlled change

Make promotion and rollback one decision

Promote: The challenger clears the acceptance bar and operating limits, and rollout, monitoring, fallback, and rollback have owners.
Promote with restrictions: Approve a smaller traffic slice, data class, task type, review level, or budget while uncertainty remains visible.
Retest: An attributable adapter, dataset, state, cache, policy, or evaluator defect has an owner and exit condition.
Reject: The candidate misses the acceptance bar or operating boundary after a fair path and enough evidence for this decision.

Specify the first traffic slice, monitoring period, rollback trigger, fallback, decision owner, and evidence required for wider rollout. After a provider outage or suspension, replay the same critical cases and acceptance bar before restoring the path. The model-access continuity guide turns that return test into an owned decision.

10 · Living facts

Put current facts in the register and run evidence elsewhere

Model Facts Register

Approved model path, region, quota, limits, price mode, lifecycle, data policy, incompatibilities, owner, refresh date, review trigger, fallback, and stop rule.

Evaluation record

Case inputs and labels, traces, tool calls, receipts, reviewer decisions, attributed failures, measurements, disagreements, rollout scope, and rollback evidence.

Open the Model Facts Register

Sources and limits

Keep reported behavior separate from the reusable method

Ploy’s GPT-5.6 account is a first-party report about its production website-building agent and a small reported redesign sample. Its harness, adapter, cache, state, cost, and timing findings explain mechanisms; they are not independent benchmarks or expected outcomes for another workload.

The method also draws on current evaluation, function-calling, caching, conversation-state, model-selection, and lifecycle-risk guidance from OpenAI, Anthropic, Google Cloud, AWS, and NIST. Provider prices, quotas, model states, API behavior, and lifecycle details can change; recheck them before a production decision.

Related resources

FAQ

Production AI model evaluation questions

Should both models use the exact same prompt and adapter?

Keep the workload and acceptance criteria the same. Provider-specific prompts, schemas, cache controls, or state mechanisms may differ when they are required to express the same intended work correctly. Record every difference and test the resulting complete path.

How many evaluation cases do we need?

There is no universal number. Cover common work, difficult variants, known failures, policy exceptions, tool and state failures, and an untouched holdout deeply enough to support the consequence of the decision. State the sample and its limits.

Which metrics should decide the winner?

Use business acceptance, workflow and tool completion, attributed failures, latency, cost per accepted case, review effort, and relevant safety behavior. Keep the underlying measures visible instead of collapsing them into one unexplained score.

How do we know whether the model or the integration failed?

Keep the exact request, adapter translation, tool arguments and effects, tool result, state and cache condition, evaluator result, and reviewer decision. Classify the first boundary where evidence shows a failure and retain unknown when it does not.

What goes in the Model Facts Register after the evaluation?

Record the approved model path, region, quota, limits, price mode, lifecycle, data policy, known incompatibilities, owner, refresh date, review trigger, fallback, and stop rule. Keep case-level traces and measurements in the linked evaluation evidence.

Need a second set of eyes?

Review one production model change before rollout

Bring one workflow, its incumbent traces, the challenger path, and the result your team currently accepts. BaristaLabs can help make the comparison fair, classify the failures, and define the evidence for promotion or rollback.

Review one model change Explore AI consulting

Best fit when a model, provider, runtime, or adapter change is close to production and the current comparison cannot yet explain why cases pass or fail.