When a label change makes a stable model look worse

A classifier can produce the same output for the same support case on two evaluation runs and still receive a different result. This can happen when the business changes the expected label in the test set while the model, prompt, and input stay fixed.

Retraining before you identify that change can waste work and damage the evidence trail. Before you decide what to update, separate a model or prompt change, an input-data change, and a changed label definition. Preserve the old evaluation while you test the rule the business uses now.

A classifier assigns an input to one of a defined set of classes, such as refund, exchange, or billing. The expected label is the answer that an evaluation treats as correct. It is often called ground truth, but for a business routing task, that answer depends on the current operating rule and the people who own it.

Separate the model, inputs, and expected answer

A model or prompt version change affects the system that produces the prediction. The model weights may change. A prompt-driven router may receive revised class definitions, or surrounding code may change how the output is parsed. An unchanged model name does not prove that this complete path stayed the same.

An input or data change affects what the classifier receives. Ticket topics can move, customers can use new language, a source field can disappear, or the serving pipeline can transform data differently from the training pipeline. Google describes differences between training and serving inputs as training-serving skew. Its monitoring guidance separates schema changes from changes in engineered features and recommends checks on both.

A target or label-definition change affects the answer the business expects. The same ticket can move from refund to exchange after a return policy changes, even when the ticket text and classifier output are identical. Some teams use terms such as concept drift or label drift for parts of this problem. This article uses the exact operational question: which rule changed, and which examples received a new expected label?

The evaluator can also fail. A fixture can load the wrong version, a join can attach a label to the wrong case, or a grading rule can change without a new identifier. Keep evaluator and fixture defects available as a separate diagnosis instead of forcing every change into the model, data, or labels.

Constructed example: one return rule changes

The following support-routing example is constructed to explain the mechanism. It is not a customer case, production evaluation, or benchmark. The same four case descriptions and model predictions appear in both columns of the comparison.

Under business-rule version 1, an unopened return within 30 days routes to refund. A replacement request routes to exchange, and a payment correction routes to billing. Under version 2, the business narrows refunds to 14 days. Return requests from day 15 through day 30 now route to exchange for another resolution step.

Scroll sideways to see all 5 columns.

Fixed support case	Fixed model prediction	Expected label under rule v1	Expected label under rule v2	What moved
Unopened item, 7 days after delivery, customer requests a refund	refund	refund	refund	Nothing
Unopened item, 21 days after delivery, customer requests a refund	refund	refund	exchange	Return-rule boundary and expected label
Wrong size, customer requests a replacement	exchange	exchange	exchange	Nothing
Duplicate card charge	billing	billing	billing	Nothing

The 21-day case matches the expected answer under version 1. The same prediction fails under version 2. The model output stayed fixed; the expected label moved.

This change can create two opposite mistakes in the evaluation process. If the team scores against version 2 labels, the result can look worse and trigger an unsupported model-drift diagnosis. If the team keeps version 1 labels after the rule takes effect, the evaluation can pass a route that no longer matches the workflow.

Preserve the old result and create a new evaluation version

The earlier evaluation remains evidence about rule version 1. Keep its case inputs, expected labels, model and prompt version, evaluator version, date, and result. Do not overwrite its labels to make the old run look current. A changed historical label makes the earlier result impossible to reproduce and hides when the operating decision changed.

Create a new expected-label set for rule version 2. Relabel a representative set under the new definition, record reviewer disagreements, and publish it with a new identifier. The earlier BaristaLabs article on versioned agent evaluation sets explains why fixed inputs and ground truth are necessary for comparison. When expected behavior changes, versioning lets the team keep the old question and ask the new one without mixing them.

Two results can appear in the same history. Different label definitions put them on separate trend lines. A result against rule version 1 answers whether the model matched the old workflow. A result against rule version 2 answers whether it matches the current workflow. Any comparison must show that boundary beside the numbers.

Google’s production monitoring guidance recommends tracking performance by versions of code, model, and data so a team can identify the source of degradation. Add the business-rule version and expected-label-set version for policy-driven classifiers. This is a BaristaLabs extension of the version record, not a field list prescribed by Google.

Hold the output fixed to isolate the label change

If the evaluation stored the model’s case-level outputs, score those same outputs once against label version 1 and once against label version 2. This controlled comparison isolates the effect of the expected-answer change because the model output does not move between scores.

For the constructed 21-day return, the stored output remains refund. It matches label version 1 and misses label version 2. That result shows the size and location of the rule-definition effect on the saved cases. It does not prove that current production inputs stayed stable or that a new model run will reproduce every saved output.

Then run the unchanged model path against a representative version 2 set. Preserve the input source and sampling period, because a current set can contain new language or case types as well as new labels. Compare the saved and new outputs at case level before relying on an aggregate score. Google’s Rules of Machine Learning recommends measuring the difference between systems and first checking that a system compared with itself is stable.

For a non-deterministic path, exact output equality may not be available. Keep the actual outputs, model and prompt identifiers, runtime settings, and repeated-run behavior. If the evidence cannot isolate one source, record the cause as unknown and design the next comparison around that gap.

Keep the versions that make the result reproducible

A useful evaluation record needs enough information to reconstruct the question it answered.

Scroll sideways to see all 2 columns.

Field	What it establishes
Business-rule version and source	Which operating definition controlled the expected route
Expected-label or test-set version	Which answers the evaluator treated as correct
Model, prompt, parser, and evaluator versions	Which path produced and scored the output
Input sample and source period	Which cases and data conditions entered the comparison
Evaluation date	When the run occurred relative to rule and system changes
Workflow owner and label owner	Who approved the operating rule and resolves ambiguous labels

The rule source can be a policy revision, taxonomy change, routing decision, or approved workflow note. Use an identifier that the operations owner and technical owner can both resolve. A date alone is weak when two revisions can occur on the same day or take effect at different times.

The label owner also matters because business classes can contain judgment. Reviewers can disagree about a borderline return even when they read the same rule. Keep the disagreement and final disposition rather than presenting one person’s silent edit as timeless ground truth.

NIST’s AI Risk Management Framework gives voluntary guidance for adding trustworthiness considerations to AI design, development, use, and evaluation. Its AI RMF Playbook describes regular monitoring as work that covers system behavior, real-world impacts, and alignment with the values and norms of the context of use. It also names business rules as controls that can limit outputs in specified contexts. NIST does not prescribe the table above. Its wider point still applies: monitoring must account for the operating context and the model artifact.

A rule change can occur with real drift

A changed label definition does not exclude model or data problems. The support team might revise its return policy during the same week that ticket language changes, a source field disappears, and a prompt update reaches production. One diagnosis does not cancel the others.

Inspect the change history across all four parts of the evaluation: rule and labels, inputs, model path, and evaluator. Check case-level output differences, schema and feature checks, new or missing input slices, evaluator code, and reviewer notes. Google recommends monitoring both sudden and slow degradation in live prediction quality because serving conditions can move after a validation set is recorded.

The same caution applies when an aggregate result does not change. Newly incorrect cases can offset newly correct cases. A stable total can hide a changed route, minority class, or reviewer burden. Keep the affected cases and classes visible beside any summary measure.

Choose the update after the changed source is clear

When only the business definition changed, update the rule record and expected labels first. If the unchanged classifier already matches the new labels on representative cases, the model may need no change. The evaluation did.

If the current model misses the new definition, the implementation depends on the model path. A prompt-driven router may need revised class instructions and a new prompt version. A classifier trained on fixed classes may need revised examples and another training run. The BaristaLabs article on bounded labels and model choice explains why changing a fixed class or boundary can create relabeling and retraining work.

When input data changed, repair or adapt the data path before attributing every miss to the classifier. When the evaluator changed, fix the evaluation and rerun the controlled case set. When several parts moved together, separate them in staged comparisons or keep the affected cases in manual routing until the evidence supports a change.

Check one evaluation against the rule version

Take the most recent evaluation that caused concern. Attach the business-rule version, expected-label-set version, model and prompt version, input source, evaluation date, and owner. Preserve the earlier run. If the expected-answer version changed, stop using the score difference as proof that the model changed.

Use the production AI model-evaluation guide to check the complete path, representative cases, failure attribution, and next decision. If the version history is missing or the current evaluation cannot separate rule, label, input, evaluator, and model changes, BaristaLabs can review one evaluation change before the team retrains or replaces the path.

Source note

Google controls the label definition, version-tracking, training-serving-skew, and live-monitoring statements attributed to its guidance. NIST controls the voluntary AI RMF and Playbook statements. BaristaLabs supplies the constructed support example, the four-part version record, the controlled re-scoring method, and the implementation recommendations.

The example contains no measured performance, customer event, benchmark, or production outcome. A policy change can coincide with model, prompt, input, pipeline, or evaluator changes; the article does not treat changed labels as a universal diagnosis.