Developing an AI-assisted feature often begins with a few impressive examples. A new prompt creates a better summary in a demo, another model writes more fluently, or a retrieval adjustment finds more source material. A good demo is not enough: it can hide regressions in accuracy, cost, latency or safety.
Treat an AI feature as measurable product behaviour. Ask not “is the model good?” but: Is this version more reliable on important user tasks than the previous version, and does it fail in an acceptable way?
Write the product decision first
Before collecting an evaluation set, clarify what the feature must accomplish. “Generate helpful answers” is too broad. Define a concrete task instead: extracting action items from a meeting transcript, summarising a long document under given headings, routing a customer request to the right category, or answering a research question with sources.
Success and failure have different meanings for every task. In an action-item feature, assigning the wrong owner can be more costly than missing one item. In a document summary, adding details absent from the source is worse than a shorter summary. In classification, returning “review required” for ambiguous input may be better than forcing a wrong category.
Metrics must represent product risk.
Build a small but representative set
An initial evaluation set does not need thousands of examples. Twenty to fifty carefully chosen cases can be more useful than hundreds of random ones. The set should combine ordinary user behaviour with risky edge cases:
- common, normal inputs,
- short, long and badly formatted inputs,
- ambiguous requests with several possible interpretations,
- missing context or cases with no correct answer,
- sensitive content, permission boundaries or requests that should be refused,
- examples that previously led to support contacts or failures.
Where possible, every case should include an expected outcome, acceptable variations and an evaluation note. For free-text tasks, one “golden answer” is not always the right approach. A checklist of required facts, prohibited claims, desired tone or source requirements can be more useful.
Keeping the set in version control matters. It shows when a case was added, which production failure created a new test and which data versions were compared.
Inspect output across several dimensions
One score does not explain all AI behaviour. Evaluation should separate at least these dimensions:
- Task success: Does the response actually complete the requested work?
- Accuracy and grounding: Do claims match the input or provided sources?
- Format: Does it preserve the required JSON, table, field structure or language rule?
- Safety: Does the feature respect authorisation, privacy and content boundaries?
- Usability: Is the response understandable, appropriately sized and actionable?
- Cost and latency: Is the result produced within an acceptable budget and time?
A model response can be grammatically excellent but break an automation flow because it does not follow the required schema. Or a sourced answer can look polished while offering no evidence. Seeing scores separately explains which improvement came with which cost.
Combine automatic checks with human review
Some criteria can be checked reliably in code. Is the JSON schema valid? Is a required field present? Was a source identifier returned? Is the response within the allowed length? Does it contain a personal-data pattern? These checks are fast and should run on every version.
Qualities such as accuracy, usefulness and tone often need human review. Rather than limiting review to random cases, focus it on risky examples and cases with large score changes. Reviewers need an explicit rubric; otherwise “I think it is better” cannot be compared across versions.
Using another model for judging can be a useful signal, but it must not be treated as final truth. A judge model can be wrong, biased or overly sensitive to form. Human verification and rule checks for known failure types balance that risk.
Record every input to a version
When output changes, knowing only the model name is not enough. An evaluation record should include model and provider version, prompt version, system instruction, tool definitions, retrieval settings, generation parameters and evaluation-set version.
This makes it possible to investigate “why did a test that passed yesterday fail today?” It also makes rollback easier. If a new prompt causes trouble in production, the team needs to restore not only code but the full configuration that produced known-good behaviour.
If real user content must be retained from the client, privacy policy, masking and retention period are part of evaluation design. Collecting more personal data than needed to measure quality is not a sound trade-off.
Track change and error budgets, not only a threshold
A high average score does not automatically make a version production-ready. A small regression in a critical error category can be unacceptable. Define red lines alongside the general score: an unsupported claim, invalid structured output or an answer to a request that should have been refused.
The decision is usually made by comparing two versions. Which cases improved, which regressed, and how did cost and latency change? This comparison exposes not only impressive examples from a model change, but also its losses.
A gradual production rollout does not replace the evaluation set; it complements it. Error rates, feedback and safety incidents from a small user group add evidence about real behaviour. Rollback conditions should be defined before release.
A useful evaluation pipeline does not need to start as a large platform. A versioned small case set, a few automatic checks, a clear human rubric and comparison across versions already turn prompt experiments into repeatable engineering decisions. This is how an AI feature stops being an impressive demo and becomes a product capability that can improve safely.