← Stackzilla Blog
AI Evaluations: Test the Task Before Changing the Model
Published November 15, 2026
· 3 min read
· AI engineering, evaluations, testing, LLMs
Build a representative evaluation set so prompt and model changes are judged against the behavior your product actually needs.
A new model produces an impressive answer to one example, and the team wants to switch. That example may be real, but it does not show whether the change improves the product's full workload.
AI evaluation starts by defining the task and the failures that matter. Without that definition, teams can optimize for answers that look polished while overlooking factual mistakes, format violations, or unsafe actions.
## Build a representative test set
Collect examples from the intended workflow with appropriate permission and privacy handling. Include straightforward cases, ambiguous requests, missing information, and situations where the correct response is to refuse or ask for clarification.
Label what success means. A document extraction task may need exact fields and evidence locations. A support assistant may need policy accuracy, grounded answers, and correct escalation behavior.
Keep a held-out set that is not used for every prompt adjustment. Otherwise repeated tuning can overfit the examples while giving a misleading impression of progress.
## Use the right kind of check
Deterministic checks are valuable for output schemas, required fields, tool-call constraints, and exact computations. Use them where possible rather than asking another model to judge everything.
Semantic judgments can help evaluate qualities that are harder to encode, but model-based graders have limitations. Define a rubric, compare judgments with human review, and inspect disagreements. A grader can prefer verbosity or share the same factual blind spots as the system being tested.
For retrieval-based applications, evaluate retrieval separately from answer generation. The model cannot reliably cite the right document if the retriever never supplied it.
## Compare changes as an experiment
Run the same cases against the current and proposed systems. Record the prompt, model identifier, configuration, retrieval setup, and tool versions needed to explain differences.
Measure latency and cost alongside quality. A modest quality gain may be unacceptable if interactive responses become too slow. Conversely, a cheaper configuration is not an improvement if it fails the most important task category.
Review failures by type instead of relying only on one aggregate score. A better overall result can hide a regression on high-impact cases such as account-specific policy or financial calculations.
After release, monitor real outcomes and feed verified failures into future evaluation sets without leaking sensitive data. Offline evaluation cannot perfectly predict a changing production workload.
The objective is not to prove that the AI is generally intelligent. It is to gather evidence that this system handles this task well enough, under the constraints and risks of the product you are building.
## Further reading
- [OpenAI: Evaluation best practices](https://platform.openai.com/docs/guides/evaluation-best-practices)
- [Anthropic: Define success criteria and build evaluations](https://docs.anthropic.com/en/docs/test-and-evaluate/define-success)
Read the full article on Stackzilla →