Evaluation glossary
Terms for testing and improving prompts. Shorter definitions of the core API terms are on the main Glossary.
Baseline
The current production setup, included in a run for comparison.
Calibration
Checking a model grader against human grades until they agree often enough. See human review.
Canary phrase
A meaningless phrase in a system prompt whose appearance in output reveals leakage. See safety.
Dataset
A fixed set of test inputs used to judge every change on the same cases. See datasets.
Disagreement
An item where variants differ in pass or fail. See reading results.
Grader
A rule that scores an output: code-based or a model with a rubric. See graders.
Held-out set
Cases kept out of everyday tuning and run only before release, to catch overfitting.
Ideal
The reference answer graders compare against.
Item
One test case: an input, optionally an ideal answer, and tags.
Label
A movable pointer such as production to a template version.
Pass rate
Share of items where every grader passed.
Prompt template
A named, versioned prompt with variables. See prompt templates.
Regression floor
The lowest acceptable score on any criterion; changes that fall below it do not ship. See success criteria.
Rubric
Instructions telling a model grader what each score means. See rubrics.
Run
Generating and grading every dataset item for each variant. See runs.
Variant
A model and prompt combination in an evaluation run.