Choose graders
A grader turns an output into a score. Pick the cheapest grader that measures what you care about.
| Kind | Checks | Cost | Good for |
|---|---|---|---|
exact_match | Output equals ideal after trimming | Free | Labels, ids, yes/no |
contains | Output contains (or does not contain) given strings | Free | Required disclaimers, banned phrases |
regex | Output matches a pattern | Free | Formats such as order numbers, dates |
json_schema | Output parses and fits a schema | Free | Structured output, tool arguments |
similarity | Embedding similarity to ideal above a threshold | Embedding tokens | Paraphrase-tolerant factual answers |
model_rubric | A model scores the output 1 to 5 against your rubric | Model tokens | Tone, completeness, faithfulness, reasoning |
Combine them
A run can apply up to five graders; an item passes only if all pass. A typical support set uses:
json_schemawhen the reply must be structured,containswithnot: ["guarantee", "100%"]for words legal does not allow,model_rubricfor overall quality.
Code graders catch hard failures for free; the rubric judges what code cannot.
When to trust a model grader
Model graders are only as good as their rubric and calibration. Before relying on one, grade 30 to 50 outputs by hand, run the grader on the same outputs with Test a grader, and compare. Agreement of 85% or more on pass/fail is a reasonable bar. Where they disagree, read both judgments; usually the rubric was ambiguous. See Write rubrics.
Grader independence
Use a different model, or at least a different prompt, for grading than for generating, and prefer a stronger model for grading (Amara Deep is a common choice). A model grading its own outputs with the instructions it was given tends to agree with itself.