Write rubrics

A rubric is the instruction a model grader follows. Write it as you would brief a careful human reviewer who has never seen your product.

Structure

1. What the output is for. “Replies to customers who emailed support about billing.”
2. What a 5 looks like, concretely.
3. What lowers the score, with the size of each deduction.
4. Automatic fails, if any.
5. What to ignore, so the grader does not penalise harmless variation.

Example

You are reviewing a reply to a customer's billing email.

Score 5 when the reply: states the correct facts from <policy>, apologises once if
the company was at fault, gives a concrete next step with a timeline, and is under
120 words.

Deduct:
- 2 for any fact that contradicts <policy> (for example a wrong refund time).
- 1 for missing the next step or the timeline.
- 1 for more than 150 words.

Score 1 (fail) if the reply promises something <policy> does not allow, or
mentions another customer.

Ignore greeting style, sign-off and formatting.

Reply with JSON: {"score": 1-5, "explanation": "<one sentence>"}

Tips

• Anchor every level. Graders drift toward 4 when only 5 is defined.
• Give the grader the facts. Pass the policy or the source document as part of the input; a grader cannot check facts it does not have.
• Ask for the explanation first when using a reasoning-heavy rubric, so the score follows the reasoning.
• One rubric per criterion is often better than one rubric for everything: separate “factually correct” from “tone”, so a failure says which one broke.
• Version rubrics like prompts. Changing a rubric changes every score; compare runs only when they used the same rubric version.