Define success
“The answers should be good” cannot be tested. Before you write or change a prompt, write down what good means in a form a grader can check.
Make each criterion specific and measurable
| Vague | Measurable |
|---|---|
| Accurate | At least 95% of extracted invoice totals equal the total on the document. |
| Helpful | At least 90% of replies resolve the question without a follow-up, judged by a rubric. |
| Safe | 0 replies reveal another customer’s data across 300 adversarial cases. |
| Concise | Median reply under 120 words; none over 250. |
| Fast | 95th-percentile time to first token under 1.5 s. |
| Affordable | Mean cost per conversation under $0.02. |
Cover several dimensions
Most applications need a criterion in each of: task quality (correct, complete), format (parsable, right length), tone and policy (brand voice, no promises you do not keep), safety (no leaks, resists manipulation), and operations (latency, cost). A change that improves one often costs another; measuring all of them makes the trade-off visible.
Set thresholds from reality
Start by measuring what you have. If today’s prompt passes 78% of a rubric, a target of 99% next week is a wish. Aim for steady gains and set a regression floor: no change ships if it lowers any criterion by over two points against its current value.
Write them down where the team sees them
Keep criteria next to the prompt template in your repository, and name graders after the criterion they check (total_matches_document, under_250_words). When a run fails, the grader’s name says which promise was broken.