Build a dataset
A dataset is the set of cases every candidate is judged on. A good one decides whether your evaluations mean anything.
Where cases come from
- Real traffic. Sample logged requests (with consent and personal data removed). This is the best source; it contains the phrasing and mess real users produce.
- Known failures. Every bug report and complaint becomes a case, so the same mistake cannot return unnoticed.
- Edge cases written on purpose. Empty input, very long input, other languages, contradictory requests, questions outside your product.
- Generated variations. Ask a model to rephrase real cases. Useful for volume, but check them; generated cases tend to be cleaner than reality.
How many
| Purpose | Size |
|---|---|
| Quick check while editing a prompt | 20 to 50 |
| Decision between two variants | 200 to 500 |
| Release gate | 500+, stratified by tag |
With 200 cases, a difference of 5 points in pass rate is usually meaningful; with 30 it often is not.
Tag everything
Give each item tags such as billing, how-to, angry-customer, de. Results can then be broken down by tag, which shows where a change helps or hurts. A variant that wins overall but drops 20 points on de is not a win for your German customers.
Ideal answers
An ideal is needed for exact-match and similarity graders, and helps rubric graders. Write ideals the way you want the model to answer, not just with the facts. Have a second person review them; a wrong ideal teaches you to prefer wrong answers.
Keep a held-out set
If you tune prompts against the same 200 cases for weeks, you will overfit to them. Keep 20% of cases aside and only run them before a release.
Load it
Small sets: Add items. Large sets: write JSONL, upload with purpose eval_dataset, and import:
{"input": {"ticket": "I was charged twice for order 88123."}, "ideal": "Apologise, confirm the duplicate charge, refund within 3 working days.", "tags": ["billing"]}