Evaluations API
⚠️
Preview — this feature is being built; the design below is the spec.
The Evaluations API runs your prompts against a fixed set of test cases and scores the results, so a change to a prompt, a model or a parameter is judged on evidence instead of a few hand-picked examples. It has three resources:
| Resource | What it is |
|---|---|
| Datasets | Test inputs, each with an optional ideal answer and tags. |
| Graders | Rules that score an output: exact match, contains, regex, JSON schema, similarity, or a model with a rubric. |
| Runs | Up to four prompt and model variants, run over a dataset and graded. |
Preview flag: syntic-beta: evals-2026-12-01. All three resources belong to the key’s workspace.
A typical loop
- Collect 50 to 500 real inputs, including the awkward ones, into a dataset. Add ideal answers where a reference exists.
- Write code graders for anything mechanical (valid JSON, required fields, banned phrases) and one rubric grader for quality.
- Calibrate the rubric with Test a grader on answers you already judged by hand.
- Run your current prompt as a baseline variant, and your candidate as a second variant.
- Read the failures with List results filtered to
passed=false, fix the prompt, run again.
Cost
Generation is billed like a batch request, at the batch discount, unless the run uses priority: interactive. Rubric graders are billed as ordinary requests to their grading model. Code graders are free. A run’s summary[].cost_usd reports generation cost per variant; grading cost appears in usage under the eval_grading line.
Limits
| Limit | Value |
|---|---|
| Items per dataset | 50,000 |
| Variants per run | 4 |
| Graders per run | 5 |
| Concurrent runs per workspace | 5 |
| Results kept | 90 days after the run ends |
Was this page helpful?