Evaluate and improve
⚠️
Preview — this feature is being built; the design below is the spec.
Prompts and model choices are code: they change behaviour, and changes need tests. This section covers how to measure whether your application is doing its job and how to improve it with evidence, using Syntic’s evaluation tools or your own.
The loop
- Define success in measurable terms before writing prompts. Define success
- Collect test cases that look like real traffic, including the hard ones. Build a dataset
- Decide how to score each output. Choose graders, Write rubrics
- Run your current setup to get a baseline, then each candidate change. Run evaluations
- Read the failures, not just the score, and fix the cause. Read results
- Guard against regressions on every change. Evaluations in CI
Tools
| Tool | Use |
|---|---|
| Evaluations API | Datasets, graders and runs. |
| Prompt tools | Draft, improve and templatize prompts. |
| Prompt templates | Version prompts so runs and production refer to the same text. |
| Human review | Where model graders are not trusted yet. |
| Safety evaluations | Prompt injection, data leakage and refusal checks. |
The Prompting section covers how to write the prompts themselves; Test your prompts is a short introduction to the same ideas.
Was this page helpful?