Read results

The summary tells you which variant won. The individual results tell you why, and what to change next.

Start with the summary

"summary": [
  {"label": "baseline", "pass_rate": 0.82, "mean_score": 4.11, "mean_output_tokens": 141, "cost_usd": "1.8410"},
  {"label": "short", "pass_rate": 0.86, "mean_score": 4.25, "mean_output_tokens": 98, "cost_usd": "1.4022"}
]

Here the shorter prompt passes more, scores higher and costs less. Before adopting it, check the breakdown and the failures.

Break down by tag

Fetch results per tag (filter items by tag in your own analysis, or run with item_tags). Look for any tag where the candidate is worse than the baseline even if the total is better.

Read the failures

curl -sS "https://app.syntic.ai/v1/evals/runs/evrun_01JD35D7F9H1K3M5P7R9/results?label=short&passed=false" \
  -H "Authorization: Bearer $SYNTIC_API_KEY" -H "syntic-beta: evals-2026-12-01"

Read at least 20 failures. Group them by cause: wrong fact, missing step, wrong format, grader mistake. Fix the biggest group first.

Look at disagreements

disagreements=true returns items where one variant passed and another failed. These are the cases that decide between variants, and often reveal what each prompt does differently.

Watch for grader errors

If many failures look fine to you, the grader is wrong, not the model. Fix the rubric and rerun the grading, rather than tuning the prompt to please a broken grader.

Record the decision

Keep a short note with each run: what changed, the result, the decision. Months later it explains why the prompt says what it says.