Run evaluations

A run executes one to four variants over a dataset and grades every output. A variant is a model plus a prompt.

Start a run

curl -sS https://app.syntic.ai/v1/evals/runs \
  -H "Authorization: Bearer $SYNTIC_API_KEY" \
  -H "syntic-beta: evals-2026-12-01" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "quick vs s1.2 on billing",
    "dataset_id": "evds_01JD13B5D7F9H1K3M5P7",
    "variants": [
      {"label": "s12", "model": "Amara S1.2", "prompt_version": "support-v13"},
      {"label": "quick", "model": "Amara Quick", "prompt_version": "support-v13"}
    ],
    "grader_ids": ["evgr_01JD46E8G0J2L4N6Q8S0"],
    "item_tags": ["billing"]
  }'

Compare one thing at a time

Change the prompt or the model in a run, not both, or you will not know which change caused the difference. A useful pattern is always including the production setup as the first variant (label: "baseline").

Timing and cost

Runs use the batch path by default: half the token price and usually done within an hour. Pass priority: "interactive" while iterating on a small set if you want results in minutes. Watch progress with Get a run or subscribe to the eval.run.completed webhook.

Repeatability

Each item is generated once per variant. Model outputs vary a little between runs, so a difference of one or two points on a few hundred items can be noise. For close decisions, run twice or use a larger set.

Templates and variables

Inline templates use {{variable}} placeholders filled from each item’s input:

{
  "label": "short",
  "model": "Amara S1.2",
  "messages": [
    {"role": "system", "content": "You answer billing questions for Acme. Under 120 words."},
    {"role": "user", "content": "Customer tier: {{customer_tier}}\n\n{{ticket}}"}
  ]
}

A placeholder with no matching input key fails the run at creation with unknown_variable, before anything is billed.