Evaluations in CI
⚠️
Preview — this feature is being built; the design below is the spec.
Run a small evaluation automatically on every change to a prompt template, model setting or tool definition, and block the merge if quality drops.
A GitHub Actions job
name: prompt-evals
on:
pull_request:
paths: ["prompts/**", "tools/**"]
jobs:
eval:
runs-on: ubuntu-latest
permissions: { id-token: write, contents: read }
steps:
- uses: actions/checkout@v4
- run: pip install syntic
- run: python scripts/run_eval.py --dataset evds_01JD13B5D7F9H1K3M5P7 --candidate prompts/support.md --min-delta -0.02The id-token: write permission lets the job use workload identity instead of a stored key.
The script
import sys, time
from syntic import Syntic
client = Syntic()
run = client.evals.runs.create(
dataset_id=DATASET,
variants=[
{"label": "baseline", "model": "Amara S1.2", "prompt_version": "support@production"},
{"label": "candidate", "model": "Amara S1.2", "messages": load_template(CANDIDATE)},
],
grader_ids=GRADERS,
priority="interactive",
)
while run.status in ("queued", "running"):
time.sleep(15)
run = client.evals.runs.retrieve(run.id)
s = {v.label: v.pass_rate for v in run.summary}
print(f"baseline {s['baseline']:.2%} candidate {s['candidate']:.2%}")
sys.exit(0 if s["candidate"] - s["baseline"] >= MIN_DELTA else 1)Keep CI runs small and fast
Use a 50 to 100 item smoke set in CI and the full set before release. With priority: "interactive" a 100-item run with one rubric grader typically finishes in a few minutes and costs well under a dollar on Amara S1.2.
Post results to the pull request
Print the summary table and links to failing results; reviewers then judge the change on evidence instead of reading the prompt diff alone.
Was this page helpful?