PromptingTest your prompts

Test your prompts

A prompt is code that runs on a probabilistic interpreter. Changing a word can fix one case and break three others, and switching models can change behaviour in ways no one notices until users complain. A small, automated test set catches these changes before they ship. This page shows how to build one and run it against the Syntic API.

Define success first

Write down what “good” means before writing the prompt. Good criteria are specific and measurable:

VagueMeasurable
”Accurate answers""At least 45 of 50 test questions answered correctly; zero answers that contradict the policy."
"Appropriate tone""No exclamation marks; average reply under 90 words; grader rates politeness 4 or 5 on every case."
"Handles edge cases""All 8 out-of-scope questions get the fixed fallback sentence."
"Fast""95% of replies complete in under 4 seconds with streaming.”

Build the test set

Collect real inputs

Use logs, support tickets, or a sample from the people who will use the feature. Twenty to fifty cases is enough to start; two hundred catches subtler regressions.

Add the hard cases on purpose

Empty input, very long input, another language, typos, sarcasm, out-of-scope requests, attempts to change the rules. These are where prompts fail.

Record the expected outcome

For each case, write either the exact expected output (for classification and extraction) or the properties a good answer must have (for open-ended text).

Keep every past failure

When a user reports a bad answer, add it to the set. The set then grows into a record of everything that has ever gone wrong.

A JSON Lines file works well:

{"id": "refund-14d", "input": "My order arrived damaged 10 days ago", "expect": {"contains": ["replacement"], "max_words": 90}}
{"id": "oos-essay", "input": "Write my essay on the French revolution", "expect": {"equals": "Sorry, I can only help with orders and deliveries."}}
{"id": "label-billing", "input": "Charged twice for October", "expect": {"label": "billing"}}

Three ways to grade

GraderGood forExample
CodeExact answers, labels, JSON validity, length, forbidden wordslabel == expected, len(words) <= 90
ModelQualities that need judgement: correctness against a reference, tone, completenessA grading request with a rubric that returns a score
PersonFinal sign-off, subtle quality, anything safety-criticalA reviewer reads a sample each release

Use code wherever possible; it is fast, free and exact. Use model grading for the rest, and spot-check the grader itself against human judgement.

A test runner

# run_tests.py — python run_tests.py cases.jsonl
import json
import os
import sys
import time
from concurrent.futures import ThreadPoolExecutor
 
from openai import OpenAI
 
client = OpenAI(base_url="https://app.syntic.ai/v1", api_key=os.environ["SYNTIC_API_KEY"])
MODEL = os.environ.get("TEST_MODEL", "Amara S1.2")
SYSTEM = open("prompt.txt").read()
 
GRADER = """
You grade an assistant reply against a reference answer.
Reply with JSON only: {"score": 1-5, "reason": "one sentence"}.
5 = same facts as the reference, nothing wrong added. 1 = wrong or contradicts the reference.
""".strip()
 
 
def run_case(case):
    t0 = time.monotonic()
    r = client.chat.completions.create(
        model=MODEL, stream=False, max_tokens=500,
        messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": case["input"]}],
    )
    out = r.choices[0].message.content or ""
    exp, problems = case["expect"], []
    if "equals" in exp and out.strip() != exp["equals"]:
        problems.append("not the exact expected text")
    for word in exp.get("contains", []):
        if word.lower() not in out.lower():
            problems.append(f"missing '{word}'")
    if "max_words" in exp and len(out.split()) > exp["max_words"]:
        problems.append(f"{len(out.split())} words")
    if "reference" in exp:
        g = client.chat.completions.create(
            model="Amara S1.2", stream=False, max_tokens=120,
            messages=[{"role": "system", "content": GRADER},
                      {"role": "user", "content": f"<reference>{exp['reference']}</reference>\n<reply>{out}</reply>"}],
        ).choices[0].message.content or "{}"
        score = json.loads(g[g.find("{"): g.rfind("}") + 1]).get("score", 0)
        if score < 4:
            problems.append(f"grader score {score}")
    return {"id": case["id"], "ok": not problems, "problems": problems,
            "seconds": round(time.monotonic() - t0, 1), "tokens": r.usage.total_tokens}
 
 
cases = [json.loads(line) for line in open(sys.argv[1]) if line.strip()]
with ThreadPoolExecutor(max_workers=6) as pool:
    results = list(pool.map(run_case, cases))
 
for res in results:
    if not res["ok"]:
        print(f"FAIL {res['id']}: {', '.join(res['problems'])}")
passed = sum(r["ok"] for r in results)
print(f"{passed}/{len(results)} passed, {sum(r['tokens'] for r in results)} tokens, model {MODEL}")
sys.exit(0 if passed == len(results) else 1)

Run it with TEST_MODEL="Amara Quick" python run_tests.py cases.jsonl to compare models on the same cases.

Dealing with variation

The same prompt can give slightly different replies on different runs, and sampling settings cannot be pinned on the Syntic API. Account for this in the tests:

  • Grade properties (“mentions the 30-day window”) rather than exact wording, except where the output really must be exact.
  • Run important cases three times and require all three to pass. A case that passes two out of three times is a weak spot.
  • Compare pass rates between versions, not single runs.

When to run the tests

  • Every time the prompt changes, before merging.
  • Before switching models, and when the model list changes; see Deprecations.
  • On a schedule (weekly) to catch drift in your own data.
  • In CI, with a dedicated API key and a spend limit; see Managing API keys.

Test runs are normal requests and use your plan allowance or credits. A 200-case suite with model grading is typically a few hundred thousand tokens; check the total the runner prints.

Related pages

Interrupted & declined replies
Was this page helpful?