Increase consistency

The same prompt can produce differently worded answers on different runs. For chat that is usually fine. For features where code reads the output, where answers are compared over time, or where customers expect the same answer to the same question, you want the variation as small as possible. This page collects the techniques that work on the Syntic API today.

Specify the output exactly

Variation lives in whatever the prompt leaves open. Close the gaps:

Fixed labels and templates are the single most effective tool. See Format the output.

Show examples of the exact shape

Two or three examples of correct output teach the shape more reliably than prose. Keep every example identical in format; inconsistencies in examples become inconsistencies in output. See Use examples.

Validate and retry

Treat the model’s output like any external input: check it, and ask again when it fails.

import json
import os
 
from openai import OpenAI
 
client = OpenAI(base_url="https://app.syntic.ai/v1", api_key=os.environ["SYNTIC_API_KEY"])
LABELS = {"urgent", "normal", "low"}
 
SYSTEM = """
Classify the urgency of a support ticket.
urgent: service down, payment failing, data loss, or a security problem.
normal: something is wrong but there is a workaround.
low: questions, feedback, feature requests.
Reply with one word: urgent, normal or low.
""".strip()
 
 
def urgency(ticket: str) -> str:
    messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": ticket}]
    for _ in range(2):
        r = client.chat.completions.create(model="Amara Quick", stream=False, max_tokens=5, messages=messages)
        label = (r.choices[0].message.content or "").strip().lower().strip(".")
        if label in LABELS:
            return label
        messages += [{"role": "assistant", "content": label},
                     {"role": "user", "content": "Reply with exactly one of: urgent, normal, low."}]
    return "normal"   # a safe default you have chosen deliberately

Cache answers you have already given

For repeated questions (FAQ bots, product descriptions, translations of fixed strings), the most consistent answer is the one you stored last time. Key a cache on a normalised form of the input and the prompt version, and serve stored answers before calling the API. It is also cheaper and faster.

import hashlib
 
PROMPT_VERSION = "faq-v7"
 
def cache_key(question: str) -> str:
    normalised = " ".join(question.lower().split())
    return hashlib.sha256(f"{PROMPT_VERSION}|{normalised}".encode()).hexdigest()

Bump PROMPT_VERSION whenever the prompt changes, so old answers are not served for a new prompt.

Ground answers in the same material

Answers drift most when the model fills gaps from general knowledge. Give it the facts (a policy text, retrieved passages, tool results) and tell it to answer only from them. Two people asking the same question then get answers built from the same sentences. See Reduce hallucinations.

Break complex judgements into steps

A single prompt that makes several decisions at once varies more than a chain where each step makes one decision with its own fixed output. See Chain prompts.

Measure variation

Run each test case several times and count how often the answers agree:

from collections import Counter
 
def agreement(fn, case: str, runs: int = 5) -> float:
    answers = Counter(fn(case) for _ in range(runs))
    return answers.most_common(1)[0][1] / runs     # 1.0 = always the same

Cases with low agreement are where the prompt is ambiguous. Read them, clarify the instruction, and re-measure. See Test your prompts.

Sampling control

Related pages

• Format the output
• Extract data to JSON
• Classify text