Python SDK: Evaluations and prompts

Datasets, graders, runs, prompt tools and templates.

from syntic import Syntic
 
client = Syntic()  # reads SYNTIC_API_KEY

Methods

evals.datasets.create

Create a dataset. Maps to POST /v1/evals/datasets.

result = client.evals.datasets.create(name="support-replies-v3")

evals.datasets.items.add

Add up to 1,000 items. Maps to POST /v1/evals/datasets/{dataset_id}/items.

result = client.evals.datasets.items.add(
    'evds_01JD13B5D7F9H1K3M5P7',
    items=[{"input": {"ticket": "…"}, "ideal": "…"}],
)

evals.graders.create

Define a grader. Maps to POST /v1/evals/graders.

result = client.evals.graders.create(
    name="no-guarantees",
    kind="contains",
    config={"not": ["guarantee"]},
)

evals.runs.create

Start a run with up to four variants. Maps to POST /v1/evals/runs.

result = client.evals.runs.create(
    dataset_id="evds_01JD13B5D7F9H1K3M5P7",
    variants=[{"label": "baseline", "model": "Amara S1.2", "prompt_version": "support@production"}],
    grader_ids=["evgr_01JD46E8G0J2L4N6Q8S0"],
)

evals.runs.wait

Helper: wait for a run to finish.

result = client.evals.runs.wait('evrun_01JD35D7F9H1K3M5P7R9')

evals.runs.results

Iterate graded results. Maps to GET /v1/evals/runs/{eval_run_id}/results.

for item in client.evals.runs.results('evrun_01JD35D7F9H1K3M5P7R9', passed=False):
    print(item.id if hasattr(item, 'id') else item)

prompts.improve

Rewrite a prompt against feedback. Maps to POST /v1/prompts/improve.

result = client.prompts.improve(system="…", feedback="Too long.")

prompts.templates.set_label

Promote a version. Maps to PUT /v1/prompts/templates/{template}/labels/{label}.

result = client.prompts.templates.set_label('support', 'production', version=13)

Conventions

Every method accepts extra_headers, extra_query, extra_body and timeout keyword arguments, and has an async twin on AsyncSyntic. List methods return pages that iterate across all pages automatically.

See the Python SDK overview for installation, errors and retries.