EvaluatePrompt generator & improver

Prompt generator and improver

⚠️
Preview — this feature is being built; the design below is the spec.

Three helpers speed up prompt work. They are ordinary model calls with careful instructions behind them, priced at the target model’s rate.

Generate a first draft

Describe the task; get a system prompt and a user template back.

draft = client.prompts.generate(
    task="Classify support emails into billing, technical, account or other, and pull out any order number.",
    variables=["email_body"],
    output_format="json",
)
print(draft.system)
print(draft.notes)  # gaps the generator could not fill, e.g. what 'other' covers

Drafts follow the prompting guidance: role, context, explicit rules, output format, and placeholders for examples. Read notes: they list decisions only you can make.

Improve an existing prompt

Give the current prompt, what is wrong, and a few failing cases:

better = client.prompts.improve(
    system=current_system,
    feedback="Too long; sometimes promises refunds on sale items.",
    failing_examples=[{"input": "Can I return a sale item?", "output": "Of course!…", "expected": "Sale items are exchange-only within 14 days."}],
)
for c in better.changes:
    print("-", c.change, "because", c.reason)

Every change comes with its reason, so you can accept some and reject others.

Templatize

Turn a prompt with real values in it into a reusable template plus a first dataset item:

t = client.prompts.templatize(text=filled_prompt)
print(t.template)    # … into {{target_language}} for our {{store_location}} store …
print(t.variables)   # {"target_language": "German", "store_location": "Berlin", …}

Then evaluate

Generated and improved prompts are hypotheses. Save them as template versions and compare them with production in an evaluation run before adopting them. API reference: Prompt tools.

Was this page helpful?