Reduce hallucinations

A hallucination is a confident statement that is not true or not supported by the material the model was given: a policy clause that does not exist, an invented order status, a citation to a page that says something else. No prompt removes the risk completely, but a few techniques reduce it sharply, and good system design catches most of what remains.

Why it happens

The model is built to produce a plausible continuation. When the true answer is not in front of it, a plausible-sounding answer is still available, and nothing in a plain prompt tells the model that “I don’t know” is acceptable. Most fixes therefore do one of two things: make the true answer available, or make “I don’t know” an explicit, welcome option.

1. Give the facts, then restrict to them

Put the material in the prompt (see Work with long documents) and say plainly that it is the only allowed source:

Answer only from the policy text inside <policy> tags. If the policy does not answer the
question, reply exactly: "The policy doesn't cover that. Please contact HR."

A fixed fallback sentence is easy to detect in code, so you can count how often it fires and route those questions to a person.

2. Permit uncertainty explicitly

Tell the model that saying it is unsure is a correct answer, not a failure:

If you are not certain, say so and explain what information would settle it.
An honest "I'm not sure" is better than a guess.

3. Quote first, then answer

For questions over documents, ask for the supporting quotes before the answer, and instruct the model to retract any claim it cannot back with a quote:

1. In <quotes>, copy the sentences from the documents that answer the question, word for word.
2. In <answer>, answer using only those quotes, citing them as [1], [2].
3. If <quotes> is empty, the answer is "Not found in the documents."

Quoting is checkable. Your code can confirm each quote really appears in the source:

import re
 
def unsupported_quotes(reply: str, sources: list[str]) -> list[str]:
    """Return quotes from the reply that do not appear in any source text."""
    block = re.search(r"<quotes>(.*?)</quotes>", reply, re.S)
    quotes = re.findall(r'"([^"]{20,})"', block.group(1)) if block else []
    norm = lambda s: " ".join(s.split()).lower()
    corpus = [norm(s) for s in sources]
    return [q for q in quotes if not any(norm(q) in c for c in corpus)]

export function unsupportedQuotes(reply, sources) {
  const block = reply.match(/<quotes>([\s\S]*?)<\/quotes>/)?.[1] ?? ''
  const quotes = [...block.matchAll(/"([^"]{20,})"/g)].map((m) => m[1])
  const norm = (s) => s.split(/\s+/).join(' ').toLowerCase()
  const corpus = sources.map(norm)
  return quotes.filter((q) => !corpus.some((c) => c.includes(norm(q))))
}

If any quote is not found, treat the answer as unreliable: retry, or show the user the sources instead of the answer.

4. Look things up instead of remembering them

For anything that changes (prices, stock, order status, today’s date), give the model a tool and instruct it to use the tool rather than its own knowledge:

Never state an order's status from memory or from earlier in the conversation.
Call get_order_status every time the customer asks.

Also give the current date in the system message when dates matter; the model has no clock.

5. Ask for less

Hallucinations concentrate in details: exact figures, names, dates, URLs. If the user does not need them, do not ask for them. “Summarise the main argument” is safer than “list every statistic in the report”. When you do need details, extract them in a separate, focused request where each one can be checked (see Extract data to JSON).

6. Check with a second request

For high-stakes output, run a verification step: a separate request receives the source material and the draft answer and lists any statement not supported by the sources.

VERIFY = """
You check answers against source documents.
List every statement in <answer> that is not directly supported by <sources>.
Reply with JSON: {"unsupported": ["...", ...]} (an empty list if everything is supported).
""".strip()

A reviewer prompt that only checks is more reliable than asking the original prompt to “double-check your work”, because it is not invested in the draft.

7. Pick the right model

Models differ in how readily they admit uncertainty. If grounded prompts still produce unsupported statements in your tests, compare Amara S1.2 with Amara Deep on the same test set. See Choosing a model.

Design so mistakes are caught

Assume some answers will be wrong, and design the product so that matters less:

• Show sources next to answers so users can check them (Answer from your documents).
• Keep consequential actions behind your own code and human approval (Build an agent).
• Log answers that hit the “not covered” fallback and review them weekly; they show gaps in your material.
• Measure the rate of unsupported answers on a fixed test set after every change (Test your prompts).

Related pages

• Work with long documents
• Citations: planned server-side source attribution.
• Answer from your documents