Answer questions from your documents

Retrieval-augmented generation (RAG) lets a model answer from material it has never seen: your handbook, product manuals, internal wiki. For each question your code finds the few passages most likely to hold the answer and puts them in the request; the model answers from those passages and says which ones it used.

This guide builds a question-answering service over the staff handbook of a fictional hotel group, Harbourline. It uses keyword search, which needs no extra services, and shows where semantic search slots in.

You will need: an API key in SYNTIC_API_KEY, Python 3.9 or later, and pip install openai rank-bm25.

Do you need retrieval at all?

The context window holds 262,144 tokens, which is roughly 600 pages of prose. If your whole knowledge base fits comfortably (say under 150,000 tokens) and does not change often, the simplest design is to put all of it in the system message and let prompt caching make repeat requests cheap. Read Long documents for how to lay it out.

Retrieval earns its complexity when the material is larger than that, changes daily, or must be filtered per user (for example, documents a member of staff is allowed to see).

How it fits together

 Indexing (when documents change)          Answering (per question)
 ────────────────────────────────          ─────────────────────────
 documents ─▶ split into passages          question ─▶ search index ─▶ top 6 passages
           ─▶ build search index                                       │
                                            system rules + passages + question
                                                       │
                                            POST /v1/chat/completions ─▶ answer with [n] markers

Split documents into passages

Passages should be small enough that six of them fit easily in one request, and large enough to hold a complete answer. Splitting on headings first and then on paragraphs keeps related sentences together.

import re
from dataclasses import dataclass
 
 
@dataclass
class Passage:
    id: int
    source: str      # file name or URL, shown to the reader
    heading: str
    text: str
 
 
def split_markdown(source: str, text: str, max_chars: int = 1800) -> list[Passage]:
    passages, heading, buf = [], "", []
 
    def flush():
        body = "\n".join(buf).strip()
        if body:
            passages.append(Passage(0, source, heading, body))
        buf.clear()
 
    for line in text.splitlines():
        if re.match(r"^#{1,3} ", line):
            flush()
            heading = line.lstrip("# ").strip()
        else:
            buf.append(line)
            if sum(len(l) for l in buf) > max_chars and line.strip() == "":
                flush()
    flush()
    return passages

Keeping the heading with each passage matters: “Requests must be made 14 days ahead” means nothing until you know it sits under “Annual leave”.

Build a keyword index

from pathlib import Path
 
from rank_bm25 import BM25Okapi
 
passages: list[Passage] = []
for path in Path("handbook").glob("**/*.md"):
    passages += split_markdown(str(path), path.read_text())
for i, p in enumerate(passages):
    p.id = i
 
def tokens(s: str) -> list[str]:
    return re.findall(r"[a-z0-9]+", s.lower())
 
index = BM25Okapi([tokens(p.heading + " " + p.text) for p in passages])
 
 
def search(question: str, k: int = 6) -> list[Passage]:
    scores = index.get_scores(tokens(question))
    best = sorted(range(len(passages)), key=lambda i: scores[i], reverse=True)[:k]
    return [passages[i] for i in best if scores[i] > 0]

Keyword search is strong on names, codes and exact terms (“form HR-12”, “night audit”), and weak on paraphrase (“time off” against “annual leave”). The query rewrite step below closes much of that gap.

⚠️

Preview — this feature is being built; the design below is the spec. A Syntic embeddings endpoint for semantic search is specified on Embeddings. Until it ships, you can combine keyword search with an embedding service you already run; the rest of this guide does not change.

Rewrite the question for search

People ask “can I swap my Sunday shift?”; the handbook says “shift exchanges”. A short, cheap request turns the question into search terms first:

import os
 
from openai import OpenAI
 
client = OpenAI(base_url="https://app.syntic.ai/v1", api_key=os.environ["SYNTIC_API_KEY"])
 
 
def search_terms(question: str) -> str:
    r = client.chat.completions.create(
        model="Amara Quick",
        stream=False,
        max_tokens=60,
        messages=[
            {"role": "system", "content": "Rewrite the staff question as 5-10 search keywords a hotel HR handbook would use, including synonyms. Reply with the keywords only, space separated."},
            {"role": "user", "content": question},
        ],
    )
    return question + " " + (r.choices[0].message.content or "")

Answer from the passages

RULES = """
You answer questions from Harbourline hotel staff using only the handbook passages provided.
 
- Base every statement on the passages. After each sentence that uses a passage, add its number in square brackets, e.g. [2].
- If the passages do not contain the answer, say "The handbook doesn't cover this" and suggest asking the duty manager. Do not answer from general knowledge.
- If passages disagree, say so and cite both.
- Keep answers under 120 words unless the question asks for a full procedure.
""".strip()
 
 
def answer(question: str) -> dict:
    found = search(search_terms(question))
    context = "\n\n".join(
        f'<passage n="{n}" source="{p.source}" section="{p.heading}">\n{p.text}\n</passage>'
        for n, p in enumerate(found, start=1)
    )
    r = client.chat.completions.create(
        model="Amara S1.2",
        stream=False,
        max_tokens=500,
        messages=[
            {"role": "system", "content": RULES},
            {"role": "user", "content": f"{context}\n\nQuestion: {question}"},
        ],
    )
    text = r.choices[0].message.content or ""
    used = sorted({int(n) for n in re.findall(r"\[(\d+)\]", text) if 0 < int(n) <= len(found)})
    return {
        "answer": text,
        "sources": [{"n": n, "source": found[n - 1].source, "section": found[n - 1].heading} for n in used],
    }
 
 
print(answer("Can I swap my Sunday shift with someone?"))

Why it is laid out this way:

Passages first, question last. With long material, putting the question at the end gives better answers.
Numbered passages with their source. The model cites by number; your code turns numbers back into links, so a reader can check the answer.
An explicit “not covered” path. Without it, the model fills gaps from general knowledge, which for a staff handbook is worse than no answer.

Server-side citations, where the response marks exactly which document span backs each sentence, are planned; see Citations. The bracket convention above works today and costs nothing extra.

Keep access rules in your code

If some documents are restricted (management-only policies, one hotel’s procedures), filter passages before they reach the prompt. A model told “do not reveal the management section” can still be talked into it; a passage that was never sent cannot leak.

def search_for(user, question):
    return [p for p in search(question, k=20) if user.may_read(p.source)][:6]

Evaluate

Write 50 questions with the answer and the passage that holds it, including ten questions the handbook does not answer. Then check three things per question:

Check	How
Retrieval hit	Was the right passage among those found? If not, fix splitting or search, not the prompt.
Answer correct	Compare with the expected answer, by hand or with a second grading request.
Honest gaps	For the unanswerable questions, did it say the handbook does not cover it?

Retrieval misses are the most common cause of wrong answers, so measure them separately. See Test your prompts for a grading loop.

Costs and limits

Six passages of about 400 tokens each plus rules and question come to roughly 3,000 input tokens per answer, plus a few dozen for the rewrite step on Amara Quick. The rules text is identical on every request and is served from cache. Requests are limited to about 2 MB of JSON and 500 messages; see Limits.

Prompting: long documents
Prompting: reduce hallucinations
Summarize long documents
Build a chatbot with memory: add follow-up questions to this service.

Review contracts Build semantic search

Was this page helpful?