Count tokens before sending
⚠️
Preview — this feature is being built; the design below is the spec.
Long conversations eventually outgrow the context window. Count first, trim if needed, then send, so users never see context_length_exceeded.
import os, requests
API = "https://app.syntic.ai/v1"
H = {"Authorization": f"Bearer {os.environ['SYNTIC_API_KEY']}"}
WINDOW = 262_144
RESERVE = 4_096 # room for the reply (max_tokens)
SAFETY = 1_000 # tool definitions and formatting overhead vary slightly
def count(body: dict) -> int:
r = requests.post(f"{API}/chat/completions/count_tokens", headers=H, json=body, timeout=30)
r.raise_for_status()
return r.json()["input_tokens"]
def fit(system: dict, history: list[dict], model="Amara S1.2") -> list[dict]:
msgs = [system, *history]
while len(msgs) > 2 and count({"model": model, "messages": msgs}) > WINDOW - RESERVE - SAFETY:
msgs.pop(1) # drop the oldest turn after the system message
if msgs[1]["role"] == "tool": # never leave a tool result without its call
msgs.pop(1)
return msgs
messages = fit({"role": "system", "content": SYSTEM_PROMPT}, conversation)
reply = requests.post(f"{API}/chat/completions", headers=H,
json={"model": "Amara S1.2", "max_tokens": RESERVE, "messages": messages}, timeout=120).json()Make it cheaper
Counting is free but is a round trip. For chat UIs, count once per turn and keep a running estimate; only call the counting endpoint again when the estimate is within 10% of the limit. Dropping whole turns keeps the remaining conversation coherent; for agents, prefer server-side context management.
See Token counting.
Was this page helpful?