Reduce latency

Two numbers describe how fast a model feature feels: time to first token (how long until text starts to appear) and total time (until the reply is complete). People reading a chat care most about the first; a pipeline waiting for a JSON object cares about the second. Measure both before changing anything, then work through the levers below, roughly in order of effect.

Measure first

import os
import time
 
from openai import OpenAI
 
client = OpenAI(base_url="https://app.syntic.ai/v1", api_key=os.environ["SYNTIC_API_KEY"])
 
 
def timed(model: str, messages: list, max_tokens: int = 400):
    start = time.perf_counter()
    first = None
    pieces = []
    stream = client.chat.completions.create(model=model, stream=True, max_tokens=max_tokens, messages=messages)
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            first = first or time.perf_counter()
            pieces.append(chunk.choices[0].delta.content)
    end = time.perf_counter()
    return {"model": model, "ttft_s": round((first or end) - start, 2), "total_s": round(end - start, 2), "chars": len("".join(pieces))}
 
 
msgs = [{"role": "user", "content": "Write three short tips for keeping houseplants alive in winter."}]
for m in ["Amara Quick", "Amara S1.2"]:
    print(timed(m, msgs))

import OpenAI from 'openai'
 
const client = new OpenAI({ baseURL: 'https://app.syntic.ai/v1', apiKey: process.env.SYNTIC_API_KEY })
 
async function timed(model, messages, max_tokens = 400) {
  const start = performance.now()
  let first = null
  let chars = 0
  const stream = await client.chat.completions.create({ model, stream: true, max_tokens, messages })
  for await (const chunk of stream) {
    const piece = chunk.choices[0]?.delta?.content
    if (piece) { first ??= performance.now(); chars += piece.length }
  }
  const end = performance.now()
  return { model, ttft_ms: Math.round((first ?? end) - start), total_ms: Math.round(end - start), chars }
}
 
const msgs = [{ role: 'user', content: 'Write three short tips for keeping houseplants alive in winter.' }]
for (const m of ['Amara Quick', 'Amara S1.2']) console.log(await timed(m, msgs))

Run each measurement several times and compare medians; single requests vary.

The levers

1. Stream anything a person waits for

Streaming does not make the model faster, but text starts to appear as soon as the first tokens exist, which changes how fast the feature feels more than anything else. Streaming is on by default on the Syntic API. See Streaming.

2. Ask for less output

Total time grows with the number of output tokens. The prompt usually decides that number:

• Ask for the length you need in concrete terms: “two sentences”, “at most five bullets”.
• For data, request compact JSON without explanations.
• Set max_tokens as a ceiling so a runaway reply cannot take a minute.

Cutting a 600-token answer to 150 tokens typically saves more time than any other change.

3. Pick the lighter model where it is good enough

Amara Quick is built for short, frequent exchanges where speed matters. For classification, routing, extraction of a few fields and short chat replies, test it against your quality bar before defaulting to Amara S1.2. For software work where turnaround matters more than price, Syntic Code Fast is the faster counterpart of Syntic Code. See Choosing a model.

4. Keep prompts stable and lean

• Put fixed content (instructions, examples, tool definitions) first and keep it byte-for-byte identical between requests, so the beginning of the prompt is served from cache.
• Trim old turns and long tool results from conversations. A request carrying 80,000 tokens of history takes longer to start than one carrying 4,000.
• Offer only the tools a request can use; each definition adds input to every call.

5. Cut round trips

Every tool round is another full request. Where you can:

• Fetch data the model will certainly need before the first request and put it in the prompt, rather than offering a tool for it.
• Let one tool return everything a typical question needs, instead of three narrow tools called in sequence.
• Run independent requests in parallel (classification and summarisation of the same email, for example) instead of chaining them.

6. Parallelise batch work

For many independent items, run several requests at once within your rate limit (600 requests a minute per key) instead of one after another. A worker pool of 8 to 16 concurrent requests is a common starting point. See Retries with back-off for handling the occasional 429.

7. Reuse connections

Create one HTTP client and reuse it, so TLS connections are kept open between requests. Creating a new client for every call adds a handshake to each one.

What does not help

• Lowering temperature. Sampling settings are ignored on the Syntic API and do not affect speed.
• Asking the model to “be quick”. Speed comes from fewer tokens and smaller prompts, not from instructions about speed.
• Retrying slow requests early. A second request competes with the first and doubles the cost; set a sensible timeout instead and retry only on failure.

Related pages

• Streaming
• Control costs
• Choosing a model