Classify text

Classification means assigning each piece of text one or more labels from a list you control: the intent of a chat message, the sentiment of a review, the department a document belongs to. This guide builds a review tagger for a fictional coffee-subscription service, Roastbox, which sorts app-store reviews by topic and sentiment so the product team can count them each week.

For a classifier that routes tickets to teams with a priority and a summary, see Route support tickets. This page focuses on the general technique: label design, multi-label output, confidence, and measuring accuracy.

You will need: an API key in SYNTIC_API_KEY and Node.js 18 or later (the example uses the built-in fetch).

Design the labels

The prompt

// prompt.js
export const SYSTEM = `
You tag customer reviews of Roastbox, a coffee subscription delivered by post.
 
Topics (choose every topic the review discusses, at least one):
- delivery: speed, lost or damaged parcels, letterbox fit, courier problems.
- taste: flavour, freshness, roast level, variety of beans.
- price: cost of coffee or delivery fees, discounts, value for money.
- app: the mobile app or website, bugs, login, notifications.
- subscription_management: pausing, skipping, cancelling, changing frequency or grind.
- customer_service: contact with the support team, response time, how problems were solved.
- other: anything not covered above.
 
Sentiment (one value for the whole review): positive, negative, mixed, neutral.
 
Reply with exactly one JSON object and no other text:
{"topics": ["..."], "sentiment": "...", "confidence": "high" | "low"}
Use "low" confidence when the review is sarcastic, very short, or could fit several readings.
`.trim()

Notes on the choices:

• Asking for a coarse confidence (high or low) gives you something to route on without pretending to be a probability.
• Listing the allowed values right in the output description reduces invented labels such as shipping or flavour. The code below still checks for them.

Classify one review

// classify.js  (node classify.js)
import { SYSTEM } from './prompt.js'
 
const TOPICS = new Set(['delivery', 'taste', 'price', 'app', 'subscription_management', 'customer_service', 'other'])
const SENTIMENTS = new Set(['positive', 'negative', 'mixed', 'neutral'])
 
export async function classify(review) {
  const res = await fetch('https://app.syntic.ai/v1/chat/completions', {
    method: 'POST',
    headers: { Authorization: `Bearer ${process.env.SYNTIC_API_KEY}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: 'Amara Quick',
      stream: false,
      max_tokens: 80,
      messages: [
        { role: 'system', content: SYSTEM },
        { role: 'user', content: `<review>\n${review}\n</review>` },
      ],
    }),
  })
  const data = await res.json()
  if (!res.ok) throw new Error(`${res.status} ${data.error}`)
 
  const text = data.choices[0].message.content ?? ''
  let out
  try {
    out = JSON.parse(text.slice(text.indexOf('{'), text.lastIndexOf('}') + 1))
  } catch {
    return { topics: ['other'], sentiment: 'neutral', confidence: 'low', invalid: true }
  }
  const topics = (out.topics ?? []).filter((t) => TOPICS.has(t))
  return {
    topics: topics.length ? topics : ['other'],
    sentiment: SENTIMENTS.has(out.sentiment) ? out.sentiment : 'neutral',
    confidence: out.confidence === 'high' ? 'high' : 'low',
  }
}
 
console.log(await classify('Beans are lovely but the parcel never fits through my letterbox, so I queue at the depot every month.'))
// { topics: [ 'taste', 'delivery' ], sentiment: 'mixed', confidence: 'high' }

curl https://app.syntic.ai/v1/chat/completions \
  -H "Authorization: Bearer $SYNTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Amara Quick",
    "stream": false,
    "max_tokens": 80,
    "messages": [
      {"role": "system", "content": "You tag customer reviews of Roastbox ... Reply with exactly one JSON object: {\"topics\": [...], \"sentiment\": ..., \"confidence\": ...}"},
      {"role": "user", "content": "<review>\nApp logs me out every day.\n</review>"}
    ]
  }'

The code never trusts the labels blindly: unknown topics are dropped, an empty list becomes other, and an unreadable reply becomes a low-confidence other. Count those invalid results; more than one or two percent means the prompt needs work.

Force the shape with a tool

You can also ask for the labels through a function definition. When tool_choice names the function, the reply carries the labels as the function’s arguments, a JSON string shaped by the parameters schema, rather than as free text:

const TAG_TOOL = {
  type: 'function',
  function: {
    name: 'record_tags',
    description: 'Record the topics and sentiment of one review.',
    parameters: {
      type: 'object',
      properties: {
        topics: { type: 'array', items: { type: 'string', enum: [...TOPICS] } },
        sentiment: { type: 'string', enum: [...SENTIMENTS] },
        confidence: { type: 'string', enum: ['high', 'low'] },
      },
      required: ['topics', 'sentiment', 'confidence'],
    },
  },
}
 
// In the request body:
//   tools: [TAG_TOOL],
//   tool_choice: { type: 'function', function: { name: 'record_tags' } },
// Then read: JSON.parse(data.choices[0].message.tool_calls[0].function.arguments)

The schema steers the model strongly, but the arguments are not yet validated against it on the server, so keep the same checks in your code. Guaranteed schema matching is planned; see JSON output and Strict tool arguments.

Add examples for hard cases

When the test run below shows a repeated confusion, add two or three short labelled examples to the system message. Pick real reviews from the confused pair, not easy ones:

Examples:
<review>Skipped two months in the app and still got charged.</review>
{"topics": ["subscription_management", "app", "price"], "sentiment": "negative", "confidence": "high"}

<review>Fine I guess.</review>
{"topics": ["other"], "sentiment": "neutral", "confidence": "low"}

More on choosing examples is in Use examples.

Measure accuracy

Hand-label 150 to 300 reviews, then compare. For multi-label output, measure each topic separately: how often the model tagged it when it should have (recall) and how often a tag it gave was right (precision).

// evaluate.js
import fs from 'node:fs'
import { classify } from './classify.js'
 
const rows = JSON.parse(fs.readFileSync('labelled_reviews.json', 'utf8')) // [{ text, topics: [...], sentiment }]
const stats = {}
for (const row of rows) {
  const got = await classify(row.text)
  for (const t of new Set([...row.topics, ...got.topics])) {
    stats[t] ??= { tp: 0, fp: 0, fn: 0 }
    const want = row.topics.includes(t)
    const have = got.topics.includes(t)
    if (want && have) stats[t].tp++
    else if (have) stats[t].fp++
    else stats[t].fn++
  }
}
for (const [t, s] of Object.entries(stats)) {
  const precision = s.tp / (s.tp + s.fp || 1)
  const recall = s.tp / (s.tp + s.fn || 1)
  console.log(t.padEnd(26), 'precision', precision.toFixed(2), 'recall', recall.toFixed(2))
}

Typical fixes, roughly from cheapest to most effort:

1. Rewrite the definition of a label with low precision so it excludes what was wrongly tagged.
2. Add examples for the confused pair.
3. Merge labels the team cannot tell apart reliably either; if two people disagree, the model will too.
4. Move from Amara Quick to Amara S1.2 only if the steps above stop helping.

Running at volume

• Throughput: each key allows 600 requests a minute. For a backlog of 50,000 reviews, run about eight requests in parallel and use the back-off from Retries with back-off.
• Cost: a review request is roughly 350 input tokens (mostly the cached system message) and 30 output tokens. See Count cost from usage to log the exact figure per request.
• Drift: re-run the evaluation monthly with fresh labelled reviews; new features create new topics.
• Batches: an asynchronous batch endpoint at a lower price is planned; see Batches.

Related pages

• Moderate user content
• Extract data to JSON
• Prompting: use examples