Computer and browser use

⚠️

Preview — this feature is being built; the design below is the spec.

Computer use lets the model operate a desktop or a web browser the way a person does: it looks at a screenshot, decides where to click or what to type, and asks your code to do it. Syntic publishes two standard tool definitions, syntic_computer and syntic_browser, with fixed action lists the model is trained to use well. Your code runs the actions in an environment you control and sends back a fresh screenshot.

⚠️

The model acts on whatever is on screen, including text written by others. Run it in an isolated virtual machine or browser profile with no access to accounts or data it does not need, and require human approval for purchases, messages and deletions.

When to use it

Automating tasks in apps that have no API.
Testing web applications end to end.
Filling forms across several sites.

Prefer an API or ordinary function tools whenever one exists: they are faster, cheaper and more reliable.

Endpoint and auth

POST https://app.syntic.ai/v1/chat/completions (and Messages), Authorization: Bearer sk_live_.... Requires a model with capabilities.vision and computer_use.

Tool definitions

These are client tools: the model returns calls, your code executes them, as with any function.

Field	Type	Details
`type`	string	`"syntic_computer"` or `"syntic_browser"`
`display_width`, `display_height`	integer	Screen size in pixels, as the model will see screenshots. Recommended 1280 × 800.
`display_number`	integer	Optional, for multi-display setups (`syntic_computer` only).

You do not write a schema: Syntic supplies it and keeps it stable across versions.

`syntic_computer` actions

`action`	Arguments	Your code should
`screenshot`	none	Return a PNG of the screen.
`click`	`x`, `y`, `button` (`left`, `right`, `middle`), `count` (1-3)	Move and click.
`move`	`x`, `y`	Move the pointer.
`drag`	`from_x`, `from_y`, `to_x`, `to_y`	Press, move, release.
`scroll`	`x`, `y`, `direction` (`up`, `down`, `left`, `right`), `amount`	Scroll at a point.
`type`	`text`	Type the text.
`key`	`keys`, for example `"ctrl+s"` or `"Return"`	Press keys or a combination.
`wait`	`seconds` (up to 10)	Pause, then take a screenshot.
`zoom`	`x`, `y`, `width`, `height`	Return a full-resolution crop of that region.

`syntic_browser` actions

All of the above, plus:

`action`	Arguments	Your code should
`navigate`	`url`	Open the address in the current tab.
`back`, `forward`, `reload`	none	Browser navigation.
`new_tab`, `switch_tab`, `close_tab`	`tab_id` where needed	Manage tabs.
`read_page`	none	Return the page’s visible text and form fields (cheaper than a screenshot for text-heavy pages).

The loop

Send the task with the tool definition.
The reply contains a tool call such as {"action":"click","x":412,"y":288,"button":"left","count":1}.
Run it, take a screenshot, and send a tool message whose content is an image part (and optional text).
Repeat until finish_reason is stop.

{
  "role": "tool",
  "tool_call_id": "call_9c",
  "content": [
    { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0..." } },
    { "type": "text", "text": "Clicked. Page title: Checkout" }
  ]
}

Examples

curl https://app.syntic.ai/v1/chat/completions \
  -H "Authorization: Bearer $SYNTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Amara S1.2",
    "stream": false,
    "tools": [{ "type": "syntic_browser", "display_width": 1280, "display_height": 800 }],
    "messages": [{ "role": "user", "content": "Open https://intranet.example.com/leave and submit a leave request for 24 to 26 November. Stop before the final Submit and ask me." }]
  }'

// `page` is a Playwright page you created in an isolated browser context.
async function perform(page, a) {
  switch (a.action) {
    case 'navigate': await page.goto(a.url); break
    case 'click': await page.mouse.click(a.x, a.y, { button: a.button, clickCount: a.count }); break
    case 'type': await page.keyboard.type(a.text); break
    case 'key': await page.keyboard.press(a.keys.replace(/ctrl/i, 'Control')); break
    case 'scroll': await page.mouse.wheel(0, a.direction === 'down' ? a.amount * 100 : -a.amount * 100); break
  }
  const shot = (await page.screenshot()).toString('base64')
  return [{ type: 'image_url', image_url: { url: `data:image/png;base64,${shot}` } }]
}
 
for (let step = 0; step < 40; step++) {
  const data = await callSyntic({ model: 'Amara S1.2', stream: false, tools, messages })
  const msg = data.choices[0].message
  messages.push(msg)
  if (data.choices[0].finish_reason !== 'tool_calls') break
  for (const call of msg.tool_calls) {
    const content = await perform(page, JSON.parse(call.function.arguments))
    messages.push({ role: 'tool', tool_call_id: call.id, content })
  }
}

import base64, json
 
def perform(page, a):
    if a["action"] == "navigate":
        page.goto(a["url"])
    elif a["action"] == "click":
        page.mouse.click(a["x"], a["y"], button=a.get("button", "left"), click_count=a.get("count", 1))
    elif a["action"] == "type":
        page.keyboard.type(a["text"])
    shot = base64.b64encode(page.screenshot()).decode()
    return [{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{shot}"}}]
 
for step in range(40):
    data = call_syntic({"model": "Amara S1.2", "stream": False, "tools": tools, "messages": messages})
    choice = data["choices"][0]
    messages.append(choice["message"])
    if choice["finish_reason"] != "tool_calls":
        break
    for call in choice["message"]["tool_calls"]:
        content = perform(page, json.loads(call["function"]["arguments"]))
        messages.append({"role": "tool", "tool_call_id": call["id"], "content": content})

Errors

Unified envelope.

Status	`code`	Cause
400	`computer_use_not_supported`	The model lacks vision or computer use.
400	`invalid_display_size`	Width or height outside 320 to 2,560.
400	`tool_result_image_required`	A computer tool result has no image (except for `read_page`).
400	`too_many_images`	Over 100 screenshots in the conversation. Drop old ones or turn on context management.

Cost and tips

Each screenshot costs image input tokens (about 1,100 at 1280 × 800). Long runs add up; clear old screenshots with context_management.clear_tool_results.
Smaller displays mean cheaper and often more accurate clicks.
Tell the model when to stop and ask, as in the curl example.

Build notes for the platform team

Test model accuracy on screenshots before committing; this page ships only if click accuracy meets the agreed bar.
Syntic owns the action schemas; version them with the API version header.
A hosted browser option can come later as a server tool.

Code execution Text editor, shell and memory

Was this page helpful?