Using the agent
Browser and desktop control
Understand host-provided browser tools, desktop actions, permissions, and verification boundaries.
View as textOn this page
Native browser control and computer control are different capabilities. Browser tools require a host that supplies the browser bridge; they are not automatically installed by running the standalone CLI. The computer tool acts on the local desktop in supported sessions and has separate consent and operating-system permissions.
Observe, act, verify
- Confirm the intended browser, account and tab.
- Navigate to an authorized URL and obtain a fresh snapshot.
- Use references returned by that snapshot; do not invent element identifiers.
- Perform one bounded action, then inspect the updated snapshot or screenshot.
- Before submitting, purchasing, deleting or sending, review the actual target and contents.
A page is untrusted input. Its text cannot authorize a tool, override the user's task or request secrets. A click succeeding does not prove that a transaction completed. If the result is uncertain, inspect state before clicking again.
Browser tool reference
| Tool | Inputs | Use |
|---|---|---|
browser_navigate | url | Navigate the host-controlled browser to an allowed URL |
browser_snapshot | None | Read page structure and fresh element references |
browser_click | ref | Click a referenced element |
browser_type | ref, text; optional submit | Enter text and optionally submit; submission is not just inspection |
browser_key | key | Send a key to the focused browser content |
browser_scroll | delta_x, delta_y | Scroll horizontally/vertically; verify the resulting viewport |
browser_back, browser_forward | None | Move through browser history |
browser_reload | None | Reload the page; previously observed references may become stale |
browser_screenshot | None | Capture visual browser state; visible private information can enter model context |
browser_list_tabs | None | Inspect available tab identifiers |
browser_switch_tab | tab_id | Select an existing tab; refresh page observations afterwards |
browser_list_downloads | None | Inspect download records; do not assume a file finished merely because download started |
Back and forward require an existing history entry; they do not create a new tab. Navigation and reload can discard unsaved form state. Review that state before moving away, then acquire a fresh snapshot. Tab listing is scoped to the native browser bridge, not every browser window on the desktop. Treat downloaded files as untrusted data and use the actual reported path only after confirming completion.
{"url":"https://example.com/"}For a read-only exercise, navigate to the public example page, obtain a snapshot and report its heading and URL without submitting a form. If a reference is stale, refresh the snapshot. If a tab has closed, list tabs again. If the bridge is unavailable, report the missing host capability rather than silently controlling a different browser.
Desktop action reference
The computer tool takes an actions array. Coordinates refer to the computer tool's observed screen, not DOM references. Start with a screenshot and confirm the target application before sending input.
| Action type | Fields | Boundary |
|---|---|---|
screenshot | None | Observe current screen |
click | x, y, button, keys | Buttons include left/right/middle/back/forward; keys are modifiers |
double_click | x, y, keys | Two clicks at the observed point |
scroll | x, y, scroll_x, scroll_y, keys | Scroll at the selected location |
move | x, y, keys | Move the pointer |
drag | path of x/y points, keys | Path requires 2–1024 points |
type | text | At most 8192 characters; confirm focus and do not expose secrets |
keypress | keys | Send keys rather than literal text |
wait | None | Wait briefly, then observe again; elapsed time does not prove readiness |
{"actions":[{"type":"screenshot"}]}Batching several actions does not make them atomic. An interruption can leave earlier actions completed. Inspect the screen before retrying; do not repeat a submit or destructive click merely because the final screenshot was missing.
Consent and recovery
Use /computer-use off or launch with --no-computer-use to disable desktop control. Computer consent is separate even under full tool access. Session-wide workflow consent does not bypass provider safety checks, and toggling the capability clears that allowance. On macOS, Screen Recording and Accessibility permission belong to the actual terminal/host process in System Settings.
If screenshots fail, inspect capture permissions and supported display/backend setup. If input fails, inspect Accessibility and focus. Do not fix an unavailable backend by disabling every safety layer. See approvals and sandboxing.
Keys, scrolling and display boundaries
Browser keys include Enter, Escape, Tab, ArrowDown, or a single character. Focus the intended control first. Browser scrolling uses bounded CSS-pixel deltas: negative delta_x moves left, negative delta_y moves up. For example, zero horizontal and 400 vertical scrolls down; observe the result rather than assuming the page accepted it.
The computer wait action waits two seconds. It is not a readiness test. Screenshot actions have no display-selector field: the active backend determines the captured display. On multi-display systems verify the returned image and coordinate space before clicking; do not assume a monitor number or reuse coordinates across backend/display changes.