How Computer-Use Agents See and Click: Claude, OpenAI and Gemini APIs Compared
A computer-use agent doesn't touch your screen: it receives screenshots, answers with a click or a keystroke, and waits for your code to carry it out and send back the next screenshot. This explainer follows that loop and compares how the Claude, OpenAI and Gemini APIs handle coordinates, actions, image limits and safety.
Checked against Anthropic's computer_toolset_20260801, OpenAI's computer use guide (GPT-6 Astra) and Google's Computer Use tool (Gemini 3.x models)
- Your application: runs actions and returns screenshots
- The model's reply: one action, or a short batch
As of (vendor documentation)All entries vendor-stated
| Attribute | Anthropic (Claude API) | OpenAI (Responses API) | Google (Gemini API) | Figure |
|---|---|---|---|---|
| How the model acts | One client toolset, computer_toolset_20260801, that gives Claude 17 member tools such as screenshot, left_click, type and zoom; your application runs every call1 |
Two routes: the model writes code that drives the interface (PyAutoGUI or Playwright), which OpenAI recommends for GPT-6 Astra, or it returns structured actions in a computer_call2 |
A Computer Use tool that returns a function_call UI action; Gemini 3.x models add an intent field explaining each step3 |
Vendor-stated |
| Environments | A desktop; the reference implementation uses a virtual X11 display (Xvfb) in a container1 | Browser or desktop, in an isolated environment you provide2 | Browser, mobile or desktop environments, run in a sandbox you provide3 | Vendor-stated |
| Coordinate space | Pixels of the screenshots you return, origin top left; no display size is declared, so you scale them back to the real screen1 | The screenshot's coordinate space; if you downscale a screenshot, map the model's coordinates back before acting2 | A normalized grid of 0–999 on each axis; your code converts with x / 1000 × screen width3 |
Vendor-stated |
| What you send back | One tool_result per tool_use block, with an image for screenshot and zoom1 |
A computer_call_output carrying a computer_screenshot image (detail: "original"), matched by call_id2 |
A fresh screenshot in a function_result after each action3 |
Vendor-stated |
| Image limits | Up to 2,576 px on the long edge and 4,784 visual tokens (about 3.75 megapixels) on models that support the toolset; an oversized image is rejected. Docs advise avoiding resolutions above 1920×10801 | detail: "original" preserves resolution; the page we read gives no pixel cap2 |
No display size is specified in the request; the sample code uses a 1440×900 viewport3 | Vendor-stated |
| Action vocabulary | Click variants (left, right, middle, double, triple), drag, mouse move and up/down, scroll, type, key, hold key, wait, cursor position, zoom, screenshot1 | click, double_click, drag, move, scroll, keypress, type, wait, screenshot; a single call can hold an ordered batch of actions2 | Browser set includes click variants, type, scroll, drag_and_drop, hotkey, key press, wait, navigate, go_back, go_forward and take_screenshot; mobile adds open_app and long_press3 | Vendor-stated |
| Safety hooks | Classifiers scan tool results such as screenshots for prompt injection, with an opt-out through support; docs advise human confirmation for consequential actions1 | Guidance on confirmation, consent and human handoff; examples keep PyAutoGUI's fail-safe on2 | A safety_decision of allowed, require_confirmation or blocked; opt-in screenshot scanning for hidden instructions on 3.x models3 |
Vendor-stated |
| Status in the docs | Client toolset on the Claude API and Google Cloud; not available in Claude Managed Agents1 | Computer tool "remains supported as an alternative" to code execution2 | Labeled a Preview capability that may contain errors and security vulnerabilities3 | Vendor-stated |
A side-by-side of the three documented computer use interfaces, read from each lab's own documentation on October 15, 2026. It describes interfaces, not results: no agent here is scored or ranked, and nothing is a measured figure. The docs change often, so version-specific names are given as the pages state them. Figure labels: how they work.
How it works
A loop, not a remote control
All three APIs use the same pattern. Your application sends the task and a screenshot; the model replies with an action; your code carries it out in an environment you control and sends back a new screenshot. Anthropic calls the repetition the "agent loop" and notes that Claude's requests are run by your application, not by Anthropic.1 OpenAI's guide says the same: "You provide the environment and execute the model's requests."2 Google's describes it as a "continuous loop between your application and the API", and advises running it in a sandboxed VM or container.3 This makes computer use a form of tool use in which the tool is a screen.
The loop ends when the model answers without requesting an action, or when your code hits a limit. Anthropic's sample loop caps iterations (10 by default in its example) to avoid runaway API costs.1
Seeing: a screenshot is the observation
The model's only view of the world is the images you return. That has costs. Anthropic says each screenshot takes roughly 1,000 to 1,800 input tokens, so long loops fill the context window quickly, and recommends pruning old screenshots in batches to keep prompt caching effective.1 Image size is also a hard limit: Anthropic's models that support the toolset accept up to 2,576 pixels on the long edge, earlier models about 1,568, and an oversized image is rejected rather than downscaled for you.1
Small text is the usual casualty of downscaling. Claude's toolset includes a zoom action that returns a region at full resolution, and Anthropic suggests asking about a specific region when text is hard to read.1 OpenAI's guidance is to pass detail: "original" so the image keeps its resolution.2
Pointing: three coordinate conventions
The model answers with coordinates, and the three APIs define them differently. Claude's are pixels in the screenshot you returned, with no display size declared.1 OpenAI's are in the screenshot's space too, and you map them back if you scaled the image.2 Gemini's are a normalized 0–999 grid on each axis, converted by your code with x / 1000 × screen width.3
That is why a scaling mistake shows up as a pattern. Anthropic's troubleshooting table lists "clicks consistently offset in one direction" as the symptom of applying coordinates to a display of a different size without scaling, and notes that on macOS Retina displays screenshots are captured at twice the logical resolution.1
Acting: a small vocabulary, sometimes in batches
The actions are what a person could do with a mouse and keyboard: click, drag, scroll, type, press keys, wait. Claude's toolset has 17 members;1 OpenAI lists click, double_click, drag, move, scroll, keypress, type, wait and screenshot;2 Gemini's browser set adds navigation actions such as navigate and go_back, and it has separate mobile and desktop sets.3
The model may send several actions at once. In Claude's API a "batch action" is a run of blocks you execute in order, stopping at the first failure and returning an error for every action not run.1 OpenAI's computer_call carries an ordered actions array.2 OpenAI also describes a second route, code as the action: the model writes a short script for a library such as PyAutoGUI or Playwright, and your application runs it and returns the output and screenshots. For GPT-6 Astra, OpenAI recommends this route and says the computer tool remains supported as an alternative.2
Checking: look again before moving on
A click isn't proof that the page changed. Anthropic notes that Claude sometimes assumes an action worked without checking, and suggests prompting it to take a screenshot after each step and state whether the step succeeded; it also says to place the instruction text before the screenshot in a turn to improve click accuracy.1 OpenAI's guide likewise advises returning another screenshot after a short group of actions.2 Claude typically ends a batch with a screenshot for the same reason.1
The risk of reading pixels
Because the model acts on whatever the screen shows, text on a web page or in an image can try to steer it, an attack called prompt injection. Anthropic says Claude will sometimes follow commands found in content even when they conflict with the user's instructions, and describes classifiers that scan tool results such as screenshots for injection attempts.1 Google's tool returns a safety_decision (allowed, require confirmation or blocked) with each suggested action, and offers opt-in screenshot scanning on Gemini 3.x models.3 All three labs advise isolating the agent from sensitive data and asking a person to confirm consequential steps, such as purchases or accepting terms.123 Google labels the capability a Preview that may contain errors and security vulnerabilities.3
How it is measured
The standard test is a benchmark where agents operate real software. OSWorld, introduced in a 2024 paper, built 369 computer tasks across real web and desktop apps, each with an execution-based check. In that paper humans completed over 72.36% of the tasks and the best model 12.24%, with failures concentrated in GUI grounding (mapping an intent to the right place on screen) and operational knowledge.4 Those are 2024 results from the paper, not today's scores, and any current ranking must name its harness and step budget.
Sources
- Anthropic, Computer use tool (Claude Platform docs), retrieved 2026-10-15.
- OpenAI, Computer use (API guide), retrieved 2026-10-15.
- Google, Computer Use (Gemini API docs), retrieved 2026-10-15.
- Tianbao Xie et al., OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, arXiv:2404.07972 (v2, May 30, 2024), retrieved 2026-10-15.