Vercel agent-desktop: Desktop Automation Through the Accessibility Tree, Not Pixel Coordinates

There are two approaches to desktop automation for AI agents. One captures screenshots, calculates pixel coordinates, and clicks. The other reads the structure of UI elements through OS accessibility APIs and operates on semantic units. Vercel Labs' agent-desktop chose the latter. With over 21,500 stars, this project extracts accessibility trees as JSON across macOS, Windows, and Linux, then operates by per-element reference IDs (refs).

The fact that no model is bundled matters. agent-desktop provides tools; the calling agent must construct the observe-act-verify loop. This article is based on repository and documentation review. CodeBridge did not reproduce snapshots on macOS.

Screenshots vs. accessibility trees

Existing computer-use agents capture a screen, use OCR or object detection to find buttons, and click at screen coordinates. This approach breaks when resolution changes and locations shift with dynamic UI.

agent-desktop works differently. It reads the accessibility tree of a target application through macOS AX API, Windows UI Automation, or Linux AT-SPI. Each element receives a role, a name, available actions, and a unique ref_id.

{
  "version": "2.6",
  "ok": true,
  "command": "snapshot",
  "data": {
    "app": "Finder",
    "snapshot_id": "s8f3k2p9",
    "ref_count": 59,
    "complete": true,
    "tree": {
      "role": "window",
      "name": "Downloads",
      "children": [
        { "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"] }
      ]
    }
  }
}

The ref @s8f3k2p9:e8 combines the snapshot ID (s8f3k2p9) and element index (e8). This ref is scoped to that snapshot only. When the UI changes, a STALE_REF error is returned. A new snapshot must be taken with new refs.

Commands: 57 operations, MCP server mode

agent-desktop provides 57 operational commands—snapshot, click, type, wait, and more—plus 4 incomplete commands (key-down, key-up, mouse-down, mouse-up, awaiting daemon release). Commands are grouped into observation, ref-based actions, keyboard and mouse, app and window management, notifications, clipboard, sessions, sandboxes, and system utilities.

A simple workflow:

agent-desktop snapshot --app Finder -i
agent-desktop click @s8f3k2p9:e8
agent-desktop snapshot --app Finder -i

Take a snapshot to read the current UI, click a ref, then snapshot again to verify. This loop lives in the calling agent, not in agent-desktop.

The MCP server mode is also notable. agent-desktop mcp serves every command as an MCP tool over stdio, so it integrates directly with MCP-compatible agent frameworks.

{ "mcpServers": { "agent-desktop": { "command": "agent-desktop", "args": ["mcp"] } } }

Headless by default, security policy, platform support

agent-desktop runs headless by default. Ref actions operate through the accessibility API without moving the pointer or taking focus. The global --headed flag is required for physical input.

The security policy is explicit: snapshots, clipboard reads, screenshots, command output, and debug artifacts can contain private desktop content. They must be kept out of public logs and uploads.

Supported platforms:

Each platform page lists what works headless, what needs --headed, and what refuses.

The loop the agent must construct

agent-desktop is not a finished agent. The calling agent must construct this loop:

1. Observe: Read the current UI state as JSON via snapshot. Review the ref_id list.
2. Act: Manipulate refs through click, type, set-value. Check error codes for retry eligibility.
3. Verify: Take another snapshot to confirm changes match intent.

Error handling matters. INVALID_ARGS means the ref format is wrong. STALE_REF means the UI changed and the previous ref is invalid. AMBIGLE_TARGET means multiple windows overlap. Retry only when error.disposition.retry is safe.

Practical use cases

agent-desktop is most useful when:

• Settings dialogs: Automating macOS System Settings or Windows Settings apps. Screen coordinates vary by resolution, but refs remain stable.
• Desktop app testing: Writing GUI test automation using element structure instead of pixel matching.
• Accessibility testing: Simulating screen-reader user paths through the accessibility tree.

Conversely, games, graphics-specialized apps, canvas-based UI, and some game engines may not provide accessibility tree information and will not work. Platform support varies, so check the platform page for your target app.

Limitations: the boundaries of accessibility APIs

First, not all apps expose accessibility information. Some require manual accessibility permission grants; others expose no tree at all.

Second, refs are snapshot-scoped. When the UI changes, refs become invalid, requiring a fresh snapshot each time. This is faster than network calls but consumes agent context when frequent.

Third, security artifact management. Snapshots and clipboard contents may contain personal data. Accidental inclusion in public logs creates problems.

Insight: automating by meaning, not coordinates

The core of agent-desktop is "semantic UI elements instead of pixel coordinates." Screenshots, OCR, and object detection are vulnerable to resolution changes and dynamic UI, while accessibility trees provide structural stability. Not all apps expose accessibility information, and refs are snapshot-scoped.

When evaluating adoption, the question is not "is agent-desktop better than screenshot-based methods?" but "does the target app provide accessibility APIs, can ref scoping be managed, and can security artifacts be handled?" After answering these, testing snapshot → click → snapshot on a simple app (Finder, etc.) is the first step.

For more on desktop automation loops, see the Agent Flow course for observe-act-verify patterns.