Cua (Computer-Use 2.0): How Agents Actually 'Use' Computers

Agents that look at screens and click buttons is already well known. But that approach has fundamental limitations.

Screen reading requires agents to look at pixels, identify UI elements, and move the mouse. It's slow, fragile, and breaks accessibility.

Cua solves this with Computer-Use 2.0 — infrastructure that lets agents communicate directly with native apps instead of looking at pixels.

Computer-Use 1.0 vs 2.0

Computer-Use 1.0 (Screen-based):

  ┌─────────────┐
  │   Desktop    │ ← Agent reads pixels
  └─────────────┘
       ↑↓ Screenshot
  ┌─────────────┐
  │ Agent Model  │ ← "Click this button"
  └─────────────┘
       ↑↓ Mouse movement
  ┌─────────────┐
  │   Desktop    │ ← Click executes
  └─────────────┘

Problems: Slow, fragile, breaks accessibility

Computer-Use 2.0 (Native-based):

  ┌─────────────┐
  │   Agent      │ ← API calls
  └─────────────┘
       ↑↓ Cua Driver (API)
  ┌─────────────┐
  │ Native Apps  │ ← Direct manipulation
  └─────────────┘

Advantages: Fast, stable, accessibility-friendly

Five Components of Cua

Cua is not a single product — it's a combination of multiple components.

1. Cua Spaces — Desktops for Agents

Provides dedicated desktop environments where agents can work.

Local: macOS VM (Virtualization.Framework)
Remote: Linux / Omarchy (servers managed by Cua)

Agents work in isolated spaces without interfering with the user's actual desktop. The Background Delivery Mode lets agents complete tasks in the background without moving the user's cursor.

2. Cua Driver — Native App Automation

Inspects and manipulates native apps on macOS, Windows, and Linux.

What Cua Driver provides:
├── UI element tree (Accessibility API)
├── Click, type, scroll
├── Menu manipulation
└── State reading (active app, window title, etc.)

Instead of taking screenshots and doing OCR, it reads UI element names, roles, and states directly through the Accessibility API. This is much faster and more accurate.

3. Lume — Local Virtual Machines

Creates macOS and Linux VMs on Apple Silicon Macs using Virtualization.Framework.

# Create a macOS VM with Lume (simplified)
lume create --os macos --name agent-workspace

# Create a Linux VM
lume create --os ubuntu --name dev-env

VMs provide isolated environments where agents can test safely. Heavier than Docker containers, but essential for GUI app testing.

4. CUA-S1 — Computer-Use-Specific Models

General LLMs are not specialized for computer manipulation. CUA-S1 is a small dedicated model for computer-use decisions.

Input: Current screen + task goal
Output: Next action (click, type, scroll, etc.)

Example:
  Screen: Google search results in Safari
  Goal: "Find React vs Vue comparison doc"
  Action: [Click input field] → ["React vs Vue comparison" typed] → [Enter]

This model is much faster than general LLMs and more accurate for computer-use tasks because it's specialized.

5. Cua Bench — Agent Evaluation Tool

Evaluates agents' computer-use capabilities and exports execution trajectories.

Cua Bench evaluation metrics:
├── Task completion rate
├── Step efficiency
├── Error recovery ability
└── Time to complete

MCP Integration — Connect with Various Agents

Cua connects with various agents through MCP (Multi-Channel Protocol).

Claude Code ──┐
Codex    ────┼──▶ MCP Server ──▶ Cua Driver
Cursor ──────┘    (Cua)        ├── macOS app manipulation
OpenClaw ──────┘               ├── Windows app manipulation
                               └── Linux app manipulation

No need to write separate integration code for each agent. Just follow the MCP standard.

Pricing and Licensing

The Cua Spaces app uses the FSL (Functional Source License). The source is public but commercial redistribution is restricted. Note that the code (MIT) and app (FSL) have different licenses.

Limitations and Considerations

1. macOS-first: Lume depends on Apple Silicon's Virtualization.Framework. Support for Intel Macs and Windows is limited.
2. VM overhead: VMs created by Lume are heavier than Docker containers and use more memory.
3. FSL app license: Cua Spaces follows FSL, unlike the MIT-licensed code.
4. Agent compatibility: While 20+ agents are supported, not all combinations have been fully tested.

Developer Takeaway

There are two ways agents "use" computers.

Screen-reading agents look human. But they're slow, fragile, and break accessibility.

Native API agents (Cua) are fast, stable, and leverage accessibility APIs.

When deploying agents to production, avoid screen-based approaches. Cua solves this at the infrastructure level.

Don't let agents move your mouse — let them manipulate apps directly.

• GitHub
• Website
• Documentation