Computer use CLI for AI agents https://agent-browser.dev/desktop agent-desktop
Computer use for AI agents: see and operate desktop apps on macOS, Windows, and Linux through the OS accessibility tree.
agent-desktop gives an AI agent computer use without screenshots or pixel matching. It reads the accessibility tree of a desktop app and returns it as JSON, with a ref for every element the agent can act on.
The agent clicks or types on a ref, reads the UI again to check the result, and repeats. The loop lives in your agent; agent-desktop has no model inside.
It ships as a CLI and a C library.
Snapshots, clipboard reads, screenshots, command output, and debug artifacts can contain private desktop content. Keep them out of public logs and uploads.
Install
npm install -g agent-desktop agent-desktop status # platform, permissions, supported surfaces agent-desktop permissions --guide # macOS: walks you through the Accessibility grant
Bun, Yarn and pnpm can install the same package. It includes native binaries and platform helpers for x64 and ARM64. For direct execution without a JavaScript launcher, download the matching native archive from GitHub Releases and keep its helper beside the executable.
On macOS and Linux, Homebrew also works: brew install agent-desktop . There is no official Scoop package yet. The repository no longer maintains a custom Homebrew tap or Scoop bucket.
Release archives and building from source are covered in Installation .
Quick start
agent-desktop snapshot --app Finder -i # interactive elements, each with a ref agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot agent-desktop snapshot --app Finder -i # observe again to verify
Each command prints one JSON envelope. A trimmed snapshot:
{ "version": "2.6", "ok": true, "command": "snapshot", "data": { "app": "Finder", "snapshot_id": "s8f3k2p9", "ref_count": 59, "complete": true, "tree": { "role": "window", "name": "Downloads", "children": [ { "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"] } ] } } }
Refs carry their snapshot id. A bare @e8 fails with INVALID_ARGS .
If the UI changed, an action returns STALE_REF or AMBIGUOUS_TARGET . Take a new snapshot and use its refs.
Actions are headless by default: no pointer movement, no focus change. Pass --headed for physical input.
In PowerShell, quote every ref ( '@s8f3k2p9:e8' ), because a bare @token is the splatting operator.
The full walkthrough is in the quick start .
How it works
Your agent calls the CLI, or the C library in process. Both run the same core, which reaches the OS only through PlatformAdapter . Each binary is built for one OS and talks to that OS's accessibility API.
For AI agents
The binary carries guides that match its version:
agent-desktop skills get desktop # the loop, refs, verification and retry rules agent-desktop skills get platform # what differs on the OS you run on agent-desktop --help # flags for one command
Install the skill into an agent harness from skills.sh :
npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop
Or add this to your AGENTS.md :
Use `agent-desktop` to see and operate desktop apps. Before the first use, run `agent-desktop skills get desktop` and `agent-desktop skills get platform`, and follow them.
Retry a failed command only when error.disposition.retry is safe . See JSON output for the envelope, error codes, and exit codes.
MCP
agent-desktop mcp serves every command as an MCP tool over stdio. A tool is named after its command ( desktop_snapshot , desktop_click , desktop_session_start ), takes that command's arguments, and returns its JSON envelope:
{ "mcpServers": { "agent-desktop": { "command": "agent-desktop", "args": ["mcp"] } } }
Global flags given to mcp , such as --session , --headed or --trace , apply to every call.
Commands
agent-desktop has 60 command names, 56 operational commands. The other four ( key-down , key-up , mouse-down , mouse-up ) refuse to run until a daemon can release a held key or button. Commands documents each one.
Group
Commands
Observe
snapshot , find , get , is , screenshot , list-surfaces
Act on a ref
click , double-click , triple-click , right-click , type , set-value , clear , focus , select , toggle , check , uncheck , expand , collapse , scroll , scroll-to
Keyboard and mouse
press , hover , drag , mouse-move , mouse-click , mouse-wheel , key-down , key-up , mouse-down , mouse-up
Apps and windows
launch , close-app , list-apps , list-windows , list-displays , focus-window , resize-window , move-window , minimize , maximize , restore , open-system-surface
Notifications
list-notifications , dismiss-notification , dismiss-all-notifications , notification-action
Clipboard
clipboard-get , clipboard-set , clipboard-clear
Wait
wait
Sessions
session , trace , cursor-overlay
System
status , permissions , version , batch , skills
Platform support
Platform
Requirement
Accessibility API
macOS
macOS 13 or later, Intel or Apple Silicon
AX API
Windows
Windows 10 1809 or later, x64 or ARM64
UI Automation
Linux
Ubuntu 24.04 or later, x86_64 or aarch64, X11 or Wayland
AT-SPI
Each platform page lists what works headless, what needs --headed , and what refuses.
C library
Each release ships libagent_desktop_ffi , so a program can call agent-desktop in process. See the C library docs .
import ctypes lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib") lib.ad_adapter_create.restype = ctypes.c_void_p lib.ad_adapter_destroy.argtypes = [ctypes.c_void_p] assert lib.ad_init(7) == 0 # AD_RESULT_OK when the header's AD_ABI_VERSION_MAJOR matches the library adapter = lib.ad_adapter_create() lib.ad_adapter_destroy(adapter)
Development
Each platform adapter builds only on its own OS, so scope cargo to the host's packages. On macOS:
HOST_PKGS=(-p agent-desktop-core -p agent-desktop-macos -p agent-desktop -p agent-desktop-ffi) cargo test "${HOST_PKGS[@]}" cargo clippy "${HOST_PKGS[@]}" --all-targets -- -D warnings
On Windows, use agent-desktop-windows instead of agent-desktop-macos ; on Linux, agent-desktop-linux . CONTRIBUTING.md lists every gate.
Documentation
Introduction : what it is and how it works
Snapshots and refs
Errors and recovery
Skills
CONCEPTS.md : vocabulary
CONTRIBUTING.md , SECURITY.md , SUPPORT.md
License
Apache-2.0