vercel-labs/agent-desktop

▲ 4,451 stars today★ 20,973⑂ 1,555

Computer use CLI for AI agents

About vercel-labs/agent-desktop

vercel-labs/agent-desktop is an open-source project on GitHub, mainly written in Rust. Computer use CLI for AI agents It currently holds 20,973 stars and 1,555 forks with 208 open issues, and was last pushed on 2026-11-10 (repository created 2026-02-19).

Project Overview

Git Homed tracks it on the Today's Trending board.

GitHub Repository Details

Repository vercel-labs/agent-desktop · default branch main · size 71465 KB · watchers 46 · source: GitHub REST API and repository README

README

agent-desktop

Computer use for AI agents: see and operate desktop apps on macOS, Windows, and Linux through the OS accessibility tree.

https://github.com/vercel-labs/agent-desktop/blob/HEAD/Vercel Labs Product https://github.com/vercel-labs/agent-desktop/blob/HEAD/npm version: agent-desktop https://github.com/vercel-labs/agent-desktop/blob/HEAD/License: Apache-2.0 https://github.com/vercel-labs/agent-desktop/blob/HEAD/npm downloads per month: agent-desktop

agent-desktop gives an AI agent computer use without screenshots or pixel matching. It reads the accessibility tree of a desktop app and returns it as JSON, with a ref for every element the agent can act on. The agent clicks or types on a ref, reads the UI again to check the result, and repeats. The loop lives in your agent; agent-desktop has no model inside. It ships as a CLI and a C library.

Snapshots, clipboard reads, screenshots, command output, and debug artifacts can contain private desktop content. Keep them out of public logs and uploads.

Install

npm install -g agent-desktop
agent-desktop status                # platform, permissions, supported surfaces
agent-desktop permissions --guide   # macOS: walks you through the Accessibility grant

Bun, Yarn and pnpm can install the same package. It includes native binaries and platform helpers for x64 and ARM64. For direct execution without a JavaScript launcher, download the matching native archive from GitHub Releases and keep its helper beside the executable.

On macOS and Linux, Homebrew also works: brew install agent-desktop. There is no official Scoop package yet. The repository no longer maintains a custom Homebrew tap or Scoop bucket.

Release archives and building from source are covered in Installation.

Quick start

agent-desktop snapshot --app Finder -i      # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8            # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i      # observe again to verify

Each command prints one JSON envelope. A trimmed snapshot:

{
  "version": "2.6",
  "ok": true,
  "command": "snapshot",
  "data": {
    "app": "Finder",
    "snapshot_id": "s8f3k2p9",
    "ref_count": 59,
    "complete": true,
    "tree": {
      "role": "window",
      "name": "Downloads",
      "children": [
        { "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"] }
      ]
    }
  }
}
returns STALE_REF or AMBIGUOUS_TARGET: take a new snapshot and use its refs. with --window-id. snapshot --root @s8f3k2p9:e3 -i --compact. --headed flag allows physical input. including refs captured from a sheet, alert, popover, focused view, or Linux menu. macOS application menus keep their application-owned surface and need no document window. The wait reads the whole pinned window, so overlay selectors share its scope. Wait scope and results describe the selector limits. The full walkthrough is in the quick start.

How it works

https://github.com/vercel-labs/agent-desktop/blob/HEAD/The calling agent runs the agent-desktop CLI or the C library. Both go through agent-desktop-core and the PlatformAdapter trait to one platform crate, chosen at compile time, which reads the target app through the AX API on macOS, UI Automation on Windows, or AT-SPI on Linux.

Your agent calls the CLI, or the C library in process. Both run the same core, which reaches the OS only through PlatformAdapter. Each binary is built for one OS and talks to that OS's accessibility API.

For AI agents

The binary carries guides that match its version:

agent-desktop skills get desktop      # the loop, refs, verification and retry rules
agent-desktop skills get platform     # what differs on the OS you run on
agent-desktop  --help        # flags for one command

Install the skill into an agent harness from skills.sh:

npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop

Or add this to your AGENTS.md:

Use agent-desktop to see and operate desktop apps. Before the first use, run
agent-desktop skills get desktop and agent-desktop skills get platform, and follow them.

Retry a failed command only when error.disposition.retry is safe. See JSON output for the envelope, error codes, and exit codes.

MCP

agent-desktop mcp serves every command as an MCP tool over stdio. A tool is named after its command (desktop_snapshot, desktop_click, desktop_session_start), takes that command's arguments, and returns its JSON envelope:

{ "mcpServers": { "agent-desktop": { "command": "agent-desktop", "args": ["mcp"] } } }

Global flags given to mcp, such as --session, --headed or --trace, apply to every call except sandbox exec, whose guest command takes its own flags.

Commands

agent-desktop has 61 command names, 57 operational commands. The other four (key-down, key-up, mouse-down, mouse-up) refuse to run until a daemon can release a held key or button. Commands documents each one.

| Group | Commands | |---|---| | Observe | snapshot, find, get, is, screenshot, list-surfaces | | Act on a ref | click, double-click, triple-click, right-click, type, set-value, clear, focus, select, toggle, check, uncheck, expand, collapse, scroll, scroll-to | | Keyboard and mouse | press, hover, drag, mouse-move, mouse-click, mouse-wheel, key-down, key-up, mouse-down, mouse-up | | Apps and windows | launch, close-app, list-apps, list-windows, list-displays, focus-window, resize-window, move-window, minimize, maximize, restore, open-system-surface | | Notifications | list-notifications, dismiss-notification, dismiss-all-notifications, notification-action | | Clipboard | clipboard-get, clipboard-set, clipboard-clear | | Wait | wait | | Sessions | session, trace, cursor-overlay | | Sandboxes | sandbox | | System | status, permissions, version, batch, skills |

Platform support

| Platform | Requirement | Accessibility API | |---|---|---| | macOS | macOS 13 or later, Intel or Apple Silicon | AX API | | Windows | Windows 10 1809 or later, x64 or ARM64 | UI Automation | | Linux | Ubuntu 24.04 or later, x86_64 or aarch64, X11 or Wayland | AT-SPI |

Each platform page lists what works headless, what needs --headed, and what refuses.

C library

Each release ships libagent_desktop_ffi, so a program can call agent-desktop in process. See the C library docs.

import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_adapter_create.restype = ctypes.c_void_p
lib.ad_adapter_destroy.argtypes = [ctypes.c_void_p]
assert lib.ad_init(7) == 0  # AD_RESULT_OK when the header's AD_ABI_VERSION_MAJOR matches the library
adapter = lib.ad_adapter_create()
lib.ad_adapter_destroy(adapter)

Development

Each platform adapter builds only on its own OS, so scope cargo to the host's packages. On macOS:

HOST_PKGS=(-p agent-desktop-core -p agent-desktop-macos -p agent-desktop -p agent-desktop-ffi)
cargo test "${HOST_PKGS[@]}"
cargo clippy "${HOST_PKGS[@]}" --all-targets -- -D warnings

On Windows, use agent-desktop-windows instead of agent-desktop-macos; on Linux, agent-desktop-linux. CONTRIBUTING.md lists every gate.

Documentation

License

Apache-2.0

GitHub Stars & Activity

20,973Stars
1,555Forks
208Open issues
RustLanguage

GitHub Popularity

GitHub stars20,973
Forks1,555
Open issues208
Primary languageRust
LicenseApache-2.0
Stars gained today4,451
Created2026-02-19
Last pushed2026-11-10

Trending History

Weekly boardrank #10 · ▲ 4,451 stars
Monthly boardrank #8 · ▲ 22,859 stars

Related GitHub Projects

1

farion1231 / cc-switch

Rust★ 155,112⑂ 10,154▲ 312 stars
→
2

pbakaus / impeccable

Rust★ 111,729⑂ 6,673▲ 693 stars
→
3

rustfs / rustfs

Rust★ 36,197⑂ 1,679▲ 55 stars
→
4

t8y2 / dbx

Rust★ 30,593⑂ 2,785▲ 101 stars
→
5

PrimeIntellect-ai / prime-agent

Rust★ 22,795⑂ 2,530▲ 67 stars
→
6

anomalyco / hex

Rust★ 7,199⑂ 469▲ 198 stars
→
7

NVIDIA-NeMo / Switchyard

Rust★ 3,616⑂ 374▲ 41 stars
→
8

BurntSushi / jiff

Rust★ 3,116⑂ 141▲ 61 stars
→

More Trending Repositories