Tool Suggestions
Vercel’s agent-desktop Gives Any AI Agent Computer Use Without Screenshots
Vercel Labs has open-sourced agent-desktop, a tool that gives any AI agent computer use on macOS, Windows and Linux — without screenshots. Announced on October 31 by Chris Tate of Vercel Labs as “agent-browser, for your whole computer”, it is a Rust command-line tool plus a C library, licensed under Apache-2.0, and it contains no model of its own. The project began in February 2026 with developer Lahfir, moved to Vercel Labs on October 30, reached version 1.0.0 on October 27 with Linux support, and stands at 1.0.2 as of October 30.
How it works
Most computer-use systems look at pixels: they screenshot the screen, guess where the button is, and click a coordinate. agent-desktop reads the app’s real interface structure through the operating system’s accessibility tree — the macOS AX API, Windows UI Automation, and Linux AT-SPI — and returns a JSON snapshot with a reference for every actionable element. The agent acts by reference, then takes another snapshot to verify the change, and errors state whether a retry is safe. Refs stay stable and actions stay safe to retry, instead of guessing from pixels.
The tool ships about 60 commands and supports MCP, the protocol that lets agents call tools. It runs in the background: structured tree actions do not move the pointer or take focus, and a --headed flag enables physical input when a task genuinely needs it. A C library, libagent_desktop_ffi, lets Python, Swift, Go, Ruby and Node call it directly, and bundled skill files teach an agent the observe-act-verify loop before first use.
Install and requirements
Install with npm install -g agent-desktop — Bun, Yarn and pnpm also work, and native binaries ship for x64 and ARM64. Homebrew and Scoop packages have been submitted but are not yet in the default catalogs. The platform requirements are macOS 13 or later with Accessibility permission granted to the running app, Windows 10 version 1809 or later, and Ubuntu 24.04 or later with AT-SPI running.
Limits, stated plainly
Two limits define what this tool is not. It cannot see applications that expose no accessibility tree — the quality of the automation depends on how well each app’s interface is built. And it does not automate web pages; Vercel’s agent-browser does that job, while agent-desktop covers the desktop.
Two cautions on the numbers around it. Token-reduction figures of 78% to 96% against full-tree dumps are the builder’s own measurements, not an independent benchmark, and no public comparison on computer-use tests exists yet. Star counts in coverage conflict — one outlet counted about 2,200, another said it had passed 5,000 — so treat popularity as “thousands of stars” and nothing more precise. A Windows crash bug was also reported against 1.0.1, so test on your own builds before putting it into any workflow you depend on.
One risk note from the coverage deserves emphasis: snapshots and clipboard reads can contain private desktop content. An agent that can read every control on your screen can also read whatever those controls display. Run it first with a test account, a known application and one reversible action, verify the target and the resulting state, and only then decide which desktop data the model may receive.