agent-desktop is an open-source Rust CLI that lets AI agents control any macOS app via OS accessibility trees — no screenshots or pixel matching needed.
Developer lahfir released agent-desktop, a Rust-built CLI tool that exposes native OS accessibility trees to AI agents for structured desktop automation. It ships with a C-ABI cdylib (libagent_desktop_ffi) loadable from Python, Node, Go, Swift, Ruby, and C — eliminating the need to fork a subprocess per command. The tool works with any macOS app that has an accessibility tree (Finder, Slack, Xcode, Safari, etc.) and falls back to mouse events only when accessibility APIs are exhausted. It's available via npm install or cargo build and requires macOS Accessibility permissions.
This replaces the fragile screenshot-then-click loop that most desktop agents rely on. Instead of Vision API calls to interpret pixels, you get a structured accessibility tree with stable element refs (@e1, @e2...) that map directly to interactive UI elements. The C-ABI cdylib means you load it once in-process from Python or Node — no subprocess spawning per action, which kills the latency problem that makes desktop agents feel sluggish.
Wire agent-desktop into a Python agent loop this week: load libagent_desktop_ffi via ctypes, call snapshot on your most-automated app (e.g. Slack or Finder), and replace your current screenshot+OCR step — measure whether element targeting accuracy improves and round-trip latency drops below 100ms.
Run: npm install -g agent-desktop && agent-desktop permissions --request
Tags