mirror of https://github.com/lahfir/agent-desktop.git synced 2026-07-26 09:02:17 +00:00

feat!: implement Playwright-grade foundation contract

Settle the Playwright-grade reliability contract in agent-desktop-core
before the Windows/Linux adapters are built, so they inherit it instead
of redesigning it. Every command now observes, waits, verifies, and
reports honestly instead of firing blindly.

Highlights: capability-supertrait split of PlatformAdapter with
not_supported() defaults; canonical role/state vocabulary with live
`is --property visible`; display enumeration (`list-displays`) and honest
`--screen` with scale factor; truthful Automation permission; `native_id`
identity spine; window-id-first resolution; serializable `LocatorQuery`
with live `find`; default-on auto-wait before every ref action; three-way
`hit_test` occlusion gate; `scroll_into_view` in core; core accessible-name
precedence; typed `ActionStep` delivery tier; `ProcessState` and
`APP_UNRESPONSIVE`; `LaunchOptions`; baseline-diff desktop signals
(`wait --event`); typed clipboard (`Text`/`Image`/`FileUrls`); mouse
modifier chords and `mouse-wheel`. Hardened through a 35-reviewer pass with
independent validation and a green live e2e gate (109/0), plus a
head-vs-main performance comparison harness.

BREAKING CHANGE: default-on auto-wait changes the timing of every
previously-untouched ref-action call (bounded 5000 ms default; `--timeout-ms 0`
restores single-shot). `ENVELOPE_VERSION` is now `2.1` (adds the
`APP_UNRESPONSIVE` code and process state in error details). FFI ABI major
is `3` (append-only struct evolution; `wait --event` is intentionally not
exposed over FFI). The legacy string clipboard API is removed in favor of
typed content. `key-down`/`key-up` fail closed until daemon-owned held input
exists. `close-app` verifies termination and the osascript fallback path is
removed. `--text` matching is subtree containment: `find --text X --first`
returns the outermost matching container.

2026-07-20 00:21:38 -07:00

4.2 KiB

Raw Permalink Blame History

agent-desktop FAQ

What is agent-desktop?

agent-desktop is a native desktop automation CLI for AI agents. It lets agents observe and control desktop apps through OS accessibility trees, using structured JSON instead of screenshots, pixel matching, or browser-only automation.

Does agent-desktop require screenshots or pixel matching?

No. The core workflow reads native accessibility trees and assigns refs to interactive elements. Screenshots are available as a separate command, but agents do not need screenshots or pixel matching to click buttons, type into fields, inspect menus, or navigate app windows.

How does agent-desktop work?

Component	Function
Native Rust CLI	Fast, single binary, no runtime dependencies
C-ABI cdylib	Load once from Python, Swift, Go, Ruby, Node, or C instead of forking
58 Commands	Observation, interaction, keyboard, mouse, notifications, clipboard, window management, session lifecycle, trace read/export, and bundled `skills` docs
Snapshot & Refs	Compact snapshot IDs and deterministic element refs like `@e1`, `@e2`
Structured JSON	Machine-readable responses with error codes and recovery hints

What makes agent-desktop useful for AI agents?

Feature	Benefit
Progressive Skeleton Traversal	78–96% token reduction on dense apps
Headless-by-Default Actions	Ref actions use accessibility APIs and block unintended physical side effects
Snapshot Refs	Agents act on stable refs within a snapshot instead of guessing coordinates
Recovery Hints	Errors include machine-readable codes and suggestions for the next agent step
Cross-Language FFI	Python, Swift, Go, Ruby, Node, C, and C++ hosts can call the native library directly

Which platforms are supported?

Feature	macOS	Windows	Linux
Accessibility tree	Yes	Planned	Planned
Click/type/keyboard	Yes	Planned	Planned
Mouse input	Yes	Planned	Planned
Screenshot	Yes	Planned	Planned
Clipboard	Yes	Planned	Planned
App/window management	Yes	Planned	Planned
Notifications	Yes	Planned	Planned

How do I install agent-desktop?

Install the CLI from npm:

npm install -g agent-desktop
agent-desktop snapshot --app Safari

Build the FFI library from source:

cargo build --profile release-ffi -p agent-desktop-ffi
# Outputs under target/release-ffi/: libagent_desktop_ffi.dylib/.so or agent_desktop_ffi.dll

What is the ref system?

snapshot assigns qualified refs to interactive elements in depth-first order, such as @s8f3k2p9:e1. The embedded snapshot ID removes the separate --snapshot flag. Snapshot lookup remains confined to the selected namespace, so refs created under a session still require that session's --session or AGENT_DESKTOP_SESSION scope. Legacy bare refs such as @e1 require an explicit --snapshot in the same namespace.

Interactive roles that receive refs:

button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell, radiobutton, incrementor, menubutton, switch, colorwell, dockitem.

Stale ref recovery:

snapshot -> act -> STALE_REF? -> snapshot again -> retry

Is agent-desktop free and open source?

Yes. agent-desktop is Apache-2.0 licensed for personal and commercial use.

Where can I get help?

Resource	Link
Repository	github.com/lahfir/agent-desktop
ClawHub Skill	clawhub.ai/lahfir/agent-desktop
skills.sh Listing	skills.sh/lahfir/agent-desktop/agent-desktop
npm Package	npmjs.com/package/agent-desktop
CI Status	GitHub Actions
Releases	GitHub Releases
Issues	GitHub Issues

4.2 KiB Raw Permalink Blame History Unescape Escape