mirror of
https://github.com/lahfir/agent-desktop.git
synced 2026-08-18 13:05:46 +00:00
No description
d4ddb5531d
* docs: add sub-phase 2.9 system lifecycle plan Plan the Windows System Lifecycle adapter fill — launch_app, close_app, window_op, process_state, is_protected_process, press_key_for_app, and the full window-activation focus policy over 2.6's minimal focus_window — grounded in a five-dossier research pass and hardened by an adversarial multi-lens review (verified-termination loop, WaitForSingleObject-gated exit reads, fail-closed cross-integrity activation, exact-match protected list, and press_key_for_app that verifies rather than re-activates). Correct the docs/phases.md contradictions and register the deferrals the planning research surfaced: drop the phantom WindowOp::Close variant the core enum never had; extend the §2.12 split-integrity item to own the cross-integrity window-activation/focus effect; assign the wait --menu parity hole and its menu-detection primitive to §2.11; and record the two press --app divergences (no semantic accelerator path, no headless pid-targeted delivery) in §2.15's settlement list. * docs: close the 2.9 plan's open questions with named owners The three residuals left after review were all deferrals without a receiving owner, which the repo's own rule forbids. Settle the U1 leg-7 shell-binding branch at planning time instead of "reconciling in U10": whichever way it measures, 2.9 keeps CreateProcessW and Win32_UI_Shell stays out of the manifest, because the manifest surface is a supply-chain decision rather than a probe outcome and KTD4 is load-bearing for U3's verification design. Reconciling after the fact would have surfaced the KTD10 stop-condition contradiction only once the code had shipped. Launch-by-display-name/AUMID now belongs to 2.14, the sub-phase that already owns Windows shell-integration commands and would already carry the shell dependency, written into its scope and its dependency line. Define envelope parity for failures macOS cannot produce: shared failures assert equal to the macOS pair, platform-only failures assert against the envelope contract and are recorded in 2.15's divergence list, so a Windows-only pair can never be asserted as if it were parity. Turn the FFI press-surface default from a floating concern into a U7 verification step against crates/ffi/src/actions/execute.rs, so a changed FFI path corrects the stated corner rather than relaxing a test. * test: measure windows system lifecycle gaps (A21) Settle CreateProcessW/attach, exit-code, hang, window-op tolerance, activation budget, and shell-binding facts before the adapter fills the lifecycle surface. * feat: add windows process_state and protected-process gate Give close-app a wait-gated liveness classifier and a Windows-reasoned exact-name blocklist before verified termination lands. * feat: launch windows apps via CreateProcessW Honor LaunchOptions with system-dirs-only bare-name resolution, ToolHelp attach policy, and verified first-window wait. * feat: close windows apps with verified termination Report close success only after the process handle and creation-time token show the target is gone. * feat: verify windows window ops via placement re-read Execute resize/move/minimize/maximize/restore against Win32 placement truth with an 8px tolerance pinned by A21-5. * feat: harden windows window-activation focus policy Add restore-versus-raise ordering, a finite focus-steal budget, and fail-closed cross-integrity activation mapping. * feat: compose windows press-key-for-app delivery Verify foreground ownership and integrity before synthesizing through the existing keyboard primitive. * test: stabilize press-key focus-lost invert under parallel tests Arm a deterministic focus-lost gate after the verify hook so the refusal does not depend on SetForegroundWindow races. * test: pin lifecycle envelope parity and cost baseline Lock shared macOS code/disposition pairs, class-b Windows-only failures, and A20-style hot-path cost captures. * docs: dogfood windows system lifecycle layer Judge launch/close/window-op/press-app and protected-process refusal against real Notepad and Explorer targets. * docs: sync windows lifecycle docs with shipped adapter Record CreateProcessW system-dirs launch, verified close, and press-app divergences against A21 and dogfood evidence. * refactor: split window-activate test hooks under file-size cap Keep window_activate.rs under the 400-line limit by moving invert-test hooks to a sibling module. * style: rustfmt elevation modules Apply pending rustfmt import ordering left after the lifecycle fill. * fix: bind terminate and wm-close to live ownership Verify creation-time token on the open process handle before TerminateProcess, and re-check HWND owner before each WM_CLOSE. * fix: unblock the windows lib-test gate after the ownership re-bind The ownership re-bind landed after the last local clippy run, so two gate failures reached CI. Drop a redundant isize cast in the new close test: LocalFixture::handle already returns isize, so `as isize` trips -D clippy::unnecessary_cast and fails the lib-test build. Take the on-screen stage lock in the focused-filter test. It reads the foreground twice - once through the filter, once to corroborate - and this sub-phase is the first to add tests that deliberately raise their own windows, so a sibling activation test landing between those two reads made it fail for the sibling's reason. It stages nothing itself, but the lock guards screen state rather than only screen real estate. Verified by running the suite in isolation (passes) and in parallel (failed before, three consecutive clean runs after). * fix: survive the mid-listing identity race in the live census test This sub-phase's suite spawns and terminates real processes, which is exactly the condition list_windows_live refuses on: a window whose owning process changes mid-walk fails the whole inventory. The census test expect()ed the listing instead, so a concurrent lifecycle test could panic it - reproduced on CI, not on the quieter dev box. Retry the listing rather than tolerate the first refusal, so the identity assertions still run wherever the race is not permanent, and accept only a refusal that survives every attempt and only as the exact WindowNotFound the inventory exists to report. Its sibling focused-filter test already handles the same refusal this way. Invert-verified: breaking the token assertion turns the test red, so the retry did not make it vacuous. * fix: deliver close to every owned window and stop tests passing vacuously Two defects the review pass found. graceful_close aborted the WM_CLOSE fan-out on the first window that refused the post, so every window enumerated after it never received the request. R2 requires the request to reach every window the pid owns precisely because the window that owns an app's shutdown may be enumerated after one that already tore itself down - which is the expected shape once an app starts closing in response to an earlier post in that same loop. The fan-out now continues and reports a failure only when no window accepted, leaving wait_for_exit's independent exit observation as the source of truth. Extracted broadcast_close as a seam over the poster so the behaviour is testable without Win32, and post_wm_close_if_still_owned now reports whether it delivered so a deliberate skip is distinguishable from a send. Four live press_key_for_app tests gated their whole body behind a foreground-staging bool and returned when staging did not land, so on any run where activation lost the race they passed while asserting nothing - including the only test proving the method is wired through SystemOps. Each now asserts the fail-closed contract instead of returning: a target that never reached the foreground must be refused with no synthesis, which is real coverage of the documented no-pid-targeting divergence. The trait-wiring test always calls through the trait object and cross-checks the outcome against whether staging succeeded. Invert-verified both: restoring abort-on-first-failure turns the fan-out test red, and the close fan-out tests fail if a skip is counted as a send. * fix: refuse window writes to a target that stops dispatching messages window_op and focus_window issued ShowWindow/SetWindowPos/ SetForegroundWindow with no liveness check. Those calls reach the owning thread's message queue, so a window whose thread never dispatches blocks the caller inside the OS call - the same shape A14-11 recorded for ElementFromHandle, and measured here as a call that never returned. A Deadline cannot rescue it: the budget is checked before the call and the block happens inside it, and the activation retry budget is consulted between attempts rather than during one. Both paths now ask the crate's existing SMTO_ABORTIFHUNG ping first and report APP_UNRESPONSIVE, which is what R9 already required of any path that could hang. The ping runs after identity verification, not before it, so a destroyed or re-owned handle still reports the stale-identity envelope it earned rather than being mislabelled unresponsive. Bound the live title read while here. GetWindowTextW sends WM_GETTEXT for a window owned by the calling process, so a same-process non-pumping window blocked identity verification itself before any write was reached - the documented cross-process behaviour returns the caption without sending, so this only bites in-process, but the title is best-effort evidence either way and identity still rests on the owner and generation checks, which never touch the message queue. Both guards are proven against the non-pumping StalledFixture, which hangs indefinitely without them. * fix: let the wait signal alone decide that a process is gone close.rs's exit gate vetoed on an exit code of 259, which is STILL_ACTIVE - the same value GetExitCodeProcess reports for a live process. A process that legitimately exits with that code therefore read as still running for as long as any handle kept it from being reaped, which the caller's own child handle routinely does, so close_app spun to TIMEOUT and delivered_unverified for a process that had already terminated cleanly. KTD3 says the wait decides rather than the code, and process_state gates that way already; close.rs had re-derived the check and got it wrong. Also stop broadcast_close discarding a request it already delivered. Its per-iteration budget check returned not_delivered without consulting the delivered flag, so a deadline expiring after a real WM_CLOSE had been posted told the caller nothing had happened and a retry was free. It now stops the fan-out and lets wait_for_exit report the honest delivered_unverified timeout instead. Invert-verified: restoring the 259 veto turns the new regression test red. * refactor: share the window liveness probe instead of duplicating it The hang guard landed the same probe call and the same timeout constant in both window_op.rs and window_activate.rs. It is one question about a window - is its thread dispatching messages - so it is one predicate, placed beside the handle type it takes in window_enum.rs. Only the probe is shared: activation and window operations report the same condition with different envelopes, so each caller still words its own refusal. * fix: treat restore as undoing the minimize, not forcing a normal placement show_verb demanded an exact SW_SHOWNORMAL placement to confirm a restore, but a window minimized while maximized carries WPF_RESTORETOMAXIMIZED, so SW_RESTORE correctly returns it to maximized. The ordinary maximize -> minimize -> restore sequence therefore reported ACTION_FAILED with delivered_unverified for an operation Windows had performed exactly as asked, telling the caller a retry was unsafe while the window sat correctly restored. Restore now succeeds when the window is no longer minimized, which is what the verb promises; it never promised to un-maximize. The verb and its predicate moved to their own module to stay under the file-size cap. Invert-verified: demanding the exact normal placement again turns the new maximize-minimize-restore test red. * fix: stop a trailing backslash swallowing the next launch argument Backslashes are literal except immediately before a quote, so an argument ending in one escaped the closing quote quote_arg appends: the quoted region never closed and every later argument was absorbed into it. A Windows path ending in a separator is an ordinary value, and the merge was silent - the launch still reported success. The trailing run is now doubled. An embedded quote stays doubled rather than backslash-escaped. Doubling is a valid encoding for the consecutive-quote rule and is the form cmd accepts; backslash-escaping is correct only for CommandLineToArgvW consumers and breaks a cmd redirection that works today, so the narrower fix closes the corruption without trading it for a different one. Also collapse the second copy of the window-title reader. window_ops kept its own unbounded GetWindowTextW while the identity module's copy was already bounded, so list_windows could still block on a window whose thread had stopped dispatching. One reader, already guarded. Invert-verified: removing the trailing-run doubling turns the new test red. * fix: correct the activation evidence and close the review's remaining gaps The A21-6 ledger row asserted the opposite of its own committed capture. The row claimed the uncontended first attempt always lands, 5/5; the capture records the foreground transition landing on neither attempt in any of five trials, with first_attempt_success_rate 0. The false sentence had propagated into docs/phases.md and into a doc comment on the shipped retry constant, so the product cited a measurement its evidence contradicted. Row, phase document and doc comment now state what was measured. The design stands: a finite budget with fail-closed exhaustion is better supported by 0/5 than by 5/5, because an unbounded retry against a target that never foregrounds would hang instead of failing. Two lifecycle tests targeted std::process::id() and drove the real graceful close, whose fan-out posts WM_CLOSE to every window the pid owns - including every in-process fixture window belonging to tests running in parallel, which DefWindowProcW then destroys. Both now target a child-process fixture whose window set the test owns. Production honesty: an all-skips fan-out no longer claims the OS accepted a request that was never posted, and WaitForSingleObject's failure is no longer read as "alive" when the wait could not judge liveness at all. Launch closes its process handles before the first fallible step and merges environment overrides case-insensitively, so an override of Path no longer loses to an inherited PATH. Test integrity: the launch probes no longer terminate every notepad.exe on the host - a developer's open Notepad and its unsaved work were being killed by a test run. A copied notepad is not a usable probe either; it re-execs and exits before the process table is read, and which system applications survive copying differs per image. The probe is now a copy of the test binary hosting a real window, which is repo-controlled and present everywhere. Also pins the previously untested keyboard-focus timeout branch, replaces a tautological activation assertion with independently read facts, stops a parity helper claiming a cross-platform match it only restated from literals, and removes the cwd-marker read race that CI reproduced. Captures three learnings in docs/solutions: a cited measurement must match its capture, a deadline cannot interrupt a blocking OS call, and a test that acts on its own runner acts on every other test. * docs: refresh the learnings this sub-phase moved or extended The identity-fingerprint learning pointed at window_resolve.rs for focus_window and its per-write ownership re-checks, which this sub-phase moved to window_activate.rs. The solution it describes is unchanged; only the references had drifted, so they now name where the code lives. Record the falsifiability shape this review found: a setup guard that returns instead of asserting. Four live tests opened by staging a real OS activation and returning when it did not land, so a degraded desktop passed them while asserting nothing - including the only test proving the method was wired through its trait. It is the hardest shape to catch in review because the early return reads as hygiene rather than a hole. |
||
|---|---|---|
| .githooks | ||
| .github | ||
| assets | ||
| benchmarks/locator-resolution | ||
| crates | ||
| docs | ||
| npm | ||
| probes/windows | ||
| scripts | ||
| skills | ||
| src | ||
| tests | ||
| .gitignore | ||
| .gitleaks.toml | ||
| .release-please-manifest.json | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| clippy.toml | ||
| CODE_OF_CONDUCT.md | ||
| CONCEPTS.md | ||
| CONTRIBUTING.md | ||
| deny.toml | ||
| LICENSE | ||
| README.md | ||
| release-please-config.json | ||
| rust-toolchain.toml | ||
| SECURITY.md | ||
| SUPPORT.md | ||
AGENT DESKTOP
OBSERVE. DECIDE. ACT.
agent-desktop is a native desktop automation CLI designed for AI agents, built with Rust. It gives structured access to any application through OS accessibility trees — no screenshots, no pixel matching, no browser required.
Architecture
Key Features
- Native Rust CLI: Fast, single binary, no runtime dependencies
- C-ABI cdylib (
libagent_desktop_ffi): Load once from Python / Swift / Go / Ruby / Node / C instead of forking the CLI per call - 58 command names, 54 operational commands: Observation, interaction, keyboard, mouse, notifications, clipboard, window management, session lifecycle, trace read/export, plus a bundled
skillsdoc loader. The four held-input names are reserved for a stateful daemon and fail closed in the stateless CLI. - Progressive skeleton traversal: 78–96% token reduction on dense apps via shallow overview + targeted drill-down
- Snapshot & refs: AI-optimized workflow using compact snapshot IDs and qualified element references (
@s8f3k2p9:e1,@s8f3k2p9:e2) - Headless-by-default interactions: Ref actions use accessibility APIs and block silent focus, cursor, keyboard, or pasteboard side effects
- Structured JSON output: Machine-readable responses with error codes and recovery hints
- Works with any app: Finder, Safari, System Settings, Xcode, Slack — anything with an accessibility tree
Installation
npm (recommended)
npm install -g agent-desktop # downloads prebuilt binary automatically
Or without installing:
npx agent-desktop snapshot --app Finder -i
From source
git clone https://github.com/lahfir/agent-desktop
cd agent-desktop
cargo build --release
cp target/release/agent-desktop /usr/local/bin/
Requires Rust 1.89+ and macOS 13.0+.
Permissions
macOS requires Accessibility permission. Screenshots also require Screen Recording permission, and the Notification Center opener requires Automation permission for System Events. Plain permission checks never prompt. Request missing permissions in a bounded isolated helper with:
agent-desktop permissions --request # request missing permissions in an isolated helper
Permission fields are explicit objects, for example:
{
"accessibility": { "state": "granted" },
"screen_recording": { "state": "denied", "suggestion": "Grant Screen Recording permission" },
"automation": { "state": "unknown" }
}
Automation reports granted, denied, or unknown; unknown means macOS would need to prompt or System Events could not be probed without prompting.
Language bindings (FFI)
Every GitHub Release ships a prebuilt C-ABI cdylib (libagent_desktop_ffi) for macOS, Linux, and Windows alongside the CLI tarballs. dlopen it and call the functions declared in agent_desktop.h for in-process calls instead of fork-exec per command.
import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_init(3) # verify ABI major (AD_ABI_VERSION_MAJOR) before any call
adapter = lib.ad_adapter_create()
# observe -> act: ad_snapshot -> parse a qualified ref -> ad_execute_by_ref ...
lib.ad_adapter_destroy(adapter)
Full consumer guide — entrypoints, ownership, threading, error-handling, build/link, release archives, and verification: skills/agent-desktop-ffi/.
Core Workflow for AI
For dense apps (Slack, VS Code, Notion), use progressive skeleton traversal to minimize token usage:
# 1. Shallow overview — depth-3 map, truncated containers show children_count
agent-desktop snapshot --skeleton --app Slack -i --compact
# Keep snapshot_id, for example s8f3k2p9
# 2. Drill into a region of interest (named containers get refs as drill targets)
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact
# 3. Act on an element found in the drill-down
agent-desktop click @e12 --snapshot s8f3k2p9
# 4. Re-drill the same region to verify the state change
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact
For simple apps, a full snapshot is fine:
agent-desktop snapshot --app Finder -i # get interactive elements with refs and snapshot_id
agent-desktop click @e3 --snapshot s8f3k2p9 # click a button by ref
agent-desktop type @e5 --snapshot s8f3k2p9 "quarterly report" # insert text into a field
agent-desktop press cmd+s # keyboard shortcut
agent-desktop snapshot -i # re-observe after UI changes
Agent loop: snapshot → decide → act → snapshot → decide → act → ...
Trace viewer (read back a session)
session_id=$(agent-desktop session start --screenshots | jq -r '.data.session_id')
export AGENT_DESKTOP_SESSION="$session_id"
agent-desktop snapshot --app Finder -i # work inside the explicit session scope
agent-desktop click @s8f3k2p9:e5
agent-desktop trace show --limit 500 # bounded JSON timeline for agents
agent-desktop trace export --out run.html # single-file HTML viewer (works from file://)
trace show merges all segment files deterministically and requires no permissions. trace export embeds the timeline plus screenshots as base64 in one static HTML file. Without --out, the HTML is written to the session directory (~/.agent-desktop/sessions/<id>/trace-<id>.html), not the current directory; --out overrides the path. Treat exported HTML like a screenshot when artifacts: full was enabled.
Shared sessions for multi-agent workflows
Run session start once per agent run to create a trace-enabled session (manifest trace: on by default), then pass the returned ID with global --session <id> or AGENT_DESKTOP_SESSION=<id>. Commands in that explicit scope get automatic JSONL segments under ~/.agent-desktop/sessions/<id>/trace/ and share the session's latest-snapshot namespace — no --trace on every call.
For concurrent independent agents, set AGENT_DESKTOP_SESSION=<id> per process. When multiple agents share one session ID, each agent should act on the qualified refs from its own snapshot call rather than assuming the namespace's latest snapshot is unchanged.
Bare --session <id> without a manifest (no session start) still scopes the snapshot namespace only and writes no trace files. Snapshot IDs resolve only inside the selected session namespace; they never trigger a cross-session search.
agent-desktop session start --name release-fix # note data.session_id
export AGENT_DESKTOP_SESSION=<session_id>
agent-desktop snapshot --app Xcode -i --compact # uses selected session + tracing
agent-desktop wait --element @s8f3k2p9:e9 --predicate actionable --timeout 5000
agent-desktop click @s8f3k2p9:e9
agent-desktop click @e9 --snapshot s2 # legacy bare ref, explicitly pinned
agent-desktop session end "$AGENT_DESKTOP_SESSION"
agent-desktop session gc
Commands
Observation
agent-desktop snapshot --app Safari -i # accessibility tree with refs
agent-desktop snapshot --surface menu # capture open menu
agent-desktop screenshot --app Finder # PNG screenshot
agent-desktop find --role button --app TextEdit # search by role, name, value, text
agent-desktop get @e3 --snapshot s8f3k2p9 --property value # read element property
agent-desktop is @e7 --snapshot s8f3k2p9 --property checked # check boolean state
agent-desktop list-surfaces --app Notes # list menus, sheets, popovers, alerts
get and is resolve the ref once, prefer live platform reads when available, and fall back only when that live read is unsupported by the adapter.
Interaction
agent-desktop click @s8f3k2p9:e3 # strict headless AX click
agent-desktop --headed click @s8f3k2p9:e3 # physical click, focus/cursor allowed
agent-desktop --headed double-click @s8f3k2p9:e3 # physical double-click
agent-desktop --headed triple-click @s8f3k2p9:e3 # physical triple-click
agent-desktop right-click @s8f3k2p9:e3 # open context menu; inspect effect before retrying
agent-desktop type @s8f3k2p9:e5 "hello world" # insert text into element
agent-desktop set-value @s8f3k2p9:e5 "new value" # set value directly via AX
agent-desktop clear @s8f3k2p9:e5 # clear element value
agent-desktop focus @s8f3k2p9:e5 # set keyboard focus
agent-desktop select @s8f3k2p9:e9 "Option B" # select verified dropdown/list option
agent-desktop toggle @s8f3k2p9:e12 # flip checkbox or switch
agent-desktop check @s8f3k2p9:e12 # idempotent check
agent-desktop uncheck @s8f3k2p9:e12 # idempotent uncheck
agent-desktop expand @s8f3k2p9:e15 # expand disclosure/tree item
agent-desktop collapse @s8f3k2p9:e15 # collapse disclosure/tree item
agent-desktop scroll @s8f3k2p9:e1 --direction down --amount 3 # strict headless AX scroll
agent-desktop scroll-to @s8f3k2p9:e20 # scroll element into view
(macOS, Phase 1) Default ref actions are strict headless semantic operations. In headed mode, core focuses the exact ref window before dispatch; pointer commands additionally require a verified target point, while the adapter owns physical delivery.
click,right-click,type,clear, andscrollare physical-first; double/triple-click, hover, and drag are physical-only; expand/collapse and other semantic commands remain semantic. Raw coordinates never imply a target window and therefore never steal focus. Seeskills/agent-desktop/references/commands-interaction.md.
Keyboard
agent-desktop press cmd+s # key combo
agent-desktop press cmd+shift+z # multi-modifier
agent-desktop press escape # single key
key-down and key-up are reserved command names and return ACTION_NOT_SUPPORTED until a stateful daemon can own the held-key lifetime.
Mouse
agent-desktop --headed hover @s8f3k2p9:e3 # move cursor to element
agent-desktop --headed hover --xy 500,300 # move cursor to coordinates
agent-desktop --headed drag --from @s8f3k2p9:e3 --to @s8f3k2p9:e8 # drag between elements
agent-desktop --headed drag --from-xy 100,200 --to-xy 400,200 # drag between coordinates
agent-desktop --headed mouse-click --xy 500,300 # click at coordinates
mouse-down and mouse-up are likewise reserved; use the atomic mouse-click or drag commands.
App & Window Management
agent-desktop launch Safari # launch app by name
agent-desktop launch com.apple.Safari # launch by bundle ID
agent-desktop close-app Safari # quit app
agent-desktop close-app Safari --force # force quit (SIGTERM, then SIGKILL if needed)
agent-desktop list-apps # list running GUI apps
agent-desktop list-windows # list visible windows
agent-desktop list-windows --app Finder # windows for specific app
agent-desktop focus-window --window-id w-4521 # bring exact window to front
agent-desktop resize-window --window-id w-4521 --width 800 --height 600
agent-desktop move-window --window-id w-4521 --x 100 --y 100
agent-desktop minimize --window-id w-4521
agent-desktop maximize --window-id w-4521
agent-desktop restore --window-id w-4521
Notifications (macOS only)
agent-desktop --headed list-notifications # open Notification Center if needed, then list
agent-desktop --headed list-notifications --app "Slack" # filter by app
agent-desktop --headed list-notifications --text "deploy" --limit 5 # filter by text
agent-desktop --headed dismiss-notification 1 --expected-app "Slack" --expected-title "Deploy complete"
agent-desktop --headed dismiss-all-notifications # dismiss all
agent-desktop --headed dismiss-all-notifications --app "Slack" # dismiss all from app
agent-desktop --headed notification-action 1 "Reply" --expected-app "Slack" --expected-title "Deploy complete"
Single-notification mutations require an app or title fingerprint from the
same listing. Every mutation requires --headed because it opens and focuses
Notification Center. Headless listing can only observe an already-open center;
headed listing may open it and restore the prior frontmost app afterward.
Clipboard
agent-desktop clipboard-get # read clipboard text
agent-desktop clipboard-set "copied" # write to clipboard
agent-desktop clipboard-clear # clear clipboard
Wait
agent-desktop wait 500 # sleep 500ms
agent-desktop wait --element @s8f3k2p9:e3 --timeout 5000 # wait for element
agent-desktop wait --element @s8f3k2p9:e3 --predicate actionable # wait until safe to act
agent-desktop wait --element @s8f3k2p9:e5 --predicate value --value ready
agent-desktop wait --window "Save" --timeout 10000 # wait for window
agent-desktop wait --text "Loading complete" --app Safari # wait for text
agent-desktop wait --text "Done" --count 1 --app Xcode # wait for exact match count
agent-desktop wait --notification --text "Build Succeeded" # wait for new matching notification
agent-desktop wait --menu --timeout 3000 # wait for menu
Batch
agent-desktop batch '[
{"command": "click", "args": {"ref_id": "@e2", "snapshot": "<snapshot_id>"}},
{"command": "type", "args": {"ref_id": "@e5", "snapshot": "<snapshot_id>", "text": "hello"}},
{"command": "press", "args": {"combo": "return"}}
]' --stop-on-error
agent-desktop --session run-a batch '[
{"command": "snapshot", "args": {"app": "Finder", "interactive_only": true}},
{"command": "status", "session": "run-b", "args": {}}
]'
System
agent-desktop session start [--name LABEL] [--no-trace] # create session; pass returned ID explicitly
agent-desktop session end [id]
agent-desktop session list
agent-desktop session gc [--older-than SECS] [--ended]
agent-desktop status # platform, permissions, session_id, tracing, latest snapshot
agent-desktop permissions # check accessibility/screen-recording/automation
agent-desktop permissions --request # request in the bounded isolated helper
agent-desktop version # version string
agent-desktop skills get desktop --full # bundled agent guidance
Snapshot Options
agent-desktop snapshot [OPTIONS]
| Flag | Default | Description |
|---|---|---|
--app <NAME> |
focused app | Filter to a specific application |
--window-id <ID> |
- | Filter to a specific window |
-i / --interactive-only |
off | Only include interactive elements |
--compact |
off | Omit empty structural nodes |
--include-bounds |
off | Include pixel bounds (x, y, width, height) |
--max-depth <N> |
10 | Maximum tree depth |
--skeleton |
off | Shallow 3-level overview; truncated containers show children_count and get refs as drill targets |
--root <REF> |
- | Start traversal from this ref; merges into existing refmap with scoped invalidation |
--snapshot <snapshot_id> |
latest | Snapshot ID to use when resolving --root |
--surface <TYPE> |
window | window, focused, menu, menubar, sheet, popover, alert |
JSON Output
See the versioned JSON envelope, error-code, and exit-code contract.
Ref System
snapshot assigns local positions in depth-first order and emits qualified refs such as @s8f3k2p9:e1, @s8f3k2p9:e2, and @s8f3k2p9:e3. A qualified ref embeds the exact snapshot ID and needs no separate --snapshot. Legacy bare refs such as @e3 remain accepted only with an explicit --snapshot s8f3k2p9. Snapshot lookup stays inside the selected session namespace.
Interactive roles that receive refs: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell, radiobutton, incrementor, menubutton, switch, colorwell, dockitem.
Static elements (labels, groups, containers) appear in the tree for context but have no ref.
Reliability contract:
session startcreates and returns a manifest-gated session with automatic trace segments. It does not activate later processes. Activation resolves explicit--sessionfirst, thenAGENT_DESKTOP_SESSION; otherwise the command uses the global, non-session namespace.- Bare
--session <id>without a manifest scopes snapshots only — no surprise trace files for existing callers. - Snapshot lookup is confined to the selected namespace. A session-owned snapshot requires the same explicit
--sessionorAGENT_DESKTOP_SESSIONscope. - Ref actions re-identify targets at action time: a moved unique target can proceed, while missing or changed stable identity returns
STALE_REF. - Mutable value text is not treated as stable identity, so text fields and timers can keep resolving when the saved window, path, role, and bounds evidence still identify the same element.
- Multiple plausible targets return
AMBIGUOUS_TARGETinstead of choosing arbitrarily. - Actions run an actionability preflight before dispatch: visibility, stability, enabled state, supported action, policy, and editability.
wait --element @s8f3k2p9:e3 --predicate actionablepolls until the target can be acted on.- With an active trace-enabled session, JSONL segments land under
sessions/<id>/trace/<pid>-*.jsonlautomatically.--trace <path>overrides to one file;--trace-strictfails on setup and pre-action writes (post-action traces are best-effort).
Stale ref recovery:
snapshot → act → STALE_REF or AMBIGUOUS_TARGET? → wait/snapshot again → retry with the new ref
Platform Support
| macOS | Windows | Linux | |
|---|---|---|---|
| Accessibility tree | Yes | Planned | Planned |
| Click / type / keyboard | Yes | Planned | Planned |
| Mouse input | Yes | Planned | Planned |
| Screenshot | Yes | Planned | Planned |
| Clipboard | Yes | Planned | Planned |
| App & window management | Yes | Planned | Planned |
| Notifications | Yes | Planned | Planned |
Development
cargo build # debug build
cargo build --release # optimized (<15MB)
cargo test --lib --workspace # run tests
cargo clippy --all-targets -- -D warnings # lint (must pass with zero warnings)
FAQ
See the complete FAQ for architecture, platform support, installation, refs, licensing, and support links.
License
Apache-2.0