No description
Find a file
Lahfir 4f7bad337c
feat: windows input synthesis (sub-phase 2.8) (#123)
* docs: add sub-phase 2.8 input synthesis plan

* docs: harden the 2.8 input synthesis plan after adversarial review

* docs: give every 2.8 deferral a receiving owner in phases.md

* feat: measure windows input synthesis gaps (area 20)

Probe the unmeasured SendInput and token facts 2.8 depends on so coordinate, abort-drag, double-click, integrity, and cost branches are settled before the adapter InputOps land.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: implement windows mouse_event via SendInput

Give bare-coordinate hover/click/wheel a real physical path with primary-vs-virtual-desktop normalization, FFI held-edge guards, and a release-safe modifier hold around multi-click.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: implement windows drag with origin release guard

Port the macOS abort-state contract onto SendInput so an interrupted drag posts corrective release at the origin and reports delivered_unverified instead of claiming a clean failure.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: implement windows keyboard synthesis and blocked combos

Ship chord and UTF-16 type_text primitives with release guards, macOS key-name parity, and a Windows-specific dangerous-combo list so press/type can land without stuck keys or silent session-kill shortcuts.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: detect uipi elevation before physical input

Compare caller and target integrity levels so a Medium-into-High SendInput attempt fails closed as PERM_DENIED instead of reporting a silent no-op as success.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: wire windows physical execute_action legs

Replace the 2.7 not-supported stubs for type, press, multi-click, and right-click with SendInput synthesis gated by focus, elevation, and headed policy.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test: pin windows physical input envelope parity

Lock AdapterError and ActionResult wire shapes for the physical legs against the macOS disposition contract, and cite the A20-6 hot-path cost baseline already captured in area 20.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: dogfood windows input synthesis and harden delivery

Prove the physical path on Notepad and ScratchForms under the corpus safety envelope, and fix the dogfood-found root-window foreground gate plus SendInput absolute normalization and drag pickup timing.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs: align windows input synthesis docs with shipped evidence

Restate phases, concepts, skill wording, and ledger rows so 2.8 reads as what shipped, including UIPI detection closure and the type command divergence handed to 2.15.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: harden windows input abort guards after review

Strip plan decision ids from crates so CI phase-reference scan passes, arm a click release guard around each down/up, and re-verify keyboard focus at every type_text chunk boundary.

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor: give the mouse input constructors and release-guard state one home

button_input existed verbatim in four files and move_input in three, so the
corrective batch an abort depends on could drift from the normal post. The
click and drag guards also carried the same armed-and-counted state machine
and the same delivery report, differing only in their recovery hint.

Both now have a single home: mouse_send.rs owns the SendInput surface -
the MOUSEEVENTF_* flags with their compile-time assertions against
windows-sys, plus the two input constructors - and release_state.rs owns
the guard state and the delivery report it produces. Each guard keeps its
own Drop, because only the guard knows whether the corrective post is a
button-up in place or a move back to the origin first.

* fix: resolve the host window before refusing a physical click

target_window_is_foreground read the leaf element's NativeWindowHandle,
which UIA reports as 0 - success, not failure - for every element that is
not itself a window. That is the normal shape for WPF, WinUI, UWP and
Chromium content, so double-click, triple-click and right-click were
refused before injection on most modern desktop UI. The handle is now
resolved the way the rest of the crate resolves it, by climbing to the
first ancestor that owns one.

Also from the same review pass:
- mouse-wheel dropped the caller's modifiers, so the documented
  --modifiers shift horizontal-scroll chord silently degraded to a plain
  vertical scroll while still reporting success.
- The drag preflight reserved one pickup delay where the sequence spends
  two, admitting drags that then aborted mid-gesture with the button
  down. Both delays now come from one constant.
- The skill docs carried plan decision ids and sub-phase numbers, and
  they are include_str!d into the binary, so agent-desktop skills served
  them to agents. The phase-reference scan now covers skills/ for exactly
  that reason and caught a second instance in SKILL.md.

* fix: close the remaining input-synthesis review findings

- The click sequence counted delivered events per click and let the
  inter-click gap fail through a bare ?, so a --count 3 that aborted late
  reported two events, or none at all with an unknown disposition.
  Accounting now spans the sequence while cleanup stays per click. Wheel
  chunks report the same way: a partial scroll says how much landed.
- mouse_modifier opened a second SendInput keyboard path with its own
  fake sink, so the one-seam invariant keyboard_send documents was
  already false. It now posts through that seam and reuses its VK map.
- type_text's focus gate and per-chunk re-verify were exercised through a
  cfg(test) clone that omitted the policy gate, the preflight and the real
  verifier, so deleting either guard left the suite green. The gate is one
  production function now, with the verifier injected, and a test drives
  it with a verifier that fails mid-sequence.
- Two drag tests asserted a std-library identity and re-typed the
  production step arithmetic. They drive synthesize_drag and read the
  posted event stream instead. Inverting the first showed the zero-dwell
  early return was dead - the loop guard already covered it - so it is
  gone.
- elevation's module doc blamed unavailable token manufacture for the
  unproven cross-boundary effect; the captures record manufacture working
  with no higher-integrity target to aim at. keyboard's docs claimed its
  primitives were unreachable from execute_action, which calls both.

* docs: capture the input-synthesis learnings and correct what the run overstated

Two new solutions docs and one extended:
- An enforcement gate must cover everything the binary embeds. The
  phase-reference scan read crates/ and src/ for .rs files while the skill
  markdown it never looked at was include_str!d into the binary, so plan
  ids were being served to agents by the skills command. Widening the
  scope immediately found a ninth instance nobody had reported.
- A zero success value is not the answer you asked for. UIA returns
  NativeWindowHandle 0 as a success for any element that is not a window,
  and the crate already had the climb that treats it as keep-looking; the
  physical-click gate re-derived a naive version and refused delivery on
  WPF, WinUI and Chromium.
- The cannot-fail taxonomy gains the cfg(test) re-implementation shape,
  and a prevention note to invert one production site per run: two
  mutations at once made a test fail on the other one's arithmetic and
  hid the failure that mattered.

The dogfood report claimed the multi-click path more broadly than it
measured it. Both multi-click judgements ran against HWND-bearing targets,
which is precisely where the defect was invisible, so J4 now states its
scope and the residual records that a non-HWND judgement is still owed.

* docs: judge multi-click on a non-HWND target and close the residual

The original run judged multi-click only against WinForms and Notepad,
both of which own a window handle - the one shape where the foreground
gate's leaf-handle read could not fail. J10 runs the same WPF button with
the pre-fix and fixed binaries back to back: the first reports
ACTION_FAILED and not_delivered with the app's click counter untouched,
the second lands and the counter advances to action:2, read back
independently of the envelope.

Two correct refusals fell out of the same run and are recorded with it:
the occlusion gate naming the terminal window that actually covered the
fixture, and the private-file guard rejecting a store an elevated run had
left owned by the Administrators group rather than the user.

* fix: block modifier supersets of a dangerous shortcut

Exact canonical-string matching let alt+shift+tab through to SendInput.
That is the reverse task switcher and takes the foreground exactly as the
alt+tab it extends, which is the harm the list exists to prevent. A test
asserted the old behaviour as correct, reasoning that an extra modifier
makes a shortcut unrelated; shift does not neutralize alt+f4 and does not
neutralize alt+tab, so the premise was wrong and the test is gone.

Matching is now key equality plus modifier superset. It over-blocks a few
combinations nobody presses, and that trade is deliberate: a wrongly
blocked combo costs one --force, a wrongly allowed one moves input to
another window mid-run.

macOS answers the same problem by enumerating variants (it lists cmd+q and
cmd+shift+q separately), so the two adapters now decide danger by
different rules. Recorded as a settlement item for the hardening gate
rather than changed here, since the macOS crate is the GA line.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-08 05:29:03 -07:00
.githooks feat: windows resolution and live locator (sub-phase 2.5) (#120) 2026-08-06 16:14:04 -07:00
.github feat: windows input synthesis (sub-phase 2.8) (#123) 2026-08-08 05:29:03 -07:00
assets refactor!: unify command execution contracts 2026-05-19 18:27:08 -07:00
benchmarks/locator-resolution feat!: implement Playwright-grade foundation contract 2026-07-20 00:21:38 -07:00
crates feat: windows input synthesis (sub-phase 2.8) (#123) 2026-08-08 05:29:03 -07:00
docs feat: windows input synthesis (sub-phase 2.8) (#123) 2026-08-08 05:29:03 -07:00
npm chore(main): release 0.7.0 (#118) 2026-08-02 01:07:39 -04:00
probes/windows feat: windows input synthesis (sub-phase 2.8) (#123) 2026-08-08 05:29:03 -07:00
scripts feat: windows input synthesis (sub-phase 2.8) (#123) 2026-08-08 05:29:03 -07:00
skills feat: windows input synthesis (sub-phase 2.8) (#123) 2026-08-08 05:29:03 -07:00
src feat: windows observation read path — snapshot, inventories, chromium settle, drill-down (#119) 2026-08-02 20:31:09 -07:00
tests feat!: implement Playwright-grade foundation contract 2026-07-20 00:21:38 -07:00
.gitignore feat: windows vocabulary — roles, states, native_id and name evidence (#115) 2026-08-01 14:25:02 -07:00
.gitleaks.toml chore: track planning artifacts with secret and privacy scanning (#109) 2026-07-26 01:41:55 -07:00
.release-please-manifest.json chore(main): release 0.7.0 (#118) 2026-08-02 01:07:39 -04:00
Cargo.lock feat: windows observation read path — snapshot, inventories, chromium settle, drill-down (#119) 2026-08-02 20:31:09 -07:00
Cargo.toml chore(main): release 0.7.0 (#118) 2026-08-02 01:07:39 -04:00
CHANGELOG.md chore(main): release 0.7.0 (#118) 2026-08-02 01:07:39 -04:00
CLAUDE.md feat: windows input synthesis (sub-phase 2.8) (#123) 2026-08-08 05:29:03 -07:00
clippy.toml feat: Phase 1 foundation — workspace scaffold, core engine, macOS adapter, 31 commands 2026-02-19 10:44:38 -08:00
CODE_OF_CONDUCT.md docs: add community health files for GitHub community standards (#83) 2026-06-28 21:41:39 -04:00
CONCEPTS.md feat: windows input synthesis (sub-phase 2.8) (#123) 2026-08-08 05:29:03 -07:00
CONTRIBUTING.md docs: add community health files for GitHub community standards (#83) 2026-06-28 21:41:39 -04:00
deny.toml refactor!: unify command execution contracts 2026-05-19 18:27:08 -07:00
LICENSE feat(ffi): ship C-ABI cdylib with review hardening and release pipeline (#26) 2026-04-17 04:00:43 -07:00
README.md feat!: implement Playwright-grade foundation contract 2026-07-20 00:21:38 -07:00
release-please-config.json fix: correct GitHub Release download URL and simplify tag format 2026-02-23 01:20:16 -08:00
rust-toolchain.toml refactor!: remove speculative Win32 private-file layer from core, add real Windows/Linux test lanes (#106) 2026-07-25 23:25:28 -07:00
SECURITY.md refactor!: unify command execution contracts 2026-05-19 18:27:08 -07:00
SUPPORT.md docs: add community health files for GitHub community standards (#83) 2026-06-28 21:41:39 -04:00

AGENT DESKTOP

OBSERVE. DECIDE. ACT.

CI status GitHub release npm version ClawHub skill skills.sh listing Apache-2.0 License

agent-desktop tutorial demo

agent-desktop is a native desktop automation CLI designed for AI agents, built with Rust. It gives structured access to any application through OS accessibility trees — no screenshots, no pixel matching, no browser required.

Architecture

agent-desktop architecture diagram

agent-desktop real-world example — Slack accessibility tree with 97% token savings

Star history for lahfir/agent-desktop

Key Features

  • Native Rust CLI: Fast, single binary, no runtime dependencies
  • C-ABI cdylib (libagent_desktop_ffi): Load once from Python / Swift / Go / Ruby / Node / C instead of forking the CLI per call
  • 58 command names, 54 operational commands: Observation, interaction, keyboard, mouse, notifications, clipboard, window management, session lifecycle, trace read/export, plus a bundled skills doc loader. The four held-input names are reserved for a stateful daemon and fail closed in the stateless CLI.
  • Progressive skeleton traversal: 7896% token reduction on dense apps via shallow overview + targeted drill-down
  • Snapshot & refs: AI-optimized workflow using compact snapshot IDs and qualified element references (@s8f3k2p9:e1, @s8f3k2p9:e2)
  • Headless-by-default interactions: Ref actions use accessibility APIs and block silent focus, cursor, keyboard, or pasteboard side effects
  • Structured JSON output: Machine-readable responses with error codes and recovery hints
  • Works with any app: Finder, Safari, System Settings, Xcode, Slack — anything with an accessibility tree

Installation

npm install -g agent-desktop        # downloads prebuilt binary automatically

Or without installing:

npx agent-desktop snapshot --app Finder -i

From source

git clone https://github.com/lahfir/agent-desktop
cd agent-desktop
cargo build --release
cp target/release/agent-desktop /usr/local/bin/

Requires Rust 1.89+ and macOS 13.0+.

Permissions

macOS requires Accessibility permission. Screenshots also require Screen Recording permission, and the Notification Center opener requires Automation permission for System Events. Plain permission checks never prompt. Request missing permissions in a bounded isolated helper with:

agent-desktop permissions --request   # request missing permissions in an isolated helper

Permission fields are explicit objects, for example:

{
  "accessibility": { "state": "granted" },
  "screen_recording": { "state": "denied", "suggestion": "Grant Screen Recording permission" },
  "automation": { "state": "unknown" }
}

Automation reports granted, denied, or unknown; unknown means macOS would need to prompt or System Events could not be probed without prompting.

Language bindings (FFI)

Every GitHub Release ships a prebuilt C-ABI cdylib (libagent_desktop_ffi) for macOS, Linux, and Windows alongside the CLI tarballs. dlopen it and call the functions declared in agent_desktop.h for in-process calls instead of fork-exec per command.

import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_init(3)  # verify ABI major (AD_ABI_VERSION_MAJOR) before any call
adapter = lib.ad_adapter_create()
# observe -> act: ad_snapshot -> parse a qualified ref -> ad_execute_by_ref ...
lib.ad_adapter_destroy(adapter)

Full consumer guide — entrypoints, ownership, threading, error-handling, build/link, release archives, and verification: skills/agent-desktop-ffi/.

Core Workflow for AI

For dense apps (Slack, VS Code, Notion), use progressive skeleton traversal to minimize token usage:

# 1. Shallow overview — depth-3 map, truncated containers show children_count
agent-desktop snapshot --skeleton --app Slack -i --compact
# Keep snapshot_id, for example s8f3k2p9

# 2. Drill into a region of interest (named containers get refs as drill targets)
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact

# 3. Act on an element found in the drill-down
agent-desktop click @e12 --snapshot s8f3k2p9

# 4. Re-drill the same region to verify the state change
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact

For simple apps, a full snapshot is fine:

agent-desktop snapshot --app Finder -i   # get interactive elements with refs and snapshot_id
agent-desktop click @e3 --snapshot s8f3k2p9  # click a button by ref
agent-desktop type @e5 --snapshot s8f3k2p9 "quarterly report"  # insert text into a field
agent-desktop press cmd+s               # keyboard shortcut
agent-desktop snapshot -i               # re-observe after UI changes
Agent loop:  snapshot → decide → act → snapshot → decide → act → ...

Trace viewer (read back a session)

session_id=$(agent-desktop session start --screenshots | jq -r '.data.session_id')
export AGENT_DESKTOP_SESSION="$session_id"
agent-desktop snapshot --app Finder -i       # work inside the explicit session scope
agent-desktop click @s8f3k2p9:e5
agent-desktop trace show --limit 500         # bounded JSON timeline for agents
agent-desktop trace export --out run.html    # single-file HTML viewer (works from file://)

trace show merges all segment files deterministically and requires no permissions. trace export embeds the timeline plus screenshots as base64 in one static HTML file. Without --out, the HTML is written to the session directory (~/.agent-desktop/sessions/<id>/trace-<id>.html), not the current directory; --out overrides the path. Treat exported HTML like a screenshot when artifacts: full was enabled.

Shared sessions for multi-agent workflows

Run session start once per agent run to create a trace-enabled session (manifest trace: on by default), then pass the returned ID with global --session <id> or AGENT_DESKTOP_SESSION=<id>. Commands in that explicit scope get automatic JSONL segments under ~/.agent-desktop/sessions/<id>/trace/ and share the session's latest-snapshot namespace — no --trace on every call.

For concurrent independent agents, set AGENT_DESKTOP_SESSION=<id> per process. When multiple agents share one session ID, each agent should act on the qualified refs from its own snapshot call rather than assuming the namespace's latest snapshot is unchanged.

Bare --session <id> without a manifest (no session start) still scopes the snapshot namespace only and writes no trace files. Snapshot IDs resolve only inside the selected session namespace; they never trigger a cross-session search.

agent-desktop session start --name release-fix          # note data.session_id
export AGENT_DESKTOP_SESSION=<session_id>
agent-desktop snapshot --app Xcode -i --compact          # uses selected session + tracing
agent-desktop wait --element @s8f3k2p9:e9 --predicate actionable --timeout 5000
agent-desktop click @s8f3k2p9:e9
agent-desktop click @e9 --snapshot s2                    # legacy bare ref, explicitly pinned
agent-desktop session end "$AGENT_DESKTOP_SESSION"
agent-desktop session gc

Commands

Observation

agent-desktop snapshot --app Safari -i           # accessibility tree with refs
agent-desktop snapshot --surface menu            # capture open menu
agent-desktop screenshot --app Finder            # PNG screenshot
agent-desktop find --role button --app TextEdit  # search by role, name, value, text
agent-desktop get @e3 --snapshot s8f3k2p9 --property value  # read element property
agent-desktop is @e7 --snapshot s8f3k2p9 --property checked # check boolean state
agent-desktop list-surfaces --app Notes          # list menus, sheets, popovers, alerts

get and is resolve the ref once, prefer live platform reads when available, and fall back only when that live read is unsupported by the adapter.

Interaction

agent-desktop click @s8f3k2p9:e3                  # strict headless AX click
agent-desktop --headed click @s8f3k2p9:e3         # physical click, focus/cursor allowed
agent-desktop --headed double-click @s8f3k2p9:e3  # physical double-click
agent-desktop --headed triple-click @s8f3k2p9:e3  # physical triple-click
agent-desktop right-click @s8f3k2p9:e3            # open context menu; inspect effect before retrying
agent-desktop type @s8f3k2p9:e5 "hello world"     # insert text into element
agent-desktop set-value @s8f3k2p9:e5 "new value"  # set value directly via AX
agent-desktop clear @s8f3k2p9:e5                  # clear element value
agent-desktop focus @s8f3k2p9:e5                  # set keyboard focus
agent-desktop select @s8f3k2p9:e9 "Option B"      # select verified dropdown/list option
agent-desktop toggle @s8f3k2p9:e12                # flip checkbox or switch
agent-desktop check @s8f3k2p9:e12                 # idempotent check
agent-desktop uncheck @s8f3k2p9:e12               # idempotent uncheck
agent-desktop expand @s8f3k2p9:e15                # expand disclosure/tree item
agent-desktop collapse @s8f3k2p9:e15              # collapse disclosure/tree item
agent-desktop scroll @s8f3k2p9:e1 --direction down --amount 3  # strict headless AX scroll
agent-desktop scroll-to @s8f3k2p9:e20             # scroll element into view

(macOS, Phase 1) Default ref actions are strict headless semantic operations. In headed mode, core focuses the exact ref window before dispatch; pointer commands additionally require a verified target point, while the adapter owns physical delivery. click, right-click, type, clear, and scroll are physical-first; double/triple-click, hover, and drag are physical-only; expand/collapse and other semantic commands remain semantic. Raw coordinates never imply a target window and therefore never steal focus. See skills/agent-desktop/references/commands-interaction.md.

Keyboard

agent-desktop press cmd+s               # key combo
agent-desktop press cmd+shift+z          # multi-modifier
agent-desktop press escape               # single key

key-down and key-up are reserved command names and return ACTION_NOT_SUPPORTED until a stateful daemon can own the held-key lifetime.

Mouse

agent-desktop --headed hover @s8f3k2p9:e3                  # move cursor to element
agent-desktop --headed hover --xy 500,300         # move cursor to coordinates
agent-desktop --headed drag --from @s8f3k2p9:e3 --to @s8f3k2p9:e8   # drag between elements
agent-desktop --headed drag --from-xy 100,200 --to-xy 400,200  # drag between coordinates
agent-desktop --headed mouse-click --xy 500,300   # click at coordinates

mouse-down and mouse-up are likewise reserved; use the atomic mouse-click or drag commands.

App & Window Management

agent-desktop launch Safari              # launch app by name
agent-desktop launch com.apple.Safari    # launch by bundle ID
agent-desktop close-app Safari           # quit app
agent-desktop close-app Safari --force   # force quit (SIGTERM, then SIGKILL if needed)
agent-desktop list-apps                  # list running GUI apps
agent-desktop list-windows               # list visible windows
agent-desktop list-windows --app Finder  # windows for specific app
agent-desktop focus-window --window-id w-4521  # bring exact window to front
agent-desktop resize-window --window-id w-4521 --width 800 --height 600
agent-desktop move-window --window-id w-4521 --x 100 --y 100
agent-desktop minimize --window-id w-4521
agent-desktop maximize --window-id w-4521
agent-desktop restore --window-id w-4521

Notifications (macOS only)

agent-desktop --headed list-notifications              # open Notification Center if needed, then list
agent-desktop --headed list-notifications --app "Slack"         # filter by app
agent-desktop --headed list-notifications --text "deploy" --limit 5  # filter by text
agent-desktop --headed dismiss-notification 1 --expected-app "Slack" --expected-title "Deploy complete"
agent-desktop --headed dismiss-all-notifications                # dismiss all
agent-desktop --headed dismiss-all-notifications --app "Slack"  # dismiss all from app
agent-desktop --headed notification-action 1 "Reply" --expected-app "Slack" --expected-title "Deploy complete"

Single-notification mutations require an app or title fingerprint from the same listing. Every mutation requires --headed because it opens and focuses Notification Center. Headless listing can only observe an already-open center; headed listing may open it and restore the prior frontmost app afterward.

Clipboard

agent-desktop clipboard-get              # read clipboard text
agent-desktop clipboard-set "copied"     # write to clipboard
agent-desktop clipboard-clear            # clear clipboard

Wait

agent-desktop wait 500                                       # sleep 500ms
agent-desktop wait --element @s8f3k2p9:e3 --timeout 5000              # wait for element
agent-desktop wait --element @s8f3k2p9:e3 --predicate actionable      # wait until safe to act
agent-desktop wait --element @s8f3k2p9:e5 --predicate value --value ready
agent-desktop wait --window "Save" --timeout 10000           # wait for window
agent-desktop wait --text "Loading complete" --app Safari    # wait for text
agent-desktop wait --text "Done" --count 1 --app Xcode       # wait for exact match count
agent-desktop wait --notification --text "Build Succeeded"   # wait for new matching notification
agent-desktop wait --menu --timeout 3000                     # wait for menu

Batch

agent-desktop batch '[
  {"command": "click", "args": {"ref_id": "@e2", "snapshot": "<snapshot_id>"}},
  {"command": "type", "args": {"ref_id": "@e5", "snapshot": "<snapshot_id>", "text": "hello"}},
  {"command": "press", "args": {"combo": "return"}}
]' --stop-on-error

agent-desktop --session run-a batch '[
  {"command": "snapshot", "args": {"app": "Finder", "interactive_only": true}},
  {"command": "status", "session": "run-b", "args": {}}
]'

System

agent-desktop session start [--name LABEL] [--no-trace]  # create session; pass returned ID explicitly
agent-desktop session end [id]
agent-desktop session list
agent-desktop session gc [--older-than SECS] [--ended]
agent-desktop status                     # platform, permissions, session_id, tracing, latest snapshot
agent-desktop permissions                # check accessibility/screen-recording/automation
agent-desktop permissions --request      # request in the bounded isolated helper
agent-desktop version                    # version string
agent-desktop skills get desktop --full  # bundled agent guidance

Snapshot Options

agent-desktop snapshot [OPTIONS]
Flag Default Description
--app <NAME> focused app Filter to a specific application
--window-id <ID> - Filter to a specific window
-i / --interactive-only off Only include interactive elements
--compact off Omit empty structural nodes
--include-bounds off Include pixel bounds (x, y, width, height)
--max-depth <N> 10 Maximum tree depth
--skeleton off Shallow 3-level overview; truncated containers show children_count and get refs as drill targets
--root <REF> - Start traversal from this ref; merges into existing refmap with scoped invalidation
--snapshot <snapshot_id> latest Snapshot ID to use when resolving --root
--surface <TYPE> window window, focused, menu, menubar, sheet, popover, alert

JSON Output

See the versioned JSON envelope, error-code, and exit-code contract.

Ref System

snapshot assigns local positions in depth-first order and emits qualified refs such as @s8f3k2p9:e1, @s8f3k2p9:e2, and @s8f3k2p9:e3. A qualified ref embeds the exact snapshot ID and needs no separate --snapshot. Legacy bare refs such as @e3 remain accepted only with an explicit --snapshot s8f3k2p9. Snapshot lookup stays inside the selected session namespace.

Interactive roles that receive refs: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell, radiobutton, incrementor, menubutton, switch, colorwell, dockitem.

Static elements (labels, groups, containers) appear in the tree for context but have no ref.

Reliability contract:

  • session start creates and returns a manifest-gated session with automatic trace segments. It does not activate later processes. Activation resolves explicit --session first, then AGENT_DESKTOP_SESSION; otherwise the command uses the global, non-session namespace.
  • Bare --session <id> without a manifest scopes snapshots only — no surprise trace files for existing callers.
  • Snapshot lookup is confined to the selected namespace. A session-owned snapshot requires the same explicit --session or AGENT_DESKTOP_SESSION scope.
  • Ref actions re-identify targets at action time: a moved unique target can proceed, while missing or changed stable identity returns STALE_REF.
  • Mutable value text is not treated as stable identity, so text fields and timers can keep resolving when the saved window, path, role, and bounds evidence still identify the same element.
  • Multiple plausible targets return AMBIGUOUS_TARGET instead of choosing arbitrarily.
  • Actions run an actionability preflight before dispatch: visibility, stability, enabled state, supported action, policy, and editability.
  • wait --element @s8f3k2p9:e3 --predicate actionable polls until the target can be acted on.
  • With an active trace-enabled session, JSONL segments land under sessions/<id>/trace/<pid>-*.jsonl automatically. --trace <path> overrides to one file; --trace-strict fails on setup and pre-action writes (post-action traces are best-effort).

Stale ref recovery:

snapshot → act → STALE_REF or AMBIGUOUS_TARGET? → wait/snapshot again → retry with the new ref

Platform Support

macOS Windows Linux
Accessibility tree Yes Planned Planned
Click / type / keyboard Yes Planned Planned
Mouse input Yes Planned Planned
Screenshot Yes Planned Planned
Clipboard Yes Planned Planned
App & window management Yes Planned Planned
Notifications Yes Planned Planned

Development

cargo build                               # debug build
cargo build --release                     # optimized (<15MB)
cargo test --lib --workspace              # run tests
cargo clippy --all-targets -- -D warnings # lint (must pass with zero warnings)

FAQ

See the complete FAQ for architecture, platform support, installation, refs, licensing, and support links.

License

Apache-2.0