mirror of
https://github.com/lahfir/agent-desktop.git
synced 2026-07-27 17:42:13 +00:00
No description
5fd9543a79
* docs: add the sub-phase 2.0 probe corpus plan The plan for the Windows platform-exploration sub-phase: an eleven-area probe corpus, its findings ledger, and the same-PR phases.md corrections that ledger obliges. Lands on the 2.0 sub-phase branch so the sub-phase PR reviews as one self-contained unit. * feat: add windows probe harness with redaction gate and integrity helpers The harness is the foundation every 2.0 probe codes against: common.ps1 owns the R11 redaction gate, BOM-less UTF-8 capture writers, KTD9 normalization, scratch-process lifecycle with pid tracking, the KTD5 foreground assertion, and a Medium-integrity process launcher. runas.exe /trustlevel:0x20000 was measured on this box and produces a High integrity process (S-1-16-12288) with a deny-only Administrators group, not the Medium label the UIPI probe needs. Start-MediumIntegrityProcess therefore duplicates the token and sets S-1-16-8192 explicitly, asserting the label on read-back so a UIPI row can never be graded against a fake boundary. * feat: add winforms and wpf scratch fixtures with explicit automation ids The fixtures are the interaction targets for the pattern census, identity stability diff, raw interaction exercises, and the COM shim's event sources. Two measured platform facts shaped them. First, the .NET Core rule that a WinForms control's Name becomes its UIA AutomationId does not hold on .NET Framework 4.8 -- the HWND provider supplies raw window handles instead, which change every launch and are useless for an identity-stability diff. The fixture installs a server-side provider per control, with a switch to fall back to explicit Win32 control ids. Second, Win32 UIA client-side proxies are inert on this VM: the WinForms fixture exposes 23 controls with zero patterns, every one a Pane, matching stock Notepad's Edit reporting as a bare Pane. The WPF fixture exposes 25 controls with a full pattern surface. Interaction probes therefore target WPF and record the WinForms absence as evidence rather than working around it. * feat: capture windows environment, session, and integrity baseline Every ledger row inherits these machine facts, so they are captured reproducibly rather than asserted: build 17763.7434, session and interactivity, mandatory integrity label, UAC policy state, default file ownership, display topology, and tool inventory. Identity is recorded as a shape, not a value. The account SID keeps its well-known RID tail (S-1-5-21-<redacted>-500) because the RID is the load-bearing fact, while integrity and well-known group SIDs stay verbatim for the same reason. Three findings the later units depend on. The GAC UIAutomationClient assembly exports 57 types, all managed System.Windows.Automation, and no IUIAutomation -- while CUIAutomation8 is registered against uiautomationcore.dll, so UIA3 is reachable only through a compiled interop shim. Newly created files are owned by S-1-5-32-544 via TokenOwner rather than TokenUser, pre-answering half of the private-file ownership question. The locale is split: UI culture es-ES while the current culture is en-US, so OS strings are Spanish but .NET number and date formatting is not. * feat: probe windows private-file primitives for the 2.1 hardening design Answers the four questions sub-phase 2.1 rebuilds its private-file layer against, measured on the real OS with verbatim Win32 error codes. Three of the four contradict the framing this work was planned under. Rename: MoveFileEx cannot replace a target that has any open handle, and it reports ERROR_ACCESS_DENIED rather than the sharing violation the plan assumed -- FILE_SHARE_DELETE does not rescue it, because MoveFileEx issues ReplaceIfExists rather than POSIX-semantics rename. ReplaceFile is the operation that honors share-delete on the destination, but it is the one that refuses an open handle on the replacement source. The two APIs have opposite tolerances on opposite sides, so code treating error 32 as the sharing case misclassifies the most likely production failure. Ownership: new files take their owner from TokenOwner, never TokenUser, at both High and Medium integrity. Integrity level is not the variable -- the default follows admin-group membership, so an owner-only DACL check breaks for any admin account regardless of elevation. Locality: GetFileInformationByHandleEx(FileRemoteProtocolInfo) does distinguish local from remote, but signals local by failing with ERROR_INVALID_PARAMETER rather than succeeding with a local protocol value. An out-of-range info class returns that same error, so the code is ambiguous and a control call is required before reading it as a locality signal. ACL: neither leaf carries an explicit ACE. The restriction is authored at the nearest protected ancestor and merely propagated, so a leaf-only check sees the effective grants but cannot tell whether they are anchored or silently re-derivable, and cannot see FILE_DELETE_CHILD on the parent. * feat: probe uia3 com for events, caching, walkers, and pattern availability This is the stack the Rust adapter actually wraps, and it does not agree with the managed client. Notepad's edit control is a bare Pane with no patterns to System.Windows.Automation but a Document with Value, Text, Text2, Scroll and LegacyIAccessible to UIA3 COM, which names the responsible EDIT proxy in ProviderDescription. Twenty-six nodes versus three. Win32 client-side proxies are alive here; they are inert only for the managed client, so a managed census would have recorded false absences for patterns the adapter will see. The scratch fixture's earlier zero-pattern reading was a fixture artifact rather than a platform fact: a custom server-side provider suppresses both client-side proxies and WinForms' own providers. Both fixture modes are censused so the ledger states the mechanism instead of the symptom. CacheRequest measures 2.73x against the documented 3-5x, but the headline number hides the shape: building the cache makes the find phase slower, and the entire win lands in the read phase at roughly 299x. The claim is wrong in both directions. Handler teardown is safe with events in flight, costing 69ms against a backlog of 281. The hazard is elsewhere: hiding and showing a top-level window while handlers are registered poisons subsequent removal for up to 86 seconds, superlinearly in repetitions, and cheap callbacks do not avoid it. A watch implementation needs a teardown timeout rather than an assumption. Pattern ids are discovered from the OS at runtime rather than hardcoded, which caught that the annotation availability property is 30118. * feat: dump notepad, explorer, settings, and obsidian trees with real geometry Targets launch restored-but-not-activated, so bounds are real: 100/95.9/100/99.2 percent non-empty across the four dumps, with focus asserted identical before and after all seven walks. Settings resolves through ApplicationFrameWindow to its CoreWindow's SystemSettings pid, because the frame window belongs to ApplicationFrameHost and no predicate in either script keys on a display name. Minimizing a window does not shrink its tree -- the node count is identical at 3 -- but it degenerates geometry in two different shapes. Only the top-level window reports an empty rect; its descendants report real dimensions anchored at -32000, and every node still reports IsOffscreen as false. An occlusion gate testing emptiness or IsOffscreen alone would accept both as visible. Chromium activation is not a client-stack difference. The managed client takes Obsidian from 9 to 119 nodes across an 8 second settle with no flag, matching the COM shim's shape. What suppresses it is occlusion: an earlier revision that left other targets restored on top held the count at first contact for a 16 second instrumented hold across three runs. The mechanism is not isolated, so it is recorded as a probe-placement hazard rather than a product claim. The managed timing cross-check reproduces the COM stack's direction but not its magnitude, and on Notepad caching is a pessimization at 0.21x, because client-side providers run in the client process and the cache is pure overhead. * feat: census patterns and automation ids, and measure identity survival Identity survival is what sub-phase 2.5 designs its Windows RefEntry against, and the three properties behave nothing alike. AutomationId survives process restart on every stack measured, at 100 percent across WinForms, WPF, and real Win32. RuntimeId survives restart nowhere, at zero percent on all three. Path survives restart perfectly and is the first thing a content change breaks. Under content mutation the failure is worse than loss. Explorer keys list rows by row index, so after a folder changes, 29 AutomationId keys still resolve and 5 of them land on a different file. A ref keyed on AutomationId alone does not fail -- it succeeds against the wrong element. That silent-wrong-target count, not the survival percentage, is what makes stable text identity load-bearing. Measuring it needed a real refresh wait: a 4 second settle reported a wholly unchanged tree and would have recorded Explorer identity as perfectly stable, the exact inverse of what the window does once it refreshes at 20 seconds. Two stack facts. WPF automation-peer binding is a one-shot race -- a client that reads before the peer exists binds the generic HWND provider and never re-resolves, which a 30 second poll never recovers. And the managed and COM clients report different AutomationId values for the same window, so the divergence between stacks is in values as well as in visibility. The honest pattern divergence is two, not eleven: LegacyIAccessible and Text2 are the only patterns the managed stack structurally cannot name. The larger figure was an artifact of the WPF peer race and was retracted after the fix. * feat: probe uipi, dpi awareness, and chromium activation UIPI is measured across a boundary that actually exists. This box runs the built-in Administrator at High with Admin Approval Mode off, so RunAs yields High-vs-High and runas /trustlevel leaves the mandatory label untouched; the probe manufactures a real Medium token instead and refuses to record a verdict unless the worker reads back S-1-16-8192. Both arms run identical code against the same High Notepad, so the High arm is a control proving the injection works. UIA reads succeed from Medium; SendInput does not land, and the two arms return the identical success code -- only re-reading the target separates them. PostMessage is denied with error 5 while SendMessage WM_GETTEXT succeeds, so reads cross the boundary and writes do not. The 125 percent DPI measurement is not achievable here and the reason is measured rather than assumed: the VMware display carries no EDID and offers a single scale step. DisplayConfigSetDeviceInfo returns success and persists the registry value while effective DPI stays at 96, which is itself worth recording -- that API's success return is not evidence the scale changed. The non-zero delta defers to 2.4, which owns per-monitor scale. What does land for 2.1 is that PER_MONITOR_AWARE_V2 succeeds on 17763 with both arms verified at the awareness levels claimed. Chromium activation is graded, not guessed. The bundled version was read off this installation by scanning the shipped binary's user-agent string: Chromium 142, Electron 39.8.3. Both stacks reach the same settled tree with no flag, so the exposure claim holds and the plan's candidate contradiction is superseded. The flag is still not redundant: without it first contact is deterministically the pre-activation shell, and with it first contact is a race. Neither arm removes the need to settle before trusting a first snapshot. * feat: exercise interactions, input synthesis, and hit-testing raw Every interaction records pre-state, action, then an independent re-read -- never the call's return value. Toggle proves why: TogglePattern flips WPF ToggleState without raising Click, so the fixture's status sink stays silent while the element itself changed. Trusting the sink alone would have recorded a failure that did not happen. Posted keystrokes are not uniformly dead. WM_KEYDOWN posted to a Win32 edit control does register, because TranslateMessage runs in the target thread's own pump and synthesises the character regardless of how the message arrived; what the path cannot carry is modifier state. Against Chromium it does not register at all, with every call still returning success. Establishing that needed an idle control pass: the Chromium tree moved on its own between two reads, and without a quiet baseline that drift would have been filed as a keystroke landing. The astral-plane payload survives typing. SendInput forces a surrogate pair into two separate unicode events and the target reassembles it intact, read back through WM_GETTEXT so the check is independent of both the injection path and UIA. Notepad's edit control is unreachable from the managed client, and not merely unenumerated: TryGetCurrentPattern returns false for Text, Value and Scroll on the handle-resolved element, while the COM census sees the same window as a Document carrying all three. Hit-testing is the only sound visibility primitive of the three tested. The zero-size control is addressable by handle, enumerable by no walk, and returned by no point. A minimized window reports an empty rect at the top level while its descendants report real dimensions at the -32000 anchor, with IsOffscreen false throughout -- so neither emptiness nor IsOffscreen can gate occlusion, but ElementFromPoint correctly returns the occluder. * feat: add findings ledger, session evidence, and phases.md truth-sync Seventy-nine rows across all eleven evidence areas, each carrying the stack that produced it and whether it generalizes, so later sub-phases can tell an API-contract fact from an observation about this box and these app versions. Forty-three confirm the roadmap, five contradict it, twenty-six are new edges, and five defer to a named Phase 2 sub-phase. No row is unknown. Six in-place corrections land with it. The CacheRequest multiplier is replaced by its phase decomposition, because a single number is wrong in both directions. The private-file requirements now name ReplaceFile rather than MoveFileEx for a replace over an open destination, record that share-delete is necessary but never sufficient, and permit locality inference only behind a control call, since an out-of-range info class is indistinguishable from a local volume by error code. The Chromium guidance now requires a settle before concluding a tree is thin; its exposure claim was left standing because the probe confirmed it. The event-teardown mitigation now names the hazard that actually dominates removal cost. The completeness check is a script rather than a checklist, and it is negative-tested: blanking a scope, setting a verdict to unknown, or dropping a closure point each fail it loudly. It asserts the hunk index is bijective against the measured diff rather than against a number written in prose. * fix: wake a suspended settings instance instead of failing the tree dump The probe skipped its own launch when SystemSettings was already running, then waited the full window for a frame that a backgrounded UWP instance never presents, so a second harness run failed where a standalone run passed. That is the plan's recorded process-lifetime suspension risk arriving in practice. After eight seconds without a frame the probe now issues ms-settings: to wake a suspended instance, and the failure text names suspension as the cause rather than reporting only that no frame was found. * fix: close four teardown and gating defects found by review None of these changed a measurement; all four were failure paths that could report success. The COM shim killed Obsidian by process name with a raw Stop-Process inside an empty catch, so those pids never reached the ledger. A survivor there was invisible to both the probe's own teardown and the harness survivor check -- the one place in the corpus a leaked process could go entirely undetected. Obsidian pids are now registered and terminated through the confirming path. The tree dump registered its Settings pid only after frame resolution succeeded, so a resolution timeout threw first and leaked a process the probe had launched. Registration now happens as soon as the pid exists, and only when the probe launched it, since killing a pre-existing instance would be the operator's window. The modifier-release sweep injected without a foreground assertion, the only unbracketed SendInput site in the corpus. It now asserts and logs rather than aborting, because it runs during teardown. The input probe swallowed a failed teardown capture write, so the file that proves clipboard, cursor and process cleanliness could vanish while the probe still reported ok. It now fails loudly, and a surviving process downgrades the status too. Separately, a focus change is now classified rather than assumed hostile. A target the probe launched taking focus during its own startup is the target activating itself; Electron does this asynchronously when its renderer becomes ready, so under load it lands mid-dump and failed the run intermittently while passing standalone. Only a focus move to a window the probe never launched is interference. * chore: retrigger ci after an unrelated interaction-lease flake The macOS lib-test job failed on interaction_lease::tests::home_changes_do_not_change_the_physical_interaction_lock, which timed out acquiring the physical lock inside a 100 ms deadline at a contention count of 14. This branch changes no Rust, and the other 975 tests in that job passed, so the failure is runner contention rather than anything in the probe corpus. * chore: stop committing normalized capture twins Every capture was committed twice: once raw and once with pids, handles, timestamps and bounds canonicalized. The twin is produced by common.ps1 on every run and is byte-reproducible from the capture beside it -- verified across all thirty-seven of them -- so tracking both doubled the evidence diff for no information a re-runner cannot regenerate locally. The twins are still written on disk and -Compare still diffs them, so re-runnability is unchanged. |
||
|---|---|---|
| .githooks | ||
| .github | ||
| assets | ||
| benchmarks/locator-resolution | ||
| crates | ||
| docs | ||
| npm | ||
| probes/windows | ||
| scripts | ||
| skills | ||
| src | ||
| tests | ||
| .gitignore | ||
| .gitleaks.toml | ||
| .release-please-manifest.json | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| clippy.toml | ||
| CODE_OF_CONDUCT.md | ||
| CONCEPTS.md | ||
| CONTRIBUTING.md | ||
| deny.toml | ||
| LICENSE | ||
| README.md | ||
| release-please-config.json | ||
| rust-toolchain.toml | ||
| SECURITY.md | ||
| SUPPORT.md | ||
AGENT DESKTOP
OBSERVE. DECIDE. ACT.
agent-desktop is a native desktop automation CLI designed for AI agents, built with Rust. It gives structured access to any application through OS accessibility trees — no screenshots, no pixel matching, no browser required.
Architecture
Key Features
- Native Rust CLI: Fast, single binary, no runtime dependencies
- C-ABI cdylib (
libagent_desktop_ffi): Load once from Python / Swift / Go / Ruby / Node / C instead of forking the CLI per call - 58 command names, 54 operational commands: Observation, interaction, keyboard, mouse, notifications, clipboard, window management, session lifecycle, trace read/export, plus a bundled
skillsdoc loader. The four held-input names are reserved for a stateful daemon and fail closed in the stateless CLI. - Progressive skeleton traversal: 78–96% token reduction on dense apps via shallow overview + targeted drill-down
- Snapshot & refs: AI-optimized workflow using compact snapshot IDs and qualified element references (
@s8f3k2p9:e1,@s8f3k2p9:e2) - Headless-by-default interactions: Ref actions use accessibility APIs and block silent focus, cursor, keyboard, or pasteboard side effects
- Structured JSON output: Machine-readable responses with error codes and recovery hints
- Works with any app: Finder, Safari, System Settings, Xcode, Slack — anything with an accessibility tree
Installation
npm (recommended)
npm install -g agent-desktop # downloads prebuilt binary automatically
Or without installing:
npx agent-desktop snapshot --app Finder -i
From source
git clone https://github.com/lahfir/agent-desktop
cd agent-desktop
cargo build --release
cp target/release/agent-desktop /usr/local/bin/
Requires Rust 1.89+ and macOS 13.0+.
Permissions
macOS requires Accessibility permission. Screenshots also require Screen Recording permission, and the Notification Center opener requires Automation permission for System Events. Plain permission checks never prompt. Request missing permissions in a bounded isolated helper with:
agent-desktop permissions --request # request missing permissions in an isolated helper
Permission fields are explicit objects, for example:
{
"accessibility": { "state": "granted" },
"screen_recording": { "state": "denied", "suggestion": "Grant Screen Recording permission" },
"automation": { "state": "unknown" }
}
Automation reports granted, denied, or unknown; unknown means macOS would need to prompt or System Events could not be probed without prompting.
Language bindings (FFI)
Every GitHub Release ships a prebuilt C-ABI cdylib (libagent_desktop_ffi) for macOS, Linux, and Windows alongside the CLI tarballs. dlopen it and call the functions declared in agent_desktop.h for in-process calls instead of fork-exec per command.
import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_init(3) # verify ABI major (AD_ABI_VERSION_MAJOR) before any call
adapter = lib.ad_adapter_create()
# observe -> act: ad_snapshot -> parse a qualified ref -> ad_execute_by_ref ...
lib.ad_adapter_destroy(adapter)
Full consumer guide — entrypoints, ownership, threading, error-handling, build/link, release archives, and verification: skills/agent-desktop-ffi/.
Core Workflow for AI
For dense apps (Slack, VS Code, Notion), use progressive skeleton traversal to minimize token usage:
# 1. Shallow overview — depth-3 map, truncated containers show children_count
agent-desktop snapshot --skeleton --app Slack -i --compact
# Keep snapshot_id, for example s8f3k2p9
# 2. Drill into a region of interest (named containers get refs as drill targets)
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact
# 3. Act on an element found in the drill-down
agent-desktop click @e12 --snapshot s8f3k2p9
# 4. Re-drill the same region to verify the state change
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact
For simple apps, a full snapshot is fine:
agent-desktop snapshot --app Finder -i # get interactive elements with refs and snapshot_id
agent-desktop click @e3 --snapshot s8f3k2p9 # click a button by ref
agent-desktop type @e5 --snapshot s8f3k2p9 "quarterly report" # insert text into a field
agent-desktop press cmd+s # keyboard shortcut
agent-desktop snapshot -i # re-observe after UI changes
Agent loop: snapshot → decide → act → snapshot → decide → act → ...
Trace viewer (read back a session)
session_id=$(agent-desktop session start --screenshots | jq -r '.data.session_id')
export AGENT_DESKTOP_SESSION="$session_id"
agent-desktop snapshot --app Finder -i # work inside the explicit session scope
agent-desktop click @s8f3k2p9:e5
agent-desktop trace show --limit 500 # bounded JSON timeline for agents
agent-desktop trace export --out run.html # single-file HTML viewer (works from file://)
trace show merges all segment files deterministically and requires no permissions. trace export embeds the timeline plus screenshots as base64 in one static HTML file. Without --out, the HTML is written to the session directory (~/.agent-desktop/sessions/<id>/trace-<id>.html), not the current directory; --out overrides the path. Treat exported HTML like a screenshot when artifacts: full was enabled.
Shared sessions for multi-agent workflows
Run session start once per agent run to create a trace-enabled session (manifest trace: on by default), then pass the returned ID with global --session <id> or AGENT_DESKTOP_SESSION=<id>. Commands in that explicit scope get automatic JSONL segments under ~/.agent-desktop/sessions/<id>/trace/ and share the session's latest-snapshot namespace — no --trace on every call.
For concurrent independent agents, set AGENT_DESKTOP_SESSION=<id> per process. When multiple agents share one session ID, each agent should act on the qualified refs from its own snapshot call rather than assuming the namespace's latest snapshot is unchanged.
Bare --session <id> without a manifest (no session start) still scopes the snapshot namespace only and writes no trace files. Snapshot IDs resolve only inside the selected session namespace; they never trigger a cross-session search.
agent-desktop session start --name release-fix # note data.session_id
export AGENT_DESKTOP_SESSION=<session_id>
agent-desktop snapshot --app Xcode -i --compact # uses selected session + tracing
agent-desktop wait --element @s8f3k2p9:e9 --predicate actionable --timeout 5000
agent-desktop click @s8f3k2p9:e9
agent-desktop click @e9 --snapshot s2 # legacy bare ref, explicitly pinned
agent-desktop session end "$AGENT_DESKTOP_SESSION"
agent-desktop session gc
Commands
Observation
agent-desktop snapshot --app Safari -i # accessibility tree with refs
agent-desktop snapshot --surface menu # capture open menu
agent-desktop screenshot --app Finder # PNG screenshot
agent-desktop find --role button --app TextEdit # search by role, name, value, text
agent-desktop get @e3 --snapshot s8f3k2p9 --property value # read element property
agent-desktop is @e7 --snapshot s8f3k2p9 --property checked # check boolean state
agent-desktop list-surfaces --app Notes # list menus, sheets, popovers, alerts
get and is resolve the ref once, prefer live platform reads when available, and fall back only when that live read is unsupported by the adapter.
Interaction
agent-desktop click @s8f3k2p9:e3 # strict headless AX click
agent-desktop --headed click @s8f3k2p9:e3 # physical click, focus/cursor allowed
agent-desktop --headed double-click @s8f3k2p9:e3 # physical double-click
agent-desktop --headed triple-click @s8f3k2p9:e3 # physical triple-click
agent-desktop right-click @s8f3k2p9:e3 # open context menu; inspect effect before retrying
agent-desktop type @s8f3k2p9:e5 "hello world" # insert text into element
agent-desktop set-value @s8f3k2p9:e5 "new value" # set value directly via AX
agent-desktop clear @s8f3k2p9:e5 # clear element value
agent-desktop focus @s8f3k2p9:e5 # set keyboard focus
agent-desktop select @s8f3k2p9:e9 "Option B" # select verified dropdown/list option
agent-desktop toggle @s8f3k2p9:e12 # flip checkbox or switch
agent-desktop check @s8f3k2p9:e12 # idempotent check
agent-desktop uncheck @s8f3k2p9:e12 # idempotent uncheck
agent-desktop expand @s8f3k2p9:e15 # expand disclosure/tree item
agent-desktop collapse @s8f3k2p9:e15 # collapse disclosure/tree item
agent-desktop scroll @s8f3k2p9:e1 --direction down --amount 3 # strict headless AX scroll
agent-desktop scroll-to @s8f3k2p9:e20 # scroll element into view
(macOS, Phase 1) Default ref actions are strict headless semantic operations. In headed mode, core focuses the exact ref window before dispatch; pointer commands additionally require a verified target point, while the adapter owns physical delivery.
click,right-click,type,clear, andscrollare physical-first; double/triple-click, hover, and drag are physical-only; expand/collapse and other semantic commands remain semantic. Raw coordinates never imply a target window and therefore never steal focus. Seeskills/agent-desktop/references/commands-interaction.md.
Keyboard
agent-desktop press cmd+s # key combo
agent-desktop press cmd+shift+z # multi-modifier
agent-desktop press escape # single key
key-down and key-up are reserved command names and return ACTION_NOT_SUPPORTED until a stateful daemon can own the held-key lifetime.
Mouse
agent-desktop --headed hover @s8f3k2p9:e3 # move cursor to element
agent-desktop --headed hover --xy 500,300 # move cursor to coordinates
agent-desktop --headed drag --from @s8f3k2p9:e3 --to @s8f3k2p9:e8 # drag between elements
agent-desktop --headed drag --from-xy 100,200 --to-xy 400,200 # drag between coordinates
agent-desktop --headed mouse-click --xy 500,300 # click at coordinates
mouse-down and mouse-up are likewise reserved; use the atomic mouse-click or drag commands.
App & Window Management
agent-desktop launch Safari # launch app by name
agent-desktop launch com.apple.Safari # launch by bundle ID
agent-desktop close-app Safari # quit app
agent-desktop close-app Safari --force # force quit (SIGTERM, then SIGKILL if needed)
agent-desktop list-apps # list running GUI apps
agent-desktop list-windows # list visible windows
agent-desktop list-windows --app Finder # windows for specific app
agent-desktop focus-window --window-id w-4521 # bring exact window to front
agent-desktop resize-window --window-id w-4521 --width 800 --height 600
agent-desktop move-window --window-id w-4521 --x 100 --y 100
agent-desktop minimize --window-id w-4521
agent-desktop maximize --window-id w-4521
agent-desktop restore --window-id w-4521
Notifications (macOS only)
agent-desktop --headed list-notifications # open Notification Center if needed, then list
agent-desktop --headed list-notifications --app "Slack" # filter by app
agent-desktop --headed list-notifications --text "deploy" --limit 5 # filter by text
agent-desktop --headed dismiss-notification 1 --expected-app "Slack" --expected-title "Deploy complete"
agent-desktop --headed dismiss-all-notifications # dismiss all
agent-desktop --headed dismiss-all-notifications --app "Slack" # dismiss all from app
agent-desktop --headed notification-action 1 "Reply" --expected-app "Slack" --expected-title "Deploy complete"
Single-notification mutations require an app or title fingerprint from the
same listing. Every mutation requires --headed because it opens and focuses
Notification Center. Headless listing can only observe an already-open center;
headed listing may open it and restore the prior frontmost app afterward.
Clipboard
agent-desktop clipboard-get # read clipboard text
agent-desktop clipboard-set "copied" # write to clipboard
agent-desktop clipboard-clear # clear clipboard
Wait
agent-desktop wait 500 # sleep 500ms
agent-desktop wait --element @s8f3k2p9:e3 --timeout 5000 # wait for element
agent-desktop wait --element @s8f3k2p9:e3 --predicate actionable # wait until safe to act
agent-desktop wait --element @s8f3k2p9:e5 --predicate value --value ready
agent-desktop wait --window "Save" --timeout 10000 # wait for window
agent-desktop wait --text "Loading complete" --app Safari # wait for text
agent-desktop wait --text "Done" --count 1 --app Xcode # wait for exact match count
agent-desktop wait --notification --text "Build Succeeded" # wait for new matching notification
agent-desktop wait --menu --timeout 3000 # wait for menu
Batch
agent-desktop batch '[
{"command": "click", "args": {"ref_id": "@e2", "snapshot": "<snapshot_id>"}},
{"command": "type", "args": {"ref_id": "@e5", "snapshot": "<snapshot_id>", "text": "hello"}},
{"command": "press", "args": {"combo": "return"}}
]' --stop-on-error
agent-desktop --session run-a batch '[
{"command": "snapshot", "args": {"app": "Finder", "interactive_only": true}},
{"command": "status", "session": "run-b", "args": {}}
]'
System
agent-desktop session start [--name LABEL] [--no-trace] # create session; pass returned ID explicitly
agent-desktop session end [id]
agent-desktop session list
agent-desktop session gc [--older-than SECS] [--ended]
agent-desktop status # platform, permissions, session_id, tracing, latest snapshot
agent-desktop permissions # check accessibility/screen-recording/automation
agent-desktop permissions --request # request in the bounded isolated helper
agent-desktop version # version string
agent-desktop skills get desktop --full # bundled agent guidance
Snapshot Options
agent-desktop snapshot [OPTIONS]
| Flag | Default | Description |
|---|---|---|
--app <NAME> |
focused app | Filter to a specific application |
--window-id <ID> |
- | Filter to a specific window |
-i / --interactive-only |
off | Only include interactive elements |
--compact |
off | Omit empty structural nodes |
--include-bounds |
off | Include pixel bounds (x, y, width, height) |
--max-depth <N> |
10 | Maximum tree depth |
--skeleton |
off | Shallow 3-level overview; truncated containers show children_count and get refs as drill targets |
--root <REF> |
- | Start traversal from this ref; merges into existing refmap with scoped invalidation |
--snapshot <snapshot_id> |
latest | Snapshot ID to use when resolving --root |
--surface <TYPE> |
window | window, focused, menu, menubar, sheet, popover, alert |
JSON Output
See the versioned JSON envelope, error-code, and exit-code contract.
Ref System
snapshot assigns local positions in depth-first order and emits qualified refs such as @s8f3k2p9:e1, @s8f3k2p9:e2, and @s8f3k2p9:e3. A qualified ref embeds the exact snapshot ID and needs no separate --snapshot. Legacy bare refs such as @e3 remain accepted only with an explicit --snapshot s8f3k2p9. Snapshot lookup stays inside the selected session namespace.
Interactive roles that receive refs: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell, radiobutton, incrementor, menubutton, switch, colorwell, dockitem.
Static elements (labels, groups, containers) appear in the tree for context but have no ref.
Reliability contract:
session startcreates and returns a manifest-gated session with automatic trace segments. It does not activate later processes. Activation resolves explicit--sessionfirst, thenAGENT_DESKTOP_SESSION; otherwise the command uses the global, non-session namespace.- Bare
--session <id>without a manifest scopes snapshots only — no surprise trace files for existing callers. - Snapshot lookup is confined to the selected namespace. A session-owned snapshot requires the same explicit
--sessionorAGENT_DESKTOP_SESSIONscope. - Ref actions re-identify targets at action time: a moved unique target can proceed, while missing or changed stable identity returns
STALE_REF. - Mutable value text is not treated as stable identity, so text fields and timers can keep resolving when the saved window, path, role, and bounds evidence still identify the same element.
- Multiple plausible targets return
AMBIGUOUS_TARGETinstead of choosing arbitrarily. - Actions run an actionability preflight before dispatch: visibility, stability, enabled state, supported action, policy, and editability.
wait --element @s8f3k2p9:e3 --predicate actionablepolls until the target can be acted on.- With an active trace-enabled session, JSONL segments land under
sessions/<id>/trace/<pid>-*.jsonlautomatically.--trace <path>overrides to one file;--trace-strictfails on setup and pre-action writes (post-action traces are best-effort).
Stale ref recovery:
snapshot → act → STALE_REF or AMBIGUOUS_TARGET? → wait/snapshot again → retry with the new ref
Platform Support
| macOS | Windows | Linux | |
|---|---|---|---|
| Accessibility tree | Yes | Planned | Planned |
| Click / type / keyboard | Yes | Planned | Planned |
| Mouse input | Yes | Planned | Planned |
| Screenshot | Yes | Planned | Planned |
| Clipboard | Yes | Planned | Planned |
| App & window management | Yes | Planned | Planned |
| Notifications | Yes | Planned | Planned |
Development
cargo build # debug build
cargo build --release # optimized (<15MB)
cargo test --lib --workspace # run tests
cargo clippy --all-targets -- -D warnings # lint (must pass with zero warnings)
FAQ
See the complete FAQ for architecture, platform support, installation, refs, licensing, and support links.
License
Apache-2.0