mirror of
https://github.com/lahfir/agent-desktop.git
synced 2026-07-26 17:12:15 +00:00
No description
8de8f904db
* build: make cbindgen regenerate the C ABI static-assert guards
Add [const] allow_static_const = false to cbindgen.toml so every
pub const AD_*_SIZE emits as a #define macro (a C constant expression)
rather than static const (which is not valid inside _Static_assert).
Add a cbindgen.toml trailer with all 19 C11 ABI guards (6 sizeof, 6
_Alignof, 7 offsetof). Each sizeof guard references the corresponding
AD_*_SIZE macro so the size literal lives in exactly one place: the
Rust source. The Rust source already has compile-time asserts tying each
const to its struct, so the chain is: Rust const → #define macro →
_Static_assert → C compile gate.
Rename pub const MAX_C_STRING_BYTES to AD_MAX_STRING_BYTES so the
public ABI name matches the header macro (#define AD_MAX_STRING_BYTES)
and the internal const name no longer leaks. Update all call sites and
the describe() error string.
Pin cbindgen 0.29.4 in scripts/update-ffi-header.sh so regen is
reproducible.
* test: stabilize flaky wait-resolution test under CI load
element_wait_retries_transient_ambiguous_resolution and its sibling used a
250ms budget that the happy path clears instantly but a loaded CI runner
(full --lib --workspace in parallel) can exceed before the transient
AmbiguousTarget retry resolves — an intermittent red CI unrelated to any
code change. Raise the ceiling to 2000ms; the success path still returns on
the second resolve, so the test stays fast while becoming load-robust.
* refactor: enforce cbindgen version in update-ffi-header.sh
The script documented 0.29.4 as required but accepted any installed
version silently. A mismatched cbindgen can produce a semantically
different header with no diagnostic. Fail fast with a clear install
command if the version does not match exactly.
* feat: add ad_abi_version and ad_init ABI handshake
* refactor: address review on ad_abi_version
* feat: add ad_version FFI entrypoint
* refactor: move guard_non_null inside trap_panic for convention parity
All other FFI entrypoints validate input pointers as the first statement
inside trap_panic(|| unsafe { … }). ad_version placed the guard outside
the closure; align it with the uniform convention throughout the crate.
* feat: add session-scoped FFI adapter constructor
* refactor: address review on session context
* feat: add ad_set_log_callback with tracing layer
* refactor: address review on log callback
- Tighten core visibility: revert `pub mod trace` to `pub(crate) mod trace`,
re-export only `sanitize_trace_value` at the crate root so the FFI crate
imports it without exposing `TraceConfig` and internal helpers publicly
- Add per-thread re-entrancy guard (`IN_CALLBACK` + `CallbackGuard` RAII
drop) to `on_event`: a consumer callback that emits `tracing` events now
silently drops the recursive invocation instead of risking a stack overflow
that `catch_unwind` cannot stop
- Update module doc and `ad_set_log_callback` `///` doc to document the
re-entrancy protection; regenerate committed FFI header via cbindgen
* feat: add ad_snapshot with refmap pipeline
* refactor: address review on ad_snapshot
Promote app_error_to_adapter to pub(crate) in commands/mod.rs so future
command files share one conversion instead of copying the match per file.
Document the tri-state *out contract (null on arg/infra errors, populated
JSON envelope on command-level errors) in both the Rust doc comment and
the C header.
* feat: add ad_status FFI entrypoint
* refactor: address review on ad_status
* feat: add ad_wait with size-pinned AdWaitArgs
* refactor: DRY ad_wait field decoding
* fix: align command_context error return with errno-style last-error invariant
The context failure arm in ad_wait hardcoded ErrInternal while
set_last_error stored the actual mapped code; callers relying on
ad_last_error_code() == return value would see a mismatch. Mirror the
execute() Err arm and return last_error_code() instead.
* feat: add ad_execute_by_ref with strict resolution
* refactor: address review on ad_execute_by_ref
- extract app_error_to_adapter to error.rs pub(crate), removing
duplicate copies from snapshot.rs and execute_by_ref.rs
- replace 24-line tri-state ref_id decode with required_adapter_string
- use last_error_code() on validate_ref_id failure path for consistency
- fix *out dual-channel doc (null on guard/decode failure, non-null
JSON envelope on command-level errors) in both Rust doc and C header
- rename misleading test: stale_ref_returns_error → returns_error_envelope
(missing refmap surfaces SNAPSHOT_NOT_FOUND, not STALE_REF)
* docs: restore ABI header comments lost in cbindgen regen
Add /// docs on the Rust source so cbindgen re-emits the privacy note on
ad_last_error_details, the AdResult forward-compat note, and the behavioral
descriptions for the ad_execute_action* family.
* refactor: dedup ad_wait error conversion via shared helper
Replace wait.rs's local app_error_to_adapter_error with the shared
commands::app_error_to_adapter, giving one canonical AppError->AdapterError
conversion across every ffi command.
* fix: harden ABI assert trailer against double-include, document panic guard
Add a one-shot #ifndef AGENT_DESKTOP_ABI_ASSERTS guard around the
_Static_assert trailer in cbindgen.toml so re-including agent_desktop.h
in a single translation unit is unambiguously safe in all C standards
(C11 already allows repeated file-scope _Static_assert, but the guard
makes the property explicit and unconditional).
Add c_header_double_include.rs regression test that includes the committed
header twice via the system cc and asserts the compile succeeds, mirroring
the existing c_header_compile.rs harness pattern.
For Finding #14 (panic=abort guard): empirical probe confirms that Cargo
always sets CARGO_CFG_PANIC="unwind" in the build script for this crate
regardless of the active profile's panic setting — because the crate
declares both cdylib and rlib crate-types. A build-time check would be
silently inert. Document the invariant and the infeasibility of the
CARGO_CFG_PANIC guard in build.rs module docs instead; the release-ffi
profile enforces unwind at the profile level.
* fix(ffi): surface foreign-subscriber conflict, soften ad_init wording, direct agents to ad_snapshot
Finding #15: replace discarded try_init result with OnceLock<bool> so the first
ad_set_log_callback call with a non-null callback returns ErrInternal when a
foreign global subscriber already owns the process, rather than silently no-oping.
Re-registration and NULL-unregister paths are unaffected. Adds three unit tests
covering the new routing logic.
Finding #8: soften ad_init and AD_ABI_VERSION_MAJOR rustdoc from mandatory
pre-call requirement to recommended ABI-compatibility check; behaviour unchanged.
Finding #12: rewrite ad_get_tree rustdoc to direct observe-act agents to
ad_snapshot (refs, refmap, JSON envelope) and reserve ad_get_tree for ref-less
raw-tree consumers; removes stale "invoke the CLI" guidance.
* refactor: unify FFI command envelope serialisation and fix wait/status error contract
Extract write_command_envelope into commands/envelope_out.rs, eliminating
the ~35-line guard→serialize→string_to_c block duplicated across all five
command modules (snapshot, execute_by_ref, wait, status, version).
ad_wait and ad_status now write the error JSON envelope into *out on
command-level failures (TIMEOUT, ELEMENT_NOT_FOUND, etc.), matching the
behaviour already present in ad_snapshot and ad_execute_by_ref. Guard and
infrastructure rejections (null adapter/out/args, off-main-thread, invalid
UTF-8) continue to leave *out null — the infra/command boundary is
preserved exactly.
Add ad_wait_command_error_writes_error_envelope_into_out to verify the
unified contract: a zero-timeout element wait that cannot be satisfied must
produce an ok:false envelope in *out, not a null pointer.
* test: add unit tests to write_command_envelope proving #3 contract
* build: regenerate FFI header after round-1 review fixes
* docs(ffi): document ownership/lifetime contracts on adapter destroy and wait
Add Safety note to ad_adapter_destroy that callers must not destroy the
handle while any call on it is in flight on another thread — concurrent
destroy + in-flight call is use-after-free (confirmed reachable: destroy
is main-thread-exempt while ad_wait holds &*adapter through its blocking
wait::execute loop).
Add blocking-duration and adapter-lifetime note to ad_wait.
Extend ad_snapshot's existing partial note to cover explicit window
targeting (window_id, not yet ABI-exposed) alongside skeleton/drill-down,
and add CLI guidance for agents that need those features today.
* refactor(ffi): route ad_execute_by_ref through canonical core pipeline
Fix four coupled architectural findings in the FFI execute_by_ref path:
#2 (P1): Delete bespoke run_ref_action that bypassed CommandContext tracing
and hard-coded ref_id "<ffi>". Route ad_execute_by_ref through a new
commands::execute_by_ref::execute in core, which calls the same
execute_ref_action_with_context pipeline the CLI click/type/etc. commands
use — full trace, strict resolution, and actionability preflight.
#7 (P2): Add nullable snapshot_id: *const c_char parameter as the third
argument. Tri-state: null → latest snapshot (prior behaviour), valid UTF-8
→ pin that snapshot id, non-null invalid UTF-8 → ErrInvalidArgs. Passes
snapshot_str.as_deref() to RefArgs.snapshot_id mirroring CLI --snapshot
semantics. Update all 7 call sites in c_abi_lifecycle.rs and the extern
decl in tests/common/mod.rs (c_abi_actions.rs had no sites).
#10 (P2): Make core the single source of policy truth. Add
Action::base_interaction_policy() (delegates to may_use_focus_fallback —
TypeText + PressKey → focus_fallback, everything else → headless) and
InteractionPolicy::join(self, other) (bitwise-OR on the two capability
flags: elevate-only, never downgrade). FFI maps AdPolicyKind → core
InteractionPolicy and calls base.join(caller_ip); core::execute_by_ref
does the base+elevation. Delete the bespoke effective_action_policy fn and
its six duplicated unit tests from the FFI; repointed truth-table coverage
lives in the six FFI tests that delegate to core and the join unit tests in
interaction_policy.rs. Note: PressKey's base shifts headless→focus_fallback
vs the old FFI-only fn — intentional alignment with the full CLI table.
#6 (P1, light): Success and error paths already route through the shared
write_command_envelope helper (no reintroduction of bespoke serialization).
Add a /// note on ad_execute_by_ref documenting dispatch-before-serialize
ordering so callers understand the near-impossible ErrInternal-after-action
scenario without requiring heavy pre-validation machinery.
* build: regenerate FFI header after core-routing fixes
* test(ffi): split c_abi_lifecycle and add observe→act roundtrip tests
Finding #4: c_abi_lifecycle.rs was 1127 LOC, violating the 400-LOC cap.
Extracted focused modules each well under 400 lines:
- c_abi_init.rs (54 LOC) — ad_abi_version / ad_init
- c_abi_session.rs (88 LOC) — session adapter ctor
- c_abi_log_callback.rs(182 LOC) — log callback tests + statics
- c_abi_json_commands.rs(165 LOC) — ad_version / ad_status
- c_abi_snapshot.rs (131 LOC) — ad_snapshot guard + envelope tests
- c_abi_wait.rs (178 LOC) — ad_wait guard + error-envelope tests
- c_abi_execute_by_ref.rs(167 LOC) — ad_execute_by_ref guard tests
c_abi_lifecycle.rs shrunk to 176 LOC (null-safety + list lifecycle).
Finding #5: add snapshot→execute_by_ref roundtrip tests.
- stale_ref_returns_ok_false_error_envelope: always runs in CI; sets a
temp HOME (empty refmap), calls ad_execute_by_ref(@e1), asserts
ok:false + error.code when the command path executes, tolerates
ErrInternal/null-out for the macOS main-thread guard.
- snapshot_execute_by_ref_live_roundtrip: #[ignore]; needs AX permission,
a live app, and main-thread execution (E2E harness).
* fix(ffi): zero ad_wait *out before args guard; correct PressKey policy docs
Review of the review fixes:
- ad_wait now zeroes *out before the args null-check so a null-args
rejection honours the documented *out-zeroed contract (matches the
other command entrypoints).
- Corrected the PressKey policy doc wording: focus_fallback is the
shared base from Action::base_interaction_policy, not a claim of CLI
ref-PressKey parity (no CLI ref-PressKey action exists).
* chore: remove stray rust_out build artifact and gitignore it
* fix(ffi): thread session context through legacy ref-action path and correct doc steering
Add execute_entry_with_context to core so the FFI legacy path
(ad_execute_ref_action_with_policy) threads the adapter's real
CommandContext — built from the session id set at adapter creation —
instead of CommandContext::default(). The existing execute_entry
delegates to it with default() so all other callers are unchanged.
Rewrite the /// docs on all three native-handle / legacy-struct
entrypoints (ad_execute_action, ad_execute_action_with_policy,
ad_execute_ref_action_with_policy) to remove the false "CLI parity"
steering and describe each accurately as a low-level escape hatch with
verbatim-policy dispatch. Point observe→act callers at ad_execute_by_ref.
Add core tests confirming execute_entry_with_context succeeds, delegates
correctly, and emits trace events when a trace path is configured. Add FFI
test confirming the session-adapter path resolves to the same error class
as the no-session path (session id is wired into trace, not the error
surface).
* refactor: unify CLI ref-action policy via CommandContext::request_base
All 16 ref-action commands previously hardcoded InteractionPolicy literals
that duplicated the intent already encoded in Action::base_interaction_policy().
Add CommandContext::request_base(action) which reads the canonical base from
the action itself and delegates to request(), then replace every hardcoded
call site so there is a single policy source of truth for CLI and FFI.
Update the all_context_request_callers_are_policy_tested guard to search for
context.request_base( rather than context.request(.
Behavior is byte-identical: headless() and focus_fallback() literals were
exact mirrors of what base_interaction_policy() returns for each action.
* docs(ffi): fix Safety doc, add header preamble, document live-test deferral
Finding #9: correct the # Safety section on ad_execute_by_ref to distinguish
ref_id (non-null required; null is defined but rejected with ErrInvalidArgs)
from snapshot_id (null is meaningful — latest snapshot). The param tri-state
doc was already accurate; only the Safety section implied false symmetry.
Finding #10: add an agent-workflow orientation preamble to cbindgen.toml via
the after_includes key. The 13-line C block-comment lands after sys-includes
and before declarations (inside the include guard), covering the full
ad_init → ad_adapter_create → ad_snapshot → ad_execute_by_ref →
ad_free_string / ad_adapter_destroy loop with the macOS main-thread
requirement. Verified via local cbindgen 0.29.4 regeneration; header not
committed (orchestrator regenerates).
Findings #2/#8: expand module doc and live-roundtrip #[ignore] test doc to
precisely explain why the full observe→act loop cannot run under libtest on
macOS (off-main-thread scheduler + AX guard), how to run manually, and that
the full-loop CI proof is deferred to plan unit U9 / Phase B (Python ctypes
external-consumer harness). Strengthen the always-running error-envelope test
with an error.message assertion (guaranteed by the error contract) without
pinning error.code (which varies by load path; pinning would cause flakiness).
* build: regenerate FFI header after round-2 review fixes
* fix: correct FFI doc/test gaps found in PR #67 audit
#7: replace non-existent ad_click/ad_type_text references in ad_get_tree
rustdoc with the real entrypoint (ad_execute_by_ref + AdAction).
#6: extend cbindgen.toml preamble with snapshot_id round-trip (pass
data.snapshot_id back to pin snapshot vs NULL for latest), AdAction
construction (zero-init + set kind + kind-specific fields), and policy
semantics (0=keeps built-in base, 2=Headed allows cursor/focus fallbacks).
#2: add dispatch-before-serialize note to ad_snapshot matching the
existing note on ad_execute_by_ref — refmap is written before JSON
serialisation, so a serialisation failure leaves *out null + ErrInternal
while the refmap is already on disk.
#10: add out-of-range policy test in c_abi_execute_by_ref.rs that passes
policy=99 and asserts ErrInvalidArgs|ErrInternal + out null, with the
last-error assertion gated on ErrInvalidArgs to tolerate macOS off-main-
thread CI returning ErrInternal before the policy check is reached.
* fix: make error suggestions transport-neutral and populate retry_command
Finding #4: STALE_REF, AMBIGUOUS_TARGET, ELEMENT_NOT_FOUND, SNAPSHOT_NOT_FOUND,
NOTIFICATION_NOT_FOUND, and POLICY_DENIED suggestion strings previously referenced
CLI-only flags (--skeleton, --snapshot, --session, --headed) or bare CLI subcommand
syntax (snapshot, list-notifications), making them meaningless to FFI consumers.
Rewritten to name the operation neutrally with both CLI and FFI equivalents in
parentheses.
Finding #15: ErrorPayload::retry_command was always None; the with_retry builder
existed but was never called from from_app_error. Added retry_token_for_code() to
populate the field for mechanically retryable codes: STALE_REF and SNAPSHOT_NOT_FOUND
get "snapshot;execute_by_ref" (re-snapshot then re-execute by ref); POLICY_DENIED gets
"escalate_policy" (policy escalation, not a re-run of the same call). All other codes
remain None to avoid misleading auto-retry loops.
Tests updated: stale_ref_suggestion_mentions_skeleton renamed and tightened to assert
transport-neutral content; snapshot_ref_tests.rs skeleton assertion updated to match;
four new output.rs tests cover the populated tokens and the None cases.
* fix(trace): wire session_id into every JSONL record, improve FFI ref label, drop duplicate test
#9: thread session_id (Option<&str>) through TraceConfig::emit/emit_lazy/write_event and
insert it as a top-level field after the sanitized fields block. This makes the docs true
for everyone (CLI + FFI when both --session and --trace are provided). session_id is not
in SENSITIVE_KEYS and is intentionally inserted after sanitize_trace_value so it cannot
be shadowed or redacted. Omitted entirely when context has no session (no null key).
#8: replace the opaque "<ffi>" ref label in execute_entry_with_context with a
role/path-index-derived label (e.g. "<button/2/0/3>"). Uses only role and numeric path
indices — no content fields — so the label is safe to emit in the unredacted "ref"
trace key. Improves per-record correlation in multi-element FFI trace logs.
#14: delete execute_entry_delegates_to_entry_with_context from ref_action_tests.rs.
The test was byte-identical in assertions to failed_action_still_releases_resolved_handle
and added no new coverage. Removed rather than forced a weak rewrite.
Tests added/updated:
- context.rs: trace_injects_session_id_as_top_level_unredacted_field (session set),
trace_writes_jsonl_without_stdout_dependency extended with session_id absence assert
- ref_action_tests.rs: execute_entry_with_context_emits_trace_events extended to assert
session_id in every emitted record; trace_records_omit_session_id_when_context_has_none;
ref_label_from_entry_uses_role_and_path_indices
* fix(core): make stale-ref actionability suggestion transport-neutral; regen header
Mirrors the round-3 transport-neutral suggestion rewrite for the
live-staleness path in actionability, plus the regenerated FFI header
carrying the round-3 doc/preamble updates.
* refactor: dedup ffi policy mapping and error conversion helpers
Replace the duplicated AdPolicyKind→InteractionPolicy three-arm match in
execute_by_ref.rs with a canonical to_interaction_policy() method on
AdPolicyKind. Delegate action_request() in actions/execute.rs to the same
method, replacing its own redundant match. Replace the inline AppError→
AdapterError match in ad_execute_ref_action_with_policy with the shared
app_error_to_adapter helper already used everywhere else.
* refactor(ffi): apply ponytail/simplify cuts (then_some, trim core doc)
Use bool::then_some for ad_wait optional scalars; drop the FFI-history
sentence from the core execute_by_ref doc (the FFI doc retains it for C
binding authors). Behavior unchanged.
|
||
|---|---|---|
| .githooks | ||
| .github | ||
| assets | ||
| crates | ||
| docs | ||
| npm | ||
| scripts | ||
| skills | ||
| src | ||
| tests | ||
| .gitignore | ||
| .release-please-manifest.json | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| clippy.toml | ||
| CONCEPTS.md | ||
| deny.toml | ||
| LICENSE | ||
| README.md | ||
| release-please-config.json | ||
| rust-toolchain.toml | ||
| SECURITY.md | ||
AGENT DESKTOP
OBSERVE. DECIDE. ACT.
agent-desktop is a native desktop automation CLI designed for AI agents, built with Rust. It gives structured access to any application through OS accessibility trees — no screenshots, no pixel matching, no browser required.
Architecture
Key Features
- Native Rust CLI: Fast, single binary, no runtime dependencies
- C-ABI cdylib (
libagent_desktop_ffi): Load once from Python / Swift / Go / Ruby / Node / C instead of forking the CLI per call - 54 commands: Observation, interaction, keyboard, mouse, notifications, clipboard, window management, plus a bundled
skillsdoc loader - Progressive skeleton traversal: 78–96% token reduction on dense apps via shallow overview + targeted drill-down
- Snapshot & refs: AI-optimized workflow using compact snapshot IDs and deterministic element references (
@e1,@e2) - Headless-by-default interactions: Ref actions use accessibility APIs and block silent focus, cursor, keyboard, or pasteboard side effects
- Structured JSON output: Machine-readable responses with error codes and recovery hints
- Works with any app: Finder, Safari, System Settings, Xcode, Slack — anything with an accessibility tree
Installation
npm (recommended)
npm install -g agent-desktop # downloads prebuilt binary automatically
Or without installing:
npx agent-desktop snapshot --app Finder -i
From source
git clone https://github.com/lahfir/agent-desktop
cd agent-desktop
cargo build --release
cp target/release/agent-desktop /usr/local/bin/
Requires Rust 1.85+ and macOS 13.0+.
Permissions
macOS requires Accessibility permission. Screenshots also require Screen Recording permission. Grant them in System Settings > Privacy & Security by adding the app that launches agent-desktop, or:
agent-desktop permissions --request # trigger platform permission request path
Permission fields are explicit objects, for example:
{
"accessibility": { "state": "granted" },
"screen_recording": { "state": "denied", "suggestion": "Grant Screen Recording permission" },
"automation": { "state": "not_required" }
}
Language bindings (FFI)
Every GitHub Release ships a prebuilt C-ABI cdylib alongside the CLI tarballs. Hosts that need in-process calls (Python agents, Swift apps, Go services, Node tools, Ruby scripts, C/C++ code) dlopen the dylib and call the functions declared in agent_desktop.h — no fork-exec per command.
| Platform | Artifact |
|---|---|
| macOS arm64 | agent-desktop-ffi-v<ver>-aarch64-apple-darwin.tar.gz |
| macOS x86_64 | agent-desktop-ffi-v<ver>-x86_64-apple-darwin.tar.gz |
| Linux x86_64 (glibc) | agent-desktop-ffi-v<ver>-x86_64-unknown-linux-gnu.tar.gz |
| Linux arm64 (glibc) | agent-desktop-ffi-v<ver>-aarch64-unknown-linux-gnu.tar.gz |
| Windows x86_64 (MSVC) | agent-desktop-ffi-v<ver>-x86_64-pc-windows-msvc.zip |
Each archive contains lib/libagent_desktop_ffi.{dylib,so,dll}, include/agent_desktop.h, LICENSE, and a short README. Verify the download with the release's checksums.txt:
shasum -a 256 -c checksums.txt
gh attestation verify agent-desktop-ffi-v*.tar.gz --repo lahfir/agent-desktop # Sigstore provenance
Minimal Python round-trip:
import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_adapter_create.restype = ctypes.c_void_p
adapter = lib.ad_adapter_create()
# ... call ad_list_apps / ad_get_tree / ad_execute_action, see docs below
lib.ad_adapter_destroy(adapter)
Full consumer guide — error-handling contract, ownership rules, threading constraints, every entrypoint with Safety docs: skills/agent-desktop-ffi/.
Core Workflow for AI
For dense apps (Slack, VS Code, Notion), use progressive skeleton traversal to minimize token usage:
# 1. Shallow overview — depth-3 map, truncated containers show children_count
agent-desktop snapshot --skeleton --app Slack -i --compact
# Keep snapshot_id, for example s8f3k2p9
# 2. Drill into a region of interest (named containers get refs as drill targets)
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact
# 3. Act on an element found in the drill-down
agent-desktop click @e12 --snapshot s8f3k2p9
# 4. Re-drill the same region to verify the state change
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact
For simple apps, a full snapshot is fine:
agent-desktop snapshot --app Finder -i # get interactive elements with refs and snapshot_id
agent-desktop click @e3 --snapshot s8f3k2p9 # click a button by ref
agent-desktop type @e5 --snapshot s8f3k2p9 "quarterly report" # insert text into a field
agent-desktop press cmd+s # keyboard shortcut
agent-desktop snapshot -i # re-observe after UI changes
Agent loop: snapshot → decide → act → snapshot → decide → act → ...
Shared sessions for multi-agent workflows
Use the same --session <id> when multiple agents coordinate on one desktop task. A session owns a latest-snapshot pointer, not a security boundary. Each snapshot gets its own snapshot_id; pass --snapshot <id> when an agent must act on a specific observation. Explicit snapshot IDs can be used without repeating --session; keep --session when you omit --snapshot and want that session's latest snapshot.
flowchart LR
S["--session release-fix"] --> A["snapshot -> s1"]
S --> B["snapshot -> s2"]
A --> C["Agent A: click @e4 --snapshot s1"]
B --> D["Agent B: wait --element @e9 --predicate actionable"]
S --> E["latest_snapshot_id points at newest snapshot"]
C --> F["Explicit snapshot id works outside session too"]
agent-desktop --session release-fix snapshot --app Xcode -i --compact
agent-desktop --session release-fix wait --element @e9 --predicate actionable --timeout 5000
agent-desktop --session release-fix click @e9
agent-desktop click @e9 --snapshot s2
Commands
Observation
agent-desktop snapshot --app Safari -i # accessibility tree with refs
agent-desktop snapshot --surface menu # capture open menu
agent-desktop screenshot --app Finder # PNG screenshot
agent-desktop find --role button --app TextEdit # search by role, name, value, text
agent-desktop get @e3 --snapshot s8f3k2p9 --property value # read element property
agent-desktop is @e7 --snapshot s8f3k2p9 --property checked # check boolean state
agent-desktop list-surfaces --app Notes # list menus, sheets, popovers, alerts
get and is resolve the ref once, prefer live platform reads when available, and fall back only when that live read is unsupported by the adapter.
Interaction
agent-desktop click @e3 # semantic AX-first click
agent-desktop double-click @e3 # AXOpen; physical double-click uses --headed mouse-click --count 2
agent-desktop triple-click @e3 # POLICY_DENIED if physical input is disabled
agent-desktop right-click @e3 # open verified context menu
agent-desktop type @e5 "hello world" # insert text into element
agent-desktop set-value @e5 "new value" # set value directly via AX
agent-desktop clear @e5 # clear element value
agent-desktop focus @e5 # set keyboard focus
agent-desktop select @e9 "Option B" # select verified dropdown/list option
agent-desktop toggle @e12 # flip checkbox or switch
agent-desktop check @e12 # idempotent check
agent-desktop uncheck @e12 # idempotent uncheck
agent-desktop expand @e15 # expand disclosure/tree item
agent-desktop collapse @e15 # collapse disclosure/tree item
agent-desktop scroll @e1 --direction down --amount 3 # scroll (AX-first)
agent-desktop scroll-to @e20 # scroll element into view
(macOS, Phase 1) Pure cursor gestures have no accessibility equivalent, so
triple-click,hover, anddragare always physical;double-clickis headless viaAXOpenand only needs--headedfor gesture-only targets. Windows (UIA) and Linux (AT-SPI) adapters may expose different capabilities. Seeskills/agent-desktop/references/commands-interaction.md.
Keyboard
agent-desktop press cmd+s # key combo
agent-desktop press cmd+shift+z # multi-modifier
agent-desktop press escape # single key
agent-desktop key-down shift # hold key
agent-desktop key-up shift # release key
Mouse
agent-desktop --headed hover @e3 # move cursor to element
agent-desktop --headed hover --xy 500,300 # move cursor to coordinates
agent-desktop --headed drag --from @e3 --to @e8 # drag between elements
agent-desktop --headed drag --from-xy 100,200 --to-xy 400,200 # drag between coordinates
agent-desktop --headed mouse-click --xy 500,300 # click at coordinates
agent-desktop --headed mouse-down --xy 500,300 # press at coordinates
agent-desktop --headed mouse-up --xy 500,300 # release at coordinates
App & Window Management
agent-desktop launch Safari # launch app by name
agent-desktop launch com.apple.Safari # launch by bundle ID
agent-desktop close-app Safari # quit app
agent-desktop close-app Safari --force # force quit (SIGTERM, then SIGKILL if needed)
agent-desktop list-apps # list running GUI apps
agent-desktop list-windows # list visible windows
agent-desktop list-windows --app Finder # windows for specific app
agent-desktop focus-window w-4521 # bring window to front
agent-desktop resize-window w-4521 800 600 # resize
agent-desktop move-window w-4521 100 100 # move
agent-desktop minimize w-4521 # minimize
agent-desktop maximize w-4521 # maximize
agent-desktop restore w-4521 # restore
Notifications (macOS only)
agent-desktop list-notifications # list all notifications
agent-desktop list-notifications --app "Slack" # filter by app
agent-desktop list-notifications --text "deploy" --limit 5 # filter by text
agent-desktop dismiss-notification 1 # dismiss by index
agent-desktop dismiss-all-notifications # dismiss all
agent-desktop dismiss-all-notifications --app "Slack" # dismiss all from app
agent-desktop notification-action 1 --action "Reply" # click action button
Clipboard
agent-desktop clipboard-get # read clipboard text
agent-desktop clipboard-set "copied" # write to clipboard
agent-desktop clipboard-clear # clear clipboard
Wait
agent-desktop wait 500 # sleep 500ms
agent-desktop wait --element @e3 --timeout 5000 # wait for element
agent-desktop wait --element @e3 --predicate actionable # wait until safe to act
agent-desktop wait --element @e5 --predicate value --value ready
agent-desktop wait --window "Save" --timeout 10000 # wait for window
agent-desktop wait --text "Loading complete" --app Safari # wait for text
agent-desktop wait --text "Done" --count 1 --app Xcode # wait for exact match count
agent-desktop wait --notification --text "Build Succeeded" # wait for new matching notification
agent-desktop wait --menu --timeout 3000 # wait for menu
Batch
agent-desktop batch '[
{"command": "click", "args": {"ref_id": "@e2", "snapshot": "<snapshot_id>"}},
{"command": "type", "args": {"ref_id": "@e5", "snapshot": "<snapshot_id>", "text": "hello"}},
{"command": "press", "args": {"combo": "return"}}
]' --stop-on-error
agent-desktop --session run-a batch '[
{"command": "snapshot", "args": {"app": "Finder", "interactive_only": true}},
{"command": "status", "session": "run-b", "args": {}}
]'
System
agent-desktop status # platform, permission report, latest snapshot
agent-desktop permissions # check accessibility/screen-recording/automation
agent-desktop permissions --request # invoke platform request path
agent-desktop version # version string
Snapshot Options
agent-desktop snapshot [OPTIONS]
| Flag | Default | Description |
|---|---|---|
--app <NAME> |
focused app | Filter to a specific application |
--window-id <ID> |
- | Filter to a specific window |
-i / --interactive-only |
off | Only include interactive elements |
--compact |
off | Omit empty structural nodes |
--include-bounds |
off | Include pixel bounds (x, y, width, height) |
--max-depth <N> |
10 | Maximum tree depth |
--skeleton |
off | Shallow 3-level overview; truncated containers show children_count and get refs as drill targets |
--root <REF> |
- | Start traversal from this ref; merges into existing refmap with scoped invalidation |
--snapshot <snapshot_id> |
latest | Snapshot ID to use when resolving --root |
--surface <TYPE> |
window | window, focused, menu, menubar, sheet, popover, alert |
JSON Output
Every command returns structured JSON:
{
"version": "2.0",
"ok": true,
"command": "click",
"data": { "action": "click" }
}
Errors include machine-readable codes and recovery hints:
{
"version": "2.0",
"ok": false,
"command": "click",
"error": {
"code": "STALE_REF",
"message": "Element at @e7 no longer matches the last snapshot",
"suggestion": "Run 'snapshot' to refresh refs, then retry"
}
}
Error Codes
| Code | Meaning |
|---|---|
PERM_DENIED |
Accessibility permission not granted |
ELEMENT_NOT_FOUND |
No element matched the ref or query |
APP_NOT_FOUND |
Application not running or no windows |
STALE_REF |
Ref could not be re-identified in the live UI |
AMBIGUOUS_TARGET |
Ref recovery matched multiple plausible targets |
SNAPSHOT_NOT_FOUND |
Snapshot ID is missing or expired |
POLICY_DENIED |
Physical/headed path blocked by policy |
ACTION_FAILED |
The OS rejected the action |
PLATFORM_NOT_SUPPORTED |
Adapter method not implemented on this platform |
TIMEOUT |
Wait condition expired |
INVALID_ARGS |
Invalid argument values |
Exit Codes
0 success, 1 structured error (JSON on stdout), 2 argument parse error.
Ref System
snapshot assigns refs to interactive elements in depth-first order: @e1, @e2, @e3, etc. Refs are scoped to a compact snapshot_id such as s8f3k2p9. Commands can omit --snapshot to use the active session's latest snapshot pointer, but passing the ID is more deterministic in multi-step flows and does not require also passing --session.
Interactive roles that receive refs: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell, radiobutton, incrementor, menubutton, switch, colorwell, dockitem.
Static elements (labels, groups, containers) appear in the tree for context but have no ref.
Reliability contract:
--session <id>scopes the latest snapshot pointer to one caller or agent team; explicit--snapshot <id>resolves the saved snapshot directly.- Ref actions re-identify targets at action time: a moved unique target can proceed, while missing or changed stable identity returns
STALE_REF. - Mutable value text is not treated as stable identity, so text fields and timers can keep resolving when the saved window, path, role, and bounds evidence still identify the same element.
- Multiple plausible targets return
AMBIGUOUS_TARGETinstead of choosing arbitrarily. - Actions run an actionability preflight before dispatch: visibility, stability, enabled state, supported action, policy, and editability.
wait --element @e3 --predicate actionablepolls until the target can be acted on.--trace <path>appends JSONL diagnostics outside stdout;--trace-strictfails on trace setup and pre-action trace writes, while post-action success traces are best-effort after the desktop mutation has already happened.
Stale ref recovery:
snapshot → act → STALE_REF or AMBIGUOUS_TARGET? → wait/snapshot again → retry with the new ref
Platform Support
| macOS | Windows | Linux | |
|---|---|---|---|
| Accessibility tree | Yes | Planned | Planned |
| Click / type / keyboard | Yes | Planned | Planned |
| Mouse input | Yes | Planned | Planned |
| Screenshot | Yes | Planned | Planned |
| Clipboard | Yes | Planned | Planned |
| App & window management | Yes | Planned | Planned |
| Notifications | Yes | Planned | Planned |
Development
cargo build # debug build
cargo build --release # optimized (<15MB)
cargo test --lib --workspace # run tests
cargo clippy --all-targets -- -D warnings # lint (must pass with zero warnings)
FAQ
What is agent-desktop?
agent-desktop is a native desktop automation CLI for AI agents. It lets agents observe and control desktop apps through OS accessibility trees, using structured JSON instead of screenshots, pixel matching, or browser-only automation.
Does agent-desktop require screenshots or pixel matching?
No. The core workflow reads native accessibility trees and assigns refs to interactive elements. Screenshots are available as a separate command, but agents do not need screenshots or pixel matching to click buttons, type into fields, inspect menus, or navigate app windows.
How does agent-desktop work?
| Component | Function |
|---|---|
| Native Rust CLI | Fast, single binary, no runtime dependencies |
| C-ABI cdylib | Load once from Python, Swift, Go, Ruby, Node, or C instead of forking |
| 54 Commands | Observation, interaction, keyboard, mouse, notifications, clipboard, window management, and bundled skills docs |
| Snapshot & Refs | Compact snapshot IDs and deterministic element refs like @e1, @e2 |
| Structured JSON | Machine-readable responses with error codes and recovery hints |
What makes agent-desktop useful for AI agents?
| Feature | Benefit |
|---|---|
| Progressive Skeleton Traversal | 78–96% token reduction on dense apps |
| Headless-by-Default Actions | Ref actions use accessibility APIs and block unintended physical side effects |
| Snapshot Refs | Agents act on stable refs within a snapshot instead of guessing coordinates |
| Recovery Hints | Errors include machine-readable codes and suggestions for the next agent step |
| Cross-Language FFI | Python, Swift, Go, Ruby, Node, C, and C++ hosts can call the native library directly |
Which platforms are supported?
| Feature | macOS | Windows | Linux |
|---|---|---|---|
| Accessibility tree | Yes | Planned | Planned |
| Click/type/keyboard | Yes | Planned | Planned |
| Mouse input | Yes | Planned | Planned |
| Screenshot | Yes | Planned | Planned |
| Clipboard | Yes | Planned | Planned |
| App/window management | Yes | Planned | Planned |
| Notifications | Yes | Planned | Planned |
How do I install agent-desktop?
Install the CLI from npm:
npm install -g agent-desktop
agent-desktop snapshot --app Safari
Build the FFI library from source:
cargo build --release
# Outputs: libagent_desktop_ffi.dylib/.so/.dll
What is the ref system?
snapshot assigns refs to interactive elements in depth-first order: @e1, @e2, @e3, etc. Refs are scoped to a compact snapshot_id such as s8f3k2p9. Commands can omit --snapshot to use the active session's latest snapshot pointer, but explicit snapshot IDs are the deterministic path and do not require also passing --session.
Interactive roles that receive refs:
button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell, radiobutton, incrementor, menubutton, switch, colorwell, dockitem.
Stale ref recovery:
snapshot -> act -> STALE_REF? -> snapshot again -> retry
Is agent-desktop free and open source?
Yes. agent-desktop is Apache-2.0 licensed for personal and commercial use.
Where can I get help?
| Resource | Link |
|---|---|
| Repository | github.com/lahfir/agent-desktop |
| ClawHub Skill | clawhub.ai/lahfir/agent-desktop |
| skills.sh Listing | skills.sh/lahfir/agent-desktop/agent-desktop |
| npm Package | npmjs.com/package/agent-desktop |
| CI Status | GitHub Actions |
| Releases | GitHub Releases |
| Issues | GitHub Issues |
License
Apache-2.0