15 commits

Author	SHA1	Message	Date
Lahfir	3cffbd67f6	feat(ffi): ship C-ABI cdylib with review hardening and release pipeline (#26) ... Lands the full agent-desktop FFI layer with every PR #22 review finding resolved, modular refactor applied, todo-resolve batches (011 + 006) closed, and a production release pipeline that bundles the prebuilt cdylib for 5 target triples alongside the CLI on every GitHub Release. **FFI surface** - Panic-unwind cdylib boundary via `trap_panic` / `trap_panic_ptr` / `trap_panic_const_ptr` / `trap_panic_void` - Runtime main-thread enforcement on every macOS-sensitive entrypoint; TLS errno-style last-error lifetime - Every `#[repr(i32)]` field validated at the C boundary via `try_from_c_enum!` — arbitrary bit patterns return `ErrInvalidArgs` without UB - BFS flat-tree layout with `child_start` / `child_count`; iterative traversal - Opaque list handles (`AdAppList`, `AdWindowList`, `AdSurfaceList`, `AdNotificationList`); opaque `AdImageBuffer` with `_data` / `_size` / `_width` / `_height` / `_format` accessors - `AdNativeHandle` single-owner single-thread contract; zero-on-free makes double-call deterministic - Fail-closed UTF-8 for optional filter pointers (`try_c_to_string` tri-state) - `AdTreeOptions` fully honored in `ad_get_tree` (include_bounds / interactive_only / compact) - Verified notification action identity (`NotificationIdentity` fingerprint) — refuses to press if NC reordered between list and act **Release pipeline** - New `build-ffi` matrix in `.github/workflows/release.yml` producing `libagent_desktop_ffi.{dylib,so,dll}` tarballs for aarch64/x86_64-apple-darwin, x86_64/aarch64-unknown-linux-gnu, x86_64-pc-windows-msvc - macOS `install_name = @rpath/libagent_desktop_ffi.dylib` baked in by `build.rs` and CI-verified via `otool -D` - `actions/attest-build-provenance@v4.1.0` — keyless Sigstore provenance over every release artifact; `gh attestation verify` - `checksums.txt` covers both CLI and FFI assets; asset-count assertion bumped 3 → 8 - npm package stays CLI-only; Python/Swift/Go/Ruby/Node/C hosts pull dylib tarball directly from the Release - README gains a "Language bindings (FFI)" section with platform→artifact table **Docs** - `skills/agent-desktop-ffi/` — SKILL.md + build-and-link.md + ownership.md + threading.md + error-handling.md - `docs/solutions/best-practices/` — two new solution docs (deterministic build-artifact marker, identity fingerprint against OS reorder) **Verification** - `cargo clippy --all-targets -- -D warnings` — clean - `cargo test --lib --workspace` — 139 passed (5 suites) - `cargo test -p agent-desktop-ffi --tests` — 87 passed (4 suites, including new `c_header_compile` C-ABI harness) - macOS PR CI green on multiple runs; latest: 24561160455 - Python `ctypes.CDLL` round-trip from a freshly extracted FFI tarball confirmed; adapter lifecycle + `ad_list_apps` enumeration clean; `install_name` survives the tarball Closes #22 (superseded).	2026-04-17 04:00:43 -07:00
Lahfir	c17f2fae7a	feat: progressive skeleton traversal with ref-rooted drill-down (#20) ... * feat: add data model foundation for progressive skeleton traversal Add children_count, root_ref, skeleton fields to core types. Add get_subtree() to PlatformAdapter trait. Add remove_by_root_ref() and write-side size check to RefMap. No behavioral changes yet. * feat: implement progressive skeleton traversal with ref-rooted drill-down Add --skeleton and --root flags to snapshot command for token-efficient accessibility tree exploration. Skeleton mode clamps depth to 3 levels and annotates truncated containers with children_count, allowing AI agents to discover regions before drilling into them. Named containers at skeleton boundaries (via name or description) receive refs as drill-down targets. The --root flag starts traversal from a previous ref with scoped invalidation — only refs from that drill-down are replaced on re-drill. Key changes: - New ref_alloc.rs: shared ref helpers (INTERACTIVE_ROLES, actions_for_role, ref_entry_from_node, is_collapsible) extracted from snapshot.rs - New snapshot_ref.rs: drill-down logic with DrillDownConfig, scoped invalidation via root_ref tagging on RefEntry - macOS count_children() uses raw CFArrayGetCount without materializing AXElement wrappers for performance at skeleton boundaries - RefMap write-side size check prevents >1MB files - Skeleton anchors consider both name and description for Electron compat * fix: mention --skeleton in STALE_REF error suggestion * docs: document --skeleton and --root flags in skill reference * docs: update phases.md and CLAUDE.md for progressive skeleton traversal Add skeleton traversal as Phase 1 objective P1-O10. Document --skeleton and --root flags, get_subtree() trait method, new core modules, and platform-agnostic notes for Phase 2/3. Update risk mitigations and performance optimizations table. * docs: make progressive skeleton traversal the default agent workflow Update SKILL.md observe-act loop to skeleton-first approach. Add progressive skeleton traversal as the primary workflow pattern. Update anti-patterns, key principles, and command quick reference to lead with --skeleton + --root. Full snapshot remains documented as fallback for simple apps. * fix: preserve skeleton drill-down anchors * fix: preserve drill-down refs across skeleton re-snapshots Skeleton snapshots now load the existing refmap and remove only skeleton-level refs (root_ref: None), preserving drill-down refs accumulated via --root. Previously, build() always created a fresh RefMap, breaking the skeleton → drill → act → skeleton(verify) workflow by wiping all drill-down refs on the verify step. * fix: preserve drill-down depth for root snapshots * fix: verify AX action effect on Electron elements before trusting success Chromium's AX implementation returns kAXErrorSuccess for AXPress, AXConfirm, etc. but only toggles ARIA state without firing DOM event handlers. This caused the click chain to short-circuit on false positives, preventing CGClick from ever being reached. The fix detects web elements via AXWebArea ancestor walk, then verifies each AX action actually had a DOM effect by comparing focused element pointers before and after. When all AX methods produce no real effect, CGClick fires as the genuine last resort. Native elements are unaffected — the original verified_press behavior is preserved in a separate code path. * chore: ignore .context/ for local tooling artifacts * style: collapse web_action_had_effect signature to single line * test: cover RefMap save oversize rejection Extract the serialize+size-check step into a private helper so the 1MB write rejection path can be unit-tested without filesystem I/O. Also moves the size check above the directory creation in save() so an oversized refmap no longer creates ~/.agent-desktop on rejection. * test: assert stale-ref suggestion mentions --skeleton Locks in the Phase 4 polish requirement that STALE_REF errors guide agents back to a skeleton refresh, not just a plain snapshot. * test: cover skeleton-to-drill-down counter continuity Asserts that skeleton refs (@e1..@e10) survive a scoped invalidation of @e3, that drill-down refs allocated after the skeleton continue from @e11 instead of resetting, and that remove_by_root_ref drops only the drill-down children. * test: cover --root + --surface rejection at execute() boundary Adds a NoopAdapter that uses every PlatformAdapter trait default to exercise execute()'s validation guards without standing up a real adapter. Verifies that combining --root with a non-Window surface returns INVALID_ARGS, and that the Window surface does not trigger the guard. * test: add filesystem-redirected integration tests for run_from_ref Adds a thread-local HOME override + RAII HomeGuard so RefMap::save and RefMap::load can be exercised against an isolated temp directory without env-var racing across parallel tests. Also adds a StubAdapter that implements PlatformAdapter with a canned subtree response so the full run_from_ref drill-down flow can be unit-tested without standing up macOS AX state. Closes seven Phase 3 plan acceptance items in one pass: - save+load roundtrip with HOME override - oversize save rejection preserves previous file on disk - run_from_ref returns subtree and persists drill refs - stale root ref → STALE_REF with skeleton suggestion - re-drill replaces drill refs only, counter continues - multiple drill-downs from @e1 and @e2 coexist - empty subtree drill-down produces no new refs * test: add golden fixtures for skeleton output and drill-down refmap Adds two committed JSON fixtures under tests/fixtures/ plus matching unit tests that build the same input trees, run them through allocate_refs / run_from_ref, and assert the produced ref ids, parent-child layout, and root_ref tagging match the golden expectations. Locks in the JSON shape so future serialization changes have to be deliberate. * fix: bound build_subtree recursion on Electron wrapper chains Anonymous AXGroup/AXGenericElement wrappers do not advance the semantic depth counter (web-wrapper depth-skip), so a long chain of nested wrappers could recurse arbitrarily deep without ever hitting either max_depth or ABSOLUTE_MAX_DEPTH. The previous code checked `depth >= ABSOLUTE_MAX_DEPTH` against the semantic depth, which stayed at 0 throughout a wrapper chain. On pathological Electron trees this risked stack exhaustion before any cap engaged. Add a separate `raw_depth` parameter that always increments and use it for the absolute cap. Semantic `depth` still drives `max_depth` and the skeleton boundary so the wrapper-flattening behavior is preserved on normal trees. Also drops the long-unused `_include_bounds` parameter; bounds filtering happens in `allocate_refs` in core, not here. * chore: add pre-commit hook running fmt + clippy + tests Mirrors the CI quality gates (cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test --lib --workspace) so a failing commit never reaches origin. Skips automatically when no Rust/TOML files are staged. Bypass with --no-verify or SKIP_PRECOMMIT=1 when a non-Rust hotfix is genuinely needed. Hook lives under .githooks/ and is opt-in per clone: git config core.hooksPath .githooks Setup instructions added to CLAUDE.md. * fix: prevent orphaned drill-down refs leaking across skeleton refresh remove_skeleton_refs() previously dropped every entry whose root_ref was None and kept every entry whose root_ref was Some. After a series of skeleton refreshes that pattern leaked drill-down refs whose root anchor had already been removed: they could never be cleaned up by a future remove_by_root_ref(target) because target referred to a non-existent anchor, and they accumulated until the 1MB write guard fired. Restructure the function to keep only: - skeleton anchors that have at least one drill-down pointing at them - drill-down refs whose root anchor is among the kept anchors Unreferenced anchors are still dropped (the original intent), and orphaned drill-downs are dropped at the same time so the refmap can no longer accumulate dead entries. Updated existing test to assert the new keep-when-referenced semantics and added a regression test for the orphan-drilldown leak path. * fix: align resolve traversal with snapshot child-attribute set find_element_recursive previously walked AXChildren first then fell back to AXContents only, and resolve_element_name only derived its fallback label from AXChildren. Snapshot building, however, uses the full child_attributes() list (AXChildren, AXContents, AXChildrenInNavigationOrder) via copy_children. That asymmetry meant refs minted for containers exposed only through AXChildrenInNavigationOrder appeared in the snapshot output but produced false STALE_REF errors on drill-down or action commands because the resolver could not find them again. Route both call sites through child_attributes() so resolution sees the same children as snapshotting. * fix: bypass AXConfirm on web elements and add skeleton anchors to drill-down * fix: compare AX elements via CFEqual in web_action_had_effect AXUIElementCopyAttributeValue follows the CoreFoundation Create rule and returns a freshly-allocated CF object on every call, so raw pointer equality (`before.0 != after.0`) between two separate copies of AXFocusedUIElement was ALWAYS true even when the focused element was unchanged. That made Step 1 (AXPress) of activate_web_element appear successful every time and left Steps 2-4 of the escalation chain (AXConfirm bypass, child actions, CGClick fallback) dead code on web elements. Replace the pointer comparison with CFEqual, which compares accessibility element identity rather than heap addresses. * refactor: unify allocate_refs across snapshot and drill-down paths allocate_refs in snapshot.rs and allocate_refs_with_root in snapshot_ref.rs were near-exact copies that differed only in whether each allocated ref carried a root_ref tag. The duplication meant any bug fix or behavior change in one path risked silently drifting from the other (bot flagged this after the first round of fixes added even more duplicated skeleton-anchor logic to allocate_refs_with_root). Move the shared logic into ref_alloc.rs behind a new RefAllocConfig struct with Option<&str> for the root_ref_id, and delete the snapshot_ref.rs copy. Both snapshot::build and snapshot_ref::run_from_ref now route through the same function with their respective configs. append_surface_refs and the existing unit tests in both modules are updated to use the shared function + config. * chore: drop with_root from drill test names after allocator unification The snapshot_ref tests kept their original test_allocate_refs_with_root_* names after the allocator dedupe even though allocate_refs_with_root no longer exists. Rename them to test_drill_alloc_* so grep for allocate_refs_with_root returns zero matches and nobody assumes a second allocator path exists. * docs: compound DRY ref-allocator dedupe into knowledge base Adds docs/solutions/best-practices/deduplicate-ref-allocator-via-config-struct-2026-04-14.md documenting the root cause, guidance, consequences, trigger conditions, and before/after of the allocate_refs / allocate_refs_with_root duplication that landed on PR #20 and was unified in d06a6c2. Also: - gitignore allows docs/solutions/** (was caught by docs/* rule) - CLAUDE.md workspace tree surfaces docs/solutions/ so fresh agents discover the knowledge store - docs/phases.md drops two stale DrillDownConfig references (lines 62 and 226) — the symbol no longer exists in the codebase * refactor: drop unused root_ref field from TreeOptions TreeOptions.root_ref was populated from SnapshotArgs.root_ref by tree_options() and then never read anywhere. The actual routing lives in execute(): `if let Some(ref root) = args.root_ref` branches to snapshot_ref::run_from_ref(adapter, &opts, root), passing the root id as an explicit argument — opts.root_ref was never consulted by any caller. Delete the field from TreeOptions and its Default impl and stop populating it in tree_options(). SnapshotArgs.root_ref remains as the single source of truth for the drill-down target. * fix: suppress skeleton flag when --root is set tree_options() clamped depth based on skeleton && root_ref.is_none() but still forwarded skeleton: args.skeleton unconditionally. When a caller passed --skeleton --root @e3, opts.skeleton stayed true, so adapter.get_subtree on macOS built a truncated skeleton tree instead of the full drill-down subtree, AND ref_alloc::allocate_refs tagged skeleton-anchor refs with the drill-down root_ref — both unintended. Tie the skeleton flag to the same root_ref.is_none() guard the depth clamp already uses. A drill-down is always a full subtree now, never a skeleton view. Adds test_tree_options_suppresses_skeleton_for_drill_down to lock the behavior in and extends the existing clamp test to assert the flag still propagates on non-drill paths. * fix: detect web action effect via value + selected + focus change The previous focus-change-only heuristic in web_action_had_effect was wrong for non-Chromium AXWebArea elements (Safari WKWebView, Mail, Notes): AXPress on a checkbox toggles AXValue correctly but does NOT shift the focused UI element. web_action_had_effect returned false, the chain escalated to CGClick, and CGClick toggled the checkbox a second time — net zero visible change. The pointer-equality bug that ae78cbc replaced was masking this by always returning true; CFEqual exposed the real signal gap. Capture element state into a PreActionState struct (focused, value, selected) before any AX action runs, and consider the action to have had an effect if ANY of those three fields differs afterward. Element value covers WKWebView checkboxes, text fields, sliders. Selected covers tabs, radio buttons, and list items. Focused still covers Chromium's DOM-driven focus shifts. Triggers on whichever signal is most relevant to the element type without needing role-specific code. Also re-exports copy_bool_attr from crates/macos/src/tree for use by the chain-steps module. * fix: emit skeleton boundary when drill path is about to hit raw cap In skeleton mode, a chain of anonymous web wrappers (AXGroup / AXGenericElement with empty title and value) never advances the semantic `depth` counter, so `child_depth > max_depth` never fires no matter how deep the chain runs. The recursion then hits the `raw_depth >= ABSOLUTE_MAX_DEPTH` top-of-function guard and silently returns None, dropping the subtree without any children_count marker. Agents see truncated output with no indication that anything was dropped. Extend the at_skeleton_boundary condition to also fire when child_raw_depth is about to hit ABSOLUTE_MAX_DEPTH, so skeleton mode always emits a visible truncation node for deep wrapper chains instead of disappearing them. The old `raw_depth < ABSOLUTE_MAX_DEPTH` half of the guard was redundant — the top-of-function check already ensures the current frame has raw budget; the meaningful question is whether the CHILD frames will. * perf: compute is_in_webarea once per verified-press call try_focus_then_verified_confirm_or_press called is_in_webarea(el) to gate AXConfirm, then fell through to do_verified_press which called is_in_webarea(el) again on its first line. Each call walks up to 20 AX parents via per-element IPC (AXUIElementCopyAttributeValue on AXParent), so every web-area click through this path paid the cost twice. Extract a private dispatch_verified_press(el, caps, in_web) that takes the web-area flag as an input. do_verified_press keeps its existing (el, caps) signature for chain_defs compatibility and computes is_in_webarea once, then delegates. The focus-then-confirm path also computes it once and passes it through. The result is identical on both paths with one AX traversal instead of two. * fix: skeleton anchors in drill-downs must not inherit root_ref; restore AXBrowser AXContents fallback Skeleton anchors discovered while processing a drill-down subtree were being tagged with the drill root_ref, making them indistinguishable from regular drill entries. re-drills would delete those anchors via remove_by_root_ref, breaking sub-drilling. They now always carry root_ref=None so they survive re-drills as stable targets. AXBrowser child resolution dropped the AXContents fallback when child_attributes() was unified — the resolver now uses ["AXColumns","AXContents"] matching the original traversal order. * fix: suppress skeleton anchor creation in drill-down mode to prevent orphaned ref accumulation * fix: bounds-based resolver pruning and CGClick fallback on chain timeout Resolver was doing exhaustive DFS through entire AX tree (depth 50) to find a single element. For large documents like a dense spreadsheet this meant visiting tens of thousands of cells via AX IPC, causing multi-minute hangs before the click chain even started. Two changes: - find_element_recursive now prunes subtrees whose spatial bounds do not contain the target element's centre point, and aborts the whole search after five seconds. Numbers table -> button not inside -> entire table skipped. Resolution went from >1m49s to <50ms. - execute_chain now sets a 1s app-level AX messaging timeout so calls to children/parents fail fast instead of blocking for the system default 6s, and when the 10s chain deadline fires it attempts the CGClick step before returning failure so the coordinate fallback is always reached. * refactor: resolve 9 code-review todos for progressive skeleton traversal - skeleton refresh now starts from RefMap::new() (removes stale ref accumulation across refreshes) - drill-down snapshot resolves real WindowInfo via PID lookup instead of synthetic empty id - --root flag validates ref format before RefMap lookup (returns INVALID_ARGS not STALE_REF) - ABSOLUTE_MAX_DEPTH truncation emits boundary node with children_count instead of silently dropping - fix ref vs ref_id field name in commands-observation.md JSON examples - fix phases.md TreeOptions listing root_ref (it lives in SnapshotArgs, not TreeOptions) - add batch snapshot skeleton/root examples to commands-system.md - split snapshot.rs and snapshot_ref.rs test modules into separate _tests.rs files (143/69 LOC) - add integration tests for skeleton + drill-down workflow including invalid --root validation * docs: compound progressive snapshot review hardening * docs: tighten progressive snapshot compound write-up * docs: update README with progressive skeleton traversal and fix command count to 50 * docs: correct command count to 53, add Notifications section and platform notes --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-04-16 01:46:31 -07:00
Lahfir	4a300c8cb0	feat: implement --compact flag to collapse single-child unnamed nodes ... Activate the existing --compact CLI flag (previously a no-op) to reduce tree verbosity by collapsing pass-through container nodes that carry no semantic information (no ref, no name, no value, no description, no states, exactly one child). Slack with -i --compact: 264 → 214 nodes, ~5,967 → ~5,117 tokens (14% reduction). All 161 refs preserved. Finder unaffected (3% reduction). Handles Electron's empty-string-vs-None pattern using is_none_or.	2026-03-01 00:36:20 -08:00
Lahfir	8aa672c668	refactor: resolve code review findings from notification PR ... - fix(core): wait --notification uses index-diff detection instead of count, passes text filter, checks deadline before sleep - fix(core): dismiss-all-notifications uses single NC session with batch adapter method and reports individual failures - refactor(macos): extract dismiss_entry helper to eliminate duplicate dismiss logic between single and batch operations - refactor(macos): restrict nc_session visibility to pub(crate), use absolute paths for pgrep/osascript, add bounded wait with kill fallback - refactor(macos): single-pass is_notification_group check using matches! macro - refactor: extract notification CLI args and dispatch into separate files to keep cli_args.rs and dispatch.rs under 400 LOC - refactor: rename DismissAllNotificationsArgs to DismissAllNotificationsCliArgs for consistency - fix(core): remove unused timestamp field from NotificationInfo - fix: remove poll_interval_ms from wait command (hardcode 500ms) - docs: update phases.md with completed notification status and cross-platform reference notes	2026-02-27 12:42:46 -08:00
Lahfir	c5b05bab60	feat: add notification command types, adapter trait, and CLI wiring ... Core types (NotificationInfo, NotificationFilter), 3 adapter methods with not_supported defaults, NotificationNotFound error code, 4 command handlers, CLI args with index>=1 validation, dispatch arms, wait --notification extension, and batch_dispatch split to stay under 400 LOC.	2026-02-27 10:40:07 -08:00
Lahfir	e154cc0cc0	refactor: address code review findings ... - dispatch.rs: 460 → 392 LOC by removing duplicate command_name(), using cmd.name() from cli.rs, and moving cli_surface_to_core to Surface::to_core() in cli_args.rs - ElementCaps: trim from 8 to 3 used fields, remove has_action() and unused discovery calls (actions, role, has_children, pid, settable_value) - chain.rs: deduplicate ChainStep/ChainDef/ChainContext out of cfg gates - ax_helpers.rs: delete dead set_ax_string	2026-02-23 04:10:50 -08:00
Lahfir	c7316e8b51	feat: add structured verbose logging across all layers ... Adds tracing::debug! calls throughout command and adapter layers, activated via -v flag. Logs command dispatch, ref resolution, chain step execution, clipboard/keyboard/mouse synthesis, and system ops. Command layer logging (free for all platforms): - dispatch: command name - resolve: ref lookup with pid/role/name, match result - tree: snapshot app/window/ref_count Adapter layer logging (macOS, other platforms add their own): - chain: step-by-step [N/total] with success/skip for each - action: perform entry - clipboard/keyboard/mouse: operation details - system: app focus/launch/close, window ops, screenshots	2026-02-23 03:52:57 -08:00
Lahfir	a2319623b4	fix: add menubar surface, fix press --app crash and modifier mapping ... - Add --surface menubar to expose full menu bar hierarchy - Fix use-after-free in press_for_app_impl (ManuallyDrop) - Fix AX modifier bit mapping (Shift=1<<0, Alt=1<<1) - Improve alert detection to search all app windows	2026-02-21 00:21:40 -08:00
Lahfir	3a796023f0	style: apply cargo fmt to all files	2026-02-19 20:00:18 -08:00
Lahfir	eca04e8392	feat: add 19 new commands, AX-first rewrites, LOC compliance ... - New commands: check, uncheck, triple-click, scroll-to, clear, clipboard-clear, hover, drag, mouse-move/click/down/up, resize/move/minimize/maximize/restore-window, key-down, key-up - Rewrite input.rs to AX-first keyboard synthesis - Add mouse synthesis (CGEvent) and window ops (AX) adapter methods - Split dispatch.rs into dispatch + batch_dispatch for LOC compliance - Split actions.rs → action_extras.rs (scroll/select helpers) - Split app_ops.rs → key_dispatch.rs (press-for-app + key dispatch) - Split cli.rs → cli_args.rs (arg structs) - Improve is-check with applicability field, list-apps with wrapped shape - Enhance wait with --text/--menu/--menu-closed support	2026-02-19 18:17:15 -08:00
Lahfir	39178b2916	feat: surface-targeted snapshot, menu wait, list-surfaces command ... Add snapshot --surface flag (menu/sheet/popover/alert/focused/window) for direct O(1) AX attribute reads rather than full-tree traversal. Add wait --menu/--menu-closed for polling-based context-menu gate. Add list-surfaces command to enumerate open transient surfaces. Remove all inline // comments from macos crate per 400-LOC/no-comment rule.	2026-02-19 15:21:07 -08:00
Lahfir	1d98ab828c	fix: make all 30 commands work end-to-end on macOS ... - tree: window_element_for() starts traversal from correct AXWindow element (matching by title) instead of app root, fixing mixed-window trees and 'disabled group' noise in Electron apps - tree: AXUIElementCopyMultipleAttributeValues batch fetch (2.5x faster) - tree: copy_ax_array() CFRetains each element before CFArray drops, fixing the dangling-pointer bug that returned kAXErrorInvalidUIElement - roles: add AXApplication->application and AXSplitGroup->splitter - actions: CGEvent is now primary click mechanism (AXPress is best-effort first try); fixes clicks in Electron/web apps that don't support kAXPressAction - actions: DoubleClick uses MOUSE_EVENT_CLICK_STATE=2, RightClick uses RightMouseDown/Up, Scroll sets event.set_location(element_center) - screenshot: rewrite using screencapture CLI + CGWindowListCopyWindowInfo to find the largest CGWindowID for the app PID; resolves app/window_id to pid before calling adapter (was producing all-zero data placeholder) - screenshot: ScreenshotTarget::Window now carries pid (i32) instead of our hash string; commands/screenshot.rs resolves via list_windows - input: add 8ms inter-keystroke delay in synthesize_text so apps can process events (was truncating long strings) - snapshot: return WindowInfo in SnapshotResult; include app and window fields in JSON output per contract - find: switch to snapshot::run so found refs are persisted to refmap - dispatch: implement dispatch_batch_command covering all 29 commands (was stub returning 'not yet implemented' for every batch sub-command)	2026-02-19 12:51:37 -08:00
Lahfir	6dc567a4ae	fix: align error codes with spec (APP_NOT_FOUND, PERM_DENIED) and add -i shorthand	2026-02-19 11:49:17 -08:00
Lahfir	218503a7eb	fix: resolve all 47 code review findings from Phase 1 audit ... Security (P1): - Replace AppleScript app-name interpolation with PID-based scripts (001, 002) - Switch pkill -f regex to pkill -x exact-name match (003) - Validate launch_app id against path traversal (006) - Remove CFRetain+mem::forget; use ManuallyDrop correctly (005, 029) - Add cycle detection (visited set) to resolve_element (009) Correctness (P1/P2): - Implement read_bounds via kAXPositionAttribute+kAXSizeAttribute (023) - Fix Scroll action to use CGEventCreateScrollWheelEvent (027) - Propagate Expand/Collapse AX errors instead of discarding (042) - Store AXDescription separately from AXTitle (046) - Fix interactive_only to recurse into container children (043) - Pass window.pid to RefEntry instead of hardcoded 0 (019) - Add batch::parse_commands() for dispatch-layer batch execution (022) - Fix find command to use snapshot::build() — no RefMap overwrite (030) - Return non-interactive elements from find with ref:null (044) - Wire permissions --request to adapter.check_permissions (031) - Fix screenshot --app to resolve window ID (032) - Close-app protected process check uses exact match (014) - Wait unbounded sleep capped at 30s (012) - Fix wait double-lookup unwrap (038) - Fix press.rs unwrap, normalize modifier order (021, 038) - emit_json no longer silently discards write errors (041) Data / API: - WindowInfo.app serializes as "app_name" per spec (045) - Add value/states/bounds fields to RefEntry; populate from snapshot (024, 025) - Stable window IDs via FxHasher(pid+title) (028) - Use FxHasher for bounds_hash, replacing unstable DefaultHasher (020) - Flush refmap temp file before rename (039) - Add HOME fallback to USERPROFILE (015) Architecture: - Split click.rs → click, double_click, right_click per one-command-per-file (047) - Split clipboard.rs → clipboard_get, clipboard_set (047) - Consolidate RefArgs to helpers.rs (047) - Move focus/launch/close impl to app_ops.rs; adapter.rs 389→311 LOC (001) - Deduplicate ErrorCode::code() via ErrorCode::as_str() (034) - Add doc comment to Response struct explaining Phase 3 intent (035) - Add snapshot::build() for read-only tree access (030) - Remove duplicate ABSOLUTE_MAX_DEPTH from snapshot.rs (033)	2026-02-19 11:38:17 -08:00
Lahfir	a346f242c2	feat: Phase 1 foundation — workspace scaffold, core engine, macOS adapter, 31 commands ... Implements the complete agent-desktop Phase 1 specification: - Workspace: 5-crate layout (core, macos, windows/linux stubs, binary) - Core: AccessibilityNode, Action, ErrorCode, PlatformAdapter trait, RefMap with atomic writes, SnapshotEngine with depth-first ref allocation - macOS adapter: AXUIElement tree traversal, action execution, input synthesis via CGEvent, screenshot via CGWindowListCreateImage, clipboard via pbpaste/pbcopy, window listing and app management via osascript - 31 CLI subcommands via clap derive: snapshot, find, screenshot, get, is, click, double-click, right-click, type, set-value, focus, select, toggle, expand, collapse, scroll, press, launch, close-app, list-windows, list-apps, focus-window, clipboard-get/set, wait, status, permissions, version, batch - Windows/Linux: not-supported stubs ready for Phase 2 implementation - JSON output contract: {version,ok,command,data} envelope with structured error payloads including SCREAMING_SNAKE_CASE error codes - Ref system: @e{N} sequential refs for interactive elements, stored at ~/.agent-desktop/last_refmap.json with 0o600/0o700 permissions - CI: GitHub Actions macOS runner with dependency isolation check, clippy, unit tests, release build, and 15MB binary size gate	2026-02-19 10:44:38 -08:00