Creepso/agent-desktop
1
0
Fork 0
mirror of https://github.com/lahfir/agent-desktop.git synced 2026-08-09 00:17:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
19 commits 34 branches 31 tags 24 MiB
Commit graph

10 commits

Author SHA1 Message Date
Claude
a7a1adcf07
feat: add diff-snapshot command for tree-level change detection 
Enables agents to detect UI changes between snapshots without
re-parsing the entire tree. Saves each snapshot to disk and diffs
against the previous one, reporting added/removed/modified nodes.

https://claude.ai/code/session_01WCp4PmZcXWCo7XeCfHPm3K
 2026-02-21 02:26:24 +00:00
Lahfir
3a796023f0 style: apply cargo fmt to all files 2026-02-19 20:00:18 -08:00
Lahfir
eca04e8392 feat: add 19 new commands, AX-first rewrites, LOC compliance 
- New commands: check, uncheck, triple-click, scroll-to, clear,
  clipboard-clear, hover, drag, mouse-move/click/down/up,
  resize/move/minimize/maximize/restore-window, key-down, key-up
- Rewrite input.rs to AX-first keyboard synthesis
- Add mouse synthesis (CGEvent) and window ops (AX) adapter methods
- Split dispatch.rs into dispatch + batch_dispatch for LOC compliance
- Split actions.rs → action_extras.rs (scroll/select helpers)
- Split app_ops.rs → key_dispatch.rs (press-for-app + key dispatch)
- Split cli.rs → cli_args.rs (arg structs)
- Improve is-check with applicability field, list-apps with wrapped shape
- Enhance wait with --text/--menu/--menu-closed support
 2026-02-19 18:17:15 -08:00
Lahfir
39178b2916 feat: surface-targeted snapshot, menu wait, list-surfaces command 
Add snapshot --surface flag (menu/sheet/popover/alert/focused/window)
for direct O(1) AX attribute reads rather than full-tree traversal.
Add wait --menu/--menu-closed for polling-based context-menu gate.
Add list-surfaces command to enumerate open transient surfaces.
Remove all inline // comments from macos crate per 400-LOC/no-comment rule.
 2026-02-19 15:21:07 -08:00
Lahfir
f1ed36046c refactor: Phase A quality fixes — dead code, bugs, LOC compliance 
- Delete clipboard.rs (superseded by clipboard_get/set; zero callers)
- Remove batch::execute() stub (dispatch layer owns batch execution)
- Fix wait.rs: replace double get+unwrap with if-let pattern
- Fix press.rs: replace dead is_empty check + unwrap with ok_or_else
- Add doc comment to is_check.rs documenting stale-state semantics
- Trim tree.rs to 394 LOC (was 403; compress non-macos stub + remove redundant comment)
- Move probe binaries to examples/ with required-features = ["dev-tools"]
- Fix clippy::explicit_auto_deref in adapter.rs
- Fix clippy::needless_borrows_for_generic_args in screenshot.rs
- Apply prior session core/macos fixes (adapter, actions, roles, snapshot, get)

All targets pass cargo clippy -D warnings.
 2026-02-19 15:04:18 -08:00
Lahfir
1d98ab828c fix: make all 30 commands work end-to-end on macOS 
- tree: window_element_for() starts traversal from correct AXWindow
  element (matching by title) instead of app root, fixing mixed-window
  trees and 'disabled group' noise in Electron apps
- tree: AXUIElementCopyMultipleAttributeValues batch fetch (2.5x faster)
- tree: copy_ax_array() CFRetains each element before CFArray drops,
  fixing the dangling-pointer bug that returned kAXErrorInvalidUIElement
- roles: add AXApplication->application and AXSplitGroup->splitter
- actions: CGEvent is now primary click mechanism (AXPress is best-effort
  first try); fixes clicks in Electron/web apps that don't support kAXPressAction
- actions: DoubleClick uses MOUSE_EVENT_CLICK_STATE=2, RightClick uses
  RightMouseDown/Up, Scroll sets event.set_location(element_center)
- screenshot: rewrite using screencapture CLI + CGWindowListCopyWindowInfo
  to find the largest CGWindowID for the app PID; resolves app/window_id
  to pid before calling adapter (was producing all-zero data placeholder)
- screenshot: ScreenshotTarget::Window now carries pid (i32) instead of
  our hash string; commands/screenshot.rs resolves via list_windows
- input: add 8ms inter-keystroke delay in synthesize_text so apps can
  process events (was truncating long strings)
- snapshot: return WindowInfo in SnapshotResult; include app and window
  fields in JSON output per contract
- find: switch to snapshot::run so found refs are persisted to refmap
- dispatch: implement dispatch_batch_command covering all 29 commands
  (was stub returning 'not yet implemented' for every batch sub-command)
 2026-02-19 12:51:37 -08:00
Lahfir
6dc567a4ae fix: align error codes with spec (APP_NOT_FOUND, PERM_DENIED) and add -i shorthand 2026-02-19 11:49:17 -08:00
Lahfir
218503a7eb fix: resolve all 47 code review findings from Phase 1 audit 
Security (P1):
- Replace AppleScript app-name interpolation with PID-based scripts (001, 002)
- Switch pkill -f regex to pkill -x exact-name match (003)
- Validate launch_app id against path traversal (006)
- Remove CFRetain+mem::forget; use ManuallyDrop correctly (005, 029)
- Add cycle detection (visited set) to resolve_element (009)

Correctness (P1/P2):
- Implement read_bounds via kAXPositionAttribute+kAXSizeAttribute (023)
- Fix Scroll action to use CGEventCreateScrollWheelEvent (027)
- Propagate Expand/Collapse AX errors instead of discarding (042)
- Store AXDescription separately from AXTitle (046)
- Fix interactive_only to recurse into container children (043)
- Pass window.pid to RefEntry instead of hardcoded 0 (019)
- Add batch::parse_commands() for dispatch-layer batch execution (022)
- Fix find command to use snapshot::build() — no RefMap overwrite (030)
- Return non-interactive elements from find with ref:null (044)
- Wire permissions --request to adapter.check_permissions (031)
- Fix screenshot --app to resolve window ID (032)
- Close-app protected process check uses exact match (014)
- Wait unbounded sleep capped at 30s (012)
- Fix wait double-lookup unwrap (038)
- Fix press.rs unwrap, normalize modifier order (021, 038)
- emit_json no longer silently discards write errors (041)

Data / API:
- WindowInfo.app serializes as "app_name" per spec (045)
- Add value/states/bounds fields to RefEntry; populate from snapshot (024, 025)
- Stable window IDs via FxHasher(pid+title) (028)
- Use FxHasher for bounds_hash, replacing unstable DefaultHasher (020)
- Flush refmap temp file before rename (039)
- Add HOME fallback to USERPROFILE (015)

Architecture:
- Split click.rs → click, double_click, right_click per one-command-per-file (047)
- Split clipboard.rs → clipboard_get, clipboard_set (047)
- Consolidate RefArgs to helpers.rs (047)
- Move focus/launch/close impl to app_ops.rs; adapter.rs 389→311 LOC (001)
- Deduplicate ErrorCode::code() via ErrorCode::as_str() (034)
- Add doc comment to Response struct explaining Phase 3 intent (035)
- Add snapshot::build() for read-only tree access (030)
- Remove duplicate ABSOLUTE_MAX_DEPTH from snapshot.rs (033)
 2026-02-19 11:38:17 -08:00
Lahfir
608d4aaaa1 fix: suppress dead_code lint on BatchCommand deserializer struct 2026-02-19 10:46:47 -08:00
Lahfir
a346f242c2 feat: Phase 1 foundation — workspace scaffold, core engine, macOS adapter, 31 commands 
Implements the complete agent-desktop Phase 1 specification:

- Workspace: 5-crate layout (core, macos, windows/linux stubs, binary)
- Core: AccessibilityNode, Action, ErrorCode, PlatformAdapter trait, RefMap
  with atomic writes, SnapshotEngine with depth-first ref allocation
- macOS adapter: AXUIElement tree traversal, action execution, input
  synthesis via CGEvent, screenshot via CGWindowListCreateImage, clipboard
  via pbpaste/pbcopy, window listing and app management via osascript
- 31 CLI subcommands via clap derive: snapshot, find, screenshot, get, is,
  click, double-click, right-click, type, set-value, focus, select, toggle,
  expand, collapse, scroll, press, launch, close-app, list-windows,
  list-apps, focus-window, clipboard-get/set, wait, status, permissions,
  version, batch
- Windows/Linux: not-supported stubs ready for Phase 2 implementation
- JSON output contract: {version,ok,command,data} envelope with structured
  error payloads including SCREAMING_SNAKE_CASE error codes
- Ref system: @e{N} sequential refs for interactive elements, stored at
  ~/.agent-desktop/last_refmap.json with 0o600/0o700 permissions
- CI: GitHub Actions macOS runner with dependency isolation check, clippy,
  unit tests, release build, and 15MB binary size gate
 2026-02-19 10:44:38 -08:00