- dispatch.rs: 460 → 392 LOC by removing duplicate command_name(),
using cmd.name() from cli.rs, and moving cli_surface_to_core to
Surface::to_core() in cli_args.rs
- ElementCaps: trim from 8 to 3 used fields, remove has_action() and
unused discovery calls (actions, role, has_children, pid, settable_value)
- chain.rs: deduplicate ChainStep/ChainDef/ChainContext out of cfg gates
- ax_helpers.rs: delete dead set_ax_string
Adds tracing::debug! calls throughout command and adapter layers,
activated via -v flag. Logs command dispatch, ref resolution, chain
step execution, clipboard/keyboard/mouse synthesis, and system ops.
Command layer logging (free for all platforms):
- dispatch: command name
- resolve: ref lookup with pid/role/name, match result
- tree: snapshot app/window/ref_count
Adapter layer logging (macOS, other platforms add their own):
- chain: step-by-step [N/total] with success/skip for each
- action: perform entry
- clipboard/keyboard/mouse: operation details
- system: app focus/launch/close, window ops, screenshots
Graph of skills covering all 50 commands with platform-specific sub-skills:
- Main index skill with core concepts, ref system, JSON contract
- Observation, interaction, and system command references
- Common workflow patterns (forms, menus, dialogs, drag-drop)
- macOS platform skill (permissions, AX API, troubleshooting)
- /desktop slash command for quick reference
- PRD addendum for skill maintenance across phases
cargo-workspace plugin cannot parse version.workspace = true in member
crates. Switch to simple strategy with generic file updater using
x-release-please-version annotation in root Cargo.toml.
Automated release pipeline using release-please for gated Release PRs,
conventional commits for SemVer version bumps, and npm distribution
with postinstall binary download from GitHub Releases.
Multi-column apps like Notes, Mail, and Finder bury the editor
textfield deep in the tree (e.g. ref 256 of 272). Agents picking
"the first textfield" grab the wrong one. Column hints let agents
distinguish sidebar vs content-list vs editor panes.
- Add 7-step right-click chain: AXShowMenu direct, focus-app +
AXShowMenu, select + AXShowMenu, focus-element + AXShowMenu,
parent AXShowMenu, child AXShowMenu, CGEvent last resort
- Key fix: AXShowMenu returns -25204 (CannotComplete) when app
isn't frontmost; ensure_app_focused via AX resolves this
- Right-click command now returns full context menu tree inline
with ref_ids on all menuitems for immediate agent interaction
- Remove surface-based menu detection (menu lives in regular tree)
- Clean up debug tracing from resolve.rs
- Rewrite ax_scroll with 10-step AX-first chain (scroll-to-visible,
increment/decrement, page scroll, value shift, sub-element press,
focus child, select rows, keyboard, arrow keys, CGEvent last resort)
- Add ensure_app_focused() guard before all CGEvent calls
- Enhance smart_activate with AXScrollToVisible pre-step,
AXShowAlternateUI, AXCustomActions, and keyboard space fallback
- Fix bounds_hash always null in RefMap (always read bounds in builder)
- Add right-click menu capture in response data
- Add append_surface_refs for post-action menu detection
- Add --surface menubar to expose full menu bar hierarchy
- Fix use-after-free in press_for_app_impl (ManuallyDrop)
- Fix AX modifier bit mapping (Shift=1<<0, Alt=1<<1)
- Improve alert detection to search all app windows
AXShowDefaultUI returns success on Finder sidebar rows but doesn't
navigate — it's a presentation action, not activation. Remove it so
the chain falls through to child activation which fires AXOpen on the
child AXCell, triggering actual navigation.
Also expand child activation to try AXOpen and AXShowDefaultUI on
children (not just AXPress/AXConfirm), and fire-and-forget since some
apps navigate as side effect even on error return codes.
Restructure crates/macos/src/ from flat files into tree/, actions/,
input/, system/ subfolders. No file exceeds 400 LOC.
Add actions/activate.rs with a 10-step discovery-based activation chain:
steps 1-9 are pure AX (AXPress, AXConfirm, AXOpen, AXPick, AXSelected,
AXSelectedRows on parent, focus+retry, child walk, parent walk),
step 10 falls back to CGEvent only when every AX path fails.
Wire smart_activate() into Click/DoubleClick/RightClick/TripleClick/Toggle.
Add AXDisclosing attribute fallback for Expand/Collapse.
Rewrite README with full command reference.
Add Platform Crate Folder Structure section to CLAUDE.md enforcing
consistent subfolder layout (tree/, actions/, input/, system/) across
all platform crates. Update smart AX-first click chain plan with
new file paths reflecting the subfolder structure. Un-ignore
docs/plans/ and docs/brainstorms/ so design documents are tracked.
Two critical fixes for macOS accessibility tree traversal:
1. Pointer reuse bug: macOS returns different logical AX elements
sharing the same AXUIElementRef pointer value across sibling
branches (e.g., split view panes). The global visited set
treated these as duplicates, silently dropping entire UI
sections (search fields, content panes, toolbars).
Fix: changed build_subtree and find_element_recursive from
global visited sets to ancestor-path sets. Pointers are added
on entry and removed after children are processed, allowing
the same pointer in different branches while still preventing
true ancestor→descendant cycles.
Impact: System Settings snapshot went from 45 to 121 refs,
now exposing sidebar, search, toolbar, AND content pane.
2. CGEvent click fallback: when AXPress/AXConfirm fail (common
for treeitems, custom SwiftUI controls), read live bounds
and synthesize a CGEvent mouse click at the element center.
Fallback chains: Click (AXPress→AXConfirm→CGEvent),
DoubleClick (AXOpen→AXPress×2→CGEvent), RightClick
(AXShowMenu→CGEvent), TripleClick, Toggle, Check/Uncheck.
Documents the root cause (sequential AX IPC calls per node), the fix
(AXUIElementCopyMultipleAttributeValues batch fetching), measured wall-time
benchmarks across TextEdit/Finder/System Settings/Xcode, and the secondary
optimisations: FxHashSet cycle detection, streaming JSON serialisation, and
per-element messaging timeout.
AXShowMenu is the accessibility-native way to trigger a context menu on
an element without moving the cursor or posting CGEvents. After the
action, the resulting AXMenu appears as a child of the triggered element.
- actions.rs: RightClick tries AXShowMenu first, falls back to CGEvent
- surfaces.rs: context_menu_from_app checks AXFocusedUIElement's children
for AXMenu, then falls back to direct app children (Electron apps)
- list_surfaces_for_pid: surfaces context menus as 'context_menu' type
(distinct from menu-bar 'menu' entries)
AXMenus on AXApplication returns kAXErrorAttributeUnsupported (-25205)
for standard Cocoa apps. Menu bar menus are always children of their
AXMenuBarItem; the only reliable open-state indicator is AXSelected=true
on that item.
- open_menubar_menu: scan AXMenuBar.AXChildren for AXMenuBarItem with
AXSelected=true, return its AXMenu child
- context_menu_from_app: kept as fallback for Electron-style apps that
do expose context menus as AXMenu in AXApplication.AXChildren
- list_surfaces_for_pid: reports menu title and item_count when open
- is_menu_open: checks both paths
Add snapshot --surface flag (menu/sheet/popover/alert/focused/window)
for direct O(1) AX attribute reads rather than full-tree traversal.
Add wait --menu/--menu-closed for polling-based context-menu gate.
Add list-surfaces command to enumerate open transient surfaces.
Remove all inline // comments from macos crate per 400-LOC/no-comment rule.
- tree: window_element_for() starts traversal from correct AXWindow
element (matching by title) instead of app root, fixing mixed-window
trees and 'disabled group' noise in Electron apps
- tree: AXUIElementCopyMultipleAttributeValues batch fetch (2.5x faster)
- tree: copy_ax_array() CFRetains each element before CFArray drops,
fixing the dangling-pointer bug that returned kAXErrorInvalidUIElement
- roles: add AXApplication->application and AXSplitGroup->splitter
- actions: CGEvent is now primary click mechanism (AXPress is best-effort
first try); fixes clicks in Electron/web apps that don't support kAXPressAction
- actions: DoubleClick uses MOUSE_EVENT_CLICK_STATE=2, RightClick uses
RightMouseDown/Up, Scroll sets event.set_location(element_center)
- screenshot: rewrite using screencapture CLI + CGWindowListCopyWindowInfo
to find the largest CGWindowID for the app PID; resolves app/window_id
to pid before calling adapter (was producing all-zero data placeholder)
- screenshot: ScreenshotTarget::Window now carries pid (i32) instead of
our hash string; commands/screenshot.rs resolves via list_windows
- input: add 8ms inter-keystroke delay in synthesize_text so apps can
process events (was truncating long strings)
- snapshot: return WindowInfo in SnapshotResult; include app and window
fields in JSON output per contract
- find: switch to snapshot::run so found refs are persisted to refmap
- dispatch: implement dispatch_batch_command covering all 29 commands
(was stub returning 'not yet implemented' for every batch sub-command)
