Rect deserialization now tolerates null/missing fields (defaults to 0.0),
preventing corrupt refmaps from breaking all subsequent click commands.
read_bounds rejects NaN/Inf at the source. Sidebar cells now resolve via
parent row AXSelected instead of falling through to CGClick. Extracted
chain step functions into chain_steps.rs to stay within 400 LOC limit.
- dispatch.rs: 460 → 392 LOC by removing duplicate command_name(),
using cmd.name() from cli.rs, and moving cli_surface_to_core to
Surface::to_core() in cli_args.rs
- ElementCaps: trim from 8 to 3 used fields, remove has_action() and
unused discovery calls (actions, role, has_children, pid, settable_value)
- chain.rs: deduplicate ChainStep/ChainDef/ChainContext out of cfg gates
- ax_helpers.rs: delete dead set_ax_string
Adds tracing::debug! calls throughout command and adapter layers,
activated via -v flag. Logs command dispatch, ref resolution, chain
step execution, clipboard/keyboard/mouse synthesis, and system ops.
Command layer logging (free for all platforms):
- dispatch: command name
- resolve: ref lookup with pid/role/name, match result
- tree: snapshot app/window/ref_count
Adapter layer logging (macOS, other platforms add their own):
- chain: step-by-step [N/total] with success/skip for each
- action: perform entry
- clipboard/keyboard/mouse: operation details
- system: app focus/launch/close, window ops, screenshots
Graph of skills covering all 50 commands with platform-specific sub-skills:
- Main index skill with core concepts, ref system, JSON contract
- Observation, interaction, and system command references
- Common workflow patterns (forms, menus, dialogs, drag-drop)
- macOS platform skill (permissions, AX API, troubleshooting)
- /desktop slash command for quick reference
- PRD addendum for skill maintenance across phases
cargo-workspace plugin cannot parse version.workspace = true in member
crates. Switch to simple strategy with generic file updater using
x-release-please-version annotation in root Cargo.toml.
Automated release pipeline using release-please for gated Release PRs,
conventional commits for SemVer version bumps, and npm distribution
with postinstall binary download from GitHub Releases.
Multi-column apps like Notes, Mail, and Finder bury the editor
textfield deep in the tree (e.g. ref 256 of 272). Agents picking
"the first textfield" grab the wrong one. Column hints let agents
distinguish sidebar vs content-list vs editor panes.
- Add 7-step right-click chain: AXShowMenu direct, focus-app +
AXShowMenu, select + AXShowMenu, focus-element + AXShowMenu,
parent AXShowMenu, child AXShowMenu, CGEvent last resort
- Key fix: AXShowMenu returns -25204 (CannotComplete) when app
isn't frontmost; ensure_app_focused via AX resolves this
- Right-click command now returns full context menu tree inline
with ref_ids on all menuitems for immediate agent interaction
- Remove surface-based menu detection (menu lives in regular tree)
- Clean up debug tracing from resolve.rs
- Rewrite ax_scroll with 10-step AX-first chain (scroll-to-visible,
increment/decrement, page scroll, value shift, sub-element press,
focus child, select rows, keyboard, arrow keys, CGEvent last resort)
- Add ensure_app_focused() guard before all CGEvent calls
- Enhance smart_activate with AXScrollToVisible pre-step,
AXShowAlternateUI, AXCustomActions, and keyboard space fallback
- Fix bounds_hash always null in RefMap (always read bounds in builder)
- Add right-click menu capture in response data
- Add append_surface_refs for post-action menu detection
- Add --surface menubar to expose full menu bar hierarchy
- Fix use-after-free in press_for_app_impl (ManuallyDrop)
- Fix AX modifier bit mapping (Shift=1<<0, Alt=1<<1)
- Improve alert detection to search all app windows
AXShowDefaultUI returns success on Finder sidebar rows but doesn't
navigate — it's a presentation action, not activation. Remove it so
the chain falls through to child activation which fires AXOpen on the
child AXCell, triggering actual navigation.
Also expand child activation to try AXOpen and AXShowDefaultUI on
children (not just AXPress/AXConfirm), and fire-and-forget since some
apps navigate as side effect even on error return codes.
Restructure crates/macos/src/ from flat files into tree/, actions/,
input/, system/ subfolders. No file exceeds 400 LOC.
Add actions/activate.rs with a 10-step discovery-based activation chain:
steps 1-9 are pure AX (AXPress, AXConfirm, AXOpen, AXPick, AXSelected,
AXSelectedRows on parent, focus+retry, child walk, parent walk),
step 10 falls back to CGEvent only when every AX path fails.
Wire smart_activate() into Click/DoubleClick/RightClick/TripleClick/Toggle.
Add AXDisclosing attribute fallback for Expand/Collapse.
Rewrite README with full command reference.
Add Platform Crate Folder Structure section to CLAUDE.md enforcing
consistent subfolder layout (tree/, actions/, input/, system/) across
all platform crates. Update smart AX-first click chain plan with
new file paths reflecting the subfolder structure. Un-ignore
docs/plans/ and docs/brainstorms/ so design documents are tracked.
Two critical fixes for macOS accessibility tree traversal:
1. Pointer reuse bug: macOS returns different logical AX elements
sharing the same AXUIElementRef pointer value across sibling
branches (e.g., split view panes). The global visited set
treated these as duplicates, silently dropping entire UI
sections (search fields, content panes, toolbars).
Fix: changed build_subtree and find_element_recursive from
global visited sets to ancestor-path sets. Pointers are added
on entry and removed after children are processed, allowing
the same pointer in different branches while still preventing
true ancestor→descendant cycles.
Impact: System Settings snapshot went from 45 to 121 refs,
now exposing sidebar, search, toolbar, AND content pane.
2. CGEvent click fallback: when AXPress/AXConfirm fail (common
for treeitems, custom SwiftUI controls), read live bounds
and synthesize a CGEvent mouse click at the element center.
Fallback chains: Click (AXPress→AXConfirm→CGEvent),
DoubleClick (AXOpen→AXPress×2→CGEvent), RightClick
(AXShowMenu→CGEvent), TripleClick, Toggle, Check/Uncheck.
Documents the root cause (sequential AX IPC calls per node), the fix
(AXUIElementCopyMultipleAttributeValues batch fetching), measured wall-time
benchmarks across TextEdit/Finder/System Settings/Xcode, and the secondary
optimisations: FxHashSet cycle detection, streaming JSON serialisation, and
per-element messaging timeout.
AXShowMenu is the accessibility-native way to trigger a context menu on
an element without moving the cursor or posting CGEvents. After the
action, the resulting AXMenu appears as a child of the triggered element.
- actions.rs: RightClick tries AXShowMenu first, falls back to CGEvent
- surfaces.rs: context_menu_from_app checks AXFocusedUIElement's children
for AXMenu, then falls back to direct app children (Electron apps)
- list_surfaces_for_pid: surfaces context menus as 'context_menu' type
(distinct from menu-bar 'menu' entries)
AXMenus on AXApplication returns kAXErrorAttributeUnsupported (-25205)
for standard Cocoa apps. Menu bar menus are always children of their
AXMenuBarItem; the only reliable open-state indicator is AXSelected=true
on that item.
- open_menubar_menu: scan AXMenuBar.AXChildren for AXMenuBarItem with
AXSelected=true, return its AXMenu child
- context_menu_from_app: kept as fallback for Electron-style apps that
do expose context menus as AXMenu in AXApplication.AXChildren
- list_surfaces_for_pid: reports menu title and item_count when open
- is_menu_open: checks both paths
Add snapshot --surface flag (menu/sheet/popover/alert/focused/window)
for direct O(1) AX attribute reads rather than full-tree traversal.
Add wait --menu/--menu-closed for polling-based context-menu gate.
Add list-surfaces command to enumerate open transient surfaces.
Remove all inline // comments from macos crate per 400-LOC/no-comment rule.
