AXUIElementCopyAttributeValue and CopyMultipleAttributeValues can
return non-array CF types for attributes that are normally arrays
(observed with Mail.app on macOS 26.3 beta). Casting the raw
CFTypeRef to CFArray<CFType> without verifying the type ID is
undefined behaviour and can cause a panic with 'entered unreachable
code' when the resulting fake array is iterated.
Add CFGetTypeID / CFArrayGetTypeID guards in copy_ax_array and
fetch_node_attrs before the unsafe casts. On mismatch, release the
value and fall back gracefully (None / per-attribute fallback path).
Fixes#49
Unify CLI and batch dispatch around the typed command path, centralize command policy and ref resolution, harden macOS action verification, split command tests from implementation, and add package/release guardrails.
BREAKING CHANGE: CLI and batch execution now share the typed command path and current command argument contracts.
BREAKING CHANGE: Ref-consuming commands use snapshot-scoped refs; deterministic consumers should pass snapshot_id and handle SNAPSHOT_NOT_FOUND.
BREAKING CHANGE: permissions and status now return PermissionReport fields for accessibility, screen_recording, and automation instead of a single boolean status.
BREAKING CHANGE: PermissionState gains NotRequired; macOS automation now reports not_required instead of unknown.
BREAKING CHANGE: right-click now separates action success from menu verification; consumers should inspect menu or menu_probe instead of assuming every right-click returns an inline menu.
BREAKING CHANGE: focus-window now confirms OS focus and returns ACTION_FAILED when focus does not settle; data.focused.is_focused is true on success.
BREAKING CHANGE: PlatformAdapter::execute_action now takes ActionRequest, and permission probing uses permission_report/request_permissions.
BREAKING CHANGE: FFI ad_execute_action now defaults to headless policy. Consumers that need focus fallback or cursor-moving behavior must call ad_execute_action_with_policy with AD_POLICY_KIND_FOCUS_FALLBACK or AD_POLICY_KIND_PHYSICAL.
BREAKING CHANGE: FFI ad_check_permissions no longer treats unknown accessibility permission as success; stub-style unknown probes return ERR_PLATFORM_NOT_SUPPORTED and macOS ambiguous unknown returns ERR_INTERNAL with last-error detail.
BREAKING CHANGE: JSON response envelopes now report version 2.0; parsers pinned to 1.0 must branch or update.
BREAKING CHANGE: focus now uses accessibility focus without cursor movement; callers that need physical focus must use explicit mouse or physical-policy paths.
BREAKING CHANGE: chain execution deadlines now return TIMEOUT instead of ACTION_FAILED when the target app does not respond before the chain deadline.
Add `agent-desktop skills` (list/get/path) so agents can load
version-matched usage guidance straight from the binary, and rework
`--help` with a top-of-help "For AI agents — read this first" section
that points there. Skills are embedded via `include_str!`, with `desktop`
and `ffi` aliases resolving to canonical names.
Also extract the long after_help text out of cli.rs into help_after.txt
and a new help_before.txt to keep the file under the 400 LOC limit, and
sweep the 53→54 command count across CLAUDE.md, README.md, and SKILL.md.
* feat: add data model foundation for progressive skeleton traversal
Add children_count, root_ref, skeleton fields to core types. Add
get_subtree() to PlatformAdapter trait. Add remove_by_root_ref() and
write-side size check to RefMap. No behavioral changes yet.
* feat: implement progressive skeleton traversal with ref-rooted drill-down
Add --skeleton and --root flags to snapshot command for token-efficient
accessibility tree exploration. Skeleton mode clamps depth to 3 levels
and annotates truncated containers with children_count, allowing AI
agents to discover regions before drilling into them. Named containers
at skeleton boundaries (via name or description) receive refs as
drill-down targets. The --root flag starts traversal from a previous
ref with scoped invalidation — only refs from that drill-down are
replaced on re-drill.
Key changes:
- New ref_alloc.rs: shared ref helpers (INTERACTIVE_ROLES, actions_for_role,
ref_entry_from_node, is_collapsible) extracted from snapshot.rs
- New snapshot_ref.rs: drill-down logic with DrillDownConfig, scoped
invalidation via root_ref tagging on RefEntry
- macOS count_children() uses raw CFArrayGetCount without materializing
AXElement wrappers for performance at skeleton boundaries
- RefMap write-side size check prevents >1MB files
- Skeleton anchors consider both name and description for Electron compat
* fix: mention --skeleton in STALE_REF error suggestion
* docs: document --skeleton and --root flags in skill reference
* docs: update phases.md and CLAUDE.md for progressive skeleton traversal
Add skeleton traversal as Phase 1 objective P1-O10. Document --skeleton
and --root flags, get_subtree() trait method, new core modules, and
platform-agnostic notes for Phase 2/3. Update risk mitigations and
performance optimizations table.
* docs: make progressive skeleton traversal the default agent workflow
Update SKILL.md observe-act loop to skeleton-first approach. Add
progressive skeleton traversal as the primary workflow pattern. Update
anti-patterns, key principles, and command quick reference to lead
with --skeleton + --root. Full snapshot remains documented as fallback
for simple apps.
* fix: preserve skeleton drill-down anchors
* fix: preserve drill-down refs across skeleton re-snapshots
Skeleton snapshots now load the existing refmap and remove only
skeleton-level refs (root_ref: None), preserving drill-down refs
accumulated via --root. Previously, build() always created a fresh
RefMap, breaking the skeleton → drill → act → skeleton(verify) workflow
by wiping all drill-down refs on the verify step.
* fix: preserve drill-down depth for root snapshots
* fix: verify AX action effect on Electron elements before trusting success
Chromium's AX implementation returns kAXErrorSuccess for AXPress,
AXConfirm, etc. but only toggles ARIA state without firing DOM event
handlers. This caused the click chain to short-circuit on false
positives, preventing CGClick from ever being reached.
The fix detects web elements via AXWebArea ancestor walk, then verifies
each AX action actually had a DOM effect by comparing focused element
pointers before and after. When all AX methods produce no real effect,
CGClick fires as the genuine last resort.
Native elements are unaffected — the original verified_press behavior
is preserved in a separate code path.
* chore: ignore .context/ for local tooling artifacts
* style: collapse web_action_had_effect signature to single line
* test: cover RefMap save oversize rejection
Extract the serialize+size-check step into a private helper so the 1MB
write rejection path can be unit-tested without filesystem I/O. Also
moves the size check above the directory creation in save() so an
oversized refmap no longer creates ~/.agent-desktop on rejection.
* test: assert stale-ref suggestion mentions --skeleton
Locks in the Phase 4 polish requirement that STALE_REF errors guide
agents back to a skeleton refresh, not just a plain snapshot.
* test: cover skeleton-to-drill-down counter continuity
Asserts that skeleton refs (@e1..@e10) survive a scoped invalidation
of @e3, that drill-down refs allocated after the skeleton continue from
@e11 instead of resetting, and that remove_by_root_ref drops only the
drill-down children.
* test: cover --root + --surface rejection at execute() boundary
Adds a NoopAdapter that uses every PlatformAdapter trait default to
exercise execute()'s validation guards without standing up a real
adapter. Verifies that combining --root with a non-Window surface
returns INVALID_ARGS, and that the Window surface does not trigger the
guard.
* test: add filesystem-redirected integration tests for run_from_ref
Adds a thread-local HOME override + RAII HomeGuard so RefMap::save and
RefMap::load can be exercised against an isolated temp directory without
env-var racing across parallel tests. Also adds a StubAdapter that
implements PlatformAdapter with a canned subtree response so the full
run_from_ref drill-down flow can be unit-tested without standing up
macOS AX state.
Closes seven Phase 3 plan acceptance items in one pass:
- save+load roundtrip with HOME override
- oversize save rejection preserves previous file on disk
- run_from_ref returns subtree and persists drill refs
- stale root ref → STALE_REF with skeleton suggestion
- re-drill replaces drill refs only, counter continues
- multiple drill-downs from @e1 and @e2 coexist
- empty subtree drill-down produces no new refs
* test: add golden fixtures for skeleton output and drill-down refmap
Adds two committed JSON fixtures under tests/fixtures/ plus matching
unit tests that build the same input trees, run them through
allocate_refs / run_from_ref, and assert the produced ref ids,
parent-child layout, and root_ref tagging match the golden expectations.
Locks in the JSON shape so future serialization changes have to be
deliberate.
* fix: bound build_subtree recursion on Electron wrapper chains
Anonymous AXGroup/AXGenericElement wrappers do not advance the semantic
depth counter (web-wrapper depth-skip), so a long chain of nested
wrappers could recurse arbitrarily deep without ever hitting either
max_depth or ABSOLUTE_MAX_DEPTH. The previous code checked
`depth >= ABSOLUTE_MAX_DEPTH` against the semantic depth, which stayed
at 0 throughout a wrapper chain. On pathological Electron trees this
risked stack exhaustion before any cap engaged.
Add a separate `raw_depth` parameter that always increments and use it
for the absolute cap. Semantic `depth` still drives `max_depth` and the
skeleton boundary so the wrapper-flattening behavior is preserved on
normal trees.
Also drops the long-unused `_include_bounds` parameter; bounds
filtering happens in `allocate_refs` in core, not here.
* chore: add pre-commit hook running fmt + clippy + tests
Mirrors the CI quality gates (cargo fmt --check, cargo clippy
--all-targets -- -D warnings, cargo test --lib --workspace) so a
failing commit never reaches origin. Skips automatically when no
Rust/TOML files are staged. Bypass with --no-verify or SKIP_PRECOMMIT=1
when a non-Rust hotfix is genuinely needed.
Hook lives under .githooks/ and is opt-in per clone:
git config core.hooksPath .githooks
Setup instructions added to CLAUDE.md.
* fix: prevent orphaned drill-down refs leaking across skeleton refresh
remove_skeleton_refs() previously dropped every entry whose root_ref
was None and kept every entry whose root_ref was Some. After a series
of skeleton refreshes that pattern leaked drill-down refs whose root
anchor had already been removed: they could never be cleaned up by a
future remove_by_root_ref(target) because target referred to a
non-existent anchor, and they accumulated until the 1MB write guard
fired.
Restructure the function to keep only:
- skeleton anchors that have at least one drill-down pointing at them
- drill-down refs whose root anchor is among the kept anchors
Unreferenced anchors are still dropped (the original intent), and
orphaned drill-downs are dropped at the same time so the refmap can
no longer accumulate dead entries.
Updated existing test to assert the new keep-when-referenced semantics
and added a regression test for the orphan-drilldown leak path.
* fix: align resolve traversal with snapshot child-attribute set
find_element_recursive previously walked AXChildren first then fell
back to AXContents only, and resolve_element_name only derived its
fallback label from AXChildren. Snapshot building, however, uses the
full child_attributes() list (AXChildren, AXContents,
AXChildrenInNavigationOrder) via copy_children. That asymmetry meant
refs minted for containers exposed only through AXChildrenInNavigationOrder
appeared in the snapshot output but produced false STALE_REF errors on
drill-down or action commands because the resolver could not find them
again.
Route both call sites through child_attributes() so resolution sees
the same children as snapshotting.
* fix: bypass AXConfirm on web elements and add skeleton anchors to drill-down
* fix: compare AX elements via CFEqual in web_action_had_effect
AXUIElementCopyAttributeValue follows the CoreFoundation Create rule
and returns a freshly-allocated CF object on every call, so raw
pointer equality (`before.0 != after.0`) between two separate copies
of AXFocusedUIElement was ALWAYS true even when the focused element
was unchanged. That made Step 1 (AXPress) of activate_web_element
appear successful every time and left Steps 2-4 of the escalation
chain (AXConfirm bypass, child actions, CGClick fallback) dead code
on web elements.
Replace the pointer comparison with CFEqual, which compares
accessibility element identity rather than heap addresses.
* refactor: unify allocate_refs across snapshot and drill-down paths
allocate_refs in snapshot.rs and allocate_refs_with_root in
snapshot_ref.rs were near-exact copies that differed only in whether
each allocated ref carried a root_ref tag. The duplication meant any
bug fix or behavior change in one path risked silently drifting from
the other (bot flagged this after the first round of fixes added even
more duplicated skeleton-anchor logic to allocate_refs_with_root).
Move the shared logic into ref_alloc.rs behind a new RefAllocConfig
struct with Option<&str> for the root_ref_id, and delete the
snapshot_ref.rs copy. Both snapshot::build and snapshot_ref::run_from_ref
now route through the same function with their respective configs.
append_surface_refs and the existing unit tests in both modules are
updated to use the shared function + config.
* chore: drop with_root from drill test names after allocator unification
The snapshot_ref tests kept their original test_allocate_refs_with_root_*
names after the allocator dedupe even though allocate_refs_with_root no
longer exists. Rename them to test_drill_alloc_* so grep for
allocate_refs_with_root returns zero matches and nobody assumes a second
allocator path exists.
* docs: compound DRY ref-allocator dedupe into knowledge base
Adds docs/solutions/best-practices/deduplicate-ref-allocator-via-config-struct-2026-04-14.md
documenting the root cause, guidance, consequences, trigger conditions,
and before/after of the allocate_refs / allocate_refs_with_root
duplication that landed on PR #20 and was unified in d06a6c2.
Also:
- gitignore allows docs/solutions/** (was caught by docs/* rule)
- CLAUDE.md workspace tree surfaces docs/solutions/ so fresh agents
discover the knowledge store
- docs/phases.md drops two stale DrillDownConfig references (lines 62
and 226) — the symbol no longer exists in the codebase
* refactor: drop unused root_ref field from TreeOptions
TreeOptions.root_ref was populated from SnapshotArgs.root_ref by
tree_options() and then never read anywhere. The actual routing
lives in execute(): `if let Some(ref root) = args.root_ref` branches
to snapshot_ref::run_from_ref(adapter, &opts, root), passing the
root id as an explicit argument — opts.root_ref was never consulted
by any caller.
Delete the field from TreeOptions and its Default impl and stop
populating it in tree_options(). SnapshotArgs.root_ref remains as
the single source of truth for the drill-down target.
* fix: suppress skeleton flag when --root is set
tree_options() clamped depth based on skeleton && root_ref.is_none()
but still forwarded skeleton: args.skeleton unconditionally. When a
caller passed --skeleton --root @e3, opts.skeleton stayed true, so
adapter.get_subtree on macOS built a truncated skeleton tree instead
of the full drill-down subtree, AND ref_alloc::allocate_refs tagged
skeleton-anchor refs with the drill-down root_ref — both unintended.
Tie the skeleton flag to the same root_ref.is_none() guard the depth
clamp already uses. A drill-down is always a full subtree now, never
a skeleton view.
Adds test_tree_options_suppresses_skeleton_for_drill_down to lock the
behavior in and extends the existing clamp test to assert the flag
still propagates on non-drill paths.
* fix: detect web action effect via value + selected + focus change
The previous focus-change-only heuristic in web_action_had_effect was
wrong for non-Chromium AXWebArea elements (Safari WKWebView, Mail,
Notes): AXPress on a checkbox toggles AXValue correctly but does NOT
shift the focused UI element. web_action_had_effect returned false, the
chain escalated to CGClick, and CGClick toggled the checkbox a second
time — net zero visible change. The pointer-equality bug that ae78cbc
replaced was masking this by always returning true; CFEqual exposed
the real signal gap.
Capture element state into a PreActionState struct (focused, value,
selected) before any AX action runs, and consider the action to have
had an effect if ANY of those three fields differs afterward. Element
value covers WKWebView checkboxes, text fields, sliders. Selected
covers tabs, radio buttons, and list items. Focused still covers
Chromium's DOM-driven focus shifts. Triggers on whichever signal is
most relevant to the element type without needing role-specific code.
Also re-exports copy_bool_attr from crates/macos/src/tree for use by
the chain-steps module.
* fix: emit skeleton boundary when drill path is about to hit raw cap
In skeleton mode, a chain of anonymous web wrappers (AXGroup /
AXGenericElement with empty title and value) never advances the
semantic `depth` counter, so `child_depth > max_depth` never fires
no matter how deep the chain runs. The recursion then hits the
`raw_depth >= ABSOLUTE_MAX_DEPTH` top-of-function guard and silently
returns None, dropping the subtree without any children_count marker.
Agents see truncated output with no indication that anything was
dropped.
Extend the at_skeleton_boundary condition to also fire when
child_raw_depth is about to hit ABSOLUTE_MAX_DEPTH, so skeleton mode
always emits a visible truncation node for deep wrapper chains
instead of disappearing them. The old `raw_depth < ABSOLUTE_MAX_DEPTH`
half of the guard was redundant — the top-of-function check already
ensures the current frame has raw budget; the meaningful question is
whether the CHILD frames will.
* perf: compute is_in_webarea once per verified-press call
try_focus_then_verified_confirm_or_press called is_in_webarea(el) to
gate AXConfirm, then fell through to do_verified_press which called
is_in_webarea(el) again on its first line. Each call walks up to 20
AX parents via per-element IPC (AXUIElementCopyAttributeValue on
AXParent), so every web-area click through this path paid the cost
twice.
Extract a private dispatch_verified_press(el, caps, in_web) that
takes the web-area flag as an input. do_verified_press keeps its
existing (el, caps) signature for chain_defs compatibility and
computes is_in_webarea once, then delegates. The focus-then-confirm
path also computes it once and passes it through. The result is
identical on both paths with one AX traversal instead of two.
* fix: skeleton anchors in drill-downs must not inherit root_ref; restore AXBrowser AXContents fallback
Skeleton anchors discovered while processing a drill-down subtree were
being tagged with the drill root_ref, making them indistinguishable from
regular drill entries. re-drills would delete those anchors via
remove_by_root_ref, breaking sub-drilling. They now always carry
root_ref=None so they survive re-drills as stable targets.
AXBrowser child resolution dropped the AXContents fallback when
child_attributes() was unified — the resolver now uses
["AXColumns","AXContents"] matching the original traversal order.
* fix: suppress skeleton anchor creation in drill-down mode to prevent orphaned ref accumulation
* fix: bounds-based resolver pruning and CGClick fallback on chain timeout
Resolver was doing exhaustive DFS through entire AX tree (depth 50) to
find a single element. For large documents like a dense spreadsheet this
meant visiting tens of thousands of cells via AX IPC, causing multi-minute
hangs before the click chain even started.
Two changes:
- find_element_recursive now prunes subtrees whose spatial bounds do not
contain the target element's centre point, and aborts the whole search
after five seconds. Numbers table -> button not inside -> entire table
skipped. Resolution went from >1m49s to <50ms.
- execute_chain now sets a 1s app-level AX messaging timeout so calls to
children/parents fail fast instead of blocking for the system default 6s,
and when the 10s chain deadline fires it attempts the CGClick step before
returning failure so the coordinate fallback is always reached.
* refactor: resolve 9 code-review todos for progressive skeleton traversal
- skeleton refresh now starts from RefMap::new() (removes stale ref accumulation across refreshes)
- drill-down snapshot resolves real WindowInfo via PID lookup instead of synthetic empty id
- --root flag validates ref format before RefMap lookup (returns INVALID_ARGS not STALE_REF)
- ABSOLUTE_MAX_DEPTH truncation emits boundary node with children_count instead of silently dropping
- fix ref vs ref_id field name in commands-observation.md JSON examples
- fix phases.md TreeOptions listing root_ref (it lives in SnapshotArgs, not TreeOptions)
- add batch snapshot skeleton/root examples to commands-system.md
- split snapshot.rs and snapshot_ref.rs test modules into separate _tests.rs files (143/69 LOC)
- add integration tests for skeleton + drill-down workflow including invalid --root validation
* docs: compound progressive snapshot review hardening
* docs: tighten progressive snapshot compound write-up
* docs: update README with progressive skeleton traversal and fix command count to 50
* docs: correct command count to 53, add Notifications section and platform notes
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Activate the existing --compact CLI flag (previously a no-op) to reduce
tree verbosity by collapsing pass-through container nodes that carry no
semantic information (no ref, no name, no value, no description, no
states, exactly one child).
Slack with -i --compact: 264 → 214 nodes, ~5,967 → ~5,117 tokens (14%
reduction). All 161 refs preserved. Finder unaffected (3% reduction).
Handles Electron's empty-string-vs-None pattern using is_none_or.
Skip depth budget for non-semantic AXGroup/AXGenericElement wrappers with
empty name and value, allowing default --max-depth 10 to find 100+ refs
in Electron apps like Slack and VS Code (previously found only 3).
Raise resolver search depth from 20 to ABSOLUTE_MAX_DEPTH (50) so ref
resolution succeeds for deeply nested Electron elements.
Fix surface detection to check if focused window itself is the target
surface (Electron reports dialogs as focused window, not as children)
and check both AXRole and AXSubrole for surface matching.
Update Phase 2 (Windows) and Phase 3 (Linux) docs with equivalent
web/Electron compatibility patterns for their respective APIs.
NcSession now captures the frontmost application before opening
Notification Center and reactivates it on close, preventing the
source app (e.g. Script Editor) from stealing focus during dismiss.
AXPress "clicks" the notification body (opening the source app) but
does not actually dismiss it. Removing it from the initial action list
forces the code to fall through to the hover-and-close-button strategy,
which reliably removes the notification from Notification Center.
- fix(core): wait --notification uses index-diff detection instead of
count, passes text filter, checks deadline before sleep
- fix(core): dismiss-all-notifications uses single NC session with
batch adapter method and reports individual failures
- refactor(macos): extract dismiss_entry helper to eliminate duplicate
dismiss logic between single and batch operations
- refactor(macos): restrict nc_session visibility to pub(crate), use
absolute paths for pgrep/osascript, add bounded wait with kill
fallback
- refactor(macos): single-pass is_notification_group check using
matches! macro
- refactor: extract notification CLI args and dispatch into separate
files to keep cli_args.rs and dispatch.rs under 400 LOC
- refactor: rename DismissAllNotificationsArgs to
DismissAllNotificationsCliArgs for consistency
- fix(core): remove unused timestamp field from NotificationInfo
- fix: remove poll_interval_ms from wait command (hardcode 500ms)
- docs: update phases.md with completed notification status and
cross-platform reference notes
Try AXDismiss/AXRemoveFromParent/AXPress on the notification group
element before falling back to cursor hover. Extract try_dismiss_button
helper. Cursor-based hover only triggers if all headless strategies
fail.
CGWindowList requires Screen Recording permission on macOS Sequoia to
read other processes' window names, and SystemUIServer blocks AX access
to its menu bar (crashes with Obj-C foreign exception). Switch to pgrep
for PID lookup and osascript clicking ControlCenter's Clock menu bar
item, spawned async to avoid blocking.
Add error code serialization and notification_not_found tests in core.
Add matches_filters unit tests in macOS list module covering app,
text, combined, and no-filter cases.
Dismiss synthesizes mouse hover to reveal close button (Sequoia), then
finds and AXPress it. Falls back to AXPress on notification group.
Notification action finds matching AXButton by name and presses it.
Both reuse shared list_entries for AX element access.
Traverses NC's AX tree to extract notification info (app, title, body,
actions). Handles variable nesting depth across Sonoma/Sequoia via
recursive heuristic matching. Supports --app, --text, and --limit
filters applied during traversal for early termination.
Implements NcSession for safe Notification Center lifecycle management
(open via AX press on clock, close via Escape, with Drop fallback).
Wires list_notifications, dismiss_notification, and notification_action
adapter methods to placeholder stubs pending Phase 3/4 implementation.
Rect deserialization now tolerates null/missing fields (defaults to 0.0),
preventing corrupt refmaps from breaking all subsequent click commands.
read_bounds rejects NaN/Inf at the source. Sidebar cells now resolve via
parent row AXSelected instead of falling through to CGClick. Extracted
chain step functions into chain_steps.rs to stay within 400 LOC limit.
- dispatch.rs: 460 → 392 LOC by removing duplicate command_name(),
using cmd.name() from cli.rs, and moving cli_surface_to_core to
Surface::to_core() in cli_args.rs
- ElementCaps: trim from 8 to 3 used fields, remove has_action() and
unused discovery calls (actions, role, has_children, pid, settable_value)
- chain.rs: deduplicate ChainStep/ChainDef/ChainContext out of cfg gates
- ax_helpers.rs: delete dead set_ax_string
Adds tracing::debug! calls throughout command and adapter layers,
activated via -v flag. Logs command dispatch, ref resolution, chain
step execution, clipboard/keyboard/mouse synthesis, and system ops.
Command layer logging (free for all platforms):
- dispatch: command name
- resolve: ref lookup with pid/role/name, match result
- tree: snapshot app/window/ref_count
Adapter layer logging (macOS, other platforms add their own):
- chain: step-by-step [N/total] with success/skip for each
- action: perform entry
- clipboard/keyboard/mouse: operation details
- system: app focus/launch/close, window ops, screenshots
Multi-column apps like Notes, Mail, and Finder bury the editor
textfield deep in the tree (e.g. ref 256 of 272). Agents picking
"the first textfield" grab the wrong one. Column hints let agents
distinguish sidebar vs content-list vs editor panes.
- Add 7-step right-click chain: AXShowMenu direct, focus-app +
AXShowMenu, select + AXShowMenu, focus-element + AXShowMenu,
parent AXShowMenu, child AXShowMenu, CGEvent last resort
- Key fix: AXShowMenu returns -25204 (CannotComplete) when app
isn't frontmost; ensure_app_focused via AX resolves this
- Right-click command now returns full context menu tree inline
with ref_ids on all menuitems for immediate agent interaction
- Remove surface-based menu detection (menu lives in regular tree)
- Clean up debug tracing from resolve.rs
- Rewrite ax_scroll with 10-step AX-first chain (scroll-to-visible,
increment/decrement, page scroll, value shift, sub-element press,
focus child, select rows, keyboard, arrow keys, CGEvent last resort)
- Add ensure_app_focused() guard before all CGEvent calls
- Enhance smart_activate with AXScrollToVisible pre-step,
AXShowAlternateUI, AXCustomActions, and keyboard space fallback
- Fix bounds_hash always null in RefMap (always read bounds in builder)
- Add right-click menu capture in response data
- Add append_surface_refs for post-action menu detection
- Add --surface menubar to expose full menu bar hierarchy
- Fix use-after-free in press_for_app_impl (ManuallyDrop)
- Fix AX modifier bit mapping (Shift=1<<0, Alt=1<<1)
- Improve alert detection to search all app windows
AXShowDefaultUI returns success on Finder sidebar rows but doesn't
navigate — it's a presentation action, not activation. Remove it so
the chain falls through to child activation which fires AXOpen on the
child AXCell, triggering actual navigation.
Also expand child activation to try AXOpen and AXShowDefaultUI on
children (not just AXPress/AXConfirm), and fire-and-forget since some
apps navigate as side effect even on error return codes.
Restructure crates/macos/src/ from flat files into tree/, actions/,
input/, system/ subfolders. No file exceeds 400 LOC.
Add actions/activate.rs with a 10-step discovery-based activation chain:
steps 1-9 are pure AX (AXPress, AXConfirm, AXOpen, AXPick, AXSelected,
AXSelectedRows on parent, focus+retry, child walk, parent walk),
step 10 falls back to CGEvent only when every AX path fails.
Wire smart_activate() into Click/DoubleClick/RightClick/TripleClick/Toggle.
Add AXDisclosing attribute fallback for Expand/Collapse.
Rewrite README with full command reference.
Two critical fixes for macOS accessibility tree traversal:
1. Pointer reuse bug: macOS returns different logical AX elements
sharing the same AXUIElementRef pointer value across sibling
branches (e.g., split view panes). The global visited set
treated these as duplicates, silently dropping entire UI
sections (search fields, content panes, toolbars).
Fix: changed build_subtree and find_element_recursive from
global visited sets to ancestor-path sets. Pointers are added
on entry and removed after children are processed, allowing
the same pointer in different branches while still preventing
true ancestor→descendant cycles.
Impact: System Settings snapshot went from 45 to 121 refs,
now exposing sidebar, search, toolbar, AND content pane.
2. CGEvent click fallback: when AXPress/AXConfirm fail (common
for treeitems, custom SwiftUI controls), read live bounds
and synthesize a CGEvent mouse click at the element center.
Fallback chains: Click (AXPress→AXConfirm→CGEvent),
DoubleClick (AXOpen→AXPress×2→CGEvent), RightClick
(AXShowMenu→CGEvent), TripleClick, Toggle, Check/Uncheck.
AXShowMenu is the accessibility-native way to trigger a context menu on
an element without moving the cursor or posting CGEvents. After the
action, the resulting AXMenu appears as a child of the triggered element.
- actions.rs: RightClick tries AXShowMenu first, falls back to CGEvent
- surfaces.rs: context_menu_from_app checks AXFocusedUIElement's children
for AXMenu, then falls back to direct app children (Electron apps)
- list_surfaces_for_pid: surfaces context menus as 'context_menu' type
(distinct from menu-bar 'menu' entries)
AXMenus on AXApplication returns kAXErrorAttributeUnsupported (-25205)
for standard Cocoa apps. Menu bar menus are always children of their
AXMenuBarItem; the only reliable open-state indicator is AXSelected=true
on that item.
- open_menubar_menu: scan AXMenuBar.AXChildren for AXMenuBarItem with
AXSelected=true, return its AXMenu child
- context_menu_from_app: kept as fallback for Electron-style apps that
do expose context menus as AXMenu in AXApplication.AXChildren
- list_surfaces_for_pid: reports menu title and item_count when open
- is_menu_open: checks both paths
Add snapshot --surface flag (menu/sheet/popover/alert/focused/window)
for direct O(1) AX attribute reads rather than full-tree traversal.
Add wait --menu/--menu-closed for polling-based context-menu gate.
Add list-surfaces command to enumerate open transient surfaces.
Remove all inline // comments from macos crate per 400-LOC/no-comment rule.
- tree: window_element_for() starts traversal from correct AXWindow
element (matching by title) instead of app root, fixing mixed-window
trees and 'disabled group' noise in Electron apps
- tree: AXUIElementCopyMultipleAttributeValues batch fetch (2.5x faster)
- tree: copy_ax_array() CFRetains each element before CFArray drops,
fixing the dangling-pointer bug that returned kAXErrorInvalidUIElement
- roles: add AXApplication->application and AXSplitGroup->splitter
- actions: CGEvent is now primary click mechanism (AXPress is best-effort
first try); fixes clicks in Electron/web apps that don't support kAXPressAction
- actions: DoubleClick uses MOUSE_EVENT_CLICK_STATE=2, RightClick uses
RightMouseDown/Up, Scroll sets event.set_location(element_center)
- screenshot: rewrite using screencapture CLI + CGWindowListCopyWindowInfo
to find the largest CGWindowID for the app PID; resolves app/window_id
to pid before calling adapter (was producing all-zero data placeholder)
- screenshot: ScreenshotTarget::Window now carries pid (i32) instead of
our hash string; commands/screenshot.rs resolves via list_windows
- input: add 8ms inter-keystroke delay in synthesize_text so apps can
process events (was truncating long strings)
- snapshot: return WindowInfo in SnapshotResult; include app and window
fields in JSON output per contract
- find: switch to snapshot::run so found refs are persisted to refmap
- dispatch: implement dispatch_batch_command covering all 29 commands
(was stub returning 'not yet implemented' for every batch sub-command)
