Drives the release binary against a real SwiftUI/AppKit fixture and verifies
every effect by independent before/after observation — never the command's own
ok:true — so a command that reports success without an effect is caught. This is
the layer mock-adapter unit tests cannot cover: it exercises the contract
against the real macOS Accessibility API.
- AgentDeskFixture.swift exposes a fixed, diverse AX surface (native AppKit
slider/stepper, gesture-only and ambiguous controls, a sheet, a press-toggled
disclosure, async-appearing elements, a drag canvas). It is never tuned to
make a command pass; a failure is a finding about the CLI or the harness.
- run.sh drives every ref-action command in BOTH headless and --headed mode with
mode-specific target values, plus the double-click discriminator (headless
fails closed with POLICY_DENIED, --headed completes) that proves the two modes
differ. It also covers strict resolution, wait predicates, skeleton drill-down,
sessions, trace redaction, surfaces, drag, expand, and force-close.
- The compiled fixture .app is a build artifact (gitignored; built on demand).
Run: cargo build --release && bash tests/e2e/run.sh (needs AX permission).
Ref actions now run in exactly two modes. Headless is the default: semantic
accessibility operations only, no cursor movement, and a fail-closed
POLICY_DENIED when only a physical gesture would work. The global --headed flag
upgrades every ref action to permit focus stealing and cursor movement, so the
chain's physical click/double-click/scroll/keypress fallbacks can complete. The
AX path is always tried first, so --headed never regresses headless-capable
elements; it only adds fallbacks for elements that need a real gesture.
- CommandContext::request(action, base) builds the per-command request: each
command declares its headless base (pure-AX headless; type uses focus_fallback
because typing requires focus but never moves the cursor) and --headed upgrades
any base to the headed policy.
- The internal/FFI "physical" policy is renamed "headed" throughout, including
the C ABI enum (AD_POLICY_KIND_HEADED keeps discriminant 2) and bindings.
- Raw-input commands (press, hover, drag, mouse-*, key-down/up) are unchanged:
always physical, mode-independent low-level escape hatch.
- Unit tests assert every ref command is headless by default and headed under
--headed; docs (CONCEPTS, CLAUDE, skills) describe the two-mode contract.
Dogfooding the binary against a real fixture app surfaced five cases where a
command reported success without producing the effect, or failed with the wrong
error. Each is verified by independent before/after observation in the E2E
harness.
- is_menu_open no longer treats a latent AXMenuBar as an open menu, so select
and wait --menu-closed stop seeing a permanently-open menu.
- set-value coerces the written AXValue to the element's existing CFNumber/
CFBoolean/CFString type and verifies numerically, fixing sliders; steppers
converge via AXIncrement/AXDecrement when AXValue writes are vacuous.
- double-click only claims success when the element advertises AXOpen; otherwise
it fails closed instead of reporting a non-existent double-click.
- a completed resolution pass that proves a ref absent downgrades a deadline
TIMEOUT to STALE_REF so removed elements fail with the correct code.
- expand/collapse verify the disclosure state and fall back to a press-toggle
for press-driven disclosures; press-toggled containers expose EXPAND/COLLAPSE.
roles.rs adds the disclosure expandable role and normalizes textarea/textbox/
searchfield role queries to textfield.
E2E testing against a diverse fixture app surfaced that disclosure
(Expand/Collapse/Click) and scrollarea (Scroll) advertise actions but
never received refs — they are not in INTERACTIVE_ROLES — so the scroll,
expand, and collapse commands required a <REF> their own target roles
could never have. The commands were uninvokable against their primary
targets.
Ref allocation now gates on addressability, not role alone: an element
is ref-able if its role is interactive OR it advertises a primary action
(any action other than a bare SetFocus, which would ref-allocate inert
focusable containers). scrollarea and disclosure become ref-able;
scroll now works against a real app. Ref-count impact is modest
(fixture 61->72, Finder ~262).
Tests assert action-bearing containers get refs, SetFocus-only and inert
elements do not, and interactive roles stay ref-able without actions.
Contract docs (CLAUDE.md, SKILL.md) updated.
macOS drop targets need the dragged item to dwell over them before they
register as the destination; too short and the gesture lands as a drag
with no drop. The dwell was a hardcoded 500ms dead sleep. Expose it as
--drop-delay <ms> (CLI), drop_delay_ms (DragParams/AdDragParams, 0 =
adapter default sentinel matching duration_ms), and replace the dead
sleep with an event-driven dwell that posts LeftMouseDragged over the
destination every 16ms so the target stays highlighted instead of
dropping the drag mid-pause.
DRY: the C-to-core drag conversion (duration/drop-delay zero-sentinel)
was copied across three FFI sites; collapse them into AdDragParams::
to_core(). FFI ABI: AdDragParams gains drop_delay_ms (header + repr +
header-compile test). Tests: core threads the value into params and
response and omits the field when unset; FFI maps both optionals.
close-app returned closed:true the instant a graceful quit was *sent*,
while the app was still running behind an unsaved-changes dialog —
a false completion claim. Empirically (NSWorkspace.runningApplications):
a clean quit completes in ~0.2s, a dialog-blocked quit never completes
on its own, and macOS confirms only that the quit request was sent, not
that the app terminated. Verifying by polling would add seconds of
latency on the exact (blocked) case it is meant to catch, so we do not
poll.
Graceful close now reports { method: graceful, requested: true } —
truthful and instant, no closed claim. --force is a synchronous SIGKILL,
so it reports { method: force, requested: true, closed: true }. Callers
needing graceful confirmation observe via list-apps / wait --window and
can drive a save dialog with snapshot + find, which is the agent-native
path.
close-app hardcoded macOS/Unix process names (loginwindow, windowserver,
dock, launchd, finder) inside core, baking platform-specific knowledge
into the platform-agnostic crate. Windows would need csrss.exe/
winlogon.exe, Linux gnome-shell/Xorg. Add PlatformAdapter::
is_protected_process (default denies nothing); the macOS adapter owns its
list with substring matching over display and bundle identifiers. core's
close-app just asks the adapter. Also genericize a macOS-flavored test
fixture string so core carries zero native vocabulary even in tests.
Verified: core has no platform-native references in non-test source, no
cfg(target_os) gates; the only remaining cfg(unix)/libc use is securing
core's own refmap/trace/lock files with non-unix fallbacks.
Replace the canonical-role allow-list (and its hard INVALID_ARGS
rejection) with a tree-derived approach. A role filter that matches
nothing now returns ok with roles_present — the distinct roles actually
in the searched tree — so the caller distinguishes 'none on screen' from
a wrong role name and self-corrects. This needs no central role list: a
role any adapter newly emits surfaces in roles_present automatically,
with nothing to keep in sync across core and the platform crates.
A tiny role-query normalizer keeps the ergonomic win (textarea, textbox,
searchfield fold to textfield, case-insensitive) but never gates or
rejects — it is a synonym shim, not a vocabulary. The macOS role table
returns to its plain match form; the cross-crate canonical-vocabulary
list and its conformance test are gone.
find --role with a role no adapter can emit (textarea, typos) silently
returned ok with zero matches, reading as 'element absent' when the
query could never match. Role queries now resolve through a canonical
vocabulary in core: common text-input aliases (textarea, textbox,
searchfield) normalize to textfield case-insensitively, and unknown
roles fail with INVALID_ARGS carrying details.valid_roles so agents can
self-correct.
The macOS role mapping becomes a sorted single-source table with
binary-search lookup, and a conformance test pins every emitted role
(plus the unknown fallback and the synthesized cell role) to core's
CANONICAL_ROLES — the cross-platform contract Windows/Linux adapters
must map their native vocabularies into, enforced by the same
table + test pattern rather than convention.
Note transient-error retry and last_error timeout detail on wait
--notification; make explicit that agents branch on error.code (message
and suggestion text is informational); warn FFI consumers that
ad_last_error_details may carry on-screen element names, values, and
window titles and should stay out of shared log surfaces.
Fold AXPosition/AXSize and the scrollbar probe into the existing
AXUIElementCopyMultipleAttributeValues batch so tree traversal and the
actionability preflight pay one IPC per element instead of up to four;
A/B benchmark shows strictly-faster snapshots with identical ref counts
and scroll capabilities (Finder 4.5s -> 2.4s same-session, Docker
Desktop parity at 440 refs / 933 scroll-capable nodes).
Consolidate the CLI and FFI ref-action pipelines into one core
execute_resolved path (actionability, tracing, and dispatch semantics
live once; FFI passes a default context). Remove the no-context
execute() shims from is/right-click/snapshot/wait and the test-only
helper shims; every command now takes an explicit CommandContext.
Split the macOS resolver into resolve (orchestration), resolve_search
(candidate collection), and resolve_classify (strict classification),
clearing the 400-LOC ceiling with room to grow.
Notification waits now retry transient baseline failures inside the
timeout budget with the same retryable gate as window/text waits and
report last_error in timeout details instead of aborting on the first
flake; a baseline is never fabricated.
Refmap writes clean up their temp file on failure, stale *.tmp orphans
are swept under the store lock, and save_existing_snapshot re-verifies
snapshot ownership inside the owning store's write lock with bounded
re-discovery before deterministically recreating in the caller's store.
Coverage hardening: zero-budget wait timeout shape, wait --text
--count 0 absence detection, ref-action pipeline call-count guard
(1 resolve / 1 live read / 1 dispatch), duplicate snapshot-id collision
on load, pruned-everywhere recreation, tmp sweep and rename-failure
cleanup, FFI AMBIGUOUS_TARGET last-error code assertion.
Apply verified code-review fixes: ref-action release failures no longer
mask successful action results (prevents double-dispatch on retry);
resolve_element_strict_with_timeout defaults to delegating so strict-only
adapters support wait --element; wait --text reports count only when
--count is requested; latest-refmap refresh logs load failures instead of
silently serving stale refs; InteractionPolicy moved to its own module and
actionability/trace modules scoped pub(crate) per file rules; duplicate
wait test helper extracted to shared support module; timeout error
constructors deduplicated; redaction test covers description; policy
focus-denial path covered; skills document steps array, actionability
details, trace redaction, and batch trace inheritance.
Replace inline // comments with /// docstrings where they carry non-obvious
contract, and add a pre-commit guard so inline comments cannot regress.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Add !t.is_empty() check to prevent false positives in partial title matching
- Add fallback to select the first window with children > 0 using count_children API
- Ensures reliable active window selection in macOS native fullscreen mode
* fix(macos): guard CFArray casts with type-ID check
AXUIElementCopyAttributeValue and CopyMultipleAttributeValues can
return non-array CF types for attributes that are normally arrays
(observed with Mail.app on macOS 26.3 beta). Casting the raw
CFTypeRef to CFArray<CFType> without verifying the type ID is
undefined behaviour and can cause a panic with 'entered unreachable
code' when the resulting fake array is iterated.
Add CFGetTypeID / CFArrayGetTypeID guards in copy_ax_array and
fetch_node_attrs before the unsafe casts. On mismatch, release the
value and fall back gracefully (None / per-attribute fallback path).
Fixes#49
* refactor: extract fetch_node_attrs_slow, remove inline comments
* fix: harden macos accessibility inventory
* fix: guard null ax element refs
* fix: harden macos app inventory fallbacks
* test: cover macos inventory edge cases
* fix: keep macos launch in background
* fix: tighten macos app fallback matching
* fix: harden macos app inventory fallbacks
* refactor: simplify macos inventory cleanup
---------
Co-authored-by: Agent-Mouses <agent-mouses@users.noreply.github.com>
Co-authored-by: Lahfir <nmhlahfir2@gmail.com>
Co-authored-by: Lahfir <70215676+lahfir@users.noreply.github.com>
