Run strict headless click and type against visible background controls, validate only real fixture observables, and emit exact invariant diagnostics without changing fixture behavior.
Keep semantic ref actions strictly headless, make natural headed actions physical-first, and prevent notification observation from opening system UI without headed permission. Gate the native suite on background non-interference and assert delivered mechanisms.
Three failures that tonight's --no-verify checkpoints hid:
- nc_session.rs::nc_pid used Deadline::after (Result<Deadline>) where
operation_deadline yields Result<Instant> — E0308 broke Native check
(macOS/Windows) and the FFI cdylib builds. Reverted to the compiling form;
the speculative deadline floor did not fix NT3 anyway.
- resolve_query_tests.rs hit 403 lines; split the hydration adapters/tests
into resolve_query_hydration_tests.rs (both now well under 400).
- test_harness_contract.py pinned 9 abort_suite sites; the branch legitimately
has 10 (all required-fixture guards). Updated the invariant.
cargo check --all-targets --locked, clippy, fmt, source rules, harness
contract, and 1562 unit tests all green locally.
Remove the scroll-to preamble (the compacted fixture makes scroll-area
directly actionable; raw scroll verified 0->805). Assert disclosure state
by revealed child content instead of AXValue, which SwiftUI DisclosureGroup
does not expose — the product's expand_verified was truthful all along.
Product unchanged since 51c87b2 (1562 unit tests, clippy, release green);
these are test-harness shell edits only.
Containers no longer derive names from descendants (INTERACTIVE_ROLES gate)
and a per-node child-label cap no longer fails whole queries closed — the
root cause of every intermittent e2e abort. Fixture NSWindow double-release
(isReleasedWhenClosed) fixed: zero new crashes. Harness now pins the fixture
window to a deterministic frame on the largest display, clearing the whole
physical-gesture failure family. Selector-timeout diagnostic snapshot budget
raised 75ms->600ms so evidence capture can actually complete. Suite: 73 pass,
7 known remainders (scroll oracle, AE4 assert, three reliability envelope
extractions, sheet confirm lookup, NT1 observation).
Two root-cause fixes surfaced by a real session trace where clicking a
non-clickable group hung the full 5s auto-wait budget and returned an opaque
TIMEOUT whose failing check was unreadable.
Issue 1 (observability): ActionabilityCheck's check-identifier field was named
'name', colliding with the sensitive 'name' key the trace sanitizer redacts
(element accessible names, incl. Occluder.name). So the bounded-vocabulary
check identifier (visible/supported_action/...) was scrubbed from traces. Root
fix: rename the field to 'check' — a token the sanitizer leaves readable —
disambiguating it from Occluder.name, which stays correctly redacted. The
sanitizer keeps failing closed; the non-sensitive field just no longer collides.
Issue 2 (behavior): check_with_stability collapsed every actionability failure
to ACTION_FAILED, so the poll loop retried structurally-permanent failures for
the whole budget then returned a generic TIMEOUT. Root fix: model the intrinsic
transient-vs-terminal nature of each check. The terminal checks (supported_action,
policy, editable) — which waiting cannot heal — now carry their semantic permanent
code (ACTION_NOT_SUPPORTED / POLICY_DENIED); the report derives the error code
from the failing checks. The auto-wait poll loop's existing is_permanent_error
then fails them fast with a precise code, no poll-loop change. Transient checks
(visible/stable/enabled/receives_events) still surface ACTION_FAILED and retry.
Verified on a real app: click of a non-clickable scrollarea now returns
ACTION_NOT_SUPPORTED in 0.09s (was ~5s TIMEOUT) with the trace showing
{check: supported_action, reason: 'Click is not available'}. Full workspace tests
pass, e2e 72/0.
Follow-up polish on the actionability_timeout diagnostics (2nd-pass review P2s):
- Both failure sources now emit a phase-tagged envelope: resolve failures
{"phase":"resolve", code, message, details}, dispatch failures
{"phase":"dispatch", report}. Previously resolve emitted a resolve_error
wrapper while dispatch emitted a raw ActionabilityReport, so a consumer reading
the timeout report couldn't assume one shape.
- The dispatch branch now only overwrites last_report when it actually has
details; a details-less dispatch failure no longer wipes a prior resolve
failure's context. Verified with cargo test --workspace (1186 passed).
- actionability::visibility_check: run HIDDEN/OFFSCREEN state checks BEFORE the
bounds read (P1 #2). Previously bounds=None short-circuited to unknown, so a
hidden/offscreen element whose live bounds read failed slipped the visibility
gate. Regression tests added (hidden/offscreen + bounds=None now fail, not unknown).
- mouse_wheel: add unit tests (P0 #1) mirroring the sibling mouse-command tests —
args reach the adapter unchanged, the scrolled envelope shape, error propagation.
- ref_action_wait poll loop: on a retryable resolve failure, record the resolve
error into last_report so an actionability_timeout on deadline expiry carries
the last STALE_REF/AMBIGUOUS_TARGET context instead of an empty report (P2 #7).
Already-fixed in prior passes (review base predates 640e8e8): StepMechanism serde
tests (#3), clipboard trait removal doc note (#10). Deferred/accepted: test-adapter
boilerplate (#5 -> issue #95), hit-test ancestor + scroll thrashing (#15/#16/#17,
intentional design tradeoffs). Verified with cargo test --workspace (1186 passed).
The CLI command-module contract test (every_core_command_module_is_registered_or_declared_helper)
enumerates commands/*.rs and requires each to be a registered command or a
declared helper. The stale_retry_test_support module added in the #8 dedup
(640e8e8) ends in _test_support (not _tests), so it was neither skipped nor
allow-listed and failed CI. Add it to NON_COMMAND_MODULES alongside the sibling
helpers_test_support/wait_test_support entries. Local gap: this is a binary-crate
test that 'cargo test --lib' skips; 'cargo test --workspace' now green.
Document that hover/drag skip the actionability battery (occlusion-only via
require_receives_events, immediate ACTION_FAILED) while dispatch actions run
check_live's full battery and poll to TIMEOUT — the divergence that produced an
inaccurate skill-doc this cycle. Prevention: validate agent-facing capability
docs against the real per-command code path, not the mental model.
Reconcile four docs/solutions learnings against current code (their core
guidance is unchanged; only cited symbols/signatures/counts drifted):
- playwright-grade-desktop-reliability: add the actionability_timeout trace
kind; fix the check_actionability_with_trace signature (bundled
ResolvedRefAction) and the ref-allocation example (allocate_refs folds in
bounds-hiding; strip_ref_bounds_when_hidden is gone)
- real-app-tests-are-the-platform-adapter-gate: attribute the accessible-name
reducer to tree/builder.rs::accessible_name with element.rs::resolve_element_name
as the thin wrapper (unified in adda4c9)
- macos-gesture-headless-capability: 54 -> 58 commands; double_click now returns
Vec<ActionStep> tagged SemanticApi/PhysicalSynthetic
- exhaustiveness-guards: context.request( -> context.request_base(
Triage + validate (the review's line numbers were bogus diff-offset
artifacts, so every finding was checked against the real code) then fix the
real ones. 4 findings were false-positives (#2 core-internal fn, #5
VisibilityEvidence not in the actionability gate, #15 supertrait forward-design,
#20 as_str duplication), 2 were deferred as disproportionate (see below).
Verified by an adversarial code-read review + full gate: fmt, clippy
-D warnings, 1006 lib + 7 ABI tests, core isolation, 1.6MB, e2e 72/0, FFI
header regenerated with matching offset asserts.
- scroll gate: maybe_scroll_into_view now reads live state/bounds
(fail-open to the snapshot entry on capability-less adapters) so an element
that scrolled off-screen after the snapshot is still scrolled into view
- auto-wait: is_permanent_error now treats ErrorCode::Internal as permanent so
an internal dispatch error fails fast instead of retrying the whole budget
- FFI header: document ad_execute_by_ref's 5000ms auto-wait default (and the
timeout=0 single-shot escape hatch); correct the stale envelope version
2.0 -> 2.1; add the 22 per-field offsetof static asserts for AdRefEntry
- skill docs: document --modifiers on the mouse commands, the receives_events
occlusion check + occluder detail, the implicit scroll-into-view preflight,
and --timeout-ms — and correct the actionability section to reflect that
hover/drag run only the receives_events check (not the full battery) and
fail fast with ACTION_FAILED rather than polling to TIMEOUT
- adapter: /// note that get_clipboard/set_clipboard were removed pre-1.0 for
the typed content methods (C ABI unaffected)
- tests: StepMechanism serde roundtrip, ActionRequest legacy-no-timeout_ms
deserialization, execute_by_ref unit tests; dedup the byte-identical
StaleThenOkAdapter retry-counter into stale_retry_test_support
Deferred (disproportionate blast radius, tracked as follow-up issues):
process-state PID-reuse start-time corroboration (needs plumbing through the
already-large RefEntry + snapshot + platform adapter); a stub_ops! macro +
84-site test-adapter retrofit (large mechanical churn on the deliberate U0
four-trait split).
Remediate the validated PR #93 review findings. Each was triaged and
validated against the real code (0 false-positives; 4 deferred as
design/maintainability follow-ups), fixed on disjoint file sets, then
confirmed by an adversarial code-read review and the full gate (fmt, clippy
-D warnings, 996 unit tests, core isolation, 1.6MB binary, FFI codegen
drift-stable, e2e 72/0). Trace changes were additionally verified on a real
app: a ref-action click now emits ref.resolve.start/entry/ok, and a stale ref
emits ref.resolve.start/error(STALE_REF) with no dangling start.
- ref actions: emit ref.resolve.start/entry/error via a shared load_ref_entry
owner, so element-mutating actions trace resolution identically to get/is
(previously they emitted no start/entry, then a dangling start once added)
- scroll: surface a ref.scroll_into_view.error trace instead of silently
swallowing a failed scroll_into_view (non-blocking; the actionability check
still gates the action)
- actionability: attach the actionability report to the success trace so an
Unknown receives_events verdict is visible instead of vanishing
- launch: stop interpolating the raw app identifier into validate_app_identifier's
error message; carry it in redacted details.app_name (trace-leak fix)
- process_state: preserve original error details when upgrading to APP_UNRESPONSIVE
- execute_by_ref / resolve_point_with_wait: fold 7-arg signatures into config
structs (ExecuteByRefArgs; reuse PointResolveArgs) under the 5-param limit
- skills/README: correct the stale "56 commands" to 58
- tests: ProcessState::Crashed serde + Crashed/AppNotFound enrichment coverage,
requires_scroll_into_view exhaustiveness guard, normalize_action_timeout_ms(0)
guard, validate_app_identifier accept/reject + leak-regression, value-based
legacy ActionStep JSON comparison; split the process-state test file to stay
under the 400 LOC limit
Remediate the validated findings from the whole-branch review. Each was
triaged, validated against the real code, fixed on a disjoint file set, then
confirmed by an adversarial code-read review and the full gate (fmt, clippy
-D warnings, 985 unit tests, core isolation, 1.6MB binary, e2e 72/0). The
ref.resolve trace restoration was additionally verified on a real app (Finder
hover under a traced session emits start/entry/ok again).
- signals: propagate app-enumeration failures in the signal baseline instead of
swallowing them with unwrap_or_default, mirroring list_windows_impl
- image: validate the PNG signature and IHDR chunk in one shared core parser
(parse_png_dimensions), dedup three hand-rolled copies, and reject non-PNG
--image input instead of tagging arbitrary bytes as a zero-dimensioned PNG
- roles: delete the dead Role enum and inline is_interactive_role as a direct
INTERACTIVE_ROLES membership check, keeping the vocabulary-contract test
- helpers: cap each ref-resolve attempt to the wait deadline through the single
traced resolver, so budgeted hover/drag can no longer spend the whole budget
on one slow attempt and once again emit ref.resolve.* trace events
- actionability: run the hit_test occlusion gate even when get_live_element is
unsupported, so occlusion is checked on hit_test-capable adapters
- launch: share the app-identifier traversal guard across both launch paths
- window_resolve: fail closed when the window-title fallback is ambiguous
(resolve only when exactly one window matches the title)
- main: surface stdout write/flush failures from the JSON envelope instead of
reporting success, exiting with a distinct code on a broken pipe
- cli_args: add a one-to-one Surface<->SnapshotSurface parity tripwire test
Addresses the code review's remaining validated findings (test + robustness
hardening; no P0 correctness left after the name and window-bridge fixes):
- find --window-id: MockAdapter test (two windows, distinct trees) proving the
flag scopes the search to the requested window and a swap with app would fail.
- WindowScope serde: batch-JSON back-compat + typo-rejection tests for the
flattened app/window_id group on find/snapshot/screenshot.
- Offscreen bounds: extracted window_bounds_for_children as a pure helper and
restored the AXWindow-own-bounds / inherited-fallback unit coverage that was
lost when the live query matcher was deleted.
- Real-app guards: the #[ignore] snapshot_test.rs guards now print a SKIP reason
before each early return, so a no-op run is visible instead of a silent pass.
- Hit-test occluder naming now applies promoted_label.or_else(resolve_element_name),
matching the strict resolver and ambiguity classifier.
- Hover e2e asserts a clean (non-"hovered") baseline after the cursor reset.
Verified: e2e 71/0, Finder sidebar cells still re-resolve, clippy + workspace
tests green.
The earlier window fix replaced the nonexistent AXWindowNumber attribute with
_AXUIElementGetWindow in window_resolve.rs only; code review found three more
live sites still reading AXWindowNumber (which AppKit/SwiftUI never publish, so
it always returns None):
- resolve_roots.rs window_by_number: scope_verified could never become true, so
strict ref re-resolution silently degraded to bounds-hash-only matching and
failed closed (STALE_REF) on any window move/resize.
- window_inventory.rs: the AX-fallback path stamped every window id "w-0"
(collisions), and focus detection fell back to title-equality — wrong for any
app with two or more same-titled windows.
Extracted ax_window_id (the _AXUIElementGetWindow bridge) as the single shared
owner and routed all three sites through it; removed the now-dead copy_i64_attr
helper entirely (its only callers were the AXWindowNumber reads); switched the
bridge's success check to kAXErrorSuccess; and added a verified-title fallback
so a transient _AXUIElementGetWindow error on an already CG-verified window
retries by title instead of returning a false WINDOW_NOT_FOUND.
Verified: list-windows focus detection returns non-zero ids, e2e 71/0.
Code review found the STALE_REF name-divergence class was re-introduced: the
snapshot builder stores a ref's name via its own chain (title -> description ->
static value -> label_from_children child text), but resolve_element_name — used
by strict ref re-resolution — dropped the child-label rung and trimmed blanks
differently. So an interactive element named only by descendant text (Finder /
Mail / System Settings sidebar cells) or by a whitespace/blank title stored one
name and recomputed another, failing identity_matches -> STALE_REF on
click/type/get. Confirmed: 5/5 Finder sidebar cells returned STALE_REF.
- One shared reducer `builder::accessible_name` (title -> description ->
static-text value -> aggregated child label, each trimmed and blank-as-absent),
with the own-text portion factored into the pure, unit-testable
`reduce_text_name`. Both the snapshot builder and resolve_element_name reduce
through it, so a stored ref name always equals what the resolver recomputes.
- Deleted the now-single-producer/single-consumer NameEvidence indirection
(crates/core/src/accname.rs, crates/macos/src/tree/name_evidence.rs) and the
now-dead label_from_child_attrs.
- Added reduce_text_name unit tests covering the rung precedence and the
blank/whitespace handling that accname_tests used to guard.
Verified: 5/5 Finder sidebar cells now re-resolve, e2e 71/0, clippy clean,
workspace tests green.
Records the durable learning from the foundation-branch remediation: green
mock/stub unit CI cannot cover platform-adapter mechanics (window bridge,
accessible-name computation), so the e2e and #[ignore] real-app tests are the
mandatory gate before merging adapter changes. Cross-references the existing
playwright-grade reliability contract.
These are the safeguards a MockAdapter cannot provide: they drive the release
binary against real macOS apps, exercising the AX plumbing the mock stubs out.
- snapshot_test.rs: three #[ignore] real-app guards, each fails closed on a bug
this branch shipped green:
- a window id from list-windows must resolve back through snapshot (the
AX-to-CGWindowID bridge);
- an element found by role must be findable by the accessible name it reports
(name-computation consistency across builder / matcher / resolver);
- a ref from find must re-resolve through get (strict ref identity).
- e2e run.sh: reset the cursor away before the headed hover assertion so
onHover fires on a genuine mouse entry regardless of where the prior test
left the cursor.
The foundation-contract branch passed unit CI but broke observation and
interaction against real macOS apps. The unit suite runs on an in-memory
MockAdapter that cannot exercise the platform's AX plumbing, so a batch of
adapter regressions shipped green; running the live e2e surfaced them.
- Window resolution: match AX windows to their CGWindowID via the private
but stable _AXUIElementGetWindow bridge instead of the nonexistent
AXWindowNumber attribute, which had made snapshot/find return
WINDOW_NOT_FOUND for every app. Verified across single- and multi-window
apps.
- Accessible name: collapse the builder, strict resolver, hit-test, and
ambiguity classifier onto one resolve_element_name (title -> description ->
static-text value) so a ref's stored name always matches what the resolver
recomputes. Fixes STALE_REF on elements named via a non-title rung (e.g.
textfields labelled through AXDescription).
- find: route through the single snapshot matcher (full traversal, correct
names, real refs) and drop the redundant live resolve_query path that was
correlated to the snapshot by index.
- find --window-id: scope a search to one window, via a shared WindowScope arg
group flattened into snapshot/find/screenshot. Ref-based and keyboard
commands intentionally omit it -- a ref already carries its source window and
keyboard input targets the focused window.
- Trace: restore ref.resolve.ok on successful ref resolution.
- Remove the now-dead resolve_query, the macOS live query matcher,
get_live_name_evidence, and the superseded accname compute_name reduction.
Verified by the live e2e (71/71) and the full unit/clippy/fmt/isolation gates.
Splits every file over the 400 LOC cap that this branch's changes pushed
past it, and flattens the god-object arg/param structs the ledger flagged,
all behavior-preserving.
LOC splits:
- crates/macos/src/adapter.rs (437) -> adapter.rs (struct + shared helper)
plus adapter_observation.rs/adapter_actions.rs/adapter_input.rs/
adapter_system.rs, one per PlatformAdapter capability trait impl
- crates/macos/src/actions/chain.rs (458) -> chain.rs (orchestration) plus
chain_step_exec.rs (per-step dispatch) and chain_value_write.rs (verified
value writes); also moves its inline mod tests {} to sibling chain_tests.rs
and chain_value_write_tests.rs per the no-inline-tests contract
- src/dispatch/mod.rs (453) -> mod.rs (routing only) plus observation.rs,
interaction.rs, keyboard_mouse.rs, app_window.rs, clipboard.rs, system.rs,
mirroring its own notifications/parse/session/trace submodule split
- crates/core/src/commands/wait_tests.rs (422) -> notification-scenario
tests stay, text/menu-scenario tests move to wait_scenario_tests.rs; the
shared wait_args() baseline moves to the established wait_test_support.rs
- crates/core/src/commands/helpers_ref_action_tests.rs (489) -> split by
scenario into helpers_ref_action_dispatch_tests.rs and
helpers_ref_action_wait_tests.rs
God-object fixes (CLAUDE.md: no struct >7 fields, no fn >5 params):
- src/cli_args/system.rs WaitModeArgs (9 fields) -> WaitEventArgs{event,
window_id} flattens onto WaitArgs as a sibling of mode/predicate (not
nested inside WaitModeArgs): serde's #[serde(flatten)] cannot coexist
with #[serde(deny_unknown_fields)] on a struct that is both a flatten
target and a flatten owner, verified with an isolated repro. WaitModeArgs
now sits at exactly 7 fields
- src/cli_args/mod.rs and crates/core/src/commands/find.rs FindArgs (14
fields, both the CLI struct and its core twin) -> FindFilterArgs (7:
role/name/value/text/description/native_id/exact) + FindSelectionArgs (5:
count/first/last/nth/limit), both flattened back onto a 4-field FindArgs
- crates/core/src/locator.rs LocatorQuery (10 fields) -> IdentityPredicate
(role/name/description/native_id/value) + ContainmentPredicate
(has/has_not), flattened back on; call sites in commands/query.rs and
macos/tree/query.rs updated
- crates/core/src/ref_action_wait.rs execute_with_auto_wait/
execute_single_shot/execute_poll_loop (6-7 params) -> RefActionWaitCtx
{adapter, entry, ref_id, context}, mirroring ref_action::ResolvedRefAction
All #[command(flatten)]/#[serde(flatten)] regroupings preserve the exact
CLI flag surface and flat batch-JSON wire shape; verified via clap
try_parse_from and serde_json::from_value round-trip tests, plus a manual
end-to-end run of the built binary through find/wait --event --window-id.
Extracts a single resolve_within_deadline poll-resolve helper shared by
ref_action_wait's auto-wait loop and wait_element's element wait loop
(F13), so the 750ms per-attempt cap and deadline math has one owner;
each caller keeps its own retry classification and side effects
(LatestRefCache refresh, ElementNotFound retry) as before.
Deletes classify_query_result/ambiguous_candidate_summaries/
QueryCandidateSummary from locator.rs (zero callers) and simplifies
find.rs's materialize_match to its snapshot-clone, dropping the
index/tree/query params it silently discarded. Chosen over wiring the
live path to build responses from live handles, which would change
find's --count/--last semantics.
Role::is_interactive now delegates to roles::INTERACTIVE_ROLES instead
of re-encoding the 16 role names (F22).
Consolidates three byte-identical NoopAdapter/StubSystemOps blanket
test doubles (src/batch/tests.rs, src/dispatch/notifications.rs,
tests/conformance/window_identity_contract.rs) into one file at
tests/support/noop_ops.rs, included via #[path] since the binary's own
unit tests and the standalone conformance integration crate cannot
share a Rust module across the crate boundary.
Extracts locate_verified_record in window_resolve.rs (parse -> find ->
verify, previously duplicated by resolve_window_strict and
resolve_window_element_strict) and compute_readonly in element.rs
(previously duplicated identically by fetch_node_attrs and
fetch_node_attrs_slow).
Hoists element.rs's 18-entry CFString attribute-name array behind a
thread-local cache (CFString/CFArray aren't Send+Sync so a process-wide
LazyLock isn't viable) instead of rebuilding it on every node. Makes
query.rs's collect_matches short-circuit on the cheap role check before
building live-tree state/children context for elements that can never
match.
