Remove the scroll-to preamble (the compacted fixture makes scroll-area
directly actionable; raw scroll verified 0->805). Assert disclosure state
by revealed child content instead of AXValue, which SwiftUI DisclosureGroup
does not expose — the product's expand_verified was truthful all along.
Product unchanged since 51c87b2 (1562 unit tests, clippy, release green);
these are test-harness shell edits only.
Containers no longer derive names from descendants (INTERACTIVE_ROLES gate)
and a per-node child-label cap no longer fails whole queries closed — the
root cause of every intermittent e2e abort. Fixture NSWindow double-release
(isReleasedWhenClosed) fixed: zero new crashes. Harness now pins the fixture
window to a deterministic frame on the largest display, clearing the whole
physical-gesture failure family. Selector-timeout diagnostic snapshot budget
raised 75ms->600ms so evidence capture can actually complete. Suite: 73 pass,
7 known remainders (scroll oracle, AE4 assert, three reliability envelope
extractions, sheet confirm lookup, NT1 observation).
Two root-cause fixes surfaced by a real session trace where clicking a
non-clickable group hung the full 5s auto-wait budget and returned an opaque
TIMEOUT whose failing check was unreadable.
Issue 1 (observability): ActionabilityCheck's check-identifier field was named
'name', colliding with the sensitive 'name' key the trace sanitizer redacts
(element accessible names, incl. Occluder.name). So the bounded-vocabulary
check identifier (visible/supported_action/...) was scrubbed from traces. Root
fix: rename the field to 'check' — a token the sanitizer leaves readable —
disambiguating it from Occluder.name, which stays correctly redacted. The
sanitizer keeps failing closed; the non-sensitive field just no longer collides.
Issue 2 (behavior): check_with_stability collapsed every actionability failure
to ACTION_FAILED, so the poll loop retried structurally-permanent failures for
the whole budget then returned a generic TIMEOUT. Root fix: model the intrinsic
transient-vs-terminal nature of each check. The terminal checks (supported_action,
policy, editable) — which waiting cannot heal — now carry their semantic permanent
code (ACTION_NOT_SUPPORTED / POLICY_DENIED); the report derives the error code
from the failing checks. The auto-wait poll loop's existing is_permanent_error
then fails them fast with a precise code, no poll-loop change. Transient checks
(visible/stable/enabled/receives_events) still surface ACTION_FAILED and retry.
Verified on a real app: click of a non-clickable scrollarea now returns
ACTION_NOT_SUPPORTED in 0.09s (was ~5s TIMEOUT) with the trace showing
{check: supported_action, reason: 'Click is not available'}. Full workspace tests
pass, e2e 72/0.
Follow-up polish on the actionability_timeout diagnostics (2nd-pass review P2s):
- Both failure sources now emit a phase-tagged envelope: resolve failures
{"phase":"resolve", code, message, details}, dispatch failures
{"phase":"dispatch", report}. Previously resolve emitted a resolve_error
wrapper while dispatch emitted a raw ActionabilityReport, so a consumer reading
the timeout report couldn't assume one shape.
- The dispatch branch now only overwrites last_report when it actually has
details; a details-less dispatch failure no longer wipes a prior resolve
failure's context. Verified with cargo test --workspace (1186 passed).
- actionability::visibility_check: run HIDDEN/OFFSCREEN state checks BEFORE the
bounds read (P1 #2). Previously bounds=None short-circuited to unknown, so a
hidden/offscreen element whose live bounds read failed slipped the visibility
gate. Regression tests added (hidden/offscreen + bounds=None now fail, not unknown).
- mouse_wheel: add unit tests (P0 #1) mirroring the sibling mouse-command tests —
args reach the adapter unchanged, the scrolled envelope shape, error propagation.
- ref_action_wait poll loop: on a retryable resolve failure, record the resolve
error into last_report so an actionability_timeout on deadline expiry carries
the last STALE_REF/AMBIGUOUS_TARGET context instead of an empty report (P2 #7).
Already-fixed in prior passes (review base predates 640e8e8): StepMechanism serde
tests (#3), clipboard trait removal doc note (#10). Deferred/accepted: test-adapter
boilerplate (#5 -> issue #95), hit-test ancestor + scroll thrashing (#15/#16/#17,
intentional design tradeoffs). Verified with cargo test --workspace (1186 passed).
The CLI command-module contract test (every_core_command_module_is_registered_or_declared_helper)
enumerates commands/*.rs and requires each to be a registered command or a
declared helper. The stale_retry_test_support module added in the #8 dedup
(640e8e8) ends in _test_support (not _tests), so it was neither skipped nor
allow-listed and failed CI. Add it to NON_COMMAND_MODULES alongside the sibling
helpers_test_support/wait_test_support entries. Local gap: this is a binary-crate
test that 'cargo test --lib' skips; 'cargo test --workspace' now green.
Document that hover/drag skip the actionability battery (occlusion-only via
require_receives_events, immediate ACTION_FAILED) while dispatch actions run
check_live's full battery and poll to TIMEOUT — the divergence that produced an
inaccurate skill-doc this cycle. Prevention: validate agent-facing capability
docs against the real per-command code path, not the mental model.
Reconcile four docs/solutions learnings against current code (their core
guidance is unchanged; only cited symbols/signatures/counts drifted):
- playwright-grade-desktop-reliability: add the actionability_timeout trace
kind; fix the check_actionability_with_trace signature (bundled
ResolvedRefAction) and the ref-allocation example (allocate_refs folds in
bounds-hiding; strip_ref_bounds_when_hidden is gone)
- real-app-tests-are-the-platform-adapter-gate: attribute the accessible-name
reducer to tree/builder.rs::accessible_name with element.rs::resolve_element_name
as the thin wrapper (unified in adda4c9)
- macos-gesture-headless-capability: 54 -> 58 commands; double_click now returns
Vec<ActionStep> tagged SemanticApi/PhysicalSynthetic
- exhaustiveness-guards: context.request( -> context.request_base(
Triage + validate (the review's line numbers were bogus diff-offset
artifacts, so every finding was checked against the real code) then fix the
real ones. 4 findings were false-positives (#2 core-internal fn, #5
VisibilityEvidence not in the actionability gate, #15 supertrait forward-design,
#20 as_str duplication), 2 were deferred as disproportionate (see below).
Verified by an adversarial code-read review + full gate: fmt, clippy
-D warnings, 1006 lib + 7 ABI tests, core isolation, 1.6MB, e2e 72/0, FFI
header regenerated with matching offset asserts.
- scroll gate: maybe_scroll_into_view now reads live state/bounds
(fail-open to the snapshot entry on capability-less adapters) so an element
that scrolled off-screen after the snapshot is still scrolled into view
- auto-wait: is_permanent_error now treats ErrorCode::Internal as permanent so
an internal dispatch error fails fast instead of retrying the whole budget
- FFI header: document ad_execute_by_ref's 5000ms auto-wait default (and the
timeout=0 single-shot escape hatch); correct the stale envelope version
2.0 -> 2.1; add the 22 per-field offsetof static asserts for AdRefEntry
- skill docs: document --modifiers on the mouse commands, the receives_events
occlusion check + occluder detail, the implicit scroll-into-view preflight,
and --timeout-ms — and correct the actionability section to reflect that
hover/drag run only the receives_events check (not the full battery) and
fail fast with ACTION_FAILED rather than polling to TIMEOUT
- adapter: /// note that get_clipboard/set_clipboard were removed pre-1.0 for
the typed content methods (C ABI unaffected)
- tests: StepMechanism serde roundtrip, ActionRequest legacy-no-timeout_ms
deserialization, execute_by_ref unit tests; dedup the byte-identical
StaleThenOkAdapter retry-counter into stale_retry_test_support
Deferred (disproportionate blast radius, tracked as follow-up issues):
process-state PID-reuse start-time corroboration (needs plumbing through the
already-large RefEntry + snapshot + platform adapter); a stub_ops! macro +
84-site test-adapter retrofit (large mechanical churn on the deliberate U0
four-trait split).
Remediate the validated PR #93 review findings. Each was triaged and
validated against the real code (0 false-positives; 4 deferred as
design/maintainability follow-ups), fixed on disjoint file sets, then
confirmed by an adversarial code-read review and the full gate (fmt, clippy
-D warnings, 996 unit tests, core isolation, 1.6MB binary, FFI codegen
drift-stable, e2e 72/0). Trace changes were additionally verified on a real
app: a ref-action click now emits ref.resolve.start/entry/ok, and a stale ref
emits ref.resolve.start/error(STALE_REF) with no dangling start.
- ref actions: emit ref.resolve.start/entry/error via a shared load_ref_entry
owner, so element-mutating actions trace resolution identically to get/is
(previously they emitted no start/entry, then a dangling start once added)
- scroll: surface a ref.scroll_into_view.error trace instead of silently
swallowing a failed scroll_into_view (non-blocking; the actionability check
still gates the action)
- actionability: attach the actionability report to the success trace so an
Unknown receives_events verdict is visible instead of vanishing
- launch: stop interpolating the raw app identifier into validate_app_identifier's
error message; carry it in redacted details.app_name (trace-leak fix)
- process_state: preserve original error details when upgrading to APP_UNRESPONSIVE
- execute_by_ref / resolve_point_with_wait: fold 7-arg signatures into config
structs (ExecuteByRefArgs; reuse PointResolveArgs) under the 5-param limit
- skills/README: correct the stale "56 commands" to 58
- tests: ProcessState::Crashed serde + Crashed/AppNotFound enrichment coverage,
requires_scroll_into_view exhaustiveness guard, normalize_action_timeout_ms(0)
guard, validate_app_identifier accept/reject + leak-regression, value-based
legacy ActionStep JSON comparison; split the process-state test file to stay
under the 400 LOC limit
Remediate the validated findings from the whole-branch review. Each was
triaged, validated against the real code, fixed on a disjoint file set, then
confirmed by an adversarial code-read review and the full gate (fmt, clippy
-D warnings, 985 unit tests, core isolation, 1.6MB binary, e2e 72/0). The
ref.resolve trace restoration was additionally verified on a real app (Finder
hover under a traced session emits start/entry/ok again).
- signals: propagate app-enumeration failures in the signal baseline instead of
swallowing them with unwrap_or_default, mirroring list_windows_impl
- image: validate the PNG signature and IHDR chunk in one shared core parser
(parse_png_dimensions), dedup three hand-rolled copies, and reject non-PNG
--image input instead of tagging arbitrary bytes as a zero-dimensioned PNG
- roles: delete the dead Role enum and inline is_interactive_role as a direct
INTERACTIVE_ROLES membership check, keeping the vocabulary-contract test
- helpers: cap each ref-resolve attempt to the wait deadline through the single
traced resolver, so budgeted hover/drag can no longer spend the whole budget
on one slow attempt and once again emit ref.resolve.* trace events
- actionability: run the hit_test occlusion gate even when get_live_element is
unsupported, so occlusion is checked on hit_test-capable adapters
- launch: share the app-identifier traversal guard across both launch paths
- window_resolve: fail closed when the window-title fallback is ambiguous
(resolve only when exactly one window matches the title)
- main: surface stdout write/flush failures from the JSON envelope instead of
reporting success, exiting with a distinct code on a broken pipe
- cli_args: add a one-to-one Surface<->SnapshotSurface parity tripwire test
Addresses the code review's remaining validated findings (test + robustness
hardening; no P0 correctness left after the name and window-bridge fixes):
- find --window-id: MockAdapter test (two windows, distinct trees) proving the
flag scopes the search to the requested window and a swap with app would fail.
- WindowScope serde: batch-JSON back-compat + typo-rejection tests for the
flattened app/window_id group on find/snapshot/screenshot.
- Offscreen bounds: extracted window_bounds_for_children as a pure helper and
restored the AXWindow-own-bounds / inherited-fallback unit coverage that was
lost when the live query matcher was deleted.
- Real-app guards: the #[ignore] snapshot_test.rs guards now print a SKIP reason
before each early return, so a no-op run is visible instead of a silent pass.
- Hit-test occluder naming now applies promoted_label.or_else(resolve_element_name),
matching the strict resolver and ambiguity classifier.
- Hover e2e asserts a clean (non-"hovered") baseline after the cursor reset.
Verified: e2e 71/0, Finder sidebar cells still re-resolve, clippy + workspace
tests green.
The earlier window fix replaced the nonexistent AXWindowNumber attribute with
_AXUIElementGetWindow in window_resolve.rs only; code review found three more
live sites still reading AXWindowNumber (which AppKit/SwiftUI never publish, so
it always returns None):
- resolve_roots.rs window_by_number: scope_verified could never become true, so
strict ref re-resolution silently degraded to bounds-hash-only matching and
failed closed (STALE_REF) on any window move/resize.
- window_inventory.rs: the AX-fallback path stamped every window id "w-0"
(collisions), and focus detection fell back to title-equality — wrong for any
app with two or more same-titled windows.
Extracted ax_window_id (the _AXUIElementGetWindow bridge) as the single shared
owner and routed all three sites through it; removed the now-dead copy_i64_attr
helper entirely (its only callers were the AXWindowNumber reads); switched the
bridge's success check to kAXErrorSuccess; and added a verified-title fallback
so a transient _AXUIElementGetWindow error on an already CG-verified window
retries by title instead of returning a false WINDOW_NOT_FOUND.
Verified: list-windows focus detection returns non-zero ids, e2e 71/0.
Code review found the STALE_REF name-divergence class was re-introduced: the
snapshot builder stores a ref's name via its own chain (title -> description ->
static value -> label_from_children child text), but resolve_element_name — used
by strict ref re-resolution — dropped the child-label rung and trimmed blanks
differently. So an interactive element named only by descendant text (Finder /
Mail / System Settings sidebar cells) or by a whitespace/blank title stored one
name and recomputed another, failing identity_matches -> STALE_REF on
click/type/get. Confirmed: 5/5 Finder sidebar cells returned STALE_REF.
- One shared reducer `builder::accessible_name` (title -> description ->
static-text value -> aggregated child label, each trimmed and blank-as-absent),
with the own-text portion factored into the pure, unit-testable
`reduce_text_name`. Both the snapshot builder and resolve_element_name reduce
through it, so a stored ref name always equals what the resolver recomputes.
- Deleted the now-single-producer/single-consumer NameEvidence indirection
(crates/core/src/accname.rs, crates/macos/src/tree/name_evidence.rs) and the
now-dead label_from_child_attrs.
- Added reduce_text_name unit tests covering the rung precedence and the
blank/whitespace handling that accname_tests used to guard.
Verified: 5/5 Finder sidebar cells now re-resolve, e2e 71/0, clippy clean,
workspace tests green.
Records the durable learning from the foundation-branch remediation: green
mock/stub unit CI cannot cover platform-adapter mechanics (window bridge,
accessible-name computation), so the e2e and #[ignore] real-app tests are the
mandatory gate before merging adapter changes. Cross-references the existing
playwright-grade reliability contract.
These are the safeguards a MockAdapter cannot provide: they drive the release
binary against real macOS apps, exercising the AX plumbing the mock stubs out.
- snapshot_test.rs: three #[ignore] real-app guards, each fails closed on a bug
this branch shipped green:
- a window id from list-windows must resolve back through snapshot (the
AX-to-CGWindowID bridge);
- an element found by role must be findable by the accessible name it reports
(name-computation consistency across builder / matcher / resolver);
- a ref from find must re-resolve through get (strict ref identity).
- e2e run.sh: reset the cursor away before the headed hover assertion so
onHover fires on a genuine mouse entry regardless of where the prior test
left the cursor.
The foundation-contract branch passed unit CI but broke observation and
interaction against real macOS apps. The unit suite runs on an in-memory
MockAdapter that cannot exercise the platform's AX plumbing, so a batch of
adapter regressions shipped green; running the live e2e surfaced them.
- Window resolution: match AX windows to their CGWindowID via the private
but stable _AXUIElementGetWindow bridge instead of the nonexistent
AXWindowNumber attribute, which had made snapshot/find return
WINDOW_NOT_FOUND for every app. Verified across single- and multi-window
apps.
- Accessible name: collapse the builder, strict resolver, hit-test, and
ambiguity classifier onto one resolve_element_name (title -> description ->
static-text value) so a ref's stored name always matches what the resolver
recomputes. Fixes STALE_REF on elements named via a non-title rung (e.g.
textfields labelled through AXDescription).
- find: route through the single snapshot matcher (full traversal, correct
names, real refs) and drop the redundant live resolve_query path that was
correlated to the snapshot by index.
- find --window-id: scope a search to one window, via a shared WindowScope arg
group flattened into snapshot/find/screenshot. Ref-based and keyboard
commands intentionally omit it -- a ref already carries its source window and
keyboard input targets the focused window.
- Trace: restore ref.resolve.ok on successful ref resolution.
- Remove the now-dead resolve_query, the macOS live query matcher,
get_live_name_evidence, and the superseded accname compute_name reduction.
Verified by the live e2e (71/71) and the full unit/clippy/fmt/isolation gates.
Splits every file over the 400 LOC cap that this branch's changes pushed
past it, and flattens the god-object arg/param structs the ledger flagged,
all behavior-preserving.
LOC splits:
- crates/macos/src/adapter.rs (437) -> adapter.rs (struct + shared helper)
plus adapter_observation.rs/adapter_actions.rs/adapter_input.rs/
adapter_system.rs, one per PlatformAdapter capability trait impl
- crates/macos/src/actions/chain.rs (458) -> chain.rs (orchestration) plus
chain_step_exec.rs (per-step dispatch) and chain_value_write.rs (verified
value writes); also moves its inline mod tests {} to sibling chain_tests.rs
and chain_value_write_tests.rs per the no-inline-tests contract
- src/dispatch/mod.rs (453) -> mod.rs (routing only) plus observation.rs,
interaction.rs, keyboard_mouse.rs, app_window.rs, clipboard.rs, system.rs,
mirroring its own notifications/parse/session/trace submodule split
- crates/core/src/commands/wait_tests.rs (422) -> notification-scenario
tests stay, text/menu-scenario tests move to wait_scenario_tests.rs; the
shared wait_args() baseline moves to the established wait_test_support.rs
- crates/core/src/commands/helpers_ref_action_tests.rs (489) -> split by
scenario into helpers_ref_action_dispatch_tests.rs and
helpers_ref_action_wait_tests.rs
God-object fixes (CLAUDE.md: no struct >7 fields, no fn >5 params):
- src/cli_args/system.rs WaitModeArgs (9 fields) -> WaitEventArgs{event,
window_id} flattens onto WaitArgs as a sibling of mode/predicate (not
nested inside WaitModeArgs): serde's #[serde(flatten)] cannot coexist
with #[serde(deny_unknown_fields)] on a struct that is both a flatten
target and a flatten owner, verified with an isolated repro. WaitModeArgs
now sits at exactly 7 fields
- src/cli_args/mod.rs and crates/core/src/commands/find.rs FindArgs (14
fields, both the CLI struct and its core twin) -> FindFilterArgs (7:
role/name/value/text/description/native_id/exact) + FindSelectionArgs (5:
count/first/last/nth/limit), both flattened back onto a 4-field FindArgs
- crates/core/src/locator.rs LocatorQuery (10 fields) -> IdentityPredicate
(role/name/description/native_id/value) + ContainmentPredicate
(has/has_not), flattened back on; call sites in commands/query.rs and
macos/tree/query.rs updated
- crates/core/src/ref_action_wait.rs execute_with_auto_wait/
execute_single_shot/execute_poll_loop (6-7 params) -> RefActionWaitCtx
{adapter, entry, ref_id, context}, mirroring ref_action::ResolvedRefAction
All #[command(flatten)]/#[serde(flatten)] regroupings preserve the exact
CLI flag surface and flat batch-JSON wire shape; verified via clap
try_parse_from and serde_json::from_value round-trip tests, plus a manual
end-to-end run of the built binary through find/wait --event --window-id.
Extracts a single resolve_within_deadline poll-resolve helper shared by
ref_action_wait's auto-wait loop and wait_element's element wait loop
(F13), so the 750ms per-attempt cap and deadline math has one owner;
each caller keeps its own retry classification and side effects
(LatestRefCache refresh, ElementNotFound retry) as before.
Deletes classify_query_result/ambiguous_candidate_summaries/
QueryCandidateSummary from locator.rs (zero callers) and simplifies
find.rs's materialize_match to its snapshot-clone, dropping the
index/tree/query params it silently discarded. Chosen over wiring the
live path to build responses from live handles, which would change
find's --count/--last semantics.
Role::is_interactive now delegates to roles::INTERACTIVE_ROLES instead
of re-encoding the 16 role names (F22).
Consolidates three byte-identical NoopAdapter/StubSystemOps blanket
test doubles (src/batch/tests.rs, src/dispatch/notifications.rs,
tests/conformance/window_identity_contract.rs) into one file at
tests/support/noop_ops.rs, included via #[path] since the binary's own
unit tests and the standalone conformance integration crate cannot
share a Rust module across the crate boundary.
Extracts locate_verified_record in window_resolve.rs (parse -> find ->
verify, previously duplicated by resolve_window_strict and
resolve_window_element_strict) and compute_readonly in element.rs
(previously duplicated identically by fetch_node_attrs and
fetch_node_attrs_slow).
Hoists element.rs's 18-entry CFString attribute-name array behind a
thread-local cache (CFString/CFArray aren't Send+Sync so a process-wide
LazyLock isn't viable) instead of rebuilding it on every node. Makes
query.rs's collect_matches short-circuit on the cheap role check before
building live-tree state/children context for elements that can never
match.
Replace the tautological state-vocabulary conformance tests (they rebuilt
their expected set from the same state:: constants they checked, so they
could never fail) with real guards: a source-scan test in core that reads
the actual state-consuming call sites and flags bare literals that bypass
state::, and a macOS test that drives the real states_from_element producer
over representative inputs and asserts the emitted tokens are a subset of
STATE_VOCABULARY. Both are paired with a should_panic test proving
assert_states_in_vocabulary genuinely rejects a bogus token.
Migrate the live "disabled" string literal in
actionability::states_are_enabled to state::DISABLED, and document the
three vocabulary-only tokens (invalid/multiselectable/haspopup) that have
no macOS AX producer today.
Thread real window bounds into the two macOS call sites that build
StateReaderContext with window_bounds hardcoded to None (post_state.rs's
post-action/live state reads and query.rs's find/query tree walk), so the
offscreen token can actually be computed there instead of being silently
unreachable. post_state.rs resolves the owning window via the AXWindow
attribute; query.rs inherits window bounds down the recursive walk the same
way builder.rs does, capturing a node's own bounds once its role is
AXWindow.
Add an ObservationOps call (resolve_element_strict) to the U0 capability
conformance test's exercise() helper, which previously called zero
ObservationOps methods and would not have caught a break in that
supertrait.
accname.rs previously reduced NameEvidence with a 2-way title-or-description
fallback despite the trait already exposing 7 evidence fields. Implement the
documented KTD6 precedence (explicit label -> labelled-by text -> native
title -> static-role value -> aggregated child label -> placeholder ->
description last) as compute_name/compute_description over NameEvidence,
plus a join_child_labels aggregation primitive.
Migrate macOS's resolve_element_name to a thin wrapper that gathers raw
NameEvidence (name_evidence_impl) and reduces it via core's compute_name,
removing the inline fallback chain it used to own. name_evidence_impl now
reads AXTitleUIElement (labelled-by), AXPlaceholderValue, and aggregates
multiple child labels via join_child_labels instead of returning only the
first match. resolve_search.rs, resolve_classify.rs, and chain_menu_steps.rs
need no changes: they already consume resolve_element_name's return value
and now get the correct precedence for free.
open_session silently returned Ok(None) instead of following the
blanket not_supported convention every other adapter default uses,
AdapterSession had lost its Sync bound, and the SessionAffinity
parameter the plan specified never shipped. Introduces
session_affinity.rs (SessionAffinity { session_id }), restores
AdapterSession: Send + Sync, and changes open_session's signature to
open_session(&SessionAffinity) -> Result<Box<dyn AdapterSession>,
AdapterError> with a default Err(not_supported("open_session")).
native_id was inserted mid-struct in AdRefEntry (offset 40), shifting every
field from `states` onward and silently corrupting a prebuilt C consumer's
field reads. Move it to the end, matching AdActionStep's append-only
evolution pattern, and bump AD_ABI_VERSION_MAJOR so a consumer built against
the old layout fails ad_init instead of misreading memory.
Also collapse execute_by_ref_timeout.rs, which hand-duplicated ~85 lines of
the @generated ad_execute_by_ref body, into a proper Family-B codegen
template so it stays in lockstep with the canonical wrapper instead of
drifting from it.
- crates/ffi/src/types/ref_entry.rs: move native_id to the end of AdRefEntry
- crates/ffi/src/abi_version.rs: bump AD_ABI_VERSION_MAJOR 1 -> 2, +regression
test pinning the bump so a revert fails a real assertion
- crates/ffi/tests/c_abi_layout.rs: explicit per-field offset_of! asserts for
every AdRefEntry field (not just monotonic ordering) + zeroed-read sentinel
now asserts native_id.is_null()
- crates/ffi/codegen_templates/execute_by_ref_timeout.rs.in +
crates/ffi/build.rs + tests/codegen_exhaustiveness.rs: generate
ad_execute_by_ref_timeout through the same Family-B pipeline as
ad_execute_by_ref instead of a hand-maintained duplicate
- crates/ffi/include/agent_desktop.h: regenerated via
scripts/update-ffi-header.sh (cbindgen 0.29.4)
Modifiers were a silent no-op: MouseEvent already carried a modifiers
field but synthesize_mouse never read it, mouse_wheel dropped its
_modifiers param outright, and mouse-click/mouse-down/mouse-up had no
--modifiers flag to even request a chord.
- macos/input/mouse.rs: derive CGEventFlags from the requested modifier
chord and set it explicitly on every synthesized CGEvent (including
the empty case), so no ambient or prior-call flags can leak onto an
unmodified click; thread modifiers through synthesize_scroll_at and
set them on the wheel event too.
- macos/adapter.rs: stop dropping mouse_wheel's modifiers parameter.
- cli_args/actions.rs + dispatch: add --modifiers to mouse-click,
mouse-down, and mouse-up (mirroring the existing mouse-wheel flag)
and thread it through parse_modifiers.
- core mouse_click/mouse_down/mouse_up commands: stop hardcoding
modifiers: Vec::new() and pass the requested chord through.
- ffi/input/mouse.rs: add the additive ad_mouse_event_with_modifiers
entrypoint (modifiers as an array + count, mirroring AdKeyCombo) so
FFI callers can reach chorded clicks without changing the pinned
AdMouseEvent layout; regenerate the committed header.
--args was emitted once per --arg, handing the launched app a stray
--args token instead of its intended argument; argv assembly is now a
pure, unit-tested fn that emits --args exactly once. --no-attach was
silently ignored: launch_app_with_options_impl always attached-or-waited
regardless of the flag. It now fails with a structured error naming the
running pid when the app is already running, and returns immediately
without the window-wait loop when it is not. The raw app id no longer
lands in trace-reachable error messages; it travels only in
details.app_name, which redacts on trace export.
Split the launch path out of app_ops.rs into a new launch.rs sibling
module (app_ops.rs would otherwise exceed the 400-line cap).
Replaces the flat {format,text,bytes_base64} ClipboardContent struct with
the plan's ClipboardContent { Text, Image, FileUrls } enum and removes the
string-only get_clipboard/set_clipboard adapter methods (KTD13: remove, not
wrap) in favor of get_clipboard_content/set_clipboard_content as the sole
surface, keeping clear_clipboard unchanged.
macOS now round-trips text, PNG images, and public.file-url references
through NSPasteboard instead of returning not_supported for image/file-urls.
clipboard-get defaults to text when --format is omitted, writes image bytes
to an --out path (or a private 0600/O_NOFOLLOW/atomic temp file under the
session dir when --out is omitted, since clipboard images can carry copied
secrets), and reports {type,path,width,height}. clipboard-set gains --image
and repeatable --file-url inputs; missing file-url paths report only a
count and entry index, never the path content, since that can reach traces.
FFI's ad_get_clipboard/ad_set_clipboard now delegate through the content
API with unchanged C signatures.
Replace the 3-kind DesktopSignal vocabulary with a full EventKind set
(WindowOpened, WindowClosed, AppLaunched, AppTerminated,
FocusChangedWindow, SurfaceAppeared, SurfaceDismissed) and a pure,
adapter-double-testable diff_signals(baseline, current) over an
id/pid-keyed SignalBaseline. wait --event window-opened/app-launched/
surface-appeared now works from just --app, without the caller naming
a window id or title up front (R16/AE6) — wait_event.rs captures a
baseline at wait start and polls diff_signals against fresh captures,
same shape as the existing notification baseline loop.
Folding window/app identity into id- and pid-keyed sets (rather than
count comparisons or id-or-title matching) closes two real bugs: a
title-matching window could win a window_focused wait for the wrong
id, and a concurrent unrelated window open could mask a window_closed
event gated on window_count. Event timeouts now carry
details.kind == "wait_timeout" like every sibling wait mode.
--window alongside --event now narrows the event wait to a window
title instead of tripping the "exactly one mode" validator.
Reshape ProcessState to the plan's {Running, Exited{code}, Crashed,
Unresponsive} contract instead of the ad hoc {Responsive, Unresponsive,
Unknown}. macOS now does a kill(pid,0)-style liveness check before
probing AX at all, and only classifies Unresponsive after a second
consecutive kAXErrorCannotComplete, so one transient AX blip on a
healthy-but-busy app no longer hard-fails the action. The classification
threshold is isolated behind a pure, platform-independent classify() fn.
ensure_process_responsive was an unconditional preflight hard-gate that
could turn a would-succeed action into a failure and propagated a
transient probe error as the terminal error. Replace it with
enrich_with_process_state: best-effort, terminal-only enrichment that
runs exactly once (never per auto-wait tick), never converts a success
into a failure, attaches details.process_state on STALE_REF/APP_NOT_FOUND,
and only surfaces APP_UNRESPONSIVE when the process is genuinely
classified Unresponsive.
Add the missing retry_token_for_code arm for AppUnresponsive.
HitTestResult becomes ReachesTarget | InterceptedBy { role, name, bounds }
| Unknown instead of a boolean, so a probe failure or a hit on the
target's own ancestor (composited/custom-drawn containers) is never
reported as a false occlusion Fail. receives_events_check maps the
three states to Pass/Fail/Unknown and carries the occluder's role and
redactable name in ActionabilityCheck.occluder instead of a hardcoded
string. requires_hit_test() now covers Drag. visibility_check also
fails on the HIDDEN/OFFSCREEN state vocabulary, not just zero bounds.
macOS hit_test_impl reuses the bounded ax_helpers::try_each_ancestor
walk instead of a hand-rolled unbounded AXParent loop, and treats
every probe failure (null/zero bounds, missing pid, AX error) as
Unknown rather than a false Fail.
hover --ref and drag --from/--to previously resolved a ref straight to
a center point via point_resolve without ever consulting hit_test, so
an occluded target dispatched blind. resolve_point_from_ref_or_xy_with_context
now runs the new receives_events-only check on the ref-targeted path
before returning the point; raw --xy input is unaffected by design.
type, set-value, select, scroll, hover, and drag were the only 6 of 18
ref-addressed commands without a --timeout-ms flag: their core Args
structs hardcoded RefArgs.timeout_ms to None, so auto-wait retry
(landed for click and friends) silently never applied to them. Add the
paired clap/serde --timeout-ms default (5000, matching RefArgs) to
TypeArgs/SetValueArgs/SelectArgs/ScrollArgs/HoverArgs/DragCliArgs and
thread the normalized value through dispatch into the core command
structs. hover/drag resolve coordinates rather than dispatching an
action, so they gain a dedicated resolve-retry helper
(helpers::resolve_point_with_wait) that retries a transient
STALE_REF/AMBIGUOUS_TARGET/TIMEOUT within the same budget.
Also fix ref_action_wait::execute_poll_loop double-checking
actionability: it ran actionability::check_live itself and then handed
off to dispatch (execute_resolved), which runs check_live again. A
failure on that second, redundant check propagated via a bare `?` that
bypassed the retry/permanent classification entirely, so a transient
actionability flip between the two checks failed the action outright
instead of retrying. The loop now calls dispatch directly per
iteration and classifies whatever it returns, so there is exactly one
check_live per attempt and every actionability failure goes through
retry/permanent classification. Single-shot (timeout_ms: None) is
unaffected — it already only checked once.
Also drop the dead `timeout_ms: None` literal in execute_by_ref's
ActionRequest construction (it was always overwritten by
execute_ref_action_with_context) in favor of the actual normalized
value, so the struct literal doesn't misrepresent what ships.
An unbounded --timeout-ms flowed into Instant::now() + Duration::from_millis(ms)
and panicked on overflow; with no catch_unwind on the CLI path this was a raw
crash on trivial input (e.g. --timeout-ms 99999999999999999999). Clamp the
budget to a 24h ceiling before deadline construction, with a regression test.
Also add APP_UNRESPONSIVE to the error-code as_str/serde consistency test, which
was silently non-exhaustive after the 16th variant landed.
The --env KEY=VALUE parser interpolated the raw pair (including the secret
value) into INVALID_ARGS messages, which reach the unredacted `message` field
of command.end trace events (redaction is a field-name allowlist, not a
content scanner). Report the entry index and rule only, never the value, per
docs/solutions/conventions/keep-raw-arguments-out-of-trace-reachable-error-messages.md,
with a regression test asserting a secret marker never appears in the message.
scroll_into_view reused the hand-rolled AXScrollToVisible CFString+perform
sequence that ax_helpers::try_ax_action already provides.
Scroll offscreen ref targets into view before actions, add accname
NameEvidence and supported_surfaces in status, gate on ProcessState with
APP_UNRESPONSIVE and envelope 2.1, extend launch with args/env/cwd,
wire desktop signal waits via --event, typed clipboard formats, and
mouse-wheel with modifier support.
Co-authored-by: Cursor <cursoragent@cursor.com>
Expose ObservationOps::hit_test with macOS AX element-at-position probing
and fail pointer-targeting ref actions when the center point is occluded.
Co-authored-by: Cursor <cursoragent@cursor.com>
Poll resolve and actionability until the budget expires so transient stale
refs settle before dispatch; CLI and FFI default to 5000 ms with --timeout-ms
and ad_execute_by_ref_timeout for overrides.
BREAKING CHANGE: ref actions now auto-wait up to 5000 ms by default; pass
--timeout-ms 0 or ad_execute_by_ref_timeout(..., 0, ...) to restore
single-shot behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
Introduce a serializable LocatorQuery contract with core matching and
classify_query_result, wire find through resolve_query on macOS with
snapshot fallback for stub adapters, and add --exact, --state, --native-id,
and --description CLI flags.
Co-authored-by: Cursor <cursoragent@cursor.com>
Window ids are now the primary lookup key for macOS snapshot, focus, and
window-op paths, with pid/title corroboration to fail closed on recycled
CGWindow numbers instead of silently matching the first same-titled window.
Co-authored-by: Cursor <cursoragent@cursor.com>
Capture AXIdentifier as native_id with auto-generated filtering, prioritize it
in identity_matches, and extend AdRefEntry via the size-pin sequence.
Co-authored-by: Cursor <cursoragent@cursor.com>
Complete the display capture contract with enumerable displays, scale_factor
on captures, and INVALID_ARGS when --screen is out of range.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add a shared state_reader for tree and live reads, expand AX batch
attributes, and emit hidden/offscreen/indeterminate tokens from evidence.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add role/state vocabulary modules and move is --property visible onto live
bounds plus hidden/offscreen tokens so off-screen elements no longer pass.
Co-authored-by: Cursor <cursoragent@cursor.com>
Restructure the 397-LOC adapter contract into ObservationOps, ActionOps,
InputOps, and SystemOps with a blanket composed trait so platform adapters
and test doubles gain file-budget headroom without changing behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
Adds the trace read/replay layer on the session-first foundation: `trace show` merges per-process segments into one deterministic timeline (bounded JSON for agents), and `trace export` renders a single self-contained, XSS-safe HTML viewer for humans. Opt-in `session start --screenshots` captures pre/post-action screenshots and refmap copies; command.start/end boundary events and a versioned trace.meta header make a step-by-step replay reconstructable. Redaction is hardened so raw caller arguments never leak into trace-reachable error messages. Available across CLI, batch, and FFI.
