3f322728b4
Settle the Playwright-grade reliability contract in agent-desktop-core before the Windows/Linux adapters are built, so they inherit it instead of redesigning it. Every command now observes, waits, verifies, and reports honestly instead of firing blindly. Highlights: capability-supertrait split of PlatformAdapter with not_supported() defaults; canonical role/state vocabulary with live `is --property visible`; display enumeration (`list-displays`) and honest `--screen` with scale factor; truthful Automation permission; `native_id` identity spine; window-id-first resolution; serializable `LocatorQuery` with live `find`; default-on auto-wait before every ref action; three-way `hit_test` occlusion gate; `scroll_into_view` in core; core accessible-name precedence; typed `ActionStep` delivery tier; `ProcessState` and `APP_UNRESPONSIVE`; `LaunchOptions`; baseline-diff desktop signals (`wait --event`); typed clipboard (`Text`/`Image`/`FileUrls`); mouse modifier chords and `mouse-wheel`. Hardened through a 35-reviewer pass with independent validation and a green live e2e gate (109/0), plus a head-vs-main performance comparison harness. BREAKING CHANGE: default-on auto-wait changes the timing of every previously-untouched ref-action call (bounded 5000 ms default; `--timeout-ms 0` restores single-shot). `ENVELOPE_VERSION` is now `2.1` (adds the `APP_UNRESPONSIVE` code and process state in error details). FFI ABI major is `3` (append-only struct evolution; `wait --event` is intentionally not exposed over FFI). The legacy string clipboard API is removed in favor of typed content. `key-down`/`key-up` fail closed until daemon-owned held input exists. `close-app` verifies termination and the osascript fallback path is removed. `--text` matching is subtree containment: `find --text X --first` returns the outermost matching container. |
||
|---|---|---|
| .. | ||
| README.md | ||
Locator resolution benchmark
This deterministic harness compares the legacy full-snapshot locator path with
the handle-free observed-tree evaluator over synthetic Chromium/Electron accessibility
trees. It exercises deep anonymous wrapper chains, duplicate role/name
candidates, moving bounds, simultaneous AXIdentifier and AXDOMIdentifier
values, containment predicates, and large trees.
Run it without network access:
rtk cargo run -p agent-desktop-core --release --example locator_benchmark \
> /private/tmp/agent-desktop-locator-synthetic.json
The JSON report includes 31-run p50/p95 wall-clock latency, candidate nodes,
predicate work, requested synthetic attribute values, cardinality, and
correctness. live_find_selected_refs uses the same selected-match-only
materialization contract as the default CLI find path. live_arena_direct is
reported separately as a lower-overhead direct-target metric, and
live_count_no_refmap verifies that count-only requests perform no action or
settable evidence reads. The harness does not claim absolute native AX calls:
role-dependent action and settable probes make that model inaccurate. Native
Slack/Electron IPC measurements require Accessibility permission and belong in
the privileged macOS integration suite.
This synthetic benchmark JSON is consumed by the repo-wide performance report:
run bash scripts/perf-baseline-compare.sh (optionally with --apps "Slack,Google Chrome") to produce report.html, covering the HEAD-vs-main fixture A/B,
optional real-app read-only probes, and this synthetic benchmark.
For privileged Electron/Chromium runs, the macOS adapter first asks whether the
application root exposes AXManualAccessibility as settable, reads its current
value, and sets it at most once when supported. The adapter then waits within
the request's existing absolute deadline until the attribute reports ready,
recording attempted/succeeded/ready activation in locator stats. Unsupported
native applications are left unchanged. This follows
Electron's official third-party accessibility guidance,
which documents setting AXManualAccessibility from native assistive software
to expose Chromium's accessibility tree before automatic assistive-technology
detection has enabled it. The
current Electron macOS implementation
advertises the attribute as settable, reports true only in complete mode, and
handles AXEnhancedUserInterface separately; that source behavior is why the
benchmark methodology requires a post-set readiness read instead of treating a
successful setter as immediate readiness.