mirror of
https://github.com/lahfir/agent-desktop.git
synced 2026-08-06 14:10:43 +00:00
docs: refresh reliability/actionability solutions for symbol drift
Reconcile four docs/solutions learnings against current code (their core
guidance is unchanged; only cited symbols/signatures/counts drifted):
- playwright-grade-desktop-reliability: add the actionability_timeout trace
kind; fix the check_actionability_with_trace signature (bundled
ResolvedRefAction) and the ref-allocation example (allocate_refs folds in
bounds-hiding; strip_ref_bounds_when_hidden is gone)
- real-app-tests-are-the-platform-adapter-gate: attribute the accessible-name
reducer to tree/builder.rs::accessible_name with element.rs::resolve_element_name
as the thin wrapper (unified in adda4c9)
- macos-gesture-headless-capability: 54 -> 58 commands; double_click now returns
Vec<ActionStep> tagged SemanticApi/PhysicalSynthetic
- exhaustiveness-guards: context.request( -> context.request_base(
This commit is contained in:
parent
640e8e83e4
commit
604844d22d
4 changed files with 55 additions and 20 deletions
|
|
@ -80,12 +80,12 @@ const POLICY_TESTED_COMMANDS: &[&str] = &[
|
|||
#[test]
|
||||
fn all_context_request_callers_are_policy_tested() {
|
||||
// scans crates/core/src/commands/*.rs (excluding *_tests) for files
|
||||
// containing `context.request(` and fails, naming each stem, when one
|
||||
// containing `context.request_base(` and fails, naming each stem, when one
|
||||
// is absent from POLICY_TESTED_COMMANDS
|
||||
}
|
||||
```
|
||||
|
||||
The universe is not hand-maintained: the test scans the filesystem for the call-site signature every ref-action command shares. A new command file that calls `context.request(` without a registered policy assertion fails CI with a message naming the stem and the required follow-up.
|
||||
The universe is not hand-maintained: the test scans the filesystem for the call-site signature every ref-action command shares. A new command file that calls `context.request_base(` without a registered policy assertion fails CI with a message naming the stem and the required follow-up.
|
||||
|
||||
**Leg 3 — per-case value pins.**
|
||||
|
||||
|
|
|
|||
|
|
@ -53,7 +53,7 @@ The answer is **per-gesture and per-platform**, because a gesture is headless-ca
|
|||
## Why This Matters
|
||||
|
||||
- It keeps the **headless-first reliability guarantee** honest: the tool only claims a headless effect when the OS actually provides one, and fails closed otherwise.
|
||||
- It preserves **cross-platform extensibility**: the same 54-command surface works identically across macOS/Windows/Linux, and each adapter contributes whatever headless capability its platform has — without touching the command layer.
|
||||
- It preserves **cross-platform extensibility**: the same 58-command surface works identically across macOS/Windows/Linux, and each adapter contributes whatever headless capability its platform has — without touching the command layer.
|
||||
- It prevents the **vacuous-success trap**: assuming `ok:true` means the gesture happened, when an AX action succeeded at the API layer but the control ignored it.
|
||||
|
||||
## When to Apply
|
||||
|
|
@ -79,11 +79,19 @@ The macOS dispatch gates the physical path on the policy (so it is reachable onl
|
|||
|
||||
```rust
|
||||
// crates/macos/src/actions/chain_defs.rs
|
||||
pub(crate) fn double_click(el, _caps, policy) -> Result<(), AdapterError> {
|
||||
pub(crate) fn double_click(
|
||||
el: &AXElement,
|
||||
policy: InteractionPolicy,
|
||||
) -> Result<Vec<ActionStep>, AdapterError> {
|
||||
if ax_helpers::has_ax_action(el, "AXOpen") && ax_helpers::try_ax_action(el, "AXOpen") {
|
||||
return Ok(()); // headless AX path
|
||||
return Ok(vec![
|
||||
ActionStep::succeeded("AXOpen").with_mechanism(StepMechanism::SemanticApi), // headless AX path
|
||||
]);
|
||||
}
|
||||
crate::actions::dispatch::click_via_bounds(el, MouseButton::Left, 2, policy) // gated; POLICY_DENIED headless
|
||||
crate::actions::dispatch::click_via_bounds(el, MouseButton::Left, 2, policy)?; // gated; POLICY_DENIED headless
|
||||
Ok(vec![
|
||||
ActionStep::succeeded("CGClick").with_mechanism(StepMechanism::PhysicalSynthetic),
|
||||
])
|
||||
}
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -134,8 +134,11 @@ The reliable split is:
|
|||
details carry a `kind` discriminant: `"wait_timeout"` for wait-loop expiry
|
||||
(predicate, timeout_ms, last observed state) and `"chain_deadline"` for a
|
||||
chain step expiring mid-increment or mid-disclosure (observed value or
|
||||
expanded state, plus a `mutated` flag) — agents key on `kind` before
|
||||
inspecting other fields.
|
||||
expanded state, plus a `mutated` flag), and `"actionability_timeout"` for
|
||||
the ref-action auto-wait poll loop (click/type/set-value/etc.) exhausting
|
||||
`--timeout-ms` while resolution or actionability keeps failing transiently
|
||||
(carries an optional `report` field with the last actionability failure
|
||||
detail) — agents key on `kind` before inspecting other fields.
|
||||
- For `wait --element` without `--snapshot`, refresh the latest-ref cache on a
|
||||
bounded cadence; for a fixed `--snapshot`, treat missing refs as invalid input
|
||||
instead of silently switching snapshots.
|
||||
|
|
@ -261,8 +264,20 @@ presentation bounds when needed:
|
|||
```rust
|
||||
let identity_opts = opts.with_ref_identity_bounds();
|
||||
let tree = adapter.get_tree(&window, &identity_opts)?;
|
||||
let (tree, refmap) = allocate_refs(tree, opts)?;
|
||||
let tree = strip_ref_bounds_when_hidden(tree, opts);
|
||||
let mut refmap = RefMap::new();
|
||||
let config = RefAllocConfig {
|
||||
include_bounds: opts.include_bounds,
|
||||
interactive_only: opts.interactive_only,
|
||||
compact: opts.compact,
|
||||
pid: window.pid,
|
||||
source_app: Some(window.app.as_str()),
|
||||
source_window_id: Some(window.id.as_str()),
|
||||
source_window_title: Some(window.title.as_str()),
|
||||
source_surface: opts.surface,
|
||||
root_ref_id: None,
|
||||
path_prefix: &[],
|
||||
};
|
||||
let tree = ref_alloc::allocate_refs(tree, &mut refmap, &config);
|
||||
```
|
||||
|
||||
Ref action execution should keep the command-selected policy while centralizing
|
||||
|
|
@ -270,7 +285,8 @@ the strict ladder:
|
|||
|
||||
```rust
|
||||
let (entry, handle) = resolve_ref_with_context(ref_id, snapshot_id, adapter, context)?;
|
||||
check_actionability_with_trace(ref_id, &entry, handle.handle(), adapter, &request, context)?;
|
||||
let target = ResolvedRefAction { adapter, entry: &entry, handle: handle.handle(), ref_id, context };
|
||||
check_actionability_with_trace(&target, &request)?;
|
||||
let result = adapter.execute_action(handle.handle(), request)?;
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -92,12 +92,16 @@ a test that drives the real platform. Invest there, not in a more elaborate mock
|
|||
|
||||
The `STALE_REF` regression was caused by three copies of "what is this element's
|
||||
name" drifting apart. An element's accessible name must be computed by a single
|
||||
canonical function (`crates/macos/src/tree/element.rs::resolve_element_name`)
|
||||
that the snapshot builder, the live matcher, and the strict ref resolver all
|
||||
call. A ref's stored name and the resolver's recomputed name must come from the
|
||||
same code, or freshly-created refs go stale. Any new consumer of an element's
|
||||
name or identity must call the canonical resolver — never re-read `AXTitle` (or
|
||||
any single attribute) itself.
|
||||
canonical reducer, `crates/macos/src/tree/builder.rs::accessible_name`
|
||||
(precedence: title → description → static-text value → aggregated child
|
||||
label), which the snapshot builder calls directly when it stores a ref's name.
|
||||
`crates/macos/src/tree/element.rs::resolve_element_name` is a thin
|
||||
AXElement-only wrapper (`accessible_name(el, &fetch_node_attrs(el))`) called by
|
||||
the strict ref resolver, the live matcher, and hit-test occluder naming. A
|
||||
ref's stored name and the resolver's recomputed name must come from the same
|
||||
reducer, or freshly-created refs go stale. Any new consumer of an element's
|
||||
name or identity must call the canonical reducer or its wrapper — never
|
||||
re-read `AXTitle` (or any single attribute) itself.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
|
|
@ -157,10 +161,17 @@ assert!(get(ref_id, property = "role")["ok"]);
|
|||
And the single-owner name computation the resolver, matcher, and builder share:
|
||||
|
||||
```rust
|
||||
// crates/macos/src/tree/element.rs — the one accessible-name owner.
|
||||
// crates/macos/src/tree/builder.rs — the one accessible-name reducer.
|
||||
pub(crate) fn accessible_name(el: &AXElement, attrs: &NodeAttrs) -> Option<String> {
|
||||
// title, else description, else (static text only) value, else a label
|
||||
// aggregated from descendant text.
|
||||
}
|
||||
|
||||
// crates/macos/src/tree/element.rs — thin wrapper the strict resolver, live
|
||||
// matcher, and hit-test occluder naming call; the snapshot builder calls
|
||||
// accessible_name directly.
|
||||
pub fn resolve_element_name(el: &AXElement) -> Option<String> {
|
||||
// AXTitle, else AXDescription, else (static text only) AXValue.
|
||||
// Every name/identity consumer calls this; none re-reads AXTitle itself.
|
||||
accessible_name(el, &fetch_node_attrs(el))
|
||||
}
|
||||
```
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue