From 052ced9de398abaf4480d83b1ce7d710101972fd Mon Sep 17 00:00:00 2001 From: Lahfir Date: Tue, 2 Jun 2026 02:06:11 -0700 Subject: [PATCH] docs: update reliability docs and skills --- README.md | 4 ++-- ...laywright-grade-desktop-reliability-2026-06-02.md | 12 ++++++++++-- skills/agent-desktop/SKILL.md | 3 ++- skills/agent-desktop/references/commands-system.md | 3 +++ tests/conformance/README.md | 4 ++-- 5 files changed, 19 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 6fe1f3d9..bb34ad08 100644 --- a/README.md +++ b/README.md @@ -206,8 +206,8 @@ agent-desktop key-up shift # release key ```bash agent-desktop hover @e3 # move cursor to element agent-desktop hover --xy 500,300 # move cursor to coordinates -agent-desktop drag @e3 --to @e8 # drag between elements -agent-desktop drag --xy 100,200 --to-xy 400,200 # drag between coordinates +agent-desktop drag --from @e3 --to @e8 # drag between elements +agent-desktop drag --from-xy 100,200 --to-xy 400,200 # drag between coordinates agent-desktop mouse-click --xy 500,300 # click at coordinates agent-desktop mouse-down --xy 500,300 # press at coordinates agent-desktop mouse-up --xy 500,300 # release at coordinates diff --git a/docs/solutions/best-practices/playwright-grade-desktop-reliability-2026-06-02.md b/docs/solutions/best-practices/playwright-grade-desktop-reliability-2026-06-02.md index ebfdafbb..445a328c 100644 --- a/docs/solutions/best-practices/playwright-grade-desktop-reliability-2026-06-02.md +++ b/docs/solutions/best-practices/playwright-grade-desktop-reliability-2026-06-02.md @@ -36,8 +36,9 @@ Ref actions must pass through the shared reliability path: 2. Resolve the ref with strict platform identity checks. 3. Return `STALE_REF` when the old element no longer matches. 4. Return `AMBIGUOUS_TARGET` when multiple candidates match. -5. Run actionability checks before adapter dispatch. -6. Emit trace events only to the requested JSONL trace path, never stdout. +5. Run live actionability checks before adapter dispatch. +6. Keep waits bounded by the caller timeout, including platform resolution retries. +7. Emit trace events only to the requested JSONL trace path, never stdout. ## Cross-Platform Rule @@ -47,6 +48,12 @@ identity fields into the same `RefEntry` concepts: role, name, value, description, state, bounds, supported actions, source surface, root ref, and tree path. +Actionability should prefer one native live-state read that returns state, +bounds, and supported actions together. Platform adapters may fall back to +separate reads, but the CLI behavior must remain identical: empty transient +action reads do not erase snapshot capabilities, while a non-empty live action +set that lacks the required action can block dispatch. + ## Review Rule Any change to ref resolution or action dispatch must include tests for: @@ -54,6 +61,7 @@ Any change to ref resolution or action dispatch must include tests for: - stale ref rejection - ambiguous target rejection - actionability failure before dispatch +- retrying waits that honor timeout and report last observed state - session isolation - FFI parity when the behavior is exposed through C ABI diff --git a/skills/agent-desktop/SKILL.md b/skills/agent-desktop/SKILL.md index fde64611..499bff30 100644 --- a/skills/agent-desktop/SKILL.md +++ b/skills/agent-desktop/SKILL.md @@ -92,7 +92,7 @@ Use **progressive skeleton traversal** as the default approach. It reduces token - After any action that changes UI, re-drill the affected region or re-snapshot - **Scoped invalidation:** re-drilling `--root @e3` only replaces refs from @e3's previous drill — refs from other regions and the skeleton itself are preserved - **Strict resolution:** stale refs return `STALE_REF`; duplicate plausible targets return `AMBIGUOUS_TARGET` instead of choosing arbitrarily. -- **Actionability:** ref actions check visibility, enabled state, supported action, policy, and editability before dispatch. +- **Actionability:** ref actions check live visibility, enabled state, supported action, policy, and editability before dispatch. - **Sessions:** use `--session ` for concurrent or multi-agent runs; batch entries may override with `"session": "id"`. - **Trace:** use `--trace ` for JSONL diagnostics outside stdout; add `--trace-strict` only when trace write failures should fail the command. @@ -212,6 +212,7 @@ agent-desktop clipboard-clear # Clear clipboard agent-desktop wait 1000 # Pause 1 second agent-desktop wait --element @e5 --snapshot --timeout 5000 # Wait for element agent-desktop wait --element @e5 --predicate actionable --timeout 5000 # Wait until actionable +agent-desktop wait --element @e5 --predicate value --value "Done" --timeout 5000 # Wait for value agent-desktop wait --window "Title" # Wait for window agent-desktop wait --text "Done" --app "App" # Wait for text agent-desktop wait --menu --app "App" # Wait for menu surface diff --git a/skills/agent-desktop/references/commands-system.md b/skills/agent-desktop/references/commands-system.md index e683dd46..72dad753 100644 --- a/skills/agent-desktop/references/commands-system.md +++ b/skills/agent-desktop/references/commands-system.md @@ -169,8 +169,11 @@ Pauses for N milliseconds. Use between actions that need time to settle. ### wait (element) ```bash agent-desktop wait --element @e5 --snapshot --timeout 5000 --app "App" +agent-desktop wait --element @e5 --predicate actionable --timeout 5000 +agent-desktop wait --element @e5 --predicate value --value "Done" --timeout 5000 ``` Blocks until the element ref appears in the accessibility tree. Useful after triggering UI changes. +When `--snapshot` is omitted, the command polls the caller's latest session refmap and refreshes it on the built-in debounce. When `--snapshot` is passed, it stays pinned to that refmap. Element resolution is capped by the remaining `--timeout`, and timeout errors include the last observed predicate/actionability state. ### wait (window) ```bash diff --git a/tests/conformance/README.md b/tests/conformance/README.md index c66172e2..4fd1022c 100644 --- a/tests/conformance/README.md +++ b/tests/conformance/README.md @@ -12,8 +12,8 @@ expectations. | Snapshot refs | Refs are depth-first, snapshot-scoped, and persisted in the caller session | | Strict resolve | A ref resolves only when identity still matches; stale refs return `STALE_REF` | | Ambiguity | Multiple plausible matches return `AMBIGUOUS_TARGET`, never an arbitrary click | -| Actionability | Ref actions check visibility, enabled state, supported action, policy, and editability before dispatch | -| Wait recovery | `wait --element` can poll the latest session refmap when no snapshot is pinned | +| Actionability | Ref actions check live visibility, enabled state, supported action, policy, and editability before dispatch | +| Wait recovery | `wait --element` can poll the latest session refmap when no snapshot is pinned, honors the caller timeout while resolving, and reports the last observed predicate state | | Session scope | `--session ` and batch item `session` never read or write another session's refmap | | Trace | `--trace ` writes JSONL diagnostics outside stdout and is best-effort unless strict | | FFI parity | FFI ref actions use strict resolve and actionability checks before adapter dispatch |