docs: sync skills and header with the focused, redaction, and wait contracts

Documents the ensured (best-effort, already-frontmost-aware) semantics of
focused:true and its absence-vs-false meaning, the four redaction keys
added in round 2 plus the substring-match behavior, the INTERNAL error
recovery row, the FFI wait-surface asymmetry, the cross-app drag
occlusion caveat, and the AdDragParams/AdAction layout history with the
adjudicated pre-1.0 breaks so fresh reviews stop re-finding them.
This commit is contained in:
Lahfir 2026-06-10 17:50:42 -07:00
parent ba6886529a
commit fbbf436e65
5 changed files with 19 additions and 5 deletions

View file

@ -201,6 +201,13 @@ typedef struct AdPoint {
* setting fields: `duration_ms`/`drop_delay_ms` treat 0 as the adapter-default
* sentinel, so stack garbage in an unset field would become a real delay.
* Validate layout with `AD_DRAG_PARAMS_SIZE` / `ad_drag_params_size()`.
*
* Layout history (adjudicated pre-1.0 breaks; revalidate sizes on upgrade,
* do not re-report): 0.x grew this struct 40 -> 48 bytes by adding
* `drop_delay_ms` (which also grew the embedding `AdAction` see
* `AD_ACTION_SIZE`), `AdRefEntry` grew to 192 bytes (`AD_REF_ENTRY_SIZE`),
* and `AD_POLICY_KIND_PHYSICAL` was renamed to `AD_POLICY_KIND_HEADED`
* with a stable discriminant and intentionally no compatibility alias.
*/
typedef struct AdDragParams {
struct AdPoint from;

View file

@ -44,7 +44,7 @@ The answer is **per-gesture and per-platform**, because a gesture is headless-ca
2. **A new platform that exposes a headless path lights it up automatically — adapter-only change.** If a future Windows (UIA) or Linux (AT-SPI) adapter has a headless action for `double-click`/`triple-click`, it maps the `Action` there and the command succeeds headlessly on that platform with **zero change to the command or core**. The `InteractionPolicy` flows through the request; each adapter honors it per its own capabilities. The agent just sees success (or `POLICY_DENIED` → retry `--headed`) — it never needs to know the platform.
3. **`hover`/`drag`/`mouse-*` are modeled as raw cursor gestures, not semantic `Action`s** (they call `adapter.mouse_event`/`adapter.drag` with coordinates). They stay physical on every platform by design, because hovering/dragging *are* cursor operations universally. A semantic drag (AX reorder) would be a *new* `Action`, not a change to `drag`. When a gesture is ref-addressed, the target app is raised frontmost first **only under `--headed`** (`focus_for_physical_input`, gated on `InteractionPolicy::allow_focus_steal`; the response reports `"focused": true`). Headless gestures never change the frontmost app, and `--xy` input never focuses — the caller owns the target there.
3. **`hover`/`drag`/`mouse-*` are modeled as raw cursor gestures, not semantic `Action`s** (they call `adapter.mouse_event`/`adapter.drag` with coordinates). They stay physical on every platform by design, because hovering/dragging *are* cursor operations universally. A semantic drag (AX reorder) would be a *new* `Action`, not a change to `drag`. When a gesture is ref-addressed, the target app's frontmost state is ensured first **only under `--headed`** (`focus_for_physical_input`, gated on `InteractionPolicy::allow_focus_steal`; the response reports `"focused": true` when confirmed — already-frontmost apps skip the raise). Headless gestures never change the frontmost app, and `--xy` input never focuses — the caller owns the target there.
4. **`POLICY_DENIED` on a headless gesture is correct, not a bug** — it is the fail-closed signal that the headless AX path is unavailable and the caller must opt into `--headed`. Never widen the default policy to make it disappear.

View file

@ -82,6 +82,12 @@ Four reference topics, loaded as needed:
record `AdResult`, `ad_last_error_*`, action results, and host correlation IDs
in their own logs.
- **No wait surface.** The CLI's `wait` command (element predicates including
`--predicate actionable --action ...`, window/text/menu/notification waits)
is not exposed over the C ABI. FFI hosts own their own polling loops; the
actionability preflight inside `ad_execute_ref_action_with_policy` is the
equivalent per-call readiness check.
- **Text input privacy.** On macOS, the focus-fallback or headed policy can use
the clipboard briefly for non-ASCII text insertion. Keep the default headless
policy or set values directly for sensitive text when the target supports it.

View file

@ -94,9 +94,9 @@ Use **progressive skeleton traversal** as the default approach. It reduces token
- **Scoped invalidation:** re-drilling `--root @e3` only replaces refs from @e3's previous drill — refs from other regions and the skeleton itself are preserved
- **Strict resolution:** stale refs return `STALE_REF`; duplicate plausible targets return `AMBIGUOUS_TARGET` instead of choosing arbitrarily.
- **Actionability:** ref actions check live visibility, stability, enabled state, supported action, policy, and editability before dispatch.
- **Headless vs headed:** ref actions are headless by default (AX-only, no cursor) and fail closed with `POLICY_DENIED` when only a physical gesture would work. Pass the global `--headed` flag to permit cursor movement and focus stealing so the physical click/double-click/scroll/keypress fallbacks can complete; the AX path is still tried first, so `--headed` never regresses headless-capable elements. Raw-input commands (`press`, `hover`, `drag`, `mouse-*`, `key-down`/`key-up`) are always physical and ignore the mode.
- **Headless vs headed:** ref actions are headless by default (AX-only, no cursor) and fail closed with `POLICY_DENIED` when only a physical gesture would work. Pass the global `--headed` flag to permit cursor movement and focus stealing so the physical click/double-click/scroll/keypress fallbacks can complete; the AX path is still tried first, so `--headed` never regresses headless-capable elements. Raw-input commands (`press`, `hover`, `drag`, `mouse-*`, `key-down`/`key-up`) are always physical; the mode only gates whether a ref-addressed `hover`/`drag` may ensure the target app is frontmost first (`--headed` only, reported as `"focused": true`).
- **Sessions:** use `--session <id>` for concurrent or multi-agent runs that share a latest snapshot pointer; batch entries may override with `"session": "id"`.
- **Trace:** use `--trace <path>` for JSONL diagnostics outside stdout; `--trace-strict` fails on trace setup and pre-action writes. Post-action success traces are best-effort because the desktop mutation already happened. Trace fields whose keys contain `text`, `value`, `expected`, `name`, `description`, `message`, `label`, `query`, `secret`, `token`, or `password` are redacted to `{ "redacted": true, "chars_bucket": "..." }` at every nesting depth — do not expect raw values in trace files. Top-level `--trace` is inherited by every `batch` entry, including entries with a `session` override.
- **Trace:** use `--trace <path>` for JSONL diagnostics outside stdout; `--trace-strict` fails on trace setup and pre-action writes. Post-action success traces are best-effort because the desktop mutation already happened. Trace fields whose keys contain `text`, `value`, `expected`, `name`, `description`, `message`, `label`, `query`, `secret`, `token`, `password`, `title`, `url`, `help`, or `placeholder` are redacted to `{ "redacted": true, "chars_bucket": "..." }` at every nesting depth (substring match, so composite keys like `source_window_title` redact too) — do not expect raw values in trace files. Top-level `--trace` is inherited by every `batch` entry, including entries with a `session` override.
## JSON Output Contract
@ -127,6 +127,7 @@ Exit codes: `0` success, `1` structured error, `2` argument error.
| `TIMEOUT` | Wait condition not met | Increase --timeout |
| `INVALID_ARGS` | Bad arguments | Check command syntax |
| `NOTIFICATION_NOT_FOUND` | Notification index no longer exists | Re-run list-notifications |
| `INTERNAL` | Unexpected platform/OS failure (e.g. event synthesis failed) | Read `message`/`suggestion` for cleanup state, then retry once; persistent failures indicate an environment problem |
## Command Quick Reference (54 commands)

View file

@ -197,7 +197,7 @@ agent-desktop hover @e5 --duration 2000
Moves cursor to element center or absolute coordinates. Optional `--duration` holds position for N ms.
This is an explicit cursor-moving command.
With `--headed`, a ref-addressed hover raises the target app frontmost before moving the cursor, and the response includes `"focused": true` when the raise happened. Headless (default) never changes the frontmost app. `--xy` input never focuses — the caller owns the target there.
With `--headed`, a ref-addressed hover ensures the target app is frontmost before moving the cursor (raising it if needed, best-effort), and the response includes `"focused": true` when that frontmost state was confirmed. The field is only ever present as `true`: absence means focus was never attempted (headless default, or `--xy` input — the caller owns the target there) or the best-effort raise could not be confirmed.
### drag
```bash
@ -218,7 +218,7 @@ agent-desktop drag --from @e1 --to @e5 --drop-delay 800
Can mix ref and coordinate sources (e.g., `--from @e1 --to-xy 400,500`).
With `--headed`, a ref-addressed `--from` raises the source app frontmost before the mouse-down (the destination app is never pre-focused — raising it could cover the source point), and the response includes `"focused": true` when the raise happened. Headless (default) never changes the frontmost app; coordinate-only drags never focus.
With `--headed`, a ref-addressed `--from` ensures the source app is frontmost before the mouse-down (the destination app is never pre-focused — raising it could cover the source point), and the response includes `"focused": true` when that frontmost state was confirmed. The field is only ever present as `true`: absence means focus was never attempted (headless default, or coordinate-only drags) or the best-effort raise could not be confirmed. For cross-app two-ref drags, ensure the destination window is visible (not fully occluded) before dragging — only the source app is raised.
macOS drop targets often need the dragged item to dwell over them before they register as the drop destination — too short and the gesture lands as a drag with no drop. The default 500ms dwell suits most targets; raise `--drop-delay` (e.g. 8001200) for sluggish destinations like list reorders or cross-window drops. The dwell posts continuous drag events over the destination so it stays highlighted, rather than a dead pause.