diff --git a/crates/ffi/include/agent_desktop.h b/crates/ffi/include/agent_desktop.h index 3099ac51..6400054b 100644 --- a/crates/ffi/include/agent_desktop.h +++ b/crates/ffi/include/agent_desktop.h @@ -201,6 +201,13 @@ typedef struct AdPoint { * setting fields: `duration_ms`/`drop_delay_ms` treat 0 as the adapter-default * sentinel, so stack garbage in an unset field would become a real delay. * Validate layout with `AD_DRAG_PARAMS_SIZE` / `ad_drag_params_size()`. + * + * Layout history (adjudicated pre-1.0 breaks; revalidate sizes on upgrade, + * do not re-report): 0.x grew this struct 40 -> 48 bytes by adding + * `drop_delay_ms` (which also grew the embedding `AdAction` — see + * `AD_ACTION_SIZE`), `AdRefEntry` grew to 192 bytes (`AD_REF_ENTRY_SIZE`), + * and `AD_POLICY_KIND_PHYSICAL` was renamed to `AD_POLICY_KIND_HEADED` + * with a stable discriminant and intentionally no compatibility alias. */ typedef struct AdDragParams { struct AdPoint from; diff --git a/docs/solutions/best-practices/macos-gesture-headless-capability-2026-06-10.md b/docs/solutions/best-practices/macos-gesture-headless-capability-2026-06-10.md index c816ff13..f561fe44 100644 --- a/docs/solutions/best-practices/macos-gesture-headless-capability-2026-06-10.md +++ b/docs/solutions/best-practices/macos-gesture-headless-capability-2026-06-10.md @@ -44,7 +44,7 @@ The answer is **per-gesture and per-platform**, because a gesture is headless-ca 2. **A new platform that exposes a headless path lights it up automatically — adapter-only change.** If a future Windows (UIA) or Linux (AT-SPI) adapter has a headless action for `double-click`/`triple-click`, it maps the `Action` there and the command succeeds headlessly on that platform with **zero change to the command or core**. The `InteractionPolicy` flows through the request; each adapter honors it per its own capabilities. The agent just sees success (or `POLICY_DENIED` → retry `--headed`) — it never needs to know the platform. -3. **`hover`/`drag`/`mouse-*` are modeled as raw cursor gestures, not semantic `Action`s** (they call `adapter.mouse_event`/`adapter.drag` with coordinates). They stay physical on every platform by design, because hovering/dragging *are* cursor operations universally. A semantic drag (AX reorder) would be a *new* `Action`, not a change to `drag`. When a gesture is ref-addressed, the target app is raised frontmost first **only under `--headed`** (`focus_for_physical_input`, gated on `InteractionPolicy::allow_focus_steal`; the response reports `"focused": true`). Headless gestures never change the frontmost app, and `--xy` input never focuses — the caller owns the target there. +3. **`hover`/`drag`/`mouse-*` are modeled as raw cursor gestures, not semantic `Action`s** (they call `adapter.mouse_event`/`adapter.drag` with coordinates). They stay physical on every platform by design, because hovering/dragging *are* cursor operations universally. A semantic drag (AX reorder) would be a *new* `Action`, not a change to `drag`. When a gesture is ref-addressed, the target app's frontmost state is ensured first **only under `--headed`** (`focus_for_physical_input`, gated on `InteractionPolicy::allow_focus_steal`; the response reports `"focused": true` when confirmed — already-frontmost apps skip the raise). Headless gestures never change the frontmost app, and `--xy` input never focuses — the caller owns the target there. 4. **`POLICY_DENIED` on a headless gesture is correct, not a bug** — it is the fail-closed signal that the headless AX path is unavailable and the caller must opt into `--headed`. Never widen the default policy to make it disappear. diff --git a/skills/agent-desktop-ffi/SKILL.md b/skills/agent-desktop-ffi/SKILL.md index 5f7321ba..9ecffc27 100644 --- a/skills/agent-desktop-ffi/SKILL.md +++ b/skills/agent-desktop-ffi/SKILL.md @@ -82,6 +82,12 @@ Four reference topics, loaded as needed: record `AdResult`, `ad_last_error_*`, action results, and host correlation IDs in their own logs. +- **No wait surface.** The CLI's `wait` command (element predicates including + `--predicate actionable --action ...`, window/text/menu/notification waits) + is not exposed over the C ABI. FFI hosts own their own polling loops; the + actionability preflight inside `ad_execute_ref_action_with_policy` is the + equivalent per-call readiness check. + - **Text input privacy.** On macOS, the focus-fallback or headed policy can use the clipboard briefly for non-ASCII text insertion. Keep the default headless policy or set values directly for sensitive text when the target supports it. diff --git a/skills/agent-desktop/SKILL.md b/skills/agent-desktop/SKILL.md index b884cd11..1917f01e 100644 --- a/skills/agent-desktop/SKILL.md +++ b/skills/agent-desktop/SKILL.md @@ -94,9 +94,9 @@ Use **progressive skeleton traversal** as the default approach. It reduces token - **Scoped invalidation:** re-drilling `--root @e3` only replaces refs from @e3's previous drill — refs from other regions and the skeleton itself are preserved - **Strict resolution:** stale refs return `STALE_REF`; duplicate plausible targets return `AMBIGUOUS_TARGET` instead of choosing arbitrarily. - **Actionability:** ref actions check live visibility, stability, enabled state, supported action, policy, and editability before dispatch. -- **Headless vs headed:** ref actions are headless by default (AX-only, no cursor) and fail closed with `POLICY_DENIED` when only a physical gesture would work. Pass the global `--headed` flag to permit cursor movement and focus stealing so the physical click/double-click/scroll/keypress fallbacks can complete; the AX path is still tried first, so `--headed` never regresses headless-capable elements. Raw-input commands (`press`, `hover`, `drag`, `mouse-*`, `key-down`/`key-up`) are always physical and ignore the mode. +- **Headless vs headed:** ref actions are headless by default (AX-only, no cursor) and fail closed with `POLICY_DENIED` when only a physical gesture would work. Pass the global `--headed` flag to permit cursor movement and focus stealing so the physical click/double-click/scroll/keypress fallbacks can complete; the AX path is still tried first, so `--headed` never regresses headless-capable elements. Raw-input commands (`press`, `hover`, `drag`, `mouse-*`, `key-down`/`key-up`) are always physical; the mode only gates whether a ref-addressed `hover`/`drag` may ensure the target app is frontmost first (`--headed` only, reported as `"focused": true`). - **Sessions:** use `--session ` for concurrent or multi-agent runs that share a latest snapshot pointer; batch entries may override with `"session": "id"`. -- **Trace:** use `--trace ` for JSONL diagnostics outside stdout; `--trace-strict` fails on trace setup and pre-action writes. Post-action success traces are best-effort because the desktop mutation already happened. Trace fields whose keys contain `text`, `value`, `expected`, `name`, `description`, `message`, `label`, `query`, `secret`, `token`, or `password` are redacted to `{ "redacted": true, "chars_bucket": "..." }` at every nesting depth — do not expect raw values in trace files. Top-level `--trace` is inherited by every `batch` entry, including entries with a `session` override. +- **Trace:** use `--trace ` for JSONL diagnostics outside stdout; `--trace-strict` fails on trace setup and pre-action writes. Post-action success traces are best-effort because the desktop mutation already happened. Trace fields whose keys contain `text`, `value`, `expected`, `name`, `description`, `message`, `label`, `query`, `secret`, `token`, `password`, `title`, `url`, `help`, or `placeholder` are redacted to `{ "redacted": true, "chars_bucket": "..." }` at every nesting depth (substring match, so composite keys like `source_window_title` redact too) — do not expect raw values in trace files. Top-level `--trace` is inherited by every `batch` entry, including entries with a `session` override. ## JSON Output Contract @@ -127,6 +127,7 @@ Exit codes: `0` success, `1` structured error, `2` argument error. | `TIMEOUT` | Wait condition not met | Increase --timeout | | `INVALID_ARGS` | Bad arguments | Check command syntax | | `NOTIFICATION_NOT_FOUND` | Notification index no longer exists | Re-run list-notifications | +| `INTERNAL` | Unexpected platform/OS failure (e.g. event synthesis failed) | Read `message`/`suggestion` for cleanup state, then retry once; persistent failures indicate an environment problem | ## Command Quick Reference (54 commands) diff --git a/skills/agent-desktop/references/commands-interaction.md b/skills/agent-desktop/references/commands-interaction.md index 067310a8..f38a7cc5 100644 --- a/skills/agent-desktop/references/commands-interaction.md +++ b/skills/agent-desktop/references/commands-interaction.md @@ -197,7 +197,7 @@ agent-desktop hover @e5 --duration 2000 Moves cursor to element center or absolute coordinates. Optional `--duration` holds position for N ms. This is an explicit cursor-moving command. -With `--headed`, a ref-addressed hover raises the target app frontmost before moving the cursor, and the response includes `"focused": true` when the raise happened. Headless (default) never changes the frontmost app. `--xy` input never focuses — the caller owns the target there. +With `--headed`, a ref-addressed hover ensures the target app is frontmost before moving the cursor (raising it if needed, best-effort), and the response includes `"focused": true` when that frontmost state was confirmed. The field is only ever present as `true`: absence means focus was never attempted (headless default, or `--xy` input — the caller owns the target there) or the best-effort raise could not be confirmed. ### drag ```bash @@ -218,7 +218,7 @@ agent-desktop drag --from @e1 --to @e5 --drop-delay 800 Can mix ref and coordinate sources (e.g., `--from @e1 --to-xy 400,500`). -With `--headed`, a ref-addressed `--from` raises the source app frontmost before the mouse-down (the destination app is never pre-focused — raising it could cover the source point), and the response includes `"focused": true` when the raise happened. Headless (default) never changes the frontmost app; coordinate-only drags never focus. +With `--headed`, a ref-addressed `--from` ensures the source app is frontmost before the mouse-down (the destination app is never pre-focused — raising it could cover the source point), and the response includes `"focused": true` when that frontmost state was confirmed. The field is only ever present as `true`: absence means focus was never attempted (headless default, or coordinate-only drags) or the best-effort raise could not be confirmed. For cross-app two-ref drags, ensure the destination window is visible (not fully occluded) before dragging — only the source app is raised. macOS drop targets often need the dragged item to dwell over them before they register as the drop destination — too short and the gesture lands as a drag with no drop. The default 500ms dwell suits most targets; raise `--drop-delay` (e.g. 800–1200) for sluggish destinations like list reorders or cross-window drops. The dwell posts continuous drag events over the destination so it stays highlighted, rather than a dead pause.