From d5869be5b70e400c2d6cb2fd424cec46da65490d Mon Sep 17 00:00:00 2001 From: Lahfir Date: Fri, 3 Jul 2026 04:53:36 -0700 Subject: [PATCH] docs: document new command surface and APP_UNRESPONSIVE --- CLAUDE.md | 8 +-- .../references/commands-interaction.md | 17 +++++ .../references/commands-observation.md | 30 +++++++++ .../references/commands-system.md | 67 ++++++++++++++++++- 4 files changed, 115 insertions(+), 7 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 29a1b28..9185c43 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -160,7 +160,7 @@ Batch is not a second dispatcher. `src/batch/mod.rs` deserializes JSON entries i ### Additive Phase Model -- **Phase 1:** Foundation + macOS MVP (56 commands, core engine, macOS adapter) +- **Phase 1:** Foundation + macOS MVP (58 commands, core engine, macOS adapter) - **Phase 2:** Windows + Linux adapters, 10+ new commands — core untouched - **Phase 3:** MCP server mode via `--mcp` flag — wraps existing commands - **Phase 4:** Daemon, sessions, enterprise quality gates @@ -192,7 +192,7 @@ Phases 2–4 add adapters, transports, and production readiness work. Nothing in PERM_DENIED, ELEMENT_NOT_FOUND, APP_NOT_FOUND, ACTION_FAILED, ACTION_NOT_SUPPORTED, STALE_REF, AMBIGUOUS_TARGET, WINDOW_NOT_FOUND, PLATFORM_NOT_SUPPORTED, TIMEOUT, INVALID_ARGS, NOTIFICATION_NOT_FOUND, -SNAPSHOT_NOT_FOUND, POLICY_DENIED, INTERNAL +SNAPSHOT_NOT_FOUND, POLICY_DENIED, APP_UNRESPONSIVE, INTERNAL ``` ### Exit Codes @@ -369,10 +369,10 @@ for the actionability preflight (`get_live_*`), and `is_protected_process` ## Commands -56 commands spanning App/Window, Observation, Interaction, Scroll, Keyboard, +58 commands spanning App/Window, Observation, Interaction, Scroll, Keyboard, Mouse, Notifications (macOS), Clipboard, Wait, System (including `session`), and Batch. The full surface and per-command reference live in `skills/agent-desktop/`. -All 56 are implemented on macOS (Phase 1); Windows/Linux (Phase 2/3) target the +All 58 are implemented on macOS (Phase 1); Windows/Linux (Phase 2/3) target the same surface. Adding a command: see the Extensibility Pattern above. ## Non-Goals diff --git a/skills/agent-desktop/references/commands-interaction.md b/skills/agent-desktop/references/commands-interaction.md index 410122a..bbff610 100644 --- a/skills/agent-desktop/references/commands-interaction.md +++ b/skills/agent-desktop/references/commands-interaction.md @@ -284,6 +284,23 @@ Low-level press/release for custom drag or hold interactions. | `--xy` | (required) | Coordinates as `x,y` | | `--button` | left | `left`, `right`, `middle` | +### mouse-wheel +```bash +agent-desktop --headed mouse-wheel --x 500 --y 300 +agent-desktop --headed mouse-wheel --x 500 --y 300 --dy 240 +agent-desktop --headed mouse-wheel --x 500 --y 300 --dx -60 --dy 0 +agent-desktop --headed mouse-wheel --x 500 --y 300 --modifiers shift +``` +Synthesizes a scroll-wheel event at absolute coordinates, distinct from `scroll `: `scroll` targets an element through AX scroll semantics, `mouse-wheel` posts a raw wheel event at a screen point (for custom scroll surfaces or canvases with no AX scroll action). Held modifiers are applied to the event, so `--modifiers shift` produces the horizontal-scroll chord some apps expect. + +| Flag | Default | Description | +|------|---------|-------------| +| `--x` | (required) | Absolute X coordinate | +| `--y` | (required) | Absolute Y coordinate | +| `--dy` | -120 | Vertical scroll delta | +| `--dx` | 0 | Horizontal scroll delta | +| `--modifiers` | | Held modifiers: `shift`, `cmd`, `ctrl`, `alt` (repeatable) | + ## Choosing the Right Command | Goal | Preferred | Alternative | diff --git a/skills/agent-desktop/references/commands-observation.md b/skills/agent-desktop/references/commands-observation.md index ab2b1e1..9803c94 100644 --- a/skills/agent-desktop/references/commands-observation.md +++ b/skills/agent-desktop/references/commands-observation.md @@ -114,6 +114,10 @@ agent-desktop find --app "Safari" --text "Sign In" --first agent-desktop find --app "App" --role checkbox --count agent-desktop find --app "App" --role button --nth 2 agent-desktop find --app "App" --role button --limit 20 +agent-desktop find --app "App" --role button --name "OK" --exact +agent-desktop find --app "App" --description "Closes the dialog" +agent-desktop find --app "App" --native-id "submitButton" +agent-desktop find --app "App" --state enabled --state focused=false ``` | Flag | Description | @@ -123,6 +127,10 @@ agent-desktop find --app "App" --role button --limit 20 | `--name` | Accessible name or label | | `--value` | Current value | | `--text` | Fuzzy match across name, value, title, and description | +| `--description` | Match by accessible description | +| `--native-id` | Match by native automation id (`AXIdentifier`) | +| `--exact` | Require exact (case-insensitive) matches for `--name`/`--description`/`--value` instead of fuzzy/substring matching | +| `--state TOKEN[=BOOL]` | Filter by state token; repeatable. Bare `TOKEN` requires the state present, `TOKEN=true`/`TOKEN=false` asserts its value (e.g. `--state enabled --state focused=false`) | | `--first` | Return first match only | | `--last` | Return last match only | | `--nth N` | Return Nth match (0-indexed) | @@ -209,18 +217,40 @@ Capture a PNG screenshot of an application window. agent-desktop screenshot --app "Finder" agent-desktop screenshot --app "Finder" output.png agent-desktop screenshot --window-id "w-1234" capture.png +agent-desktop screenshot --screen 0 display.png ``` | Flag | Description | |------|-------------| | `--app` | Application name | | `--window-id` | Specific window ID | +| `--screen` | Capture display by index instead of an app window (from `list-displays`; `0` = primary) | | (positional) | File path to save PNG (omit for base64 in JSON) | When no output path is given, the screenshot is returned as a base64-encoded string in the JSON `data` field. Screenshots require Screen Recording permission. Permission denial is reported as `PERM_DENIED`, not `INTERNAL`. +## list-displays + +List connected displays with bounds and scale factor. + +```bash +agent-desktop list-displays +``` + +Returns an array of `{ id, bounds: { x, y, width, height }, is_primary, scale }`, sorted primary-first. Use the array index (not `id`) with `screenshot --screen ` — `0` is always the primary display after sorting. + +**Output:** +```json +{ + "data": [ + { "id": "1", "bounds": { "x": 0, "y": 0, "width": 2560, "height": 1440 }, "is_primary": true, "scale": 2.0 }, + { "id": "2", "bounds": { "x": 2560, "y": 0, "width": 1920, "height": 1080 }, "is_primary": false, "scale": 1.0 } + ] +} +``` + ## list-surfaces List available accessibility surfaces for an application. diff --git a/skills/agent-desktop/references/commands-system.md b/skills/agent-desktop/references/commands-system.md index 56e7751..b4fe575 100644 --- a/skills/agent-desktop/references/commands-system.md +++ b/skills/agent-desktop/references/commands-system.md @@ -8,12 +8,21 @@ App lifecycle, window management, notifications, clipboard, wait, and system hea ```bash agent-desktop launch "System Settings" agent-desktop launch "com.apple.Safari" --timeout 10000 +agent-desktop launch "TextEdit" --arg /tmp/notes.txt +agent-desktop launch "MyTool" --arg --flag --arg value --env KEY=VALUE --cwd /tmp +agent-desktop launch "MyTool" --no-attach ``` Launches an application by name or bundle ID and waits until its window is visible. | Flag | Default | Description | |------|---------|-------------| | `--timeout` | 30000 | Max wait time in ms for window to appear | +| `--arg` | | Command-line argument passed to the launched app; repeatable, order preserved | +| `--env` | | `KEY=VALUE` environment variable for the launched process; repeatable | +| `--cwd` | | Working directory for the launched process | +| `--no-attach` | false | Require a fresh launch instead of the default attach-if-running behavior | + +By default, `launch` attaches to an already-running instance of the app (returning its window) instead of failing. `--no-attach` changes both branches: if the app is already running, the command returns `ACTION_FAILED` naming the running pid instead of attaching; if it is not running, the command returns immediately after spawning the process without polling for a window (the response has empty `id`/`title`), which is useful for apps that legitimately have no window (menu-bar-only utilities, background agents). ### close-app ```bash @@ -150,13 +159,39 @@ Blocks until a new notification appears (detects index-diff from a baseline capt ### clipboard-get ```bash agent-desktop clipboard-get +agent-desktop clipboard-get --format auto +agent-desktop clipboard-get --format image --out /tmp/clip.png +agent-desktop clipboard-get --format file-urls ``` -Returns `{ "data": { "text": "clipboard contents" } }`. +Reads a typed clipboard representation. + +| Flag | Default | Description | +|------|---------|-------------| +| `--format` | text | Representation to read: `text`, `auto` (richest available: file references, then image, then text), `image`, `file-urls` | +| `--out` | private temp file | Where to write image bytes when `--format image`/`auto` resolves to an image; defaults to a private file under the active session's directory, or `~/.agent-desktop/tmp` with no active session | + +**Output by format:** +```json +{ "data": { "type": "text", "text": "clipboard contents" } } +{ "data": { "type": "file_urls", "file_urls": ["/Users/me/Documents/report.pdf"] } } +{ "data": { "type": "image", "path": "/Users/me/.agent-desktop/sessions//clipboard/clipboard-...png", "width": 800, "height": 600 } } +``` +When the pasteboard has nothing in the requested representation, the response is `{ "data": { "type": "", "found": false } }` with no other payload fields. ### clipboard-set ```bash agent-desktop clipboard-set "Hello, world!" +agent-desktop clipboard-set --image /tmp/screenshot.png +agent-desktop clipboard-set --file-url /Users/me/Documents/report.pdf +agent-desktop clipboard-set --file-url /tmp/a.txt --file-url /tmp/b.txt ``` +Writes typed content to the clipboard. `--file-url` (repeatable) and `--image` each take priority over the positional text argument when present; only one representation is written per call. + +| Flag | Description | +|------|-------------| +| (positional) | Text to write (ignored if `--image` or `--file-url` is given) | +| `--image` | Path to a PNG file to write to the clipboard | +| `--file-url` | File path to write as a file reference; repeatable. Every path must exist on disk or the command returns `INVALID_ARGS` | ### clipboard-clear ```bash @@ -207,6 +242,30 @@ agent-desktop wait --menu-closed --app "Finder" --timeout 3000 ``` Blocks until the menu surface is dismissed. +### wait (event) +```bash +agent-desktop wait --event window-opened --app "Finder" --timeout 10000 +agent-desktop wait --event window-closed --window-id "w-1234" --timeout 10000 +agent-desktop wait --event app-launched --app "Safari" --timeout 15000 +agent-desktop wait --event app-terminated --app "Safari" --timeout 15000 +agent-desktop wait --event focus-changed --timeout 10000 +agent-desktop wait --event surface-appeared --app "Finder" --timeout 5000 +agent-desktop wait --event window-opened --window "Untitled" --timeout 10000 +``` +Blocks until a desktop lifecycle signal is observed, detected by diffing a baseline captured at wait start against fresh reads — no need to know a new window's id or title up front. `--window-id`/`--window` are optional narrowing filters on top of `--event`, never a requirement by themselves (bare `--window` without `--event` instead selects the `wait (window)` mode above). + +| Token | Fires when | +|-------|------------| +| `window-opened` | A window not present in the baseline appears | +| `window-closed` | A baseline window disappears | +| `app-launched` | A process not present in the baseline starts | +| `app-terminated` | A baseline process exits | +| `focus-changed` | The OS-focused window differs from the baseline's | +| `surface-appeared` | A menu/sheet/popover/alert surface count increases | +| `surface-dismissed` | A menu/sheet/popover/alert surface count decreases | + +Transient errors (timeouts, element-not-found) are retried within the `--timeout` budget for both the baseline capture and polling; other errors fail immediately. Timeout errors include `baseline_counts` and, when a poll errored, `last_error`. + | Flag | Default | Description | |------|---------|-------------| | (positional) | | Milliseconds to pause | @@ -216,12 +275,14 @@ Blocks until the menu surface is dismissed. | `--value` | | Expected text for `--predicate value` | | `--action` | click | Action checked by `--predicate actionable`: `click`, `type`, `set-value`, `clear` | | `--count` | | Expected match count for `--text` waits | -| `--window` | | Window title to wait for | +| `--window` | | Window title to wait for; with `--event`, narrows the event to that window's title instead of selecting a mode | | `--text` | | Text to wait for; with `--notification`, filters notification title/body | | `--menu` | false | Wait for menu surface to open | | `--menu-closed` | false | Wait for menu surface to close | | `--notification` | false | Wait for a new notification | -| `--timeout` | 30000 | Timeout in ms (for element/window/text/menu waits) | +| `--event` | | Desktop lifecycle signal to wait for: `window-opened`, `window-closed`, `app-launched`, `app-terminated`, `focus-changed`, `surface-appeared`, `surface-dismissed` | +| `--window-id` | | Narrows `--event` to one window ID (window/focus events only) | +| `--timeout` | 30000 | Timeout in ms (for element/window/text/menu/event waits) | | `--app` | | Scope the wait to a specific application | ## Batch