Creepso/agent-desktop
1
0
Fork 0
mirror of https://github.com/lahfir/agent-desktop.git synced 2026-08-04 13:16:06 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

docs: document new command surface and APP_UNRESPONSIVE

Browse source
This commit is contained in:
Lahfir 2026-07-03 04:53:36 -07:00
parent 34de713132
commit d5869be5b7
4 changed files with 115 additions and 7 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
CLAUDE.md
View file

@ -160,7 +160,7 @@ Batch is not a second dispatcher. `src/batch/mod.rs` deserializes JSON entries i
### Additive Phase Model
- **Phase 1:** Foundation + macOS MVP (56 commands, core engine, macOS adapter)
- **Phase 1:** Foundation + macOS MVP (58 commands, core engine, macOS adapter)
- **Phase 2:** Windows + Linux adapters, 10+ new commands — core untouched
- **Phase 3:** MCP server mode via `--mcp` flag — wraps existing commands
- **Phase 4:** Daemon, sessions, enterprise quality gates
@ -192,7 +192,7 @@ Phases 2–4 add adapters, transports, and production readiness work. Nothing in
PERM_DENIED, ELEMENT_NOT_FOUND, APP_NOT_FOUND, ACTION_FAILED,
ACTION_NOT_SUPPORTED, STALE_REF, AMBIGUOUS_TARGET, WINDOW_NOT_FOUND,
PLATFORM_NOT_SUPPORTED, TIMEOUT, INVALID_ARGS, NOTIFICATION_NOT_FOUND,
SNAPSHOT_NOT_FOUND, POLICY_DENIED, INTERNAL
SNAPSHOT_NOT_FOUND, POLICY_DENIED, APP_UNRESPONSIVE, INTERNAL
```
### Exit Codes
@ -369,10 +369,10 @@ for the actionability preflight (`get_live_*`), and `is_protected_process`
## Commands
56 commands spanning App/Window, Observation, Interaction, Scroll, Keyboard,
58 commands spanning App/Window, Observation, Interaction, Scroll, Keyboard,
Mouse, Notifications (macOS), Clipboard, Wait, System (including `session`), and
Batch. The full surface and per-command reference live in `skills/agent-desktop/`.
All 56 are implemented on macOS (Phase 1); Windows/Linux (Phase 2/3) target the
All 58 are implemented on macOS (Phase 1); Windows/Linux (Phase 2/3) target the
same surface. Adding a command: see the Extensibility Pattern above.
## Non-Goals

17
skills/agent-desktop/references/commands-interaction.md
View file

@ -284,6 +284,23 @@ Low-level press/release for custom drag or hold interactions.
| `--xy` | (required) | Coordinates as `x,y` |
| `--button` | left | `left`, `right`, `middle` |
### mouse-wheel
```bash
agent-desktop --headed mouse-wheel --x 500 --y 300
agent-desktop --headed mouse-wheel --x 500 --y 300 --dy 240
agent-desktop --headed mouse-wheel --x 500 --y 300 --dx -60 --dy 0
agent-desktop --headed mouse-wheel --x 500 --y 300 --modifiers shift
```
Synthesizes a scroll-wheel event at absolute coordinates, distinct from `scroll <ref>`: `scroll` targets an element through AX scroll semantics, `mouse-wheel` posts a raw wheel event at a screen point (for custom scroll surfaces or canvases with no AX scroll action). Held modifiers are applied to the event, so `--modifiers shift` produces the horizontal-scroll chord some apps expect.
| Flag | Default | Description |
|------|---------|-------------|
| `--x` | (required) | Absolute X coordinate |
| `--y` | (required) | Absolute Y coordinate |
| `--dy` | -120 | Vertical scroll delta |
| `--dx` | 0 | Horizontal scroll delta |
| `--modifiers` | | Held modifiers: `shift`, `cmd`, `ctrl`, `alt` (repeatable) |
## Choosing the Right Command
| Goal | Preferred | Alternative |

30
skills/agent-desktop/references/commands-observation.md
View file

@ -114,6 +114,10 @@ agent-desktop find --app "Safari" --text "Sign In" --first
agent-desktop find --app "App" --role checkbox --count
agent-desktop find --app "App" --role button --nth 2
agent-desktop find --app "App" --role button --limit 20
agent-desktop find --app "App" --role button --name "OK" --exact
agent-desktop find --app "App" --description "Closes the dialog"
agent-desktop find --app "App" --native-id "submitButton"
agent-desktop find --app "App" --state enabled --state focused=false
```
| Flag | Description |
@ -123,6 +127,10 @@ agent-desktop find --app "App" --role button --limit 20
| `--name` | Accessible name or label |
| `--value` | Current value |
| `--text` | Fuzzy match across name, value, title, and description |
| `--description` | Match by accessible description |
| `--native-id` | Match by native automation id (`AXIdentifier`) |
| `--exact` | Require exact (case-insensitive) matches for `--name`/`--description`/`--value` instead of fuzzy/substring matching |
| `--state TOKEN[=BOOL]` | Filter by state token; repeatable. Bare `TOKEN` requires the state present, `TOKEN=true`/`TOKEN=false` asserts its value (e.g. `--state enabled --state focused=false`) |
| `--first` | Return first match only |
| `--last` | Return last match only |
| `--nth N` | Return Nth match (0-indexed) |
@ -209,18 +217,40 @@ Capture a PNG screenshot of an application window.
agent-desktop screenshot --app "Finder"
agent-desktop screenshot --app "Finder" output.png
agent-desktop screenshot --window-id "w-1234" capture.png
agent-desktop screenshot --screen 0 display.png
```
| Flag | Description |
|------|-------------|
| `--app` | Application name |
| `--window-id` | Specific window ID |
| `--screen` | Capture display by index instead of an app window (from `list-displays`; `0` = primary) |
| (positional) | File path to save PNG (omit for base64 in JSON) |
When no output path is given, the screenshot is returned as a base64-encoded string in the JSON `data` field.
Screenshots require Screen Recording permission. Permission denial is reported as `PERM_DENIED`, not `INTERNAL`.
## list-displays
List connected displays with bounds and scale factor.
```bash
agent-desktop list-displays
```
Returns an array of `{ id, bounds: { x, y, width, height }, is_primary, scale }`, sorted primary-first. Use the array index (not `id`) with `screenshot --screen <index>` — `0` is always the primary display after sorting.
**Output:**
```json
{
"data": [
{ "id": "1", "bounds": { "x": 0, "y": 0, "width": 2560, "height": 1440 }, "is_primary": true, "scale": 2.0 },
{ "id": "2", "bounds": { "x": 2560, "y": 0, "width": 1920, "height": 1080 }, "is_primary": false, "scale": 1.0 }
]
}
```
## list-surfaces
List available accessibility surfaces for an application.

67
skills/agent-desktop/references/commands-system.md
View file

@ -8,12 +8,21 @@ App lifecycle, window management, notifications, clipboard, wait, and system hea
```bash
agent-desktop launch "System Settings"
agent-desktop launch "com.apple.Safari" --timeout 10000
agent-desktop launch "TextEdit" --arg /tmp/notes.txt
agent-desktop launch "MyTool" --arg --flag --arg value --env KEY=VALUE --cwd /tmp
agent-desktop launch "MyTool" --no-attach
```
Launches an application by name or bundle ID and waits until its window is visible.
| Flag | Default | Description |
|------|---------|-------------|
| `--timeout` | 30000 | Max wait time in ms for window to appear |
| `--arg` | | Command-line argument passed to the launched app; repeatable, order preserved |
| `--env` | | `KEY=VALUE` environment variable for the launched process; repeatable |
| `--cwd` | | Working directory for the launched process |
| `--no-attach` | false | Require a fresh launch instead of the default attach-if-running behavior |
By default, `launch` attaches to an already-running instance of the app (returning its window) instead of failing. `--no-attach` changes both branches: if the app is already running, the command returns `ACTION_FAILED` naming the running pid instead of attaching; if it is not running, the command returns immediately after spawning the process without polling for a window (the response has empty `id`/`title`), which is useful for apps that legitimately have no window (menu-bar-only utilities, background agents).
### close-app
```bash
@ -150,13 +159,39 @@ Blocks until a new notification appears (detects index-diff from a baseline capt
### clipboard-get
```bash
agent-desktop clipboard-get
agent-desktop clipboard-get --format auto
agent-desktop clipboard-get --format image --out /tmp/clip.png
agent-desktop clipboard-get --format file-urls
```
Returns `{ "data": { "text": "clipboard contents" } }`.
Reads a typed clipboard representation.
| Flag | Default | Description |
|------|---------|-------------|
| `--format` | text | Representation to read: `text`, `auto` (richest available: file references, then image, then text), `image`, `file-urls` |
| `--out` | private temp file | Where to write image bytes when `--format image`/`auto` resolves to an image; defaults to a private file under the active session's directory, or `~/.agent-desktop/tmp` with no active session |
**Output by format:**
```json
{ "data": { "type": "text", "text": "clipboard contents" } }
{ "data": { "type": "file_urls", "file_urls": ["/Users/me/Documents/report.pdf"] } }
{ "data": { "type": "image", "path": "/Users/me/.agent-desktop/sessions/<id>/clipboard/clipboard-...png", "width": 800, "height": 600 } }
```
When the pasteboard has nothing in the requested representation, the response is `{ "data": { "type": "<requested format>", "found": false } }` with no other payload fields.
### clipboard-set
```bash
agent-desktop clipboard-set "Hello, world!"
agent-desktop clipboard-set --image /tmp/screenshot.png
agent-desktop clipboard-set --file-url /Users/me/Documents/report.pdf
agent-desktop clipboard-set --file-url /tmp/a.txt --file-url /tmp/b.txt
```
Writes typed content to the clipboard. `--file-url` (repeatable) and `--image` each take priority over the positional text argument when present; only one representation is written per call.
| Flag | Description |
|------|-------------|
| (positional) | Text to write (ignored if `--image` or `--file-url` is given) |
| `--image` | Path to a PNG file to write to the clipboard |
| `--file-url` | File path to write as a file reference; repeatable. Every path must exist on disk or the command returns `INVALID_ARGS` |
### clipboard-clear
```bash
@ -207,6 +242,30 @@ agent-desktop wait --menu-closed --app "Finder" --timeout 3000
```
Blocks until the menu surface is dismissed.
### wait (event)
```bash
agent-desktop wait --event window-opened --app "Finder" --timeout 10000
agent-desktop wait --event window-closed --window-id "w-1234" --timeout 10000
agent-desktop wait --event app-launched --app "Safari" --timeout 15000
agent-desktop wait --event app-terminated --app "Safari" --timeout 15000
agent-desktop wait --event focus-changed --timeout 10000
agent-desktop wait --event surface-appeared --app "Finder" --timeout 5000
agent-desktop wait --event window-opened --window "Untitled" --timeout 10000
```
Blocks until a desktop lifecycle signal is observed, detected by diffing a baseline captured at wait start against fresh reads — no need to know a new window's id or title up front. `--window-id`/`--window` are optional narrowing filters on top of `--event`, never a requirement by themselves (bare `--window` without `--event` instead selects the `wait (window)` mode above).
| Token | Fires when |
|-------|------------|
| `window-opened` | A window not present in the baseline appears |
| `window-closed` | A baseline window disappears |
| `app-launched` | A process not present in the baseline starts |
| `app-terminated` | A baseline process exits |
| `focus-changed` | The OS-focused window differs from the baseline's |
| `surface-appeared` | A menu/sheet/popover/alert surface count increases |
| `surface-dismissed` | A menu/sheet/popover/alert surface count decreases |
Transient errors (timeouts, element-not-found) are retried within the `--timeout` budget for both the baseline capture and polling; other errors fail immediately. Timeout errors include `baseline_counts` and, when a poll errored, `last_error`.
| Flag | Default | Description |
|------|---------|-------------|
| (positional) | | Milliseconds to pause |
@ -216,12 +275,14 @@ Blocks until the menu surface is dismissed.
| `--value` | | Expected text for `--predicate value` |
| `--action` | click | Action checked by `--predicate actionable`: `click`, `type`, `set-value`, `clear` |
| `--count` | | Expected match count for `--text` waits |
| `--window` | | Window title to wait for |
| `--window` | | Window title to wait for; with `--event`, narrows the event to that window's title instead of selecting a mode |
| `--text` | | Text to wait for; with `--notification`, filters notification title/body |
| `--menu` | false | Wait for menu surface to open |
| `--menu-closed` | false | Wait for menu surface to close |
| `--notification` | false | Wait for a new notification |
| `--timeout` | 30000 | Timeout in ms (for element/window/text/menu waits) |
| `--event` | | Desktop lifecycle signal to wait for: `window-opened`, `window-closed`, `app-launched`, `app-terminated`, `focus-changed`, `surface-appeared`, `surface-dismissed` |
| `--window-id` | | Narrows `--event` to one window ID (window/focus events only) |
| `--timeout` | 30000 | Timeout in ms (for element/window/text/menu/event waits) |
| `--app` | | Scope the wait to a specific application |
## Batch