From ad91cd32cf2de1c1c8dcda4c0dcae37f0022b4c6 Mon Sep 17 00:00:00 2001 From: Lahfir Date: Mon, 23 Feb 2026 01:51:22 -0800 Subject: [PATCH] feat: add Claude Code skills for agent-desktop automation Graph of skills covering all 50 commands with platform-specific sub-skills: - Main index skill with core concepts, ref system, JSON contract - Observation, interaction, and system command references - Common workflow patterns (forms, menus, dialogs, drag-drop) - macOS platform skill (permissions, AX API, troubleshooting) - /desktop slash command for quick reference - PRD addendum for skill maintenance across phases --- .claude/commands/desktop.md | 78 ++++++ .claude/skills/agent-desktop-macos/SKILL.md | 209 ++++++++++++++++ .claude/skills/agent-desktop/SKILL.md | 220 +++++++++++++++++ .../agent-desktop/commands-interaction.md | 223 ++++++++++++++++++ .../agent-desktop/commands-observation.md | 182 ++++++++++++++ .../skills/agent-desktop/commands-system.md | 184 +++++++++++++++ .claude/skills/agent-desktop/workflows.md | 219 +++++++++++++++++ .gitignore | 5 +- docs/prd-addendum-skill-maintenance.md | 69 ++++++ 9 files changed, 1387 insertions(+), 2 deletions(-) create mode 100644 .claude/commands/desktop.md create mode 100644 .claude/skills/agent-desktop-macos/SKILL.md create mode 100644 .claude/skills/agent-desktop/SKILL.md create mode 100644 .claude/skills/agent-desktop/commands-interaction.md create mode 100644 .claude/skills/agent-desktop/commands-observation.md create mode 100644 .claude/skills/agent-desktop/commands-system.md create mode 100644 .claude/skills/agent-desktop/workflows.md create mode 100644 docs/prd-addendum-skill-maintenance.md diff --git a/.claude/commands/desktop.md b/.claude/commands/desktop.md new file mode 100644 index 0000000..3dbdbb1 --- /dev/null +++ b/.claude/commands/desktop.md @@ -0,0 +1,78 @@ +# agent-desktop Quick Reference + +Use this command when you need to automate desktop applications with agent-desktop. + +## Core Loop + +```bash +# 1. OBSERVE +agent-desktop snapshot --app "$APP" -i + +# 2. Find target element by parsing JSON output (look for ref like @e5) + +# 3. ACT +agent-desktop click @e5 # or type, select, toggle, etc. + +# 4. VERIFY +agent-desktop snapshot --app "$APP" -i +``` + +## Common Tasks + +**Click a button:** +```bash +agent-desktop snapshot --app "App" -i +agent-desktop click @e5 +``` + +**Fill a text field:** +```bash +agent-desktop clear @e2 +agent-desktop type @e2 "new text" +``` + +**Toggle a checkbox:** +```bash +agent-desktop check @e3 # idempotent on +agent-desktop uncheck @e3 # idempotent off +``` + +**Open context menu:** +```bash +agent-desktop right-click @e5 +agent-desktop wait --menu --app "App" +agent-desktop snapshot --app "App" --surface menu -i +agent-desktop click @e7 +``` + +**Navigate menus:** +```bash +agent-desktop snapshot --app "App" --surface menubar -i +agent-desktop click @e1 # File menu +agent-desktop wait --menu --app "App" +agent-desktop snapshot --app "App" --surface menu -i +agent-desktop click @e5 # Menu item +``` + +**Wait for UI:** +```bash +agent-desktop wait --text "Done" --app "App" --timeout 5000 +agent-desktop wait --window "Save" --timeout 5000 +``` + +## First-Time Setup + +```bash +agent-desktop permissions --request +# Grant in System Settings > Privacy & Security > Accessibility +``` + +## Detailed Skills + +For comprehensive reference, see: +- `.claude/skills/agent-desktop/SKILL.md` — Core concepts and full command index +- `.claude/skills/agent-desktop/commands-observation.md` — snapshot, find, get, is, screenshot +- `.claude/skills/agent-desktop/commands-interaction.md` — click, type, select, toggle, scroll, drag +- `.claude/skills/agent-desktop/commands-system.md` — launch, close, windows, clipboard, wait, batch +- `.claude/skills/agent-desktop/workflows.md` — Common automation patterns +- `.claude/skills/agent-desktop-macos/SKILL.md` — macOS permissions, troubleshooting, AX details diff --git a/.claude/skills/agent-desktop-macos/SKILL.md b/.claude/skills/agent-desktop-macos/SKILL.md new file mode 100644 index 0000000..a9db2bc --- /dev/null +++ b/.claude/skills/agent-desktop-macos/SKILL.md @@ -0,0 +1,209 @@ +--- +name: agent-desktop-macos +description: > + macOS platform details for agent-desktop. Use when troubleshooting macOS accessibility + permissions (TCC), debugging AX API failures, understanding the smart activation chain, + working with macOS surfaces (menu, sheet, alert), or handling macOS-specific errors like + PERM_DENIED, ACTION_FAILED, or empty accessibility trees. +--- + +# agent-desktop: macOS Platform + +macOS-specific details for agent-desktop. Covers permissions, accessibility API behavior, troubleshooting, and platform quirks. + +## Prerequisites + +### Accessibility Permission (TCC) + +macOS requires explicit Accessibility permission for any process that reads or controls UI elements across applications. + +```bash +# Check permission status +agent-desktop permissions + +# Trigger the system dialog +agent-desktop permissions --request +``` + +**To grant manually:** +1. Open System Settings > Privacy & Security > Accessibility +2. Click the lock to make changes +3. Add your terminal application (Terminal.app, iTerm2, Warp, VS Code, etc.) +4. Toggle it ON + +**Important:** The permission is granted to the **terminal application**, not to agent-desktop itself. If you run agent-desktop from a different terminal, you need to grant that terminal too. + +After granting, restart the terminal for the permission to take effect. + +### Supported macOS Versions + +- macOS 12 (Monterey) and later +- Both Intel (x86_64) and Apple Silicon (ARM64) + +## macOS Accessibility API (AX) + +agent-desktop uses the macOS Accessibility API (`AXUIElement`) to read and manipulate UI elements. + +### How It Works + +1. `AXUIElementCreateApplication(pid)` creates a handle to an app's accessibility tree +2. Tree traversal reads `kAXChildrenAttribute` recursively +3. Attributes like `kAXRoleAttribute`, `kAXNameAttribute`, `kAXValueAttribute` provide element details +4. Actions like `kAXPressAction`, `kAXConfirmAction` trigger element behavior + +### Smart Activation Chain + +When you run `click @ref`, agent-desktop doesn't just do a simple click. It runs a multi-step activation chain: + +1. **AXScrollToVisible** — ensure element is on screen +2. **AXPress** — standard press action +3. **AXConfirm** — confirmation action +4. **AXOpen** — open action (for links, files) +5. **AXPick** — picker action +6. **AXShowAlternateUI** — reveal hidden UI, then press child +7. **Child activation** — try pressing child elements +8. **AXSelected** — set selected attribute +9. **Select via parent** — set parent's selected rows (for tables/lists) +10. **Custom actions** — AXPerformCustomAction +11. **Focus + activate** — set focus then press/confirm +12. **Keyboard activate** — focus + synthesize space key +13. **Parent activation** — try pressing ancestor elements +14. **Coordinate click** — final fallback: CGEvent click at element bounds center + +For `right-click`, the chain tries AXShowMenu first, then various focus/select combinations before falling back to a coordinate-based right-click. + +### Surfaces + +macOS apps can have multiple accessibility surfaces: + +| Surface | Description | When to use | +|---------|-------------|-------------| +| `window` | Main application window (default) | General UI interaction | +| `focused` | Currently focused element's context | Inspecting active element | +| `menu` | Open dropdown or context menu | After click/right-click on menu triggers | +| `menubar` | Application menu bar | Navigating File/Edit/View menus | +| `sheet` | Modal sheet (Save dialog, etc.) | After triggering sheet dialogs | +| `popover` | Popover/popup content | Inspecting tooltips, popovers | +| `alert` | System or app alert | Handling alert dialogs | + +```bash +# List available surfaces +agent-desktop list-surfaces --app "Finder" + +# Snapshot a specific surface +agent-desktop snapshot --app "Finder" --surface menu -i +``` + +## Troubleshooting + +### PERM_DENIED + +``` +"code": "PERM_DENIED", +"message": "Accessibility permission not granted" +``` + +**Fix:** +1. `agent-desktop permissions --request` to trigger dialog +2. System Settings > Privacy & Security > Accessibility +3. Add and enable your terminal +4. Restart terminal + +### Empty or Sparse Tree + +Some apps don't expose full accessibility trees: +- **Electron apps** (VS Code, Slack): Generally good accessibility support +- **Custom-rendered UIs** (games, some creative tools): May have no accessibility tree +- **Web views**: Content inside WebViews may be limited + +**Try:** +- Remove `-i` flag to see all elements including non-interactive ones +- Increase `--max-depth` to explore deeper +- Use `screenshot` as a visual fallback + +### STALE_REF + +``` +"code": "STALE_REF", +"message": "RefMap is from a previous snapshot" +``` + +The UI changed between your snapshot and action. Run `snapshot` again and use the new refs. + +### ACTION_FAILED + +``` +"code": "ACTION_FAILED" +``` + +The accessibility action was rejected. This can happen when: +- The element is disabled +- The app is busy or unresponsive +- The element doesn't support the requested action + +**Try:** +1. Check `is @ref --property enabled` first +2. Try coordinate-based click: get bounds with `get @ref --property bounds`, then `mouse-click --xy x,y` +3. Use keyboard: `focus @ref` then `press return` + +### APP_NOT_FOUND + +The application isn't running. Launch it first: +```bash +agent-desktop launch "App Name" +``` + +### Slow Snapshots + +Large apps (Xcode, Safari with many tabs) can have deep trees. + +**Optimize:** +- Use `-i` to filter to interactive elements only +- Use `--max-depth 5` to limit depth +- Use `--compact` to remove empty structural nodes +- Target a specific window with `--window-id` +- Use `find` instead of full snapshot when you know what you're looking for + +### Context Menu Doesn't Appear + +After `right-click @ref`, if no menu appears: +1. The element may not support context menus +2. The app may need to be focused first: `focus-window --app "App"` before right-clicking +3. Try `mouse-click --xy x,y --button right` with coordinates from `get @ref --property bounds` + +## macOS-Specific Behavior + +### App Identification + +Apps can be referenced by: +- **Display name:** `"System Settings"`, `"Finder"`, `"TextEdit"` +- **Bundle ID:** `"com.apple.systempreferences"`, `"com.apple.finder"` + +`launch` accepts both. Other commands use the display name with `--app`. + +### Window IDs + +Window IDs (like `w-4521`) are assigned by macOS and are stable for the lifetime of the window. Use `list-windows` to discover them. + +### Keyboard Shortcuts + +macOS uses `cmd` (Command) as the primary modifier: +```bash +agent-desktop press cmd+c # Copy +agent-desktop press cmd+v # Paste +agent-desktop press cmd+z # Undo +agent-desktop press cmd+s # Save +agent-desktop press cmd+w # Close window +agent-desktop press cmd+q # Quit app (use with caution!) +``` + +### Full-Screen Apps + +Apps in full-screen mode are accessible but may behave differently: +- `list-windows` still shows them +- Bounds may report full screen dimensions +- Some animations may delay UI state updates (use `wait`) + +### Menu Bar Apps + +Menu bar apps (status bar items) can be accessed via `--surface menubar`. The menu bar is a separate surface from the application window. diff --git a/.claude/skills/agent-desktop/SKILL.md b/.claude/skills/agent-desktop/SKILL.md new file mode 100644 index 0000000..38ba117 --- /dev/null +++ b/.claude/skills/agent-desktop/SKILL.md @@ -0,0 +1,220 @@ +--- +name: agent-desktop +description: > + Desktop automation via native OS accessibility trees using the agent-desktop CLI. + Use when an AI agent needs to observe, interact with, or automate desktop applications + (click buttons, fill forms, navigate menus, read UI state, toggle checkboxes, scroll, + drag, type text, take screenshots, manage windows, use clipboard). Covers 50 commands + across observation, interaction, keyboard/mouse, app lifecycle, clipboard, and wait. + Triggers on: "click button", "fill form", "open app", "read UI", "automate desktop", + "accessibility tree", "snapshot app", "type into field", "navigate menu", "toggle checkbox", + "take screenshot", "desktop automation", "agent-desktop", or any desktop GUI interaction task. + Supports macOS (Phase 1), with Windows and Linux planned. +--- + +# agent-desktop + +Cross-platform CLI tool enabling AI agents to observe and control desktop applications via native OS accessibility trees. + +**Core principle:** agent-desktop is NOT an AI agent. It is a tool that AI agents invoke. It outputs structured JSON with ref-based element identifiers. The observation-action loop lives in the calling agent. + +## When to Use + +Use agent-desktop when you need to: +- Read UI state from desktop applications (buttons, text fields, menus, checkboxes) +- Interact with desktop app elements (click, type, select, toggle, scroll) +- Automate multi-step desktop workflows (fill forms, navigate menus, transfer data) +- Wait for UI state changes before proceeding +- Take screenshots of application windows + +Do NOT use agent-desktop for: +- Web browser automation (use agent-browser instead) +- Custom-rendered or game-engine UIs lacking accessibility exposure +- Applications that don't expose accessibility trees + +## Skill Graph + +This skill is the index. Platform-specific details and advanced patterns are in sub-skills: + +| Skill | When to use | +|-------|-------------| +| `agent-desktop-macos` | macOS permissions, AX API quirks, troubleshooting | +| `commands-observation.md` | Snapshot, find, get, is, screenshot, list-surfaces | +| `commands-interaction.md` | Click, type, set-value, select, toggle, expand, scroll, drag | +| `commands-system.md` | Launch, close, windows, clipboard, wait, batch | +| `workflows.md` | Common automation patterns (forms, menus, dialogs, navigation) | + +## The Observe-Act Loop + +Every automation follows this pattern: + +``` +1. OBSERVE → agent-desktop snapshot --app "App Name" -i +2. REASON → Parse JSON, find target element by ref (@e1, @e2...) +3. ACT → agent-desktop click @e5 (or type, select, toggle...) +4. VERIFY → agent-desktop snapshot again to confirm state change +5. REPEAT → Continue until task is complete +``` + +Always snapshot before acting. Refs are snapshot-scoped and become stale after UI changes. + +## Ref System + +- Refs are assigned in depth-first document order: `@e1`, `@e2`, `@e3`... +- Only interactive elements receive refs: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell +- Static text, groups, and containers do NOT get refs (they remain in the tree for context) +- Refs are deterministic within a snapshot but NOT stable across snapshots if the UI changed +- After any action that changes UI, run `snapshot` again to get fresh refs + +## JSON Output Contract + +Every command returns a JSON envelope on stdout: + +### Success +```json +{ + "version": "1.0", + "ok": true, + "command": "snapshot", + "data": { ... } +} +``` + +### Error +```json +{ + "version": "1.0", + "ok": false, + "command": "click", + "error": { + "code": "STALE_REF", + "message": "RefMap is from a previous snapshot", + "suggestion": "Run 'snapshot' to refresh, then retry with updated ref" + } +} +``` + +### Exit Codes +- `0` — success (check `ok: true`) +- `1` — structured error (JSON with error code) +- `2` — argument/parse error + +### Error Codes +| Code | Meaning | Recovery | +|------|---------|----------| +| `PERM_DENIED` | Accessibility permission not granted | Grant in System Settings > Privacy > Accessibility | +| `ELEMENT_NOT_FOUND` | Ref not found in current refmap | Re-run snapshot, use fresh ref | +| `APP_NOT_FOUND` | Application not running | Launch it first with `launch` | +| `ACTION_FAILED` | Accessibility action rejected | Try alternative approach (different action or coordinate-based) | +| `ACTION_NOT_SUPPORTED` | Element doesn't support this action | Check available actions, use different command | +| `STALE_REF` | Ref from old snapshot | Re-run snapshot to get fresh refs | +| `WINDOW_NOT_FOUND` | No matching window | Check app name, use `list-windows` | +| `PLATFORM_NOT_SUPPORTED` | Feature not available on this OS | Check platform support | +| `TIMEOUT` | Wait condition not met in time | Increase `--timeout` or check condition | +| `INVALID_ARGS` | Bad arguments | Check command syntax | +| `INTERNAL` | Unexpected internal error | Report bug with `-v` verbose output | + +## Command Quick Reference + +### Observation (see commands-observation.md) +``` +snapshot --app "App" -i # Accessibility tree with refs +screenshot --app "App" out.png # PNG screenshot +find --app "App" --role button # Search elements +get @e1 --property text # Read element property +is @e1 --property enabled # Check element state +list-surfaces --app "App" # Available surfaces +``` + +### Interaction (see commands-interaction.md) +``` +click @e5 # Click element +double-click @e3 # Double-click +right-click @e5 # Right-click (context menu) +type @e2 "hello" # Type text into element +set-value @e2 "new value" # Set value directly +clear @e2 # Clear element value +focus @e2 # Set keyboard focus +select @e4 "Option B" # Select dropdown option +toggle @e6 # Toggle checkbox/switch +check @e6 # Idempotent check +uncheck @e6 # Idempotent uncheck +expand @e7 # Expand disclosure +collapse @e7 # Collapse disclosure +scroll @e1 --direction down # Scroll element +scroll-to @e8 # Scroll element into view +``` + +### Keyboard & Mouse (see commands-interaction.md) +``` +press cmd+c # Key combo +press return --app "App" # Key combo targeted at app +key-down shift # Hold key +key-up shift # Release key +hover @e5 # Move cursor to element +hover --xy 500,300 # Move cursor to coordinates +drag --from @e1 --to @e5 # Drag between elements +mouse-click --xy 500,300 # Click at coordinates +mouse-move --xy 100,200 # Move cursor +mouse-down --xy 100,200 # Press mouse button +mouse-up --xy 300,400 # Release mouse button +``` + +### App & Window (see commands-system.md) +``` +launch "System Settings" # Launch and wait for window +close-app "TextEdit" # Quit gracefully +close-app "TextEdit" --force # Force kill +list-windows --app "Finder" # List windows +list-apps # List running GUI apps +focus-window --app "Finder" # Bring window to front +resize-window --app "App" --width 800 --height 600 +move-window --app "App" --x 0 --y 0 +minimize --app "App" +maximize --app "App" +restore --app "App" +``` + +### Clipboard (see commands-system.md) +``` +clipboard-get # Read clipboard +clipboard-set "text" # Write to clipboard +clipboard-clear # Clear clipboard +``` + +### Wait (see commands-system.md) +``` +wait 1000 # Pause 1 second +wait --element @e5 --timeout 5000 # Wait for element +wait --window "Title" # Wait for window +wait --text "Done" --app "App" # Wait for text +wait --menu --app "App" # Wait for context menu +wait --menu-closed --app "App" # Wait for menu dismissal +``` + +### System (see commands-system.md) +``` +status # Health check +permissions # Check accessibility permission +permissions --request # Trigger system permission dialog +version # Version string +version --json # Machine-readable version +batch '[...]' # Run multiple commands +``` + +## Global Flag + +All commands accept `--verbose` / `-v` for debug logging to stderr. + +## Key Principles for Agents + +1. **Always snapshot first.** Never assume UI state. Snapshot, parse, then act. +2. **Use `-i` flag.** `snapshot --app "App" -i` filters to interactive elements only, reducing token count. +3. **Refs are ephemeral.** After any action that changes UI, snapshot again. +4. **Prefer AX actions over coordinates.** `click @e5` is more reliable than `mouse-click --xy 500,300`. +5. **Use `wait` for async UI.** After launching apps or triggering dialogs, wait for the expected state. +6. **Check permissions first.** Run `permissions` on first use. macOS requires Accessibility permission. +7. **Handle errors gracefully.** Parse the JSON `error.code` field and follow `error.suggestion`. +8. **Use `find` for targeted searches.** When you know the role and name, `find` is faster than parsing a full snapshot. +9. **Use surfaces for menus.** `snapshot --surface menu` captures open menus and context menus. +10. **Batch for performance.** Use `batch` to run multiple commands in a single invocation when order matters. diff --git a/.claude/skills/agent-desktop/commands-interaction.md b/.claude/skills/agent-desktop/commands-interaction.md new file mode 100644 index 0000000..2088f7b --- /dev/null +++ b/.claude/skills/agent-desktop/commands-interaction.md @@ -0,0 +1,223 @@ +# Interaction Commands + +Commands for modifying UI state — clicking, typing, selecting, scrolling, and input synthesis. + +## Click Actions + +All click commands use a smart activation chain (AX-first) that tries accessibility actions before falling back to coordinate-based clicks. + +### click +```bash +agent-desktop click @e5 +``` +Primary activation. Tries AXPress > AXConfirm > AXOpen > AXPick > child activation > focus+activate > coordinate click. + +### double-click +```bash +agent-desktop double-click @e3 +``` +Tries AXOpen first, then two smart activations with 50ms gap, then CGEvent double-click. + +### triple-click +```bash +agent-desktop triple-click @e2 +``` +Three smart activations with 30ms gaps, then CGEvent triple-click. Useful for select-all in text fields. + +### right-click +```bash +agent-desktop right-click @e5 +``` +Opens context menu. Tries AXShowMenu > focus+AXShowMenu > parent/child AXShowMenu > coordinate right-click. Use `wait --menu` after to capture the menu, then `snapshot --surface menu` to read it. + +## Text Input + +### type +```bash +agent-desktop type @e2 "hello@example.com" +agent-desktop type @e2 "multi line\ntext" +``` +Focuses the element then types each character via keyboard synthesis. Handles special characters. + +### set-value +```bash +agent-desktop set-value @e2 "new value" +``` +Sets the value directly via the AX value attribute. Faster than `type` but may not trigger all UI callbacks. Use for text fields, text areas, and sliders. + +### clear +```bash +agent-desktop clear @e2 +``` +Clears the element's value to an empty string. Equivalent to `set-value @e2 ""`. + +### focus +```bash +agent-desktop focus @e2 +``` +Sets keyboard focus on the element without clicking it. + +## Selection & Toggle + +### select +```bash +agent-desktop select @e4 "Option B" +``` +Selects an option in a list, dropdown, or combobox by its display text. + +### toggle +```bash +agent-desktop toggle @e6 +``` +Toggles a checkbox or switch to the opposite state. + +### check +```bash +agent-desktop check @e6 +``` +Sets a checkbox or switch to the checked/on state. Idempotent — does nothing if already checked. + +### uncheck +```bash +agent-desktop uncheck @e6 +``` +Sets a checkbox or switch to the unchecked/off state. Idempotent. + +## Expand & Collapse + +### expand +```bash +agent-desktop expand @e7 +``` +Expands a disclosure triangle, tree item, or accordion. + +### collapse +```bash +agent-desktop collapse @e7 +``` +Collapses an expanded disclosure/tree item. + +## Scrolling + +### scroll +```bash +agent-desktop scroll @e1 --direction down --amount 3 +agent-desktop scroll @e1 --direction up --amount 5 +agent-desktop scroll @e1 --direction left --amount 2 +agent-desktop scroll @e1 --direction right --amount 2 +``` + +| Flag | Default | Description | +|------|---------|-------------| +| `--direction` | down | `up`, `down`, `left`, `right` | +| `--amount` | 3 | Number of scroll units | + +### scroll-to +```bash +agent-desktop scroll-to @e8 +``` +Scrolls the element into the visible area of its scroll container. + +## Keyboard + +### press +```bash +agent-desktop press return +agent-desktop press escape +agent-desktop press cmd+c +agent-desktop press cmd+shift+z +agent-desktop press shift+tab +agent-desktop press f5 +agent-desktop press cmd+a --app "TextEdit" +``` + +| Flag | Description | +|------|-------------| +| `--app` | Target application (focuses app before pressing) | + +**Key names:** `return`, `escape`, `tab`, `space`, `delete`, `up`, `down`, `left`, `right`, `f1`-`f12` +**Modifiers:** `cmd`, `ctrl`, `alt`, `shift` — combine with `+` + +### key-down +```bash +agent-desktop key-down shift +``` +Holds a key or modifier down. Must be paired with `key-up`. + +### key-up +```bash +agent-desktop key-up shift +``` +Releases a held key or modifier. + +## Mouse + +### hover +```bash +agent-desktop hover @e5 +agent-desktop hover --xy 500,300 +agent-desktop hover @e5 --duration 2000 +``` +Moves cursor to element center or absolute coordinates. Optional `--duration` holds position for N ms. + +### drag +```bash +agent-desktop drag --from @e1 --to @e5 +agent-desktop drag --from-xy 100,200 --to-xy 400,500 +agent-desktop drag --from @e1 --to-xy 400,500 --duration 500 +``` + +| Flag | Description | +|------|-------------| +| `--from` | Source element ref | +| `--from-xy` | Source coordinates as `x,y` | +| `--to` | Destination element ref | +| `--to-xy` | Destination coordinates as `x,y` | +| `--duration` | Drag duration in milliseconds | + +Can mix ref and coordinate sources (e.g., `--from @e1 --to-xy 400,500`). + +### mouse-move +```bash +agent-desktop mouse-move --xy 500,300 +``` +Moves cursor to absolute screen coordinates. + +### mouse-click +```bash +agent-desktop mouse-click --xy 500,300 +agent-desktop mouse-click --xy 500,300 --button right +agent-desktop mouse-click --xy 500,300 --count 2 +``` + +| Flag | Default | Description | +|------|---------|-------------| +| `--xy` | (required) | Coordinates as `x,y` | +| `--button` | left | `left`, `right`, `middle` | +| `--count` | 1 | Number of clicks | + +### mouse-down / mouse-up +```bash +agent-desktop mouse-down --xy 100,200 +agent-desktop mouse-up --xy 300,400 +``` +Low-level press/release for custom drag or hold interactions. + +| Flag | Default | Description | +|------|---------|-------------| +| `--xy` | (required) | Coordinates as `x,y` | +| `--button` | left | `left`, `right`, `middle` | + +## Choosing the Right Command + +| Goal | Preferred | Alternative | +|------|-----------|-------------| +| Click a button | `click @ref` | `mouse-click --xy` if AX fails | +| Fill a text field | `type @ref "text"` | `set-value @ref "text"` for direct set | +| Clear then type | `clear @ref` then `type @ref "new"` | `triple-click @ref` then `type @ref "new"` | +| Toggle a checkbox | `check @ref` / `uncheck @ref` | `toggle @ref` if you don't know current state | +| Open context menu | `right-click @ref` then `wait --menu` | `mouse-click --xy --button right` | +| Select dropdown option | `select @ref "Option"` | `click @ref` then `find` the option | +| Navigate a form | `press tab` between fields | `focus @ref` to jump directly | +| Copy text | `press cmd+c --app "App"` | `clipboard-set` to set directly | +| Scroll to find elements | `scroll @ref --direction down` | `scroll-to @ref` if you have the ref | diff --git a/.claude/skills/agent-desktop/commands-observation.md b/.claude/skills/agent-desktop/commands-observation.md new file mode 100644 index 0000000..abc0e88 --- /dev/null +++ b/.claude/skills/agent-desktop/commands-observation.md @@ -0,0 +1,182 @@ +# Observation Commands + +Commands for reading UI state without modifying it. + +## snapshot + +Capture the accessibility tree as structured JSON with `@ref` IDs. + +```bash +agent-desktop snapshot --app "System Settings" -i +agent-desktop snapshot --app "Finder" --max-depth 5 --include-bounds +agent-desktop snapshot --app "App" --surface menu +agent-desktop snapshot --app "App" --window-id "w-1234" +agent-desktop snapshot --app "App" -i --compact +``` + +| Flag | Default | Description | +|------|---------|-------------| +| `--app` | (required) | Application name | +| `--window-id` | | Specific window ID from `list-windows` | +| `-i` / `--interactive-only` | false | Only include interactive elements (buttons, fields, etc.) | +| `--max-depth` | 10 | Maximum tree traversal depth | +| `--include-bounds` | false | Include `{x, y, width, height}` for each element | +| `--compact` | false | Omit empty structural nodes | +| `--surface` | window | Target surface: `window`, `focused`, `menu`, `menubar`, `sheet`, `popover`, `alert` | + +**Output structure:** +```json +{ + "version": "1.0", + "ok": true, + "command": "snapshot", + "data": { + "app": "System Settings", + "window": { "id": "w-4521", "title": "General" }, + "ref_count": 14, + "tree": { + "role": "window", + "name": "General", + "children": [ + { + "ref": "@e1", + "role": "button", + "name": "About", + "states": ["focused"] + }, + { + "role": "group", + "name": "Appearance", + "children": [ + { + "ref": "@e2", + "role": "checkbox", + "name": "Dark Mode", + "value": "0", + "states": ["enabled"] + } + ] + } + ] + } + } +} +``` + +**Tips:** +- Always use `-i` to keep output compact for LLM context windows +- Use `--surface menu` to capture open context menus or dropdown menus +- Use `--surface sheet` for modal dialogs +- Use `--compact` with `-i` for maximum token efficiency +- Combine `--max-depth 5` to limit deep trees (e.g., Xcode) + +## find + +Search elements by role, name, value, or text content. + +```bash +agent-desktop find --app "Finder" --role button --name "OK" +agent-desktop find --app "TextEdit" --role textfield +agent-desktop find --app "Safari" --text "Sign In" --first +agent-desktop find --app "App" --role checkbox --count +agent-desktop find --app "App" --role button --nth 2 +``` + +| Flag | Description | +|------|-------------| +| `--app` | Application name | +| `--role` | Accessibility role: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell | +| `--name` | Accessible name or label | +| `--value` | Current value | +| `--text` | Fuzzy match across name, value, title, and description | +| `--first` | Return first match only | +| `--last` | Return last match only | +| `--nth N` | Return Nth match (0-indexed) | +| `--count` | Return match count only | + +**Output (matches):** +```json +{ + "data": { + "matches": [ + { "ref": "@e5", "role": "button", "name": "OK", "states": ["enabled"] } + ], + "count": 1 + } +} +``` + +## get + +Read a specific property from an element. + +```bash +agent-desktop get @e1 --property text +agent-desktop get @e2 --property value +agent-desktop get @e3 --property bounds +agent-desktop get @e4 --property role +agent-desktop get @e5 --property states +agent-desktop get @e1 --property title +``` + +| Property | Returns | +|----------|---------| +| `text` | Accessible name/label (default) | +| `value` | Current value (text content, slider position, etc.) | +| `title` | Window or element title | +| `bounds` | `{ x, y, width, height }` rectangle | +| `role` | Element role string | +| `states` | Array of active states | + +## is + +Check a boolean state on an element. + +```bash +agent-desktop is @e1 --property visible +agent-desktop is @e2 --property enabled +agent-desktop is @e3 --property checked +agent-desktop is @e4 --property focused +agent-desktop is @e5 --property expanded +``` + +| Property | Checks | +|----------|--------| +| `visible` | Element is on screen (default) | +| `enabled` | Element is interactable | +| `checked` | Checkbox/switch is checked | +| `focused` | Element has keyboard focus | +| `expanded` | Disclosure/tree item is expanded | + +**Output:** +```json +{ "data": { "ref": "@e3", "property": "checked", "result": true } } +``` + +## screenshot + +Capture a PNG screenshot of an application window. + +```bash +agent-desktop screenshot --app "Finder" +agent-desktop screenshot --app "Finder" output.png +agent-desktop screenshot --window-id "w-1234" capture.png +``` + +| Flag | Description | +|------|-------------| +| `--app` | Application name | +| `--window-id` | Specific window ID | +| (positional) | File path to save PNG (omit for base64 in JSON) | + +When no output path is given, the screenshot is returned as a base64-encoded string in the JSON `data` field. + +## list-surfaces + +List available accessibility surfaces for an application. + +```bash +agent-desktop list-surfaces --app "Finder" +``` + +Returns the available surfaces (window, menu, menubar, sheet, popover, alert) for snapshotting. Use this to discover what surfaces are currently available before targeting a specific one with `snapshot --surface`. diff --git a/.claude/skills/agent-desktop/commands-system.md b/.claude/skills/agent-desktop/commands-system.md new file mode 100644 index 0000000..9513b25 --- /dev/null +++ b/.claude/skills/agent-desktop/commands-system.md @@ -0,0 +1,184 @@ +# System Commands + +App lifecycle, window management, clipboard, wait, and system health commands. + +## App Lifecycle + +### launch +```bash +agent-desktop launch "System Settings" +agent-desktop launch "com.apple.Safari" --timeout 10000 +``` +Launches an application by name or bundle ID and waits until its window is visible. + +| Flag | Default | Description | +|------|---------|-------------| +| `--timeout` | 30000 | Max wait time in ms for window to appear | + +### close-app +```bash +agent-desktop close-app "TextEdit" +agent-desktop close-app "TextEdit" --force +``` +Quits an application gracefully. Use `--force` to kill the process. + +### list-apps +```bash +agent-desktop list-apps +``` +Lists all running GUI applications. Returns array of `{ name, pid, bundle_id }`. + +## Window Management + +### list-windows +```bash +agent-desktop list-windows +agent-desktop list-windows --app "Finder" +``` +Lists all visible windows, optionally filtered by app. Returns array of `{ id, title, app_name, pid, bounds }`. + +### focus-window +```bash +agent-desktop focus-window --app "Finder" +agent-desktop focus-window --title "Documents" +agent-desktop focus-window --window-id "w-4521" +``` +Brings a window to the front. At least one identifier required. + +### resize-window +```bash +agent-desktop resize-window --app "TextEdit" --width 800 --height 600 +``` + +### move-window +```bash +agent-desktop move-window --app "TextEdit" --x 0 --y 0 +``` + +### minimize +```bash +agent-desktop minimize --app "TextEdit" +``` + +### maximize +```bash +agent-desktop maximize --app "TextEdit" +``` +Zooms the window to fill the screen. + +### restore +```bash +agent-desktop restore --app "TextEdit" +``` +Restores a minimized or maximized window to its previous size. + +## Clipboard + +### clipboard-get +```bash +agent-desktop clipboard-get +``` +Returns `{ "data": { "text": "clipboard contents" } }`. + +### clipboard-set +```bash +agent-desktop clipboard-set "Hello, world!" +``` + +### clipboard-clear +```bash +agent-desktop clipboard-clear +``` + +## Wait + +### wait (time) +```bash +agent-desktop wait 1000 +``` +Pauses for N milliseconds. Use between actions that need time to settle. + +### wait (element) +```bash +agent-desktop wait --element @e5 --timeout 5000 --app "App" +``` +Blocks until the element ref appears in the accessibility tree. Useful after triggering UI changes. + +### wait (window) +```bash +agent-desktop wait --window "Save As" --timeout 10000 +``` +Blocks until a window with the given title appears. + +### wait (text) +```bash +agent-desktop wait --text "Loading complete" --app "Safari" --timeout 5000 +``` +Blocks until the specified text appears anywhere in the app's accessibility tree. + +### wait (menu) +```bash +agent-desktop wait --menu --app "Finder" --timeout 3000 +``` +Blocks until a context menu is detected as open. + +### wait (menu-closed) +```bash +agent-desktop wait --menu-closed --app "Finder" --timeout 3000 +``` +Blocks until the context menu is dismissed. + +| Flag | Default | Description | +|------|---------|-------------| +| (positional) | | Milliseconds to pause | +| `--element` | | Ref to wait for | +| `--window` | | Window title to wait for | +| `--text` | | Text to wait for | +| `--menu` | false | Wait for context menu to open | +| `--menu-closed` | false | Wait for context menu to close | +| `--timeout` | 30000 | Timeout in ms (for element/window/text/menu waits) | +| `--app` | | Scope the wait to a specific application | + +## Batch + +### batch +```bash +agent-desktop batch '[{"command":"click","args":{"ref_id":"@e1"}},{"command":"wait","args":{"ms":500}},{"command":"click","args":{"ref_id":"@e2"}}]' +agent-desktop batch '[...]' --stop-on-error +``` +Execute multiple commands in sequence from a JSON array. Each entry has `command` (string) and `args` (object). + +| Flag | Default | Description | +|------|---------|-------------| +| `--stop-on-error` | false | Halt on first failed command | + +**Batch format:** +```json +[ + { "command": "click", "args": { "ref_id": "@e1" } }, + { "command": "wait", "args": { "ms": 500 } }, + { "command": "type", "args": { "ref_id": "@e2", "text": "hello" } } +] +``` + +## System Health + +### status +```bash +agent-desktop status +``` +Returns adapter health, platform info, and permission state. + +### permissions +```bash +agent-desktop permissions +agent-desktop permissions --request +``` +Checks accessibility permission status. Use `--request` to trigger the macOS system dialog. + +### version +```bash +agent-desktop version +agent-desktop version --json +``` +Returns version string. Use `--json` for `{ "version": "0.1.3", "platform": "macos", "arch": "aarch64" }`. diff --git a/.claude/skills/agent-desktop/workflows.md b/.claude/skills/agent-desktop/workflows.md new file mode 100644 index 0000000..ad8b3f4 --- /dev/null +++ b/.claude/skills/agent-desktop/workflows.md @@ -0,0 +1,219 @@ +# Common Automation Workflows + +Patterns for using agent-desktop effectively in multi-step desktop automation tasks. + +## First-Time Setup + +Before any automation, verify permissions: + +```bash +agent-desktop permissions +# If PERM_DENIED: +agent-desktop permissions --request +# Then: System Settings > Privacy & Security > Accessibility > enable your terminal +``` + +## Pattern: Fill a Form + +```bash +# 1. Snapshot the form +agent-desktop snapshot --app "System Settings" -i + +# 2. Parse output, identify text fields by name +# Found: @e3 = "Computer Name" textfield, @e5 = "Local Hostname" textfield + +# 3. Clear and fill each field +agent-desktop clear @e3 +agent-desktop type @e3 "My MacBook Pro" +agent-desktop clear @e5 +agent-desktop type @e5 "my-macbook-pro" + +# 4. Click the save/apply button +agent-desktop click @e8 + +# 5. Verify success +agent-desktop snapshot --app "System Settings" -i +``` + +## Pattern: Navigate Menus + +```bash +# 1. Click the menu item +agent-desktop snapshot --app "TextEdit" --surface menubar -i +# Found: @e1 = "File" menuitem + +agent-desktop click @e1 +agent-desktop wait --menu --app "TextEdit" +agent-desktop snapshot --app "TextEdit" --surface menu -i +# Found: @e5 = "Save As..." menuitem + +agent-desktop click @e5 + +# 2. Wait for the dialog +agent-desktop wait --window "Save" +agent-desktop snapshot --app "TextEdit" -i +``` + +## Pattern: Right-Click Context Menu + +```bash +# 1. Right-click the target element +agent-desktop right-click @e3 + +# 2. Wait for context menu to appear +agent-desktop wait --menu --app "Finder" --timeout 3000 + +# 3. Snapshot the menu surface +agent-desktop snapshot --app "Finder" --surface menu -i + +# 4. Click the desired menu item +agent-desktop click @e7 + +# 5. Wait for menu to close +agent-desktop wait --menu-closed --app "Finder" --timeout 2000 +``` + +## Pattern: Handle a Dialog + +```bash +# After triggering a dialog (save, alert, confirmation): +agent-desktop wait --window "Save As" --timeout 5000 +agent-desktop snapshot --app "TextEdit" -i + +# Fill dialog fields +agent-desktop type @e2 "my-document.txt" + +# Click OK/Save +agent-desktop click @e5 +``` + +## Pattern: Scroll and Find + +When the target element isn't visible and you need to scroll to find it: + +```bash +# 1. Find the scroll area +agent-desktop snapshot --app "App" -i +# Found: @e1 = scroll area + +# 2. Scroll and search in a loop +agent-desktop scroll @e1 --direction down --amount 5 +agent-desktop find --app "App" --name "Target Item" +# If no matches, scroll again +agent-desktop scroll @e1 --direction down --amount 5 +agent-desktop find --app "App" --name "Target Item" +# Found: @e14 = "Target Item" +agent-desktop click @e14 +``` + +## Pattern: Tab Through Fields + +```bash +# For sequential form filling without needing refs for each field: +agent-desktop click @e1 # Focus first field +agent-desktop type @e1 "value1" +agent-desktop press tab +# Now in next field — type directly since focus moved +agent-desktop press tab # Skip a field +agent-desktop type @e3 "value3" # Or snapshot again to get new refs +``` + +## Pattern: Copy Text from Element + +```bash +# Option A: Read directly via accessibility +agent-desktop get @e5 --property value + +# Option B: Copy via keyboard +agent-desktop click @e5 +agent-desktop press cmd+a +agent-desktop press cmd+c +agent-desktop clipboard-get +``` + +## Pattern: Drag and Drop + +```bash +# Between elements (by ref) +agent-desktop drag --from @e3 --to @e8 + +# Between coordinates +agent-desktop drag --from-xy 100,200 --to-xy 500,400 + +# Mixed: element to coordinates +agent-desktop drag --from @e3 --to-xy 500,400 --duration 500 +``` + +## Pattern: Wait for Async UI + +```bash +# After triggering a long operation: +agent-desktop click @e5 # "Download" button + +# Wait for completion text +agent-desktop wait --text "Download complete" --app "App" --timeout 30000 + +# Or wait for a specific element to appear +agent-desktop wait --element @e10 --timeout 10000 +``` + +## Pattern: Launch, Automate, Close + +```bash +# Full lifecycle +agent-desktop launch "Calculator" +agent-desktop snapshot --app "Calculator" -i + +# ... perform automation ... + +agent-desktop close-app "Calculator" +``` + +## Pattern: Multi-Window Workflow + +```bash +# List windows to find the right one +agent-desktop list-windows --app "Finder" +# Returns: [{id: "w-1234", title: "Documents"}, {id: "w-5678", title: "Downloads"}] + +# Focus a specific window +agent-desktop focus-window --window-id "w-5678" + +# Snapshot that specific window +agent-desktop snapshot --app "Finder" --window-id "w-5678" -i +``` + +## Pattern: Check Before Act (Idempotent) + +```bash +# Check if already in desired state +agent-desktop is @e6 --property checked +# If result is false, then check it +agent-desktop check @e6 + +# Or use check/uncheck directly (they're idempotent) +agent-desktop check @e6 # No-op if already checked +agent-desktop uncheck @e6 # No-op if already unchecked +``` + +## Pattern: Batch Operations + +```bash +# Run multiple commands atomically +agent-desktop batch '[ + {"command":"click","args":{"ref_id":"@e1"}}, + {"command":"wait","args":{"ms":200}}, + {"command":"type","args":{"ref_id":"@e2","text":"hello"}}, + {"command":"press","args":{"combo":"return"}} +]' --stop-on-error +``` + +## Anti-Patterns to Avoid + +1. **Acting without observing.** Never click a ref without a recent snapshot. +2. **Hardcoding refs.** Refs change between snapshots. Always use fresh refs. +3. **Ignoring wait.** After launch, dialog triggers, or menu clicks — always wait before snapshotting. +4. **Using coordinates when refs exist.** AX-based actions are more reliable than coordinate clicks. +5. **Not checking permissions.** Always verify accessibility permission before starting automation. +6. **Deep snapshots of large apps.** Use `--max-depth 5` and `-i` for Xcode, VS Code, etc. +7. **Assuming UI stability.** Re-snapshot after every action that could change the UI. diff --git a/.gitignore b/.gitignore index ef7334e..2254495 100644 --- a/.gitignore +++ b/.gitignore @@ -20,8 +20,9 @@ Cargo.lock *~ .vim/ -# Claude Code -.claude/ +# Claude Code (keep skills and commands tracked, ignore settings) +.claude/settings.json +.claude/settings.local.json .agents/ # Environment diff --git a/docs/prd-addendum-skill-maintenance.md b/docs/prd-addendum-skill-maintenance.md new file mode 100644 index 0000000..c62551e --- /dev/null +++ b/docs/prd-addendum-skill-maintenance.md @@ -0,0 +1,69 @@ +# PRD Addendum: Skill Maintenance Across Phases + +**Date:** 2026-02-23 +**Applies to:** agent-desktop PRD v2.0, all phases + +## Overview + +agent-desktop ships Claude Code skills (`.claude/skills/` and `.claude/commands/`) that teach AI coding agents how to use agent-desktop effectively. These skills must be kept in sync with the tool's capabilities across all phases. + +## Skill Graph Structure + +``` +.claude/ +├── commands/ +│ └── desktop.md # Quick reference slash command +└── skills/ + ├── agent-desktop/ # Core skill (platform-agnostic) + │ ├── SKILL.md # Index: concepts, JSON contract, command overview + │ ├── commands-observation.md # snapshot, find, get, is, screenshot + │ ├── commands-interaction.md # click, type, select, toggle, scroll, drag + │ ├── commands-system.md # launch, close, windows, clipboard, wait, batch + │ └── workflows.md # Common automation patterns + └── agent-desktop-macos/ # macOS platform skill + └── SKILL.md # Permissions, AX API, troubleshooting +``` + +## Phase-Specific Skill Updates + +### Phase 1 (Foundation + macOS MVP) — CURRENT +- All skill files created covering 50 commands +- macOS platform skill with AX API details, permissions, troubleshooting +- Core workflows: form filling, menu navigation, context menus, scrolling, dialogs + +### Phase 2 (Cross-Platform Expansion) +When Windows and Linux adapters ship: +- [ ] Create `.claude/skills/agent-desktop-windows/SKILL.md` + - UIA permission model + - Windows-specific behaviors (UAC, WinUI3 quirks) + - Troubleshooting guide +- [ ] Create `.claude/skills/agent-desktop-linux/SKILL.md` + - AT-SPI2/D-Bus setup + - Wayland vs X11 differences + - Troubleshooting guide +- [ ] Update core `SKILL.md` skill graph table to include new platform skills +- [ ] Update `workflows.md` with cross-platform patterns + +### Phase 3 (MCP Server Mode) +When `--mcp` flag ships: +- [ ] Create `.claude/skills/agent-desktop-mcp/SKILL.md` + - MCP tool surface documentation + - Transport configuration (stdio, SSE) + - Session management + - Tool-to-CLI mapping reference +- [ ] Update core `SKILL.md` with MCP mode section +- [ ] Add MCP workflow patterns to `workflows.md` + +### Phase 4 (Production Hardening) +When daemon mode and sessions ship: +- [ ] Update `commands-system.md` with session commands +- [ ] Add daemon lifecycle patterns to `workflows.md` +- [ ] Document enterprise quality gates in platform skills + +## Maintenance Rules + +1. **Every new command** must be added to the appropriate commands-*.md file +2. **Every new platform** gets its own skill directory under `.claude/skills/agent-desktop-{platform}/` +3. **Every new mode** (MCP, daemon) gets its own skill file +4. **Breaking changes** to JSON output or CLI flags must update all affected skill files +5. **Skill files are reviewed** as part of the PR checklist for any command-surface change