diff --git a/.claude/commands/desktop.md b/.claude/commands/desktop.md deleted file mode 100644 index 3dbdbb1..0000000 --- a/.claude/commands/desktop.md +++ /dev/null @@ -1,78 +0,0 @@ -# agent-desktop Quick Reference - -Use this command when you need to automate desktop applications with agent-desktop. - -## Core Loop - -```bash -# 1. OBSERVE -agent-desktop snapshot --app "$APP" -i - -# 2. Find target element by parsing JSON output (look for ref like @e5) - -# 3. ACT -agent-desktop click @e5 # or type, select, toggle, etc. - -# 4. VERIFY -agent-desktop snapshot --app "$APP" -i -``` - -## Common Tasks - -**Click a button:** -```bash -agent-desktop snapshot --app "App" -i -agent-desktop click @e5 -``` - -**Fill a text field:** -```bash -agent-desktop clear @e2 -agent-desktop type @e2 "new text" -``` - -**Toggle a checkbox:** -```bash -agent-desktop check @e3 # idempotent on -agent-desktop uncheck @e3 # idempotent off -``` - -**Open context menu:** -```bash -agent-desktop right-click @e5 -agent-desktop wait --menu --app "App" -agent-desktop snapshot --app "App" --surface menu -i -agent-desktop click @e7 -``` - -**Navigate menus:** -```bash -agent-desktop snapshot --app "App" --surface menubar -i -agent-desktop click @e1 # File menu -agent-desktop wait --menu --app "App" -agent-desktop snapshot --app "App" --surface menu -i -agent-desktop click @e5 # Menu item -``` - -**Wait for UI:** -```bash -agent-desktop wait --text "Done" --app "App" --timeout 5000 -agent-desktop wait --window "Save" --timeout 5000 -``` - -## First-Time Setup - -```bash -agent-desktop permissions --request -# Grant in System Settings > Privacy & Security > Accessibility -``` - -## Detailed Skills - -For comprehensive reference, see: -- `.claude/skills/agent-desktop/SKILL.md` — Core concepts and full command index -- `.claude/skills/agent-desktop/commands-observation.md` — snapshot, find, get, is, screenshot -- `.claude/skills/agent-desktop/commands-interaction.md` — click, type, select, toggle, scroll, drag -- `.claude/skills/agent-desktop/commands-system.md` — launch, close, windows, clipboard, wait, batch -- `.claude/skills/agent-desktop/workflows.md` — Common automation patterns -- `.claude/skills/agent-desktop-macos/SKILL.md` — macOS permissions, troubleshooting, AX details diff --git a/.claude/skills/agent-desktop-macos/SKILL.md b/.claude/skills/agent-desktop-macos/SKILL.md deleted file mode 100644 index a9db2bc..0000000 --- a/.claude/skills/agent-desktop-macos/SKILL.md +++ /dev/null @@ -1,209 +0,0 @@ ---- -name: agent-desktop-macos -description: > - macOS platform details for agent-desktop. Use when troubleshooting macOS accessibility - permissions (TCC), debugging AX API failures, understanding the smart activation chain, - working with macOS surfaces (menu, sheet, alert), or handling macOS-specific errors like - PERM_DENIED, ACTION_FAILED, or empty accessibility trees. ---- - -# agent-desktop: macOS Platform - -macOS-specific details for agent-desktop. Covers permissions, accessibility API behavior, troubleshooting, and platform quirks. - -## Prerequisites - -### Accessibility Permission (TCC) - -macOS requires explicit Accessibility permission for any process that reads or controls UI elements across applications. - -```bash -# Check permission status -agent-desktop permissions - -# Trigger the system dialog -agent-desktop permissions --request -``` - -**To grant manually:** -1. Open System Settings > Privacy & Security > Accessibility -2. Click the lock to make changes -3. Add your terminal application (Terminal.app, iTerm2, Warp, VS Code, etc.) -4. Toggle it ON - -**Important:** The permission is granted to the **terminal application**, not to agent-desktop itself. If you run agent-desktop from a different terminal, you need to grant that terminal too. - -After granting, restart the terminal for the permission to take effect. - -### Supported macOS Versions - -- macOS 12 (Monterey) and later -- Both Intel (x86_64) and Apple Silicon (ARM64) - -## macOS Accessibility API (AX) - -agent-desktop uses the macOS Accessibility API (`AXUIElement`) to read and manipulate UI elements. - -### How It Works - -1. `AXUIElementCreateApplication(pid)` creates a handle to an app's accessibility tree -2. Tree traversal reads `kAXChildrenAttribute` recursively -3. Attributes like `kAXRoleAttribute`, `kAXNameAttribute`, `kAXValueAttribute` provide element details -4. Actions like `kAXPressAction`, `kAXConfirmAction` trigger element behavior - -### Smart Activation Chain - -When you run `click @ref`, agent-desktop doesn't just do a simple click. It runs a multi-step activation chain: - -1. **AXScrollToVisible** — ensure element is on screen -2. **AXPress** — standard press action -3. **AXConfirm** — confirmation action -4. **AXOpen** — open action (for links, files) -5. **AXPick** — picker action -6. **AXShowAlternateUI** — reveal hidden UI, then press child -7. **Child activation** — try pressing child elements -8. **AXSelected** — set selected attribute -9. **Select via parent** — set parent's selected rows (for tables/lists) -10. **Custom actions** — AXPerformCustomAction -11. **Focus + activate** — set focus then press/confirm -12. **Keyboard activate** — focus + synthesize space key -13. **Parent activation** — try pressing ancestor elements -14. **Coordinate click** — final fallback: CGEvent click at element bounds center - -For `right-click`, the chain tries AXShowMenu first, then various focus/select combinations before falling back to a coordinate-based right-click. - -### Surfaces - -macOS apps can have multiple accessibility surfaces: - -| Surface | Description | When to use | -|---------|-------------|-------------| -| `window` | Main application window (default) | General UI interaction | -| `focused` | Currently focused element's context | Inspecting active element | -| `menu` | Open dropdown or context menu | After click/right-click on menu triggers | -| `menubar` | Application menu bar | Navigating File/Edit/View menus | -| `sheet` | Modal sheet (Save dialog, etc.) | After triggering sheet dialogs | -| `popover` | Popover/popup content | Inspecting tooltips, popovers | -| `alert` | System or app alert | Handling alert dialogs | - -```bash -# List available surfaces -agent-desktop list-surfaces --app "Finder" - -# Snapshot a specific surface -agent-desktop snapshot --app "Finder" --surface menu -i -``` - -## Troubleshooting - -### PERM_DENIED - -``` -"code": "PERM_DENIED", -"message": "Accessibility permission not granted" -``` - -**Fix:** -1. `agent-desktop permissions --request` to trigger dialog -2. System Settings > Privacy & Security > Accessibility -3. Add and enable your terminal -4. Restart terminal - -### Empty or Sparse Tree - -Some apps don't expose full accessibility trees: -- **Electron apps** (VS Code, Slack): Generally good accessibility support -- **Custom-rendered UIs** (games, some creative tools): May have no accessibility tree -- **Web views**: Content inside WebViews may be limited - -**Try:** -- Remove `-i` flag to see all elements including non-interactive ones -- Increase `--max-depth` to explore deeper -- Use `screenshot` as a visual fallback - -### STALE_REF - -``` -"code": "STALE_REF", -"message": "RefMap is from a previous snapshot" -``` - -The UI changed between your snapshot and action. Run `snapshot` again and use the new refs. - -### ACTION_FAILED - -``` -"code": "ACTION_FAILED" -``` - -The accessibility action was rejected. This can happen when: -- The element is disabled -- The app is busy or unresponsive -- The element doesn't support the requested action - -**Try:** -1. Check `is @ref --property enabled` first -2. Try coordinate-based click: get bounds with `get @ref --property bounds`, then `mouse-click --xy x,y` -3. Use keyboard: `focus @ref` then `press return` - -### APP_NOT_FOUND - -The application isn't running. Launch it first: -```bash -agent-desktop launch "App Name" -``` - -### Slow Snapshots - -Large apps (Xcode, Safari with many tabs) can have deep trees. - -**Optimize:** -- Use `-i` to filter to interactive elements only -- Use `--max-depth 5` to limit depth -- Use `--compact` to remove empty structural nodes -- Target a specific window with `--window-id` -- Use `find` instead of full snapshot when you know what you're looking for - -### Context Menu Doesn't Appear - -After `right-click @ref`, if no menu appears: -1. The element may not support context menus -2. The app may need to be focused first: `focus-window --app "App"` before right-clicking -3. Try `mouse-click --xy x,y --button right` with coordinates from `get @ref --property bounds` - -## macOS-Specific Behavior - -### App Identification - -Apps can be referenced by: -- **Display name:** `"System Settings"`, `"Finder"`, `"TextEdit"` -- **Bundle ID:** `"com.apple.systempreferences"`, `"com.apple.finder"` - -`launch` accepts both. Other commands use the display name with `--app`. - -### Window IDs - -Window IDs (like `w-4521`) are assigned by macOS and are stable for the lifetime of the window. Use `list-windows` to discover them. - -### Keyboard Shortcuts - -macOS uses `cmd` (Command) as the primary modifier: -```bash -agent-desktop press cmd+c # Copy -agent-desktop press cmd+v # Paste -agent-desktop press cmd+z # Undo -agent-desktop press cmd+s # Save -agent-desktop press cmd+w # Close window -agent-desktop press cmd+q # Quit app (use with caution!) -``` - -### Full-Screen Apps - -Apps in full-screen mode are accessible but may behave differently: -- `list-windows` still shows them -- Bounds may report full screen dimensions -- Some animations may delay UI state updates (use `wait`) - -### Menu Bar Apps - -Menu bar apps (status bar items) can be accessed via `--surface menubar`. The menu bar is a separate surface from the application window. diff --git a/.claude/skills/agent-desktop/SKILL.md b/.claude/skills/agent-desktop/SKILL.md deleted file mode 100644 index 38ba117..0000000 --- a/.claude/skills/agent-desktop/SKILL.md +++ /dev/null @@ -1,220 +0,0 @@ ---- -name: agent-desktop -description: > - Desktop automation via native OS accessibility trees using the agent-desktop CLI. - Use when an AI agent needs to observe, interact with, or automate desktop applications - (click buttons, fill forms, navigate menus, read UI state, toggle checkboxes, scroll, - drag, type text, take screenshots, manage windows, use clipboard). Covers 50 commands - across observation, interaction, keyboard/mouse, app lifecycle, clipboard, and wait. - Triggers on: "click button", "fill form", "open app", "read UI", "automate desktop", - "accessibility tree", "snapshot app", "type into field", "navigate menu", "toggle checkbox", - "take screenshot", "desktop automation", "agent-desktop", or any desktop GUI interaction task. - Supports macOS (Phase 1), with Windows and Linux planned. ---- - -# agent-desktop - -Cross-platform CLI tool enabling AI agents to observe and control desktop applications via native OS accessibility trees. - -**Core principle:** agent-desktop is NOT an AI agent. It is a tool that AI agents invoke. It outputs structured JSON with ref-based element identifiers. The observation-action loop lives in the calling agent. - -## When to Use - -Use agent-desktop when you need to: -- Read UI state from desktop applications (buttons, text fields, menus, checkboxes) -- Interact with desktop app elements (click, type, select, toggle, scroll) -- Automate multi-step desktop workflows (fill forms, navigate menus, transfer data) -- Wait for UI state changes before proceeding -- Take screenshots of application windows - -Do NOT use agent-desktop for: -- Web browser automation (use agent-browser instead) -- Custom-rendered or game-engine UIs lacking accessibility exposure -- Applications that don't expose accessibility trees - -## Skill Graph - -This skill is the index. Platform-specific details and advanced patterns are in sub-skills: - -| Skill | When to use | -|-------|-------------| -| `agent-desktop-macos` | macOS permissions, AX API quirks, troubleshooting | -| `commands-observation.md` | Snapshot, find, get, is, screenshot, list-surfaces | -| `commands-interaction.md` | Click, type, set-value, select, toggle, expand, scroll, drag | -| `commands-system.md` | Launch, close, windows, clipboard, wait, batch | -| `workflows.md` | Common automation patterns (forms, menus, dialogs, navigation) | - -## The Observe-Act Loop - -Every automation follows this pattern: - -``` -1. OBSERVE → agent-desktop snapshot --app "App Name" -i -2. REASON → Parse JSON, find target element by ref (@e1, @e2...) -3. ACT → agent-desktop click @e5 (or type, select, toggle...) -4. VERIFY → agent-desktop snapshot again to confirm state change -5. REPEAT → Continue until task is complete -``` - -Always snapshot before acting. Refs are snapshot-scoped and become stale after UI changes. - -## Ref System - -- Refs are assigned in depth-first document order: `@e1`, `@e2`, `@e3`... -- Only interactive elements receive refs: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell -- Static text, groups, and containers do NOT get refs (they remain in the tree for context) -- Refs are deterministic within a snapshot but NOT stable across snapshots if the UI changed -- After any action that changes UI, run `snapshot` again to get fresh refs - -## JSON Output Contract - -Every command returns a JSON envelope on stdout: - -### Success -```json -{ - "version": "1.0", - "ok": true, - "command": "snapshot", - "data": { ... } -} -``` - -### Error -```json -{ - "version": "1.0", - "ok": false, - "command": "click", - "error": { - "code": "STALE_REF", - "message": "RefMap is from a previous snapshot", - "suggestion": "Run 'snapshot' to refresh, then retry with updated ref" - } -} -``` - -### Exit Codes -- `0` — success (check `ok: true`) -- `1` — structured error (JSON with error code) -- `2` — argument/parse error - -### Error Codes -| Code | Meaning | Recovery | -|------|---------|----------| -| `PERM_DENIED` | Accessibility permission not granted | Grant in System Settings > Privacy > Accessibility | -| `ELEMENT_NOT_FOUND` | Ref not found in current refmap | Re-run snapshot, use fresh ref | -| `APP_NOT_FOUND` | Application not running | Launch it first with `launch` | -| `ACTION_FAILED` | Accessibility action rejected | Try alternative approach (different action or coordinate-based) | -| `ACTION_NOT_SUPPORTED` | Element doesn't support this action | Check available actions, use different command | -| `STALE_REF` | Ref from old snapshot | Re-run snapshot to get fresh refs | -| `WINDOW_NOT_FOUND` | No matching window | Check app name, use `list-windows` | -| `PLATFORM_NOT_SUPPORTED` | Feature not available on this OS | Check platform support | -| `TIMEOUT` | Wait condition not met in time | Increase `--timeout` or check condition | -| `INVALID_ARGS` | Bad arguments | Check command syntax | -| `INTERNAL` | Unexpected internal error | Report bug with `-v` verbose output | - -## Command Quick Reference - -### Observation (see commands-observation.md) -``` -snapshot --app "App" -i # Accessibility tree with refs -screenshot --app "App" out.png # PNG screenshot -find --app "App" --role button # Search elements -get @e1 --property text # Read element property -is @e1 --property enabled # Check element state -list-surfaces --app "App" # Available surfaces -``` - -### Interaction (see commands-interaction.md) -``` -click @e5 # Click element -double-click @e3 # Double-click -right-click @e5 # Right-click (context menu) -type @e2 "hello" # Type text into element -set-value @e2 "new value" # Set value directly -clear @e2 # Clear element value -focus @e2 # Set keyboard focus -select @e4 "Option B" # Select dropdown option -toggle @e6 # Toggle checkbox/switch -check @e6 # Idempotent check -uncheck @e6 # Idempotent uncheck -expand @e7 # Expand disclosure -collapse @e7 # Collapse disclosure -scroll @e1 --direction down # Scroll element -scroll-to @e8 # Scroll element into view -``` - -### Keyboard & Mouse (see commands-interaction.md) -``` -press cmd+c # Key combo -press return --app "App" # Key combo targeted at app -key-down shift # Hold key -key-up shift # Release key -hover @e5 # Move cursor to element -hover --xy 500,300 # Move cursor to coordinates -drag --from @e1 --to @e5 # Drag between elements -mouse-click --xy 500,300 # Click at coordinates -mouse-move --xy 100,200 # Move cursor -mouse-down --xy 100,200 # Press mouse button -mouse-up --xy 300,400 # Release mouse button -``` - -### App & Window (see commands-system.md) -``` -launch "System Settings" # Launch and wait for window -close-app "TextEdit" # Quit gracefully -close-app "TextEdit" --force # Force kill -list-windows --app "Finder" # List windows -list-apps # List running GUI apps -focus-window --app "Finder" # Bring window to front -resize-window --app "App" --width 800 --height 600 -move-window --app "App" --x 0 --y 0 -minimize --app "App" -maximize --app "App" -restore --app "App" -``` - -### Clipboard (see commands-system.md) -``` -clipboard-get # Read clipboard -clipboard-set "text" # Write to clipboard -clipboard-clear # Clear clipboard -``` - -### Wait (see commands-system.md) -``` -wait 1000 # Pause 1 second -wait --element @e5 --timeout 5000 # Wait for element -wait --window "Title" # Wait for window -wait --text "Done" --app "App" # Wait for text -wait --menu --app "App" # Wait for context menu -wait --menu-closed --app "App" # Wait for menu dismissal -``` - -### System (see commands-system.md) -``` -status # Health check -permissions # Check accessibility permission -permissions --request # Trigger system permission dialog -version # Version string -version --json # Machine-readable version -batch '[...]' # Run multiple commands -``` - -## Global Flag - -All commands accept `--verbose` / `-v` for debug logging to stderr. - -## Key Principles for Agents - -1. **Always snapshot first.** Never assume UI state. Snapshot, parse, then act. -2. **Use `-i` flag.** `snapshot --app "App" -i` filters to interactive elements only, reducing token count. -3. **Refs are ephemeral.** After any action that changes UI, snapshot again. -4. **Prefer AX actions over coordinates.** `click @e5` is more reliable than `mouse-click --xy 500,300`. -5. **Use `wait` for async UI.** After launching apps or triggering dialogs, wait for the expected state. -6. **Check permissions first.** Run `permissions` on first use. macOS requires Accessibility permission. -7. **Handle errors gracefully.** Parse the JSON `error.code` field and follow `error.suggestion`. -8. **Use `find` for targeted searches.** When you know the role and name, `find` is faster than parsing a full snapshot. -9. **Use surfaces for menus.** `snapshot --surface menu` captures open menus and context menus. -10. **Batch for performance.** Use `batch` to run multiple commands in a single invocation when order matters. diff --git a/.claude/skills/agent-desktop/commands-interaction.md b/.claude/skills/agent-desktop/commands-interaction.md deleted file mode 100644 index 2088f7b..0000000 --- a/.claude/skills/agent-desktop/commands-interaction.md +++ /dev/null @@ -1,223 +0,0 @@ -# Interaction Commands - -Commands for modifying UI state — clicking, typing, selecting, scrolling, and input synthesis. - -## Click Actions - -All click commands use a smart activation chain (AX-first) that tries accessibility actions before falling back to coordinate-based clicks. - -### click -```bash -agent-desktop click @e5 -``` -Primary activation. Tries AXPress > AXConfirm > AXOpen > AXPick > child activation > focus+activate > coordinate click. - -### double-click -```bash -agent-desktop double-click @e3 -``` -Tries AXOpen first, then two smart activations with 50ms gap, then CGEvent double-click. - -### triple-click -```bash -agent-desktop triple-click @e2 -``` -Three smart activations with 30ms gaps, then CGEvent triple-click. Useful for select-all in text fields. - -### right-click -```bash -agent-desktop right-click @e5 -``` -Opens context menu. Tries AXShowMenu > focus+AXShowMenu > parent/child AXShowMenu > coordinate right-click. Use `wait --menu` after to capture the menu, then `snapshot --surface menu` to read it. - -## Text Input - -### type -```bash -agent-desktop type @e2 "hello@example.com" -agent-desktop type @e2 "multi line\ntext" -``` -Focuses the element then types each character via keyboard synthesis. Handles special characters. - -### set-value -```bash -agent-desktop set-value @e2 "new value" -``` -Sets the value directly via the AX value attribute. Faster than `type` but may not trigger all UI callbacks. Use for text fields, text areas, and sliders. - -### clear -```bash -agent-desktop clear @e2 -``` -Clears the element's value to an empty string. Equivalent to `set-value @e2 ""`. - -### focus -```bash -agent-desktop focus @e2 -``` -Sets keyboard focus on the element without clicking it. - -## Selection & Toggle - -### select -```bash -agent-desktop select @e4 "Option B" -``` -Selects an option in a list, dropdown, or combobox by its display text. - -### toggle -```bash -agent-desktop toggle @e6 -``` -Toggles a checkbox or switch to the opposite state. - -### check -```bash -agent-desktop check @e6 -``` -Sets a checkbox or switch to the checked/on state. Idempotent — does nothing if already checked. - -### uncheck -```bash -agent-desktop uncheck @e6 -``` -Sets a checkbox or switch to the unchecked/off state. Idempotent. - -## Expand & Collapse - -### expand -```bash -agent-desktop expand @e7 -``` -Expands a disclosure triangle, tree item, or accordion. - -### collapse -```bash -agent-desktop collapse @e7 -``` -Collapses an expanded disclosure/tree item. - -## Scrolling - -### scroll -```bash -agent-desktop scroll @e1 --direction down --amount 3 -agent-desktop scroll @e1 --direction up --amount 5 -agent-desktop scroll @e1 --direction left --amount 2 -agent-desktop scroll @e1 --direction right --amount 2 -``` - -| Flag | Default | Description | -|------|---------|-------------| -| `--direction` | down | `up`, `down`, `left`, `right` | -| `--amount` | 3 | Number of scroll units | - -### scroll-to -```bash -agent-desktop scroll-to @e8 -``` -Scrolls the element into the visible area of its scroll container. - -## Keyboard - -### press -```bash -agent-desktop press return -agent-desktop press escape -agent-desktop press cmd+c -agent-desktop press cmd+shift+z -agent-desktop press shift+tab -agent-desktop press f5 -agent-desktop press cmd+a --app "TextEdit" -``` - -| Flag | Description | -|------|-------------| -| `--app` | Target application (focuses app before pressing) | - -**Key names:** `return`, `escape`, `tab`, `space`, `delete`, `up`, `down`, `left`, `right`, `f1`-`f12` -**Modifiers:** `cmd`, `ctrl`, `alt`, `shift` — combine with `+` - -### key-down -```bash -agent-desktop key-down shift -``` -Holds a key or modifier down. Must be paired with `key-up`. - -### key-up -```bash -agent-desktop key-up shift -``` -Releases a held key or modifier. - -## Mouse - -### hover -```bash -agent-desktop hover @e5 -agent-desktop hover --xy 500,300 -agent-desktop hover @e5 --duration 2000 -``` -Moves cursor to element center or absolute coordinates. Optional `--duration` holds position for N ms. - -### drag -```bash -agent-desktop drag --from @e1 --to @e5 -agent-desktop drag --from-xy 100,200 --to-xy 400,500 -agent-desktop drag --from @e1 --to-xy 400,500 --duration 500 -``` - -| Flag | Description | -|------|-------------| -| `--from` | Source element ref | -| `--from-xy` | Source coordinates as `x,y` | -| `--to` | Destination element ref | -| `--to-xy` | Destination coordinates as `x,y` | -| `--duration` | Drag duration in milliseconds | - -Can mix ref and coordinate sources (e.g., `--from @e1 --to-xy 400,500`). - -### mouse-move -```bash -agent-desktop mouse-move --xy 500,300 -``` -Moves cursor to absolute screen coordinates. - -### mouse-click -```bash -agent-desktop mouse-click --xy 500,300 -agent-desktop mouse-click --xy 500,300 --button right -agent-desktop mouse-click --xy 500,300 --count 2 -``` - -| Flag | Default | Description | -|------|---------|-------------| -| `--xy` | (required) | Coordinates as `x,y` | -| `--button` | left | `left`, `right`, `middle` | -| `--count` | 1 | Number of clicks | - -### mouse-down / mouse-up -```bash -agent-desktop mouse-down --xy 100,200 -agent-desktop mouse-up --xy 300,400 -``` -Low-level press/release for custom drag or hold interactions. - -| Flag | Default | Description | -|------|---------|-------------| -| `--xy` | (required) | Coordinates as `x,y` | -| `--button` | left | `left`, `right`, `middle` | - -## Choosing the Right Command - -| Goal | Preferred | Alternative | -|------|-----------|-------------| -| Click a button | `click @ref` | `mouse-click --xy` if AX fails | -| Fill a text field | `type @ref "text"` | `set-value @ref "text"` for direct set | -| Clear then type | `clear @ref` then `type @ref "new"` | `triple-click @ref` then `type @ref "new"` | -| Toggle a checkbox | `check @ref` / `uncheck @ref` | `toggle @ref` if you don't know current state | -| Open context menu | `right-click @ref` then `wait --menu` | `mouse-click --xy --button right` | -| Select dropdown option | `select @ref "Option"` | `click @ref` then `find` the option | -| Navigate a form | `press tab` between fields | `focus @ref` to jump directly | -| Copy text | `press cmd+c --app "App"` | `clipboard-set` to set directly | -| Scroll to find elements | `scroll @ref --direction down` | `scroll-to @ref` if you have the ref | diff --git a/.claude/skills/agent-desktop/commands-observation.md b/.claude/skills/agent-desktop/commands-observation.md deleted file mode 100644 index abc0e88..0000000 --- a/.claude/skills/agent-desktop/commands-observation.md +++ /dev/null @@ -1,182 +0,0 @@ -# Observation Commands - -Commands for reading UI state without modifying it. - -## snapshot - -Capture the accessibility tree as structured JSON with `@ref` IDs. - -```bash -agent-desktop snapshot --app "System Settings" -i -agent-desktop snapshot --app "Finder" --max-depth 5 --include-bounds -agent-desktop snapshot --app "App" --surface menu -agent-desktop snapshot --app "App" --window-id "w-1234" -agent-desktop snapshot --app "App" -i --compact -``` - -| Flag | Default | Description | -|------|---------|-------------| -| `--app` | (required) | Application name | -| `--window-id` | | Specific window ID from `list-windows` | -| `-i` / `--interactive-only` | false | Only include interactive elements (buttons, fields, etc.) | -| `--max-depth` | 10 | Maximum tree traversal depth | -| `--include-bounds` | false | Include `{x, y, width, height}` for each element | -| `--compact` | false | Omit empty structural nodes | -| `--surface` | window | Target surface: `window`, `focused`, `menu`, `menubar`, `sheet`, `popover`, `alert` | - -**Output structure:** -```json -{ - "version": "1.0", - "ok": true, - "command": "snapshot", - "data": { - "app": "System Settings", - "window": { "id": "w-4521", "title": "General" }, - "ref_count": 14, - "tree": { - "role": "window", - "name": "General", - "children": [ - { - "ref": "@e1", - "role": "button", - "name": "About", - "states": ["focused"] - }, - { - "role": "group", - "name": "Appearance", - "children": [ - { - "ref": "@e2", - "role": "checkbox", - "name": "Dark Mode", - "value": "0", - "states": ["enabled"] - } - ] - } - ] - } - } -} -``` - -**Tips:** -- Always use `-i` to keep output compact for LLM context windows -- Use `--surface menu` to capture open context menus or dropdown menus -- Use `--surface sheet` for modal dialogs -- Use `--compact` with `-i` for maximum token efficiency -- Combine `--max-depth 5` to limit deep trees (e.g., Xcode) - -## find - -Search elements by role, name, value, or text content. - -```bash -agent-desktop find --app "Finder" --role button --name "OK" -agent-desktop find --app "TextEdit" --role textfield -agent-desktop find --app "Safari" --text "Sign In" --first -agent-desktop find --app "App" --role checkbox --count -agent-desktop find --app "App" --role button --nth 2 -``` - -| Flag | Description | -|------|-------------| -| `--app` | Application name | -| `--role` | Accessibility role: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell | -| `--name` | Accessible name or label | -| `--value` | Current value | -| `--text` | Fuzzy match across name, value, title, and description | -| `--first` | Return first match only | -| `--last` | Return last match only | -| `--nth N` | Return Nth match (0-indexed) | -| `--count` | Return match count only | - -**Output (matches):** -```json -{ - "data": { - "matches": [ - { "ref": "@e5", "role": "button", "name": "OK", "states": ["enabled"] } - ], - "count": 1 - } -} -``` - -## get - -Read a specific property from an element. - -```bash -agent-desktop get @e1 --property text -agent-desktop get @e2 --property value -agent-desktop get @e3 --property bounds -agent-desktop get @e4 --property role -agent-desktop get @e5 --property states -agent-desktop get @e1 --property title -``` - -| Property | Returns | -|----------|---------| -| `text` | Accessible name/label (default) | -| `value` | Current value (text content, slider position, etc.) | -| `title` | Window or element title | -| `bounds` | `{ x, y, width, height }` rectangle | -| `role` | Element role string | -| `states` | Array of active states | - -## is - -Check a boolean state on an element. - -```bash -agent-desktop is @e1 --property visible -agent-desktop is @e2 --property enabled -agent-desktop is @e3 --property checked -agent-desktop is @e4 --property focused -agent-desktop is @e5 --property expanded -``` - -| Property | Checks | -|----------|--------| -| `visible` | Element is on screen (default) | -| `enabled` | Element is interactable | -| `checked` | Checkbox/switch is checked | -| `focused` | Element has keyboard focus | -| `expanded` | Disclosure/tree item is expanded | - -**Output:** -```json -{ "data": { "ref": "@e3", "property": "checked", "result": true } } -``` - -## screenshot - -Capture a PNG screenshot of an application window. - -```bash -agent-desktop screenshot --app "Finder" -agent-desktop screenshot --app "Finder" output.png -agent-desktop screenshot --window-id "w-1234" capture.png -``` - -| Flag | Description | -|------|-------------| -| `--app` | Application name | -| `--window-id` | Specific window ID | -| (positional) | File path to save PNG (omit for base64 in JSON) | - -When no output path is given, the screenshot is returned as a base64-encoded string in the JSON `data` field. - -## list-surfaces - -List available accessibility surfaces for an application. - -```bash -agent-desktop list-surfaces --app "Finder" -``` - -Returns the available surfaces (window, menu, menubar, sheet, popover, alert) for snapshotting. Use this to discover what surfaces are currently available before targeting a specific one with `snapshot --surface`. diff --git a/.claude/skills/agent-desktop/commands-system.md b/.claude/skills/agent-desktop/commands-system.md deleted file mode 100644 index 9513b25..0000000 --- a/.claude/skills/agent-desktop/commands-system.md +++ /dev/null @@ -1,184 +0,0 @@ -# System Commands - -App lifecycle, window management, clipboard, wait, and system health commands. - -## App Lifecycle - -### launch -```bash -agent-desktop launch "System Settings" -agent-desktop launch "com.apple.Safari" --timeout 10000 -``` -Launches an application by name or bundle ID and waits until its window is visible. - -| Flag | Default | Description | -|------|---------|-------------| -| `--timeout` | 30000 | Max wait time in ms for window to appear | - -### close-app -```bash -agent-desktop close-app "TextEdit" -agent-desktop close-app "TextEdit" --force -``` -Quits an application gracefully. Use `--force` to kill the process. - -### list-apps -```bash -agent-desktop list-apps -``` -Lists all running GUI applications. Returns array of `{ name, pid, bundle_id }`. - -## Window Management - -### list-windows -```bash -agent-desktop list-windows -agent-desktop list-windows --app "Finder" -``` -Lists all visible windows, optionally filtered by app. Returns array of `{ id, title, app_name, pid, bounds }`. - -### focus-window -```bash -agent-desktop focus-window --app "Finder" -agent-desktop focus-window --title "Documents" -agent-desktop focus-window --window-id "w-4521" -``` -Brings a window to the front. At least one identifier required. - -### resize-window -```bash -agent-desktop resize-window --app "TextEdit" --width 800 --height 600 -``` - -### move-window -```bash -agent-desktop move-window --app "TextEdit" --x 0 --y 0 -``` - -### minimize -```bash -agent-desktop minimize --app "TextEdit" -``` - -### maximize -```bash -agent-desktop maximize --app "TextEdit" -``` -Zooms the window to fill the screen. - -### restore -```bash -agent-desktop restore --app "TextEdit" -``` -Restores a minimized or maximized window to its previous size. - -## Clipboard - -### clipboard-get -```bash -agent-desktop clipboard-get -``` -Returns `{ "data": { "text": "clipboard contents" } }`. - -### clipboard-set -```bash -agent-desktop clipboard-set "Hello, world!" -``` - -### clipboard-clear -```bash -agent-desktop clipboard-clear -``` - -## Wait - -### wait (time) -```bash -agent-desktop wait 1000 -``` -Pauses for N milliseconds. Use between actions that need time to settle. - -### wait (element) -```bash -agent-desktop wait --element @e5 --timeout 5000 --app "App" -``` -Blocks until the element ref appears in the accessibility tree. Useful after triggering UI changes. - -### wait (window) -```bash -agent-desktop wait --window "Save As" --timeout 10000 -``` -Blocks until a window with the given title appears. - -### wait (text) -```bash -agent-desktop wait --text "Loading complete" --app "Safari" --timeout 5000 -``` -Blocks until the specified text appears anywhere in the app's accessibility tree. - -### wait (menu) -```bash -agent-desktop wait --menu --app "Finder" --timeout 3000 -``` -Blocks until a context menu is detected as open. - -### wait (menu-closed) -```bash -agent-desktop wait --menu-closed --app "Finder" --timeout 3000 -``` -Blocks until the context menu is dismissed. - -| Flag | Default | Description | -|------|---------|-------------| -| (positional) | | Milliseconds to pause | -| `--element` | | Ref to wait for | -| `--window` | | Window title to wait for | -| `--text` | | Text to wait for | -| `--menu` | false | Wait for context menu to open | -| `--menu-closed` | false | Wait for context menu to close | -| `--timeout` | 30000 | Timeout in ms (for element/window/text/menu waits) | -| `--app` | | Scope the wait to a specific application | - -## Batch - -### batch -```bash -agent-desktop batch '[{"command":"click","args":{"ref_id":"@e1"}},{"command":"wait","args":{"ms":500}},{"command":"click","args":{"ref_id":"@e2"}}]' -agent-desktop batch '[...]' --stop-on-error -``` -Execute multiple commands in sequence from a JSON array. Each entry has `command` (string) and `args` (object). - -| Flag | Default | Description | -|------|---------|-------------| -| `--stop-on-error` | false | Halt on first failed command | - -**Batch format:** -```json -[ - { "command": "click", "args": { "ref_id": "@e1" } }, - { "command": "wait", "args": { "ms": 500 } }, - { "command": "type", "args": { "ref_id": "@e2", "text": "hello" } } -] -``` - -## System Health - -### status -```bash -agent-desktop status -``` -Returns adapter health, platform info, and permission state. - -### permissions -```bash -agent-desktop permissions -agent-desktop permissions --request -``` -Checks accessibility permission status. Use `--request` to trigger the macOS system dialog. - -### version -```bash -agent-desktop version -agent-desktop version --json -``` -Returns version string. Use `--json` for `{ "version": "0.1.3", "platform": "macos", "arch": "aarch64" }`. diff --git a/.claude/skills/agent-desktop/workflows.md b/.claude/skills/agent-desktop/workflows.md deleted file mode 100644 index ad8b3f4..0000000 --- a/.claude/skills/agent-desktop/workflows.md +++ /dev/null @@ -1,219 +0,0 @@ -# Common Automation Workflows - -Patterns for using agent-desktop effectively in multi-step desktop automation tasks. - -## First-Time Setup - -Before any automation, verify permissions: - -```bash -agent-desktop permissions -# If PERM_DENIED: -agent-desktop permissions --request -# Then: System Settings > Privacy & Security > Accessibility > enable your terminal -``` - -## Pattern: Fill a Form - -```bash -# 1. Snapshot the form -agent-desktop snapshot --app "System Settings" -i - -# 2. Parse output, identify text fields by name -# Found: @e3 = "Computer Name" textfield, @e5 = "Local Hostname" textfield - -# 3. Clear and fill each field -agent-desktop clear @e3 -agent-desktop type @e3 "My MacBook Pro" -agent-desktop clear @e5 -agent-desktop type @e5 "my-macbook-pro" - -# 4. Click the save/apply button -agent-desktop click @e8 - -# 5. Verify success -agent-desktop snapshot --app "System Settings" -i -``` - -## Pattern: Navigate Menus - -```bash -# 1. Click the menu item -agent-desktop snapshot --app "TextEdit" --surface menubar -i -# Found: @e1 = "File" menuitem - -agent-desktop click @e1 -agent-desktop wait --menu --app "TextEdit" -agent-desktop snapshot --app "TextEdit" --surface menu -i -# Found: @e5 = "Save As..." menuitem - -agent-desktop click @e5 - -# 2. Wait for the dialog -agent-desktop wait --window "Save" -agent-desktop snapshot --app "TextEdit" -i -``` - -## Pattern: Right-Click Context Menu - -```bash -# 1. Right-click the target element -agent-desktop right-click @e3 - -# 2. Wait for context menu to appear -agent-desktop wait --menu --app "Finder" --timeout 3000 - -# 3. Snapshot the menu surface -agent-desktop snapshot --app "Finder" --surface menu -i - -# 4. Click the desired menu item -agent-desktop click @e7 - -# 5. Wait for menu to close -agent-desktop wait --menu-closed --app "Finder" --timeout 2000 -``` - -## Pattern: Handle a Dialog - -```bash -# After triggering a dialog (save, alert, confirmation): -agent-desktop wait --window "Save As" --timeout 5000 -agent-desktop snapshot --app "TextEdit" -i - -# Fill dialog fields -agent-desktop type @e2 "my-document.txt" - -# Click OK/Save -agent-desktop click @e5 -``` - -## Pattern: Scroll and Find - -When the target element isn't visible and you need to scroll to find it: - -```bash -# 1. Find the scroll area -agent-desktop snapshot --app "App" -i -# Found: @e1 = scroll area - -# 2. Scroll and search in a loop -agent-desktop scroll @e1 --direction down --amount 5 -agent-desktop find --app "App" --name "Target Item" -# If no matches, scroll again -agent-desktop scroll @e1 --direction down --amount 5 -agent-desktop find --app "App" --name "Target Item" -# Found: @e14 = "Target Item" -agent-desktop click @e14 -``` - -## Pattern: Tab Through Fields - -```bash -# For sequential form filling without needing refs for each field: -agent-desktop click @e1 # Focus first field -agent-desktop type @e1 "value1" -agent-desktop press tab -# Now in next field — type directly since focus moved -agent-desktop press tab # Skip a field -agent-desktop type @e3 "value3" # Or snapshot again to get new refs -``` - -## Pattern: Copy Text from Element - -```bash -# Option A: Read directly via accessibility -agent-desktop get @e5 --property value - -# Option B: Copy via keyboard -agent-desktop click @e5 -agent-desktop press cmd+a -agent-desktop press cmd+c -agent-desktop clipboard-get -``` - -## Pattern: Drag and Drop - -```bash -# Between elements (by ref) -agent-desktop drag --from @e3 --to @e8 - -# Between coordinates -agent-desktop drag --from-xy 100,200 --to-xy 500,400 - -# Mixed: element to coordinates -agent-desktop drag --from @e3 --to-xy 500,400 --duration 500 -``` - -## Pattern: Wait for Async UI - -```bash -# After triggering a long operation: -agent-desktop click @e5 # "Download" button - -# Wait for completion text -agent-desktop wait --text "Download complete" --app "App" --timeout 30000 - -# Or wait for a specific element to appear -agent-desktop wait --element @e10 --timeout 10000 -``` - -## Pattern: Launch, Automate, Close - -```bash -# Full lifecycle -agent-desktop launch "Calculator" -agent-desktop snapshot --app "Calculator" -i - -# ... perform automation ... - -agent-desktop close-app "Calculator" -``` - -## Pattern: Multi-Window Workflow - -```bash -# List windows to find the right one -agent-desktop list-windows --app "Finder" -# Returns: [{id: "w-1234", title: "Documents"}, {id: "w-5678", title: "Downloads"}] - -# Focus a specific window -agent-desktop focus-window --window-id "w-5678" - -# Snapshot that specific window -agent-desktop snapshot --app "Finder" --window-id "w-5678" -i -``` - -## Pattern: Check Before Act (Idempotent) - -```bash -# Check if already in desired state -agent-desktop is @e6 --property checked -# If result is false, then check it -agent-desktop check @e6 - -# Or use check/uncheck directly (they're idempotent) -agent-desktop check @e6 # No-op if already checked -agent-desktop uncheck @e6 # No-op if already unchecked -``` - -## Pattern: Batch Operations - -```bash -# Run multiple commands atomically -agent-desktop batch '[ - {"command":"click","args":{"ref_id":"@e1"}}, - {"command":"wait","args":{"ms":200}}, - {"command":"type","args":{"ref_id":"@e2","text":"hello"}}, - {"command":"press","args":{"combo":"return"}} -]' --stop-on-error -``` - -## Anti-Patterns to Avoid - -1. **Acting without observing.** Never click a ref without a recent snapshot. -2. **Hardcoding refs.** Refs change between snapshots. Always use fresh refs. -3. **Ignoring wait.** After launch, dialog triggers, or menu clicks — always wait before snapshotting. -4. **Using coordinates when refs exist.** AX-based actions are more reliable than coordinate clicks. -5. **Not checking permissions.** Always verify accessibility permission before starting automation. -6. **Deep snapshots of large apps.** Use `--max-depth 5` and `-i` for Xcode, VS Code, etc. -7. **Assuming UI stability.** Re-snapshot after every action that could change the UI. diff --git a/.gitignore b/.gitignore index 2254495..ef7334e 100644 --- a/.gitignore +++ b/.gitignore @@ -20,9 +20,8 @@ Cargo.lock *~ .vim/ -# Claude Code (keep skills and commands tracked, ignore settings) -.claude/settings.json -.claude/settings.local.json +# Claude Code +.claude/ .agents/ # Environment