mirror of
https://github.com/lahfir/agent-desktop.git
synced 2026-08-05 05:30:21 +00:00
feat: add Claude Code skills for agent-desktop automation
Graph of skills covering all 50 commands with platform-specific sub-skills: - Main index skill with core concepts, ref system, JSON contract - Observation, interaction, and system command references - Common workflow patterns (forms, menus, dialogs, drag-drop) - macOS platform skill (permissions, AX API, troubleshooting) - /desktop slash command for quick reference - PRD addendum for skill maintenance across phases
This commit is contained in:
Lahfir
parent
572a2a3f25
commit
ad91cd32cf
9 changed files with 1387 additions and 2 deletions
78
.claude/commands/desktop.md
Normal file
78
.claude/commands/desktop.md
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
# agent-desktop Quick Reference
|
||||
|
||||
Use this command when you need to automate desktop applications with agent-desktop.
|
||||
|
||||
## Core Loop
|
||||
|
||||
```bash
|
||||
# 1. OBSERVE
|
||||
agent-desktop snapshot --app "$APP" -i
|
||||
|
||||
# 2. Find target element by parsing JSON output (look for ref like @e5)
|
||||
|
||||
# 3. ACT
|
||||
agent-desktop click @e5 # or type, select, toggle, etc.
|
||||
|
||||
# 4. VERIFY
|
||||
agent-desktop snapshot --app "$APP" -i
|
||||
```
|
||||
|
||||
## Common Tasks
|
||||
|
||||
**Click a button:**
|
||||
```bash
|
||||
agent-desktop snapshot --app "App" -i
|
||||
agent-desktop click @e5
|
||||
```
|
||||
|
||||
**Fill a text field:**
|
||||
```bash
|
||||
agent-desktop clear @e2
|
||||
agent-desktop type @e2 "new text"
|
||||
```
|
||||
|
||||
**Toggle a checkbox:**
|
||||
```bash
|
||||
agent-desktop check @e3 # idempotent on
|
||||
agent-desktop uncheck @e3 # idempotent off
|
||||
```
|
||||
|
||||
**Open context menu:**
|
||||
```bash
|
||||
agent-desktop right-click @e5
|
||||
agent-desktop wait --menu --app "App"
|
||||
agent-desktop snapshot --app "App" --surface menu -i
|
||||
agent-desktop click @e7
|
||||
```
|
||||
|
||||
**Navigate menus:**
|
||||
```bash
|
||||
agent-desktop snapshot --app "App" --surface menubar -i
|
||||
agent-desktop click @e1 # File menu
|
||||
agent-desktop wait --menu --app "App"
|
||||
agent-desktop snapshot --app "App" --surface menu -i
|
||||
agent-desktop click @e5 # Menu item
|
||||
```
|
||||
|
||||
**Wait for UI:**
|
||||
```bash
|
||||
agent-desktop wait --text "Done" --app "App" --timeout 5000
|
||||
agent-desktop wait --window "Save" --timeout 5000
|
||||
```
|
||||
|
||||
## First-Time Setup
|
||||
|
||||
```bash
|
||||
agent-desktop permissions --request
|
||||
# Grant in System Settings > Privacy & Security > Accessibility
|
||||
```
|
||||
|
||||
## Detailed Skills
|
||||
|
||||
For comprehensive reference, see:
|
||||
- `.claude/skills/agent-desktop/SKILL.md` — Core concepts and full command index
|
||||
- `.claude/skills/agent-desktop/commands-observation.md` — snapshot, find, get, is, screenshot
|
||||
- `.claude/skills/agent-desktop/commands-interaction.md` — click, type, select, toggle, scroll, drag
|
||||
- `.claude/skills/agent-desktop/commands-system.md` — launch, close, windows, clipboard, wait, batch
|
||||
- `.claude/skills/agent-desktop/workflows.md` — Common automation patterns
|
||||
- `.claude/skills/agent-desktop-macos/SKILL.md` — macOS permissions, troubleshooting, AX details
|
||||
209
.claude/skills/agent-desktop-macos/SKILL.md
Normal file
209
.claude/skills/agent-desktop-macos/SKILL.md
Normal file
|
|
@ -0,0 +1,209 @@
|
|||
---
|
||||
name: agent-desktop-macos
|
||||
description: >
|
||||
macOS platform details for agent-desktop. Use when troubleshooting macOS accessibility
|
||||
permissions (TCC), debugging AX API failures, understanding the smart activation chain,
|
||||
working with macOS surfaces (menu, sheet, alert), or handling macOS-specific errors like
|
||||
PERM_DENIED, ACTION_FAILED, or empty accessibility trees.
|
||||
---
|
||||
|
||||
# agent-desktop: macOS Platform
|
||||
|
||||
macOS-specific details for agent-desktop. Covers permissions, accessibility API behavior, troubleshooting, and platform quirks.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Accessibility Permission (TCC)
|
||||
|
||||
macOS requires explicit Accessibility permission for any process that reads or controls UI elements across applications.
|
||||
|
||||
```bash
|
||||
# Check permission status
|
||||
agent-desktop permissions
|
||||
|
||||
# Trigger the system dialog
|
||||
agent-desktop permissions --request
|
||||
```
|
||||
|
||||
**To grant manually:**
|
||||
1. Open System Settings > Privacy & Security > Accessibility
|
||||
2. Click the lock to make changes
|
||||
3. Add your terminal application (Terminal.app, iTerm2, Warp, VS Code, etc.)
|
||||
4. Toggle it ON
|
||||
|
||||
**Important:** The permission is granted to the **terminal application**, not to agent-desktop itself. If you run agent-desktop from a different terminal, you need to grant that terminal too.
|
||||
|
||||
After granting, restart the terminal for the permission to take effect.
|
||||
|
||||
### Supported macOS Versions
|
||||
|
||||
- macOS 12 (Monterey) and later
|
||||
- Both Intel (x86_64) and Apple Silicon (ARM64)
|
||||
|
||||
## macOS Accessibility API (AX)
|
||||
|
||||
agent-desktop uses the macOS Accessibility API (`AXUIElement`) to read and manipulate UI elements.
|
||||
|
||||
### How It Works
|
||||
|
||||
1. `AXUIElementCreateApplication(pid)` creates a handle to an app's accessibility tree
|
||||
2. Tree traversal reads `kAXChildrenAttribute` recursively
|
||||
3. Attributes like `kAXRoleAttribute`, `kAXNameAttribute`, `kAXValueAttribute` provide element details
|
||||
4. Actions like `kAXPressAction`, `kAXConfirmAction` trigger element behavior
|
||||
|
||||
### Smart Activation Chain
|
||||
|
||||
When you run `click @ref`, agent-desktop doesn't just do a simple click. It runs a multi-step activation chain:
|
||||
|
||||
1. **AXScrollToVisible** — ensure element is on screen
|
||||
2. **AXPress** — standard press action
|
||||
3. **AXConfirm** — confirmation action
|
||||
4. **AXOpen** — open action (for links, files)
|
||||
5. **AXPick** — picker action
|
||||
6. **AXShowAlternateUI** — reveal hidden UI, then press child
|
||||
7. **Child activation** — try pressing child elements
|
||||
8. **AXSelected** — set selected attribute
|
||||
9. **Select via parent** — set parent's selected rows (for tables/lists)
|
||||
10. **Custom actions** — AXPerformCustomAction
|
||||
11. **Focus + activate** — set focus then press/confirm
|
||||
12. **Keyboard activate** — focus + synthesize space key
|
||||
13. **Parent activation** — try pressing ancestor elements
|
||||
14. **Coordinate click** — final fallback: CGEvent click at element bounds center
|
||||
|
||||
For `right-click`, the chain tries AXShowMenu first, then various focus/select combinations before falling back to a coordinate-based right-click.
|
||||
|
||||
### Surfaces
|
||||
|
||||
macOS apps can have multiple accessibility surfaces:
|
||||
|
||||
| Surface | Description | When to use |
|
||||
|---------|-------------|-------------|
|
||||
| `window` | Main application window (default) | General UI interaction |
|
||||
| `focused` | Currently focused element's context | Inspecting active element |
|
||||
| `menu` | Open dropdown or context menu | After click/right-click on menu triggers |
|
||||
| `menubar` | Application menu bar | Navigating File/Edit/View menus |
|
||||
| `sheet` | Modal sheet (Save dialog, etc.) | After triggering sheet dialogs |
|
||||
| `popover` | Popover/popup content | Inspecting tooltips, popovers |
|
||||
| `alert` | System or app alert | Handling alert dialogs |
|
||||
|
||||
```bash
|
||||
# List available surfaces
|
||||
agent-desktop list-surfaces --app "Finder"
|
||||
|
||||
# Snapshot a specific surface
|
||||
agent-desktop snapshot --app "Finder" --surface menu -i
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### PERM_DENIED
|
||||
|
||||
```
|
||||
"code": "PERM_DENIED",
|
||||
"message": "Accessibility permission not granted"
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
1. `agent-desktop permissions --request` to trigger dialog
|
||||
2. System Settings > Privacy & Security > Accessibility
|
||||
3. Add and enable your terminal
|
||||
4. Restart terminal
|
||||
|
||||
### Empty or Sparse Tree
|
||||
|
||||
Some apps don't expose full accessibility trees:
|
||||
- **Electron apps** (VS Code, Slack): Generally good accessibility support
|
||||
- **Custom-rendered UIs** (games, some creative tools): May have no accessibility tree
|
||||
- **Web views**: Content inside WebViews may be limited
|
||||
|
||||
**Try:**
|
||||
- Remove `-i` flag to see all elements including non-interactive ones
|
||||
- Increase `--max-depth` to explore deeper
|
||||
- Use `screenshot` as a visual fallback
|
||||
|
||||
### STALE_REF
|
||||
|
||||
```
|
||||
"code": "STALE_REF",
|
||||
"message": "RefMap is from a previous snapshot"
|
||||
```
|
||||
|
||||
The UI changed between your snapshot and action. Run `snapshot` again and use the new refs.
|
||||
|
||||
### ACTION_FAILED
|
||||
|
||||
```
|
||||
"code": "ACTION_FAILED"
|
||||
```
|
||||
|
||||
The accessibility action was rejected. This can happen when:
|
||||
- The element is disabled
|
||||
- The app is busy or unresponsive
|
||||
- The element doesn't support the requested action
|
||||
|
||||
**Try:**
|
||||
1. Check `is @ref --property enabled` first
|
||||
2. Try coordinate-based click: get bounds with `get @ref --property bounds`, then `mouse-click --xy x,y`
|
||||
3. Use keyboard: `focus @ref` then `press return`
|
||||
|
||||
### APP_NOT_FOUND
|
||||
|
||||
The application isn't running. Launch it first:
|
||||
```bash
|
||||
agent-desktop launch "App Name"
|
||||
```
|
||||
|
||||
### Slow Snapshots
|
||||
|
||||
Large apps (Xcode, Safari with many tabs) can have deep trees.
|
||||
|
||||
**Optimize:**
|
||||
- Use `-i` to filter to interactive elements only
|
||||
- Use `--max-depth 5` to limit depth
|
||||
- Use `--compact` to remove empty structural nodes
|
||||
- Target a specific window with `--window-id`
|
||||
- Use `find` instead of full snapshot when you know what you're looking for
|
||||
|
||||
### Context Menu Doesn't Appear
|
||||
|
||||
After `right-click @ref`, if no menu appears:
|
||||
1. The element may not support context menus
|
||||
2. The app may need to be focused first: `focus-window --app "App"` before right-clicking
|
||||
3. Try `mouse-click --xy x,y --button right` with coordinates from `get @ref --property bounds`
|
||||
|
||||
## macOS-Specific Behavior
|
||||
|
||||
### App Identification
|
||||
|
||||
Apps can be referenced by:
|
||||
- **Display name:** `"System Settings"`, `"Finder"`, `"TextEdit"`
|
||||
- **Bundle ID:** `"com.apple.systempreferences"`, `"com.apple.finder"`
|
||||
|
||||
`launch` accepts both. Other commands use the display name with `--app`.
|
||||
|
||||
### Window IDs
|
||||
|
||||
Window IDs (like `w-4521`) are assigned by macOS and are stable for the lifetime of the window. Use `list-windows` to discover them.
|
||||
|
||||
### Keyboard Shortcuts
|
||||
|
||||
macOS uses `cmd` (Command) as the primary modifier:
|
||||
```bash
|
||||
agent-desktop press cmd+c # Copy
|
||||
agent-desktop press cmd+v # Paste
|
||||
agent-desktop press cmd+z # Undo
|
||||
agent-desktop press cmd+s # Save
|
||||
agent-desktop press cmd+w # Close window
|
||||
agent-desktop press cmd+q # Quit app (use with caution!)
|
||||
```
|
||||
|
||||
### Full-Screen Apps
|
||||
|
||||
Apps in full-screen mode are accessible but may behave differently:
|
||||
- `list-windows` still shows them
|
||||
- Bounds may report full screen dimensions
|
||||
- Some animations may delay UI state updates (use `wait`)
|
||||
|
||||
### Menu Bar Apps
|
||||
|
||||
Menu bar apps (status bar items) can be accessed via `--surface menubar`. The menu bar is a separate surface from the application window.
|
||||
220
.claude/skills/agent-desktop/SKILL.md
Normal file
220
.claude/skills/agent-desktop/SKILL.md
Normal file
|
|
@ -0,0 +1,220 @@
|
|||
---
|
||||
name: agent-desktop
|
||||
description: >
|
||||
Desktop automation via native OS accessibility trees using the agent-desktop CLI.
|
||||
Use when an AI agent needs to observe, interact with, or automate desktop applications
|
||||
(click buttons, fill forms, navigate menus, read UI state, toggle checkboxes, scroll,
|
||||
drag, type text, take screenshots, manage windows, use clipboard). Covers 50 commands
|
||||
across observation, interaction, keyboard/mouse, app lifecycle, clipboard, and wait.
|
||||
Triggers on: "click button", "fill form", "open app", "read UI", "automate desktop",
|
||||
"accessibility tree", "snapshot app", "type into field", "navigate menu", "toggle checkbox",
|
||||
"take screenshot", "desktop automation", "agent-desktop", or any desktop GUI interaction task.
|
||||
Supports macOS (Phase 1), with Windows and Linux planned.
|
||||
---
|
||||
|
||||
# agent-desktop
|
||||
|
||||
Cross-platform CLI tool enabling AI agents to observe and control desktop applications via native OS accessibility trees.
|
||||
|
||||
**Core principle:** agent-desktop is NOT an AI agent. It is a tool that AI agents invoke. It outputs structured JSON with ref-based element identifiers. The observation-action loop lives in the calling agent.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use agent-desktop when you need to:
|
||||
- Read UI state from desktop applications (buttons, text fields, menus, checkboxes)
|
||||
- Interact with desktop app elements (click, type, select, toggle, scroll)
|
||||
- Automate multi-step desktop workflows (fill forms, navigate menus, transfer data)
|
||||
- Wait for UI state changes before proceeding
|
||||
- Take screenshots of application windows
|
||||
|
||||
Do NOT use agent-desktop for:
|
||||
- Web browser automation (use agent-browser instead)
|
||||
- Custom-rendered or game-engine UIs lacking accessibility exposure
|
||||
- Applications that don't expose accessibility trees
|
||||
|
||||
## Skill Graph
|
||||
|
||||
This skill is the index. Platform-specific details and advanced patterns are in sub-skills:
|
||||
|
||||
| Skill | When to use |
|
||||
|-------|-------------|
|
||||
| `agent-desktop-macos` | macOS permissions, AX API quirks, troubleshooting |
|
||||
| `commands-observation.md` | Snapshot, find, get, is, screenshot, list-surfaces |
|
||||
| `commands-interaction.md` | Click, type, set-value, select, toggle, expand, scroll, drag |
|
||||
| `commands-system.md` | Launch, close, windows, clipboard, wait, batch |
|
||||
| `workflows.md` | Common automation patterns (forms, menus, dialogs, navigation) |
|
||||
|
||||
## The Observe-Act Loop
|
||||
|
||||
Every automation follows this pattern:
|
||||
|
||||
```
|
||||
1. OBSERVE → agent-desktop snapshot --app "App Name" -i
|
||||
2. REASON → Parse JSON, find target element by ref (@e1, @e2...)
|
||||
3. ACT → agent-desktop click @e5 (or type, select, toggle...)
|
||||
4. VERIFY → agent-desktop snapshot again to confirm state change
|
||||
5. REPEAT → Continue until task is complete
|
||||
```
|
||||
|
||||
Always snapshot before acting. Refs are snapshot-scoped and become stale after UI changes.
|
||||
|
||||
## Ref System
|
||||
|
||||
- Refs are assigned in depth-first document order: `@e1`, `@e2`, `@e3`...
|
||||
- Only interactive elements receive refs: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell
|
||||
- Static text, groups, and containers do NOT get refs (they remain in the tree for context)
|
||||
- Refs are deterministic within a snapshot but NOT stable across snapshots if the UI changed
|
||||
- After any action that changes UI, run `snapshot` again to get fresh refs
|
||||
|
||||
## JSON Output Contract
|
||||
|
||||
Every command returns a JSON envelope on stdout:
|
||||
|
||||
### Success
|
||||
```json
|
||||
{
|
||||
"version": "1.0",
|
||||
"ok": true,
|
||||
"command": "snapshot",
|
||||
"data": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
### Error
|
||||
```json
|
||||
{
|
||||
"version": "1.0",
|
||||
"ok": false,
|
||||
"command": "click",
|
||||
"error": {
|
||||
"code": "STALE_REF",
|
||||
"message": "RefMap is from a previous snapshot",
|
||||
"suggestion": "Run 'snapshot' to refresh, then retry with updated ref"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Exit Codes
|
||||
- `0` — success (check `ok: true`)
|
||||
- `1` — structured error (JSON with error code)
|
||||
- `2` — argument/parse error
|
||||
|
||||
### Error Codes
|
||||
| Code | Meaning | Recovery |
|
||||
|------|---------|----------|
|
||||
| `PERM_DENIED` | Accessibility permission not granted | Grant in System Settings > Privacy > Accessibility |
|
||||
| `ELEMENT_NOT_FOUND` | Ref not found in current refmap | Re-run snapshot, use fresh ref |
|
||||
| `APP_NOT_FOUND` | Application not running | Launch it first with `launch` |
|
||||
| `ACTION_FAILED` | Accessibility action rejected | Try alternative approach (different action or coordinate-based) |
|
||||
| `ACTION_NOT_SUPPORTED` | Element doesn't support this action | Check available actions, use different command |
|
||||
| `STALE_REF` | Ref from old snapshot | Re-run snapshot to get fresh refs |
|
||||
| `WINDOW_NOT_FOUND` | No matching window | Check app name, use `list-windows` |
|
||||
| `PLATFORM_NOT_SUPPORTED` | Feature not available on this OS | Check platform support |
|
||||
| `TIMEOUT` | Wait condition not met in time | Increase `--timeout` or check condition |
|
||||
| `INVALID_ARGS` | Bad arguments | Check command syntax |
|
||||
| `INTERNAL` | Unexpected internal error | Report bug with `-v` verbose output |
|
||||
|
||||
## Command Quick Reference
|
||||
|
||||
### Observation (see commands-observation.md)
|
||||
```
|
||||
snapshot --app "App" -i # Accessibility tree with refs
|
||||
screenshot --app "App" out.png # PNG screenshot
|
||||
find --app "App" --role button # Search elements
|
||||
get @e1 --property text # Read element property
|
||||
is @e1 --property enabled # Check element state
|
||||
list-surfaces --app "App" # Available surfaces
|
||||
```
|
||||
|
||||
### Interaction (see commands-interaction.md)
|
||||
```
|
||||
click @e5 # Click element
|
||||
double-click @e3 # Double-click
|
||||
right-click @e5 # Right-click (context menu)
|
||||
type @e2 "hello" # Type text into element
|
||||
set-value @e2 "new value" # Set value directly
|
||||
clear @e2 # Clear element value
|
||||
focus @e2 # Set keyboard focus
|
||||
select @e4 "Option B" # Select dropdown option
|
||||
toggle @e6 # Toggle checkbox/switch
|
||||
check @e6 # Idempotent check
|
||||
uncheck @e6 # Idempotent uncheck
|
||||
expand @e7 # Expand disclosure
|
||||
collapse @e7 # Collapse disclosure
|
||||
scroll @e1 --direction down # Scroll element
|
||||
scroll-to @e8 # Scroll element into view
|
||||
```
|
||||
|
||||
### Keyboard & Mouse (see commands-interaction.md)
|
||||
```
|
||||
press cmd+c # Key combo
|
||||
press return --app "App" # Key combo targeted at app
|
||||
key-down shift # Hold key
|
||||
key-up shift # Release key
|
||||
hover @e5 # Move cursor to element
|
||||
hover --xy 500,300 # Move cursor to coordinates
|
||||
drag --from @e1 --to @e5 # Drag between elements
|
||||
mouse-click --xy 500,300 # Click at coordinates
|
||||
mouse-move --xy 100,200 # Move cursor
|
||||
mouse-down --xy 100,200 # Press mouse button
|
||||
mouse-up --xy 300,400 # Release mouse button
|
||||
```
|
||||
|
||||
### App & Window (see commands-system.md)
|
||||
```
|
||||
launch "System Settings" # Launch and wait for window
|
||||
close-app "TextEdit" # Quit gracefully
|
||||
close-app "TextEdit" --force # Force kill
|
||||
list-windows --app "Finder" # List windows
|
||||
list-apps # List running GUI apps
|
||||
focus-window --app "Finder" # Bring window to front
|
||||
resize-window --app "App" --width 800 --height 600
|
||||
move-window --app "App" --x 0 --y 0
|
||||
minimize --app "App"
|
||||
maximize --app "App"
|
||||
restore --app "App"
|
||||
```
|
||||
|
||||
### Clipboard (see commands-system.md)
|
||||
```
|
||||
clipboard-get # Read clipboard
|
||||
clipboard-set "text" # Write to clipboard
|
||||
clipboard-clear # Clear clipboard
|
||||
```
|
||||
|
||||
### Wait (see commands-system.md)
|
||||
```
|
||||
wait 1000 # Pause 1 second
|
||||
wait --element @e5 --timeout 5000 # Wait for element
|
||||
wait --window "Title" # Wait for window
|
||||
wait --text "Done" --app "App" # Wait for text
|
||||
wait --menu --app "App" # Wait for context menu
|
||||
wait --menu-closed --app "App" # Wait for menu dismissal
|
||||
```
|
||||
|
||||
### System (see commands-system.md)
|
||||
```
|
||||
status # Health check
|
||||
permissions # Check accessibility permission
|
||||
permissions --request # Trigger system permission dialog
|
||||
version # Version string
|
||||
version --json # Machine-readable version
|
||||
batch '[...]' # Run multiple commands
|
||||
```
|
||||
|
||||
## Global Flag
|
||||
|
||||
All commands accept `--verbose` / `-v` for debug logging to stderr.
|
||||
|
||||
## Key Principles for Agents
|
||||
|
||||
1. **Always snapshot first.** Never assume UI state. Snapshot, parse, then act.
|
||||
2. **Use `-i` flag.** `snapshot --app "App" -i` filters to interactive elements only, reducing token count.
|
||||
3. **Refs are ephemeral.** After any action that changes UI, snapshot again.
|
||||
4. **Prefer AX actions over coordinates.** `click @e5` is more reliable than `mouse-click --xy 500,300`.
|
||||
5. **Use `wait` for async UI.** After launching apps or triggering dialogs, wait for the expected state.
|
||||
6. **Check permissions first.** Run `permissions` on first use. macOS requires Accessibility permission.
|
||||
7. **Handle errors gracefully.** Parse the JSON `error.code` field and follow `error.suggestion`.
|
||||
8. **Use `find` for targeted searches.** When you know the role and name, `find` is faster than parsing a full snapshot.
|
||||
9. **Use surfaces for menus.** `snapshot --surface menu` captures open menus and context menus.
|
||||
10. **Batch for performance.** Use `batch` to run multiple commands in a single invocation when order matters.
|
||||
223
.claude/skills/agent-desktop/commands-interaction.md
Normal file
223
.claude/skills/agent-desktop/commands-interaction.md
Normal file
|
|
@ -0,0 +1,223 @@
|
|||
# Interaction Commands
|
||||
|
||||
Commands for modifying UI state — clicking, typing, selecting, scrolling, and input synthesis.
|
||||
|
||||
## Click Actions
|
||||
|
||||
All click commands use a smart activation chain (AX-first) that tries accessibility actions before falling back to coordinate-based clicks.
|
||||
|
||||
### click
|
||||
```bash
|
||||
agent-desktop click @e5
|
||||
```
|
||||
Primary activation. Tries AXPress > AXConfirm > AXOpen > AXPick > child activation > focus+activate > coordinate click.
|
||||
|
||||
### double-click
|
||||
```bash
|
||||
agent-desktop double-click @e3
|
||||
```
|
||||
Tries AXOpen first, then two smart activations with 50ms gap, then CGEvent double-click.
|
||||
|
||||
### triple-click
|
||||
```bash
|
||||
agent-desktop triple-click @e2
|
||||
```
|
||||
Three smart activations with 30ms gaps, then CGEvent triple-click. Useful for select-all in text fields.
|
||||
|
||||
### right-click
|
||||
```bash
|
||||
agent-desktop right-click @e5
|
||||
```
|
||||
Opens context menu. Tries AXShowMenu > focus+AXShowMenu > parent/child AXShowMenu > coordinate right-click. Use `wait --menu` after to capture the menu, then `snapshot --surface menu` to read it.
|
||||
|
||||
## Text Input
|
||||
|
||||
### type
|
||||
```bash
|
||||
agent-desktop type @e2 "hello@example.com"
|
||||
agent-desktop type @e2 "multi line\ntext"
|
||||
```
|
||||
Focuses the element then types each character via keyboard synthesis. Handles special characters.
|
||||
|
||||
### set-value
|
||||
```bash
|
||||
agent-desktop set-value @e2 "new value"
|
||||
```
|
||||
Sets the value directly via the AX value attribute. Faster than `type` but may not trigger all UI callbacks. Use for text fields, text areas, and sliders.
|
||||
|
||||
### clear
|
||||
```bash
|
||||
agent-desktop clear @e2
|
||||
```
|
||||
Clears the element's value to an empty string. Equivalent to `set-value @e2 ""`.
|
||||
|
||||
### focus
|
||||
```bash
|
||||
agent-desktop focus @e2
|
||||
```
|
||||
Sets keyboard focus on the element without clicking it.
|
||||
|
||||
## Selection & Toggle
|
||||
|
||||
### select
|
||||
```bash
|
||||
agent-desktop select @e4 "Option B"
|
||||
```
|
||||
Selects an option in a list, dropdown, or combobox by its display text.
|
||||
|
||||
### toggle
|
||||
```bash
|
||||
agent-desktop toggle @e6
|
||||
```
|
||||
Toggles a checkbox or switch to the opposite state.
|
||||
|
||||
### check
|
||||
```bash
|
||||
agent-desktop check @e6
|
||||
```
|
||||
Sets a checkbox or switch to the checked/on state. Idempotent — does nothing if already checked.
|
||||
|
||||
### uncheck
|
||||
```bash
|
||||
agent-desktop uncheck @e6
|
||||
```
|
||||
Sets a checkbox or switch to the unchecked/off state. Idempotent.
|
||||
|
||||
## Expand & Collapse
|
||||
|
||||
### expand
|
||||
```bash
|
||||
agent-desktop expand @e7
|
||||
```
|
||||
Expands a disclosure triangle, tree item, or accordion.
|
||||
|
||||
### collapse
|
||||
```bash
|
||||
agent-desktop collapse @e7
|
||||
```
|
||||
Collapses an expanded disclosure/tree item.
|
||||
|
||||
## Scrolling
|
||||
|
||||
### scroll
|
||||
```bash
|
||||
agent-desktop scroll @e1 --direction down --amount 3
|
||||
agent-desktop scroll @e1 --direction up --amount 5
|
||||
agent-desktop scroll @e1 --direction left --amount 2
|
||||
agent-desktop scroll @e1 --direction right --amount 2
|
||||
```
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--direction` | down | `up`, `down`, `left`, `right` |
|
||||
| `--amount` | 3 | Number of scroll units |
|
||||
|
||||
### scroll-to
|
||||
```bash
|
||||
agent-desktop scroll-to @e8
|
||||
```
|
||||
Scrolls the element into the visible area of its scroll container.
|
||||
|
||||
## Keyboard
|
||||
|
||||
### press
|
||||
```bash
|
||||
agent-desktop press return
|
||||
agent-desktop press escape
|
||||
agent-desktop press cmd+c
|
||||
agent-desktop press cmd+shift+z
|
||||
agent-desktop press shift+tab
|
||||
agent-desktop press f5
|
||||
agent-desktop press cmd+a --app "TextEdit"
|
||||
```
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--app` | Target application (focuses app before pressing) |
|
||||
|
||||
**Key names:** `return`, `escape`, `tab`, `space`, `delete`, `up`, `down`, `left`, `right`, `f1`-`f12`
|
||||
**Modifiers:** `cmd`, `ctrl`, `alt`, `shift` — combine with `+`
|
||||
|
||||
### key-down
|
||||
```bash
|
||||
agent-desktop key-down shift
|
||||
```
|
||||
Holds a key or modifier down. Must be paired with `key-up`.
|
||||
|
||||
### key-up
|
||||
```bash
|
||||
agent-desktop key-up shift
|
||||
```
|
||||
Releases a held key or modifier.
|
||||
|
||||
## Mouse
|
||||
|
||||
### hover
|
||||
```bash
|
||||
agent-desktop hover @e5
|
||||
agent-desktop hover --xy 500,300
|
||||
agent-desktop hover @e5 --duration 2000
|
||||
```
|
||||
Moves cursor to element center or absolute coordinates. Optional `--duration` holds position for N ms.
|
||||
|
||||
### drag
|
||||
```bash
|
||||
agent-desktop drag --from @e1 --to @e5
|
||||
agent-desktop drag --from-xy 100,200 --to-xy 400,500
|
||||
agent-desktop drag --from @e1 --to-xy 400,500 --duration 500
|
||||
```
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--from` | Source element ref |
|
||||
| `--from-xy` | Source coordinates as `x,y` |
|
||||
| `--to` | Destination element ref |
|
||||
| `--to-xy` | Destination coordinates as `x,y` |
|
||||
| `--duration` | Drag duration in milliseconds |
|
||||
|
||||
Can mix ref and coordinate sources (e.g., `--from @e1 --to-xy 400,500`).
|
||||
|
||||
### mouse-move
|
||||
```bash
|
||||
agent-desktop mouse-move --xy 500,300
|
||||
```
|
||||
Moves cursor to absolute screen coordinates.
|
||||
|
||||
### mouse-click
|
||||
```bash
|
||||
agent-desktop mouse-click --xy 500,300
|
||||
agent-desktop mouse-click --xy 500,300 --button right
|
||||
agent-desktop mouse-click --xy 500,300 --count 2
|
||||
```
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--xy` | (required) | Coordinates as `x,y` |
|
||||
| `--button` | left | `left`, `right`, `middle` |
|
||||
| `--count` | 1 | Number of clicks |
|
||||
|
||||
### mouse-down / mouse-up
|
||||
```bash
|
||||
agent-desktop mouse-down --xy 100,200
|
||||
agent-desktop mouse-up --xy 300,400
|
||||
```
|
||||
Low-level press/release for custom drag or hold interactions.
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--xy` | (required) | Coordinates as `x,y` |
|
||||
| `--button` | left | `left`, `right`, `middle` |
|
||||
|
||||
## Choosing the Right Command
|
||||
|
||||
| Goal | Preferred | Alternative |
|
||||
|------|-----------|-------------|
|
||||
| Click a button | `click @ref` | `mouse-click --xy` if AX fails |
|
||||
| Fill a text field | `type @ref "text"` | `set-value @ref "text"` for direct set |
|
||||
| Clear then type | `clear @ref` then `type @ref "new"` | `triple-click @ref` then `type @ref "new"` |
|
||||
| Toggle a checkbox | `check @ref` / `uncheck @ref` | `toggle @ref` if you don't know current state |
|
||||
| Open context menu | `right-click @ref` then `wait --menu` | `mouse-click --xy --button right` |
|
||||
| Select dropdown option | `select @ref "Option"` | `click @ref` then `find` the option |
|
||||
| Navigate a form | `press tab` between fields | `focus @ref` to jump directly |
|
||||
| Copy text | `press cmd+c --app "App"` | `clipboard-set` to set directly |
|
||||
| Scroll to find elements | `scroll @ref --direction down` | `scroll-to @ref` if you have the ref |
|
||||
182
.claude/skills/agent-desktop/commands-observation.md
Normal file
182
.claude/skills/agent-desktop/commands-observation.md
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
# Observation Commands
|
||||
|
||||
Commands for reading UI state without modifying it.
|
||||
|
||||
## snapshot
|
||||
|
||||
Capture the accessibility tree as structured JSON with `@ref` IDs.
|
||||
|
||||
```bash
|
||||
agent-desktop snapshot --app "System Settings" -i
|
||||
agent-desktop snapshot --app "Finder" --max-depth 5 --include-bounds
|
||||
agent-desktop snapshot --app "App" --surface menu
|
||||
agent-desktop snapshot --app "App" --window-id "w-1234"
|
||||
agent-desktop snapshot --app "App" -i --compact
|
||||
```
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--app` | (required) | Application name |
|
||||
| `--window-id` | | Specific window ID from `list-windows` |
|
||||
| `-i` / `--interactive-only` | false | Only include interactive elements (buttons, fields, etc.) |
|
||||
| `--max-depth` | 10 | Maximum tree traversal depth |
|
||||
| `--include-bounds` | false | Include `{x, y, width, height}` for each element |
|
||||
| `--compact` | false | Omit empty structural nodes |
|
||||
| `--surface` | window | Target surface: `window`, `focused`, `menu`, `menubar`, `sheet`, `popover`, `alert` |
|
||||
|
||||
**Output structure:**
|
||||
```json
|
||||
{
|
||||
"version": "1.0",
|
||||
"ok": true,
|
||||
"command": "snapshot",
|
||||
"data": {
|
||||
"app": "System Settings",
|
||||
"window": { "id": "w-4521", "title": "General" },
|
||||
"ref_count": 14,
|
||||
"tree": {
|
||||
"role": "window",
|
||||
"name": "General",
|
||||
"children": [
|
||||
{
|
||||
"ref": "@e1",
|
||||
"role": "button",
|
||||
"name": "About",
|
||||
"states": ["focused"]
|
||||
},
|
||||
{
|
||||
"role": "group",
|
||||
"name": "Appearance",
|
||||
"children": [
|
||||
{
|
||||
"ref": "@e2",
|
||||
"role": "checkbox",
|
||||
"name": "Dark Mode",
|
||||
"value": "0",
|
||||
"states": ["enabled"]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Tips:**
|
||||
- Always use `-i` to keep output compact for LLM context windows
|
||||
- Use `--surface menu` to capture open context menus or dropdown menus
|
||||
- Use `--surface sheet` for modal dialogs
|
||||
- Use `--compact` with `-i` for maximum token efficiency
|
||||
- Combine `--max-depth 5` to limit deep trees (e.g., Xcode)
|
||||
|
||||
## find
|
||||
|
||||
Search elements by role, name, value, or text content.
|
||||
|
||||
```bash
|
||||
agent-desktop find --app "Finder" --role button --name "OK"
|
||||
agent-desktop find --app "TextEdit" --role textfield
|
||||
agent-desktop find --app "Safari" --text "Sign In" --first
|
||||
agent-desktop find --app "App" --role checkbox --count
|
||||
agent-desktop find --app "App" --role button --nth 2
|
||||
```
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--app` | Application name |
|
||||
| `--role` | Accessibility role: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell |
|
||||
| `--name` | Accessible name or label |
|
||||
| `--value` | Current value |
|
||||
| `--text` | Fuzzy match across name, value, title, and description |
|
||||
| `--first` | Return first match only |
|
||||
| `--last` | Return last match only |
|
||||
| `--nth N` | Return Nth match (0-indexed) |
|
||||
| `--count` | Return match count only |
|
||||
|
||||
**Output (matches):**
|
||||
```json
|
||||
{
|
||||
"data": {
|
||||
"matches": [
|
||||
{ "ref": "@e5", "role": "button", "name": "OK", "states": ["enabled"] }
|
||||
],
|
||||
"count": 1
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## get
|
||||
|
||||
Read a specific property from an element.
|
||||
|
||||
```bash
|
||||
agent-desktop get @e1 --property text
|
||||
agent-desktop get @e2 --property value
|
||||
agent-desktop get @e3 --property bounds
|
||||
agent-desktop get @e4 --property role
|
||||
agent-desktop get @e5 --property states
|
||||
agent-desktop get @e1 --property title
|
||||
```
|
||||
|
||||
| Property | Returns |
|
||||
|----------|---------|
|
||||
| `text` | Accessible name/label (default) |
|
||||
| `value` | Current value (text content, slider position, etc.) |
|
||||
| `title` | Window or element title |
|
||||
| `bounds` | `{ x, y, width, height }` rectangle |
|
||||
| `role` | Element role string |
|
||||
| `states` | Array of active states |
|
||||
|
||||
## is
|
||||
|
||||
Check a boolean state on an element.
|
||||
|
||||
```bash
|
||||
agent-desktop is @e1 --property visible
|
||||
agent-desktop is @e2 --property enabled
|
||||
agent-desktop is @e3 --property checked
|
||||
agent-desktop is @e4 --property focused
|
||||
agent-desktop is @e5 --property expanded
|
||||
```
|
||||
|
||||
| Property | Checks |
|
||||
|----------|--------|
|
||||
| `visible` | Element is on screen (default) |
|
||||
| `enabled` | Element is interactable |
|
||||
| `checked` | Checkbox/switch is checked |
|
||||
| `focused` | Element has keyboard focus |
|
||||
| `expanded` | Disclosure/tree item is expanded |
|
||||
|
||||
**Output:**
|
||||
```json
|
||||
{ "data": { "ref": "@e3", "property": "checked", "result": true } }
|
||||
```
|
||||
|
||||
## screenshot
|
||||
|
||||
Capture a PNG screenshot of an application window.
|
||||
|
||||
```bash
|
||||
agent-desktop screenshot --app "Finder"
|
||||
agent-desktop screenshot --app "Finder" output.png
|
||||
agent-desktop screenshot --window-id "w-1234" capture.png
|
||||
```
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--app` | Application name |
|
||||
| `--window-id` | Specific window ID |
|
||||
| (positional) | File path to save PNG (omit for base64 in JSON) |
|
||||
|
||||
When no output path is given, the screenshot is returned as a base64-encoded string in the JSON `data` field.
|
||||
|
||||
## list-surfaces
|
||||
|
||||
List available accessibility surfaces for an application.
|
||||
|
||||
```bash
|
||||
agent-desktop list-surfaces --app "Finder"
|
||||
```
|
||||
|
||||
Returns the available surfaces (window, menu, menubar, sheet, popover, alert) for snapshotting. Use this to discover what surfaces are currently available before targeting a specific one with `snapshot --surface`.
|
||||
184
.claude/skills/agent-desktop/commands-system.md
Normal file
184
.claude/skills/agent-desktop/commands-system.md
Normal file
|
|
@ -0,0 +1,184 @@
|
|||
# System Commands
|
||||
|
||||
App lifecycle, window management, clipboard, wait, and system health commands.
|
||||
|
||||
## App Lifecycle
|
||||
|
||||
### launch
|
||||
```bash
|
||||
agent-desktop launch "System Settings"
|
||||
agent-desktop launch "com.apple.Safari" --timeout 10000
|
||||
```
|
||||
Launches an application by name or bundle ID and waits until its window is visible.
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--timeout` | 30000 | Max wait time in ms for window to appear |
|
||||
|
||||
### close-app
|
||||
```bash
|
||||
agent-desktop close-app "TextEdit"
|
||||
agent-desktop close-app "TextEdit" --force
|
||||
```
|
||||
Quits an application gracefully. Use `--force` to kill the process.
|
||||
|
||||
### list-apps
|
||||
```bash
|
||||
agent-desktop list-apps
|
||||
```
|
||||
Lists all running GUI applications. Returns array of `{ name, pid, bundle_id }`.
|
||||
|
||||
## Window Management
|
||||
|
||||
### list-windows
|
||||
```bash
|
||||
agent-desktop list-windows
|
||||
agent-desktop list-windows --app "Finder"
|
||||
```
|
||||
Lists all visible windows, optionally filtered by app. Returns array of `{ id, title, app_name, pid, bounds }`.
|
||||
|
||||
### focus-window
|
||||
```bash
|
||||
agent-desktop focus-window --app "Finder"
|
||||
agent-desktop focus-window --title "Documents"
|
||||
agent-desktop focus-window --window-id "w-4521"
|
||||
```
|
||||
Brings a window to the front. At least one identifier required.
|
||||
|
||||
### resize-window
|
||||
```bash
|
||||
agent-desktop resize-window --app "TextEdit" --width 800 --height 600
|
||||
```
|
||||
|
||||
### move-window
|
||||
```bash
|
||||
agent-desktop move-window --app "TextEdit" --x 0 --y 0
|
||||
```
|
||||
|
||||
### minimize
|
||||
```bash
|
||||
agent-desktop minimize --app "TextEdit"
|
||||
```
|
||||
|
||||
### maximize
|
||||
```bash
|
||||
agent-desktop maximize --app "TextEdit"
|
||||
```
|
||||
Zooms the window to fill the screen.
|
||||
|
||||
### restore
|
||||
```bash
|
||||
agent-desktop restore --app "TextEdit"
|
||||
```
|
||||
Restores a minimized or maximized window to its previous size.
|
||||
|
||||
## Clipboard
|
||||
|
||||
### clipboard-get
|
||||
```bash
|
||||
agent-desktop clipboard-get
|
||||
```
|
||||
Returns `{ "data": { "text": "clipboard contents" } }`.
|
||||
|
||||
### clipboard-set
|
||||
```bash
|
||||
agent-desktop clipboard-set "Hello, world!"
|
||||
```
|
||||
|
||||
### clipboard-clear
|
||||
```bash
|
||||
agent-desktop clipboard-clear
|
||||
```
|
||||
|
||||
## Wait
|
||||
|
||||
### wait (time)
|
||||
```bash
|
||||
agent-desktop wait 1000
|
||||
```
|
||||
Pauses for N milliseconds. Use between actions that need time to settle.
|
||||
|
||||
### wait (element)
|
||||
```bash
|
||||
agent-desktop wait --element @e5 --timeout 5000 --app "App"
|
||||
```
|
||||
Blocks until the element ref appears in the accessibility tree. Useful after triggering UI changes.
|
||||
|
||||
### wait (window)
|
||||
```bash
|
||||
agent-desktop wait --window "Save As" --timeout 10000
|
||||
```
|
||||
Blocks until a window with the given title appears.
|
||||
|
||||
### wait (text)
|
||||
```bash
|
||||
agent-desktop wait --text "Loading complete" --app "Safari" --timeout 5000
|
||||
```
|
||||
Blocks until the specified text appears anywhere in the app's accessibility tree.
|
||||
|
||||
### wait (menu)
|
||||
```bash
|
||||
agent-desktop wait --menu --app "Finder" --timeout 3000
|
||||
```
|
||||
Blocks until a context menu is detected as open.
|
||||
|
||||
### wait (menu-closed)
|
||||
```bash
|
||||
agent-desktop wait --menu-closed --app "Finder" --timeout 3000
|
||||
```
|
||||
Blocks until the context menu is dismissed.
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| (positional) | | Milliseconds to pause |
|
||||
| `--element` | | Ref to wait for |
|
||||
| `--window` | | Window title to wait for |
|
||||
| `--text` | | Text to wait for |
|
||||
| `--menu` | false | Wait for context menu to open |
|
||||
| `--menu-closed` | false | Wait for context menu to close |
|
||||
| `--timeout` | 30000 | Timeout in ms (for element/window/text/menu waits) |
|
||||
| `--app` | | Scope the wait to a specific application |
|
||||
|
||||
## Batch
|
||||
|
||||
### batch
|
||||
```bash
|
||||
agent-desktop batch '[{"command":"click","args":{"ref_id":"@e1"}},{"command":"wait","args":{"ms":500}},{"command":"click","args":{"ref_id":"@e2"}}]'
|
||||
agent-desktop batch '[...]' --stop-on-error
|
||||
```
|
||||
Execute multiple commands in sequence from a JSON array. Each entry has `command` (string) and `args` (object).
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--stop-on-error` | false | Halt on first failed command |
|
||||
|
||||
**Batch format:**
|
||||
```json
|
||||
[
|
||||
{ "command": "click", "args": { "ref_id": "@e1" } },
|
||||
{ "command": "wait", "args": { "ms": 500 } },
|
||||
{ "command": "type", "args": { "ref_id": "@e2", "text": "hello" } }
|
||||
]
|
||||
```
|
||||
|
||||
## System Health
|
||||
|
||||
### status
|
||||
```bash
|
||||
agent-desktop status
|
||||
```
|
||||
Returns adapter health, platform info, and permission state.
|
||||
|
||||
### permissions
|
||||
```bash
|
||||
agent-desktop permissions
|
||||
agent-desktop permissions --request
|
||||
```
|
||||
Checks accessibility permission status. Use `--request` to trigger the macOS system dialog.
|
||||
|
||||
### version
|
||||
```bash
|
||||
agent-desktop version
|
||||
agent-desktop version --json
|
||||
```
|
||||
Returns version string. Use `--json` for `{ "version": "0.1.3", "platform": "macos", "arch": "aarch64" }`.
|
||||
219
.claude/skills/agent-desktop/workflows.md
Normal file
219
.claude/skills/agent-desktop/workflows.md
Normal file
|
|
@ -0,0 +1,219 @@
|
|||
# Common Automation Workflows
|
||||
|
||||
Patterns for using agent-desktop effectively in multi-step desktop automation tasks.
|
||||
|
||||
## First-Time Setup
|
||||
|
||||
Before any automation, verify permissions:
|
||||
|
||||
```bash
|
||||
agent-desktop permissions
|
||||
# If PERM_DENIED:
|
||||
agent-desktop permissions --request
|
||||
# Then: System Settings > Privacy & Security > Accessibility > enable your terminal
|
||||
```
|
||||
|
||||
## Pattern: Fill a Form
|
||||
|
||||
```bash
|
||||
# 1. Snapshot the form
|
||||
agent-desktop snapshot --app "System Settings" -i
|
||||
|
||||
# 2. Parse output, identify text fields by name
|
||||
# Found: @e3 = "Computer Name" textfield, @e5 = "Local Hostname" textfield
|
||||
|
||||
# 3. Clear and fill each field
|
||||
agent-desktop clear @e3
|
||||
agent-desktop type @e3 "My MacBook Pro"
|
||||
agent-desktop clear @e5
|
||||
agent-desktop type @e5 "my-macbook-pro"
|
||||
|
||||
# 4. Click the save/apply button
|
||||
agent-desktop click @e8
|
||||
|
||||
# 5. Verify success
|
||||
agent-desktop snapshot --app "System Settings" -i
|
||||
```
|
||||
|
||||
## Pattern: Navigate Menus
|
||||
|
||||
```bash
|
||||
# 1. Click the menu item
|
||||
agent-desktop snapshot --app "TextEdit" --surface menubar -i
|
||||
# Found: @e1 = "File" menuitem
|
||||
|
||||
agent-desktop click @e1
|
||||
agent-desktop wait --menu --app "TextEdit"
|
||||
agent-desktop snapshot --app "TextEdit" --surface menu -i
|
||||
# Found: @e5 = "Save As..." menuitem
|
||||
|
||||
agent-desktop click @e5
|
||||
|
||||
# 2. Wait for the dialog
|
||||
agent-desktop wait --window "Save"
|
||||
agent-desktop snapshot --app "TextEdit" -i
|
||||
```
|
||||
|
||||
## Pattern: Right-Click Context Menu
|
||||
|
||||
```bash
|
||||
# 1. Right-click the target element
|
||||
agent-desktop right-click @e3
|
||||
|
||||
# 2. Wait for context menu to appear
|
||||
agent-desktop wait --menu --app "Finder" --timeout 3000
|
||||
|
||||
# 3. Snapshot the menu surface
|
||||
agent-desktop snapshot --app "Finder" --surface menu -i
|
||||
|
||||
# 4. Click the desired menu item
|
||||
agent-desktop click @e7
|
||||
|
||||
# 5. Wait for menu to close
|
||||
agent-desktop wait --menu-closed --app "Finder" --timeout 2000
|
||||
```
|
||||
|
||||
## Pattern: Handle a Dialog
|
||||
|
||||
```bash
|
||||
# After triggering a dialog (save, alert, confirmation):
|
||||
agent-desktop wait --window "Save As" --timeout 5000
|
||||
agent-desktop snapshot --app "TextEdit" -i
|
||||
|
||||
# Fill dialog fields
|
||||
agent-desktop type @e2 "my-document.txt"
|
||||
|
||||
# Click OK/Save
|
||||
agent-desktop click @e5
|
||||
```
|
||||
|
||||
## Pattern: Scroll and Find
|
||||
|
||||
When the target element isn't visible and you need to scroll to find it:
|
||||
|
||||
```bash
|
||||
# 1. Find the scroll area
|
||||
agent-desktop snapshot --app "App" -i
|
||||
# Found: @e1 = scroll area
|
||||
|
||||
# 2. Scroll and search in a loop
|
||||
agent-desktop scroll @e1 --direction down --amount 5
|
||||
agent-desktop find --app "App" --name "Target Item"
|
||||
# If no matches, scroll again
|
||||
agent-desktop scroll @e1 --direction down --amount 5
|
||||
agent-desktop find --app "App" --name "Target Item"
|
||||
# Found: @e14 = "Target Item"
|
||||
agent-desktop click @e14
|
||||
```
|
||||
|
||||
## Pattern: Tab Through Fields
|
||||
|
||||
```bash
|
||||
# For sequential form filling without needing refs for each field:
|
||||
agent-desktop click @e1 # Focus first field
|
||||
agent-desktop type @e1 "value1"
|
||||
agent-desktop press tab
|
||||
# Now in next field — type directly since focus moved
|
||||
agent-desktop press tab # Skip a field
|
||||
agent-desktop type @e3 "value3" # Or snapshot again to get new refs
|
||||
```
|
||||
|
||||
## Pattern: Copy Text from Element
|
||||
|
||||
```bash
|
||||
# Option A: Read directly via accessibility
|
||||
agent-desktop get @e5 --property value
|
||||
|
||||
# Option B: Copy via keyboard
|
||||
agent-desktop click @e5
|
||||
agent-desktop press cmd+a
|
||||
agent-desktop press cmd+c
|
||||
agent-desktop clipboard-get
|
||||
```
|
||||
|
||||
## Pattern: Drag and Drop
|
||||
|
||||
```bash
|
||||
# Between elements (by ref)
|
||||
agent-desktop drag --from @e3 --to @e8
|
||||
|
||||
# Between coordinates
|
||||
agent-desktop drag --from-xy 100,200 --to-xy 500,400
|
||||
|
||||
# Mixed: element to coordinates
|
||||
agent-desktop drag --from @e3 --to-xy 500,400 --duration 500
|
||||
```
|
||||
|
||||
## Pattern: Wait for Async UI
|
||||
|
||||
```bash
|
||||
# After triggering a long operation:
|
||||
agent-desktop click @e5 # "Download" button
|
||||
|
||||
# Wait for completion text
|
||||
agent-desktop wait --text "Download complete" --app "App" --timeout 30000
|
||||
|
||||
# Or wait for a specific element to appear
|
||||
agent-desktop wait --element @e10 --timeout 10000
|
||||
```
|
||||
|
||||
## Pattern: Launch, Automate, Close
|
||||
|
||||
```bash
|
||||
# Full lifecycle
|
||||
agent-desktop launch "Calculator"
|
||||
agent-desktop snapshot --app "Calculator" -i
|
||||
|
||||
# ... perform automation ...
|
||||
|
||||
agent-desktop close-app "Calculator"
|
||||
```
|
||||
|
||||
## Pattern: Multi-Window Workflow
|
||||
|
||||
```bash
|
||||
# List windows to find the right one
|
||||
agent-desktop list-windows --app "Finder"
|
||||
# Returns: [{id: "w-1234", title: "Documents"}, {id: "w-5678", title: "Downloads"}]
|
||||
|
||||
# Focus a specific window
|
||||
agent-desktop focus-window --window-id "w-5678"
|
||||
|
||||
# Snapshot that specific window
|
||||
agent-desktop snapshot --app "Finder" --window-id "w-5678" -i
|
||||
```
|
||||
|
||||
## Pattern: Check Before Act (Idempotent)
|
||||
|
||||
```bash
|
||||
# Check if already in desired state
|
||||
agent-desktop is @e6 --property checked
|
||||
# If result is false, then check it
|
||||
agent-desktop check @e6
|
||||
|
||||
# Or use check/uncheck directly (they're idempotent)
|
||||
agent-desktop check @e6 # No-op if already checked
|
||||
agent-desktop uncheck @e6 # No-op if already unchecked
|
||||
```
|
||||
|
||||
## Pattern: Batch Operations
|
||||
|
||||
```bash
|
||||
# Run multiple commands atomically
|
||||
agent-desktop batch '[
|
||||
{"command":"click","args":{"ref_id":"@e1"}},
|
||||
{"command":"wait","args":{"ms":200}},
|
||||
{"command":"type","args":{"ref_id":"@e2","text":"hello"}},
|
||||
{"command":"press","args":{"combo":"return"}}
|
||||
]' --stop-on-error
|
||||
```
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
1. **Acting without observing.** Never click a ref without a recent snapshot.
|
||||
2. **Hardcoding refs.** Refs change between snapshots. Always use fresh refs.
|
||||
3. **Ignoring wait.** After launch, dialog triggers, or menu clicks — always wait before snapshotting.
|
||||
4. **Using coordinates when refs exist.** AX-based actions are more reliable than coordinate clicks.
|
||||
5. **Not checking permissions.** Always verify accessibility permission before starting automation.
|
||||
6. **Deep snapshots of large apps.** Use `--max-depth 5` and `-i` for Xcode, VS Code, etc.
|
||||
7. **Assuming UI stability.** Re-snapshot after every action that could change the UI.
|
||||
5
.gitignore
vendored
5
.gitignore
vendored
|
|
@ -20,8 +20,9 @@ Cargo.lock
|
|||
*~
|
||||
.vim/
|
||||
|
||||
# Claude Code
|
||||
.claude/
|
||||
# Claude Code (keep skills and commands tracked, ignore settings)
|
||||
.claude/settings.json
|
||||
.claude/settings.local.json
|
||||
.agents/
|
||||
|
||||
# Environment
|
||||
|
|
|
|||
69
docs/prd-addendum-skill-maintenance.md
Normal file
69
docs/prd-addendum-skill-maintenance.md
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
# PRD Addendum: Skill Maintenance Across Phases
|
||||
|
||||
**Date:** 2026-02-23
|
||||
**Applies to:** agent-desktop PRD v2.0, all phases
|
||||
|
||||
## Overview
|
||||
|
||||
agent-desktop ships Claude Code skills (`.claude/skills/` and `.claude/commands/`) that teach AI coding agents how to use agent-desktop effectively. These skills must be kept in sync with the tool's capabilities across all phases.
|
||||
|
||||
## Skill Graph Structure
|
||||
|
||||
```
|
||||
.claude/
|
||||
├── commands/
|
||||
│ └── desktop.md # Quick reference slash command
|
||||
└── skills/
|
||||
├── agent-desktop/ # Core skill (platform-agnostic)
|
||||
│ ├── SKILL.md # Index: concepts, JSON contract, command overview
|
||||
│ ├── commands-observation.md # snapshot, find, get, is, screenshot
|
||||
│ ├── commands-interaction.md # click, type, select, toggle, scroll, drag
|
||||
│ ├── commands-system.md # launch, close, windows, clipboard, wait, batch
|
||||
│ └── workflows.md # Common automation patterns
|
||||
└── agent-desktop-macos/ # macOS platform skill
|
||||
└── SKILL.md # Permissions, AX API, troubleshooting
|
||||
```
|
||||
|
||||
## Phase-Specific Skill Updates
|
||||
|
||||
### Phase 1 (Foundation + macOS MVP) — CURRENT
|
||||
- All skill files created covering 50 commands
|
||||
- macOS platform skill with AX API details, permissions, troubleshooting
|
||||
- Core workflows: form filling, menu navigation, context menus, scrolling, dialogs
|
||||
|
||||
### Phase 2 (Cross-Platform Expansion)
|
||||
When Windows and Linux adapters ship:
|
||||
- [ ] Create `.claude/skills/agent-desktop-windows/SKILL.md`
|
||||
- UIA permission model
|
||||
- Windows-specific behaviors (UAC, WinUI3 quirks)
|
||||
- Troubleshooting guide
|
||||
- [ ] Create `.claude/skills/agent-desktop-linux/SKILL.md`
|
||||
- AT-SPI2/D-Bus setup
|
||||
- Wayland vs X11 differences
|
||||
- Troubleshooting guide
|
||||
- [ ] Update core `SKILL.md` skill graph table to include new platform skills
|
||||
- [ ] Update `workflows.md` with cross-platform patterns
|
||||
|
||||
### Phase 3 (MCP Server Mode)
|
||||
When `--mcp` flag ships:
|
||||
- [ ] Create `.claude/skills/agent-desktop-mcp/SKILL.md`
|
||||
- MCP tool surface documentation
|
||||
- Transport configuration (stdio, SSE)
|
||||
- Session management
|
||||
- Tool-to-CLI mapping reference
|
||||
- [ ] Update core `SKILL.md` with MCP mode section
|
||||
- [ ] Add MCP workflow patterns to `workflows.md`
|
||||
|
||||
### Phase 4 (Production Hardening)
|
||||
When daemon mode and sessions ship:
|
||||
- [ ] Update `commands-system.md` with session commands
|
||||
- [ ] Add daemon lifecycle patterns to `workflows.md`
|
||||
- [ ] Document enterprise quality gates in platform skills
|
||||
|
||||
## Maintenance Rules
|
||||
|
||||
1. **Every new command** must be added to the appropriate commands-*.md file
|
||||
2. **Every new platform** gets its own skill directory under `.claude/skills/agent-desktop-{platform}/`
|
||||
3. **Every new mode** (MCP, daemon) gets its own skill file
|
||||
4. **Breaking changes** to JSON output or CLI flags must update all affected skill files
|
||||
5. **Skill files are reviewed** as part of the PR checklist for any command-surface change
|
||||
Loading…
Reference in a new issue