mirror of
https://github.com/lahfir/agent-desktop.git
synced 2026-08-05 13:40:17 +00:00
132 lines
7 KiB
Text
132 lines
7 KiB
Text
OBSERVATION
|
|
snapshot Accessibility tree as JSON with @ref IDs
|
|
screenshot PNG screenshot of an application window
|
|
find Search elements by role, name, value, or text (--limit defaults to 50)
|
|
get <ref> --property <p> Read element property: text, value, title, bounds, role, states
|
|
is <ref> --property <p> Check state: visible, enabled, checked, focused, expanded
|
|
list-surfaces Available surfaces for an app
|
|
|
|
INTERACTION
|
|
click <ref> Click element (kAXPress)
|
|
double-click <ref> Open via AXOpen; physical double-click uses mouse-click
|
|
triple-click <ref> Triple-click element; POLICY_DENIED if physical input is disabled
|
|
right-click <ref> Right-click; includes menu when verified
|
|
type <ref> <text> Insert text; may use focus fallback only for explicit policy paths
|
|
set-value <ref> <value> Set value attribute directly
|
|
clear <ref> Clear element value to empty string
|
|
focus <ref> Set keyboard focus
|
|
select <ref> <value> Select option in list or dropdown
|
|
toggle <ref> Toggle checkbox or switch
|
|
check <ref> Set checkbox/switch to checked (idempotent)
|
|
uncheck <ref> Set checkbox/switch to unchecked (idempotent)
|
|
expand <ref> Expand disclosure triangle or tree item
|
|
collapse <ref> Collapse disclosure triangle or tree item
|
|
scroll <ref> Scroll element (--direction, --amount)
|
|
scroll-to <ref> Scroll element into visible area
|
|
|
|
KEYBOARD
|
|
press <combo> Key combo: return, escape, cmd+c, shift+tab ...
|
|
key-down <combo> Hold a key or modifier down
|
|
key-up <combo> Release a held key or modifier
|
|
|
|
MOUSE
|
|
hover <ref|--xy> Move cursor to element or coordinates
|
|
drag Drag from one element/point to another
|
|
mouse-move --xy x,y Move cursor to absolute coordinates
|
|
mouse-click --xy x,y Click at coordinates (--button, --count)
|
|
mouse-down --xy x,y Press mouse button at coordinates
|
|
mouse-up --xy x,y Release mouse button at coordinates
|
|
|
|
APP & WINDOW
|
|
launch <app> Launch app and wait until window is visible
|
|
close-app <app> Quit app gracefully (--force to kill)
|
|
list-windows All visible windows (--app to filter)
|
|
list-apps All running GUI applications (--app substring filter)
|
|
focus-window Bring window to front and confirm focus
|
|
resize-window Resize window (--width, --height)
|
|
move-window Move window (--x, --y)
|
|
minimize Minimize window
|
|
maximize Maximize/zoom window
|
|
restore Restore minimized/maximized window
|
|
|
|
NOTIFICATIONS
|
|
list-notifications List notifications from Notification Center
|
|
dismiss-notification <n> Dismiss notification by index
|
|
dismiss-all-notifications Dismiss all notifications
|
|
notification-action <n> <action> Click action button on notification
|
|
|
|
CLIPBOARD
|
|
clipboard-get Read plain-text clipboard
|
|
clipboard-set <text> Write text to clipboard
|
|
clipboard-clear Clear the clipboard
|
|
|
|
WAIT
|
|
wait [ms] Pause for N milliseconds
|
|
wait --element <ref> Block until element appears (--timeout ms)
|
|
wait --element <ref> --predicate actionable Block until element is actionable
|
|
wait --element <ref> --predicate value --value <text> Block until value matches
|
|
wait --element <ref> --snapshot <snapshot_id> Wait against a specific snapshot refmap
|
|
wait --window <title> Block until window appears
|
|
wait --text <text> Block until text appears in app
|
|
wait --menu --app <app> Block until a menu surface is open
|
|
wait --menu-closed --app <app> Block until a menu surface is dismissed
|
|
wait --notification Block until a new notification arrives (--text filters it)
|
|
|
|
SYSTEM
|
|
status Adapter health, platform, and permission state
|
|
permissions Check nested permission states: {state,...}
|
|
version Version string (--json for machine-readable)
|
|
skills Bundled skill docs for AI agents (list, get, path)
|
|
|
|
BATCH
|
|
batch <json> Run commands from a JSON array (--stop-on-error)
|
|
batch items may set "session": "id" to override the inherited --session
|
|
|
|
REF IDs
|
|
snapshot assigns @e1, @e2, ... to interactive elements in depth-first order.
|
|
Use a ref wherever <ref> appears. Refs are snapshot-scoped. Pass
|
|
--snapshot <snapshot_id> to pin a command to a known snapshot; when omitted,
|
|
ref commands use the latest saved snapshot. Run snapshot again after UI changes.
|
|
Use --session <id> as a shared namespace for concurrent agents; each session
|
|
stores multiple snapshot IDs plus its own latest-snapshot pointer.
|
|
Ref actions use strict resolution: stale targets return STALE_REF; duplicate
|
|
plausible targets return AMBIGUOUS_TARGET instead of choosing arbitrarily.
|
|
Ref actions run actionability checks before dispatch. Use --trace <path> to
|
|
write JSONL diagnostics outside stdout; add --trace-strict to fail on trace
|
|
write errors.
|
|
|
|
KEY COMBOS
|
|
Single keys: return, escape, tab, space, delete, up, down, left, right
|
|
Function keys: f1 - f12
|
|
With modifiers: cmd+c, cmd+v, cmd+z, cmd+shift+z, ctrl+a, shift+tab
|
|
Modifiers: cmd, ctrl, alt, shift
|
|
|
|
EXAMPLES
|
|
agent-desktop skills get desktop --full Load the primary skill (start here)
|
|
agent-desktop --session task-1 snapshot --app "System Settings" -i
|
|
agent-desktop find --role button --name "OK"
|
|
agent-desktop find --role button --limit 20
|
|
agent-desktop click @e5 --snapshot <snapshot_id>
|
|
agent-desktop check @e3 --snapshot <snapshot_id>
|
|
agent-desktop type @e2 --snapshot <snapshot_id> "hello@example.com"
|
|
agent-desktop press cmd+z
|
|
agent-desktop drag --from @e1 --to @e5
|
|
agent-desktop hover @e5
|
|
agent-desktop minimize --app TextEdit
|
|
agent-desktop resize-window --app TextEdit --width 800 --height 600
|
|
agent-desktop mouse-click --xy 500,300
|
|
agent-desktop wait --text "Loading complete" --app Safari --timeout 5000
|
|
agent-desktop wait --element @e5 --predicate actionable --timeout 5000
|
|
agent-desktop --session run-a --trace /tmp/ad.jsonl click @e5
|
|
agent-desktop batch '[{"command":"click","args":{"ref_id":"@e1","snapshot":"<snapshot_id>"}}]'
|
|
agent-desktop --session run-a batch '[{"command":"status","session":"run-b","args":{}}]'
|
|
|
|
AGENT LOOP
|
|
1. Use `skills get desktop --full` when you need detailed guidance.
|
|
2. Use `--session <id>` when another agent or task may also run commands.
|
|
3. Observe with `snapshot -i` or `snapshot --skeleton -i --compact`.
|
|
4. Keep `snapshot_id`; pass it to every ref-consuming action.
|
|
5. Use `wait --predicate actionable`, `wait --text`, or `wait --window`
|
|
instead of sleeping blindly.
|
|
6. On STALE_REF or AMBIGUOUS_TARGET, snapshot again and choose a fresh ref.
|
|
7. Add `--trace <path>` when diagnosing stale, ambiguous, or actionability failures.
|