agent-desktop/src/cli/help_after.txt
2026-06-17 13:43:18 -07:00

132 lines
7 KiB
Text

OBSERVATION
snapshot Accessibility tree as JSON with @ref IDs
screenshot PNG screenshot of an application window
find Search elements by role, name, value, or text (--limit defaults to 50)
get <ref> --property <p> Read element property: text, value, title, bounds, role, states
is <ref> --property <p> Check state: visible, enabled, checked, focused, expanded
list-surfaces Available surfaces for an app
INTERACTION
click <ref> Click element (kAXPress)
double-click <ref> Open via AXOpen; physical double-click uses mouse-click
triple-click <ref> Triple-click element; POLICY_DENIED if physical input is disabled
right-click <ref> Right-click; includes menu when verified
type <ref> <text> Insert text; may use focus fallback only for explicit policy paths
set-value <ref> <value> Set value attribute directly
clear <ref> Clear element value to empty string
focus <ref> Set keyboard focus
select <ref> <value> Select option in list or dropdown
toggle <ref> Toggle checkbox or switch
check <ref> Set checkbox/switch to checked (idempotent)
uncheck <ref> Set checkbox/switch to unchecked (idempotent)
expand <ref> Expand disclosure triangle or tree item
collapse <ref> Collapse disclosure triangle or tree item
scroll <ref> Scroll element (--direction, --amount)
scroll-to <ref> Scroll element into visible area
KEYBOARD
press <combo> Key combo: return, escape, cmd+c, shift+tab ...
key-down <combo> Hold a key or modifier down
key-up <combo> Release a held key or modifier
MOUSE
hover <ref|--xy> Move cursor to element or coordinates
drag Drag from one element/point to another
mouse-move --xy x,y Move cursor to absolute coordinates
mouse-click --xy x,y Click at coordinates (--button, --count)
mouse-down --xy x,y Press mouse button at coordinates
mouse-up --xy x,y Release mouse button at coordinates
APP & WINDOW
launch <app> Launch app and wait until window is visible
close-app <app> Quit app gracefully (--force to kill)
list-windows All visible windows (--app to filter)
list-apps All running GUI applications (--app substring filter)
focus-window Bring window to front and confirm focus
resize-window Resize window (--width, --height)
move-window Move window (--x, --y)
minimize Minimize window
maximize Maximize/zoom window
restore Restore minimized/maximized window
NOTIFICATIONS
list-notifications List notifications from Notification Center
dismiss-notification <n> Dismiss notification by index
dismiss-all-notifications Dismiss all notifications
notification-action <n> <action> Click action button on notification
CLIPBOARD
clipboard-get Read plain-text clipboard
clipboard-set <text> Write text to clipboard
clipboard-clear Clear the clipboard
WAIT
wait [ms] Pause for N milliseconds
wait --element <ref> Block until element appears (--timeout ms)
wait --element <ref> --predicate actionable Block until element is actionable
wait --element <ref> --predicate value --value <text> Block until value matches
wait --element <ref> --snapshot <snapshot_id> Wait against a specific snapshot refmap
wait --window <title> Block until window appears
wait --text <text> Block until text appears in app
wait --menu --app <app> Block until a menu surface is open
wait --menu-closed --app <app> Block until a menu surface is dismissed
wait --notification Block until a new notification arrives (--text filters it)
SYSTEM
status Adapter health, platform, and permission state
permissions Check nested permission states: {state,...}
version Version string (--json for machine-readable)
skills Bundled skill docs for AI agents (list, get, path)
BATCH
batch <json> Run commands from a JSON array (--stop-on-error)
batch items may set "session": "id" to override the inherited --session
REF IDs
snapshot assigns @e1, @e2, ... to interactive elements in depth-first order.
Use a ref wherever <ref> appears. Refs are snapshot-scoped. Pass
--snapshot <snapshot_id> to pin a command to a known snapshot; when omitted,
ref commands use the latest saved snapshot. Run snapshot again after UI changes.
Use --session <id> as a shared namespace for concurrent agents; each session
stores multiple snapshot IDs plus its own latest-snapshot pointer.
Ref actions use strict resolution: stale targets return STALE_REF; duplicate
plausible targets return AMBIGUOUS_TARGET instead of choosing arbitrarily.
Ref actions run actionability checks before dispatch. Use --trace <path> to
write JSONL diagnostics outside stdout; add --trace-strict to fail on trace
write errors.
KEY COMBOS
Single keys: return, escape, tab, space, delete, up, down, left, right
Function keys: f1 - f12
With modifiers: cmd+c, cmd+v, cmd+z, cmd+shift+z, ctrl+a, shift+tab
Modifiers: cmd, ctrl, alt, shift
EXAMPLES
agent-desktop skills get desktop --full Load the primary skill (start here)
agent-desktop --session task-1 snapshot --app "System Settings" -i
agent-desktop find --role button --name "OK"
agent-desktop find --role button --limit 20
agent-desktop click @e5 --snapshot <snapshot_id>
agent-desktop check @e3 --snapshot <snapshot_id>
agent-desktop type @e2 --snapshot <snapshot_id> "hello@example.com"
agent-desktop press cmd+z
agent-desktop drag --from @e1 --to @e5
agent-desktop hover @e5
agent-desktop minimize --app TextEdit
agent-desktop resize-window --app TextEdit --width 800 --height 600
agent-desktop mouse-click --xy 500,300
agent-desktop wait --text "Loading complete" --app Safari --timeout 5000
agent-desktop wait --element @e5 --predicate actionable --timeout 5000
agent-desktop --session run-a --trace /tmp/ad.jsonl click @e5
agent-desktop batch '[{"command":"click","args":{"ref_id":"@e1","snapshot":"<snapshot_id>"}}]'
agent-desktop --session run-a batch '[{"command":"status","session":"run-b","args":{}}]'
AGENT LOOP
1. Use `skills get desktop --full` when you need detailed guidance.
2. Use `--session <id>` when another agent or task may also run commands.
3. Observe with `snapshot -i` or `snapshot --skeleton -i --compact`.
4. Keep `snapshot_id`; pass it to every ref-consuming action.
5. Use `wait --predicate actionable`, `wait --text`, or `wait --window`
instead of sleeping blindly.
6. On STALE_REF or AMBIGUOUS_TARGET, snapshot again and choose a fresh ref.
7. Add `--trace <path>` when diagnosing stale, ambiguous, or actionability failures.