Creepso/agent-desktop
1
0
Fork 0
mirror of https://github.com/lahfir/agent-desktop.git synced 2026-07-27 09:32:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 6

docs: add surface-first snapshot rule for overlays

Browse source 
When a sheet, alert, popover, or menu is open, agents should snapshot
the specific surface instead of the full window. Background refs are
irrelevant while an overlay has focus and waste tokens.
This commit is contained in:
Lahfir 2026-03-03 12:34:50 -08:00
parent eb607e2b8e
commit ccc734ecbc
2 changed files with 23 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
skills/agent-desktop/references/macos.md
View file

@ -85,6 +85,19 @@ agent-desktop list-surfaces --app "Finder"
agent-desktop snapshot --app "Finder" --surface menu -i
```
### Surface-First Snapshot Rule
**When a snapshot shows a sheet, alert, popover, or menu is open, ALWAYS re-snapshot with the matching `--surface` flag.** The full app tree includes refs for the entire window behind the overlay — those refs are irrelevant and waste tokens. The surface snapshot returns only the overlay's refs, which are the only ones you can meaningfully interact with.
| Snapshot shows... | Next command |
|-------------------|-------------|
| A sheet is open (Save, Export, Print dialog) | `snapshot --app "App" --surface sheet -i` |
| An alert is visible (confirmation, error) | `snapshot --app "App" --surface alert -i` |
| A menu is open (dropdown, context menu) | `snapshot --app "App" --surface menu -i` |
| A popover appeared (tooltip, picker) | `snapshot --app "App" --surface popover -i` |
**Do not interact with refs from the full window snapshot while an overlay is open.** The overlay captures focus — clicks on background refs will either fail or dismiss the overlay unexpectedly. Snapshot the surface, interact with it, then snapshot the window again after the overlay closes.
## Notification Center
agent-desktop interacts with macOS Notification Center via the accessibility API.

13
skills/agent-desktop/references/workflows.md
View file

@ -49,9 +49,9 @@ agent-desktop snapshot --app "TextEdit" --surface menu -i
agent-desktop click @e5
# 2. Wait for the dialog
# 2. Wait for the dialog, then snapshot the SHEET surface (not the full window)
agent-desktop wait --window "Save"
agent-desktop snapshot --app "TextEdit" -i
agent-desktop snapshot --app "TextEdit" --surface sheet -i
```
## Pattern: Right-Click Context Menu
@ -78,13 +78,19 @@ agent-desktop wait --menu-closed --app "Finder" --timeout 2000
```bash
# After triggering a dialog (save, alert, confirmation):
agent-desktop wait --window "Save As" --timeout 5000
agent-desktop snapshot --app "TextEdit" -i
# Snapshot the SURFACE, not the full window — only overlay refs matter
agent-desktop snapshot --app "TextEdit" --surface sheet -i
# For alerts: --surface alert | For popovers: --surface popover
# Fill dialog fields
agent-desktop type @e2 "my-document.txt"
# Click OK/Save
agent-desktop click @e5
# After dialog closes, snapshot the window again for fresh refs
agent-desktop snapshot --app "TextEdit" -i
```
## Pattern: Scroll and Find
@ -217,3 +223,4 @@ agent-desktop batch '[
5. **Not checking permissions.** Always verify accessibility permission before starting automation.
6. **Deep snapshots of large apps.** Use `--max-depth 5` and `-i` for Xcode, VS Code, etc.
7. **Assuming UI stability.** Re-snapshot after every action that could change the UI.
8. **Snapshotting the full window when an overlay is open.** When a sheet, alert, popover, or menu is visible, use `--surface sheet/alert/popover/menu` instead. The full window tree includes irrelevant background refs that waste tokens and can't be interacted with while the overlay has focus.