From b2769cfad18e47ea0065b8d925e4edf1ddc91b88 Mon Sep 17 00:00:00 2001 From: Lahfir Date: Tue, 30 Jun 2026 21:40:16 -0700 Subject: [PATCH] docs: document session-first trace in bundled skills Agents need the session start/end contract, manifest-gated tracing, activation precedence, and FFI structured-trace requirements in the embedded skill docs that ship with the binary. Co-authored-by: Cursor --- skills/agent-desktop-ffi/SKILL.md | 49 +++++++++++------ skills/agent-desktop/SKILL.md | 23 ++++---- .../references/commands-interaction.md | 2 +- .../references/commands-system.md | 55 ++++++++++++++++++- skills/agent-desktop/references/workflows.md | 25 +++++++++ 5 files changed, 124 insertions(+), 30 deletions(-) diff --git a/skills/agent-desktop-ffi/SKILL.md b/skills/agent-desktop-ffi/SKILL.md index 2eb0e47..c6d6253 100644 --- a/skills/agent-desktop-ffi/SKILL.md +++ b/skills/agent-desktop-ffi/SKILL.md @@ -1,6 +1,6 @@ --- name: agent-desktop-ffi -version: 0.4.0 +version: 0.4.1 tags: ffi, c-bindings, cdylib, python, swift, node, go, rust-ffi requirements: - agent-desktop-ffi @@ -76,10 +76,20 @@ to `ad_execute_by_ref` to drive the CLI-semantics ref-action pipeline symbols increment the major. - **Session adapters.** `ad_adapter_create_with_session("session-id")` associates - the adapter with a named session for refmap persistence. Passing the same session - ID across adapter lifetimes lets `ad_execute_by_ref` with `snapshot_id=NULL` - target the latest snapshot from that session. Session IDs: 1–64 chars, ASCII - alphanumeric / `-` / `_`. Invalid IDs return null (check `ad_last_error_*`). + the adapter with a session namespace for refmap persistence — the same as CLI + `--session `. Passing the same session ID across adapter lifetimes lets + `ad_execute_by_ref` with `snapshot_id=NULL` target the latest snapshot from + that session. Session IDs: 1–64 chars, ASCII alphanumeric / `-` / `_`. + Invalid IDs return null (check `ad_last_error_*`). + +- **Structured session trace (no ABI change).** File-based JSONL tracing activates + only when the session has a manifest with `trace: on` from `session start` + (CLI) or equivalent on-disk setup. `ad_adapter_create_with_session` alone does + **not** create trace files. When tracing is active, `command_context()`-backed + commands append to one segment per OS process under + `~/.agent-desktop/sessions//trace/-.jsonl`. A long-lived host + reuses the same segment filename for all calls in that process. For unstructured + diagnostics regardless of session manifest, use `ad_set_log_callback` (below). - **Main thread only (macOS).** Call every adapter-touching entrypoint (`ad_snapshot`, `ad_execute_by_ref`, `ad_wait`, `ad_get_tree`, `ad_find`, @@ -141,17 +151,24 @@ to `ad_execute_by_ref` to drive the CLI-semantics ref-action pipeline for activation-chain actions. Each entry has `label` and `outcome` strings and is owned by the result; release with `ad_free_action_result(&out)`. -- **Tracing / log callback.** `ad_set_log_callback(cb)` installs a `tracing` - subscriber layer that delivers events as JSON to your callback. `cb` receives an - int32_t level (1=ERROR … 5=TRACE) and a `const char *msg` valid only for the - duration of the call. Pass `NULL` to unregister. The layer is installed on the - first non-null call; if a foreign global subscriber already owns the process at - that point, the install fails with `AD_RESULT_ERR_INTERNAL` and no events are - ever delivered. Sensitive field values (password, token, text, …) are replaced - with `{"redacted":true}` before formatting. A panicking callback is caught and - silently discarded. The callback may fire from threads other than the registering - thread, and may still fire briefly after a `NULL` unregister — keep the callback - and any data it captures valid for the process lifetime. +- **Tracing / log callback.** Two tracing surfaces coexist: + + 1. **Structured file trace** — same JSONL contract as CLI `--trace`, gated by a + `trace: on` session manifest. Segments include `event`, `ts_ms`, `seq`, and + redacted fields. Requires `session start` (or equivalent manifest on disk) + before creating the adapter; plain session-id adapters write nothing to disk. + + 2. **`ad_set_log_callback(cb)`** — installs a `tracing` subscriber layer that + delivers events as JSON to your callback. `cb` receives an int32_t level + (1=ERROR … 5=TRACE) and a `const char *msg` valid only for the duration of + the call. Pass `NULL` to unregister. The layer is installed on the first + non-null call; if a foreign global subscriber already owns the process at + that point, the install fails with `AD_RESULT_ERR_INTERNAL` and no events are + ever delivered. Sensitive field values (password, token, text, …) are + replaced with `{"redacted":true}` before formatting. A panicking callback is + caught and silently discarded. The callback may fire from threads other than + the registering thread, and may still fire briefly after a `NULL` unregister + — keep the callback and any data it captures valid for the process lifetime. - **Wait.** `ad_wait(adapter, args, &out)` runs the full CLI `wait` command (element-appear, window-appear, text-appear, menu-open/close, notification, diff --git a/skills/agent-desktop/SKILL.md b/skills/agent-desktop/SKILL.md index 9e9ca17..2a7c0b6 100644 --- a/skills/agent-desktop/SKILL.md +++ b/skills/agent-desktop/SKILL.md @@ -1,6 +1,6 @@ --- name: agent-desktop -version: 0.3.0 +version: 0.4.0 tags: desktop-automation, accessibility, ai-agent, gui-automation, cli requirements: - agent-desktop @@ -9,8 +9,8 @@ description: > Use when an AI agent needs to observe, interact with, or automate desktop applications (click buttons, fill forms, navigate menus, read UI state, toggle checkboxes, scroll, drag, type text, take screenshots, manage windows, use clipboard, manage notifications). - Covers 54 commands across observation, interaction, keyboard/mouse, app lifecycle, - notifications (macOS), clipboard, wait, and a `skills` command that prints these + Covers 55 commands across observation, interaction, keyboard/mouse, app lifecycle, + notifications (macOS), clipboard, wait, session lifecycle, and a `skills` command bundled docs straight from the binary. Triggers on: "click button", "fill form", "open app", "read UI", "automate desktop", "accessibility tree", "snapshot app", "type into field", "navigate menu", "toggle checkbox", @@ -43,7 +43,7 @@ Detailed documentation is split into focused reference files. Read them as neede |-----------|----------| | `references/commands-observation.md` | snapshot, find, get, is, screenshot, list-surfaces — all flags, output examples | | `references/commands-interaction.md` | click, type, set-value, select, toggle, scroll, drag, keyboard, mouse — choosing the right command | -| `references/commands-system.md` | launch, close, windows, clipboard, wait, batch, status, permissions, version | +| `references/commands-system.md` | launch, close, windows, clipboard, wait, batch, session, status, permissions, version | | `references/workflows.md` | 12 common patterns: forms, menus, dialogs, scroll-find, drag-drop, async wait, anti-patterns | | `references/macos.md` | macOS permissions/TCC, AX API internals, smart activation chain, surfaces, Notification Center, troubleshooting | @@ -95,8 +95,7 @@ Use **progressive skeleton traversal** as the default approach. It reduces token - **Strict resolution:** stale refs return `STALE_REF`; duplicate plausible targets return `AMBIGUOUS_TARGET` instead of choosing arbitrarily. - **Actionability:** ref actions check live visibility, stability, enabled state, supported action, policy, and editability before dispatch. - **Headless vs headed:** ref actions are headless by default (AX-only, no cursor) and fail closed when only a physical gesture would work. `type` uses a focus-fallback base policy because typing needs focus but never moves the cursor. Pass the global `--headed` flag to permit cursor movement and focus stealing so physical fallbacks can complete; the AX path is still tried first, so `--headed` never regresses headless-capable elements. Raw cursor commands (`hover`, `drag`, `mouse-*`) are physical and require `--headed`; keyboard commands (`press`, `key-down`, `key-up`) are explicit low-level input. -- **Sessions:** use `--session ` for concurrent or multi-agent runs that share a latest snapshot pointer; batch entries may override with `"session": "id"`. -- **Trace:** use `--trace ` for JSONL diagnostics outside stdout; `--trace-strict` fails on trace setup and pre-action writes. Post-action success traces are best-effort because the desktop mutation already happened. Trace fields with sensitive key tokens such as `text`, `value`, `expected`, `name`, `username`, `description`, `label`, `query`, `secret`, `token`, `password`, `title`, `url`, `help`, or `placeholder` are redacted to `{ "redacted": true }` at every nesting depth. Token matching handles snake_case, kebab-case, and camelCase keys (for example, `source_window_title`, `api_token`, and `typedText` redact, while `filename` stays readable). Top-level `--trace` is inherited by every `batch` entry, including entries with a `session` override. +- **Sessions and tracing:** run `session start` once per agent run to create a manifest with `trace: on` (default). Subsequent commands in that run record JSONL automatically to per-process segments under `~/.agent-desktop/sessions//trace/-.jsonl` — no `--trace` on every call. A session owns both its trace and its latest-snapshot namespace; activating it (via pointer, env, or flag) relocates implicit "latest" to that session. Explicit `--snapshot ` still resolves cross-session. **`--session ` alone** (no manifest from `session start`) selects only the snapshot namespace — existing callers see no surprise trace files. **`--trace `** still overrides to one atomic file for CI or one-offs. Activation precedence: `--session` > `AGENT_DESKTOP_SESSION` > `~/.agent-desktop/current_session` (written only by `session start`). Concurrent independent agents set `AGENT_DESKTOP_SESSION` per process; the pointer is a single-active-session convenience. Multi-agent shared sessions: each agent acts on the `snapshot_id` from its own `snapshot` call — implicit latest is not a cross-agent guarantee. Run `status` to see `session_id` and `tracing`. Trace lines include `ts_ms`, monotonic per-process `seq`, and redacted sensitive fields (`text`, `value`, `expected`, `name`, `username`, `description`, `label`, `query`, `secret`, `token`, `password`, `title`, `url`, `help`, `placeholder` → `{ "redacted": true }`). `--trace-strict` fails on trace setup and pre-action writes; post-action success traces are best-effort. ## JSON Output Contract @@ -131,7 +130,7 @@ Exit codes: `0` success, `1` structured error, `2` argument error. `TIMEOUT` errors carry a `details` object whose `kind` field selects the schema. `kind: "wait_timeout"` includes `predicate`, `timeout_ms`, and `last_observed` or `last_error`, plus `ref`/`title`/`text_chars` depending on the wait mode. `kind: "chain_deadline"` includes `value_before`, `value_at_timeout`, `target`, and `mutated` (increment waits) or `wanted_expanded`/`observed_expanded` (disclosure waits). `mutated: true` — or an unknown `observed_expanded` state — means re-read the element before retrying; `mutated: false` means the state did not change and retrying directly is safe. -## Command Quick Reference (54 commands) +## Command Quick Reference (55 commands) ### Observation ``` @@ -229,7 +228,11 @@ agent-desktop wait --notification --app "App" # Wait for new notification ### System ``` -agent-desktop status # Health check +agent-desktop session start [--name LABEL] [--no-trace] [--force] # Create trace-enabled session + pointer +agent-desktop session end [id] # Seal manifest, clear pointer +agent-desktop session list # List session manifests +agent-desktop session gc [--older-than SECS] [--ended] # Reclaim ended/stale sessions +agent-desktop status # Health, session_id, tracing, permissions agent-desktop permissions # Check permission agent-desktop permissions --request # Trigger permission dialog agent-desktop version # Version info (always JSON envelope) @@ -251,5 +254,5 @@ agent-desktop skills get desktop --full # Load this skill + all referenc 9. **Use surfaces for overlays.** `snapshot --surface menu` for menus, `--surface sheet` for dialogs. Never `--skeleton` for surfaces — they're already focused. 10. **Batch for performance.** Multiple commands in one invocation. 11. **Headless by default.** Ref actions use semantic AX paths and block silent focus stealing, cursor movement, keyboard synthesis, and pasteboard insertion. Use explicit `focus`, `press`, `hover`, `drag`, or `mouse-*` commands only when physical/headed interaction is intended. -12. **Use sessions for parallel work.** Add `--session ` when multiple agents or batches can run at once. -13. **Trace hard failures.** Add `--trace /tmp/agent-desktop.jsonl` when diagnosing stale, ambiguous, or actionability failures. +12. **Start a session once per run.** `session start` enables automatic tracing and relocates the latest-snapshot namespace. Use `AGENT_DESKTOP_SESSION` for concurrent independent agents; use `--session ` to override the active pointer for a single command. +13. **Trace hard failures.** With an active trace-enabled session, segments are written automatically. Add `--trace /tmp/agent-desktop.jsonl` only when you need a single override file (CI, one-offs). Check `status` when unsure whether tracing is active. diff --git a/skills/agent-desktop/references/commands-interaction.md b/skills/agent-desktop/references/commands-interaction.md index a71a0d9..410122a 100644 --- a/skills/agent-desktop/references/commands-interaction.md +++ b/skills/agent-desktop/references/commands-interaction.md @@ -55,7 +55,7 @@ The command surface is platform-agnostic: every ref action builds an `Action` an | `drag` / drop | no | dragging *is* a cursor press-move-release; no general AX drag. Native cross-app drop needs the OS dragging-session/pasteboard protocol that synthetic events cannot start (works for same-view source-tracked gestures and web/Electron mouse-DnD) | | menu bar (`--surface menubar`) | enumerate/open | the app menu bar is readable and openable; SwiftUI `CommandMenu` items accept AXPress but do not route to their action closure (a SwiftUI limitation, like its Slider) — native AppKit menu items fire. `.contextMenu` item selection works. | -All ref-based interaction commands accept `--snapshot `. Omit it for the active session's latest saved snapshot, or pass the `snapshot_id` returned by `snapshot` to keep scripts pinned to the exact ref map they observed. Explicit snapshot IDs do not require also passing `--session`. +All ref-based interaction commands accept `--snapshot `. Omit it for the active session's latest saved snapshot, or pass the `snapshot_id` returned by `snapshot` to keep scripts pinned to the exact ref map they observed. Explicit snapshot IDs do not require also passing `--session`. After `session start`, implicit latest resolves inside the new session; snapshots taken before the boundary need explicit `--snapshot `. Success responses for ref actions include a `steps` array when the activation chain recorded attempts: each entry is `{ "label": "AXPress", "outcome": "attempted" | "skipped" | "succeeded" }` in execution order, showing which activation path produced the result. diff --git a/skills/agent-desktop/references/commands-system.md b/skills/agent-desktop/references/commands-system.md index 16c54bf..878961c 100644 --- a/skills/agent-desktop/references/commands-system.md +++ b/skills/agent-desktop/references/commands-system.md @@ -236,7 +236,9 @@ Execute multiple commands in sequence from a JSON array. Each entry has `command Batch uses the same typed `Commands` enum, command policy preflight, permission report, and dispatch path as the CLI. Unknown fields are rejected instead of being silently ignored. Nested `batch` is rejected. -Each entry may include `"session": "id"` beside `command` and `args`. If omitted, the entry inherits the top-level `--session`. Use per-entry sessions only when intentionally inspecting or coordinating separate agent runs. Top-level `--trace` is inherited by every entry — including entries with a `session` override — so one JSONL file captures the whole batch. +Each entry may include `"session": "id"` beside `command` and `args`. If omitted, the entry inherits the top-level resolved session. Use per-entry sessions only when intentionally inspecting or coordinating separate agent runs. + +**Trace in batch:** when the top-level CLI passes `--trace `, every entry writes to that single file (override). Without `--trace`, entries inherit the resolved session's manifest-gated segment sink; a per-entry `"session"` override re-derives the sink for that session (events never land in the parent session's segment). Session subcommands (`session start`, etc.) are also available in batch JSON via `"action": "start"|"end"|"list"|"gc"`. | Flag | Default | Description | |------|---------|-------------| @@ -248,7 +250,8 @@ Each entry may include `"session": "id"` beside `command` and `args`. If omitted { "command": "click", "args": { "ref_id": "@e1", "snapshot": "" } }, { "command": "wait", "args": { "ms": 500 } }, { "command": "type", "args": { "ref_id": "@e2", "snapshot": "", "text": "hello" } }, - { "command": "status", "session": "other-agent", "args": {} } + { "command": "status", "session": "other-agent", "args": {} }, + { "command": "session", "args": { "action": "start", "name": "batch-run" } } ] ``` @@ -276,13 +279,59 @@ Each entry may include `"session": "id"` beside `command` and `args`. If omitted `skeleton: true` clamps depth to 3 and tags truncated containers with `children_count`. `root: "@eN"` starts traversal from that ref instead of the window root; it cannot be combined with `surface`. +## Session lifecycle + +Sessions are on-disk containers under `~/.agent-desktop/sessions//` with a `session.json` manifest, snapshot refmaps, and (when tracing is on) a `trace/` directory. **`session start` is the only command that writes `~/.agent-desktop/current_session`.** + +### session start +```bash +agent-desktop session start +agent-desktop session start --name "nightly-run" +agent-desktop session start --no-trace # Namespace only — no automatic JSONL +agent-desktop session start --force # Override pointer even if it references a live session +``` +Creates the session directory, pre-creates `trace/` (when tracing is on), writes `session.json` (`trace: on` unless `--no-trace`), sets the current-session pointer, and prints `{ "session_id", "name", "trace", "created_at" }`. + +Refuses to clobber a pointer that still references a **live** session unless `--force`. Live means an active `refstore.lock` holder or recent writes under `trace/`. + +### session end +```bash +agent-desktop session end +agent-desktop session end run-1719763200123-0 +``` +Seals the manifest with `ended_at` and clears the pointer when it still points at this session. + +### session list +```bash +agent-desktop session list +``` +Returns manifest fields only (`session_id`, `name`, `created_at`, `ended_at`, `trace`) — no subtree walk. + +### session gc +```bash +agent-desktop session gc +agent-desktop session gc --ended +agent-desktop session gc --older-than 3600 +``` +Removes ended sessions that are not live and not pointer-referenced. Never reaps a session with a live lock holder or recent `trace/` activity. Refuses symlinked session directories. + +### Activation (all commands) + +| Source | Precedence | +|--------|------------| +| `--session ` | Highest | +| `AGENT_DESKTOP_SESSION` env var | Middle | +| `~/.agent-desktop/current_session` | Lowest (set only by `session start`) | + +Trace-on requires a manifest with `trace: on` from `session start`. Bare `--session` or FFI `ad_adapter_create_with_session` without that manifest selects the snapshot namespace only. + ## System Health ### status ```bash agent-desktop status ``` -Returns adapter health, platform info, permission report, and latest snapshot metadata (`snapshot_id`, `ref_count`) when available. +Returns adapter health, platform info, permission report, latest snapshot metadata (`snapshot_id`, `ref_count`) when available, plus **`session_id`** (resolved active session, if any) and **`tracing`** (whether structured trace output is configured for this process — explicit `--trace`, or a trace-enabled session manifest). ### permissions ```bash diff --git a/skills/agent-desktop/references/workflows.md b/skills/agent-desktop/references/workflows.md index 86faf03..d2ccadd 100644 --- a/skills/agent-desktop/references/workflows.md +++ b/skills/agent-desktop/references/workflows.md @@ -15,6 +15,31 @@ agent-desktop permissions --request For screenshots, also grant Screen Recording. `permissions` reports `accessibility`, `screen_recording`, and `automation` separately. +## Pattern: Session-Scoped Tracing (Default for Multi-Step Runs) + +Start one session per agent run so tracing and the latest-snapshot namespace follow automatically — no `--trace` on every command. + +```bash +# 1. Start once — creates manifest (trace: on), pointer, and trace/ directory +agent-desktop session start --name "invoice-bot" +# Note session_id from data.session_id (also written to ~/.agent-desktop/current_session) + +# 2. Observe-act loop — segments land under sessions//trace/-*.jsonl +agent-desktop snapshot --app "Preview" -i --compact +agent-desktop click @e3 --snapshot +agent-desktop status # confirms session_id + tracing: true + +# 3. End and reclaim when finished +agent-desktop session end +agent-desktop session gc +``` + +**Concurrent independent agents:** set `AGENT_DESKTOP_SESSION=` in each process instead of sharing the global pointer. Each agent still uses the `snapshot_id` from its own `snapshot` call when sharing a session id. + +**Namespace without tracing:** `session start --no-trace` or bare `--session legacy-id` (no manifest) — snapshots namespaced, no JSONL files. + +**Override file trace:** `--trace /tmp/run.jsonl` still forces a single file regardless of session manifest. + ## Pattern: Progressive Skeleton Traversal (Default for Dense Apps) The recommended approach for Electron apps (Slack, VS Code, Discord) and any app with 50+ interactive elements. Reduces token consumption 78-96%.