mirror of
https://github.com/lahfir/agent-desktop.git
synced 2026-07-26 09:02:17 +00:00
docs: document session-first trace in bundled skills
Agents need the session start/end contract, manifest-gated tracing, activation precedence, and FFI structured-trace requirements in the embedded skill docs that ship with the binary. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Lahfir
parent
c1530d5742
commit
b2769cfad1
5 changed files with 124 additions and 30 deletions
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
name: agent-desktop-ffi
|
||||
version: 0.4.0
|
||||
version: 0.4.1
|
||||
tags: ffi, c-bindings, cdylib, python, swift, node, go, rust-ffi
|
||||
requirements:
|
||||
- agent-desktop-ffi
|
||||
|
|
@ -76,10 +76,20 @@ to `ad_execute_by_ref` to drive the CLI-semantics ref-action pipeline
|
|||
symbols increment the major.
|
||||
|
||||
- **Session adapters.** `ad_adapter_create_with_session("session-id")` associates
|
||||
the adapter with a named session for refmap persistence. Passing the same session
|
||||
ID across adapter lifetimes lets `ad_execute_by_ref` with `snapshot_id=NULL`
|
||||
target the latest snapshot from that session. Session IDs: 1–64 chars, ASCII
|
||||
alphanumeric / `-` / `_`. Invalid IDs return null (check `ad_last_error_*`).
|
||||
the adapter with a session namespace for refmap persistence — the same as CLI
|
||||
`--session <id>`. Passing the same session ID across adapter lifetimes lets
|
||||
`ad_execute_by_ref` with `snapshot_id=NULL` target the latest snapshot from
|
||||
that session. Session IDs: 1–64 chars, ASCII alphanumeric / `-` / `_`.
|
||||
Invalid IDs return null (check `ad_last_error_*`).
|
||||
|
||||
- **Structured session trace (no ABI change).** File-based JSONL tracing activates
|
||||
only when the session has a manifest with `trace: on` from `session start`
|
||||
(CLI) or equivalent on-disk setup. `ad_adapter_create_with_session` alone does
|
||||
**not** create trace files. When tracing is active, `command_context()`-backed
|
||||
commands append to one segment per OS process under
|
||||
`~/.agent-desktop/sessions/<id>/trace/<pid>-<procTs>.jsonl`. A long-lived host
|
||||
reuses the same segment filename for all calls in that process. For unstructured
|
||||
diagnostics regardless of session manifest, use `ad_set_log_callback` (below).
|
||||
|
||||
- **Main thread only (macOS).** Call every adapter-touching entrypoint
|
||||
(`ad_snapshot`, `ad_execute_by_ref`, `ad_wait`, `ad_get_tree`, `ad_find`,
|
||||
|
|
@ -141,17 +151,24 @@ to `ad_execute_by_ref` to drive the CLI-semantics ref-action pipeline
|
|||
for activation-chain actions. Each entry has `label` and `outcome` strings and
|
||||
is owned by the result; release with `ad_free_action_result(&out)`.
|
||||
|
||||
- **Tracing / log callback.** `ad_set_log_callback(cb)` installs a `tracing`
|
||||
subscriber layer that delivers events as JSON to your callback. `cb` receives an
|
||||
int32_t level (1=ERROR … 5=TRACE) and a `const char *msg` valid only for the
|
||||
duration of the call. Pass `NULL` to unregister. The layer is installed on the
|
||||
first non-null call; if a foreign global subscriber already owns the process at
|
||||
that point, the install fails with `AD_RESULT_ERR_INTERNAL` and no events are
|
||||
ever delivered. Sensitive field values (password, token, text, …) are replaced
|
||||
with `{"redacted":true}` before formatting. A panicking callback is caught and
|
||||
silently discarded. The callback may fire from threads other than the registering
|
||||
thread, and may still fire briefly after a `NULL` unregister — keep the callback
|
||||
and any data it captures valid for the process lifetime.
|
||||
- **Tracing / log callback.** Two tracing surfaces coexist:
|
||||
|
||||
1. **Structured file trace** — same JSONL contract as CLI `--trace`, gated by a
|
||||
`trace: on` session manifest. Segments include `event`, `ts_ms`, `seq`, and
|
||||
redacted fields. Requires `session start` (or equivalent manifest on disk)
|
||||
before creating the adapter; plain session-id adapters write nothing to disk.
|
||||
|
||||
2. **`ad_set_log_callback(cb)`** — installs a `tracing` subscriber layer that
|
||||
delivers events as JSON to your callback. `cb` receives an int32_t level
|
||||
(1=ERROR … 5=TRACE) and a `const char *msg` valid only for the duration of
|
||||
the call. Pass `NULL` to unregister. The layer is installed on the first
|
||||
non-null call; if a foreign global subscriber already owns the process at
|
||||
that point, the install fails with `AD_RESULT_ERR_INTERNAL` and no events are
|
||||
ever delivered. Sensitive field values (password, token, text, …) are
|
||||
replaced with `{"redacted":true}` before formatting. A panicking callback is
|
||||
caught and silently discarded. The callback may fire from threads other than
|
||||
the registering thread, and may still fire briefly after a `NULL` unregister
|
||||
— keep the callback and any data it captures valid for the process lifetime.
|
||||
|
||||
- **Wait.** `ad_wait(adapter, args, &out)` runs the full CLI `wait` command
|
||||
(element-appear, window-appear, text-appear, menu-open/close, notification,
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
name: agent-desktop
|
||||
version: 0.3.0
|
||||
version: 0.4.0
|
||||
tags: desktop-automation, accessibility, ai-agent, gui-automation, cli
|
||||
requirements:
|
||||
- agent-desktop
|
||||
|
|
@ -9,8 +9,8 @@ description: >
|
|||
Use when an AI agent needs to observe, interact with, or automate desktop applications
|
||||
(click buttons, fill forms, navigate menus, read UI state, toggle checkboxes, scroll,
|
||||
drag, type text, take screenshots, manage windows, use clipboard, manage notifications).
|
||||
Covers 54 commands across observation, interaction, keyboard/mouse, app lifecycle,
|
||||
notifications (macOS), clipboard, wait, and a `skills` command that prints these
|
||||
Covers 55 commands across observation, interaction, keyboard/mouse, app lifecycle,
|
||||
notifications (macOS), clipboard, wait, session lifecycle, and a `skills` command
|
||||
bundled docs straight from the binary.
|
||||
Triggers on: "click button", "fill form", "open app", "read UI", "automate desktop",
|
||||
"accessibility tree", "snapshot app", "type into field", "navigate menu", "toggle checkbox",
|
||||
|
|
@ -43,7 +43,7 @@ Detailed documentation is split into focused reference files. Read them as neede
|
|||
|-----------|----------|
|
||||
| `references/commands-observation.md` | snapshot, find, get, is, screenshot, list-surfaces — all flags, output examples |
|
||||
| `references/commands-interaction.md` | click, type, set-value, select, toggle, scroll, drag, keyboard, mouse — choosing the right command |
|
||||
| `references/commands-system.md` | launch, close, windows, clipboard, wait, batch, status, permissions, version |
|
||||
| `references/commands-system.md` | launch, close, windows, clipboard, wait, batch, session, status, permissions, version |
|
||||
| `references/workflows.md` | 12 common patterns: forms, menus, dialogs, scroll-find, drag-drop, async wait, anti-patterns |
|
||||
| `references/macos.md` | macOS permissions/TCC, AX API internals, smart activation chain, surfaces, Notification Center, troubleshooting |
|
||||
|
||||
|
|
@ -95,8 +95,7 @@ Use **progressive skeleton traversal** as the default approach. It reduces token
|
|||
- **Strict resolution:** stale refs return `STALE_REF`; duplicate plausible targets return `AMBIGUOUS_TARGET` instead of choosing arbitrarily.
|
||||
- **Actionability:** ref actions check live visibility, stability, enabled state, supported action, policy, and editability before dispatch.
|
||||
- **Headless vs headed:** ref actions are headless by default (AX-only, no cursor) and fail closed when only a physical gesture would work. `type` uses a focus-fallback base policy because typing needs focus but never moves the cursor. Pass the global `--headed` flag to permit cursor movement and focus stealing so physical fallbacks can complete; the AX path is still tried first, so `--headed` never regresses headless-capable elements. Raw cursor commands (`hover`, `drag`, `mouse-*`) are physical and require `--headed`; keyboard commands (`press`, `key-down`, `key-up`) are explicit low-level input.
|
||||
- **Sessions:** use `--session <id>` for concurrent or multi-agent runs that share a latest snapshot pointer; batch entries may override with `"session": "id"`.
|
||||
- **Trace:** use `--trace <path>` for JSONL diagnostics outside stdout; `--trace-strict` fails on trace setup and pre-action writes. Post-action success traces are best-effort because the desktop mutation already happened. Trace fields with sensitive key tokens such as `text`, `value`, `expected`, `name`, `username`, `description`, `label`, `query`, `secret`, `token`, `password`, `title`, `url`, `help`, or `placeholder` are redacted to `{ "redacted": true }` at every nesting depth. Token matching handles snake_case, kebab-case, and camelCase keys (for example, `source_window_title`, `api_token`, and `typedText` redact, while `filename` stays readable). Top-level `--trace` is inherited by every `batch` entry, including entries with a `session` override.
|
||||
- **Sessions and tracing:** run `session start` once per agent run to create a manifest with `trace: on` (default). Subsequent commands in that run record JSONL automatically to per-process segments under `~/.agent-desktop/sessions/<id>/trace/<pid>-<procTs>.jsonl` — no `--trace` on every call. A session owns both its trace and its latest-snapshot namespace; activating it (via pointer, env, or flag) relocates implicit "latest" to that session. Explicit `--snapshot <id>` still resolves cross-session. **`--session <id>` alone** (no manifest from `session start`) selects only the snapshot namespace — existing callers see no surprise trace files. **`--trace <path>`** still overrides to one atomic file for CI or one-offs. Activation precedence: `--session` > `AGENT_DESKTOP_SESSION` > `~/.agent-desktop/current_session` (written only by `session start`). Concurrent independent agents set `AGENT_DESKTOP_SESSION` per process; the pointer is a single-active-session convenience. Multi-agent shared sessions: each agent acts on the `snapshot_id` from its own `snapshot` call — implicit latest is not a cross-agent guarantee. Run `status` to see `session_id` and `tracing`. Trace lines include `ts_ms`, monotonic per-process `seq`, and redacted sensitive fields (`text`, `value`, `expected`, `name`, `username`, `description`, `label`, `query`, `secret`, `token`, `password`, `title`, `url`, `help`, `placeholder` → `{ "redacted": true }`). `--trace-strict` fails on trace setup and pre-action writes; post-action success traces are best-effort.
|
||||
|
||||
## JSON Output Contract
|
||||
|
||||
|
|
@ -131,7 +130,7 @@ Exit codes: `0` success, `1` structured error, `2` argument error.
|
|||
|
||||
`TIMEOUT` errors carry a `details` object whose `kind` field selects the schema. `kind: "wait_timeout"` includes `predicate`, `timeout_ms`, and `last_observed` or `last_error`, plus `ref`/`title`/`text_chars` depending on the wait mode. `kind: "chain_deadline"` includes `value_before`, `value_at_timeout`, `target`, and `mutated` (increment waits) or `wanted_expanded`/`observed_expanded` (disclosure waits). `mutated: true` — or an unknown `observed_expanded` state — means re-read the element before retrying; `mutated: false` means the state did not change and retrying directly is safe.
|
||||
|
||||
## Command Quick Reference (54 commands)
|
||||
## Command Quick Reference (55 commands)
|
||||
|
||||
### Observation
|
||||
```
|
||||
|
|
@ -229,7 +228,11 @@ agent-desktop wait --notification --app "App" # Wait for new notification
|
|||
|
||||
### System
|
||||
```
|
||||
agent-desktop status # Health check
|
||||
agent-desktop session start [--name LABEL] [--no-trace] [--force] # Create trace-enabled session + pointer
|
||||
agent-desktop session end [id] # Seal manifest, clear pointer
|
||||
agent-desktop session list # List session manifests
|
||||
agent-desktop session gc [--older-than SECS] [--ended] # Reclaim ended/stale sessions
|
||||
agent-desktop status # Health, session_id, tracing, permissions
|
||||
agent-desktop permissions # Check permission
|
||||
agent-desktop permissions --request # Trigger permission dialog
|
||||
agent-desktop version # Version info (always JSON envelope)
|
||||
|
|
@ -251,5 +254,5 @@ agent-desktop skills get desktop --full # Load this skill + all referenc
|
|||
9. **Use surfaces for overlays.** `snapshot --surface menu` for menus, `--surface sheet` for dialogs. Never `--skeleton` for surfaces — they're already focused.
|
||||
10. **Batch for performance.** Multiple commands in one invocation.
|
||||
11. **Headless by default.** Ref actions use semantic AX paths and block silent focus stealing, cursor movement, keyboard synthesis, and pasteboard insertion. Use explicit `focus`, `press`, `hover`, `drag`, or `mouse-*` commands only when physical/headed interaction is intended.
|
||||
12. **Use sessions for parallel work.** Add `--session <id>` when multiple agents or batches can run at once.
|
||||
13. **Trace hard failures.** Add `--trace /tmp/agent-desktop.jsonl` when diagnosing stale, ambiguous, or actionability failures.
|
||||
12. **Start a session once per run.** `session start` enables automatic tracing and relocates the latest-snapshot namespace. Use `AGENT_DESKTOP_SESSION` for concurrent independent agents; use `--session <id>` to override the active pointer for a single command.
|
||||
13. **Trace hard failures.** With an active trace-enabled session, segments are written automatically. Add `--trace /tmp/agent-desktop.jsonl` only when you need a single override file (CI, one-offs). Check `status` when unsure whether tracing is active.
|
||||
|
|
|
|||
|
|
@ -55,7 +55,7 @@ The command surface is platform-agnostic: every ref action builds an `Action` an
|
|||
| `drag` / drop | no | dragging *is* a cursor press-move-release; no general AX drag. Native cross-app drop needs the OS dragging-session/pasteboard protocol that synthetic events cannot start (works for same-view source-tracked gestures and web/Electron mouse-DnD) |
|
||||
| menu bar (`--surface menubar`) | enumerate/open | the app menu bar is readable and openable; SwiftUI `CommandMenu` items accept AXPress but do not route to their action closure (a SwiftUI limitation, like its Slider) — native AppKit menu items fire. `.contextMenu` item selection works. |
|
||||
|
||||
All ref-based interaction commands accept `--snapshot <snapshot_id>`. Omit it for the active session's latest saved snapshot, or pass the `snapshot_id` returned by `snapshot` to keep scripts pinned to the exact ref map they observed. Explicit snapshot IDs do not require also passing `--session`.
|
||||
All ref-based interaction commands accept `--snapshot <snapshot_id>`. Omit it for the active session's latest saved snapshot, or pass the `snapshot_id` returned by `snapshot` to keep scripts pinned to the exact ref map they observed. Explicit snapshot IDs do not require also passing `--session`. After `session start`, implicit latest resolves inside the new session; snapshots taken before the boundary need explicit `--snapshot <old-id>`.
|
||||
|
||||
Success responses for ref actions include a `steps` array when the activation chain recorded attempts: each entry is `{ "label": "AXPress", "outcome": "attempted" | "skipped" | "succeeded" }` in execution order, showing which activation path produced the result.
|
||||
|
||||
|
|
|
|||
|
|
@ -236,7 +236,9 @@ Execute multiple commands in sequence from a JSON array. Each entry has `command
|
|||
|
||||
Batch uses the same typed `Commands` enum, command policy preflight, permission report, and dispatch path as the CLI. Unknown fields are rejected instead of being silently ignored. Nested `batch` is rejected.
|
||||
|
||||
Each entry may include `"session": "id"` beside `command` and `args`. If omitted, the entry inherits the top-level `--session`. Use per-entry sessions only when intentionally inspecting or coordinating separate agent runs. Top-level `--trace` is inherited by every entry — including entries with a `session` override — so one JSONL file captures the whole batch.
|
||||
Each entry may include `"session": "id"` beside `command` and `args`. If omitted, the entry inherits the top-level resolved session. Use per-entry sessions only when intentionally inspecting or coordinating separate agent runs.
|
||||
|
||||
**Trace in batch:** when the top-level CLI passes `--trace <path>`, every entry writes to that single file (override). Without `--trace`, entries inherit the resolved session's manifest-gated segment sink; a per-entry `"session"` override re-derives the sink for that session (events never land in the parent session's segment). Session subcommands (`session start`, etc.) are also available in batch JSON via `"action": "start"|"end"|"list"|"gc"`.
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
|
|
@ -248,7 +250,8 @@ Each entry may include `"session": "id"` beside `command` and `args`. If omitted
|
|||
{ "command": "click", "args": { "ref_id": "@e1", "snapshot": "<snapshot_id>" } },
|
||||
{ "command": "wait", "args": { "ms": 500 } },
|
||||
{ "command": "type", "args": { "ref_id": "@e2", "snapshot": "<snapshot_id>", "text": "hello" } },
|
||||
{ "command": "status", "session": "other-agent", "args": {} }
|
||||
{ "command": "status", "session": "other-agent", "args": {} },
|
||||
{ "command": "session", "args": { "action": "start", "name": "batch-run" } }
|
||||
]
|
||||
```
|
||||
|
||||
|
|
@ -276,13 +279,59 @@ Each entry may include `"session": "id"` beside `command` and `args`. If omitted
|
|||
|
||||
`skeleton: true` clamps depth to 3 and tags truncated containers with `children_count`. `root: "@eN"` starts traversal from that ref instead of the window root; it cannot be combined with `surface`.
|
||||
|
||||
## Session lifecycle
|
||||
|
||||
Sessions are on-disk containers under `~/.agent-desktop/sessions/<id>/` with a `session.json` manifest, snapshot refmaps, and (when tracing is on) a `trace/` directory. **`session start` is the only command that writes `~/.agent-desktop/current_session`.**
|
||||
|
||||
### session start
|
||||
```bash
|
||||
agent-desktop session start
|
||||
agent-desktop session start --name "nightly-run"
|
||||
agent-desktop session start --no-trace # Namespace only — no automatic JSONL
|
||||
agent-desktop session start --force # Override pointer even if it references a live session
|
||||
```
|
||||
Creates the session directory, pre-creates `trace/` (when tracing is on), writes `session.json` (`trace: on` unless `--no-trace`), sets the current-session pointer, and prints `{ "session_id", "name", "trace", "created_at" }`.
|
||||
|
||||
Refuses to clobber a pointer that still references a **live** session unless `--force`. Live means an active `refstore.lock` holder or recent writes under `trace/`.
|
||||
|
||||
### session end
|
||||
```bash
|
||||
agent-desktop session end
|
||||
agent-desktop session end run-1719763200123-0
|
||||
```
|
||||
Seals the manifest with `ended_at` and clears the pointer when it still points at this session.
|
||||
|
||||
### session list
|
||||
```bash
|
||||
agent-desktop session list
|
||||
```
|
||||
Returns manifest fields only (`session_id`, `name`, `created_at`, `ended_at`, `trace`) — no subtree walk.
|
||||
|
||||
### session gc
|
||||
```bash
|
||||
agent-desktop session gc
|
||||
agent-desktop session gc --ended
|
||||
agent-desktop session gc --older-than 3600
|
||||
```
|
||||
Removes ended sessions that are not live and not pointer-referenced. Never reaps a session with a live lock holder or recent `trace/` activity. Refuses symlinked session directories.
|
||||
|
||||
### Activation (all commands)
|
||||
|
||||
| Source | Precedence |
|
||||
|--------|------------|
|
||||
| `--session <id>` | Highest |
|
||||
| `AGENT_DESKTOP_SESSION` env var | Middle |
|
||||
| `~/.agent-desktop/current_session` | Lowest (set only by `session start`) |
|
||||
|
||||
Trace-on requires a manifest with `trace: on` from `session start`. Bare `--session` or FFI `ad_adapter_create_with_session` without that manifest selects the snapshot namespace only.
|
||||
|
||||
## System Health
|
||||
|
||||
### status
|
||||
```bash
|
||||
agent-desktop status
|
||||
```
|
||||
Returns adapter health, platform info, permission report, and latest snapshot metadata (`snapshot_id`, `ref_count`) when available.
|
||||
Returns adapter health, platform info, permission report, latest snapshot metadata (`snapshot_id`, `ref_count`) when available, plus **`session_id`** (resolved active session, if any) and **`tracing`** (whether structured trace output is configured for this process — explicit `--trace`, or a trace-enabled session manifest).
|
||||
|
||||
### permissions
|
||||
```bash
|
||||
|
|
|
|||
|
|
@ -15,6 +15,31 @@ agent-desktop permissions --request
|
|||
|
||||
For screenshots, also grant Screen Recording. `permissions` reports `accessibility`, `screen_recording`, and `automation` separately.
|
||||
|
||||
## Pattern: Session-Scoped Tracing (Default for Multi-Step Runs)
|
||||
|
||||
Start one session per agent run so tracing and the latest-snapshot namespace follow automatically — no `--trace` on every command.
|
||||
|
||||
```bash
|
||||
# 1. Start once — creates manifest (trace: on), pointer, and trace/ directory
|
||||
agent-desktop session start --name "invoice-bot"
|
||||
# Note session_id from data.session_id (also written to ~/.agent-desktop/current_session)
|
||||
|
||||
# 2. Observe-act loop — segments land under sessions/<id>/trace/<pid>-*.jsonl
|
||||
agent-desktop snapshot --app "Preview" -i --compact
|
||||
agent-desktop click @e3 --snapshot <snapshot_id>
|
||||
agent-desktop status # confirms session_id + tracing: true
|
||||
|
||||
# 3. End and reclaim when finished
|
||||
agent-desktop session end
|
||||
agent-desktop session gc
|
||||
```
|
||||
|
||||
**Concurrent independent agents:** set `AGENT_DESKTOP_SESSION=<id>` in each process instead of sharing the global pointer. Each agent still uses the `snapshot_id` from its own `snapshot` call when sharing a session id.
|
||||
|
||||
**Namespace without tracing:** `session start --no-trace` or bare `--session legacy-id` (no manifest) — snapshots namespaced, no JSONL files.
|
||||
|
||||
**Override file trace:** `--trace /tmp/run.jsonl` still forces a single file regardless of session manifest.
|
||||
|
||||
## Pattern: Progressive Skeleton Traversal (Default for Dense Apps)
|
||||
|
||||
The recommended approach for Electron apps (Slack, VS Code, Discord) and any app with 50+ interactive elements. Reduces token consumption 78-96%.
|
||||
|
|
|
|||
Loading…
Reference in a new issue