mirror of
https://github.com/lahfir/agent-desktop.git
synced 2026-08-06 06:00:39 +00:00
32175e44c5
* fix: return observed trees, stop demanding activation from shallow walks Six defects found by measuring the observation and action paths against real applications. Each was code computing something correct and then discarding it. Snapshot discarded a fully-observed tree when its budget expired. Finder observed 247 nodes, threw away all of them, and returned TIMEOUT with zero refs. It now returns what it observed with `complete: false`, and every node whose descendants were cut carries `subtree_truncated`, which propagates to its ancestors so a reader can walk from the root to the boundary. Only a full snapshot may be partial: a drill-down replaces refs inside an existing map, so it still requires a complete observation rather than destroying descendants it may not be able to re-allocate. `kAXErrorIllegalArgument` was classified as retryable. It is the window bridge rejecting an element outright, which no amount of retrying will change, so strict resolution burned its full budget on a call that could not succeed and then blamed the application. Finder refs went from 0 of 9 resolving at ~814ms to 6 of 7 at ~120ms. A depth-clamped observation was treated as evidence that a renderer had no accessibility surface. A shallow walk stops above the web content by design, so every skeleton snapshot of a Chromium application demanded an activation it did not need and re-walked the tree until the deadline expired. Slack `--skeleton` went from failing at 3.4s to 0.2s, and depth now scales monotonically instead of shallower being slower than deeper. The retry that follows a genuine activation also backs off, because each attempt costs a full tree walk: a 3s budget spent about 120 of them and now spends 11. A boundary node is read for its child count alone. On a renderer that materialises children lazily that count can cost more than the traversal it describes, so it is now best-effort; a boundary that cannot afford one is still reported as truncated, just without a number. Refmap retention kept 512 snapshots and swept every one of them for orphaned temporary files on every save. Retention is now 128 with eviction to 96 so the sort-and-stat pass is amortised, the per-save sweep covers only directories that save could have written, and the exhaustive sweep runs with eviction. `session end` drops the ref scaffolding it accumulated, but only under `ArtifactsMode::Full` where the trace keeps its own copy; the default mode never copies them, and discarding them there would sever snapshot resolution for anyone reading the trace after. `is --property` read element bounds for properties that never use them. BREAKING CHANGE: ENVELOPE_VERSION is now 2.2. `data.complete` is present on every successful snapshot, and a snapshot that exhausts its budget returns `ok: true` with `complete: false` where it previously returned a TIMEOUT error. Callers that branched on TIMEOUT to detect an oversized tree must read `complete` instead. * docs: correct solution docs that contradicted the code and each other A refresh pass over docs/solutions/ against the current tree. Three of the four corrections were internal contradictions that reading the doc alone could not reveal. The pointer-action doc cited `resolve_point_with_deadline`, which has no matches in the repository; hover and drag resolve in two phases, before and under the interaction lease. It also now distinguishes the two hit-tests that exist, since the shared battery's multi-candidate check has grown to cover the click family while the pointer pipeline keeps its own single-point check. The drag-abort doc claimed the release guard "arms only after mouse-down is posted". It arms before, which is what the doc's own prevention rule requires and what the code does. The progressive-snapshot contract described one truncation path. There are two, and the second was silently dropping descendants with no marker at all, violating that doc's own rule. CONCEPTS.md gains Interaction Lease, Partial Observation and Delivery Semantics, and corrects Interaction Policy, which claimed ref commands expose exactly two modes when a third is the base policy of an explicit key press and is directly selectable by language bindings. * fix: only discard refmaps the trace actually copied Being in ArtifactsMode::Full was treated as proof that every refmap had been duplicated into the trace, so ending a session deleted the whole snapshot store. It is not proof. The artifact byte budget rejects a copy once a session's refmaps exceed it, and a serialisation failure skips one too; both report success to the caller. A long full-artifacts session therefore holds snapshots whose only refmap lives in the store, and ending it destroyed them permanently, severing snapshot resolution for anyone reading that trace afterwards. Each snapshot directory is now removed only against its own duplicate in the trace, and the latest-snapshot pointer survives unless the snapshot it names is gone. The existing full-artifacts test encoded the defect: it seeded a snapshot with no trace copy and asserted the store was emptied. It now seeds the copy it claims exists, and a new test covers the case that was losing data — a refmap the trace never copied must survive.
132 lines
12 KiB
Markdown
132 lines
12 KiB
Markdown
# Concepts
|
|
|
|
Shared domain vocabulary for this project -- entities, named processes, and status concepts with project-specific meaning. Seeded with core domain vocabulary, then accretes as ce-compound and ce-compound-refresh process learnings; direct edits are fine. Glossary only, not a spec or catch-all.
|
|
|
|
## Desktop Observation
|
|
|
|
### Accessibility Tree
|
|
A structured representation of an application's user interface exposed by the operating system accessibility APIs and used by agent-desktop as the source of truth for observation and semantic interaction.
|
|
|
|
### Snapshot
|
|
An observation of an accessibility tree at a point in time, persisted with the element refs allocated from that observation.
|
|
|
|
### Snapshot ID
|
|
A compact identifier for one persisted snapshot. Lookup is confined to the selected session namespace, so an ID created in a session is not a cross-session handle.
|
|
|
|
### Surface
|
|
A scoped UI layer that can be observed separately from the whole window, such as an open menu, sheet, popover, alert, or focused area.
|
|
|
|
### Drill-down
|
|
A snapshot operation that starts from an existing ref to observe that element's subtree instead of re-reading the entire window.
|
|
|
|
### Partial Observation
|
|
A snapshot that ran out of its allotted time before finishing the tree and returns what it did observe rather than discarding the walk.
|
|
|
|
Completeness is reported on the observation as a whole and on each node whose descendants were cut, so a reader can walk from the root to the boundary. A depth clamp knows how many children it skipped and says so; budget exhaustion cannot afford that count and marks the node without one. Only a full snapshot may be partial — a drill-down replaces refs inside an existing map, so it requires a complete observation and fails rather than destroying descendants it cannot re-allocate.
|
|
|
|
## Refs And Identity
|
|
|
|
### Ref
|
|
A short element identifier assigned by agent-desktop to an actionable or drillable node in a snapshot.
|
|
|
|
Refs are deterministic inside one snapshot but are not stable across UI changes. Snapshot and find output qualify each ref with its snapshot ID. Legacy bare refs require the producing snapshot ID as a separate argument.
|
|
|
|
### RefMap
|
|
The persisted mapping from refs to the identity evidence needed to re-identify elements later.
|
|
|
|
### Stable Text Identity
|
|
The role-conditional text evidence used during strict ref resolution.
|
|
|
|
Names and descriptions can identify a ref when they are stable labels. Mutable control values, including text field content and value text promoted into an accessibility name, are volatile and do not identify the element by themselves. Core owns this policy so macOS, Windows, Linux, CLI, and FFI consumers share the same semantics.
|
|
|
|
### Stale Ref
|
|
A ref whose stored identity no longer matches a live element strongly enough to act safely.
|
|
|
|
### Strict Ref Resolution
|
|
The fail-closed process of re-identifying a ref from stored identity evidence before a command acts on it.
|
|
|
|
Strict ref resolution rejects missing, stale, and ambiguous matches instead of guessing. It is the boundary between an old observation and a live desktop mutation.
|
|
|
|
## Coordination
|
|
|
|
### Session
|
|
An on-disk container under `~/.agent-desktop/sessions/<id>/` that owns snapshot refmaps, an optional trace directory, and a `session.json` manifest.
|
|
|
|
`session start` writes the manifest (`trace: on` unless `--no-trace`) and pre-creates `trace/` when tracing is on. It returns the new ID but does not activate it for later processes. Explicit `--session` takes precedence over `AGENT_DESKTOP_SESSION`; with neither, commands use the global, non-session namespace. Bare `--session <id>` without a manifest remains snapshot-namespace-only for backward compatibility.
|
|
|
|
Use sessions when callers want a coordinated snapshot namespace and trace sink. Every lookup is confined to its selected namespace, so a snapshot created under a session requires that same `--session` or `AGENT_DESKTOP_SESSION` scope later. Qualified refs remain the deterministic path for pinned actions inside that namespace.
|
|
|
|
### Session Manifest
|
|
The `session.json` file describing one session: id, optional name, created/ended timestamps, and `trace: on|off`.
|
|
|
|
Structured file tracing activates only when the manifest has `trace: on`. FFI adapters and bare `--session` ids without this manifest do not write trace segments.
|
|
|
|
### Trace Segment
|
|
One append-only JSONL file per OS process under `<session>/trace/<pid>-<procStartTs>.jsonl`, written lazily with atomic lines. Each new segment opens with a `trace.meta` header (`schema`, binary version, `os`, `pid`, `proc_start_ms`, `session_id`). Older traces without meta read as schema 0. Explicit `--trace <path>` overrides to a single file.
|
|
|
|
### Trace Timeline
|
|
The merged, deterministic ordering of all events from every segment in a session, produced by `trace show` and `trace export`. Merge key is `(ts_ms, writer pid, in-file position)`; the reader tolerates truncated tails, corrupt lines, and foreign files with counted warnings rather than hard errors.
|
|
|
|
### Trace Schema
|
|
Additive-only evolution contract: new event types and optional fields may appear; existing meanings never change. Readers ignore unknown content. Segments declare their schema in the leading `trace.meta` line; unknown future schemas warn and parse best-effort.
|
|
|
|
### Replay Artifacts
|
|
Opt-in capture mode (`session start --screenshots`, manifest `artifacts: full`) that stores pre/post-action PNGs under `<session>/trace/screens/` and refmap copies under `<session>/trace/refmaps/`. Event-mode traces (`artifacts: events`, the default) record JSONL only. Artifacts are unredacted and may appear in exported HTML — treat them like screenshots.
|
|
|
|
### Protected Process
|
|
A session-critical operating-system process that agent-desktop refuses to close on every surface, because terminating it would break the user's desktop session.
|
|
|
|
The refusal is enforced where the close happens, so CLI, FFI, and any future consumer behave identically. Matching is exact — a process name or a bundle-identifier component, never a substring — so lookalike applications that merely contain a protected name stay closable.
|
|
|
|
## Action Reliability
|
|
|
|
### Actionability
|
|
The pre-dispatch judgement that a resolved element is safe to act on, based on native evidence such as visibility, stability, enabled state, supported action, policy, and editability.
|
|
|
|
### Delivery Semantics
|
|
What a failed or uncertain action says about whether input actually reached the application, and therefore whether repeating it is safe.
|
|
|
|
The distinction that matters is not success versus failure but delivered versus not: an action that never reached the target can be retried freely, while one that may have landed cannot be repeated without risking a duplicate. Verification is a third axis — an action can be known-delivered yet unverified, meaning the input was posted but its effect was not confirmed. Errors carry this alongside the recovery hint so a caller never has to infer retry safety from an error code.
|
|
|
|
### Interaction Lease
|
|
Machine-wide exclusivity over desktop input, held by one process at a time so concurrent callers cannot interleave synthetic input into each other's actions.
|
|
|
|
The lease covers dispatch only, never the waiting that precedes it: waiting for an element to become actionable can run long, and holding exclusivity across it would serialize every caller on the machine. Anything resolved while waiting was therefore observed without exclusivity and is re-resolved once the lease is held. That second resolution is the correctness boundary, not redundant work.
|
|
|
|
### Capability Vocabulary
|
|
The platform-neutral set of supported action names that core uses to compare command intent with native adapter evidence.
|
|
|
|
Each adapter maps native primitives into this shared vocabulary before core evaluates actionability. New commands should extend the central vocabulary first, then reuse it from actionability, ref allocation, predicates, FFI tests, and platform adapters.
|
|
|
|
### Interaction Policy
|
|
The side-effect contract attached to an action request, controlling whether the command may steal focus, move the cursor, or use physical input. The CLI exposes two: **headless** (the default — accessibility-only, no cursor, fails closed when the semantic path is unavailable) and **headed** (opt-in via the global `--headed` flag — authorizes the action's declared focus and cursor preconditions). A third, **focus fallback**, sits between them: it permits focus but not cursor movement. It is not reachable from the CLI flag — it is the base policy of an explicit key press, and language bindings may select it directly.
|
|
|
|
Core owns those preconditions through `HeadedRequirement`: `FocusedWindow` for keyboard or focus-sensitive work, and `FocusedWindowAndCursor` for pointer delivery. For ref actions, core focuses the exact source window before dispatch and resolves a verified target point for pointer work; the platform adapter owns the OS-specific focus primitive and delivery mechanism. Raw coordinate input has no ref identity, so it never infers or focuses a window. On macOS, headed `click`, `right-click`, `type`, `clear`, and `scroll` are physical-first; double/triple-click, hover, and drag are physical-only; expand/collapse and the remaining semantic actions stay semantic after the core focus precondition.
|
|
|
|
### Headless Ref Action
|
|
A ref-based action that uses semantic accessibility operations without implicit focus stealing, cursor movement, synthetic keyboard input, or pasteboard use. This is the default mode.
|
|
|
|
Headless ref actions may still fail when the native accessibility API cannot perform the requested semantic operation; they fail closed with structured actionability or policy errors rather than silently substituting physical input. The broader **headed** policy must be selected explicitly with `--headed`.
|
|
|
|
### Action Chain
|
|
The ordered ladder of strategies a ref action walks to perform one intent, with each step gated by policy and its delivery evidence recorded. The order is action-specific: natural input may put a headed physical step first, while semantic state changes use accessibility actions or settable attributes only.
|
|
|
|
The chain pins one execution deadline at its start (distinct from the Resolver Deadline, which budgets re-identification) and every step observes it. Expiry while a step may have partially mutated the element surfaces as a structured timeout carrying the observed state, never as a plain step failure — the caller must be able to tell "nothing happened" from "something may have happened".
|
|
|
|
### Wait Predicate
|
|
The condition a wait command polls for before returning, such as element actionability, text presence, window appearance, menu state, or notification arrival.
|
|
|
|
### Resolver Deadline
|
|
The remaining time budget carried through strict ref resolution so every native read can fail with a structured timeout instead of using an unrelated platform default timeout.
|
|
|
|
### Coordinate Fallback
|
|
An explicit opt-in path that uses screen coordinates or physical input when semantic accessibility operations cannot perform the requested action.
|
|
|
|
Ref-targeted physical input lands on the topmost window at the resolved point, so core first ensures the target element's exact window is frontmost — the app being frontmost is not sufficient when the element lives in a background window of that app. Raw `--xy` input carries no window identity and therefore moves/clicks at the requested coordinates without focusing any application.
|
|
|
|
### FFI Ref-Action Parity
|
|
The requirement that language bindings using refs follow the same strict resolution, actionability, and interaction-policy semantics as CLI ref commands.
|
|
|
|
## Relationships
|
|
|
|
A session owns one latest-snapshot pointer, an optional manifest-gated trace directory, and persisted snapshot refmaps. A snapshot persists a ref map and can be selected by ID within that same namespace. A ref resolves through strict ref resolution into live native evidence, then actionability decides whether the action can safely dispatch. In headed mode, core applies the action's focus/cursor requirement before the platform adapter executes the action-specific chain under its own deadline. FFI ref-action parity keeps that same relationship true for language bindings.
|