3.9 KiB
Concepts
Shared domain vocabulary for this project -- entities, named processes, and status concepts with project-specific meaning. Seeded with core domain vocabulary, then accretes as ce-compound and ce-compound-refresh process learnings; direct edits are fine. Glossary only, not a spec or catch-all.
Desktop Observation
Accessibility Tree
A structured representation of an application's user interface exposed by the operating system accessibility APIs and used by agent-desktop as the source of truth for observation and semantic interaction.
Snapshot
An observation of an accessibility tree at a point in time, persisted with the element refs allocated from that observation.
Snapshot ID
A compact identifier for one persisted snapshot inside a session.
Surface
A scoped UI layer that can be observed separately from the whole window, such as an open menu, sheet, popover, alert, or focused area.
Drill-down
A snapshot operation that starts from an existing ref to observe that element's subtree instead of re-reading the entire window.
Refs And Identity
Ref
A short element identifier assigned by agent-desktop to an actionable or drillable node in a snapshot.
Refs are deterministic inside one snapshot but are not stable across UI changes. Callers either pass the snapshot ID that produced the ref or intentionally use the session's latest snapshot pointer.
RefMap
The persisted mapping from refs to the identity evidence needed to re-identify elements later.
Stale Ref
A ref whose stored identity no longer matches a live element strongly enough to act safely.
Strict Ref Resolution
The fail-closed process of re-identifying a ref from stored identity evidence before a command acts on it.
Strict ref resolution rejects missing, stale, and ambiguous matches instead of guessing. It is the boundary between an old observation and a live desktop mutation.
Coordination
Session
A namespace for snapshots, ref maps, and the latest-snapshot pointer shared by one agent or a coordinated group of agents.
A session can contain many snapshots. The latest-snapshot pointer is a convenience for fluid workflows, not a replacement for explicit snapshot IDs when deterministic replay matters.
Action Reliability
Actionability
The pre-dispatch judgement that a resolved element is safe to act on, based on native evidence such as visibility, stability, enabled state, supported action, policy, and editability.
Interaction Policy
The side-effect contract attached to an action request, controlling whether the command may steal focus, move the cursor, or use physical input fallbacks.
Headless Ref Action
A ref-based action that uses semantic accessibility operations without implicit focus stealing, cursor movement, synthetic keyboard input, or pasteboard use.
Headless ref actions may still fail when the native accessibility API cannot perform the requested semantic operation. A broader interaction policy must be explicit rather than silently substituting physical input.
Wait Predicate
The condition a wait command polls for before returning, such as element actionability, text presence, window appearance, menu state, or notification arrival.
Coordinate Fallback
An explicit opt-in path that uses screen coordinates or physical input when semantic accessibility operations cannot perform the requested action.
FFI Ref-Action Parity
The requirement that language bindings using refs follow the same strict resolution, actionability, and interaction-policy semantics as CLI ref commands.
Relationships
A session contains many snapshots and owns one latest-snapshot pointer. A snapshot persists a ref map. A ref resolves through strict ref resolution into live native evidence, then actionability decides whether a headless ref action can safely dispatch. FFI ref-action parity keeps that same relationship true for language bindings.