mirror of
https://github.com/lahfir/agent-desktop.git
synced 2026-08-03 20:56:01 +00:00
6 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Lahfir
|
a2712f18a5 |
chore: merge main into the windows adapter branch
Brings 0.7.0 and the observation and action fixes (#117) onto the Windows integration branch so the promotion PR is no longer conflicting. One conflict, in CONCEPTS.md, and it was additive on both sides: main added Partial Observation, this branch added the Vocabulary section. Both are kept — Partial Observation belongs to Desktop Observation, beside Drill-down, which is where it now sits, and Vocabulary follows it. |
||
|
Lahfir
|
32175e44c5
|
fix: return observed trees and stop demanding renderer activation from shallow walks (#117)
* fix: return observed trees, stop demanding activation from shallow walks Six defects found by measuring the observation and action paths against real applications. Each was code computing something correct and then discarding it. Snapshot discarded a fully-observed tree when its budget expired. Finder observed 247 nodes, threw away all of them, and returned TIMEOUT with zero refs. It now returns what it observed with `complete: false`, and every node whose descendants were cut carries `subtree_truncated`, which propagates to its ancestors so a reader can walk from the root to the boundary. Only a full snapshot may be partial: a drill-down replaces refs inside an existing map, so it still requires a complete observation rather than destroying descendants it may not be able to re-allocate. `kAXErrorIllegalArgument` was classified as retryable. It is the window bridge rejecting an element outright, which no amount of retrying will change, so strict resolution burned its full budget on a call that could not succeed and then blamed the application. Finder refs went from 0 of 9 resolving at ~814ms to 6 of 7 at ~120ms. A depth-clamped observation was treated as evidence that a renderer had no accessibility surface. A shallow walk stops above the web content by design, so every skeleton snapshot of a Chromium application demanded an activation it did not need and re-walked the tree until the deadline expired. Slack `--skeleton` went from failing at 3.4s to 0.2s, and depth now scales monotonically instead of shallower being slower than deeper. The retry that follows a genuine activation also backs off, because each attempt costs a full tree walk: a 3s budget spent about 120 of them and now spends 11. A boundary node is read for its child count alone. On a renderer that materialises children lazily that count can cost more than the traversal it describes, so it is now best-effort; a boundary that cannot afford one is still reported as truncated, just without a number. Refmap retention kept 512 snapshots and swept every one of them for orphaned temporary files on every save. Retention is now 128 with eviction to 96 so the sort-and-stat pass is amortised, the per-save sweep covers only directories that save could have written, and the exhaustive sweep runs with eviction. `session end` drops the ref scaffolding it accumulated, but only under `ArtifactsMode::Full` where the trace keeps its own copy; the default mode never copies them, and discarding them there would sever snapshot resolution for anyone reading the trace after. `is --property` read element bounds for properties that never use them. BREAKING CHANGE: ENVELOPE_VERSION is now 2.2. `data.complete` is present on every successful snapshot, and a snapshot that exhausts its budget returns `ok: true` with `complete: false` where it previously returned a TIMEOUT error. Callers that branched on TIMEOUT to detect an oversized tree must read `complete` instead. * docs: correct solution docs that contradicted the code and each other A refresh pass over docs/solutions/ against the current tree. Three of the four corrections were internal contradictions that reading the doc alone could not reveal. The pointer-action doc cited `resolve_point_with_deadline`, which has no matches in the repository; hover and drag resolve in two phases, before and under the interaction lease. It also now distinguishes the two hit-tests that exist, since the shared battery's multi-candidate check has grown to cover the click family while the pointer pipeline keeps its own single-point check. The drag-abort doc claimed the release guard "arms only after mouse-down is posted". It arms before, which is what the doc's own prevention rule requires and what the code does. The progressive-snapshot contract described one truncation path. There are two, and the second was silently dropping descendants with no marker at all, violating that doc's own rule. CONCEPTS.md gains Interaction Lease, Partial Observation and Delivery Semantics, and corrects Interaction Policy, which claimed ref commands expose exactly two modes when a third is the base policy of an explicit key press and is directly selectable by language bindings. * fix: only discard refmaps the trace actually copied Being in ArtifactsMode::Full was treated as proof that every refmap had been duplicated into the trace, so ending a session deleted the whole snapshot store. It is not proof. The artifact byte budget rejects a copy once a session's refmaps exceed it, and a serialisation failure skips one too; both report success to the caller. A long full-artifacts session therefore holds snapshots whose only refmap lives in the store, and ending it destroyed them permanently, severing snapshot resolution for anyone reading that trace afterwards. Each snapshot directory is now removed only against its own duplicate in the trace, and the latest-snapshot pointer survives unless the snapshot it names is gone. The existing full-artifacts test encoded the defect: it seeded a snapshot with no trace copy and asserted the store was emptied. It now seeds the copy it claims exists, and a new test covers the case that was losing data — a refmap the trace never copied must survive. |
||
|
Lahfir
|
8f24f04f5a
|
feat: windows vocabulary — roles, states, native_id and name evidence (#115)
Gives the Windows UIA tree its vocabulary: a ControlType→Role map with no catch-all, available actions, a gated state vocabulary, native_id from AutomationId, and name evidence resolved through core's single shared precedence. Core is touched exactly twice; macOS output is byte-identical. Pattern state is read as plain batched properties, each gated on its own Is*PatternAvailable flag — a provider returns a plausible default for a pattern it never implemented, so an ungated read decorates every inert node. invalid and pressed are deliberately unproduced on Windows: the first has no source that is a positive claim, the second no reachable role precondition. Evidence is tri-state throughout. Absent is an answer and Unknown is the lack of one, and the two are never collapsed — a failed read withholds a role, state or affordance rather than granting it. Includes the probe corpus and findings ledger behind these decisions, a census tool that reports shapes and counts without ever serializing a real application's text, a dogfood run against four real UI stacks, and a gate that fails the build when shipped source references the delivery plan. |
||
|
Lahfir
|
3f322728b4
|
feat!: implement Playwright-grade foundation contract
Settle the Playwright-grade reliability contract in agent-desktop-core before the Windows/Linux adapters are built, so they inherit it instead of redesigning it. Every command now observes, waits, verifies, and reports honestly instead of firing blindly. Highlights: capability-supertrait split of PlatformAdapter with not_supported() defaults; canonical role/state vocabulary with live `is --property visible`; display enumeration (`list-displays`) and honest `--screen` with scale factor; truthful Automation permission; `native_id` identity spine; window-id-first resolution; serializable `LocatorQuery` with live `find`; default-on auto-wait before every ref action; three-way `hit_test` occlusion gate; `scroll_into_view` in core; core accessible-name precedence; typed `ActionStep` delivery tier; `ProcessState` and `APP_UNRESPONSIVE`; `LaunchOptions`; baseline-diff desktop signals (`wait --event`); typed clipboard (`Text`/`Image`/`FileUrls`); mouse modifier chords and `mouse-wheel`. Hardened through a 35-reviewer pass with independent validation and a green live e2e gate (109/0), plus a head-vs-main performance comparison harness. BREAKING CHANGE: default-on auto-wait changes the timing of every previously-untouched ref-action call (bounded 5000 ms default; `--timeout-ms 0` restores single-shot). `ENVELOPE_VERSION` is now `2.1` (adds the `APP_UNRESPONSIVE` code and process state in error details). FFI ABI major is `3` (append-only struct evolution; `wait --event` is intentionally not exposed over FFI). The legacy string clipboard API is removed in favor of typed content. `key-down`/`key-up` fail closed until daemon-owned held input exists. `close-app` verifies termination and the osascript fallback path is removed. `--text` matching is subtree containment: `find --text X --first` returns the outermost matching container. |
||
|
Lahfir
|
1291a9cdbf
|
refactor!: unify command execution contracts
Unify CLI and batch dispatch around the typed command path, centralize command policy and ref resolution, harden macOS action verification, split command tests from implementation, and add package/release guardrails. BREAKING CHANGE: CLI and batch execution now share the typed command path and current command argument contracts. BREAKING CHANGE: Ref-consuming commands use snapshot-scoped refs; deterministic consumers should pass snapshot_id and handle SNAPSHOT_NOT_FOUND. BREAKING CHANGE: permissions and status now return PermissionReport fields for accessibility, screen_recording, and automation instead of a single boolean status. BREAKING CHANGE: PermissionState gains NotRequired; macOS automation now reports not_required instead of unknown. BREAKING CHANGE: right-click now separates action success from menu verification; consumers should inspect menu or menu_probe instead of assuming every right-click returns an inline menu. BREAKING CHANGE: focus-window now confirms OS focus and returns ACTION_FAILED when focus does not settle; data.focused.is_focused is true on success. BREAKING CHANGE: PlatformAdapter::execute_action now takes ActionRequest, and permission probing uses permission_report/request_permissions. BREAKING CHANGE: FFI ad_execute_action now defaults to headless policy. Consumers that need focus fallback or cursor-moving behavior must call ad_execute_action_with_policy with AD_POLICY_KIND_FOCUS_FALLBACK or AD_POLICY_KIND_PHYSICAL. BREAKING CHANGE: FFI ad_check_permissions no longer treats unknown accessibility permission as success; stub-style unknown probes return ERR_PLATFORM_NOT_SUPPORTED and macOS ambiguous unknown returns ERR_INTERNAL with last-error detail. BREAKING CHANGE: JSON response envelopes now report version 2.0; parsers pinned to 1.0 must branch or update. BREAKING CHANGE: focus now uses accessibility focus without cursor movement; callers that need physical focus must use explicit mouse or physical-policy paths. BREAKING CHANGE: chain execution deadlines now return TIMEOUT instead of ACTION_FAILED when the target app does not respond before the chain deadline. |
||
|
Lahfir
|
c17f2fae7a
|
feat: progressive skeleton traversal with ref-rooted drill-down (#20)
* feat: add data model foundation for progressive skeleton traversal
Add children_count, root_ref, skeleton fields to core types. Add
get_subtree() to PlatformAdapter trait. Add remove_by_root_ref() and
write-side size check to RefMap. No behavioral changes yet.
* feat: implement progressive skeleton traversal with ref-rooted drill-down
Add --skeleton and --root flags to snapshot command for token-efficient
accessibility tree exploration. Skeleton mode clamps depth to 3 levels
and annotates truncated containers with children_count, allowing AI
agents to discover regions before drilling into them. Named containers
at skeleton boundaries (via name or description) receive refs as
drill-down targets. The --root flag starts traversal from a previous
ref with scoped invalidation — only refs from that drill-down are
replaced on re-drill.
Key changes:
- New ref_alloc.rs: shared ref helpers (INTERACTIVE_ROLES, actions_for_role,
ref_entry_from_node, is_collapsible) extracted from snapshot.rs
- New snapshot_ref.rs: drill-down logic with DrillDownConfig, scoped
invalidation via root_ref tagging on RefEntry
- macOS count_children() uses raw CFArrayGetCount without materializing
AXElement wrappers for performance at skeleton boundaries
- RefMap write-side size check prevents >1MB files
- Skeleton anchors consider both name and description for Electron compat
* fix: mention --skeleton in STALE_REF error suggestion
* docs: document --skeleton and --root flags in skill reference
* docs: update phases.md and CLAUDE.md for progressive skeleton traversal
Add skeleton traversal as Phase 1 objective P1-O10. Document --skeleton
and --root flags, get_subtree() trait method, new core modules, and
platform-agnostic notes for Phase 2/3. Update risk mitigations and
performance optimizations table.
* docs: make progressive skeleton traversal the default agent workflow
Update SKILL.md observe-act loop to skeleton-first approach. Add
progressive skeleton traversal as the primary workflow pattern. Update
anti-patterns, key principles, and command quick reference to lead
with --skeleton + --root. Full snapshot remains documented as fallback
for simple apps.
* fix: preserve skeleton drill-down anchors
* fix: preserve drill-down refs across skeleton re-snapshots
Skeleton snapshots now load the existing refmap and remove only
skeleton-level refs (root_ref: None), preserving drill-down refs
accumulated via --root. Previously, build() always created a fresh
RefMap, breaking the skeleton → drill → act → skeleton(verify) workflow
by wiping all drill-down refs on the verify step.
* fix: preserve drill-down depth for root snapshots
* fix: verify AX action effect on Electron elements before trusting success
Chromium's AX implementation returns kAXErrorSuccess for AXPress,
AXConfirm, etc. but only toggles ARIA state without firing DOM event
handlers. This caused the click chain to short-circuit on false
positives, preventing CGClick from ever being reached.
The fix detects web elements via AXWebArea ancestor walk, then verifies
each AX action actually had a DOM effect by comparing focused element
pointers before and after. When all AX methods produce no real effect,
CGClick fires as the genuine last resort.
Native elements are unaffected — the original verified_press behavior
is preserved in a separate code path.
* chore: ignore .context/ for local tooling artifacts
* style: collapse web_action_had_effect signature to single line
* test: cover RefMap save oversize rejection
Extract the serialize+size-check step into a private helper so the 1MB
write rejection path can be unit-tested without filesystem I/O. Also
moves the size check above the directory creation in save() so an
oversized refmap no longer creates ~/.agent-desktop on rejection.
* test: assert stale-ref suggestion mentions --skeleton
Locks in the Phase 4 polish requirement that STALE_REF errors guide
agents back to a skeleton refresh, not just a plain snapshot.
* test: cover skeleton-to-drill-down counter continuity
Asserts that skeleton refs (@e1..@e10) survive a scoped invalidation
of @e3, that drill-down refs allocated after the skeleton continue from
@e11 instead of resetting, and that remove_by_root_ref drops only the
drill-down children.
* test: cover --root + --surface rejection at execute() boundary
Adds a NoopAdapter that uses every PlatformAdapter trait default to
exercise execute()'s validation guards without standing up a real
adapter. Verifies that combining --root with a non-Window surface
returns INVALID_ARGS, and that the Window surface does not trigger the
guard.
* test: add filesystem-redirected integration tests for run_from_ref
Adds a thread-local HOME override + RAII HomeGuard so RefMap::save and
RefMap::load can be exercised against an isolated temp directory without
env-var racing across parallel tests. Also adds a StubAdapter that
implements PlatformAdapter with a canned subtree response so the full
run_from_ref drill-down flow can be unit-tested without standing up
macOS AX state.
Closes seven Phase 3 plan acceptance items in one pass:
- save+load roundtrip with HOME override
- oversize save rejection preserves previous file on disk
- run_from_ref returns subtree and persists drill refs
- stale root ref → STALE_REF with skeleton suggestion
- re-drill replaces drill refs only, counter continues
- multiple drill-downs from @e1 and @e2 coexist
- empty subtree drill-down produces no new refs
* test: add golden fixtures for skeleton output and drill-down refmap
Adds two committed JSON fixtures under tests/fixtures/ plus matching
unit tests that build the same input trees, run them through
allocate_refs / run_from_ref, and assert the produced ref ids,
parent-child layout, and root_ref tagging match the golden expectations.
Locks in the JSON shape so future serialization changes have to be
deliberate.
* fix: bound build_subtree recursion on Electron wrapper chains
Anonymous AXGroup/AXGenericElement wrappers do not advance the semantic
depth counter (web-wrapper depth-skip), so a long chain of nested
wrappers could recurse arbitrarily deep without ever hitting either
max_depth or ABSOLUTE_MAX_DEPTH. The previous code checked
`depth >= ABSOLUTE_MAX_DEPTH` against the semantic depth, which stayed
at 0 throughout a wrapper chain. On pathological Electron trees this
risked stack exhaustion before any cap engaged.
Add a separate `raw_depth` parameter that always increments and use it
for the absolute cap. Semantic `depth` still drives `max_depth` and the
skeleton boundary so the wrapper-flattening behavior is preserved on
normal trees.
Also drops the long-unused `_include_bounds` parameter; bounds
filtering happens in `allocate_refs` in core, not here.
* chore: add pre-commit hook running fmt + clippy + tests
Mirrors the CI quality gates (cargo fmt --check, cargo clippy
--all-targets -- -D warnings, cargo test --lib --workspace) so a
failing commit never reaches origin. Skips automatically when no
Rust/TOML files are staged. Bypass with --no-verify or SKIP_PRECOMMIT=1
when a non-Rust hotfix is genuinely needed.
Hook lives under .githooks/ and is opt-in per clone:
git config core.hooksPath .githooks
Setup instructions added to CLAUDE.md.
* fix: prevent orphaned drill-down refs leaking across skeleton refresh
remove_skeleton_refs() previously dropped every entry whose root_ref
was None and kept every entry whose root_ref was Some. After a series
of skeleton refreshes that pattern leaked drill-down refs whose root
anchor had already been removed: they could never be cleaned up by a
future remove_by_root_ref(target) because target referred to a
non-existent anchor, and they accumulated until the 1MB write guard
fired.
Restructure the function to keep only:
- skeleton anchors that have at least one drill-down pointing at them
- drill-down refs whose root anchor is among the kept anchors
Unreferenced anchors are still dropped (the original intent), and
orphaned drill-downs are dropped at the same time so the refmap can
no longer accumulate dead entries.
Updated existing test to assert the new keep-when-referenced semantics
and added a regression test for the orphan-drilldown leak path.
* fix: align resolve traversal with snapshot child-attribute set
find_element_recursive previously walked AXChildren first then fell
back to AXContents only, and resolve_element_name only derived its
fallback label from AXChildren. Snapshot building, however, uses the
full child_attributes() list (AXChildren, AXContents,
AXChildrenInNavigationOrder) via copy_children. That asymmetry meant
refs minted for containers exposed only through AXChildrenInNavigationOrder
appeared in the snapshot output but produced false STALE_REF errors on
drill-down or action commands because the resolver could not find them
again.
Route both call sites through child_attributes() so resolution sees
the same children as snapshotting.
* fix: bypass AXConfirm on web elements and add skeleton anchors to drill-down
* fix: compare AX elements via CFEqual in web_action_had_effect
AXUIElementCopyAttributeValue follows the CoreFoundation Create rule
and returns a freshly-allocated CF object on every call, so raw
pointer equality (`before.0 != after.0`) between two separate copies
of AXFocusedUIElement was ALWAYS true even when the focused element
was unchanged. That made Step 1 (AXPress) of activate_web_element
appear successful every time and left Steps 2-4 of the escalation
chain (AXConfirm bypass, child actions, CGClick fallback) dead code
on web elements.
Replace the pointer comparison with CFEqual, which compares
accessibility element identity rather than heap addresses.
* refactor: unify allocate_refs across snapshot and drill-down paths
allocate_refs in snapshot.rs and allocate_refs_with_root in
snapshot_ref.rs were near-exact copies that differed only in whether
each allocated ref carried a root_ref tag. The duplication meant any
bug fix or behavior change in one path risked silently drifting from
the other (bot flagged this after the first round of fixes added even
more duplicated skeleton-anchor logic to allocate_refs_with_root).
Move the shared logic into ref_alloc.rs behind a new RefAllocConfig
struct with Option<&str> for the root_ref_id, and delete the
snapshot_ref.rs copy. Both snapshot::build and snapshot_ref::run_from_ref
now route through the same function with their respective configs.
append_surface_refs and the existing unit tests in both modules are
updated to use the shared function + config.
* chore: drop with_root from drill test names after allocator unification
The snapshot_ref tests kept their original test_allocate_refs_with_root_*
names after the allocator dedupe even though allocate_refs_with_root no
longer exists. Rename them to test_drill_alloc_* so grep for
allocate_refs_with_root returns zero matches and nobody assumes a second
allocator path exists.
* docs: compound DRY ref-allocator dedupe into knowledge base
Adds docs/solutions/best-practices/deduplicate-ref-allocator-via-config-struct-2026-04-14.md
documenting the root cause, guidance, consequences, trigger conditions,
and before/after of the allocate_refs / allocate_refs_with_root
duplication that landed on PR #20 and was unified in
|