mirror of
https://github.com/lahfir/agent-desktop.git
synced 2026-07-27 01:22:16 +00:00
feat: progressive skeleton traversal with ref-rooted drill-down (#20)
* feat: add data model foundation for progressive skeleton traversal
Add children_count, root_ref, skeleton fields to core types. Add
get_subtree() to PlatformAdapter trait. Add remove_by_root_ref() and
write-side size check to RefMap. No behavioral changes yet.
* feat: implement progressive skeleton traversal with ref-rooted drill-down
Add --skeleton and --root flags to snapshot command for token-efficient
accessibility tree exploration. Skeleton mode clamps depth to 3 levels
and annotates truncated containers with children_count, allowing AI
agents to discover regions before drilling into them. Named containers
at skeleton boundaries (via name or description) receive refs as
drill-down targets. The --root flag starts traversal from a previous
ref with scoped invalidation — only refs from that drill-down are
replaced on re-drill.
Key changes:
- New ref_alloc.rs: shared ref helpers (INTERACTIVE_ROLES, actions_for_role,
ref_entry_from_node, is_collapsible) extracted from snapshot.rs
- New snapshot_ref.rs: drill-down logic with DrillDownConfig, scoped
invalidation via root_ref tagging on RefEntry
- macOS count_children() uses raw CFArrayGetCount without materializing
AXElement wrappers for performance at skeleton boundaries
- RefMap write-side size check prevents >1MB files
- Skeleton anchors consider both name and description for Electron compat
* fix: mention --skeleton in STALE_REF error suggestion
* docs: document --skeleton and --root flags in skill reference
* docs: update phases.md and CLAUDE.md for progressive skeleton traversal
Add skeleton traversal as Phase 1 objective P1-O10. Document --skeleton
and --root flags, get_subtree() trait method, new core modules, and
platform-agnostic notes for Phase 2/3. Update risk mitigations and
performance optimizations table.
* docs: make progressive skeleton traversal the default agent workflow
Update SKILL.md observe-act loop to skeleton-first approach. Add
progressive skeleton traversal as the primary workflow pattern. Update
anti-patterns, key principles, and command quick reference to lead
with --skeleton + --root. Full snapshot remains documented as fallback
for simple apps.
* fix: preserve skeleton drill-down anchors
* fix: preserve drill-down refs across skeleton re-snapshots
Skeleton snapshots now load the existing refmap and remove only
skeleton-level refs (root_ref: None), preserving drill-down refs
accumulated via --root. Previously, build() always created a fresh
RefMap, breaking the skeleton → drill → act → skeleton(verify) workflow
by wiping all drill-down refs on the verify step.
* fix: preserve drill-down depth for root snapshots
* fix: verify AX action effect on Electron elements before trusting success
Chromium's AX implementation returns kAXErrorSuccess for AXPress,
AXConfirm, etc. but only toggles ARIA state without firing DOM event
handlers. This caused the click chain to short-circuit on false
positives, preventing CGClick from ever being reached.
The fix detects web elements via AXWebArea ancestor walk, then verifies
each AX action actually had a DOM effect by comparing focused element
pointers before and after. When all AX methods produce no real effect,
CGClick fires as the genuine last resort.
Native elements are unaffected — the original verified_press behavior
is preserved in a separate code path.
* chore: ignore .context/ for local tooling artifacts
* style: collapse web_action_had_effect signature to single line
* test: cover RefMap save oversize rejection
Extract the serialize+size-check step into a private helper so the 1MB
write rejection path can be unit-tested without filesystem I/O. Also
moves the size check above the directory creation in save() so an
oversized refmap no longer creates ~/.agent-desktop on rejection.
* test: assert stale-ref suggestion mentions --skeleton
Locks in the Phase 4 polish requirement that STALE_REF errors guide
agents back to a skeleton refresh, not just a plain snapshot.
* test: cover skeleton-to-drill-down counter continuity
Asserts that skeleton refs (@e1..@e10) survive a scoped invalidation
of @e3, that drill-down refs allocated after the skeleton continue from
@e11 instead of resetting, and that remove_by_root_ref drops only the
drill-down children.
* test: cover --root + --surface rejection at execute() boundary
Adds a NoopAdapter that uses every PlatformAdapter trait default to
exercise execute()'s validation guards without standing up a real
adapter. Verifies that combining --root with a non-Window surface
returns INVALID_ARGS, and that the Window surface does not trigger the
guard.
* test: add filesystem-redirected integration tests for run_from_ref
Adds a thread-local HOME override + RAII HomeGuard so RefMap::save and
RefMap::load can be exercised against an isolated temp directory without
env-var racing across parallel tests. Also adds a StubAdapter that
implements PlatformAdapter with a canned subtree response so the full
run_from_ref drill-down flow can be unit-tested without standing up
macOS AX state.
Closes seven Phase 3 plan acceptance items in one pass:
- save+load roundtrip with HOME override
- oversize save rejection preserves previous file on disk
- run_from_ref returns subtree and persists drill refs
- stale root ref → STALE_REF with skeleton suggestion
- re-drill replaces drill refs only, counter continues
- multiple drill-downs from @e1 and @e2 coexist
- empty subtree drill-down produces no new refs
* test: add golden fixtures for skeleton output and drill-down refmap
Adds two committed JSON fixtures under tests/fixtures/ plus matching
unit tests that build the same input trees, run them through
allocate_refs / run_from_ref, and assert the produced ref ids,
parent-child layout, and root_ref tagging match the golden expectations.
Locks in the JSON shape so future serialization changes have to be
deliberate.
* fix: bound build_subtree recursion on Electron wrapper chains
Anonymous AXGroup/AXGenericElement wrappers do not advance the semantic
depth counter (web-wrapper depth-skip), so a long chain of nested
wrappers could recurse arbitrarily deep without ever hitting either
max_depth or ABSOLUTE_MAX_DEPTH. The previous code checked
`depth >= ABSOLUTE_MAX_DEPTH` against the semantic depth, which stayed
at 0 throughout a wrapper chain. On pathological Electron trees this
risked stack exhaustion before any cap engaged.
Add a separate `raw_depth` parameter that always increments and use it
for the absolute cap. Semantic `depth` still drives `max_depth` and the
skeleton boundary so the wrapper-flattening behavior is preserved on
normal trees.
Also drops the long-unused `_include_bounds` parameter; bounds
filtering happens in `allocate_refs` in core, not here.
* chore: add pre-commit hook running fmt + clippy + tests
Mirrors the CI quality gates (cargo fmt --check, cargo clippy
--all-targets -- -D warnings, cargo test --lib --workspace) so a
failing commit never reaches origin. Skips automatically when no
Rust/TOML files are staged. Bypass with --no-verify or SKIP_PRECOMMIT=1
when a non-Rust hotfix is genuinely needed.
Hook lives under .githooks/ and is opt-in per clone:
git config core.hooksPath .githooks
Setup instructions added to CLAUDE.md.
* fix: prevent orphaned drill-down refs leaking across skeleton refresh
remove_skeleton_refs() previously dropped every entry whose root_ref
was None and kept every entry whose root_ref was Some. After a series
of skeleton refreshes that pattern leaked drill-down refs whose root
anchor had already been removed: they could never be cleaned up by a
future remove_by_root_ref(target) because target referred to a
non-existent anchor, and they accumulated until the 1MB write guard
fired.
Restructure the function to keep only:
- skeleton anchors that have at least one drill-down pointing at them
- drill-down refs whose root anchor is among the kept anchors
Unreferenced anchors are still dropped (the original intent), and
orphaned drill-downs are dropped at the same time so the refmap can
no longer accumulate dead entries.
Updated existing test to assert the new keep-when-referenced semantics
and added a regression test for the orphan-drilldown leak path.
* fix: align resolve traversal with snapshot child-attribute set
find_element_recursive previously walked AXChildren first then fell
back to AXContents only, and resolve_element_name only derived its
fallback label from AXChildren. Snapshot building, however, uses the
full child_attributes() list (AXChildren, AXContents,
AXChildrenInNavigationOrder) via copy_children. That asymmetry meant
refs minted for containers exposed only through AXChildrenInNavigationOrder
appeared in the snapshot output but produced false STALE_REF errors on
drill-down or action commands because the resolver could not find them
again.
Route both call sites through child_attributes() so resolution sees
the same children as snapshotting.
* fix: bypass AXConfirm on web elements and add skeleton anchors to drill-down
* fix: compare AX elements via CFEqual in web_action_had_effect
AXUIElementCopyAttributeValue follows the CoreFoundation Create rule
and returns a freshly-allocated CF object on every call, so raw
pointer equality (`before.0 != after.0`) between two separate copies
of AXFocusedUIElement was ALWAYS true even when the focused element
was unchanged. That made Step 1 (AXPress) of activate_web_element
appear successful every time and left Steps 2-4 of the escalation
chain (AXConfirm bypass, child actions, CGClick fallback) dead code
on web elements.
Replace the pointer comparison with CFEqual, which compares
accessibility element identity rather than heap addresses.
* refactor: unify allocate_refs across snapshot and drill-down paths
allocate_refs in snapshot.rs and allocate_refs_with_root in
snapshot_ref.rs were near-exact copies that differed only in whether
each allocated ref carried a root_ref tag. The duplication meant any
bug fix or behavior change in one path risked silently drifting from
the other (bot flagged this after the first round of fixes added even
more duplicated skeleton-anchor logic to allocate_refs_with_root).
Move the shared logic into ref_alloc.rs behind a new RefAllocConfig
struct with Option<&str> for the root_ref_id, and delete the
snapshot_ref.rs copy. Both snapshot::build and snapshot_ref::run_from_ref
now route through the same function with their respective configs.
append_surface_refs and the existing unit tests in both modules are
updated to use the shared function + config.
* chore: drop with_root from drill test names after allocator unification
The snapshot_ref tests kept their original test_allocate_refs_with_root_*
names after the allocator dedupe even though allocate_refs_with_root no
longer exists. Rename them to test_drill_alloc_* so grep for
allocate_refs_with_root returns zero matches and nobody assumes a second
allocator path exists.
* docs: compound DRY ref-allocator dedupe into knowledge base
Adds docs/solutions/best-practices/deduplicate-ref-allocator-via-config-struct-2026-04-14.md
documenting the root cause, guidance, consequences, trigger conditions,
and before/after of the allocate_refs / allocate_refs_with_root
duplication that landed on PR #20 and was unified in d06a6c2.
Also:
- gitignore allows docs/solutions/** (was caught by docs/* rule)
- CLAUDE.md workspace tree surfaces docs/solutions/ so fresh agents
discover the knowledge store
- docs/phases.md drops two stale DrillDownConfig references (lines 62
and 226) — the symbol no longer exists in the codebase
* refactor: drop unused root_ref field from TreeOptions
TreeOptions.root_ref was populated from SnapshotArgs.root_ref by
tree_options() and then never read anywhere. The actual routing
lives in execute(): `if let Some(ref root) = args.root_ref` branches
to snapshot_ref::run_from_ref(adapter, &opts, root), passing the
root id as an explicit argument — opts.root_ref was never consulted
by any caller.
Delete the field from TreeOptions and its Default impl and stop
populating it in tree_options(). SnapshotArgs.root_ref remains as
the single source of truth for the drill-down target.
* fix: suppress skeleton flag when --root is set
tree_options() clamped depth based on skeleton && root_ref.is_none()
but still forwarded skeleton: args.skeleton unconditionally. When a
caller passed --skeleton --root @e3, opts.skeleton stayed true, so
adapter.get_subtree on macOS built a truncated skeleton tree instead
of the full drill-down subtree, AND ref_alloc::allocate_refs tagged
skeleton-anchor refs with the drill-down root_ref — both unintended.
Tie the skeleton flag to the same root_ref.is_none() guard the depth
clamp already uses. A drill-down is always a full subtree now, never
a skeleton view.
Adds test_tree_options_suppresses_skeleton_for_drill_down to lock the
behavior in and extends the existing clamp test to assert the flag
still propagates on non-drill paths.
* fix: detect web action effect via value + selected + focus change
The previous focus-change-only heuristic in web_action_had_effect was
wrong for non-Chromium AXWebArea elements (Safari WKWebView, Mail,
Notes): AXPress on a checkbox toggles AXValue correctly but does NOT
shift the focused UI element. web_action_had_effect returned false, the
chain escalated to CGClick, and CGClick toggled the checkbox a second
time — net zero visible change. The pointer-equality bug that ae78cbc
replaced was masking this by always returning true; CFEqual exposed
the real signal gap.
Capture element state into a PreActionState struct (focused, value,
selected) before any AX action runs, and consider the action to have
had an effect if ANY of those three fields differs afterward. Element
value covers WKWebView checkboxes, text fields, sliders. Selected
covers tabs, radio buttons, and list items. Focused still covers
Chromium's DOM-driven focus shifts. Triggers on whichever signal is
most relevant to the element type without needing role-specific code.
Also re-exports copy_bool_attr from crates/macos/src/tree for use by
the chain-steps module.
* fix: emit skeleton boundary when drill path is about to hit raw cap
In skeleton mode, a chain of anonymous web wrappers (AXGroup /
AXGenericElement with empty title and value) never advances the
semantic `depth` counter, so `child_depth > max_depth` never fires
no matter how deep the chain runs. The recursion then hits the
`raw_depth >= ABSOLUTE_MAX_DEPTH` top-of-function guard and silently
returns None, dropping the subtree without any children_count marker.
Agents see truncated output with no indication that anything was
dropped.
Extend the at_skeleton_boundary condition to also fire when
child_raw_depth is about to hit ABSOLUTE_MAX_DEPTH, so skeleton mode
always emits a visible truncation node for deep wrapper chains
instead of disappearing them. The old `raw_depth < ABSOLUTE_MAX_DEPTH`
half of the guard was redundant — the top-of-function check already
ensures the current frame has raw budget; the meaningful question is
whether the CHILD frames will.
* perf: compute is_in_webarea once per verified-press call
try_focus_then_verified_confirm_or_press called is_in_webarea(el) to
gate AXConfirm, then fell through to do_verified_press which called
is_in_webarea(el) again on its first line. Each call walks up to 20
AX parents via per-element IPC (AXUIElementCopyAttributeValue on
AXParent), so every web-area click through this path paid the cost
twice.
Extract a private dispatch_verified_press(el, caps, in_web) that
takes the web-area flag as an input. do_verified_press keeps its
existing (el, caps) signature for chain_defs compatibility and
computes is_in_webarea once, then delegates. The focus-then-confirm
path also computes it once and passes it through. The result is
identical on both paths with one AX traversal instead of two.
* fix: skeleton anchors in drill-downs must not inherit root_ref; restore AXBrowser AXContents fallback
Skeleton anchors discovered while processing a drill-down subtree were
being tagged with the drill root_ref, making them indistinguishable from
regular drill entries. re-drills would delete those anchors via
remove_by_root_ref, breaking sub-drilling. They now always carry
root_ref=None so they survive re-drills as stable targets.
AXBrowser child resolution dropped the AXContents fallback when
child_attributes() was unified — the resolver now uses
["AXColumns","AXContents"] matching the original traversal order.
* fix: suppress skeleton anchor creation in drill-down mode to prevent orphaned ref accumulation
* fix: bounds-based resolver pruning and CGClick fallback on chain timeout
Resolver was doing exhaustive DFS through entire AX tree (depth 50) to
find a single element. For large documents like a dense spreadsheet this
meant visiting tens of thousands of cells via AX IPC, causing multi-minute
hangs before the click chain even started.
Two changes:
- find_element_recursive now prunes subtrees whose spatial bounds do not
contain the target element's centre point, and aborts the whole search
after five seconds. Numbers table -> button not inside -> entire table
skipped. Resolution went from >1m49s to <50ms.
- execute_chain now sets a 1s app-level AX messaging timeout so calls to
children/parents fail fast instead of blocking for the system default 6s,
and when the 10s chain deadline fires it attempts the CGClick step before
returning failure so the coordinate fallback is always reached.
* refactor: resolve 9 code-review todos for progressive skeleton traversal
- skeleton refresh now starts from RefMap::new() (removes stale ref accumulation across refreshes)
- drill-down snapshot resolves real WindowInfo via PID lookup instead of synthetic empty id
- --root flag validates ref format before RefMap lookup (returns INVALID_ARGS not STALE_REF)
- ABSOLUTE_MAX_DEPTH truncation emits boundary node with children_count instead of silently dropping
- fix ref vs ref_id field name in commands-observation.md JSON examples
- fix phases.md TreeOptions listing root_ref (it lives in SnapshotArgs, not TreeOptions)
- add batch snapshot skeleton/root examples to commands-system.md
- split snapshot.rs and snapshot_ref.rs test modules into separate _tests.rs files (143/69 LOC)
- add integration tests for skeleton + drill-down workflow including invalid --root validation
* docs: compound progressive snapshot review hardening
* docs: tighten progressive snapshot compound write-up
* docs: update README with progressive skeleton traversal and fix command count to 50
* docs: correct command count to 53, add Notifications section and platform notes
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
This commit is contained in:
parent
ccc734ecbc
commit
c17f2fae7a
36 changed files with 2766 additions and 426 deletions
52
.githooks/pre-commit
Executable file
52
.githooks/pre-commit
Executable file
|
|
@ -0,0 +1,52 @@
|
|||
#!/usr/bin/env bash
|
||||
# Pre-commit hook for agent-desktop.
|
||||
# Mirrors the CI quality gates so a failing commit never reaches origin.
|
||||
#
|
||||
# Setup (once per clone):
|
||||
# git config core.hooksPath .githooks
|
||||
#
|
||||
# Bypass for a single commit (only when truly necessary):
|
||||
# git commit --no-verify
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
if [ -n "${SKIP_PRECOMMIT:-}" ]; then
|
||||
echo "pre-commit: SKIP_PRECOMMIT set, skipping"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! command -v cargo >/dev/null 2>&1; then
|
||||
echo "pre-commit: cargo not found in PATH, skipping (install Rust to enable)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
|
||||
if git diff --cached --name-only --diff-filter=ACMR | grep -qE '\.(rs|toml)$'; then
|
||||
HAS_RUST_CHANGES=1
|
||||
else
|
||||
HAS_RUST_CHANGES=0
|
||||
fi
|
||||
|
||||
if [ "$HAS_RUST_CHANGES" -eq 0 ]; then
|
||||
echo "pre-commit: no Rust changes staged, skipping cargo checks"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
run() {
|
||||
local label="$1"
|
||||
shift
|
||||
printf '\033[1;34m▶ %s\033[0m\n' "$label"
|
||||
if ! "$@"; then
|
||||
printf '\033[1;31m✗ %s failed\033[0m\n' "$label" >&2
|
||||
echo "" >&2
|
||||
echo "pre-commit: refusing the commit. Fix the issues above or rerun with SKIP_PRECOMMIT=1 to bypass." >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
run "cargo fmt --all -- --check" cargo fmt --all -- --check
|
||||
run "cargo clippy --all-targets -- -D warnings" cargo clippy --all-targets -- -D warnings
|
||||
run "cargo test --lib --workspace" cargo test --lib --workspace
|
||||
|
||||
printf '\033[1;32m✓ pre-commit checks passed\033[0m\n'
|
||||
5
.gitignore
vendored
5
.gitignore
vendored
|
|
@ -73,5 +73,8 @@ docs/*
|
|||
!docs/architecture.excalidraw
|
||||
!docs/architecture.png
|
||||
!docs/phases.md
|
||||
!docs/solutions/
|
||||
!docs/solutions/**
|
||||
todos/
|
||||
.cursor/
|
||||
.cursor/
|
||||
.context/
|
||||
31
CLAUDE.md
31
CLAUDE.md
|
|
@ -19,6 +19,16 @@ cargo tree -p agent-desktop-core # Verify no platform crate leaks
|
|||
|
||||
Run the binary: `./target/release/agent-desktop snapshot --app Finder -i`
|
||||
|
||||
## Pre-commit Hook
|
||||
|
||||
The repo ships a pre-commit hook at `.githooks/pre-commit` that runs `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, and `cargo test --lib --workspace` against staged Rust changes. Wire it up once after cloning:
|
||||
|
||||
```bash
|
||||
git config core.hooksPath .githooks
|
||||
```
|
||||
|
||||
Bypass for an emergency commit with `git commit --no-verify` or `SKIP_PRECOMMIT=1 git commit ...`.
|
||||
|
||||
## Project Overview
|
||||
|
||||
Cross-platform Rust CLI + MCP server enabling AI agents to observe and control desktop applications via native OS accessibility trees.
|
||||
|
|
@ -57,6 +67,10 @@ agent-desktop/
|
|||
├── clippy.toml # project-wide lint config
|
||||
├── crates/
|
||||
│ ├── core/ # agent-desktop-core (platform-agnostic)
|
||||
│ │ └── src/
|
||||
│ │ ├── ref_alloc.rs # Shared ref helpers (INTERACTIVE_ROLES, is_collapsible)
|
||||
│ │ ├── snapshot_ref.rs # Ref-rooted drill-down (run_from_ref)
|
||||
│ │ └── commands/ # one file per command
|
||||
│ ├── macos/ # agent-desktop-macos (Phase 1)
|
||||
│ ├── windows/ # agent-desktop-windows (stub → Phase 2)
|
||||
│ └── linux/ # agent-desktop-linux (stub → Phase 2)
|
||||
|
|
@ -66,6 +80,8 @@ agent-desktop/
|
|||
│ ├── cli_args.rs # all command argument structs
|
||||
│ ├── dispatch.rs # command dispatcher + parse helpers
|
||||
│ └── batch_dispatch.rs # batch command execution
|
||||
├── docs/
|
||||
│ └── solutions/ # documented solutions to past problems (bugs, best practices, workflow patterns), organized by category with YAML frontmatter (module, tags, problem_type); relevant when implementing or debugging in documented areas
|
||||
└── tests/
|
||||
├── fixtures/ # golden JSON snapshots
|
||||
└── integration/ # macOS CI integration tests
|
||||
|
|
@ -284,16 +300,20 @@ Error responses:
|
|||
- RefMap stored at `~/.agent-desktop/last_refmap.json` with `0o600` permissions, directory at `0o700`
|
||||
- Each snapshot REPLACES the refmap file entirely (atomic write via temp + rename)
|
||||
- Action commands use optimistic re-identification: `(pid, role, name, bounds_hash)`. Return `STALE_REF` on mismatch.
|
||||
- Progressive traversal: `--skeleton` clamps depth to 3, annotates truncated containers with `children_count`. Named/described containers at boundary receive refs as drill-down targets
|
||||
- Drill-down: `--root @ref` starts from a previously-discovered ref with scoped invalidation (only that ref's subtree refs are replaced on re-drill)
|
||||
- RefMap size check: write-side guard prevents >1MB refmap files
|
||||
|
||||
## PlatformAdapter Trait
|
||||
|
||||
12 methods with default implementations returning `not_supported()`:
|
||||
13 methods with default implementations returning `not_supported()`:
|
||||
|
||||
```rust
|
||||
pub trait PlatformAdapter: Send + Sync {
|
||||
fn list_windows(&self, filter: &WindowFilter) -> Result<Vec<WindowInfo>, AdapterError>;
|
||||
fn list_apps(&self) -> Result<Vec<AppInfo>, AdapterError>;
|
||||
fn get_tree(&self, win: &WindowInfo, opts: &TreeOptions) -> Result<AccessibilityNode, AdapterError>;
|
||||
fn get_subtree(&self, handle: &NativeHandle, opts: &TreeOptions) -> Result<AccessibilityNode, AdapterError>;
|
||||
fn execute_action(&self, handle: &NativeHandle, action: Action) -> Result<ActionResult, AdapterError>;
|
||||
fn resolve_element(&self, entry: &RefEntry) -> Result<NativeHandle, AdapterError>;
|
||||
fn check_permissions(&self) -> PermissionStatus;
|
||||
|
|
@ -406,7 +426,9 @@ Target binary size: <15MB per platform.
|
|||
- `cargo test --workspace`
|
||||
- Binary size check: fail if release binary exceeds 15MB
|
||||
|
||||
## Implemented Commands (50)
|
||||
## Implemented Commands (53)
|
||||
|
||||
> **Platform note:** All 53 commands are implemented on macOS (Phase 1). Windows and Linux adapters are planned (Phase 2/3) and will support the same command surface; notification commands depend on platform-specific notification APIs.
|
||||
|
||||
| Category | Commands |
|
||||
|----------|----------|
|
||||
|
|
@ -415,9 +437,10 @@ Target binary size: <15MB per platform.
|
|||
| Interaction (14) | `click`, `double-click`, `triple-click`, `right-click`, `type`, `set-value`, `clear`, `focus`, `select`, `toggle`, `check`, `uncheck`, `expand`, `collapse` |
|
||||
| Scroll (2) | `scroll`, `scroll-to` |
|
||||
| Keyboard (3) | `press`, `key-down`, `key-up` |
|
||||
| Mouse (5) | `hover`, `drag`, `mouse-move`, `mouse-click`, `mouse-down`, `mouse-up` |
|
||||
| Mouse (6) | `hover`, `drag`, `mouse-move`, `mouse-click`, `mouse-down`, `mouse-up` |
|
||||
| Notifications (4) *(macOS)* | `list-notifications`, `dismiss-notification`, `dismiss-all-notifications`, `notification-action` |
|
||||
| Clipboard (3) | `clipboard-get`, `clipboard-set`, `clipboard-clear` |
|
||||
| Wait (1) | `wait` (with `--element`, `--window`, `--text`, `--menu` flags) |
|
||||
| Wait (1) | `wait` (with `--element`, `--window`, `--text`, `--menu`, `--notification` flags) |
|
||||
| System (3) | `status`, `permissions`, `version` |
|
||||
| Batch (1) | `batch` |
|
||||
|
||||
|
|
|
|||
38
README.md
38
README.md
|
|
@ -11,7 +11,8 @@
|
|||
## Key Features
|
||||
|
||||
- **Native Rust CLI**: Fast, single binary, no runtime dependencies
|
||||
- **50 commands**: Observation, interaction, keyboard, mouse, clipboard, window management
|
||||
- **53 commands**: Observation, interaction, keyboard, mouse, notifications, clipboard, window management
|
||||
- **Progressive skeleton traversal**: 78–96% token reduction on dense apps via shallow overview + targeted drill-down
|
||||
- **Snapshot & refs**: AI-optimized workflow using deterministic element references (`@e1`, `@e2`)
|
||||
- **AX-first interactions**: Every action exhausts pure accessibility API strategies before falling back to mouse events
|
||||
- **Structured JSON output**: Machine-readable responses with error codes and recovery hints
|
||||
|
|
@ -52,6 +53,24 @@ agent-desktop permissions --request # trigger system dialog
|
|||
|
||||
## Core Workflow for AI
|
||||
|
||||
For dense apps (Slack, VS Code, Notion), use **progressive skeleton traversal** to minimize token usage:
|
||||
|
||||
```bash
|
||||
# 1. Shallow overview — depth-3 map, truncated containers show children_count
|
||||
agent-desktop snapshot --skeleton --app Slack -i --compact
|
||||
|
||||
# 2. Drill into a region of interest (named containers get refs as drill targets)
|
||||
agent-desktop snapshot --root @e3 -i --compact
|
||||
|
||||
# 3. Act on an element found in the drill-down
|
||||
agent-desktop click @e12
|
||||
|
||||
# 4. Re-drill the same region to verify the state change
|
||||
agent-desktop snapshot --root @e3 -i --compact
|
||||
```
|
||||
|
||||
For simple apps, a full snapshot is fine:
|
||||
|
||||
```bash
|
||||
agent-desktop snapshot --app Finder -i # get interactive elements with refs
|
||||
agent-desktop click @e3 # click a button by ref
|
||||
|
|
@ -60,8 +79,6 @@ agent-desktop press cmd+s # keyboard shortcut
|
|||
agent-desktop snapshot -i # re-observe after UI changes
|
||||
```
|
||||
|
||||
The snapshot + ref pattern is optimal for LLMs: refs provide deterministic element selection without re-querying the accessibility tree.
|
||||
|
||||
```
|
||||
Agent loop: snapshot → decide → act → snapshot → decide → act → ...
|
||||
```
|
||||
|
|
@ -141,6 +158,18 @@ agent-desktop maximize w-4521 # maximize
|
|||
agent-desktop restore w-4521 # restore
|
||||
```
|
||||
|
||||
### Notifications *(macOS only)*
|
||||
|
||||
```bash
|
||||
agent-desktop list-notifications # list all notifications
|
||||
agent-desktop list-notifications --app "Slack" # filter by app
|
||||
agent-desktop list-notifications --text "deploy" --limit 5 # filter by text
|
||||
agent-desktop dismiss-notification 1 # dismiss by index
|
||||
agent-desktop dismiss-all-notifications # dismiss all
|
||||
agent-desktop dismiss-all-notifications --app "Slack" # dismiss all from app
|
||||
agent-desktop notification-action 1 --action "Reply" # click action button
|
||||
```
|
||||
|
||||
### Clipboard
|
||||
|
||||
```bash
|
||||
|
|
@ -192,6 +221,8 @@ agent-desktop snapshot [OPTIONS]
|
|||
| `--compact` | off | Omit empty structural nodes |
|
||||
| `--include-bounds` | off | Include pixel bounds (x, y, width, height) |
|
||||
| `--max-depth <N>` | 10 | Maximum tree depth |
|
||||
| `--skeleton` | off | Shallow 3-level overview; truncated containers show `children_count` and get refs as drill targets |
|
||||
| `--root <REF>` | - | Start traversal from this ref; merges into existing refmap with scoped invalidation |
|
||||
| `--surface <TYPE>` | window | `window`, `focused`, `menu`, `menubar`, `sheet`, `popover`, `alert` |
|
||||
|
||||
## JSON Output
|
||||
|
|
@ -262,6 +293,7 @@ snapshot → act → STALE_REF? → snapshot again → retry
|
|||
| Screenshot | **Yes** | Planned | Planned |
|
||||
| Clipboard | **Yes** | Planned | Planned |
|
||||
| App & window management | **Yes** | Planned | Planned |
|
||||
| Notifications | **Yes** | Planned | Planned |
|
||||
|
||||
## Development
|
||||
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ pub struct TreeOptions {
|
|||
pub interactive_only: bool,
|
||||
pub compact: bool,
|
||||
pub surface: SnapshotSurface,
|
||||
pub skeleton: bool,
|
||||
}
|
||||
|
||||
impl Default for TreeOptions {
|
||||
|
|
@ -40,6 +41,7 @@ impl Default for TreeOptions {
|
|||
interactive_only: false,
|
||||
compact: false,
|
||||
surface: SnapshotSurface::Window,
|
||||
skeleton: false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -244,4 +246,12 @@ pub trait PlatformAdapter: Send + Sync {
|
|||
) -> Result<ActionResult, AdapterError> {
|
||||
Err(AdapterError::not_supported("notification_action"))
|
||||
}
|
||||
|
||||
fn get_subtree(
|
||||
&self,
|
||||
_handle: &NativeHandle,
|
||||
_opts: &TreeOptions,
|
||||
) -> Result<AccessibilityNode, AdapterError> {
|
||||
Err(AdapterError::not_supported("get_subtree"))
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
use crate::{
|
||||
adapter::{PlatformAdapter, SnapshotSurface},
|
||||
commands::helpers::validate_ref_id,
|
||||
error::AppError,
|
||||
snapshot,
|
||||
snapshot, snapshot_ref,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
|
|
@ -13,6 +14,26 @@ pub struct SnapshotArgs {
|
|||
pub interactive_only: bool,
|
||||
pub compact: bool,
|
||||
pub surface: SnapshotSurface,
|
||||
pub skeleton: bool,
|
||||
pub root_ref: Option<String>,
|
||||
}
|
||||
|
||||
fn tree_options(args: &SnapshotArgs) -> crate::adapter::TreeOptions {
|
||||
let skeleton_applies = args.skeleton && args.root_ref.is_none();
|
||||
let effective_depth = if skeleton_applies {
|
||||
args.max_depth.min(3)
|
||||
} else {
|
||||
args.max_depth
|
||||
};
|
||||
|
||||
crate::adapter::TreeOptions {
|
||||
max_depth: effective_depth,
|
||||
include_bounds: args.include_bounds,
|
||||
interactive_only: args.interactive_only,
|
||||
compact: args.compact,
|
||||
surface: args.surface,
|
||||
skeleton: skeleton_applies,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn execute(args: SnapshotArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
|
|
@ -25,13 +46,17 @@ pub fn execute(args: SnapshotArgs, adapter: &dyn PlatformAdapter) -> Result<Valu
|
|||
args.compact
|
||||
);
|
||||
|
||||
let opts = crate::adapter::TreeOptions {
|
||||
max_depth: args.max_depth,
|
||||
include_bounds: args.include_bounds,
|
||||
interactive_only: args.interactive_only,
|
||||
compact: args.compact,
|
||||
surface: args.surface,
|
||||
};
|
||||
let opts = tree_options(&args);
|
||||
|
||||
if let Some(ref root) = args.root_ref {
|
||||
if !matches!(args.surface, SnapshotSurface::Window) {
|
||||
return Err(AppError::invalid_input(
|
||||
"--root cannot be combined with --surface",
|
||||
));
|
||||
}
|
||||
validate_ref_id(root)?;
|
||||
return format_result(snapshot_ref::run_from_ref(adapter, &opts, root)?);
|
||||
}
|
||||
|
||||
let result = snapshot::run(
|
||||
adapter,
|
||||
|
|
@ -40,6 +65,10 @@ pub fn execute(args: SnapshotArgs, adapter: &dyn PlatformAdapter) -> Result<Valu
|
|||
args.window_id.as_deref(),
|
||||
)?;
|
||||
|
||||
format_result(result)
|
||||
}
|
||||
|
||||
fn format_result(result: snapshot::SnapshotResult) -> Result<Value, AppError> {
|
||||
let ref_count = result.refmap.len();
|
||||
let tree = serde_json::to_value(&result.tree)?;
|
||||
let win = &result.window;
|
||||
|
|
@ -61,3 +90,137 @@ pub fn execute(args: SnapshotArgs, adapter: &dyn PlatformAdapter) -> Result<Valu
|
|||
"tree": tree
|
||||
}))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::error::ErrorCode;
|
||||
|
||||
struct NoopAdapter;
|
||||
impl PlatformAdapter for NoopAdapter {}
|
||||
|
||||
fn base_args() -> SnapshotArgs {
|
||||
SnapshotArgs {
|
||||
app: None,
|
||||
window_id: None,
|
||||
max_depth: 8,
|
||||
include_bounds: false,
|
||||
interactive_only: false,
|
||||
compact: false,
|
||||
surface: SnapshotSurface::Window,
|
||||
skeleton: false,
|
||||
root_ref: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn args_with_surface(surface: SnapshotSurface) -> SnapshotArgs {
|
||||
SnapshotArgs {
|
||||
surface,
|
||||
root_ref: Some("@e3".into()),
|
||||
..base_args()
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_tree_options_clamps_skeleton_depth() {
|
||||
let mut args = base_args();
|
||||
args.skeleton = true;
|
||||
|
||||
let opts = tree_options(&args);
|
||||
|
||||
assert_eq!(opts.max_depth, 3);
|
||||
assert!(
|
||||
opts.skeleton,
|
||||
"skeleton flag must propagate for full snapshots"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_tree_options_suppresses_skeleton_for_drill_down() {
|
||||
let mut args = base_args();
|
||||
args.skeleton = true;
|
||||
args.root_ref = Some("@e3".into());
|
||||
|
||||
let opts = tree_options(&args);
|
||||
|
||||
assert_eq!(
|
||||
opts.max_depth, 8,
|
||||
"depth must not be clamped for drill-down"
|
||||
);
|
||||
assert!(
|
||||
!opts.skeleton,
|
||||
"skeleton flag must be suppressed for drill-down so build_subtree \
|
||||
returns the full subtree and allocate_refs does not tag anchors"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_root_with_menu_surface_rejected() {
|
||||
let args = args_with_surface(SnapshotSurface::Menu);
|
||||
let err = execute(args, &NoopAdapter).expect_err("should reject --root + --surface");
|
||||
match err {
|
||||
AppError::Adapter(adapter_err) => {
|
||||
assert_eq!(adapter_err.code, ErrorCode::InvalidArgs);
|
||||
assert!(
|
||||
adapter_err.message.contains("--root")
|
||||
&& adapter_err.message.contains("--surface"),
|
||||
"error message should name both flags, got: {}",
|
||||
adapter_err.message
|
||||
);
|
||||
}
|
||||
other => panic!("expected Adapter(InvalidArgs), got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_root_with_window_surface_does_not_short_circuit_validation() {
|
||||
let args = args_with_surface(SnapshotSurface::Window);
|
||||
let result = execute(args, &NoopAdapter);
|
||||
assert!(
|
||||
result.is_err(),
|
||||
"NoopAdapter cannot satisfy run_from_ref so this must error"
|
||||
);
|
||||
if let AppError::Adapter(adapter_err) = result.unwrap_err() {
|
||||
assert_ne!(
|
||||
adapter_err.code,
|
||||
ErrorCode::InvalidArgs,
|
||||
"Window surface must NOT trigger the --surface validation guard"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_invalid_root_ref_format_returns_invalid_args() {
|
||||
let args = SnapshotArgs {
|
||||
root_ref: Some("not-a-ref".into()),
|
||||
..base_args()
|
||||
};
|
||||
let err = execute(args, &NoopAdapter).expect_err("malformed --root should fail");
|
||||
match err {
|
||||
AppError::Adapter(adapter_err) => {
|
||||
assert_eq!(
|
||||
adapter_err.code,
|
||||
ErrorCode::InvalidArgs,
|
||||
"malformed ref must return INVALID_ARGS, not STALE_REF"
|
||||
);
|
||||
}
|
||||
other => panic!("expected Adapter(InvalidArgs), got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_valid_root_ref_format_does_not_trigger_invalid_args() {
|
||||
let args = SnapshotArgs {
|
||||
root_ref: Some("@e42".into()),
|
||||
..base_args()
|
||||
};
|
||||
let err = execute(args, &NoopAdapter).expect_err("NoopAdapter cannot resolve ref");
|
||||
if let AppError::Adapter(adapter_err) = err {
|
||||
assert_ne!(
|
||||
adapter_err.code,
|
||||
ErrorCode::InvalidArgs,
|
||||
"well-formed ref must not trigger INVALID_ARGS"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -78,7 +78,9 @@ impl AdapterError {
|
|||
ErrorCode::StaleRef,
|
||||
format!("{ref_id} not found in current RefMap"),
|
||||
)
|
||||
.with_suggestion("Run 'snapshot' to refresh, then retry with updated ref")
|
||||
.with_suggestion(
|
||||
"Run 'snapshot' (or 'snapshot --skeleton') to refresh, then retry with updated ref",
|
||||
)
|
||||
}
|
||||
|
||||
pub fn not_supported(method: &str) -> Self {
|
||||
|
|
@ -182,4 +184,19 @@ mod tests {
|
|||
let json = serde_json::to_string(&code).unwrap();
|
||||
assert_eq!(json, "\"NOTIFICATION_NOT_FOUND\"");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stale_ref_suggestion_mentions_skeleton() {
|
||||
let err = AdapterError::stale_ref("@e7");
|
||||
assert_eq!(err.code, ErrorCode::StaleRef);
|
||||
assert!(err.message.contains("@e7"));
|
||||
let suggestion = err
|
||||
.suggestion
|
||||
.as_deref()
|
||||
.expect("stale_ref should carry a suggestion");
|
||||
assert!(
|
||||
suggestion.contains("skeleton"),
|
||||
"stale-ref suggestion should mention skeleton refresh, got: {suggestion}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,8 +6,10 @@ pub mod hints;
|
|||
pub mod node;
|
||||
pub mod notification;
|
||||
pub mod output;
|
||||
pub mod ref_alloc;
|
||||
pub mod refs;
|
||||
pub mod snapshot;
|
||||
pub mod snapshot_ref;
|
||||
|
||||
pub use action::{
|
||||
Action, ActionResult, Direction, DragParams, ElementState, KeyCombo, Modifier, MouseButton,
|
||||
|
|
|
|||
|
|
@ -25,6 +25,9 @@ pub struct AccessibilityNode {
|
|||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub bounds: Option<Rect>,
|
||||
|
||||
#[serde(skip_serializing_if = "Option::is_none", default)]
|
||||
pub children_count: Option<u32>,
|
||||
|
||||
#[serde(skip_serializing_if = "Vec::is_empty", default)]
|
||||
pub children: Vec<AccessibilityNode>,
|
||||
}
|
||||
|
|
@ -112,6 +115,49 @@ mod tests {
|
|||
assert_eq!(rect.width, 100.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_children_count_omitted_when_none() {
|
||||
let node = AccessibilityNode {
|
||||
ref_id: None,
|
||||
role: "group".into(),
|
||||
name: Some("Sidebar".into()),
|
||||
value: None,
|
||||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
children_count: None,
|
||||
children: vec![],
|
||||
};
|
||||
let json = serde_json::to_string(&node).unwrap();
|
||||
assert!(!json.contains("children_count"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_children_count_present_when_set() {
|
||||
let node = AccessibilityNode {
|
||||
ref_id: None,
|
||||
role: "group".into(),
|
||||
name: Some("Sidebar".into()),
|
||||
value: None,
|
||||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
children_count: Some(47),
|
||||
children: vec![],
|
||||
};
|
||||
let json = serde_json::to_string(&node).unwrap();
|
||||
assert!(json.contains("\"children_count\":47"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_children_count_backward_compat() {
|
||||
let json = r#"{"role":"button","name":"OK"}"#;
|
||||
let node: AccessibilityNode = serde_json::from_str(json).unwrap();
|
||||
assert!(node.children_count.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_rect_normal_roundtrip() {
|
||||
let rect = Rect {
|
||||
|
|
|
|||
126
crates/core/src/ref_alloc.rs
Normal file
126
crates/core/src/ref_alloc.rs
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
use crate::node::AccessibilityNode;
|
||||
use crate::refs::{RefEntry, RefMap};
|
||||
|
||||
pub(crate) const INTERACTIVE_ROLES: &[&str] = &[
|
||||
"button",
|
||||
"textfield",
|
||||
"checkbox",
|
||||
"link",
|
||||
"menuitem",
|
||||
"tab",
|
||||
"slider",
|
||||
"combobox",
|
||||
"treeitem",
|
||||
"cell",
|
||||
"radiobutton",
|
||||
"incrementor",
|
||||
"menubutton",
|
||||
"switch",
|
||||
"colorwell",
|
||||
"dockitem",
|
||||
];
|
||||
|
||||
pub(crate) fn actions_for_role(role: &str) -> Vec<String> {
|
||||
match role {
|
||||
"button" | "link" | "menuitem" | "tab" | "radiobutton" => vec!["Click".into()],
|
||||
"textfield" | "incrementor" => vec!["Click".into(), "SetValue".into(), "SetFocus".into()],
|
||||
"checkbox" => vec!["Click".into(), "Toggle".into()],
|
||||
"combobox" => vec!["Click".into(), "Select".into()],
|
||||
"treeitem" => vec!["Click".into(), "Expand".into(), "Collapse".into()],
|
||||
"slider" => vec!["SetValue".into()],
|
||||
"cell" => vec!["Click".into()],
|
||||
_ => vec!["Click".into()],
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn ref_entry_from_node(
|
||||
node: &AccessibilityNode,
|
||||
pid: i32,
|
||||
source_app: Option<&str>,
|
||||
root_ref: Option<String>,
|
||||
) -> RefEntry {
|
||||
RefEntry {
|
||||
pid,
|
||||
role: node.role.clone(),
|
||||
name: node.name.clone(),
|
||||
value: node.value.clone(),
|
||||
states: node.states.clone(),
|
||||
bounds: node.bounds,
|
||||
bounds_hash: node.bounds.as_ref().map(|b| b.bounds_hash()),
|
||||
available_actions: actions_for_role(&node.role),
|
||||
source_app: source_app.map(str::to_string),
|
||||
root_ref,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn is_collapsible(node: &AccessibilityNode) -> bool {
|
||||
node.ref_id.is_none()
|
||||
&& node.name.as_deref().is_none_or(str::is_empty)
|
||||
&& node.value.as_deref().is_none_or(str::is_empty)
|
||||
&& node.description.as_deref().is_none_or(str::is_empty)
|
||||
&& node.states.is_empty()
|
||||
&& node.children.len() == 1
|
||||
}
|
||||
|
||||
pub(crate) struct RefAllocConfig<'a> {
|
||||
pub include_bounds: bool,
|
||||
pub interactive_only: bool,
|
||||
pub compact: bool,
|
||||
pub pid: i32,
|
||||
pub source_app: Option<&'a str>,
|
||||
pub root_ref_id: Option<&'a str>,
|
||||
}
|
||||
|
||||
pub(crate) fn allocate_refs(
|
||||
mut node: AccessibilityNode,
|
||||
refmap: &mut RefMap,
|
||||
config: &RefAllocConfig,
|
||||
) -> AccessibilityNode {
|
||||
let root_ref_owned = config.root_ref_id.map(str::to_string);
|
||||
let is_interactive = INTERACTIVE_ROLES.contains(&node.role.as_str());
|
||||
|
||||
if is_interactive {
|
||||
let entry =
|
||||
ref_entry_from_node(&node, config.pid, config.source_app, root_ref_owned.clone());
|
||||
node.ref_id = Some(refmap.allocate(entry));
|
||||
}
|
||||
|
||||
let has_label = node.name.as_deref().is_some_and(|n| !n.is_empty())
|
||||
|| node.description.as_deref().is_some_and(|d| !d.is_empty());
|
||||
let is_skeleton_anchor = !is_interactive
|
||||
&& node.children_count.is_some()
|
||||
&& has_label
|
||||
&& config.root_ref_id.is_none();
|
||||
|
||||
if is_skeleton_anchor {
|
||||
let mut entry = ref_entry_from_node(&node, config.pid, config.source_app, None);
|
||||
entry.available_actions = vec![];
|
||||
node.ref_id = Some(refmap.allocate(entry));
|
||||
}
|
||||
|
||||
if !config.include_bounds {
|
||||
node.bounds = None;
|
||||
}
|
||||
|
||||
node.children = node
|
||||
.children
|
||||
.into_iter()
|
||||
.filter_map(|child| {
|
||||
let child = allocate_refs(child, refmap, config);
|
||||
if config.compact && is_collapsible(&child) {
|
||||
return child.children.into_iter().next();
|
||||
}
|
||||
if config.interactive_only
|
||||
&& child.ref_id.is_none()
|
||||
&& child.children.is_empty()
|
||||
&& child.children_count.is_none()
|
||||
{
|
||||
None
|
||||
} else {
|
||||
Some(child)
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
|
||||
node
|
||||
}
|
||||
|
|
@ -1,10 +1,15 @@
|
|||
use crate::error::AppError;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::cell::RefCell;
|
||||
use std::collections::HashMap;
|
||||
use std::path::PathBuf;
|
||||
|
||||
const MAX_REFMAP_BYTES: u64 = 1_048_576; // 1 MB
|
||||
|
||||
thread_local! {
|
||||
static HOME_OVERRIDE: RefCell<Option<PathBuf>> = const { RefCell::new(None) };
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct RefEntry {
|
||||
pub pid: i32,
|
||||
|
|
@ -19,6 +24,8 @@ pub struct RefEntry {
|
|||
pub bounds_hash: Option<u64>,
|
||||
pub available_actions: Vec<String>,
|
||||
pub source_app: Option<String>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub root_ref: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
|
|
@ -54,7 +61,24 @@ impl RefMap {
|
|||
self.inner.is_empty()
|
||||
}
|
||||
|
||||
pub fn remove_by_root_ref(&mut self, root: &str) {
|
||||
self.inner
|
||||
.retain(|_, entry| entry.root_ref.as_deref() != Some(root));
|
||||
}
|
||||
|
||||
fn serialize_with_size_check(&self) -> Result<String, AppError> {
|
||||
let json = serde_json::to_string(self)?;
|
||||
if json.len() as u64 > MAX_REFMAP_BYTES {
|
||||
return Err(AppError::Internal(
|
||||
"RefMap exceeds 1MB size limit on write".into(),
|
||||
));
|
||||
}
|
||||
Ok(json)
|
||||
}
|
||||
|
||||
pub fn save(&self) -> Result<(), AppError> {
|
||||
let json = self.serialize_with_size_check()?;
|
||||
|
||||
let path = refmap_path()?;
|
||||
let dir = path
|
||||
.parent()
|
||||
|
|
@ -71,7 +95,6 @@ impl RefMap {
|
|||
#[cfg(not(unix))]
|
||||
std::fs::create_dir_all(dir)?;
|
||||
|
||||
let json = serde_json::to_string(self)?;
|
||||
let tmp = path.with_extension("tmp");
|
||||
|
||||
#[cfg(unix)]
|
||||
|
|
@ -122,11 +145,73 @@ fn refmap_path() -> Result<PathBuf, AppError> {
|
|||
}
|
||||
|
||||
fn home_dir() -> Option<PathBuf> {
|
||||
if let Some(p) = HOME_OVERRIDE.with(|cell| cell.borrow().clone()) {
|
||||
return Some(p);
|
||||
}
|
||||
std::env::var_os("HOME")
|
||||
.map(PathBuf::from)
|
||||
.or_else(|| std::env::var_os("USERPROFILE").map(PathBuf::from))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) struct HomeGuard {
|
||||
_dir: tempdir::TempDir,
|
||||
prev: Option<PathBuf>,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tempdir {
|
||||
use std::fs;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
static COUNTER: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
pub struct TempDir(PathBuf);
|
||||
|
||||
impl TempDir {
|
||||
pub fn new() -> Self {
|
||||
let n = COUNTER.fetch_add(1, Ordering::SeqCst);
|
||||
let nanos = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos())
|
||||
.unwrap_or(0);
|
||||
let path = std::env::temp_dir().join(format!("agent-desktop-test-{nanos}-{n}"));
|
||||
fs::create_dir_all(&path).expect("create tempdir");
|
||||
Self(path)
|
||||
}
|
||||
|
||||
pub fn path(&self) -> &std::path::Path {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for TempDir {
|
||||
fn drop(&mut self) {
|
||||
let _ = fs::remove_dir_all(&self.0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
impl HomeGuard {
|
||||
pub fn new() -> Self {
|
||||
let dir = tempdir::TempDir::new();
|
||||
let prev = HOME_OVERRIDE.with(|cell| cell.borrow().clone());
|
||||
HOME_OVERRIDE.with(|cell| *cell.borrow_mut() = Some(dir.path().to_path_buf()));
|
||||
Self { _dir: dir, prev }
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
impl Drop for HomeGuard {
|
||||
fn drop(&mut self) {
|
||||
let prev = self.prev.take();
|
||||
HOME_OVERRIDE.with(|cell| *cell.borrow_mut() = prev);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
|
@ -144,6 +229,7 @@ mod tests {
|
|||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
};
|
||||
let r1 = map.allocate(entry.clone());
|
||||
let r2 = map.allocate(entry);
|
||||
|
|
@ -165,6 +251,7 @@ mod tests {
|
|||
bounds_hash: Some(12345),
|
||||
available_actions: vec![],
|
||||
source_app: Some("Finder".into()),
|
||||
root_ref: None,
|
||||
};
|
||||
let ref_id = map.allocate(entry);
|
||||
let retrieved = map.get(&ref_id).unwrap();
|
||||
|
|
@ -177,4 +264,238 @@ mod tests {
|
|||
let map = RefMap::new();
|
||||
assert!(map.get("@e99").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_remove_by_root_ref() {
|
||||
let mut map = RefMap::new();
|
||||
let base = RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some("OK".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
};
|
||||
|
||||
map.allocate(base.clone());
|
||||
|
||||
let drilled = RefEntry {
|
||||
root_ref: Some("@e1".into()),
|
||||
..base.clone()
|
||||
};
|
||||
map.allocate(drilled.clone());
|
||||
map.allocate(drilled);
|
||||
assert_eq!(map.len(), 3);
|
||||
|
||||
map.remove_by_root_ref("@e1");
|
||||
assert_eq!(map.len(), 1);
|
||||
assert!(map.get("@e1").is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_counter_continues_after_skeleton_into_drill_down() {
|
||||
let mut map = RefMap::new();
|
||||
let skeleton_entry = RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some("Skeleton".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec![],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
};
|
||||
|
||||
let last_skeleton = (0..10)
|
||||
.map(|_| map.allocate(skeleton_entry.clone()))
|
||||
.last()
|
||||
.unwrap();
|
||||
assert_eq!(last_skeleton, "@e10");
|
||||
|
||||
let drilled = RefEntry {
|
||||
root_ref: Some("@e3".into()),
|
||||
..skeleton_entry
|
||||
};
|
||||
|
||||
let first_drilled = map.allocate(drilled.clone());
|
||||
let second_drilled = map.allocate(drilled);
|
||||
assert_eq!(
|
||||
first_drilled, "@e11",
|
||||
"counter should continue past skeleton ids, not reset"
|
||||
);
|
||||
assert_eq!(second_drilled, "@e12");
|
||||
assert_eq!(map.len(), 12);
|
||||
|
||||
map.remove_by_root_ref("@e3");
|
||||
assert_eq!(
|
||||
map.len(),
|
||||
10,
|
||||
"scoped invalidation should drop only the drill-down refs"
|
||||
);
|
||||
assert!(map.get("@e3").is_some(), "skeleton @e3 must survive");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_root_ref_serde_roundtrip() {
|
||||
let entry = RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: None,
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec![],
|
||||
source_app: None,
|
||||
root_ref: Some("@e5".into()),
|
||||
};
|
||||
let json = serde_json::to_string(&entry).unwrap();
|
||||
assert!(json.contains("root_ref"));
|
||||
let back: RefEntry = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(back.root_ref.as_deref(), Some("@e5"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_serialize_with_size_check_rejects_oversized() {
|
||||
let mut map = RefMap::new();
|
||||
let big_name = "x".repeat(2048);
|
||||
for _ in 0..600 {
|
||||
map.allocate(RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some(big_name.clone()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
});
|
||||
}
|
||||
|
||||
let result = map.serialize_with_size_check();
|
||||
assert!(result.is_err(), "oversized refmap should be rejected");
|
||||
let err = result.unwrap_err();
|
||||
let msg = err.to_string();
|
||||
assert!(
|
||||
msg.contains("1MB"),
|
||||
"error should mention the 1MB limit, got: {msg}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_serialize_with_size_check_accepts_normal() {
|
||||
let mut map = RefMap::new();
|
||||
for _ in 0..50 {
|
||||
map.allocate(RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some("OK".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
});
|
||||
}
|
||||
|
||||
let result = map.serialize_with_size_check();
|
||||
assert!(result.is_ok(), "normal-sized refmap should serialize");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_save_load_roundtrip_with_home_override() {
|
||||
let _guard = HomeGuard::new();
|
||||
let mut map = RefMap::new();
|
||||
map.allocate(RefEntry {
|
||||
pid: 7,
|
||||
role: "button".into(),
|
||||
name: Some("Send".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: Some(42),
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: Some("TestApp".into()),
|
||||
root_ref: None,
|
||||
});
|
||||
map.save().expect("save should succeed under HomeGuard");
|
||||
|
||||
let loaded = RefMap::load().expect("load should succeed");
|
||||
assert_eq!(loaded.len(), 1);
|
||||
let entry = loaded.get("@e1").unwrap();
|
||||
assert_eq!(entry.pid, 7);
|
||||
assert_eq!(entry.name.as_deref(), Some("Send"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_save_oversize_preserves_previous_file() {
|
||||
let _guard = HomeGuard::new();
|
||||
|
||||
let mut original = RefMap::new();
|
||||
original.allocate(RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some("Original".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
});
|
||||
original.save().expect("baseline save");
|
||||
|
||||
let mut oversize = RefMap::new();
|
||||
let big = "x".repeat(2048);
|
||||
for _ in 0..600 {
|
||||
oversize.allocate(RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some(big.clone()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
});
|
||||
}
|
||||
let result = oversize.save();
|
||||
assert!(result.is_err(), "oversize save must reject");
|
||||
|
||||
let reloaded = RefMap::load().expect("previous file must still load");
|
||||
assert_eq!(reloaded.len(), 1);
|
||||
let entry = reloaded.get("@e1").unwrap();
|
||||
assert_eq!(entry.name.as_deref(), Some("Original"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_root_ref_none_omitted() {
|
||||
let entry = RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: None,
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec![],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
};
|
||||
let json = serde_json::to_string(&entry).unwrap();
|
||||
assert!(!json.contains("root_ref"));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,28 +2,10 @@ use crate::{
|
|||
adapter::{PlatformAdapter, SnapshotSurface, TreeOptions, WindowFilter},
|
||||
error::AppError,
|
||||
node::{AccessibilityNode, WindowInfo},
|
||||
refs::{RefEntry, RefMap},
|
||||
ref_alloc::{self, RefAllocConfig},
|
||||
refs::RefMap,
|
||||
};
|
||||
|
||||
const INTERACTIVE_ROLES: &[&str] = &[
|
||||
"button",
|
||||
"textfield",
|
||||
"checkbox",
|
||||
"link",
|
||||
"menuitem",
|
||||
"tab",
|
||||
"slider",
|
||||
"combobox",
|
||||
"treeitem",
|
||||
"cell",
|
||||
"radiobutton",
|
||||
"incrementor",
|
||||
"menubutton",
|
||||
"switch",
|
||||
"colorwell",
|
||||
"dockitem",
|
||||
];
|
||||
|
||||
pub struct SnapshotResult {
|
||||
pub tree: AccessibilityNode,
|
||||
pub refmap: RefMap,
|
||||
|
|
@ -94,15 +76,15 @@ pub fn build(
|
|||
let raw_tree = adapter.get_tree(&window, opts)?;
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let mut tree = allocate_refs(
|
||||
raw_tree,
|
||||
&mut refmap,
|
||||
opts.include_bounds,
|
||||
opts.interactive_only,
|
||||
opts.compact,
|
||||
window.pid,
|
||||
Some(window.app.as_str()),
|
||||
);
|
||||
let config = RefAllocConfig {
|
||||
include_bounds: opts.include_bounds,
|
||||
interactive_only: opts.interactive_only,
|
||||
compact: opts.compact,
|
||||
pid: window.pid,
|
||||
source_app: Some(window.app.as_str()),
|
||||
root_ref_id: None,
|
||||
};
|
||||
let mut tree = ref_alloc::allocate_refs(raw_tree, &mut refmap, &config);
|
||||
|
||||
crate::hints::add_structural_hints(&mut tree);
|
||||
|
||||
|
|
@ -143,212 +125,19 @@ pub fn append_surface_refs(
|
|||
};
|
||||
let raw_tree = adapter.get_tree(&window, &opts).ok()?;
|
||||
let mut refmap = RefMap::load().ok()?;
|
||||
let tree = allocate_refs(raw_tree, &mut refmap, false, true, false, pid, source_app);
|
||||
let config = RefAllocConfig {
|
||||
include_bounds: false,
|
||||
interactive_only: true,
|
||||
compact: false,
|
||||
pid,
|
||||
source_app,
|
||||
root_ref_id: None,
|
||||
};
|
||||
let tree = ref_alloc::allocate_refs(raw_tree, &mut refmap, &config);
|
||||
refmap.save().ok()?;
|
||||
Some(tree)
|
||||
}
|
||||
|
||||
fn is_collapsible(node: &AccessibilityNode) -> bool {
|
||||
node.ref_id.is_none()
|
||||
&& node.name.as_deref().is_none_or(str::is_empty)
|
||||
&& node.value.as_deref().is_none_or(str::is_empty)
|
||||
&& node.description.as_deref().is_none_or(str::is_empty)
|
||||
&& node.states.is_empty()
|
||||
&& node.children.len() == 1
|
||||
}
|
||||
|
||||
fn allocate_refs(
|
||||
mut node: AccessibilityNode,
|
||||
refmap: &mut RefMap,
|
||||
include_bounds: bool,
|
||||
interactive_only: bool,
|
||||
compact: bool,
|
||||
window_pid: i32,
|
||||
source_app: Option<&str>,
|
||||
) -> AccessibilityNode {
|
||||
let is_interactive = INTERACTIVE_ROLES.contains(&node.role.as_str());
|
||||
|
||||
if is_interactive {
|
||||
let entry = RefEntry {
|
||||
pid: window_pid,
|
||||
role: node.role.clone(),
|
||||
name: node.name.clone(),
|
||||
value: node.value.clone(),
|
||||
states: node.states.clone(),
|
||||
bounds: node.bounds,
|
||||
bounds_hash: node.bounds.as_ref().map(|b| b.bounds_hash()),
|
||||
available_actions: actions_for_role(&node.role),
|
||||
source_app: source_app.map(str::to_string),
|
||||
};
|
||||
node.ref_id = Some(refmap.allocate(entry));
|
||||
}
|
||||
|
||||
if !include_bounds {
|
||||
node.bounds = None;
|
||||
}
|
||||
|
||||
node.children = node
|
||||
.children
|
||||
.into_iter()
|
||||
.filter_map(|child| {
|
||||
let child = allocate_refs(
|
||||
child,
|
||||
refmap,
|
||||
include_bounds,
|
||||
interactive_only,
|
||||
compact,
|
||||
window_pid,
|
||||
source_app,
|
||||
);
|
||||
if compact && is_collapsible(&child) {
|
||||
return child.children.into_iter().next();
|
||||
}
|
||||
if interactive_only && child.ref_id.is_none() && child.children.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(child)
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
|
||||
node
|
||||
}
|
||||
|
||||
fn actions_for_role(role: &str) -> Vec<String> {
|
||||
match role {
|
||||
"button" | "link" | "menuitem" | "tab" | "radiobutton" => vec!["Click".into()],
|
||||
"textfield" | "incrementor" => vec!["Click".into(), "SetValue".into(), "SetFocus".into()],
|
||||
"checkbox" => vec!["Click".into(), "Toggle".into()],
|
||||
"combobox" => vec!["Click".into(), "Select".into()],
|
||||
"treeitem" => vec!["Click".into(), "Expand".into(), "Collapse".into()],
|
||||
"slider" => vec!["SetValue".into()],
|
||||
"cell" => vec!["Click".into()],
|
||||
_ => vec!["Click".into()],
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::node::AccessibilityNode;
|
||||
|
||||
fn node(role: &str) -> AccessibilityNode {
|
||||
AccessibilityNode {
|
||||
ref_id: None,
|
||||
role: role.into(),
|
||||
name: None,
|
||||
value: None,
|
||||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
children: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
fn run_compact(tree: AccessibilityNode) -> AccessibilityNode {
|
||||
let mut refmap = RefMap::new();
|
||||
allocate_refs(tree, &mut refmap, false, false, true, 1, Some("Test"))
|
||||
}
|
||||
|
||||
fn run_compact_interactive(tree: AccessibilityNode) -> AccessibilityNode {
|
||||
let mut refmap = RefMap::new();
|
||||
allocate_refs(tree, &mut refmap, false, true, true, 1, Some("Test"))
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_compact_collapses_single_child_chain() {
|
||||
let mut btn = node("button");
|
||||
btn.name = Some("Send".into());
|
||||
let mut g1 = node("group");
|
||||
g1.children = vec![btn];
|
||||
let mut g2 = node("group");
|
||||
g2.children = vec![g1];
|
||||
let mut root = node("window");
|
||||
root.children = vec![g2];
|
||||
|
||||
let result = run_compact(root);
|
||||
assert_eq!(result.role, "window");
|
||||
assert_eq!(result.children.len(), 1);
|
||||
assert_eq!(result.children[0].role, "button");
|
||||
assert_eq!(result.children[0].name.as_deref(), Some("Send"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_compact_preserves_named_containers() {
|
||||
let btn = node("button");
|
||||
let mut named = node("group");
|
||||
named.name = Some("Sidebar".into());
|
||||
named.children = vec![btn];
|
||||
let mut root = node("window");
|
||||
root.children = vec![named];
|
||||
|
||||
let result = run_compact(root);
|
||||
assert_eq!(result.children.len(), 1);
|
||||
assert_eq!(result.children[0].role, "group");
|
||||
assert_eq!(result.children[0].name.as_deref(), Some("Sidebar"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_compact_preserves_description() {
|
||||
let btn = node("button");
|
||||
let mut desc_node = node("group");
|
||||
desc_node.description = Some("toolbar".into());
|
||||
desc_node.children = vec![btn];
|
||||
let mut root = node("window");
|
||||
root.children = vec![desc_node];
|
||||
|
||||
let result = run_compact(root);
|
||||
assert_eq!(result.children.len(), 1);
|
||||
assert_eq!(result.children[0].role, "group");
|
||||
assert_eq!(result.children[0].description.as_deref(), Some("toolbar"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_compact_preserves_states() {
|
||||
let btn = node("button");
|
||||
let mut disabled = node("group");
|
||||
disabled.states = vec!["disabled".into()];
|
||||
disabled.children = vec![btn];
|
||||
let mut root = node("window");
|
||||
root.children = vec![disabled];
|
||||
|
||||
let result = run_compact(root);
|
||||
assert_eq!(result.children.len(), 1);
|
||||
assert_eq!(result.children[0].role, "group");
|
||||
assert_eq!(result.children[0].states, vec!["disabled"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_compact_preserves_multi_child() {
|
||||
let btn = node("button");
|
||||
let tf = node("textfield");
|
||||
let mut group = node("group");
|
||||
group.children = vec![btn, tf];
|
||||
let mut root = node("window");
|
||||
root.children = vec![group];
|
||||
|
||||
let result = run_compact(root);
|
||||
assert_eq!(result.children.len(), 1);
|
||||
assert_eq!(result.children[0].role, "group");
|
||||
assert_eq!(result.children[0].children.len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_compact_with_interactive_only() {
|
||||
let mut btn = node("button");
|
||||
btn.name = Some("OK".into());
|
||||
let text = node("statictext");
|
||||
let mut g1 = node("group");
|
||||
g1.children = vec![btn];
|
||||
let mut g2 = node("group");
|
||||
g2.children = vec![text];
|
||||
let mut root = node("window");
|
||||
root.children = vec![g1, g2];
|
||||
|
||||
let result = run_compact_interactive(root);
|
||||
assert_eq!(result.children.len(), 1);
|
||||
assert_eq!(result.children[0].role, "button");
|
||||
assert!(result.children[0].ref_id.is_some());
|
||||
}
|
||||
}
|
||||
#[path = "snapshot_tests.rs"]
|
||||
mod tests;
|
||||
|
|
|
|||
69
crates/core/src/snapshot_ref.rs
Normal file
69
crates/core/src/snapshot_ref.rs
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
use crate::{
|
||||
adapter::{PlatformAdapter, TreeOptions, WindowFilter},
|
||||
error::AppError,
|
||||
node::WindowInfo,
|
||||
ref_alloc::{self, RefAllocConfig},
|
||||
refs::RefMap,
|
||||
snapshot::SnapshotResult,
|
||||
};
|
||||
|
||||
pub fn run_from_ref(
|
||||
adapter: &dyn PlatformAdapter,
|
||||
opts: &TreeOptions,
|
||||
root_ref_id: &str,
|
||||
) -> Result<SnapshotResult, AppError> {
|
||||
let mut refmap = RefMap::load()?;
|
||||
|
||||
let entry = refmap
|
||||
.get(root_ref_id)
|
||||
.ok_or_else(|| AppError::stale_ref(root_ref_id))?
|
||||
.clone();
|
||||
|
||||
let handle = adapter.resolve_element(&entry)?;
|
||||
|
||||
let raw_tree = adapter.get_subtree(&handle, opts)?;
|
||||
|
||||
refmap.remove_by_root_ref(root_ref_id);
|
||||
|
||||
let source_app = entry.source_app.as_deref();
|
||||
let config = RefAllocConfig {
|
||||
include_bounds: opts.include_bounds,
|
||||
interactive_only: opts.interactive_only,
|
||||
compact: opts.compact,
|
||||
pid: entry.pid,
|
||||
source_app,
|
||||
root_ref_id: Some(root_ref_id),
|
||||
};
|
||||
|
||||
let mut tree = ref_alloc::allocate_refs(raw_tree, &mut refmap, &config);
|
||||
|
||||
crate::hints::add_structural_hints(&mut tree);
|
||||
|
||||
refmap.save()?;
|
||||
|
||||
let window = adapter
|
||||
.list_windows(&WindowFilter {
|
||||
focused_only: false,
|
||||
app: None,
|
||||
})
|
||||
.ok()
|
||||
.and_then(|ws| ws.into_iter().find(|w| w.pid == entry.pid))
|
||||
.unwrap_or(WindowInfo {
|
||||
id: String::new(),
|
||||
title: format!("subtree from {root_ref_id}"),
|
||||
app: entry.source_app.unwrap_or_default(),
|
||||
pid: entry.pid,
|
||||
bounds: None,
|
||||
is_focused: true,
|
||||
});
|
||||
|
||||
Ok(SnapshotResult {
|
||||
tree,
|
||||
refmap,
|
||||
window,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "snapshot_ref_tests.rs"]
|
||||
mod tests;
|
||||
364
crates/core/src/snapshot_ref_tests.rs
Normal file
364
crates/core/src/snapshot_ref_tests.rs
Normal file
|
|
@ -0,0 +1,364 @@
|
|||
use super::*;
|
||||
use crate::action::Action;
|
||||
use crate::adapter::{NativeHandle, PermissionStatus, PlatformAdapter};
|
||||
use crate::error::AdapterError;
|
||||
use crate::node::AccessibilityNode;
|
||||
use crate::ref_alloc::ref_entry_from_node;
|
||||
use crate::refs::HomeGuard;
|
||||
use std::cell::Cell;
|
||||
|
||||
fn node(role: &str) -> AccessibilityNode {
|
||||
AccessibilityNode {
|
||||
ref_id: None,
|
||||
role: role.into(),
|
||||
name: None,
|
||||
value: None,
|
||||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
children_count: None,
|
||||
children: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
fn named(role: &str, name: &str) -> AccessibilityNode {
|
||||
let mut n = node(role);
|
||||
n.name = Some(name.into());
|
||||
n
|
||||
}
|
||||
|
||||
struct StubAdapter {
|
||||
subtree: AccessibilityNode,
|
||||
resolve_calls: Cell<u32>,
|
||||
}
|
||||
|
||||
impl StubAdapter {
|
||||
fn new(subtree: AccessibilityNode) -> Self {
|
||||
Self {
|
||||
subtree,
|
||||
resolve_calls: Cell::new(0),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
unsafe impl Send for StubAdapter {}
|
||||
unsafe impl Sync for StubAdapter {}
|
||||
|
||||
impl PlatformAdapter for StubAdapter {
|
||||
fn check_permissions(&self) -> PermissionStatus {
|
||||
PermissionStatus::Granted
|
||||
}
|
||||
|
||||
fn resolve_element(
|
||||
&self,
|
||||
_entry: &crate::refs::RefEntry,
|
||||
) -> Result<NativeHandle, AdapterError> {
|
||||
self.resolve_calls.set(self.resolve_calls.get() + 1);
|
||||
Ok(NativeHandle::null())
|
||||
}
|
||||
|
||||
fn get_subtree(
|
||||
&self,
|
||||
_handle: &NativeHandle,
|
||||
_opts: &TreeOptions,
|
||||
) -> Result<AccessibilityNode, AdapterError> {
|
||||
Ok(self.subtree.clone())
|
||||
}
|
||||
|
||||
fn execute_action(
|
||||
&self,
|
||||
_handle: &NativeHandle,
|
||||
_action: Action,
|
||||
) -> Result<crate::action::ActionResult, AdapterError> {
|
||||
Err(AdapterError::not_supported("execute_action"))
|
||||
}
|
||||
}
|
||||
|
||||
fn seed_skeleton_refmap() -> RefMap {
|
||||
let mut map = RefMap::new();
|
||||
let anchor = ref_entry_from_node(&named("group", "Sidebar"), 42, Some("TestApp"), None);
|
||||
let _ = map.allocate(anchor);
|
||||
let other = ref_entry_from_node(&named("button", "Toolbar"), 42, Some("TestApp"), None);
|
||||
let _ = map.allocate(other);
|
||||
map
|
||||
}
|
||||
|
||||
fn drill_opts() -> TreeOptions {
|
||||
TreeOptions {
|
||||
interactive_only: false,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_run_from_ref_returns_subtree_and_persists_refs() {
|
||||
let _guard = HomeGuard::new();
|
||||
seed_skeleton_refmap().save().unwrap();
|
||||
|
||||
let mut child_btn = named("button", "Save");
|
||||
child_btn.children = vec![];
|
||||
let mut subtree_root = named("group", "Sidebar");
|
||||
subtree_root.children = vec![child_btn];
|
||||
|
||||
let adapter = StubAdapter::new(subtree_root);
|
||||
let result = run_from_ref(&adapter, &drill_opts(), "@e1").expect("drill should succeed");
|
||||
|
||||
let on_disk = RefMap::load().unwrap();
|
||||
assert_eq!(on_disk.len(), result.refmap.len());
|
||||
assert!(
|
||||
result.refmap.len() >= 3,
|
||||
"expected at least 2 skeleton + 1 drill ref, got {}",
|
||||
result.refmap.len()
|
||||
);
|
||||
|
||||
let drill_ref = result
|
||||
.tree
|
||||
.children
|
||||
.iter()
|
||||
.find(|c| c.role == "button")
|
||||
.and_then(|c| c.ref_id.as_deref())
|
||||
.expect("button child should carry a ref");
|
||||
let drill_entry = on_disk.get(drill_ref).expect("entry persisted");
|
||||
assert_eq!(drill_entry.root_ref.as_deref(), Some("@e1"));
|
||||
assert_eq!(adapter.resolve_calls.get(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_run_from_ref_stale_root_returns_stale_ref() {
|
||||
let _guard = HomeGuard::new();
|
||||
RefMap::new().save().unwrap();
|
||||
|
||||
let adapter = StubAdapter::new(named("group", "Sidebar"));
|
||||
let result = run_from_ref(&adapter, &drill_opts(), "@e99");
|
||||
let err = match result {
|
||||
Ok(_) => panic!("stale root must error"),
|
||||
Err(e) => e,
|
||||
};
|
||||
match err {
|
||||
AppError::Adapter(adapter_err) => {
|
||||
assert_eq!(adapter_err.code, crate::error::ErrorCode::StaleRef);
|
||||
let suggestion = adapter_err.suggestion.as_deref().unwrap_or("");
|
||||
assert!(
|
||||
suggestion.contains("skeleton"),
|
||||
"stale-ref suggestion should mention skeleton, got: {suggestion}"
|
||||
);
|
||||
}
|
||||
other => panic!("expected Adapter(StaleRef), got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_run_from_ref_re_drill_replaces_drill_refs_only() {
|
||||
let _guard = HomeGuard::new();
|
||||
seed_skeleton_refmap().save().unwrap();
|
||||
|
||||
let subtree = named("button", "Save");
|
||||
let adapter = StubAdapter::new(subtree);
|
||||
|
||||
let first = run_from_ref(&adapter, &drill_opts(), "@e1").unwrap();
|
||||
let first_count = first.refmap.len();
|
||||
let first_button_ref = first.tree.ref_id.clone().expect("button should get a ref");
|
||||
|
||||
let second = run_from_ref(&adapter, &drill_opts(), "@e1").unwrap();
|
||||
let second_count = second.refmap.len();
|
||||
let second_button_ref = second.tree.ref_id.clone().expect("button should get a ref");
|
||||
|
||||
assert_eq!(
|
||||
first_count, second_count,
|
||||
"ref count stable across re-drill"
|
||||
);
|
||||
assert_ne!(
|
||||
first_button_ref, second_button_ref,
|
||||
"re-drill should issue a fresh ref id (counter continues)"
|
||||
);
|
||||
let on_disk = RefMap::load().unwrap();
|
||||
assert!(on_disk.get("@e1").is_some(), "skeleton anchor preserved");
|
||||
assert!(on_disk.get(&second_button_ref).is_some());
|
||||
assert!(
|
||||
on_disk.get(&first_button_ref).is_none(),
|
||||
"first drill ref must be invalidated by remove_by_root_ref"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_run_from_ref_multiple_drill_downs_accumulate() {
|
||||
let _guard = HomeGuard::new();
|
||||
seed_skeleton_refmap().save().unwrap();
|
||||
|
||||
let adapter_one = StubAdapter::new(named("button", "FromE1"));
|
||||
let first = run_from_ref(&adapter_one, &drill_opts(), "@e1").unwrap();
|
||||
let from_e1_ref = first.tree.ref_id.clone().expect("first drill ref");
|
||||
|
||||
let adapter_two = StubAdapter::new(named("button", "FromE2"));
|
||||
let second = run_from_ref(&adapter_two, &drill_opts(), "@e2").unwrap();
|
||||
let from_e2_ref = second.tree.ref_id.clone().expect("second drill ref");
|
||||
|
||||
let on_disk = RefMap::load().unwrap();
|
||||
assert!(on_disk.get("@e1").is_some(), "skeleton @e1 preserved");
|
||||
assert!(on_disk.get("@e2").is_some(), "skeleton @e2 preserved");
|
||||
let entry_one = on_disk.get(&from_e1_ref).expect("@e1 drill survives");
|
||||
assert_eq!(entry_one.root_ref.as_deref(), Some("@e1"));
|
||||
let entry_two = on_disk.get(&from_e2_ref).expect("@e2 drill survives");
|
||||
assert_eq!(entry_two.root_ref.as_deref(), Some("@e2"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_drilldown_refmap_matches_golden_fixture() {
|
||||
let golden = include_str!("../../../tests/fixtures/drilldown-refmap.json");
|
||||
let golden_value: serde_json::Value = serde_json::from_str(golden).unwrap();
|
||||
let expected_total = golden_value["expected_total"].as_u64().unwrap() as usize;
|
||||
|
||||
let _guard = HomeGuard::new();
|
||||
let mut seed = RefMap::new();
|
||||
seed.allocate(ref_entry_from_node(
|
||||
&named("group", "Sidebar"),
|
||||
42,
|
||||
Some("Fixture"),
|
||||
None,
|
||||
));
|
||||
seed.allocate(ref_entry_from_node(
|
||||
&named("group", "Toolbar"),
|
||||
42,
|
||||
Some("Fixture"),
|
||||
None,
|
||||
));
|
||||
seed.save().unwrap();
|
||||
|
||||
let mut sidebar_subtree = named("outline", "Sidebar");
|
||||
sidebar_subtree.children = vec![named("treeitem", "Recents"), named("treeitem", "Documents")];
|
||||
let adapter = StubAdapter::new(sidebar_subtree);
|
||||
let _ = run_from_ref(&adapter, &drill_opts(), "@e1").unwrap();
|
||||
|
||||
let toolbar_subtree = named("button", "Back");
|
||||
let adapter = StubAdapter::new(toolbar_subtree);
|
||||
let _ = run_from_ref(&adapter, &drill_opts(), "@e2").unwrap();
|
||||
|
||||
let on_disk = RefMap::load().unwrap();
|
||||
assert_eq!(
|
||||
on_disk.len(),
|
||||
expected_total,
|
||||
"merged refmap should match golden fixture's expected_total"
|
||||
);
|
||||
|
||||
for anchor in golden_value["skeleton_anchors"].as_array().unwrap() {
|
||||
let id = anchor["ref_id"].as_str().unwrap();
|
||||
let entry = on_disk.get(id).unwrap_or_else(|| panic!("missing {id}"));
|
||||
assert_eq!(entry.role, anchor["role"].as_str().unwrap());
|
||||
assert_eq!(entry.name.as_deref(), anchor["name"].as_str());
|
||||
assert!(
|
||||
entry.root_ref.is_none(),
|
||||
"skeleton {id} must have null root_ref"
|
||||
);
|
||||
}
|
||||
|
||||
for drill in golden_value["drilled_from_e1"].as_array().unwrap() {
|
||||
let id = drill["ref_id"].as_str().unwrap();
|
||||
if let Some(entry) = on_disk.get(id) {
|
||||
assert_eq!(entry.root_ref.as_deref(), Some("@e1"));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_run_from_ref_empty_subtree() {
|
||||
let _guard = HomeGuard::new();
|
||||
seed_skeleton_refmap().save().unwrap();
|
||||
|
||||
let adapter = StubAdapter::new(node("group"));
|
||||
let result = run_from_ref(&adapter, &drill_opts(), "@e1").unwrap();
|
||||
|
||||
assert!(result.tree.children.is_empty());
|
||||
assert_eq!(
|
||||
result.refmap.len(),
|
||||
2,
|
||||
"no new refs added for empty subtree"
|
||||
);
|
||||
}
|
||||
|
||||
fn drill_config<'a>(
|
||||
source_app: Option<&'a str>,
|
||||
pid: i32,
|
||||
root_ref_id: &'a str,
|
||||
interactive_only: bool,
|
||||
compact: bool,
|
||||
) -> RefAllocConfig<'a> {
|
||||
RefAllocConfig {
|
||||
include_bounds: false,
|
||||
interactive_only,
|
||||
compact,
|
||||
pid,
|
||||
source_app,
|
||||
root_ref_id: Some(root_ref_id),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_drill_alloc_tags_entries() {
|
||||
let mut btn = node("button");
|
||||
btn.name = Some("Submit".into());
|
||||
let mut root = node("group");
|
||||
root.children = vec![btn];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = drill_config(Some("TestApp"), 42, "@e5", false, false);
|
||||
let tree = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert_eq!(refmap.len(), 1);
|
||||
let btn_ref = tree.children[0]
|
||||
.ref_id
|
||||
.as_deref()
|
||||
.expect("button should have ref");
|
||||
let entry = refmap.get(btn_ref).expect("entry should exist");
|
||||
assert_eq!(entry.root_ref.as_deref(), Some("@e5"));
|
||||
assert_eq!(entry.pid, 42);
|
||||
assert_eq!(entry.source_app.as_deref(), Some("TestApp"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_drill_alloc_respects_interactive_only() {
|
||||
let btn = node("button");
|
||||
let text = node("statictext");
|
||||
let mut root = node("group");
|
||||
root.children = vec![btn, text];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = drill_config(None, 1, "@e1", true, false);
|
||||
let tree = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert_eq!(tree.children.len(), 1);
|
||||
assert_eq!(tree.children[0].role, "button");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_drill_alloc_preserves_truncated_child() {
|
||||
let mut container = node("group");
|
||||
container.name = Some("Sidebar".into());
|
||||
container.children_count = Some(4);
|
||||
let mut root = node("window");
|
||||
root.children = vec![container];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = drill_config(None, 1, "@e1", true, false);
|
||||
let tree = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert_eq!(tree.children.len(), 1);
|
||||
assert_eq!(tree.children[0].children_count, Some(4));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_drill_alloc_compact() {
|
||||
let mut btn = node("button");
|
||||
btn.name = Some("OK".into());
|
||||
let mut wrapper = node("group");
|
||||
wrapper.children = vec![btn];
|
||||
let mut root = node("window");
|
||||
root.children = vec![wrapper];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = drill_config(None, 1, "@e1", false, true);
|
||||
let tree = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert_eq!(tree.children.len(), 1);
|
||||
assert_eq!(tree.children[0].role, "button");
|
||||
}
|
||||
288
crates/core/src/snapshot_tests.rs
Normal file
288
crates/core/src/snapshot_tests.rs
Normal file
|
|
@ -0,0 +1,288 @@
|
|||
use super::*;
|
||||
use crate::node::AccessibilityNode;
|
||||
|
||||
fn node(role: &str) -> AccessibilityNode {
|
||||
AccessibilityNode {
|
||||
ref_id: None,
|
||||
role: role.into(),
|
||||
name: None,
|
||||
value: None,
|
||||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
children_count: None,
|
||||
children: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
fn run_config(compact: bool, interactive_only: bool) -> RefAllocConfig<'static> {
|
||||
RefAllocConfig {
|
||||
include_bounds: false,
|
||||
interactive_only,
|
||||
compact,
|
||||
pid: 1,
|
||||
source_app: Some("Test"),
|
||||
root_ref_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn run_compact(tree: AccessibilityNode) -> AccessibilityNode {
|
||||
let mut refmap = RefMap::new();
|
||||
ref_alloc::allocate_refs(tree, &mut refmap, &run_config(true, false))
|
||||
}
|
||||
|
||||
fn run_compact_interactive(tree: AccessibilityNode) -> AccessibilityNode {
|
||||
let mut refmap = RefMap::new();
|
||||
ref_alloc::allocate_refs(tree, &mut refmap, &run_config(true, true))
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_compact_collapses_single_child_chain() {
|
||||
let mut btn = node("button");
|
||||
btn.name = Some("Send".into());
|
||||
let mut g1 = node("group");
|
||||
g1.children = vec![btn];
|
||||
let mut g2 = node("group");
|
||||
g2.children = vec![g1];
|
||||
let mut root = node("window");
|
||||
root.children = vec![g2];
|
||||
|
||||
let result = run_compact(root);
|
||||
assert_eq!(result.role, "window");
|
||||
assert_eq!(result.children.len(), 1);
|
||||
assert_eq!(result.children[0].role, "button");
|
||||
assert_eq!(result.children[0].name.as_deref(), Some("Send"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_compact_preserves_named_containers() {
|
||||
let btn = node("button");
|
||||
let mut named = node("group");
|
||||
named.name = Some("Sidebar".into());
|
||||
named.children = vec![btn];
|
||||
let mut root = node("window");
|
||||
root.children = vec![named];
|
||||
|
||||
let result = run_compact(root);
|
||||
assert_eq!(result.children.len(), 1);
|
||||
assert_eq!(result.children[0].role, "group");
|
||||
assert_eq!(result.children[0].name.as_deref(), Some("Sidebar"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_compact_preserves_description() {
|
||||
let btn = node("button");
|
||||
let mut desc_node = node("group");
|
||||
desc_node.description = Some("toolbar".into());
|
||||
desc_node.children = vec![btn];
|
||||
let mut root = node("window");
|
||||
root.children = vec![desc_node];
|
||||
|
||||
let result = run_compact(root);
|
||||
assert_eq!(result.children.len(), 1);
|
||||
assert_eq!(result.children[0].role, "group");
|
||||
assert_eq!(result.children[0].description.as_deref(), Some("toolbar"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_compact_preserves_states() {
|
||||
let btn = node("button");
|
||||
let mut disabled = node("group");
|
||||
disabled.states = vec!["disabled".into()];
|
||||
disabled.children = vec![btn];
|
||||
let mut root = node("window");
|
||||
root.children = vec![disabled];
|
||||
|
||||
let result = run_compact(root);
|
||||
assert_eq!(result.children.len(), 1);
|
||||
assert_eq!(result.children[0].role, "group");
|
||||
assert_eq!(result.children[0].states, vec!["disabled"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_compact_preserves_multi_child() {
|
||||
let btn = node("button");
|
||||
let tf = node("textfield");
|
||||
let mut group = node("group");
|
||||
group.children = vec![btn, tf];
|
||||
let mut root = node("window");
|
||||
root.children = vec![group];
|
||||
|
||||
let result = run_compact(root);
|
||||
assert_eq!(result.children.len(), 1);
|
||||
assert_eq!(result.children[0].role, "group");
|
||||
assert_eq!(result.children[0].children.len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_compact_with_interactive_only() {
|
||||
let mut btn = node("button");
|
||||
btn.name = Some("OK".into());
|
||||
let text = node("statictext");
|
||||
let mut g1 = node("group");
|
||||
g1.children = vec![btn];
|
||||
let mut g2 = node("group");
|
||||
g2.children = vec![text];
|
||||
let mut root = node("window");
|
||||
root.children = vec![g1, g2];
|
||||
|
||||
let result = run_compact_interactive(root);
|
||||
assert_eq!(result.children.len(), 1);
|
||||
assert_eq!(result.children[0].role, "button");
|
||||
assert!(result.children[0].ref_id.is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_skeleton_named_container_gets_ref() {
|
||||
let mut container = node("group");
|
||||
container.name = Some("Sidebar".into());
|
||||
container.children_count = Some(5);
|
||||
let mut root = node("window");
|
||||
root.children = vec![container];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let result = ref_alloc::allocate_refs(root, &mut refmap, &run_config(false, false));
|
||||
|
||||
assert!(result.children[0].ref_id.is_some());
|
||||
assert_eq!(refmap.len(), 1);
|
||||
let entry = refmap
|
||||
.get(result.children[0].ref_id.as_deref().unwrap())
|
||||
.unwrap();
|
||||
assert!(entry.available_actions.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_skeleton_unnamed_container_no_ref() {
|
||||
let mut container = node("group");
|
||||
container.children_count = Some(5);
|
||||
let mut root = node("window");
|
||||
root.children = vec![container];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let result = ref_alloc::allocate_refs(root, &mut refmap, &run_config(false, false));
|
||||
|
||||
assert!(result.children[0].ref_id.is_none());
|
||||
assert_eq!(refmap.len(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_skeleton_anchor_suppressed_in_drilldown() {
|
||||
let mut anchor = node("group");
|
||||
anchor.name = Some("Channels".into());
|
||||
anchor.children_count = Some(8);
|
||||
let mut root = node("group");
|
||||
root.children = vec![anchor];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = RefAllocConfig {
|
||||
include_bounds: false,
|
||||
interactive_only: false,
|
||||
compact: false,
|
||||
pid: 1,
|
||||
source_app: Some("Test"),
|
||||
root_ref_id: Some("@e3"),
|
||||
};
|
||||
let result = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert!(
|
||||
result.children[0].ref_id.is_none(),
|
||||
"skeleton anchors must not be created during drill-down to prevent orphaned ref accumulation"
|
||||
);
|
||||
assert_eq!(refmap.len(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_skeleton_described_container_gets_ref() {
|
||||
let mut container = node("group");
|
||||
container.description = Some("Channels and direct messages".into());
|
||||
container.children_count = Some(12);
|
||||
let mut root = node("window");
|
||||
root.children = vec![container];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let result = ref_alloc::allocate_refs(root, &mut refmap, &run_config(false, false));
|
||||
|
||||
assert!(result.children[0].ref_id.is_some());
|
||||
assert_eq!(refmap.len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_skeleton_truncated_node_survives_interactive_only() {
|
||||
let mut container = node("group");
|
||||
container.name = Some("Content".into());
|
||||
container.children_count = Some(10);
|
||||
let mut root = node("window");
|
||||
root.children = vec![container];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let result = ref_alloc::allocate_refs(root, &mut refmap, &run_config(false, true));
|
||||
|
||||
assert_eq!(result.children.len(), 1);
|
||||
assert_eq!(result.children[0].children_count, Some(10));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_skeleton_fixture_matches_golden() {
|
||||
let golden = include_str!("../../../tests/fixtures/skeleton-tree.json");
|
||||
let golden_value: serde_json::Value = serde_json::from_str(golden).unwrap();
|
||||
|
||||
let mut sidebar = node("group");
|
||||
sidebar.name = Some("Sidebar".into());
|
||||
sidebar.children_count = Some(26);
|
||||
|
||||
let mut described = node("group");
|
||||
described.description = Some("Channels and direct messages".into());
|
||||
described.children_count = Some(12);
|
||||
|
||||
let mut send = node("button");
|
||||
send.name = Some("Send".into());
|
||||
let mut msg = node("textfield");
|
||||
msg.name = Some("Message".into());
|
||||
let mut content = node("group");
|
||||
content.name = Some("Content".into());
|
||||
content.children = vec![send, msg];
|
||||
|
||||
let mut root = node("window");
|
||||
root.name = Some("Test Window".into());
|
||||
root.children = vec![sidebar, described, content];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = RefAllocConfig {
|
||||
include_bounds: false,
|
||||
interactive_only: false,
|
||||
compact: false,
|
||||
pid: 42,
|
||||
source_app: Some("Fixture"),
|
||||
root_ref_id: None,
|
||||
};
|
||||
let result = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert_eq!(refmap.len(), 4, "should allocate 4 refs total");
|
||||
let result_value = serde_json::to_value(&result).unwrap();
|
||||
|
||||
assert_eq!(result_value["role"], golden_value["role"]);
|
||||
assert_eq!(result_value["name"], golden_value["name"]);
|
||||
assert_eq!(
|
||||
result_value["children"][0]["ref_id"], golden_value["children"][0]["ref_id"],
|
||||
"named skeleton anchor should be @e1"
|
||||
);
|
||||
assert_eq!(
|
||||
result_value["children"][0]["children_count"],
|
||||
golden_value["children"][0]["children_count"]
|
||||
);
|
||||
assert_eq!(
|
||||
result_value["children"][1]["ref_id"], golden_value["children"][1]["ref_id"],
|
||||
"described skeleton anchor should be @e2"
|
||||
);
|
||||
assert_eq!(
|
||||
result_value["children"][2]["children"][0]["ref_id"],
|
||||
golden_value["children"][2]["children"][0]["ref_id"],
|
||||
"interactive button should be @e3"
|
||||
);
|
||||
assert_eq!(
|
||||
result_value["children"][2]["children"][1]["ref_id"],
|
||||
golden_value["children"][2]["children"][1]["ref_id"],
|
||||
"interactive textfield should be @e4"
|
||||
);
|
||||
}
|
||||
|
|
@ -64,7 +64,11 @@ mod imp {
|
|||
let deadline = Instant::now() + CHAIN_TIMEOUT;
|
||||
let total = def.steps.len();
|
||||
|
||||
ax_helpers::set_messaging_timeout(el, 3.0);
|
||||
if let Some(pid) = crate::system::app_ops::pid_from_element(el) {
|
||||
ax_helpers::set_messaging_timeout(&crate::tree::element_for_pid(pid), 1.0);
|
||||
}
|
||||
ax_helpers::set_messaging_timeout(el, 1.0);
|
||||
|
||||
if def.pre_scroll {
|
||||
tracing::debug!("chain: pre-scroll AXScrollToVisible");
|
||||
ax_helpers::ensure_visible(el);
|
||||
|
|
@ -72,7 +76,17 @@ mod imp {
|
|||
|
||||
for (i, step) in def.steps.iter().enumerate() {
|
||||
if Instant::now() > deadline {
|
||||
tracing::debug!("chain: timeout after {i}/{total} steps");
|
||||
tracing::debug!("chain: timeout after {i}/{total} steps, trying CGClick fallback");
|
||||
if let Some(cg) = def
|
||||
.steps
|
||||
.iter()
|
||||
.find(|s| matches!(s, ChainStep::CGClick { .. }))
|
||||
{
|
||||
if execute_step(el, caps, cg, ctx) {
|
||||
tracing::debug!("chain: CGClick fallback succeeded");
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
return Err(AdapterError::timeout("Chain execution exceeded 10s"));
|
||||
}
|
||||
let label = step_label(step);
|
||||
|
|
|
|||
|
|
@ -3,7 +3,21 @@ mod imp {
|
|||
use crate::actions::{ax_helpers, discovery::ElementCaps};
|
||||
use crate::tree::AXElement;
|
||||
|
||||
pub fn do_verified_press(el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
pub fn do_verified_press(el: &AXElement, caps: &ElementCaps) -> bool {
|
||||
dispatch_verified_press(el, caps, is_in_webarea(el))
|
||||
}
|
||||
|
||||
fn dispatch_verified_press(el: &AXElement, _caps: &ElementCaps, in_web: bool) -> bool {
|
||||
if !in_web {
|
||||
return verified_press_native(el);
|
||||
}
|
||||
tracing::debug!("verified_press: web element detected");
|
||||
activate_web_element(el)
|
||||
}
|
||||
|
||||
/// Native (non-web) elements: AXPress with selection verification
|
||||
/// for elements inside selection containers.
|
||||
fn verified_press_native(el: &AXElement) -> bool {
|
||||
use accessibility_sys::kAXRoleAttribute;
|
||||
let parent = crate::tree::copy_element_attr(el, "AXParent");
|
||||
let in_container = parent.as_ref().is_some_and(|p| {
|
||||
|
|
@ -15,6 +29,7 @@ mod imp {
|
|||
if !in_container {
|
||||
return ax_helpers::try_ax_action_retried(el, "AXPress");
|
||||
}
|
||||
tracing::debug!("verified_press: native element in container, using AXPress");
|
||||
let selected_before = crate::tree::element::copy_bool_attr(el, "AXSelected");
|
||||
if !ax_helpers::try_ax_action_retried(el, "AXPress") {
|
||||
return false;
|
||||
|
|
@ -30,19 +45,144 @@ mod imp {
|
|||
if crate::tree::copy_string_attr(el, kAXRoleAttribute).is_none() {
|
||||
return true;
|
||||
}
|
||||
tracing::debug!("verified_press: AXPress ok but no state change in selection container");
|
||||
tracing::debug!("verified_press: AXPress ok but no state change");
|
||||
false
|
||||
}
|
||||
|
||||
fn is_in_webarea(el: &AXElement) -> bool {
|
||||
use accessibility_sys::kAXRoleAttribute;
|
||||
let mut current = crate::tree::copy_element_attr(el, "AXParent");
|
||||
for _ in 0..20 {
|
||||
let Some(ref parent) = current else {
|
||||
return false;
|
||||
};
|
||||
if crate::tree::copy_string_attr(parent, kAXRoleAttribute).as_deref()
|
||||
== Some("AXWebArea")
|
||||
{
|
||||
return true;
|
||||
}
|
||||
current = crate::tree::copy_element_attr(parent, "AXParent");
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
/// Activate any Electron/web element with verified effect detection.
|
||||
///
|
||||
/// Chromium's AX implementation lies: AXPress/AXConfirm return
|
||||
/// kAXErrorSuccess but only toggle ARIA state — DOM click handlers
|
||||
/// never fire. We verify by checking if the focused UI element
|
||||
/// changed after AXPress. If nothing changed, CGClick is the
|
||||
/// genuine last resort.
|
||||
///
|
||||
/// This handles the FULL escalation for web elements so the chain
|
||||
/// engine never reaches false-positive AX steps (AXConfirm, SetBool
|
||||
/// AXSelected, etc. all lie on Chromium).
|
||||
struct PreActionState {
|
||||
focused: Option<AXElement>,
|
||||
value: Option<String>,
|
||||
selected: Option<bool>,
|
||||
}
|
||||
|
||||
impl PreActionState {
|
||||
fn capture(app: &AXElement, el: &AXElement) -> Self {
|
||||
Self {
|
||||
focused: crate::tree::copy_element_attr(app, "AXFocusedUIElement"),
|
||||
value: crate::tree::copy_string_attr(el, "AXValue"),
|
||||
selected: crate::tree::copy_bool_attr(el, "AXSelected"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn activate_web_element(el: &AXElement) -> bool {
|
||||
let Some(pid) = crate::system::app_ops::pid_from_element(el) else {
|
||||
return false;
|
||||
};
|
||||
|
||||
let app = crate::tree::element_for_pid(pid);
|
||||
let before = PreActionState::capture(&app, el);
|
||||
|
||||
if ax_helpers::try_ax_action_retried(el, "AXPress") {
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
if web_action_had_effect(&app, el, &before) {
|
||||
tracing::debug!("activate_web: AXPress had real effect");
|
||||
return true;
|
||||
}
|
||||
tracing::debug!("activate_web: AXPress returned success but no DOM effect");
|
||||
}
|
||||
|
||||
let actions = ax_helpers::list_ax_actions(el);
|
||||
for action in &["AXConfirm", "AXOpen"] {
|
||||
if actions.iter().any(|a| a == action) && ax_helpers::try_ax_action(el, action) {
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
if web_action_had_effect(&app, el, &before) {
|
||||
tracing::debug!("activate_web: {action} had real effect");
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if ax_helpers::try_each_child(
|
||||
el,
|
||||
|child| {
|
||||
let child_actions = ax_helpers::list_ax_actions(child);
|
||||
ax_helpers::try_action_from_list(
|
||||
child,
|
||||
&child_actions,
|
||||
&["AXPress", "AXConfirm", "AXOpen"],
|
||||
)
|
||||
},
|
||||
5,
|
||||
) {
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
if web_action_had_effect(&app, el, &before) {
|
||||
tracing::debug!("activate_web: child action had real effect");
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
tracing::debug!("activate_web: all AX methods had no effect, CGClick");
|
||||
let _ = crate::system::app_ops::ensure_app_focused(pid);
|
||||
crate::actions::dispatch::click_via_bounds(
|
||||
el,
|
||||
agent_desktop_core::action::MouseButton::Left,
|
||||
1,
|
||||
)
|
||||
.is_ok()
|
||||
}
|
||||
|
||||
fn web_action_had_effect(app: &AXElement, el: &AXElement, before: &PreActionState) -> bool {
|
||||
use core_foundation::base::{CFEqual, CFTypeRef};
|
||||
|
||||
let value_after = crate::tree::copy_string_attr(el, "AXValue");
|
||||
if before.value != value_after {
|
||||
return true;
|
||||
}
|
||||
|
||||
let selected_after = crate::tree::copy_bool_attr(el, "AXSelected");
|
||||
if before.selected != selected_after {
|
||||
return true;
|
||||
}
|
||||
|
||||
let focused_after = crate::tree::copy_element_attr(app, "AXFocusedUIElement");
|
||||
match (&before.focused, &focused_after) {
|
||||
(Some(before_f), Some(after_f)) => unsafe {
|
||||
CFEqual(before_f.0 as CFTypeRef, after_f.0 as CFTypeRef) == 0
|
||||
},
|
||||
(None, Some(_)) => true,
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn try_focus_then_verified_confirm_or_press(el: &AXElement, caps: &ElementCaps) -> bool {
|
||||
if !ax_helpers::ax_focus(el) {
|
||||
return false;
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||
if ax_helpers::try_ax_action_retried(el, "AXConfirm") {
|
||||
let in_web = is_in_webarea(el);
|
||||
if !in_web && ax_helpers::try_ax_action_retried(el, "AXConfirm") {
|
||||
return true;
|
||||
}
|
||||
do_verified_press(el, caps)
|
||||
dispatch_verified_press(el, caps, in_web)
|
||||
}
|
||||
|
||||
pub fn try_value_relay(el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
|
|
|
|||
|
|
@ -51,7 +51,7 @@ impl PlatformAdapter for MacOSAdapter {
|
|||
.ok_or_else(|| AdapterError::element_not_found("No open alert or dialog"))?,
|
||||
};
|
||||
let mut visited = FxHashSet::default();
|
||||
crate::tree::build_subtree(&el, 0, opts.max_depth, opts.include_bounds, &mut visited)
|
||||
crate::tree::build_subtree(&el, 0, 0, opts.max_depth, &mut visited, opts.skeleton)
|
||||
.ok_or_else(|| AdapterError::internal("Empty AX tree for surface"))
|
||||
}
|
||||
|
||||
|
|
@ -72,7 +72,7 @@ impl PlatformAdapter for MacOSAdapter {
|
|||
}
|
||||
|
||||
fn list_apps(&self) -> Result<Vec<AppInfo>, AdapterError> {
|
||||
list_apps_impl()
|
||||
crate::system::app_ops::list_apps_impl()
|
||||
}
|
||||
|
||||
fn focus_window(&self, win: &WindowInfo) -> Result<(), AdapterError> {
|
||||
|
|
@ -205,6 +205,28 @@ impl PlatformAdapter for MacOSAdapter {
|
|||
) -> Result<ActionResult, AdapterError> {
|
||||
crate::notifications::actions::notification_action(index, action_name)
|
||||
}
|
||||
|
||||
fn get_subtree(
|
||||
&self,
|
||||
handle: &NativeHandle,
|
||||
opts: &TreeOptions,
|
||||
) -> Result<AccessibilityNode, AdapterError> {
|
||||
use crate::tree::AXElement;
|
||||
use std::mem::ManuallyDrop;
|
||||
|
||||
let el = ManuallyDrop::new(AXElement(
|
||||
handle.as_raw() as accessibility_sys::AXUIElementRef
|
||||
));
|
||||
let mut ancestors = FxHashSet::default();
|
||||
crate::tree::build_subtree(&el, 0, 0, opts.max_depth, &mut ancestors, opts.skeleton)
|
||||
.ok_or_else(|| {
|
||||
AdapterError::new(
|
||||
agent_desktop_core::error::ErrorCode::ElementNotFound,
|
||||
"Element no longer exists in accessibility tree",
|
||||
)
|
||||
.with_suggestion("Run 'snapshot' to refresh refs, then retry.")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
|
|
@ -318,80 +340,3 @@ pub(crate) fn list_windows_impl(filter: &WindowFilter) -> Result<Vec<WindowInfo>
|
|||
Err(AdapterError::not_supported("list_windows"))
|
||||
}
|
||||
}
|
||||
|
||||
fn list_apps_impl() -> Result<Vec<AppInfo>, AdapterError> {
|
||||
#[cfg(target_os = "macos")]
|
||||
{
|
||||
use core_foundation::base::{CFType, TCFType};
|
||||
use core_foundation::number::CFNumber;
|
||||
use core_foundation::string::CFString;
|
||||
use core_foundation_sys::dictionary::CFDictionaryGetValue;
|
||||
use core_graphics::display::CGDisplay;
|
||||
use core_graphics::window::{
|
||||
kCGWindowLayer, kCGWindowListOptionOnScreenOnly, kCGWindowOwnerName, kCGWindowOwnerPID,
|
||||
};
|
||||
|
||||
let arr = match CGDisplay::window_list_info(kCGWindowListOptionOnScreenOnly, None) {
|
||||
Some(a) => a,
|
||||
None => return Ok(vec![]),
|
||||
};
|
||||
|
||||
let mut seen_pids = std::collections::HashSet::new();
|
||||
let mut apps = Vec::new();
|
||||
|
||||
for raw in arr.get_all_values() {
|
||||
if raw.is_null() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let layer = unsafe {
|
||||
let v = CFDictionaryGetValue(raw as _, kCGWindowLayer as _);
|
||||
if v.is_null() {
|
||||
continue;
|
||||
}
|
||||
CFType::wrap_under_get_rule(v as _)
|
||||
.downcast::<CFNumber>()
|
||||
.and_then(|n| n.to_i64())
|
||||
.unwrap_or(99)
|
||||
};
|
||||
if layer != 0 {
|
||||
continue;
|
||||
}
|
||||
|
||||
let pid = unsafe {
|
||||
let v = CFDictionaryGetValue(raw as _, kCGWindowOwnerPID as _);
|
||||
if v.is_null() {
|
||||
continue;
|
||||
}
|
||||
CFType::wrap_under_get_rule(v as _)
|
||||
.downcast::<CFNumber>()
|
||||
.and_then(|n| n.to_i64())
|
||||
.unwrap_or(0) as i32
|
||||
};
|
||||
if !seen_pids.insert(pid) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let name = unsafe {
|
||||
let v = CFDictionaryGetValue(raw as _, kCGWindowOwnerName as _);
|
||||
if v.is_null() {
|
||||
continue;
|
||||
}
|
||||
CFType::wrap_under_get_rule(v as _)
|
||||
.downcast::<CFString>()
|
||||
.map(|s| s.to_string())
|
||||
};
|
||||
|
||||
if let Some(n) = name {
|
||||
apps.push(AppInfo {
|
||||
name: n,
|
||||
pid,
|
||||
bundle_id: None,
|
||||
});
|
||||
}
|
||||
}
|
||||
Ok(apps)
|
||||
}
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
Err(AdapterError::not_supported("list_apps"))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,4 +1,8 @@
|
|||
use agent_desktop_core::{adapter::WindowFilter, error::AdapterError, node::WindowInfo};
|
||||
use agent_desktop_core::{
|
||||
adapter::WindowFilter,
|
||||
error::AdapterError,
|
||||
node::{AppInfo, WindowInfo},
|
||||
};
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
pub fn pid_from_element(el: &crate::tree::AXElement) -> Option<i32> {
|
||||
|
|
@ -234,3 +238,80 @@ fn try_quit_via_menu_bar(app_el: &crate::tree::AXElement) -> bool {
|
|||
pub fn close_app_impl(_id: &str, _force: bool) -> Result<(), AdapterError> {
|
||||
Err(AdapterError::not_supported("close_app"))
|
||||
}
|
||||
|
||||
pub fn list_apps_impl() -> Result<Vec<AppInfo>, AdapterError> {
|
||||
#[cfg(target_os = "macos")]
|
||||
{
|
||||
use core_foundation::base::{CFType, TCFType};
|
||||
use core_foundation::number::CFNumber;
|
||||
use core_foundation::string::CFString;
|
||||
use core_foundation_sys::dictionary::CFDictionaryGetValue;
|
||||
use core_graphics::display::CGDisplay;
|
||||
use core_graphics::window::{
|
||||
kCGWindowLayer, kCGWindowListOptionOnScreenOnly, kCGWindowOwnerName, kCGWindowOwnerPID,
|
||||
};
|
||||
|
||||
let arr = match CGDisplay::window_list_info(kCGWindowListOptionOnScreenOnly, None) {
|
||||
Some(a) => a,
|
||||
None => return Ok(vec![]),
|
||||
};
|
||||
|
||||
let mut seen_pids = std::collections::HashSet::new();
|
||||
let mut apps = Vec::new();
|
||||
|
||||
for raw in arr.get_all_values() {
|
||||
if raw.is_null() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let layer = unsafe {
|
||||
let v = CFDictionaryGetValue(raw as _, kCGWindowLayer as _);
|
||||
if v.is_null() {
|
||||
continue;
|
||||
}
|
||||
CFType::wrap_under_get_rule(v as _)
|
||||
.downcast::<CFNumber>()
|
||||
.and_then(|n| n.to_i64())
|
||||
.unwrap_or(99)
|
||||
};
|
||||
if layer != 0 {
|
||||
continue;
|
||||
}
|
||||
|
||||
let pid = unsafe {
|
||||
let v = CFDictionaryGetValue(raw as _, kCGWindowOwnerPID as _);
|
||||
if v.is_null() {
|
||||
continue;
|
||||
}
|
||||
CFType::wrap_under_get_rule(v as _)
|
||||
.downcast::<CFNumber>()
|
||||
.and_then(|n| n.to_i64())
|
||||
.unwrap_or(0) as i32
|
||||
};
|
||||
if !seen_pids.insert(pid) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let name = unsafe {
|
||||
let v = CFDictionaryGetValue(raw as _, kCGWindowOwnerName as _);
|
||||
if v.is_null() {
|
||||
continue;
|
||||
}
|
||||
CFType::wrap_under_get_rule(v as _)
|
||||
.downcast::<CFString>()
|
||||
.map(|s| s.to_string())
|
||||
};
|
||||
|
||||
if let Some(n) = name {
|
||||
apps.push(AppInfo {
|
||||
name: n,
|
||||
pid,
|
||||
bundle_id: None,
|
||||
});
|
||||
}
|
||||
}
|
||||
Ok(apps)
|
||||
}
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
Err(AdapterError::not_supported("list_apps"))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,14 +2,14 @@ use agent_desktop_core::node::AccessibilityNode;
|
|||
use rustc_hash::FxHashSet;
|
||||
|
||||
use super::element::{
|
||||
copy_ax_array, copy_string_attr, element_for_pid, fetch_node_attrs, read_bounds, AXElement,
|
||||
ABSOLUTE_MAX_DEPTH,
|
||||
child_attributes, copy_ax_array, copy_string_attr, count_children, element_for_pid,
|
||||
fetch_node_attrs, read_bounds, AXElement, ABSOLUTE_MAX_DEPTH,
|
||||
};
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
use accessibility_sys::{
|
||||
kAXChildrenAttribute, kAXContentsAttribute, kAXRoleAttribute, kAXTitleAttribute,
|
||||
kAXValueAttribute, kAXWindowsAttribute,
|
||||
kAXChildrenAttribute, kAXRoleAttribute, kAXTitleAttribute, kAXValueAttribute,
|
||||
kAXWindowsAttribute,
|
||||
};
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
|
|
@ -44,13 +44,41 @@ pub fn window_element_for(pid: i32, win_title: &str) -> AXElement {
|
|||
pub fn build_subtree(
|
||||
el: &AXElement,
|
||||
depth: u8,
|
||||
raw_depth: u8,
|
||||
max_depth: u8,
|
||||
_include_bounds: bool,
|
||||
ancestors: &mut FxHashSet<usize>,
|
||||
skeleton: bool,
|
||||
) -> Option<AccessibilityNode> {
|
||||
if depth > max_depth || depth >= ABSOLUTE_MAX_DEPTH {
|
||||
if depth > max_depth {
|
||||
return None;
|
||||
}
|
||||
if raw_depth >= ABSOLUTE_MAX_DEPTH {
|
||||
let (ax_role, title, ax_desc, value, _, _) = fetch_node_attrs(el);
|
||||
let role = ax_role
|
||||
.as_deref()
|
||||
.map(crate::tree::roles::ax_role_to_str)
|
||||
.unwrap_or("unknown")
|
||||
.to_string();
|
||||
let name = title.or(ax_desc);
|
||||
let child_count = count_children(el, ax_role.as_deref());
|
||||
let bounds = read_bounds(el);
|
||||
return Some(AccessibilityNode {
|
||||
ref_id: None,
|
||||
role,
|
||||
name,
|
||||
value,
|
||||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
bounds,
|
||||
children_count: if child_count > 0 {
|
||||
Some(child_count)
|
||||
} else {
|
||||
None
|
||||
},
|
||||
children: vec![],
|
||||
});
|
||||
}
|
||||
let ptr_key = el.0 as usize;
|
||||
if !ancestors.insert(ptr_key) {
|
||||
return None;
|
||||
|
|
@ -83,23 +111,62 @@ pub fn build_subtree(
|
|||
|
||||
let bounds = read_bounds(el);
|
||||
|
||||
let children_raw = copy_children(el, ax_role.as_deref()).unwrap_or_default();
|
||||
let name = name.or_else(|| label_from_children(&children_raw));
|
||||
|
||||
// Non-semantic groups inside web content don't cost depth budget.
|
||||
// A nameless AXGroup/AXGenericElement is just a <div> wrapper — skip it.
|
||||
let is_web_wrapper = matches!(
|
||||
ax_role.as_deref(),
|
||||
Some("AXGroup") | Some("AXGenericElement")
|
||||
) && title.as_deref().is_none_or(str::is_empty)
|
||||
&& value.as_deref().is_none_or(str::is_empty);
|
||||
|
||||
// Web wrappers do not consume a logical depth slot so that Electron/Chromium
|
||||
// structural layers (AXGroup/AXGenericElement with no label) are transparent to
|
||||
// agents. A chain of wrappers only stops at ABSOLUTE_MAX_DEPTH, not max_depth.
|
||||
// This is intentional: skeleton depth tracks semantic content depth, not raw DOM depth.
|
||||
let child_depth = if is_web_wrapper { depth } else { depth + 1 };
|
||||
let child_raw_depth = raw_depth + 1;
|
||||
|
||||
let at_skeleton_boundary =
|
||||
skeleton && (child_depth > max_depth || child_raw_depth >= ABSOLUTE_MAX_DEPTH);
|
||||
|
||||
if at_skeleton_boundary {
|
||||
let child_count = count_children(el, ax_role.as_deref());
|
||||
let children_count = if child_count > 0 {
|
||||
Some(child_count)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let name = name.or_else(|| {
|
||||
let children_raw = copy_children(el, ax_role.as_deref()).unwrap_or_default();
|
||||
label_from_children(&children_raw)
|
||||
});
|
||||
ancestors.remove(&ptr_key);
|
||||
return Some(AccessibilityNode {
|
||||
ref_id: None,
|
||||
role,
|
||||
name,
|
||||
value,
|
||||
description,
|
||||
hint: None,
|
||||
states,
|
||||
bounds,
|
||||
children_count,
|
||||
children: vec![],
|
||||
});
|
||||
}
|
||||
|
||||
let children_raw = copy_children(el, ax_role.as_deref()).unwrap_or_default();
|
||||
let name = name.or_else(|| label_from_children(&children_raw));
|
||||
|
||||
let children = children_raw
|
||||
.into_iter()
|
||||
.filter_map(|child| {
|
||||
build_subtree(&child, child_depth, max_depth, _include_bounds, ancestors)
|
||||
build_subtree(
|
||||
&child,
|
||||
child_depth,
|
||||
child_raw_depth,
|
||||
max_depth,
|
||||
ancestors,
|
||||
skeleton,
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
|
||||
|
|
@ -114,6 +181,7 @@ pub fn build_subtree(
|
|||
hint: None,
|
||||
states,
|
||||
bounds,
|
||||
children_count: None,
|
||||
children,
|
||||
})
|
||||
}
|
||||
|
|
@ -159,14 +227,7 @@ pub fn label_from_children(children: &[AXElement]) -> Option<String> {
|
|||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn copy_children(el: &AXElement, ax_role: Option<&str>) -> Option<Vec<AXElement>> {
|
||||
if ax_role == Some("AXBrowser") {
|
||||
return copy_ax_array(el, "AXColumns");
|
||||
}
|
||||
for attr in &[
|
||||
kAXChildrenAttribute,
|
||||
kAXContentsAttribute,
|
||||
"AXChildrenInNavigationOrder",
|
||||
] {
|
||||
for attr in child_attributes(ax_role) {
|
||||
if let Some(v) = copy_ax_array(el, attr) {
|
||||
if !v.is_empty() {
|
||||
return Some(v);
|
||||
|
|
@ -185,9 +246,31 @@ pub fn window_element_for(_pid: i32, _win_title: &str) -> AXElement {
|
|||
pub fn build_subtree(
|
||||
_el: &AXElement,
|
||||
_depth: u8,
|
||||
_raw_depth: u8,
|
||||
_max_depth: u8,
|
||||
_include_bounds: bool,
|
||||
_visited: &mut FxHashSet<usize>,
|
||||
_skeleton: bool,
|
||||
) -> Option<AccessibilityNode> {
|
||||
None
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::child_attributes;
|
||||
|
||||
#[test]
|
||||
fn test_browser_children_use_columns() {
|
||||
assert_eq!(
|
||||
child_attributes(Some("AXBrowser")),
|
||||
["AXColumns", "AXContents"]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_default_children_follow_fallback_order() {
|
||||
assert_eq!(
|
||||
child_attributes(Some("AXGroup")),
|
||||
["AXChildren", "AXContents", "AXChildrenInNavigationOrder"]
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,14 +2,22 @@ use agent_desktop_core::node::Rect;
|
|||
|
||||
pub const ABSOLUTE_MAX_DEPTH: u8 = 50;
|
||||
|
||||
pub(crate) fn child_attributes(ax_role: Option<&str>) -> &'static [&'static str] {
|
||||
if ax_role == Some("AXBrowser") {
|
||||
&["AXColumns", "AXContents"]
|
||||
} else {
|
||||
&["AXChildren", "AXContents", "AXChildrenInNavigationOrder"]
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
mod imp {
|
||||
use super::*;
|
||||
use accessibility_sys::{
|
||||
kAXChildrenAttribute, kAXDescriptionAttribute, kAXEnabledAttribute, kAXErrorSuccess,
|
||||
kAXFocusedAttribute, kAXRoleAttribute, kAXTitleAttribute, kAXValueAttribute,
|
||||
AXUIElementCopyAttributeValue, AXUIElementCopyMultipleAttributeValues,
|
||||
AXUIElementCreateApplication, AXUIElementRef, AXUIElementSetMessagingTimeout,
|
||||
kAXDescriptionAttribute, kAXEnabledAttribute, kAXErrorSuccess, kAXFocusedAttribute,
|
||||
kAXRoleAttribute, kAXTitleAttribute, kAXValueAttribute, AXUIElementCopyAttributeValue,
|
||||
AXUIElementCopyMultipleAttributeValues, AXUIElementCreateApplication, AXUIElementRef,
|
||||
AXUIElementSetMessagingTimeout,
|
||||
};
|
||||
use core_foundation::{
|
||||
array::CFArray,
|
||||
|
|
@ -143,7 +151,10 @@ mod imp {
|
|||
};
|
||||
|
||||
name.or_else(|| {
|
||||
let children = copy_ax_array(el, kAXChildrenAttribute).unwrap_or_default();
|
||||
let children = super::child_attributes(ax_role.as_deref())
|
||||
.iter()
|
||||
.find_map(|attr| copy_ax_array(el, attr).filter(|v| !v.is_empty()))
|
||||
.unwrap_or_default();
|
||||
crate::tree::builder::label_from_children(&children)
|
||||
})
|
||||
}
|
||||
|
|
@ -238,6 +249,29 @@ mod imp {
|
|||
Some(AXElement(ptr))
|
||||
}
|
||||
|
||||
pub fn count_children(element: &AXElement, ax_role: Option<&str>) -> u32 {
|
||||
unsafe {
|
||||
for attr_name in child_attributes(ax_role) {
|
||||
let mut value: core_foundation::base::CFTypeRef = std::ptr::null();
|
||||
let attr = CFString::from_static_string(attr_name);
|
||||
let err = AXUIElementCopyAttributeValue(
|
||||
element.0,
|
||||
attr.as_concrete_TypeRef(),
|
||||
&mut value,
|
||||
);
|
||||
if err != kAXErrorSuccess || value.is_null() {
|
||||
continue;
|
||||
}
|
||||
let count = core_foundation_sys::array::CFArrayGetCount(value as _);
|
||||
CFRelease(value);
|
||||
if count > 0 {
|
||||
return count as u32;
|
||||
}
|
||||
}
|
||||
0
|
||||
}
|
||||
}
|
||||
|
||||
pub fn read_bounds(el: &AXElement) -> Option<Rect> {
|
||||
use accessibility_sys::{
|
||||
kAXPositionAttribute, kAXSizeAttribute, kAXValueTypeCGPoint, kAXValueTypeCGSize,
|
||||
|
|
@ -337,6 +371,9 @@ mod imp {
|
|||
pub fn copy_element_attr(_el: &AXElement, _attr: &str) -> Option<AXElement> {
|
||||
None
|
||||
}
|
||||
pub fn count_children(_element: &AXElement, _ax_role: Option<&str>) -> u32 {
|
||||
0
|
||||
}
|
||||
pub fn read_bounds(_el: &AXElement) -> Option<Rect> {
|
||||
None
|
||||
}
|
||||
|
|
@ -362,5 +399,6 @@ mod imp {
|
|||
|
||||
pub use imp::{
|
||||
copy_ax_array, copy_bool_attr, copy_element_attr, copy_string_attr, copy_value_typed,
|
||||
element_for_pid, fetch_node_attrs, read_bounds, resolve_element_name, AXElement,
|
||||
count_children, element_for_pid, fetch_node_attrs, read_bounds, resolve_element_name,
|
||||
AXElement,
|
||||
};
|
||||
|
|
|
|||
|
|
@ -6,8 +6,9 @@ pub mod surfaces;
|
|||
|
||||
pub use builder::{build_subtree, window_element_for};
|
||||
pub use element::{
|
||||
copy_ax_array, copy_element_attr, copy_string_attr, copy_value_typed, element_for_pid,
|
||||
read_bounds, resolve_element_name, AXElement, ABSOLUTE_MAX_DEPTH,
|
||||
copy_ax_array, copy_bool_attr, copy_element_attr, copy_string_attr, copy_value_typed,
|
||||
count_children, element_for_pid, read_bounds, resolve_element_name, AXElement,
|
||||
ABSOLUTE_MAX_DEPTH,
|
||||
};
|
||||
pub use resolve::{find_element_recursive, resolve_element_impl};
|
||||
pub use roles::{ax_role_to_str, is_interactive_role};
|
||||
|
|
|
|||
|
|
@ -2,7 +2,8 @@ use agent_desktop_core::{adapter::NativeHandle, error::AdapterError, refs::RefEn
|
|||
use rustc_hash::FxHashSet;
|
||||
|
||||
use super::element::{
|
||||
copy_ax_array, copy_string_attr, element_for_pid, resolve_element_name, AXElement,
|
||||
child_attributes, copy_ax_array, copy_string_attr, element_for_pid, resolve_element_name,
|
||||
AXElement,
|
||||
};
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
|
|
@ -16,9 +17,11 @@ pub fn resolve_element_impl(entry: &RefEntry) -> Result<NativeHandle, AdapterErr
|
|||
);
|
||||
let root = element_for_pid(entry.pid);
|
||||
let mut visited = FxHashSet::default();
|
||||
// Electron/web apps nest 25+ levels deep; use ABSOLUTE_MAX_DEPTH (50) for resolution.
|
||||
let resolve_depth: u8 = 50;
|
||||
if let Ok(handle) = find_element_recursive(&root, entry, 0, resolve_depth, &mut visited) {
|
||||
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5);
|
||||
if let Ok(handle) =
|
||||
find_element_recursive(&root, entry, 0, resolve_depth, &mut visited, deadline)
|
||||
{
|
||||
tracing::debug!("resolve: found exact match");
|
||||
return Ok(handle);
|
||||
}
|
||||
|
|
@ -29,7 +32,8 @@ pub fn resolve_element_impl(entry: &RefEntry) -> Result<NativeHandle, AdapterErr
|
|||
..entry.clone()
|
||||
};
|
||||
visited.clear();
|
||||
if let Ok(handle) = find_element_recursive(&root, &relaxed, 0, resolve_depth, &mut visited)
|
||||
if let Ok(handle) =
|
||||
find_element_recursive(&root, &relaxed, 0, resolve_depth, &mut visited, deadline)
|
||||
{
|
||||
tracing::debug!("resolve: found via relaxed match (bounds changed)");
|
||||
return Ok(handle);
|
||||
|
|
@ -47,6 +51,13 @@ pub fn resolve_element_impl(entry: &RefEntry) -> Result<NativeHandle, AdapterErr
|
|||
.with_suggestion("Run 'snapshot' to refresh, then retry with the updated ref."))
|
||||
}
|
||||
|
||||
/// Depth-first search for a single element matching `entry`.
|
||||
///
|
||||
/// `deadline` is shared across the full resolve call (exact + relaxed passes)
|
||||
/// so the total wall-clock time is bounded at five seconds. Subtrees whose
|
||||
/// spatial bounds do not contain the target's centre point are pruned, which
|
||||
/// makes resolution over large documents (e.g. dense spreadsheets) fast even
|
||||
/// when the AX tree has thousands of nodes.
|
||||
#[cfg(target_os = "macos")]
|
||||
pub fn find_element_recursive(
|
||||
el: &AXElement,
|
||||
|
|
@ -54,10 +65,19 @@ pub fn find_element_recursive(
|
|||
depth: u8,
|
||||
max_depth: u8,
|
||||
ancestors: &mut FxHashSet<usize>,
|
||||
deadline: std::time::Instant,
|
||||
) -> Result<NativeHandle, AdapterError> {
|
||||
use accessibility_sys::kAXRoleAttribute;
|
||||
use core_foundation::base::{CFRetain, CFTypeRef};
|
||||
|
||||
if std::time::Instant::now() > deadline {
|
||||
return Err(AdapterError::new(
|
||||
agent_desktop_core::error::ErrorCode::StaleRef,
|
||||
"Element resolution timed out",
|
||||
)
|
||||
.with_suggestion("Run 'snapshot' to refresh, then retry with the updated ref."));
|
||||
}
|
||||
|
||||
let ptr_key = el.0 as usize;
|
||||
if !ancestors.insert(ptr_key) {
|
||||
return Err(AdapterError::element_not_found("element"));
|
||||
|
|
@ -95,18 +115,32 @@ pub fn find_element_recursive(
|
|||
return Err(AdapterError::element_not_found("element"));
|
||||
}
|
||||
|
||||
let child_attr = if ax_role.as_deref() == Some("AXBrowser") {
|
||||
"AXColumns"
|
||||
} else {
|
||||
"AXChildren"
|
||||
};
|
||||
let children = copy_ax_array(el, child_attr)
|
||||
.filter(|v| !v.is_empty())
|
||||
.or_else(|| copy_ax_array(el, "AXContents").filter(|v| !v.is_empty()))
|
||||
if let Some(target) = &entry.bounds {
|
||||
if let Some(el_bounds) = crate::tree::read_bounds(el) {
|
||||
if el_bounds.width > 0.0 && el_bounds.height > 0.0 {
|
||||
let cx = target.x + target.width / 2.0;
|
||||
let cy = target.y + target.height / 2.0;
|
||||
if cx < el_bounds.x
|
||||
|| cx > el_bounds.x + el_bounds.width
|
||||
|| cy < el_bounds.y
|
||||
|| cy > el_bounds.y + el_bounds.height
|
||||
{
|
||||
ancestors.remove(&ptr_key);
|
||||
return Err(AdapterError::element_not_found("element"));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let children = child_attributes(ax_role.as_deref())
|
||||
.iter()
|
||||
.find_map(|attr| copy_ax_array(el, attr).filter(|v| !v.is_empty()))
|
||||
.unwrap_or_default();
|
||||
|
||||
for child in &children {
|
||||
if let Ok(handle) = find_element_recursive(child, entry, depth + 1, max_depth, ancestors) {
|
||||
if let Ok(handle) =
|
||||
find_element_recursive(child, entry, depth + 1, max_depth, ancestors, deadline)
|
||||
{
|
||||
ancestors.remove(&ptr_key);
|
||||
return Ok(handle);
|
||||
}
|
||||
|
|
@ -128,6 +162,7 @@ pub fn find_element_recursive(
|
|||
_depth: u8,
|
||||
_max_depth: u8,
|
||||
_ancestors: &mut FxHashSet<usize>,
|
||||
_deadline: std::time::Instant,
|
||||
) -> Result<NativeHandle, AdapterError> {
|
||||
Err(AdapterError::not_supported("find_element_recursive"))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -37,6 +37,7 @@ Phase 1 is the load-bearing phase. It establishes every shared abstraction, ever
|
|||
| P1-O7 | Command extensibility | Adding a new command requires exactly 1 new file + 2 registration lines |
|
||||
| P1-O8 | 50 working commands | All commands pass integration tests |
|
||||
| P1-O9 | CI pipeline | GitHub Actions macOS runner executes full test suite on every PR |
|
||||
| P1-O10 | Progressive skeleton traversal | Skeleton + drill-down workflow achieves 78%+ token savings on Electron apps |
|
||||
|
||||
### Workspace Structure
|
||||
|
||||
|
|
@ -57,6 +58,8 @@ agent-desktop/
|
|||
│ │ ├── adapter.rs # PlatformAdapter trait
|
||||
│ │ ├── action.rs # Action enum, ActionResult, InputEvent, WindowOp
|
||||
│ │ ├── refs.rs # RefAllocator, RefMap, RefEntry
|
||||
│ │ ├── ref_alloc.rs # Shared ref-allocation logic (RefAllocConfig, allocate_refs, INTERACTIVE_ROLES, is_collapsible)
|
||||
│ │ ├── snapshot_ref.rs # Ref-rooted drill-down (run_from_ref) — delegates allocation to ref_alloc
|
||||
│ │ ├── snapshot.rs # SnapshotEngine (filter, allocate, serialize)
|
||||
│ │ ├── error.rs # ErrorCode enum, AdapterError, AppError
|
||||
│ │ ├── output.rs # Response envelope, JSON formatting
|
||||
|
|
@ -96,6 +99,7 @@ pub trait PlatformAdapter: Send + Sync {
|
|||
fn set_clipboard(&self, text: &str) -> Result<()>;
|
||||
fn synthesize_input(&self, input: InputEvent) -> Result<()>;
|
||||
fn manage_window(&self, win: &WindowInfo, op: WindowOp) -> Result<()>;
|
||||
fn get_subtree(&self, handle: &NativeHandle, opts: &TreeOptions) -> Result<AccessibilityNode>;
|
||||
fn list_notifications(&self) -> Result<Vec<NotificationInfo>>;
|
||||
fn dismiss_notification(&self, id: &str) -> Result<()>;
|
||||
fn interact_notification(&self, id: &str, action_name: &str) -> Result<ActionResult>;
|
||||
|
|
@ -112,6 +116,7 @@ pub trait PlatformAdapter: Send + Sync {
|
|||
- `ScreenshotTarget` — Screen(index), Window(id), Element(NativeHandle), FullScreen
|
||||
- `NotificationInfo` — id, app_name, title, body, timestamp, actions: Vec\<String\>, is_persistent
|
||||
- `TrayItemInfo` — id, app_name, title, tooltip, has_menu
|
||||
- `TreeOptions` — `max_depth`, `include_bounds`, `interactive_only`, `compact`, `surface`, `skeleton` (root is CLI-only via `SnapshotArgs.root_ref`, not plumbed into `TreeOptions`)
|
||||
|
||||
### macOS Adapter Implementation
|
||||
|
||||
|
|
@ -213,6 +218,16 @@ Platform-agnostic, lives in `agent-desktop-core`:
|
|||
|
||||
RefMap persisted at `~/.agent-desktop/last_refmap.json` with `0o600` permissions, directory at `0o700`. Each snapshot replaces the refmap file entirely (atomic write via temp + rename). Action commands use optimistic re-identification: `(pid, role, name, bounds_hash)`. Return `STALE_REF` on mismatch.
|
||||
|
||||
**Progressive Skeleton Traversal:**
|
||||
- `--skeleton` flag clamps depth to `min(max_depth, 3)`, annotates truncated containers with `children_count` for agent discovery
|
||||
- `--root <REF>` flag starts traversal from a previously-discovered ref instead of window root
|
||||
- Named or described containers at skeleton boundary receive refs as drill-down targets (with empty `available_actions`)
|
||||
- Scoped invalidation: re-drilling a ref replaces only that ref's subtree refs, preserving all others
|
||||
- Core modules: `ref_alloc.rs` (canonical `allocate_refs` + `RefAllocConfig`), `snapshot_ref.rs` (drill-down flow that delegates allocation to `ref_alloc`)
|
||||
- macOS: `count_children()` uses raw `CFArrayGetCount` without materializing `AXElement` wrappers for performance
|
||||
- RefMap write-side size check prevents >1MB files
|
||||
- Token savings: 78-96% reduction for dense Electron apps (Slack skeleton: ~3.6KB vs ~17.3KB full)
|
||||
|
||||
### New Commands — Notification & System Tray (Post Phase 1)
|
||||
|
||||
> **Note:** Notification management and system tray interaction were not part of the original Phase 1 delivery. These are **new features to be implemented across all platforms** as each platform adapter is built. The macOS implementations were added as a follow-up to Phase 1. Windows (Phase 2) and Linux (Phase 3) implementations follow the same pattern.
|
||||
|
|
@ -512,6 +527,8 @@ Chromium-based apps (Electron, Chrome, Edge, VS Code) expose deep, noisy accessi
|
|||
- Check both `ControlType` and `LocalizedControlType` / UIA patterns (analogous to macOS checking both AXRole and AXSubrole)
|
||||
- Implement in `crates/windows/src/tree/surfaces.rs`
|
||||
|
||||
**Progressive skeleton traversal** works identically on Windows — `--skeleton` and `--root` flags are platform-agnostic, handled entirely by core. The Windows adapter only needs to implement `get_subtree()` (which delegates to the same `build_subtree()` as `get_tree()`). Token savings for Electron apps (VS Code, Slack) apply equally.
|
||||
|
||||
### Minimum OS Requirements
|
||||
|
||||
- Windows 10 1809+ (October 2018 update)
|
||||
|
|
@ -758,6 +775,8 @@ Same Chromium/Electron compatibility patterns as Phase 2 (Windows), adapted for
|
|||
- If AT-SPI tree is empty for a Chromium app, warn about `--force-renderer-accessibility`
|
||||
- On Linux, Chromium respects `ACCESSIBILITY_ENABLED=1` environment variable as an alternative
|
||||
|
||||
**Progressive skeleton traversal** works identically on Linux — `--skeleton` and `--root` flags are platform-agnostic, handled entirely by core. The Linux adapter only needs to implement `get_subtree()` (which delegates to the same async tree walker). Token savings for Electron apps apply equally.
|
||||
|
||||
### AT-SPI2 Bus Detection
|
||||
|
||||
- Check for `org.a11y.Bus` presence on the D-Bus session bus
|
||||
|
|
@ -1131,6 +1150,7 @@ When daemon is not running, CLI falls back to direct execution (same as Phases 1
|
|||
| Async tree walking | Linux | Parallel D-Bus calls for tree traversal — concurrent child fetching |
|
||||
| Cached subtrees | All (daemon) | Reuse unchanged subtrees between snapshots in same session — skip re-traversal of stable UI regions |
|
||||
| Warm adapter | All (daemon) | Adapter stays initialized between commands — skip COM init (Win), D-Bus connect (Linux), AX bootstrap (macOS) |
|
||||
| Progressive skeleton drill | All | Skeleton overview + targeted drill-down reduces token consumption 78-96% for dense apps — fewer tokens per snapshot means more budget for actions |
|
||||
|
||||
### Package Manager Distribution
|
||||
|
||||
|
|
@ -1285,5 +1305,5 @@ All runners enforce: `cargo clippy --all-targets -- -D warnings`, `cargo test --
|
|||
| R4 | Wayland a11y gaps | Medium | Medium | Focus on GNOME (best AT-SPI2 support). Prefer AT-SPI actions over coordinate input. Document gaps. |
|
||||
| R5 | Rust a11y crate maintenance stalls | Low | High | Pin versions, maintain patches. `atspi` backed by Odilia project. Fork-ready. |
|
||||
| R6 | MCP spec changes break compat | Low | Medium | Pin `rmcp` version. Monitor spec under Linux Foundation governance. |
|
||||
| R7 | Tree traversal too slow (>5s) | Medium | Medium | Depth limiting via `--max-depth`. Focused-window-only. Cached subtrees in Phase 5 daemon. |
|
||||
| R8 | Ref instability confuses agents | Medium | High | Clear docs: refs are snapshot-scoped. `STALE_REF` error with recovery hint. Stable hashing in Phase 5. |
|
||||
| R7 | Tree traversal too slow (>5s) | Medium | Medium | Depth limiting via `--max-depth`. Focused-window-only. Cached subtrees in Phase 5 daemon. Progressive skeleton traversal (`--skeleton` + `--root`) reduces token consumption 78-96% for dense apps. |
|
||||
| R8 | Ref instability confuses agents | Medium | High | Clear docs: refs are snapshot-scoped. `STALE_REF` error with recovery hint. Stable hashing in Phase 5. Progressive skeleton traversal with scoped invalidation provides a stable drill-down workflow for navigating complex UIs. |
|
||||
|
|
|
|||
|
|
@ -0,0 +1,268 @@
|
|||
---
|
||||
title: Deduplicate ref allocator via RefAllocConfig instead of a _with_X copy
|
||||
date: 2026-04-14
|
||||
category: best-practices
|
||||
module: crates/core
|
||||
problem_type: best_practice
|
||||
component: tooling
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Two functions share identical bodies differing only in one Optional/nullable field (classic _with_X naming)
|
||||
- A positional parameter list carries five or more args, including bool flags
|
||||
- A patch commit touches only one copy of duplicated traversal logic, causing silent drift in the other
|
||||
- A code review flags a DRY violation on ref-allocation or tree-traversal paths
|
||||
- Adding a new per-call config axis (e.g. root_ref_id) would require another _with_X variant
|
||||
tags:
|
||||
- dry
|
||||
- ref-allocation
|
||||
- config-struct
|
||||
- skeleton-traversal
|
||||
- drill-down
|
||||
- deduplication
|
||||
- rust-patterns
|
||||
- code-review
|
||||
---
|
||||
|
||||
# Deduplicate ref allocator via RefAllocConfig instead of a _with_X copy
|
||||
|
||||
## Context
|
||||
|
||||
The progressive skeleton traversal feature shipped two near-identical ref allocators into `crates/core/`:
|
||||
|
||||
- `allocate_refs` in `crates/core/src/snapshot.rs` — 7 positional parameters, called from the full-snapshot path
|
||||
- `allocate_refs_with_root` in `crates/core/src/snapshot_ref.rs` — took a local `DrillDownConfig<'a>` struct, called from the drill-down path
|
||||
|
||||
Both bodies were ~95% identical: same `INTERACTIVE_ROLES` check, same `ref_entry_from_node` call, same skeleton-anchor detection (`!is_interactive && node.children_count.is_some() && has_label`), same bounds filtering, same compact-collapse, same `interactive_only` pruning. They differed in exactly one thing: whether each allocated `RefEntry` carried a `root_ref: None` or `root_ref: Some(root_ref_id.to_string())` tag.
|
||||
|
||||
The duplication did not land because an author was lazy. (session history) It landed because the March 10 implementation session used two parallel worktree agents — one owning `snapshot.rs` + `builder.rs`, the other owning `snapshot_ref.rs` + the macOS adapter. Each agent wrote its own allocator in isolation. When the merge agent integrated both worktrees, it hit the project's 400 LOC file limit on `snapshot.rs` and extracted `ref_alloc.rs` purely as a LOC-budget fix — pulling out the *small* helpers (`INTERACTIVE_ROLES`, `actions_for_role`, `ref_entry_from_node`, `is_collapsible`) while leaving the two full allocator bodies in their original files. The merge agent's summary explicitly listed `allocate_refs_with_root` and `DrillDownConfig` as distinct items belonging to `snapshot_ref.rs`. No decision was ever made to leave them separate for design reasons; it was the path of least resistance under a LOC deadline.
|
||||
|
||||
The brainstorm at `docs/brainstorms/2026-03-10-progressive-skeleton-traversal-brainstorm.md` described `ref_alloc.rs` as the home for "shared ref-allocation logic" but only named `INTERACTIVE_ROLES` and `is_collapsible`. It did not specify that the full allocator body should live there, leaving a gap the implementation filled with a copy.
|
||||
|
||||
## Guidance
|
||||
|
||||
**Any two function bodies that are structurally identical except for a small number of parameter values or literals are a single function waiting to be written.** The correct fix is a shared config struct whose fields default to — or accept an `Option<T>` for — the differentiating value. Copy-and-modify with a name suffix (`foo_with_X`, `foo_v2`, `foo_alternate`, `do_thing_for_Y`) is never the right tool; it is technical debt that generates silent divergence under the first patch.
|
||||
|
||||
**Concrete rule for this codebase:** `crates/core/src/ref_alloc.rs` is the single source of truth for the ref-allocation pass. `RefAllocConfig::root_ref_id` is `Option<&'a str>`:
|
||||
|
||||
- `snapshot.rs::build` (and `append_surface_refs`) passes `root_ref_id: None`
|
||||
- `snapshot_ref.rs::run_from_ref` passes `root_ref_id: Some(root_ref_id)`
|
||||
|
||||
The function body is identical for both callers. Any future change to allocation logic — adding a new role to `INTERACTIVE_ROLES`, changing skeleton-anchor detection, altering `is_collapsible`, tuning the `interactive_only` pruning rule — is made once in `ref_alloc::allocate_refs` and immediately applies to both the full-snapshot and the drill-down path.
|
||||
|
||||
**The threshold to extract a config struct:** (auto memory [claude]) when a function already takes four or more positional parameters and the new variant adds one more distinguishing value. Do not add an eighth positional parameter. Do not copy the body. Extract a config struct with all fields (including the new one as `Option<T>`) and unify.
|
||||
|
||||
**Applies to tests too.** Two test helper closures that differ only in a flag value are a single parametrised helper. The drill-down tests in `snapshot_ref.rs` already follow this via `drill_config(source_app, pid, root_ref_id, interactive_only, compact)` returning a `RefAllocConfig`.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
The divergence was not theoretical. (session history) Two separate commits drifted the copies in opposite directions before the duplication was even two weeks old:
|
||||
|
||||
**Incident 1 — `cec176d fix: preserve skeleton drill-down anchors`**: repaired logic that ensured skeleton anchors retained their `root_ref` tag correctly. The fix was applied only to `allocate_refs_with_root` in `snapshot_ref.rs`. The matching logic in `allocate_refs` in `snapshot.rs` was not touched. Snapshot and drill-down paths became silently inconsistent on anchor tagging.
|
||||
|
||||
**Incident 2 — `0fcf4e8 fix: bypass AXConfirm on web elements and add skeleton anchors to drill-down`**: added the `!is_interactive && node.children_count.is_some() && has_label` skeleton-anchor detection to `allocate_refs_with_root`. The same logic already existed in `allocate_refs` — it had been added there in the original `b13dc69 feat: implement progressive skeleton traversal` commit. The author of `0fcf4e8` had to check both copies, noticed the gap in the drill-down copy, and patched it. But the patch direction was reversed from `cec176d`: this time the snapshot copy was ahead and the drill-down copy was behind.
|
||||
|
||||
Two patches, two weeks, opposite directions of drift, within the same feature branch. That is the expected rate of rot for any copied core algorithm in this codebase.
|
||||
|
||||
**Review cost**: every reviewer on PR #20 had to understand both bodies to confirm they were equivalent. Because they lived in different files with different struct names (`window_pid` vs `pid`, seven positional args vs `&DrillDownConfig`), confirming equivalence required careful side-by-side reading rather than a single glance. The Cursor Bugbot eventually flagged it as "Low Severity". **Low was wrong.** (auto memory [claude]) Per `feedback_dry_is_core.md`, any duplicated core algorithm is P1 minimum, because the next fix will always land in exactly one copy.
|
||||
|
||||
## When to Apply
|
||||
|
||||
**Trigger 1 — identical recursive body.** If you are writing a recursive tree-walk and the only difference from an existing recursive tree-walk is one field on the entries it produces, unify. Recursive duplicates are particularly high-risk because the recursive self-call must also be updated in both copies on every future change.
|
||||
|
||||
**Trigger 2 — name suffix pattern.** Function names ending in `_with_X`, `_v2`, `_alternate`, `_for_Y` are red flags. Before adding such a function, ask: can the original accept an `Option<X>` parameter or a config struct with `X` as an optional field?
|
||||
|
||||
**Trigger 3 — config struct threshold.** Four or more positional parameters + one more distinguishing value = extract a config struct. This rule applies even if the existing function "looks fine" with seven args today; adding an eighth makes the call sites unreadable and invites the copy-and-modify shortcut at the next variant.
|
||||
|
||||
**Trigger 4 — parallel worktree seams.** (session history) When an implementation plan fans out into parallel worktree agents by file, be explicit in the plan about which functions are shared and name the exact helpers that will live in the shared module *before* the agents diverge. The March 10 plan named `INTERACTIVE_ROLES` and `is_collapsible` as shared but omitted `allocate_refs` — and the duplication followed from that omission.
|
||||
|
||||
**Trigger 5 — LOC-budget extraction.** When extracting code into a new module purely to fit a file-size limit, resist the urge to stop at the smallest extraction that clears the limit. Check whether the code that *stays behind* on one side now mirrors code on the other side. If it does, the LOC fix is also a DRY fix and both extractions belong together.
|
||||
|
||||
**Applies to review too.** (auto memory [claude]) Any two function bodies flagged by review tooling as "duplicated" get P1 severity minimum in this project. Low or P3 severity labels on duplication findings are wrong and should be escalated.
|
||||
|
||||
## Examples
|
||||
|
||||
### Before — two independent bodies, two files
|
||||
|
||||
**`crates/core/src/snapshot.rs`** (pre-`d06a6c2`, 7 positional params):
|
||||
|
||||
```rust
|
||||
fn allocate_refs(
|
||||
mut node: AccessibilityNode,
|
||||
refmap: &mut RefMap,
|
||||
include_bounds: bool,
|
||||
interactive_only: bool,
|
||||
compact: bool,
|
||||
window_pid: i32,
|
||||
source_app: Option<&str>,
|
||||
) -> AccessibilityNode {
|
||||
let is_interactive = INTERACTIVE_ROLES.contains(&node.role.as_str());
|
||||
|
||||
if is_interactive {
|
||||
let entry = ref_entry_from_node(&node, window_pid, source_app, None);
|
||||
node.ref_id = Some(refmap.allocate(entry));
|
||||
}
|
||||
|
||||
let has_label = node.name.as_deref().is_some_and(|n| !n.is_empty())
|
||||
|| node.description.as_deref().is_some_and(|d| !d.is_empty());
|
||||
let is_skeleton_anchor = !is_interactive && node.children_count.is_some() && has_label;
|
||||
|
||||
if is_skeleton_anchor {
|
||||
let mut entry = ref_entry_from_node(&node, window_pid, source_app, None);
|
||||
entry.available_actions = vec![];
|
||||
node.ref_id = Some(refmap.allocate(entry));
|
||||
}
|
||||
|
||||
if !include_bounds { node.bounds = None; }
|
||||
|
||||
node.children = node.children.into_iter()
|
||||
.filter_map(|child| {
|
||||
let child = allocate_refs(child, refmap, include_bounds,
|
||||
interactive_only, compact, window_pid, source_app);
|
||||
if compact && is_collapsible(&child) {
|
||||
return child.children.into_iter().next();
|
||||
}
|
||||
if interactive_only && child.ref_id.is_none()
|
||||
&& child.children.is_empty() && child.children_count.is_none() {
|
||||
None
|
||||
} else { Some(child) }
|
||||
})
|
||||
.collect();
|
||||
node
|
||||
}
|
||||
```
|
||||
|
||||
**`crates/core/src/snapshot_ref.rs`** (pre-`d06a6c2`, local `DrillDownConfig`):
|
||||
|
||||
```rust
|
||||
struct DrillDownConfig<'a> {
|
||||
include_bounds: bool,
|
||||
interactive_only: bool,
|
||||
compact: bool,
|
||||
pid: i32,
|
||||
source_app: Option<&'a str>,
|
||||
root_ref_id: &'a str,
|
||||
}
|
||||
|
||||
fn allocate_refs_with_root(
|
||||
mut node: AccessibilityNode,
|
||||
refmap: &mut RefMap,
|
||||
config: &DrillDownConfig,
|
||||
) -> AccessibilityNode {
|
||||
let is_interactive = INTERACTIVE_ROLES.contains(&node.role.as_str());
|
||||
|
||||
if is_interactive {
|
||||
let entry = ref_entry_from_node(
|
||||
&node, config.pid, config.source_app,
|
||||
Some(config.root_ref_id.to_string()), // <-- the only substantive difference
|
||||
);
|
||||
node.ref_id = Some(refmap.allocate(entry));
|
||||
}
|
||||
|
||||
// ... 40+ more lines structurally identical to allocate_refs
|
||||
}
|
||||
```
|
||||
|
||||
The only substantive difference between the two bodies was `None` vs `Some(config.root_ref_id.to_string())` in exactly two call sites.
|
||||
|
||||
### After — one body, one file, two callers
|
||||
|
||||
**`crates/core/src/ref_alloc.rs`** (post-`d06a6c2`):
|
||||
|
||||
```rust
|
||||
pub(crate) struct RefAllocConfig<'a> {
|
||||
pub include_bounds: bool,
|
||||
pub interactive_only: bool,
|
||||
pub compact: bool,
|
||||
pub pid: i32,
|
||||
pub source_app: Option<&'a str>,
|
||||
pub root_ref_id: Option<&'a str>, // None = full snapshot, Some(_) = drill-down
|
||||
}
|
||||
|
||||
pub(crate) fn allocate_refs(
|
||||
mut node: AccessibilityNode,
|
||||
refmap: &mut RefMap,
|
||||
config: &RefAllocConfig,
|
||||
) -> AccessibilityNode {
|
||||
let root_ref_owned = config.root_ref_id.map(str::to_string);
|
||||
let is_interactive = INTERACTIVE_ROLES.contains(&node.role.as_str());
|
||||
|
||||
if is_interactive {
|
||||
let entry = ref_entry_from_node(&node, config.pid, config.source_app, root_ref_owned.clone());
|
||||
node.ref_id = Some(refmap.allocate(entry));
|
||||
}
|
||||
|
||||
let has_label = node.name.as_deref().is_some_and(|n| !n.is_empty())
|
||||
|| node.description.as_deref().is_some_and(|d| !d.is_empty());
|
||||
let is_skeleton_anchor = !is_interactive && node.children_count.is_some() && has_label;
|
||||
|
||||
if is_skeleton_anchor {
|
||||
let mut entry = ref_entry_from_node(&node, config.pid, config.source_app, root_ref_owned);
|
||||
entry.available_actions = vec![];
|
||||
node.ref_id = Some(refmap.allocate(entry));
|
||||
}
|
||||
|
||||
if !config.include_bounds { node.bounds = None; }
|
||||
|
||||
node.children = node.children.into_iter()
|
||||
.filter_map(|child| {
|
||||
let child = allocate_refs(child, refmap, config);
|
||||
if config.compact && is_collapsible(&child) {
|
||||
return child.children.into_iter().next();
|
||||
}
|
||||
if config.interactive_only
|
||||
&& child.ref_id.is_none()
|
||||
&& child.children.is_empty()
|
||||
&& child.children_count.is_none()
|
||||
{
|
||||
None
|
||||
} else {
|
||||
Some(child)
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
|
||||
node
|
||||
}
|
||||
```
|
||||
|
||||
**`crates/core/src/snapshot.rs`** caller:
|
||||
|
||||
```rust
|
||||
let config = RefAllocConfig {
|
||||
include_bounds: opts.include_bounds,
|
||||
interactive_only: opts.interactive_only,
|
||||
compact: opts.compact,
|
||||
pid: window.pid,
|
||||
source_app: Some(window.app.as_str()),
|
||||
root_ref_id: None,
|
||||
};
|
||||
let mut tree = ref_alloc::allocate_refs(raw_tree, &mut refmap, &config);
|
||||
```
|
||||
|
||||
**`crates/core/src/snapshot_ref.rs`** caller:
|
||||
|
||||
```rust
|
||||
let config = RefAllocConfig {
|
||||
include_bounds: opts.include_bounds,
|
||||
interactive_only: opts.interactive_only,
|
||||
compact: opts.compact,
|
||||
pid: entry.pid,
|
||||
source_app,
|
||||
root_ref_id: Some(root_ref_id),
|
||||
};
|
||||
let mut tree = ref_alloc::allocate_refs(raw_tree, &mut refmap, &config);
|
||||
```
|
||||
|
||||
`DrillDownConfig` is deleted. `allocate_refs_with_root` is deleted. A future change to `INTERACTIVE_ROLES`, the skeleton-anchor condition, the `is_collapsible` rule, or the `interactive_only` pruning touches exactly one place.
|
||||
|
||||
**Net diff** (`d06a6c2 refactor: unify allocate_refs across snapshot and drill-down paths`): +139 / −190 across `ref_alloc.rs`, `snapshot.rs`, `snapshot_ref.rs`. 51 net LOC removed. `grep allocate_refs_with_root` returns zero matches. The follow-up `7c7837a chore: drop with_root from drill test names after allocator unification` renamed the leftover `test_allocate_refs_with_root_*` helpers to `test_drill_alloc_*` so the name is gone from the codebase entirely.
|
||||
|
||||
## Related
|
||||
|
||||
- `crates/core/src/ref_alloc.rs` — single source of truth for ref allocation
|
||||
- `crates/core/src/snapshot.rs` — full-snapshot caller
|
||||
- `crates/core/src/snapshot_ref.rs` — drill-down caller
|
||||
- Commit `d06a6c2` — the unifying refactor
|
||||
- Commit `7c7837a` — test name cleanup
|
||||
- `docs/plans/2026-03-10-feat-progressive-skeleton-traversal-plan.md` — original plan that split into parallel worktrees but did not name `allocate_refs` as a shared helper (moderate overlap but predates the duplication problem)
|
||||
- `docs/brainstorms/2026-03-10-progressive-skeleton-traversal-brainstorm.md` — feature brainstorm; named `INTERACTIVE_ROLES` and `is_collapsible` as shared helpers but omitted the allocator body
|
||||
|
|
@ -0,0 +1,70 @@
|
|||
---
|
||||
|
||||
## title: Harden progressive snapshot after review (skeleton reset, drill window, validation, depth)
|
||||
date: 2026-04-16
|
||||
category: logic-errors
|
||||
module: agent-desktop-core
|
||||
problem_type: logic_error
|
||||
component: tooling
|
||||
symptoms:
|
||||
- Repeated `snapshot --skeleton` runs grew `ref_count` and retained stale drill-down refs because skeleton reused a merged on-disk `RefMap` instead of replacing it.
|
||||
- `snapshot --root @eN` returned `window.id` empty and a synthetic title, breaking the same JSON contract as window-rooted snapshots.
|
||||
- Malformed `--root` values surfaced as `STALE_REF` after a failed map lookup instead of `INVALID_ARGS` from format validation.
|
||||
- Deep non-skeleton subtree builds could return `None` at `ABSOLUTE_MAX_DEPTH` with no truncation marker, so agents saw missing branches without explanation.
|
||||
root_cause: logic_error
|
||||
resolution_type: code_fix
|
||||
severity: high
|
||||
tags:
|
||||
- progressive-snapshot
|
||||
- skeleton
|
||||
- drill-down
|
||||
- refmap
|
||||
- snapshot
|
||||
- macos
|
||||
- agent-contract
|
||||
|
||||
# Harden progressive snapshot after review (skeleton reset, drill window, validation, depth)
|
||||
|
||||
## Problem
|
||||
|
||||
A focused review of the `feat/progressive-skeleton-traversal` work found several **contract and state** issues: skeleton mode did not behave like a clean refresh of overview refs, drill-down responses did not carry real window identity, invalid `--root` strings were classified like missing refs, and the macOS tree builder could silently stop at the absolute depth cap. Follow-up work aligned behavior with agent expectations and locked the behavior with tests and docs.
|
||||
|
||||
## Symptoms
|
||||
|
||||
- Refmap growth and stale drill refs after multiple skeleton snapshots (merged load + selective removal).
|
||||
- Drill-down JSON with empty `window.id` and generic title.
|
||||
- `bad-ref` style `--root` values reported as stale rather than invalid input.
|
||||
- Possible empty/missing subtrees at extreme depth without `children_count` or a boundary node.
|
||||
|
||||
## What Didn't Work
|
||||
|
||||
- **Treating skeleton as incremental merge** — convenient for caching, but agents and token budgets assume a fresh overview map unless documented otherwise.
|
||||
- **Synthetic `WindowInfo` only** — fast to ship, but breaks any workflow that keys on `window.id` or correlates with `list-windows`.
|
||||
- **Lookup-before-validate for `--root`** — reuses `stale_ref` for every missing key, including syntactically invalid IDs.
|
||||
|
||||
## Solution
|
||||
|
||||
1. **Full refmap replace on window snapshot** — `snapshot::build` always starts from `RefMap::new()` before `allocate_refs`, then `run` persists the result. Skeleton mode still passes `TreeOptions.skeleton` into `get_tree` for shallow traversal and boundary labeling; it no longer rehydrates prior refs for the overview path.
|
||||
2. **Resolve real window for drill-down** — `snapshot_ref::run_from_ref` calls `adapter.list_windows` and picks the window matching `entry.pid`, with a fallback to the previous synthetic `WindowInfo` if listing fails.
|
||||
3. **Validate `--root` early** — `commands/snapshot::execute` calls `validate_ref_id(root)` before `run_from_ref`, so malformed IDs return `invalid_input` / `INVALID_ARGS` instead of `STALE_REF`.
|
||||
4. **Observable cap at `ABSOLUTE_MAX_DEPTH`** — `build_subtree` returns a **leaf boundary node** with `children_count` when `raw_depth >= ABSOLUTE_MAX_DEPTH` instead of `None`, so deep drill-downs are not silently dropped.
|
||||
5. **Docs and tests** — `docs/phases.md` clarifies that `root` is CLI-only (`SnapshotArgs`), not `TreeOptions`; skills document `ref_id` in examples and batch `snapshot` args for `skeleton` / `root`. Integration tests cover skeleton depth, ref-count stability across refresh, invalid `--root`, and skeleton→drill flow.
|
||||
|
||||
## Why This Works
|
||||
|
||||
- **Reset semantics** match the mental model “snapshot the window again” — one `RefMap` per successful window snapshot save, no hidden accumulation from prior sessions on the same tree.
|
||||
- **Window resolution** restores a stable JSON shape for automation that round-trips with window listing APIs.
|
||||
- **Validation order** separates **bad syntax** from **stale or missing** refs, which is what agents need for recovery.
|
||||
- **Boundary nodes** make the absolute depth cap **visible** in the tree instead of a silent prune.
|
||||
|
||||
## Prevention
|
||||
|
||||
- When changing snapshot or ref persistence, add or extend an integration test that asserts `**ref_count` is stable** across two identical `snapshot --skeleton` runs for the same app/window.
|
||||
- Any new CLI flag that accepts a ref id should call `**validate_ref_id`** before map or adapter lookups.
|
||||
- For subtree-only APIs, if a hard depth cap exists, return a **node with `children_count`** (or an explicit hint field) rather than `None`.
|
||||
- Keep `**TreeOptions` vs CLI args** documented in `docs/phases.md` when adding root-like concepts so public architecture docs do not drift.
|
||||
|
||||
## Related Issues
|
||||
|
||||
- [Known pattern] DRY ref allocation — `[docs/solutions/best-practices/deduplicate-ref-allocator-via-config-struct-2026-04-14.md](../best-practices/deduplicate-ref-allocator-via-config-struct-2026-04-14.md)`
|
||||
- Plan: `docs/plans/2026-03-10-feat-progressive-skeleton-traversal-plan.md`
|
||||
|
|
@ -8,8 +8,9 @@ description: >
|
|||
Desktop automation via native OS accessibility trees using the agent-desktop CLI.
|
||||
Use when an AI agent needs to observe, interact with, or automate desktop applications
|
||||
(click buttons, fill forms, navigate menus, read UI state, toggle checkboxes, scroll,
|
||||
drag, type text, take screenshots, manage windows, use clipboard). Covers 54 commands
|
||||
across observation, interaction, keyboard/mouse, app lifecycle, clipboard, and wait.
|
||||
drag, type text, take screenshots, manage windows, use clipboard, manage notifications).
|
||||
Covers 53 commands across observation, interaction, keyboard/mouse, app lifecycle,
|
||||
notifications (macOS), clipboard, and wait.
|
||||
Triggers on: "click button", "fill form", "open app", "read UI", "automate desktop",
|
||||
"accessibility tree", "snapshot app", "type into field", "navigate menu", "toggle checkbox",
|
||||
"take screenshot", "desktop automation", "agent-desktop", or any desktop GUI interaction task.
|
||||
|
|
@ -44,27 +45,47 @@ Detailed documentation is split into focused reference files. Read them as neede
|
|||
| `references/workflows.md` | 12 common patterns: forms, menus, dialogs, scroll-find, drag-drop, async wait, anti-patterns |
|
||||
| `references/macos.md` | macOS permissions/TCC, AX API internals, smart activation chain, surfaces, Notification Center, troubleshooting |
|
||||
|
||||
## The Observe-Act Loop
|
||||
## The Observe-Act Loop (Progressive Skeleton Traversal)
|
||||
|
||||
Every automation follows this pattern:
|
||||
Use **progressive skeleton traversal** as the default approach. It reduces token consumption 78-96% for dense apps by exploring the UI in two phases: a shallow skeleton overview, then targeted drill-downs into regions of interest.
|
||||
|
||||
```
|
||||
1. OBSERVE → agent-desktop snapshot --app "App Name" -i
|
||||
2. REASON → Parse JSON, find target element by ref (@e1, @e2...)
|
||||
3. ACT → agent-desktop click @e5 (or type, select, toggle...)
|
||||
4. VERIFY → agent-desktop snapshot again to confirm state change
|
||||
5. REPEAT → Continue until task is complete
|
||||
1. SKELETON → agent-desktop snapshot --skeleton --app "App" -i --compact
|
||||
Parse the overview. Identify the region containing your target.
|
||||
Regions show children_count (e.g., "Sidebar" with children_count: 42).
|
||||
Named containers at truncation boundary have refs for drill-down.
|
||||
|
||||
2. DRILL → agent-desktop snapshot --root @e3 -i --compact
|
||||
Expand the target region. Now you see its interactive elements.
|
||||
|
||||
3. ACT → agent-desktop click @e12 (or type, select, toggle...)
|
||||
|
||||
4. VERIFY → agent-desktop snapshot --root @e3 -i --compact
|
||||
Re-drill the same region to confirm the state change.
|
||||
Scoped invalidation: only @e3's subtree refs are replaced.
|
||||
|
||||
5. REPEAT → Continue drilling other regions or acting as needed.
|
||||
```
|
||||
|
||||
Always snapshot before acting. Refs are snapshot-scoped and become stale after UI changes.
|
||||
**When to skip skeleton and use full snapshot instead:**
|
||||
- Simple apps with few elements (Finder, Calculator, TextEdit)
|
||||
- You already know the exact element name — use `find` instead
|
||||
- Surface snapshots (menus, sheets, alerts) — these are already focused
|
||||
|
||||
**When skeleton shines:**
|
||||
- Dense Electron apps (Slack, VS Code, Discord, Notion)
|
||||
- Any app where full snapshot exceeds ~50 refs
|
||||
- Multi-region workflows (sidebar + main content + toolbar)
|
||||
|
||||
## Ref System
|
||||
|
||||
- Refs assigned depth-first: `@e1`, `@e2`, `@e3`...
|
||||
- Only interactive elements get refs: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell
|
||||
- In skeleton mode, named/described containers at truncation boundary also get refs (drill-down targets with empty `available_actions`)
|
||||
- Static text, groups, containers remain in tree for context but have no ref
|
||||
- Refs are deterministic within a snapshot but NOT stable across snapshots if UI changed
|
||||
- After any action that changes UI, run `snapshot` again for fresh refs
|
||||
- After any action that changes UI, re-drill the affected region or re-snapshot
|
||||
- **Scoped invalidation:** re-drilling `--root @e3` only replaces refs from @e3's previous drill — refs from other regions and the skeleton itself are preserved
|
||||
|
||||
## JSON Output Contract
|
||||
|
||||
|
|
@ -89,16 +110,19 @@ Exit codes: `0` success, `1` structured error, `2` argument error.
|
|||
| `TIMEOUT` | Wait condition not met | Increase --timeout |
|
||||
| `INVALID_ARGS` | Bad arguments | Check command syntax |
|
||||
|
||||
## Command Quick Reference (54 commands)
|
||||
## Command Quick Reference (53 commands)
|
||||
|
||||
### Observation
|
||||
```
|
||||
agent-desktop snapshot --app "App" -i # Accessibility tree with refs
|
||||
agent-desktop screenshot --app "App" out.png # PNG screenshot
|
||||
agent-desktop find --app "App" --role button # Search elements
|
||||
agent-desktop get @e1 --property text # Read element property
|
||||
agent-desktop is @e1 --property enabled # Check element state
|
||||
agent-desktop list-surfaces --app "App" # Available surfaces
|
||||
agent-desktop snapshot --skeleton --app "App" -i --compact # Skeleton overview (preferred)
|
||||
agent-desktop snapshot --root @e3 -i --compact # Drill into region
|
||||
agent-desktop snapshot --app "App" -i # Full tree (simple apps)
|
||||
agent-desktop snapshot --app "App" --surface menu -i # Surface snapshot
|
||||
agent-desktop screenshot --app "App" out.png # PNG screenshot
|
||||
agent-desktop find --app "App" --role button # Search elements
|
||||
agent-desktop get @e1 --property text # Read element property
|
||||
agent-desktop is @e1 --property enabled # Check element state
|
||||
agent-desktop list-surfaces --app "App" # Available surfaces
|
||||
```
|
||||
|
||||
### Interaction
|
||||
|
|
@ -191,13 +215,13 @@ agent-desktop batch '[...]' --stop-on-error # Batch commands
|
|||
|
||||
## Key Principles for Agents
|
||||
|
||||
1. **Always snapshot first.** Never assume UI state.
|
||||
2. **Use `-i` flag.** Filters to interactive elements only, reducing tokens.
|
||||
3. **Refs are ephemeral.** Snapshot again after any UI-changing action.
|
||||
1. **Skeleton first, drill second.** Start with `--skeleton -i --compact` for dense apps. Drill into regions with `--root @ref`. Full snapshot only for simple apps.
|
||||
2. **Use `-i --compact` flags.** Filters to interactive elements and collapses empty wrappers, minimizing tokens.
|
||||
3. **Refs are ephemeral.** Re-drill the affected region after any UI-changing action. Scoped invalidation keeps other refs intact.
|
||||
4. **Prefer refs over coordinates.** `click @e5` > `mouse-click --xy 500,300`.
|
||||
5. **Use `wait` for async UI.** After launch/dialog triggers, wait for expected state.
|
||||
6. **Check permissions first.** Run `permissions` on first use.
|
||||
7. **Handle errors.** Parse `error.code` and follow `error.suggestion`.
|
||||
8. **Use `find` for targeted searches.** Faster than full snapshot when you know role/name.
|
||||
9. **Use surfaces for menus.** `snapshot --surface menu` captures open menus.
|
||||
8. **Use `find` for targeted searches.** Faster than any snapshot when you know role/name.
|
||||
9. **Use surfaces for overlays.** `snapshot --surface menu` for menus, `--surface sheet` for dialogs. Never `--skeleton` for surfaces — they're already focused.
|
||||
10. **Batch for performance.** Multiple commands in one invocation.
|
||||
|
|
|
|||
|
|
@ -22,6 +22,8 @@ agent-desktop snapshot --app "App" -i --compact
|
|||
| `--max-depth` | 10 | Maximum tree traversal depth |
|
||||
| `--include-bounds` | false | Include `{x, y, width, height}` for each element |
|
||||
| `--compact` | false | Omit empty structural nodes |
|
||||
| `--skeleton` | false | Shallow overview: clamps depth to min(max_depth, 3), adds `children_count` on truncated containers |
|
||||
| `--root <REF>` | | Start traversal from this ref instead of window root. Cannot be combined with `--surface` |
|
||||
| `--surface` | window | Target surface: `window`, `focused`, `menu`, `menubar`, `sheet`, `popover`, `alert` |
|
||||
|
||||
**Output structure:**
|
||||
|
|
@ -39,7 +41,7 @@ agent-desktop snapshot --app "App" -i --compact
|
|||
"name": "General",
|
||||
"children": [
|
||||
{
|
||||
"ref": "@e1",
|
||||
"ref_id": "@e1",
|
||||
"role": "button",
|
||||
"name": "About",
|
||||
"states": ["focused"]
|
||||
|
|
@ -49,7 +51,7 @@ agent-desktop snapshot --app "App" -i --compact
|
|||
"name": "Appearance",
|
||||
"children": [
|
||||
{
|
||||
"ref": "@e2",
|
||||
"ref_id": "@e2",
|
||||
"role": "checkbox",
|
||||
"name": "Dark Mode",
|
||||
"value": "0",
|
||||
|
|
@ -63,12 +65,36 @@ agent-desktop snapshot --app "App" -i --compact
|
|||
}
|
||||
```
|
||||
|
||||
**Skeleton mode (`--skeleton`):**
|
||||
- Produces a shallow overview by clamping depth to `min(max_depth, 3)`
|
||||
- Truncated containers include a `children_count` field showing how many children were omitted
|
||||
- Named or described containers at the truncation boundary receive refs with empty `available_actions`, serving as drill-down targets for `--root`
|
||||
|
||||
**Root mode (`--root <REF>`):**
|
||||
- Starts tree traversal from the given ref instead of the window root
|
||||
- Merges new refs into the existing refmap with scoped invalidation: only refs from the previous drill of the same root are replaced, leaving all other refs intact
|
||||
- Cannot be combined with `--surface`
|
||||
|
||||
**Progressive drill-down workflow:**
|
||||
```bash
|
||||
# Step 1: Get skeleton overview
|
||||
agent-desktop snapshot --skeleton --app Slack -i
|
||||
|
||||
# Step 2: Drill into a discovered region
|
||||
agent-desktop snapshot --root @e3 -i
|
||||
|
||||
# Step 3: Re-drill same region (scoped invalidation replaces @e3's refs)
|
||||
agent-desktop snapshot --root @e3 -i
|
||||
```
|
||||
|
||||
**Tips:**
|
||||
- Always use `-i` to keep output compact for LLM context windows
|
||||
- Use `--surface menu` to capture open context menus or dropdown menus
|
||||
- Use `--surface sheet` for modal dialogs
|
||||
- Use `--compact` with `-i` for maximum token efficiency
|
||||
- Combine `--max-depth 5` to limit deep trees (e.g., Xcode)
|
||||
- Use `--skeleton` first to get a high-level map, then `--root` to drill into specific regions
|
||||
- Combine `--skeleton` with `-i` and `--compact` for the most token-efficient initial overview
|
||||
|
||||
## find
|
||||
|
||||
|
|
@ -99,7 +125,7 @@ agent-desktop find --app "App" --role button --nth 2
|
|||
{
|
||||
"data": {
|
||||
"matches": [
|
||||
{ "ref": "@e5", "role": "button", "name": "OK", "states": ["enabled"] }
|
||||
{ "ref_id": "@e5", "role": "button", "name": "OK", "states": ["enabled"] }
|
||||
],
|
||||
"count": 1
|
||||
}
|
||||
|
|
|
|||
|
|
@ -217,6 +217,16 @@ Execute multiple commands in sequence from a JSON array. Each entry has `command
|
|||
]
|
||||
```
|
||||
|
||||
**Progressive snapshot in batch** — use `skeleton` and `root` fields inside `snapshot` args:
|
||||
```json
|
||||
[
|
||||
{ "command": "snapshot", "args": { "app": "Slack", "skeleton": true, "interactive_only": true } },
|
||||
{ "command": "snapshot", "args": { "app": "Slack", "root": "@e3", "interactive_only": true } }
|
||||
]
|
||||
```
|
||||
|
||||
`skeleton: true` clamps depth to 3 and tags truncated containers with `children_count`. `root: "@eN"` starts traversal from that ref instead of the window root; it cannot be combined with `surface`.
|
||||
|
||||
## System Health
|
||||
|
||||
### status
|
||||
|
|
|
|||
|
|
@ -13,25 +13,65 @@ agent-desktop permissions --request
|
|||
# Then: System Settings > Privacy & Security > Accessibility > enable your terminal
|
||||
```
|
||||
|
||||
## Pattern: Progressive Skeleton Traversal (Default for Dense Apps)
|
||||
|
||||
The recommended approach for Electron apps (Slack, VS Code, Discord) and any app with 50+ interactive elements. Reduces token consumption 78-96%.
|
||||
|
||||
```bash
|
||||
# 1. Get skeleton overview — shallow 3-level map with children_count hints
|
||||
agent-desktop snapshot --skeleton --app "Slack" -i --compact
|
||||
# Output shows regions like:
|
||||
# @e1 = group "Workspaces" (children_count: 4)
|
||||
# @e2 = group "Channels" (children_count: 42)
|
||||
# @e3 = group "Messages" (children_count: 156)
|
||||
# @e4 = button "New Message" ← interactive elements at top levels still get refs
|
||||
|
||||
# 2. Identify the region you need and drill into it
|
||||
agent-desktop snapshot --root @e2 -i --compact
|
||||
# Now you see all 42 children inside "Channels" with full refs
|
||||
|
||||
# 3. Act on an element found in the drill-down
|
||||
agent-desktop click @e18 # Click "general" channel
|
||||
|
||||
# 4. Re-drill the same or a different region to verify / continue
|
||||
agent-desktop snapshot --root @e3 -i --compact
|
||||
# Scoped invalidation: only @e3's previous refs are replaced
|
||||
# @e2's drill-down refs and the skeleton refs are preserved
|
||||
|
||||
# 5. Drill into another region as needed — refs accumulate
|
||||
agent-desktop snapshot --root @e1 -i --compact
|
||||
# Now you have refs from skeleton + @e2 drill + @e3 drill + @e1 drill
|
||||
```
|
||||
|
||||
**Key behaviors:**
|
||||
- `--skeleton` clamps depth to min(max_depth, 3) automatically
|
||||
- Named/described containers at the boundary get refs as drill-down targets
|
||||
- `--root @ref` merges new refs into the existing refmap
|
||||
- Re-drilling the same root replaces only that root's subtree refs
|
||||
- Interactive elements (buttons, textfields) within skeleton depth still get normal refs
|
||||
|
||||
## Pattern: Fill a Form
|
||||
|
||||
```bash
|
||||
# 1. Snapshot the form
|
||||
# For simple apps, full snapshot is fine
|
||||
agent-desktop snapshot --app "System Settings" -i
|
||||
|
||||
# 2. Parse output, identify text fields by name
|
||||
# For dense apps, use skeleton first to find the form region, then drill
|
||||
# agent-desktop snapshot --skeleton --app "System Settings" -i --compact
|
||||
# agent-desktop snapshot --root @e5 -i --compact
|
||||
|
||||
# Found: @e3 = "Computer Name" textfield, @e5 = "Local Hostname" textfield
|
||||
|
||||
# 3. Clear and fill each field
|
||||
# Clear and fill each field
|
||||
agent-desktop clear @e3
|
||||
agent-desktop type @e3 "My MacBook Pro"
|
||||
agent-desktop clear @e5
|
||||
agent-desktop type @e5 "my-macbook-pro"
|
||||
|
||||
# 4. Click the save/apply button
|
||||
# Click the save/apply button
|
||||
agent-desktop click @e8
|
||||
|
||||
# 5. Verify success
|
||||
# Verify success — re-snapshot or re-drill
|
||||
agent-desktop snapshot --app "System Settings" -i
|
||||
```
|
||||
|
||||
|
|
@ -98,15 +138,19 @@ agent-desktop snapshot --app "TextEdit" -i
|
|||
When the target element isn't visible and you need to scroll to find it:
|
||||
|
||||
```bash
|
||||
# 1. Find the scroll area
|
||||
agent-desktop snapshot --app "App" -i
|
||||
# Found: @e1 = scroll area
|
||||
# 1. Use skeleton to find the scrollable region
|
||||
agent-desktop snapshot --skeleton --app "App" -i --compact
|
||||
# Found: @e2 = group "Content" (children_count: 200)
|
||||
|
||||
# 2. Scroll and search in a loop
|
||||
agent-desktop scroll @e1 --direction down --amount 5
|
||||
# 2. Drill into the region to get a scroll area ref
|
||||
agent-desktop snapshot --root @e2 -i --compact
|
||||
# Found: @e8 = scroll area
|
||||
|
||||
# 3. Scroll and search in a loop
|
||||
agent-desktop scroll @e8 --direction down --amount 5
|
||||
agent-desktop find --app "App" --name "Target Item"
|
||||
# If no matches, scroll again
|
||||
agent-desktop scroll @e1 --direction down --amount 5
|
||||
agent-desktop scroll @e8 --direction down --amount 5
|
||||
agent-desktop find --app "App" --name "Target Item"
|
||||
# Found: @e14 = "Target Item"
|
||||
agent-desktop click @e14
|
||||
|
|
@ -168,8 +212,14 @@ agent-desktop wait --element @e10 --timeout 10000
|
|||
```bash
|
||||
# Full lifecycle
|
||||
agent-desktop launch "Calculator"
|
||||
# Simple app → full snapshot is fine
|
||||
agent-desktop snapshot --app "Calculator" -i
|
||||
|
||||
# Dense app → skeleton first
|
||||
# agent-desktop launch "Slack"
|
||||
# agent-desktop snapshot --skeleton --app "Slack" -i --compact
|
||||
# agent-desktop snapshot --root @e2 -i --compact
|
||||
|
||||
# ... perform automation ...
|
||||
|
||||
agent-desktop close-app "Calculator"
|
||||
|
|
@ -216,11 +266,12 @@ agent-desktop batch '[
|
|||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
1. **Acting without observing.** Never click a ref without a recent snapshot.
|
||||
2. **Hardcoding refs.** Refs change between snapshots. Always use fresh refs.
|
||||
3. **Ignoring wait.** After launch, dialog triggers, or menu clicks — always wait before snapshotting.
|
||||
4. **Using coordinates when refs exist.** AX-based actions are more reliable than coordinate clicks.
|
||||
5. **Not checking permissions.** Always verify accessibility permission before starting automation.
|
||||
6. **Deep snapshots of large apps.** Use `--max-depth 5` and `-i` for Xcode, VS Code, etc.
|
||||
7. **Assuming UI stability.** Re-snapshot after every action that could change the UI.
|
||||
8. **Snapshotting the full window when an overlay is open.** When a sheet, alert, popover, or menu is visible, use `--surface sheet/alert/popover/menu` instead. The full window tree includes irrelevant background refs that waste tokens and can't be interacted with while the overlay has focus.
|
||||
1. **Full snapshot on dense apps.** Use `--skeleton` + `--root` for Electron apps (Slack, VS Code, Discord). Full snapshot wastes 4-25x more tokens.
|
||||
2. **Acting without observing.** Never click a ref without a recent snapshot or drill-down.
|
||||
3. **Hardcoding refs.** Refs change between snapshots. Always use fresh refs.
|
||||
4. **Ignoring wait.** After launch, dialog triggers, or menu clicks — always wait before snapshotting.
|
||||
5. **Using coordinates when refs exist.** AX-based actions are more reliable than coordinate clicks.
|
||||
6. **Not checking permissions.** Always verify accessibility permission before starting automation.
|
||||
7. **Assuming UI stability.** Re-drill the affected region after every action that could change the UI.
|
||||
8. **Snapshotting the full window when an overlay is open.** Use `--surface sheet/alert/popover/menu` instead. Never `--skeleton` for surfaces — they're already focused.
|
||||
9. **Re-snapshotting everything after one action.** Use scoped re-drill (`--root @ref`) to refresh only the affected region. Other refs stay valid.
|
||||
|
|
|
|||
|
|
@ -39,6 +39,11 @@ pub fn dispatch_batch_command(
|
|||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false),
|
||||
surface: parse_batch_surface(args.get("surface").and_then(|v| v.as_str())),
|
||||
skeleton: args
|
||||
.get("skeleton")
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false),
|
||||
root_ref: str_field(&args, "root"),
|
||||
},
|
||||
adapter,
|
||||
),
|
||||
|
|
|
|||
|
|
@ -50,6 +50,13 @@ pub struct SnapshotArgs {
|
|||
pub compact: bool,
|
||||
#[arg(long, value_enum, default_value_t = Surface::Window, help = "Surface to snapshot")]
|
||||
pub surface: Surface,
|
||||
#[arg(
|
||||
long,
|
||||
help = "Shallow overview with children_count on truncated containers"
|
||||
)]
|
||||
pub skeleton: bool,
|
||||
#[arg(long, help = "Start traversal from this ref instead of window root")]
|
||||
pub root: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Parser, Debug)]
|
||||
|
|
|
|||
|
|
@ -27,6 +27,8 @@ pub fn dispatch(cmd: Commands, adapter: &dyn PlatformAdapter) -> Result<Value, A
|
|||
interactive_only: a.interactive_only,
|
||||
compact: a.compact,
|
||||
surface: a.surface.to_core(),
|
||||
skeleton: a.skeleton,
|
||||
root_ref: a.root,
|
||||
},
|
||||
adapter,
|
||||
),
|
||||
|
|
|
|||
44
tests/fixtures/drilldown-refmap.json
vendored
Normal file
44
tests/fixtures/drilldown-refmap.json
vendored
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
{
|
||||
"skeleton_anchors": [
|
||||
{
|
||||
"ref_id": "@e1",
|
||||
"pid": 42,
|
||||
"role": "group",
|
||||
"name": "Sidebar",
|
||||
"root_ref": null
|
||||
},
|
||||
{
|
||||
"ref_id": "@e2",
|
||||
"pid": 42,
|
||||
"role": "group",
|
||||
"name": "Toolbar",
|
||||
"root_ref": null
|
||||
}
|
||||
],
|
||||
"drilled_from_e1": [
|
||||
{
|
||||
"ref_id": "@e3",
|
||||
"pid": 42,
|
||||
"role": "treeitem",
|
||||
"name": "Recents",
|
||||
"root_ref": "@e1"
|
||||
},
|
||||
{
|
||||
"ref_id": "@e4",
|
||||
"pid": 42,
|
||||
"role": "treeitem",
|
||||
"name": "Documents",
|
||||
"root_ref": "@e1"
|
||||
}
|
||||
],
|
||||
"drilled_from_e2": [
|
||||
{
|
||||
"ref_id": "@e5",
|
||||
"pid": 42,
|
||||
"role": "button",
|
||||
"name": "Back",
|
||||
"root_ref": "@e2"
|
||||
}
|
||||
],
|
||||
"expected_total": 5
|
||||
}
|
||||
35
tests/fixtures/skeleton-tree.json
vendored
Normal file
35
tests/fixtures/skeleton-tree.json
vendored
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
{
|
||||
"ref_id": null,
|
||||
"role": "window",
|
||||
"name": "Test Window",
|
||||
"children": [
|
||||
{
|
||||
"ref_id": "@e1",
|
||||
"role": "group",
|
||||
"name": "Sidebar",
|
||||
"children_count": 26
|
||||
},
|
||||
{
|
||||
"ref_id": "@e2",
|
||||
"role": "group",
|
||||
"description": "Channels and direct messages",
|
||||
"children_count": 12
|
||||
},
|
||||
{
|
||||
"role": "group",
|
||||
"name": "Content",
|
||||
"children": [
|
||||
{
|
||||
"ref_id": "@e3",
|
||||
"role": "button",
|
||||
"name": "Send"
|
||||
},
|
||||
{
|
||||
"ref_id": "@e4",
|
||||
"role": "textfield",
|
||||
"name": "Message"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -69,6 +69,142 @@ mod tests {
|
|||
assert!(json["data"]["version"].is_string());
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[cfg(target_os = "macos")]
|
||||
#[ignore = "requires Accessibility permissions and running macOS apps"]
|
||||
fn snapshot_skeleton_returns_shallow_tree_with_children_count() {
|
||||
let bin = agent_desktop_bin();
|
||||
let output = Command::new(&bin)
|
||||
.args(["snapshot", "--app", "Finder", "--skeleton", "-i"])
|
||||
.output()
|
||||
.expect("failed to run agent-desktop");
|
||||
|
||||
let stdout = String::from_utf8_lossy(&output.stdout);
|
||||
let json: serde_json::Value =
|
||||
serde_json::from_str(&stdout).expect("output is not valid JSON");
|
||||
|
||||
assert_eq!(json["ok"], true);
|
||||
let tree = &json["data"]["tree"];
|
||||
let max_depth = find_max_depth(tree, 0);
|
||||
assert!(
|
||||
max_depth <= 4,
|
||||
"skeleton must clamp to depth ~3, got depth {max_depth}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[cfg(target_os = "macos")]
|
||||
#[ignore = "requires Accessibility permissions and running macOS apps"]
|
||||
fn snapshot_skeleton_refresh_does_not_accumulate_stale_refs() {
|
||||
let bin = agent_desktop_bin();
|
||||
let run = |extra: &[&str]| {
|
||||
let mut args = vec!["snapshot", "--app", "Finder", "--skeleton", "-i"];
|
||||
args.extend_from_slice(extra);
|
||||
Command::new(&bin)
|
||||
.args(&args)
|
||||
.output()
|
||||
.expect("failed to run agent-desktop")
|
||||
};
|
||||
|
||||
let first = run(&[]);
|
||||
let first_json: serde_json::Value =
|
||||
serde_json::from_str(&String::from_utf8_lossy(&first.stdout)).unwrap();
|
||||
let first_count = first_json["data"]["ref_count"].as_u64().unwrap_or(0);
|
||||
|
||||
let second = run(&[]);
|
||||
let second_json: serde_json::Value =
|
||||
serde_json::from_str(&String::from_utf8_lossy(&second.stdout)).unwrap();
|
||||
let second_count = second_json["data"]["ref_count"].as_u64().unwrap_or(0);
|
||||
|
||||
assert_eq!(
|
||||
first_count, second_count,
|
||||
"repeated skeleton refresh must produce identical ref_count (no accumulation)"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn snapshot_invalid_root_ref_format_returns_invalid_args() {
|
||||
let bin = agent_desktop_bin();
|
||||
if !bin.exists() {
|
||||
return;
|
||||
}
|
||||
let output = Command::new(&bin)
|
||||
.args(["snapshot", "--app", "Finder", "--root", "bad-ref"])
|
||||
.output()
|
||||
.expect("failed to run agent-desktop");
|
||||
|
||||
let stdout = String::from_utf8_lossy(&output.stdout);
|
||||
let json: serde_json::Value =
|
||||
serde_json::from_str(&stdout).expect("output is not valid JSON");
|
||||
|
||||
assert_eq!(json["ok"], false);
|
||||
assert_eq!(
|
||||
json["error"]["code"], "INVALID_ARGS",
|
||||
"malformed --root must return INVALID_ARGS, got: {}",
|
||||
json["error"]["code"]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[cfg(target_os = "macos")]
|
||||
#[ignore = "requires Accessibility permissions and running macOS apps"]
|
||||
fn snapshot_root_drill_returns_non_empty_subtree() {
|
||||
let bin = agent_desktop_bin();
|
||||
let skeleton_out = Command::new(&bin)
|
||||
.args(["snapshot", "--app", "Finder", "--skeleton", "-i"])
|
||||
.output()
|
||||
.expect("failed to run agent-desktop");
|
||||
|
||||
let skeleton_json: serde_json::Value =
|
||||
serde_json::from_str(&String::from_utf8_lossy(&skeleton_out.stdout)).unwrap();
|
||||
assert_eq!(skeleton_json["ok"], true);
|
||||
|
||||
let first_ref = first_ref_id(&skeleton_json["data"]["tree"]);
|
||||
let Some(ref_id) = first_ref else {
|
||||
return;
|
||||
};
|
||||
|
||||
let drill_out = Command::new(&bin)
|
||||
.args(["snapshot", "--app", "Finder", "--root", &ref_id, "-i"])
|
||||
.output()
|
||||
.expect("failed to run agent-desktop");
|
||||
|
||||
let drill_json: serde_json::Value =
|
||||
serde_json::from_str(&String::from_utf8_lossy(&drill_out.stdout)).unwrap();
|
||||
|
||||
assert_eq!(drill_json["ok"], true);
|
||||
assert!(
|
||||
drill_json["data"]["ref_count"].as_u64().unwrap_or(0) > 0,
|
||||
"drill-down must return refs"
|
||||
);
|
||||
}
|
||||
|
||||
fn find_max_depth(node: &serde_json::Value, depth: usize) -> usize {
|
||||
let children = match node.get("children").and_then(|c| c.as_array()) {
|
||||
Some(c) if !c.is_empty() => c,
|
||||
_ => return depth,
|
||||
};
|
||||
children
|
||||
.iter()
|
||||
.map(|c| find_max_depth(c, depth + 1))
|
||||
.max()
|
||||
.unwrap_or(depth)
|
||||
}
|
||||
|
||||
fn first_ref_id(node: &serde_json::Value) -> Option<String> {
|
||||
if let Some(r) = node.get("ref_id").and_then(|v| v.as_str()) {
|
||||
return Some(r.to_string());
|
||||
}
|
||||
if let Some(children) = node.get("children").and_then(|c| c.as_array()) {
|
||||
for child in children {
|
||||
if let Some(r) = first_ref_id(child) {
|
||||
return Some(r);
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn list_apps_on_non_macos_errors_gracefully() {
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
|
|
|
|||
Loading…
Reference in a new issue