mirror of
https://github.com/lahfir/agent-desktop.git
synced 2026-07-26 17:12:15 +00:00
9 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Lahfir
|
53d6800aca |
docs: align README and repo docs with session-first trace
Co-authored-by: Cursor <cursoragent@cursor.com> |
||
Lahfir
|
ce232787b5
|
feat: add --wait-for selector polling flags (#86)
Global --wait-for / --wait-for-gone (-w) and --wait-timeout flags that poll the accessibility tree until a role:text selector appears or disappears, then return the snapshot. Honored by snapshot and the 16 ref-action commands; post-action waits scope to the acted-on window. Closes #84. |
||
|
Lahfir
|
9023f331b3
|
feat(ffi): Phase B and C — Python smoke harness, parity gates, build.rs codegen (#77)
Completes the FFI plan: U9 adds a Python ctypes smoke harness proving the C ABI from a non-Rust host; U10 adds cross-platform parity CI gates for header drift, codegen drift, panic-unwind, and stub-adapter passthrough; U11 generates the command-backed wrappers from build.rs templates with a committed drift-gated output. The C ABI stays byte-identical and the surface is Windows and Linux ready with no new FFI code. |
||
|
Lahfir
|
0c0a5b8dbc
|
fix: harden agent-desktop reliability (#54)
* feat: add strict ref reliability core
* feat: add session-scoped reliability diagnostics
* fix: skip finder pseudo windows for snapshots
* fix: report wait and trace failure context
* fix: harden ref action reliability
* fix: close reliability review findings
* fix: harden reliability edge cases
* fix: close final reliability edge cases
* fix: address reliability follow-ups
* docs: update reliability docs and skills
* fix: harden wait and ref action reliability
* refactor: centralize reliability helpers
* fix: stabilize macos ref resolution
* refactor: organize binary crate modules
* fix: harden ref action reliability
* fix: harden ref action reliability
* docs: compound reliability patterns
* fix: harden ref reliability edge cases
* fix: harden source-window ref resolution
* fix: preserve safe window title fallback
* fix: make explicit snapshots session-independent
* fix: harden ref fallback resolution
* fix: fail closed on uncertain ref fallback
* chore: strip inline comments and enforce docstrings
Replace inline // comments with /// docstrings where they carry non-obvious
contract, and add a pre-commit guard so inline comments cannot regress.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(review): preserve action results, delegate timeout resolution, tighten core boundaries
Apply verified code-review fixes: ref-action release failures no longer
mask successful action results (prevents double-dispatch on retry);
resolve_element_strict_with_timeout defaults to delegating so strict-only
adapters support wait --element; wait --text reports count only when
--count is requested; latest-refmap refresh logs load failures instead of
silently serving stale refs; InteractionPolicy moved to its own module and
actionability/trace modules scoped pub(crate) per file rules; duplicate
wait test helper extracted to shared support module; timeout error
constructors deduplicated; redaction test covers description; policy
focus-denial path covered; skills document steps array, actionability
details, trace redaction, and batch trace inheritance.
* fix(review): eliminate per-element AX round trips and close remaining review findings
Fold AXPosition/AXSize and the scrollbar probe into the existing
AXUIElementCopyMultipleAttributeValues batch so tree traversal and the
actionability preflight pay one IPC per element instead of up to four;
A/B benchmark shows strictly-faster snapshots with identical ref counts
and scroll capabilities (Finder 4.5s -> 2.4s same-session, Docker
Desktop parity at 440 refs / 933 scroll-capable nodes).
Consolidate the CLI and FFI ref-action pipelines into one core
execute_resolved path (actionability, tracing, and dispatch semantics
live once; FFI passes a default context). Remove the no-context
execute() shims from is/right-click/snapshot/wait and the test-only
helper shims; every command now takes an explicit CommandContext.
Split the macOS resolver into resolve (orchestration), resolve_search
(candidate collection), and resolve_classify (strict classification),
clearing the 400-LOC ceiling with room to grow.
Notification waits now retry transient baseline failures inside the
timeout budget with the same retryable gate as window/text waits and
report last_error in timeout details instead of aborting on the first
flake; a baseline is never fabricated.
Refmap writes clean up their temp file on failure, stale *.tmp orphans
are swept under the store lock, and save_existing_snapshot re-verifies
snapshot ownership inside the owning store's write lock with bounded
re-discovery before deterministically recreating in the caller's store.
Coverage hardening: zero-budget wait timeout shape, wait --text
--count 0 absence detection, ref-action pipeline call-count guard
(1 resolve / 1 live read / 1 dispatch), duplicate snapshot-id collision
on load, pruned-everywhere recreation, tmp sweep and rename-failure
cleanup, FFI AMBIGUOUS_TARGET last-error code assertion.
* docs: document notification retry, error-code contract, and diagnostics sensitivity
Note transient-error retry and last_error timeout detail on wait
--notification; make explicit that agents branch on error.code (message
and suggestion text is informational); warn FFI consumers that
ad_last_error_details may carry on-screen element names, values, and
window titles and should stay out of shared log surfaces.
* fix: validate find roles against the canonical vocabulary
find --role with a role no adapter can emit (textarea, typos) silently
returned ok with zero matches, reading as 'element absent' when the
query could never match. Role queries now resolve through a canonical
vocabulary in core: common text-input aliases (textarea, textbox,
searchfield) normalize to textfield case-insensitively, and unknown
roles fail with INVALID_ARGS carrying details.valid_roles so agents can
self-correct.
The macOS role mapping becomes a sorted single-source table with
binary-search lookup, and a conformance test pins every emitted role
(plus the unknown fallback and the synthesized cell role) to core's
CANONICAL_ROLES — the cross-platform contract Windows/Linux adapters
must map their native vocabularies into, enforced by the same
table + test pattern rather than convention.
* refactor: derive find role hints from the live tree, drop hardcoded vocabulary
Replace the canonical-role allow-list (and its hard INVALID_ARGS
rejection) with a tree-derived approach. A role filter that matches
nothing now returns ok with roles_present — the distinct roles actually
in the searched tree — so the caller distinguishes 'none on screen' from
a wrong role name and self-corrects. This needs no central role list: a
role any adapter newly emits surfaces in roles_present automatically,
with nothing to keep in sync across core and the platform crates.
A tiny role-query normalizer keeps the ergonomic win (textarea, textbox,
searchfield fold to textfield, case-insensitive) but never gates or
rejects — it is a synonym shim, not a vocabulary. The macOS role table
returns to its plain match form; the cross-crate canonical-vocabulary
list and its conformance test are gone.
* refactor: move protected-process knowledge out of core into the adapter
close-app hardcoded macOS/Unix process names (loginwindow, windowserver,
dock, launchd, finder) inside core, baking platform-specific knowledge
into the platform-agnostic crate. Windows would need csrss.exe/
winlogon.exe, Linux gnome-shell/Xorg. Add PlatformAdapter::
is_protected_process (default denies nothing); the macOS adapter owns its
list with substring matching over display and bundle identifiers. core's
close-app just asks the adapter. Also genericize a macOS-flavored test
fixture string so core carries zero native vocabulary even in tests.
Verified: core has no platform-native references in non-test source, no
cfg(target_os) gates; the only remaining cfg(unix)/libc use is securing
core's own refmap/trace/lock files with non-unix fallbacks.
* fix: stop close-app claiming a graceful quit it cannot confirm
close-app returned closed:true the instant a graceful quit was *sent*,
while the app was still running behind an unsaved-changes dialog —
a false completion claim. Empirically (NSWorkspace.runningApplications):
a clean quit completes in ~0.2s, a dialog-blocked quit never completes
on its own, and macOS confirms only that the quit request was sent, not
that the app terminated. Verifying by polling would add seconds of
latency on the exact (blocked) case it is meant to catch, so we do not
poll.
Graceful close now reports { method: graceful, requested: true } —
truthful and instant, no closed claim. --force is a synchronous SIGKILL,
so it reports { method: force, requested: true, closed: true }. Callers
needing graceful confirmation observe via list-apps / wait --window and
can drive a save dialog with snapshot + find, which is the agent-native
path.
* feat: add drag --drop-delay for reliable macOS drop registration
macOS drop targets need the dragged item to dwell over them before they
register as the destination; too short and the gesture lands as a drag
with no drop. The dwell was a hardcoded 500ms dead sleep. Expose it as
--drop-delay <ms> (CLI), drop_delay_ms (DragParams/AdDragParams, 0 =
adapter default sentinel matching duration_ms), and replace the dead
sleep with an event-driven dwell that posts LeftMouseDragged over the
destination every 16ms so the target stays highlighted instead of
dropping the drag mid-pause.
DRY: the C-to-core drag conversion (duration/drop-delay zero-sentinel)
was copied across three FFI sites; collapse them into AdDragParams::
to_core(). FFI ABI: AdDragParams gains drop_delay_ms (header + repr +
header-compile test). Tests: core threads the value into params and
response and omits the field when unset; FFI maps both optionals.
* fix: make action-bearing elements ref-able so scroll/expand can target them
E2E testing against a diverse fixture app surfaced that disclosure
(Expand/Collapse/Click) and scrollarea (Scroll) advertise actions but
never received refs — they are not in INTERACTIVE_ROLES — so the scroll,
expand, and collapse commands required a <REF> their own target roles
could never have. The commands were uninvokable against their primary
targets.
Ref allocation now gates on addressability, not role alone: an element
is ref-able if its role is interactive OR it advertises a primary action
(any action other than a bare SetFocus, which would ref-allocate inert
focusable containers). scrollarea and disclosure become ref-able;
scroll now works against a real app. Ref-count impact is modest
(fixture 61->72, Finder ~262).
Tests assert action-bearing containers get refs, SetFocus-only and inert
elements do not, and interactive roles stay ref-able without actions.
Contract docs (CLAUDE.md, SKILL.md) updated.
* fix: eliminate vacuous AX successes and harden resolution
Dogfooding the binary against a real fixture app surfaced five cases where a
command reported success without producing the effect, or failed with the wrong
error. Each is verified by independent before/after observation in the E2E
harness.
- is_menu_open no longer treats a latent AXMenuBar as an open menu, so select
and wait --menu-closed stop seeing a permanently-open menu.
- set-value coerces the written AXValue to the element's existing CFNumber/
CFBoolean/CFString type and verifies numerically, fixing sliders; steppers
converge via AXIncrement/AXDecrement when AXValue writes are vacuous.
- double-click only claims success when the element advertises AXOpen; otherwise
it fails closed instead of reporting a non-existent double-click.
- a completed resolution pass that proves a ref absent downgrades a deadline
TIMEOUT to STALE_REF so removed elements fail with the correct code.
- expand/collapse verify the disclosure state and fall back to a press-toggle
for press-driven disclosures; press-toggled containers expose EXPAND/COLLAPSE.
roles.rs adds the disclosure expandable role and normalizes textarea/textbox/
searchfield role queries to textfield.
* feat: add Playwright-style headed/headless interaction mode
Ref actions now run in exactly two modes. Headless is the default: semantic
accessibility operations only, no cursor movement, and a fail-closed
POLICY_DENIED when only a physical gesture would work. The global --headed flag
upgrades every ref action to permit focus stealing and cursor movement, so the
chain's physical click/double-click/scroll/keypress fallbacks can complete. The
AX path is always tried first, so --headed never regresses headless-capable
elements; it only adds fallbacks for elements that need a real gesture.
- CommandContext::request(action, base) builds the per-command request: each
command declares its headless base (pure-AX headless; type uses focus_fallback
because typing requires focus but never moves the cursor) and --headed upgrades
any base to the headed policy.
- The internal/FFI "physical" policy is renamed "headed" throughout, including
the C ABI enum (AD_POLICY_KIND_HEADED keeps discriminant 2) and bindings.
- Raw-input commands (press, hover, drag, mouse-*, key-down/up) are unchanged:
always physical, mode-independent low-level escape hatch.
- Unit tests assert every ref command is headless by default and headed under
--headed; docs (CONCEPTS, CLAUDE, skills) describe the two-mode contract.
* test: add E2E fixture app and dual-mode harness
Drives the release binary against a real SwiftUI/AppKit fixture and verifies
every effect by independent before/after observation — never the command's own
ok:true — so a command that reports success without an effect is caught. This is
the layer mock-adapter unit tests cannot cover: it exercises the contract
against the real macOS Accessibility API.
- AgentDeskFixture.swift exposes a fixed, diverse AX surface (native AppKit
slider/stepper, gesture-only and ambiguous controls, a sheet, a press-toggled
disclosure, async-appearing elements, a drag canvas). It is never tuned to
make a command pass; a failure is a finding about the CLI or the harness.
- run.sh drives every ref-action command in BOTH headless and --headed mode with
mode-specific target values, plus the double-click discriminator (headless
fails closed with POLICY_DENIED, --headed completes) that proves the two modes
differ. It also covers strict resolution, wait predicates, skeleton drill-down,
sessions, trace redaction, surfaces, drag, expand, and force-close.
- The compiled fixture .app is a build artifact (gitignored; built on demand).
Run: cargo build --release && bash tests/e2e/run.sh (needs AX permission).
* fix: close review runtime, correctness, and security gaps
Addresses validated findings from the branch code review:
- chain: thread the chain deadline into increment_to_value so a non-converging
stepper cannot spin up to 1024 AX round trips and blow past the timeout.
- mouse: a RAII guard posts LeftMouseUp if any fallible step of a drag returns
early, so an error can never leave the mouse button held down system-wide.
- ffi: cap caller-supplied state/action/path counts before from_raw_parts,
mirroring the existing MAX_MODIFIERS_PER_COMBO guard, to reject out-of-bounds
reads from a garbage C count.
- wait: the actionable predicate now forwards the structured ActionabilityReport
from error.details instead of dropping it to a flat message, so agents can see
which check is blocking.
- scroll: gate the row-select fallback on policy.allow_focus_steal so a headless
scroll can no longer silently change the user's table selection.
- actionability: delete the unreachable stable->StaleRef branch (stability_check
never fails by design) and its now-dead failed_check helper.
- status: delete two pub wrappers with no production callers; the test now drives
the real execute_with_report_with_context entry point.
* refactor: extract disclosure chain steps under the 400-LOC limit
chain_steps.rs had grown to 402 lines, over the hard per-file limit. The six
press-toggle disclosure helpers form a cohesive group and move cleanly into a
sibling chain_disclosure_steps.rs (following the chain_web_steps/
chain_menu_steps pattern); chain_defs.rs references the new module. No behavior
change.
* feat: focus the target window before ref-addressed physical input
Ref-action physical fallbacks (click/scroll/type) already brought the target
app frontmost before synthesizing CGEvents. The raw-input commands (drag, hover,
mouse-*) resolving a point from a ref did not — the resolver had the pid but
discarded it, so the adapter saw only coordinates and synthetic events could
land on whatever window happened to be frontmost.
Add a best-effort focus_app(pid) to PlatformAdapter (macOS uses
ensure_app_focused; other adapters default to not_supported). The point resolver
now focuses the ref's app before returning, so every physical interaction that
targets a known element raises its window first. Coordinate (--xy) input is
unchanged: the caller owns the target there.
* fix: add AdDragParams size guard and document the ABI breaks
AdDragParams gained a drop_delay_ms field but, unlike AdRefEntry, had no size
guard — an old caller's smaller allocation would let Rust read past it and turn
stack garbage into a real drop delay. Add AD_DRAG_PARAMS_SIZE, ad_drag_params_size(),
a compile-time layout assertion, and a zero-init note, matching the ref-entry
pattern.
This branch makes several consumer-visible contract changes that release
tooling must cut as a major. They are gathered here because the release workflow
ships the C header as an artifact.
BREAKING CHANGE: the C ABI and CLI/JSON contract changed on this branch.
- AdPolicyKind: AD_POLICY_KIND_PHYSICAL is renamed AD_POLICY_KIND_HEADED
(discriminant 2 unchanged, so compiled binaries are safe; source-level C
consumers must rename). No back-compat alias is kept — "physical" is gone.
- AdRefEntry grew (caller-allocated input); validate layout with
AD_REF_ENTRY_SIZE / ad_ref_entry_size().
- AdDragParams grew; validate with AD_DRAG_PARAMS_SIZE / ad_drag_params_size()
and zero-initialize before use.
- close-app graceful response no longer includes closed:true; it returns
{ method: "graceful", requested: true } because a graceful quit cannot be
synchronously confirmed.
* test: harden and expand the E2E proof layer
Closes the honesty gaps the review found and covers the interactions that were
missing, all verified by independent before/after observation:
- twins: the fixture twins now record distinct effects (twin-a/twin-b) and the
assert requires the ADDRESSED twin to fire (or AMBIGUOUS_TARGET), instead of
passing on any ok:true.
- click: click-status is a counter, so the headed pass must observe a fresh
increment rather than inheriting the headless pass's value.
- adds triple-click + hover (headed gestures), tab selection (TabView tabs are
radiobuttons), context-menu open + item selection, and menu-bar enumeration
via --surface menubar.
- adds a performance section reporting per-command CLI wall-clock (snapshot,
find, get, click, set-value, type) with a soft <2s snapshot gate.
- documents the SwiftUI CommandMenu and cross-app drop limitations as tracked
notes, not silent skips.
* docs: trim CLAUDE.md to standards and document gesture headless-capability
- CLAUDE.md: remove ~130 lines of reference material that duplicated code,
Cargo.toml, or the skills (full PlatformAdapter trait dump, Key Types listing,
macOS API listings, dependency/build-config tables, the 54-command table).
Replaced the stale trait dump with a pointer to adapter.rs (which also fixes
the review's stale-trait-docs finding) and kept only the non-obvious gotchas.
CLAUDE.md is now standards, invariants, and conventions.
- Document, on macOS (Phase 1), which gestures have a headless path: most ref
actions do; double-click via AXOpen; triple-click/hover/drag are cursor
gestures with no AX equivalent (physical only). The command surface is
platform-agnostic — a future Windows/Linux adapter that exposes a headless
path lights it up with no command or core change. Added to README and the
interaction reference.
- FFI skill: AD_POLICY_KIND_PHYSICAL is now AD_POLICY_KIND_HEADED.
* docs: capture gesture headless-capability learning and refresh policy docs
- Add best-practices/macos-gesture-headless-capability: which desktop gestures
have a headless AX path on macOS (double-click via AXOpen; triple-click/hover/
drag are physical-only; SwiftUI controls vs native AppKit), and why the command
never decides — the platform adapter owns headless-vs-physical.
- Refresh two policy learnings for the physical->headed rename: ActionRequest::
physical -> headed and AD_POLICY_KIND_PHYSICAL -> AD_POLICY_KIND_HEADED, noting
the new global --headed upgrade path via CommandContext::request.
* fix: address review P2 correctness and reliability findings
- wait: cap each ref-resolution attempt (750ms) so a slow resolve cannot
consume the whole wait budget on the first poll; the predicate is re-checked
across the full timeout.
- wait: make LatestRefCache timing fields private (no external readers).
- wait: add a unit test for --menu-closed (asserts it waits for open=false).
- close-app: make the graceful and force responses symmetric — both carry
`closed` (force confirms true; graceful cannot confirm, so false) instead of
graceful silently omitting the field.
- close-app: test adapter-error propagation.
- adapter: the default resolve_element_strict_with_timeout now logs that it does
not enforce the deadline, so an adapter that forgets to override it is visible
in traces.
- type: a RAII guard restores the user's clipboard on every scope exit (success,
error, panic) during the paste-based non-ASCII path, shrinking the clobber
window to an unpreventable SIGKILL.
Note: the reviewer's "fail fast on AMBIGUOUS_TARGET with a pinned snapshot"
suggestion is not applied — existing tests prove transient ambiguity resolves on
retry even with a pinned ref (ambiguity is a property of the live tree, not the
refmap), so failing fast would regress intentional, tested behavior.
* perf: trim actionability preflight and resolve-search allocations
- action_list: gate the AXValue and AXExpanded `is_settable` probes on whether
the role could plausibly carry that capability (unknown roles always probe),
skipping up to two AX round trips per preflight on common click-only targets.
No capability is lost — value/expandable roles still probe.
- resolve_search: reuse one scratch FxHashSet across nodes instead of allocating
a fresh dedup set per node during path and recursive search.
* refactor: split actionability types into one file each
ActionabilityStatus, ActionabilityCheck, and ActionabilityReport move into their
own files under a new actionability/ module (following the tree/ and actions/
folder pattern); mod.rs keeps the check logic and re-exports the types. Honors
the one-domain-type-per-file rule without changing behavior.
* test: split fixture under the LOC limit and fix swift-ios issues
- Extract the reusable fixture components (status readout, native AppKit
slider/stepper, drag canvas, card) into FixtureComponents.swift so each file
is under the 400-LOC limit; build.sh now compiles every .swift in the dir.
- NativeSlider/NativeStepper implement updateNSView so a SwiftUI binding change
syncs back to the NSView.
- The fixture no longer steals focus unconditionally on launch
(activate ignoringOtherApps:false), so it cannot mask headless-policy focus
violations; the harness drives focus explicitly.
- build.sh pins the SDK and a macOS 13 deployment target for reproducible builds.
* docs: document the optional error.details field and roles_present shape
- SKILL.md: note that the error object may carry an optional `details` (the
actionability report, AMBIGUOUS_TARGET candidates, or a wait TIMEOUT's last
observed state) and that responses should be parsed leniently — `details` and
future fields are additive.
- commands-observation: show the no-match `find` response with the
`roles_present` hint so callers can tell a wrong role name from "none on
screen".
* fix: gate the ref-gesture focus raise on the interaction policy
Ref-addressed hover/drag raised the target app unconditionally, violating
the headless no-implicit-focus-steal contract. The point resolver now
returns the owning pid instead of focusing, and commands decide: headless
never raises, --headed raises once (drag focuses only the from-app, fixing
the cross-app double-focus). Responses report focused:true so multi-app
agents can detect the frontmost change.
* fix: abort failed drags at the origin and disarm the guard only on success
The mouse-up guard disarmed before the final fallible up-event, so a
failed final post left the button held. Worse, its corrective release
fired at the unreached destination, silently committing an aborted drag
as a completed drop (CGEvents resolve at their embedded coordinates).
The guard now owns the release: it disarms only after the up actually
posts, and an early return cancels by dragging back to the origin and
releasing there.
* fix: enforce the chain deadline inside increment steps
All dispatch sites construct ChainContext with deadline: None, so the
remediation parameter on increment_to_value never received a value and
the 1024-iteration loop ran unbounded by the chain timeout. The chain now
pins its resolved deadline into the context every step observes. Also
extracts the pure write-verification predicates and their tests to
chain_verify.rs, bringing chain.rs back under the 400-LOC limit.
* fix: pin the AdAction ABI layout and bound FFI string and array inputs
AdDragParams is embedded by value in AdAction, so its 8-byte growth grew
the struct C callers pass to ad_execute_action with no size guard —
old-layout callers under-allocate and stack garbage becomes a live
drop_delay_ms. Adds AD_ACTION_SIZE / ad_action_size() with a layout pin,
matching the AdRefEntry pattern.
Also hardens the input boundary: C strings are decoded with a bounded
NUL scan (AD_MAX_STRING_BYTES, sized for CLI argv parity) so a missing
terminator cannot walk arbitrary memory, and the single coarse 1024
array cap becomes published per-field caps (AD_MAX_REF_STATES/ACTIONS/
PATH_DEPTH) with tests just over each limit.
* perf: replace fixed input settle sleeps with state polls
ensure_app_focused slept 50ms per physical input even when the app was
already frontmost; it now polls AXFrontmost (1ms, 50ms deadline).
disclosure_settled slept an unconditional 40ms up to three times per
expand/collapse; it now polls the disclosed state (5ms, 200ms deadline),
converging immediately on fast UIs. The kAXFocusedAttribute settability
probe is gated by role_may_accept_focus, mirroring role_may_bear_value,
and key dispatch reuses ensure_app_focused instead of its inline
duplicate.
* feat: check a specific action in wait --predicate actionable
The actionable predicate hardcoded Click, so wait-then-type flows got a
false ready on fields that cannot accept text (the editability check only
runs for editing actions). --action selects click (default), type,
set-value, or clear, and the preflight mirrors each command's real base
policy (type uses its focus-fallback base).
* fix: redact title, url, help, and placeholder keys in traces
Window titles, URLs, tooltips, and placeholder text carry user content
just like names and values; the redaction list now covers them.
* test: harden the e2e harness and unify fixture AX labels
The harness now fails setup loudly when the fixture build or AX trust is
missing, rebuilds the fixture when sources are newer than the bundle,
asserts hover only against a freshly observed state, and force-collapses
the disclosure so the expand test proves a real flip. The slider/stepper
labels live solely on the NSViews (the AX-actionable elements), removing
the macOS-version-dependent race between two label sources, and the drag
canvas reports a zero frame when detached from a window.
* docs: record the perf commit type, pre-1.0 bump policy, and error details field
Adds perf: to the allowed commit types (release-please already maps it to
a Performance changelog section), records the pre-1.0 versioning policy so
a BREAKING footer is expected to cut a minor rather than a major, and
shows the optional error.details object in the error envelope docs.
The gitignored local AGENTS.md mirror got the same contract sync.
* fix: surface increment deadline expiry as timeout with the observed value
A chain deadline firing mid-increment returned Ok(false), so the step was
recorded as skipped, the chain exhausted into ACTION_FAILED, and the
control sat at a half-applied value the caller could not see — post-state
is only read on success, and ACTION_FAILED recovery guidance points away
from retrying. Expiry is now a TIMEOUT error carrying value_before,
value_at_timeout, target, and a mutated flag in details.
* perf: cap the disclosure settle poll to the chain deadline and widen its interval
The settle poll could spend 3 x 200ms x 5ms-interval reads (~360 IPCs)
per expand/collapse and overshoot the chain's own deadline. A new
CustomWithDeadline chain step threads the chain deadline into the
disclosure steps, the settle budget is min(200ms, remaining chain
budget), and the interval widens to 20ms (~30 IPCs worst case).
* fix: enforce the protected-process guard inside the adapter close path
The guard lived only in the CLI command layer, so ad_close_app could
force-kill session-critical processes (loginwindow, WindowServer, Dock)
that the CLI refuses. close_app_impl now refuses them before any side
effect with the exact CLI error contract, making CLI, FFI, and any future
consumer behave identically; the CLI preflight remains as an earlier
check against the same predicate.
* fix: make focused semantics honest and confirm window focus by polling
ensure_app_focused set AXFrontmost unconditionally and reported success
identically whether or not a raise happened; it now no-ops when the app
is already frontmost, so Ok (and the focused:true response field) means
"frontmost ensured" exactly as documented. focus_window_impl gains the
same confirmation poll after its raise, and the poll interval widens to
5ms (10 reads max in the 50ms window).
* fix: attach abort-state guidance to drag failures and document cancel limits
Drag synthesis errors surfaced as bare INTERNAL with no hint about the
gesture's end state. Failures now carry a suggestion stating the button
was released back at the origin (best-effort), no drop was committed,
and where the cursor ends; the guard doc spells out the two best-effort
limits (corrective posts can fail; a self-drop at the origin is a no-op
for most targets).
* refactor: split oversized files by responsibility under the 400-LOC limit
helpers_tests (426) splits into resolution/window/pipeline tests, a
ref-action+trace test file, and a shared entry-builder support module.
wait_element_tests (414) splits into predicate-behavior tests and
resolution/lifecycle tests over a widened wait_test_support. wait.rs
(395) loses the element-wait loop to wait_element.rs, and refs_store
(397) moves its tmp-cleanup/retention methods to a refs_store_prune
child module (declared via #[path] so the split keeps base_dir and
snapshots_dir private to the store).
* refactor: build the actionable preflight request per action name at parse
The policy mirror lived in a separate helper with a catch-all arm, so a
future action name could silently inherit the headless policy. Parse now
maps every --action name to the exact ActionRequest its real command
runs (type is the only focus-fallback), the catch-all is gone, and a
test pins each name's policy.
* refactor: move point resolution and the focus helper to point_resolve
PointResolveArgs, ResolvedPoint, the ref-or-xy resolver, and
focus_for_physical_input were accumulating in helpers.rs alongside
unrelated ref-action plumbing; they now live in a dedicated
point_resolve module consumed by hover and drag.
* test: keep the drag canvas AX label on the NSView only
The DragCanvas carried two label sources (the NSView and a SwiftUI
modifier on its representable), the same macOS-version-dependent race
the slider/stepper fix removed; the harness-facing label now lives
solely on the AX-actionable NSView.
* docs: sync skills and header with the focused, redaction, and wait contracts
Documents the ensured (best-effort, already-frontmost-aware) semantics of
focused:true and its absence-vs-false meaning, the four redaction keys
added in round 2 plus the substring-match behavior, the INTERNAL error
recovery row, the FFI wait-surface asymmetry, the cross-app drag
occlusion caveat, and the AdDragParams/AdAction layout history with the
adjudicated pre-1.0 breaks so fresh reviews stop re-finding them.
* fix: surface settle-wait deadline truncation as timeout with a schema discriminant
A chain deadline truncating the disclosure settle wait returned a plain
step failure, exhausting into ACTION_FAILED — the same masking class
fixed for increments — even though the triggering action may still land
after the truncated wait. Settle exits are now classified: full-budget
misses stay step failures, deadline-truncated waits raise TIMEOUT with
the wanted/observed state, and the poll sleeps are clamped so a tight
deadline still gets at least one read. All TIMEOUT details now carry a
kind discriminant (wait_timeout vs chain_deadline) so agents can branch
without sniffing field names.
* fix: match protected processes exactly, not by substring
'docker'.contains('dock') permanently blocked close-app for Docker,
FinderSync-class apps, and anything else embedding a protected name.
Matching is now an exact lowercase name or an exact dot-separated
bundle-id component, so Dock and com.apple.dock stay protected while
Docker, Docker Desktop, FinderSync, and PathFinder stay closable —
pinned by false-positive tests.
* fix: raise the element's window before the physical click fallback
CGEvents land on the topmost window at the click point, so an app being
frontmost is not enough when the target element lives in a background
window of that app — the physical fallback clicked whatever overlapped
it, and the skip-raise-when-frontmost optimization widened the window
for that. click_via_bounds now raises the element's own AXWindow (AXRaise,
AXMain fallback, brief confirmation poll) via a shared window_ops helper
that focus_window_impl reuses. Verified live: a headed click on a ref in
an occluded Finder window raises that window and lands the click in it.
* test: guard ref-action policy coverage against silent gaps
A new ref-action command could ship without a base-policy assertion. A
guard test now scans crates/core/src/commands/ for files calling
context.request( and fails unless each stem appears in the
POLICY_TESTED_COMMANDS list backing the policy assertions.
* fix: carry the protected-process suggestion on the CLI preflight
The CLI-layer guard returned bare INVALID_ARGS while the adapter layer
carried recovery guidance, so agents on the primary surface got 'check
command syntax' for a permanently-disallowed operation and looped on
argument fixes. Both layers now state the same suggestion.
* docs: document the TIMEOUT schemas, chain deadline knob, and roles_present scope
Names the two TIMEOUT details schemas by their kind discriminant with
the mutated-flag retry rule, points chain-deadline recovery at
AGENT_DESKTOP_CHAIN_TIMEOUT_MS instead of --timeout, extends the
roles_present hint to all non-count selection-mode misses, and aligns
the STALE_REF recovery row with the richer error.rs suggestion.
* test: extract the scroll card and document drag-canvas data flow
AgentDeskFixture.swift sat at exactly 400 lines; the scroll card is
fully self-contained (its offset state never leaves the card), so it
moves to FixtureCards.swift as a standalone view with zero bindings,
landing the main file at 374. Harness-facing labels are byte-identical.
DragCanvas gains two intent comments distinguishing the deliberately
empty updateNSView from a forgotten sync.
* chore: pin ABI sizes for C consumers and explain the prune module split
C11-gated _Static_asserts mirror the Rust-side layout pins so a C
consumer compiling against a drifted header fails at build time, and
the production #[path] prune module carries its privacy rationale.
* test: give racing wait tests a deterministic budget
Three wait tests used a 1ms timeout that can elapse before the loop's
first resolution attempt under load, flaking on machine pressure (the
ambiguous-resolution test needs at least one attempt to record its
observation). 50ms guarantees the first attempt without slowing the
suite.
* docs: capture three round-4 review learnings and grow the concept map
Documents the abort-state contract for multi-step physical input (guard
disarm ordering, origin release, end-state suggestions), the three-layer
repr(C) size-pinning discipline born from the AdAction silent-growth
incident, and the named-arms-plus-exhaustiveness-guard pattern for
policy/dispatch mirrors. CONCEPTS.md gains Action Chain and Protected
Process and refreshes Coordinate Fallback with the window-topmost rule.
* docs: refresh six learnings against the enhanced-reliability branch
Brings the learning corpus back in line with code that moved this
branch: the gesture-capability and policy docs now describe the
window-level raise in the physical path and link the new abort-state
doc, the reliability contract documents the TIMEOUT kind discriminant
and the wait --action per-name policy variant, the FFI review rule
covers structural repr(C) size drift alongside behavioral parity, the
allocator doc records how the config struct absorbed four more fields
in one place, and the fingerprint doc names the real tri-state decode
error type. Three docs verified accurate with no edits.
* docs: sync roadmap with current reliability contracts
* fix: harden desktop action reliability
* fix: harden reliability review regressions
* fix: close final reliability review gaps
* fix: close reliability review gaps
* test: avoid raw pointer mutation in ffi free tests
* refactor: trim reliability branch dead code
* fix: close reliability review gaps
BREAKING CHANGE: the C ABI AdActionResult layout now includes action steps; C consumers must rebuild against the updated agent_desktop.h header.
* fix: close reliability review gaps
* ci: scope cache hash inputs
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Lahfir
|
1291a9cdbf
|
refactor!: unify command execution contracts
Unify CLI and batch dispatch around the typed command path, centralize command policy and ref resolution, harden macOS action verification, split command tests from implementation, and add package/release guardrails. BREAKING CHANGE: CLI and batch execution now share the typed command path and current command argument contracts. BREAKING CHANGE: Ref-consuming commands use snapshot-scoped refs; deterministic consumers should pass snapshot_id and handle SNAPSHOT_NOT_FOUND. BREAKING CHANGE: permissions and status now return PermissionReport fields for accessibility, screen_recording, and automation instead of a single boolean status. BREAKING CHANGE: PermissionState gains NotRequired; macOS automation now reports not_required instead of unknown. BREAKING CHANGE: right-click now separates action success from menu verification; consumers should inspect menu or menu_probe instead of assuming every right-click returns an inline menu. BREAKING CHANGE: focus-window now confirms OS focus and returns ACTION_FAILED when focus does not settle; data.focused.is_focused is true on success. BREAKING CHANGE: PlatformAdapter::execute_action now takes ActionRequest, and permission probing uses permission_report/request_permissions. BREAKING CHANGE: FFI ad_execute_action now defaults to headless policy. Consumers that need focus fallback or cursor-moving behavior must call ad_execute_action_with_policy with AD_POLICY_KIND_FOCUS_FALLBACK or AD_POLICY_KIND_PHYSICAL. BREAKING CHANGE: FFI ad_check_permissions no longer treats unknown accessibility permission as success; stub-style unknown probes return ERR_PLATFORM_NOT_SUPPORTED and macOS ambiguous unknown returns ERR_INTERNAL with last-error detail. BREAKING CHANGE: JSON response envelopes now report version 2.0; parsers pinned to 1.0 must branch or update. BREAKING CHANGE: focus now uses accessibility focus without cursor movement; callers that need physical focus must use explicit mouse or physical-policy paths. BREAKING CHANGE: chain execution deadlines now return TIMEOUT instead of ACTION_FAILED when the target app does not respond before the chain deadline. |
||
|
Lahfir
|
c17f2fae7a
|
feat: progressive skeleton traversal with ref-rooted drill-down (#20)
* feat: add data model foundation for progressive skeleton traversal
Add children_count, root_ref, skeleton fields to core types. Add
get_subtree() to PlatformAdapter trait. Add remove_by_root_ref() and
write-side size check to RefMap. No behavioral changes yet.
* feat: implement progressive skeleton traversal with ref-rooted drill-down
Add --skeleton and --root flags to snapshot command for token-efficient
accessibility tree exploration. Skeleton mode clamps depth to 3 levels
and annotates truncated containers with children_count, allowing AI
agents to discover regions before drilling into them. Named containers
at skeleton boundaries (via name or description) receive refs as
drill-down targets. The --root flag starts traversal from a previous
ref with scoped invalidation — only refs from that drill-down are
replaced on re-drill.
Key changes:
- New ref_alloc.rs: shared ref helpers (INTERACTIVE_ROLES, actions_for_role,
ref_entry_from_node, is_collapsible) extracted from snapshot.rs
- New snapshot_ref.rs: drill-down logic with DrillDownConfig, scoped
invalidation via root_ref tagging on RefEntry
- macOS count_children() uses raw CFArrayGetCount without materializing
AXElement wrappers for performance at skeleton boundaries
- RefMap write-side size check prevents >1MB files
- Skeleton anchors consider both name and description for Electron compat
* fix: mention --skeleton in STALE_REF error suggestion
* docs: document --skeleton and --root flags in skill reference
* docs: update phases.md and CLAUDE.md for progressive skeleton traversal
Add skeleton traversal as Phase 1 objective P1-O10. Document --skeleton
and --root flags, get_subtree() trait method, new core modules, and
platform-agnostic notes for Phase 2/3. Update risk mitigations and
performance optimizations table.
* docs: make progressive skeleton traversal the default agent workflow
Update SKILL.md observe-act loop to skeleton-first approach. Add
progressive skeleton traversal as the primary workflow pattern. Update
anti-patterns, key principles, and command quick reference to lead
with --skeleton + --root. Full snapshot remains documented as fallback
for simple apps.
* fix: preserve skeleton drill-down anchors
* fix: preserve drill-down refs across skeleton re-snapshots
Skeleton snapshots now load the existing refmap and remove only
skeleton-level refs (root_ref: None), preserving drill-down refs
accumulated via --root. Previously, build() always created a fresh
RefMap, breaking the skeleton → drill → act → skeleton(verify) workflow
by wiping all drill-down refs on the verify step.
* fix: preserve drill-down depth for root snapshots
* fix: verify AX action effect on Electron elements before trusting success
Chromium's AX implementation returns kAXErrorSuccess for AXPress,
AXConfirm, etc. but only toggles ARIA state without firing DOM event
handlers. This caused the click chain to short-circuit on false
positives, preventing CGClick from ever being reached.
The fix detects web elements via AXWebArea ancestor walk, then verifies
each AX action actually had a DOM effect by comparing focused element
pointers before and after. When all AX methods produce no real effect,
CGClick fires as the genuine last resort.
Native elements are unaffected — the original verified_press behavior
is preserved in a separate code path.
* chore: ignore .context/ for local tooling artifacts
* style: collapse web_action_had_effect signature to single line
* test: cover RefMap save oversize rejection
Extract the serialize+size-check step into a private helper so the 1MB
write rejection path can be unit-tested without filesystem I/O. Also
moves the size check above the directory creation in save() so an
oversized refmap no longer creates ~/.agent-desktop on rejection.
* test: assert stale-ref suggestion mentions --skeleton
Locks in the Phase 4 polish requirement that STALE_REF errors guide
agents back to a skeleton refresh, not just a plain snapshot.
* test: cover skeleton-to-drill-down counter continuity
Asserts that skeleton refs (@e1..@e10) survive a scoped invalidation
of @e3, that drill-down refs allocated after the skeleton continue from
@e11 instead of resetting, and that remove_by_root_ref drops only the
drill-down children.
* test: cover --root + --surface rejection at execute() boundary
Adds a NoopAdapter that uses every PlatformAdapter trait default to
exercise execute()'s validation guards without standing up a real
adapter. Verifies that combining --root with a non-Window surface
returns INVALID_ARGS, and that the Window surface does not trigger the
guard.
* test: add filesystem-redirected integration tests for run_from_ref
Adds a thread-local HOME override + RAII HomeGuard so RefMap::save and
RefMap::load can be exercised against an isolated temp directory without
env-var racing across parallel tests. Also adds a StubAdapter that
implements PlatformAdapter with a canned subtree response so the full
run_from_ref drill-down flow can be unit-tested without standing up
macOS AX state.
Closes seven Phase 3 plan acceptance items in one pass:
- save+load roundtrip with HOME override
- oversize save rejection preserves previous file on disk
- run_from_ref returns subtree and persists drill refs
- stale root ref → STALE_REF with skeleton suggestion
- re-drill replaces drill refs only, counter continues
- multiple drill-downs from @e1 and @e2 coexist
- empty subtree drill-down produces no new refs
* test: add golden fixtures for skeleton output and drill-down refmap
Adds two committed JSON fixtures under tests/fixtures/ plus matching
unit tests that build the same input trees, run them through
allocate_refs / run_from_ref, and assert the produced ref ids,
parent-child layout, and root_ref tagging match the golden expectations.
Locks in the JSON shape so future serialization changes have to be
deliberate.
* fix: bound build_subtree recursion on Electron wrapper chains
Anonymous AXGroup/AXGenericElement wrappers do not advance the semantic
depth counter (web-wrapper depth-skip), so a long chain of nested
wrappers could recurse arbitrarily deep without ever hitting either
max_depth or ABSOLUTE_MAX_DEPTH. The previous code checked
`depth >= ABSOLUTE_MAX_DEPTH` against the semantic depth, which stayed
at 0 throughout a wrapper chain. On pathological Electron trees this
risked stack exhaustion before any cap engaged.
Add a separate `raw_depth` parameter that always increments and use it
for the absolute cap. Semantic `depth` still drives `max_depth` and the
skeleton boundary so the wrapper-flattening behavior is preserved on
normal trees.
Also drops the long-unused `_include_bounds` parameter; bounds
filtering happens in `allocate_refs` in core, not here.
* chore: add pre-commit hook running fmt + clippy + tests
Mirrors the CI quality gates (cargo fmt --check, cargo clippy
--all-targets -- -D warnings, cargo test --lib --workspace) so a
failing commit never reaches origin. Skips automatically when no
Rust/TOML files are staged. Bypass with --no-verify or SKIP_PRECOMMIT=1
when a non-Rust hotfix is genuinely needed.
Hook lives under .githooks/ and is opt-in per clone:
git config core.hooksPath .githooks
Setup instructions added to CLAUDE.md.
* fix: prevent orphaned drill-down refs leaking across skeleton refresh
remove_skeleton_refs() previously dropped every entry whose root_ref
was None and kept every entry whose root_ref was Some. After a series
of skeleton refreshes that pattern leaked drill-down refs whose root
anchor had already been removed: they could never be cleaned up by a
future remove_by_root_ref(target) because target referred to a
non-existent anchor, and they accumulated until the 1MB write guard
fired.
Restructure the function to keep only:
- skeleton anchors that have at least one drill-down pointing at them
- drill-down refs whose root anchor is among the kept anchors
Unreferenced anchors are still dropped (the original intent), and
orphaned drill-downs are dropped at the same time so the refmap can
no longer accumulate dead entries.
Updated existing test to assert the new keep-when-referenced semantics
and added a regression test for the orphan-drilldown leak path.
* fix: align resolve traversal with snapshot child-attribute set
find_element_recursive previously walked AXChildren first then fell
back to AXContents only, and resolve_element_name only derived its
fallback label from AXChildren. Snapshot building, however, uses the
full child_attributes() list (AXChildren, AXContents,
AXChildrenInNavigationOrder) via copy_children. That asymmetry meant
refs minted for containers exposed only through AXChildrenInNavigationOrder
appeared in the snapshot output but produced false STALE_REF errors on
drill-down or action commands because the resolver could not find them
again.
Route both call sites through child_attributes() so resolution sees
the same children as snapshotting.
* fix: bypass AXConfirm on web elements and add skeleton anchors to drill-down
* fix: compare AX elements via CFEqual in web_action_had_effect
AXUIElementCopyAttributeValue follows the CoreFoundation Create rule
and returns a freshly-allocated CF object on every call, so raw
pointer equality (`before.0 != after.0`) between two separate copies
of AXFocusedUIElement was ALWAYS true even when the focused element
was unchanged. That made Step 1 (AXPress) of activate_web_element
appear successful every time and left Steps 2-4 of the escalation
chain (AXConfirm bypass, child actions, CGClick fallback) dead code
on web elements.
Replace the pointer comparison with CFEqual, which compares
accessibility element identity rather than heap addresses.
* refactor: unify allocate_refs across snapshot and drill-down paths
allocate_refs in snapshot.rs and allocate_refs_with_root in
snapshot_ref.rs were near-exact copies that differed only in whether
each allocated ref carried a root_ref tag. The duplication meant any
bug fix or behavior change in one path risked silently drifting from
the other (bot flagged this after the first round of fixes added even
more duplicated skeleton-anchor logic to allocate_refs_with_root).
Move the shared logic into ref_alloc.rs behind a new RefAllocConfig
struct with Option<&str> for the root_ref_id, and delete the
snapshot_ref.rs copy. Both snapshot::build and snapshot_ref::run_from_ref
now route through the same function with their respective configs.
append_surface_refs and the existing unit tests in both modules are
updated to use the shared function + config.
* chore: drop with_root from drill test names after allocator unification
The snapshot_ref tests kept their original test_allocate_refs_with_root_*
names after the allocator dedupe even though allocate_refs_with_root no
longer exists. Rename them to test_drill_alloc_* so grep for
allocate_refs_with_root returns zero matches and nobody assumes a second
allocator path exists.
* docs: compound DRY ref-allocator dedupe into knowledge base
Adds docs/solutions/best-practices/deduplicate-ref-allocator-via-config-struct-2026-04-14.md
documenting the root cause, guidance, consequences, trigger conditions,
and before/after of the allocate_refs / allocate_refs_with_root
duplication that landed on PR #20 and was unified in
|
||
|
Lahfir
|
ccbf736f9b | chore: remove agentic test notes from tracking | ||
|
Lahfir
|
3a796023f0 | style: apply cargo fmt to all files | ||
|
Lahfir
|
a346f242c2 |
feat: Phase 1 foundation — workspace scaffold, core engine, macOS adapter, 31 commands
Implements the complete agent-desktop Phase 1 specification:
- Workspace: 5-crate layout (core, macos, windows/linux stubs, binary)
- Core: AccessibilityNode, Action, ErrorCode, PlatformAdapter trait, RefMap
with atomic writes, SnapshotEngine with depth-first ref allocation
- macOS adapter: AXUIElement tree traversal, action execution, input
synthesis via CGEvent, screenshot via CGWindowListCreateImage, clipboard
via pbpaste/pbcopy, window listing and app management via osascript
- 31 CLI subcommands via clap derive: snapshot, find, screenshot, get, is,
click, double-click, right-click, type, set-value, focus, select, toggle,
expand, collapse, scroll, press, launch, close-app, list-windows,
list-apps, focus-window, clipboard-get/set, wait, status, permissions,
version, batch
- Windows/Linux: not-supported stubs ready for Phase 2 implementation
- JSON output contract: {version,ok,command,data} envelope with structured
error payloads including SCREAMING_SNAKE_CASE error codes
- Ref system: @e{N} sequential refs for interactive elements, stored at
~/.agent-desktop/last_refmap.json with 0o600/0o700 permissions
- CI: GitHub Actions macOS runner with dependency isolation check, clippy,
unit tests, release build, and 15MB binary size gate
|