36 commits

Author	SHA1	Message	Date
Lahfir	69371f92f3	docs: cut acl validation from the windows private-file design ... Reverses the ancestor-walk decision on measurement. A plain leaf under the user profile already inherits SYSTEM, Administrators and the user with no Users entry, all inherited and none explicit, so the walk would verify a property Windows already guarantees. Unix has to author its permission because the default there is world-readable; Windows does not, because the default is already private. The only principal the walk would additionally exclude is another administrator, who holds SeTakeOwnershipPrivilege and can seize any object whatever its DACL. Building a chain-walking validator against someone who can bypass it costs more than it buys, and the cost is specific: parsing ACEs is what the previous layer did, and its AceSize handling is the defect that took the layer down. Removing the parser removes the bug class rather than fixing it, so no successor module exists and a test asserts the ACE symbols never come back. What survives is the control with real unix parity and a real attack behind it: per-component reparse-point rejection. That is an integrity control, not a confidentiality one -- a junction planted on the path redirects where the product writes, which no permission on the intended destination prevents. Cross-admin isolation is now answered rather than open: out of scope, for the same reason macOS never raised the question.	2026-07-27 01:47:57 -06:00
Lahfir	2d51c680e8	docs: plan the windows toolchain, ci, and com bootstrap sub-phase ... Eight units covering the CI lane extension, Win32 bindings in the platform crate, the COM and DPI bootstrap, the adapter session, the permission probe, and a private-file layer rebuilt from scratch against sub-phase 2.0's measured evidence. Three reviewers found what the first draft got wrong. It installed both the apartment and the private-file seam from ad_init, which the FFI header labels optional and whose own doc promises it initialises nothing -- so any host that skipped it would have run every private write through the unhardened default with nothing to distinguish the two states. The install moves to the site the macOS arm already uses, where a consumer cannot skip it. It also planned a layer that validates security state it never authors. The deleted implementation at least built an owner-only protected descriptor at creation; the rebuild now does too, and rejects reparse points per component the way the unix path rejects symlinks. The ACE validation order was still wrong in the same shape as the defect it exists to prevent: checking the type before forming the SID pointer still dereferences the header at an offset nothing has bounds-checked. The order is now extent-first, with the ACE count treated as advisory rather than trusted. The seam routes at core's five private-file primitives rather than at a list of call sites, which covers the lock and liveness paths a list had missed and needs no call-site edits at all.	2026-07-27 00:14:08 -06:00
Lahfir	4206c722c9	docs: move the windows runner registration to the sub-phase that needs it ... Sub-phase 2.1 was to register a self-hosted interactive Windows runner that nothing in 2.1 through 2.11 uses. On a public repository that is a persistently registered runner sitting idle behind ten sub-phases, and GitHub's own guidance is that self-hosted runners should almost never be used for public repositories at all. Registration moves to 2.12, the first sub-phase whose gate needs a real desktop, and it takes the hardening requirements with it: dispatch-only triggering, a written fork-PR approval policy, and an explicit ephemeral-versus-persistent decision. The deferred RDP row moves with it. Its closure was never documentation -- it is a measurement that only exists once a second, non-console session does, so 2.12 now owns closing it rather than 2.1 owning a promise it could not keep. Two probe scripts hardcoded the old closure sub-phase in text they emit. The scripts were corrected and their probes re-run, so the captures were regenerated rather than edited by hand.	2026-07-27 00:02:23 -06:00
Lahfir	31ffd5f657	docs: correct the windows sub-phase facts research disproved ... Sixteen replacements, no annotations. The document is the product's source of truth and now reads true. The COM bootstrap is split by consumer. CoInitializeEx is right for the CLI and wrong for the cdylib, which fails RPC_E_CHANGED_MODE against any host thread already in an STA and whose per-thread balance can never be released from a Drop on another thread; the library path uses CoIncrementMTAUsage, whose cookie is thread-agnostic. MTA is recorded as a requirement rather than a preference, because the single-threaded apartment can prevent handler removal outright. Two things the sub-phase asserted were not reachable. Its private-file seam named a boundary that does not exist: every write site is in core with no adapter handle, and core may not depend on the platform crate. The document now states that constraint and leaves the mechanism to the plan. Its CI paragraph claimed all runners enforce clippy, isolation, and the size cap; only the macOS job does, and the other two lanes each run a single lib invocation. The event invariant said handlers are drained on the dedicated MTA thread. They are not: delivery arrives on several UIA-owned threads at once, the registering worker among them and the main thread never, so handler state has to be safe for concurrent delivery. Runner registration gains the hardening a public repository requires, and gains the measurement it already owed the ledger: it closes the deferred RDP row rather than merely documenting session isolation. The DPI bootstrap records that awareness cannot be verified by read-back, since the query API has no enumerant for the value being set. The ledger's hunk index moves with them, 37 to 40. Hunks backed by research rather than by a probe are labelled as such, so an external citation cannot pass itself off as a measurement.	2026-07-26 23:45:25 -06:00
Lahfir	5fd9543a79	feat: add windows platform exploration probe corpus (phase 2.0) (#111) ... * docs: add the sub-phase 2.0 probe corpus plan The plan for the Windows platform-exploration sub-phase: an eleven-area probe corpus, its findings ledger, and the same-PR phases.md corrections that ledger obliges. Lands on the 2.0 sub-phase branch so the sub-phase PR reviews as one self-contained unit. * feat: add windows probe harness with redaction gate and integrity helpers The harness is the foundation every 2.0 probe codes against: common.ps1 owns the R11 redaction gate, BOM-less UTF-8 capture writers, KTD9 normalization, scratch-process lifecycle with pid tracking, the KTD5 foreground assertion, and a Medium-integrity process launcher. runas.exe /trustlevel:0x20000 was measured on this box and produces a High integrity process (S-1-16-12288) with a deny-only Administrators group, not the Medium label the UIPI probe needs. Start-MediumIntegrityProcess therefore duplicates the token and sets S-1-16-8192 explicitly, asserting the label on read-back so a UIPI row can never be graded against a fake boundary. * feat: add winforms and wpf scratch fixtures with explicit automation ids The fixtures are the interaction targets for the pattern census, identity stability diff, raw interaction exercises, and the COM shim's event sources. Two measured platform facts shaped them. First, the .NET Core rule that a WinForms control's Name becomes its UIA AutomationId does not hold on .NET Framework 4.8 -- the HWND provider supplies raw window handles instead, which change every launch and are useless for an identity-stability diff. The fixture installs a server-side provider per control, with a switch to fall back to explicit Win32 control ids. Second, Win32 UIA client-side proxies are inert on this VM: the WinForms fixture exposes 23 controls with zero patterns, every one a Pane, matching stock Notepad's Edit reporting as a bare Pane. The WPF fixture exposes 25 controls with a full pattern surface. Interaction probes therefore target WPF and record the WinForms absence as evidence rather than working around it. * feat: capture windows environment, session, and integrity baseline Every ledger row inherits these machine facts, so they are captured reproducibly rather than asserted: build 17763.7434, session and interactivity, mandatory integrity label, UAC policy state, default file ownership, display topology, and tool inventory. Identity is recorded as a shape, not a value. The account SID keeps its well-known RID tail (S-1-5-21-<redacted>-500) because the RID is the load-bearing fact, while integrity and well-known group SIDs stay verbatim for the same reason. Three findings the later units depend on. The GAC UIAutomationClient assembly exports 57 types, all managed System.Windows.Automation, and no IUIAutomation -- while CUIAutomation8 is registered against uiautomationcore.dll, so UIA3 is reachable only through a compiled interop shim. Newly created files are owned by S-1-5-32-544 via TokenOwner rather than TokenUser, pre-answering half of the private-file ownership question. The locale is split: UI culture es-ES while the current culture is en-US, so OS strings are Spanish but .NET number and date formatting is not. * feat: probe windows private-file primitives for the 2.1 hardening design Answers the four questions sub-phase 2.1 rebuilds its private-file layer against, measured on the real OS with verbatim Win32 error codes. Three of the four contradict the framing this work was planned under. Rename: MoveFileEx cannot replace a target that has any open handle, and it reports ERROR_ACCESS_DENIED rather than the sharing violation the plan assumed -- FILE_SHARE_DELETE does not rescue it, because MoveFileEx issues ReplaceIfExists rather than POSIX-semantics rename. ReplaceFile is the operation that honors share-delete on the destination, but it is the one that refuses an open handle on the replacement source. The two APIs have opposite tolerances on opposite sides, so code treating error 32 as the sharing case misclassifies the most likely production failure. Ownership: new files take their owner from TokenOwner, never TokenUser, at both High and Medium integrity. Integrity level is not the variable -- the default follows admin-group membership, so an owner-only DACL check breaks for any admin account regardless of elevation. Locality: GetFileInformationByHandleEx(FileRemoteProtocolInfo) does distinguish local from remote, but signals local by failing with ERROR_INVALID_PARAMETER rather than succeeding with a local protocol value. An out-of-range info class returns that same error, so the code is ambiguous and a control call is required before reading it as a locality signal. ACL: neither leaf carries an explicit ACE. The restriction is authored at the nearest protected ancestor and merely propagated, so a leaf-only check sees the effective grants but cannot tell whether they are anchored or silently re-derivable, and cannot see FILE_DELETE_CHILD on the parent. * feat: probe uia3 com for events, caching, walkers, and pattern availability This is the stack the Rust adapter actually wraps, and it does not agree with the managed client. Notepad's edit control is a bare Pane with no patterns to System.Windows.Automation but a Document with Value, Text, Text2, Scroll and LegacyIAccessible to UIA3 COM, which names the responsible EDIT proxy in ProviderDescription. Twenty-six nodes versus three. Win32 client-side proxies are alive here; they are inert only for the managed client, so a managed census would have recorded false absences for patterns the adapter will see. The scratch fixture's earlier zero-pattern reading was a fixture artifact rather than a platform fact: a custom server-side provider suppresses both client-side proxies and WinForms' own providers. Both fixture modes are censused so the ledger states the mechanism instead of the symptom. CacheRequest measures 2.73x against the documented 3-5x, but the headline number hides the shape: building the cache makes the find phase slower, and the entire win lands in the read phase at roughly 299x. The claim is wrong in both directions. Handler teardown is safe with events in flight, costing 69ms against a backlog of 281. The hazard is elsewhere: hiding and showing a top-level window while handlers are registered poisons subsequent removal for up to 86 seconds, superlinearly in repetitions, and cheap callbacks do not avoid it. A watch implementation needs a teardown timeout rather than an assumption. Pattern ids are discovered from the OS at runtime rather than hardcoded, which caught that the annotation availability property is 30118. * feat: dump notepad, explorer, settings, and obsidian trees with real geometry Targets launch restored-but-not-activated, so bounds are real: 100/95.9/100/99.2 percent non-empty across the four dumps, with focus asserted identical before and after all seven walks. Settings resolves through ApplicationFrameWindow to its CoreWindow's SystemSettings pid, because the frame window belongs to ApplicationFrameHost and no predicate in either script keys on a display name. Minimizing a window does not shrink its tree -- the node count is identical at 3 -- but it degenerates geometry in two different shapes. Only the top-level window reports an empty rect; its descendants report real dimensions anchored at -32000, and every node still reports IsOffscreen as false. An occlusion gate testing emptiness or IsOffscreen alone would accept both as visible. Chromium activation is not a client-stack difference. The managed client takes Obsidian from 9 to 119 nodes across an 8 second settle with no flag, matching the COM shim's shape. What suppresses it is occlusion: an earlier revision that left other targets restored on top held the count at first contact for a 16 second instrumented hold across three runs. The mechanism is not isolated, so it is recorded as a probe-placement hazard rather than a product claim. The managed timing cross-check reproduces the COM stack's direction but not its magnitude, and on Notepad caching is a pessimization at 0.21x, because client-side providers run in the client process and the cache is pure overhead. * feat: census patterns and automation ids, and measure identity survival Identity survival is what sub-phase 2.5 designs its Windows RefEntry against, and the three properties behave nothing alike. AutomationId survives process restart on every stack measured, at 100 percent across WinForms, WPF, and real Win32. RuntimeId survives restart nowhere, at zero percent on all three. Path survives restart perfectly and is the first thing a content change breaks. Under content mutation the failure is worse than loss. Explorer keys list rows by row index, so after a folder changes, 29 AutomationId keys still resolve and 5 of them land on a different file. A ref keyed on AutomationId alone does not fail -- it succeeds against the wrong element. That silent-wrong-target count, not the survival percentage, is what makes stable text identity load-bearing. Measuring it needed a real refresh wait: a 4 second settle reported a wholly unchanged tree and would have recorded Explorer identity as perfectly stable, the exact inverse of what the window does once it refreshes at 20 seconds. Two stack facts. WPF automation-peer binding is a one-shot race -- a client that reads before the peer exists binds the generic HWND provider and never re-resolves, which a 30 second poll never recovers. And the managed and COM clients report different AutomationId values for the same window, so the divergence between stacks is in values as well as in visibility. The honest pattern divergence is two, not eleven: LegacyIAccessible and Text2 are the only patterns the managed stack structurally cannot name. The larger figure was an artifact of the WPF peer race and was retracted after the fix. * feat: probe uipi, dpi awareness, and chromium activation UIPI is measured across a boundary that actually exists. This box runs the built-in Administrator at High with Admin Approval Mode off, so RunAs yields High-vs-High and runas /trustlevel leaves the mandatory label untouched; the probe manufactures a real Medium token instead and refuses to record a verdict unless the worker reads back S-1-16-8192. Both arms run identical code against the same High Notepad, so the High arm is a control proving the injection works. UIA reads succeed from Medium; SendInput does not land, and the two arms return the identical success code -- only re-reading the target separates them. PostMessage is denied with error 5 while SendMessage WM_GETTEXT succeeds, so reads cross the boundary and writes do not. The 125 percent DPI measurement is not achievable here and the reason is measured rather than assumed: the VMware display carries no EDID and offers a single scale step. DisplayConfigSetDeviceInfo returns success and persists the registry value while effective DPI stays at 96, which is itself worth recording -- that API's success return is not evidence the scale changed. The non-zero delta defers to 2.4, which owns per-monitor scale. What does land for 2.1 is that PER_MONITOR_AWARE_V2 succeeds on 17763 with both arms verified at the awareness levels claimed. Chromium activation is graded, not guessed. The bundled version was read off this installation by scanning the shipped binary's user-agent string: Chromium 142, Electron 39.8.3. Both stacks reach the same settled tree with no flag, so the exposure claim holds and the plan's candidate contradiction is superseded. The flag is still not redundant: without it first contact is deterministically the pre-activation shell, and with it first contact is a race. Neither arm removes the need to settle before trusting a first snapshot. * feat: exercise interactions, input synthesis, and hit-testing raw Every interaction records pre-state, action, then an independent re-read -- never the call's return value. Toggle proves why: TogglePattern flips WPF ToggleState without raising Click, so the fixture's status sink stays silent while the element itself changed. Trusting the sink alone would have recorded a failure that did not happen. Posted keystrokes are not uniformly dead. WM_KEYDOWN posted to a Win32 edit control does register, because TranslateMessage runs in the target thread's own pump and synthesises the character regardless of how the message arrived; what the path cannot carry is modifier state. Against Chromium it does not register at all, with every call still returning success. Establishing that needed an idle control pass: the Chromium tree moved on its own between two reads, and without a quiet baseline that drift would have been filed as a keystroke landing. The astral-plane payload survives typing. SendInput forces a surrogate pair into two separate unicode events and the target reassembles it intact, read back through WM_GETTEXT so the check is independent of both the injection path and UIA. Notepad's edit control is unreachable from the managed client, and not merely unenumerated: TryGetCurrentPattern returns false for Text, Value and Scroll on the handle-resolved element, while the COM census sees the same window as a Document carrying all three. Hit-testing is the only sound visibility primitive of the three tested. The zero-size control is addressable by handle, enumerable by no walk, and returned by no point. A minimized window reports an empty rect at the top level while its descendants report real dimensions at the -32000 anchor, with IsOffscreen false throughout -- so neither emptiness nor IsOffscreen can gate occlusion, but ElementFromPoint correctly returns the occluder. * feat: add findings ledger, session evidence, and phases.md truth-sync Seventy-nine rows across all eleven evidence areas, each carrying the stack that produced it and whether it generalizes, so later sub-phases can tell an API-contract fact from an observation about this box and these app versions. Forty-three confirm the roadmap, five contradict it, twenty-six are new edges, and five defer to a named Phase 2 sub-phase. No row is unknown. Six in-place corrections land with it. The CacheRequest multiplier is replaced by its phase decomposition, because a single number is wrong in both directions. The private-file requirements now name ReplaceFile rather than MoveFileEx for a replace over an open destination, record that share-delete is necessary but never sufficient, and permit locality inference only behind a control call, since an out-of-range info class is indistinguishable from a local volume by error code. The Chromium guidance now requires a settle before concluding a tree is thin; its exposure claim was left standing because the probe confirmed it. The event-teardown mitigation now names the hazard that actually dominates removal cost. The completeness check is a script rather than a checklist, and it is negative-tested: blanking a scope, setting a verdict to unknown, or dropping a closure point each fail it loudly. It asserts the hunk index is bijective against the measured diff rather than against a number written in prose. * fix: wake a suspended settings instance instead of failing the tree dump The probe skipped its own launch when SystemSettings was already running, then waited the full window for a frame that a backgrounded UWP instance never presents, so a second harness run failed where a standalone run passed. That is the plan's recorded process-lifetime suspension risk arriving in practice. After eight seconds without a frame the probe now issues ms-settings: to wake a suspended instance, and the failure text names suspension as the cause rather than reporting only that no frame was found. * fix: close four teardown and gating defects found by review None of these changed a measurement; all four were failure paths that could report success. The COM shim killed Obsidian by process name with a raw Stop-Process inside an empty catch, so those pids never reached the ledger. A survivor there was invisible to both the probe's own teardown and the harness survivor check -- the one place in the corpus a leaked process could go entirely undetected. Obsidian pids are now registered and terminated through the confirming path. The tree dump registered its Settings pid only after frame resolution succeeded, so a resolution timeout threw first and leaked a process the probe had launched. Registration now happens as soon as the pid exists, and only when the probe launched it, since killing a pre-existing instance would be the operator's window. The modifier-release sweep injected without a foreground assertion, the only unbracketed SendInput site in the corpus. It now asserts and logs rather than aborting, because it runs during teardown. The input probe swallowed a failed teardown capture write, so the file that proves clipboard, cursor and process cleanliness could vanish while the probe still reported ok. It now fails loudly, and a surviving process downgrades the status too. Separately, a focus change is now classified rather than assumed hostile. A target the probe launched taking focus during its own startup is the target activating itself; Electron does this asynchronously when its renderer becomes ready, so under load it lands mid-dump and failed the run intermittently while passing standalone. Only a focus move to a window the probe never launched is interference. * chore: retrigger ci after an unrelated interaction-lease flake The macOS lib-test job failed on interaction_lease::tests::home_changes_do_not_change_the_physical_interaction_lock, which timed out acquiring the physical lock inside a 100 ms deadline at a contention count of 14. This branch changes no Rust, and the other 975 tests in that job passed, so the failure is runner contention rather than anything in the probe corpus. * chore: stop committing normalized capture twins Every capture was committed twice: once raw and once with pids, handles, timestamps and bounds canonicalized. The twin is produced by common.ps1 on every run and is byte-reproducible from the capture beside it -- verified across all thirty-seven of them -- so tracking both doubled the evidence diff for no information a re-runner cannot regenerate locally. The twins are still written on disk and -Compare still diffs them, so re-runnability is unchanged.	2026-07-26 22:21:35 -07:00
Lahfir	4fa7661608	feat: open the Windows adapter phase with corrected platform facts ... Establishes feat/windows-adapter as the base branch for all Windows work. Sub-phases 2.0 through 2.15 are cut from here and merge back here; main stays the macOS-GA line and gains Windows once, at the end, when the adapter is production-solid as a whole. Corrects the Phase 2 Windows sections, which were written from documentation research in 2026-04 and re-verified against current sources and a real Windows machine: - uiautomation 0.25 (a "0.24" requirement will not resolve to it under 0.x semver); windows-capture 2.0.0 - the recorded 1.5.4 was never published - Chromium 138 ships native UIA by default, so the web-wrapper depth-skip is the primary lever and --force-renderer-accessibility is a fallback - Windows 11 22H2+ moved tray overflow to TopLevelWindowForOverflowXamlIsland - windows-11-arm runners have been GA for public repos since 2025-08-07 - Windows 10 1809 remains the API floor, now qualified with its servicing reality (LTSC 2019 / Server 2019 are the serviced targets) - Notepad is two different apps across Server and Win11 client SKUs - CI tables reflect the real test-windows and test-linux lanes from v0.6.0 Adds the no-convenience-deferral rule: scope moves between sub-phases of a phase, never out of it. The only sanctioned deferral is proven impossibility, evidenced by a ledger row, shipping an honest PLATFORM_NOT_SUPPORTED. 2.14 and 3.14 are no longer deferrable stretch sub-phases.	2026-07-26 02:42:08 -06:00
Lahfir	dde0b98dc0	chore: track planning artifacts with secret and privacy scanning (#109) ... Tracks docs/plans and docs/brainstorms, adds .gitleaks.toml with privacy rules, and runs a history-mode gitleaks scan in CI. Docs and tooling only - no releasable change.	2026-07-26 01:41:55 -07:00
Lahfir	8ad66b8f21	refactor!: remove speculative Win32 private-file layer from core, add real Windows/Linux test lanes (#106)	2026-07-25 23:25:28 -07:00
Lahfir	3f322728b4	feat!: implement Playwright-grade foundation contract ... Settle the Playwright-grade reliability contract in agent-desktop-core before the Windows/Linux adapters are built, so they inherit it instead of redesigning it. Every command now observes, waits, verifies, and reports honestly instead of firing blindly. Highlights: capability-supertrait split of PlatformAdapter with not_supported() defaults; canonical role/state vocabulary with live `is --property visible`; display enumeration (`list-displays`) and honest `--screen` with scale factor; truthful Automation permission; `native_id` identity spine; window-id-first resolution; serializable `LocatorQuery` with live `find`; default-on auto-wait before every ref action; three-way `hit_test` occlusion gate; `scroll_into_view` in core; core accessible-name precedence; typed `ActionStep` delivery tier; `ProcessState` and `APP_UNRESPONSIVE`; `LaunchOptions`; baseline-diff desktop signals (`wait --event`); typed clipboard (`Text`/`Image`/`FileUrls`); mouse modifier chords and `mouse-wheel`. Hardened through a 35-reviewer pass with independent validation and a green live e2e gate (109/0), plus a head-vs-main performance comparison harness. BREAKING CHANGE: default-on auto-wait changes the timing of every previously-untouched ref-action call (bounded 5000 ms default; `--timeout-ms 0` restores single-shot). `ENVELOPE_VERSION` is now `2.1` (adds the `APP_UNRESPONSIVE` code and process state in error details). FFI ABI major is `3` (append-only struct evolution; `wait --event` is intentionally not exposed over FFI). The legacy string clipboard API is removed in favor of typed content. `key-down`/`key-up` fail closed until daemon-owned held input exists. `close-app` verifies termination and the osascript fallback path is removed. `--text` matching is subtree containment: `find --text X --first` returns the outermost matching container.	2026-07-20 00:21:38 -07:00
Lahfir	e3e1872ff3	feat: add trace viewer and replay artifacts ... Adds the trace read/replay layer on the session-first foundation: `trace show` merges per-process segments into one deterministic timeline (bounded JSON for agents), and `trace export` renders a single self-contained, XSS-safe HTML viewer for humans. Opt-in `session start --screenshots` captures pre/post-action screenshots and refmap copies; command.start/end boundary events and a versioned trace.meta header make a step-by-step replay reconstructable. Redaction is hardened so raw caller arguments never leak into trace-reachable error messages. Available across CLI, batch, and FFI.	2026-07-02 11:48:06 -07:00
Lahfir	35fa914b52	feat: make sessions the first-class trace container ... Sessions now own tracing: `session start` creates a manifest-gated trace sink with per-process JSONL segments under ~/.agent-desktop/sessions/<id>/trace/, so agents set the session once instead of passing --trace on every command. Bare --session stays snapshot-namespace-only; --trace <path> still overrides for CI and one-offs. Adds session start/end/list/gc, activation resolution (flag > AGENT_DESKTOP_SESSION > pointer), FFI verification tests, and status fields for session_id + tracing.	2026-07-01 17:11:33 -07:00
Lahfir	9023f331b3	feat(ffi): Phase B and C — Python smoke harness, parity gates, build.rs codegen (#77) ... Completes the FFI plan: U9 adds a Python ctypes smoke harness proving the C ABI from a non-Rust host; U10 adds cross-platform parity CI gates for header drift, codegen drift, panic-unwind, and stub-adapter passthrough; U11 generates the command-backed wrappers from build.rs templates with a committed drift-gated output. The C ABI stays byte-identical and the surface is Windows and Linux ready with no new FFI code.	2026-06-26 19:14:14 -07:00
Lahfir	9f144c2caf	fix: harden macos stale ref resolution (#62) ... * fix: harden macos stale ref resolution * fix: tighten stale ref fallback resolution * fix: fail closed for gone titled windows * refactor: tidy stale ref resolver naming	2026-06-20 21:37:46 -04:00
Lahfir	c0d52334c5	docs: update phase roadmap for v0.3.0	2026-06-20 13:13:33 -07:00
Lahfir	0c0a5b8dbc	fix: harden agent-desktop reliability (#54) ... * feat: add strict ref reliability core * feat: add session-scoped reliability diagnostics * fix: skip finder pseudo windows for snapshots * fix: report wait and trace failure context * fix: harden ref action reliability * fix: close reliability review findings * fix: harden reliability edge cases * fix: close final reliability edge cases * fix: address reliability follow-ups * docs: update reliability docs and skills * fix: harden wait and ref action reliability * refactor: centralize reliability helpers * fix: stabilize macos ref resolution * refactor: organize binary crate modules * fix: harden ref action reliability * fix: harden ref action reliability * docs: compound reliability patterns * fix: harden ref reliability edge cases * fix: harden source-window ref resolution * fix: preserve safe window title fallback * fix: make explicit snapshots session-independent * fix: harden ref fallback resolution * fix: fail closed on uncertain ref fallback * chore: strip inline comments and enforce docstrings Replace inline // comments with /// docstrings where they carry non-obvious contract, and add a pre-commit guard so inline comments cannot regress. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(review): preserve action results, delegate timeout resolution, tighten core boundaries Apply verified code-review fixes: ref-action release failures no longer mask successful action results (prevents double-dispatch on retry); resolve_element_strict_with_timeout defaults to delegating so strict-only adapters support wait --element; wait --text reports count only when --count is requested; latest-refmap refresh logs load failures instead of silently serving stale refs; InteractionPolicy moved to its own module and actionability/trace modules scoped pub(crate) per file rules; duplicate wait test helper extracted to shared support module; timeout error constructors deduplicated; redaction test covers description; policy focus-denial path covered; skills document steps array, actionability details, trace redaction, and batch trace inheritance. * fix(review): eliminate per-element AX round trips and close remaining review findings Fold AXPosition/AXSize and the scrollbar probe into the existing AXUIElementCopyMultipleAttributeValues batch so tree traversal and the actionability preflight pay one IPC per element instead of up to four; A/B benchmark shows strictly-faster snapshots with identical ref counts and scroll capabilities (Finder 4.5s -> 2.4s same-session, Docker Desktop parity at 440 refs / 933 scroll-capable nodes). Consolidate the CLI and FFI ref-action pipelines into one core execute_resolved path (actionability, tracing, and dispatch semantics live once; FFI passes a default context). Remove the no-context execute() shims from is/right-click/snapshot/wait and the test-only helper shims; every command now takes an explicit CommandContext. Split the macOS resolver into resolve (orchestration), resolve_search (candidate collection), and resolve_classify (strict classification), clearing the 400-LOC ceiling with room to grow. Notification waits now retry transient baseline failures inside the timeout budget with the same retryable gate as window/text waits and report last_error in timeout details instead of aborting on the first flake; a baseline is never fabricated. Refmap writes clean up their temp file on failure, stale *.tmp orphans are swept under the store lock, and save_existing_snapshot re-verifies snapshot ownership inside the owning store's write lock with bounded re-discovery before deterministically recreating in the caller's store. Coverage hardening: zero-budget wait timeout shape, wait --text --count 0 absence detection, ref-action pipeline call-count guard (1 resolve / 1 live read / 1 dispatch), duplicate snapshot-id collision on load, pruned-everywhere recreation, tmp sweep and rename-failure cleanup, FFI AMBIGUOUS_TARGET last-error code assertion. * docs: document notification retry, error-code contract, and diagnostics sensitivity Note transient-error retry and last_error timeout detail on wait --notification; make explicit that agents branch on error.code (message and suggestion text is informational); warn FFI consumers that ad_last_error_details may carry on-screen element names, values, and window titles and should stay out of shared log surfaces. * fix: validate find roles against the canonical vocabulary find --role with a role no adapter can emit (textarea, typos) silently returned ok with zero matches, reading as 'element absent' when the query could never match. Role queries now resolve through a canonical vocabulary in core: common text-input aliases (textarea, textbox, searchfield) normalize to textfield case-insensitively, and unknown roles fail with INVALID_ARGS carrying details.valid_roles so agents can self-correct. The macOS role mapping becomes a sorted single-source table with binary-search lookup, and a conformance test pins every emitted role (plus the unknown fallback and the synthesized cell role) to core's CANONICAL_ROLES — the cross-platform contract Windows/Linux adapters must map their native vocabularies into, enforced by the same table + test pattern rather than convention. * refactor: derive find role hints from the live tree, drop hardcoded vocabulary Replace the canonical-role allow-list (and its hard INVALID_ARGS rejection) with a tree-derived approach. A role filter that matches nothing now returns ok with roles_present — the distinct roles actually in the searched tree — so the caller distinguishes 'none on screen' from a wrong role name and self-corrects. This needs no central role list: a role any adapter newly emits surfaces in roles_present automatically, with nothing to keep in sync across core and the platform crates. A tiny role-query normalizer keeps the ergonomic win (textarea, textbox, searchfield fold to textfield, case-insensitive) but never gates or rejects — it is a synonym shim, not a vocabulary. The macOS role table returns to its plain match form; the cross-crate canonical-vocabulary list and its conformance test are gone. * refactor: move protected-process knowledge out of core into the adapter close-app hardcoded macOS/Unix process names (loginwindow, windowserver, dock, launchd, finder) inside core, baking platform-specific knowledge into the platform-agnostic crate. Windows would need csrss.exe/ winlogon.exe, Linux gnome-shell/Xorg. Add PlatformAdapter:: is_protected_process (default denies nothing); the macOS adapter owns its list with substring matching over display and bundle identifiers. core's close-app just asks the adapter. Also genericize a macOS-flavored test fixture string so core carries zero native vocabulary even in tests. Verified: core has no platform-native references in non-test source, no cfg(target_os) gates; the only remaining cfg(unix)/libc use is securing core's own refmap/trace/lock files with non-unix fallbacks. * fix: stop close-app claiming a graceful quit it cannot confirm close-app returned closed:true the instant a graceful quit was *sent*, while the app was still running behind an unsaved-changes dialog — a false completion claim. Empirically (NSWorkspace.runningApplications): a clean quit completes in ~0.2s, a dialog-blocked quit never completes on its own, and macOS confirms only that the quit request was sent, not that the app terminated. Verifying by polling would add seconds of latency on the exact (blocked) case it is meant to catch, so we do not poll. Graceful close now reports { method: graceful, requested: true } — truthful and instant, no closed claim. --force is a synchronous SIGKILL, so it reports { method: force, requested: true, closed: true }. Callers needing graceful confirmation observe via list-apps / wait --window and can drive a save dialog with snapshot + find, which is the agent-native path. * feat: add drag --drop-delay for reliable macOS drop registration macOS drop targets need the dragged item to dwell over them before they register as the destination; too short and the gesture lands as a drag with no drop. The dwell was a hardcoded 500ms dead sleep. Expose it as --drop-delay <ms> (CLI), drop_delay_ms (DragParams/AdDragParams, 0 = adapter default sentinel matching duration_ms), and replace the dead sleep with an event-driven dwell that posts LeftMouseDragged over the destination every 16ms so the target stays highlighted instead of dropping the drag mid-pause. DRY: the C-to-core drag conversion (duration/drop-delay zero-sentinel) was copied across three FFI sites; collapse them into AdDragParams:: to_core(). FFI ABI: AdDragParams gains drop_delay_ms (header + repr + header-compile test). Tests: core threads the value into params and response and omits the field when unset; FFI maps both optionals. * fix: make action-bearing elements ref-able so scroll/expand can target them E2E testing against a diverse fixture app surfaced that disclosure (Expand/Collapse/Click) and scrollarea (Scroll) advertise actions but never received refs — they are not in INTERACTIVE_ROLES — so the scroll, expand, and collapse commands required a <REF> their own target roles could never have. The commands were uninvokable against their primary targets. Ref allocation now gates on addressability, not role alone: an element is ref-able if its role is interactive OR it advertises a primary action (any action other than a bare SetFocus, which would ref-allocate inert focusable containers). scrollarea and disclosure become ref-able; scroll now works against a real app. Ref-count impact is modest (fixture 61->72, Finder ~262). Tests assert action-bearing containers get refs, SetFocus-only and inert elements do not, and interactive roles stay ref-able without actions. Contract docs (CLAUDE.md, SKILL.md) updated. * fix: eliminate vacuous AX successes and harden resolution Dogfooding the binary against a real fixture app surfaced five cases where a command reported success without producing the effect, or failed with the wrong error. Each is verified by independent before/after observation in the E2E harness. - is_menu_open no longer treats a latent AXMenuBar as an open menu, so select and wait --menu-closed stop seeing a permanently-open menu. - set-value coerces the written AXValue to the element's existing CFNumber/ CFBoolean/CFString type and verifies numerically, fixing sliders; steppers converge via AXIncrement/AXDecrement when AXValue writes are vacuous. - double-click only claims success when the element advertises AXOpen; otherwise it fails closed instead of reporting a non-existent double-click. - a completed resolution pass that proves a ref absent downgrades a deadline TIMEOUT to STALE_REF so removed elements fail with the correct code. - expand/collapse verify the disclosure state and fall back to a press-toggle for press-driven disclosures; press-toggled containers expose EXPAND/COLLAPSE. roles.rs adds the disclosure expandable role and normalizes textarea/textbox/ searchfield role queries to textfield. * feat: add Playwright-style headed/headless interaction mode Ref actions now run in exactly two modes. Headless is the default: semantic accessibility operations only, no cursor movement, and a fail-closed POLICY_DENIED when only a physical gesture would work. The global --headed flag upgrades every ref action to permit focus stealing and cursor movement, so the chain's physical click/double-click/scroll/keypress fallbacks can complete. The AX path is always tried first, so --headed never regresses headless-capable elements; it only adds fallbacks for elements that need a real gesture. - CommandContext::request(action, base) builds the per-command request: each command declares its headless base (pure-AX headless; type uses focus_fallback because typing requires focus but never moves the cursor) and --headed upgrades any base to the headed policy. - The internal/FFI "physical" policy is renamed "headed" throughout, including the C ABI enum (AD_POLICY_KIND_HEADED keeps discriminant 2) and bindings. - Raw-input commands (press, hover, drag, mouse-*, key-down/up) are unchanged: always physical, mode-independent low-level escape hatch. - Unit tests assert every ref command is headless by default and headed under --headed; docs (CONCEPTS, CLAUDE, skills) describe the two-mode contract. * test: add E2E fixture app and dual-mode harness Drives the release binary against a real SwiftUI/AppKit fixture and verifies every effect by independent before/after observation — never the command's own ok:true — so a command that reports success without an effect is caught. This is the layer mock-adapter unit tests cannot cover: it exercises the contract against the real macOS Accessibility API. - AgentDeskFixture.swift exposes a fixed, diverse AX surface (native AppKit slider/stepper, gesture-only and ambiguous controls, a sheet, a press-toggled disclosure, async-appearing elements, a drag canvas). It is never tuned to make a command pass; a failure is a finding about the CLI or the harness. - run.sh drives every ref-action command in BOTH headless and --headed mode with mode-specific target values, plus the double-click discriminator (headless fails closed with POLICY_DENIED, --headed completes) that proves the two modes differ. It also covers strict resolution, wait predicates, skeleton drill-down, sessions, trace redaction, surfaces, drag, expand, and force-close. - The compiled fixture .app is a build artifact (gitignored; built on demand). Run: cargo build --release && bash tests/e2e/run.sh (needs AX permission). * fix: close review runtime, correctness, and security gaps Addresses validated findings from the branch code review: - chain: thread the chain deadline into increment_to_value so a non-converging stepper cannot spin up to 1024 AX round trips and blow past the timeout. - mouse: a RAII guard posts LeftMouseUp if any fallible step of a drag returns early, so an error can never leave the mouse button held down system-wide. - ffi: cap caller-supplied state/action/path counts before from_raw_parts, mirroring the existing MAX_MODIFIERS_PER_COMBO guard, to reject out-of-bounds reads from a garbage C count. - wait: the actionable predicate now forwards the structured ActionabilityReport from error.details instead of dropping it to a flat message, so agents can see which check is blocking. - scroll: gate the row-select fallback on policy.allow_focus_steal so a headless scroll can no longer silently change the user's table selection. - actionability: delete the unreachable stable->StaleRef branch (stability_check never fails by design) and its now-dead failed_check helper. - status: delete two pub wrappers with no production callers; the test now drives the real execute_with_report_with_context entry point. * refactor: extract disclosure chain steps under the 400-LOC limit chain_steps.rs had grown to 402 lines, over the hard per-file limit. The six press-toggle disclosure helpers form a cohesive group and move cleanly into a sibling chain_disclosure_steps.rs (following the chain_web_steps/ chain_menu_steps pattern); chain_defs.rs references the new module. No behavior change. * feat: focus the target window before ref-addressed physical input Ref-action physical fallbacks (click/scroll/type) already brought the target app frontmost before synthesizing CGEvents. The raw-input commands (drag, hover, mouse-*) resolving a point from a ref did not — the resolver had the pid but discarded it, so the adapter saw only coordinates and synthetic events could land on whatever window happened to be frontmost. Add a best-effort focus_app(pid) to PlatformAdapter (macOS uses ensure_app_focused; other adapters default to not_supported). The point resolver now focuses the ref's app before returning, so every physical interaction that targets a known element raises its window first. Coordinate (--xy) input is unchanged: the caller owns the target there. * fix: add AdDragParams size guard and document the ABI breaks AdDragParams gained a drop_delay_ms field but, unlike AdRefEntry, had no size guard — an old caller's smaller allocation would let Rust read past it and turn stack garbage into a real drop delay. Add AD_DRAG_PARAMS_SIZE, ad_drag_params_size(), a compile-time layout assertion, and a zero-init note, matching the ref-entry pattern. This branch makes several consumer-visible contract changes that release tooling must cut as a major. They are gathered here because the release workflow ships the C header as an artifact. BREAKING CHANGE: the C ABI and CLI/JSON contract changed on this branch. - AdPolicyKind: AD_POLICY_KIND_PHYSICAL is renamed AD_POLICY_KIND_HEADED (discriminant 2 unchanged, so compiled binaries are safe; source-level C consumers must rename). No back-compat alias is kept — "physical" is gone. - AdRefEntry grew (caller-allocated input); validate layout with AD_REF_ENTRY_SIZE / ad_ref_entry_size(). - AdDragParams grew; validate with AD_DRAG_PARAMS_SIZE / ad_drag_params_size() and zero-initialize before use. - close-app graceful response no longer includes closed:true; it returns { method: "graceful", requested: true } because a graceful quit cannot be synchronously confirmed. * test: harden and expand the E2E proof layer Closes the honesty gaps the review found and covers the interactions that were missing, all verified by independent before/after observation: - twins: the fixture twins now record distinct effects (twin-a/twin-b) and the assert requires the ADDRESSED twin to fire (or AMBIGUOUS_TARGET), instead of passing on any ok:true. - click: click-status is a counter, so the headed pass must observe a fresh increment rather than inheriting the headless pass's value. - adds triple-click + hover (headed gestures), tab selection (TabView tabs are radiobuttons), context-menu open + item selection, and menu-bar enumeration via --surface menubar. - adds a performance section reporting per-command CLI wall-clock (snapshot, find, get, click, set-value, type) with a soft <2s snapshot gate. - documents the SwiftUI CommandMenu and cross-app drop limitations as tracked notes, not silent skips. * docs: trim CLAUDE.md to standards and document gesture headless-capability - CLAUDE.md: remove ~130 lines of reference material that duplicated code, Cargo.toml, or the skills (full PlatformAdapter trait dump, Key Types listing, macOS API listings, dependency/build-config tables, the 54-command table). Replaced the stale trait dump with a pointer to adapter.rs (which also fixes the review's stale-trait-docs finding) and kept only the non-obvious gotchas. CLAUDE.md is now standards, invariants, and conventions. - Document, on macOS (Phase 1), which gestures have a headless path: most ref actions do; double-click via AXOpen; triple-click/hover/drag are cursor gestures with no AX equivalent (physical only). The command surface is platform-agnostic — a future Windows/Linux adapter that exposes a headless path lights it up with no command or core change. Added to README and the interaction reference. - FFI skill: AD_POLICY_KIND_PHYSICAL is now AD_POLICY_KIND_HEADED. * docs: capture gesture headless-capability learning and refresh policy docs - Add best-practices/macos-gesture-headless-capability: which desktop gestures have a headless AX path on macOS (double-click via AXOpen; triple-click/hover/ drag are physical-only; SwiftUI controls vs native AppKit), and why the command never decides — the platform adapter owns headless-vs-physical. - Refresh two policy learnings for the physical->headed rename: ActionRequest:: physical -> headed and AD_POLICY_KIND_PHYSICAL -> AD_POLICY_KIND_HEADED, noting the new global --headed upgrade path via CommandContext::request. * fix: address review P2 correctness and reliability findings - wait: cap each ref-resolution attempt (750ms) so a slow resolve cannot consume the whole wait budget on the first poll; the predicate is re-checked across the full timeout. - wait: make LatestRefCache timing fields private (no external readers). - wait: add a unit test for --menu-closed (asserts it waits for open=false). - close-app: make the graceful and force responses symmetric — both carry `closed` (force confirms true; graceful cannot confirm, so false) instead of graceful silently omitting the field. - close-app: test adapter-error propagation. - adapter: the default resolve_element_strict_with_timeout now logs that it does not enforce the deadline, so an adapter that forgets to override it is visible in traces. - type: a RAII guard restores the user's clipboard on every scope exit (success, error, panic) during the paste-based non-ASCII path, shrinking the clobber window to an unpreventable SIGKILL. Note: the reviewer's "fail fast on AMBIGUOUS_TARGET with a pinned snapshot" suggestion is not applied — existing tests prove transient ambiguity resolves on retry even with a pinned ref (ambiguity is a property of the live tree, not the refmap), so failing fast would regress intentional, tested behavior. * perf: trim actionability preflight and resolve-search allocations - action_list: gate the AXValue and AXExpanded `is_settable` probes on whether the role could plausibly carry that capability (unknown roles always probe), skipping up to two AX round trips per preflight on common click-only targets. No capability is lost — value/expandable roles still probe. - resolve_search: reuse one scratch FxHashSet across nodes instead of allocating a fresh dedup set per node during path and recursive search. * refactor: split actionability types into one file each ActionabilityStatus, ActionabilityCheck, and ActionabilityReport move into their own files under a new actionability/ module (following the tree/ and actions/ folder pattern); mod.rs keeps the check logic and re-exports the types. Honors the one-domain-type-per-file rule without changing behavior. * test: split fixture under the LOC limit and fix swift-ios issues - Extract the reusable fixture components (status readout, native AppKit slider/stepper, drag canvas, card) into FixtureComponents.swift so each file is under the 400-LOC limit; build.sh now compiles every .swift in the dir. - NativeSlider/NativeStepper implement updateNSView so a SwiftUI binding change syncs back to the NSView. - The fixture no longer steals focus unconditionally on launch (activate ignoringOtherApps:false), so it cannot mask headless-policy focus violations; the harness drives focus explicitly. - build.sh pins the SDK and a macOS 13 deployment target for reproducible builds. * docs: document the optional error.details field and roles_present shape - SKILL.md: note that the error object may carry an optional `details` (the actionability report, AMBIGUOUS_TARGET candidates, or a wait TIMEOUT's last observed state) and that responses should be parsed leniently — `details` and future fields are additive. - commands-observation: show the no-match `find` response with the `roles_present` hint so callers can tell a wrong role name from "none on screen". * fix: gate the ref-gesture focus raise on the interaction policy Ref-addressed hover/drag raised the target app unconditionally, violating the headless no-implicit-focus-steal contract. The point resolver now returns the owning pid instead of focusing, and commands decide: headless never raises, --headed raises once (drag focuses only the from-app, fixing the cross-app double-focus). Responses report focused:true so multi-app agents can detect the frontmost change. * fix: abort failed drags at the origin and disarm the guard only on success The mouse-up guard disarmed before the final fallible up-event, so a failed final post left the button held. Worse, its corrective release fired at the unreached destination, silently committing an aborted drag as a completed drop (CGEvents resolve at their embedded coordinates). The guard now owns the release: it disarms only after the up actually posts, and an early return cancels by dragging back to the origin and releasing there. * fix: enforce the chain deadline inside increment steps All dispatch sites construct ChainContext with deadline: None, so the remediation parameter on increment_to_value never received a value and the 1024-iteration loop ran unbounded by the chain timeout. The chain now pins its resolved deadline into the context every step observes. Also extracts the pure write-verification predicates and their tests to chain_verify.rs, bringing chain.rs back under the 400-LOC limit. * fix: pin the AdAction ABI layout and bound FFI string and array inputs AdDragParams is embedded by value in AdAction, so its 8-byte growth grew the struct C callers pass to ad_execute_action with no size guard — old-layout callers under-allocate and stack garbage becomes a live drop_delay_ms. Adds AD_ACTION_SIZE / ad_action_size() with a layout pin, matching the AdRefEntry pattern. Also hardens the input boundary: C strings are decoded with a bounded NUL scan (AD_MAX_STRING_BYTES, sized for CLI argv parity) so a missing terminator cannot walk arbitrary memory, and the single coarse 1024 array cap becomes published per-field caps (AD_MAX_REF_STATES/ACTIONS/ PATH_DEPTH) with tests just over each limit. * perf: replace fixed input settle sleeps with state polls ensure_app_focused slept 50ms per physical input even when the app was already frontmost; it now polls AXFrontmost (1ms, 50ms deadline). disclosure_settled slept an unconditional 40ms up to three times per expand/collapse; it now polls the disclosed state (5ms, 200ms deadline), converging immediately on fast UIs. The kAXFocusedAttribute settability probe is gated by role_may_accept_focus, mirroring role_may_bear_value, and key dispatch reuses ensure_app_focused instead of its inline duplicate. * feat: check a specific action in wait --predicate actionable The actionable predicate hardcoded Click, so wait-then-type flows got a false ready on fields that cannot accept text (the editability check only runs for editing actions). --action selects click (default), type, set-value, or clear, and the preflight mirrors each command's real base policy (type uses its focus-fallback base). * fix: redact title, url, help, and placeholder keys in traces Window titles, URLs, tooltips, and placeholder text carry user content just like names and values; the redaction list now covers them. * test: harden the e2e harness and unify fixture AX labels The harness now fails setup loudly when the fixture build or AX trust is missing, rebuilds the fixture when sources are newer than the bundle, asserts hover only against a freshly observed state, and force-collapses the disclosure so the expand test proves a real flip. The slider/stepper labels live solely on the NSViews (the AX-actionable elements), removing the macOS-version-dependent race between two label sources, and the drag canvas reports a zero frame when detached from a window. * docs: record the perf commit type, pre-1.0 bump policy, and error details field Adds perf: to the allowed commit types (release-please already maps it to a Performance changelog section), records the pre-1.0 versioning policy so a BREAKING footer is expected to cut a minor rather than a major, and shows the optional error.details object in the error envelope docs. The gitignored local AGENTS.md mirror got the same contract sync. * fix: surface increment deadline expiry as timeout with the observed value A chain deadline firing mid-increment returned Ok(false), so the step was recorded as skipped, the chain exhausted into ACTION_FAILED, and the control sat at a half-applied value the caller could not see — post-state is only read on success, and ACTION_FAILED recovery guidance points away from retrying. Expiry is now a TIMEOUT error carrying value_before, value_at_timeout, target, and a mutated flag in details. * perf: cap the disclosure settle poll to the chain deadline and widen its interval The settle poll could spend 3 x 200ms x 5ms-interval reads (~360 IPCs) per expand/collapse and overshoot the chain's own deadline. A new CustomWithDeadline chain step threads the chain deadline into the disclosure steps, the settle budget is min(200ms, remaining chain budget), and the interval widens to 20ms (~30 IPCs worst case). * fix: enforce the protected-process guard inside the adapter close path The guard lived only in the CLI command layer, so ad_close_app could force-kill session-critical processes (loginwindow, WindowServer, Dock) that the CLI refuses. close_app_impl now refuses them before any side effect with the exact CLI error contract, making CLI, FFI, and any future consumer behave identically; the CLI preflight remains as an earlier check against the same predicate. * fix: make focused semantics honest and confirm window focus by polling ensure_app_focused set AXFrontmost unconditionally and reported success identically whether or not a raise happened; it now no-ops when the app is already frontmost, so Ok (and the focused:true response field) means "frontmost ensured" exactly as documented. focus_window_impl gains the same confirmation poll after its raise, and the poll interval widens to 5ms (10 reads max in the 50ms window). * fix: attach abort-state guidance to drag failures and document cancel limits Drag synthesis errors surfaced as bare INTERNAL with no hint about the gesture's end state. Failures now carry a suggestion stating the button was released back at the origin (best-effort), no drop was committed, and where the cursor ends; the guard doc spells out the two best-effort limits (corrective posts can fail; a self-drop at the origin is a no-op for most targets). * refactor: split oversized files by responsibility under the 400-LOC limit helpers_tests (426) splits into resolution/window/pipeline tests, a ref-action+trace test file, and a shared entry-builder support module. wait_element_tests (414) splits into predicate-behavior tests and resolution/lifecycle tests over a widened wait_test_support. wait.rs (395) loses the element-wait loop to wait_element.rs, and refs_store (397) moves its tmp-cleanup/retention methods to a refs_store_prune child module (declared via #[path] so the split keeps base_dir and snapshots_dir private to the store). * refactor: build the actionable preflight request per action name at parse The policy mirror lived in a separate helper with a catch-all arm, so a future action name could silently inherit the headless policy. Parse now maps every --action name to the exact ActionRequest its real command runs (type is the only focus-fallback), the catch-all is gone, and a test pins each name's policy. * refactor: move point resolution and the focus helper to point_resolve PointResolveArgs, ResolvedPoint, the ref-or-xy resolver, and focus_for_physical_input were accumulating in helpers.rs alongside unrelated ref-action plumbing; they now live in a dedicated point_resolve module consumed by hover and drag. * test: keep the drag canvas AX label on the NSView only The DragCanvas carried two label sources (the NSView and a SwiftUI modifier on its representable), the same macOS-version-dependent race the slider/stepper fix removed; the harness-facing label now lives solely on the AX-actionable NSView. * docs: sync skills and header with the focused, redaction, and wait contracts Documents the ensured (best-effort, already-frontmost-aware) semantics of focused:true and its absence-vs-false meaning, the four redaction keys added in round 2 plus the substring-match behavior, the INTERNAL error recovery row, the FFI wait-surface asymmetry, the cross-app drag occlusion caveat, and the AdDragParams/AdAction layout history with the adjudicated pre-1.0 breaks so fresh reviews stop re-finding them. * fix: surface settle-wait deadline truncation as timeout with a schema discriminant A chain deadline truncating the disclosure settle wait returned a plain step failure, exhausting into ACTION_FAILED — the same masking class fixed for increments — even though the triggering action may still land after the truncated wait. Settle exits are now classified: full-budget misses stay step failures, deadline-truncated waits raise TIMEOUT with the wanted/observed state, and the poll sleeps are clamped so a tight deadline still gets at least one read. All TIMEOUT details now carry a kind discriminant (wait_timeout vs chain_deadline) so agents can branch without sniffing field names. * fix: match protected processes exactly, not by substring 'docker'.contains('dock') permanently blocked close-app for Docker, FinderSync-class apps, and anything else embedding a protected name. Matching is now an exact lowercase name or an exact dot-separated bundle-id component, so Dock and com.apple.dock stay protected while Docker, Docker Desktop, FinderSync, and PathFinder stay closable — pinned by false-positive tests. * fix: raise the element's window before the physical click fallback CGEvents land on the topmost window at the click point, so an app being frontmost is not enough when the target element lives in a background window of that app — the physical fallback clicked whatever overlapped it, and the skip-raise-when-frontmost optimization widened the window for that. click_via_bounds now raises the element's own AXWindow (AXRaise, AXMain fallback, brief confirmation poll) via a shared window_ops helper that focus_window_impl reuses. Verified live: a headed click on a ref in an occluded Finder window raises that window and lands the click in it. * test: guard ref-action policy coverage against silent gaps A new ref-action command could ship without a base-policy assertion. A guard test now scans crates/core/src/commands/ for files calling context.request( and fails unless each stem appears in the POLICY_TESTED_COMMANDS list backing the policy assertions. * fix: carry the protected-process suggestion on the CLI preflight The CLI-layer guard returned bare INVALID_ARGS while the adapter layer carried recovery guidance, so agents on the primary surface got 'check command syntax' for a permanently-disallowed operation and looped on argument fixes. Both layers now state the same suggestion. * docs: document the TIMEOUT schemas, chain deadline knob, and roles_present scope Names the two TIMEOUT details schemas by their kind discriminant with the mutated-flag retry rule, points chain-deadline recovery at AGENT_DESKTOP_CHAIN_TIMEOUT_MS instead of --timeout, extends the roles_present hint to all non-count selection-mode misses, and aligns the STALE_REF recovery row with the richer error.rs suggestion. * test: extract the scroll card and document drag-canvas data flow AgentDeskFixture.swift sat at exactly 400 lines; the scroll card is fully self-contained (its offset state never leaves the card), so it moves to FixtureCards.swift as a standalone view with zero bindings, landing the main file at 374. Harness-facing labels are byte-identical. DragCanvas gains two intent comments distinguishing the deliberately empty updateNSView from a forgotten sync. * chore: pin ABI sizes for C consumers and explain the prune module split C11-gated _Static_asserts mirror the Rust-side layout pins so a C consumer compiling against a drifted header fails at build time, and the production #[path] prune module carries its privacy rationale. * test: give racing wait tests a deterministic budget Three wait tests used a 1ms timeout that can elapse before the loop's first resolution attempt under load, flaking on machine pressure (the ambiguous-resolution test needs at least one attempt to record its observation). 50ms guarantees the first attempt without slowing the suite. * docs: capture three round-4 review learnings and grow the concept map Documents the abort-state contract for multi-step physical input (guard disarm ordering, origin release, end-state suggestions), the three-layer repr(C) size-pinning discipline born from the AdAction silent-growth incident, and the named-arms-plus-exhaustiveness-guard pattern for policy/dispatch mirrors. CONCEPTS.md gains Action Chain and Protected Process and refreshes Coordinate Fallback with the window-topmost rule. * docs: refresh six learnings against the enhanced-reliability branch Brings the learning corpus back in line with code that moved this branch: the gesture-capability and policy docs now describe the window-level raise in the physical path and link the new abort-state doc, the reliability contract documents the TIMEOUT kind discriminant and the wait --action per-name policy variant, the FFI review rule covers structural repr(C) size drift alongside behavioral parity, the allocator doc records how the config struct absorbed four more fields in one place, and the fingerprint doc names the real tri-state decode error type. Three docs verified accurate with no edits. * docs: sync roadmap with current reliability contracts * fix: harden desktop action reliability * fix: harden reliability review regressions * fix: close final reliability review gaps * fix: close reliability review gaps * test: avoid raw pointer mutation in ffi free tests * refactor: trim reliability branch dead code * fix: close reliability review gaps BREAKING CHANGE: the C ABI AdActionResult layout now includes action steps; C consumers must rebuild against the updated agent_desktop.h header. * fix: close reliability review gaps * ci: scope cache hash inputs --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-20 15:43:42 -04:00
Lahfir	a520b7b574	docs: scrub public phase roadmap	2026-05-24 02:13:10 -07:00
Lahfir	c16a91e9bd	docs: keep phase planning artifacts local	2026-05-24 02:06:27 -07:00
Lahfir	9435b3c193	docs: harden phase 2 windows plan	2026-05-24 02:01:33 -07:00
Lahfir	1fb5a7d51e	fix: stabilize empty accessibility identity refs ... * fix: treat empty accessibility text as missing identity * fix: align help and skill snapshot docs * fix: preserve accessibility descriptions in refs * fix: resolve refs after window title drift * fix: harden stale ref identity resolution * ci: add explicit CodeQL workflow * fix: fail closed when scoped refs drift	2026-05-23 06:07:32 -04:00
Lahfir	1291a9cdbf	refactor!: unify command execution contracts ... Unify CLI and batch dispatch around the typed command path, centralize command policy and ref resolution, harden macOS action verification, split command tests from implementation, and add package/release guardrails. BREAKING CHANGE: CLI and batch execution now share the typed command path and current command argument contracts. BREAKING CHANGE: Ref-consuming commands use snapshot-scoped refs; deterministic consumers should pass snapshot_id and handle SNAPSHOT_NOT_FOUND. BREAKING CHANGE: permissions and status now return PermissionReport fields for accessibility, screen_recording, and automation instead of a single boolean status. BREAKING CHANGE: PermissionState gains NotRequired; macOS automation now reports not_required instead of unknown. BREAKING CHANGE: right-click now separates action success from menu verification; consumers should inspect menu or menu_probe instead of assuming every right-click returns an inline menu. BREAKING CHANGE: focus-window now confirms OS focus and returns ACTION_FAILED when focus does not settle; data.focused.is_focused is true on success. BREAKING CHANGE: PlatformAdapter::execute_action now takes ActionRequest, and permission probing uses permission_report/request_permissions. BREAKING CHANGE: FFI ad_execute_action now defaults to headless policy. Consumers that need focus fallback or cursor-moving behavior must call ad_execute_action_with_policy with AD_POLICY_KIND_FOCUS_FALLBACK or AD_POLICY_KIND_PHYSICAL. BREAKING CHANGE: FFI ad_check_permissions no longer treats unknown accessibility permission as success; stub-style unknown probes return ERR_PLATFORM_NOT_SUPPORTED and macOS ambiguous unknown returns ERR_INTERNAL with last-error detail. BREAKING CHANGE: JSON response envelopes now report version 2.0; parsers pinned to 1.0 must branch or update. BREAKING CHANGE: focus now uses accessibility focus without cursor movement; callers that need physical focus must use explicit mouse or physical-policy paths. BREAKING CHANGE: chain execution deadlines now return TIMEOUT instead of ACTION_FAILED when the target app does not respond before the chain deadline.	2026-05-19 18:27:08 -07:00
Lahfir	99fd6799ec	docs(phases): expand Phase 2 plan invariants and dependency table ... Distributes Phase 2 plan research findings into roadmap: headless-first invariant, Windows engineering invariants, DeliverFiles rationale, build.rs registry decision, and updated dependency pins.	2026-04-24 18:04:02 -07:00
Lahfir	2d373708e7	docs(phases): sync roadmap + distribute gap findings + codify DRY invariant (#28) ... Docs-only sync of `docs/phases.md` against the v0.1.13 shipped reality, with four research agents' findings distributed into Phase 2-5 as concrete objectives and the command-surface DRY invariant codified at the top of the document.	2026-04-17 05:21:47 -07:00
Lahfir	3cffbd67f6	feat(ffi): ship C-ABI cdylib with review hardening and release pipeline (#26) ... Lands the full agent-desktop FFI layer with every PR #22 review finding resolved, modular refactor applied, todo-resolve batches (011 + 006) closed, and a production release pipeline that bundles the prebuilt cdylib for 5 target triples alongside the CLI on every GitHub Release. **FFI surface** - Panic-unwind cdylib boundary via `trap_panic` / `trap_panic_ptr` / `trap_panic_const_ptr` / `trap_panic_void` - Runtime main-thread enforcement on every macOS-sensitive entrypoint; TLS errno-style last-error lifetime - Every `#[repr(i32)]` field validated at the C boundary via `try_from_c_enum!` — arbitrary bit patterns return `ErrInvalidArgs` without UB - BFS flat-tree layout with `child_start` / `child_count`; iterative traversal - Opaque list handles (`AdAppList`, `AdWindowList`, `AdSurfaceList`, `AdNotificationList`); opaque `AdImageBuffer` with `_data` / `_size` / `_width` / `_height` / `_format` accessors - `AdNativeHandle` single-owner single-thread contract; zero-on-free makes double-call deterministic - Fail-closed UTF-8 for optional filter pointers (`try_c_to_string` tri-state) - `AdTreeOptions` fully honored in `ad_get_tree` (include_bounds / interactive_only / compact) - Verified notification action identity (`NotificationIdentity` fingerprint) — refuses to press if NC reordered between list and act **Release pipeline** - New `build-ffi` matrix in `.github/workflows/release.yml` producing `libagent_desktop_ffi.{dylib,so,dll}` tarballs for aarch64/x86_64-apple-darwin, x86_64/aarch64-unknown-linux-gnu, x86_64-pc-windows-msvc - macOS `install_name = @rpath/libagent_desktop_ffi.dylib` baked in by `build.rs` and CI-verified via `otool -D` - `actions/attest-build-provenance@v4.1.0` — keyless Sigstore provenance over every release artifact; `gh attestation verify` - `checksums.txt` covers both CLI and FFI assets; asset-count assertion bumped 3 → 8 - npm package stays CLI-only; Python/Swift/Go/Ruby/Node/C hosts pull dylib tarball directly from the Release - README gains a "Language bindings (FFI)" section with platform→artifact table **Docs** - `skills/agent-desktop-ffi/` — SKILL.md + build-and-link.md + ownership.md + threading.md + error-handling.md - `docs/solutions/best-practices/` — two new solution docs (deterministic build-artifact marker, identity fingerprint against OS reorder) **Verification** - `cargo clippy --all-targets -- -D warnings` — clean - `cargo test --lib --workspace` — 139 passed (5 suites) - `cargo test -p agent-desktop-ffi --tests` — 87 passed (4 suites, including new `c_header_compile` C-ABI harness) - macOS PR CI green on multiple runs; latest: 24561160455 - Python `ctypes.CDLL` round-trip from a freshly extracted FFI tarball confirmed; adapter lifecycle + `ad_list_apps` enumeration clean; `install_name` survives the tarball Closes #22 (superseded).	2026-04-17 04:00:43 -07:00
Lahfir	c17f2fae7a	feat: progressive skeleton traversal with ref-rooted drill-down (#20) ... * feat: add data model foundation for progressive skeleton traversal Add children_count, root_ref, skeleton fields to core types. Add get_subtree() to PlatformAdapter trait. Add remove_by_root_ref() and write-side size check to RefMap. No behavioral changes yet. * feat: implement progressive skeleton traversal with ref-rooted drill-down Add --skeleton and --root flags to snapshot command for token-efficient accessibility tree exploration. Skeleton mode clamps depth to 3 levels and annotates truncated containers with children_count, allowing AI agents to discover regions before drilling into them. Named containers at skeleton boundaries (via name or description) receive refs as drill-down targets. The --root flag starts traversal from a previous ref with scoped invalidation — only refs from that drill-down are replaced on re-drill. Key changes: - New ref_alloc.rs: shared ref helpers (INTERACTIVE_ROLES, actions_for_role, ref_entry_from_node, is_collapsible) extracted from snapshot.rs - New snapshot_ref.rs: drill-down logic with DrillDownConfig, scoped invalidation via root_ref tagging on RefEntry - macOS count_children() uses raw CFArrayGetCount without materializing AXElement wrappers for performance at skeleton boundaries - RefMap write-side size check prevents >1MB files - Skeleton anchors consider both name and description for Electron compat * fix: mention --skeleton in STALE_REF error suggestion * docs: document --skeleton and --root flags in skill reference * docs: update phases.md and CLAUDE.md for progressive skeleton traversal Add skeleton traversal as Phase 1 objective P1-O10. Document --skeleton and --root flags, get_subtree() trait method, new core modules, and platform-agnostic notes for Phase 2/3. Update risk mitigations and performance optimizations table. * docs: make progressive skeleton traversal the default agent workflow Update SKILL.md observe-act loop to skeleton-first approach. Add progressive skeleton traversal as the primary workflow pattern. Update anti-patterns, key principles, and command quick reference to lead with --skeleton + --root. Full snapshot remains documented as fallback for simple apps. * fix: preserve skeleton drill-down anchors * fix: preserve drill-down refs across skeleton re-snapshots Skeleton snapshots now load the existing refmap and remove only skeleton-level refs (root_ref: None), preserving drill-down refs accumulated via --root. Previously, build() always created a fresh RefMap, breaking the skeleton → drill → act → skeleton(verify) workflow by wiping all drill-down refs on the verify step. * fix: preserve drill-down depth for root snapshots * fix: verify AX action effect on Electron elements before trusting success Chromium's AX implementation returns kAXErrorSuccess for AXPress, AXConfirm, etc. but only toggles ARIA state without firing DOM event handlers. This caused the click chain to short-circuit on false positives, preventing CGClick from ever being reached. The fix detects web elements via AXWebArea ancestor walk, then verifies each AX action actually had a DOM effect by comparing focused element pointers before and after. When all AX methods produce no real effect, CGClick fires as the genuine last resort. Native elements are unaffected — the original verified_press behavior is preserved in a separate code path. * chore: ignore .context/ for local tooling artifacts * style: collapse web_action_had_effect signature to single line * test: cover RefMap save oversize rejection Extract the serialize+size-check step into a private helper so the 1MB write rejection path can be unit-tested without filesystem I/O. Also moves the size check above the directory creation in save() so an oversized refmap no longer creates ~/.agent-desktop on rejection. * test: assert stale-ref suggestion mentions --skeleton Locks in the Phase 4 polish requirement that STALE_REF errors guide agents back to a skeleton refresh, not just a plain snapshot. * test: cover skeleton-to-drill-down counter continuity Asserts that skeleton refs (@e1..@e10) survive a scoped invalidation of @e3, that drill-down refs allocated after the skeleton continue from @e11 instead of resetting, and that remove_by_root_ref drops only the drill-down children. * test: cover --root + --surface rejection at execute() boundary Adds a NoopAdapter that uses every PlatformAdapter trait default to exercise execute()'s validation guards without standing up a real adapter. Verifies that combining --root with a non-Window surface returns INVALID_ARGS, and that the Window surface does not trigger the guard. * test: add filesystem-redirected integration tests for run_from_ref Adds a thread-local HOME override + RAII HomeGuard so RefMap::save and RefMap::load can be exercised against an isolated temp directory without env-var racing across parallel tests. Also adds a StubAdapter that implements PlatformAdapter with a canned subtree response so the full run_from_ref drill-down flow can be unit-tested without standing up macOS AX state. Closes seven Phase 3 plan acceptance items in one pass: - save+load roundtrip with HOME override - oversize save rejection preserves previous file on disk - run_from_ref returns subtree and persists drill refs - stale root ref → STALE_REF with skeleton suggestion - re-drill replaces drill refs only, counter continues - multiple drill-downs from @e1 and @e2 coexist - empty subtree drill-down produces no new refs * test: add golden fixtures for skeleton output and drill-down refmap Adds two committed JSON fixtures under tests/fixtures/ plus matching unit tests that build the same input trees, run them through allocate_refs / run_from_ref, and assert the produced ref ids, parent-child layout, and root_ref tagging match the golden expectations. Locks in the JSON shape so future serialization changes have to be deliberate. * fix: bound build_subtree recursion on Electron wrapper chains Anonymous AXGroup/AXGenericElement wrappers do not advance the semantic depth counter (web-wrapper depth-skip), so a long chain of nested wrappers could recurse arbitrarily deep without ever hitting either max_depth or ABSOLUTE_MAX_DEPTH. The previous code checked `depth >= ABSOLUTE_MAX_DEPTH` against the semantic depth, which stayed at 0 throughout a wrapper chain. On pathological Electron trees this risked stack exhaustion before any cap engaged. Add a separate `raw_depth` parameter that always increments and use it for the absolute cap. Semantic `depth` still drives `max_depth` and the skeleton boundary so the wrapper-flattening behavior is preserved on normal trees. Also drops the long-unused `_include_bounds` parameter; bounds filtering happens in `allocate_refs` in core, not here. * chore: add pre-commit hook running fmt + clippy + tests Mirrors the CI quality gates (cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test --lib --workspace) so a failing commit never reaches origin. Skips automatically when no Rust/TOML files are staged. Bypass with --no-verify or SKIP_PRECOMMIT=1 when a non-Rust hotfix is genuinely needed. Hook lives under .githooks/ and is opt-in per clone: git config core.hooksPath .githooks Setup instructions added to CLAUDE.md. * fix: prevent orphaned drill-down refs leaking across skeleton refresh remove_skeleton_refs() previously dropped every entry whose root_ref was None and kept every entry whose root_ref was Some. After a series of skeleton refreshes that pattern leaked drill-down refs whose root anchor had already been removed: they could never be cleaned up by a future remove_by_root_ref(target) because target referred to a non-existent anchor, and they accumulated until the 1MB write guard fired. Restructure the function to keep only: - skeleton anchors that have at least one drill-down pointing at them - drill-down refs whose root anchor is among the kept anchors Unreferenced anchors are still dropped (the original intent), and orphaned drill-downs are dropped at the same time so the refmap can no longer accumulate dead entries. Updated existing test to assert the new keep-when-referenced semantics and added a regression test for the orphan-drilldown leak path. * fix: align resolve traversal with snapshot child-attribute set find_element_recursive previously walked AXChildren first then fell back to AXContents only, and resolve_element_name only derived its fallback label from AXChildren. Snapshot building, however, uses the full child_attributes() list (AXChildren, AXContents, AXChildrenInNavigationOrder) via copy_children. That asymmetry meant refs minted for containers exposed only through AXChildrenInNavigationOrder appeared in the snapshot output but produced false STALE_REF errors on drill-down or action commands because the resolver could not find them again. Route both call sites through child_attributes() so resolution sees the same children as snapshotting. * fix: bypass AXConfirm on web elements and add skeleton anchors to drill-down * fix: compare AX elements via CFEqual in web_action_had_effect AXUIElementCopyAttributeValue follows the CoreFoundation Create rule and returns a freshly-allocated CF object on every call, so raw pointer equality (`before.0 != after.0`) between two separate copies of AXFocusedUIElement was ALWAYS true even when the focused element was unchanged. That made Step 1 (AXPress) of activate_web_element appear successful every time and left Steps 2-4 of the escalation chain (AXConfirm bypass, child actions, CGClick fallback) dead code on web elements. Replace the pointer comparison with CFEqual, which compares accessibility element identity rather than heap addresses. * refactor: unify allocate_refs across snapshot and drill-down paths allocate_refs in snapshot.rs and allocate_refs_with_root in snapshot_ref.rs were near-exact copies that differed only in whether each allocated ref carried a root_ref tag. The duplication meant any bug fix or behavior change in one path risked silently drifting from the other (bot flagged this after the first round of fixes added even more duplicated skeleton-anchor logic to allocate_refs_with_root). Move the shared logic into ref_alloc.rs behind a new RefAllocConfig struct with Option<&str> for the root_ref_id, and delete the snapshot_ref.rs copy. Both snapshot::build and snapshot_ref::run_from_ref now route through the same function with their respective configs. append_surface_refs and the existing unit tests in both modules are updated to use the shared function + config. * chore: drop with_root from drill test names after allocator unification The snapshot_ref tests kept their original test_allocate_refs_with_root_* names after the allocator dedupe even though allocate_refs_with_root no longer exists. Rename them to test_drill_alloc_* so grep for allocate_refs_with_root returns zero matches and nobody assumes a second allocator path exists. * docs: compound DRY ref-allocator dedupe into knowledge base Adds docs/solutions/best-practices/deduplicate-ref-allocator-via-config-struct-2026-04-14.md documenting the root cause, guidance, consequences, trigger conditions, and before/after of the allocate_refs / allocate_refs_with_root duplication that landed on PR #20 and was unified in d06a6c2. Also: - gitignore allows docs/solutions/** (was caught by docs/* rule) - CLAUDE.md workspace tree surfaces docs/solutions/ so fresh agents discover the knowledge store - docs/phases.md drops two stale DrillDownConfig references (lines 62 and 226) — the symbol no longer exists in the codebase * refactor: drop unused root_ref field from TreeOptions TreeOptions.root_ref was populated from SnapshotArgs.root_ref by tree_options() and then never read anywhere. The actual routing lives in execute(): `if let Some(ref root) = args.root_ref` branches to snapshot_ref::run_from_ref(adapter, &opts, root), passing the root id as an explicit argument — opts.root_ref was never consulted by any caller. Delete the field from TreeOptions and its Default impl and stop populating it in tree_options(). SnapshotArgs.root_ref remains as the single source of truth for the drill-down target. * fix: suppress skeleton flag when --root is set tree_options() clamped depth based on skeleton && root_ref.is_none() but still forwarded skeleton: args.skeleton unconditionally. When a caller passed --skeleton --root @e3, opts.skeleton stayed true, so adapter.get_subtree on macOS built a truncated skeleton tree instead of the full drill-down subtree, AND ref_alloc::allocate_refs tagged skeleton-anchor refs with the drill-down root_ref — both unintended. Tie the skeleton flag to the same root_ref.is_none() guard the depth clamp already uses. A drill-down is always a full subtree now, never a skeleton view. Adds test_tree_options_suppresses_skeleton_for_drill_down to lock the behavior in and extends the existing clamp test to assert the flag still propagates on non-drill paths. * fix: detect web action effect via value + selected + focus change The previous focus-change-only heuristic in web_action_had_effect was wrong for non-Chromium AXWebArea elements (Safari WKWebView, Mail, Notes): AXPress on a checkbox toggles AXValue correctly but does NOT shift the focused UI element. web_action_had_effect returned false, the chain escalated to CGClick, and CGClick toggled the checkbox a second time — net zero visible change. The pointer-equality bug that ae78cbc replaced was masking this by always returning true; CFEqual exposed the real signal gap. Capture element state into a PreActionState struct (focused, value, selected) before any AX action runs, and consider the action to have had an effect if ANY of those three fields differs afterward. Element value covers WKWebView checkboxes, text fields, sliders. Selected covers tabs, radio buttons, and list items. Focused still covers Chromium's DOM-driven focus shifts. Triggers on whichever signal is most relevant to the element type without needing role-specific code. Also re-exports copy_bool_attr from crates/macos/src/tree for use by the chain-steps module. * fix: emit skeleton boundary when drill path is about to hit raw cap In skeleton mode, a chain of anonymous web wrappers (AXGroup / AXGenericElement with empty title and value) never advances the semantic `depth` counter, so `child_depth > max_depth` never fires no matter how deep the chain runs. The recursion then hits the `raw_depth >= ABSOLUTE_MAX_DEPTH` top-of-function guard and silently returns None, dropping the subtree without any children_count marker. Agents see truncated output with no indication that anything was dropped. Extend the at_skeleton_boundary condition to also fire when child_raw_depth is about to hit ABSOLUTE_MAX_DEPTH, so skeleton mode always emits a visible truncation node for deep wrapper chains instead of disappearing them. The old `raw_depth < ABSOLUTE_MAX_DEPTH` half of the guard was redundant — the top-of-function check already ensures the current frame has raw budget; the meaningful question is whether the CHILD frames will. * perf: compute is_in_webarea once per verified-press call try_focus_then_verified_confirm_or_press called is_in_webarea(el) to gate AXConfirm, then fell through to do_verified_press which called is_in_webarea(el) again on its first line. Each call walks up to 20 AX parents via per-element IPC (AXUIElementCopyAttributeValue on AXParent), so every web-area click through this path paid the cost twice. Extract a private dispatch_verified_press(el, caps, in_web) that takes the web-area flag as an input. do_verified_press keeps its existing (el, caps) signature for chain_defs compatibility and computes is_in_webarea once, then delegates. The focus-then-confirm path also computes it once and passes it through. The result is identical on both paths with one AX traversal instead of two. * fix: skeleton anchors in drill-downs must not inherit root_ref; restore AXBrowser AXContents fallback Skeleton anchors discovered while processing a drill-down subtree were being tagged with the drill root_ref, making them indistinguishable from regular drill entries. re-drills would delete those anchors via remove_by_root_ref, breaking sub-drilling. They now always carry root_ref=None so they survive re-drills as stable targets. AXBrowser child resolution dropped the AXContents fallback when child_attributes() was unified — the resolver now uses ["AXColumns","AXContents"] matching the original traversal order. * fix: suppress skeleton anchor creation in drill-down mode to prevent orphaned ref accumulation * fix: bounds-based resolver pruning and CGClick fallback on chain timeout Resolver was doing exhaustive DFS through entire AX tree (depth 50) to find a single element. For large documents like a dense spreadsheet this meant visiting tens of thousands of cells via AX IPC, causing multi-minute hangs before the click chain even started. Two changes: - find_element_recursive now prunes subtrees whose spatial bounds do not contain the target element's centre point, and aborts the whole search after five seconds. Numbers table -> button not inside -> entire table skipped. Resolution went from >1m49s to <50ms. - execute_chain now sets a 1s app-level AX messaging timeout so calls to children/parents fail fast instead of blocking for the system default 6s, and when the 10s chain deadline fires it attempts the CGClick step before returning failure so the coordinate fallback is always reached. * refactor: resolve 9 code-review todos for progressive skeleton traversal - skeleton refresh now starts from RefMap::new() (removes stale ref accumulation across refreshes) - drill-down snapshot resolves real WindowInfo via PID lookup instead of synthetic empty id - --root flag validates ref format before RefMap lookup (returns INVALID_ARGS not STALE_REF) - ABSOLUTE_MAX_DEPTH truncation emits boundary node with children_count instead of silently dropping - fix ref vs ref_id field name in commands-observation.md JSON examples - fix phases.md TreeOptions listing root_ref (it lives in SnapshotArgs, not TreeOptions) - add batch snapshot skeleton/root examples to commands-system.md - split snapshot.rs and snapshot_ref.rs test modules into separate _tests.rs files (143/69 LOC) - add integration tests for skeleton + drill-down workflow including invalid --root validation * docs: compound progressive snapshot review hardening * docs: tighten progressive snapshot compound write-up * docs: update README with progressive skeleton traversal and fix command count to 50 * docs: correct command count to 53, add Notifications section and platform notes --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-04-16 01:46:31 -07:00
Lahfir	a19c1b5132	feat: add electron/web app compatibility for accessibility tree traversal ... Skip depth budget for non-semantic AXGroup/AXGenericElement wrappers with empty name and value, allowing default --max-depth 10 to find 100+ refs in Electron apps like Slack and VS Code (previously found only 3). Raise resolver search depth from 20 to ABSOLUTE_MAX_DEPTH (50) so ref resolution succeeds for deeply nested Electron elements. Fix surface detection to check if focused window itself is the target surface (Electron reports dialogs as focused window, not as children) and check both AXRole and AXSubrole for surface matching. Update Phase 2 (Windows) and Phase 3 (Linux) docs with equivalent web/Electron compatibility patterns for their respective APIs.	2026-02-28 19:03:11 -08:00
Lahfir	8aa672c668	refactor: resolve code review findings from notification PR ... - fix(core): wait --notification uses index-diff detection instead of count, passes text filter, checks deadline before sleep - fix(core): dismiss-all-notifications uses single NC session with batch adapter method and reports individual failures - refactor(macos): extract dismiss_entry helper to eliminate duplicate dismiss logic between single and batch operations - refactor(macos): restrict nc_session visibility to pub(crate), use absolute paths for pgrep/osascript, add bounded wait with kill fallback - refactor(macos): single-pass is_notification_group check using matches! macro - refactor: extract notification CLI args and dispatch into separate files to keep cli_args.rs and dispatch.rs under 400 LOC - refactor: rename DismissAllNotificationsArgs to DismissAllNotificationsCliArgs for consistency - fix(core): remove unused timestamp field from NotificationInfo - fix: remove poll_interval_ms from wait command (hardcode 500ms) - docs: update phases.md with completed notification status and cross-platform reference notes	2026-02-27 12:42:46 -08:00
Lahfir	844e964b82	docs: update phases with notification and tray trait methods	2026-02-27 11:57:18 -08:00
Lahfir	03268ce18e	docs: add comprehensive phase roadmap	2026-02-25 01:50:57 -08:00
Lahfir	50335f4576	chore: remove internal docs from tracking and fix gitignore	2026-02-24 06:24:26 -08:00
Lahfir	a73fe2e6c1	docs: add architecture diagram to README	2026-02-24 06:22:52 -08:00
Lahfir	ad91cd32cf	feat: add Claude Code skills for agent-desktop automation ... Graph of skills covering all 50 commands with platform-specific sub-skills: - Main index skill with core concepts, ref system, JSON contract - Observation, interaction, and system command references - Common workflow patterns (forms, menus, dialogs, drag-drop) - macOS platform skill (permissions, AX API, troubleshooting) - /desktop slash command for quick reference - PRD addendum for skill maintenance across phases	2026-02-23 01:51:31 -08:00
Lahfir	b2c34f6ec5	Revert "docs: add platform crate folder structure convention and track plans" ... This reverts commit 775892ee8e.	2026-02-20 23:31:50 -08:00
Lahfir	775892ee8e	docs: add platform crate folder structure convention and track plans ... Add Platform Crate Folder Structure section to CLAUDE.md enforcing consistent subfolder layout (tree/, actions/, input/, system/) across all platform crates. Update smart AX-first click chain plan with new file paths reflecting the subfolder structure. Un-ignore docs/plans/ and docs/brainstorms/ so design documents are tracked.	2026-02-20 23:26:56 -08:00
Lahfir	3a796023f0	style: apply cargo fmt to all files	2026-02-19 20:00:18 -08:00
Lahfir	29d6cbaac4	docs: check off all Phase 1 Must-Fix agent-native items	2026-02-19 10:46:04 -08:00
Lahfir	a346f242c2	feat: Phase 1 foundation — workspace scaffold, core engine, macOS adapter, 31 commands ... Implements the complete agent-desktop Phase 1 specification: - Workspace: 5-crate layout (core, macos, windows/linux stubs, binary) - Core: AccessibilityNode, Action, ErrorCode, PlatformAdapter trait, RefMap with atomic writes, SnapshotEngine with depth-first ref allocation - macOS adapter: AXUIElement tree traversal, action execution, input synthesis via CGEvent, screenshot via CGWindowListCreateImage, clipboard via pbpaste/pbcopy, window listing and app management via osascript - 31 CLI subcommands via clap derive: snapshot, find, screenshot, get, is, click, double-click, right-click, type, set-value, focus, select, toggle, expand, collapse, scroll, press, launch, close-app, list-windows, list-apps, focus-window, clipboard-get/set, wait, status, permissions, version, batch - Windows/Linux: not-supported stubs ready for Phase 2 implementation - JSON output contract: {version,ok,command,data} envelope with structured error payloads including SCREAMING_SNAKE_CASE error codes - Ref system: @e{N} sequential refs for interactive elements, stored at ~/.agent-desktop/last_refmap.json with 0o600/0o700 permissions - CI: GitHub Actions macOS runner with dependency isolation check, clippy, unit tests, release build, and 15MB binary size gate	2026-02-19 10:44:38 -08:00