Creepso/agent-desktop
1
0
Fork 0
mirror of https://github.com/lahfir/agent-desktop.git synced 2026-08-20 05:57:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

docs: record in phases.md what 2.2 changed and what it hands over

Browse source 
Two gaps, one of my own making.

The client construction correction stopped being true the moment the bounded
client landed. phases.md still said 'construct with UIAutomation::new_direct()'
at the 2.2 key APIs and in the API mapping table, while the shipped code
builds from CUIAutomation8 - so the document contradicted the code, which is
the exact failure U9 exists to prevent. Both now state the construction that
ships and why: new_direct() builds CUIAutomation, which has no IUIAutomation2
and therefore no call timeout, and A14-11 measured that as a 59 s block where
the bounded client returns UIA_E_TIMEOUT in 2.05 s.

The deferrals were recorded only in the ledger and the PR, not in the
sub-phase that has to act on them. 2.4 owns the snapshot path, so 2.4's scope
now carries what 2.2 could not resolve: a target that dies mid-walk is
invisible on both the sibling axis (A14-4) and the property axis (A14-9), so a
subtree whose provider died reports complete with fewer siblings than exist,
and only descent surfaces it. A snapshot that must tell 'no more children'
from 'the app went away' needs an independent liveness check. The hang bound
is the client's connection timeout, never the operation deadline.

The hunk index moves with them, 46 to 47.
This commit is contained in:
Lahfir 2026-07-28 18:52:02 -06:00
parent eb32b03f99
commit f4a505e5f6
2 changed files with 28 additions and 26 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
docs/phases.md
View file

@ -996,7 +996,7 @@ Every sub-phase 2.0–2.15 below is held to the same definition of done, stated
- `CacheRequest` attribute reads (the UIA analogue of `AXUIElementCopyMultipleAttributeValues`), batched conditionally rather than always. A6-1 measured a 220-node Explorer window at 2.69x overall with the find phase 1.5x *slower* and the read phase ~298x faster, and A6-2 measured classic Notepad — 3 nodes, served by `UIAutomationClientsideProviders` inside the client process, so an uncached read costs no cross-process RPC and the request adds pure setup — as a net pessimization at 0.5763x overall and 0.436x on the find phase. The rule that ships is the one the Windows API Mapping table below already carries: cache only the properties that will be read, and skip caching when a root-level `ProviderDescription` read indicates an in-process client-side provider. There is deliberately no node-count arm — the node count is unknown at the moment the cache request is built
- Committed probe examples: raw UIA dumps of Notepad and Explorer, checked in as evidence alongside the sub-phase plan
**Key APIs:** `IUIAutomation.ElementFromHandle()`, `IUIAutomationTreeWalker.GetFirstChild`/`GetNextSibling`, `CacheRequest` (`uiautomation` crate 0.25+ wrapping the `windows` crate's COM bindings; construct the client with `UIAutomation::new_direct()` — verified against the `uiautomation` 0.25.0 source, `UIAutomation::new()` calls `CoInitializeEx(None, COINIT_MULTITHREADED)` itself and proceeds on any non-negative HRESULT, so on a thread already in the MTA it reads `S_FALSE` as success and permanently leaks one initialization count, the type having no `Drop` and the crate never calling `CoUninitialize` — a leak Phase 5's long-lived daemon accumulates — while on any STA host thread it returns `Err(RPC_E_CHANGED_MODE)` outright)
**Key APIs:** `IUIAutomation.ElementFromHandle()`, `IUIAutomationTreeWalker.GetFirstChild`/`GetNextSibling`, `CacheRequest` (`uiautomation` crate 0.25+ wrapping the `windows` crate's COM bindings; construct the client by direct `CoCreateInstance` on **`CUIAutomation8`**, never through `UIAutomation::new()`. `new()` calls `CoInitializeEx(None, COINIT_MULTITHREADED)` itself and proceeds on any non-negative HRESULT, so on a thread already in the MTA it reads `S_FALSE` as success and permanently leaks one initialization count — the type has no `Drop` and the crate never calls `CoUninitialize` — a leak Phase 5's long-lived daemon accumulates, and on any STA host thread it returns `Err(RPC_E_CHANGED_MODE)` outright. `UIAutomation::new_direct()` avoids both, but it is `CoCreateInstance(&CUIAutomation, …)`, and that object returns `E_NOINTERFACE` for `IUIAutomation2` — so its calls carry no timeout and `ElementFromHandle` against a window that stopped dispatching blocks indefinitely (A14-11, A14-12: 59.09 s through a 30 s watchdog, against `UIA_E_TIMEOUT` in 2.05 s once `SetConnectionTimeout` is set). `CUIAutomation8` is the same direct `CoCreateInstance` with the same no-`CoInitializeEx` property, and it exposes the timeout; there is no fallback to the unbounded client)
**Depends on:** 2.1
@ -1036,6 +1036,7 @@ Every sub-phase 2.0–2.15 below is held to the same definition of done, stated
- **Resolver depth:** element re-identification searches to `ABSOLUTE_MAX_DEPTH` (50) — Electron elements commonly sit at depth 25+; implement in `crates/windows/src/tree/resolve.rs`
- **Surface detection for Electron:** an Electron modal (file picker, dialog) may report as the focused window itself rather than a child; surface detection must check whether the focused window IS the target surface, checking both `ControlType` and `LocalizedControlType`/UIA patterns (analogous to macOS AXRole + AXSubrole); implement in `crates/windows/src/tree/surfaces.rs`
- Progressive skeleton traversal (`--skeleton`, `--root`) needs no Windows-specific work beyond `get_subtree()` — core owns the flow
- **What 2.2 hands over unresolved, and 2.4 must not assume away.** A target that dies part-way through a walk is **not detectable from the tree API**: A14-4 measured `get_next_sibling` returning the exact `code()`/`result()` pair a live provider returns at end-of-list, and A14-9 measured property reads answering locally with an empty string and no error, so a subtree whose provider died mid-list reports *complete* with fewer siblings than exist. Descent (`get_first_child`) does surface it as `E_FAIL`. A snapshot that must distinguish "this app has no more children" from "this app went away" needs an independent liveness check — re-resolving the root, or a process read — not the sibling terminator. The hang bound is the client's `ConnectionTimeout`, not the operation `Deadline`, which cannot interrupt a blocking `SendMessage` (A14-11)
**Key APIs:** `ControlViewWalker`, `FindAll`, `UICacheRequest`, `Chrome_WidgetWin_1` class match, `IVirtualDesktopManager` (read-only, for diagnostics)
@ -1102,7 +1103,7 @@ Every sub-phase 2.0–2.15 below is held to the same definition of done, stated
| Capability | Technology | Details |
|------------|-----------|---------|
| Tree root | `IUIAutomation.ElementFromHandle()` | Via `uiautomation` crate (v0.25+) wrapping UIA COM APIs via `windows` crate. Construct with `UIAutomation::new_direct()`, never `UIAutomation::new()` — verified against `uiautomation` 0.25.0 source, the latter initializes the COM library itself and proceeds on any non-negative HRESULT, so it takes `S_FALSE` as success on an MTA thread and leaks one initialization count per call (no `Drop`, no `CoUninitialize`), and returns `Err(RPC_E_CHANGED_MODE)` on an STA host thread |
| Tree root | `IUIAutomation.ElementFromHandle()` | Via `uiautomation` crate (v0.25+) wrapping UIA COM APIs via `windows` crate. Construct by direct `CoCreateInstance` on `CUIAutomation8`, never `UIAutomation::new()` — the latter initializes COM itself, takes `S_FALSE` as success on an MTA thread and leaks one initialization count per call, and returns `Err(RPC_E_CHANGED_MODE)` on an STA host thread. `UIAutomation::new_direct()` avoids that but builds `CUIAutomation`, which has no `IUIAutomation2` and therefore no call timeout (A14-12); `CUIAutomation8` sets `ConnectionTimeout`, which is what stops a non-dispatching target hanging the caller |
| Children | `IUIAutomationTreeWalker.GetFirstChild` / `GetNextSibling` | With `CacheRequest` for batch attribute retrieval. The speedup is a phase split, not one multiplier: measured on UIA3 COM over a 220-node Explorer window reading 8 properties, building the cache makes the find pass *slower* (180 ms vs 117 ms) and the property-read pass ~300x faster (372 ms vs 1.2 ms), netting ~2.7x for a single full-tree read. Cache only the properties that will be read, and expect the win from repeated reads over a cached tree rather than from the walk. On plain Win32 trees served by in-process client-side providers, unconditional caching is a net pessimization |
| Role mapping | `UIA ControlType` integers | Map to unified role enum in `tree/roles.rs` — e.g. `UIA_ButtonControlTypeId` → `button` |
| Click | `InvokePattern.Invoke()` | Pattern-based; coordinate click via SendInput only under explicit physical policy |

49
probes/windows/FINDINGS.md
View file

@ -256,32 +256,33 @@ The five `CONTRADICTS` rows are A1-5, A6-1, A8-3, A11-1 and A11-3; each appears
| H18 | `@@ -981 +984 @@` 2.1 exit criteria: package scope, adapter-backed commands and portable private-file assertions, with no runner or RDP measurement left in the gate | C-12, R6-3, R6-5, A11-2, A11-4, C-11 |
| H19 | `@@ -996 +999 @@` 2.2 key APIs, `uiautomation` 0.25+ constructed with `new_direct()`, and the reason for it corrected | R6-11, C-2 (research: the crate's own two constructor doc strings, then the 0.25.0 source itself - `new()` proceeds on any non-negative HRESULT, so it reads `S_FALSE` as success on an MTA thread and leaks one initialization count, and fails `RPC_E_CHANGED_MODE` on an STA host thread) |
| H20 | `@@ -1032 +1035 @@` 2.4 Chromium detection requires a settle before judging thinness | A1-5, C-4 |
| H21 | `@@ -1102,2 +1105,2 @@` API mapping table, tree-root pin plus `new_direct()` and its corrected reason, and the `CacheRequest` phase split | R6-11, A6-1, A6-2, C-2 (research: `uiautomation` 0.25.0 source, the same correction H19 carries - the table stated the disproved reason in a second place) |
| H22 | `@@ -1211 +1214,5 @@` 2.12 registers the self-hosted interactive runner, hardens that registration for a public repository, owns the deferred-row closure, and records the measured hosted-runner session without weakening the justification | A10-2, A14-1 (research: GitHub's own guidance against self-hosted runners on public repositories, and Microsoft's contrary Azure Pipelines UI-testing guidance) |
| H23 | `@@ -1217 +1224 @@` 2.12 exit criteria gain the runner, its written hardening policy and the RDP measurement | A10-2 |
| H24 | `@@ -1219 +1226 @@` 2.12 estimate covers runner registration alongside the fixture and harness | A10-2 |
| H25 | `@@ -1227 +1234 @@` npm postinstall gains `win32-arm64` | C-6 |
| H26 | `@@ -1240 +1247 @@` 2.14 title loses the stretch qualifier | A10-2 |
| H27 | `@@ -1244 +1251 @@` 2.14 ships before the 2.15 merge | A10-2 |
| H28 | `@@ -1261 +1268 @@` tray command table, overflow flyout class | C-5 |
| H29 | `@@ -1280 +1287 @@` 2.14 list-items bullet, overflow flyout class | C-5 |
| H30 | `@@ -1298 +1305 @@` 2.15 depends on all of 2.0 through 2.14 | A10-2 |
| H31 | `@@ -1306,2 +1313,2 @@` OS floors, Windows 10 servicing reality and the WGC feature floors | C-7, C-8, A10-5 |
| H32 | `@@ -1316,3 +1323,3 @@` new-dependency pin table | R6-11, C-1, C-2, C-3 |
| H33 | `@@ -1322 +1329 @@` pin re-verification note | R6-11 |
| H34 | `@@ -1335,3 +1342,3 @@` `Cargo.toml` snippet: `uiautomation = "0.25"`, the five `windows` features 2.1's own scope needs, and `windows-capture = "2.0.0"` | R6-11, C-1, C-2, C-3 (research: all five feature gates verified present in `windows` 0.62.2 and `windows-sys` 0.61.2 on docs.rs) |
| H35 | `@@ -1389 +1396 @@` docs checklist, Windows permissions and Chromium 138 | C-4, A1-5 |
| H36 | `@@ -1716 +1723 @@` 3.14 title loses the stretch qualifier | A10-2 |
| H37 | `@@ -1720 +1727 @@` 3.14 ships before the 3.15 merge | A10-2 |
| H38 | `@@ -1768 +1775 @@` 3.15 depends on all of 3.0 through 3.14 | A10-2 |
| H39 | `@@ -2476,3 +2483,4 @@` CI evolution table, v0.6.0 row plus the Phase 2 and 3 rows, with runner registration named at 2.12 | C-14, R6-1, R6-6, A10-2 |
| H40 | `@@ -2482 +2490 @@` runner enforcement: package scope, and clippy, core isolation and the size cap stated as macOS-only today | C-12, C-13, R6-3, R6-6, R6-7 |
| H41 | `@@ -2493,3 +2501,3 @@` new-dependencies summary table | R6-11, C-1, C-2, C-3 |
| H42 | `@@ -2536 +2544 @@` R2 risk row, settle before judging a Chromium tree thin | A1-5, C-4 |
| H43 | `@@ -2546,2 +2554,2 @@` R12 risk row, the `tscon` workaround is owed by the sub-phase that registers the runner, and R13, handler removal cost is window churn rather than the barrier | A10-2, A8-3 |
| H21 | `@@ -1102,2 +1106,2 @@` API mapping table, tree-root pin plus `new_direct()` and its corrected reason, and the `CacheRequest` phase split | R6-11, A6-1, A6-2, C-2 (research: `uiautomation` 0.25.0 source, the same correction H19 carries - the table stated the disproved reason in a second place) |
| H22 | `@@ -1211 +1215,5 @@` 2.12 registers the self-hosted interactive runner, hardens that registration for a public repository, owns the deferred-row closure, and records the measured hosted-runner session without weakening the justification | A10-2, A14-1 (research: GitHub's own guidance against self-hosted runners on public repositories, and Microsoft's contrary Azure Pipelines UI-testing guidance) |
| H23 | `@@ -1217 +1225 @@` 2.12 exit criteria gain the runner, its written hardening policy and the RDP measurement | A10-2 |
| H24 | `@@ -1219 +1227 @@` 2.12 estimate covers runner registration alongside the fixture and harness | A10-2 |
| H25 | `@@ -1227 +1235 @@` npm postinstall gains `win32-arm64` | C-6 |
| H26 | `@@ -1240 +1248 @@` 2.14 title loses the stretch qualifier | A10-2 |
| H27 | `@@ -1244 +1252 @@` 2.14 ships before the 2.15 merge | A10-2 |
| H28 | `@@ -1261 +1269 @@` tray command table, overflow flyout class | C-5 |
| H29 | `@@ -1280 +1288 @@` 2.14 list-items bullet, overflow flyout class | C-5 |
| H30 | `@@ -1298 +1306 @@` 2.15 depends on all of 2.0 through 2.14 | A10-2 |
| H31 | `@@ -1306,2 +1314,2 @@` OS floors, Windows 10 servicing reality and the WGC feature floors | C-7, C-8, A10-5 |
| H32 | `@@ -1316,3 +1324,3 @@` new-dependency pin table | R6-11, C-1, C-2, C-3 |
| H33 | `@@ -1322 +1330 @@` pin re-verification note | R6-11 |
| H34 | `@@ -1335,3 +1343,3 @@` `Cargo.toml` snippet: `uiautomation = "0.25"`, the five `windows` features 2.1's own scope needs, and `windows-capture = "2.0.0"` | R6-11, C-1, C-2, C-3 (research: all five feature gates verified present in `windows` 0.62.2 and `windows-sys` 0.61.2 on docs.rs) |
| H35 | `@@ -1389 +1397 @@` docs checklist, Windows permissions and Chromium 138 | C-4, A1-5 |
| H36 | `@@ -1716 +1724 @@` 3.14 title loses the stretch qualifier | A10-2 |
| H37 | `@@ -1720 +1728 @@` 3.14 ships before the 3.15 merge | A10-2 |
| H38 | `@@ -1768 +1776 @@` 3.15 depends on all of 3.0 through 3.14 | A10-2 |
| H39 | `@@ -2476,3 +2484,4 @@` CI evolution table, v0.6.0 row plus the Phase 2 and 3 rows, with runner registration named at 2.12 | C-14, R6-1, R6-6, A10-2 |
| H40 | `@@ -2482 +2491 @@` runner enforcement: package scope, and clippy, core isolation and the size cap stated as macOS-only today | C-12, C-13, R6-3, R6-6, R6-7 |
| H41 | `@@ -2493,3 +2502,3 @@` new-dependencies summary table | R6-11, C-1, C-2, C-3 |
| H42 | `@@ -2536 +2545 @@` R2 risk row, settle before judging a Chromium tree thin | A1-5, C-4 |
| H43 | `@@ -2546,2 +2555,2 @@` R12 risk row, the `tscon` workaround is owed by the sub-phase that registers the runner, and R13, handler removal cost is window churn rather than the barrier | A10-2, A8-3 |
| H44 | `@@ -992,2 +995,2 @@` 2.2 scope: the cycle-guard rationale, and `CacheRequest` batching stated as conditional with no node-count arm | A6-1, A6-2 (research: 2.0 measured nothing about UIA element-identity reuse and neither 2.0 walker calls `GetParent`, so the imported macOS pointer-reuse rationale had no backing) |
| H45 | `@@ -1000 +1003 @@` 2.2 exit criteria: the dump binary batches by provider class, and the cache unit test asserts equality rather than speed | A6-1, A6-2 |
| H46 | `@@ -1041 +1044 @@` 2.4 exit criteria restated rule-shaped, with named apps and fixed ref thresholds removed and an explicit skip-with-reason | A6-2, A2-4, A1-1, C-9, C-11 (research: the probe corpus's own KTD7 scope rule, which keeps an `app/provider` row from travelling) |
| H46 | `@@ -1041 +1045 @@` 2.4 exit criteria restated rule-shaped, with named apps and fixed ref thresholds removed and an explicit skip-with-reason | A6-2, A2-4, A1-1, C-9, C-11 (research: the probe corpus's own KTD7 scope rule, which keeps an `app/provider` row from travelling) |
| H48 | `@@ -1035,0 +1039 @@` 2.4 scope records what 2.2 hands over unresolved: mid-walk process death is invisible on both the sibling and property axes, and the hang bound is the client's connection timeout rather than the operation deadline | A14-4, A14-9, A14-11 |
## Completeness self-check