From aa94260594a09b4363dff61cd51169115d844e35 Mon Sep 17 00:00:00 2001 From: Lahfir Date: Fri, 17 Apr 2026 05:11:56 -0700 Subject: [PATCH] docs(phases): distribute gap-analysis findings into phases 2-5 as concrete tasks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the freestanding "Gap Analysis — 2026-04-17 Research" appendix with real objectives + API-specific implementation tables inside the phases where each item actually ships. Every cross-platform primitive now has matching rows on macOS, Windows, and Linux so the three adapters stay in lockstep. Phase 2 — Windows Adapter + Cross-Platform Feature Parity + P2-O8 AccessibilityNode stable-selector fields (identifier / subrole / role_description / placeholder / dom_id / dom_classes) populated by UIA AutomationId on Windows, kAXIdentifier / kAXSubrole / kAXDOMIdentifier on macOS backfill, AT-SPI accessible-id in P3. + P2-O9 Action enum expansion: LongPress, ForceClick, ShowMenu, FileDrop, WindowRaise, Cancel, SelectRange, InsertAtCaret, Watch. + P2-O10 ErrorCode expansion: PermissionRevoked, ResourceExhausted, AxMessagingTimeout, AutomationPermissionDenied. + P2-O11 watch_element trait method — AXObserver + CFRunLoopSource (macOS), IUIAutomation.AddAutomationEventHandler (Windows). Replaces polling in system/wait.rs. + P2-O12 Text range primitives — AXSelectedTextRange / AXStringForRange (macOS), TextPattern.GetSelection / TextRange.Select (Windows). New `text` subcommand tree. + P2-O13 Modern per-window screenshot — SCScreenshotManager (macOS) / Windows.Graphics.Capture (Windows). ~10x faster than subprocess. + P2-O14 Toolbar + Spotlight + Dock + MenuBarExtras surfaces. + P2-O15 Electron / WebView2 AXEnhancedUserInterface + HtmlId toggles. + P2-O16 FFI parity — ad_snapshot, ad_execute_by_ref, ad_wait, ad_version, ad_abi_version, ad_set_log_callback, plus wrappers for every new trait method. + P2-O17 Screen Recording + Automation permission detection (macOS). + New "Cross-Platform Trait Extensions" section with the actual Rust signatures and supporting types. + New "Cross-platform capability map" matrix tying every capability to concrete Apple / Microsoft / freedesktop APIs. + Testing section grows to cover P2-O8 through O17 plus FFI parity harnesses. + Dependencies add objc2, screencapturekit, windows crate features for Graphics_Capture + Direct3D11. Phase 3 — Linux Adapter + Cross-Platform Parity Completion + P3-O8 AT-SPI accessible-id populates AccessibilityNode.identifier. + P3-O9 watch_element via zbus signal subscription on org.a11y.atspi.Event.Object.PropertyChange / ChildrenChanged / StateChanged / Window events. + P3-O10 org.a11y.atspi.Text + EditableText for text range primitives. + P3-O11 PipeWire org.freedesktop.portal.ScreenCast for modern screenshot. + P3-O12 Toolbar via AT-SPI Role::ToolBar; per-DE panel surfaces. + P3-O13 Action variants — LongPress via xdotool/ydotool, ShowMenu via Action.DoAction("popup"), FileDrop via XDND / FileTransfer portal, ForceClick returns ActionNotSupported (no Linux equivalent). + P3-O14 FFI cdylib continues; each new trait method adds its Linux impl. + P3-O15 Flatpak/Snap AT-SPI bus permission documentation. + Testing expands to cover all extensions. Phase 4 — MCP Server Mode + P4-O5 Ref-based tool shape ({ref: "e5"}) matching Playwright-MCP idiom. + P4-O6 MCP Resource types: agent-desktop://refmap/current, snapshot/latest, permissions/current, events/stream, audit/{id}. + P4-O7 watch_element events stream via notifications/message. + P4-O8 notifications/progress on wait + drill-down chains. + P4-O9 Tool-level permission tiers (observation / interactive / destructive) negotiated at initialize. + P4-O10 Session-scoped RefMap — no leakage across MCP sessions. + P4-O11 Tri-state permission matrix in initialize response. + P4-O12 Streamable HTTP transport alongside stdio + SSE. + MCP Tool Surface table split into observation / interactive / destructive tiers matching the capability gates. + Framework Integration table with ready-to-use configs for Claude Desktop, Cursor, VS Code Copilot, Gemini CLI, Microsoft Agent Framework 1.0, Zed, Continue.dev — each with a 60-second acceptance demo. + Tests expanded to host-specific acceptance + capability negotiation + event streaming. Phase 5 — Production Hardening + P5-O5 Safety trio (--dry-run / --confirm / append-only audit log) — hash-chained audit file maps to EU AI Act Art. 14 + OWASP Agentic Top-10 AA-02 / AA-06. + P5-O6 Policy engine — ~/.agent-desktop/policy.yaml with allow/deny/ require-confirm/dry-run-only decisions, SIGHUP reload, audit entries per evaluation. + P5-O7 OCR / vision fallback — find --visual routes to macOS Vision, Windows MediaOCR, Linux Tesseract; synthetic @vN refs. + P5-O8 Session trace + OpenTelemetry — every command writes OTLP- compliant spans; trace view / trace export --otlp --har commands. + P5-O9 Screencast receipt via --record-trace at 2 Hz using Phase 2 modern screenshot APIs. + P5-O10 Sigstore attestation verification at install time — brew / winget / scoop manifests run gh attestation verify before extracting. + New "Safety Trio", "Policy Engine", "OCR / Vision Fallback", "Session Trace + OpenTelemetry" sections with concrete code snippets. + Package Manager Distribution table gains a Signing column and a crates.io row. + Enterprise Quality Gates grows to include Safety, FFI stability, and updated Observability / Reliability rows. + Testing grows to cover safety trio, OCR fallback, session trace, and install-time Sigstore verification. The Gap Analysis appendix is removed — every item now lives in its correct phase as a first-class objective with concrete APIs, file paths, and testable metrics. File: 1538 → 1787 lines (+249). 10 top-level sections (one fewer than before since the appendix is gone). Objective counts: P2=17, P3=15, P4=12, P5=10. --- docs/phases.md | 575 +++++++++++++++++++++++++++++++++++-------------- 1 file changed, 412 insertions(+), 163 deletions(-) diff --git a/docs/phases.md b/docs/phases.md index 7e23330..fc0e782 100644 --- a/docs/phases.md +++ b/docs/phases.md @@ -538,23 +538,105 @@ These items are tracked under **Gap Analysis — 2026-04-17 Research** at the bo --- -## Phase 2 — Windows Adapter +## Phase 2 — Windows Adapter + Cross-Platform Feature Parity **Status: Planned** -Phase 2 brings agent-desktop to Windows. Core engine, CLI parser, JSON contract, error types, snapshot engine, and command registry are untouched. Only the new `WindowsAdapter` implementation is added inside `crates/windows/`. The existing stub is replaced with a full implementation. +Phase 2 brings agent-desktop to Windows. It is also the phase that closes the cross-platform feature-parity gaps surfaced after the v0.1.13 FFI ship — shipping Windows meaningfully requires new core abstractions (stable identifiers, event subscriptions, text-range primitives) that Windows UIA exposes natively and the macOS adapter currently does not surface. Every new trait method added here is implemented on both platforms in the same PR pair: Windows ships the native version, macOS backfills using the equivalent AX API. Linux (Phase 3) mirrors both against AT-SPI2. + +Core engine, CLI parser, JSON contract invariants, and command-registration pattern are preserved. What Phase 2 legitimately changes: `AccessibilityNode` field set, `Action` enum variants, `ErrorCode` variants, `PlatformAdapter` trait size. Every change is additive (`#[non_exhaustive]` already guards the enums) and every macOS backfill lands atomically with the Windows implementation so the two platforms never drift. ### Objectives +Core + Windows parity (original scope): + | ID | Objective | Metric | |----|-----------|--------| | P2-O1 | Windows adapter | `snapshot` on Windows returns valid tree for Explorer, Notepad, Settings | | P2-O2 | All existing commands cross-platform | Identical JSON schema output on macOS and Windows for every command | | P2-O3 | Windows input synthesis | `click`, `type`, `press`, all mouse commands working via UIA + SendInput | -| P2-O4 | Windows screenshot | `screenshot` produces PNG via BitBlt / PrintWindow or `xcap` crate | +| P2-O4 | Windows screenshot | `screenshot` produces PNG via `Windows.Graphics.Capture` API | | P2-O5 | Windows clipboard | `clipboard-get` / `clipboard-set` / `clipboard-clear` working via Win32 Clipboard API | | P2-O6 | Windows CI | GitHub Actions Windows runner executes full test suite on every PR | -| P2-O7 | Windows binary release | Prebuilt `.exe` published via GitHub Releases and npm | +| P2-O7 | Windows binary release | Prebuilt `.exe` published via GitHub Releases and npm; Phase 1.5 FFI cdylib for Windows already ships | + +Cross-platform core extensions (new, landed alongside Windows): + +| ID | Objective | Metric | +|----|-----------|--------| +| P2-O8 | `AccessibilityNode` stable-selector fields | Every node carries `identifier`, `subrole`, `role_description`, `placeholder`, `dom_id`, `dom_classes` (all `Option` / `Vec` with `skip_serializing_if`). Populated by Windows UIA `AutomationId` / `LocalizedControlType` / `HelpText`, by macOS `kAXIdentifierAttribute` / `kAXSubroleAttribute` / `kAXRoleDescriptionAttribute` / `kAXPlaceholderValueAttribute` / `kAXDOMIdentifierAttribute` / `kAXDOMClassListAttribute`. Snapshot regression fixtures show stable IDs across re-drills | +| P2-O9 | `Action` enum expansion for 2026 agent workloads | New variants: `LongPress { duration_ms }`, `ForceClick`, `ShowMenu`, `FileDrop(Vec)`, `WindowRaise`, `Cancel`, `SelectRange { start, len }`, `InsertAtCaret(String)`, `Watch(WatchSpec)`. Each has a macOS AX API mapping, a Windows UIA pattern mapping, and a new CLI subcommand. The `#[non_exhaustive]` attribute keeps this SemVer-safe | +| P2-O10 | `ErrorCode` expansion | Add `PermissionRevoked` (distinct from `PermDenied` — TCC yanked mid-session), `ResourceExhausted` (refmap >1 MB, tree node-count cap), `AxMessagingTimeout` (AX-specific timeout separate from orchestration `Timeout`), `AutomationPermissionDenied` (macOS `osascript` grant). Tri-state permission probe at startup distinguishes "never granted" from "revoked" | +| P2-O11 | Event-subscription primitive (push, not poll) | New trait method `watch_element(handle, events: &[EventKind], timeout_ms: u64) -> Result>`. macOS: `AXObserverCreate` + `AXObserverAddNotification` + `CFRunLoopSource` (no more polling in `system/wait.rs`). Windows: `IUIAutomation.AddAutomationEventHandler` + `AddFocusChangedEventHandler` + `AddPropertyChangedEventHandler`. New `wait --event value-changed --ref @e5 --timeout 3000` CLI flag. Linux mirrors in Phase 3 via AT-SPI2 D-Bus signals | +| P2-O12 | Text range primitives | Read caret, read selection, select a range by offsets, read text at range, insert at caret. macOS: `kAXSelectedTextRangeAttribute` (settable), `AXStringForRangeParameterizedAttribute`, `AXBoundsForRangeParameterizedAttribute`, `AXRangeForLineParameterizedAttribute`, `AXValueCreate(kAXValueCFRangeType, …)`. Windows: `TextPattern.GetSelection`, `TextPattern.DocumentRange`, `TextRange.Select`, `TextRange.Move`, `TextRange.GetText`, `TextRange.GetBoundingRectangles`. Commands: `text get-selection`, `text select-range `, `text insert-at-caret `, `text at-offset ` | +| P2-O13 | Modern per-window screenshot APIs | macOS: replace `/usr/sbin/screencapture` subprocess with `SCScreenshotManager.captureImage(contentFilter:config:)` filtered to a specific `CGWindowID` from `SCShareableContent.windows`. Windows: `Windows.Graphics.Capture` via `GraphicsCaptureItem.CreateFromWindowHandle(HWND)` + `Direct3D11CaptureFramePool`. No subprocess, honours platform capture permission, ~10× faster for per-window captures | +| P2-O14 | Toolbar and missing surfaces | Both platforms add `SnapshotSurface::Toolbar`. macOS additionally adds `Spotlight` (pid of `/System/Library/CoreServices/Spotlight.app`), `Dock` (pid of `/System/Library/CoreServices/Dock.app`), and `MenuBarExtras` (enumerates `SystemUIServer`, `ControlCenter`, and per-app `AXExtrasMenuBar`). Windows adds `SystemTray` (as structured surface, not just tray commands) | +| P2-O15 | Electron / WebView2 deep-tree toggles | macOS: `build_subtree` writes `AXEnhancedUserInterface = YES` on app root for known Electron bundle IDs (VS Code, Cursor, Slack post-Sept-2024, Teams, Discord, Figma Desktop, Notion). Windows: detect Edge WebView2 via UIA `ClassName = "Chrome_WidgetWin_1"` and the equivalent flag; apply same web-wrapper depth-skip. Both: new `--force-electron-a11y` CLI override | +| P2-O16 | FFI parity expansion | Every new trait method from P2-O11 / O12 / O13 / O14 gets a matching `ad_*` extern "C" entry point shipped in the same PR as the Rust method. Backfill pre-existing CLI-parity gaps: `ad_snapshot` (runs the full refmap pipeline, returns `AdSnapshot { tree, refmap }`), `ad_execute_by_ref(adapter, "@e5", action, out)`, `ad_wait(adapter, AdWaitCondition, timeout, out)`, `ad_version() -> *const c_char`, `ad_abi_version() -> u32` (packed major/minor/patch) with `AD_ABI_VERSION_MAJOR` cbindgen `[defines]` export, `ad_status(adapter, out)`, `ad_set_log_callback(fn(level, msg))` that installs a `tracing_subscriber` layer so dlopen consumers see debug output | +| P2-O17 | Screen Recording / Automation permission detection (macOS backfill) | `check_permissions()` returns a richer `PermissionStatus` with a tri-state for AX, Screen Recording (`CGPreflightScreenCaptureAccess` / `CGRequestScreenCaptureAccess`), and Automation (`AEDeterminePermissionToAutomateTarget`). Failures surface as `PermDenied` / `AutomationPermissionDenied` with concrete System Settings paths | + +### Cross-Platform Trait Extensions + +All methods land as `#[non_exhaustive]` additions in `crates/core/src/adapter.rs` with default implementations returning `AdapterError::not_supported(method)`. Windows implements them natively. macOS backfills in the same PR pair. Linux (Phase 3) adds the AT-SPI2 implementations. + +```rust +impl PlatformAdapter for … { + // P2-O11 — event subscription + fn watch_element( + &self, + handle: &NativeHandle, + events: &[EventKind], + timeout: Duration, + ) -> Result, AdapterError> { /* default: not_supported */ } + + // P2-O12 — text ranges + fn get_text_selection(&self, handle: &NativeHandle) -> Result; + fn set_text_selection(&self, handle: &NativeHandle, range: TextRange) -> Result<(), AdapterError>; + fn get_text_at(&self, handle: &NativeHandle, range: TextRange) -> Result; + fn insert_text_at_caret(&self, handle: &NativeHandle, text: &str) -> Result<(), AdapterError>; + + // P2-O13 — modern screenshot + // (screenshot() gains a new `ScreenshotBackend::Modern` variant; platforms pick the + // native modern API; a `Legacy` fallback preserves the Phase 1 subprocess path.) + + // P2-O14 — new surfaces + fn list_surfaces(&self, pid: i32) -> Result, AdapterError> // extended kinds +} +``` + +New supporting types (land in `crates/core/src/`): + +- `EventKind` — `FocusChanged`, `ValueChanged`, `SelectionChanged`, `ChildrenChanged`, `WindowOpened`, `WindowClosed`, `MenuOpened`, `MenuClosed`, `NotificationPosted`, `ElementDestroyed` +- `ElementEvent` — `{ kind, handle_ref_id: Option, timestamp, attr_snapshot: Option }` +- `TextRange` — `{ start: u32, length: u32 }` (UTF-16 code units to match both AX CFRange and UIA TextRange conventions) +- `TextSelection` — `{ range: TextRange, caret_offset: u32, lines_in_view: Vec }` +- `ScreenshotBackend` — `Modern` (ScreenCaptureKit / Windows.Graphics.Capture / PipeWire) or `Legacy` (preserves Phase 1 subprocess path as fallback for restricted environments) +- `PermissionStatus` extends to `{ accessibility: TriState, screen_recording: TriState, automation: TriState }` where `TriState = Granted | Denied { suggestion } | NotDetermined` + +### Cross-platform capability map (P2-O8 through O17) + +| Capability | macOS API | Windows API | Linux API (Phase 3) | +|------------|-----------|-------------|----------------------| +| Stable `identifier` | `kAXIdentifierAttribute` | UIA `AutomationId` | AT-SPI2 `accessible-id` + GTK `gtk-id` | +| `subrole` | `kAXSubroleAttribute` | UIA `LocalizedControlType` + pattern-based heuristic | AT-SPI2 `role-name` + `state-set` | +| `role_description` | `kAXRoleDescriptionAttribute` | UIA `LocalizedControlType` | AT-SPI2 `role-description` | +| `placeholder` | `kAXPlaceholderValueAttribute` | UIA `HelpText` + `IsTextEditPatternAvailable` placeholder | AT-SPI2 `description` + HTML `placeholder` via `object-attributes` | +| `dom_id` / `dom_classes` | `kAXDOMIdentifierAttribute` / `kAXDOMClassListAttribute` | Edge WebView2 UIA `HtmlId` / `HtmlClass` properties | AT-SPI2 `object-attributes` HTML keys | +| Event subscription | `AXObserverCreate` + `AXObserverAddNotification` on `CFRunLoop` | `IUIAutomation.AddAutomationEventHandler` + `AddFocusChangedEventHandler` + `AddPropertyChangedEventHandler` | AT-SPI2 D-Bus signals via `zbus::StreamFactory` | +| Text range read | `AXStringForRangeParameterizedAttribute` + `AXSelectedTextRangeAttribute` | `TextPattern.GetSelection`, `TextPattern.DocumentRange.GetText` | AT-SPI2 `Text.GetText(start, end)` + `Text.GetCaretOffset` | +| Text range write | `AXSelectedTextRange = AXValueCreate(kAXValueCFRangeType, …)` | `TextRange.Select` + `TextRange.Move` | AT-SPI2 `EditableText.InsertText` + `Text.SetCaretOffset` | +| Modern per-window screenshot | `SCScreenshotManager.captureImage(contentFilter:config:)` | `GraphicsCaptureItem.CreateFromWindowHandle` + `Direct3D11CaptureFramePool` | PipeWire `org.freedesktop.portal.ScreenCast` | +| Toolbar surface | `AXRole == AXToolbar` or `AXUnifiedTitleAndToolbar` | UIA `ControlType.ToolBar` | AT-SPI2 `Role::ToolBar` | +| Menu-bar extras surface | `SystemUIServer` + `ControlCenter` pid walk | UIA `Shell_TrayWnd` + `NotifyIconOverflowWindow` | AT-SPI2 `StatusNotifierWatcher` D-Bus | +| Dock / taskbar surface | `Dock.app` pid walk | UIA `Shell_TrayWnd` `TaskListButton` children | AT-SPI2 per-DE panel walk | +| `LongPress` | `CGEventCreateMouseEvent(…Down…)` + sleep + `…Up` | `SendInput` hold + release | Coordinate via `ydotool/xdotool` | +| `ForceClick` | `CGEventSetIntegerValueField(kCGMouseEventPressure, …)` + `kCGEventMouseSubtypeTabletPoint` | Pen input `SendInput` with `PEN_FLAGS_BARREL` | Not natively supported — return `ActionNotSupported` | +| `ShowMenu` action | `AXPerformAction(kAXShowMenuAction)` | `ExpandCollapsePattern.Expand` + UIA right-click fallback | AT-SPI2 `Action.DoAction("popup")` | +| `WindowRaise` | `AXUIElementSetAttributeValue(kAXRaiseAction)` | `SetForegroundWindow` + `SetWindowPos(HWND_TOP)` | `wmctrl -a` / `xdotool windowactivate` | +| `Cancel` | `AXPerformAction(kAXCancelAction)` | UIA `WindowPattern.Close` on dialog or `InvokePattern` on cancel button | AT-SPI2 `Action.DoAction("cancel")` or synthesize Escape | +| `FileDrop(Vec)` | `NSPasteboard` file-promise drag via `NSFilePromiseProvider` | `IDataObject` + `DoDragDrop` via `ole32` | XDND protocol via `xdotool` file-drag extension | +| Screen Recording permission | `CGPreflightScreenCaptureAccess` / `CGRequestScreenCaptureAccess` | Implicit for `Windows.Graphics.Capture` — prompts on first use | PipeWire portal permission dialog | +| Automation permission | `AEDeterminePermissionToAutomateTarget` | N/A (no equivalent restriction) | N/A | ### Windows Adapter Implementation @@ -616,7 +698,7 @@ crates/windows/src/ | Keyboard | `SendInput` API | `INPUT_KEYBOARD` structs with virtual key codes and scan codes | | Mouse | `SendInput` API | `INPUT_MOUSE` structs with `MOUSEEVENTF_*` flags | | Clipboard | `OpenClipboard` / `GetClipboardData` / `SetClipboardData` | Win32 APIs, handle `CF_UNICODETEXT` format | -| Screenshot | `BitBlt` / `PrintWindow` | Window capture; or `xcap` crate for cross-platform consistency | +| Screenshot | `Windows.Graphics.Capture` | Modern per-window capture via `GraphicsCaptureItem.CreateFromWindowHandle` + `Direct3D11CaptureFramePool`. Prompts for capture permission on first use, no subprocess, respects DWM compositing. `BitBlt` / `PrintWindow` retained as `ScreenshotBackend::Legacy` fallback for pre-Windows-10 1903 environments | | App launch | `CreateProcess` / `ShellExecuteEx` | Launch by name or path, wait for main window | | App close | `WM_CLOSE` / `TerminateProcess` | Graceful close first, force kill with `--force` | | Window ops | `SetWindowPos` / `ShowWindow` | Resize, move, minimize (`SW_MINIMIZE`), maximize (`SW_MAXIMIZE`), restore (`SW_RESTORE`) | @@ -677,16 +759,25 @@ Chromium-based apps (Electron, Chrome, Edge, VS Code) expose deep, noisy accessi - Windows 10 1809+ (October 2018 update) - UIA COM interfaces available since Windows 7, but modern patterns require 10+ -### New Dependency +### New Dependencies -| Crate | Version | Purpose | License | -|-------|---------|---------|---------| -| `uiautomation` | 0.24+ | Windows UIA wrapper via `windows` crate | Apache-2.0 | +| Crate | Version | Scope | Purpose | +|-------|---------|-------|---------| +| `uiautomation` | 0.24+ | Windows | UIA client wrapper, tree walker, patterns | +| `windows` | 0.58+ | Windows | Raw Win32 / WinRT bindings for SendInput, clipboard, `Windows.Graphics.Capture`, D3D11 frame pool | +| `objc2` | 0.5+ | macOS (new for P2-O13 / O17) | Safe bridging to `SCScreenshotManager`, `CGPreflightScreenCaptureAccess`, `NSFilePromiseProvider` — replaces ad-hoc `objc` message sends | +| `screencapturekit` | 0.3+ | macOS (new for P2-O13) | Thin wrapper around the `ScreenCaptureKit` framework (`SCShareableContent`, `SCScreenshotManager`) | -Added to `Cargo.toml` as target-gated dependency: +Added to `Cargo.toml` as target-gated dependencies: ```toml [target.'cfg(target_os = "windows")'.dependencies] agent-desktop-windows = { path = "crates/windows" } +uiautomation = "0.24" +windows = { version = "0.58", features = ["Win32_UI_Input", "Win32_System_Com", "Graphics_Capture", "Win32_Graphics_Direct3D11"] } + +[target.'cfg(target_os = "macos")'.dependencies] +objc2 = "0.5" +screencapturekit = "0.3" ``` ### Testing @@ -723,6 +814,24 @@ agent-desktop-windows = { path = "crates/windows" } - Same snapshot of a cross-platform app (e.g., VS Code) produces structurally identical JSON on macOS and Windows - All error codes produce identical JSON envelope format +**Cross-platform extension tests (P2-O8 through O17):** +- Stable-selector fields: every interactive node emits `identifier` on both platforms (UIA `AutomationId` on Windows, `AXIdentifier` on macOS) for a target app with known IDs (e.g. Calculator's `AutomationId` on Windows, a test harness app exporting `accessibilityIdentifier` on macOS) +- Event subscription: `watch --event value-changed --ref @e3 --timeout 2000` receives an event within 500 ms of a programmatic value change on both platforms +- Text ranges: `text select-range @e1 5 10` + `text get-selection @e1` round-trips to `{start:5, length:10}` on both platforms for a multi-line text editor (TextEdit / Notepad) +- Text insert-at-caret: `text insert-at-caret @e1 "hello"` produces matching `value` on both platforms with the caret advanced correctly +- Modern screenshot: `screenshot --window ` PNG matches a reference capture within SSIM threshold; cold latency <50 ms on both platforms (vs ~300 ms macOS subprocess baseline) +- Toolbar surface: `snapshot --surface toolbar` on Safari (macOS) and Edge (Windows) returns the toolbar's children with refs +- Electron deep-tree: VS Code snapshot with `--force-electron-a11y` exposes ≥100 refs at default depth on both platforms +- Screen Recording permission: on a macOS runner without Screen Recording, `screenshot --window` returns `PermDenied` with the Screen Recording suggestion (distinct from AX denial) +- Automation permission: on a macOS runner without Automation for a target app, `close-app` returns `AutomationPermissionDenied` rather than squeezing into `ActionFailed` + +**FFI parity tests (P2-O16):** +- `ad_abi_version()` returns a packed `u32` matching the Cargo version; consumer built against 0.2.0 refuses to load 0.3.0 +- `ad_snapshot` writes a refmap and the same `@e5` resolves via `ad_execute_by_ref` without a prior CLI snapshot on disk +- `ad_execute_by_ref(adapter, "@e5", AD_ACTION_KIND_CLICK, &out)` produces identical `AdActionResult` to `ad_resolve_element` + `ad_execute_action` +- `ad_set_log_callback` receives at least one `tracing::debug!` event during a `ad_get_tree` call +- Every new `Action` variant round-trips through the `AdAction.kind` i32 → Rust enum conversion without UB on arbitrary bit patterns (extends the existing `fuzz_arbitrary_bit_patterns_never_panic_across_all_enums` suite) + ### CI - Add GitHub Actions Windows runner alongside existing macOS runner @@ -732,9 +841,11 @@ agent-desktop-windows = { path = "crates/windows" } ### Release -- [ ] Prebuilt Windows `.exe` binary published to GitHub Releases via `cargo-dist` -- [ ] npm package updated to include Windows binary (platform-specific download) -- [ ] GitHub Release notes document Windows support and installation +- [ ] Prebuilt Windows `.exe` binary added to the existing `.github/workflows/release.yml` `build` matrix (alongside the macOS CLI targets). Uses the same tarball + sha256 + attestation pipeline shipped in Phase 1.5. +- [ ] npm `postinstall.js` gains a `win32-x64` / `win32-arm64` branch so `npm install -g agent-desktop` works on Windows without changes to package shape. +- [ ] The Phase 1.5 FFI cdylib for Windows (`x86_64-pc-windows-msvc`) is already shipping; Phase 2 adds `aarch64-pc-windows-msvc` for ARM64 parity. +- [ ] Every new `ad_*` FFI entrypoint (P2-O16) is included in the `release-ffi` build and CI header drift check. +- [ ] GitHub Release notes document Windows support and installation. ### Skill Update @@ -769,23 +880,38 @@ Per [Skill Maintenance Addendum](./prd-addendum-skill-maintenance.md): --- -## Phase 3 — Linux Adapter +## Phase 3 — Linux Adapter + Cross-Platform Parity Completion **Status: Planned** -Phase 3 brings agent-desktop to Linux, completing the three-platform story. Same additive approach — only the `LinuxAdapter` implementation is added inside `crates/linux/`. The existing stub is replaced with a full implementation. +Phase 3 completes the three-platform story. The Linux adapter implements the original adapter surface **plus** every cross-platform extension landed in Phase 2 (event subscriptions, text ranges, modern screenshot, stable-selector fields, Toolbar surface, new Action variants, new ErrorCode variants). Each has a canonical AT-SPI2 / D-Bus / Wayland-portal implementation. Core engine, trait contract, and FFI surface are untouched — Phase 3 is pure adapter code plus AT-SPI-specific compat notes. ### Objectives +Linux parity (original scope): + | ID | Objective | Metric | |----|-----------|--------| | P3-O1 | Linux adapter | `snapshot` on Ubuntu GNOME returns valid tree for Files, Terminal, Settings | | P3-O2 | All commands cross-platform | Identical JSON schema output on all 3 platforms for every command | | P3-O3 | Linux input synthesis | `click`, `type`, `press`, all mouse commands via AT-SPI actions + xdotool/ydotool | -| P3-O4 | Linux screenshot | `screenshot` produces PNG via PipeWire ScreenCast (Wayland) / XGetImage (X11) | +| P3-O4 | Linux screenshot | `screenshot` produces PNG via PipeWire ScreenCast portal (Wayland) / XGetImage (X11) | | P3-O5 | Linux clipboard | `clipboard-get` / `clipboard-set` / `clipboard-clear` via `wl-clipboard` (Wayland) / `xclip` (X11) | | P3-O6 | Cross-platform CI | GitHub Actions matrix: macOS + Windows + Ubuntu | -| P3-O7 | Linux binary release | Prebuilt binary published via GitHub Releases and npm | +| P3-O7 | Linux binary release | Prebuilt CLI binary added to the release pipeline (Phase 1.5 already ships the Linux FFI cdylib) | + +Cross-platform extensions (Linux implementations of Phase 2 primitives): + +| ID | Objective | Metric | +|----|-----------|--------| +| P3-O8 | Stable-selector fields on Linux | `AccessibilityNode.identifier` populated from AT-SPI2 `accessible-id` attribute (standard since AT-SPI 2.18) with GTK `gtk-id` / Qt `objectName` fallback; `dom_id` / `dom_classes` populated from AT-SPI2 `object-attributes` HTML keys (`id`, `class`) on `WebKitGTK` / `Chromium-Content` embeds | +| P3-O9 | AT-SPI2 event subscriptions (P2-O11 parity) | `watch_element` implemented via `zbus::Proxy::receive_signal` on AT-SPI2 signals: `org.a11y.atspi.Event.Object.PropertyChange`, `ChildrenChanged`, `StateChanged:focused`, `Window:Create`, `Window:Destroy`. Same `wait --event` CLI shape as macOS/Windows. Replaces polling in `crates/linux/src/system/wait.rs` before it's even written | +| P3-O10 | AT-SPI2 Text interface (P2-O12 parity) | Text range primitives via `org.a11y.atspi.Text` D-Bus methods: `GetText(start, end)`, `GetCaretOffset`, `SetCaretOffset`, `GetNSelections`, `GetSelection(n)`, `AddSelection(start, end)`, `RemoveSelection(n)`. `InsertAtCaret` uses `org.a11y.atspi.EditableText.InsertText(position, text, length)` | +| P3-O11 | PipeWire modern screenshot (P2-O13 parity) | `screenshot --window ` via `org.freedesktop.portal.ScreenCast` (Wayland) + `org.freedesktop.portal.RemoteDesktop` for capture permission flow. XDG desktop portal handles the user consent dialog exactly like `SCScreenshotManager` does on macOS. X11 fallback uses `XGetImage` for the lowest-permission path | +| P3-O12 | Toolbar + surfaces (P2-O14 parity) | `SnapshotSurface::Toolbar` via AT-SPI2 `Role::ToolBar`. Dock / taskbar surface via per-DE panel process walk (GNOME Shell process for gnome-shell extensions, Plasma `plasmashell` for KDE). StatusNotifierWatcher already scoped in the original Phase 3 tray spec | +| P3-O13 | Action variants on Linux (P2-O9 parity) | `Action::LongPress` via timed `xdotool/ydotool` button-hold; `Action::ShowMenu` via `org.a11y.atspi.Action.DoAction("popup")`; `Action::Cancel` via `Action.DoAction("cancel")` or Escape synthesis; `Action::FileDrop` via XDND (`xdotool key`+selection protocol on X11, portal-based FileTransfer on Wayland); `Action::ForceClick` returns `ActionNotSupported` on Linux (no pressure input primitive) | +| P3-O14 | FFI cdylib continues to ship | Phase 1.5 already publishes Linux FFI for x86_64 + aarch64; Phase 3 adds each new `ad_*` entrypoint's Linux implementation and extends the header drift check. No new FFI bindings to design — just implementations for the platform-specific methods under the existing trait | +| P3-O15 | Flatpak / Snap compatibility note | AT-SPI2 requires `--talk-name=org.a11y.Bus` permission inside sandboxed runtimes. Skill docs include the exact Flatpak override and Snap plug grants, so sandboxed consumers aren't silently empty-tree | ### Linux Adapter Implementation @@ -993,7 +1119,15 @@ Note: `tokio` is introduced here for the first time. Phases 1-2 are fully synchr - Same snapshot of a cross-platform app (e.g., VS Code) produces structurally identical JSON on all 3 platforms - All error codes produce identical JSON envelope format on all 3 platforms - Notification commands return identical JSON envelope structure across all 3 platforms (list, dismiss, action) -- Tray commands return identical JSON envelope structure across all 3 platforms +- Tray / StatusNotifierItem commands return identical JSON envelope structure across all 3 platforms + +**Extension tests for P3-O8 through O15 (Linux-specific parity):** +- AT-SPI `accessible-id` populated for every interactive node in GNOME Calculator, GNOME Files, Firefox (with `ACCESSIBILITY_ENABLED=1`) +- `watch --event value-changed` via `zbus` signal subscription delivers an event within 500 ms for a programmatic value change in a test harness app (GTK4 + pygobject) +- `text select-range` / `get-selection` / `insert-at-caret` round-trips correctly in GNOME Text Editor via `org.a11y.atspi.Text` + `EditableText` +- PipeWire portal screenshot flow: user approves via XDG portal dialog, subsequent calls bypass the dialog within the session grant window; screenshot matches reference +- Toolbar surface: Firefox toolbar + GNOME Files toolbar both enumerate via `Role::ToolBar` +- Flatpak compatibility: a Flatpak-packaged GNOME Text Editor snapshot is non-empty when `--talk-name=org.a11y.Bus` is granted; returns clear diagnostic otherwise ### CI @@ -1004,9 +1138,10 @@ Note: `tokio` is introduced here for the first time. Phases 1-2 are fully synchr ### Release -- [ ] Prebuilt Linux binary published to GitHub Releases via `cargo-dist` -- [ ] npm package updated to include Linux binary (platform-specific download) -- [ ] GitHub Release notes document Linux support, requirements, and installation +- [ ] Prebuilt Linux CLI binary added to `.github/workflows/release.yml` matrix for `x86_64-unknown-linux-gnu` and `aarch64-unknown-linux-gnu` (Phase 1.5 already builds the FFI cdylib for both triples on the same runners — Phase 3 reuses those runners) +- [ ] npm `postinstall.js` gains `linux-x64` / `linux-arm64` branches +- [ ] Every new `ad_*` Linux implementation from P3-O9 / O10 / O11 is covered by the existing FFI drift check + Sigstore attestation pipeline +- [ ] GitHub Release notes document Linux support, minimum glibc (2.35, Ubuntu 22.04 baseline), display-server requirements, and Flatpak/Snap compatibility ### Skill Update @@ -1047,16 +1182,26 @@ Per [Skill Maintenance Addendum](./prd-addendum-skill-maintenance.md): **Status: Planned** -Phase 4 adds a new I/O layer. Core engine and all three platform adapters are unchanged. The MCP server wraps existing command logic in JSON-RPC tool definitions, enabling agent-desktop to work as an MCP server for Claude Desktop, Cursor, and other MCP-compatible hosts. +Phase 4 adds a new I/O layer. Core engine and all three platform adapters are unchanged. The MCP server wraps existing command logic in JSON-RPC tool definitions, enabling agent-desktop to work as an MCP-native desktop automation server for Claude Desktop, Cursor, VS Code Copilot, Gemini CLI, Microsoft Agent Framework 1.0, and any other MCP-compatible host. + +By Phase 4 the CLI already covers 53 commands on three platforms, the FFI ships as a shared library for in-process consumers, and the cross-platform event / text-range / stable-selector primitives from Phase 2 / 3 are in place. MCP mode is primarily a **transport + discovery layer** on top of them. Every MCP tool maps 1:1 to a CLI command — no fork in semantics, no parallel codepath. ### Objectives | ID | Objective | Metric | |----|-----------|--------| -| P4-O1 | MCP server mode via `--mcp` | Responds to MCP `initialize` handshake, reports capabilities | -| P4-O2 | All commands as MCP tools | `tools/list` returns all tools with JSON Schema specs | -| P4-O3 | Claude Desktop validated | Claude Desktop invokes tools to control desktop apps end-to-end on all platforms | -| P4-O4 | Tool annotations | `readOnlyHint`, `destructiveHint`, `idempotentHint` on every tool | +| P4-O1 | MCP server mode via `--mcp` | Responds to MCP `initialize` handshake, reports capabilities, per-host hello-world passes | +| P4-O2 | All commands as MCP tools | `tools/list` returns 53+ tools with JSON Schemas generated from the CLI arg structs via `schemars`; tool names prefixed `desktop_` | +| P4-O3 | Claude Desktop + Cursor + VS Code + Gemini CLI + MS Agent Framework validated | Each host invokes tools to control a desktop app end-to-end on all three platforms; repo ships `mcp.json` / `claude_desktop_config.json` / `.cursor/mcp.json` examples per host | +| P4-O4 | Tool annotations | `readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint` on every tool; Claude Desktop surfaces destructive tools with a confirmation prompt | +| P4-O5 | Ref-based MCP tool shape (Playwright-MCP idiom) | Tools take `{ref: "e5"}` not raw `element_handle`, matching Playwright MCP so agents can swap between the two without relearning selectors. Tree snapshots return as MCP resources with refs inline | +| P4-O6 | MCP resource types | `agent-desktop://refmap/current`, `agent-desktop://snapshot/latest`, `agent-desktop://audit/{trace_id}` (audit log under Phase 5). `resources/list` + `resources/read` expose the current RefMap and last snapshot without re-running the command | +| P4-O7 | Tree-diff notifications | `watch_element` events (Phase 2 P2-O11) stream as MCP `notifications/message` during a long-running wait, so the host sees value-changed / focus-changed events as they happen rather than polling | +| P4-O8 | Progress notifications | `notifications/progress` for `wait`, `snapshot --skeleton` → `--root` drill-down chains, and large-tree traversals. Agents surface progress to users instead of hanging | +| P4-O9 | Tool-level permission tiers | Observation tools (`desktop_snapshot`, `desktop_find`, `desktop_get`, `desktop_is`, `desktop_list_*`) are freely callable. Interaction tools (`desktop_click`, `desktop_type_text`, `desktop_set_value`, `desktop_drag`) are gated behind an `interactive` capability negotiated at `initialize`. Destructive tools (`desktop_close_app`, `desktop_dismiss_all_notifications`) require the `destructive` capability plus the Phase 5 audit log | +| P4-O10 | Session-scoped RefMap | Each MCP session has its own in-memory RefMap keyed by `session_id` — no conflict with the on-disk CLI RefMap, no cross-session leakage when a host runs multiple agent-desktop-mcp instances | +| P4-O11 | MCP `initialize` returns tri-platform capability matrix | The `initialize` response declares platform (macOS / Windows / Linux), permission status (AX + Screen Recording + Automation tri-state from Phase 2 P2-O17), display-server (Linux), and the set of actually-supported tools given current permissions. A host can decide whether to prompt for missing permissions before the first tool call | +| P4-O12 | SSE + Streamable HTTP transports | Stdio remains primary. SSE (pre-March-2025 spec) and **Streamable HTTP** (post-March-2025 replacement) are implemented for remote scenarios — MS Agent Framework and future MCP hosts prefer the HTTP transport | ### Entry Point @@ -1078,43 +1223,90 @@ crates/mcp/src/ ### MCP Tool Surface -Each MCP tool maps 1:1 to a CLI command. Tool names are prefixed with `desktop_` to avoid collision with other MCP servers. +Each MCP tool maps 1:1 to a CLI command. Tool names are prefixed with `desktop_` to avoid collision with other MCP servers. Annotations match the Phase 4 capability tiers (P4-O9): tools with `destructiveHint: true` only run when the `destructive` capability was negotiated at `initialize`. -| MCP Tool | CLI Equivalent | readOnly | destructive | -|----------|---------------|----------|-------------| -| `desktop_snapshot` | `snapshot` | true | false | -| `desktop_click` | `click ` | false | false | -| `desktop_type_text` | `type ` | false | false | -| `desktop_set_value` | `set-value ` | false | false | -| `desktop_press_key` | `press ` | false | false | -| `desktop_find` | `find ` | true | false | -| `desktop_list_windows` | `list-windows` | true | false | -| `desktop_focus_window` | `focus-window` | false | false | -| `desktop_launch_app` | `launch ` | false | false | -| `desktop_close_app` | `close-app ` | false | true | -| `desktop_screenshot` | `screenshot` | true | false | -| `desktop_scroll` | `scroll ` | false | false | -| `desktop_drag` | `drag ` | false | false | -| `desktop_select` | `select ` | false | false | -| `desktop_toggle` | `toggle ` | false | false | -| `desktop_clipboard_get` | `clipboard get` | true | false | -| `desktop_clipboard_set` | `clipboard set ` | false | false | -| `desktop_wait` | `wait` | true | false | -| `desktop_get` | `get ` | true | false | -| `desktop_is` | `is ` | true | false | -| `desktop_list_notifications` | `list-notifications` | true | false | -| `desktop_dismiss_notification` | `dismiss-notification ` | false | true | -| `desktop_dismiss_all_notifications` | `dismiss-all-notifications` | false | true | -| `desktop_notification_action` | `notification-action ` | false | false | -| `desktop_list_tray_items` | `list-tray-items` | true | false | -| `desktop_click_tray_item` | `click-tray-item ` | false | false | -| `desktop_open_tray_menu` | `open-tray-menu ` | false | false | +Observation tools (always available): + +| MCP Tool | CLI | Returns | +|----------|-----|---------| +| `desktop_snapshot` | `snapshot` | Tree + refmap in response; also published as `agent-desktop://snapshot/latest` resource | +| `desktop_find` | `find ` | Matching refs (array) | +| `desktop_get` | `get ` | Property value | +| `desktop_is` | `is ` | Boolean | +| `desktop_list_windows` | `list-windows` | Array of windows | +| `desktop_list_apps` | `list-apps` | Array of apps | +| `desktop_list_surfaces` | `list-surfaces` | Array of surfaces (incl. Toolbar / Spotlight / Dock / MenuBarExtras from P2-O14) | +| `desktop_list_notifications` | `list-notifications` | Array of notifications | +| `desktop_screenshot` | `screenshot` | Base64 PNG (or MCP resource link) | +| `desktop_clipboard_get` | `clipboard-get` | Clipboard text | +| `desktop_permissions` | `permissions` | Tri-state permission report (AX + Screen Recording + Automation) | +| `desktop_status` | `status` | Daemon + adapter status | +| `desktop_version` | `version` | Version + ABI version | + +Interaction tools (gated by `interactive` capability): + +| MCP Tool | CLI | Shape | +|----------|-----|-------| +| `desktop_click` / `desktop_double_click` / `desktop_triple_click` / `desktop_right_click` | `click @e5` (and variants) | `{ref: "e5"}` | +| `desktop_type_text` | `type @e5 "hello"` | `{ref: "e5", text: "hello"}` | +| `desktop_set_value` | `set-value @e5 "hello"` | `{ref: "e5", value: "hello"}` | +| `desktop_clear` | `clear @e5` | `{ref: "e5"}` | +| `desktop_focus` | `focus @e5` | `{ref: "e5"}` | +| `desktop_select` / `desktop_toggle` / `desktop_check` / `desktop_uncheck` / `desktop_expand` / `desktop_collapse` | — | `{ref: "e5"}` (+ `value` for select) | +| `desktop_scroll` / `desktop_scroll_to` | `scroll ` | `{ref: "e5", direction, amount}` | +| `desktop_press_key` / `desktop_key_down` / `desktop_key_up` | `press ` | `{key, modifiers}` | +| `desktop_hover` / `desktop_drag` | `hover`/`drag` | `{ref: "e5"}` or `{from, to}` | +| `desktop_mouse_move` / `desktop_mouse_click` / `desktop_mouse_down` / `desktop_mouse_up` | — | `{x, y, button}` | +| `desktop_wait` | `wait --element / --window / --text / --menu / --notification` | `{condition, timeout_ms}` | +| `desktop_watch_element` (P2-O11) | `watch --event …` | `{ref: "e5", events: [EventKind], timeout_ms}` — streams via `notifications/message` | +| `desktop_launch_app` / `desktop_focus_window` / `desktop_resize_window` / `desktop_move_window` / `desktop_minimize` / `desktop_maximize` / `desktop_restore` | app / window ops | App / window args | +| `desktop_clipboard_set` / `desktop_clipboard_clear` | — | `{text}` / `{}` | +| `desktop_notification_action` | `notification-action ` | `{index, expected_app?, expected_title?, action}` (NC-reorder safe) | +| `desktop_text_select_range` / `desktop_text_get_selection` / `desktop_text_insert_at_caret` / `desktop_text_at_offset` (P2-O12) | `text …` subcommands | `{ref, start, length, text?}` | + +Destructive tools (gated by both `interactive` and `destructive` capabilities; always write to the Phase 5 audit log): + +| MCP Tool | CLI | +|----------|-----| +| `desktop_close_app` | `close-app [--force]` | +| `desktop_dismiss_notification` | `dismiss-notification ` | +| `desktop_dismiss_all_notifications` | `dismiss-all-notifications` | +| `desktop_batch` | `batch` — accepts destructive sub-commands, each evaluated against its own annotation | + +### MCP Resource Types + +Resources let hosts pull structured state without re-issuing a tool call: + +| URI | Content | Update model | +|-----|---------|--------------| +| `agent-desktop://refmap/current` | JSON RefMap for the current MCP session (not the on-disk CLI refmap) | Replaced on every `desktop_snapshot` invocation; subscribable via `notifications/resources/updated` | +| `agent-desktop://snapshot/latest` | Last `desktop_snapshot` response as JSON (tree + refmap + metadata) | Same update model | +| `agent-desktop://permissions/current` | Tri-state permission report (AX, Screen Recording, Automation, display-server) | Refreshed on request; subscribable when Phase 2 P2-O17 permission observer is available | +| `agent-desktop://events/stream` | Merged `watch_element` event stream for the session | Real-time, subscribable | +| `agent-desktop://audit/{trace_id}` | Phase 5 append-only audit log entries for a trace | Growable; new entries as `notifications/resources/updated` | + +### Framework Integration Targets + +Every major 2026 MCP host gets a validated config example committed to `examples/mcp-hosts/`: + +| Host | Config file | Transport | Notes | +|------|-------------|-----------|-------| +| Claude Desktop | `claude_desktop_config.json` | stdio | Already widespread; our reference host | +| Cursor | `.cursor/mcp.json` | stdio | Per-workspace config | +| VS Code (Copilot) | `.vscode/mcp.json` + `settings.json` | stdio | Copilot Chat 2026 adds MCP tool discovery | +| Gemini CLI | `~/.config/gemini-cli/mcp.json` | stdio | Google's first-party MCP integration | +| Microsoft Agent Framework 1.0 | `agentframework.yaml` MCP section | Streamable HTTP | Cloud-first host, requires HTTP transport (P4-O12) | +| Zed editor | `~/.config/zed/settings.json` | stdio | Desktop IDE with MCP-native agents | +| Continue.dev | `config.json` MCP section | stdio | OSS agent framework | + +Each host gets a ~30-line config + a 60-second "hello agent" demo (launch Calculator → compute something → verify result) in the `examples/` directory as a runnable acceptance test. ### Transport -- **Stdio (primary):** MCP host spawns `agent-desktop --mcp` as child process. JSON-RPC over stdin/stdout. This is the only required transport. -- **HTTP+SSE (optional, stretch goal):** For remote scenarios. Additive, non-blocking for core milestone. -- **Session:** On MCP `initialize`, detect platform, check accessibility permissions, report capabilities. RefMap is session-scoped (held in memory, not persisted to disk like CLI mode). +- **Stdio (primary):** MCP host spawns `agent-desktop --mcp` as a child process. JSON-RPC over stdin/stdout. Required; validated against all hosts in the Framework Integration table. +- **Streamable HTTP (P4-O12, required for MS Agent Framework):** Single HTTP endpoint at `POST /mcp` with chunked response streaming; replaces the pre-March-2025 SSE transport. Used when the host declares `transport: http` in its MCP config. Binds to `127.0.0.1` by default; `--mcp-bind ` CLI flag overrides. +- **SSE (legacy):** Retained for hosts that haven't migrated to Streamable HTTP. Gated on `--mcp-transport sse`. +- **Session:** On `initialize`, detect platform, probe permissions (AX + Screen Recording + Automation tri-state), report tool capabilities given current permissions. Each MCP session has its own in-memory RefMap keyed by `session_id` — never reads or writes the on-disk CLI refmap at `~/.agent-desktop/last_refmap.json`. Sessions are isolated so the same host can run multiple agent-desktop-mcp instances concurrently without cross-contamination. ### Initialize Handler @@ -1158,6 +1350,24 @@ Note: If `tokio` was already introduced in Phase 3 (Linux), it is already availa - MCP server works identically on macOS, Windows, and Linux - Same tool invocations produce same JSON structure on all platforms +**Framework host acceptance tests (one per row in the Framework Integration table):** +- Claude Desktop: launch Calculator → snapshot → click buttons → verify result string via `desktop_get` +- Cursor: open a code file → snapshot editor → `desktop_text_insert_at_caret` a function → verify file content +- VS Code Copilot: same as Cursor on the VS Code host +- Gemini CLI: text-only interaction — list open windows, focus one, dismiss a notification +- Microsoft Agent Framework 1.0 (Streamable HTTP): HTTP-based MCP client runs the same Calculator demo against `http://127.0.0.1:/mcp` +- Zed: editor-focused scenario (open file → select range → replace) +- Continue.dev: Claude Opus 4.7 with our server runs a 3-step canvas test in TextEdit + +**Capability negotiation tests (P4-O9):** +- Host that negotiates only `observation` cannot invoke `desktop_click` — MCP error with clear `-32601 Method not found within capability set` message +- Host that negotiates `interactive` but not `destructive` cannot invoke `desktop_close_app` +- `initialize` response's `supported_tools` list shrinks correctly when AX permission is denied (only `desktop_permissions`, `desktop_version`, `desktop_status` remain) + +**Event streaming tests (P4-O7):** +- `desktop_watch_element` subscription receives `notifications/message` events for a programmatic value change within 500 ms of the change on all three platforms +- Two concurrent watches on different refs get their events routed to the correct subscription ID + ### MCP Config Examples Provide ready-to-use config snippets for: @@ -1220,7 +1430,7 @@ Per [Skill Maintenance Addendum](./prd-addendum-skill-maintenance.md): **Status: Planned** -Phase 5 transforms agent-desktop from functional to enterprise-grade. Persistent daemon process, session isolation for concurrent agents, comprehensive quality gates, and distribution via native package managers. +Phase 5 transforms agent-desktop from functional to enterprise-grade. Persistent daemon process, session isolation for concurrent agents, the safety trio required for enterprise and regulated deployments (dry-run + confirm + audit log), an OCR/vision fallback for custom-rendered UIs where the accessibility tree is empty, session tracing with OpenTelemetry-compatible event streams, and first-class distribution via native package managers. ### Objectives @@ -1229,7 +1439,13 @@ Phase 5 transforms agent-desktop from functional to enterprise-grade. Persistent | P5-O1 | Persistent daemon | Warm snapshot completes in <50ms (vs 200ms+ cold start) | | P5-O2 | Session isolation | Two agents hold independent RefMaps without interference | | P5-O3 | Enterprise quality gates | All gates in quality gates table pass | -| P5-O4 | Package manager distribution | Available via brew (macOS), winget/scoop (Windows), snap/apt (Linux) | +| P5-O4 | Package manager distribution | Available via brew (macOS), winget/scoop (Windows), snap/apt (Linux) with Sigstore attestation verification on install | +| P5-O5 | Safety trio: `--dry-run` / `--confirm` / append-only audit log | Every destructive command supports `--dry-run` (resolves ref, computes the action, emits the would-be JSON response, does not execute), `--confirm` (stderr prompt with configurable timeout), and `~/.agent-desktop/audit.jsonl` append-only log with trace_id, actor, tool, args, decision (allowed / dry-run / denied / confirmed), exit code, timestamp. Covers EU AI Act Article 14 and OWASP Agentic Top-10 (2026) requirements | +| P5-O6 | Policy allowlist / denylist | `~/.agent-desktop/policy.yaml` defines per-tool rules — e.g. "never call `desktop_close_app` for `com.apple.finder`", "require confirm for any action on bundle ID `com.apple.mail`". Loaded at daemon start, reload-on-SIGHUP. Policy decisions land in the audit log | +| P5-O7 | OCR / vision fallback (`find --visual`) | When the AX tree is empty or the target isn't exposed (Canvas apps, Flutter-desktop, games, remote desktop, Figma plugins), `find --visual "label"` falls back to a per-window screenshot + OCR to locate text. macOS: `Vision` framework `VNRecognizeTextRequest`. Windows: `Windows.Media.Ocr.OcrEngine`. Linux: Tesseract via `tesseract` crate. Returns a synthetic ref that routes to coordinate events; clearly marked `source: "visual"` in output to signal reduced reliability | +| P5-O8 | Session trace + OpenTelemetry-compatible event stream | `--trace-id ` on every CLI invocation; generated if not provided. Each command appends structured events (command start, adapter call, action dispatched, exit) to `~/.agent-desktop/traces/{trace_id}.jsonl`. Events are OpenTelemetry-compliant (`span_id`, `trace_id`, `parent_span_id`, `span_name`, `attributes`). New `agent-desktop trace view ` pretty-prints; `trace export ` emits OTLP JSON | +| P5-O9 | Screencast / screenshot-per-action receipt | `--record-trace ` on long-running MCP sessions or CLI batches. Uses Phase 2 P2-O13 modern screenshot APIs at 2 Hz by default. Parity with Playwright 1.59 `page.screencast`. Mutually exclusive with `--dry-run` (nothing to record) | +| P5-O10 | Sigstore attestation verification at install time | `brew install` formula and `winget` manifest run `cosign verify-blob` / `gh attestation verify` against the downloaded tarball before installing. Prevents supply-chain tampering. apt/snap use distro-native signatures; the formula publishes both Sigstore bundle and the checksum | ### Daemon Architecture @@ -1260,6 +1476,16 @@ The daemon is a long-running process that maintains state between CLI/MCP invoca |---------|-------------| | `session list` | List active daemon sessions with IDs, creation time, last activity | | `session kill ` | Terminate a specific daemon session, release its RefMap | +| `trace view ` | Pretty-print a session trace from `~/.agent-desktop/traces/{uuid}.jsonl` | +| `trace export [--otlp \| --har]` | Export a session trace as OpenTelemetry OTLP JSON or HAR for post-mortem inspection | +| `audit tail [--follow]` | Tail `~/.agent-desktop/audit.jsonl`, optionally streaming new entries | +| `audit verify ` | Verify the append-only integrity of an audit log (hash-chain check) | +| `policy check ` | Evaluate the policy file against a would-be command without executing | +| `find --visual "