# CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Common Commands ```bash cargo build # Debug build cargo build --release # Release build (<15MB target) cargo test --workspace # Run every test (lib, binary, and integration targets) cargo test --lib --workspace # Library unit tests only; SKIPS the agent-desktop binary crate cargo test -p agent-desktop # Binary crate tests (CLI contract, dispatch, batch, policy) cargo test --lib -p agent-desktop-core # Test core crate only cargo test --lib -p agent-desktop-macos # Test macOS crate only cargo test test_name # Run a single test by name cargo check -p agent-desktop-core --all-targets --target x86_64-pc-windows-msvc # Core must cross-compile cargo check -p agent-desktop-core --all-targets --target x86_64-unknown-linux-gnu cargo clippy --all-targets -- -D warnings # Lint (must pass, zero warnings) cargo fmt --all -- --check # Format check cargo fmt --all # Auto-format cargo tree -p agent-desktop-core # Verify no platform crate leaks (CI enforces) bash tests/e2e/run.sh # E2E: real binary vs fixture app, verify by observation (needs --release + AX permission) bash scripts/perf-baseline-compare.sh # Perf A/B vs merge-base -> report.html (add --apps "Slack,Google Chrome" for Electron coverage) ``` Run the binary: `./target/release/agent-desktop snapshot --app Finder -i` The E2E harness drives the release binary against a real SwiftUI/AppKit fixture and asserts every effect by independent observation (never the command's own `ok:true`), covering every ref action in **both** headless and `--headed` mode. See `tests/e2e/README.md`. ## Pre-commit Hook The repo ships a pre-commit hook at `.githooks/pre-commit` that runs `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, and `cargo test --lib --workspace` against staged Rust changes. Wire it up once after cloning: ```bash git config core.hooksPath .githooks ``` Bypass for an emergency commit with `git commit --no-verify` or `SKIP_PRECOMMIT=1 git commit ...`. ## Project Overview Cross-platform Rust CLI + MCP server enabling AI agents to observe and control desktop applications via native OS accessibility trees. ## Git & Commits ### Branching during a platform phase (Phase 2 = Windows, in progress) - **`feat/windows-adapter` is the base branch for all Windows work**, not `main`. Every sub-phase (2.0 → 2.15) is cut from it and merges back into it. - Sub-phase branches: `feat/windows--` (e.g. `feat/windows-2.0-probes`), PR'd into `feat/windows-adapter`. - **Never branch Windows work off `main`, never PR a sub-phase into `main`, never rebase a sub-phase onto `main`.** - `main` is the macOS-GA line for the whole phase. It gains Windows exactly once, at the end, when the adapter is production-solid as a whole — one release-noted `feat!` promotion after full-branch review, live e2e, and a perf baseline. Phase 3 repeats this with `feat/linux-adapter`. - Plan docs: historical plans and brainstorms are tracked on `main`. Every sub-phase's plan doc lands on its own sub-phase branch, so each sub-phase PR reviews as one self-contained unit. - See `docs/phases.md` §Platform Delivery Model for the full rule. - All commits are authored by **Lahfir** - NEVER add `Co-Authored-By` lines, AI attribution badges, or "Generated with" footers - NEVER include co-committers of any kind - **Conventional Commits required.** Every commit message must use a type prefix: - `feat:` — new feature (triggers minor version bump) - `fix:` — bug fix (triggers patch version bump) - `feat!:` or `BREAKING CHANGE:` footer — breaking change (triggers major version bump) - `docs:` — documentation only - `style:` — formatting, no code change - `refactor:` — code change that neither fixes a bug nor adds a feature - `perf:` — performance improvement with no behavior change - `chore:` — maintenance tasks, dependencies - `ci:` — CI/CD changes - `test:` — adding or fixing tests - Format: `type: concise imperative description` (lowercase type, no capital after colon) - Focus on "why" not "what" - Examples: `feat: add scroll-to command`, `fix: prevent stale ref on window resize`, `ci: add binary size check` - **Pre-1.0 versioning policy** (release-please `bump-minor-pre-major` + `bump-patch-for-minor-pre-major`): while the version is 0.x, a `BREAKING CHANGE` cuts a **minor** (0.2 → 0.3) and a `feat:` cuts a **patch**. Do not expect a major release before 1.0. ## Core Principle agent-desktop is NOT an AI agent. It is a tool that AI agents invoke. It outputs structured JSON with ref-based element identifiers. The observation-action loop lives in the calling agent. ## Architecture ### Workspace Layout ``` agent-desktop/ ├── Cargo.toml # workspace: members, shared deps ├── CONCEPTS.md # shared domain vocabulary for refs, snapshots, sessions, actionability, and related concepts ├── rust-toolchain.toml # pinned Rust version ├── clippy.toml # project-wide lint config ├── crates/ │ ├── core/ # agent-desktop-core (platform-agnostic) │ │ └── src/ │ │ ├── ref_alloc.rs # Shared ref helpers (INTERACTIVE_ROLES, is_collapsible) │ │ ├── snapshot_ref.rs # Ref-rooted drill-down (run_from_ref) │ │ └── commands/ # one file per command │ ├── macos/ # agent-desktop-macos (Phase 1) │ ├── windows/ # agent-desktop-windows (stub → Phase 2) │ ├── linux/ # agent-desktop-linux (stub → Phase 2) │ └── ffi/ # agent-desktop-ffi (cdylib + committed C ABI header) ├── src/ # agent-desktop binary (entry point) │ ├── main.rs # entry point, permission check, JSON envelope │ ├── batch/ # batch JSON → typed Commands │ ├── cli/ # clap derive enum, help text, CLI contract tests │ ├── cli_args/ # command argument structs by domain │ ├── command_policy/ # permission/ref/side-effect policy │ ├── dispatch/ # command dispatcher, parse helpers, notifications │ └── tests/ # binary-level conformance tests ├── docs/ │ └── solutions/ # documented solutions to past problems (bugs, best practices, workflow patterns), organized by category with YAML frontmatter (module, tags, problem_type); relevant when implementing or debugging in documented areas └── tests/ ├── fixtures/ # golden JSON snapshots └── integration/ # macOS CI integration tests ``` ### Dependency Inversion (Non-Negotiable) - `agent-desktop-core` defines the `PlatformAdapter` trait and all shared types - Platform crates (`macos`, `windows`, `linux`) implement the trait - **Core NEVER imports platform crates.** Platform crates NEVER import each other. - Two legitimate wiring points bring platform → core together: 1. The binary crate (`src/`) — CLI consumers 2. The FFI crate (`crates/ffi/`) — cdylib consumers (Python, Swift, Go, Node, C++) - CI enforces core isolation: `cargo tree -p agent-desktop-core` must contain zero platform crate names ### Platform Selection Compile-time via `#[cfg(target_os)]` in `build_adapter()`. Agents never specify platform — `agent-desktop snapshot -i` works identically on macOS, Windows, and Linux. ```rust fn build_adapter() -> impl PlatformAdapter { #[cfg(target_os = "macos")] { agent_desktop_macos::MacOSAdapter::new() } #[cfg(target_os = "windows")] { agent_desktop_windows::WindowsAdapter::new() } #[cfg(target_os = "linux")] { agent_desktop_linux::LinuxAdapter::new() } } ``` ### Target-Gated Dependencies Binary crate `Cargo.toml` uses platform-specific deps, NOT unconditional deps with `#[cfg]` in source: ```toml [target.'cfg(target_os = "macos")'.dependencies] agent-desktop-macos = { path = "crates/macos" } [target.'cfg(target_os = "windows")'.dependencies] agent-desktop-windows = { path = "crates/windows" } [target.'cfg(target_os = "linux")'.dependencies] agent-desktop-linux = { path = "crates/linux" } ``` ### Command Dispatch Direct `match` in the binary crate. No `Command` trait, no `CommandRegistry`. Each command is a standalone `execute()` function under `crates/core/src/commands/`. ```rust pub fn dispatch( cmd: Commands, adapter: &dyn PlatformAdapter, permission_report: &PermissionReport, ) -> Result { match cmd { Commands::Snapshot(args) => commands::snapshot::execute(args, adapter), Commands::Click(args) => commands::click::execute(args, adapter), // one arm per command } } ``` Batch is not a second dispatcher. `src/batch/mod.rs` deserializes JSON entries into the same typed `Commands` enum, runs the same `CommandPolicy` preflight, and calls the same `dispatch()` path as CLI. ### Additive Phase Model Numbering follows `docs/phases.md` (the source of truth for phase scope): - **Phase 1 / 1.5 / 1.6 (completed):** Foundation + macOS MVP, FFI cdylib distribution, and the Playwright-grade foundation contract (capability supertraits, auto-wait, occlusion gate, live locator, ProcessState, envelope 2.1) - **Phase 2:** Windows adapter — delivered as sub-phases 2.0–2.15, beginning with a raw-script platform exploration phase, each a <=2,000-LOC PR into the `feat/windows-adapter` integration branch - **Phase 3:** Linux adapter — same sub-phase template onto `feat/linux-adapter` - **Phase 4:** MCP server mode via `--mcp` flag — wraps existing commands - **Phase 5:** Daemon, sessions, enterprise quality gates Later phases add adapters, transports, and production readiness work. Nothing in core is rebuilt; platform work implements into the settled contract. ## Coding Standards ### File Rules - **400 LOC hard limit per hand-written file.** If approaching 400, split by responsibility. The committed C header at `crates/ffi/include/agent_desktop.h` is generated by cbindgen and verified by the `ffi-header-drift` CI job; change the Rust ABI declarations or cbindgen configuration and regenerate it rather than hand-editing generated declarations. - **No inline comments.** Code must be self-documenting through naming. Only Rust doc-comments (`///`) on public items when the name alone is insufficient. - **One struct/enum per file** for domain types. `node.rs` defines `AccessibilityNode`. `action.rs` defines `Action`. - **One command per file.** Each CLI command lives in its own file under `commands/`. Filename matches the command name. - **No God objects.** No struct with more than 7 fields. No function with more than 5 parameters. Use builder patterns or config structs. - **Explicit pub boundaries.** Only `lib.rs` re-exports public items. Internal modules use `pub(crate)`. No wildcard re-exports. ### Error Handling - **Zero `unwrap()` in non-test code.** All `Result`s propagated with `?` or matched explicitly. Panics are test-only. - Every error carries: `ErrorCode` enum (machine-readable), `message: String` (human-readable), `suggestion: Option` (recovery hint), `platform_detail: Option` (OS-specific detail) - All platform adapter functions return `Result` - All command handlers return `Result` - The binary's `main()` converts `AppError` to JSON and sets the exit code ### Error Codes ``` PERM_DENIED, ELEMENT_NOT_FOUND, APP_NOT_FOUND, ACTION_FAILED, ACTION_NOT_SUPPORTED, STALE_REF, AMBIGUOUS_TARGET, WINDOW_NOT_FOUND, PLATFORM_NOT_SUPPORTED, TIMEOUT, INVALID_ARGS, NOTIFICATION_NOT_FOUND, SNAPSHOT_NOT_FOUND, POLICY_DENIED, APP_UNRESPONSIVE, INTERNAL ``` ### Exit Codes - `0` — success - `1` — structured error (JSON with error code) - `2` — argument/parse error ### Naming Conventions | Element | Convention | Example | |---------|-----------|---------| | Crate names | `agent-desktop-{name}` | `agent-desktop-core`, `agent-desktop-macos` | | Module files | `snake_case`, singular | `snapshot.rs`, `list_windows.rs` | | Structs | PascalCase, descriptive noun | `SnapshotEngine`, `RefAllocator` | | Traits | PascalCase, adjective/capability | `PlatformAdapter`, `Executable` | | Enums | PascalCase, variants PascalCase | `Action::Click`, `ErrorCode::PermDenied` | | Functions | `snake_case`, verb-first | `build_tree()`, `allocate_refs()` | | Constants | `SCREAMING_SNAKE_CASE` | `MAX_TREE_DEPTH`, `DEFAULT_TIMEOUT_MS` | | CLI flags | kebab-case | `--max-depth`, `--include-bounds` | | Ref IDs | `@:e{n}` qualified output | `@s8f3k2p9:e1`, `@s8f3k2p9:e14` | ### Platform Crate Folder Structure All platform crates (`macos`, `windows`, `linux`) follow an identical subfolder layout. New files must be placed in the correct subfolder. ``` crates/{macos,windows,linux}/src/ ├── lib.rs # mod declarations + re-exports only ├── adapter.rs # PlatformAdapter trait impl (~175 LOC) ├── tree/ # Reading & understanding the UI │ ├── mod.rs # re-exports │ ├── element.rs # AXElement struct + attribute readers │ ├── capabilities.rs # AX-supported actions and settable attributes │ ├── builder.rs # build_subtree, tree traversal │ ├── roles.rs # Role mapping │ ├── resolve.rs # Element re-identification │ └── surfaces.rs # Surface detection ├── actions/ # Interacting with elements │ ├── mod.rs # re-exports │ ├── dispatch.rs # perform_action match arms │ ├── activate.rs # Smart AX-first activation chain │ ├── extras.rs # select_value helpers │ ├── scroll.rs # scroll semantics and gated physical fallback │ └── type_text.rs # headless text insertion and physical typing ├── input/ # Low-level OS input synthesis │ ├── mod.rs # re-exports │ ├── keyboard.rs # Key synthesis, text typing │ ├── mouse.rs # Mouse events │ └── clipboard.rs # Clipboard get/set └── system/ # App lifecycle, windows, permissions ├── mod.rs # re-exports ├── app_ops.rs # launch, close, focus ├── window_ops.rs # window operations ├── key_dispatch.rs # app-targeted key press ├── permissions.rs # permission checks ├── screenshot.rs # screen capture └── wait.rs # wait utilities ``` **Placement rules:** - Tree reading/traversal/resolution → `tree/` - Element interaction/activation → `actions/` - Raw OS input (keyboard, mouse, clipboard) → `input/` - App lifecycle, windows, permissions, screenshots → `system/` - `adapter.rs` stays at root — it's the PlatformAdapter impl that wires everything together ### Extensibility Pattern Adding a new command requires exactly these steps: 1. Create `crates/core/src/commands/{name}.rs` with an `execute()` function 2. Register it in `crates/core/src/commands/mod.rs` 3. Add the CLI subcommand variant to `src/cli/mod.rs` and arguments under `src/cli_args/` 4. Add a match arm in `dispatch()` in the binary crate 5. If new `Action` variant needed, add to `crates/core/src/action.rs` 6. If new adapter method needed, add to `PlatformAdapter` trait with a default returning `Err(AdapterError::not_supported())` No existing files are modified beyond the registration points. Enforce via code review. ## JSON Output Contract Every command produces a response envelope: ```json { "version": "2.1", "ok": true, "command": "snapshot", "data": { "app": "Finder", "window": { "id": "w-4521", "title": "Documents" }, "ref_count": 14, "tree": { ... } } } ``` Error responses: ```json { "version": "2.1", "ok": false, "command": "click", "error": { "code": "STALE_REF", "message": "Element could not be resolved from the requested snapshot", "suggestion": "Run 'snapshot' to refresh, then retry with updated ref", "recovery": { "strategy": "refresh_snapshot_then_retry_original", "retryable": true, "requires_fresh_snapshot": true }, "disposition": { "delivery": "not_delivered", "retry": "safe" } } } ``` The `error` object may also carry optional `details` and `recovery` objects. Version 2.1 removed the 2.0 `retry_command` string without a compatibility alias; consumers must use `recovery.strategy` only when `disposition.retry` is `safe`. ### Serialization Rules - Omit null/None fields (`#[serde(skip_serializing_if = "Option::is_none")]`) - Omit empty arrays (`#[serde(skip_serializing_if = "Vec::is_empty")]`) - Omit bounds in compact mode - `ref_count` and `tree` go inside `data`, not as top-level siblings ## Ref System - Refs are allocated in depth-first document order and emitted as snapshot-qualified IDs such as `@s8f3k2p9:e1`. Legacy bare IDs such as `@e1` remain valid input only with an explicit `--snapshot s8f3k2p9`. - An element receives a ref when it is **addressable for an action**: its role is interactive (`button`, `textfield`, `checkbox`, `link`, `menuitem`, `tab`, `slider`, `combobox`, `treeitem`, `cell`, `radiobutton`, `switch`, `colorwell`, `menubutton`, `incrementor`, `dockitem`), **or** it advertises an available action regardless of role. Container roles such as `scrollarea` (Scroll) and `disclosure` (Expand/Collapse/Click) are not interactive by role but are genuinely actionable, so they are ref-able — `scroll` / `expand` / `collapse` need a ref to target them - A bare `SetFocus` affordance does not qualify on its own (focusability is not a primary action), so inert focusable containers stay ref-less - Static text and non-actionable groups/containers do NOT get refs (they remain in tree for context) - Refs are deterministic within a snapshot but NOT stable across snapshots if UI changed - Snapshot refs are stored by snapshot ID under `~/.agent-desktop/snapshots/{snapshot_id}/refmap.json`, with a `latest_snapshot_id` pointer for commands that omit `--snapshot` - `~/.agent-desktop/last_refmap.json` is written only as a latest-snapshot inspection artifact; command code must use `RefStore` - Action commands use strict re-identification from platform-neutral `RefEntry` evidence: pid, role, path/source surface, role-conditional stable text identity, and bounds hash. Mutable control values are volatile and must not be treated as stable text identity. Return `STALE_REF` on mismatch and `AMBIGUOUS_TARGET` when multiple plausible live candidates remain. - Progressive traversal: `--skeleton` clamps depth to 3, annotates truncated containers with `children_count`. Named/described containers at boundary receive refs as drill-down targets - Drill-down: `--root @ref` starts from a previously-discovered ref with scoped invalidation (only that ref's subtree refs are replaced on re-drill) - RefMap size check: write-side guard prevents >1MB refmap files - **Sessions:** `session start` creates and returns a manifest-gated session under `~/.agent-desktop/sessions//` and enables automatic trace segments by default. It does not activate that session for later processes. Pass the returned ID through `--session` or `AGENT_DESKTOP_SESSION`. Bare `--session ` without a manifest scopes only the snapshot namespace — no surprise trace files. - **Trace:** manifest `trace: on` writes per-process JSONL segments under `/trace/-.jsonl`; `--trace ` overrides to one file; activation resolves `--session` > `AGENT_DESKTOP_SESSION` > no session. Snapshot lookup is confined to that selected namespace and never searches other sessions. ## PlatformAdapter Trait Core defines `PlatformAdapter`; platform crates implement it. Methods default to `not_supported()`, so an adapter only implements what it supports. Read the current signatures in `crates/core/src/adapter/mod.rs` and its `actions.rs`, `input.rs`, `observation.rs`, and `system.rs` capability fragments — notably strict resolution (`resolve_element_strict*` → STALE_REF on 0, AMBIGUOUS_TARGET on 2+), live reads for the actionability preflight (`get_live_*`), and `is_protected_process` (keeps platform-specific process names out of core). ## macOS Adapter Gotchas - **Ancestor-path set, not a global visited set** — macOS reuses `AXUIElementRef` pointers across sibling branches, so a global visited set would prune real subtrees. - **`AXElement` memory safety** — inner field is `pub(crate)` (prevents double-free via raw pointer extraction); `Clone` must `CFRetain`, `Drop` must `CFRelease`. - **Batch attribute reads** — use `AXUIElementCopyMultipleAttributeValues` (3-5x faster than per-attribute fetches). ## Testing - Unit tests use an in-memory `MockAdapter`; golden fixtures in `tests/fixtures/` regression-test serialization. - macOS CI integration tests drive real apps (Finder, TextEdit, System Settings). - `tests/e2e/run.sh` drives the release binary against the SwiftUI fixture and verifies every effect by independent observation in both headless and `--headed` mode (see `tests/e2e/README.md`). ## CI Requirements - GitHub Actions macOS runner executes full test suite on every PR - Windows and Linux runners execute the core unit tests plus their native platform crate on every PR - `cargo tree -p agent-desktop-core` must not contain platform crate names - `cargo clippy --all-targets -- -D warnings` - `cargo test --workspace` - Binary size check: fail if release binary exceeds 15MB ### Core platform-conditional code `agent-desktop-core` must stay executable on every supported OS. A `#[cfg]` branch in core that no CI lane runs is a hypothesis, not shipped code — either add a lane that executes it or keep it out of core. Platform hardening belongs behind `PlatformAdapter` or in the platform crate, written on that platform against a lane that runs it. Core carried 1,062 LOC of never-executed Win32 file I/O into v0.5.0; it failed 225 of 940 tests on first contact with Windows and was deleted. See `docs/solutions/best-practices/never-ship-platform-code-that-ci-cannot-execute.md`. ## Commands 58 commands spanning App/Window, Observation, Interaction, Scroll, Keyboard, Mouse, Notifications (macOS), Clipboard, Wait, System (including `session`), and Batch. The full surface and per-command reference live in `skills/agent-desktop/`. All 58 are implemented on macOS (Phase 1); Windows/Linux (Phase 2/3) target the same surface. Adding a command: see the Extensibility Pattern above. ## Non-Goals - Does NOT embed or invoke LLMs - Does NOT provide a GUI, TUI, or interactive prompt — machine-facing only - Does NOT automate web browsers (use agent-browser for that) - Does NOT record or replay macros (stateless per invocation until Phase 4 daemon) - Does NOT work with custom-rendered or game-engine UIs lacking accessibility exposure ## Reference Documents - PRD v2.0: `docs/agent_desktop_prd_v2.pdf` - Architecture Brainstorm: `docs/brainstorms/2026-02-19-architecture-validation-brainstorm.md` - Phase 1 Plan: `docs/plans/2026-02-19-feat-agent-desktop-phase1-foundation-plan.md` ## Definition of Done: Performance Baseline Every substantive change ends with a performance baseline check before merge: run `bash scripts/perf-baseline-compare.sh` (optionally `--apps "Slack,Google Chrome"` for dense Electron/Chromium coverage) and review the generated `report.html` against the merge-base. Latency deltas must be intentional and explainable — never discovered by users.