agent-desktop/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Common Commands

cargo build                                    # Debug build
cargo build --release                          # Release build (<15MB target)
cargo test --lib --workspace                   # Run all unit tests
cargo test --lib -p agent-desktop-core         # Test core crate only
cargo test --lib -p agent-desktop-macos        # Test macOS crate only
cargo test test_name                           # Run a single test by name
cargo clippy --all-targets -- -D warnings      # Lint (must pass, zero warnings)
cargo fmt --all -- --check                     # Format check
cargo fmt --all                                # Auto-format
cargo tree -p agent-desktop-core               # Verify no platform crate leaks (CI enforces)

Run the binary: ./target/release/agent-desktop snapshot --app Finder -i

Pre-commit Hook

The repo ships a pre-commit hook at .githooks/pre-commit that runs cargo fmt --check, cargo clippy --all-targets -- -D warnings, and cargo test --lib --workspace against staged Rust changes. Wire it up once after cloning:

git config core.hooksPath .githooks

Bypass for an emergency commit with git commit --no-verify or SKIP_PRECOMMIT=1 git commit ....

Project Overview

Cross-platform Rust CLI + MCP server enabling AI agents to observe and control desktop applications via native OS accessibility trees.

Git & Commits

• All commits are authored by Lahfir
• NEVER add Co-Authored-By lines, AI attribution badges, or "Generated with" footers
• NEVER include co-committers of any kind
• Conventional Commits required. Every commit message must use a type prefix:
  • feat: — new feature (triggers minor version bump)
  • fix: — bug fix (triggers patch version bump)
  • feat!: or BREAKING CHANGE: footer — breaking change (triggers major version bump)
  • docs: — documentation only
  • style: — formatting, no code change
  • refactor: — code change that neither fixes a bug nor adds a feature
  • chore: — maintenance tasks, dependencies
  • ci: — CI/CD changes
  • test: — adding or fixing tests
• Format: type: concise imperative description (lowercase type, no capital after colon)
• Focus on "why" not "what"
• Examples: feat: add scroll-to command, fix: prevent stale ref on window resize, ci: add binary size check

Core Principle

agent-desktop is NOT an AI agent. It is a tool that AI agents invoke. It outputs structured JSON with ref-based element identifiers. The observation-action loop lives in the calling agent.

Architecture

Workspace Layout

agent-desktop/
├── Cargo.toml              # workspace: members, shared deps
├── rust-toolchain.toml     # pinned Rust version
├── clippy.toml             # project-wide lint config
├── crates/
│   ├── core/               # agent-desktop-core (platform-agnostic)
│   │   └── src/
│   │       ├── ref_alloc.rs      # Shared ref helpers (INTERACTIVE_ROLES, is_collapsible)
│   │       ├── snapshot_ref.rs   # Ref-rooted drill-down (run_from_ref)
│   │       └── commands/         # one file per command
│   ├── macos/              # agent-desktop-macos (Phase 1)
│   ├── windows/            # agent-desktop-windows (stub → Phase 2)
│   ├── linux/              # agent-desktop-linux (stub → Phase 2)
│   └── ffi/                # agent-desktop-ffi (cdylib + committed C ABI header)
├── src/                    # agent-desktop binary (entry point)
│   ├── main.rs             # entry point, permission check, JSON envelope
│   ├── cli.rs              # clap derive enum (Commands)
│   ├── cli_args.rs         # all command argument structs
│   ├── batch.rs            # batch JSON → typed Commands
│   ├── command_policy.rs   # command permission/ref/side-effect policy
│   ├── dispatch.rs         # command dispatcher + parse helpers
│   └── dispatch_notifications.rs
├── docs/
│   └── solutions/          # documented solutions to past problems (bugs, best practices, workflow patterns), organized by category with YAML frontmatter (module, tags, problem_type); relevant when implementing or debugging in documented areas
└── tests/
    ├── fixtures/           # golden JSON snapshots
    └── integration/        # macOS CI integration tests

Dependency Inversion (Non-Negotiable)

• agent-desktop-core defines the PlatformAdapter trait and all shared types
• Platform crates (macos, windows, linux) implement the trait
• Core NEVER imports platform crates. Platform crates NEVER import each other.
• Two legitimate wiring points bring platform → core together:
  1. The binary crate (src/) — CLI consumers
  2. The FFI crate (crates/ffi/) — cdylib consumers (Python, Swift, Go, Node, C++)
• CI enforces core isolation: cargo tree -p agent-desktop-core must contain zero platform crate names

Platform Selection

Compile-time via #[cfg(target_os)] in build_adapter(). Agents never specify platform — agent-desktop snapshot -i works identically on macOS, Windows, and Linux.

fn build_adapter() -> impl PlatformAdapter {
    #[cfg(target_os = "macos")]
    { agent_desktop_macos::MacOSAdapter::new() }

    #[cfg(target_os = "windows")]
    { agent_desktop_windows::WindowsAdapter::new() }

    #[cfg(target_os = "linux")]
    { agent_desktop_linux::LinuxAdapter::new() }
}

Target-Gated Dependencies

Binary crate Cargo.toml uses platform-specific deps, NOT unconditional deps with #[cfg] in source:

[target.'cfg(target_os = "macos")'.dependencies]
agent-desktop-macos = { path = "crates/macos" }

[target.'cfg(target_os = "windows")'.dependencies]
agent-desktop-windows = { path = "crates/windows" }

[target.'cfg(target_os = "linux")'.dependencies]
agent-desktop-linux = { path = "crates/linux" }

Command Dispatch

Direct match in the binary crate. No Command trait, no CommandRegistry. Each command is a standalone execute() function under crates/core/src/commands/.

pub fn dispatch(
    cmd: Commands,
    adapter: &dyn PlatformAdapter,
    permission_report: &PermissionReport,
) -> Result<serde_json::Value, AppError> {
    match cmd {
        Commands::Snapshot(args) => commands::snapshot::execute(args, adapter),
        Commands::Click(args) => commands::click::execute(args, adapter),
        // one arm per command
    }
}

Batch is not a second dispatcher. src/batch.rs deserializes JSON entries into the same typed Commands enum, runs the same CommandPolicy preflight, and calls the same dispatch() path as CLI.

Additive Phase Model

• Phase 1: Foundation + macOS MVP (54 commands, core engine, macOS adapter)
• Phase 2: Windows + Linux adapters, 10+ new commands — core untouched
• Phase 3: MCP server mode via --mcp flag — wraps existing commands
• Phase 4: Daemon, sessions, enterprise quality gates

Phases 2–4 add adapters, transports, and production readiness work. Nothing in core is rebuilt.

Coding Standards

File Rules

• 400 LOC hard limit per file. If approaching 400, split by responsibility. No exceptions.
• No inline comments. Code must be self-documenting through naming. Only Rust doc-comments (///) on public items when the name alone is insufficient.
• One struct/enum per file for domain types. node.rs defines AccessibilityNode. action.rs defines Action.
• One command per file. Each CLI command lives in its own file under commands/. Filename matches the command name.
• No God objects. No struct with more than 7 fields. No function with more than 5 parameters. Use builder patterns or config structs.
• Explicit pub boundaries. Only lib.rs re-exports public items. Internal modules use pub(crate). No wildcard re-exports.

Error Handling

• Zero unwrap() in non-test code. All Results propagated with ? or matched explicitly. Panics are test-only.
• Every error carries: ErrorCode enum (machine-readable), message: String (human-readable), suggestion: Option<String> (recovery hint), platform_detail: Option<String> (OS-specific detail)
• All platform adapter functions return Result<T, AdapterError>
• All command handlers return Result<serde_json::Value, AppError>
• The binary's main() converts AppError to JSON and sets the exit code

Error Codes

PERM_DENIED, ELEMENT_NOT_FOUND, APP_NOT_FOUND, ACTION_FAILED,
ACTION_NOT_SUPPORTED, STALE_REF, AMBIGUOUS_TARGET, WINDOW_NOT_FOUND,
PLATFORM_NOT_SUPPORTED, TIMEOUT, INVALID_ARGS, NOTIFICATION_NOT_FOUND,
SNAPSHOT_NOT_FOUND, POLICY_DENIED, INTERNAL

Exit Codes

• 0 — success
• 1 — structured error (JSON with error code)
• 2 — argument/parse error

Naming Conventions

Element	Convention	Example
Crate names	agent-desktop-{name}	agent-desktop-core, agent-desktop-macos
Module files	snake_case, singular	snapshot.rs, list_windows.rs
Structs	PascalCase, descriptive noun	SnapshotEngine, RefAllocator
Traits	PascalCase, adjective/capability	PlatformAdapter, Executable
Enums	PascalCase, variants PascalCase	Action::Click, ErrorCode::PermDenied
Functions	snake_case, verb-first	build_tree(), allocate_refs()
Constants	SCREAMING_SNAKE_CASE	MAX_TREE_DEPTH, DEFAULT_TIMEOUT_MS
CLI flags	kebab-case	--max-depth, --include-bounds
Ref IDs	@e{n} sequential	@e1, @e2, @e14

Platform Crate Folder Structure

All platform crates (macos, windows, linux) follow an identical subfolder layout. New files must be placed in the correct subfolder.

crates/{macos,windows,linux}/src/
├── lib.rs              # mod declarations + re-exports only
├── adapter.rs          # PlatformAdapter trait impl (~175 LOC)
├── tree/               # Reading & understanding the UI
│   ├── mod.rs          # re-exports
│   ├── element.rs      # AXElement struct + attribute readers
│   ├── capabilities.rs # AX-supported actions and settable attributes
│   ├── builder.rs      # build_subtree, tree traversal
│   ├── roles.rs        # Role mapping
│   ├── resolve.rs      # Element re-identification
│   └── surfaces.rs     # Surface detection
├── actions/            # Interacting with elements
│   ├── mod.rs          # re-exports
│   ├── dispatch.rs     # perform_action match arms
│   ├── activate.rs     # Smart AX-first activation chain
│   ├── extras.rs       # select_value helpers
│   ├── scroll.rs       # scroll semantics and gated physical fallback
│   └── type_text.rs    # headless text insertion and physical typing
├── input/              # Low-level OS input synthesis
│   ├── mod.rs          # re-exports
│   ├── keyboard.rs     # Key synthesis, text typing
│   ├── mouse.rs        # Mouse events
│   └── clipboard.rs    # Clipboard get/set
└── system/             # App lifecycle, windows, permissions
    ├── mod.rs          # re-exports
    ├── app_ops.rs      # launch, close, focus
    ├── window_ops.rs   # window operations
    ├── key_dispatch.rs # app-targeted key press
    ├── permissions.rs  # permission checks
    ├── screenshot.rs   # screen capture
    └── wait.rs         # wait utilities

Placement rules:

• Tree reading/traversal/resolution → tree/
• Element interaction/activation → actions/
• Raw OS input (keyboard, mouse, clipboard) → input/
• App lifecycle, windows, permissions, screenshots → system/
• adapter.rs stays at root — it's the PlatformAdapter impl that wires everything together

Extensibility Pattern

Adding a new command requires exactly these steps:

1. Create crates/core/src/commands/{name}.rs with an execute() function
2. Register it in crates/core/src/commands/mod.rs
3. Add the CLI subcommand variant to src/cli.rs (clap derive enum)
4. Add a match arm in dispatch() in the binary crate
5. If new Action variant needed, add to crates/core/src/action.rs
6. If new adapter method needed, add to PlatformAdapter trait with a default returning Err(AdapterError::not_supported())

No existing files are modified beyond the registration points. Enforce via code review.

JSON Output Contract

Every command produces a response envelope:

{
  "version": "2.0",
  "ok": true,
  "command": "snapshot",
  "data": {
    "app": "Finder",
    "window": { "id": "w-4521", "title": "Documents" },
    "ref_count": 14,
    "tree": { ... }
  }
}

Error responses:

{
  "version": "2.0",
  "ok": false,
  "command": "click",
  "error": {
    "code": "STALE_REF",
    "message": "Element could not be resolved from the requested snapshot",
    "suggestion": "Run 'snapshot' to refresh, then retry with updated ref"
  }
}

Serialization Rules

• Omit null/None fields (#[serde(skip_serializing_if = "Option::is_none")])
• Omit empty arrays (#[serde(skip_serializing_if = "Vec::is_empty")])
• Omit bounds in compact mode
• ref_count and tree go inside data, not as top-level siblings

Ref System

• Refs allocated in depth-first document order: @e1, @e2, etc.
• Only interactive roles receive refs: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell
• Static text, groups, containers do NOT get refs (they remain in tree for context)
• Refs are deterministic within a snapshot but NOT stable across snapshots if UI changed
• Snapshot refs are stored by snapshot ID under ~/.agent-desktop/snapshots/{snapshot_id}/refmap.json, with a latest_snapshot_id pointer for commands that omit --snapshot
• ~/.agent-desktop/last_refmap.json is written only as a latest-snapshot inspection artifact; command code must use RefStore
• Action commands use optimistic re-identification: (pid, role, name, bounds_hash). Return STALE_REF on mismatch.
• Progressive traversal: --skeleton clamps depth to 3, annotates truncated containers with children_count. Named/described containers at boundary receive refs as drill-down targets
• Drill-down: --root @ref starts from a previously-discovered ref with scoped invalidation (only that ref's subtree refs are replaced on re-drill)
• RefMap size check: write-side guard prevents >1MB refmap files

PlatformAdapter Trait

Platform-facing methods default to not_supported() unless implemented by an adapter:

pub trait PlatformAdapter {
    fn list_windows(&self, filter: &WindowFilter) -> Result<Vec<WindowInfo>, AdapterError>;
    fn list_apps(&self) -> Result<Vec<AppInfo>, AdapterError>;
    fn get_tree(&self, win: &WindowInfo, opts: &TreeOptions) -> Result<AccessibilityNode, AdapterError>;
    fn get_subtree(&self, handle: &NativeHandle, opts: &TreeOptions) -> Result<AccessibilityNode, AdapterError>;
    fn execute_action(&self, handle: &NativeHandle, request: ActionRequest) -> Result<ActionResult, AdapterError>;
    fn resolve_element(&self, entry: &RefEntry) -> Result<NativeHandle, AdapterError>;
    fn permission_report(&self) -> PermissionReport;
    fn request_permissions(&self) -> PermissionReport;
    fn focus_window(&self, win: &WindowInfo) -> Result<(), AdapterError>;
    fn launch_app(&self, id: &str, wait: bool) -> Result<WindowInfo, AdapterError>;
    fn close_app(&self, id: &str, force: bool) -> Result<(), AdapterError>;
    fn screenshot(&self, target: ScreenshotTarget) -> Result<ImageBuffer, AdapterError>;
    fn get_clipboard(&self) -> Result<String, AdapterError>;
    fn set_clipboard(&self, text: &str) -> Result<(), AdapterError>;
}

Key Types

• AccessibilityNode — platform-agnostic tree node: ref, role, name, value, description, states, available_actions, bounds, children
• Action — Click, DoubleClick, RightClick, SetValue(String), SetFocus, Expand, Collapse, Select(String), Toggle, Scroll(Direction, Amount), PressKey(KeyCombo)
• ActionRequest — { action, policy }, where the default InteractionPolicy forbids focus stealing and cursor movement
• NativeHandle — opaque platform pointer with PhantomData<*const ()> to prevent auto-Send/Sync. Inner field is pub(crate).
• RefEntry — { pid, role, name, bounds_hash, available_actions }
• WindowInfo — { id, title, app_name, pid, bounds }
• PermissionReport — { accessibility, screen_recording, automation }, each { "state": "granted" }, { "state": "denied", "suggestion": "..." }, or { "state": "unknown" }
• ErrorCode — machine-readable enum with #[serde(rename_all = "SCREAMING_SNAKE_CASE")]
• AdapterError — struct with code, message, suggestion, platform_detail
• AppError — enum with #[from] impls for AdapterError, std::io::Error, serde_json::Error

macOS Adapter (Phase 1)

Tree Traversal

• Entry: AXUIElementCreateApplication(pid) for app root
• Children: kAXChildrenAttribute recursively with ancestor-path set (not global visited set — macOS reuses AXUIElementRef pointers across sibling branches)
• Use AXUIElementCopyMultipleAttributeValues for batch attribute fetch (3-5x faster)
• Role mapping: AXRole strings → unified role enum in tree/roles.rs
• Max depth default: 10. Configurable via --max-depth

Action Execution

• Click: AXUIElementPerformAction(kAXPressAction)
• SetValue: AXUIElementSetAttributeValue(kAXValueAttribute, value)
• SetFocus: AXUIElementSetAttributeValue(kAXFocusedAttribute, true)
• Keyboard/Mouse: CGEventCreateKeyboardEvent / CGEventCreateMouseEvent
• Clipboard: NSPasteboard.generalPasteboard via Cocoa FFI
• Screenshot: ScreenshotBackend boundary with secure screencapture temp files

Permission Detection

• Call AXIsProcessTrusted() on startup
• If false, return PERM_DENIED with guidance: "Open System Settings > Privacy > Accessibility and add your terminal"
• Optionally call AXIsProcessTrustedWithOptions(prompt: true) to trigger system dialog

AXElement Safety

• Inner field: pub(crate) not pub (prevents double-free via raw pointer extraction)
• Clone impl must call CFRetain
• Drop impl must call CFRelease

Testing Strategy

Unit Tests (core)

• AccessibilityNode ser/de roundtrips
• Ref allocator only assigns interactive roles
• SnapshotEngine filtering
• Error serialization
• MockAdapter: in-memory PlatformAdapter returning hardcoded trees

Golden Fixtures (tests/fixtures/)

• Real snapshots from Finder, TextEdit, etc. checked into repo
• Regression-test serialization format changes

Integration Tests (macOS CI)

• Snapshot Finder, TextEdit, System Settings — non-empty trees with refs
• Click button in test app — verify action succeeded
• Type text into TextEdit via ref — verify content changed
• Clipboard get/set roundtrip
• Permission denied scenario — correct error code and guidance
• Large tree (Xcode) snapshot in under 2 seconds

Dependencies (Phase 1)

Crate	Version	Purpose
clap	4.x	CLI parsing with derive macros
serde + serde_json	1.x	JSON serialization
thiserror	2.x	Error derive macros
tracing	0.1+	Structured logging
base64	0.22+	Screenshot encoding
accessibility-sys	0.1+	macOS AXUIElement FFI
core-foundation	0.10+	macOS CF types
core-graphics	0.24+	macOS CG types

Deferred Dependencies

• tokio — Phase 2/3 (all Phase 1 ops are synchronous)
• rmcp (0.15.0) — Phase 3 (MCP server)
• schemars — Phase 3 (JSON Schema generation)
• uiautomation (0.24+) — Phase 2 (Windows)
• atspi (0.28+) + zbus (5.x) — Phase 2 (Linux)

Build Configuration

[profile.release]
opt-level = "z"
lto = true
codegen-units = 1
strip = true
panic = "abort"

Target binary size: <15MB per platform.

CI Requirements

• GitHub Actions macOS runner executes full test suite on every PR
• cargo tree -p agent-desktop-core must not contain platform crate names
• cargo clippy --all-targets -- -D warnings
• cargo test --workspace
• Binary size check: fail if release binary exceeds 15MB

Implemented Commands (54)

Platform note: All 54 commands are implemented on macOS (Phase 1). Windows and Linux adapters are planned (Phase 2/3) and will support the same command surface; notification commands depend on platform-specific notification APIs.

Category	Commands
App/Window (10)	launch, close-app, list-windows, list-apps, focus-window, resize-window, move-window, minimize, maximize, restore
Observation (6)	snapshot, screenshot, find, get, is, list-surfaces
Interaction (14)	click, double-click, triple-click, right-click, type, set-value, clear, focus, select, toggle, check, uncheck, expand, collapse
Scroll (2)	scroll, scroll-to
Keyboard (3)	press, key-down, key-up
Mouse (6)	hover, drag, mouse-move, mouse-click, mouse-down, mouse-up
Notifications (4) (macOS)	list-notifications, dismiss-notification, dismiss-all-notifications, notification-action
Clipboard (3)	clipboard-get, clipboard-set, clipboard-clear
Wait (1)	wait (with --element, --window, --text, --menu, --notification flags)
System (4)	status, permissions, version, skills
Batch (1)	batch

Non-Goals

• Does NOT embed or invoke LLMs
• Does NOT provide a GUI, TUI, or interactive prompt — machine-facing only
• Does NOT automate web browsers (use agent-browser for that)
• Does NOT record or replay macros (stateless per invocation until Phase 4 daemon)
• Does NOT work with custom-rendered or game-engine UIs lacking accessibility exposure

Reference Documents

• PRD v2.0: docs/agent_desktop_prd_v2.pdf
• Architecture Brainstorm: docs/brainstorms/2026-02-19-architecture-validation-brainstorm.md
• Phase 1 Plan: docs/plans/2026-02-19-feat-agent-desktop-phase1-foundation-plan.md