agent-desktop/CLAUDE.md

agent-desktop

Cross-platform Rust CLI + MCP server enabling AI agents to observe and control desktop applications via native OS accessibility trees.

Git & Commits

• All commits are authored by Lahfir
• NEVER add Co-Authored-By lines, AI attribution badges, or "Generated with" footers
• NEVER include co-committers of any kind
• Commit messages: concise, imperative mood, focus on "why" not "what"

Core Principle

agent-desktop is NOT an AI agent. It is a tool that AI agents invoke. It outputs structured JSON with ref-based element identifiers. The observation-action loop lives in the calling agent.

Architecture

Workspace Layout

agent-desktop/
├── Cargo.toml              # workspace: members, shared deps
├── rust-toolchain.toml     # pinned Rust version
├── clippy.toml             # project-wide lint config
├── crates/
│   ├── core/               # agent-desktop-core (platform-agnostic)
│   ├── macos/              # agent-desktop-macos (Phase 1)
│   ├── windows/            # agent-desktop-windows (stub → Phase 2)
│   └── linux/              # agent-desktop-linux (stub → Phase 2)
├── src/                    # agent-desktop binary (entry point)
│   ├── main.rs             # mode detection, dispatch
│   └── cli.rs              # clap derive structs
└── tests/
    ├── fixtures/           # golden JSON snapshots
    └── integration/        # macOS CI integration tests

Dependency Inversion (Non-Negotiable)

• agent-desktop-core defines the PlatformAdapter trait and all shared types
• Platform crates (macos, windows, linux) implement the trait
• Core NEVER imports platform crates. Platform crates NEVER import each other.
• The binary crate (src/) is the only place that wires platform → core
• CI enforces this: cargo tree -p agent-desktop-core must contain zero platform crate names

Platform Selection

Compile-time via #[cfg(target_os)] in build_adapter(). Agents never specify platform — agent-desktop snapshot -i works identically on macOS, Windows, and Linux.

fn build_adapter() -> impl PlatformAdapter {
    #[cfg(target_os = "macos")]
    { agent_desktop_macos::MacOSAdapter::new() }

    #[cfg(target_os = "windows")]
    { agent_desktop_windows::WindowsAdapter::new() }

    #[cfg(target_os = "linux")]
    { agent_desktop_linux::LinuxAdapter::new() }
}

Target-Gated Dependencies

Binary crate Cargo.toml uses platform-specific deps, NOT unconditional deps with #[cfg] in source:

[target.'cfg(target_os = "macos")'.dependencies]
agent-desktop-macos = { path = "crates/macos" }

[target.'cfg(target_os = "windows")'.dependencies]
agent-desktop-windows = { path = "crates/windows" }

[target.'cfg(target_os = "linux")'.dependencies]
agent-desktop-linux = { path = "crates/linux" }

Command Dispatch

Direct match in the binary crate. No Command trait, no CommandRegistry. Each command is a standalone execute() function under crates/core/src/commands/.

pub fn dispatch(cmd: Commands, adapter: &dyn PlatformAdapter) -> Result<serde_json::Value, AppError> {
    match cmd {
        Commands::Snapshot(args) => commands::snapshot::execute(args, adapter),
        Commands::Click(args) => commands::click::execute(args, adapter),
        // one arm per command
    }
}

Additive Phase Model

• Phase 1: Foundation + macOS MVP (30 commands, core engine, macOS adapter)
• Phase 2: Windows + Linux adapters, 10+ new commands — core untouched
• Phase 3: MCP server mode via --mcp flag — wraps existing commands
• Phase 4: Daemon, sessions, enterprise quality gates

Phases 2–4 add adapters/transports/hardening. Nothing in core is rebuilt.

Coding Standards

File Rules

• 400 LOC hard limit per file. If approaching 400, split by responsibility. No exceptions.
• No inline comments. Code must be self-documenting through naming. Only Rust doc-comments (///) on public items when the name alone is insufficient.
• One struct/enum per file for domain types. node.rs defines AccessibilityNode. action.rs defines Action.
• One command per file. Each CLI command lives in its own file under commands/. Filename matches the command name.
• No God objects. No struct with more than 7 fields. No function with more than 5 parameters. Use builder patterns or config structs.
• Explicit pub boundaries. Only lib.rs re-exports public items. Internal modules use pub(crate). No wildcard re-exports.

Error Handling

• Zero unwrap() in non-test code. All Results propagated with ? or matched explicitly. Panics are test-only.
• Every error carries: ErrorCode enum (machine-readable), message: String (human-readable), suggestion: Option<String> (recovery hint), platform_detail: Option<String> (OS-specific detail)
• All platform adapter functions return Result<T, AdapterError>
• All command handlers return Result<serde_json::Value, AppError>
• The binary's main() converts AppError to JSON and sets the exit code

Error Codes

PERM_DENIED, ELEMENT_NOT_FOUND, APP_NOT_FOUND, ACTION_FAILED,
ACTION_NOT_SUPPORTED, STALE_REF, WINDOW_NOT_FOUND,
PLATFORM_NOT_SUPPORTED, TIMEOUT, INVALID_ARGS, INTERNAL

Exit Codes

• 0 — success
• 1 — structured error (JSON with error code)
• 2 — argument/parse error

Naming Conventions

Element	Convention	Example
Crate names	agent-desktop-{name}	agent-desktop-core, agent-desktop-macos
Module files	snake_case, singular	snapshot.rs, list_windows.rs
Structs	PascalCase, descriptive noun	SnapshotEngine, RefAllocator
Traits	PascalCase, adjective/capability	PlatformAdapter, Executable
Enums	PascalCase, variants PascalCase	Action::Click, ErrorCode::PermDenied
Functions	snake_case, verb-first	build_tree(), allocate_refs()
Constants	SCREAMING_SNAKE_CASE	MAX_TREE_DEPTH, DEFAULT_TIMEOUT_MS
CLI flags	kebab-case	--max-depth, --include-bounds
Ref IDs	@e{n} sequential	@e1, @e2, @e14

Extensibility Pattern

Adding a new command requires exactly these steps:

1. Create crates/core/src/commands/{name}.rs with an execute() function
2. Register it in crates/core/src/commands/mod.rs
3. Add the CLI subcommand variant to src/cli.rs (clap derive enum)
4. Add a match arm in dispatch() in the binary crate
5. If new Action variant needed, add to crates/core/src/action.rs
6. If new adapter method needed, add to PlatformAdapter trait with a default returning Err(AdapterError::not_supported())

No existing files are modified beyond the registration points. Enforce via code review.

JSON Output Contract

Every command produces a response envelope:

{
  "version": "1.0",
  "ok": true,
  "command": "snapshot",
  "data": {
    "app": "Finder",
    "window": { "id": "w-4521", "title": "Documents" },
    "ref_count": 14,
    "tree": { ... }
  }
}

Error responses:

{
  "version": "1.0",
  "ok": false,
  "command": "click",
  "error": {
    "code": "STALE_REF",
    "message": "RefMap is from a previous snapshot",
    "suggestion": "Run 'snapshot' to refresh, then retry with updated ref"
  }
}

Serialization Rules

• Omit null/None fields (#[serde(skip_serializing_if = "Option::is_none")])
• Omit empty arrays (#[serde(skip_serializing_if = "Vec::is_empty")])
• Omit bounds in compact mode
• ref_count and tree go inside data, not as top-level siblings

Ref System

• Refs allocated in depth-first document order: @e1, @e2, etc.
• Only interactive roles receive refs: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell
• Static text, groups, containers do NOT get refs (they remain in tree for context)
• Refs are deterministic within a snapshot but NOT stable across snapshots if UI changed
• RefMap stored at ~/.agent-desktop/last_refmap.json with 0o600 permissions, directory at 0o700
• Each snapshot REPLACES the refmap file entirely (atomic write via temp + rename)
• Action commands use optimistic re-identification: (pid, role, name, bounds_hash). Return STALE_REF on mismatch.

PlatformAdapter Trait

12 methods with default implementations returning not_supported():

pub trait PlatformAdapter: Send + Sync {
    fn list_windows(&self, filter: &WindowFilter) -> Result<Vec<WindowInfo>, AdapterError>;
    fn list_apps(&self) -> Result<Vec<AppInfo>, AdapterError>;
    fn get_tree(&self, win: &WindowInfo, opts: &TreeOptions) -> Result<AccessibilityNode, AdapterError>;
    fn execute_action(&self, handle: &NativeHandle, action: Action) -> Result<ActionResult, AdapterError>;
    fn resolve_element(&self, entry: &RefEntry) -> Result<NativeHandle, AdapterError>;
    fn check_permissions(&self) -> PermissionStatus;
    fn focus_window(&self, win: &WindowInfo) -> Result<(), AdapterError>;
    fn launch_app(&self, id: &str, wait: bool) -> Result<WindowInfo, AdapterError>;
    fn close_app(&self, id: &str, force: bool) -> Result<(), AdapterError>;
    fn screenshot(&self, target: ScreenshotTarget) -> Result<ImageBuffer, AdapterError>;
    fn get_clipboard(&self) -> Result<String, AdapterError>;
    fn set_clipboard(&self, text: &str) -> Result<(), AdapterError>;
}

Key Types

• AccessibilityNode — platform-agnostic tree node: ref, role, name, value, description, states, bounds, children
• Action — Click, DoubleClick, RightClick, SetValue(String), SetFocus, Expand, Collapse, Select(String), Toggle, Scroll(Direction, Amount), PressKey(KeyCombo)
• NativeHandle — opaque platform pointer with PhantomData<*const ()> to prevent auto-Send/Sync. Inner field is pub(crate).
• RefEntry — { pid, role, name, bounds_hash, available_actions }
• WindowInfo — { id, title, app_name, pid, bounds }
• ErrorCode — 11-variant enum with #[serde(rename_all = "SCREAMING_SNAKE_CASE")]
• AdapterError — struct with code, message, suggestion, platform_detail
• AppError — enum with #[from] impls for AdapterError, std::io::Error, serde_json::Error

macOS Adapter (Phase 1)

Tree Traversal

• Entry: AXUIElementCreateApplication(pid) for app root
• Children: kAXChildrenAttribute recursively with visited-set to prevent cycles
• Use AXUIElementCopyMultipleAttributeValues for batch attribute fetch (3-5x faster)
• Role mapping: AXRole strings → unified role enum in roles.rs
• Max depth default: 10. Configurable via --max-depth

Action Execution

• Click: AXUIElementPerformAction(kAXPressAction)
• SetValue: AXUIElementSetAttributeValue(kAXValueAttribute, value)
• SetFocus: AXUIElementSetAttributeValue(kAXFocusedAttribute, true)
• Keyboard/Mouse: CGEventCreateKeyboardEvent / CGEventCreateMouseEvent
• Clipboard: NSPasteboard.generalPasteboard via Cocoa FFI
• Screenshot: CGWindowListCreateImage

Permission Detection

• Call AXIsProcessTrusted() on startup
• If false, return PERM_DENIED with guidance: "Open System Settings > Privacy > Accessibility and add your terminal"
• Optionally call AXIsProcessTrustedWithOptions(prompt: true) to trigger system dialog

AXElement Safety

• Inner field: pub(crate) not pub (prevents double-free via raw pointer extraction)
• Clone impl must call CFRetain
• Drop impl must call CFRelease

Testing Strategy

Unit Tests (core)

• AccessibilityNode ser/de roundtrips
• Ref allocator only assigns interactive roles
• SnapshotEngine filtering
• Error serialization
• MockAdapter: in-memory PlatformAdapter returning hardcoded trees

Golden Fixtures (tests/fixtures/)

• Real snapshots from Finder, TextEdit, etc. checked into repo
• Regression-test serialization format changes

Integration Tests (macOS CI)

• Snapshot Finder, TextEdit, System Settings — non-empty trees with refs
• Click button in test app — verify action succeeded
• Type text into TextEdit via ref — verify content changed
• Clipboard get/set roundtrip
• Permission denied scenario — correct error code and guidance
• Large tree (Xcode) snapshot in under 2 seconds

Dependencies (Phase 1)

Crate	Version	Purpose
clap	4.x	CLI parsing with derive macros
serde + serde_json	1.x	JSON serialization
thiserror	2.x	Error derive macros
tracing	0.1+	Structured logging
base64	0.22+	Screenshot encoding
accessibility-sys	0.1+	macOS AXUIElement FFI
core-foundation	0.10+	macOS CF types
core-graphics	0.24+	macOS CG types

Deferred Dependencies

• tokio — Phase 2/3 (all Phase 1 ops are synchronous)
• rmcp (0.15.0) — Phase 3 (MCP server)
• schemars — Phase 3 (JSON Schema generation)
• uiautomation (0.24+) — Phase 2 (Windows)
• atspi (0.28+) + zbus (5.x) — Phase 2 (Linux)

Build Configuration

[profile.release]
opt-level = "z"
lto = true
codegen-units = 1
strip = true
panic = "abort"

Target binary size: <15MB per platform.

CI Requirements

• GitHub Actions macOS runner executes full test suite on every PR
• cargo tree -p agent-desktop-core must not contain platform crate names
• cargo clippy --all-targets -- -D warnings
• cargo test --workspace
• Binary size check: fail if release binary exceeds 15MB

Phase 1 Command Scope (30 commands)

Category	Commands
App/Window (5)	launch, close-app, list-windows, list-apps, focus-window
Observation (15)	snapshot, screenshot, find, get (text, value, title, bounds, role, states), is (visible, enabled, checked, focused, expanded)
Interaction (11)	click, double-click, right-click, type, set-value, focus, select, toggle, expand, collapse, scroll
Keyboard (1)	press
Clipboard (2)	clipboard get, clipboard set
Wait (3)	wait (ms), wait --element, wait --window
System (3)	status, permissions, version

Non-Goals

• Does NOT embed or invoke LLMs
• Does NOT provide a GUI, TUI, or interactive prompt — machine-facing only
• Does NOT automate web browsers (use agent-browser for that)
• Does NOT record or replay macros (stateless per invocation until Phase 4 daemon)
• Does NOT work with custom-rendered or game-engine UIs lacking accessibility exposure

Reference Documents

• PRD v2.0: docs/agent_desktop_prd_v2.pdf
• Architecture Brainstorm: docs/brainstorms/2026-02-19-architecture-validation-brainstorm.md
• Phase 1 Plan: docs/plans/2026-02-19-feat-agent-desktop-phase1-foundation-plan.md