mirror of https://github.com/lahfir/agent-desktop.git synced 2026-08-05 21:48:45 +00:00

feat!: implement Playwright-grade foundation contract

Settle the Playwright-grade reliability contract in agent-desktop-core
before the Windows/Linux adapters are built, so they inherit it instead
of redesigning it. Every command now observes, waits, verifies, and
reports honestly instead of firing blindly.

Highlights: capability-supertrait split of PlatformAdapter with
not_supported() defaults; canonical role/state vocabulary with live
`is --property visible`; display enumeration (`list-displays`) and honest
`--screen` with scale factor; truthful Automation permission; `native_id`
identity spine; window-id-first resolution; serializable `LocatorQuery`
with live `find`; default-on auto-wait before every ref action; three-way
`hit_test` occlusion gate; `scroll_into_view` in core; core accessible-name
precedence; typed `ActionStep` delivery tier; `ProcessState` and
`APP_UNRESPONSIVE`; `LaunchOptions`; baseline-diff desktop signals
(`wait --event`); typed clipboard (`Text`/`Image`/`FileUrls`); mouse
modifier chords and `mouse-wheel`. Hardened through a 35-reviewer pass with
independent validation and a green live e2e gate (109/0), plus a
head-vs-main performance comparison harness.

BREAKING CHANGE: default-on auto-wait changes the timing of every
previously-untouched ref-action call (bounded 5000 ms default; `--timeout-ms 0`
restores single-shot). `ENVELOPE_VERSION` is now `2.1` (adds the
`APP_UNRESPONSIVE` code and process state in error details). FFI ABI major
is `3` (append-only struct evolution; `wait --event` is intentionally not
exposed over FFI). The legacy string clipboard API is removed in favor of
typed content. `key-down`/`key-up` fail closed until daemon-owned held input
exists. `close-app` verifies termination and the osascript fallback path is
removed. `--text` matching is subtree containment: `find --text X --first`
returns the outermost matching container.

2026-07-20 00:21:38 -07:00

3.7 KiB

Raw Blame History

agent-desktop-ffi

C-ABI cdylib over PlatformAdapter. Exposes libagent_desktop_ffi.{dylib,so,dll} to Python (ctypes), Swift, Go (cgo), Node (ffi-napi), and C++ consumers without spawning a CLI subprocess per call.

Build

cargo build --profile release-ffi -p agent-desktop-ffi

The release-ffi profile keeps panic = "unwind", which is required for the catch_unwind traps that prevent panics from crossing the extern "C" boundary. Do not use --release — that profile sets panic = "abort", which silently defeats every trap.

For the CI / stub-adapter path (no AX permission required):

cargo build --profile release-ffi -p agent-desktop-ffi --features stub-adapter

--features stub-adapter replaces the real platform adapter with a no-op that returns PLATFORM_NOT_SUPPORTED for every adapter call.

ABI Surface

Group	Key symbols
ABI handshake	`ad_abi_version()`, `ad_init(expected_major)`
Adapter lifecycle	`ad_adapter_create()`, `ad_adapter_create_with_session(id)`, `ad_adapter_destroy()`
Command-backed JSON entrypoints	`ad_snapshot`, `ad_execute_by_ref`, `ad_wait`, `ad_version`, `ad_status`
Log callback	`ad_set_log_callback(fn(level, msg))` — unstructured tracing to a callback
Structured file trace	Same JSONL contract as CLI, gated by a `trace: on` session manifest (no ABI change); use `session start` before creating a session adapter
Errno last-error	`ad_last_error_code()`, `ad_last_error_message()`, `ad_last_error_suggestion()`, `ad_last_error_platform_detail()`
Type accessors / size getters	`ad__size()`, `ad__list_{count,get,free}()`, `ad_image_buffer_*()`
Free helpers	`ad_free_string()`, `ad_free_handle()`, `ad_free_tree()`, `ad_free_action_result()`

The full declaration list is in include/agent_desktop.h. Consumers must call ad_init(AD_ABI_VERSION_MAJOR) after dlopen to verify the loaded dylib matches the header the consumer was built against.

Use additive versioned AdExactRefEntry, AdExactWindowInfo, and AdExactSurfaceInfo APIs when round-tripping observed identities. Legacy ABI-v3 layouts remain binary-compatible but cannot express process generation or a surface ID; legacy direct window/ref targeting therefore fails closed.

Command wrappers

Each command-backed JSON entrypoint has one canonical module under src/commands/. The build script only configures the dylib install name and never rewrites source files.

Thread Safety (macOS)

Adapter entrypoints may run on any host thread. AX and CG calls have no blanket main-thread restriction, and Apple explicitly documents NSWorkspace.shared as safe from any thread. Cocoa-backed paths establish per-thread autorelease pools; FFI initialization starts one NSThread once so Cocoa enables the locks Apple documents for POSIX-threaded hosts.

Native handles are stricter: resolve, use, and free each handle on the same thread with the same adapter. Mutations are serialized across processes by the canonical interaction lease. See the FFI skill's threading reference for the full evidence and ordering contract.

Error Model

Every AdResult-returning function sets thread-local last-error state on failure. Read it with ad_last_error_code() / ad_last_error_message() / ad_last_error_suggestion(). The pointer returned by ad_last_error_message() stays valid until the next failing call on the same thread — successful calls leave it untouched (POSIX errno semantics).

Full Documentation

skills/agent-desktop-ffi/ — build-and-link guide, ownership model, threading rules, and error-handling patterns.

3.7 KiB Raw Blame History