mirror of https://github.com/lahfir/agent-desktop.git synced 2026-07-26 17:12:15 +00:00

feat!: implement Playwright-grade foundation contract

Settle the Playwright-grade reliability contract in agent-desktop-core
before the Windows/Linux adapters are built, so they inherit it instead
of redesigning it. Every command now observes, waits, verifies, and
reports honestly instead of firing blindly.

Highlights: capability-supertrait split of PlatformAdapter with
not_supported() defaults; canonical role/state vocabulary with live
`is --property visible`; display enumeration (`list-displays`) and honest
`--screen` with scale factor; truthful Automation permission; `native_id`
identity spine; window-id-first resolution; serializable `LocatorQuery`
with live `find`; default-on auto-wait before every ref action; three-way
`hit_test` occlusion gate; `scroll_into_view` in core; core accessible-name
precedence; typed `ActionStep` delivery tier; `ProcessState` and
`APP_UNRESPONSIVE`; `LaunchOptions`; baseline-diff desktop signals
(`wait --event`); typed clipboard (`Text`/`Image`/`FileUrls`); mouse
modifier chords and `mouse-wheel`. Hardened through a 35-reviewer pass with
independent validation and a green live e2e gate (109/0), plus a
head-vs-main performance comparison harness.

BREAKING CHANGE: default-on auto-wait changes the timing of every
previously-untouched ref-action call (bounded 5000 ms default; `--timeout-ms 0`
restores single-shot). `ENVELOPE_VERSION` is now `2.1` (adds the
`APP_UNRESPONSIVE` code and process state in error details). FFI ABI major
is `3` (append-only struct evolution; `wait --event` is intentionally not
exposed over FFI). The legacy string clipboard API is removed in favor of
typed content. `key-down`/`key-up` fail closed until daemon-owned held input
exists. `close-app` verifies termination and the osascript fallback path is
removed. `--text` matching is subtree containment: `find --text X --first`
returns the outermost matching container.

2026-07-20 00:21:38 -07:00

2 KiB

Raw Permalink Blame History

JSON output contract

Every command returns structured JSON:

{
  "version": "2.1",
  "ok": true,
  "command": "click",
  "data": { "action": "click" }
}

Errors include machine-readable codes and recovery hints:

{
  "version": "2.1",
  "ok": false,
  "command": "click",
  "error": {
    "code": "STALE_REF",
    "message": "Element at @e7 no longer matches the last snapshot",
    "suggestion": "Run 'snapshot' to refresh refs, then retry",
    "recovery": {
      "strategy": "refresh_snapshot_then_retry_original",
      "retryable": true,
      "requires_fresh_snapshot": true
    },
    "disposition": {
      "delivery": "not_delivered",
      "retry": "safe"
    }
  }
}

Version 2.1 replaces the 2.0 error.retry_command string with the structured error.recovery object and adds error.disposition. Consumers must select a recovery strategy from recovery.strategy only when disposition.retry is safe; command strings from older envelopes must not be executed blindly. The removed retry_command field has no compatibility alias.

Error codes

Code	Meaning
`PERM_DENIED`	Accessibility permission not granted
`ELEMENT_NOT_FOUND`	No element matched the ref or query
`APP_NOT_FOUND`	Application not running or no windows
`STALE_REF`	Ref could not be re-identified in the live UI
`AMBIGUOUS_TARGET`	Ref recovery matched multiple plausible targets
`SNAPSHOT_NOT_FOUND`	Snapshot ID is missing or expired
`POLICY_DENIED`	Physical/headed path blocked by policy
`ACTION_FAILED`	The OS rejected the action
`ACTION_NOT_SUPPORTED`	The target does not expose the requested action
`APP_UNRESPONSIVE`	The matching application stopped responding
`PLATFORM_NOT_SUPPORTED`	Adapter method not implemented on this platform
`TIMEOUT`	Wait condition expired
`INVALID_ARGS`	Invalid argument values

Exit codes

0 success, 1 structured error (JSON on stdout), 2 argument parse error.

2 KiB Raw Permalink Blame History

JSON output contract

Error codes

Exit codes

2 KiB

Raw Permalink Blame History