From 30e80b90bcedf1848dc16bd2ced53ab4308d2eee Mon Sep 17 00:00:00 2001 From: Lahfir Date: Wed, 10 Jun 2026 01:07:00 -0700 Subject: [PATCH] test: add E2E fixture app and dual-mode harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drives the release binary against a real SwiftUI/AppKit fixture and verifies every effect by independent before/after observation — never the command's own ok:true — so a command that reports success without an effect is caught. This is the layer mock-adapter unit tests cannot cover: it exercises the contract against the real macOS Accessibility API. - AgentDeskFixture.swift exposes a fixed, diverse AX surface (native AppKit slider/stepper, gesture-only and ambiguous controls, a sheet, a press-toggled disclosure, async-appearing elements, a drag canvas). It is never tuned to make a command pass; a failure is a finding about the CLI or the harness. - run.sh drives every ref-action command in BOTH headless and --headed mode with mode-specific target values, plus the double-click discriminator (headless fails closed with POLICY_DENIED, --headed completes) that proves the two modes differ. It also covers strict resolution, wait predicates, skeleton drill-down, sessions, trace redaction, surfaces, drag, expand, and force-close. - The compiled fixture .app is a build artifact (gitignored; built on demand). Run: cargo build --release && bash tests/e2e/run.sh (needs AX permission). --- .gitignore | 3 + tests/e2e/README.md | 87 ++++ tests/e2e/run.sh | 303 ++++++++++++++ tests/fixture-app/AgentDeskFixture.swift | 494 +++++++++++++++++++++++ tests/fixture-app/build.sh | 39 ++ 5 files changed, 926 insertions(+) create mode 100644 tests/e2e/README.md create mode 100755 tests/e2e/run.sh create mode 100644 tests/fixture-app/AgentDeskFixture.swift create mode 100755 tests/fixture-app/build.sh diff --git a/.gitignore b/.gitignore index 3900c96..01dd3ae 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,9 @@ /target/ **/*.rs.bk +# E2E fixture build artifact (compiled by tests/fixture-app/build.sh on demand) +/tests/fixture-app/build/ + # Cargo !Cargo.lock diff --git a/tests/e2e/README.md b/tests/e2e/README.md new file mode 100644 index 0000000..0d1949b --- /dev/null +++ b/tests/e2e/README.md @@ -0,0 +1,87 @@ +# End-to-End Tests (real binary vs. real app) + +These tests drive the **release `agent-desktop` binary** against a **real macOS +application** and verify every effect by **independent observation** — never the +command's own `ok: true`. A command that returns success without producing the +effect is caught, because each check re-reads the UI (status label, element +value, scroll offset, `list-apps`, …) and asserts the observed `before`/`after`. + +This is the layer that mock-adapter unit tests cannot cover: it exercises the +contract against the actual macOS Accessibility API. + +## Layout + +| Path | Role | +|------|------| +| `tests/fixture-app/AgentDeskFixture.swift` | SwiftUI + AppKit fixture exposing a fixed, diverse AX surface | +| `tests/fixture-app/build.sh` | Compiles the fixture into `build/AgentDeskFixture.app` | +| `tests/e2e/run.sh` | The harness: launches the fixture, runs the binary, asserts by observation | + +The fixture is a **principal-engineer-grade slice of real UI**, not a target +tuned to make the CLI pass. It deliberately mixes AX-actionable native AppKit +controls (`NSSlider`, `NSStepper`) with gesture-only and ambiguous patterns. **A +failure here is a finding about the CLI or the harness — never a reason to edit +the fixture so the CLI passes.** + +## Running + +```bash +cargo build --release # the harness uses target/release/agent-desktop +bash tests/e2e/run.sh +``` + +Requirements: + +- macOS with **Accessibility permission** granted to the terminal running the + harness (System Settings → Privacy & Security → Accessibility). +- The harness builds the fixture `.app` automatically if it is missing. +- `--headed` checks move the real cursor; run on a machine where that is OK. + +Exit code is `0` when every scenario passes, `1` on any failure (with the +observed values printed inline for each failing check), `2` on setup error. + +## What it verifies + +- **Observation:** snapshot role diversity, `find` vocabulary + `roles_present` + hint, textarea→textfield alias. +- **Interaction in BOTH modes (the headless/headed contract):** every ref-action + command (`click`, `type`, `set-value`, `clear`, `check`, `uncheck`, `select`, + `set-value` on slider/stepper, `scroll`) is driven **twice** — default + **headless**, then **`--headed`** — with mode-specific target values so a + regression in either mode is caught by an independent `before`/`after` + observation. Headed must not regress the AX path (it is tried first); it only + adds cursor/physical fallbacks. +- **The headless/headed discriminator:** `double-click` on a gesture-only button + (no `AXOpen`) **fails closed with `POLICY_DENIED` headless**, and **completes + with `--headed`** (physical double-click). This is the crisp proof the two + modes differ and that `--headed` unlocks the physical fallback. +- **Strict resolution:** ambiguous twins do not silently act; a removed element + fails closed with `STALE_REF`. +- **Reliability core:** `wait` predicates (`enabled`, `actionable`, `visible`, + `value`), `wait --text` async appearance, skeleton traversal + scoped + drill-down, session isolation + session-independent explicit snapshots, trace + JSONL with secret redaction. +- **Surfaces / drag / expand / close:** open a sheet and act inside it, + source-tracked drag gesture verified by a tracking canvas, expand a + press-toggled disclosure, `close-app --force` verified via `list-apps`. + +## Documented limitations (tracked, not failures) + +- **Cross-target native drag-and-drop (`onDrop`)** needs the OS dragging-session + / pasteboard protocol. Synthetic mouse events route mouse-up to the drag + origin, so they cannot drop onto a separate native target. Source-tracked + gestures (and web/Electron mouse-DnD) work; the harness verifies the gesture + via a tracking canvas. +- **SwiftUI `Slider`/`Stepper`/`DisclosureGroup` are not AX-actionable.** The + fixture uses native AppKit `NSSlider`/`NSStepper` (which are), so `set-value` + and `expand` are proven on controls that actually expose the capability. + +## Adding a scenario + +1. Expose the surface in `AgentDeskFixture.swift` with a stable + `accessibilityLabel` and a `statictext` status readout that reflects the + real state (so the harness can observe the effect, not the command's claim). +2. Add a check in `run.sh` using `verify