mirror of
https://github.com/lahfir/agent-desktop.git
synced 2026-07-26 17:12:15 +00:00
refactor!: unify command execution contracts
BREAKING CHANGE: CLI and batch execution now share the typed command path and current command argument contracts. BREAKING CHANGE: Ref-consuming commands use snapshot-scoped refs; deterministic consumers should pass snapshot_id and handle SNAPSHOT_NOT_FOUND. BREAKING CHANGE: permissions and status now return PermissionReport fields for accessibility, screen_recording, and automation instead of a single boolean status. BREAKING CHANGE: PermissionState gains NotRequired; macOS automation now reports not_required instead of unknown. BREAKING CHANGE: right-click now separates action success from menu verification; consumers should inspect menu or menu_probe instead of assuming every right-click returns an inline menu. BREAKING CHANGE: focus-window now confirms OS focus and returns ACTION_FAILED when focus does not settle; data.focused.is_focused is true on success. BREAKING CHANGE: PlatformAdapter::execute_action now takes ActionRequest, and permission probing uses permission_report/request_permissions. BREAKING CHANGE: FFI ad_execute_action now defaults to headless policy. Consumers that need focus fallback or cursor-moving behavior must call ad_execute_action_with_policy with AD_POLICY_KIND_FOCUS_FALLBACK or AD_POLICY_KIND_PHYSICAL. BREAKING CHANGE: FFI ad_check_permissions no longer treats unknown accessibility permission as success; stub-style unknown probes return ERR_PLATFORM_NOT_SUPPORTED and macOS ambiguous unknown returns ERR_INTERNAL with last-error detail. BREAKING CHANGE: JSON response envelopes now report version 2.0; parsers pinned to 1.0 must branch or update. BREAKING CHANGE: focus now uses accessibility focus without cursor movement; callers that need physical focus must use explicit mouse or physical-policy paths. BREAKING CHANGE: chain execution deadlines now return TIMEOUT instead of ACTION_FAILED when the target app does not respond before the chain deadline.
This commit is contained in:
parent
5b19d3f73f
commit
1e4667ad3a
161 changed files with 10293 additions and 4536 deletions
|
|
@ -12,7 +12,7 @@
|
|||
|
||||
### Features
|
||||
|
||||
* **ffi:** ship C-ABI cdylib with review hardening and release pipeline ([#26](https://github.com/lahfir/agent-desktop/issues/26)) ([3cffbd6](https://github.com/lahfir/agent-desktop/commit/3cffbd67f6b27f42001643bef9fd2530cb7f9003))
|
||||
* **ffi:** ship C-ABI cdylib with review fixes and release pipeline ([#26](https://github.com/lahfir/agent-desktop/issues/26)) ([3cffbd6](https://github.com/lahfir/agent-desktop/commit/3cffbd67f6b27f42001643bef9fd2530cb7f9003))
|
||||
|
||||
## [0.1.12](https://github.com/lahfir/agent-desktop/compare/v0.1.11...v0.1.12) (2026-04-16)
|
||||
|
||||
|
|
|
|||
44
CLAUDE.md
44
CLAUDE.md
|
|
@ -79,8 +79,10 @@ agent-desktop/
|
|||
│ ├── main.rs # entry point, permission check, JSON envelope
|
||||
│ ├── cli.rs # clap derive enum (Commands)
|
||||
│ ├── cli_args.rs # all command argument structs
|
||||
│ ├── batch.rs # batch JSON → typed Commands
|
||||
│ ├── command_policy.rs # command permission/ref/side-effect policy
|
||||
│ ├── dispatch.rs # command dispatcher + parse helpers
|
||||
│ └── batch_dispatch.rs # batch command execution
|
||||
│ └── dispatch_notifications.rs
|
||||
├── docs/
|
||||
│ └── solutions/ # documented solutions to past problems (bugs, best practices, workflow patterns), organized by category with YAML frontmatter (module, tags, problem_type); relevant when implementing or debugging in documented areas
|
||||
└── tests/
|
||||
|
|
@ -135,7 +137,11 @@ agent-desktop-linux = { path = "crates/linux" }
|
|||
Direct `match` in the binary crate. No `Command` trait, no `CommandRegistry`. Each command is a standalone `execute()` function under `crates/core/src/commands/`.
|
||||
|
||||
```rust
|
||||
pub fn dispatch(cmd: Commands, adapter: &dyn PlatformAdapter) -> Result<serde_json::Value, AppError> {
|
||||
pub fn dispatch(
|
||||
cmd: Commands,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
permission_report: &PermissionReport,
|
||||
) -> Result<serde_json::Value, AppError> {
|
||||
match cmd {
|
||||
Commands::Snapshot(args) => commands::snapshot::execute(args, adapter),
|
||||
Commands::Click(args) => commands::click::execute(args, adapter),
|
||||
|
|
@ -144,14 +150,16 @@ pub fn dispatch(cmd: Commands, adapter: &dyn PlatformAdapter) -> Result<serde_js
|
|||
}
|
||||
```
|
||||
|
||||
Batch is not a second dispatcher. `src/batch.rs` deserializes JSON entries into the same typed `Commands` enum, runs the same `CommandPolicy` preflight, and calls the same `dispatch()` path as CLI.
|
||||
|
||||
### Additive Phase Model
|
||||
|
||||
- **Phase 1:** Foundation + macOS MVP (30 commands, core engine, macOS adapter)
|
||||
- **Phase 1:** Foundation + macOS MVP (54 commands, core engine, macOS adapter)
|
||||
- **Phase 2:** Windows + Linux adapters, 10+ new commands — core untouched
|
||||
- **Phase 3:** MCP server mode via `--mcp` flag — wraps existing commands
|
||||
- **Phase 4:** Daemon, sessions, enterprise quality gates
|
||||
|
||||
Phases 2–4 add adapters/transports/hardening. Nothing in core is rebuilt.
|
||||
Phases 2–4 add adapters, transports, and production readiness work. Nothing in core is rebuilt.
|
||||
|
||||
## Coding Standards
|
||||
|
||||
|
|
@ -211,6 +219,7 @@ crates/{macos,windows,linux}/src/
|
|||
├── tree/ # Reading & understanding the UI
|
||||
│ ├── mod.rs # re-exports
|
||||
│ ├── element.rs # AXElement struct + attribute readers
|
||||
│ ├── capabilities.rs # AX-supported actions and settable attributes
|
||||
│ ├── builder.rs # build_subtree, tree traversal
|
||||
│ ├── roles.rs # Role mapping
|
||||
│ ├── resolve.rs # Element re-identification
|
||||
|
|
@ -219,7 +228,9 @@ crates/{macos,windows,linux}/src/
|
|||
│ ├── mod.rs # re-exports
|
||||
│ ├── dispatch.rs # perform_action match arms
|
||||
│ ├── activate.rs # Smart AX-first activation chain
|
||||
│ └── extras.rs # select_value, ax_scroll
|
||||
│ ├── extras.rs # select_value helpers
|
||||
│ ├── scroll.rs # scroll semantics and gated physical fallback
|
||||
│ └── type_text.rs # headless text insertion and physical typing
|
||||
├── input/ # Low-level OS input synthesis
|
||||
│ ├── mod.rs # re-exports
|
||||
│ ├── keyboard.rs # Key synthesis, text typing
|
||||
|
|
@ -281,7 +292,7 @@ Error responses:
|
|||
"command": "click",
|
||||
"error": {
|
||||
"code": "STALE_REF",
|
||||
"message": "RefMap is from a previous snapshot",
|
||||
"message": "Element could not be resolved from the requested snapshot",
|
||||
"suggestion": "Run 'snapshot' to refresh, then retry with updated ref"
|
||||
}
|
||||
}
|
||||
|
|
@ -300,8 +311,8 @@ Error responses:
|
|||
- Only interactive roles receive refs: `button`, `textfield`, `checkbox`, `link`, `menuitem`, `tab`, `slider`, `combobox`, `treeitem`, `cell`
|
||||
- Static text, groups, containers do NOT get refs (they remain in tree for context)
|
||||
- Refs are deterministic within a snapshot but NOT stable across snapshots if UI changed
|
||||
- RefMap stored at `~/.agent-desktop/last_refmap.json` with `0o600` permissions, directory at `0o700`
|
||||
- Each snapshot REPLACES the refmap file entirely (atomic write via temp + rename)
|
||||
- Snapshot refs are stored by snapshot ID under `~/.agent-desktop/snapshots/{snapshot_id}/refmap.json`, with a `latest_snapshot_id` pointer for commands that omit `--snapshot`
|
||||
- `~/.agent-desktop/last_refmap.json` is written only as a latest-snapshot inspection artifact; command code must use `RefStore`
|
||||
- Action commands use optimistic re-identification: `(pid, role, name, bounds_hash)`. Return `STALE_REF` on mismatch.
|
||||
- Progressive traversal: `--skeleton` clamps depth to 3, annotates truncated containers with `children_count`. Named/described containers at boundary receive refs as drill-down targets
|
||||
- Drill-down: `--root @ref` starts from a previously-discovered ref with scoped invalidation (only that ref's subtree refs are replaced on re-drill)
|
||||
|
|
@ -309,17 +320,18 @@ Error responses:
|
|||
|
||||
## PlatformAdapter Trait
|
||||
|
||||
13 methods with default implementations returning `not_supported()`:
|
||||
Platform-facing methods default to `not_supported()` unless implemented by an adapter:
|
||||
|
||||
```rust
|
||||
pub trait PlatformAdapter: Send + Sync {
|
||||
pub trait PlatformAdapter {
|
||||
fn list_windows(&self, filter: &WindowFilter) -> Result<Vec<WindowInfo>, AdapterError>;
|
||||
fn list_apps(&self) -> Result<Vec<AppInfo>, AdapterError>;
|
||||
fn get_tree(&self, win: &WindowInfo, opts: &TreeOptions) -> Result<AccessibilityNode, AdapterError>;
|
||||
fn get_subtree(&self, handle: &NativeHandle, opts: &TreeOptions) -> Result<AccessibilityNode, AdapterError>;
|
||||
fn execute_action(&self, handle: &NativeHandle, action: Action) -> Result<ActionResult, AdapterError>;
|
||||
fn execute_action(&self, handle: &NativeHandle, request: ActionRequest) -> Result<ActionResult, AdapterError>;
|
||||
fn resolve_element(&self, entry: &RefEntry) -> Result<NativeHandle, AdapterError>;
|
||||
fn check_permissions(&self) -> PermissionStatus;
|
||||
fn permission_report(&self) -> PermissionReport;
|
||||
fn request_permissions(&self) -> PermissionReport;
|
||||
fn focus_window(&self, win: &WindowInfo) -> Result<(), AdapterError>;
|
||||
fn launch_app(&self, id: &str, wait: bool) -> Result<WindowInfo, AdapterError>;
|
||||
fn close_app(&self, id: &str, force: bool) -> Result<(), AdapterError>;
|
||||
|
|
@ -331,12 +343,14 @@ pub trait PlatformAdapter: Send + Sync {
|
|||
|
||||
## Key Types
|
||||
|
||||
- `AccessibilityNode` — platform-agnostic tree node: `ref`, `role`, `name`, `value`, `description`, `states`, `bounds`, `children`
|
||||
- `AccessibilityNode` — platform-agnostic tree node: `ref`, `role`, `name`, `value`, `description`, `states`, `available_actions`, `bounds`, `children`
|
||||
- `Action` — Click, DoubleClick, RightClick, SetValue(String), SetFocus, Expand, Collapse, Select(String), Toggle, Scroll(Direction, Amount), PressKey(KeyCombo)
|
||||
- `ActionRequest` — `{ action, policy }`, where the default `InteractionPolicy` forbids focus stealing and cursor movement
|
||||
- `NativeHandle` — opaque platform pointer with `PhantomData<*const ()>` to prevent auto-Send/Sync. Inner field is `pub(crate)`.
|
||||
- `RefEntry` — `{ pid, role, name, bounds_hash, available_actions }`
|
||||
- `WindowInfo` — `{ id, title, app_name, pid, bounds }`
|
||||
- `ErrorCode` — 11-variant enum with `#[serde(rename_all = "SCREAMING_SNAKE_CASE")]`
|
||||
- `PermissionReport` — `{ accessibility, screen_recording, automation }`, each `{ "state": "granted" }`, `{ "state": "denied", "suggestion": "..." }`, or `{ "state": "unknown" }`
|
||||
- `ErrorCode` — machine-readable enum with `#[serde(rename_all = "SCREAMING_SNAKE_CASE")]`
|
||||
- `AdapterError` — struct with `code`, `message`, `suggestion`, `platform_detail`
|
||||
- `AppError` — enum with `#[from]` impls for `AdapterError`, `std::io::Error`, `serde_json::Error`
|
||||
|
||||
|
|
@ -355,7 +369,7 @@ pub trait PlatformAdapter: Send + Sync {
|
|||
- SetFocus: `AXUIElementSetAttributeValue(kAXFocusedAttribute, true)`
|
||||
- Keyboard/Mouse: `CGEventCreateKeyboardEvent` / `CGEventCreateMouseEvent`
|
||||
- Clipboard: `NSPasteboard.generalPasteboard` via Cocoa FFI
|
||||
- Screenshot: `CGWindowListCreateImage`
|
||||
- Screenshot: `ScreenshotBackend` boundary with secure `screencapture` temp files
|
||||
|
||||
### Permission Detection
|
||||
- Call `AXIsProcessTrusted()` on startup
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ tracing = "0.1"
|
|||
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||
base64 = "0.22"
|
||||
rustc-hash = "2.1"
|
||||
libc = "0.2"
|
||||
agent-desktop-core = { path = "crates/core" }
|
||||
|
||||
[profile.release]
|
||||
|
|
|
|||
69
README.md
69
README.md
|
|
@ -39,8 +39,8 @@
|
|||
- **C-ABI cdylib** (`libagent_desktop_ffi`): Load once from Python / Swift / Go / Ruby / Node / C instead of forking the CLI per call
|
||||
- **54 commands**: Observation, interaction, keyboard, mouse, notifications, clipboard, window management, plus a bundled `skills` doc loader
|
||||
- **Progressive skeleton traversal**: 78–96% token reduction on dense apps via shallow overview + targeted drill-down
|
||||
- **Snapshot & refs**: AI-optimized workflow using deterministic element references (`@e1`, `@e2`)
|
||||
- **AX-first interactions**: Every action exhausts pure accessibility API strategies before falling back to mouse events
|
||||
- **Snapshot & refs**: AI-optimized workflow using compact snapshot IDs and deterministic element references (`@e1`, `@e2`)
|
||||
- **Headless-by-default interactions**: Ref actions use accessibility APIs and block silent focus, cursor, keyboard, or pasteboard side effects
|
||||
- **Structured JSON output**: Machine-readable responses with error codes and recovery hints
|
||||
- **Works with any app**: Finder, Safari, System Settings, Xcode, Slack — anything with an accessibility tree
|
||||
|
||||
|
|
@ -71,10 +71,20 @@ Requires Rust 1.78+ and macOS 13.0+.
|
|||
|
||||
### Permissions
|
||||
|
||||
macOS requires Accessibility permission. Grant it in **System Settings > Privacy & Security > Accessibility** by adding your terminal app, or:
|
||||
macOS requires Accessibility permission. Screenshots also require Screen Recording permission. Grant them in **System Settings > Privacy & Security** by adding the app that launches agent-desktop, or:
|
||||
|
||||
```bash
|
||||
agent-desktop permissions --request # trigger system dialog
|
||||
agent-desktop permissions --request # trigger platform permission request path
|
||||
```
|
||||
|
||||
Permission fields are explicit objects, for example:
|
||||
|
||||
```json
|
||||
{
|
||||
"accessibility": { "state": "granted" },
|
||||
"screen_recording": { "state": "denied", "suggestion": "Grant Screen Recording permission" },
|
||||
"automation": { "state": "not_required" }
|
||||
}
|
||||
```
|
||||
|
||||
## Language bindings (FFI)
|
||||
|
|
@ -116,23 +126,24 @@ For dense apps (Slack, VS Code, Notion), use **progressive skeleton traversal**
|
|||
```bash
|
||||
# 1. Shallow overview — depth-3 map, truncated containers show children_count
|
||||
agent-desktop snapshot --skeleton --app Slack -i --compact
|
||||
# Keep snapshot_id, for example s8f3k2p9
|
||||
|
||||
# 2. Drill into a region of interest (named containers get refs as drill targets)
|
||||
agent-desktop snapshot --root @e3 -i --compact
|
||||
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact
|
||||
|
||||
# 3. Act on an element found in the drill-down
|
||||
agent-desktop click @e12
|
||||
agent-desktop click @e12 --snapshot s8f3k2p9
|
||||
|
||||
# 4. Re-drill the same region to verify the state change
|
||||
agent-desktop snapshot --root @e3 -i --compact
|
||||
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact
|
||||
```
|
||||
|
||||
For simple apps, a full snapshot is fine:
|
||||
|
||||
```bash
|
||||
agent-desktop snapshot --app Finder -i # get interactive elements with refs
|
||||
agent-desktop click @e3 # click a button by ref
|
||||
agent-desktop type @e5 "quarterly report" # type into a text field
|
||||
agent-desktop snapshot --app Finder -i # get interactive elements with refs and snapshot_id
|
||||
agent-desktop click @e3 --snapshot s8f3k2p9 # click a button by ref
|
||||
agent-desktop type @e5 --snapshot s8f3k2p9 "quarterly report" # insert text into a field
|
||||
agent-desktop press cmd+s # keyboard shortcut
|
||||
agent-desktop snapshot -i # re-observe after UI changes
|
||||
```
|
||||
|
|
@ -150,29 +161,29 @@ agent-desktop snapshot --app Safari -i # accessibility tree with refs
|
|||
agent-desktop snapshot --surface menu # capture open menu
|
||||
agent-desktop screenshot --app Finder # PNG screenshot
|
||||
agent-desktop find --role button --app TextEdit # search by role, name, value, text
|
||||
agent-desktop get @e3 value # read element property
|
||||
agent-desktop is @e7 checked # check boolean state
|
||||
agent-desktop get @e3 --snapshot s8f3k2p9 --property value # read element property
|
||||
agent-desktop is @e7 --snapshot s8f3k2p9 --property checked # check boolean state
|
||||
agent-desktop list-surfaces --app Notes # list menus, sheets, popovers, alerts
|
||||
```
|
||||
|
||||
### Interaction
|
||||
|
||||
```bash
|
||||
agent-desktop click @e3 # smart AX-first click (15-step chain)
|
||||
agent-desktop double-click @e3 # open files, select words
|
||||
agent-desktop triple-click @e3 # select lines/paragraphs
|
||||
agent-desktop right-click @e3 # context menu (returns menu tree inline)
|
||||
agent-desktop type @e5 "hello world" # type text into element
|
||||
agent-desktop click @e3 # semantic AX-first click
|
||||
agent-desktop double-click @e3 # AXOpen; physical double-click uses mouse-click --count 2
|
||||
agent-desktop triple-click @e3 # POLICY_DENIED if physical input is disabled
|
||||
agent-desktop right-click @e3 # open verified context menu
|
||||
agent-desktop type @e5 "hello world" # insert text into element
|
||||
agent-desktop set-value @e5 "new value" # set value directly via AX
|
||||
agent-desktop clear @e5 # clear element value
|
||||
agent-desktop focus @e5 # set keyboard focus
|
||||
agent-desktop select @e9 "Option B" # select option in dropdown/list
|
||||
agent-desktop select @e9 "Option B" # select verified dropdown/list option
|
||||
agent-desktop toggle @e12 # flip checkbox or switch
|
||||
agent-desktop check @e12 # idempotent check
|
||||
agent-desktop uncheck @e12 # idempotent uncheck
|
||||
agent-desktop expand @e15 # expand disclosure/tree item
|
||||
agent-desktop collapse @e15 # collapse disclosure/tree item
|
||||
agent-desktop scroll @e1 down 3 # scroll (AX-first, 10-step chain)
|
||||
agent-desktop scroll @e1 --direction down --amount 3 # scroll (AX-first)
|
||||
agent-desktop scroll-to @e20 # scroll element into view
|
||||
```
|
||||
|
||||
|
|
@ -250,8 +261,8 @@ agent-desktop wait --menu --timeout 3000 # wait for menu
|
|||
|
||||
```bash
|
||||
agent-desktop batch '[
|
||||
{"command": "click", "args": {"ref_id": "@e2"}},
|
||||
{"command": "type", "args": {"ref_id": "@e5", "text": "hello"}},
|
||||
{"command": "click", "args": {"ref_id": "@e2", "snapshot": "s8f3k2p9"}},
|
||||
{"command": "type", "args": {"ref_id": "@e5", "snapshot": "s8f3k2p9", "text": "hello"}},
|
||||
{"command": "press", "args": {"combo": "return"}}
|
||||
]' --stop-on-error
|
||||
```
|
||||
|
|
@ -259,9 +270,9 @@ agent-desktop batch '[
|
|||
### System
|
||||
|
||||
```bash
|
||||
agent-desktop status # platform, permission state
|
||||
agent-desktop permissions # check accessibility permission
|
||||
agent-desktop permissions --request # trigger system dialog
|
||||
agent-desktop status # platform, permission report, latest snapshot
|
||||
agent-desktop permissions # check accessibility/screen-recording/automation
|
||||
agent-desktop permissions --request # invoke platform request path
|
||||
agent-desktop version # version string
|
||||
```
|
||||
|
||||
|
|
@ -281,6 +292,7 @@ agent-desktop snapshot [OPTIONS]
|
|||
| `--max-depth <N>` | 10 | Maximum tree depth |
|
||||
| `--skeleton` | off | Shallow 3-level overview; truncated containers show `children_count` and get refs as drill targets |
|
||||
| `--root <REF>` | - | Start traversal from this ref; merges into existing refmap with scoped invalidation |
|
||||
| `--snapshot <ID>` | latest | Snapshot ID to use when resolving `--root` |
|
||||
| `--surface <TYPE>` | window | `window`, `focused`, `menu`, `menubar`, `sheet`, `popover`, `alert` |
|
||||
|
||||
## JSON Output
|
||||
|
|
@ -289,7 +301,7 @@ Every command returns structured JSON:
|
|||
|
||||
```json
|
||||
{
|
||||
"version": "1.0",
|
||||
"version": "2.0",
|
||||
"ok": true,
|
||||
"command": "click",
|
||||
"data": { "action": "click" }
|
||||
|
|
@ -300,7 +312,7 @@ Errors include machine-readable codes and recovery hints:
|
|||
|
||||
```json
|
||||
{
|
||||
"version": "1.0",
|
||||
"version": "2.0",
|
||||
"ok": false,
|
||||
"command": "click",
|
||||
"error": {
|
||||
|
|
@ -319,7 +331,10 @@ Errors include machine-readable codes and recovery hints:
|
|||
| `ELEMENT_NOT_FOUND` | No element matched the ref or query |
|
||||
| `APP_NOT_FOUND` | Application not running or no windows |
|
||||
| `STALE_REF` | Ref is from a previous snapshot |
|
||||
| `SNAPSHOT_NOT_FOUND` | Snapshot ID is missing or expired |
|
||||
| `POLICY_DENIED` | Physical/headed path blocked by policy |
|
||||
| `ACTION_FAILED` | The OS rejected the action |
|
||||
| `PLATFORM_NOT_SUPPORTED` | Adapter method not implemented on this platform |
|
||||
| `TIMEOUT` | Wait condition expired |
|
||||
| `INVALID_ARGS` | Invalid argument values |
|
||||
|
||||
|
|
@ -329,7 +344,7 @@ Errors include machine-readable codes and recovery hints:
|
|||
|
||||
## Ref System
|
||||
|
||||
`snapshot` assigns refs to interactive elements in depth-first order: `@e1`, `@e2`, `@e3`, etc. Refs are valid until the next snapshot replaces them.
|
||||
`snapshot` assigns refs to interactive elements in depth-first order: `@e1`, `@e2`, `@e3`, etc. Refs are scoped to a compact `snapshot_id` such as `s8f3k2p9`. Commands can omit `--snapshot` to use the latest snapshot pointer, but passing the ID is more deterministic in multi-step flows.
|
||||
|
||||
Interactive roles that receive refs: `button`, `textfield`, `checkbox`, `link`, `menuitem`, `tab`, `slider`, `combobox`, `treeitem`, `cell`, `radiobutton`, `incrementor`, `menubutton`, `switch`, `colorwell`, `dockitem`.
|
||||
|
||||
|
|
|
|||
|
|
@ -11,3 +11,4 @@ thiserror.workspace = true
|
|||
tracing.workspace = true
|
||||
rustc-hash.workspace = true
|
||||
base64.workspace = true
|
||||
libc.workspace = true
|
||||
|
|
|
|||
|
|
@ -26,6 +26,95 @@ pub enum Action {
|
|||
Drag(DragParams),
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ActionRequest {
|
||||
pub action: Action,
|
||||
pub policy: InteractionPolicy,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct InteractionPolicy {
|
||||
pub allow_focus_steal: bool,
|
||||
pub allow_cursor_move: bool,
|
||||
}
|
||||
|
||||
impl ActionRequest {
|
||||
pub fn headless(action: Action) -> Self {
|
||||
Self {
|
||||
action,
|
||||
policy: InteractionPolicy::headless(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn focus_fallback(action: Action) -> Self {
|
||||
Self {
|
||||
action,
|
||||
policy: InteractionPolicy::focus_fallback(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn physical(action: Action) -> Self {
|
||||
Self {
|
||||
action,
|
||||
policy: InteractionPolicy::physical(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl InteractionPolicy {
|
||||
pub fn headless() -> Self {
|
||||
Self {
|
||||
allow_focus_steal: false,
|
||||
allow_cursor_move: false,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn focus_fallback() -> Self {
|
||||
Self {
|
||||
allow_focus_steal: true,
|
||||
allow_cursor_move: false,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn physical() -> Self {
|
||||
Self {
|
||||
allow_focus_steal: true,
|
||||
allow_cursor_move: true,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for InteractionPolicy {
|
||||
fn default() -> Self {
|
||||
Self::headless()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod policy_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn default_policy_is_headless() {
|
||||
let policy = InteractionPolicy::default();
|
||||
assert!(!policy.allow_focus_steal);
|
||||
assert!(!policy.allow_cursor_move);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn headless_request_blocks_physical_side_effects() {
|
||||
let request = ActionRequest::headless(Action::Click);
|
||||
assert_eq!(request.policy, InteractionPolicy::headless());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn focus_fallback_policy_never_moves_cursor() {
|
||||
let request = ActionRequest::focus_fallback(Action::Scroll(Direction::Down, 1));
|
||||
assert!(request.policy.allow_focus_steal);
|
||||
assert!(!request.policy.allow_cursor_move);
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub enum Direction {
|
||||
Up,
|
||||
|
|
|
|||
|
|
@ -1,9 +1,12 @@
|
|||
use crate::{
|
||||
action::{Action, ActionResult, DragParams, MouseEvent, WindowOp},
|
||||
action::{
|
||||
ActionRequest, ActionResult, DragParams, ElementState, KeyCombo, MouseEvent, WindowOp,
|
||||
},
|
||||
error::AdapterError,
|
||||
node::{AccessibilityNode, AppInfo, Rect, SurfaceInfo, WindowInfo},
|
||||
notification::{NotificationFilter, NotificationIdentity, NotificationInfo},
|
||||
refs::RefEntry,
|
||||
PermissionReport, PermissionState,
|
||||
};
|
||||
use std::marker::PhantomData;
|
||||
|
||||
|
|
@ -53,11 +56,6 @@ pub enum ScreenshotTarget {
|
|||
FullScreen,
|
||||
}
|
||||
|
||||
pub enum PermissionStatus {
|
||||
Granted,
|
||||
Denied { suggestion: String },
|
||||
}
|
||||
|
||||
pub struct NativeHandle {
|
||||
pub(crate) ptr: *const std::ffi::c_void,
|
||||
_not_send_sync: PhantomData<*const ()>,
|
||||
|
|
@ -87,12 +85,6 @@ impl NativeHandle {
|
|||
}
|
||||
}
|
||||
|
||||
// SAFETY: Phase 1 is single-threaded CLI. NativeHandle is never sent across thread
|
||||
// boundaries. The unsafe impls are required for use with dyn PlatformAdapter (which
|
||||
// is Send + Sync). Remove in Phase 4 when async daemon is introduced.
|
||||
unsafe impl Send for NativeHandle {}
|
||||
unsafe impl Sync for NativeHandle {}
|
||||
|
||||
pub struct ImageBuffer {
|
||||
pub data: Vec<u8>,
|
||||
pub format: ImageFormat,
|
||||
|
|
@ -134,7 +126,7 @@ pub trait PlatformAdapter: Send + Sync {
|
|||
fn execute_action(
|
||||
&self,
|
||||
_handle: &NativeHandle,
|
||||
_action: Action,
|
||||
_request: ActionRequest,
|
||||
) -> Result<ActionResult, AdapterError> {
|
||||
Err(AdapterError::not_supported("execute_action"))
|
||||
}
|
||||
|
|
@ -144,20 +136,33 @@ pub trait PlatformAdapter: Send + Sync {
|
|||
}
|
||||
|
||||
/// Releases a platform-specific element handle returned from
|
||||
/// `resolve_element`. macOS implementations must `CFRelease` the
|
||||
/// underlying `AXUIElementRef` to balance the `CFRetain` that
|
||||
/// happened during resolve. Windows/Linux consumers can leave this
|
||||
/// as the default `not_supported` no-op.
|
||||
/// `resolve_element`. Adapter methods that receive `&NativeHandle`
|
||||
/// borrow it only; they must not consume or release it. macOS
|
||||
/// implementations must `CFRelease` here to balance the `CFRetain`
|
||||
/// that happened during resolve. Windows/Linux consumers can leave
|
||||
/// this as the default no-op.
|
||||
fn release_handle(&self, _handle: &NativeHandle) -> Result<(), AdapterError> {
|
||||
Err(AdapterError::not_supported("release_handle"))
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn check_permissions(&self) -> PermissionStatus {
|
||||
PermissionStatus::Denied {
|
||||
suggestion: "Platform adapter not available".into(),
|
||||
fn permission_report(&self) -> PermissionReport {
|
||||
PermissionReport {
|
||||
accessibility: PermissionState::Denied {
|
||||
suggestion: "Platform adapter not available".into(),
|
||||
},
|
||||
screen_recording: PermissionState::Unknown,
|
||||
automation: PermissionState::NotRequired,
|
||||
}
|
||||
}
|
||||
|
||||
fn unknown_accessibility_means_unsupported(&self) -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
fn request_permissions(&self) -> PermissionReport {
|
||||
self.permission_report()
|
||||
}
|
||||
|
||||
fn focus_window(&self, _win: &WindowInfo) -> Result<(), AdapterError> {
|
||||
Err(AdapterError::not_supported("focus_window"))
|
||||
}
|
||||
|
|
@ -190,6 +195,10 @@ pub trait PlatformAdapter: Send + Sync {
|
|||
Err(AdapterError::not_supported("get_live_value"))
|
||||
}
|
||||
|
||||
fn get_live_state(&self, _handle: &NativeHandle) -> Result<Option<ElementState>, AdapterError> {
|
||||
Err(AdapterError::not_supported("get_live_state"))
|
||||
}
|
||||
|
||||
fn press_key_for_app(
|
||||
&self,
|
||||
_app_name: &str,
|
||||
|
|
@ -218,6 +227,10 @@ pub trait PlatformAdapter: Send + Sync {
|
|||
Err(AdapterError::not_supported("mouse_event"))
|
||||
}
|
||||
|
||||
fn key_event(&self, _combo: &KeyCombo, _down: bool) -> Result<(), AdapterError> {
|
||||
Err(AdapterError::not_supported("key_event"))
|
||||
}
|
||||
|
||||
fn drag(&self, _params: DragParams) -> Result<(), AdapterError> {
|
||||
Err(AdapterError::not_supported("drag"))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,14 +1,11 @@
|
|||
use crate::{
|
||||
action::Action, adapter::PlatformAdapter, commands::helpers::resolve_ref, error::AppError,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{execute_ref_action, RefArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
pub struct CheckArgs {
|
||||
pub ref_id: String,
|
||||
}
|
||||
|
||||
pub fn execute(args: CheckArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::Check)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
pub fn execute(args: RefArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
execute_ref_action(args, adapter, ActionRequest::headless(Action::Check))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,14 +1,11 @@
|
|||
use crate::{
|
||||
action::Action, adapter::PlatformAdapter, commands::helpers::resolve_ref, error::AppError,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{execute_ref_action, RefArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
pub struct ClearArgs {
|
||||
pub ref_id: String,
|
||||
}
|
||||
|
||||
pub fn execute(args: ClearArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::Clear)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
pub fn execute(args: RefArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
execute_ref_action(args, adapter, ActionRequest::headless(Action::Clear))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,14 +1,11 @@
|
|||
use crate::{
|
||||
action::Action, adapter::PlatformAdapter, commands::helpers::resolve_ref, error::AppError,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{execute_ref_action, RefArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
pub struct ClickArgs {
|
||||
pub ref_id: String,
|
||||
}
|
||||
|
||||
pub fn execute(args: ClickArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::Click)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
pub fn execute(args: RefArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
execute_ref_action(args, adapter, ActionRequest::headless(Action::Click))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,13 +1,11 @@
|
|||
use crate::{
|
||||
action::Action,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{resolve_ref, RefArgs},
|
||||
commands::helpers::{execute_ref_action, RefArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
pub fn execute(args: RefArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::Collapse)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
execute_ref_action(args, adapter, ActionRequest::headless(Action::Collapse))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,14 +1,11 @@
|
|||
use crate::{
|
||||
action::Action, adapter::PlatformAdapter, commands::helpers::resolve_ref, error::AppError,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{execute_ref_action, RefArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
pub struct DoubleClickArgs {
|
||||
pub ref_id: String,
|
||||
}
|
||||
|
||||
pub fn execute(args: DoubleClickArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::DoubleClick)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
pub fn execute(args: RefArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
execute_ref_action(args, adapter, ActionRequest::headless(Action::DoubleClick))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,12 +11,25 @@ pub struct DragArgs {
|
|||
pub from_xy: Option<(f64, f64)>,
|
||||
pub to_ref: Option<String>,
|
||||
pub to_xy: Option<(f64, f64)>,
|
||||
pub snapshot_id: Option<String>,
|
||||
pub duration_ms: Option<u64>,
|
||||
}
|
||||
|
||||
pub fn execute(args: DragArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let from = resolve_point(&args.from_ref, args.from_xy, "from", adapter)?;
|
||||
let to = resolve_point(&args.to_ref, args.to_xy, "to", adapter)?;
|
||||
let from = resolve_point(
|
||||
&args.from_ref,
|
||||
args.from_xy,
|
||||
"from",
|
||||
args.snapshot_id.as_deref(),
|
||||
adapter,
|
||||
)?;
|
||||
let to = resolve_point(
|
||||
&args.to_ref,
|
||||
args.to_xy,
|
||||
"to",
|
||||
args.snapshot_id.as_deref(),
|
||||
adapter,
|
||||
)?;
|
||||
let params = DragParams {
|
||||
from: from.clone(),
|
||||
to: to.clone(),
|
||||
|
|
@ -34,12 +47,13 @@ fn resolve_point(
|
|||
ref_id: &Option<String>,
|
||||
xy: Option<(f64, f64)>,
|
||||
label: &str,
|
||||
snapshot_id: Option<&str>,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
) -> Result<Point, AppError> {
|
||||
if let Some(ref_id) = ref_id {
|
||||
let (_entry, handle) = resolve_ref(ref_id, adapter)?;
|
||||
let (_entry, handle) = resolve_ref(ref_id, snapshot_id, adapter)?;
|
||||
let bounds = adapter
|
||||
.get_element_bounds(&handle)?
|
||||
.get_element_bounds(handle.handle())?
|
||||
.ok_or_else(|| AppError::invalid_input(format!("Element {ref_id} has no bounds")))?;
|
||||
Ok(Point {
|
||||
x: bounds.x + bounds.width / 2.0,
|
||||
|
|
|
|||
|
|
@ -1,13 +1,11 @@
|
|||
use crate::{
|
||||
action::Action,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{resolve_ref, RefArgs},
|
||||
commands::helpers::{execute_ref_action, RefArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
pub fn execute(args: RefArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::Expand)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
execute_ref_action(args, adapter, ActionRequest::headless(Action::Expand))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,11 @@
|
|||
use crate::{adapter::PlatformAdapter, error::AppError, node::AccessibilityNode, snapshot};
|
||||
use crate::{
|
||||
adapter::PlatformAdapter, commands::search_text, error::AppError, node::AccessibilityNode,
|
||||
snapshot,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
const DEFAULT_LIMIT: usize = 50;
|
||||
|
||||
pub struct FindArgs {
|
||||
pub app: Option<String>,
|
||||
pub role: Option<String>,
|
||||
|
|
@ -11,19 +16,29 @@ pub struct FindArgs {
|
|||
pub first: bool,
|
||||
pub last: bool,
|
||||
pub nth: Option<usize>,
|
||||
pub limit: Option<usize>,
|
||||
}
|
||||
|
||||
pub fn execute(args: FindArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
validate_find_mode(&args)?;
|
||||
let opts = crate::adapter::TreeOptions::default();
|
||||
let result = snapshot::run(adapter, &opts, args.app.as_deref(), None)?;
|
||||
|
||||
let mut matches = Vec::new();
|
||||
search_tree(&result.tree, &args, &mut Vec::new(), &mut matches);
|
||||
let query = FindQuery::from_args(&args);
|
||||
|
||||
if args.count {
|
||||
return Ok(json!({ "count": matches.len() }));
|
||||
return Ok(json!({ "count": count_matches(&result.tree, &query) }));
|
||||
}
|
||||
|
||||
let mut matches = Vec::new();
|
||||
let max_matches = max_matches_for_args(&args);
|
||||
search_tree(
|
||||
&result.tree,
|
||||
&query,
|
||||
&mut Vec::new(),
|
||||
&mut matches,
|
||||
max_matches,
|
||||
);
|
||||
|
||||
if args.first {
|
||||
return Ok(json!({ "match": matches.into_iter().next() }));
|
||||
}
|
||||
|
|
@ -39,45 +54,70 @@ pub fn execute(args: FindArgs, adapter: &dyn PlatformAdapter) -> Result<Value, A
|
|||
Ok(json!({ "matches": matches }))
|
||||
}
|
||||
|
||||
fn max_matches_for_args(args: &FindArgs) -> Option<usize> {
|
||||
if args.count || args.last {
|
||||
return None;
|
||||
}
|
||||
if args.first {
|
||||
return Some(1);
|
||||
}
|
||||
if let Some(n) = args.nth {
|
||||
return Some(n.saturating_add(1));
|
||||
}
|
||||
match args.limit.unwrap_or(DEFAULT_LIMIT) {
|
||||
0 => None,
|
||||
limit => Some(limit),
|
||||
}
|
||||
}
|
||||
|
||||
fn validate_find_mode(args: &FindArgs) -> Result<(), AppError> {
|
||||
let selector_count = [args.count, args.first, args.last, args.nth.is_some()]
|
||||
.into_iter()
|
||||
.filter(|selected| *selected)
|
||||
.count();
|
||||
if selector_count > 1 || (selector_count == 1 && args.limit.is_some()) {
|
||||
return Err(AppError::invalid_input_with_suggestion(
|
||||
"find accepts only one result-shaping mode",
|
||||
"Use one of --count, --first, --last, --nth, or --limit.",
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
struct FindQuery<'a> {
|
||||
role: Option<&'a str>,
|
||||
name: Option<String>,
|
||||
value: Option<String>,
|
||||
text: Option<String>,
|
||||
}
|
||||
|
||||
impl<'a> FindQuery<'a> {
|
||||
fn from_args(args: &'a FindArgs) -> Self {
|
||||
Self {
|
||||
role: args.role.as_deref(),
|
||||
name: args.name.as_deref().map(search_text::normalize),
|
||||
value: args.value.as_deref().map(search_text::normalize),
|
||||
text: args.text.as_deref().map(search_text::normalize),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn search_tree(
|
||||
node: &AccessibilityNode,
|
||||
args: &FindArgs,
|
||||
query: &FindQuery,
|
||||
path: &mut Vec<String>,
|
||||
matches: &mut Vec<Value>,
|
||||
) {
|
||||
let role_match = args.role.as_deref().is_none_or(|r| node.role == r);
|
||||
let name_match = args.name.as_deref().is_none_or(|n| {
|
||||
node.name
|
||||
.as_deref()
|
||||
.is_some_and(|name| name.to_lowercase().contains(&n.to_lowercase()))
|
||||
});
|
||||
let value_match = args.value.as_deref().is_none_or(|v| {
|
||||
node.value
|
||||
.as_deref()
|
||||
.is_some_and(|val| val.to_lowercase().contains(&v.to_lowercase()))
|
||||
});
|
||||
let text_match = args.text.as_deref().is_none_or(|t| {
|
||||
let t_lower = t.to_lowercase();
|
||||
let in_name = node
|
||||
.name
|
||||
.as_deref()
|
||||
.is_some_and(|n| n.to_lowercase().contains(&t_lower));
|
||||
let in_value = node
|
||||
.value
|
||||
.as_deref()
|
||||
.is_some_and(|v| v.to_lowercase().contains(&t_lower));
|
||||
let in_desc = node
|
||||
.description
|
||||
.as_deref()
|
||||
.is_some_and(|d| d.to_lowercase().contains(&t_lower));
|
||||
in_name || in_value || in_desc
|
||||
});
|
||||
|
||||
if role_match && name_match && value_match && text_match {
|
||||
max_matches: Option<usize>,
|
||||
) -> bool {
|
||||
if max_matches.is_some_and(|limit| matches.len() >= limit) {
|
||||
return true;
|
||||
}
|
||||
if node_matches(node, query) {
|
||||
let interactive = node.ref_id.is_some();
|
||||
let display_name = node
|
||||
.name
|
||||
.as_deref()
|
||||
.or(node.value.as_deref())
|
||||
.or(node.description.as_deref())
|
||||
.map(String::from)
|
||||
.unwrap_or_else(|| format!("(unnamed {})", node.role));
|
||||
|
|
@ -89,18 +129,194 @@ fn search_tree(
|
|||
"interactive": interactive,
|
||||
"path": path.clone()
|
||||
}));
|
||||
if max_matches.is_some_and(|limit| matches.len() >= limit) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
let label = if let Some(name) = &node.name {
|
||||
format!("{}:{}", node.role, name)
|
||||
} else {
|
||||
node.role.clone()
|
||||
};
|
||||
let label = node
|
||||
.name
|
||||
.as_deref()
|
||||
.or(node.value.as_deref())
|
||||
.map(|label| format!("{}:{label}", node.role))
|
||||
.unwrap_or_else(|| node.role.clone());
|
||||
path.push(label);
|
||||
|
||||
for child in &node.children {
|
||||
search_tree(child, args, path, matches);
|
||||
if search_tree(child, query, path, matches, max_matches) {
|
||||
path.pop();
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
path.pop();
|
||||
false
|
||||
}
|
||||
|
||||
fn count_matches(node: &AccessibilityNode, query: &FindQuery) -> usize {
|
||||
usize::from(node_matches(node, query))
|
||||
+ node
|
||||
.children
|
||||
.iter()
|
||||
.map(|child| count_matches(child, query))
|
||||
.sum::<usize>()
|
||||
}
|
||||
|
||||
fn node_matches(node: &AccessibilityNode, query: &FindQuery) -> bool {
|
||||
let role_match = query.role.is_none_or(|r| node.role == r);
|
||||
let name_match = query.name.as_deref().is_none_or(|n| {
|
||||
node.name
|
||||
.as_deref()
|
||||
.is_some_and(|text| search_text::contains(text, n))
|
||||
});
|
||||
let value_match = query.value.as_deref().is_none_or(|v| {
|
||||
node.value
|
||||
.as_deref()
|
||||
.is_some_and(|val| search_text::contains(val, v))
|
||||
});
|
||||
let text_match = query
|
||||
.text
|
||||
.as_deref()
|
||||
.is_none_or(|t| search_text::node_contains(node, t));
|
||||
role_match && name_match && value_match && text_match
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn node(
|
||||
name: Option<&str>,
|
||||
value: Option<&str>,
|
||||
description: Option<&str>,
|
||||
) -> AccessibilityNode {
|
||||
AccessibilityNode {
|
||||
ref_id: Some("@e1".into()),
|
||||
role: "textfield".into(),
|
||||
name: name.map(String::from),
|
||||
value: value.map(String::from),
|
||||
description: description.map(String::from),
|
||||
hint: None,
|
||||
states: vec![],
|
||||
available_actions: vec![],
|
||||
bounds: None,
|
||||
children_count: None,
|
||||
children: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn display_name_prefers_value_before_description() {
|
||||
let root = node(None, Some("current value"), Some("help text"));
|
||||
let query = FindQuery {
|
||||
role: None,
|
||||
name: None,
|
||||
value: None,
|
||||
text: None,
|
||||
};
|
||||
let mut matches = Vec::new();
|
||||
|
||||
search_tree(&root, &query, &mut Vec::new(), &mut matches, None);
|
||||
|
||||
assert_eq!(matches[0]["name"], "current value");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn search_tree_matches_text_across_fields() {
|
||||
let root = node(None, Some("Primary"), Some("Secondary"));
|
||||
let query = FindQuery {
|
||||
role: None,
|
||||
name: None,
|
||||
value: None,
|
||||
text: Some(search_text::normalize("secondary")),
|
||||
};
|
||||
let mut matches = Vec::new();
|
||||
|
||||
search_tree(&root, &query, &mut Vec::new(), &mut matches, None);
|
||||
|
||||
assert_eq!(matches.len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn default_limit_caps_materialized_matches() {
|
||||
let root = AccessibilityNode {
|
||||
ref_id: None,
|
||||
role: "window".into(),
|
||||
name: None,
|
||||
value: None,
|
||||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
available_actions: vec![],
|
||||
bounds: None,
|
||||
children_count: None,
|
||||
children: (0..60)
|
||||
.map(|i| node(Some(&format!("Button {i}")), None, None))
|
||||
.collect(),
|
||||
};
|
||||
let query = FindQuery {
|
||||
role: None,
|
||||
name: None,
|
||||
value: None,
|
||||
text: Some(search_text::normalize("button")),
|
||||
};
|
||||
let mut matches = Vec::new();
|
||||
|
||||
search_tree(
|
||||
&root,
|
||||
&query,
|
||||
&mut Vec::new(),
|
||||
&mut matches,
|
||||
Some(DEFAULT_LIMIT),
|
||||
);
|
||||
|
||||
assert_eq!(matches.len(), DEFAULT_LIMIT);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn limit_conflicts_with_single_result_modes_for_batch_too() {
|
||||
let err = validate_find_mode(&FindArgs {
|
||||
app: None,
|
||||
role: None,
|
||||
name: None,
|
||||
value: None,
|
||||
text: None,
|
||||
count: false,
|
||||
first: true,
|
||||
last: false,
|
||||
nth: None,
|
||||
limit: Some(10),
|
||||
})
|
||||
.unwrap_err();
|
||||
|
||||
assert_eq!(err.code(), "INVALID_ARGS");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn count_matches_does_not_build_result_json() {
|
||||
let root = AccessibilityNode {
|
||||
ref_id: None,
|
||||
role: "window".into(),
|
||||
name: None,
|
||||
value: None,
|
||||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
available_actions: vec![],
|
||||
bounds: None,
|
||||
children_count: None,
|
||||
children: vec![
|
||||
node(Some("Save"), None, None),
|
||||
node(Some("Cancel"), None, None),
|
||||
],
|
||||
};
|
||||
let query = FindQuery {
|
||||
role: None,
|
||||
name: None,
|
||||
value: None,
|
||||
text: Some(search_text::normalize("a")),
|
||||
};
|
||||
|
||||
assert_eq!(count_matches(&root, &query), 2);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,13 +1,11 @@
|
|||
use crate::{
|
||||
action::Action,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{resolve_ref, RefArgs},
|
||||
commands::helpers::{execute_ref_action, RefArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
pub fn execute(args: RefArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::SetFocus)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
execute_ref_action(args, adapter, ActionRequest::headless(Action::SetFocus))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,8 +1,16 @@
|
|||
use crate::{
|
||||
adapter::{PlatformAdapter, WindowFilter},
|
||||
error::AppError,
|
||||
error::{AdapterError, AppError, ErrorCode},
|
||||
node::WindowInfo,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
#[cfg(not(test))]
|
||||
const FOCUS_SETTLE_TIMEOUT_MS: u64 = 750;
|
||||
#[cfg(test)]
|
||||
const FOCUS_SETTLE_TIMEOUT_MS: u64 = 250;
|
||||
const FOCUS_CONFIRMATIONS: u8 = 2;
|
||||
|
||||
pub struct FocusWindowArgs {
|
||||
pub window_id: Option<String>,
|
||||
|
|
@ -43,6 +51,211 @@ pub fn execute(args: FocusWindowArgs, adapter: &dyn PlatformAdapter) -> Result<V
|
|||
)
|
||||
})?;
|
||||
|
||||
let window_id = window.id.clone();
|
||||
adapter.focus_window(&window)?;
|
||||
Ok(json!({ "focused": window }))
|
||||
let focused = wait_for_focused_window(adapter, &window_id, args.app)?;
|
||||
Ok(json!({ "focused": focused }))
|
||||
}
|
||||
|
||||
fn wait_for_focused_window(
|
||||
adapter: &dyn PlatformAdapter,
|
||||
window_id: &str,
|
||||
app: Option<String>,
|
||||
) -> Result<WindowInfo, AppError> {
|
||||
let deadline = Instant::now() + Duration::from_millis(FOCUS_SETTLE_TIMEOUT_MS);
|
||||
let mut confirmations = 0;
|
||||
loop {
|
||||
match observed_focused_window(adapter, app.as_deref())? {
|
||||
Some(window) if window.id == window_id => {
|
||||
confirmations += 1;
|
||||
if confirmations >= FOCUS_CONFIRMATIONS {
|
||||
return Ok(window);
|
||||
}
|
||||
}
|
||||
_ => {
|
||||
confirmations = 0;
|
||||
}
|
||||
}
|
||||
|
||||
if Instant::now() >= deadline {
|
||||
return Err(AppError::Adapter(
|
||||
AdapterError::new(
|
||||
ErrorCode::ActionFailed,
|
||||
"Window focus did not settle on the requested window",
|
||||
)
|
||||
.with_suggestion("Run 'list-windows' to refresh window IDs, then retry."),
|
||||
));
|
||||
}
|
||||
|
||||
std::thread::sleep(Duration::from_millis(50));
|
||||
}
|
||||
}
|
||||
|
||||
fn observed_focused_window(
|
||||
adapter: &dyn PlatformAdapter,
|
||||
app: Option<&str>,
|
||||
) -> Result<Option<WindowInfo>, AppError> {
|
||||
match adapter.focused_window() {
|
||||
Ok(window) => Ok(window),
|
||||
Err(err) if err.code == ErrorCode::PlatformNotSupported => adapter
|
||||
.list_windows(&WindowFilter {
|
||||
focused_only: true,
|
||||
app: app.map(str::to_string),
|
||||
})
|
||||
.map(|windows| windows.into_iter().next())
|
||||
.map_err(AppError::Adapter),
|
||||
Err(err) => Err(AppError::Adapter(err)),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::sync::Mutex;
|
||||
|
||||
struct FocusAdapter {
|
||||
windows: Vec<WindowInfo>,
|
||||
focused_windows: Mutex<Vec<WindowInfo>>,
|
||||
focused_window_calls: Mutex<u32>,
|
||||
focused_window_supported: bool,
|
||||
}
|
||||
|
||||
impl PlatformAdapter for FocusAdapter {
|
||||
fn list_windows(&self, filter: &WindowFilter) -> Result<Vec<WindowInfo>, AdapterError> {
|
||||
if filter.focused_only {
|
||||
Ok(self.focused_windows.lock().unwrap().clone())
|
||||
} else {
|
||||
Ok(self.windows.clone())
|
||||
}
|
||||
}
|
||||
|
||||
fn focus_window(&self, _win: &WindowInfo) -> Result<(), AdapterError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn focused_window(&self) -> Result<Option<WindowInfo>, AdapterError> {
|
||||
*self.focused_window_calls.lock().unwrap() += 1;
|
||||
if !self.focused_window_supported {
|
||||
return Err(AdapterError::not_supported("focused_window"));
|
||||
}
|
||||
let mut focused = self.focused_windows.lock().unwrap();
|
||||
if focused.len() > 1 {
|
||||
Ok(Some(focused.remove(0)))
|
||||
} else {
|
||||
Ok(focused.first().cloned())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn window(id: &str, focused: bool) -> WindowInfo {
|
||||
WindowInfo {
|
||||
id: id.into(),
|
||||
title: "Main".into(),
|
||||
app: "TextEdit".into(),
|
||||
pid: 42,
|
||||
bounds: None,
|
||||
is_focused: focused,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reports_focused_window_after_os_confirms_focus() {
|
||||
let target = window("w1", false);
|
||||
let adapter = FocusAdapter {
|
||||
windows: vec![target.clone()],
|
||||
focused_windows: Mutex::new(vec![window("w1", true)]),
|
||||
focused_window_calls: Mutex::new(0),
|
||||
focused_window_supported: true,
|
||||
};
|
||||
|
||||
let value = execute(
|
||||
FocusWindowArgs {
|
||||
window_id: Some(target.id),
|
||||
app: None,
|
||||
title: None,
|
||||
},
|
||||
&adapter,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(value["focused"]["id"], "w1");
|
||||
assert_eq!(value["focused"]["is_focused"], true);
|
||||
assert_eq!(*adapter.focused_window_calls.lock().unwrap(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn errors_when_focus_does_not_settle_on_requested_window() {
|
||||
let target = window("w1", false);
|
||||
let adapter = FocusAdapter {
|
||||
windows: vec![target.clone()],
|
||||
focused_windows: Mutex::new(Vec::new()),
|
||||
focused_window_calls: Mutex::new(0),
|
||||
focused_window_supported: true,
|
||||
};
|
||||
|
||||
let err = execute(
|
||||
FocusWindowArgs {
|
||||
window_id: Some(target.id),
|
||||
app: None,
|
||||
title: None,
|
||||
},
|
||||
&adapter,
|
||||
)
|
||||
.unwrap_err();
|
||||
|
||||
assert_eq!(err.code(), "ACTION_FAILED");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn falls_back_to_focused_window_list_when_direct_observation_is_unsupported() {
|
||||
let target = window("w1", false);
|
||||
let adapter = FocusAdapter {
|
||||
windows: vec![target.clone()],
|
||||
focused_windows: Mutex::new(vec![window("w1", true)]),
|
||||
focused_window_calls: Mutex::new(0),
|
||||
focused_window_supported: false,
|
||||
};
|
||||
|
||||
let value = execute(
|
||||
FocusWindowArgs {
|
||||
window_id: Some(target.id),
|
||||
app: None,
|
||||
title: None,
|
||||
},
|
||||
&adapter,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(value["focused"]["id"], "w1");
|
||||
assert_eq!(*adapter.focused_window_calls.lock().unwrap(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn focus_confirmation_resets_after_transient_wrong_window() {
|
||||
let target = window("w1", false);
|
||||
let adapter = FocusAdapter {
|
||||
windows: vec![target.clone()],
|
||||
focused_windows: Mutex::new(vec![
|
||||
window("w1", true),
|
||||
window("w2", true),
|
||||
window("w1", true),
|
||||
window("w1", true),
|
||||
]),
|
||||
focused_window_calls: Mutex::new(0),
|
||||
focused_window_supported: true,
|
||||
};
|
||||
|
||||
let value = execute(
|
||||
FocusWindowArgs {
|
||||
window_id: Some(target.id),
|
||||
app: None,
|
||||
title: None,
|
||||
},
|
||||
&adapter,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(value["focused"]["id"], "w1");
|
||||
assert_eq!(*adapter.focused_window_calls.lock().unwrap(), 4);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ use serde_json::{json, Value};
|
|||
|
||||
pub struct GetArgs {
|
||||
pub ref_id: String,
|
||||
pub snapshot_id: Option<String>,
|
||||
pub property: GetProperty,
|
||||
}
|
||||
|
||||
|
|
@ -16,13 +17,13 @@ pub enum GetProperty {
|
|||
}
|
||||
|
||||
pub fn execute(args: GetArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let (entry, handle) = resolve_ref(&args.ref_id, args.snapshot_id.as_deref(), adapter)?;
|
||||
|
||||
let value = match args.property {
|
||||
GetProperty::Role => json!(entry.role),
|
||||
GetProperty::Title => json!(entry.name),
|
||||
GetProperty::Text | GetProperty::Value => {
|
||||
let live = adapter.get_live_value(&handle).ok().flatten();
|
||||
let live = adapter.get_live_value(handle.handle()).ok().flatten();
|
||||
json!(live.or(entry.value))
|
||||
}
|
||||
GetProperty::Bounds => json!(entry.bounds),
|
||||
|
|
|
|||
|
|
@ -1,19 +1,32 @@
|
|||
use crate::{
|
||||
adapter::{NativeHandle, PlatformAdapter, WindowFilter},
|
||||
action::{ActionRequest, WindowOp},
|
||||
adapter::{PlatformAdapter, WindowFilter},
|
||||
commands::resolved_element::ResolvedElement,
|
||||
error::AppError,
|
||||
refs::{RefEntry, RefMap},
|
||||
node::WindowInfo,
|
||||
refs::RefEntry,
|
||||
refs_store::RefStore,
|
||||
window_lookup,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
pub struct AppArgs {
|
||||
pub app: Option<String>,
|
||||
}
|
||||
|
||||
pub struct RefArgs {
|
||||
pub ref_id: String,
|
||||
pub snapshot_id: Option<String>,
|
||||
}
|
||||
|
||||
pub fn resolve_ref(
|
||||
pub fn resolve_ref<'a>(
|
||||
ref_id: &str,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
) -> Result<(RefEntry, NativeHandle), AppError> {
|
||||
snapshot_id: Option<&str>,
|
||||
adapter: &'a dyn PlatformAdapter,
|
||||
) -> Result<(RefEntry, ResolvedElement<'a>), AppError> {
|
||||
validate_ref_id(ref_id)?;
|
||||
let refmap = RefMap::load().map_err(|e| {
|
||||
let store = RefStore::new()?;
|
||||
let refmap = store.load(snapshot_id).map_err(|e| {
|
||||
tracing::debug!("refmap load failed: {e}");
|
||||
AppError::stale_ref(ref_id)
|
||||
})?;
|
||||
|
|
@ -30,14 +43,15 @@ pub fn resolve_ref(
|
|||
);
|
||||
let handle = adapter.resolve_element(&entry)?;
|
||||
tracing::debug!("resolve: {} resolved successfully", ref_id);
|
||||
Ok((entry, handle))
|
||||
Ok((entry, ResolvedElement::new(adapter, handle)))
|
||||
}
|
||||
|
||||
pub fn validate_ref_id(ref_id: &str) -> Result<(), AppError> {
|
||||
let valid = ref_id.starts_with("@e")
|
||||
&& ref_id.len() >= 3
|
||||
&& ref_id.len() <= 12
|
||||
&& ref_id[2..].chars().all(|c| c.is_ascii_digit());
|
||||
&& ref_id[2..].chars().all(|c| c.is_ascii_digit())
|
||||
&& ref_id[2..].parse::<u32>().is_ok_and(|n| n > 0);
|
||||
if !valid {
|
||||
return Err(AppError::invalid_input(format!(
|
||||
"Invalid ref_id '{ref_id}': must match @e{{N}} where N is a positive integer"
|
||||
|
|
@ -66,9 +80,204 @@ pub fn resolve_app_pid(app: Option<&str>, adapter: &dyn PlatformAdapter) -> Resu
|
|||
}
|
||||
}
|
||||
|
||||
pub fn execute_ref_action(
|
||||
args: RefArgs,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
request: ActionRequest,
|
||||
) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, args.snapshot_id.as_deref(), adapter)?;
|
||||
let result = adapter.execute_action(handle.handle(), request)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
}
|
||||
|
||||
pub fn window_op_command(
|
||||
args: AppArgs,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
op: WindowOp,
|
||||
response_key: &'static str,
|
||||
) -> Result<Value, AppError> {
|
||||
let pid = resolve_app_pid(args.app.as_deref(), adapter)?;
|
||||
let win = match find_window_for_pid(pid, adapter) {
|
||||
Ok(win) => win,
|
||||
Err(_) if matches!(op, WindowOp::Restore) => WindowInfo {
|
||||
id: String::new(),
|
||||
title: String::new(),
|
||||
app: args.app.unwrap_or_default(),
|
||||
pid,
|
||||
bounds: None,
|
||||
is_focused: false,
|
||||
},
|
||||
Err(err) => return Err(err),
|
||||
};
|
||||
adapter.window_op(&win, op)?;
|
||||
Ok(json!({ response_key: true }))
|
||||
}
|
||||
|
||||
pub fn find_window_for_pid(
|
||||
pid: i32,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
) -> Result<WindowInfo, AppError> {
|
||||
window_lookup::find_window_for_pid(pid, adapter)
|
||||
}
|
||||
|
||||
pub fn resolve_window_for_app(
|
||||
app: Option<&str>,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
) -> Result<WindowInfo, AppError> {
|
||||
let pid = resolve_app_pid(app, adapter)?;
|
||||
find_window_for_pid(pid, adapter)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::action::{Action, ActionResult, InteractionPolicy};
|
||||
use crate::adapter::NativeHandle;
|
||||
use crate::error::{AdapterError, ErrorCode};
|
||||
use crate::node::AppInfo;
|
||||
use crate::refs::RefMap;
|
||||
use crate::refs_test_support::HomeGuard;
|
||||
use std::sync::atomic::{AtomicU32, Ordering};
|
||||
use std::sync::Mutex;
|
||||
|
||||
struct ReleaseCountingAdapter {
|
||||
releases: AtomicU32,
|
||||
}
|
||||
|
||||
impl PlatformAdapter for ReleaseCountingAdapter {
|
||||
fn resolve_element(&self, _entry: &RefEntry) -> Result<NativeHandle, AdapterError> {
|
||||
Ok(NativeHandle::null())
|
||||
}
|
||||
|
||||
fn release_handle(&self, _handle: &NativeHandle) -> Result<(), AdapterError> {
|
||||
self.releases.fetch_add(1, Ordering::SeqCst);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
struct RecordingAdapter {
|
||||
request: Mutex<Option<ActionRequest>>,
|
||||
}
|
||||
|
||||
impl PlatformAdapter for RecordingAdapter {
|
||||
fn resolve_element(&self, _entry: &RefEntry) -> Result<NativeHandle, AdapterError> {
|
||||
Ok(NativeHandle::null())
|
||||
}
|
||||
|
||||
fn execute_action(
|
||||
&self,
|
||||
_handle: &NativeHandle,
|
||||
request: ActionRequest,
|
||||
) -> Result<ActionResult, AdapterError> {
|
||||
*self.request.lock().unwrap() = Some(request);
|
||||
Ok(ActionResult::new("ok"))
|
||||
}
|
||||
}
|
||||
|
||||
struct RestoreWithoutWindowAdapter {
|
||||
op_count: AtomicU32,
|
||||
}
|
||||
|
||||
impl PlatformAdapter for RestoreWithoutWindowAdapter {
|
||||
fn list_apps(&self) -> Result<Vec<AppInfo>, AdapterError> {
|
||||
Ok(vec![AppInfo {
|
||||
name: "TextEdit".into(),
|
||||
pid: 42,
|
||||
bundle_id: None,
|
||||
}])
|
||||
}
|
||||
|
||||
fn list_windows(
|
||||
&self,
|
||||
_filter: &crate::adapter::WindowFilter,
|
||||
) -> Result<Vec<WindowInfo>, AdapterError> {
|
||||
Err(AdapterError::new(ErrorCode::WindowNotFound, "no windows"))
|
||||
}
|
||||
|
||||
fn window_op(&self, win: &WindowInfo, op: WindowOp) -> Result<(), AdapterError> {
|
||||
assert_eq!(win.pid, 42);
|
||||
assert!(matches!(op, WindowOp::Restore));
|
||||
self.op_count.fetch_add(1, Ordering::SeqCst);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn entry() -> RefEntry {
|
||||
RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some("OK".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
source_window_title: None,
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolved_element_releases_handle_once_on_drop() {
|
||||
let _guard = HomeGuard::new();
|
||||
let mut refmap = RefMap::new();
|
||||
refmap.allocate(entry());
|
||||
let snapshot_id = RefStore::new().unwrap().save_new_snapshot(&refmap).unwrap();
|
||||
let adapter = ReleaseCountingAdapter {
|
||||
releases: AtomicU32::new(0),
|
||||
};
|
||||
|
||||
{
|
||||
let (_entry, resolved) = resolve_ref("@e1", Some(&snapshot_id), &adapter).unwrap();
|
||||
let _handle = resolved.handle();
|
||||
assert_eq!(adapter.releases.load(Ordering::SeqCst), 0);
|
||||
}
|
||||
|
||||
assert_eq!(adapter.releases.load(Ordering::SeqCst), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn execute_ref_action_preserves_action_and_policy() {
|
||||
let _guard = HomeGuard::new();
|
||||
let mut refmap = RefMap::new();
|
||||
refmap.allocate(entry());
|
||||
let snapshot_id = RefStore::new().unwrap().save_new_snapshot(&refmap).unwrap();
|
||||
let adapter = RecordingAdapter {
|
||||
request: Mutex::new(None),
|
||||
};
|
||||
let args = RefArgs {
|
||||
ref_id: "@e1".into(),
|
||||
snapshot_id: Some(snapshot_id),
|
||||
};
|
||||
|
||||
execute_ref_action(args, &adapter, ActionRequest::headless(Action::Clear)).unwrap();
|
||||
|
||||
let request = adapter.request.lock().unwrap().clone().unwrap();
|
||||
assert!(matches!(request.action, Action::Clear));
|
||||
assert_eq!(request.policy, InteractionPolicy::headless());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn restore_can_run_when_no_window_is_currently_listed() {
|
||||
let adapter = RestoreWithoutWindowAdapter {
|
||||
op_count: AtomicU32::new(0),
|
||||
};
|
||||
|
||||
let value = window_op_command(
|
||||
AppArgs {
|
||||
app: Some("TextEdit".into()),
|
||||
},
|
||||
&adapter,
|
||||
WindowOp::Restore,
|
||||
"restored",
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(value["restored"], true);
|
||||
assert_eq!(adapter.op_count.load(Ordering::SeqCst), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_valid_refs() {
|
||||
|
|
@ -82,6 +291,7 @@ mod tests {
|
|||
assert!(validate_ref_id("@").is_err());
|
||||
assert!(validate_ref_id("e1").is_err());
|
||||
assert!(validate_ref_id("@e").is_err());
|
||||
assert!(validate_ref_id("@e0").is_err());
|
||||
assert!(validate_ref_id("@e0abc").is_err());
|
||||
assert!(validate_ref_id("1").is_err());
|
||||
assert!(validate_ref_id("").is_err());
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ use serde_json::{json, Value};
|
|||
|
||||
pub struct HoverArgs {
|
||||
pub ref_id: Option<String>,
|
||||
pub snapshot_id: Option<String>,
|
||||
pub xy: Option<(f64, f64)>,
|
||||
pub duration_ms: Option<u64>,
|
||||
}
|
||||
|
|
@ -27,9 +28,9 @@ pub fn execute(args: HoverArgs, adapter: &dyn PlatformAdapter) -> Result<Value,
|
|||
|
||||
fn resolve_hover_point(args: &HoverArgs, adapter: &dyn PlatformAdapter) -> Result<Point, AppError> {
|
||||
if let Some(ref_id) = &args.ref_id {
|
||||
let (_entry, handle) = resolve_ref(ref_id, adapter)?;
|
||||
let (_entry, handle) = resolve_ref(ref_id, args.snapshot_id.as_deref(), adapter)?;
|
||||
let bounds = adapter
|
||||
.get_element_bounds(&handle)?
|
||||
.get_element_bounds(handle.handle())?
|
||||
.ok_or_else(|| AppError::invalid_input(format!("Element {ref_id} has no bounds")))?;
|
||||
Ok(Point {
|
||||
x: bounds.x + bounds.width / 2.0,
|
||||
|
|
|
|||
|
|
@ -1,8 +1,12 @@
|
|||
use crate::{adapter::PlatformAdapter, commands::helpers::resolve_ref, error::AppError};
|
||||
use crate::{
|
||||
action::ElementState, adapter::PlatformAdapter, commands::helpers::resolve_ref,
|
||||
error::AppError, refs::RefEntry,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
pub struct IsArgs {
|
||||
pub ref_id: String,
|
||||
pub snapshot_id: Option<String>,
|
||||
pub property: IsProperty,
|
||||
}
|
||||
|
||||
|
|
@ -14,11 +18,14 @@ pub enum IsProperty {
|
|||
Expanded,
|
||||
}
|
||||
|
||||
/// States are read from the last snapshot's RefMap. `resolve_ref` verifies the element
|
||||
/// is still live before returning, but the state values themselves are not re-queried
|
||||
/// from the AX API. Run `snapshot` to refresh state before calling `is`.
|
||||
/// State is read live when the platform supports it, then falls back to snapshot state.
|
||||
pub fn execute(args: IsArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (entry, _handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let (entry, handle) = resolve_ref(&args.ref_id, args.snapshot_id.as_deref(), adapter)?;
|
||||
let state = adapter
|
||||
.get_live_state(handle.handle())
|
||||
.ok()
|
||||
.flatten()
|
||||
.unwrap_or_else(|| state_from_ref_entry(&entry));
|
||||
|
||||
let prop_name = match args.property {
|
||||
IsProperty::Visible => "visible",
|
||||
|
|
@ -28,14 +35,14 @@ pub fn execute(args: IsArgs, adapter: &dyn PlatformAdapter) -> Result<Value, App
|
|||
IsProperty::Expanded => "expanded",
|
||||
};
|
||||
|
||||
let applicable = is_applicable(&args.property, &entry.role);
|
||||
let applicable = is_applicable(&args.property, &entry, &state);
|
||||
|
||||
let result = match args.property {
|
||||
IsProperty::Visible => !entry.states.contains(&"hidden".to_string()),
|
||||
IsProperty::Enabled => !entry.states.contains(&"disabled".to_string()),
|
||||
IsProperty::Checked => entry.states.contains(&"checked".to_string()),
|
||||
IsProperty::Focused => entry.states.contains(&"focused".to_string()),
|
||||
IsProperty::Expanded => entry.states.contains(&"expanded".to_string()),
|
||||
IsProperty::Visible => !has_state(&state, "hidden"),
|
||||
IsProperty::Enabled => !has_state(&state, "disabled"),
|
||||
IsProperty::Checked => has_state(&state, "checked"),
|
||||
IsProperty::Focused => has_state(&state, "focused"),
|
||||
IsProperty::Expanded => has_state(&state, "expanded"),
|
||||
};
|
||||
|
||||
Ok(
|
||||
|
|
@ -43,21 +50,226 @@ pub fn execute(args: IsArgs, adapter: &dyn PlatformAdapter) -> Result<Value, App
|
|||
)
|
||||
}
|
||||
|
||||
fn is_applicable(property: &IsProperty, role: &str) -> bool {
|
||||
match property {
|
||||
IsProperty::Visible | IsProperty::Enabled | IsProperty::Focused => true,
|
||||
IsProperty::Checked => matches!(
|
||||
role,
|
||||
"checkbox"
|
||||
| "switch"
|
||||
| "radiobutton"
|
||||
| "togglebutton"
|
||||
| "menuitemcheckbox"
|
||||
| "menuitemradio"
|
||||
),
|
||||
IsProperty::Expanded => matches!(
|
||||
role,
|
||||
"disclosuretriangle" | "treeitem" | "combobox" | "popupbutton" | "outline" | "row"
|
||||
),
|
||||
fn state_from_ref_entry(entry: &RefEntry) -> ElementState {
|
||||
ElementState {
|
||||
role: entry.role.clone(),
|
||||
states: entry.states.clone(),
|
||||
value: entry.value.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
fn has_state(state: &ElementState, name: &str) -> bool {
|
||||
state.states.iter().any(|s| s == name)
|
||||
}
|
||||
|
||||
fn is_applicable(property: &IsProperty, entry: &RefEntry, state: &ElementState) -> bool {
|
||||
match property {
|
||||
IsProperty::Visible | IsProperty::Enabled | IsProperty::Focused => true,
|
||||
IsProperty::Checked => {
|
||||
crate::roles::is_toggleable_role(&entry.role)
|
||||
|| has_state(state, "checked")
|
||||
|| has_available_action(entry, "Toggle")
|
||||
|| has_available_action(entry, "Check")
|
||||
|| has_available_action(entry, "Uncheck")
|
||||
}
|
||||
IsProperty::Expanded => {
|
||||
crate::roles::is_expandable_role(&entry.role)
|
||||
|| has_state(state, "expanded")
|
||||
|| has_available_action(entry, "Expand")
|
||||
|| has_available_action(entry, "Collapse")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn has_available_action(entry: &RefEntry, action: &str) -> bool {
|
||||
entry.available_actions.iter().any(|a| a == action)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::{
|
||||
adapter::NativeHandle, error::AdapterError, refs::RefMap, refs_store::RefStore,
|
||||
refs_test_support::HomeGuard,
|
||||
};
|
||||
use std::sync::Mutex;
|
||||
|
||||
struct LiveStateAdapter {
|
||||
state: Mutex<Option<ElementState>>,
|
||||
}
|
||||
|
||||
impl PlatformAdapter for LiveStateAdapter {
|
||||
fn resolve_element(&self, _entry: &RefEntry) -> Result<NativeHandle, AdapterError> {
|
||||
Ok(NativeHandle::null())
|
||||
}
|
||||
|
||||
fn get_live_state(
|
||||
&self,
|
||||
_handle: &NativeHandle,
|
||||
) -> Result<Option<ElementState>, AdapterError> {
|
||||
Ok(self.state.lock().unwrap().clone())
|
||||
}
|
||||
}
|
||||
|
||||
fn save_entry(entry: RefEntry) -> String {
|
||||
let mut refmap = RefMap::new();
|
||||
refmap.allocate(entry);
|
||||
RefStore::new().unwrap().save_new_snapshot(&refmap).unwrap()
|
||||
}
|
||||
|
||||
fn entry(states: Vec<String>, value: Option<&str>, actions: Vec<&str>) -> RefEntry {
|
||||
RefEntry {
|
||||
pid: 1,
|
||||
role: "checkbox".into(),
|
||||
name: Some("Target".into()),
|
||||
value: value.map(str::to_string),
|
||||
states,
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: actions.into_iter().map(str::to_string).collect(),
|
||||
source_app: None,
|
||||
source_window_title: None,
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn checked_uses_live_canonical_state() {
|
||||
let _guard = HomeGuard::new();
|
||||
let snapshot_id = save_entry(entry(vec![], None, vec!["Toggle"]));
|
||||
let adapter = LiveStateAdapter {
|
||||
state: Mutex::new(Some(ElementState {
|
||||
role: "checkbox".into(),
|
||||
states: vec!["checked".into()],
|
||||
value: Some("1".into()),
|
||||
})),
|
||||
};
|
||||
|
||||
let result = execute(
|
||||
IsArgs {
|
||||
ref_id: "@e1".into(),
|
||||
snapshot_id: Some(snapshot_id),
|
||||
property: IsProperty::Checked,
|
||||
},
|
||||
&adapter,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(result["result"], true);
|
||||
assert_eq!(result["applicable"], true);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn checked_does_not_infer_platform_values_in_core() {
|
||||
let _guard = HomeGuard::new();
|
||||
let snapshot_id = save_entry(entry(vec![], Some("1"), vec!["Toggle"]));
|
||||
let adapter = LiveStateAdapter {
|
||||
state: Mutex::new(None),
|
||||
};
|
||||
|
||||
let result = execute(
|
||||
IsArgs {
|
||||
ref_id: "@e1".into(),
|
||||
snapshot_id: Some(snapshot_id),
|
||||
property: IsProperty::Checked,
|
||||
},
|
||||
&adapter,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(result["result"], false);
|
||||
assert_eq!(result["applicable"], true);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn checked_falls_back_to_snapshot_state_when_live_state_is_missing() {
|
||||
let _guard = HomeGuard::new();
|
||||
let snapshot_id = save_entry(entry(vec!["checked".into()], None, vec!["Toggle"]));
|
||||
let adapter = LiveStateAdapter {
|
||||
state: Mutex::new(None),
|
||||
};
|
||||
|
||||
let result = execute(
|
||||
IsArgs {
|
||||
ref_id: "@e1".into(),
|
||||
snapshot_id: Some(snapshot_id),
|
||||
property: IsProperty::Checked,
|
||||
},
|
||||
&adapter,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(result["result"], true);
|
||||
assert_eq!(result["applicable"], true);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn basic_state_properties_use_live_state() {
|
||||
let _guard = HomeGuard::new();
|
||||
let snapshot_id = save_entry(entry(vec![], None, vec![]));
|
||||
let adapter = LiveStateAdapter {
|
||||
state: Mutex::new(Some(ElementState {
|
||||
role: "button".into(),
|
||||
states: vec!["focused".into(), "expanded".into()],
|
||||
value: None,
|
||||
})),
|
||||
};
|
||||
|
||||
for (property, expected) in [
|
||||
(IsProperty::Visible, true),
|
||||
(IsProperty::Enabled, true),
|
||||
(IsProperty::Focused, true),
|
||||
(IsProperty::Expanded, true),
|
||||
] {
|
||||
let result = execute(
|
||||
IsArgs {
|
||||
ref_id: "@e1".into(),
|
||||
snapshot_id: Some(snapshot_id.clone()),
|
||||
property,
|
||||
},
|
||||
&adapter,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(result["result"], expected);
|
||||
assert_eq!(result["applicable"], true);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn action_availability_makes_toggle_and_expand_applicable() {
|
||||
let _guard = HomeGuard::new();
|
||||
let snapshot_id = save_entry(RefEntry {
|
||||
pid: 1,
|
||||
role: "cell".into(),
|
||||
name: Some("Disclosure".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Check".into(), "Expand".into()],
|
||||
source_app: None,
|
||||
source_window_title: None,
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
});
|
||||
let adapter = LiveStateAdapter {
|
||||
state: Mutex::new(None),
|
||||
};
|
||||
|
||||
for property in [IsProperty::Checked, IsProperty::Expanded] {
|
||||
let result = execute(
|
||||
IsArgs {
|
||||
ref_id: "@e1".into(),
|
||||
snapshot_id: Some(snapshot_id.clone()),
|
||||
property,
|
||||
},
|
||||
&adapter,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(result["applicable"], true);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,4 @@
|
|||
use crate::{
|
||||
action::Action, adapter::PlatformAdapter, commands::press::parse_combo, error::AppError,
|
||||
};
|
||||
use crate::{adapter::PlatformAdapter, commands::press::parse_combo, error::AppError};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
pub struct KeyDownArgs {
|
||||
|
|
@ -9,7 +7,6 @@ pub struct KeyDownArgs {
|
|||
|
||||
pub fn execute(args: KeyDownArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let combo = parse_combo(&args.combo)?;
|
||||
let handle = crate::adapter::NativeHandle::null();
|
||||
adapter.execute_action(&handle, Action::KeyDown(combo))?;
|
||||
adapter.key_event(&combo, true)?;
|
||||
Ok(json!({ "key_down": args.combo }))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,4 @@
|
|||
use crate::{
|
||||
action::Action, adapter::PlatformAdapter, commands::press::parse_combo, error::AppError,
|
||||
};
|
||||
use crate::{adapter::PlatformAdapter, commands::press::parse_combo, error::AppError};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
pub struct KeyUpArgs {
|
||||
|
|
@ -9,7 +7,6 @@ pub struct KeyUpArgs {
|
|||
|
||||
pub fn execute(args: KeyUpArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let combo = parse_combo(&args.combo)?;
|
||||
let handle = crate::adapter::NativeHandle::null();
|
||||
adapter.execute_action(&handle, Action::KeyUp(combo))?;
|
||||
adapter.key_event(&combo, false)?;
|
||||
Ok(json!({ "key_up": args.combo }))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,7 +1,55 @@
|
|||
use crate::{adapter::PlatformAdapter, error::AppError};
|
||||
use crate::{adapter::PlatformAdapter, commands::search_text, error::AppError};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
pub fn execute(adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let apps = adapter.list_apps()?;
|
||||
pub struct ListAppsArgs {
|
||||
pub app: Option<String>,
|
||||
}
|
||||
|
||||
pub fn execute(args: ListAppsArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let mut apps = adapter.list_apps()?;
|
||||
if let Some(app) = args.app {
|
||||
let needle = search_text::normalize(&app);
|
||||
apps.retain(|candidate| search_text::contains(&candidate.name, &needle));
|
||||
}
|
||||
Ok(json!({ "apps": apps }))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::{adapter::PlatformAdapter, error::AdapterError, node::AppInfo};
|
||||
|
||||
struct AppsAdapter;
|
||||
|
||||
impl PlatformAdapter for AppsAdapter {
|
||||
fn list_apps(&self) -> Result<Vec<AppInfo>, AdapterError> {
|
||||
Ok(vec![
|
||||
AppInfo {
|
||||
name: "Finder".into(),
|
||||
pid: 1,
|
||||
bundle_id: Some("com.apple.finder".into()),
|
||||
},
|
||||
AppInfo {
|
||||
name: "TextEdit".into(),
|
||||
pid: 2,
|
||||
bundle_id: Some("com.apple.TextEdit".into()),
|
||||
},
|
||||
])
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn app_filter_matches_by_name_case_insensitively() {
|
||||
let value = execute(
|
||||
ListAppsArgs {
|
||||
app: Some("text".into()),
|
||||
},
|
||||
&AppsAdapter,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let apps = value["apps"].as_array().unwrap();
|
||||
assert_eq!(apps.len(), 1);
|
||||
assert_eq!(apps[0]["name"], "TextEdit");
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,30 +1,11 @@
|
|||
use crate::{
|
||||
action::WindowOp, adapter::PlatformAdapter, commands::helpers::resolve_app_pid, error::AppError,
|
||||
action::WindowOp,
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{window_op_command, AppArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
use serde_json::Value;
|
||||
|
||||
pub struct MaximizeArgs {
|
||||
pub app: Option<String>,
|
||||
}
|
||||
|
||||
pub fn execute(args: MaximizeArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let pid = resolve_app_pid(args.app.as_deref(), adapter)?;
|
||||
let win = find_window(pid, adapter)?;
|
||||
adapter.window_op(&win, WindowOp::Maximize)?;
|
||||
Ok(json!({ "maximized": true }))
|
||||
}
|
||||
|
||||
fn find_window(
|
||||
pid: i32,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
) -> Result<crate::node::WindowInfo, AppError> {
|
||||
let filter = crate::adapter::WindowFilter {
|
||||
focused_only: false,
|
||||
app: None,
|
||||
};
|
||||
let windows = adapter.list_windows(&filter)?;
|
||||
windows
|
||||
.into_iter()
|
||||
.find(|w| w.pid == pid)
|
||||
.ok_or_else(|| AppError::invalid_input("No window found for this application"))
|
||||
pub fn execute(args: AppArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
window_op_command(args, adapter, WindowOp::Maximize, "maximized")
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,30 +1,11 @@
|
|||
use crate::{
|
||||
action::WindowOp, adapter::PlatformAdapter, commands::helpers::resolve_app_pid, error::AppError,
|
||||
action::WindowOp,
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{window_op_command, AppArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
use serde_json::Value;
|
||||
|
||||
pub struct MinimizeArgs {
|
||||
pub app: Option<String>,
|
||||
}
|
||||
|
||||
pub fn execute(args: MinimizeArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let pid = resolve_app_pid(args.app.as_deref(), adapter)?;
|
||||
let win = find_window(pid, adapter)?;
|
||||
adapter.window_op(&win, WindowOp::Minimize)?;
|
||||
Ok(json!({ "minimized": true }))
|
||||
}
|
||||
|
||||
fn find_window(
|
||||
pid: i32,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
) -> Result<crate::node::WindowInfo, AppError> {
|
||||
let filter = crate::adapter::WindowFilter {
|
||||
focused_only: false,
|
||||
app: None,
|
||||
};
|
||||
let windows = adapter.list_windows(&filter)?;
|
||||
windows
|
||||
.into_iter()
|
||||
.find(|w| w.pid == pid)
|
||||
.ok_or_else(|| AppError::invalid_input("No window found for this application"))
|
||||
pub fn execute(args: AppArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
window_op_command(args, adapter, WindowOp::Minimize, "minimized")
|
||||
}
|
||||
|
|
|
|||
|
|
@ -37,11 +37,13 @@ pub mod notification_action;
|
|||
pub mod permissions;
|
||||
pub mod press;
|
||||
pub mod resize_window;
|
||||
pub mod resolved_element;
|
||||
pub mod restore;
|
||||
pub mod right_click;
|
||||
pub mod screenshot;
|
||||
pub mod scroll;
|
||||
pub mod scroll_to;
|
||||
pub(crate) mod search_text;
|
||||
pub mod select;
|
||||
pub mod set_value;
|
||||
pub mod skills;
|
||||
|
|
@ -53,3 +55,6 @@ pub mod type_text;
|
|||
pub mod uncheck;
|
||||
pub mod version;
|
||||
pub mod wait;
|
||||
|
||||
#[cfg(test)]
|
||||
mod ref_policy_tests;
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
use crate::{
|
||||
action::WindowOp, adapter::PlatformAdapter, commands::helpers::resolve_app_pid, error::AppError,
|
||||
action::WindowOp, adapter::PlatformAdapter, commands::helpers::resolve_window_for_app,
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
|
|
@ -10,8 +11,7 @@ pub struct MoveWindowArgs {
|
|||
}
|
||||
|
||||
pub fn execute(args: MoveWindowArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let pid = resolve_app_pid(args.app.as_deref(), adapter)?;
|
||||
let win = find_window(pid, adapter)?;
|
||||
let win = resolve_window_for_app(args.app.as_deref(), adapter)?;
|
||||
adapter.window_op(
|
||||
&win,
|
||||
WindowOp::Move {
|
||||
|
|
@ -21,18 +21,3 @@ pub fn execute(args: MoveWindowArgs, adapter: &dyn PlatformAdapter) -> Result<Va
|
|||
)?;
|
||||
Ok(json!({ "moved": true, "x": args.x, "y": args.y }))
|
||||
}
|
||||
|
||||
fn find_window(
|
||||
pid: i32,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
) -> Result<crate::node::WindowInfo, AppError> {
|
||||
let filter = crate::adapter::WindowFilter {
|
||||
focused_only: false,
|
||||
app: None,
|
||||
};
|
||||
let windows = adapter.list_windows(&filter)?;
|
||||
windows
|
||||
.into_iter()
|
||||
.find(|w| w.pid == pid)
|
||||
.ok_or_else(|| AppError::invalid_input("No window found for this application"))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,26 +1,27 @@
|
|||
use crate::{
|
||||
adapter::{PermissionStatus, PlatformAdapter},
|
||||
error::AppError,
|
||||
};
|
||||
use crate::{adapter::PlatformAdapter, error::AppError, PermissionReport};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
pub struct PermissionsArgs {
|
||||
pub request: bool,
|
||||
}
|
||||
|
||||
pub fn execute(args: PermissionsArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
if args.request {
|
||||
return Ok(json!({
|
||||
"requested": true,
|
||||
"note": "Permission dialog triggered via --request flag"
|
||||
}));
|
||||
}
|
||||
|
||||
match adapter.check_permissions() {
|
||||
PermissionStatus::Granted => Ok(json!({ "granted": true })),
|
||||
PermissionStatus::Denied { suggestion } => Ok(json!({
|
||||
"granted": false,
|
||||
"suggestion": suggestion
|
||||
})),
|
||||
}
|
||||
pub fn execute_with_report(
|
||||
args: PermissionsArgs,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
report: &PermissionReport,
|
||||
) -> Result<Value, AppError> {
|
||||
let report = if args.request {
|
||||
adapter.request_permissions()
|
||||
} else {
|
||||
report.clone()
|
||||
};
|
||||
Ok(render(report))
|
||||
}
|
||||
|
||||
fn render(report: PermissionReport) -> Value {
|
||||
json!({
|
||||
"accessibility": report.accessibility,
|
||||
"screen_recording": report.screen_recording,
|
||||
"automation": report.automation
|
||||
})
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
use crate::{
|
||||
action::{Action, KeyCombo, Modifier},
|
||||
action::{Action, ActionRequest, KeyCombo, Modifier},
|
||||
adapter::PlatformAdapter,
|
||||
error::AppError,
|
||||
};
|
||||
|
|
@ -35,7 +35,8 @@ pub fn execute(args: PressArgs, adapter: &dyn PlatformAdapter) -> Result<Value,
|
|||
}
|
||||
|
||||
let handle = crate::adapter::NativeHandle::null();
|
||||
let result = adapter.execute_action(&handle, Action::PressKey(combo))?;
|
||||
let result =
|
||||
adapter.execute_action(&handle, ActionRequest::physical(Action::PressKey(combo)))?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
}
|
||||
|
||||
|
|
|
|||
158
crates/core/src/commands/ref_policy_tests.rs
Normal file
158
crates/core/src/commands/ref_policy_tests.rs
Normal file
|
|
@ -0,0 +1,158 @@
|
|||
use crate::{
|
||||
action::{Action, ActionRequest, ActionResult, Direction, InteractionPolicy},
|
||||
adapter::{NativeHandle, PlatformAdapter},
|
||||
commands::{
|
||||
check, clear, click, collapse, double_click, expand, focus, helpers::RefArgs, right_click,
|
||||
scroll, scroll_to, select, set_value, toggle, triple_click, type_text, uncheck,
|
||||
},
|
||||
error::AdapterError,
|
||||
refs::{RefEntry, RefMap},
|
||||
refs_store::RefStore,
|
||||
refs_test_support::HomeGuard,
|
||||
};
|
||||
use std::sync::Mutex;
|
||||
|
||||
struct RecordingAdapter {
|
||||
requests: Mutex<Vec<ActionRequest>>,
|
||||
}
|
||||
|
||||
impl RecordingAdapter {
|
||||
fn new() -> Self {
|
||||
Self {
|
||||
requests: Mutex::new(Vec::new()),
|
||||
}
|
||||
}
|
||||
|
||||
fn last_request(&self) -> ActionRequest {
|
||||
self.requests.lock().unwrap().last().cloned().unwrap()
|
||||
}
|
||||
}
|
||||
|
||||
impl PlatformAdapter for RecordingAdapter {
|
||||
fn resolve_element(&self, _entry: &RefEntry) -> Result<NativeHandle, AdapterError> {
|
||||
Ok(NativeHandle::null())
|
||||
}
|
||||
|
||||
fn execute_action(
|
||||
&self,
|
||||
_handle: &NativeHandle,
|
||||
request: ActionRequest,
|
||||
) -> Result<ActionResult, AdapterError> {
|
||||
self.requests.lock().unwrap().push(request);
|
||||
Ok(ActionResult::new("ok"))
|
||||
}
|
||||
}
|
||||
|
||||
fn snapshot_id() -> String {
|
||||
let mut refmap = RefMap::new();
|
||||
refmap.allocate(RefEntry {
|
||||
pid: 1,
|
||||
role: "textfield".into(),
|
||||
name: Some("Target".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
source_window_title: None,
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
});
|
||||
RefStore::new().unwrap().save_new_snapshot(&refmap).unwrap()
|
||||
}
|
||||
|
||||
fn ref_args(snapshot_id: &str) -> RefArgs {
|
||||
RefArgs {
|
||||
ref_id: "@e1".into(),
|
||||
snapshot_id: Some(snapshot_id.to_owned()),
|
||||
}
|
||||
}
|
||||
|
||||
fn assert_headless(request: &ActionRequest) {
|
||||
assert_eq!(request.policy, InteractionPolicy::headless());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn default_ref_commands_are_headless() {
|
||||
let _guard = HomeGuard::new();
|
||||
let snapshot_id = snapshot_id();
|
||||
let adapter = RecordingAdapter::new();
|
||||
|
||||
click::execute(ref_args(&snapshot_id), &adapter).unwrap();
|
||||
double_click::execute(ref_args(&snapshot_id), &adapter).unwrap();
|
||||
triple_click::execute(ref_args(&snapshot_id), &adapter).unwrap();
|
||||
let before_right_click = adapter.requests.lock().unwrap().len();
|
||||
let _ = right_click::execute(ref_args(&snapshot_id), &adapter);
|
||||
assert_eq!(
|
||||
adapter.requests.lock().unwrap().len(),
|
||||
before_right_click + 1
|
||||
);
|
||||
clear::execute(ref_args(&snapshot_id), &adapter).unwrap();
|
||||
toggle::execute(ref_args(&snapshot_id), &adapter).unwrap();
|
||||
check::execute(ref_args(&snapshot_id), &adapter).unwrap();
|
||||
uncheck::execute(ref_args(&snapshot_id), &adapter).unwrap();
|
||||
expand::execute(ref_args(&snapshot_id), &adapter).unwrap();
|
||||
collapse::execute(ref_args(&snapshot_id), &adapter).unwrap();
|
||||
scroll_to::execute(ref_args(&snapshot_id), &adapter).unwrap();
|
||||
set_value::execute(
|
||||
set_value::SetValueArgs {
|
||||
ref_id: "@e1".into(),
|
||||
snapshot_id: Some(snapshot_id.clone()),
|
||||
value: "value".into(),
|
||||
},
|
||||
&adapter,
|
||||
)
|
||||
.unwrap();
|
||||
select::execute(
|
||||
select::SelectArgs {
|
||||
ref_id: "@e1".into(),
|
||||
snapshot_id: Some(snapshot_id.clone()),
|
||||
value: "choice".into(),
|
||||
},
|
||||
&adapter,
|
||||
)
|
||||
.unwrap();
|
||||
let before_type = adapter.requests.lock().unwrap().len();
|
||||
type_text::execute(
|
||||
type_text::TypeArgs {
|
||||
ref_id: "@e1".into(),
|
||||
snapshot_id: Some(snapshot_id.clone()),
|
||||
text: "text".into(),
|
||||
},
|
||||
&adapter,
|
||||
)
|
||||
.unwrap();
|
||||
let type_request = adapter.requests.lock().unwrap()[before_type].clone();
|
||||
assert_eq!(type_request.policy, InteractionPolicy::focus_fallback());
|
||||
scroll::execute(
|
||||
scroll::ScrollArgs {
|
||||
ref_id: "@e1".into(),
|
||||
snapshot_id: Some(snapshot_id),
|
||||
direction: Direction::Down,
|
||||
amount: 1,
|
||||
},
|
||||
&adapter,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
for request in adapter.requests.lock().unwrap().iter() {
|
||||
if matches!(request.action, Action::TypeText(_)) {
|
||||
continue;
|
||||
}
|
||||
assert_headless(request);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn focus_command_is_explicit_headless_policy() {
|
||||
let _guard = HomeGuard::new();
|
||||
let snapshot_id = snapshot_id();
|
||||
let adapter = RecordingAdapter::new();
|
||||
|
||||
focus::execute(ref_args(&snapshot_id), &adapter).unwrap();
|
||||
|
||||
let request = adapter.last_request();
|
||||
assert!(matches!(request.action, Action::SetFocus));
|
||||
assert_eq!(request.policy, InteractionPolicy::headless());
|
||||
}
|
||||
|
|
@ -1,5 +1,6 @@
|
|||
use crate::{
|
||||
action::WindowOp, adapter::PlatformAdapter, commands::helpers::resolve_app_pid, error::AppError,
|
||||
action::WindowOp, adapter::PlatformAdapter, commands::helpers::resolve_window_for_app,
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
|
|
@ -10,8 +11,7 @@ pub struct ResizeWindowArgs {
|
|||
}
|
||||
|
||||
pub fn execute(args: ResizeWindowArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let pid = resolve_app_pid(args.app.as_deref(), adapter)?;
|
||||
let win = find_window(pid, adapter)?;
|
||||
let win = resolve_window_for_app(args.app.as_deref(), adapter)?;
|
||||
adapter.window_op(
|
||||
&win,
|
||||
WindowOp::Resize {
|
||||
|
|
@ -21,18 +21,3 @@ pub fn execute(args: ResizeWindowArgs, adapter: &dyn PlatformAdapter) -> Result<
|
|||
)?;
|
||||
Ok(json!({ "resized": true, "width": args.width, "height": args.height }))
|
||||
}
|
||||
|
||||
fn find_window(
|
||||
pid: i32,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
) -> Result<crate::node::WindowInfo, AppError> {
|
||||
let filter = crate::adapter::WindowFilter {
|
||||
focused_only: false,
|
||||
app: None,
|
||||
};
|
||||
let windows = adapter.list_windows(&filter)?;
|
||||
windows
|
||||
.into_iter()
|
||||
.find(|w| w.pid == pid)
|
||||
.ok_or_else(|| AppError::invalid_input("No window found for this application"))
|
||||
}
|
||||
|
|
|
|||
22
crates/core/src/commands/resolved_element.rs
Normal file
22
crates/core/src/commands/resolved_element.rs
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
use crate::adapter::{NativeHandle, PlatformAdapter};
|
||||
|
||||
pub struct ResolvedElement<'a> {
|
||||
adapter: &'a dyn PlatformAdapter,
|
||||
handle: NativeHandle,
|
||||
}
|
||||
|
||||
impl<'a> ResolvedElement<'a> {
|
||||
pub fn new(adapter: &'a dyn PlatformAdapter, handle: NativeHandle) -> Self {
|
||||
Self { adapter, handle }
|
||||
}
|
||||
|
||||
pub fn handle(&self) -> &NativeHandle {
|
||||
&self.handle
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for ResolvedElement<'_> {
|
||||
fn drop(&mut self) {
|
||||
let _ = self.adapter.release_handle(&self.handle);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,30 +1,11 @@
|
|||
use crate::{
|
||||
action::WindowOp, adapter::PlatformAdapter, commands::helpers::resolve_app_pid, error::AppError,
|
||||
action::WindowOp,
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{window_op_command, AppArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
use serde_json::Value;
|
||||
|
||||
pub struct RestoreArgs {
|
||||
pub app: Option<String>,
|
||||
}
|
||||
|
||||
pub fn execute(args: RestoreArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let pid = resolve_app_pid(args.app.as_deref(), adapter)?;
|
||||
let win = find_window(pid, adapter)?;
|
||||
adapter.window_op(&win, WindowOp::Restore)?;
|
||||
Ok(json!({ "restored": true }))
|
||||
}
|
||||
|
||||
fn find_window(
|
||||
pid: i32,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
) -> Result<crate::node::WindowInfo, AppError> {
|
||||
let filter = crate::adapter::WindowFilter {
|
||||
focused_only: false,
|
||||
app: None,
|
||||
};
|
||||
let windows = adapter.list_windows(&filter)?;
|
||||
windows
|
||||
.into_iter()
|
||||
.find(|w| w.pid == pid)
|
||||
.ok_or_else(|| AppError::invalid_input("No window found for this application"))
|
||||
pub fn execute(args: AppArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
window_op_command(args, adapter, WindowOp::Restore, "restored")
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,48 +1,231 @@
|
|||
use crate::{
|
||||
action::Action,
|
||||
adapter::{PlatformAdapter, TreeOptions},
|
||||
commands::helpers::resolve_ref,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::{PlatformAdapter, SnapshotSurface, TreeOptions, WindowFilter},
|
||||
commands::helpers::{resolve_ref, RefArgs},
|
||||
error::AppError,
|
||||
node::AccessibilityNode,
|
||||
refs::RefEntry,
|
||||
snapshot,
|
||||
};
|
||||
use serde_json::Value;
|
||||
use serde_json::{json, Value};
|
||||
|
||||
pub struct RightClickArgs {
|
||||
pub ref_id: String,
|
||||
}
|
||||
|
||||
pub fn execute(args: RightClickArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::RightClick)?;
|
||||
pub fn execute(args: RefArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (entry, handle) = resolve_ref(&args.ref_id, args.snapshot_id.as_deref(), adapter)?;
|
||||
let result =
|
||||
adapter.execute_action(handle.handle(), ActionRequest::headless(Action::RightClick))?;
|
||||
let mut response = serde_json::to_value(&result)?;
|
||||
|
||||
std::thread::sleep(std::time::Duration::from_millis(200));
|
||||
|
||||
let opts = TreeOptions {
|
||||
interactive_only: true,
|
||||
surface: SnapshotSurface::Menu,
|
||||
..Default::default()
|
||||
};
|
||||
if let Ok(snap) = snapshot::build(adapter, &opts, entry.source_app.as_deref(), None) {
|
||||
snap.refmap.save().ok();
|
||||
if let Some(menu) = find_context_menu(&snap.tree) {
|
||||
if let Ok(menu_json) = serde_json::to_value(menu) {
|
||||
let probe_app = probe_app_name(adapter, &entry);
|
||||
match snapshot::run(adapter, &opts, probe_app.as_deref(), None) {
|
||||
Ok(snap) => match serde_json::to_value(&snap.tree) {
|
||||
Ok(menu_json) => {
|
||||
response["menu"] = menu_json;
|
||||
if let Some(snapshot_id) = snap.snapshot_id {
|
||||
response["menu_snapshot_id"] = json!(snapshot_id);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(err) => {
|
||||
response["menu_probe"] = json!({
|
||||
"ok": false,
|
||||
"error": {
|
||||
"code": "INTERNAL",
|
||||
"message": err.to_string(),
|
||||
}
|
||||
})
|
||||
}
|
||||
},
|
||||
Err(err) => response["menu_probe"] = probe_error_json(&err),
|
||||
}
|
||||
|
||||
Ok(response)
|
||||
}
|
||||
|
||||
fn find_context_menu(node: &AccessibilityNode) -> Option<&AccessibilityNode> {
|
||||
if node.role == "menu" && node.children.iter().any(|c| c.role == "menuitem") {
|
||||
return Some(node);
|
||||
fn probe_app_name(adapter: &dyn PlatformAdapter, entry: &RefEntry) -> Option<String> {
|
||||
if entry.source_app.is_some() {
|
||||
return entry.source_app.clone();
|
||||
}
|
||||
for child in &node.children {
|
||||
if let Some(menu) = find_context_menu(child) {
|
||||
return Some(menu);
|
||||
adapter
|
||||
.list_windows(&WindowFilter {
|
||||
focused_only: false,
|
||||
app: None,
|
||||
})
|
||||
.ok()
|
||||
.and_then(|windows| {
|
||||
windows
|
||||
.into_iter()
|
||||
.find(|window| window.pid == entry.pid)
|
||||
.map(|window| window.app)
|
||||
})
|
||||
}
|
||||
|
||||
fn probe_error_json(err: &AppError) -> Value {
|
||||
if err.code() == "ELEMENT_NOT_FOUND" {
|
||||
return json!({
|
||||
"ok": false,
|
||||
"error": {
|
||||
"code": "ELEMENT_NOT_FOUND",
|
||||
"message": "Right-click action was accepted, but no menu accessibility tree was exposed for capture.",
|
||||
"suggestion": "Use 'snapshot --surface menu' only when the app exposes the context menu through accessibility."
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
let mut error = json!({
|
||||
"code": err.code(),
|
||||
"message": err.to_string(),
|
||||
});
|
||||
if let Some(suggestion) = err.suggestion() {
|
||||
error["suggestion"] = json!(suggestion);
|
||||
}
|
||||
json!({
|
||||
"ok": false,
|
||||
"error": error,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::{
|
||||
action::ActionResult,
|
||||
adapter::NativeHandle,
|
||||
error::{AdapterError, ErrorCode},
|
||||
node::WindowInfo,
|
||||
refs::{RefEntry, RefMap},
|
||||
refs_store::RefStore,
|
||||
refs_test_support::HomeGuard,
|
||||
};
|
||||
|
||||
struct ProbeFailingAdapter {
|
||||
tree_error: Option<ErrorCode>,
|
||||
}
|
||||
|
||||
impl PlatformAdapter for ProbeFailingAdapter {
|
||||
fn resolve_element(&self, _entry: &RefEntry) -> Result<NativeHandle, AdapterError> {
|
||||
Ok(NativeHandle::null())
|
||||
}
|
||||
|
||||
fn execute_action(
|
||||
&self,
|
||||
_handle: &NativeHandle,
|
||||
_request: ActionRequest,
|
||||
) -> Result<ActionResult, AdapterError> {
|
||||
Ok(ActionResult::new("right_click"))
|
||||
}
|
||||
|
||||
fn list_windows(&self, filter: &WindowFilter) -> Result<Vec<WindowInfo>, AdapterError> {
|
||||
if filter.app.is_some() && self.tree_error.is_none() {
|
||||
return Err(AdapterError::new(
|
||||
ErrorCode::WindowNotFound,
|
||||
"menu probe failed",
|
||||
));
|
||||
}
|
||||
if filter.focused_only {
|
||||
return Err(AdapterError::new(
|
||||
ErrorCode::WindowNotFound,
|
||||
"no focused menu",
|
||||
));
|
||||
}
|
||||
Ok(vec![WindowInfo {
|
||||
id: "w1".into(),
|
||||
title: "Main".into(),
|
||||
app: "TargetApp".into(),
|
||||
pid: 7,
|
||||
bounds: None,
|
||||
is_focused: true,
|
||||
}])
|
||||
}
|
||||
|
||||
fn get_tree(
|
||||
&self,
|
||||
_win: &WindowInfo,
|
||||
_opts: &TreeOptions,
|
||||
) -> Result<crate::node::AccessibilityNode, AdapterError> {
|
||||
if let Some(code) = self.tree_error.clone() {
|
||||
return Err(AdapterError::new(code, "menu tree unavailable"));
|
||||
}
|
||||
Ok(crate::node::AccessibilityNode {
|
||||
ref_id: None,
|
||||
role: "menu".into(),
|
||||
name: None,
|
||||
value: None,
|
||||
description: None,
|
||||
hint: None,
|
||||
states: Vec::new(),
|
||||
available_actions: Vec::new(),
|
||||
bounds: None,
|
||||
children_count: None,
|
||||
children: Vec::new(),
|
||||
})
|
||||
}
|
||||
}
|
||||
None
|
||||
|
||||
fn save_refmap(source_app: Option<String>) -> String {
|
||||
let mut refmap = RefMap::new();
|
||||
refmap.allocate(RefEntry {
|
||||
pid: 7,
|
||||
role: "button".into(),
|
||||
name: Some("Open".into()),
|
||||
value: None,
|
||||
states: Vec::new(),
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["RightClick".into()],
|
||||
source_app,
|
||||
source_window_title: None,
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
});
|
||||
RefStore::new().unwrap().save_new_snapshot(&refmap).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn returns_action_success_when_menu_probe_fails() {
|
||||
let _guard = HomeGuard::new();
|
||||
let snapshot_id = save_refmap(None);
|
||||
|
||||
let value = execute(
|
||||
RefArgs {
|
||||
ref_id: "@e1".into(),
|
||||
snapshot_id: Some(snapshot_id),
|
||||
},
|
||||
&ProbeFailingAdapter { tree_error: None },
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(value["action"], "right_click");
|
||||
assert_eq!(value["menu_probe"]["ok"], false);
|
||||
assert_eq!(value["menu_probe"]["error"]["code"], "WINDOW_NOT_FOUND");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn element_not_found_menu_probe_uses_right_click_specific_guidance() {
|
||||
let _guard = HomeGuard::new();
|
||||
let snapshot_id = save_refmap(Some("TargetApp".into()));
|
||||
|
||||
let value = execute(
|
||||
RefArgs {
|
||||
ref_id: "@e1".into(),
|
||||
snapshot_id: Some(snapshot_id),
|
||||
},
|
||||
&ProbeFailingAdapter {
|
||||
tree_error: Some(ErrorCode::ElementNotFound),
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(value["action"], "right_click");
|
||||
assert_eq!(value["menu_probe"]["ok"], false);
|
||||
assert_eq!(value["menu_probe"]["error"]["code"], "ELEMENT_NOT_FOUND");
|
||||
assert!(value["menu_probe"]["error"]["suggestion"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.contains("snapshot --surface menu"));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
use crate::{
|
||||
action::{Action, Direction},
|
||||
action::{Action, ActionRequest, Direction},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::resolve_ref,
|
||||
error::AppError,
|
||||
|
|
@ -8,12 +8,16 @@ use serde_json::Value;
|
|||
|
||||
pub struct ScrollArgs {
|
||||
pub ref_id: String,
|
||||
pub snapshot_id: Option<String>,
|
||||
pub direction: Direction,
|
||||
pub amount: u32,
|
||||
}
|
||||
|
||||
pub fn execute(args: ScrollArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::Scroll(args.direction, args.amount))?;
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, args.snapshot_id.as_deref(), adapter)?;
|
||||
let result = adapter.execute_action(
|
||||
handle.handle(),
|
||||
ActionRequest::headless(Action::Scroll(args.direction, args.amount)),
|
||||
)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,14 +1,11 @@
|
|||
use crate::{
|
||||
action::Action, adapter::PlatformAdapter, commands::helpers::resolve_ref, error::AppError,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{execute_ref_action, RefArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
pub struct ScrollToArgs {
|
||||
pub ref_id: String,
|
||||
}
|
||||
|
||||
pub fn execute(args: ScrollToArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::ScrollTo)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
pub fn execute(args: RefArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
execute_ref_action(args, adapter, ActionRequest::headless(Action::ScrollTo))
|
||||
}
|
||||
|
|
|
|||
31
crates/core/src/commands/search_text.rs
Normal file
31
crates/core/src/commands/search_text.rs
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
pub(crate) fn normalize(value: &str) -> String {
|
||||
if value.is_ascii() {
|
||||
value.to_ascii_lowercase()
|
||||
} else {
|
||||
value.to_lowercase()
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn contains(haystack: &str, normalized_needle: &str) -> bool {
|
||||
if haystack.is_ascii() && normalized_needle.is_ascii() {
|
||||
return haystack
|
||||
.as_bytes()
|
||||
.windows(normalized_needle.len())
|
||||
.any(|chunk| chunk.eq_ignore_ascii_case(normalized_needle.as_bytes()));
|
||||
}
|
||||
normalize(haystack).contains(normalized_needle)
|
||||
}
|
||||
|
||||
pub(crate) fn node_contains(
|
||||
node: &crate::node::AccessibilityNode,
|
||||
normalized_needle: &str,
|
||||
) -> bool {
|
||||
[
|
||||
node.name.as_deref(),
|
||||
node.value.as_deref(),
|
||||
node.description.as_deref(),
|
||||
]
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.any(|text| contains(text, normalized_needle))
|
||||
}
|
||||
|
|
@ -1,15 +1,22 @@
|
|||
use crate::{
|
||||
action::Action, adapter::PlatformAdapter, commands::helpers::resolve_ref, error::AppError,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::resolve_ref,
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
pub struct SelectArgs {
|
||||
pub ref_id: String,
|
||||
pub snapshot_id: Option<String>,
|
||||
pub value: String,
|
||||
}
|
||||
|
||||
pub fn execute(args: SelectArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::Select(args.value))?;
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, args.snapshot_id.as_deref(), adapter)?;
|
||||
let result = adapter.execute_action(
|
||||
handle.handle(),
|
||||
ActionRequest::headless(Action::Select(args.value)),
|
||||
)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,15 +1,22 @@
|
|||
use crate::{
|
||||
action::Action, adapter::PlatformAdapter, commands::helpers::resolve_ref, error::AppError,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::resolve_ref,
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
pub struct SetValueArgs {
|
||||
pub ref_id: String,
|
||||
pub snapshot_id: Option<String>,
|
||||
pub value: String,
|
||||
}
|
||||
|
||||
pub fn execute(args: SetValueArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::SetValue(args.value))?;
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, args.snapshot_id.as_deref(), adapter)?;
|
||||
let result = adapter.execute_action(
|
||||
handle.handle(),
|
||||
ActionRequest::headless(Action::SetValue(args.value)),
|
||||
)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -40,7 +40,7 @@ const SKILLS: &[Skill] = &[
|
|||
Skill {
|
||||
canonical: "agent-desktop",
|
||||
aliases: &["desktop", "agent-desktop"],
|
||||
summary: "Primary guide. Snapshot/ref loop, JSON envelope, 53 commands across observation, interaction, keyboard/mouse, app lifecycle, notifications, clipboard, wait.",
|
||||
summary: "Primary guide. Snapshot/ref loop, JSON envelope, 54 commands across observation, interaction, keyboard/mouse, app lifecycle, notifications, clipboard, wait.",
|
||||
main: SKILL_DESKTOP_MAIN,
|
||||
refs: &[
|
||||
SkillRef { rel_path: "references/commands-observation.md", body: SKILL_DESKTOP_REF_OBSERVATION },
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ pub struct SnapshotArgs {
|
|||
pub surface: SnapshotSurface,
|
||||
pub skeleton: bool,
|
||||
pub root_ref: Option<String>,
|
||||
pub snapshot_id: Option<String>,
|
||||
}
|
||||
|
||||
fn tree_options(args: &SnapshotArgs) -> crate::adapter::TreeOptions {
|
||||
|
|
@ -55,7 +56,12 @@ pub fn execute(args: SnapshotArgs, adapter: &dyn PlatformAdapter) -> Result<Valu
|
|||
));
|
||||
}
|
||||
validate_ref_id(root)?;
|
||||
return format_result(snapshot_ref::run_from_ref(adapter, &opts, root)?);
|
||||
return format_result(snapshot_ref::run_from_ref(
|
||||
adapter,
|
||||
&opts,
|
||||
root,
|
||||
args.snapshot_id.as_deref(),
|
||||
)?);
|
||||
}
|
||||
|
||||
let result = snapshot::run(
|
||||
|
|
@ -87,6 +93,7 @@ fn format_result(result: snapshot::SnapshotResult) -> Result<Value, AppError> {
|
|||
"title": win.title
|
||||
},
|
||||
"ref_count": ref_count,
|
||||
"snapshot_id": result.snapshot_id,
|
||||
"tree": tree
|
||||
}))
|
||||
}
|
||||
|
|
@ -110,6 +117,7 @@ mod tests {
|
|||
surface: SnapshotSurface::Window,
|
||||
skeleton: false,
|
||||
root_ref: None,
|
||||
snapshot_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,25 +1,74 @@
|
|||
use crate::{
|
||||
adapter::{PermissionStatus, PlatformAdapter},
|
||||
adapter::PlatformAdapter,
|
||||
commands::permissions::{self, PermissionsArgs},
|
||||
error::AppError,
|
||||
refs::RefMap,
|
||||
refs_store::RefStore,
|
||||
PermissionReport,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
pub fn execute(adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let permissions = match adapter.check_permissions() {
|
||||
PermissionStatus::Granted => json!({ "granted": true }),
|
||||
PermissionStatus::Denied { suggestion } => json!({
|
||||
"granted": false,
|
||||
"suggestion": suggestion
|
||||
}),
|
||||
};
|
||||
let report = adapter.permission_report();
|
||||
execute_with_report(adapter, &report)
|
||||
}
|
||||
|
||||
let ref_count = RefMap::load().ok().map(|m| m.len());
|
||||
pub fn execute_with_report(
|
||||
adapter: &dyn PlatformAdapter,
|
||||
report: &PermissionReport,
|
||||
) -> Result<Value, AppError> {
|
||||
let permissions =
|
||||
permissions::execute_with_report(PermissionsArgs { request: false }, adapter, report)?;
|
||||
|
||||
let store = RefStore::new().ok();
|
||||
let ref_count = store
|
||||
.as_ref()
|
||||
.and_then(|s| s.load_latest().ok())
|
||||
.map(|m| m.len());
|
||||
let snapshot_id = store.and_then(|s| s.latest_snapshot_id());
|
||||
|
||||
Ok(json!({
|
||||
"platform": std::env::consts::OS,
|
||||
"version": env!("CARGO_PKG_VERSION"),
|
||||
"permissions": permissions,
|
||||
"snapshot_id": snapshot_id,
|
||||
"ref_count": ref_count
|
||||
}))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::PermissionState;
|
||||
|
||||
struct DeniedAdapter;
|
||||
|
||||
impl PlatformAdapter for DeniedAdapter {
|
||||
fn permission_report(&self) -> PermissionReport {
|
||||
PermissionReport {
|
||||
accessibility: PermissionState::Denied {
|
||||
suggestion: "should not be used".into(),
|
||||
},
|
||||
screen_recording: PermissionState::Denied {
|
||||
suggestion: "should not be used".into(),
|
||||
},
|
||||
automation: PermissionState::Unknown,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn status_uses_precomputed_permission_report() {
|
||||
let report = PermissionReport {
|
||||
accessibility: PermissionState::Granted,
|
||||
screen_recording: PermissionState::Granted,
|
||||
automation: PermissionState::NotRequired,
|
||||
};
|
||||
|
||||
let value = execute_with_report(&DeniedAdapter, &report).unwrap();
|
||||
let permissions = value.get("permissions").unwrap();
|
||||
|
||||
assert_eq!(permissions["accessibility"]["state"], "granted");
|
||||
assert_eq!(permissions["screen_recording"]["state"], "granted");
|
||||
assert_eq!(permissions["automation"]["state"], "not_required");
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,13 +1,11 @@
|
|||
use crate::{
|
||||
action::Action,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{resolve_ref, RefArgs},
|
||||
commands::helpers::{execute_ref_action, RefArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
pub fn execute(args: RefArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::Toggle)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
execute_ref_action(args, adapter, ActionRequest::headless(Action::Toggle))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,14 +1,11 @@
|
|||
use crate::{
|
||||
action::Action, adapter::PlatformAdapter, commands::helpers::resolve_ref, error::AppError,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{execute_ref_action, RefArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
pub struct TripleClickArgs {
|
||||
pub ref_id: String,
|
||||
}
|
||||
|
||||
pub fn execute(args: TripleClickArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::TripleClick)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
pub fn execute(args: RefArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
execute_ref_action(args, adapter, ActionRequest::headless(Action::TripleClick))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,8 @@
|
|||
use crate::{
|
||||
action::Action, adapter::PlatformAdapter, commands::helpers::resolve_ref, error::AppError,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::resolve_ref,
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
|
|
@ -7,6 +10,7 @@ const MAX_TEXT_LEN: usize = 10_000;
|
|||
|
||||
pub struct TypeArgs {
|
||||
pub ref_id: String,
|
||||
pub snapshot_id: Option<String>,
|
||||
pub text: String,
|
||||
}
|
||||
|
||||
|
|
@ -17,8 +21,10 @@ pub fn execute(args: TypeArgs, adapter: &dyn PlatformAdapter) -> Result<Value, A
|
|||
)));
|
||||
}
|
||||
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
adapter.execute_action(&handle, Action::SetFocus)?;
|
||||
let result = adapter.execute_action(&handle, Action::TypeText(args.text))?;
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, args.snapshot_id.as_deref(), adapter)?;
|
||||
let result = adapter.execute_action(
|
||||
handle.handle(),
|
||||
ActionRequest::focus_fallback(Action::TypeText(args.text)),
|
||||
)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,14 +1,11 @@
|
|||
use crate::{
|
||||
action::Action, adapter::PlatformAdapter, commands::helpers::resolve_ref, error::AppError,
|
||||
action::{Action, ActionRequest},
|
||||
adapter::PlatformAdapter,
|
||||
commands::helpers::{execute_ref_action, RefArgs},
|
||||
error::AppError,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
pub struct UncheckArgs {
|
||||
pub ref_id: String,
|
||||
}
|
||||
|
||||
pub fn execute(args: UncheckArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
let (_entry, handle) = resolve_ref(&args.ref_id, adapter)?;
|
||||
let result = adapter.execute_action(&handle, Action::Uncheck)?;
|
||||
Ok(serde_json::to_value(result)?)
|
||||
pub fn execute(args: RefArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
execute_ref_action(args, adapter, ActionRequest::headless(Action::Uncheck))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,10 +1,11 @@
|
|||
use crate::{
|
||||
adapter::{PlatformAdapter, WindowFilter},
|
||||
commands::helpers::{resolve_app_pid, validate_ref_id},
|
||||
error::AppError,
|
||||
commands::{helpers::resolve_app_pid, helpers::validate_ref_id, search_text},
|
||||
error::{AppError, ErrorCode},
|
||||
node::AccessibilityNode,
|
||||
notification::NotificationFilter,
|
||||
refs::RefMap,
|
||||
refs_store::RefStore,
|
||||
snapshot,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
|
|
@ -13,6 +14,7 @@ use std::time::{Duration, Instant};
|
|||
pub struct WaitArgs {
|
||||
pub ms: Option<u64>,
|
||||
pub element: Option<String>,
|
||||
pub snapshot_id: Option<String>,
|
||||
pub window: Option<String>,
|
||||
pub text: Option<String>,
|
||||
pub timeout_ms: u64,
|
||||
|
|
@ -23,6 +25,8 @@ pub struct WaitArgs {
|
|||
}
|
||||
|
||||
pub fn execute(args: WaitArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
|
||||
validate_wait_mode(&args)?;
|
||||
|
||||
if let Some(ms) = args.ms {
|
||||
std::thread::sleep(Duration::from_millis(ms));
|
||||
return Ok(json!({ "waited_ms": ms }));
|
||||
|
|
@ -44,7 +48,7 @@ pub fn execute(args: WaitArgs, adapter: &dyn PlatformAdapter) -> Result<Value, A
|
|||
|
||||
if let Some(ref_id) = args.element {
|
||||
validate_ref_id(&ref_id)?;
|
||||
return wait_for_element(ref_id, args.timeout_ms, adapter);
|
||||
return wait_for_element(ref_id, args.snapshot_id, args.timeout_ms, adapter);
|
||||
}
|
||||
|
||||
if let Some(title) = args.window {
|
||||
|
|
@ -60,31 +64,134 @@ pub fn execute(args: WaitArgs, adapter: &dyn PlatformAdapter) -> Result<Value, A
|
|||
))
|
||||
}
|
||||
|
||||
fn validate_wait_mode(args: &WaitArgs) -> Result<(), AppError> {
|
||||
let selected = [
|
||||
args.ms.is_some(),
|
||||
args.element.is_some(),
|
||||
args.window.is_some(),
|
||||
args.text.is_some() && !args.notification,
|
||||
args.menu,
|
||||
args.menu_closed,
|
||||
args.notification,
|
||||
]
|
||||
.into_iter()
|
||||
.filter(|selected| *selected)
|
||||
.count();
|
||||
if selected <= 1 {
|
||||
return Ok(());
|
||||
}
|
||||
Err(AppError::invalid_input_with_suggestion(
|
||||
"wait accepts exactly one mode",
|
||||
"Use one of: ms, --element, --window, --text, --menu, --menu-closed, or --notification.",
|
||||
))
|
||||
}
|
||||
|
||||
fn wait_for_element(
|
||||
ref_id: String,
|
||||
snapshot_id: Option<String>,
|
||||
timeout_ms: u64,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
) -> Result<Value, AppError> {
|
||||
let start = Instant::now();
|
||||
let timeout = Duration::from_millis(timeout_ms);
|
||||
let store = RefStore::new()?;
|
||||
let fixed_refmap = match snapshot_id.as_deref() {
|
||||
Some(id) => Some(store.load_snapshot(id)?),
|
||||
None => None,
|
||||
};
|
||||
let mut latest_cache = if fixed_refmap.is_none() {
|
||||
Some(LatestRefCache::new(&store)?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
if fixed_refmap
|
||||
.as_ref()
|
||||
.is_some_and(|refmap| refmap.get(&ref_id).is_none())
|
||||
{
|
||||
return Err(AppError::invalid_input_with_suggestion(
|
||||
format!("Ref {ref_id} is not present in the requested snapshot"),
|
||||
"Use a ref returned by that snapshot_id, or omit --snapshot to wait against the latest refmap.",
|
||||
));
|
||||
}
|
||||
|
||||
loop {
|
||||
if let Ok(refmap) = RefMap::load() {
|
||||
if let Some(entry) = refmap.get(&ref_id) {
|
||||
if adapter.resolve_element(entry).is_ok() {
|
||||
let entry = fixed_refmap
|
||||
.as_ref()
|
||||
.and_then(|r| r.get(&ref_id).cloned())
|
||||
.or_else(|| latest_cache.as_ref().and_then(|c| c.entry(&ref_id)));
|
||||
if let Some(entry) = entry {
|
||||
match adapter.resolve_element(&entry) {
|
||||
Ok(handle) => {
|
||||
let _ = adapter.release_handle(&handle);
|
||||
let elapsed = start.elapsed().as_millis();
|
||||
return Ok(json!({ "found": true, "ref": ref_id, "elapsed_ms": elapsed }));
|
||||
}
|
||||
Err(err) if fixed_refmap.is_none() && err.code == ErrorCode::StaleRef => {
|
||||
if let Some(cache) = latest_cache.as_mut() {
|
||||
cache.refresh_if_due();
|
||||
}
|
||||
}
|
||||
Err(_) => {}
|
||||
}
|
||||
} else if let Some(cache) = latest_cache.as_mut() {
|
||||
cache.refresh_if_due();
|
||||
}
|
||||
|
||||
if start.elapsed() >= timeout {
|
||||
let remaining = timeout.saturating_sub(start.elapsed());
|
||||
if remaining.is_zero() {
|
||||
return Err(AppError::Adapter(crate::error::AdapterError::timeout(
|
||||
format!("Element {ref_id} not found within {timeout_ms}ms"),
|
||||
)));
|
||||
}
|
||||
std::thread::sleep(remaining.min(Duration::from_millis(100)));
|
||||
}
|
||||
}
|
||||
|
||||
std::thread::sleep(Duration::from_millis(100));
|
||||
struct LatestRefCache<'a> {
|
||||
store: &'a RefStore,
|
||||
snapshot_id: Option<String>,
|
||||
refmap: RefMap,
|
||||
last_refresh: Instant,
|
||||
}
|
||||
|
||||
impl<'a> LatestRefCache<'a> {
|
||||
fn new(store: &'a RefStore) -> Result<Self, AppError> {
|
||||
let snapshot_id = store.latest_snapshot_id();
|
||||
let refmap = if let Some(id) = snapshot_id.as_deref() {
|
||||
store.load_snapshot(id)?
|
||||
} else {
|
||||
store.load_latest()?
|
||||
};
|
||||
Ok(Self {
|
||||
store,
|
||||
snapshot_id,
|
||||
refmap,
|
||||
last_refresh: Instant::now() - Duration::from_millis(500),
|
||||
})
|
||||
}
|
||||
|
||||
fn entry(&self, ref_id: &str) -> Option<crate::refs::RefEntry> {
|
||||
self.refmap.get(ref_id).cloned()
|
||||
}
|
||||
|
||||
fn refresh_if_due(&mut self) {
|
||||
if self.last_refresh.elapsed() < Duration::from_millis(500) {
|
||||
return;
|
||||
}
|
||||
self.last_refresh = Instant::now();
|
||||
if let Some(snapshot_id) = self.store.latest_snapshot_id() {
|
||||
if self.snapshot_id.as_deref() == Some(snapshot_id.as_str()) {
|
||||
return;
|
||||
}
|
||||
if let Ok(refmap) = self.store.load_snapshot(&snapshot_id) {
|
||||
self.snapshot_id = Some(snapshot_id);
|
||||
self.refmap = refmap;
|
||||
}
|
||||
} else if let Ok(refmap) = self.store.load_latest() {
|
||||
self.refmap = refmap;
|
||||
self.snapshot_id = self.store.latest_snapshot_id();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -127,17 +234,19 @@ fn wait_for_text(
|
|||
let start = Instant::now();
|
||||
let timeout = Duration::from_millis(timeout_ms);
|
||||
let opts = crate::adapter::TreeOptions::default();
|
||||
let text_lower = text.to_lowercase();
|
||||
let normalized_text = search_text::normalize(&text);
|
||||
|
||||
loop {
|
||||
if let Ok(result) = snapshot::run(adapter, &opts, app.as_deref(), None) {
|
||||
if let Some(found) = find_text_in_tree(&result.tree, &text_lower) {
|
||||
if let Ok(result) = snapshot::build(adapter, &opts, app.as_deref(), None) {
|
||||
if let Some(found) = find_text_in_tree(&result.tree, &normalized_text) {
|
||||
let snapshot_id = RefStore::new()?.save_new_snapshot(&result.refmap)?;
|
||||
let elapsed = start.elapsed().as_millis();
|
||||
return Ok(json!({
|
||||
"found": true,
|
||||
"text": text,
|
||||
"ref": found.ref_id,
|
||||
"role": found.role,
|
||||
"snapshot_id": snapshot_id,
|
||||
"elapsed_ms": elapsed
|
||||
}));
|
||||
}
|
||||
|
|
@ -167,7 +276,9 @@ fn wait_for_notification(
|
|||
text: args.text.clone(),
|
||||
..Default::default()
|
||||
};
|
||||
let baseline = adapter.list_notifications(&filter)?;
|
||||
let baseline = adapter
|
||||
.list_notifications(&filter)
|
||||
.map_err(AppError::Adapter)?;
|
||||
let baseline_indices: std::collections::HashSet<usize> =
|
||||
baseline.iter().map(|n| n.index).collect();
|
||||
let interval = Duration::from_millis(500);
|
||||
|
|
@ -180,40 +291,28 @@ fn wait_for_notification(
|
|||
format!("No new notification within {}ms", args.timeout_ms),
|
||||
)));
|
||||
}
|
||||
let current = adapter.list_notifications(&filter)?;
|
||||
let new_notif = current
|
||||
let current = adapter
|
||||
.list_notifications(&filter)
|
||||
.map_err(AppError::Adapter)?;
|
||||
let Some(notif) = current
|
||||
.iter()
|
||||
.find(|n| !baseline_indices.contains(&n.index));
|
||||
if let Some(notif) = new_notif.or(current.last()) {
|
||||
if current.len() > baseline_indices.len() {
|
||||
let elapsed = start.elapsed().as_millis();
|
||||
return Ok(json!({
|
||||
"condition": "notification",
|
||||
"matched": true,
|
||||
"notification": notif,
|
||||
"elapsed_ms": elapsed,
|
||||
}));
|
||||
}
|
||||
}
|
||||
std::thread::sleep(interval);
|
||||
.find(|n| !baseline_indices.contains(&n.index))
|
||||
else {
|
||||
std::thread::sleep(interval);
|
||||
continue;
|
||||
};
|
||||
let elapsed = start.elapsed().as_millis();
|
||||
return Ok(json!({
|
||||
"condition": "notification",
|
||||
"matched": true,
|
||||
"notification": notif,
|
||||
"elapsed_ms": elapsed,
|
||||
}));
|
||||
}
|
||||
}
|
||||
|
||||
fn find_text_in_tree(node: &AccessibilityNode, text_lower: &str) -> Option<TextMatch> {
|
||||
let in_name = node
|
||||
.name
|
||||
.as_deref()
|
||||
.is_some_and(|n| n.to_lowercase().contains(text_lower));
|
||||
let in_value = node
|
||||
.value
|
||||
.as_deref()
|
||||
.is_some_and(|v| v.to_lowercase().contains(text_lower));
|
||||
let in_desc = node
|
||||
.description
|
||||
.as_deref()
|
||||
.is_some_and(|d| d.to_lowercase().contains(text_lower));
|
||||
|
||||
if in_name || in_value || in_desc {
|
||||
if search_text::node_contains(node, text_lower) {
|
||||
return Some(TextMatch {
|
||||
ref_id: node.ref_id.clone(),
|
||||
role: node.role.clone(),
|
||||
|
|
@ -227,3 +326,7 @@ fn find_text_in_tree(node: &AccessibilityNode, text_lower: &str) -> Option<TextM
|
|||
}
|
||||
None
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "wait_tests.rs"]
|
||||
mod tests;
|
||||
|
|
|
|||
191
crates/core/src/commands/wait_tests.rs
Normal file
191
crates/core/src/commands/wait_tests.rs
Normal file
|
|
@ -0,0 +1,191 @@
|
|||
use super::*;
|
||||
use crate::{
|
||||
adapter::PlatformAdapter,
|
||||
error::{AdapterError, ErrorCode},
|
||||
notification::{NotificationFilter, NotificationInfo},
|
||||
refs::{RefEntry, RefMap},
|
||||
refs_store::RefStore,
|
||||
refs_test_support::HomeGuard,
|
||||
};
|
||||
|
||||
struct NoopAdapter;
|
||||
|
||||
impl PlatformAdapter for NoopAdapter {}
|
||||
|
||||
struct NotificationErrorAdapter;
|
||||
|
||||
impl PlatformAdapter for NotificationErrorAdapter {
|
||||
fn list_notifications(
|
||||
&self,
|
||||
_filter: &NotificationFilter,
|
||||
) -> Result<Vec<NotificationInfo>, AdapterError> {
|
||||
Err(AdapterError::new(
|
||||
ErrorCode::PlatformNotSupported,
|
||||
"notifications unavailable",
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
fn snapshot_with_one_ref() -> String {
|
||||
let mut refmap = RefMap::new();
|
||||
refmap.allocate(RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some("Run".into()),
|
||||
value: None,
|
||||
states: Vec::new(),
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
source_window_title: None,
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
});
|
||||
RefStore::new().unwrap().save_new_snapshot(&refmap).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn snapshot_pinned_missing_ref_is_invalid_args() {
|
||||
let _guard = HomeGuard::new();
|
||||
let snapshot_id = snapshot_with_one_ref();
|
||||
|
||||
let err = wait_for_element("@e2".into(), Some(snapshot_id), 1, &NoopAdapter).unwrap_err();
|
||||
|
||||
assert_eq!(err.code(), "INVALID_ARGS");
|
||||
assert!(err.suggestion().is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn notification_wait_propagates_adapter_error() {
|
||||
let err = execute(
|
||||
WaitArgs {
|
||||
ms: None,
|
||||
element: None,
|
||||
snapshot_id: None,
|
||||
window: None,
|
||||
text: None,
|
||||
timeout_ms: 1,
|
||||
menu: false,
|
||||
menu_closed: false,
|
||||
notification: true,
|
||||
app: None,
|
||||
},
|
||||
&NotificationErrorAdapter,
|
||||
)
|
||||
.unwrap_err();
|
||||
|
||||
assert_eq!(err.code(), "PLATFORM_NOT_SUPPORTED");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_multiple_wait_modes() {
|
||||
let err = execute(
|
||||
WaitArgs {
|
||||
ms: Some(1),
|
||||
element: Some("@e1".into()),
|
||||
snapshot_id: None,
|
||||
window: None,
|
||||
text: None,
|
||||
timeout_ms: 1,
|
||||
menu: false,
|
||||
menu_closed: false,
|
||||
notification: false,
|
||||
app: None,
|
||||
},
|
||||
&NoopAdapter,
|
||||
)
|
||||
.unwrap_err();
|
||||
|
||||
assert_eq!(err.code(), "INVALID_ARGS");
|
||||
assert!(err.suggestion().is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn notification_wait_allows_text_filter() {
|
||||
let result = validate_wait_mode(&WaitArgs {
|
||||
ms: None,
|
||||
element: None,
|
||||
snapshot_id: None,
|
||||
window: None,
|
||||
text: Some("done".into()),
|
||||
timeout_ms: 1,
|
||||
menu: false,
|
||||
menu_closed: false,
|
||||
notification: true,
|
||||
app: None,
|
||||
});
|
||||
|
||||
assert!(result.is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn latest_ref_cache_picks_up_newer_snapshot_after_refresh() {
|
||||
let _guard = HomeGuard::new();
|
||||
let _ = snapshot_with_one_ref();
|
||||
let store = RefStore::new().unwrap();
|
||||
let first_id = store.latest_snapshot_id().unwrap();
|
||||
|
||||
let mut cache = LatestRefCache::new(&store).unwrap();
|
||||
assert_eq!(cache.snapshot_id.as_deref(), Some(first_id.as_str()));
|
||||
|
||||
let mut second = RefMap::new();
|
||||
second.allocate(RefEntry {
|
||||
pid: 99,
|
||||
role: "button".into(),
|
||||
name: Some("Second".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
source_window_title: None,
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
});
|
||||
let second_id = store.save_new_snapshot(&second).unwrap();
|
||||
assert_ne!(first_id, second_id);
|
||||
|
||||
cache.last_refresh = std::time::Instant::now() - std::time::Duration::from_secs(2);
|
||||
cache.refresh_if_due();
|
||||
|
||||
assert_eq!(cache.snapshot_id.as_deref(), Some(second_id.as_str()));
|
||||
assert!(cache.entry("@e1").is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn latest_ref_cache_debounces_consecutive_refreshes() {
|
||||
let _guard = HomeGuard::new();
|
||||
let _ = snapshot_with_one_ref();
|
||||
let store = RefStore::new().unwrap();
|
||||
let first_id = store.latest_snapshot_id().unwrap();
|
||||
|
||||
let mut cache = LatestRefCache::new(&store).unwrap();
|
||||
let pinned_snapshot_id = cache.snapshot_id.clone();
|
||||
let pinned_refresh = cache.last_refresh;
|
||||
|
||||
let mut other = RefMap::new();
|
||||
other.allocate(RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: None,
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
source_window_title: None,
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
});
|
||||
let _ = store.save_new_snapshot(&other).unwrap();
|
||||
|
||||
cache.last_refresh = std::time::Instant::now();
|
||||
cache.refresh_if_due();
|
||||
|
||||
assert_eq!(cache.snapshot_id, pinned_snapshot_id);
|
||||
assert_eq!(cache.last_refresh, pinned_refresh.max(cache.last_refresh));
|
||||
let _ = first_id;
|
||||
}
|
||||
|
|
@ -15,6 +15,8 @@ pub enum ErrorCode {
|
|||
Timeout,
|
||||
InvalidArgs,
|
||||
NotificationNotFound,
|
||||
SnapshotNotFound,
|
||||
PolicyDenied,
|
||||
Internal,
|
||||
}
|
||||
|
||||
|
|
@ -32,6 +34,8 @@ impl ErrorCode {
|
|||
ErrorCode::Timeout => "TIMEOUT",
|
||||
ErrorCode::InvalidArgs => "INVALID_ARGS",
|
||||
ErrorCode::NotificationNotFound => "NOTIFICATION_NOT_FOUND",
|
||||
ErrorCode::SnapshotNotFound => "SNAPSHOT_NOT_FOUND",
|
||||
ErrorCode::PolicyDenied => "POLICY_DENIED",
|
||||
ErrorCode::Internal => "INTERNAL",
|
||||
}
|
||||
}
|
||||
|
|
@ -122,7 +126,21 @@ impl AdapterError {
|
|||
"Accessibility permission not granted",
|
||||
)
|
||||
.with_suggestion(
|
||||
"Open System Settings > Privacy & Security > Accessibility and add your terminal",
|
||||
"Open System Settings > Privacy & Security > Accessibility and add the app that launches agent-desktop",
|
||||
)
|
||||
}
|
||||
|
||||
pub fn snapshot_not_found(snapshot_id: &str) -> Self {
|
||||
Self::new(
|
||||
ErrorCode::SnapshotNotFound,
|
||||
format!("Snapshot '{snapshot_id}' not found"),
|
||||
)
|
||||
.with_suggestion("Run 'snapshot' again and retry with the returned snapshot_id")
|
||||
}
|
||||
|
||||
pub fn policy_denied(message: impl Into<String>) -> Self {
|
||||
Self::new(ErrorCode::PolicyDenied, message).with_suggestion(
|
||||
"Use an explicit mouse/focus command if physical interaction is intended",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -164,6 +182,15 @@ impl AppError {
|
|||
pub fn invalid_input(msg: impl Into<String>) -> Self {
|
||||
AppError::Adapter(AdapterError::new(ErrorCode::InvalidArgs, msg))
|
||||
}
|
||||
|
||||
pub fn invalid_input_with_suggestion(
|
||||
msg: impl Into<String>,
|
||||
suggestion: impl Into<String>,
|
||||
) -> Self {
|
||||
AppError::Adapter(
|
||||
AdapterError::new(ErrorCode::InvalidArgs, msg).with_suggestion(suggestion),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
|
|
|||
|
|
@ -6,21 +6,32 @@ pub mod hints;
|
|||
pub mod node;
|
||||
pub mod notification;
|
||||
pub mod output;
|
||||
pub mod permission_report;
|
||||
pub mod permission_state;
|
||||
pub mod ref_alloc;
|
||||
pub mod refs;
|
||||
mod refs_lock;
|
||||
pub mod refs_store;
|
||||
#[cfg(test)]
|
||||
mod refs_test_support;
|
||||
pub mod roles;
|
||||
pub mod snapshot;
|
||||
pub mod snapshot_ref;
|
||||
mod window_lookup;
|
||||
|
||||
pub use action::{
|
||||
Action, ActionResult, Direction, DragParams, ElementState, KeyCombo, Modifier, MouseButton,
|
||||
MouseEvent, MouseEventKind, Point, WindowOp,
|
||||
Action, ActionRequest, ActionResult, Direction, DragParams, ElementState, InteractionPolicy,
|
||||
KeyCombo, Modifier, MouseButton, MouseEvent, MouseEventKind, Point, WindowOp,
|
||||
};
|
||||
pub use adapter::{
|
||||
ImageBuffer, ImageFormat, NativeHandle, PermissionStatus, PlatformAdapter, ScreenshotTarget,
|
||||
TreeOptions, WindowFilter,
|
||||
ImageBuffer, ImageFormat, NativeHandle, PlatformAdapter, ScreenshotTarget, TreeOptions,
|
||||
WindowFilter,
|
||||
};
|
||||
pub use error::{AdapterError, AppError, ErrorCode};
|
||||
pub use node::{AccessibilityNode, AppInfo, Rect, WindowInfo};
|
||||
pub use notification::{NotificationFilter, NotificationInfo};
|
||||
pub use output::{AppContext, ErrorPayload, Response, WindowContext};
|
||||
pub use permission_report::PermissionReport;
|
||||
pub use permission_state::PermissionState;
|
||||
pub use refs::{RefEntry, RefMap};
|
||||
pub use refs_store::RefStore;
|
||||
|
|
|
|||
|
|
@ -22,6 +22,9 @@ pub struct AccessibilityNode {
|
|||
#[serde(skip_serializing_if = "Vec::is_empty", default)]
|
||||
pub states: Vec<String>,
|
||||
|
||||
#[serde(skip_serializing_if = "Vec::is_empty", default)]
|
||||
pub available_actions: Vec<String>,
|
||||
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub bounds: Option<Rect>,
|
||||
|
||||
|
|
@ -125,6 +128,7 @@ mod tests {
|
|||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
available_actions: vec![],
|
||||
bounds: None,
|
||||
children_count: None,
|
||||
children: vec![],
|
||||
|
|
@ -143,6 +147,7 @@ mod tests {
|
|||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
available_actions: vec![],
|
||||
bounds: None,
|
||||
children_count: Some(47),
|
||||
children: vec![],
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
use serde::Serialize;
|
||||
use serde_json::Value;
|
||||
|
||||
pub const ENVELOPE_VERSION: &str = "2.0";
|
||||
|
||||
/// Structured output envelope used by the Phase 3 MCP server transport layer.
|
||||
/// CLI commands currently build responses via inline `serde_json::json!` calls in `main.rs`;
|
||||
/// this type provides the typed equivalent for programmatic consumers.
|
||||
|
|
@ -45,7 +47,7 @@ pub struct ErrorPayload {
|
|||
impl Response {
|
||||
pub fn ok(command: impl Into<String>, data: Value) -> Self {
|
||||
Self {
|
||||
version: "1.0",
|
||||
version: ENVELOPE_VERSION,
|
||||
ok: true,
|
||||
command: command.into(),
|
||||
app: None,
|
||||
|
|
@ -56,7 +58,7 @@ impl Response {
|
|||
|
||||
pub fn err(command: impl Into<String>, payload: ErrorPayload) -> Self {
|
||||
Self {
|
||||
version: "1.0",
|
||||
version: ENVELOPE_VERSION,
|
||||
ok: false,
|
||||
command: command.into(),
|
||||
app: None,
|
||||
|
|
|
|||
80
crates/core/src/permission_report.rs
Normal file
80
crates/core/src/permission_report.rs
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
use crate::permission_state::PermissionState;
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, PartialEq, Eq)]
|
||||
pub struct PermissionReport {
|
||||
pub accessibility: PermissionState,
|
||||
pub screen_recording: PermissionState,
|
||||
pub automation: PermissionState,
|
||||
}
|
||||
|
||||
impl PermissionReport {
|
||||
pub fn accessibility_granted(&self) -> bool {
|
||||
matches!(self.accessibility, PermissionState::Granted)
|
||||
}
|
||||
|
||||
pub fn screen_recording_granted(&self) -> bool {
|
||||
matches!(self.screen_recording, PermissionState::Granted)
|
||||
}
|
||||
|
||||
pub fn accessibility_denied(&self) -> bool {
|
||||
matches!(self.accessibility, PermissionState::Denied { .. })
|
||||
}
|
||||
|
||||
pub fn screen_recording_denied(&self) -> bool {
|
||||
matches!(self.screen_recording, PermissionState::Denied { .. })
|
||||
}
|
||||
|
||||
pub fn accessibility_suggestion(&self) -> Option<&str> {
|
||||
match &self.accessibility {
|
||||
PermissionState::Denied { suggestion } => Some(suggestion.as_str()),
|
||||
PermissionState::Granted | PermissionState::NotRequired | PermissionState::Unknown => {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn screen_recording_suggestion(&self) -> Option<&str> {
|
||||
match &self.screen_recording {
|
||||
PermissionState::Denied { suggestion } => Some(suggestion.as_str()),
|
||||
PermissionState::Granted | PermissionState::NotRequired | PermissionState::Unknown => {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for PermissionReport {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
accessibility: PermissionState::Unknown,
|
||||
screen_recording: PermissionState::Unknown,
|
||||
automation: PermissionState::NotRequired,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use serde_json::json;
|
||||
|
||||
#[test]
|
||||
fn permission_state_serializes_with_stable_state_field() {
|
||||
let report = PermissionReport {
|
||||
accessibility: PermissionState::Granted,
|
||||
screen_recording: PermissionState::Denied {
|
||||
suggestion: "grant screen recording".into(),
|
||||
},
|
||||
automation: PermissionState::NotRequired,
|
||||
};
|
||||
|
||||
let value = serde_json::to_value(report).unwrap();
|
||||
|
||||
assert_eq!(value["accessibility"], json!({"state": "granted"}));
|
||||
assert_eq!(
|
||||
value["screen_recording"],
|
||||
json!({"state": "denied", "suggestion": "grant screen recording"})
|
||||
);
|
||||
assert_eq!(value["automation"], json!({"state": "not_required"}));
|
||||
}
|
||||
}
|
||||
8
crates/core/src/permission_state.rs
Normal file
8
crates/core/src/permission_state.rs
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, PartialEq, Eq)]
|
||||
#[serde(tag = "state", rename_all = "snake_case")]
|
||||
pub enum PermissionState {
|
||||
Granted,
|
||||
Denied { suggestion: String },
|
||||
NotRequired,
|
||||
Unknown,
|
||||
}
|
||||
|
|
@ -1,24 +1,7 @@
|
|||
use crate::node::AccessibilityNode;
|
||||
use crate::refs::{RefEntry, RefMap};
|
||||
|
||||
pub(crate) const INTERACTIVE_ROLES: &[&str] = &[
|
||||
"button",
|
||||
"textfield",
|
||||
"checkbox",
|
||||
"link",
|
||||
"menuitem",
|
||||
"tab",
|
||||
"slider",
|
||||
"combobox",
|
||||
"treeitem",
|
||||
"cell",
|
||||
"radiobutton",
|
||||
"incrementor",
|
||||
"menubutton",
|
||||
"switch",
|
||||
"colorwell",
|
||||
"dockitem",
|
||||
];
|
||||
pub(crate) use crate::roles::INTERACTIVE_ROLES;
|
||||
|
||||
pub(crate) fn actions_for_role(role: &str) -> Vec<String> {
|
||||
match role {
|
||||
|
|
@ -37,7 +20,9 @@ pub(crate) fn ref_entry_from_node(
|
|||
node: &AccessibilityNode,
|
||||
pid: i32,
|
||||
source_app: Option<&str>,
|
||||
source_window_title: Option<&str>,
|
||||
root_ref: Option<String>,
|
||||
path: &[usize],
|
||||
) -> RefEntry {
|
||||
RefEntry {
|
||||
pid,
|
||||
|
|
@ -47,9 +32,15 @@ pub(crate) fn ref_entry_from_node(
|
|||
states: node.states.clone(),
|
||||
bounds: node.bounds,
|
||||
bounds_hash: node.bounds.as_ref().map(|b| b.bounds_hash()),
|
||||
available_actions: actions_for_role(&node.role),
|
||||
available_actions: if node.available_actions.is_empty() {
|
||||
actions_for_role(&node.role)
|
||||
} else {
|
||||
node.available_actions.clone()
|
||||
},
|
||||
source_app: source_app.map(str::to_string),
|
||||
source_window_title: source_window_title.map(str::to_string),
|
||||
root_ref,
|
||||
path: path.to_vec(),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -113,20 +104,36 @@ pub(crate) struct RefAllocConfig<'a> {
|
|||
pub compact: bool,
|
||||
pub pid: i32,
|
||||
pub source_app: Option<&'a str>,
|
||||
pub source_window_title: Option<&'a str>,
|
||||
pub root_ref_id: Option<&'a str>,
|
||||
}
|
||||
|
||||
pub(crate) fn allocate_refs(
|
||||
node: AccessibilityNode,
|
||||
refmap: &mut RefMap,
|
||||
config: &RefAllocConfig,
|
||||
) -> AccessibilityNode {
|
||||
allocate_refs_at_path(node, refmap, config, &mut Vec::new())
|
||||
}
|
||||
|
||||
fn allocate_refs_at_path(
|
||||
mut node: AccessibilityNode,
|
||||
refmap: &mut RefMap,
|
||||
config: &RefAllocConfig,
|
||||
path: &mut Vec<usize>,
|
||||
) -> AccessibilityNode {
|
||||
let root_ref_owned = config.root_ref_id.map(str::to_string);
|
||||
let is_interactive = INTERACTIVE_ROLES.contains(&node.role.as_str());
|
||||
|
||||
if is_interactive {
|
||||
let entry =
|
||||
ref_entry_from_node(&node, config.pid, config.source_app, root_ref_owned.clone());
|
||||
let entry = ref_entry_from_node(
|
||||
&node,
|
||||
config.pid,
|
||||
config.source_app,
|
||||
config.source_window_title,
|
||||
root_ref_owned.clone(),
|
||||
path,
|
||||
);
|
||||
node.ref_id = Some(refmap.allocate(entry));
|
||||
}
|
||||
|
||||
|
|
@ -138,7 +145,14 @@ pub(crate) fn allocate_refs(
|
|||
&& config.root_ref_id.is_none();
|
||||
|
||||
if is_skeleton_anchor {
|
||||
let mut entry = ref_entry_from_node(&node, config.pid, config.source_app, None);
|
||||
let mut entry = ref_entry_from_node(
|
||||
&node,
|
||||
config.pid,
|
||||
config.source_app,
|
||||
config.source_window_title,
|
||||
None,
|
||||
path,
|
||||
);
|
||||
entry.available_actions = vec![];
|
||||
node.ref_id = Some(refmap.allocate(entry));
|
||||
}
|
||||
|
|
@ -150,8 +164,12 @@ pub(crate) fn allocate_refs(
|
|||
node.children = node
|
||||
.children
|
||||
.into_iter()
|
||||
.enumerate()
|
||||
.filter_map(|child| {
|
||||
let child = allocate_refs(child, refmap, config);
|
||||
let (idx, child) = child;
|
||||
path.push(idx);
|
||||
let child = allocate_refs_at_path(child, refmap, config, path);
|
||||
path.pop();
|
||||
if config.compact && is_collapsible(&child) {
|
||||
return child.children.into_iter().next();
|
||||
}
|
||||
|
|
@ -184,6 +202,7 @@ mod tests {
|
|||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
available_actions: vec![],
|
||||
bounds: Some(Rect {
|
||||
x: 0.0,
|
||||
y: 0.0,
|
||||
|
|
@ -253,4 +272,39 @@ mod tests {
|
|||
assert_eq!(out.children[0].role, "group");
|
||||
assert_eq!(out.children[0].name.as_deref(), Some("Toolbar"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ref_entry_prefers_platform_actions() {
|
||||
let mut button = node("button", Some("Save"));
|
||||
button.available_actions = vec!["SetFocus".into()];
|
||||
|
||||
let entry = ref_entry_from_node(&button, 7, None, None, None, &[0]);
|
||||
|
||||
assert_eq!(entry.available_actions, vec!["SetFocus"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn allocate_refs_records_structural_paths() {
|
||||
let mut root = node("window", Some("w"));
|
||||
let mut group = node("group", Some("List"));
|
||||
group.children = vec![node("button", Some("Open"))];
|
||||
root.children = vec![node("button", Some("Save")), group];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = RefAllocConfig {
|
||||
include_bounds: true,
|
||||
interactive_only: false,
|
||||
compact: false,
|
||||
pid: 7,
|
||||
source_app: Some("Finder"),
|
||||
source_window_title: Some("Documents"),
|
||||
root_ref_id: None,
|
||||
};
|
||||
let out = allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
let save_ref = out.children[0].ref_id.as_deref().unwrap();
|
||||
let open_ref = out.children[1].children[0].ref_id.as_deref().unwrap();
|
||||
assert_eq!(refmap.get(save_ref).unwrap().path, vec![0]);
|
||||
assert_eq!(refmap.get(open_ref).unwrap().path, vec![1, 0]);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,10 +1,15 @@
|
|||
use crate::error::AppError;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::cell::RefCell;
|
||||
use std::collections::hash_map::RandomState;
|
||||
use std::collections::HashMap;
|
||||
use std::path::PathBuf;
|
||||
use std::hash::BuildHasher;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
const MAX_REFMAP_BYTES: u64 = 1_048_576; // 1 MB
|
||||
pub(crate) const MAX_REFMAP_BYTES: u64 = 1_048_576;
|
||||
static SNAPSHOT_COUNTER: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
thread_local! {
|
||||
static HOME_OVERRIDE: RefCell<Option<PathBuf>> = const { RefCell::new(None) };
|
||||
|
|
@ -25,7 +30,11 @@ pub struct RefEntry {
|
|||
pub available_actions: Vec<String>,
|
||||
pub source_app: Option<String>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub source_window_title: Option<String>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub root_ref: Option<String>,
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub path: Vec<usize>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
|
|
@ -66,7 +75,7 @@ impl RefMap {
|
|||
.retain(|_, entry| entry.root_ref.as_deref() != Some(root));
|
||||
}
|
||||
|
||||
fn serialize_with_size_check(&self) -> Result<String, AppError> {
|
||||
pub(crate) fn serialize_with_size_check(&self) -> Result<String, AppError> {
|
||||
let json = serde_json::to_string(self)?;
|
||||
if json.len() as u64 > MAX_REFMAP_BYTES {
|
||||
return Err(AppError::Internal(
|
||||
|
|
@ -76,45 +85,11 @@ impl RefMap {
|
|||
Ok(json)
|
||||
}
|
||||
|
||||
pub fn save(&self) -> Result<(), AppError> {
|
||||
#[cfg(test)]
|
||||
pub(crate) fn save(&self) -> Result<(), AppError> {
|
||||
let json = self.serialize_with_size_check()?;
|
||||
|
||||
let path = refmap_path()?;
|
||||
let dir = path
|
||||
.parent()
|
||||
.ok_or_else(|| AppError::Internal("invalid refmap path".into()))?;
|
||||
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::DirBuilderExt;
|
||||
std::fs::DirBuilder::new()
|
||||
.recursive(true)
|
||||
.mode(0o700)
|
||||
.create(dir)?;
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
std::fs::create_dir_all(dir)?;
|
||||
|
||||
let tmp = path.with_extension("tmp");
|
||||
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::io::Write;
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
let mut file = std::fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.create(true)
|
||||
.truncate(true)
|
||||
.mode(0o600)
|
||||
.open(&tmp)?;
|
||||
file.write_all(json.as_bytes())?;
|
||||
file.flush()?;
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
std::fs::write(&tmp, json.as_bytes())?;
|
||||
|
||||
std::fs::rename(&tmp, &path)?;
|
||||
Ok(())
|
||||
write_private_file(&path, json.as_bytes())
|
||||
}
|
||||
|
||||
pub fn load() -> Result<Self, AppError> {
|
||||
|
|
@ -144,358 +119,130 @@ fn refmap_path() -> Result<PathBuf, AppError> {
|
|||
Ok(home.join(".agent-desktop").join("last_refmap.json"))
|
||||
}
|
||||
|
||||
fn home_dir() -> Option<PathBuf> {
|
||||
if let Some(p) = HOME_OVERRIDE.with(|cell| cell.borrow().clone()) {
|
||||
return Some(p);
|
||||
}
|
||||
std::env::var_os("HOME")
|
||||
.map(PathBuf::from)
|
||||
.or_else(|| std::env::var_os("USERPROFILE").map(PathBuf::from))
|
||||
pub(crate) fn new_snapshot_id() -> String {
|
||||
let nanos = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos() as u64)
|
||||
.unwrap_or(0);
|
||||
let counter = SNAPSHOT_COUNTER.fetch_add(1, Ordering::Relaxed);
|
||||
let seed = RandomState::new();
|
||||
let mixed = seed.hash_one((nanos, std::process::id(), counter));
|
||||
format!("s{}", base36(mixed))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) struct HomeGuard {
|
||||
_dir: tempdir::TempDir,
|
||||
prev: Option<PathBuf>,
|
||||
}
|
||||
fn base36(mut value: u64) -> String {
|
||||
const DIGITS: &[u8; 36] = b"0123456789abcdefghijklmnopqrstuvwxyz";
|
||||
const MIN_LEN: usize = 4;
|
||||
|
||||
#[cfg(test)]
|
||||
mod tempdir {
|
||||
use std::fs;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
static COUNTER: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
pub struct TempDir(PathBuf);
|
||||
|
||||
impl TempDir {
|
||||
pub fn new() -> Self {
|
||||
let n = COUNTER.fetch_add(1, Ordering::SeqCst);
|
||||
let nanos = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos())
|
||||
.unwrap_or(0);
|
||||
let path = std::env::temp_dir().join(format!("agent-desktop-test-{nanos}-{n}"));
|
||||
fs::create_dir_all(&path).expect("create tempdir");
|
||||
Self(path)
|
||||
}
|
||||
|
||||
pub fn path(&self) -> &std::path::Path {
|
||||
&self.0
|
||||
let mut buf = [b'0'; 13];
|
||||
let mut i = buf.len();
|
||||
if value == 0 {
|
||||
i -= 1;
|
||||
} else {
|
||||
while value > 0 {
|
||||
i -= 1;
|
||||
buf[i] = DIGITS[(value % 36) as usize];
|
||||
value /= 36;
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for TempDir {
|
||||
fn drop(&mut self) {
|
||||
let _ = fs::remove_dir_all(&self.0);
|
||||
}
|
||||
let digits = buf.len() - i;
|
||||
if digits < MIN_LEN {
|
||||
let pad = MIN_LEN - digits;
|
||||
i -= pad;
|
||||
}
|
||||
|
||||
String::from_utf8_lossy(&buf[i..]).into_owned()
|
||||
}
|
||||
|
||||
pub fn validate_snapshot_id(snapshot_id: &str) -> Result<(), AppError> {
|
||||
let valid = snapshot_id.len() <= 64
|
||||
&& snapshot_id.len() >= 3
|
||||
&& snapshot_id
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_');
|
||||
if !valid {
|
||||
return Err(AppError::invalid_input(format!(
|
||||
"Invalid snapshot_id '{snapshot_id}': use the value returned by snapshot"
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn write_private_file(path: &Path, bytes: &[u8]) -> Result<(), AppError> {
|
||||
let dir = path
|
||||
.parent()
|
||||
.ok_or_else(|| AppError::Internal("invalid ref store path".into()))?;
|
||||
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::DirBuilderExt;
|
||||
std::fs::DirBuilder::new()
|
||||
.recursive(true)
|
||||
.mode(0o700)
|
||||
.create(dir)?;
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
std::fs::create_dir_all(dir)?;
|
||||
|
||||
let tmp = path.with_extension("tmp");
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::io::Write;
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
let mut file = std::fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.create(true)
|
||||
.truncate(true)
|
||||
.mode(0o600)
|
||||
.open(&tmp)?;
|
||||
file.write_all(bytes)?;
|
||||
file.flush()?;
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
std::fs::write(&tmp, bytes)?;
|
||||
|
||||
std::fs::rename(tmp, path)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn home_dir() -> Option<PathBuf> {
|
||||
let home = HOME_OVERRIDE
|
||||
.with(|cell| cell.borrow().clone())
|
||||
.or_else(|| std::env::var_os("HOME").map(PathBuf::from))
|
||||
.or_else(|| std::env::var_os("USERPROFILE").map(PathBuf::from))?;
|
||||
validate_home_dir(&home).then_some(home)
|
||||
}
|
||||
|
||||
fn validate_home_dir(home: &Path) -> bool {
|
||||
let Ok(link_meta) = std::fs::symlink_metadata(home) else {
|
||||
return false;
|
||||
};
|
||||
if link_meta.file_type().is_symlink() {
|
||||
return false;
|
||||
}
|
||||
let Ok(meta) = std::fs::metadata(home) else {
|
||||
return false;
|
||||
};
|
||||
if !meta.is_dir() {
|
||||
return false;
|
||||
}
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
meta.uid() == unsafe { libc::getuid() }
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
{
|
||||
true
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
impl HomeGuard {
|
||||
pub fn new() -> Self {
|
||||
let dir = tempdir::TempDir::new();
|
||||
let prev = HOME_OVERRIDE.with(|cell| cell.borrow().clone());
|
||||
HOME_OVERRIDE.with(|cell| *cell.borrow_mut() = Some(dir.path().to_path_buf()));
|
||||
Self { _dir: dir, prev }
|
||||
}
|
||||
pub(crate) fn set_home_override(home: Option<PathBuf>) -> Option<PathBuf> {
|
||||
HOME_OVERRIDE.with(|cell| std::mem::replace(&mut *cell.borrow_mut(), home))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
impl Drop for HomeGuard {
|
||||
fn drop(&mut self) {
|
||||
let prev = self.prev.take();
|
||||
HOME_OVERRIDE.with(|cell| *cell.borrow_mut() = prev);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_allocate_sequential() {
|
||||
let mut map = RefMap::new();
|
||||
let entry = RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some("OK".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
};
|
||||
let r1 = map.allocate(entry.clone());
|
||||
let r2 = map.allocate(entry);
|
||||
assert_eq!(r1, "@e1");
|
||||
assert_eq!(r2, "@e2");
|
||||
assert_eq!(map.len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_get_existing() {
|
||||
let mut map = RefMap::new();
|
||||
let entry = RefEntry {
|
||||
pid: 42,
|
||||
role: "textfield".into(),
|
||||
name: None,
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: Some(12345),
|
||||
available_actions: vec![],
|
||||
source_app: Some("Finder".into()),
|
||||
root_ref: None,
|
||||
};
|
||||
let ref_id = map.allocate(entry);
|
||||
let retrieved = map.get(&ref_id).unwrap();
|
||||
assert_eq!(retrieved.pid, 42);
|
||||
assert_eq!(retrieved.role, "textfield");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_get_missing() {
|
||||
let map = RefMap::new();
|
||||
assert!(map.get("@e99").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_remove_by_root_ref() {
|
||||
let mut map = RefMap::new();
|
||||
let base = RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some("OK".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
};
|
||||
|
||||
map.allocate(base.clone());
|
||||
|
||||
let drilled = RefEntry {
|
||||
root_ref: Some("@e1".into()),
|
||||
..base.clone()
|
||||
};
|
||||
map.allocate(drilled.clone());
|
||||
map.allocate(drilled);
|
||||
assert_eq!(map.len(), 3);
|
||||
|
||||
map.remove_by_root_ref("@e1");
|
||||
assert_eq!(map.len(), 1);
|
||||
assert!(map.get("@e1").is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_counter_continues_after_skeleton_into_drill_down() {
|
||||
let mut map = RefMap::new();
|
||||
let skeleton_entry = RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some("Skeleton".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec![],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
};
|
||||
|
||||
let last_skeleton = (0..10)
|
||||
.map(|_| map.allocate(skeleton_entry.clone()))
|
||||
.last()
|
||||
.unwrap();
|
||||
assert_eq!(last_skeleton, "@e10");
|
||||
|
||||
let drilled = RefEntry {
|
||||
root_ref: Some("@e3".into()),
|
||||
..skeleton_entry
|
||||
};
|
||||
|
||||
let first_drilled = map.allocate(drilled.clone());
|
||||
let second_drilled = map.allocate(drilled);
|
||||
assert_eq!(
|
||||
first_drilled, "@e11",
|
||||
"counter should continue past skeleton ids, not reset"
|
||||
);
|
||||
assert_eq!(second_drilled, "@e12");
|
||||
assert_eq!(map.len(), 12);
|
||||
|
||||
map.remove_by_root_ref("@e3");
|
||||
assert_eq!(
|
||||
map.len(),
|
||||
10,
|
||||
"scoped invalidation should drop only the drill-down refs"
|
||||
);
|
||||
assert!(map.get("@e3").is_some(), "skeleton @e3 must survive");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_root_ref_serde_roundtrip() {
|
||||
let entry = RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: None,
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec![],
|
||||
source_app: None,
|
||||
root_ref: Some("@e5".into()),
|
||||
};
|
||||
let json = serde_json::to_string(&entry).unwrap();
|
||||
assert!(json.contains("root_ref"));
|
||||
let back: RefEntry = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(back.root_ref.as_deref(), Some("@e5"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_serialize_with_size_check_rejects_oversized() {
|
||||
let mut map = RefMap::new();
|
||||
let big_name = "x".repeat(2048);
|
||||
for _ in 0..600 {
|
||||
map.allocate(RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some(big_name.clone()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
});
|
||||
}
|
||||
|
||||
let result = map.serialize_with_size_check();
|
||||
assert!(result.is_err(), "oversized refmap should be rejected");
|
||||
let err = result.unwrap_err();
|
||||
let msg = err.to_string();
|
||||
assert!(
|
||||
msg.contains("1MB"),
|
||||
"error should mention the 1MB limit, got: {msg}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_serialize_with_size_check_accepts_normal() {
|
||||
let mut map = RefMap::new();
|
||||
for _ in 0..50 {
|
||||
map.allocate(RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some("OK".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
});
|
||||
}
|
||||
|
||||
let result = map.serialize_with_size_check();
|
||||
assert!(result.is_ok(), "normal-sized refmap should serialize");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_save_load_roundtrip_with_home_override() {
|
||||
let _guard = HomeGuard::new();
|
||||
let mut map = RefMap::new();
|
||||
map.allocate(RefEntry {
|
||||
pid: 7,
|
||||
role: "button".into(),
|
||||
name: Some("Send".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: Some(42),
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: Some("TestApp".into()),
|
||||
root_ref: None,
|
||||
});
|
||||
map.save().expect("save should succeed under HomeGuard");
|
||||
|
||||
let loaded = RefMap::load().expect("load should succeed");
|
||||
assert_eq!(loaded.len(), 1);
|
||||
let entry = loaded.get("@e1").unwrap();
|
||||
assert_eq!(entry.pid, 7);
|
||||
assert_eq!(entry.name.as_deref(), Some("Send"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_save_oversize_preserves_previous_file() {
|
||||
let _guard = HomeGuard::new();
|
||||
|
||||
let mut original = RefMap::new();
|
||||
original.allocate(RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some("Original".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
});
|
||||
original.save().expect("baseline save");
|
||||
|
||||
let mut oversize = RefMap::new();
|
||||
let big = "x".repeat(2048);
|
||||
for _ in 0..600 {
|
||||
oversize.allocate(RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: Some(big.clone()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
});
|
||||
}
|
||||
let result = oversize.save();
|
||||
assert!(result.is_err(), "oversize save must reject");
|
||||
|
||||
let reloaded = RefMap::load().expect("previous file must still load");
|
||||
assert_eq!(reloaded.len(), 1);
|
||||
let entry = reloaded.get("@e1").unwrap();
|
||||
assert_eq!(entry.name.as_deref(), Some("Original"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_root_ref_none_omitted() {
|
||||
let entry = RefEntry {
|
||||
pid: 1,
|
||||
role: "button".into(),
|
||||
name: None,
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec![],
|
||||
source_app: None,
|
||||
root_ref: None,
|
||||
};
|
||||
let json = serde_json::to_string(&entry).unwrap();
|
||||
assert!(!json.contains("root_ref"));
|
||||
}
|
||||
}
|
||||
#[path = "refs_tests.rs"]
|
||||
mod tests;
|
||||
|
|
|
|||
296
crates/core/src/refs_lock.rs
Normal file
296
crates/core/src/refs_lock.rs
Normal file
|
|
@ -0,0 +1,296 @@
|
|||
use crate::error::AppError;
|
||||
use std::io::{ErrorKind, Write};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
|
||||
|
||||
const LOCK_TIMEOUT: Duration = Duration::from_secs(2);
|
||||
const MALFORMED_LOCK_WINDOW: Duration = Duration::from_secs(30);
|
||||
|
||||
pub(crate) struct RefStoreLock {
|
||||
path: PathBuf,
|
||||
token: String,
|
||||
}
|
||||
|
||||
impl RefStoreLock {
|
||||
pub(crate) fn acquire(path: &Path) -> Result<Self, AppError> {
|
||||
if let Some(dir) = path.parent() {
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::DirBuilderExt;
|
||||
std::fs::DirBuilder::new()
|
||||
.recursive(true)
|
||||
.mode(0o700)
|
||||
.create(dir)?;
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
std::fs::create_dir_all(dir)?;
|
||||
}
|
||||
let start = Instant::now();
|
||||
loop {
|
||||
let token = lock_token();
|
||||
match std::fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.create_new(true)
|
||||
.open(path)
|
||||
{
|
||||
Ok(mut file) => {
|
||||
if let Err(err) =
|
||||
writeln!(file, "{} {} {}", std::process::id(), now_secs(), token)
|
||||
{
|
||||
let _ = std::fs::remove_file(path);
|
||||
return Err(err.into());
|
||||
}
|
||||
return Ok(Self {
|
||||
path: path.to_path_buf(),
|
||||
token,
|
||||
});
|
||||
}
|
||||
Err(err) if err.kind() == std::io::ErrorKind::AlreadyExists => {
|
||||
if try_remove_stale_lock(path) {
|
||||
continue;
|
||||
}
|
||||
if start.elapsed() > LOCK_TIMEOUT {
|
||||
return Err(AppError::Internal(format!(
|
||||
"Timed out waiting for ref store lock at {}",
|
||||
path.display()
|
||||
)));
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(10));
|
||||
}
|
||||
Err(err) => return Err(err.into()),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn try_remove_stale_lock(path: &Path) -> bool {
|
||||
let snapshot = match LockSnapshot::read(path) {
|
||||
Ok(Some(snapshot)) => snapshot,
|
||||
Ok(None) => return true,
|
||||
Err(()) => return false,
|
||||
};
|
||||
if !snapshot.is_stale() {
|
||||
return false;
|
||||
}
|
||||
let current = match LockSnapshot::read(path) {
|
||||
Ok(Some(current)) => current,
|
||||
Ok(None) => return true,
|
||||
Err(()) => return false,
|
||||
};
|
||||
if current.contents != snapshot.contents || current.modified != snapshot.modified {
|
||||
return false;
|
||||
}
|
||||
match std::fs::remove_file(path) {
|
||||
Ok(()) => true,
|
||||
Err(err) if err.kind() == ErrorKind::NotFound => true,
|
||||
Err(_) => false,
|
||||
}
|
||||
}
|
||||
|
||||
struct LockSnapshot {
|
||||
contents: String,
|
||||
modified: Option<SystemTime>,
|
||||
}
|
||||
|
||||
impl LockSnapshot {
|
||||
fn read(path: &Path) -> Result<Option<Self>, ()> {
|
||||
let contents = match std::fs::read_to_string(path) {
|
||||
Ok(contents) => contents,
|
||||
Err(err) if err.kind() == ErrorKind::NotFound => return Ok(None),
|
||||
Err(_) => return Err(()),
|
||||
};
|
||||
let modified = std::fs::metadata(path).and_then(|m| m.modified()).ok();
|
||||
Ok(Some(Self { contents, modified }))
|
||||
}
|
||||
|
||||
fn is_stale(&self) -> bool {
|
||||
if self.contents.trim().is_empty() {
|
||||
return self.modified_age_exceeds(MALFORMED_LOCK_WINDOW);
|
||||
}
|
||||
match self.pid() {
|
||||
Some(pid) => match process_is_alive(pid) {
|
||||
Some(true) => self.live_pid_lock_is_stale(pid),
|
||||
Some(false) => true,
|
||||
None => self.modified_age_exceeds(MALFORMED_LOCK_WINDOW),
|
||||
},
|
||||
None => self.modified_age_exceeds(MALFORMED_LOCK_WINDOW),
|
||||
}
|
||||
}
|
||||
|
||||
fn live_pid_lock_is_stale(&self, pid: u32) -> bool {
|
||||
if !self.token_shape_matches_pid(pid) {
|
||||
return self.modified_age_exceeds(LOCK_TIMEOUT);
|
||||
}
|
||||
self.modified_age_exceeds(MALFORMED_LOCK_WINDOW)
|
||||
}
|
||||
|
||||
fn pid(&self) -> Option<u32> {
|
||||
self.contents
|
||||
.split_whitespace()
|
||||
.next()
|
||||
.and_then(|s| s.parse::<u32>().ok())
|
||||
}
|
||||
|
||||
fn token(&self) -> Option<&str> {
|
||||
self.contents.split_whitespace().nth(2)
|
||||
}
|
||||
|
||||
fn owned_by(&self, token: &str) -> bool {
|
||||
self.pid() == Some(std::process::id()) && self.token() == Some(token)
|
||||
}
|
||||
|
||||
fn token_shape_matches_pid(&self, pid: u32) -> bool {
|
||||
self.token()
|
||||
.and_then(|token| token.split_once('-'))
|
||||
.and_then(|(prefix, _)| prefix.parse::<u32>().ok())
|
||||
== Some(pid)
|
||||
}
|
||||
|
||||
fn modified_age_exceeds(&self, duration: Duration) -> bool {
|
||||
self.modified
|
||||
.and_then(|modified| SystemTime::now().duration_since(modified).ok())
|
||||
.is_some_and(|age| age > duration)
|
||||
}
|
||||
}
|
||||
|
||||
fn now_secs() -> u64 {
|
||||
SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_secs())
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
fn lock_token() -> String {
|
||||
let nanos = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos())
|
||||
.unwrap_or(0);
|
||||
format!("{}-{nanos}", std::process::id())
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
fn process_is_alive(pid: u32) -> Option<bool> {
|
||||
unsafe extern "C" {
|
||||
fn kill(pid: i32, sig: i32) -> i32;
|
||||
}
|
||||
let result = unsafe { kill(pid as i32, 0) };
|
||||
if result == 0 {
|
||||
return Some(true);
|
||||
}
|
||||
match std::io::Error::last_os_error().raw_os_error() {
|
||||
Some(errno) if errno == libc::ESRCH => Some(false),
|
||||
Some(errno) if errno == libc::EPERM => None,
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(not(unix))]
|
||||
fn process_is_alive(_pid: u32) -> Option<bool> {
|
||||
None
|
||||
}
|
||||
|
||||
impl Drop for RefStoreLock {
|
||||
fn drop(&mut self) {
|
||||
let should_remove = LockSnapshot::read(&self.path)
|
||||
.ok()
|
||||
.flatten()
|
||||
.is_some_and(|snapshot| snapshot.owned_by(&self.token));
|
||||
if should_remove {
|
||||
let _ = std::fs::remove_file(&self.path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::fs;
|
||||
|
||||
fn lock_path(name: &str) -> PathBuf {
|
||||
std::env::temp_dir().join(format!(
|
||||
"agent-desktop-{name}-{}-{}.lock",
|
||||
std::process::id(),
|
||||
lock_token()
|
||||
))
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn acquire_removes_lock_on_drop() {
|
||||
let path = lock_path("drop");
|
||||
{
|
||||
let _lock = RefStoreLock::acquire(&path).unwrap();
|
||||
assert!(path.exists());
|
||||
}
|
||||
assert!(!path.exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stale_dead_pid_lock_is_replaced() {
|
||||
let path = lock_path("stale-pid");
|
||||
fs::write(&path, "999999 1 stale").unwrap();
|
||||
let _lock = RefStoreLock::acquire(&path).unwrap();
|
||||
let contents = fs::read_to_string(&path).unwrap();
|
||||
assert!(contents.starts_with(&std::process::id().to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn old_same_process_lock_with_foreign_token_shape_is_stale() {
|
||||
let snapshot = LockSnapshot {
|
||||
contents: format!("{} 1 old-token", std::process::id()),
|
||||
modified: Some(SystemTime::now() - Duration::from_secs(60)),
|
||||
};
|
||||
|
||||
assert!(snapshot.is_stale());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recent_live_pid_lock_is_not_stale() {
|
||||
let snapshot = LockSnapshot {
|
||||
contents: format!(
|
||||
"{} {} {}-token",
|
||||
std::process::id(),
|
||||
now_secs(),
|
||||
std::process::id()
|
||||
),
|
||||
modified: Some(SystemTime::now()),
|
||||
};
|
||||
|
||||
assert!(!snapshot.is_stale());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn old_live_pid_lock_with_foreign_token_shape_is_stale() {
|
||||
let snapshot = LockSnapshot {
|
||||
contents: "2000 1 1000-old".into(),
|
||||
modified: Some(SystemTime::now() - Duration::from_secs(3)),
|
||||
};
|
||||
|
||||
assert!(snapshot.live_pid_lock_is_stale(2000));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recent_live_pid_lock_with_matching_token_shape_is_not_stale() {
|
||||
let snapshot = LockSnapshot {
|
||||
contents: "2000 1 2000-token".into(),
|
||||
modified: Some(SystemTime::now() - Duration::from_secs(3)),
|
||||
};
|
||||
|
||||
assert!(!snapshot.live_pid_lock_is_stale(2000));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn drop_does_not_remove_replaced_lock() {
|
||||
let path = lock_path("replaced");
|
||||
let lock = RefStoreLock::acquire(&path).unwrap();
|
||||
fs::write(
|
||||
&path,
|
||||
format!("{} {} replacement-token", std::process::id(), now_secs()),
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
drop(lock);
|
||||
|
||||
assert!(path.exists());
|
||||
fs::remove_file(&path).unwrap();
|
||||
}
|
||||
}
|
||||
196
crates/core/src/refs_store.rs
Normal file
196
crates/core/src/refs_store.rs
Normal file
|
|
@ -0,0 +1,196 @@
|
|||
use crate::{
|
||||
error::{AdapterError, AppError},
|
||||
refs::{home_dir, new_snapshot_id, validate_snapshot_id, write_private_file, RefMap},
|
||||
refs_lock::RefStoreLock,
|
||||
};
|
||||
use std::io::Read;
|
||||
use std::path::PathBuf;
|
||||
|
||||
const LATEST_SNAPSHOT_FILE: &str = "latest_snapshot_id";
|
||||
const MAX_SAVED_SNAPSHOTS: usize = 512;
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct RefStore {
|
||||
base_dir: PathBuf,
|
||||
}
|
||||
|
||||
impl RefStore {
|
||||
pub fn new() -> Result<Self, AppError> {
|
||||
let home =
|
||||
home_dir().ok_or_else(|| AppError::Internal("HOME directory not found".into()))?;
|
||||
Ok(Self {
|
||||
base_dir: home.join(".agent-desktop"),
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub fn for_tests() -> Result<Self, AppError> {
|
||||
Self::new()
|
||||
}
|
||||
|
||||
pub fn save_new_snapshot(&self, refmap: &RefMap) -> Result<String, AppError> {
|
||||
self.with_write_lock(|| {
|
||||
let snapshot_id = new_snapshot_id();
|
||||
self.save_snapshot_unlocked(&snapshot_id, refmap)?;
|
||||
self.set_latest_unlocked(&snapshot_id)?;
|
||||
self.prune_old_snapshots_unlocked(&snapshot_id)?;
|
||||
Ok(snapshot_id)
|
||||
})
|
||||
}
|
||||
|
||||
pub fn save_snapshot(&self, snapshot_id: &str, refmap: &RefMap) -> Result<(), AppError> {
|
||||
self.with_write_lock(|| self.save_snapshot_unlocked(snapshot_id, refmap))
|
||||
}
|
||||
|
||||
pub fn save_existing_snapshot(
|
||||
&self,
|
||||
snapshot_id: &str,
|
||||
refmap: &RefMap,
|
||||
) -> Result<(), AppError> {
|
||||
self.with_write_lock(|| self.save_snapshot_unlocked(snapshot_id, refmap))
|
||||
}
|
||||
|
||||
pub fn load(&self, snapshot_id: Option<&str>) -> Result<RefMap, AppError> {
|
||||
match snapshot_id {
|
||||
Some(id) => self.load_snapshot(id),
|
||||
None => self.load_latest(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn load_latest(&self) -> Result<RefMap, AppError> {
|
||||
if let Ok(id) = std::fs::read_to_string(self.latest_path()) {
|
||||
let id = id.trim();
|
||||
if !id.is_empty() {
|
||||
return self.load_snapshot(id);
|
||||
}
|
||||
}
|
||||
if let Some(refmap) = self.migrate_legacy_latest()? {
|
||||
return Ok(refmap);
|
||||
}
|
||||
Err(AppError::Adapter(AdapterError::snapshot_not_found(
|
||||
"latest",
|
||||
)))
|
||||
}
|
||||
|
||||
pub fn load_snapshot(&self, snapshot_id: &str) -> Result<RefMap, AppError> {
|
||||
validate_snapshot_id(snapshot_id)?;
|
||||
let path = self.snapshot_path(snapshot_id);
|
||||
let mut file = std::fs::File::open(&path)
|
||||
.map_err(|_| AppError::Adapter(AdapterError::snapshot_not_found(snapshot_id)))?;
|
||||
let metadata = file.metadata()?;
|
||||
if metadata.len() > crate::refs::MAX_REFMAP_BYTES {
|
||||
return Err(AppError::Internal(
|
||||
"RefMap file exceeds 1MB size limit".into(),
|
||||
));
|
||||
}
|
||||
let mut json = String::with_capacity(metadata.len() as usize);
|
||||
file.read_to_string(&mut json)?;
|
||||
if json.len() as u64 > crate::refs::MAX_REFMAP_BYTES {
|
||||
return Err(AppError::Internal(
|
||||
"RefMap file exceeds 1MB size limit".into(),
|
||||
));
|
||||
}
|
||||
Ok(serde_json::from_str(&json)?)
|
||||
}
|
||||
|
||||
pub fn set_latest(&self, snapshot_id: &str) -> Result<(), AppError> {
|
||||
self.with_write_lock(|| self.set_latest_unlocked(snapshot_id))
|
||||
}
|
||||
|
||||
pub fn latest_snapshot_id(&self) -> Option<String> {
|
||||
std::fs::read_to_string(self.latest_path())
|
||||
.ok()
|
||||
.map(|s| s.trim().to_string())
|
||||
.filter(|s| !s.is_empty())
|
||||
}
|
||||
|
||||
fn save_snapshot_unlocked(&self, snapshot_id: &str, refmap: &RefMap) -> Result<(), AppError> {
|
||||
validate_snapshot_id(snapshot_id)?;
|
||||
let json = refmap.serialize_with_size_check()?;
|
||||
let path = self.snapshot_path(snapshot_id);
|
||||
write_private_file(&path, json.as_bytes())
|
||||
}
|
||||
|
||||
fn set_latest_unlocked(&self, snapshot_id: &str) -> Result<(), AppError> {
|
||||
validate_snapshot_id(snapshot_id)?;
|
||||
write_private_file(&self.latest_path(), snapshot_id.as_bytes())
|
||||
}
|
||||
|
||||
fn latest_path(&self) -> PathBuf {
|
||||
self.base_dir.join(LATEST_SNAPSHOT_FILE)
|
||||
}
|
||||
|
||||
fn snapshot_path(&self, snapshot_id: &str) -> PathBuf {
|
||||
self.base_dir
|
||||
.join("snapshots")
|
||||
.join(snapshot_id)
|
||||
.join("refmap.json")
|
||||
}
|
||||
|
||||
fn snapshots_dir(&self) -> PathBuf {
|
||||
self.base_dir.join("snapshots")
|
||||
}
|
||||
|
||||
fn prune_old_snapshots_unlocked(&self, latest_id: &str) -> Result<(), AppError> {
|
||||
let dir = self.snapshots_dir();
|
||||
let Ok(entries) = std::fs::read_dir(&dir) else {
|
||||
return Ok(());
|
||||
};
|
||||
let mut snapshots = Vec::new();
|
||||
for entry in entries.flatten() {
|
||||
let Ok(file_type) = entry.file_type() else {
|
||||
continue;
|
||||
};
|
||||
if !file_type.is_dir() {
|
||||
continue;
|
||||
}
|
||||
let id = entry.file_name().to_string_lossy().to_string();
|
||||
if validate_snapshot_id(&id).is_err() {
|
||||
continue;
|
||||
}
|
||||
let modified = entry
|
||||
.metadata()
|
||||
.and_then(|metadata| metadata.modified())
|
||||
.unwrap_or(std::time::SystemTime::UNIX_EPOCH);
|
||||
snapshots.push((modified, id, entry.path()));
|
||||
}
|
||||
if snapshots.len() <= MAX_SAVED_SNAPSHOTS {
|
||||
return Ok(());
|
||||
}
|
||||
snapshots.sort_by_key(|(modified, id, _)| (*modified, id.clone()));
|
||||
let remove_count = snapshots.len() - MAX_SAVED_SNAPSHOTS;
|
||||
for (_, id, path) in snapshots.into_iter().take(remove_count) {
|
||||
if id != latest_id {
|
||||
let _ = std::fs::remove_dir_all(path);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn lock_path(&self) -> PathBuf {
|
||||
self.base_dir.join("refstore.lock")
|
||||
}
|
||||
|
||||
fn with_write_lock<T>(&self, f: impl FnOnce() -> Result<T, AppError>) -> Result<T, AppError> {
|
||||
let _lock = RefStoreLock::acquire(&self.lock_path())?;
|
||||
f()
|
||||
}
|
||||
|
||||
fn migrate_legacy_latest(&self) -> Result<Option<RefMap>, AppError> {
|
||||
self.with_write_lock(|| {
|
||||
if let Ok(id) = std::fs::read_to_string(self.latest_path()) {
|
||||
let id = id.trim();
|
||||
if !id.is_empty() {
|
||||
return self.load_snapshot(id).map(Some);
|
||||
}
|
||||
}
|
||||
let Ok(refmap) = RefMap::load() else {
|
||||
return Ok(None);
|
||||
};
|
||||
let snapshot_id = new_snapshot_id();
|
||||
self.save_snapshot_unlocked(&snapshot_id, &refmap)?;
|
||||
self.set_latest_unlocked(&snapshot_id)?;
|
||||
Ok(Some(refmap))
|
||||
})
|
||||
}
|
||||
}
|
||||
51
crates/core/src/refs_test_support.rs
Normal file
51
crates/core/src/refs_test_support.rs
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
use std::fs;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
static COUNTER: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
pub(crate) struct HomeGuard {
|
||||
_dir: TempDir,
|
||||
prev: Option<PathBuf>,
|
||||
}
|
||||
|
||||
impl HomeGuard {
|
||||
pub(crate) fn new() -> Self {
|
||||
let dir = TempDir::new();
|
||||
let prev = crate::refs::set_home_override(Some(dir.path().to_path_buf()));
|
||||
Self { _dir: dir, prev }
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for HomeGuard {
|
||||
fn drop(&mut self) {
|
||||
let prev = self.prev.take();
|
||||
crate::refs::set_home_override(prev);
|
||||
}
|
||||
}
|
||||
|
||||
struct TempDir(PathBuf);
|
||||
|
||||
impl TempDir {
|
||||
fn new() -> Self {
|
||||
let n = COUNTER.fetch_add(1, Ordering::SeqCst);
|
||||
let nanos = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos())
|
||||
.unwrap_or(0);
|
||||
let path = std::env::temp_dir().join(format!("agent-desktop-test-{nanos}-{n}"));
|
||||
fs::create_dir_all(&path).expect("create tempdir");
|
||||
Self(path)
|
||||
}
|
||||
|
||||
fn path(&self) -> &Path {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for TempDir {
|
||||
fn drop(&mut self) {
|
||||
let _ = fs::remove_dir_all(&self.0);
|
||||
}
|
||||
}
|
||||
309
crates/core/src/refs_tests.rs
Normal file
309
crates/core/src/refs_tests.rs
Normal file
|
|
@ -0,0 +1,309 @@
|
|||
use super::*;
|
||||
use crate::{refs_store::RefStore, refs_test_support::HomeGuard};
|
||||
|
||||
fn entry(role: &str, name: Option<&str>) -> RefEntry {
|
||||
RefEntry {
|
||||
pid: 1,
|
||||
role: role.into(),
|
||||
name: name.map(String::from),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: None,
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: None,
|
||||
source_window_title: None,
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_allocate_sequential() {
|
||||
let mut map = RefMap::new();
|
||||
let r1 = map.allocate(entry("button", Some("OK")));
|
||||
let r2 = map.allocate(entry("button", Some("OK")));
|
||||
assert_eq!(r1, "@e1");
|
||||
assert_eq!(r2, "@e2");
|
||||
assert_eq!(map.len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_get_existing() {
|
||||
let mut map = RefMap::new();
|
||||
let ref_id = map.allocate(RefEntry {
|
||||
pid: 42,
|
||||
role: "textfield".into(),
|
||||
name: None,
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: Some(12345),
|
||||
available_actions: vec![],
|
||||
source_app: Some("Finder".into()),
|
||||
source_window_title: Some("Documents".into()),
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
});
|
||||
let retrieved = map.get(&ref_id).unwrap();
|
||||
assert_eq!(retrieved.pid, 42);
|
||||
assert_eq!(retrieved.role, "textfield");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_get_missing() {
|
||||
let map = RefMap::new();
|
||||
assert!(map.get("@e99").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_remove_by_root_ref() {
|
||||
let mut map = RefMap::new();
|
||||
let base = entry("button", Some("OK"));
|
||||
|
||||
map.allocate(base.clone());
|
||||
|
||||
let drilled = RefEntry {
|
||||
root_ref: Some("@e1".into()),
|
||||
..base
|
||||
};
|
||||
map.allocate(drilled.clone());
|
||||
map.allocate(drilled);
|
||||
assert_eq!(map.len(), 3);
|
||||
|
||||
map.remove_by_root_ref("@e1");
|
||||
assert_eq!(map.len(), 1);
|
||||
assert!(map.get("@e1").is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_counter_continues_after_skeleton_into_drill_down() {
|
||||
let mut map = RefMap::new();
|
||||
let skeleton_entry = entry("button", Some("Skeleton"));
|
||||
|
||||
let last_skeleton = (0..10)
|
||||
.map(|_| map.allocate(skeleton_entry.clone()))
|
||||
.last()
|
||||
.unwrap();
|
||||
assert_eq!(last_skeleton, "@e10");
|
||||
|
||||
let drilled = RefEntry {
|
||||
root_ref: Some("@e3".into()),
|
||||
..skeleton_entry
|
||||
};
|
||||
|
||||
let first_drilled = map.allocate(drilled.clone());
|
||||
let second_drilled = map.allocate(drilled);
|
||||
assert_eq!(
|
||||
first_drilled, "@e11",
|
||||
"counter should continue past skeleton ids, not reset"
|
||||
);
|
||||
assert_eq!(second_drilled, "@e12");
|
||||
assert_eq!(map.len(), 12);
|
||||
|
||||
map.remove_by_root_ref("@e3");
|
||||
assert_eq!(
|
||||
map.len(),
|
||||
10,
|
||||
"scoped invalidation should drop only the drill-down refs"
|
||||
);
|
||||
assert!(map.get("@e3").is_some(), "skeleton @e3 must survive");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_root_ref_serde_roundtrip() {
|
||||
let entry = RefEntry {
|
||||
root_ref: Some("@e5".into()),
|
||||
..entry("button", None)
|
||||
};
|
||||
let json = serde_json::to_string(&entry).unwrap();
|
||||
assert!(json.contains("root_ref"));
|
||||
let back: RefEntry = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(back.root_ref.as_deref(), Some("@e5"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_serialize_with_size_check_rejects_oversized() {
|
||||
let mut map = RefMap::new();
|
||||
let big_name = "x".repeat(2048);
|
||||
for _ in 0..600 {
|
||||
map.allocate(entry("button", Some(&big_name)));
|
||||
}
|
||||
|
||||
let result = map.serialize_with_size_check();
|
||||
assert!(result.is_err(), "oversized refmap should be rejected");
|
||||
let err = result.unwrap_err();
|
||||
let msg = err.to_string();
|
||||
assert!(
|
||||
msg.contains("1MB"),
|
||||
"error should mention the 1MB limit, got: {msg}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_serialize_with_size_check_accepts_normal() {
|
||||
let mut map = RefMap::new();
|
||||
for _ in 0..50 {
|
||||
map.allocate(entry("button", Some("OK")));
|
||||
}
|
||||
|
||||
let result = map.serialize_with_size_check();
|
||||
assert!(result.is_ok(), "normal-sized refmap should serialize");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_save_load_roundtrip_with_home_override() {
|
||||
let _guard = HomeGuard::new();
|
||||
let mut map = RefMap::new();
|
||||
map.allocate(RefEntry {
|
||||
pid: 7,
|
||||
role: "button".into(),
|
||||
name: Some("Send".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: Some(42),
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: Some("TestApp".into()),
|
||||
source_window_title: Some("Test Window".into()),
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
});
|
||||
map.save().expect("save should succeed under HomeGuard");
|
||||
|
||||
let loaded = RefMap::load().expect("load should succeed");
|
||||
assert_eq!(loaded.len(), 1);
|
||||
let entry = loaded.get("@e1").unwrap();
|
||||
assert_eq!(entry.pid, 7);
|
||||
assert_eq!(entry.name.as_deref(), Some("Send"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_refstore_snapshot_roundtrip_and_latest_pointer() {
|
||||
let _guard = HomeGuard::new();
|
||||
let store = RefStore::new().unwrap();
|
||||
let mut map = RefMap::new();
|
||||
map.allocate(RefEntry {
|
||||
pid: 7,
|
||||
role: "button".into(),
|
||||
name: Some("Send".into()),
|
||||
value: None,
|
||||
states: vec![],
|
||||
bounds: None,
|
||||
bounds_hash: Some(42),
|
||||
available_actions: vec!["Click".into()],
|
||||
source_app: Some("TestApp".into()),
|
||||
source_window_title: Some("Test Window".into()),
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
});
|
||||
|
||||
let snapshot_id = store.save_new_snapshot(&map).unwrap();
|
||||
|
||||
assert_eq!(
|
||||
store.latest_snapshot_id().as_deref(),
|
||||
Some(snapshot_id.as_str())
|
||||
);
|
||||
assert_eq!(store.load(Some(&snapshot_id)).unwrap().len(), 1);
|
||||
assert_eq!(store.load(None).unwrap().len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_save_existing_snapshot_does_not_promote_latest_pointer() {
|
||||
let _guard = HomeGuard::new();
|
||||
let store = RefStore::new().unwrap();
|
||||
|
||||
let mut first = RefMap::new();
|
||||
first.allocate(entry("button", Some("First")));
|
||||
let first_id = store.save_new_snapshot(&first).unwrap();
|
||||
|
||||
let mut second = RefMap::new();
|
||||
second.allocate(entry("button", Some("Second")));
|
||||
let second_id = store.save_new_snapshot(&second).unwrap();
|
||||
|
||||
first.allocate(entry("button", Some("First Child")));
|
||||
store.save_existing_snapshot(&first_id, &first).unwrap();
|
||||
|
||||
assert_eq!(
|
||||
store.latest_snapshot_id().as_deref(),
|
||||
Some(second_id.as_str())
|
||||
);
|
||||
assert_eq!(store.load(Some(&first_id)).unwrap().len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_snapshot_ids_are_compact_and_valid() {
|
||||
let id = new_snapshot_id();
|
||||
|
||||
assert!(id.starts_with('s'));
|
||||
assert!(id.len() <= 14, "snapshot id should stay token-light: {id}");
|
||||
validate_snapshot_id(&id).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_base36_encoding() {
|
||||
assert_eq!(base36(0), "0000");
|
||||
assert_eq!(base36(35), "000z");
|
||||
assert_eq!(base36(36), "0010");
|
||||
assert_eq!(base36(36 * 36 * 36 + 35), "100z");
|
||||
assert!(base36(u64::MAX).len() >= 4);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_new_snapshot_id_passes_validation() {
|
||||
for _ in 0..256 {
|
||||
let id = new_snapshot_id();
|
||||
validate_snapshot_id(&id).expect("generated snapshot id must validate");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_refstore_migrates_legacy_latest_refmap() {
|
||||
let _guard = HomeGuard::new();
|
||||
let mut map = RefMap::new();
|
||||
map.allocate(entry("button", Some("Legacy")));
|
||||
map.save().unwrap();
|
||||
|
||||
let store = RefStore::new().unwrap();
|
||||
let loaded = store.load_latest().unwrap();
|
||||
|
||||
assert_eq!(loaded.len(), 1);
|
||||
assert!(store.latest_snapshot_id().is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_validate_snapshot_id_rejects_bad_values() {
|
||||
assert!(validate_snapshot_id("").is_err());
|
||||
assert!(validate_snapshot_id("s").is_err());
|
||||
assert!(validate_snapshot_id(&format!("s{}", "x".repeat(64))).is_err());
|
||||
assert!(validate_snapshot_id("bad/id").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_save_oversize_preserves_previous_file() {
|
||||
let _guard = HomeGuard::new();
|
||||
|
||||
let mut original = RefMap::new();
|
||||
original.allocate(entry("button", Some("Original")));
|
||||
original.save().expect("baseline save");
|
||||
|
||||
let mut oversize = RefMap::new();
|
||||
let big = "x".repeat(2048);
|
||||
for _ in 0..600 {
|
||||
oversize.allocate(entry("button", Some(&big)));
|
||||
}
|
||||
let result = oversize.save();
|
||||
assert!(result.is_err(), "oversize save must reject");
|
||||
|
||||
let reloaded = RefMap::load().expect("previous file must still load");
|
||||
assert_eq!(reloaded.len(), 1);
|
||||
let entry = reloaded.get("@e1").unwrap();
|
||||
assert_eq!(entry.name.as_deref(), Some("Original"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_root_ref_none_omitted() {
|
||||
let entry = entry("button", None);
|
||||
let json = serde_json::to_string(&entry).unwrap();
|
||||
assert!(!json.contains("root_ref"));
|
||||
}
|
||||
104
crates/core/src/roles.rs
Normal file
104
crates/core/src/roles.rs
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
/// Interactive roles that receive refs during snapshot allocation.
|
||||
///
|
||||
/// Each entry must be produced by at least one platform adapter's native-to-canonical
|
||||
/// role mapping. Read-only roles (statictext, image) and container roles (group, list,
|
||||
/// table) stay out. Platform-private extensions live in the adapter, not here.
|
||||
pub const INTERACTIVE_ROLES: &[&str] = &[
|
||||
"button",
|
||||
"cell",
|
||||
"checkbox",
|
||||
"colorwell",
|
||||
"combobox",
|
||||
"dockitem",
|
||||
"incrementor",
|
||||
"link",
|
||||
"menubutton",
|
||||
"menuitem",
|
||||
"radiobutton",
|
||||
"slider",
|
||||
"switch",
|
||||
"tab",
|
||||
"textfield",
|
||||
"treeitem",
|
||||
];
|
||||
|
||||
/// Returns true when `role` is in [`INTERACTIVE_ROLES`].
|
||||
pub fn is_interactive_role(role: &str) -> bool {
|
||||
INTERACTIVE_ROLES.contains(&role)
|
||||
}
|
||||
|
||||
/// Returns true for roles whose checked/unchecked state can be queried and set.
|
||||
pub fn is_toggleable_role(role: &str) -> bool {
|
||||
matches!(role, "checkbox" | "switch" | "radiobutton")
|
||||
}
|
||||
|
||||
/// Returns true for roles that carry an expanded/collapsed surface state.
|
||||
pub fn is_expandable_role(role: &str) -> bool {
|
||||
matches!(role, "combobox" | "menubutton" | "treeitem")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn interactive_roles_are_sorted_and_unique() {
|
||||
let mut sorted = INTERACTIVE_ROLES.to_vec();
|
||||
sorted.sort_unstable();
|
||||
sorted.dedup();
|
||||
assert_eq!(sorted.as_slice(), INTERACTIVE_ROLES);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn toggleable_roles_are_a_subset_of_interactive() {
|
||||
for role in ["checkbox", "switch", "radiobutton"] {
|
||||
assert!(is_toggleable_role(role));
|
||||
assert!(is_interactive_role(role));
|
||||
}
|
||||
assert!(!is_toggleable_role("button"));
|
||||
assert!(!is_toggleable_role("textfield"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn expandable_roles_are_a_subset_of_interactive() {
|
||||
for role in ["combobox", "menubutton", "treeitem"] {
|
||||
assert!(is_expandable_role(role));
|
||||
assert!(is_interactive_role(role));
|
||||
}
|
||||
assert!(!is_expandable_role("button"));
|
||||
assert!(!is_expandable_role("checkbox"));
|
||||
assert!(!is_expandable_role("disclosure"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_expandable_role_is_interactive() {
|
||||
for role in ["combobox", "menubutton", "treeitem"] {
|
||||
assert!(
|
||||
is_expandable_role(role),
|
||||
"{role} expected expandable for subset check"
|
||||
);
|
||||
assert!(
|
||||
INTERACTIVE_ROLES.contains(&role),
|
||||
"expandable role {role} missing from INTERACTIVE_ROLES"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_toggleable_role_is_interactive() {
|
||||
for role in ["checkbox", "switch", "radiobutton"] {
|
||||
assert!(is_toggleable_role(role));
|
||||
assert!(
|
||||
INTERACTIVE_ROLES.contains(&role),
|
||||
"toggleable role {role} missing from INTERACTIVE_ROLES"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn read_only_roles_are_never_interactive() {
|
||||
for role in ["statictext", "image", "group", "list", "table"] {
|
||||
assert!(!is_interactive_role(role));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -4,12 +4,14 @@ use crate::{
|
|||
node::{AccessibilityNode, WindowInfo},
|
||||
ref_alloc::{self, RefAllocConfig},
|
||||
refs::RefMap,
|
||||
refs_store::RefStore,
|
||||
};
|
||||
|
||||
pub struct SnapshotResult {
|
||||
pub tree: AccessibilityNode,
|
||||
pub refmap: RefMap,
|
||||
pub window: WindowInfo,
|
||||
pub snapshot_id: Option<String>,
|
||||
}
|
||||
|
||||
pub fn build(
|
||||
|
|
@ -82,6 +84,7 @@ pub fn build(
|
|||
compact: opts.compact,
|
||||
pid: window.pid,
|
||||
source_app: Some(window.app.as_str()),
|
||||
source_window_title: Some(window.title.as_str()),
|
||||
root_ref_id: None,
|
||||
};
|
||||
let mut tree = ref_alloc::allocate_refs(raw_tree, &mut refmap, &config);
|
||||
|
|
@ -92,6 +95,7 @@ pub fn build(
|
|||
tree,
|
||||
refmap,
|
||||
window,
|
||||
snapshot_id: None,
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -101,8 +105,10 @@ pub fn run(
|
|||
app_name: Option<&str>,
|
||||
window_id: Option<&str>,
|
||||
) -> Result<SnapshotResult, AppError> {
|
||||
let result = build(adapter, opts, app_name, window_id)?;
|
||||
result.refmap.save()?;
|
||||
let mut result = build(adapter, opts, app_name, window_id)?;
|
||||
let store = RefStore::new()?;
|
||||
let snapshot_id = store.save_new_snapshot(&result.refmap)?;
|
||||
result.snapshot_id = Some(snapshot_id);
|
||||
Ok(result)
|
||||
}
|
||||
|
||||
|
|
@ -111,31 +117,39 @@ pub fn append_surface_refs(
|
|||
pid: i32,
|
||||
source_app: Option<&str>,
|
||||
surface: SnapshotSurface,
|
||||
) -> Option<AccessibilityNode> {
|
||||
) -> Result<Option<AccessibilityNode>, AppError> {
|
||||
let filter = WindowFilter {
|
||||
focused_only: false,
|
||||
app: None,
|
||||
};
|
||||
let windows = adapter.list_windows(&filter).ok()?;
|
||||
let window = windows.into_iter().find(|w| w.pid == pid)?;
|
||||
let windows = adapter.list_windows(&filter)?;
|
||||
let Some(window) = windows.into_iter().find(|w| w.pid == pid) else {
|
||||
return Ok(None);
|
||||
};
|
||||
let opts = TreeOptions {
|
||||
surface,
|
||||
interactive_only: true,
|
||||
..Default::default()
|
||||
};
|
||||
let raw_tree = adapter.get_tree(&window, &opts).ok()?;
|
||||
let mut refmap = RefMap::load().ok()?;
|
||||
let raw_tree = adapter.get_tree(&window, &opts)?;
|
||||
let store = RefStore::new()?;
|
||||
let mut refmap = store.load_latest()?;
|
||||
let config = RefAllocConfig {
|
||||
include_bounds: false,
|
||||
interactive_only: true,
|
||||
compact: false,
|
||||
pid,
|
||||
source_app,
|
||||
source_window_title: Some(window.title.as_str()),
|
||||
root_ref_id: None,
|
||||
};
|
||||
let tree = ref_alloc::allocate_refs(raw_tree, &mut refmap, &config);
|
||||
refmap.save().ok()?;
|
||||
Some(tree)
|
||||
if let Some(id) = store.latest_snapshot_id() {
|
||||
store.save_existing_snapshot(&id, &refmap)?;
|
||||
} else {
|
||||
store.save_new_snapshot(&refmap)?;
|
||||
}
|
||||
Ok(Some(tree))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
use crate::{
|
||||
adapter::{PlatformAdapter, TreeOptions, WindowFilter},
|
||||
adapter::{PlatformAdapter, TreeOptions},
|
||||
error::AppError,
|
||||
node::WindowInfo,
|
||||
ref_alloc::{self, RefAllocConfig},
|
||||
refs::RefMap,
|
||||
refs_store::RefStore,
|
||||
snapshot::SnapshotResult,
|
||||
};
|
||||
|
||||
|
|
@ -11,8 +11,13 @@ pub fn run_from_ref(
|
|||
adapter: &dyn PlatformAdapter,
|
||||
opts: &TreeOptions,
|
||||
root_ref_id: &str,
|
||||
snapshot_id: Option<&str>,
|
||||
) -> Result<SnapshotResult, AppError> {
|
||||
let mut refmap = RefMap::load()?;
|
||||
let store = RefStore::new()?;
|
||||
let mut refmap = store.load(snapshot_id)?;
|
||||
let active_snapshot_id = snapshot_id
|
||||
.map(str::to_string)
|
||||
.or_else(|| store.latest_snapshot_id());
|
||||
|
||||
let entry = refmap
|
||||
.get(root_ref_id)
|
||||
|
|
@ -26,12 +31,14 @@ pub fn run_from_ref(
|
|||
refmap.remove_by_root_ref(root_ref_id);
|
||||
|
||||
let source_app = entry.source_app.as_deref();
|
||||
let source_window_title = entry.source_window_title.as_deref();
|
||||
let config = RefAllocConfig {
|
||||
include_bounds: opts.include_bounds,
|
||||
interactive_only: opts.interactive_only,
|
||||
compact: opts.compact,
|
||||
pid: entry.pid,
|
||||
source_app,
|
||||
source_window_title,
|
||||
root_ref_id: Some(root_ref_id),
|
||||
};
|
||||
|
||||
|
|
@ -39,18 +46,19 @@ pub fn run_from_ref(
|
|||
|
||||
crate::hints::add_structural_hints(&mut tree);
|
||||
|
||||
refmap.save()?;
|
||||
let saved_snapshot_id = if let Some(id) = active_snapshot_id {
|
||||
store.save_existing_snapshot(&id, &refmap)?;
|
||||
Some(id)
|
||||
} else {
|
||||
Some(store.save_new_snapshot(&refmap)?)
|
||||
};
|
||||
|
||||
let window = adapter
|
||||
.list_windows(&WindowFilter {
|
||||
focused_only: false,
|
||||
app: None,
|
||||
})
|
||||
.ok()
|
||||
.and_then(|ws| ws.into_iter().find(|w| w.pid == entry.pid))
|
||||
.unwrap_or(WindowInfo {
|
||||
let window =
|
||||
crate::window_lookup::find_window_for_pid(entry.pid, adapter).unwrap_or(WindowInfo {
|
||||
id: String::new(),
|
||||
title: format!("subtree from {root_ref_id}"),
|
||||
title: entry
|
||||
.source_window_title
|
||||
.unwrap_or_else(|| format!("subtree from {root_ref_id}")),
|
||||
app: entry.source_app.unwrap_or_default(),
|
||||
pid: entry.pid,
|
||||
bounds: None,
|
||||
|
|
@ -61,9 +69,14 @@ pub fn run_from_ref(
|
|||
tree,
|
||||
refmap,
|
||||
window,
|
||||
snapshot_id: saved_snapshot_id,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "snapshot_ref_tests.rs"]
|
||||
mod tests;
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "snapshot_ref_alloc_tests.rs"]
|
||||
mod alloc_tests;
|
||||
|
|
|
|||
107
crates/core/src/snapshot_ref_alloc_tests.rs
Normal file
107
crates/core/src/snapshot_ref_alloc_tests.rs
Normal file
|
|
@ -0,0 +1,107 @@
|
|||
use crate::node::AccessibilityNode;
|
||||
use crate::ref_alloc::{self, RefAllocConfig};
|
||||
use crate::refs::RefMap;
|
||||
|
||||
fn node(role: &str) -> AccessibilityNode {
|
||||
AccessibilityNode {
|
||||
ref_id: None,
|
||||
role: role.into(),
|
||||
name: None,
|
||||
value: None,
|
||||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
available_actions: vec![],
|
||||
bounds: None,
|
||||
children_count: None,
|
||||
children: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
fn drill_config<'a>(
|
||||
source_app: Option<&'a str>,
|
||||
pid: i32,
|
||||
root_ref_id: &'a str,
|
||||
interactive_only: bool,
|
||||
compact: bool,
|
||||
) -> RefAllocConfig<'a> {
|
||||
RefAllocConfig {
|
||||
include_bounds: false,
|
||||
interactive_only,
|
||||
compact,
|
||||
pid,
|
||||
source_app,
|
||||
source_window_title: Some("Drill Window"),
|
||||
root_ref_id: Some(root_ref_id),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_drill_alloc_tags_entries() {
|
||||
let mut btn = node("button");
|
||||
btn.name = Some("Submit".into());
|
||||
let mut root = node("group");
|
||||
root.children = vec![btn];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = drill_config(Some("TestApp"), 42, "@e5", false, false);
|
||||
let tree = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert_eq!(refmap.len(), 1);
|
||||
let btn_ref = tree.children[0]
|
||||
.ref_id
|
||||
.as_deref()
|
||||
.expect("button should have ref");
|
||||
let entry = refmap.get(btn_ref).expect("entry should exist");
|
||||
assert_eq!(entry.root_ref.as_deref(), Some("@e5"));
|
||||
assert_eq!(entry.pid, 42);
|
||||
assert_eq!(entry.source_app.as_deref(), Some("TestApp"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_drill_alloc_respects_interactive_only() {
|
||||
let btn = node("button");
|
||||
let text = node("statictext");
|
||||
let mut root = node("group");
|
||||
root.children = vec![btn, text];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = drill_config(None, 1, "@e1", true, false);
|
||||
let tree = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert_eq!(tree.children.len(), 1);
|
||||
assert_eq!(tree.children[0].role, "button");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_drill_alloc_preserves_truncated_child() {
|
||||
let mut container = node("group");
|
||||
container.name = Some("Sidebar".into());
|
||||
container.children_count = Some(4);
|
||||
let mut root = node("window");
|
||||
root.children = vec![container];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = drill_config(None, 1, "@e1", true, false);
|
||||
let tree = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert_eq!(tree.children.len(), 1);
|
||||
assert_eq!(tree.children[0].children_count, Some(4));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_drill_alloc_compact() {
|
||||
let mut btn = node("button");
|
||||
btn.name = Some("OK".into());
|
||||
let mut wrapper = node("group");
|
||||
wrapper.children = vec![btn];
|
||||
let mut root = node("window");
|
||||
root.children = vec![wrapper];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = drill_config(None, 1, "@e1", false, true);
|
||||
let tree = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert_eq!(tree.children.len(), 1);
|
||||
assert_eq!(tree.children[0].role, "button");
|
||||
}
|
||||
|
|
@ -1,11 +1,12 @@
|
|||
use super::*;
|
||||
use crate::action::Action;
|
||||
use crate::adapter::{NativeHandle, PermissionStatus, PlatformAdapter};
|
||||
use crate::action::ActionRequest;
|
||||
use crate::adapter::{NativeHandle, PlatformAdapter};
|
||||
use crate::error::AdapterError;
|
||||
use crate::node::AccessibilityNode;
|
||||
use crate::ref_alloc::ref_entry_from_node;
|
||||
use crate::refs::HomeGuard;
|
||||
use std::cell::Cell;
|
||||
use crate::refs_test_support::HomeGuard;
|
||||
use crate::{refs::RefMap, refs_store::RefStore};
|
||||
use std::sync::atomic::{AtomicU32, Ordering};
|
||||
|
||||
fn node(role: &str) -> AccessibilityNode {
|
||||
AccessibilityNode {
|
||||
|
|
@ -16,6 +17,7 @@ fn node(role: &str) -> AccessibilityNode {
|
|||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
available_actions: vec![],
|
||||
bounds: None,
|
||||
children_count: None,
|
||||
children: vec![],
|
||||
|
|
@ -30,31 +32,24 @@ fn named(role: &str, name: &str) -> AccessibilityNode {
|
|||
|
||||
struct StubAdapter {
|
||||
subtree: AccessibilityNode,
|
||||
resolve_calls: Cell<u32>,
|
||||
resolve_calls: AtomicU32,
|
||||
}
|
||||
|
||||
impl StubAdapter {
|
||||
fn new(subtree: AccessibilityNode) -> Self {
|
||||
Self {
|
||||
subtree,
|
||||
resolve_calls: Cell::new(0),
|
||||
resolve_calls: AtomicU32::new(0),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
unsafe impl Send for StubAdapter {}
|
||||
unsafe impl Sync for StubAdapter {}
|
||||
|
||||
impl PlatformAdapter for StubAdapter {
|
||||
fn check_permissions(&self) -> PermissionStatus {
|
||||
PermissionStatus::Granted
|
||||
}
|
||||
|
||||
fn resolve_element(
|
||||
&self,
|
||||
_entry: &crate::refs::RefEntry,
|
||||
) -> Result<NativeHandle, AdapterError> {
|
||||
self.resolve_calls.set(self.resolve_calls.get() + 1);
|
||||
self.resolve_calls.fetch_add(1, Ordering::SeqCst);
|
||||
Ok(NativeHandle::null())
|
||||
}
|
||||
|
||||
|
|
@ -69,17 +64,45 @@ impl PlatformAdapter for StubAdapter {
|
|||
fn execute_action(
|
||||
&self,
|
||||
_handle: &NativeHandle,
|
||||
_action: Action,
|
||||
_request: ActionRequest,
|
||||
) -> Result<crate::action::ActionResult, AdapterError> {
|
||||
Err(AdapterError::not_supported("execute_action"))
|
||||
}
|
||||
}
|
||||
|
||||
fn save_latest(refmap: RefMap) -> String {
|
||||
RefStore::new()
|
||||
.unwrap()
|
||||
.save_new_snapshot(&refmap)
|
||||
.expect("snapshot refmap should save")
|
||||
}
|
||||
|
||||
fn load_latest() -> RefMap {
|
||||
RefStore::new()
|
||||
.unwrap()
|
||||
.load_latest()
|
||||
.expect("latest snapshot should load")
|
||||
}
|
||||
|
||||
fn seed_skeleton_refmap() -> RefMap {
|
||||
let mut map = RefMap::new();
|
||||
let anchor = ref_entry_from_node(&named("group", "Sidebar"), 42, Some("TestApp"), None);
|
||||
let anchor = ref_entry_from_node(
|
||||
&named("group", "Sidebar"),
|
||||
42,
|
||||
Some("TestApp"),
|
||||
None,
|
||||
None,
|
||||
&[0],
|
||||
);
|
||||
let _ = map.allocate(anchor);
|
||||
let other = ref_entry_from_node(&named("button", "Toolbar"), 42, Some("TestApp"), None);
|
||||
let other = ref_entry_from_node(
|
||||
&named("button", "Toolbar"),
|
||||
42,
|
||||
Some("TestApp"),
|
||||
None,
|
||||
None,
|
||||
&[1],
|
||||
);
|
||||
let _ = map.allocate(other);
|
||||
map
|
||||
}
|
||||
|
|
@ -94,7 +117,7 @@ fn drill_opts() -> TreeOptions {
|
|||
#[test]
|
||||
fn test_run_from_ref_returns_subtree_and_persists_refs() {
|
||||
let _guard = HomeGuard::new();
|
||||
seed_skeleton_refmap().save().unwrap();
|
||||
save_latest(seed_skeleton_refmap());
|
||||
|
||||
let mut child_btn = named("button", "Save");
|
||||
child_btn.children = vec![];
|
||||
|
|
@ -102,9 +125,9 @@ fn test_run_from_ref_returns_subtree_and_persists_refs() {
|
|||
subtree_root.children = vec![child_btn];
|
||||
|
||||
let adapter = StubAdapter::new(subtree_root);
|
||||
let result = run_from_ref(&adapter, &drill_opts(), "@e1").expect("drill should succeed");
|
||||
let result = run_from_ref(&adapter, &drill_opts(), "@e1", None).expect("drill should succeed");
|
||||
|
||||
let on_disk = RefMap::load().unwrap();
|
||||
let on_disk = load_latest();
|
||||
assert_eq!(on_disk.len(), result.refmap.len());
|
||||
assert!(
|
||||
result.refmap.len() >= 3,
|
||||
|
|
@ -121,16 +144,16 @@ fn test_run_from_ref_returns_subtree_and_persists_refs() {
|
|||
.expect("button child should carry a ref");
|
||||
let drill_entry = on_disk.get(drill_ref).expect("entry persisted");
|
||||
assert_eq!(drill_entry.root_ref.as_deref(), Some("@e1"));
|
||||
assert_eq!(adapter.resolve_calls.get(), 1);
|
||||
assert_eq!(adapter.resolve_calls.load(Ordering::SeqCst), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_run_from_ref_stale_root_returns_stale_ref() {
|
||||
let _guard = HomeGuard::new();
|
||||
RefMap::new().save().unwrap();
|
||||
save_latest(RefMap::new());
|
||||
|
||||
let adapter = StubAdapter::new(named("group", "Sidebar"));
|
||||
let result = run_from_ref(&adapter, &drill_opts(), "@e99");
|
||||
let result = run_from_ref(&adapter, &drill_opts(), "@e99", None);
|
||||
let err = match result {
|
||||
Ok(_) => panic!("stale root must error"),
|
||||
Err(e) => e,
|
||||
|
|
@ -151,16 +174,16 @@ fn test_run_from_ref_stale_root_returns_stale_ref() {
|
|||
#[test]
|
||||
fn test_run_from_ref_re_drill_replaces_drill_refs_only() {
|
||||
let _guard = HomeGuard::new();
|
||||
seed_skeleton_refmap().save().unwrap();
|
||||
save_latest(seed_skeleton_refmap());
|
||||
|
||||
let subtree = named("button", "Save");
|
||||
let adapter = StubAdapter::new(subtree);
|
||||
|
||||
let first = run_from_ref(&adapter, &drill_opts(), "@e1").unwrap();
|
||||
let first = run_from_ref(&adapter, &drill_opts(), "@e1", None).unwrap();
|
||||
let first_count = first.refmap.len();
|
||||
let first_button_ref = first.tree.ref_id.clone().expect("button should get a ref");
|
||||
|
||||
let second = run_from_ref(&adapter, &drill_opts(), "@e1").unwrap();
|
||||
let second = run_from_ref(&adapter, &drill_opts(), "@e1", None).unwrap();
|
||||
let second_count = second.refmap.len();
|
||||
let second_button_ref = second.tree.ref_id.clone().expect("button should get a ref");
|
||||
|
||||
|
|
@ -172,7 +195,7 @@ fn test_run_from_ref_re_drill_replaces_drill_refs_only() {
|
|||
first_button_ref, second_button_ref,
|
||||
"re-drill should issue a fresh ref id (counter continues)"
|
||||
);
|
||||
let on_disk = RefMap::load().unwrap();
|
||||
let on_disk = load_latest();
|
||||
assert!(on_disk.get("@e1").is_some(), "skeleton anchor preserved");
|
||||
assert!(on_disk.get(&second_button_ref).is_some());
|
||||
assert!(
|
||||
|
|
@ -184,17 +207,17 @@ fn test_run_from_ref_re_drill_replaces_drill_refs_only() {
|
|||
#[test]
|
||||
fn test_run_from_ref_multiple_drill_downs_accumulate() {
|
||||
let _guard = HomeGuard::new();
|
||||
seed_skeleton_refmap().save().unwrap();
|
||||
save_latest(seed_skeleton_refmap());
|
||||
|
||||
let adapter_one = StubAdapter::new(named("button", "FromE1"));
|
||||
let first = run_from_ref(&adapter_one, &drill_opts(), "@e1").unwrap();
|
||||
let first = run_from_ref(&adapter_one, &drill_opts(), "@e1", None).unwrap();
|
||||
let from_e1_ref = first.tree.ref_id.clone().expect("first drill ref");
|
||||
|
||||
let adapter_two = StubAdapter::new(named("button", "FromE2"));
|
||||
let second = run_from_ref(&adapter_two, &drill_opts(), "@e2").unwrap();
|
||||
let second = run_from_ref(&adapter_two, &drill_opts(), "@e2", None).unwrap();
|
||||
let from_e2_ref = second.tree.ref_id.clone().expect("second drill ref");
|
||||
|
||||
let on_disk = RefMap::load().unwrap();
|
||||
let on_disk = load_latest();
|
||||
assert!(on_disk.get("@e1").is_some(), "skeleton @e1 preserved");
|
||||
assert!(on_disk.get("@e2").is_some(), "skeleton @e2 preserved");
|
||||
let entry_one = on_disk.get(&from_e1_ref).expect("@e1 drill survives");
|
||||
|
|
@ -216,25 +239,29 @@ fn test_drilldown_refmap_matches_golden_fixture() {
|
|||
42,
|
||||
Some("Fixture"),
|
||||
None,
|
||||
None,
|
||||
&[0],
|
||||
));
|
||||
seed.allocate(ref_entry_from_node(
|
||||
&named("group", "Toolbar"),
|
||||
42,
|
||||
Some("Fixture"),
|
||||
None,
|
||||
None,
|
||||
&[1],
|
||||
));
|
||||
seed.save().unwrap();
|
||||
save_latest(seed);
|
||||
|
||||
let mut sidebar_subtree = named("outline", "Sidebar");
|
||||
sidebar_subtree.children = vec![named("treeitem", "Recents"), named("treeitem", "Documents")];
|
||||
let adapter = StubAdapter::new(sidebar_subtree);
|
||||
let _ = run_from_ref(&adapter, &drill_opts(), "@e1").unwrap();
|
||||
let _ = run_from_ref(&adapter, &drill_opts(), "@e1", None).unwrap();
|
||||
|
||||
let toolbar_subtree = named("button", "Back");
|
||||
let adapter = StubAdapter::new(toolbar_subtree);
|
||||
let _ = run_from_ref(&adapter, &drill_opts(), "@e2").unwrap();
|
||||
let _ = run_from_ref(&adapter, &drill_opts(), "@e2", None).unwrap();
|
||||
|
||||
let on_disk = RefMap::load().unwrap();
|
||||
let on_disk = load_latest();
|
||||
assert_eq!(
|
||||
on_disk.len(),
|
||||
expected_total,
|
||||
|
|
@ -263,10 +290,10 @@ fn test_drilldown_refmap_matches_golden_fixture() {
|
|||
#[test]
|
||||
fn test_run_from_ref_empty_subtree() {
|
||||
let _guard = HomeGuard::new();
|
||||
seed_skeleton_refmap().save().unwrap();
|
||||
save_latest(seed_skeleton_refmap());
|
||||
|
||||
let adapter = StubAdapter::new(node("group"));
|
||||
let result = run_from_ref(&adapter, &drill_opts(), "@e1").unwrap();
|
||||
let result = run_from_ref(&adapter, &drill_opts(), "@e1", None).unwrap();
|
||||
|
||||
assert!(result.tree.children.is_empty());
|
||||
assert_eq!(
|
||||
|
|
@ -275,90 +302,3 @@ fn test_run_from_ref_empty_subtree() {
|
|||
"no new refs added for empty subtree"
|
||||
);
|
||||
}
|
||||
|
||||
fn drill_config<'a>(
|
||||
source_app: Option<&'a str>,
|
||||
pid: i32,
|
||||
root_ref_id: &'a str,
|
||||
interactive_only: bool,
|
||||
compact: bool,
|
||||
) -> RefAllocConfig<'a> {
|
||||
RefAllocConfig {
|
||||
include_bounds: false,
|
||||
interactive_only,
|
||||
compact,
|
||||
pid,
|
||||
source_app,
|
||||
root_ref_id: Some(root_ref_id),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_drill_alloc_tags_entries() {
|
||||
let mut btn = node("button");
|
||||
btn.name = Some("Submit".into());
|
||||
let mut root = node("group");
|
||||
root.children = vec![btn];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = drill_config(Some("TestApp"), 42, "@e5", false, false);
|
||||
let tree = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert_eq!(refmap.len(), 1);
|
||||
let btn_ref = tree.children[0]
|
||||
.ref_id
|
||||
.as_deref()
|
||||
.expect("button should have ref");
|
||||
let entry = refmap.get(btn_ref).expect("entry should exist");
|
||||
assert_eq!(entry.root_ref.as_deref(), Some("@e5"));
|
||||
assert_eq!(entry.pid, 42);
|
||||
assert_eq!(entry.source_app.as_deref(), Some("TestApp"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_drill_alloc_respects_interactive_only() {
|
||||
let btn = node("button");
|
||||
let text = node("statictext");
|
||||
let mut root = node("group");
|
||||
root.children = vec![btn, text];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = drill_config(None, 1, "@e1", true, false);
|
||||
let tree = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert_eq!(tree.children.len(), 1);
|
||||
assert_eq!(tree.children[0].role, "button");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_drill_alloc_preserves_truncated_child() {
|
||||
let mut container = node("group");
|
||||
container.name = Some("Sidebar".into());
|
||||
container.children_count = Some(4);
|
||||
let mut root = node("window");
|
||||
root.children = vec![container];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = drill_config(None, 1, "@e1", true, false);
|
||||
let tree = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert_eq!(tree.children.len(), 1);
|
||||
assert_eq!(tree.children[0].children_count, Some(4));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_drill_alloc_compact() {
|
||||
let mut btn = node("button");
|
||||
btn.name = Some("OK".into());
|
||||
let mut wrapper = node("group");
|
||||
wrapper.children = vec![btn];
|
||||
let mut root = node("window");
|
||||
root.children = vec![wrapper];
|
||||
|
||||
let mut refmap = RefMap::new();
|
||||
let config = drill_config(None, 1, "@e1", false, true);
|
||||
let tree = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
||||
assert_eq!(tree.children.len(), 1);
|
||||
assert_eq!(tree.children[0].role, "button");
|
||||
}
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ fn node(role: &str) -> AccessibilityNode {
|
|||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
available_actions: vec![],
|
||||
bounds: None,
|
||||
children_count: None,
|
||||
children: vec![],
|
||||
|
|
@ -23,6 +24,7 @@ fn run_config(compact: bool, interactive_only: bool) -> RefAllocConfig<'static>
|
|||
compact,
|
||||
pid: 1,
|
||||
source_app: Some("Test"),
|
||||
source_window_title: Some("Test Window"),
|
||||
root_ref_id: None,
|
||||
}
|
||||
}
|
||||
|
|
@ -181,6 +183,7 @@ fn test_skeleton_anchor_suppressed_in_drilldown() {
|
|||
compact: false,
|
||||
pid: 1,
|
||||
source_app: Some("Test"),
|
||||
source_window_title: Some("Test Window"),
|
||||
root_ref_id: Some("@e3"),
|
||||
};
|
||||
let result = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
|
@ -254,6 +257,7 @@ fn test_skeleton_fixture_matches_golden() {
|
|||
compact: false,
|
||||
pid: 42,
|
||||
source_app: Some("Fixture"),
|
||||
source_window_title: Some("Fixture Window"),
|
||||
root_ref_id: None,
|
||||
};
|
||||
let result = ref_alloc::allocate_refs(root, &mut refmap, &config);
|
||||
|
|
|
|||
27
crates/core/src/window_lookup.rs
Normal file
27
crates/core/src/window_lookup.rs
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
use crate::{
|
||||
adapter::{PlatformAdapter, WindowFilter},
|
||||
error::AppError,
|
||||
node::WindowInfo,
|
||||
};
|
||||
|
||||
pub(crate) fn find_window_for_pid(
|
||||
pid: i32,
|
||||
adapter: &dyn PlatformAdapter,
|
||||
) -> Result<WindowInfo, AppError> {
|
||||
let filter = WindowFilter {
|
||||
focused_only: false,
|
||||
app: None,
|
||||
};
|
||||
let windows = adapter.list_windows(&filter)?;
|
||||
let mut candidates: Vec<_> = windows.into_iter().filter(|w| w.pid == pid).collect();
|
||||
if candidates.is_empty() {
|
||||
return Err(AppError::invalid_input(
|
||||
"No window found for this application",
|
||||
));
|
||||
}
|
||||
let selected = candidates
|
||||
.iter()
|
||||
.position(|w| w.is_focused)
|
||||
.unwrap_or_default();
|
||||
Ok(candidates.swap_remove(selected))
|
||||
}
|
||||
|
|
@ -19,6 +19,7 @@ include = [
|
|||
"AdModifier",
|
||||
"AdMouseButton",
|
||||
"AdMouseEventKind",
|
||||
"AdPolicyKind",
|
||||
"AdScreenshotKind",
|
||||
"AdSnapshotSurface",
|
||||
"AdWindowOpKind",
|
||||
|
|
|
|||
|
|
@ -67,6 +67,13 @@ enum AdMouseEventKind {
|
|||
};
|
||||
typedef int32_t AdMouseEventKind;
|
||||
|
||||
enum AdPolicyKind {
|
||||
AD_POLICY_KIND_HEADLESS = 0,
|
||||
AD_POLICY_KIND_FOCUS_FALLBACK = 1,
|
||||
AD_POLICY_KIND_PHYSICAL = 2,
|
||||
};
|
||||
typedef int32_t AdPolicyKind;
|
||||
|
||||
enum AdResult {
|
||||
AD_RESULT_OK = 0,
|
||||
AD_RESULT_ERR_PERM_DENIED = -1,
|
||||
|
|
@ -81,6 +88,8 @@ enum AdResult {
|
|||
AD_RESULT_ERR_INVALID_ARGS = -10,
|
||||
AD_RESULT_ERR_NOTIFICATION_NOT_FOUND = -11,
|
||||
AD_RESULT_ERR_INTERNAL = -12,
|
||||
AD_RESULT_ERR_SNAPSHOT_NOT_FOUND = -13,
|
||||
AD_RESULT_ERR_POLICY_DENIED = -14,
|
||||
};
|
||||
typedef int32_t AdResult;
|
||||
|
||||
|
|
@ -375,6 +384,20 @@ AdResult ad_execute_action(const struct AdAdapter *adapter,
|
|||
const struct AdAction *action,
|
||||
struct AdActionResult *out);
|
||||
|
||||
/**
|
||||
* # Safety
|
||||
*
|
||||
* `adapter` must be a non-null pointer returned by `ad_adapter_create`.
|
||||
* `handle` must be a non-null pointer to a valid `AdNativeHandle`.
|
||||
* `action` must be a non-null pointer to a valid `AdAction`.
|
||||
* `out` must be a non-null pointer to an `AdActionResult` to write the result into.
|
||||
*/
|
||||
AdResult ad_execute_action_with_policy(const struct AdAdapter *adapter,
|
||||
const struct AdNativeHandle *handle,
|
||||
const struct AdAction *action,
|
||||
int32_t policy,
|
||||
struct AdActionResult *out);
|
||||
|
||||
/**
|
||||
* Releases a handle previously returned by `ad_resolve_element` and
|
||||
* zeroes the caller's struct so accidentally calling this twice is
|
||||
|
|
|
|||
|
|
@ -2,9 +2,9 @@ use crate::actions::conversion::action_from_c;
|
|||
use crate::actions::result::action_result_to_c;
|
||||
use crate::error::{self, AdResult};
|
||||
use crate::ffi_try::trap_panic;
|
||||
use crate::types::{AdAction, AdActionResult, AdNativeHandle};
|
||||
use crate::types::{AdAction, AdActionResult, AdNativeHandle, AdPolicyKind};
|
||||
use crate::AdAdapter;
|
||||
use agent_desktop_core::adapter::NativeHandle;
|
||||
use agent_desktop_core::{action::ActionRequest, adapter::NativeHandle};
|
||||
|
||||
/// # Safety
|
||||
///
|
||||
|
|
@ -18,6 +18,25 @@ pub unsafe extern "C" fn ad_execute_action(
|
|||
handle: *const AdNativeHandle,
|
||||
action: *const AdAction,
|
||||
out: *mut AdActionResult,
|
||||
) -> AdResult {
|
||||
unsafe {
|
||||
ad_execute_action_with_policy(adapter, handle, action, AdPolicyKind::Headless as i32, out)
|
||||
}
|
||||
}
|
||||
|
||||
/// # Safety
|
||||
///
|
||||
/// `adapter` must be a non-null pointer returned by `ad_adapter_create`.
|
||||
/// `handle` must be a non-null pointer to a valid `AdNativeHandle`.
|
||||
/// `action` must be a non-null pointer to a valid `AdAction`.
|
||||
/// `out` must be a non-null pointer to an `AdActionResult` to write the result into.
|
||||
#[no_mangle]
|
||||
pub unsafe extern "C" fn ad_execute_action_with_policy(
|
||||
adapter: *const AdAdapter,
|
||||
handle: *const AdNativeHandle,
|
||||
action: *const AdAction,
|
||||
policy: i32,
|
||||
out: *mut AdActionResult,
|
||||
) -> AdResult {
|
||||
trap_panic(|| unsafe {
|
||||
crate::pointer_guard::guard_non_null!(out, c"out is null");
|
||||
|
|
@ -49,7 +68,15 @@ pub unsafe extern "C" fn ad_execute_action(
|
|||
}
|
||||
};
|
||||
let native_handle = NativeHandle::from_ptr(handle_ref.ptr);
|
||||
match adapter.inner.execute_action(&native_handle, core_action) {
|
||||
let Some(policy) = AdPolicyKind::from_c(policy) else {
|
||||
error::set_last_error(&agent_desktop_core::error::AdapterError::new(
|
||||
agent_desktop_core::error::ErrorCode::InvalidArgs,
|
||||
"invalid policy kind discriminant",
|
||||
));
|
||||
return AdResult::ErrInvalidArgs;
|
||||
};
|
||||
let request = action_request(policy, core_action);
|
||||
match adapter.inner.execute_action(&native_handle, request) {
|
||||
Ok(result) => {
|
||||
*out = action_result_to_c(&result);
|
||||
AdResult::Ok
|
||||
|
|
@ -61,3 +88,14 @@ pub unsafe extern "C" fn ad_execute_action(
|
|||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn action_request(
|
||||
policy: AdPolicyKind,
|
||||
action: agent_desktop_core::action::Action,
|
||||
) -> ActionRequest {
|
||||
match policy {
|
||||
AdPolicyKind::Headless => ActionRequest::headless(action),
|
||||
AdPolicyKind::FocusFallback => ActionRequest::focus_fallback(action),
|
||||
AdPolicyKind::Physical => ActionRequest::physical(action),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -66,7 +66,9 @@ pub unsafe extern "C" fn ad_resolve_element(
|
|||
bounds_hash,
|
||||
available_actions: vec![],
|
||||
source_app: None,
|
||||
source_window_title: None,
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
};
|
||||
match adapter.inner.resolve_element(&core_entry) {
|
||||
Ok(handle) => {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
use crate::error::{self, AdResult};
|
||||
use crate::ffi_try::{trap_panic, trap_panic_ptr, trap_panic_void};
|
||||
use agent_desktop_core::adapter::PlatformAdapter;
|
||||
use agent_desktop_core::{adapter::PlatformAdapter, PermissionState};
|
||||
|
||||
pub struct AdAdapter {
|
||||
pub(crate) inner: Box<dyn PlatformAdapter>,
|
||||
|
|
@ -65,9 +65,9 @@ pub unsafe extern "C" fn ad_check_permissions(adapter: *const AdAdapter) -> AdRe
|
|||
trap_panic(|| {
|
||||
crate::pointer_guard::guard_non_null!(adapter, c"adapter is null");
|
||||
let adapter = unsafe { &*adapter };
|
||||
match adapter.inner.check_permissions() {
|
||||
agent_desktop_core::adapter::PermissionStatus::Granted => AdResult::Ok,
|
||||
agent_desktop_core::adapter::PermissionStatus::Denied { suggestion } => {
|
||||
match adapter.inner.permission_report().accessibility {
|
||||
PermissionState::Granted => AdResult::Ok,
|
||||
PermissionState::Denied { suggestion } => {
|
||||
error::set_last_error(
|
||||
&agent_desktop_core::error::AdapterError::new(
|
||||
agent_desktop_core::error::ErrorCode::PermDenied,
|
||||
|
|
@ -77,10 +77,32 @@ pub unsafe extern "C" fn ad_check_permissions(adapter: *const AdAdapter) -> AdRe
|
|||
);
|
||||
AdResult::ErrPermDenied
|
||||
}
|
||||
PermissionState::NotRequired => AdResult::Ok,
|
||||
PermissionState::Unknown => unknown_permission_result(adapter.inner.as_ref()),
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn unknown_permission_result(adapter: &dyn PlatformAdapter) -> AdResult {
|
||||
let (code, message, suggestion) = if adapter.unknown_accessibility_means_unsupported() {
|
||||
(
|
||||
agent_desktop_core::error::ErrorCode::PlatformNotSupported,
|
||||
"Accessibility permission state is unknown because this platform adapter does not support permission probing",
|
||||
"Use a platform adapter with implemented permission probing before executing desktop actions.",
|
||||
)
|
||||
} else {
|
||||
(
|
||||
agent_desktop_core::error::ErrorCode::Internal,
|
||||
"Accessibility permission state is unknown",
|
||||
"Run the platform-specific permission report before executing desktop actions.",
|
||||
)
|
||||
};
|
||||
let err =
|
||||
agent_desktop_core::error::AdapterError::new(code, message).with_suggestion(suggestion);
|
||||
error::set_last_error(&err);
|
||||
crate::error::last_error_code()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
|
@ -96,4 +118,54 @@ mod tests {
|
|||
fn test_destroy_null_is_noop() {
|
||||
unsafe { ad_adapter_destroy(std::ptr::null_mut()) };
|
||||
}
|
||||
|
||||
struct UnknownPermissionAdapter;
|
||||
|
||||
impl PlatformAdapter for UnknownPermissionAdapter {
|
||||
fn permission_report(&self) -> agent_desktop_core::PermissionReport {
|
||||
agent_desktop_core::PermissionReport {
|
||||
accessibility: PermissionState::Unknown,
|
||||
screen_recording: PermissionState::Unknown,
|
||||
automation: PermissionState::NotRequired,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn check_permissions_maps_default_unknown_accessibility_to_platform_unsupported() {
|
||||
let adapter = AdAdapter {
|
||||
inner: Box::new(UnknownPermissionAdapter),
|
||||
};
|
||||
|
||||
let result = unsafe { ad_check_permissions(&adapter) };
|
||||
|
||||
assert_eq!(result, AdResult::ErrPlatformNotSupported);
|
||||
}
|
||||
|
||||
struct AmbiguousPermissionAdapter;
|
||||
|
||||
impl PlatformAdapter for AmbiguousPermissionAdapter {
|
||||
fn permission_report(&self) -> agent_desktop_core::PermissionReport {
|
||||
agent_desktop_core::PermissionReport {
|
||||
accessibility: PermissionState::Unknown,
|
||||
screen_recording: PermissionState::Unknown,
|
||||
automation: PermissionState::NotRequired,
|
||||
}
|
||||
}
|
||||
|
||||
fn unknown_accessibility_means_unsupported(&self) -> bool {
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn check_permissions_preserves_ambiguous_unknown_accessibility_as_internal() {
|
||||
let adapter = AdAdapter {
|
||||
inner: Box::new(AmbiguousPermissionAdapter),
|
||||
};
|
||||
|
||||
let result = unsafe { ad_check_permissions(&adapter) };
|
||||
|
||||
assert_eq!(result, AdResult::ErrInternal);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
|
||||
use crate::types::{
|
||||
AdActionKind, AdDirection, AdImageFormat, AdModifier, AdMouseButton, AdMouseEventKind,
|
||||
AdScreenshotKind, AdSnapshotSurface, AdWindowOpKind,
|
||||
AdPolicyKind, AdScreenshotKind, AdSnapshotSurface, AdWindowOpKind,
|
||||
};
|
||||
|
||||
macro_rules! try_from_c_enum {
|
||||
|
|
@ -37,6 +37,12 @@ try_from_c_enum! {
|
|||
}
|
||||
}
|
||||
|
||||
try_from_c_enum! {
|
||||
AdPolicyKind {
|
||||
Headless = 0, FocusFallback = 1, Physical = 2,
|
||||
}
|
||||
}
|
||||
|
||||
try_from_c_enum! {
|
||||
AdDirection {
|
||||
Up = 0, Down = 1, Left = 2, Right = 3,
|
||||
|
|
@ -114,6 +120,14 @@ mod tests {
|
|||
assert!(AdDirection::from_c(4).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_policy_kind_valid_range() {
|
||||
for raw in 0..=2 {
|
||||
assert!(AdPolicyKind::from_c(raw).is_some());
|
||||
}
|
||||
assert!(AdPolicyKind::from_c(3).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_modifier_valid_range() {
|
||||
for raw in 0..=3 {
|
||||
|
|
@ -170,6 +184,7 @@ mod tests {
|
|||
for raw in [-1_000_000_i32, -1, 0, 1, 20, 21, 999, i32::MAX, i32::MIN] {
|
||||
let _ = AdActionKind::from_c(raw);
|
||||
let _ = AdDirection::from_c(raw);
|
||||
let _ = AdPolicyKind::from_c(raw);
|
||||
let _ = AdModifier::from_c(raw);
|
||||
let _ = AdMouseButton::from_c(raw);
|
||||
let _ = AdMouseEventKind::from_c(raw);
|
||||
|
|
|
|||
|
|
@ -16,6 +16,8 @@ const fn error_code_variant_count() -> usize {
|
|||
ErrorCode::Timeout,
|
||||
ErrorCode::InvalidArgs,
|
||||
ErrorCode::NotificationNotFound,
|
||||
ErrorCode::SnapshotNotFound,
|
||||
ErrorCode::PolicyDenied,
|
||||
ErrorCode::Internal,
|
||||
];
|
||||
let mut i = 0;
|
||||
|
|
@ -40,6 +42,8 @@ const fn ad_result_error_variant_count() -> usize {
|
|||
AdResult::ErrInvalidArgs,
|
||||
AdResult::ErrNotificationNotFound,
|
||||
AdResult::ErrInternal,
|
||||
AdResult::ErrSnapshotNotFound,
|
||||
AdResult::ErrPolicyDenied,
|
||||
];
|
||||
let mut count = 0;
|
||||
let mut i = 0;
|
||||
|
|
@ -50,10 +54,6 @@ const fn ad_result_error_variant_count() -> usize {
|
|||
count
|
||||
}
|
||||
|
||||
// Compile-time parity check: every core `ErrorCode` variant must have a
|
||||
// matching `AdResult::Err*`. Adding a variant to either enum without
|
||||
// updating the other fails the build with the message below — preferable
|
||||
// to the silent-drop we'd otherwise see at the FFI boundary.
|
||||
const _: () = assert!(
|
||||
error_code_variant_count() == ad_result_error_variant_count(),
|
||||
"ErrorCode variants must match AdResult error-code variants one-to-one"
|
||||
|
|
@ -75,8 +75,25 @@ pub enum AdResult {
|
|||
ErrInvalidArgs = -10,
|
||||
ErrNotificationNotFound = -11,
|
||||
ErrInternal = -12,
|
||||
ErrSnapshotNotFound = -13,
|
||||
ErrPolicyDenied = -14,
|
||||
}
|
||||
|
||||
const _: () = assert!(AdResult::ErrPermDenied as i32 == -1);
|
||||
const _: () = assert!(AdResult::ErrElementNotFound as i32 == -2);
|
||||
const _: () = assert!(AdResult::ErrAppNotFound as i32 == -3);
|
||||
const _: () = assert!(AdResult::ErrActionFailed as i32 == -4);
|
||||
const _: () = assert!(AdResult::ErrActionNotSupported as i32 == -5);
|
||||
const _: () = assert!(AdResult::ErrStaleRef as i32 == -6);
|
||||
const _: () = assert!(AdResult::ErrWindowNotFound as i32 == -7);
|
||||
const _: () = assert!(AdResult::ErrPlatformNotSupported as i32 == -8);
|
||||
const _: () = assert!(AdResult::ErrTimeout as i32 == -9);
|
||||
const _: () = assert!(AdResult::ErrInvalidArgs as i32 == -10);
|
||||
const _: () = assert!(AdResult::ErrNotificationNotFound as i32 == -11);
|
||||
const _: () = assert!(AdResult::ErrInternal as i32 == -12);
|
||||
const _: () = assert!(AdResult::ErrSnapshotNotFound as i32 == -13);
|
||||
const _: () = assert!(AdResult::ErrPolicyDenied as i32 == -14);
|
||||
|
||||
enum MessageSource {
|
||||
Owned(CString),
|
||||
Static(&'static CStr),
|
||||
|
|
@ -126,6 +143,8 @@ fn error_code_to_result(code: &ErrorCode) -> AdResult {
|
|||
ErrorCode::InvalidArgs => AdResult::ErrInvalidArgs,
|
||||
ErrorCode::NotificationNotFound => AdResult::ErrNotificationNotFound,
|
||||
ErrorCode::Internal => AdResult::ErrInternal,
|
||||
ErrorCode::SnapshotNotFound => AdResult::ErrSnapshotNotFound,
|
||||
ErrorCode::PolicyDenied => AdResult::ErrPolicyDenied,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -249,76 +268,5 @@ pub extern "C" fn ad_last_error_platform_detail() -> *const c_char {
|
|||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn last_error_message_str() -> Option<String> {
|
||||
LAST_ERROR.with(|cell| cell.borrow().as_ref().map(|e| e.message.to_owned_string()))
|
||||
}
|
||||
|
||||
fn last_error_suggestion_str() -> Option<String> {
|
||||
LAST_ERROR.with(|cell| {
|
||||
cell.borrow().as_ref().and_then(|e| {
|
||||
e.suggestion
|
||||
.as_ref()
|
||||
.map(|s| s.to_string_lossy().into_owned())
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
fn last_error_platform_detail_str() -> Option<String> {
|
||||
LAST_ERROR.with(|cell| {
|
||||
cell.borrow().as_ref().and_then(|e| {
|
||||
e.platform_detail
|
||||
.as_ref()
|
||||
.map(|s| s.to_string_lossy().into_owned())
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_no_error_initially() {
|
||||
clear_last_error();
|
||||
assert!(last_error_message_str().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_set_and_get_error() {
|
||||
let err = AdapterError::new(ErrorCode::ElementNotFound, "element @e5 gone")
|
||||
.with_suggestion("run snapshot");
|
||||
set_last_error(&err);
|
||||
assert_eq!(last_error_code(), AdResult::ErrElementNotFound);
|
||||
assert_eq!(last_error_message_str().unwrap(), "element @e5 gone");
|
||||
assert_eq!(last_error_suggestion_str().unwrap(), "run snapshot");
|
||||
assert!(last_error_platform_detail_str().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_clear_error() {
|
||||
let err = AdapterError::internal("oops");
|
||||
set_last_error(&err);
|
||||
clear_last_error();
|
||||
assert!(last_error_message_str().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_error_isolation_across_threads() {
|
||||
clear_last_error();
|
||||
let err = AdapterError::internal("thread1");
|
||||
set_last_error(&err);
|
||||
|
||||
let handle = std::thread::spawn(|| last_error_message_str().is_none());
|
||||
assert!(handle.join().unwrap(), "other thread should see no error");
|
||||
assert_eq!(last_error_message_str().unwrap(), "thread1");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_interior_nul_falls_back_to_static() {
|
||||
let err = AdapterError::new(ErrorCode::Internal, "before\0after");
|
||||
set_last_error(&err);
|
||||
assert_eq!(
|
||||
last_error_message_str().unwrap(),
|
||||
"(message contained null byte)"
|
||||
);
|
||||
}
|
||||
}
|
||||
#[path = "error_tests.rs"]
|
||||
mod tests;
|
||||
|
|
|
|||
78
crates/ffi/src/error_tests.rs
Normal file
78
crates/ffi/src/error_tests.rs
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
use super::*;
|
||||
|
||||
fn last_error_message_str() -> Option<String> {
|
||||
LAST_ERROR.with(|cell| cell.borrow().as_ref().map(|e| e.message.to_owned_string()))
|
||||
}
|
||||
|
||||
fn last_error_suggestion_str() -> Option<String> {
|
||||
LAST_ERROR.with(|cell| {
|
||||
cell.borrow().as_ref().and_then(|e| {
|
||||
e.suggestion
|
||||
.as_ref()
|
||||
.map(|s| s.to_string_lossy().into_owned())
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
fn last_error_platform_detail_str() -> Option<String> {
|
||||
LAST_ERROR.with(|cell| {
|
||||
cell.borrow().as_ref().and_then(|e| {
|
||||
e.platform_detail
|
||||
.as_ref()
|
||||
.map(|s| s.to_string_lossy().into_owned())
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_no_error_initially() {
|
||||
clear_last_error();
|
||||
assert!(last_error_message_str().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn result_discriminants_preserve_existing_abi_values() {
|
||||
assert_eq!(AdResult::ErrInternal as i32, -12);
|
||||
assert_eq!(AdResult::ErrSnapshotNotFound as i32, -13);
|
||||
assert_eq!(AdResult::ErrPolicyDenied as i32, -14);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_set_and_get_error() {
|
||||
let err = AdapterError::new(ErrorCode::ElementNotFound, "element @e5 gone")
|
||||
.with_suggestion("run snapshot");
|
||||
set_last_error(&err);
|
||||
assert_eq!(last_error_code(), AdResult::ErrElementNotFound);
|
||||
assert_eq!(last_error_message_str().unwrap(), "element @e5 gone");
|
||||
assert_eq!(last_error_suggestion_str().unwrap(), "run snapshot");
|
||||
assert!(last_error_platform_detail_str().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_clear_error() {
|
||||
let err = AdapterError::internal("oops");
|
||||
set_last_error(&err);
|
||||
clear_last_error();
|
||||
assert!(last_error_message_str().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_error_isolation_across_threads() {
|
||||
clear_last_error();
|
||||
let err = AdapterError::internal("thread1");
|
||||
set_last_error(&err);
|
||||
|
||||
let handle = std::thread::spawn(|| last_error_message_str().is_none());
|
||||
assert!(handle.join().unwrap(), "other thread should see no error");
|
||||
assert_eq!(last_error_message_str().unwrap(), "thread1");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_interior_nul_falls_back_to_static() {
|
||||
let err = AdapterError::new(ErrorCode::Internal, "before\0after");
|
||||
set_last_error(&err);
|
||||
assert_eq!(
|
||||
last_error_message_str().unwrap(),
|
||||
"(message contained null byte)"
|
||||
);
|
||||
}
|
||||
|
|
@ -84,6 +84,7 @@ pub use types::notification_filter::AdNotificationFilter;
|
|||
pub use types::notification_info::AdNotificationInfo;
|
||||
pub use types::notification_list::AdNotificationList;
|
||||
pub use types::point::AdPoint;
|
||||
pub use types::policy_kind::AdPolicyKind;
|
||||
pub use types::rect::AdRect;
|
||||
pub use types::ref_entry::AdRefEntry;
|
||||
pub use types::screenshot_kind::AdScreenshotKind;
|
||||
|
|
|
|||
|
|
@ -100,7 +100,9 @@ pub unsafe extern "C" fn ad_find(
|
|||
bounds_hash,
|
||||
available_actions: Vec::new(),
|
||||
source_app: None,
|
||||
source_window_title: Some(core_win.title.clone()),
|
||||
root_ref: None,
|
||||
path: Vec::new(),
|
||||
};
|
||||
match adapter.inner.resolve_element(&ref_entry) {
|
||||
Ok(handle) => {
|
||||
|
|
|
|||
|
|
@ -171,6 +171,7 @@ mod tests {
|
|||
description: None,
|
||||
hint: None,
|
||||
states: states.iter().map(|s| s.to_string()).collect(),
|
||||
available_actions: vec![],
|
||||
bounds: None,
|
||||
children: vec![],
|
||||
children_count: None,
|
||||
|
|
|
|||
|
|
@ -65,6 +65,7 @@ mod tests {
|
|||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
available_actions: vec![],
|
||||
bounds: None,
|
||||
children: vec![],
|
||||
children_count: None,
|
||||
|
|
|
|||
|
|
@ -126,6 +126,7 @@ mod tests {
|
|||
description: None,
|
||||
hint: None,
|
||||
states: vec![],
|
||||
available_actions: vec![],
|
||||
bounds: None,
|
||||
children: vec![],
|
||||
children_count: None,
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ pub mod notification_filter;
|
|||
pub mod notification_info;
|
||||
pub mod notification_list;
|
||||
pub mod point;
|
||||
pub mod policy_kind;
|
||||
pub mod rect;
|
||||
pub mod ref_entry;
|
||||
pub mod screenshot_kind;
|
||||
|
|
@ -58,6 +59,7 @@ pub use notification_filter::AdNotificationFilter;
|
|||
pub use notification_info::AdNotificationInfo;
|
||||
pub use notification_list::AdNotificationList;
|
||||
pub use point::AdPoint;
|
||||
pub use policy_kind::AdPolicyKind;
|
||||
pub use rect::AdRect;
|
||||
pub use ref_entry::AdRefEntry;
|
||||
pub use screenshot_kind::AdScreenshotKind;
|
||||
|
|
|
|||
19
crates/ffi/src/types/policy_kind.rs
Normal file
19
crates/ffi/src/types/policy_kind.rs
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
#[repr(i32)]
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum AdPolicyKind {
|
||||
Headless = 0,
|
||||
FocusFallback = 1,
|
||||
Physical = 2,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn discriminants_are_abi_stable() {
|
||||
assert_eq!(AdPolicyKind::Headless as i32, 0);
|
||||
assert_eq!(AdPolicyKind::FocusFallback as i32, 1);
|
||||
assert_eq!(AdPolicyKind::Physical as i32, 2);
|
||||
}
|
||||
}
|
||||
62
crates/ffi/tests/c_abi_actions.rs
Normal file
62
crates/ffi/tests/c_abi_actions.rs
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
mod common;
|
||||
|
||||
use common::{
|
||||
ad_execute_action, ad_execute_action_with_policy, default_action, with_adapter, AdActionResult,
|
||||
AdNativeHandle, AdPolicyKind, AdResult,
|
||||
};
|
||||
|
||||
#[test]
|
||||
fn enum_fuzz_invalid_discriminant_rejected() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let mut action = default_action();
|
||||
action.kind = i32::MAX;
|
||||
let handle = AdNativeHandle {
|
||||
ptr: std::ptr::null(),
|
||||
};
|
||||
let mut out: AdActionResult = std::mem::zeroed();
|
||||
let rc = ad_execute_action(adapter, &handle, &action, &mut out);
|
||||
assert!(
|
||||
matches!(rc, AdResult::ErrInvalidArgs | AdResult::ErrInternal),
|
||||
"arbitrary enum bit pattern must be rejected, got {:?}",
|
||||
rc
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn invalid_policy_discriminant_rejected_without_ub() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let action = default_action();
|
||||
let handle = AdNativeHandle {
|
||||
ptr: std::ptr::dangling::<std::ffi::c_void>(),
|
||||
};
|
||||
let mut out: AdActionResult = std::mem::zeroed();
|
||||
let rc = ad_execute_action_with_policy(
|
||||
adapter,
|
||||
&handle,
|
||||
&action,
|
||||
AdPolicyKind::Physical as i32 + 1,
|
||||
&mut out,
|
||||
);
|
||||
assert!(matches!(
|
||||
rc,
|
||||
AdResult::ErrInvalidArgs | AdResult::ErrInternal
|
||||
));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn execute_action_rejects_null_handle_ptr() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let action = default_action();
|
||||
let handle = AdNativeHandle {
|
||||
ptr: std::ptr::null(),
|
||||
};
|
||||
let mut out: AdActionResult = std::mem::zeroed();
|
||||
let rc = ad_execute_action(adapter, &handle, &action, &mut out);
|
||||
assert!(matches!(
|
||||
rc,
|
||||
AdResult::ErrInvalidArgs | AdResult::ErrInternal
|
||||
));
|
||||
});
|
||||
}
|
||||
|
|
@ -1,468 +0,0 @@
|
|||
//! C-ABI contract harness.
|
||||
//!
|
||||
//! Drives the FFI exactly as a C program would — raw `extern "C"`
|
||||
//! declarations with `#[allow(improper_ctypes)]` over the opaque
|
||||
//! handle types, no high-level Rust access. Exercises bug classes the
|
||||
//! inline `#[cfg(test)]` modules can't reach on their own:
|
||||
//!
|
||||
//! 1. Struct layouts that a consumer memcpy-copies (AdRect, AdAction).
|
||||
//! 2. Enum fuzzing — `int32_t` bit patterns written into enum-typed
|
||||
//! fields must not UB, must return `AD_RESULT_ERR_INVALID_ARGS`.
|
||||
//! 3. Null tolerance in the free-* family and accessor family.
|
||||
//! 4. Interior-NUL inputs funnel through `string_to_c_lossy` without
|
||||
//! returning null.
|
||||
//! 5. List handle lifecycle (count on null, _free on null, _get OOB).
|
||||
|
||||
#![allow(improper_ctypes)]
|
||||
|
||||
use agent_desktop_ffi::error::AdResult;
|
||||
use agent_desktop_ffi::{
|
||||
AdAction, AdActionResult, AdAdapter, AdAppList, AdDirection, AdDragParams, AdFindQuery,
|
||||
AdKeyCombo, AdNativeHandle, AdPoint, AdRect, AdRefEntry, AdScrollParams, AdWindowInfo,
|
||||
AdWindowList,
|
||||
};
|
||||
use std::ffi::CStr;
|
||||
use std::os::raw::c_char;
|
||||
|
||||
extern "C" {
|
||||
fn ad_adapter_create() -> *mut AdAdapter;
|
||||
fn ad_adapter_destroy(adapter: *mut AdAdapter);
|
||||
fn ad_check_permissions(adapter: *const AdAdapter) -> AdResult;
|
||||
|
||||
fn ad_last_error_code() -> AdResult;
|
||||
fn ad_last_error_message() -> *const c_char;
|
||||
|
||||
fn ad_list_apps(adapter: *const AdAdapter, out: *mut *mut AdAppList) -> AdResult;
|
||||
fn ad_app_list_count(list: *const AdAppList) -> u32;
|
||||
fn ad_app_list_get(list: *const AdAppList, index: u32) -> *const u8;
|
||||
fn ad_app_list_free(list: *mut AdAppList);
|
||||
|
||||
fn ad_list_windows(
|
||||
adapter: *const AdAdapter,
|
||||
app_filter: *const c_char,
|
||||
focused_only: bool,
|
||||
out: *mut *mut AdWindowList,
|
||||
) -> AdResult;
|
||||
fn ad_window_list_count(list: *const AdWindowList) -> u32;
|
||||
fn ad_window_list_free(list: *mut AdWindowList);
|
||||
|
||||
fn ad_launch_app(
|
||||
adapter: *const AdAdapter,
|
||||
id: *const c_char,
|
||||
timeout_ms: u64,
|
||||
out: *mut AdWindowInfo,
|
||||
) -> AdResult;
|
||||
|
||||
fn ad_execute_action(
|
||||
adapter: *const AdAdapter,
|
||||
handle: *const AdNativeHandle,
|
||||
action: *const AdAction,
|
||||
out: *mut AdActionResult,
|
||||
) -> AdResult;
|
||||
|
||||
fn ad_find(
|
||||
adapter: *const AdAdapter,
|
||||
win: *const AdWindowInfo,
|
||||
query: *const AdFindQuery,
|
||||
out: *mut AdNativeHandle,
|
||||
) -> AdResult;
|
||||
|
||||
fn ad_free_handle(adapter: *const AdAdapter, handle: *mut AdNativeHandle) -> AdResult;
|
||||
|
||||
fn ad_resolve_element(
|
||||
adapter: *const AdAdapter,
|
||||
entry: *const AdRefEntry,
|
||||
out: *mut AdNativeHandle,
|
||||
) -> AdResult;
|
||||
}
|
||||
|
||||
fn with_adapter<F: FnOnce(*mut AdAdapter)>(body: F) {
|
||||
unsafe {
|
||||
let adapter = ad_adapter_create();
|
||||
assert!(!adapter.is_null(), "ad_adapter_create must not return null");
|
||||
body(adapter);
|
||||
ad_adapter_destroy(adapter);
|
||||
}
|
||||
}
|
||||
|
||||
fn default_scroll() -> AdScrollParams {
|
||||
AdScrollParams {
|
||||
direction: AdDirection::Down as i32,
|
||||
amount: 0,
|
||||
}
|
||||
}
|
||||
|
||||
fn default_key() -> AdKeyCombo {
|
||||
AdKeyCombo {
|
||||
key: std::ptr::null(),
|
||||
modifiers: std::ptr::null(),
|
||||
modifier_count: 0,
|
||||
}
|
||||
}
|
||||
|
||||
fn default_drag() -> AdDragParams {
|
||||
AdDragParams {
|
||||
from: AdPoint { x: 0.0, y: 0.0 },
|
||||
to: AdPoint { x: 0.0, y: 0.0 },
|
||||
duration_ms: 0,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rect_and_point_layouts_are_memcpyable() {
|
||||
// A C consumer that does { AdRect r; memcpy(&r, src, sizeof(r)); } must
|
||||
// see the same field values Rust wrote. Plain #[repr(C)] without
|
||||
// padding games makes this a byte copy.
|
||||
let rect = AdRect {
|
||||
x: 1.25,
|
||||
y: -2.5,
|
||||
width: 640.0,
|
||||
height: 480.0,
|
||||
};
|
||||
let copied = unsafe { std::ptr::read(&rect as *const AdRect) };
|
||||
assert_eq!(copied.x, 1.25);
|
||||
assert_eq!(copied.y, -2.5);
|
||||
assert_eq!(copied.width, 640.0);
|
||||
assert_eq!(copied.height, 480.0);
|
||||
|
||||
let point = AdPoint { x: 3.0, y: 4.0 };
|
||||
let copied = unsafe { std::ptr::read(&point as *const AdPoint) };
|
||||
assert_eq!(copied.x, 3.0);
|
||||
assert_eq!(copied.y, 4.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn enum_fuzz_invalid_discriminant_rejected() {
|
||||
// AdAction.kind is `i32` — a buggy C caller can legally stuff any
|
||||
// value in here. Must not UB the Rust side; the validator should
|
||||
// surface AD_RESULT_ERR_INVALID_ARGS before any adapter code runs.
|
||||
with_adapter(|adapter| unsafe {
|
||||
let mut action: AdAction = std::mem::zeroed();
|
||||
action.kind = i32::MAX;
|
||||
action.scroll = default_scroll();
|
||||
action.key = default_key();
|
||||
action.drag = default_drag();
|
||||
action.text = std::ptr::null();
|
||||
|
||||
let handle = AdNativeHandle {
|
||||
ptr: std::ptr::null(),
|
||||
};
|
||||
let mut out: AdActionResult = std::mem::zeroed();
|
||||
let rc = ad_execute_action(adapter, &handle, &action, &mut out);
|
||||
// Either the enum validator rejects (expected) or the cargo-test
|
||||
// worker thread trips the macOS main-thread assert and returns
|
||||
// ErrInternal. Both prove the absence of UB; what matters is that
|
||||
// the arbitrary bit pattern never results in AD_RESULT_OK.
|
||||
assert!(
|
||||
matches!(rc, AdResult::ErrInvalidArgs | AdResult::ErrInternal),
|
||||
"arbitrary enum bit pattern must be rejected, got {:?}",
|
||||
rc
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn null_adapter_rejected_without_ub() {
|
||||
unsafe {
|
||||
let mut list: *mut AdAppList = std::ptr::null_mut();
|
||||
let rc = ad_list_apps(std::ptr::null(), &mut list);
|
||||
// On cargo-test worker threads the macOS main-thread guard
|
||||
// fires first (ErrInternal); on the main thread the null-adapter
|
||||
// guard wins (ErrInvalidArgs). Either way: no dereference, no UB.
|
||||
assert!(matches!(
|
||||
rc,
|
||||
AdResult::ErrInvalidArgs | AdResult::ErrInternal
|
||||
));
|
||||
assert!(list.is_null(), "out-param must stay null on failure");
|
||||
|
||||
let rc2 = ad_check_permissions(std::ptr::null());
|
||||
// ad_check_permissions has no main-thread guard — null adapter
|
||||
// must hit the null-check and return InvalidArgs deterministically.
|
||||
assert_eq!(rc2, AdResult::ErrInvalidArgs);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dirty_out_param_is_cleared_before_early_return_on_worker_thread() {
|
||||
// Regression for todo 006: prior shape ran require_main_thread()
|
||||
// *before* zeroing *out, so a worker-thread early-return left
|
||||
// whatever the caller had in the struct. A follow-up free on that
|
||||
// stale pointer would double-free. This test seeds the out slot
|
||||
// with fake pointers that *must* be cleared before the fn returns.
|
||||
with_adapter(|adapter| unsafe {
|
||||
let fake_ptr = 0xDEAD_BEEF as *mut AdAppList;
|
||||
let mut list: *mut AdAppList = fake_ptr;
|
||||
let rc = ad_list_apps(adapter, &mut list);
|
||||
// Either the main-thread guard (ErrInternal on worker) or a
|
||||
// successful call zeroed the slot before anything else.
|
||||
if rc != AdResult::Ok {
|
||||
assert!(
|
||||
list.is_null(),
|
||||
"dirty out-param must be zeroed before early return, got {:?}",
|
||||
list
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn invalid_utf8_filter_rejected_not_silently_widened() {
|
||||
// Regression for todo 010: prior c_to_string conflated null with
|
||||
// invalid UTF-8, so a non-null buffer with bogus bytes in the
|
||||
// app_filter slot would be treated as "no filter" and widen
|
||||
// ad_list_windows to every app on the system. Must now fail closed.
|
||||
with_adapter(|adapter| unsafe {
|
||||
let bad: [u8; 2] = [0xC3, 0x00];
|
||||
let mut list: *mut AdWindowList = std::ptr::null_mut();
|
||||
let rc = ad_list_windows(adapter, bad.as_ptr() as *const c_char, false, &mut list);
|
||||
// Main-thread guard (ErrInternal on worker) or UTF-8 rejection
|
||||
// (ErrInvalidArgs) — either way we do NOT produce a list by
|
||||
// silently treating bad bytes as "no filter".
|
||||
assert!(matches!(
|
||||
rc,
|
||||
AdResult::ErrInvalidArgs | AdResult::ErrInternal
|
||||
));
|
||||
assert!(list.is_null());
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn null_out_param_rejected_before_write() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let rc = ad_list_apps(adapter, std::ptr::null_mut());
|
||||
assert!(matches!(
|
||||
rc,
|
||||
AdResult::ErrInvalidArgs | AdResult::ErrInternal
|
||||
));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn null_tolerance_on_list_accessors_and_free() {
|
||||
unsafe {
|
||||
assert_eq!(ad_app_list_count(std::ptr::null()), 0);
|
||||
assert!(ad_app_list_get(std::ptr::null(), 0).is_null());
|
||||
ad_app_list_free(std::ptr::null_mut());
|
||||
|
||||
assert_eq!(ad_window_list_count(std::ptr::null()), 0);
|
||||
ad_window_list_free(std::ptr::null_mut());
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn list_handle_lifecycle_roundtrip() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let mut list: *mut AdAppList = std::ptr::null_mut();
|
||||
let rc = ad_list_apps(adapter, &mut list);
|
||||
// On CI without accessibility permission this may return PermDenied;
|
||||
// both outcomes preserve the contract we're testing (no UB, valid
|
||||
// pointer-or-null).
|
||||
if rc == AdResult::Ok {
|
||||
assert!(!list.is_null());
|
||||
let count = ad_app_list_count(list);
|
||||
// Request an index past the end — must return null, not segfault.
|
||||
assert!(ad_app_list_get(list, count).is_null());
|
||||
ad_app_list_free(list);
|
||||
} else {
|
||||
assert!(list.is_null(), "failed list call must leave out null");
|
||||
let msg_ptr = ad_last_error_message();
|
||||
assert!(!msg_ptr.is_null());
|
||||
let _ = CStr::from_ptr(msg_ptr).to_string_lossy();
|
||||
assert_eq!(ad_last_error_code(), rc);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn list_windows_focused_only_runs() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let mut list: *mut AdWindowList = std::ptr::null_mut();
|
||||
let rc = ad_list_windows(adapter, std::ptr::null(), true, &mut list);
|
||||
if rc == AdResult::Ok {
|
||||
assert!(!list.is_null());
|
||||
let _ = ad_window_list_count(list);
|
||||
ad_window_list_free(list);
|
||||
} else {
|
||||
assert!(list.is_null());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn invalid_utf8_app_id_rejected() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
// A mid-byte of a multi-byte UTF-8 sequence, not valid on its own.
|
||||
let bad: [u8; 2] = [0xC3, 0];
|
||||
let mut out: AdWindowInfo = std::mem::zeroed();
|
||||
let rc = ad_launch_app(adapter, bad.as_ptr() as *const c_char, 0, &mut out);
|
||||
// Either the UTF-8 check (ErrInvalidArgs) or the macOS main-thread
|
||||
// guard (ErrInternal — cargo tests run on worker threads) rejects
|
||||
// the call. Either way no UB occurred.
|
||||
assert!(
|
||||
matches!(rc, AdResult::ErrInvalidArgs | AdResult::ErrInternal),
|
||||
"must reject without UB, got {:?}",
|
||||
rc
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn find_returns_not_found_on_empty_query_against_no_window() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let bad_win: AdWindowInfo = std::mem::zeroed();
|
||||
let query = AdFindQuery {
|
||||
role: std::ptr::null(),
|
||||
name_substring: std::ptr::null(),
|
||||
value_substring: std::ptr::null(),
|
||||
};
|
||||
let mut handle = AdNativeHandle {
|
||||
ptr: std::ptr::null(),
|
||||
};
|
||||
let rc = ad_find(adapter, &bad_win, &query, &mut handle);
|
||||
// A zeroed window has null id/title → InvalidArgs. On cargo-test
|
||||
// worker threads the main-thread assert trips first and returns
|
||||
// ErrInternal. Either outcome means the FFI refused a malformed
|
||||
// input without UB.
|
||||
assert!(
|
||||
matches!(rc, AdResult::ErrInvalidArgs | AdResult::ErrInternal),
|
||||
"zeroed window must not succeed, got {:?}",
|
||||
rc
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn free_handle_null_is_noop() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let mut handle = AdNativeHandle {
|
||||
ptr: std::ptr::null(),
|
||||
};
|
||||
let rc = ad_free_handle(adapter, &mut handle);
|
||||
assert_eq!(rc, AdResult::Ok);
|
||||
assert!(handle.ptr.is_null());
|
||||
|
||||
let rc2 = ad_free_handle(adapter, std::ptr::null_mut());
|
||||
assert_eq!(rc2, AdResult::Ok);
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn free_handle_zeroes_ptr_so_double_free_is_noop() {
|
||||
// macOS is excluded: ad_free_handle invokes CFRelease on the
|
||||
// underlying pointer, and a fabricated "fake live" pointer will
|
||||
// SIGBUS before we can observe the zeroing. On Windows/Linux
|
||||
// release_handle is NotSupported (no platform call), so the zeroing
|
||||
// contract is safely observable with a fake pointer.
|
||||
with_adapter(|adapter| unsafe {
|
||||
let fake_live_ptr = 0x1234 as *const std::ffi::c_void;
|
||||
let mut handle = AdNativeHandle { ptr: fake_live_ptr };
|
||||
|
||||
let _ = ad_free_handle(adapter, &mut handle);
|
||||
assert!(handle.ptr.is_null());
|
||||
|
||||
let rc = ad_free_handle(adapter, &mut handle);
|
||||
assert_eq!(rc, AdResult::Ok);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_element_rejects_invalid_utf8_name() {
|
||||
// Regression for todo 004: prior c_to_string(entry.name) conflated
|
||||
// null with invalid UTF-8, so a non-null buffer with bogus bytes in
|
||||
// the `name` slot was treated as "no name filter" and widened the
|
||||
// re-resolution match. Must now fail closed with InvalidArgs.
|
||||
with_adapter(|adapter| unsafe {
|
||||
let role = std::ffi::CString::new("button").unwrap();
|
||||
let bad_name: [u8; 2] = [0xC3, 0x00]; // partial multi-byte + NUL
|
||||
let entry = AdRefEntry {
|
||||
pid: 0,
|
||||
role: role.as_ptr(),
|
||||
name: bad_name.as_ptr() as *const c_char,
|
||||
bounds_hash: 0,
|
||||
has_bounds_hash: false,
|
||||
};
|
||||
let mut out = AdNativeHandle {
|
||||
ptr: std::ptr::null(),
|
||||
};
|
||||
let rc = ad_resolve_element(adapter, &entry, &mut out);
|
||||
// Either the UTF-8 check wins (ErrInvalidArgs) or the main-thread
|
||||
// guard wins on worker threads (ErrInternal). Either way, no
|
||||
// silent widening, no UB, and the out-handle stays null.
|
||||
assert!(
|
||||
matches!(rc, AdResult::ErrInvalidArgs | AdResult::ErrInternal),
|
||||
"must reject without UB, got {:?}",
|
||||
rc
|
||||
);
|
||||
assert!(out.ptr.is_null());
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn execute_action_rejects_null_handle_ptr() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let action = AdAction {
|
||||
kind: 0,
|
||||
text: std::ptr::null(),
|
||||
scroll: AdScrollParams {
|
||||
direction: 0,
|
||||
amount: 0,
|
||||
},
|
||||
key: AdKeyCombo {
|
||||
key: std::ptr::null(),
|
||||
modifiers: std::ptr::null(),
|
||||
modifier_count: 0,
|
||||
},
|
||||
drag: AdDragParams {
|
||||
from: AdPoint { x: 0.0, y: 0.0 },
|
||||
to: AdPoint { x: 0.0, y: 0.0 },
|
||||
duration_ms: 0,
|
||||
},
|
||||
};
|
||||
let handle = AdNativeHandle {
|
||||
ptr: std::ptr::null(),
|
||||
};
|
||||
let mut out: AdActionResult = std::mem::zeroed();
|
||||
let rc = ad_execute_action(adapter, &handle, &action, &mut out);
|
||||
// Main-thread guard or null-ptr guard wins; both avoid UB and
|
||||
// keep the struct-level handle accepted while rejecting the
|
||||
// inner null pointer.
|
||||
assert!(matches!(
|
||||
rc,
|
||||
AdResult::ErrInvalidArgs | AdResult::ErrInternal
|
||||
));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn last_error_survives_successful_calls() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let mut out: AdWindowInfo = std::mem::zeroed();
|
||||
let rc = ad_launch_app(adapter, std::ptr::null(), 0, &mut out);
|
||||
// Worker-thread cargo tests hit the main-thread guard first,
|
||||
// producing ErrInternal. Main-thread runs would see
|
||||
// ErrInvalidArgs. What we need is that *some* failure populates
|
||||
// last-error and its pointer stays stable across the success
|
||||
// calls that follow.
|
||||
assert!(
|
||||
matches!(rc, AdResult::ErrInvalidArgs | AdResult::ErrInternal),
|
||||
"must fail, got {:?}",
|
||||
rc
|
||||
);
|
||||
let msg_ptr = ad_last_error_message();
|
||||
assert!(!msg_ptr.is_null());
|
||||
|
||||
// Use accessors that are guaranteed to succeed and never set
|
||||
// last-error. ad_check_permissions is NOT safe here — it can
|
||||
// return ErrPermDenied on macOS without Accessibility permission,
|
||||
// which overwrites the TLS error slot we're testing.
|
||||
for _ in 0..5 {
|
||||
let _ = ad_app_list_count(std::ptr::null());
|
||||
let _ = ad_window_list_count(std::ptr::null());
|
||||
}
|
||||
|
||||
let after = ad_last_error_message();
|
||||
assert_eq!(msg_ptr, after);
|
||||
assert_eq!(ad_last_error_code(), rc);
|
||||
});
|
||||
}
|
||||
59
crates/ffi/tests/c_abi_inputs.rs
Normal file
59
crates/ffi/tests/c_abi_inputs.rs
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
mod common;
|
||||
|
||||
use common::{
|
||||
ad_launch_app, ad_list_windows, ad_resolve_element, c_char, with_adapter, AdNativeHandle,
|
||||
AdRefEntry, AdResult, AdWindowInfo, AdWindowList,
|
||||
};
|
||||
|
||||
#[test]
|
||||
fn invalid_utf8_filter_rejected_not_silently_widened() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let bad: [u8; 2] = [0xC3, 0x00];
|
||||
let mut list: *mut AdWindowList = std::ptr::null_mut();
|
||||
let rc = ad_list_windows(adapter, bad.as_ptr() as *const c_char, false, &mut list);
|
||||
assert!(matches!(
|
||||
rc,
|
||||
AdResult::ErrInvalidArgs | AdResult::ErrInternal
|
||||
));
|
||||
assert!(list.is_null());
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn invalid_utf8_app_id_rejected() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let bad: [u8; 2] = [0xC3, 0];
|
||||
let mut out: AdWindowInfo = std::mem::zeroed();
|
||||
let rc = ad_launch_app(adapter, bad.as_ptr() as *const c_char, 0, &mut out);
|
||||
assert!(
|
||||
matches!(rc, AdResult::ErrInvalidArgs | AdResult::ErrInternal),
|
||||
"must reject without UB, got {:?}",
|
||||
rc
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_element_rejects_invalid_utf8_name() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let role = std::ffi::CString::new("button").unwrap();
|
||||
let bad_name: [u8; 2] = [0xC3, 0x00];
|
||||
let entry = AdRefEntry {
|
||||
pid: 0,
|
||||
role: role.as_ptr(),
|
||||
name: bad_name.as_ptr() as *const c_char,
|
||||
bounds_hash: 0,
|
||||
has_bounds_hash: false,
|
||||
};
|
||||
let mut out = AdNativeHandle {
|
||||
ptr: std::ptr::null(),
|
||||
};
|
||||
let rc = ad_resolve_element(adapter, &entry, &mut out);
|
||||
assert!(
|
||||
matches!(rc, AdResult::ErrInvalidArgs | AdResult::ErrInternal),
|
||||
"must reject without UB, got {:?}",
|
||||
rc
|
||||
);
|
||||
assert!(out.ptr.is_null());
|
||||
});
|
||||
}
|
||||
23
crates/ffi/tests/c_abi_layout.rs
Normal file
23
crates/ffi/tests/c_abi_layout.rs
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
mod common;
|
||||
|
||||
use common::{AdPoint, AdRect};
|
||||
|
||||
#[test]
|
||||
fn rect_and_point_layouts_are_memcpyable() {
|
||||
let rect = AdRect {
|
||||
x: 1.25,
|
||||
y: -2.5,
|
||||
width: 640.0,
|
||||
height: 480.0,
|
||||
};
|
||||
let copied = unsafe { std::ptr::read(&rect as *const AdRect) };
|
||||
assert_eq!(copied.x, 1.25);
|
||||
assert_eq!(copied.y, -2.5);
|
||||
assert_eq!(copied.width, 640.0);
|
||||
assert_eq!(copied.height, 480.0);
|
||||
|
||||
let point = AdPoint { x: 3.0, y: 4.0 };
|
||||
let copied = unsafe { std::ptr::read(&point as *const AdPoint) };
|
||||
assert_eq!(copied.x, 3.0);
|
||||
assert_eq!(copied.y, 4.0);
|
||||
}
|
||||
176
crates/ffi/tests/c_abi_lifecycle.rs
Normal file
176
crates/ffi/tests/c_abi_lifecycle.rs
Normal file
|
|
@ -0,0 +1,176 @@
|
|||
mod common;
|
||||
|
||||
use common::{
|
||||
ad_adapter_create, ad_adapter_destroy, ad_app_list_count, ad_app_list_free, ad_app_list_get,
|
||||
ad_check_permissions, ad_find, ad_free_handle, ad_last_error_code, ad_last_error_message,
|
||||
ad_list_apps, ad_list_windows, ad_window_list_count, ad_window_list_free, with_adapter,
|
||||
AdAppList, AdFindQuery, AdNativeHandle, AdResult, AdWindowInfo, AdWindowList, CStr,
|
||||
};
|
||||
|
||||
#[test]
|
||||
fn null_adapter_rejected_without_ub() {
|
||||
unsafe {
|
||||
let mut list: *mut AdAppList = std::ptr::null_mut();
|
||||
let rc = ad_list_apps(std::ptr::null(), &mut list);
|
||||
assert!(matches!(
|
||||
rc,
|
||||
AdResult::ErrInvalidArgs | AdResult::ErrInternal
|
||||
));
|
||||
assert!(list.is_null(), "out-param must stay null on failure");
|
||||
|
||||
let rc2 = ad_check_permissions(std::ptr::null());
|
||||
assert_eq!(rc2, AdResult::ErrInvalidArgs);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn null_out_param_rejected_before_write() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let rc = ad_list_apps(adapter, std::ptr::null_mut());
|
||||
assert!(matches!(
|
||||
rc,
|
||||
AdResult::ErrInvalidArgs | AdResult::ErrInternal
|
||||
));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn null_tolerance_on_list_accessors_and_free() {
|
||||
unsafe {
|
||||
assert_eq!(ad_app_list_count(std::ptr::null()), 0);
|
||||
assert!(ad_app_list_get(std::ptr::null(), 0).is_null());
|
||||
ad_app_list_free(std::ptr::null_mut());
|
||||
|
||||
assert_eq!(ad_window_list_count(std::ptr::null()), 0);
|
||||
ad_window_list_free(std::ptr::null_mut());
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dirty_out_param_is_cleared_before_early_return_on_worker_thread() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let fake_ptr = 0xDEAD_BEEF as *mut AdAppList;
|
||||
let mut list: *mut AdAppList = fake_ptr;
|
||||
let rc = ad_list_apps(adapter, &mut list);
|
||||
if rc != AdResult::Ok {
|
||||
assert!(
|
||||
list.is_null(),
|
||||
"dirty out-param must be zeroed before early return, got {:?}",
|
||||
list
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn list_handle_lifecycle_roundtrip() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let mut list: *mut AdAppList = std::ptr::null_mut();
|
||||
let rc = ad_list_apps(adapter, &mut list);
|
||||
if rc == AdResult::Ok {
|
||||
assert!(!list.is_null());
|
||||
let count = ad_app_list_count(list);
|
||||
assert!(ad_app_list_get(list, count).is_null());
|
||||
ad_app_list_free(list);
|
||||
} else {
|
||||
assert!(list.is_null(), "failed list call must leave out null");
|
||||
let msg_ptr = ad_last_error_message();
|
||||
assert!(!msg_ptr.is_null());
|
||||
let _ = CStr::from_ptr(msg_ptr).to_string_lossy();
|
||||
assert_eq!(ad_last_error_code(), rc);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn list_windows_focused_only_runs() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let mut list: *mut AdWindowList = std::ptr::null_mut();
|
||||
let rc = ad_list_windows(adapter, std::ptr::null(), true, &mut list);
|
||||
if rc == AdResult::Ok {
|
||||
assert!(!list.is_null());
|
||||
let _ = ad_window_list_count(list);
|
||||
ad_window_list_free(list);
|
||||
} else {
|
||||
assert!(list.is_null());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn find_returns_not_found_on_empty_query_against_no_window() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let bad_win: AdWindowInfo = std::mem::zeroed();
|
||||
let query = AdFindQuery {
|
||||
role: std::ptr::null(),
|
||||
name_substring: std::ptr::null(),
|
||||
value_substring: std::ptr::null(),
|
||||
};
|
||||
let mut handle = AdNativeHandle {
|
||||
ptr: std::ptr::null(),
|
||||
};
|
||||
let rc = ad_find(adapter, &bad_win, &query, &mut handle);
|
||||
assert!(
|
||||
matches!(rc, AdResult::ErrInvalidArgs | AdResult::ErrInternal),
|
||||
"zeroed window must not succeed, got {:?}",
|
||||
rc
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn free_handle_null_is_noop() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let mut handle = AdNativeHandle {
|
||||
ptr: std::ptr::null(),
|
||||
};
|
||||
let rc = ad_free_handle(adapter, &mut handle);
|
||||
assert_eq!(rc, AdResult::Ok);
|
||||
assert!(handle.ptr.is_null());
|
||||
|
||||
let rc2 = ad_free_handle(adapter, std::ptr::null_mut());
|
||||
assert_eq!(rc2, AdResult::Ok);
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn free_handle_zeroes_ptr_so_double_free_is_noop() {
|
||||
with_adapter(|adapter| unsafe {
|
||||
let fake_live_ptr = 0x1234 as *const std::ffi::c_void;
|
||||
let mut handle = AdNativeHandle { ptr: fake_live_ptr };
|
||||
|
||||
let _ = ad_free_handle(adapter, &mut handle);
|
||||
assert!(handle.ptr.is_null());
|
||||
|
||||
let rc = ad_free_handle(adapter, &mut handle);
|
||||
assert_eq!(rc, AdResult::Ok);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn last_error_survives_successful_calls() {
|
||||
unsafe {
|
||||
let adapter = ad_adapter_create();
|
||||
assert!(!adapter.is_null());
|
||||
let mut out: AdWindowInfo = std::mem::zeroed();
|
||||
let rc = common::ad_launch_app(adapter, std::ptr::null(), 0, &mut out);
|
||||
assert!(
|
||||
matches!(rc, AdResult::ErrInvalidArgs | AdResult::ErrInternal),
|
||||
"must fail, got {:?}",
|
||||
rc
|
||||
);
|
||||
let msg_ptr = ad_last_error_message();
|
||||
assert!(!msg_ptr.is_null());
|
||||
|
||||
for _ in 0..5 {
|
||||
let _ = ad_app_list_count(std::ptr::null());
|
||||
let _ = ad_window_list_count(std::ptr::null());
|
||||
}
|
||||
|
||||
let after = ad_last_error_message();
|
||||
assert_eq!(msg_ptr, after);
|
||||
assert_eq!(ad_last_error_code(), rc);
|
||||
ad_adapter_destroy(adapter);
|
||||
}
|
||||
}
|
||||
100
crates/ffi/tests/common/mod.rs
Normal file
100
crates/ffi/tests/common/mod.rs
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
#![allow(dead_code, unused_imports)]
|
||||
#![allow(improper_ctypes)]
|
||||
|
||||
pub use agent_desktop_ffi::error::AdResult;
|
||||
pub use agent_desktop_ffi::{
|
||||
AdAction, AdActionResult, AdAdapter, AdAppList, AdDirection, AdDragParams, AdFindQuery,
|
||||
AdKeyCombo, AdNativeHandle, AdPoint, AdPolicyKind, AdRect, AdRefEntry, AdScrollParams,
|
||||
AdWindowInfo, AdWindowList,
|
||||
};
|
||||
pub use std::ffi::CStr;
|
||||
pub use std::os::raw::c_char;
|
||||
|
||||
extern "C" {
|
||||
pub fn ad_adapter_create() -> *mut AdAdapter;
|
||||
pub fn ad_adapter_destroy(adapter: *mut AdAdapter);
|
||||
pub fn ad_check_permissions(adapter: *const AdAdapter) -> AdResult;
|
||||
|
||||
pub fn ad_last_error_code() -> AdResult;
|
||||
pub fn ad_last_error_message() -> *const c_char;
|
||||
|
||||
pub fn ad_list_apps(adapter: *const AdAdapter, out: *mut *mut AdAppList) -> AdResult;
|
||||
pub fn ad_app_list_count(list: *const AdAppList) -> u32;
|
||||
pub fn ad_app_list_get(list: *const AdAppList, index: u32) -> *const u8;
|
||||
pub fn ad_app_list_free(list: *mut AdAppList);
|
||||
|
||||
pub fn ad_list_windows(
|
||||
adapter: *const AdAdapter,
|
||||
app_filter: *const c_char,
|
||||
focused_only: bool,
|
||||
out: *mut *mut AdWindowList,
|
||||
) -> AdResult;
|
||||
pub fn ad_window_list_count(list: *const AdWindowList) -> u32;
|
||||
pub fn ad_window_list_free(list: *mut AdWindowList);
|
||||
|
||||
pub fn ad_launch_app(
|
||||
adapter: *const AdAdapter,
|
||||
id: *const c_char,
|
||||
timeout_ms: u64,
|
||||
out: *mut AdWindowInfo,
|
||||
) -> AdResult;
|
||||
|
||||
pub fn ad_execute_action(
|
||||
adapter: *const AdAdapter,
|
||||
handle: *const AdNativeHandle,
|
||||
action: *const AdAction,
|
||||
out: *mut AdActionResult,
|
||||
) -> AdResult;
|
||||
pub fn ad_execute_action_with_policy(
|
||||
adapter: *const AdAdapter,
|
||||
handle: *const AdNativeHandle,
|
||||
action: *const AdAction,
|
||||
policy: i32,
|
||||
out: *mut AdActionResult,
|
||||
) -> AdResult;
|
||||
|
||||
pub fn ad_find(
|
||||
adapter: *const AdAdapter,
|
||||
win: *const AdWindowInfo,
|
||||
query: *const AdFindQuery,
|
||||
out: *mut AdNativeHandle,
|
||||
) -> AdResult;
|
||||
|
||||
pub fn ad_free_handle(adapter: *const AdAdapter, handle: *mut AdNativeHandle) -> AdResult;
|
||||
|
||||
pub fn ad_resolve_element(
|
||||
adapter: *const AdAdapter,
|
||||
entry: *const AdRefEntry,
|
||||
out: *mut AdNativeHandle,
|
||||
) -> AdResult;
|
||||
}
|
||||
|
||||
pub fn with_adapter<F: FnOnce(*mut AdAdapter)>(body: F) {
|
||||
unsafe {
|
||||
let adapter = ad_adapter_create();
|
||||
assert!(!adapter.is_null(), "ad_adapter_create must not return null");
|
||||
body(adapter);
|
||||
ad_adapter_destroy(adapter);
|
||||
}
|
||||
}
|
||||
|
||||
pub fn default_action() -> AdAction {
|
||||
AdAction {
|
||||
kind: 0,
|
||||
text: std::ptr::null(),
|
||||
scroll: AdScrollParams {
|
||||
direction: AdDirection::Down as i32,
|
||||
amount: 0,
|
||||
},
|
||||
key: AdKeyCombo {
|
||||
key: std::ptr::null(),
|
||||
modifiers: std::ptr::null(),
|
||||
modifier_count: 0,
|
||||
},
|
||||
drag: AdDragParams {
|
||||
from: AdPoint { x: 0.0, y: 0.0 },
|
||||
to: AdPoint { x: 0.0, y: 0.0 },
|
||||
duration_ms: 0,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
|
@ -13,7 +13,6 @@ extern "C" {
|
|||
) -> AdResult;
|
||||
fn ad_last_error_message() -> *const std::os::raw::c_char;
|
||||
fn ad_last_error_code() -> AdResult;
|
||||
fn ad_check_permissions(adapter: *const agent_desktop_ffi::AdAdapter) -> AdResult;
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
@ -39,7 +38,9 @@ fn last_error_pointer_survives_across_successful_calls() {
|
|||
let first_msg = CStr::from_ptr(first_msg_ptr).to_string_lossy().into_owned();
|
||||
|
||||
for _ in 0..10 {
|
||||
let _rc = ad_check_permissions(adapter);
|
||||
let next = ad_adapter_create();
|
||||
assert!(!next.is_null());
|
||||
ad_adapter_destroy(next);
|
||||
}
|
||||
|
||||
let later_msg_ptr = ad_last_error_message();
|
||||
|
|
|
|||
|
|
@ -1,13 +1,13 @@
|
|||
use agent_desktop_core::error::AdapterError;
|
||||
use agent_desktop_core::error::{AdapterError, ErrorCode};
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
mod imp {
|
||||
use super::*;
|
||||
use crate::tree::AXElement;
|
||||
use accessibility_sys::{
|
||||
kAXErrorCannotComplete, kAXErrorSuccess, kAXFocusedAttribute, kAXValueAttribute,
|
||||
AXUIElementCopyActionNames, AXUIElementIsAttributeSettable, AXUIElementPerformAction,
|
||||
AXUIElementSetAttributeValue, AXUIElementSetMessagingTimeout,
|
||||
kAXErrorAPIDisabled, kAXErrorCannotComplete, kAXErrorSuccess, kAXFocusedAttribute,
|
||||
kAXValueAttribute, AXUIElementCopyActionNames, AXUIElementIsAttributeSettable,
|
||||
AXUIElementPerformAction, AXUIElementSetAttributeValue, AXUIElementSetMessagingTimeout,
|
||||
};
|
||||
use core_foundation::{
|
||||
array::CFArray,
|
||||
|
|
@ -24,20 +24,37 @@ mod imp {
|
|||
}
|
||||
|
||||
pub fn try_ax_action_retried(el: &AXElement, name: &str) -> bool {
|
||||
try_ax_action_retried_or_err(el, name).unwrap_or(false)
|
||||
}
|
||||
|
||||
pub fn try_ax_action_retried_or_err(el: &AXElement, name: &str) -> Result<bool, AdapterError> {
|
||||
let action = CFString::new(name);
|
||||
let err = unsafe { AXUIElementPerformAction(el.0, action.as_concrete_TypeRef()) };
|
||||
if err == kAXErrorSuccess {
|
||||
return true;
|
||||
return Ok(true);
|
||||
}
|
||||
if err == kAXErrorCannotComplete {
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
let retry = unsafe { AXUIElementPerformAction(el.0, action.as_concrete_TypeRef()) };
|
||||
return retry == kAXErrorSuccess;
|
||||
if retry == kAXErrorSuccess {
|
||||
return Ok(true);
|
||||
}
|
||||
ax_error_result(name, retry)?;
|
||||
return Ok(false);
|
||||
}
|
||||
false
|
||||
ax_error_result(name, err)?;
|
||||
Ok(false)
|
||||
}
|
||||
|
||||
pub fn set_ax_bool(el: &AXElement, attr: &str, value: bool) -> bool {
|
||||
set_ax_bool_or_err(el, attr, value).unwrap_or(false)
|
||||
}
|
||||
|
||||
pub fn set_ax_bool_or_err(
|
||||
el: &AXElement,
|
||||
attr: &str,
|
||||
value: bool,
|
||||
) -> Result<bool, AdapterError> {
|
||||
let cf_attr = CFString::new(attr);
|
||||
let cf_val = if value {
|
||||
CFBoolean::true_value()
|
||||
|
|
@ -47,7 +64,11 @@ mod imp {
|
|||
let err = unsafe {
|
||||
AXUIElementSetAttributeValue(el.0, cf_attr.as_concrete_TypeRef(), cf_val.as_CFTypeRef())
|
||||
};
|
||||
err == kAXErrorSuccess
|
||||
if err == kAXErrorSuccess {
|
||||
return Ok(true);
|
||||
}
|
||||
ax_error_result(attr, err)?;
|
||||
Ok(false)
|
||||
}
|
||||
|
||||
pub fn set_ax_string_or_err(
|
||||
|
|
@ -61,8 +82,9 @@ mod imp {
|
|||
AXUIElementSetAttributeValue(el.0, cf_attr.as_concrete_TypeRef(), cf_val.as_CFTypeRef())
|
||||
};
|
||||
if err != kAXErrorSuccess {
|
||||
ax_error_result(attr, err)?;
|
||||
return Err(AdapterError::new(
|
||||
agent_desktop_core::error::ErrorCode::ActionFailed,
|
||||
ErrorCode::ActionFailed,
|
||||
format!("AXSetAttributeValue({attr}) failed (err={err})"),
|
||||
)
|
||||
.with_suggestion("Attribute may be read-only. Try 'click' or 'type' instead."));
|
||||
|
|
@ -142,8 +164,8 @@ mod imp {
|
|||
unsafe { AXUIElementSetMessagingTimeout(el.0, seconds) };
|
||||
}
|
||||
|
||||
pub fn ax_focus(el: &AXElement) -> bool {
|
||||
set_ax_bool(el, kAXFocusedAttribute, true)
|
||||
pub fn ax_focus_or_err(el: &AXElement) -> Result<bool, AdapterError> {
|
||||
set_ax_bool_or_err(el, kAXFocusedAttribute, true)
|
||||
}
|
||||
|
||||
pub fn ax_set_value(el: &AXElement, val: &str) -> Result<(), AdapterError> {
|
||||
|
|
@ -159,6 +181,14 @@ mod imp {
|
|||
crate::tree::copy_string_attr(el, kAXRoleAttribute)
|
||||
.map(|r| crate::tree::roles::ax_role_to_str(&r).to_string())
|
||||
}
|
||||
|
||||
fn ax_error_result(operation: &str, err: i32) -> Result<(), AdapterError> {
|
||||
if err == kAXErrorAPIDisabled {
|
||||
return Err(AdapterError::permission_denied()
|
||||
.with_platform_detail(format!("{operation} failed with kAXErrorAPIDisabled")));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
|
|
@ -172,9 +202,22 @@ mod imp {
|
|||
pub fn try_ax_action_retried(_el: &AXElement, _name: &str) -> bool {
|
||||
false
|
||||
}
|
||||
pub fn try_ax_action_retried_or_err(
|
||||
_el: &AXElement,
|
||||
_name: &str,
|
||||
) -> Result<bool, AdapterError> {
|
||||
Ok(false)
|
||||
}
|
||||
pub fn set_ax_bool(_el: &AXElement, _attr: &str, _value: bool) -> bool {
|
||||
false
|
||||
}
|
||||
pub fn set_ax_bool_or_err(
|
||||
_el: &AXElement,
|
||||
_attr: &str,
|
||||
_value: bool,
|
||||
) -> Result<bool, AdapterError> {
|
||||
Ok(false)
|
||||
}
|
||||
pub fn set_ax_string_or_err(
|
||||
_el: &AXElement,
|
||||
_attr: &str,
|
||||
|
|
@ -206,8 +249,8 @@ mod imp {
|
|||
}
|
||||
pub fn ensure_visible(_el: &AXElement) {}
|
||||
pub fn set_messaging_timeout(_el: &AXElement, _seconds: f32) {}
|
||||
pub fn ax_focus(_el: &AXElement) -> bool {
|
||||
false
|
||||
pub fn ax_focus_or_err(_el: &AXElement) -> Result<bool, AdapterError> {
|
||||
Ok(false)
|
||||
}
|
||||
pub fn ax_set_value(_el: &AXElement, _val: &str) -> Result<(), AdapterError> {
|
||||
Err(AdapterError::not_supported("ax_set_value"))
|
||||
|
|
@ -221,7 +264,8 @@ mod imp {
|
|||
}
|
||||
|
||||
pub(crate) use imp::{
|
||||
ax_focus, ax_press, ax_set_value, element_role, ensure_visible, has_ax_action,
|
||||
is_attr_settable, list_ax_actions, set_ax_bool, set_ax_string_or_err, set_messaging_timeout,
|
||||
try_action_from_list, try_ax_action, try_ax_action_retried, try_each_ancestor, try_each_child,
|
||||
ax_focus_or_err, ax_press, ax_set_value, element_role, ensure_visible, has_ax_action,
|
||||
is_attr_settable, list_ax_actions, set_ax_bool, set_ax_bool_or_err, set_ax_string_or_err,
|
||||
set_messaging_timeout, try_action_from_list, try_ax_action, try_ax_action_retried,
|
||||
try_ax_action_retried_or_err, try_each_ancestor, try_each_child,
|
||||
};
|
||||
|
|
|
|||
|
|
@ -1,51 +1,12 @@
|
|||
use agent_desktop_core::action::MouseButton;
|
||||
use agent_desktop_core::action::InteractionPolicy;
|
||||
use agent_desktop_core::error::{AdapterError, ErrorCode};
|
||||
|
||||
use crate::actions::discovery::ElementCaps;
|
||||
use crate::tree::AXElement;
|
||||
|
||||
#[allow(dead_code)]
|
||||
pub enum ChainStep {
|
||||
Action(&'static str),
|
||||
SetBool {
|
||||
attr: &'static str,
|
||||
value: bool,
|
||||
},
|
||||
SetDynamic {
|
||||
attr: &'static str,
|
||||
},
|
||||
FocusThenAction(&'static str),
|
||||
FocusThenSetDynamic {
|
||||
attr: &'static str,
|
||||
},
|
||||
FocusThenConfirmOrPress,
|
||||
ChildActions {
|
||||
actions: &'static [&'static str],
|
||||
limit: usize,
|
||||
},
|
||||
AncestorActions {
|
||||
actions: &'static [&'static str],
|
||||
limit: usize,
|
||||
},
|
||||
Custom {
|
||||
label: &'static str,
|
||||
func: fn(&AXElement, &ElementCaps) -> bool,
|
||||
},
|
||||
CGClick {
|
||||
button: MouseButton,
|
||||
count: u32,
|
||||
},
|
||||
}
|
||||
|
||||
pub struct ChainDef {
|
||||
pub pre_scroll: bool,
|
||||
pub steps: &'static [ChainStep],
|
||||
pub suggestion: &'static str,
|
||||
}
|
||||
|
||||
pub struct ChainContext<'a> {
|
||||
pub dynamic_value: Option<&'a str>,
|
||||
}
|
||||
pub(crate) use super::chain_context::ChainContext;
|
||||
pub(crate) use super::chain_def::ChainDef;
|
||||
pub(crate) use super::chain_step::ChainStep;
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
mod imp {
|
||||
|
|
@ -53,15 +14,19 @@ mod imp {
|
|||
use crate::actions::ax_helpers;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
const CHAIN_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
const DEFAULT_CHAIN_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
const MAX_CHAIN_TIMEOUT_MS: u64 = 300_000;
|
||||
|
||||
pub fn execute_chain(
|
||||
el: &AXElement,
|
||||
caps: &ElementCaps,
|
||||
def: &ChainDef,
|
||||
ctx: &ChainContext,
|
||||
policy: InteractionPolicy,
|
||||
) -> Result<(), AdapterError> {
|
||||
let deadline = Instant::now() + CHAIN_TIMEOUT;
|
||||
let deadline = ctx
|
||||
.deadline
|
||||
.unwrap_or_else(|| Instant::now() + chain_timeout());
|
||||
let total = def.steps.len();
|
||||
|
||||
if let Some(pid) = crate::system::app_ops::pid_from_element(el) {
|
||||
|
|
@ -82,15 +47,25 @@ mod imp {
|
|||
.iter()
|
||||
.find(|s| matches!(s, ChainStep::CGClick { .. }))
|
||||
{
|
||||
if execute_step(el, caps, cg, ctx) {
|
||||
if physical_click_permitted(policy) && execute_step(el, caps, cg, ctx, policy)?
|
||||
{
|
||||
tracing::debug!("chain: CGClick fallback succeeded");
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
return Err(AdapterError::timeout("Chain execution exceeded 10s"));
|
||||
return Err(
|
||||
AdapterError::timeout("Chain execution deadline exceeded").with_suggestion(
|
||||
"Retry the command, refresh the snapshot, or increase AGENT_DESKTOP_CHAIN_TIMEOUT_MS for slow apps.",
|
||||
),
|
||||
);
|
||||
}
|
||||
if matches!(step, ChainStep::CGClick { .. }) && !physical_click_permitted(policy) {
|
||||
return Err(AdapterError::policy_denied(
|
||||
"Physical click fallback is disabled by the current interaction policy",
|
||||
));
|
||||
}
|
||||
let label = step_label(step);
|
||||
if execute_step(el, caps, step, ctx) {
|
||||
if execute_step(el, caps, step, ctx, policy)? {
|
||||
tracing::debug!("chain: [{}/{}] {} -> success", i + 1, total, label);
|
||||
return Ok(());
|
||||
}
|
||||
|
|
@ -109,9 +84,8 @@ mod imp {
|
|||
ChainStep::Action(name) => name,
|
||||
ChainStep::SetBool { attr, .. } => attr,
|
||||
ChainStep::SetDynamic { attr } => attr,
|
||||
ChainStep::FocusThenAction(name) => name,
|
||||
ChainStep::FocusThenSetDynamic { attr } => attr,
|
||||
ChainStep::FocusThenConfirmOrPress => "FocusThenConfirmOrPress",
|
||||
ChainStep::FocusThenClearByKeyboard => "FocusThenClearByKeyboard",
|
||||
ChainStep::ChildActions { .. } => "ChildActions",
|
||||
ChainStep::AncestorActions { .. } => "AncestorActions",
|
||||
ChainStep::Custom { label, .. } => label,
|
||||
|
|
@ -124,9 +98,10 @@ mod imp {
|
|||
caps: &ElementCaps,
|
||||
step: &ChainStep,
|
||||
ctx: &ChainContext,
|
||||
) -> bool {
|
||||
policy: InteractionPolicy,
|
||||
) -> Result<bool, AdapterError> {
|
||||
match step {
|
||||
ChainStep::Action(name) => ax_helpers::try_ax_action_retried(el, name),
|
||||
ChainStep::Action(name) => ax_helpers::try_ax_action_retried_or_err(el, name),
|
||||
|
||||
ChainStep::SetBool { attr, value } => {
|
||||
let settable = match *attr {
|
||||
|
|
@ -135,71 +110,177 @@ mod imp {
|
|||
"AXFocused" => caps.settable_focus,
|
||||
_ => ax_helpers::is_attr_settable(el, attr),
|
||||
};
|
||||
settable && ax_helpers::set_ax_bool(el, attr, *value)
|
||||
Ok(settable && set_bool_verified(el, attr, *value)?)
|
||||
}
|
||||
|
||||
ChainStep::SetDynamic { attr } => {
|
||||
let value = match ctx.dynamic_value {
|
||||
Some(v) => v,
|
||||
None => return false,
|
||||
None => return Ok(false),
|
||||
};
|
||||
ax_helpers::set_ax_string_or_err(el, attr, value).is_ok()
|
||||
}
|
||||
|
||||
ChainStep::FocusThenAction(name) => {
|
||||
if !ax_helpers::ax_focus(el) {
|
||||
return false;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(50));
|
||||
ax_helpers::try_ax_action_retried(el, name)
|
||||
set_dynamic_verified(el, attr, value)
|
||||
}
|
||||
|
||||
ChainStep::FocusThenSetDynamic { attr } => {
|
||||
if !policy.allow_focus_steal {
|
||||
return Ok(false);
|
||||
}
|
||||
let value = match ctx.dynamic_value {
|
||||
Some(v) => v,
|
||||
None => return false,
|
||||
None => return Ok(false),
|
||||
};
|
||||
if !ax_helpers::ax_focus(el) {
|
||||
return false;
|
||||
if !ax_helpers::ax_focus_or_err(el)? {
|
||||
return Ok(false);
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(50));
|
||||
ax_helpers::set_ax_string_or_err(el, attr, value).is_ok()
|
||||
set_dynamic_verified(el, attr, value)
|
||||
}
|
||||
|
||||
ChainStep::FocusThenConfirmOrPress => {
|
||||
if !ax_helpers::ax_focus(el) {
|
||||
return false;
|
||||
ChainStep::FocusThenClearByKeyboard => {
|
||||
if !policy.allow_focus_steal {
|
||||
return Ok(false);
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(50));
|
||||
ax_helpers::try_ax_action_retried(el, "AXConfirm")
|
||||
|| ax_helpers::try_ax_action_retried(el, "AXPress")
|
||||
if !ax_helpers::ax_focus_or_err(el)? {
|
||||
return Ok(false);
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(20));
|
||||
Ok(crate::input::keyboard::synthesize_key_for_element(
|
||||
el,
|
||||
&agent_desktop_core::action::KeyCombo {
|
||||
key: "a".into(),
|
||||
modifiers: vec![agent_desktop_core::action::Modifier::Cmd],
|
||||
},
|
||||
)
|
||||
.and_then(|_| {
|
||||
crate::input::keyboard::synthesize_key_for_element(
|
||||
el,
|
||||
&agent_desktop_core::action::KeyCombo {
|
||||
key: "delete".into(),
|
||||
modifiers: vec![],
|
||||
},
|
||||
)
|
||||
})
|
||||
.is_ok())
|
||||
}
|
||||
|
||||
ChainStep::ChildActions { actions, limit } => ax_helpers::try_each_child(
|
||||
ChainStep::ChildActions { actions, limit } => Ok(ax_helpers::try_each_child(
|
||||
el,
|
||||
|child| {
|
||||
let child_actions = ax_helpers::list_ax_actions(child);
|
||||
ax_helpers::try_action_from_list(child, &child_actions, actions)
|
||||
},
|
||||
*limit,
|
||||
),
|
||||
)),
|
||||
|
||||
ChainStep::AncestorActions { actions, limit } => ax_helpers::try_each_ancestor(
|
||||
ChainStep::AncestorActions { actions, limit } => Ok(ax_helpers::try_each_ancestor(
|
||||
el,
|
||||
|ancestor| {
|
||||
let al = ax_helpers::list_ax_actions(ancestor);
|
||||
ax_helpers::try_action_from_list(ancestor, &al, actions)
|
||||
},
|
||||
*limit,
|
||||
),
|
||||
)),
|
||||
|
||||
ChainStep::Custom { label: _, func } => func(el, caps),
|
||||
|
||||
ChainStep::CGClick { button, count } => {
|
||||
crate::actions::dispatch::click_via_bounds(el, button.clone(), *count).is_ok()
|
||||
Ok(
|
||||
crate::actions::dispatch::click_via_bounds(el, button.clone(), *count, policy)
|
||||
.is_ok(),
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn chain_timeout() -> Duration {
|
||||
std::env::var("AGENT_DESKTOP_CHAIN_TIMEOUT_MS")
|
||||
.ok()
|
||||
.and_then(|v| v.parse::<u64>().ok())
|
||||
.filter(|ms| *ms > 0)
|
||||
.map(|ms| ms.min(MAX_CHAIN_TIMEOUT_MS))
|
||||
.map(Duration::from_millis)
|
||||
.unwrap_or(DEFAULT_CHAIN_TIMEOUT)
|
||||
}
|
||||
|
||||
fn physical_click_permitted(policy: InteractionPolicy) -> bool {
|
||||
policy.allow_focus_steal && policy.allow_cursor_move
|
||||
}
|
||||
|
||||
fn set_dynamic_verified(el: &AXElement, attr: &str, value: &str) -> Result<bool, AdapterError> {
|
||||
ax_helpers::set_ax_string_or_err(el, attr, value)?;
|
||||
Ok(dynamic_write_had_effect(
|
||||
attr,
|
||||
ax_helpers::element_role(el).as_deref(),
|
||||
value,
|
||||
crate::tree::copy_value_typed(el).as_deref(),
|
||||
))
|
||||
}
|
||||
|
||||
fn set_bool_verified(el: &AXElement, attr: &str, value: bool) -> Result<bool, AdapterError> {
|
||||
Ok(ax_helpers::set_ax_bool_or_err(el, attr, value)?
|
||||
&& bool_write_had_effect(attr, value, crate::tree::copy_bool_attr(el, attr)))
|
||||
}
|
||||
|
||||
fn bool_write_had_effect(attr: &str, expected: bool, observed: Option<bool>) -> bool {
|
||||
!matches!(
|
||||
attr,
|
||||
"AXExpanded" | "AXDisclosing" | "AXSelected" | "AXFocused"
|
||||
) || observed == Some(expected)
|
||||
}
|
||||
|
||||
fn dynamic_write_had_effect(
|
||||
attr: &str,
|
||||
role: Option<&str>,
|
||||
expected: &str,
|
||||
observed: Option<&str>,
|
||||
) -> bool {
|
||||
attr != "AXValue" || role == Some("AXSecureTextField") || observed == Some(expected)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{bool_write_had_effect, dynamic_write_had_effect};
|
||||
|
||||
#[test]
|
||||
fn ax_value_write_requires_readback_match() {
|
||||
assert!(!dynamic_write_had_effect(
|
||||
"AXValue",
|
||||
Some("AXTextField"),
|
||||
"",
|
||||
Some("unchanged")
|
||||
));
|
||||
assert!(dynamic_write_had_effect(
|
||||
"AXValue",
|
||||
Some("AXTextField"),
|
||||
"",
|
||||
Some("")
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn non_value_and_secure_writes_trust_ax_success() {
|
||||
assert!(dynamic_write_had_effect(
|
||||
"AXSelected",
|
||||
Some("AXCheckBox"),
|
||||
"true",
|
||||
None
|
||||
));
|
||||
assert!(dynamic_write_had_effect(
|
||||
"AXValue",
|
||||
Some("AXSecureTextField"),
|
||||
"secret",
|
||||
None
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bool_state_writes_require_readback_match_for_stateful_attrs() {
|
||||
assert!(bool_write_had_effect("AXExpanded", true, Some(true)));
|
||||
assert!(!bool_write_had_effect("AXExpanded", true, Some(false)));
|
||||
assert!(!bool_write_had_effect("AXExpanded", false, None));
|
||||
assert!(bool_write_had_effect("AXFoo", true, None));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
|
|
@ -211,6 +292,7 @@ mod imp {
|
|||
_caps: &ElementCaps,
|
||||
def: &ChainDef,
|
||||
_ctx: &ChainContext,
|
||||
_policy: InteractionPolicy,
|
||||
) -> Result<(), AdapterError> {
|
||||
Err(AdapterError::new(
|
||||
ErrorCode::ActionFailed,
|
||||
|
|
|
|||
4
crates/macos/src/actions/chain_context.rs
Normal file
4
crates/macos/src/actions/chain_context.rs
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
pub(crate) struct ChainContext<'a> {
|
||||
pub(crate) dynamic_value: Option<&'a str>,
|
||||
pub(crate) deadline: Option<std::time::Instant>,
|
||||
}
|
||||
7
crates/macos/src/actions/chain_def.rs
Normal file
7
crates/macos/src/actions/chain_def.rs
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
use super::chain_step::ChainStep;
|
||||
|
||||
pub(crate) struct ChainDef {
|
||||
pub(crate) pre_scroll: bool,
|
||||
pub(crate) steps: &'static [ChainStep],
|
||||
pub(crate) suggestion: &'static str,
|
||||
}
|
||||
|
|
@ -5,12 +5,12 @@ mod imp {
|
|||
use super::*;
|
||||
use crate::actions::{
|
||||
ax_helpers,
|
||||
chain::{execute_chain, ChainContext, ChainDef, ChainStep},
|
||||
chain_steps,
|
||||
chain::{ChainDef, ChainStep},
|
||||
chain_menu_steps, chain_steps,
|
||||
discovery::ElementCaps,
|
||||
};
|
||||
use crate::tree::AXElement;
|
||||
use agent_desktop_core::action::MouseButton;
|
||||
use agent_desktop_core::action::{InteractionPolicy, MouseButton};
|
||||
|
||||
pub static CLICK_CHAIN: ChainDef = ChainDef {
|
||||
pre_scroll: true,
|
||||
|
|
@ -26,13 +26,9 @@ mod imp {
|
|||
label: "show_alternate_ui",
|
||||
func: chain_steps::try_show_alternate_ui,
|
||||
},
|
||||
ChainStep::ChildActions {
|
||||
actions: &["AXPress", "AXConfirm", "AXOpen"],
|
||||
limit: 3,
|
||||
},
|
||||
ChainStep::Custom {
|
||||
label: "value_relay",
|
||||
func: chain_steps::try_value_relay,
|
||||
label: "containing_item_select",
|
||||
func: chain_steps::try_select_containing_item,
|
||||
},
|
||||
ChainStep::SetBool {
|
||||
attr: "AXSelected",
|
||||
|
|
@ -42,22 +38,22 @@ mod imp {
|
|||
label: "parent_row_select",
|
||||
func: chain_steps::try_parent_row_select,
|
||||
},
|
||||
ChainStep::Custom {
|
||||
label: "value_relay",
|
||||
func: chain_steps::try_value_relay,
|
||||
},
|
||||
ChainStep::Custom {
|
||||
label: "select_via_parent",
|
||||
func: chain_steps::try_select_via_parent,
|
||||
},
|
||||
ChainStep::ChildActions {
|
||||
actions: &["AXPress", "AXConfirm", "AXOpen"],
|
||||
limit: 3,
|
||||
},
|
||||
ChainStep::Custom {
|
||||
label: "custom_actions",
|
||||
func: chain_steps::try_custom_actions,
|
||||
},
|
||||
ChainStep::Custom {
|
||||
label: "focus_verified_confirm_or_press",
|
||||
func: chain_steps::try_focus_then_verified_confirm_or_press,
|
||||
},
|
||||
ChainStep::Custom {
|
||||
label: "keyboard_activate",
|
||||
func: chain_steps::try_keyboard_activate,
|
||||
},
|
||||
ChainStep::AncestorActions {
|
||||
actions: &["AXPress", "AXConfirm"],
|
||||
limit: 2,
|
||||
|
|
@ -73,23 +69,25 @@ mod imp {
|
|||
pub static RIGHT_CLICK_CHAIN: ChainDef = ChainDef {
|
||||
pre_scroll: false,
|
||||
steps: &[
|
||||
ChainStep::Action("AXShowMenu"),
|
||||
ChainStep::Custom {
|
||||
label: "focus_app_show_menu",
|
||||
func: chain_steps::focus_app_then_show_menu,
|
||||
label: "show_menu",
|
||||
func: chain_menu_steps::show_menu,
|
||||
},
|
||||
ChainStep::Custom {
|
||||
label: "select_then_show_menu",
|
||||
func: chain_steps::select_then_show_menu,
|
||||
func: chain_menu_steps::select_then_show_menu,
|
||||
},
|
||||
ChainStep::FocusThenAction("AXShowMenu"),
|
||||
ChainStep::AncestorActions {
|
||||
actions: &["AXShowMenu"],
|
||||
limit: 3,
|
||||
ChainStep::Custom {
|
||||
label: "selected_items_menu",
|
||||
func: chain_menu_steps::select_then_selected_items_menu,
|
||||
},
|
||||
ChainStep::ChildActions {
|
||||
actions: &["AXShowMenu"],
|
||||
limit: 5,
|
||||
ChainStep::Custom {
|
||||
label: "child_show_menu",
|
||||
func: chain_menu_steps::show_menu_on_children,
|
||||
},
|
||||
ChainStep::Custom {
|
||||
label: "ancestor_show_menu",
|
||||
func: chain_menu_steps::show_menu_on_ancestors,
|
||||
},
|
||||
ChainStep::CGClick {
|
||||
button: MouseButton::Right,
|
||||
|
|
@ -103,6 +101,10 @@ mod imp {
|
|||
pre_scroll: false,
|
||||
steps: &[
|
||||
ChainStep::Action("AXExpand"),
|
||||
ChainStep::SetBool {
|
||||
attr: "AXExpanded",
|
||||
value: true,
|
||||
},
|
||||
ChainStep::SetBool {
|
||||
attr: "AXDisclosing",
|
||||
value: true,
|
||||
|
|
@ -115,6 +117,10 @@ mod imp {
|
|||
pre_scroll: false,
|
||||
steps: &[
|
||||
ChainStep::Action("AXCollapse"),
|
||||
ChainStep::SetBool {
|
||||
attr: "AXExpanded",
|
||||
value: false,
|
||||
},
|
||||
ChainStep::SetBool {
|
||||
attr: "AXDisclosing",
|
||||
value: false,
|
||||
|
|
@ -123,12 +129,14 @@ mod imp {
|
|||
suggestion: "Try 'click' to close it instead.",
|
||||
};
|
||||
|
||||
const VALUE_STEPS: &[ChainStep] = &[
|
||||
ChainStep::SetDynamic { attr: "AXValue" },
|
||||
ChainStep::FocusThenSetDynamic { attr: "AXValue" },
|
||||
];
|
||||
|
||||
pub static SET_VALUE_CHAIN: ChainDef = ChainDef {
|
||||
pre_scroll: false,
|
||||
steps: &[
|
||||
ChainStep::SetDynamic { attr: "AXValue" },
|
||||
ChainStep::FocusThenSetDynamic { attr: "AXValue" },
|
||||
],
|
||||
steps: VALUE_STEPS,
|
||||
suggestion: "Try 'clear' then 'type', or check element is a text field.",
|
||||
};
|
||||
|
||||
|
|
@ -137,10 +145,7 @@ mod imp {
|
|||
steps: &[
|
||||
ChainStep::SetDynamic { attr: "AXValue" },
|
||||
ChainStep::FocusThenSetDynamic { attr: "AXValue" },
|
||||
ChainStep::Custom {
|
||||
label: "select_all_delete",
|
||||
func: chain_steps::select_all_then_delete,
|
||||
},
|
||||
ChainStep::FocusThenClearByKeyboard,
|
||||
],
|
||||
suggestion: "Try 'press cmd+a' then 'press delete'.",
|
||||
};
|
||||
|
|
@ -171,35 +176,30 @@ mod imp {
|
|||
steps: &[
|
||||
ChainStep::Action("AXScrollToVisible"),
|
||||
ChainStep::Custom {
|
||||
label: "walk_parents_scroll",
|
||||
func: chain_steps::walk_parents_and_scroll,
|
||||
label: "visible_in_scroll_context",
|
||||
func: chain_steps::element_is_visible_in_scroll_context,
|
||||
},
|
||||
],
|
||||
suggestion: "Element may not be in a scrollable container.",
|
||||
};
|
||||
|
||||
pub fn double_click(el: &AXElement, caps: &ElementCaps) -> Result<(), AdapterError> {
|
||||
pub fn double_click(
|
||||
el: &AXElement,
|
||||
_caps: &ElementCaps,
|
||||
policy: InteractionPolicy,
|
||||
) -> Result<(), AdapterError> {
|
||||
if ax_helpers::try_ax_action(el, "AXOpen") {
|
||||
return Ok(());
|
||||
}
|
||||
let ctx = ChainContext {
|
||||
dynamic_value: None,
|
||||
};
|
||||
let _ = execute_chain(el, caps, &CLICK_CHAIN, &ctx);
|
||||
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||
let _ = execute_chain(el, caps, &CLICK_CHAIN, &ctx);
|
||||
crate::actions::dispatch::click_via_bounds(el, MouseButton::Left, 2)
|
||||
crate::actions::dispatch::click_via_bounds(el, MouseButton::Left, 2, policy)
|
||||
}
|
||||
|
||||
pub fn triple_click(el: &AXElement, caps: &ElementCaps) -> Result<(), AdapterError> {
|
||||
let ctx = ChainContext {
|
||||
dynamic_value: None,
|
||||
};
|
||||
for _ in 0..3 {
|
||||
let _ = execute_chain(el, caps, &CLICK_CHAIN, &ctx);
|
||||
std::thread::sleep(std::time::Duration::from_millis(30));
|
||||
}
|
||||
crate::actions::dispatch::click_via_bounds(el, MouseButton::Left, 3)
|
||||
pub fn triple_click(
|
||||
el: &AXElement,
|
||||
_caps: &ElementCaps,
|
||||
policy: InteractionPolicy,
|
||||
) -> Result<(), AdapterError> {
|
||||
crate::actions::dispatch::click_via_bounds(el, MouseButton::Left, 3, policy)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
249
crates/macos/src/actions/chain_menu_steps.rs
Normal file
249
crates/macos/src/actions/chain_menu_steps.rs
Normal file
|
|
@ -0,0 +1,249 @@
|
|||
#[cfg(target_os = "macos")]
|
||||
mod imp {
|
||||
use crate::actions::{ax_helpers, discovery::ElementCaps};
|
||||
use crate::tree::AXElement;
|
||||
use agent_desktop_core::error::AdapterError;
|
||||
|
||||
pub fn show_menu(el: &AXElement, _caps: &ElementCaps) -> Result<bool, AdapterError> {
|
||||
show_menu_on_element(el)
|
||||
}
|
||||
|
||||
pub fn show_menu_on_ancestors(
|
||||
el: &AXElement,
|
||||
_caps: &ElementCaps,
|
||||
) -> Result<bool, AdapterError> {
|
||||
let mut current = crate::tree::copy_element_attr(el, "AXParent");
|
||||
for _ in 0..3 {
|
||||
let Some(parent) = current else {
|
||||
return Ok(false);
|
||||
};
|
||||
if show_menu_on_element(&parent)? {
|
||||
return Ok(true);
|
||||
}
|
||||
current = crate::tree::copy_element_attr(&parent, "AXParent");
|
||||
}
|
||||
Ok(false)
|
||||
}
|
||||
|
||||
pub fn show_menu_on_children(
|
||||
el: &AXElement,
|
||||
_caps: &ElementCaps,
|
||||
) -> Result<bool, AdapterError> {
|
||||
for child in crate::tree::copy_ax_array(el, "AXChildren")
|
||||
.unwrap_or_default()
|
||||
.iter()
|
||||
.take(5)
|
||||
{
|
||||
if show_menu_on_element(child)? {
|
||||
return Ok(true);
|
||||
}
|
||||
}
|
||||
Ok(false)
|
||||
}
|
||||
|
||||
pub fn select_then_show_menu(
|
||||
el: &AXElement,
|
||||
_caps: &ElementCaps,
|
||||
) -> Result<bool, AdapterError> {
|
||||
if !select_containing_item(el)? {
|
||||
return Ok(false);
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||
show_menu_on_element(el)
|
||||
}
|
||||
|
||||
pub fn select_then_selected_items_menu(
|
||||
el: &AXElement,
|
||||
_caps: &ElementCaps,
|
||||
) -> Result<bool, AdapterError> {
|
||||
if !select_containing_item(el)? {
|
||||
tracing::debug!("selected-items menu: could not select containing item");
|
||||
return Ok(false);
|
||||
}
|
||||
let selected_name = crate::tree::resolve_element_name(el);
|
||||
let Some(window) = window_ancestor(el) else {
|
||||
tracing::debug!("selected-items menu: no window ancestor");
|
||||
return Ok(false);
|
||||
};
|
||||
let Some(menu_button) = selected_items_menu_button(&window) else {
|
||||
tracing::debug!("selected-items menu: no selected-items menu button");
|
||||
return Ok(false);
|
||||
};
|
||||
show_menu_or_press_selected(&menu_button, selected_name.as_deref())
|
||||
}
|
||||
|
||||
fn show_menu_on_element(el: &AXElement) -> Result<bool, AdapterError> {
|
||||
let Some(pid) = crate::system::app_ops::pid_from_element(el) else {
|
||||
return Ok(false);
|
||||
};
|
||||
let was_open = is_menu_open(pid);
|
||||
if !ax_helpers::try_ax_action_retried_or_err(el, "AXShowMenu")? {
|
||||
return Ok(false);
|
||||
}
|
||||
Ok(wait_for_new_menu(pid, was_open))
|
||||
}
|
||||
|
||||
fn show_menu_or_press_selected(
|
||||
el: &AXElement,
|
||||
selected_name: Option<&str>,
|
||||
) -> Result<bool, AdapterError> {
|
||||
let Some(pid) = crate::system::app_ops::pid_from_element(el) else {
|
||||
return Ok(false);
|
||||
};
|
||||
if ax_helpers::try_ax_action_retried_or_err(el, "AXShowMenu")?
|
||||
&& wait_for_selected_menu(pid, selected_name)
|
||||
{
|
||||
return Ok(true);
|
||||
}
|
||||
Ok(ax_helpers::try_ax_action_retried_or_err(el, "AXPress")?
|
||||
&& wait_for_selected_menu(pid, selected_name))
|
||||
}
|
||||
|
||||
fn wait_for_new_menu(pid: i32, was_open: bool) -> bool {
|
||||
if was_open {
|
||||
return false;
|
||||
}
|
||||
crate::system::wait::wait_for_menu(pid, true, crate::system::wait::menu_timeout_ms())
|
||||
.is_ok()
|
||||
}
|
||||
|
||||
fn is_menu_open(pid: i32) -> bool {
|
||||
crate::system::wait::wait_for_menu(pid, true, 0).is_ok()
|
||||
}
|
||||
|
||||
fn wait_for_selected_menu(pid: i32, selected_name: Option<&str>) -> bool {
|
||||
let deadline = std::time::Instant::now()
|
||||
+ std::time::Duration::from_millis(crate::system::wait::menu_timeout_ms());
|
||||
loop {
|
||||
if menu_matches_selection(pid, selected_name) {
|
||||
return true;
|
||||
}
|
||||
if std::time::Instant::now() >= deadline {
|
||||
return false;
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||
}
|
||||
}
|
||||
|
||||
fn menu_matches_selection(pid: i32, selected_name: Option<&str>) -> bool {
|
||||
let Some(menu) = crate::tree::surfaces::menu_element_for_pid(pid) else {
|
||||
return false;
|
||||
};
|
||||
selected_name
|
||||
.filter(|name| !name.is_empty())
|
||||
.is_none_or(|name| element_text_contains(&menu, name, 0))
|
||||
}
|
||||
|
||||
fn element_text_contains(el: &AXElement, needle: &str, depth: usize) -> bool {
|
||||
if depth > 8 {
|
||||
return false;
|
||||
}
|
||||
["AXTitle", "AXDescription", "AXValue", "AXHelp"]
|
||||
.into_iter()
|
||||
.filter_map(|attr| crate::tree::copy_string_attr(el, attr))
|
||||
.any(|value| value.contains(needle))
|
||||
|| crate::tree::copy_ax_array(el, "AXChildren")
|
||||
.unwrap_or_default()
|
||||
.iter()
|
||||
.any(|child| element_text_contains(child, needle, depth + 1))
|
||||
}
|
||||
|
||||
fn select_containing_item(el: &AXElement) -> Result<bool, AdapterError> {
|
||||
Ok(ax_helpers::set_ax_bool_or_err(el, "AXSelected", true)?
|
||||
|| crate::actions::chain_steps::try_select_containing_item(
|
||||
el,
|
||||
&ElementCaps {
|
||||
settable_focus: false,
|
||||
settable_selected: false,
|
||||
settable_disclosing: false,
|
||||
},
|
||||
)?)
|
||||
}
|
||||
|
||||
fn window_ancestor(el: &AXElement) -> Option<AXElement> {
|
||||
let mut current = crate::tree::copy_element_attr(el, "AXParent");
|
||||
for _ in 0..20 {
|
||||
let ancestor = current?;
|
||||
if crate::tree::copy_string_attr(&ancestor, "AXRole").as_deref() == Some("AXWindow") {
|
||||
return Some(ancestor);
|
||||
}
|
||||
current = crate::tree::copy_element_attr(&ancestor, "AXParent");
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
fn selected_items_menu_button(root: &AXElement) -> Option<AXElement> {
|
||||
find_descendant(root, 0, &|el| {
|
||||
crate::tree::copy_string_attr(el, "AXRole").as_deref() == Some("AXMenuButton")
|
||||
&& is_selected_items_control(el)
|
||||
})
|
||||
}
|
||||
|
||||
fn is_selected_items_control(el: &AXElement) -> bool {
|
||||
["AXHelp", "AXDescription", "AXTitle"]
|
||||
.into_iter()
|
||||
.filter_map(|attr| crate::tree::copy_string_attr(el, attr))
|
||||
.any(|value| {
|
||||
let matches = selected_items_text(&value);
|
||||
if matches {
|
||||
tracing::debug!("selected-items menu: matched control text {value:?}");
|
||||
}
|
||||
matches
|
||||
})
|
||||
}
|
||||
|
||||
fn selected_items_text(value: &str) -> bool {
|
||||
let value = value.to_ascii_lowercase();
|
||||
value.contains("selected item")
|
||||
}
|
||||
|
||||
fn find_descendant(
|
||||
el: &AXElement,
|
||||
depth: usize,
|
||||
predicate: &impl Fn(&AXElement) -> bool,
|
||||
) -> Option<AXElement> {
|
||||
if depth > 8 {
|
||||
return None;
|
||||
}
|
||||
if predicate(el) {
|
||||
return Some(el.clone());
|
||||
}
|
||||
for child in crate::tree::copy_ax_array(el, "AXChildren").unwrap_or_default() {
|
||||
if let Some(found) = find_descendant(&child, depth + 1, predicate) {
|
||||
return Some(found);
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
mod imp {
|
||||
use crate::actions::discovery::ElementCaps;
|
||||
use crate::tree::AXElement;
|
||||
|
||||
pub fn show_menu(_el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
pub fn show_menu_on_ancestors(_el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
pub fn show_menu_on_children(_el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
pub fn select_then_show_menu(_el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
pub fn select_then_selected_items_menu(_el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) use imp::{
|
||||
select_then_selected_items_menu, select_then_show_menu, show_menu, show_menu_on_ancestors,
|
||||
show_menu_on_children,
|
||||
};
|
||||
34
crates/macos/src/actions/chain_step.rs
Normal file
34
crates/macos/src/actions/chain_step.rs
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
use agent_desktop_core::{action::MouseButton, error::AdapterError};
|
||||
|
||||
use crate::{actions::discovery::ElementCaps, tree::AXElement};
|
||||
|
||||
pub(crate) enum ChainStep {
|
||||
Action(&'static str),
|
||||
SetBool {
|
||||
attr: &'static str,
|
||||
value: bool,
|
||||
},
|
||||
SetDynamic {
|
||||
attr: &'static str,
|
||||
},
|
||||
FocusThenSetDynamic {
|
||||
attr: &'static str,
|
||||
},
|
||||
FocusThenClearByKeyboard,
|
||||
ChildActions {
|
||||
actions: &'static [&'static str],
|
||||
limit: usize,
|
||||
},
|
||||
AncestorActions {
|
||||
actions: &'static [&'static str],
|
||||
limit: usize,
|
||||
},
|
||||
Custom {
|
||||
label: &'static str,
|
||||
func: fn(&AXElement, &ElementCaps) -> Result<bool, AdapterError>,
|
||||
},
|
||||
CGClick {
|
||||
button: MouseButton,
|
||||
count: u32,
|
||||
},
|
||||
}
|
||||
|
|
@ -2,22 +2,25 @@
|
|||
mod imp {
|
||||
use crate::actions::{ax_helpers, discovery::ElementCaps};
|
||||
use crate::tree::AXElement;
|
||||
use agent_desktop_core::error::AdapterError;
|
||||
|
||||
pub fn do_verified_press(el: &AXElement, caps: &ElementCaps) -> bool {
|
||||
dispatch_verified_press(el, caps, is_in_webarea(el))
|
||||
pub fn do_verified_press(el: &AXElement, caps: &ElementCaps) -> Result<bool, AdapterError> {
|
||||
dispatch_verified_press(el, caps, crate::actions::chain_web_steps::is_in_webarea(el))
|
||||
}
|
||||
|
||||
fn dispatch_verified_press(el: &AXElement, _caps: &ElementCaps, in_web: bool) -> bool {
|
||||
fn dispatch_verified_press(
|
||||
el: &AXElement,
|
||||
_caps: &ElementCaps,
|
||||
in_web: bool,
|
||||
) -> Result<bool, AdapterError> {
|
||||
if !in_web {
|
||||
return verified_press_native(el);
|
||||
}
|
||||
tracing::debug!("verified_press: web element detected");
|
||||
activate_web_element(el)
|
||||
Ok(crate::actions::chain_web_steps::activate_web_element(el))
|
||||
}
|
||||
|
||||
/// Native (non-web) elements: AXPress with selection verification
|
||||
/// for elements inside selection containers.
|
||||
fn verified_press_native(el: &AXElement) -> bool {
|
||||
fn verified_press_native(el: &AXElement) -> Result<bool, AdapterError> {
|
||||
use accessibility_sys::kAXRoleAttribute;
|
||||
let parent = crate::tree::copy_element_attr(el, "AXParent");
|
||||
let in_container = parent.as_ref().is_some_and(|p| {
|
||||
|
|
@ -27,174 +30,38 @@ mod imp {
|
|||
)
|
||||
});
|
||||
if !in_container {
|
||||
return ax_helpers::try_ax_action_retried(el, "AXPress");
|
||||
return ax_helpers::try_ax_action_retried_or_err(el, "AXPress");
|
||||
}
|
||||
tracing::debug!("verified_press: native element in container, using AXPress");
|
||||
let selected_before = crate::tree::element::copy_bool_attr(el, "AXSelected");
|
||||
if !ax_helpers::try_ax_action_retried(el, "AXPress") {
|
||||
return false;
|
||||
if !ax_helpers::try_ax_action_retried_or_err(el, "AXPress")? {
|
||||
return Ok(false);
|
||||
}
|
||||
if selected_before == Some(true) {
|
||||
return true;
|
||||
return Ok(true);
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||
let selected_after = crate::tree::element::copy_bool_attr(el, "AXSelected");
|
||||
if selected_after == Some(true) {
|
||||
return true;
|
||||
return Ok(true);
|
||||
}
|
||||
if crate::tree::copy_string_attr(el, kAXRoleAttribute).is_none() {
|
||||
return true;
|
||||
return Ok(true);
|
||||
}
|
||||
tracing::debug!("verified_press: AXPress ok but no state change");
|
||||
false
|
||||
Ok(false)
|
||||
}
|
||||
|
||||
fn is_in_webarea(el: &AXElement) -> bool {
|
||||
use accessibility_sys::kAXRoleAttribute;
|
||||
let mut current = crate::tree::copy_element_attr(el, "AXParent");
|
||||
for _ in 0..20 {
|
||||
let Some(ref parent) = current else {
|
||||
return false;
|
||||
};
|
||||
if crate::tree::copy_string_attr(parent, kAXRoleAttribute).as_deref()
|
||||
== Some("AXWebArea")
|
||||
{
|
||||
return true;
|
||||
}
|
||||
current = crate::tree::copy_element_attr(parent, "AXParent");
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
/// Activate any Electron/web element with verified effect detection.
|
||||
///
|
||||
/// Chromium's AX implementation lies: AXPress/AXConfirm return
|
||||
/// kAXErrorSuccess but only toggle ARIA state — DOM click handlers
|
||||
/// never fire. We verify by checking if the focused UI element
|
||||
/// changed after AXPress. If nothing changed, CGClick is the
|
||||
/// genuine last resort.
|
||||
///
|
||||
/// This handles the FULL escalation for web elements so the chain
|
||||
/// engine never reaches false-positive AX steps (AXConfirm, SetBool
|
||||
/// AXSelected, etc. all lie on Chromium).
|
||||
struct PreActionState {
|
||||
focused: Option<AXElement>,
|
||||
value: Option<String>,
|
||||
selected: Option<bool>,
|
||||
}
|
||||
|
||||
impl PreActionState {
|
||||
fn capture(app: &AXElement, el: &AXElement) -> Self {
|
||||
Self {
|
||||
focused: crate::tree::copy_element_attr(app, "AXFocusedUIElement"),
|
||||
value: crate::tree::copy_string_attr(el, "AXValue"),
|
||||
selected: crate::tree::copy_bool_attr(el, "AXSelected"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn activate_web_element(el: &AXElement) -> bool {
|
||||
let Some(pid) = crate::system::app_ops::pid_from_element(el) else {
|
||||
return false;
|
||||
};
|
||||
|
||||
let app = crate::tree::element_for_pid(pid);
|
||||
let before = PreActionState::capture(&app, el);
|
||||
|
||||
if ax_helpers::try_ax_action_retried(el, "AXPress") {
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
if web_action_had_effect(&app, el, &before) {
|
||||
tracing::debug!("activate_web: AXPress had real effect");
|
||||
return true;
|
||||
}
|
||||
tracing::debug!("activate_web: AXPress returned success but no DOM effect");
|
||||
}
|
||||
|
||||
let actions = ax_helpers::list_ax_actions(el);
|
||||
for action in &["AXConfirm", "AXOpen"] {
|
||||
if actions.iter().any(|a| a == action) && ax_helpers::try_ax_action(el, action) {
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
if web_action_had_effect(&app, el, &before) {
|
||||
tracing::debug!("activate_web: {action} had real effect");
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if ax_helpers::try_each_child(
|
||||
el,
|
||||
|child| {
|
||||
let child_actions = ax_helpers::list_ax_actions(child);
|
||||
ax_helpers::try_action_from_list(
|
||||
child,
|
||||
&child_actions,
|
||||
&["AXPress", "AXConfirm", "AXOpen"],
|
||||
)
|
||||
},
|
||||
5,
|
||||
) {
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
if web_action_had_effect(&app, el, &before) {
|
||||
tracing::debug!("activate_web: child action had real effect");
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
tracing::debug!("activate_web: all AX methods had no effect, CGClick");
|
||||
let _ = crate::system::app_ops::ensure_app_focused(pid);
|
||||
crate::actions::dispatch::click_via_bounds(
|
||||
el,
|
||||
agent_desktop_core::action::MouseButton::Left,
|
||||
1,
|
||||
)
|
||||
.is_ok()
|
||||
}
|
||||
|
||||
fn web_action_had_effect(app: &AXElement, el: &AXElement, before: &PreActionState) -> bool {
|
||||
use core_foundation::base::{CFEqual, CFTypeRef};
|
||||
|
||||
let value_after = crate::tree::copy_string_attr(el, "AXValue");
|
||||
if before.value != value_after {
|
||||
return true;
|
||||
}
|
||||
|
||||
let selected_after = crate::tree::copy_bool_attr(el, "AXSelected");
|
||||
if before.selected != selected_after {
|
||||
return true;
|
||||
}
|
||||
|
||||
let focused_after = crate::tree::copy_element_attr(app, "AXFocusedUIElement");
|
||||
match (&before.focused, &focused_after) {
|
||||
(Some(before_f), Some(after_f)) => unsafe {
|
||||
CFEqual(before_f.0 as CFTypeRef, after_f.0 as CFTypeRef) == 0
|
||||
},
|
||||
(None, Some(_)) => true,
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn try_focus_then_verified_confirm_or_press(el: &AXElement, caps: &ElementCaps) -> bool {
|
||||
if !ax_helpers::ax_focus(el) {
|
||||
return false;
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||
let in_web = is_in_webarea(el);
|
||||
if !in_web && ax_helpers::try_ax_action_retried(el, "AXConfirm") {
|
||||
return true;
|
||||
}
|
||||
dispatch_verified_press(el, caps, in_web)
|
||||
}
|
||||
|
||||
pub fn try_value_relay(el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
pub fn try_value_relay(el: &AXElement, _caps: &ElementCaps) -> Result<bool, AdapterError> {
|
||||
if !ax_helpers::list_ax_actions(el).is_empty() {
|
||||
return false;
|
||||
return Ok(false);
|
||||
}
|
||||
let win = crate::tree::copy_element_attr(el, "AXWindow");
|
||||
let is_dialog = win.as_ref().is_some_and(|w| {
|
||||
crate::tree::copy_string_attr(w, "AXSubrole").as_deref() == Some("AXDialog")
|
||||
});
|
||||
if !is_dialog {
|
||||
return false;
|
||||
return Ok(false);
|
||||
}
|
||||
let label = std::cell::RefCell::new(None::<String>);
|
||||
ax_helpers::try_each_child(
|
||||
|
|
@ -210,123 +77,208 @@ mod imp {
|
|||
5,
|
||||
);
|
||||
let Some(label) = label.into_inner() else {
|
||||
return false;
|
||||
return Ok(false);
|
||||
};
|
||||
let Some(pid) = crate::system::app_ops::pid_from_element(el) else {
|
||||
return false;
|
||||
return Ok(false);
|
||||
};
|
||||
let app = crate::tree::element_for_pid(pid);
|
||||
let Some(owner) = crate::tree::copy_element_attr(&app, "AXFocusedUIElement") else {
|
||||
return false;
|
||||
return Ok(false);
|
||||
};
|
||||
if !same_window(&owner, win.as_ref()) {
|
||||
return Ok(false);
|
||||
}
|
||||
if !ax_helpers::is_attr_settable(&owner, "AXValue") {
|
||||
return false;
|
||||
return Ok(false);
|
||||
}
|
||||
let orig = crate::tree::copy_string_attr(&owner, "AXValue");
|
||||
if ax_helpers::set_ax_string_or_err(&owner, "AXValue", &label).is_err() {
|
||||
return false;
|
||||
}
|
||||
ax_helpers::set_ax_string_or_err(&owner, "AXValue", &label)?;
|
||||
std::thread::sleep(std::time::Duration::from_millis(150));
|
||||
if !ax_helpers::try_ax_action(&owner, "AXConfirm") {
|
||||
if !ax_helpers::try_ax_action_retried_or_err(&owner, "AXConfirm")? {
|
||||
if let Some(o) = &orig {
|
||||
let _ = ax_helpers::set_ax_string_or_err(&owner, "AXValue", o);
|
||||
}
|
||||
return false;
|
||||
return Ok(false);
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(150));
|
||||
let final_val = crate::tree::copy_string_attr(&owner, "AXValue");
|
||||
if final_val.as_deref() != Some(label.as_str()) {
|
||||
tracing::debug!("value_relay: reverted to {final_val:?}, expected {label:?}");
|
||||
}
|
||||
final_val.as_deref() == Some(label.as_str())
|
||||
Ok(final_val.as_deref() == Some(label.as_str()))
|
||||
}
|
||||
|
||||
pub fn select_all_then_delete(el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
use accessibility_sys::AXUIElementPostKeyboardEvent as PostKey;
|
||||
if !ax_helpers::ax_focus(el) {
|
||||
return false;
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||
let Some(pid) = crate::system::app_ops::pid_from_element(el) else {
|
||||
fn same_window(owner: &AXElement, expected_window: Option<&AXElement>) -> bool {
|
||||
let Some(expected_window) = expected_window else {
|
||||
return false;
|
||||
};
|
||||
let a = crate::tree::element_for_pid(pid);
|
||||
unsafe {
|
||||
PostKey(a.0, 0, 55, true);
|
||||
PostKey(a.0, 0, 0, true);
|
||||
PostKey(a.0, 0, 0, false);
|
||||
PostKey(a.0, 0, 55, false);
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(30));
|
||||
unsafe {
|
||||
PostKey(a.0, 0, 51, true);
|
||||
PostKey(a.0, 0, 51, false);
|
||||
}
|
||||
true
|
||||
let Some(owner_window) = crate::tree::copy_element_attr(owner, "AXWindow") else {
|
||||
return false;
|
||||
};
|
||||
crate::tree::same_element(&owner_window, expected_window)
|
||||
}
|
||||
|
||||
pub fn walk_parents_and_scroll(el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
pub fn element_is_visible_in_scroll_context(
|
||||
el: &AXElement,
|
||||
_caps: &ElementCaps,
|
||||
) -> Result<bool, AdapterError> {
|
||||
use accessibility_sys::kAXRoleAttribute;
|
||||
let Some(bounds) = crate::tree::read_bounds(el) else {
|
||||
return false;
|
||||
return Ok(false);
|
||||
};
|
||||
if !rect_has_area(&bounds) {
|
||||
return Ok(false);
|
||||
}
|
||||
let mut current = crate::tree::copy_element_attr(el, "AXParent");
|
||||
for _ in 0..8 {
|
||||
let Some(parent) = ¤t else { return false };
|
||||
let Some(parent) = ¤t else {
|
||||
return Ok(true);
|
||||
};
|
||||
if crate::tree::copy_string_attr(parent, kAXRoleAttribute).as_deref()
|
||||
== Some("AXScrollArea")
|
||||
{
|
||||
let Some(pb) = crate::tree::read_bounds(parent) else {
|
||||
return false;
|
||||
return Ok(false);
|
||||
};
|
||||
let ty = bounds.y + bounds.height / 2.0;
|
||||
if ty < pb.y || ty > pb.y + pb.height {
|
||||
let dy = if ty > pb.y + pb.height / 2.0 { -5 } else { 5 };
|
||||
let (cx, cy) = (pb.x + pb.width / 2.0, pb.y + pb.height / 2.0);
|
||||
for _ in 0..20 {
|
||||
let _ = crate::input::mouse::synthesize_scroll_at(cx, cy, dy, 0);
|
||||
std::thread::sleep(std::time::Duration::from_millis(16));
|
||||
}
|
||||
}
|
||||
return true;
|
||||
return Ok(rect_has_area(&pb) && center_is_inside(&bounds, &pb));
|
||||
}
|
||||
current = crate::tree::copy_element_attr(parent, "AXParent");
|
||||
}
|
||||
false
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
pub fn try_show_alternate_ui(el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
pub(crate) fn rect_has_area(rect: &agent_desktop_core::node::Rect) -> bool {
|
||||
rect.width > 0.0 && rect.height > 0.0
|
||||
}
|
||||
|
||||
pub(crate) fn center_is_inside(
|
||||
inner: &agent_desktop_core::node::Rect,
|
||||
outer: &agent_desktop_core::node::Rect,
|
||||
) -> bool {
|
||||
let x = inner.x + inner.width / 2.0;
|
||||
let y = inner.y + inner.height / 2.0;
|
||||
x >= outer.x && x <= outer.x + outer.width && y >= outer.y && y <= outer.y + outer.height
|
||||
}
|
||||
|
||||
pub fn try_show_alternate_ui(
|
||||
el: &AXElement,
|
||||
_caps: &ElementCaps,
|
||||
) -> Result<bool, AdapterError> {
|
||||
if !ax_helpers::has_ax_action(el, "AXShowAlternateUI") {
|
||||
return false;
|
||||
return Ok(false);
|
||||
}
|
||||
ax_helpers::try_ax_action(el, "AXShowAlternateUI");
|
||||
ax_helpers::try_ax_action_retried_or_err(el, "AXShowAlternateUI")?;
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
ax_helpers::try_each_child(
|
||||
Ok(ax_helpers::try_each_child(
|
||||
el,
|
||||
|child| {
|
||||
let ca = ax_helpers::list_ax_actions(child);
|
||||
ax_helpers::try_action_from_list(child, &ca, &["AXPress"])
|
||||
},
|
||||
5,
|
||||
)
|
||||
))
|
||||
}
|
||||
|
||||
pub fn try_parent_row_select(el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
pub fn try_parent_row_select(
|
||||
el: &AXElement,
|
||||
_caps: &ElementCaps,
|
||||
) -> Result<bool, AdapterError> {
|
||||
use accessibility_sys::kAXRoleAttribute;
|
||||
let Some(parent) = crate::tree::copy_element_attr(el, "AXParent") else {
|
||||
return false;
|
||||
return Ok(false);
|
||||
};
|
||||
let role = crate::tree::copy_string_attr(&parent, kAXRoleAttribute).unwrap_or_default();
|
||||
if !matches!(role.as_str(), "AXRow" | "AXOutlineRow") {
|
||||
return false;
|
||||
return Ok(false);
|
||||
}
|
||||
if !ax_helpers::is_attr_settable(&parent, "AXSelected") {
|
||||
return false;
|
||||
return Ok(false);
|
||||
}
|
||||
ax_helpers::set_ax_bool(&parent, "AXSelected", true)
|
||||
ax_helpers::set_ax_bool_or_err(&parent, "AXSelected", true)
|
||||
}
|
||||
|
||||
pub fn try_select_via_parent(el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
pub fn try_select_containing_item(
|
||||
el: &AXElement,
|
||||
_caps: &ElementCaps,
|
||||
) -> Result<bool, AdapterError> {
|
||||
let mut current = Some(el.clone());
|
||||
for _ in 0..4 {
|
||||
let Some(candidate) = current else {
|
||||
return Ok(false);
|
||||
};
|
||||
if select_candidate(&candidate)? || select_candidate_in_container(&candidate) {
|
||||
return Ok(true);
|
||||
}
|
||||
current = crate::tree::copy_element_attr(&candidate, "AXParent");
|
||||
}
|
||||
Ok(false)
|
||||
}
|
||||
|
||||
fn select_candidate(candidate: &AXElement) -> Result<bool, AdapterError> {
|
||||
Ok(ax_helpers::is_attr_settable(candidate, "AXSelected")
|
||||
&& ax_helpers::set_ax_bool_or_err(candidate, "AXSelected", true)?
|
||||
&& selected_state_settled(candidate))
|
||||
}
|
||||
|
||||
fn selected_state_settled(candidate: &AXElement) -> bool {
|
||||
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||
crate::tree::copy_bool_attr(candidate, "AXSelected") == Some(true)
|
||||
}
|
||||
|
||||
fn select_candidate_in_container(candidate: &AXElement) -> bool {
|
||||
for attr in ["AXSelectedChildren", "AXSelectedRows"] {
|
||||
if set_container_selection(candidate, attr)
|
||||
&& container_selection_contains(candidate, attr)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
fn set_container_selection(candidate: &AXElement, attr: &str) -> bool {
|
||||
use accessibility_sys::{kAXErrorSuccess, AXUIElementSetAttributeValue};
|
||||
use core_foundation::{
|
||||
array::CFArray,
|
||||
base::{CFRetain, CFType, CFTypeRef, TCFType},
|
||||
string::CFString,
|
||||
};
|
||||
let Some(container) = crate::tree::copy_element_attr(candidate, "AXParent") else {
|
||||
return false;
|
||||
};
|
||||
if !ax_helpers::is_attr_settable(&container, attr) {
|
||||
return false;
|
||||
}
|
||||
unsafe { CFRetain(candidate.0 as CFTypeRef) };
|
||||
let candidate_cf = unsafe { CFType::wrap_under_create_rule(candidate.0 as CFTypeRef) };
|
||||
let selected = CFArray::from_CFTypes(&[candidate_cf]);
|
||||
let cf_attr = CFString::new(attr);
|
||||
let err = unsafe {
|
||||
AXUIElementSetAttributeValue(
|
||||
container.0,
|
||||
cf_attr.as_concrete_TypeRef(),
|
||||
selected.as_CFTypeRef(),
|
||||
)
|
||||
};
|
||||
err == kAXErrorSuccess
|
||||
}
|
||||
|
||||
fn container_selection_contains(candidate: &AXElement, attr: &str) -> bool {
|
||||
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||
let Some(container) = crate::tree::copy_element_attr(candidate, "AXParent") else {
|
||||
return false;
|
||||
};
|
||||
crate::tree::copy_ax_array(&container, attr)
|
||||
.unwrap_or_default()
|
||||
.iter()
|
||||
.any(|selected| crate::tree::same_element(selected, candidate))
|
||||
}
|
||||
|
||||
pub fn try_select_via_parent(
|
||||
el: &AXElement,
|
||||
_caps: &ElementCaps,
|
||||
) -> Result<bool, AdapterError> {
|
||||
use accessibility_sys::{kAXErrorSuccess, kAXRoleAttribute, AXUIElementSetAttributeValue};
|
||||
use core_foundation::{
|
||||
array::CFArray,
|
||||
|
|
@ -334,16 +286,16 @@ mod imp {
|
|||
string::CFString,
|
||||
};
|
||||
let Some(parent) = crate::tree::copy_element_attr(el, "AXParent") else {
|
||||
return false;
|
||||
return Ok(false);
|
||||
};
|
||||
let Some(role) = crate::tree::copy_string_attr(&parent, kAXRoleAttribute) else {
|
||||
return false;
|
||||
return Ok(false);
|
||||
};
|
||||
if !matches!(role.as_str(), "AXTable" | "AXOutline" | "AXList") {
|
||||
return false;
|
||||
return Ok(false);
|
||||
}
|
||||
if !ax_helpers::is_attr_settable(&parent, "AXSelectedRows") {
|
||||
return false;
|
||||
return Ok(false);
|
||||
}
|
||||
unsafe { CFRetain(el.0 as CFTypeRef) };
|
||||
let el_cf = unsafe { CFType::wrap_under_create_rule(el.0 as CFTypeRef) };
|
||||
|
|
@ -356,50 +308,14 @@ mod imp {
|
|||
arr.as_CFTypeRef(),
|
||||
)
|
||||
};
|
||||
err == kAXErrorSuccess
|
||||
Ok(err == kAXErrorSuccess)
|
||||
}
|
||||
|
||||
pub fn try_custom_actions(el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
pub fn try_custom_actions(el: &AXElement, _caps: &ElementCaps) -> Result<bool, AdapterError> {
|
||||
let has = !crate::tree::copy_ax_array(el, "AXCustomActions")
|
||||
.unwrap_or_default()
|
||||
.is_empty();
|
||||
has && ax_helpers::try_ax_action(el, "AXPerformCustomAction")
|
||||
}
|
||||
|
||||
pub fn try_keyboard_activate(el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
use accessibility_sys::AXUIElementPostKeyboardEvent as PostKey;
|
||||
if !ax_helpers::ax_focus(el) {
|
||||
return false;
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||
let Some(pid) = crate::system::app_ops::pid_from_element(el) else {
|
||||
return false;
|
||||
};
|
||||
let a = crate::tree::element_for_pid(pid);
|
||||
unsafe {
|
||||
PostKey(a.0, 0, 49, true);
|
||||
PostKey(a.0, 0, 49, false);
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
pub fn focus_app_then_show_menu(el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
let Some(pid) = crate::system::app_ops::pid_from_element(el) else {
|
||||
return false;
|
||||
};
|
||||
let _ = crate::system::app_ops::ensure_app_focused(pid);
|
||||
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||
ax_helpers::try_ax_action(el, "AXShowMenu")
|
||||
}
|
||||
|
||||
pub fn select_then_show_menu(el: &AXElement, _caps: &ElementCaps) -> bool {
|
||||
if !ax_helpers::is_attr_settable(el, "AXSelected")
|
||||
|| !ax_helpers::set_ax_bool(el, "AXSelected", true)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||
ax_helpers::try_ax_action(el, "AXShowMenu")
|
||||
Ok(has && ax_helpers::try_ax_action_retried_or_err(el, "AXPerformCustomAction")?)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
47
crates/macos/src/actions/chain_steps_tests.rs
Normal file
47
crates/macos/src/actions/chain_steps_tests.rs
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
use agent_desktop_core::node::Rect;
|
||||
|
||||
use super::chain_steps::{center_is_inside, rect_has_area};
|
||||
|
||||
#[test]
|
||||
fn rect_area_requires_positive_dimensions() {
|
||||
assert!(rect_has_area(&Rect {
|
||||
x: 0.0,
|
||||
y: 0.0,
|
||||
width: 1.0,
|
||||
height: 1.0,
|
||||
}));
|
||||
assert!(!rect_has_area(&Rect {
|
||||
x: 0.0,
|
||||
y: 0.0,
|
||||
width: 0.0,
|
||||
height: 1.0,
|
||||
}));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn center_visibility_uses_parent_bounds() {
|
||||
let outer = Rect {
|
||||
x: 10.0,
|
||||
y: 10.0,
|
||||
width: 100.0,
|
||||
height: 100.0,
|
||||
};
|
||||
assert!(center_is_inside(
|
||||
&Rect {
|
||||
x: 20.0,
|
||||
y: 20.0,
|
||||
width: 10.0,
|
||||
height: 10.0,
|
||||
},
|
||||
&outer
|
||||
));
|
||||
assert!(!center_is_inside(
|
||||
&Rect {
|
||||
x: 200.0,
|
||||
y: 20.0,
|
||||
width: 10.0,
|
||||
height: 10.0,
|
||||
},
|
||||
&outer
|
||||
));
|
||||
}
|
||||
118
crates/macos/src/actions/chain_web_steps.rs
Normal file
118
crates/macos/src/actions/chain_web_steps.rs
Normal file
|
|
@ -0,0 +1,118 @@
|
|||
#[cfg(target_os = "macos")]
|
||||
mod imp {
|
||||
use crate::actions::ax_helpers;
|
||||
use crate::tree::AXElement;
|
||||
|
||||
struct PreActionState {
|
||||
focused: Option<AXElement>,
|
||||
value: Option<String>,
|
||||
selected: Option<bool>,
|
||||
}
|
||||
|
||||
impl PreActionState {
|
||||
fn capture(app: &AXElement, el: &AXElement) -> Self {
|
||||
Self {
|
||||
focused: crate::tree::copy_element_attr(app, "AXFocusedUIElement"),
|
||||
value: crate::tree::copy_string_attr(el, "AXValue"),
|
||||
selected: crate::tree::copy_bool_attr(el, "AXSelected"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_in_webarea(el: &AXElement) -> bool {
|
||||
use accessibility_sys::kAXRoleAttribute;
|
||||
let mut current = crate::tree::copy_element_attr(el, "AXParent");
|
||||
for _ in 0..20 {
|
||||
let Some(ref parent) = current else {
|
||||
return false;
|
||||
};
|
||||
if crate::tree::copy_string_attr(parent, kAXRoleAttribute).as_deref()
|
||||
== Some("AXWebArea")
|
||||
{
|
||||
return true;
|
||||
}
|
||||
current = crate::tree::copy_element_attr(parent, "AXParent");
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
pub fn activate_web_element(el: &AXElement) -> bool {
|
||||
let Some(pid) = crate::system::app_ops::pid_from_element(el) else {
|
||||
return false;
|
||||
};
|
||||
|
||||
let app = crate::tree::element_for_pid(pid);
|
||||
let before = PreActionState::capture(&app, el);
|
||||
|
||||
if ax_helpers::try_ax_action_retried(el, "AXPress") {
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
if web_action_had_effect(&app, el, &before) {
|
||||
tracing::debug!("activate_web: AXPress had real effect");
|
||||
return true;
|
||||
}
|
||||
tracing::debug!("activate_web: AXPress returned success but no DOM effect");
|
||||
}
|
||||
|
||||
let actions = ax_helpers::list_ax_actions(el);
|
||||
for action in &["AXConfirm", "AXOpen"] {
|
||||
if actions.iter().any(|a| a == action) && ax_helpers::try_ax_action(el, action) {
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
if web_action_had_effect(&app, el, &before) {
|
||||
tracing::debug!("activate_web: {action} had real effect");
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if ax_helpers::try_each_child(
|
||||
el,
|
||||
|child| {
|
||||
let child_actions = ax_helpers::list_ax_actions(child);
|
||||
ax_helpers::try_action_from_list(
|
||||
child,
|
||||
&child_actions,
|
||||
&["AXPress", "AXConfirm", "AXOpen"],
|
||||
)
|
||||
},
|
||||
5,
|
||||
) {
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
if web_action_had_effect(&app, el, &before) {
|
||||
tracing::debug!("activate_web: child action had real effect");
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
tracing::debug!("activate_web: all AX methods had no effect");
|
||||
false
|
||||
}
|
||||
|
||||
fn web_action_had_effect(app: &AXElement, el: &AXElement, before: &PreActionState) -> bool {
|
||||
use core_foundation::base::{CFEqual, CFTypeRef};
|
||||
|
||||
let value_after = crate::tree::copy_string_attr(el, "AXValue");
|
||||
if before.value != value_after {
|
||||
return true;
|
||||
}
|
||||
|
||||
let selected_after = crate::tree::copy_bool_attr(el, "AXSelected");
|
||||
if before.selected != selected_after {
|
||||
return true;
|
||||
}
|
||||
|
||||
let focused_after = crate::tree::copy_element_attr(app, "AXFocusedUIElement");
|
||||
match (&before.focused, &focused_after) {
|
||||
(Some(before_f), Some(after_f)) => unsafe {
|
||||
CFEqual(before_f.0 as CFTypeRef, after_f.0 as CFTypeRef) == 0
|
||||
},
|
||||
(None, Some(_)) => true,
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
mod imp {}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
pub(crate) use imp::*;
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue