Creepso/agent-desktop
1
0
Fork 0
mirror of https://github.com/lahfir/agent-desktop.git synced 2026-08-20 05:57:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

feat!: decide macOS delivery by observation and stop launch waiting on an uncaused event (#125)

Browse source 
macOS actions now decide delivery from what the application did rather than
from what its return codes claim, because those codes lie in both directions:
Finder answers an error to an AXOpen that navigated, and success to an
AXConfirm that did nothing.

A row that publishes no activation of its own is activated through the cell
that carries it, so a click no longer falls through to writing selection and
reporting a success the application never performed.

launch returns once its process is running. A document-based application
creates its first window in response to being brought forward, so waiting for
one without asking waits for an event that cannot fire. The wait is bounded by
the application's own startup instead of the deadline, --activate asks for a
window and waits for it, and a launch whose process exits reports that instead
of a windowless success. Launching TextEdit goes from 30.06s to 1.3s.

Accessibility elements are compared by CFEqual identity. A raw pointer answers
neither question the comparison asks: the framework returns a fresh reference
for the same element and reuses a released one for a different element.

find gains --root for element-scoped drill-down and --surface for menu bars and
other overlays, cutting a targeted lookup from a full-tree dump to a scoped
read.

BREAKING CHANGE: the response envelope is version 2.3. launch returns
{ app, pid, process_instance, window? } instead of a bare window object, and an
application that presents no window is ok:true with window omitted rather than
WINDOW_NOT_FOUND. The C ABI is unchanged: it still writes one window and
reports WINDOW_NOT_FOUND when there is none.
This commit is contained in:
Lahfir 2026-08-13 16:44:53 -07:00 committed by GitHub
parent 015307e7b9
commit 298f1ff215
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
133 changed files with 1963 additions and 538 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
CLAUDE.md
View file

@ -286,7 +286,7 @@ Every command produces a response envelope:
```json
{
"version": "2.2",
"version": "2.3",
"ok": true,
"command": "snapshot",
"data": {
@ -302,7 +302,7 @@ Error responses:
```json
{
"version": "2.2",
"version": "2.3",
"ok": false,
"command": "click",
"error": {

2
crates/core/examples/locator_benchmark/adapter.rs
View file

@ -39,7 +39,7 @@ impl ObservationOps for FixtureAdapter<'_> {
live_target_tree(
self.fixture,
index,
ObservationSource::from_root(&root),
ObservationSource::from_root(&root, request.surface),
request,
)
}

5
crates/core/examples/locator_benchmark/fixture_builder.rs
View file

@ -13,7 +13,10 @@ pub(crate) fn live_tree(
live_tree_from_roots(
fixture,
&fixture.roots,
ObservationSource::Window(fixture.window.clone()),
ObservationSource::Window {
window: fixture.window.clone(),
surface: agent_desktop_core::SnapshotSurface::Window,
},
requirements,
)
}

1
crates/core/examples/locator_benchmark/live.rs
View file

@ -66,6 +66,7 @@ fn run_live(
selection,
deadline: Deadline::after(5_000)?,
max_raw_depth: 50,
surface: None,
materialization,
};
let adapter = FixtureAdapter { fixture };

16
crates/core/src/action.rs
View file

@ -96,6 +96,22 @@ impl Action {
)
}
/// Whether the action can leave the application showing a sheet, menu, or
/// alert. Listing surfaces costs a walk of the application's overlays, so
/// only the actions that can raise one pay for it.
pub fn may_raise_surface(&self) -> bool {
matches!(
self,
Self::Click
| Self::DoubleClick
| Self::RightClick
| Self::TripleClick
| Self::Expand
| Self::Select(_)
| Self::PressKey(_)
)
}
pub fn requires_scroll_into_view(&self) -> bool {
matches!(
self,

7
crates/core/src/action_result.rs
View file

@ -11,6 +11,11 @@ pub struct ActionResult {
pub post_state: Option<ElementState>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub steps: Vec<ActionStep>,
/// Overlays the application had open once the action settled. An action can
/// leave a sheet, menu, or alert on screen, and without this the caller has
/// to go hunting through windows to discover it.
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub surfaces: Vec<crate::SurfaceInfo>,
#[serde(skip_serializing_if = "Option::is_none")]
pub details: Option<serde_json::Value>,
#[serde(
@ -49,6 +54,7 @@ impl ActionResult {
action: action.into(),
post_state: None,
steps: Vec::new(),
surfaces: Vec::new(),
details: None,
disposition: DeliverySemantics::not_delivered(),
}
@ -59,6 +65,7 @@ impl ActionResult {
action: action.into(),
post_state: None,
steps: Vec::new(),
surfaces: Vec::new(),
details: None,
disposition: default_action_disposition(),
}

8
crates/core/src/actionability/gates.rs
View file

@ -19,7 +19,13 @@ pub(super) fn visibility(evidence: &ActionabilityEvidence) -> ActionabilityCheck
}
match evidence.state.offscreen {
Some(true) => return fail("visible", "live offscreen state is true"),
None => return unknown("visible", "live offscreen state unavailable"),
None if !evidence.states_complete => {
return unknown("visible", "live offscreen state unavailable");
}
None if crate::state::has_state(&evidence.state.states, crate::state::OFFSCREEN) => {
return fail("visible", "canonical offscreen state is present");
}
None => {}
Some(false) => {}
}
let Some(bounds) = evidence.bounds else {

2
crates/core/src/adapter/system.rs
View file

@ -66,7 +66,7 @@ pub trait SystemOps: Send + Sync {
_id: &str,
_options: &crate::launch_options::LaunchOptions,
_lease: &InteractionLease,
) -> Result<WindowInfo, AdapterError> {
) -> Result<crate::launch_result::LaunchResult, AdapterError> {
Err(AdapterError::not_supported("launch_app"))
}

2
crates/core/src/adapter/test_support.rs
View file

@ -161,7 +161,7 @@ pub(crate) fn observed_tree(
ObservedTree::from_roots(
vec![subtree(node)],
ObservationSource::from_root(root),
ObservationSource::from_root(root, root.surface()),
LocatorStats::default(),
true,
)

15
crates/core/src/app_info.rs
View file

@ -2,6 +2,19 @@ use serde::{Deserialize, Serialize};
use crate::ProcessId;
/// How an application presents itself to the user, so an agent can tell a
/// window-owning app from one that only appears on a hotkey or lives in the
/// menu bar or tray.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum AppPresentation {
/// Owns ordinary windows and appears in the Dock or taskbar.
Foreground,
/// No Dock or taskbar entry. Menu bar and tray items live here, as do
/// overlays summoned by a hotkey; their windows may exist only while shown.
Background,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AppInfo {
pub name: String,
@ -10,4 +23,6 @@ pub struct AppInfo {
pub bundle_id: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub process_instance: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub presentation: Option<AppPresentation>,
}

1
crates/core/src/app_lookup_tests.rs
View file

@ -34,6 +34,7 @@ fn app(instance: Option<&str>) -> AppInfo {
pid: crate::ProcessId::new(42),
bundle_id: Some("com.example.app".into()),
process_instance: instance.map(str::to_string),
presentation: None,
}
}

2
crates/core/src/commands/close_app_tests.rs
View file

@ -11,6 +11,7 @@ impl ObservationOps for ProtectiveAdapter {
pid: crate::ProcessId::new(42),
bundle_id: Some("com.apple.TextEdit".into()),
process_instance: Some("textedit-instance".into()),
presentation: None,
}])
}
}
@ -45,6 +46,7 @@ impl ObservationOps for FailingAdapter {
pid: crate::ProcessId::new(77),
bundle_id: None,
process_instance: Some("ghost-instance".into()),
presentation: None,
}])
}
}

9
crates/core/src/commands/find.rs
View file

@ -39,6 +39,9 @@ pub struct FindSelectionArgs {
pub struct FindArgs {
pub app: Option<String>,
pub window_id: Option<String>,
pub root: Option<String>,
pub snapshot: Option<String>,
pub surface: crate::SnapshotSurface,
pub filter: FindFilterArgs,
pub states: Vec<StatePredicate>,
pub selection: FindSelectionArgs,
@ -50,6 +53,8 @@ pub fn execute(
context: &CommandContext,
) -> Result<Value, AppError> {
validate_find_mode(&args)?;
super::surface_scope::reject_root_with_surface("find", args.root.as_deref(), args.surface)?;
super::surface_scope::require_supported(args.surface, adapter)?;
let query = locator_query_from_args(&args)?;
query.validate_states().map_err(AppError::Adapter)?;
@ -230,6 +235,10 @@ mod live;
#[path = "find_tests.rs"]
mod tests;
#[cfg(test)]
#[path = "find_live_test_support.rs"]
mod test_support;
#[cfg(test)]
#[path = "find_live_tests.rs"]
mod live_tests;

43
crates/core/src/commands/find_live.rs
View file

@ -24,14 +24,39 @@ pub(super) fn execute(
context: &CommandContext,
) -> Result<Value, AppError> {
let deadline = crate::Deadline::from_duration(LOCATOR_TIMEOUT)?;
let window = snapshot::resolve_window(
adapter,
args.app.as_deref(),
args.window_id.as_deref(),
deadline,
)?;
let request = resolve_request(args, deadline);
let mut resolution = resolve_query(adapter, query, ObservationRoot::Window(&window), &request)?;
let mut resolution = match args.root.as_deref() {
Some(root_ref) => {
let (_, local_root_ref) =
crate::ref_token::resolve_ref_target(root_ref, args.snapshot.as_deref())?;
let entry = crate::commands::helpers::load_ref_entry(
root_ref,
args.snapshot.as_deref(),
context,
)?;
let handle = adapter.resolve_element_strict(&entry, deadline)?;
resolve_query(
adapter,
query,
ObservationRoot::Element {
handle: &handle,
entry: &entry,
root_ref: Some(&local_root_ref),
},
&request,
)?
}
None => {
let window = snapshot::resolve_window_for_surface(
adapter,
args.app.as_deref(),
args.window_id.as_deref(),
args.surface,
deadline,
)?;
resolve_query(adapter, query, ObservationRoot::Window(&window), &request)?
}
};
require_complete(&resolution)?;
let ref_count = resolution.refmap.as_ref().map(RefMap::len);
let snapshot_id = match resolution.refmap.take() {
@ -72,6 +97,7 @@ fn resolve_request(args: &FindArgs, deadline: crate::Deadline) -> LocatorResolve
selection,
deadline,
max_raw_depth: MAX_RAW_DEPTH,
surface: (args.surface != crate::SnapshotSurface::Window).then_some(args.surface),
materialization: if args.selection.count {
LocatorMaterialization::None
} else {
@ -194,6 +220,9 @@ mod tests {
FindArgs {
app: None,
window_id: None,
root: None,
snapshot: None,
surface: crate::SnapshotSurface::Window,
filter: crate::commands::find::FindFilterArgs {
role: None,
name: None,

134
crates/core/src/commands/find_live_test_support.rs Normal file
View file

@ -0,0 +1,134 @@
use crate::{
AdapterError, WindowInfo,
adapter::{ActionOps, InputOps, NativeHandle, ObservationOps, SystemOps, WindowFilter},
live_locator::{
IdentifierEvidence, LocatorEvidence, LocatorField, LocatorRefEvidence, LocatorStats,
ObservationRequest, ObservationRoot, ObservationSource, ObservedSubtree, ObservedTree,
},
};
pub(crate) struct LiveFindAdapter {
structurally_complete: bool,
}
impl LiveFindAdapter {
pub(crate) fn complete() -> Self {
Self {
structurally_complete: true,
}
}
pub(crate) fn incomplete() -> Self {
Self {
structurally_complete: false,
}
}
fn evidence(role: &str, name: Option<&str>) -> LocatorEvidence {
LocatorEvidence {
role: LocatorField::Known(role.into()),
name: name
.map(|value| LocatorField::Known(value.into()))
.unwrap_or(LocatorField::Absent),
description: LocatorField::Absent,
value: LocatorField::Absent,
identifiers: IdentifierEvidence::absent(),
states: LocatorField::Known(Vec::new()),
ref_evidence: LocatorRefEvidence {
bounds: LocatorField::Absent,
available_actions: LocatorField::Known(Vec::new()),
},
}
}
fn node(&self, evidence: LocatorEvidence, children: Vec<ObservedSubtree>) -> ObservedSubtree {
ObservedSubtree::new(evidence, children, self.structurally_complete, None)
}
}
impl ObservationOps for LiveFindAdapter {
fn observe_tree(
&self,
root: ObservationRoot<'_>,
request: &ObservationRequest,
) -> Result<ObservedTree, AdapterError> {
if let ObservationRoot::Element { entry, .. } = &root {
return ObservedTree::from_roots(
vec![self.node(
Self::evidence(&entry.identity.role, entry.identity.name.as_deref()),
Vec::new(),
)],
ObservationSource::from_root(&root, request.surface),
LocatorStats::default(),
self.structurally_complete,
);
}
let ObservationRoot::Window(window) = &root else {
return Err(AdapterError::internal("expected locator root"));
};
let window = *window;
let marker = if window.id == "w-2" {
"OnlyInWindowTwo"
} else {
"OnlyInWindowOne"
};
let child = self.node(Self::evidence("button", Some(marker)), Vec::new());
let root_node = self.node(Self::evidence("window", Some(&window.title)), vec![child]);
ObservedTree::from_roots(
vec![root_node],
ObservationSource::from_root(&root, request.surface),
LocatorStats::default(),
self.structurally_complete,
)
}
fn list_windows(
&self,
_filter: &WindowFilter,
_deadline: crate::Deadline,
) -> Result<Vec<WindowInfo>, AdapterError> {
Ok(vec![
WindowInfo {
id: "w-1".into(),
title: "First".into(),
app: "FixtureApp".into(),
pid: crate::ProcessId::new(101),
process_instance: Some("test-instance".into()),
bounds: None,
state: crate::WindowState {
is_focused: true,
..Default::default()
},
},
WindowInfo {
id: "w-2".into(),
title: "Second".into(),
app: "FixtureApp".into(),
pid: crate::ProcessId::new(102),
process_instance: Some("test-instance".into()),
bounds: None,
state: crate::WindowState {
is_focused: false,
..Default::default()
},
},
])
}
fn resolve_locator_anchor(
&self,
_entry: &crate::refs::RefEntry,
_deadline: crate::Deadline,
) -> Result<NativeHandle, AdapterError> {
Ok(NativeHandle::null())
}
}
impl ActionOps for LiveFindAdapter {}
impl InputOps for LiveFindAdapter {}
impl SystemOps for LiveFindAdapter {
fn supported_surfaces(&self) -> Vec<crate::SnapshotSurface> {
vec![crate::SnapshotSurface::Window]
}
}

137
crates/core/src/commands/find_live_tests.rs
View file

@ -1,142 +1,23 @@
use super::test_support::LiveFindAdapter;
use super::*;
use crate::{
AdapterError, WindowInfo,
adapter::{ActionOps, InputOps, NativeHandle, ObservationOps, SystemOps, WindowFilter},
AppError,
adapter::{ObservationOps, WindowFilter},
live_locator::{
IdentifierEvidence, LocatorEvidence, LocatorField, LocatorMaterialization,
LocatorRefEvidence, LocatorResolveRequest, LocatorSelection, LocatorStats,
ObservationRequest, ObservationRoot, ObservationSource, ObservedSubtree, ObservedTree,
LocatorMaterialization, LocatorResolveRequest, LocatorSelection, ObservationRoot,
require_unique, resolve_query,
},
refs_store::RefStore,
refs_test_support::HomeGuard,
};
pub(super) struct LiveFindAdapter {
structurally_complete: bool,
}
impl LiveFindAdapter {
pub(super) fn complete() -> Self {
Self {
structurally_complete: true,
}
}
fn incomplete() -> Self {
Self {
structurally_complete: false,
}
}
fn evidence(role: &str, name: Option<&str>) -> LocatorEvidence {
LocatorEvidence {
role: LocatorField::Known(role.into()),
name: name
.map(|value| LocatorField::Known(value.into()))
.unwrap_or(LocatorField::Absent),
description: LocatorField::Absent,
value: LocatorField::Absent,
identifiers: IdentifierEvidence::absent(),
states: LocatorField::Known(Vec::new()),
ref_evidence: LocatorRefEvidence {
bounds: LocatorField::Absent,
available_actions: LocatorField::Known(Vec::new()),
},
}
}
fn node(&self, evidence: LocatorEvidence, children: Vec<ObservedSubtree>) -> ObservedSubtree {
ObservedSubtree::new(evidence, children, self.structurally_complete, None)
}
}
impl ObservationOps for LiveFindAdapter {
fn observe_tree(
&self,
root: ObservationRoot<'_>,
_request: &ObservationRequest,
) -> Result<ObservedTree, AdapterError> {
if let ObservationRoot::Element { entry, .. } = &root {
return ObservedTree::from_roots(
vec![self.node(
Self::evidence(&entry.identity.role, entry.identity.name.as_deref()),
Vec::new(),
)],
ObservationSource::from_root(&root),
LocatorStats::default(),
self.structurally_complete,
);
}
let ObservationRoot::Window(window) = &root else {
return Err(AdapterError::internal("expected locator root"));
};
let window = *window;
let marker = if window.id == "w-2" {
"OnlyInWindowTwo"
} else {
"OnlyInWindowOne"
};
let child = self.node(Self::evidence("button", Some(marker)), Vec::new());
let root_node = self.node(Self::evidence("window", Some(&window.title)), vec![child]);
ObservedTree::from_roots(
vec![root_node],
ObservationSource::from_root(&root),
LocatorStats::default(),
self.structurally_complete,
)
}
fn list_windows(
&self,
_filter: &WindowFilter,
_deadline: crate::Deadline,
) -> Result<Vec<WindowInfo>, AdapterError> {
Ok(vec![
WindowInfo {
id: "w-1".into(),
title: "First".into(),
app: "FixtureApp".into(),
pid: crate::ProcessId::new(101),
process_instance: Some("test-instance".into()),
bounds: None,
state: crate::WindowState {
is_focused: true,
..Default::default()
},
},
WindowInfo {
id: "w-2".into(),
title: "Second".into(),
app: "FixtureApp".into(),
pid: crate::ProcessId::new(102),
process_instance: Some("test-instance".into()),
bounds: None,
state: crate::WindowState {
is_focused: false,
..Default::default()
},
},
])
}
fn resolve_locator_anchor(
&self,
_entry: &crate::refs::RefEntry,
_deadline: crate::Deadline,
) -> Result<NativeHandle, AdapterError> {
Ok(NativeHandle::null())
}
}
impl ActionOps for LiveFindAdapter {}
impl InputOps for LiveFindAdapter {}
impl SystemOps for LiveFindAdapter {}
fn named_find(window_id: &str, name: &str) -> FindArgs {
FindArgs {
app: None,
window_id: Some(window_id.into()),
root: None,
snapshot: None,
surface: crate::SnapshotSurface::Window,
filter: FindFilterArgs {
name: Some(name.into()),
role: None,
@ -161,6 +42,9 @@ fn unfiltered_find(window_id: &str, selection: FindSelectionArgs) -> FindArgs {
FindArgs {
app: None,
window_id: Some(window_id.into()),
root: None,
snapshot: None,
surface: crate::SnapshotSurface::Window,
filter: FindFilterArgs {
role: None,
name: None,
@ -366,6 +250,7 @@ fn strict_live_resolution_reports_bounded_ambiguous_candidates() {
selection: LocatorSelection::Strict,
deadline: crate::Deadline::from_duration(std::time::Duration::from_secs(1)).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::FullRefMap,
};
let resolution = resolve_query(

20
crates/core/src/commands/find_tests.rs
View file

@ -130,6 +130,9 @@ fn limit_conflicts_with_single_result_modes_for_batch_too() {
let err = validate_find_mode(&FindArgs {
app: None,
window_id: None,
root: None,
snapshot: None,
surface: crate::SnapshotSurface::Window,
filter: no_filter(),
states: vec![],
selection: FindSelectionArgs {
@ -176,6 +179,9 @@ fn role_alias_is_preserved_until_live_validation() {
let query = query_from_args(&FindArgs {
app: None,
window_id: None,
root: None,
snapshot: None,
surface: crate::SnapshotSurface::Window,
filter: FindFilterArgs {
role: Some("textarea".into()),
..no_filter()
@ -197,6 +203,9 @@ fn unknown_role_is_preserved_until_validation() {
let query = query_from_args(&FindArgs {
app: None,
window_id: None,
root: None,
snapshot: None,
surface: crate::SnapshotSurface::Window,
filter: FindFilterArgs {
role: Some("navbar".into()),
..no_filter()
@ -224,6 +233,9 @@ fn empty_role_filtered_result_reports_roles_present_from_tree() {
let query = query_from_args(&FindArgs {
app: None,
window_id: None,
root: None,
snapshot: None,
surface: crate::SnapshotSurface::Window,
filter: FindFilterArgs {
role: Some("navbar".into()),
..no_filter()
@ -247,6 +259,9 @@ fn roles_present_hint_is_omitted_when_a_match_is_found() {
let query = query_from_args(&FindArgs {
app: None,
window_id: None,
root: None,
snapshot: None,
surface: crate::SnapshotSurface::Window,
filter: FindFilterArgs {
role: Some("textfield".into()),
..no_filter()
@ -268,6 +283,9 @@ fn find_args_scoped_to_window(window_id: &str) -> FindArgs {
FindArgs {
app: None,
window_id: Some(window_id.into()),
root: None,
snapshot: None,
surface: crate::SnapshotSurface::Window,
filter: FindFilterArgs {
name: Some("OnlyInWindowTwo".into()),
..no_filter()
@ -281,7 +299,7 @@ fn find_args_scoped_to_window(window_id: &str) -> FindArgs {
fn find_scopes_matches_to_requested_window_id() {
let _guard = HomeGuard::new();
let context = CommandContext::default();
let adapter = super::live_tests::LiveFindAdapter::complete();
let adapter = super::test_support::LiveFindAdapter::complete();
let from_window_two = execute(find_args_scoped_to_window("w-2"), &adapter, &context)
.expect("find scoped to w-2 should succeed");

1
crates/core/src/commands/helpers_tests.rs
View file

@ -110,6 +110,7 @@ impl ObservationOps for RestoreWithoutWindowAdapter {
pid: crate::ProcessId::new(42),
bundle_id: None,
process_instance: Some("test-instance".into()),
presentation: None,
}])
}

4
crates/core/src/commands/launch.rs
View file

@ -14,8 +14,8 @@ pub fn execute(args: LaunchArgs, adapter: &dyn PlatformAdapter) -> Result<Value,
crate::Deadline::after(args.options.timeout_ms)?
};
let lease = adapter.acquire_interaction_lease(deadline)?;
let window = adapter.launch_app(&args.app, &args.options, &lease)?;
Ok(serde_json::to_value(window)?)
let launched = adapter.launch_app(&args.app, &args.options, &lease)?;
Ok(serde_json::to_value(launched)?)
}
#[cfg(test)]

17
crates/core/src/commands/launch_tests.rs
View file

@ -27,15 +27,20 @@ impl SystemOps for LaunchAdapter {
_id: &str,
_options: &LaunchOptions,
_lease: &InteractionLease,
) -> Result<WindowInfo, AdapterError> {
Ok(WindowInfo {
id: "w-1".into(),
title: "Fixture".into(),
) -> Result<crate::launch_result::LaunchResult, AdapterError> {
Ok(crate::launch_result::LaunchResult {
app: "Fixture".into(),
pid: ProcessId::new(42),
process_instance: Some("42:1".into()),
bounds: None,
state: WindowState::default(),
window: Some(WindowInfo {
id: "w-1".into(),
title: "Fixture".into(),
app: "Fixture".into(),
pid: ProcessId::new(42),
process_instance: Some("42:1".into()),
bounds: None,
state: WindowState::default(),
}),
})
}
}

2
crates/core/src/commands/list_apps_tests.rs
View file

@ -12,12 +12,14 @@ impl ObservationOps for AppsAdapter {
pid: crate::ProcessId::new(1),
bundle_id: Some("com.apple.finder".into()),
process_instance: Some("test-instance".into()),
presentation: None,
},
AppInfo {
name: "TextEdit".into(),
pid: crate::ProcessId::new(2),
bundle_id: Some("com.apple.TextEdit".into()),
process_instance: Some("test-instance".into()),
presentation: None,
},
])
}

1
crates/core/src/commands/mod.rs
View file

@ -62,6 +62,7 @@ pub mod set_value;
pub mod skills;
pub mod snapshot;
pub mod status;
pub(crate) mod surface_scope;
pub mod toggle;
pub mod trace;
pub mod triple_click;

1
crates/core/src/commands/press_tests.rs
View file

@ -66,6 +66,7 @@ impl ObservationOps for CapturingAdapter {
pid: crate::ProcessId::new(42),
bundle_id: Some("com.example.Editor".into()),
process_instance: Some("generation-1".into()),
presentation: None,
}])
}

1
crates/core/src/commands/screenshot_tests.rs
View file

@ -35,6 +35,7 @@ impl ObservationOps for ScreenshotAdapter {
pid: crate::ProcessId::new(700),
bundle_id: Some("com.example.app".into()),
process_instance: Some("instance-700".into()),
presentation: None,
}])
}

35
crates/core/src/commands/snapshot.rs
View file

@ -60,15 +60,11 @@ pub fn execute(
"Run snapshot without --root, or omit the wait selector flags.",
));
}
if !matches!(args.surface, SnapshotSurface::Window) {
return Err(AppError::invalid_input(
"--root cannot be combined with --surface",
));
}
super::surface_scope::reject_root_with_surface("snapshot", Some(root), args.surface)?;
validate_ref_id(root)?;
}
validate_surface_support(args.surface, adapter)?;
super::surface_scope::require_supported(args.surface, adapter)?;
let opts = tree_options(&args);
@ -108,33 +104,6 @@ pub fn execute(
format_result(result)
}
fn validate_surface_support(
requested: SnapshotSurface,
adapter: &dyn PlatformAdapter,
) -> Result<(), AppError> {
let supported = adapter.supported_surfaces();
if supported.contains(&requested) {
return Ok(());
}
let supported = supported
.into_iter()
.map(SnapshotSurface::as_str)
.collect::<Vec<_>>();
Err(crate::AdapterError::new(
crate::ErrorCode::PlatformNotSupported,
format!(
"Snapshot surface '{}' is not supported on this platform",
requested.as_str()
),
)
.with_details(json!({
"requested_surface": requested.as_str(),
"supported_surfaces": supported
}))
.with_suggestion("Choose one of the supported snapshot surfaces")
.into())
}
fn format_result(result: snapshot::SnapshotResult) -> Result<Value, AppError> {
format_snapshot_fields(&result, None, None)
}

56
crates/core/src/commands/surface_scope.rs Normal file
View file

@ -0,0 +1,56 @@
use serde_json::json;
use crate::{AppError, SnapshotSurface, adapter::PlatformAdapter};
/// A ref already carries the surface it was captured from, so naming a second
/// one asks for two roots at once.
pub(crate) fn reject_root_with_surface(
command: &str,
root: Option<&str>,
surface: SnapshotSurface,
) -> Result<(), AppError> {
if root.is_none() || matches!(surface, SnapshotSurface::Window) {
return Ok(());
}
Err(AppError::invalid_input_with_suggestion(
"--root cannot be combined with --surface",
format!(
"A ref is already scoped to its own surface. Run `{command} --surface` without \
--root, or drop --surface."
),
))
}
/// Every surface, including the window, requires the adapter to declare it. An
/// adapter that declares none is unimplemented and must fail closed rather than
/// observe through a partially wired platform.
pub(crate) fn require_supported(
requested: SnapshotSurface,
adapter: &dyn PlatformAdapter,
) -> Result<(), AppError> {
let supported = adapter.supported_surfaces();
if supported.contains(&requested) {
return Ok(());
}
let supported = supported
.into_iter()
.map(SnapshotSurface::as_str)
.collect::<Vec<_>>();
Err(crate::AdapterError::new(
crate::ErrorCode::PlatformNotSupported,
format!(
"Snapshot surface '{}' is not supported on this platform",
requested.as_str()
),
)
.with_details(json!({
"requested_surface": requested.as_str(),
"supported_surfaces": supported
}))
.with_suggestion("Choose one of the supported snapshot surfaces")
.into())
}
#[cfg(test)]
#[path = "surface_scope_tests.rs"]
mod tests;

19
crates/core/src/commands/surface_scope_tests.rs Normal file
View file

@ -0,0 +1,19 @@
use super::reject_root_with_surface;
use crate::SnapshotSurface;
#[test]
fn a_ref_root_may_keep_the_default_window_surface() {
assert!(reject_root_with_surface("find", Some("@s1:e2"), SnapshotSurface::Window).is_ok());
}
#[test]
fn a_ref_root_rejects_an_explicit_second_surface() {
let error = reject_root_with_surface("find", Some("@s1:e2"), SnapshotSurface::Menubar)
.expect_err("a ref already carries its surface");
assert!(error.to_string().contains("--surface"));
}
#[test]
fn a_surface_without_a_root_is_allowed() {
assert!(reject_root_with_surface("find", None, SnapshotSurface::Menubar).is_ok());
}

2
crates/core/src/commands/wait_event_tests.rs
View file

@ -103,6 +103,7 @@ fn app(name: &str, instance: &str) -> AppInfo {
pid: crate::ProcessId::new(42),
bundle_id: Some("com.example.editor".into()),
process_instance: Some(instance.into()),
presentation: None,
}
}
@ -329,6 +330,7 @@ fn failed_inventory_poll_never_reports_app_termination() {
pid: crate::ProcessId::new(42),
bundle_id: Some("com.apple.TextEdit".into()),
process_instance: Some("test-instance".into()),
presentation: None,
}]);
let err = wait_for_event(request, &adapter, Some(Ok(baseline))).unwrap_err();

1
crates/core/src/commands/wait_scenario_tests.rs
View file

@ -111,6 +111,7 @@ impl ObservationOps for MenuWaitAdapter {
pid: crate::ProcessId::new(42),
bundle_id: None,
process_instance: Some("test-instance".into()),
presentation: None,
}])
}
}

5
crates/core/src/commands/wait_selector.rs
View file

@ -173,10 +173,11 @@ fn observe_selector(
query: &crate::LocatorQuery,
deadline: crate::Deadline,
) -> Result<Option<bool>, AppError> {
let window = snapshot::resolve_window(
let window = snapshot::resolve_window_for_surface(
adapter,
input.app.as_deref(),
input.window_id.as_deref(),
input.opts.surface,
deadline,
)?;
let resolution = resolve_query(
@ -187,6 +188,8 @@ fn observe_selector(
selection: LocatorSelection::First,
deadline,
max_raw_depth: 50,
surface: (input.opts.surface != crate::SnapshotSurface::Window)
.then_some(input.opts.surface),
materialization: LocatorMaterialization::None,
},
)?;

6
crates/core/src/launch_options.rs
View file

@ -13,6 +13,11 @@ pub struct LaunchOptions {
pub cwd: Option<PathBuf>,
pub timeout_ms: u64,
pub attach_if_running: bool,
/// Brings the application forward so it presents a window. A document-based
/// application creates its first window in response to activation, so a
/// caller that needs a window has to ask for one; waiting without asking
/// waits for an event that never fires.
pub activate: bool,
}
impl Default for LaunchOptions {
@ -23,6 +28,7 @@ impl Default for LaunchOptions {
cwd: None,
timeout_ms: 5_000,
attach_if_running: true,
activate: false,
}
}
}

19
crates/core/src/launch_result.rs Normal file
View file

@ -0,0 +1,19 @@
use serde::{Deserialize, Serialize};
use crate::{ProcessId, WindowInfo};
/// What a launch can honestly report. The process starting and the application
/// presenting a window are separate outcomes: a background application never
/// shows one, and a document-based application creates its first window only
/// once it is brought forward. Reporting them separately lets a caller wait for
/// the window it actually asked for instead of a deadline. Run `list-apps` for
/// the presentation of an application that reports no window.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LaunchResult {
pub app: String,
pub pid: ProcessId,
#[serde(skip_serializing_if = "Option::is_none")]
pub process_instance: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub window: Option<WindowInfo>,
}

3
crates/core/src/lib.rs
View file

@ -44,6 +44,7 @@ mod interaction_lease;
pub mod interaction_policy;
mod key_combo;
pub mod launch_options;
pub mod launch_result;
pub mod live_element;
mod live_identity;
mod live_locator;
@ -160,7 +161,7 @@ pub use adapter::system::SystemOps;
pub use adapter_error::AdapterError;
pub use adapter_session::AdapterSession;
pub use app_error::AppError;
pub use app_info::AppInfo;
pub use app_info::{AppInfo, AppPresentation};
pub use clipboard_content::ClipboardContent;
pub use clipboard_format::ClipboardFormat;
pub use containment_predicate::ContainmentPredicate;

1
crates/core/src/live_locator/evaluator_identifier_tests.rs
View file

@ -14,6 +14,7 @@ fn request() -> LocatorResolveRequest {
selection: LocatorSelection::Strict,
deadline: crate::Deadline::from_duration(std::time::Duration::from_secs(5)).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::None,
}
}

1
crates/core/src/live_locator/evaluator_tests.rs
View file

@ -14,6 +14,7 @@ fn request(selection: LocatorSelection) -> LocatorResolveRequest {
selection,
deadline: crate::Deadline::from_duration(std::time::Duration::from_secs(5)).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::None,
}
}

1
crates/core/src/live_locator/evidence_requirements.rs
View file

@ -99,6 +99,7 @@ mod tests {
selection: LocatorSelection::First,
deadline: crate::Deadline::after(500).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::SelectedMatches,
};

1
crates/core/src/live_locator/hydrate.rs
View file

@ -95,6 +95,7 @@ pub(super) fn selected_matches(
query,
&super::LocatorResolveRequest {
selection: super::LocatorSelection::First,
surface: None,
materialization: super::LocatorMaterialization::None,
..*request
},

4
crates/core/src/live_locator/locator_resolve_request.rs
View file

@ -7,4 +7,8 @@ pub struct LocatorResolveRequest {
pub deadline: Deadline,
pub max_raw_depth: u8,
pub materialization: LocatorMaterialization,
/// Overrides the surface implied by the root. A window root otherwise means
/// the window surface, which leaves menu bars and other overlays
/// unreachable by a targeted search.
pub surface: Option<crate::SnapshotSurface>,
}

3
crates/core/src/live_locator/materialize.rs
View file

@ -98,13 +98,14 @@ pub(crate) fn ref_entry(
fn apply_source(entry: &mut RefEntry, source: &ObservationSource, node_path: &RefPath) {
match source {
ObservationSource::Window(window) => {
ObservationSource::Window { window, surface } => {
entry.process.pid = window.pid;
entry.source.source_app = Some(window.app.clone());
entry.source.source_window_id = Some(window.id.clone());
entry.source.source_window_title = Some(window.title.clone());
entry.source.source_window_bounds_hash =
window.bounds.as_ref().and_then(crate::Rect::bounds_hash);
entry.source.source_surface = *surface;
entry.process.process_instance = window.process_instance.clone();
}
ObservationSource::Element {

4
crates/core/src/live_locator/materialize_tests.rs
View file

@ -17,6 +17,7 @@ fn request() -> LocatorResolveRequest {
selection: LocatorSelection::Strict,
deadline: crate::Deadline::from_duration(std::time::Duration::from_secs(5)).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::FullRefMap,
}
}
@ -50,6 +51,7 @@ fn selected_materialization_persists_only_returned_matches() {
selection: LocatorSelection::First,
deadline: crate::Deadline::from_duration(std::time::Duration::from_secs(5)).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::SelectedMatches,
},
)
@ -190,7 +192,7 @@ fn window_source_materialization_preserves_geometry_generation_evidence() {
vec![0],
true,
);
let ObservationSource::Window(window) = &mut observed.source else {
let ObservationSource::Window { window, .. } = &mut observed.source else {
panic!("fixture must use a window source");
};
window.bounds = Some(bounds);

2
crates/core/src/live_locator/observation_request.rs
View file

@ -82,7 +82,7 @@ impl ObservationRequest {
deadline: Deadline,
) -> Self {
Self {
surface: root.surface(),
surface: request.surface.unwrap_or_else(|| root.surface()),
..Self::locator(query, request, deadline)
}
}

15
crates/core/src/live_locator/observation_source.rs
View file

@ -4,7 +4,13 @@ use super::ObservationRoot;
#[derive(Debug, Clone)]
pub enum ObservationSource {
Window(WindowInfo),
Window {
window: WindowInfo,
/// The surface actually walked. A ref that was found on the menu bar
/// must record that, or re-resolving it later searches the window and
/// reports the element missing.
surface: crate::SnapshotSurface,
},
Element {
entry: Box<RefEntry>,
root_ref: Option<String>,
@ -12,9 +18,12 @@ pub enum ObservationSource {
}
impl ObservationSource {
pub fn from_root(root: &ObservationRoot<'_>) -> Self {
pub fn from_root(root: &ObservationRoot<'_>, surface: crate::SnapshotSurface) -> Self {
match root {
ObservationRoot::Window(window) => Self::Window((*window).clone()),
ObservationRoot::Window(window) => Self::Window {
window: (*window).clone(),
surface,
},
ObservationRoot::Element {
entry, root_ref, ..
} => Self::Element {

26
crates/core/src/live_locator/observed_tree_tests.rs
View file

@ -10,18 +10,21 @@ use crate::{
use super::test_support::evidence;
fn source() -> ObservationSource {
ObservationSource::Window(WindowInfo {
id: "w-1".into(),
title: "Fixture".into(),
app: "FixtureApp".into(),
pid: crate::ProcessId::new(42),
process_instance: Some("test-instance".into()),
bounds: None,
state: crate::WindowState {
is_focused: true,
..Default::default()
ObservationSource::Window {
surface: crate::SnapshotSurface::Window,
window: WindowInfo {
id: "w-1".into(),
title: "Fixture".into(),
app: "FixtureApp".into(),
pid: crate::ProcessId::new(42),
process_instance: Some("test-instance".into()),
bounds: None,
state: crate::WindowState {
is_focused: true,
..Default::default()
},
},
})
}
}
fn subtree(role: &str, name: &str, children: Vec<ObservedSubtree>) -> ObservedSubtree {
@ -120,6 +123,7 @@ fn snapshot_projection_and_find_share_the_same_observation() {
selection: LocatorSelection::Count,
deadline: crate::Deadline::from_duration(std::time::Duration::from_secs(1)).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::None,
},
)

1
crates/core/src/live_locator/ownership_tests.rs
View file

@ -10,6 +10,7 @@ fn request(selection: LocatorSelection) -> LocatorResolveRequest {
selection,
deadline: crate::Deadline::from_duration(std::time::Duration::from_secs(5)).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::None,
}
}

1
crates/core/src/live_locator/resolve.rs
View file

@ -210,6 +210,7 @@ pub fn find_first_entry(
selection: super::LocatorSelection::First,
deadline: crate::Deadline::from_duration(timeout)?,
max_raw_depth: 50,
surface: None,
materialization: super::LocatorMaterialization::SelectedMatches,
},
)

2
crates/core/src/live_locator/resolve_query_hydration_tests.rs
View file

@ -73,6 +73,7 @@ fn hydration_retry_preserves_failed_attempt_statistics() {
selection: LocatorSelection::Strict,
deadline: crate::Deadline::after(5_000).unwrap(),
max_raw_depth: 10,
surface: None,
materialization: LocatorMaterialization::SelectedMatches,
};
@ -134,6 +135,7 @@ fn selected_hydration_rejects_incomplete_snapshot_evidence() {
selection: LocatorSelection::Strict,
deadline: crate::Deadline::after(35).unwrap(),
max_raw_depth: 10,
surface: None,
materialization: LocatorMaterialization::SelectedMatches,
};

1
crates/core/src/live_locator/resolve_query_tests.rs
View file

@ -41,6 +41,7 @@ fn request_with_timeout(timeout: std::time::Duration) -> LocatorResolveRequest {
selection: LocatorSelection::Strict,
deadline: crate::Deadline::from_duration(timeout).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::None,
}
}

1
crates/core/src/live_locator/resolve_tests.rs
View file

@ -11,6 +11,7 @@ fn request() -> LocatorResolveRequest {
selection: LocatorSelection::Strict,
deadline: crate::Deadline::from_duration(std::time::Duration::from_secs(5)).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::None,
}
}

1
crates/core/src/live_locator/scrollarea_name_tests.rs
View file

@ -29,6 +29,7 @@ fn exact_named_scroll_area_is_unique_among_unnamed_scroll_containers() {
selection: LocatorSelection::Strict,
deadline: crate::Deadline::after(500).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::None,
};

3
crates/core/src/live_locator/selected_hydration_churn_tests.rs
View file

@ -205,6 +205,7 @@ fn selected_request() -> LocatorResolveRequest {
selection: LocatorSelection::First,
deadline: crate::Deadline::after(5_000).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::SelectedMatches,
}
}
@ -247,7 +248,7 @@ fn single_hydration_tree(
vec![0],
true,
);
tree.source = ObservationSource::from_root(&root);
tree.source = ObservationSource::from_root(&root, crate::SnapshotSurface::Window);
tree.structurally_complete = topology_complete;
tree
}

3
crates/core/src/live_locator/selected_hydration_contract_tests.rs
View file

@ -194,7 +194,7 @@ fn hydrated_tree(root: ObservationRoot<'_>, mode: Mode) -> ObservedTree {
vec![0],
true,
);
tree.source = ObservationSource::from_root(&root);
tree.source = ObservationSource::from_root(&root, crate::SnapshotSurface::Window);
tree
}
@ -235,6 +235,7 @@ fn selected_request() -> LocatorResolveRequest {
selection: LocatorSelection::First,
deadline: crate::Deadline::after(5_000).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::SelectedMatches,
}
}

3
crates/core/src/live_locator/selected_hydration_subtree_tests.rs
View file

@ -210,7 +210,7 @@ fn selected_tree(root: ObservationRoot<'_>, mode: Mode) -> ObservedTree {
nodes[last].completeness.subtree_complete = false;
}
let mut tree = super::test_support::tree(nodes, vec![0], !incomplete);
tree.source = ObservationSource::from_root(&root);
tree.source = ObservationSource::from_root(&root, crate::SnapshotSurface::Window);
tree
}
@ -296,6 +296,7 @@ fn request() -> LocatorResolveRequest {
selection: LocatorSelection::First,
deadline: crate::Deadline::after(5_000).unwrap(),
max_raw_depth: 10,
surface: None,
materialization: LocatorMaterialization::SelectedMatches,
}
}

3
crates/core/src/live_locator/selected_hydration_tests.rs
View file

@ -101,6 +101,7 @@ fn role_only_selected_hydration_anchors_before_hydrating_without_rewalking_large
selection: LocatorSelection::First,
deadline: crate::Deadline::after(5_000).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::SelectedMatches,
};
@ -231,7 +232,7 @@ fn hydrated_button_tree(root: ObservationRoot<'_>) -> ObservedTree {
vec![0],
true,
);
tree.source = ObservationSource::from_root(&root);
tree.source = ObservationSource::from_root(&root, crate::SnapshotSurface::Window);
tree.stats.traversal.nodes_visited = 1;
tree.stats.reads.counts.attribute_batches = 1;
tree.stats.reads.counts.attributes_requested = 23;

1
crates/core/src/live_locator/selection_completeness_tests.rs
View file

@ -122,6 +122,7 @@ fn first_request() -> LocatorResolveRequest {
selection: LocatorSelection::First,
deadline: crate::Deadline::after(500).unwrap(),
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::None,
}
}

5
crates/core/src/live_locator/test_support.rs
View file

@ -61,7 +61,10 @@ pub(crate) fn tree(
ObservedTree {
nodes,
roots,
source: ObservationSource::Window(window()),
source: ObservationSource::Window {
window: window(),
surface: crate::SnapshotSurface::Window,
},
stats: LocatorStats::default(),
structurally_complete,
}

1
crates/core/src/live_locator/validation_tests.rs
View file

@ -17,6 +17,7 @@ fn request(max_raw_depth: u8) -> LocatorResolveRequest {
selection: LocatorSelection::All { limit: None },
deadline: crate::Deadline::from_duration(std::time::Duration::from_secs(5)).unwrap(),
max_raw_depth,
surface: None,
materialization: LocatorMaterialization::None,
}
}

3
crates/core/src/node_tests.rs
View file

@ -84,6 +84,7 @@ fn app_info_bundle_id_none_omitted_from_json() {
pid: crate::ProcessId::new(42),
bundle_id: None,
process_instance: Some("test-instance".into()),
presentation: None,
};
let json = serde_json::to_string(&info).unwrap();
assert!(
@ -100,6 +101,7 @@ fn app_info_bundle_id_some_present_in_json() {
pid: crate::ProcessId::new(7),
bundle_id: Some("com.apple.Safari".into()),
process_instance: Some("test-instance".into()),
presentation: None,
};
let json = serde_json::to_string(&info).unwrap();
assert!(
@ -117,6 +119,7 @@ fn app_info_roundtrip_preserves_all_fields() {
pid: crate::ProcessId::new(1234),
bundle_id: Some("com.apple.TextEdit".into()),
process_instance: Some("test-instance".into()),
presentation: None,
};
let json = serde_json::to_string(&original).unwrap();
let back: AppInfo = serde_json::from_str(&json).unwrap();

2
crates/core/src/output.rs
View file

@ -4,7 +4,7 @@ use serde_json::Value;
use crate::recovery_hint::RecoveryHint;
use crate::{AppError, DeliverySemantics, ErrorCode, RetryDisposition};
pub const ENVELOPE_VERSION: &str = "2.2";
pub const ENVELOPE_VERSION: &str = "2.3";
/// Structured output envelope used by the CLI and future programmatic transports.
#[derive(Debug, Serialize)]

22
crates/core/src/ref_action.rs
View file

@ -99,17 +99,21 @@ pub(crate) fn dispatch_resolved(
}
request = request
.with_verified_point(preflight.verified_point)
.with_expected_process(expected_process);
.with_expected_process(expected_process.clone());
let final_target = ResolvedRefAction::new(target, &handle);
final_target.context.trace_lazy(
"action.dispatch.start",
|| json!({ "ref": final_target.ref_id, "action": request.action.name() }),
)?;
let action_name = request.action.name();
let raises_surface = request.action.may_raise_surface();
let dispatch_result = final_target
.adapter
.execute_action(final_target.handle, request, lease);
let result = dispatch_result?;
let mut result = dispatch_result?;
if raises_surface {
result.surfaces = settled_surfaces(&final_target, expected_process);
}
final_target
.context
.trace_lazy(
@ -120,6 +124,20 @@ pub(crate) fn dispatch_resolved(
Ok(result)
}
/// An action that opens a sheet, menu, or alert leaves the caller staring at a
/// success envelope with no hint that the application is now waiting on a
/// dialog. Reporting the overlays saves a round of window archaeology. Purely
/// informational, so a failed read never disturbs delivered work.
fn settled_surfaces(
target: &ResolvedRefAction<'_>,
process: crate::ProcessIdentity,
) -> Vec<crate::SurfaceInfo> {
target
.adapter
.list_surfaces(process, target.deadline)
.unwrap_or_default()
}
pub(crate) fn mark_pre_dispatch_resolution_failure(error: AdapterError) -> AdapterError {
match error.disposition {
crate::DeliverySemantics::Unknown | crate::DeliverySemantics::NotDelivered => {

4
crates/core/src/ref_action_wait.rs
View file

@ -133,6 +133,10 @@ mod process_state_tests;
#[path = "ref_action_wait_app_not_found_tests.rs"]
mod app_not_found_tests;
#[cfg(test)]
#[path = "ref_action_wait_success_tests.rs"]
mod success_tests;
#[cfg(test)]
#[path = "ref_action_exactly_once_tests.rs"]
mod exactly_once_tests;

1
crates/core/src/ref_action_wait_app_not_found_tests.rs
View file

@ -180,6 +180,7 @@ impl ObservationOps for AppNotFoundUnresponsiveProcessAdapter {
pid: crate::ProcessId::new(1),
bundle_id: None,
process_instance: Some("test-instance".into()),
presentation: None,
}])
}

66
crates/core/src/ref_action_wait_process_state_tests.rs
View file

@ -75,6 +75,7 @@ fn app(name: &str) -> AppInfo {
pid: crate::ProcessId::new(1),
bundle_id: None,
process_instance: Some("test-instance".into()),
presentation: None,
}
}
@ -333,68 +334,3 @@ fn terminal_stale_ref_against_crashed_process_carries_process_state_detail() {
"STALE_REF against a crashed pid must carry details.process_state = \"crashed\""
);
}
struct SuccessWithUnresponsiveProbeAdapter {
probe_calls: AtomicU32,
}
impl ObservationOps for SuccessWithUnresponsiveProbeAdapter {
fn resolve_element_strict(
&self,
_entry: &RefEntry,
_deadline: crate::Deadline,
) -> Result<NativeHandle, AdapterError> {
Ok(NativeHandle::null())
}
crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]);
}
impl ActionOps for SuccessWithUnresponsiveProbeAdapter {
fn execute_action(
&self,
_handle: &NativeHandle,
_request: ActionRequest,
_lease: &crate::InteractionLease,
) -> Result<crate::action_result::ActionResult, AdapterError> {
Ok(crate::action_result::ActionResult::delivered_unverified(
"click",
))
}
}
impl InputOps for SuccessWithUnresponsiveProbeAdapter {}
impl SystemOps for SuccessWithUnresponsiveProbeAdapter {
crate::adapter::guarded_interaction_lease!();
fn process_state(
&self,
_process: crate::ProcessIdentity,
_deadline: crate::Deadline,
) -> Result<crate::process_state::ProcessState, AdapterError> {
self.probe_calls.fetch_add(1, Ordering::SeqCst);
Ok(crate::process_state::ProcessState::Unresponsive)
}
}
#[test]
fn enrichment_never_converts_a_successful_action_into_a_failure() {
let adapter = SuccessWithUnresponsiveProbeAdapter {
probe_calls: AtomicU32::new(0),
};
let result = execute_with_auto_wait(
RefActionWaitCtx {
adapter: &adapter,
entry: &entry(),
ref_id: "@e1",
context: &CommandContext::default(),
},
ActionRequest::headless(Action::Click),
crate::ref_action::dispatch_resolved,
)
.unwrap();
assert_eq!(result.action, "click");
assert_eq!(adapter.probe_calls.load(Ordering::SeqCst), 0);
}

119
crates/core/src/ref_action_wait_success_tests.rs Normal file
View file

@ -0,0 +1,119 @@
use super::*;
use crate::{
AdapterError,
action::Action,
adapter::{ActionOps, InputOps, NativeHandle, ObservationOps, SystemOps},
capability,
};
use std::sync::atomic::{AtomicU32, Ordering};
/// Covers the branch where a successful action must never be rewritten into a
/// failure by process-state enrichment, split out of
/// `ref_action_wait_process_state_tests.rs` to keep both files under the
/// repo's 400 LOC hard limit.
fn entry() -> RefEntry {
let bounds = crate::Rect {
x: 1.0,
y: 1.0,
width: 20.0,
height: 20.0,
};
RefEntry {
process: crate::RefProcess {
pid: crate::ProcessId::new(1),
process_instance: Some("test-instance".into()),
},
identity: crate::RefEntryIdentity {
role: "button".into(),
name: Some("Run".into()),
value: None,
description: None,
native_id: None,
},
geometry: crate::RefGeometry {
bounds: Some(bounds),
bounds_hash: bounds.bounds_hash(),
},
capabilities: crate::RefCapabilities {
states: vec![],
available_actions: vec![capability::CLICK.into()],
},
source: crate::RefSource {
source_app: Some("Original".into()),
source_window_id: None,
source_window_title: None,
source_window_bounds_hash: None,
source_surface: crate::snapshot_surface::SnapshotSurface::Window,
},
scope: crate::RefScope {
root_ref: None,
path_is_absolute: false,
path: smallvec::SmallVec::new(),
},
}
}
struct SuccessWithUnresponsiveProbeAdapter {
probe_calls: AtomicU32,
}
impl ObservationOps for SuccessWithUnresponsiveProbeAdapter {
fn resolve_element_strict(
&self,
_entry: &RefEntry,
_deadline: crate::Deadline,
) -> Result<NativeHandle, AdapterError> {
Ok(NativeHandle::null())
}
crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]);
}
impl ActionOps for SuccessWithUnresponsiveProbeAdapter {
fn execute_action(
&self,
_handle: &NativeHandle,
_request: ActionRequest,
_lease: &crate::InteractionLease,
) -> Result<crate::action_result::ActionResult, AdapterError> {
Ok(crate::action_result::ActionResult::delivered_unverified(
"click",
))
}
}
impl InputOps for SuccessWithUnresponsiveProbeAdapter {}
impl SystemOps for SuccessWithUnresponsiveProbeAdapter {
crate::adapter::guarded_interaction_lease!();
fn process_state(
&self,
_process: crate::ProcessIdentity,
_deadline: crate::Deadline,
) -> Result<crate::process_state::ProcessState, AdapterError> {
self.probe_calls.fetch_add(1, Ordering::SeqCst);
Ok(crate::process_state::ProcessState::Unresponsive)
}
}
#[test]
fn enrichment_never_converts_a_successful_action_into_a_failure() {
let adapter = SuccessWithUnresponsiveProbeAdapter {
probe_calls: AtomicU32::new(0),
};
let result = execute_with_auto_wait(
RefActionWaitCtx {
adapter: &adapter,
entry: &entry(),
ref_id: "@e1",
context: &CommandContext::default(),
},
ActionRequest::headless(Action::Click),
crate::ref_action::dispatch_resolved,
)
.unwrap();
assert_eq!(result.action, "click");
assert_eq!(adapter.probe_calls.load(Ordering::SeqCst), 0);
}

1
crates/core/src/signals_tests.rs
View file

@ -25,6 +25,7 @@ fn app_with_instance(name: &str, pid: u32, instance: &str) -> AppInfo {
pid: crate::ProcessId::new(pid),
bundle_id: None,
process_instance: Some(instance.into()),
presentation: None,
}
}

59
crates/core/src/snapshot.rs
View file

@ -37,7 +37,7 @@ pub fn build(
window_id: Option<&str>,
deadline: crate::Deadline,
) -> Result<SnapshotResult, AppError> {
let window = resolve_window(adapter, app_name, window_id, deadline)?;
let window = resolve_window_for_surface(adapter, app_name, window_id, opts.surface, deadline)?;
let observation_options = opts.with_ref_identity_bounds();
let (raw_tree, complete, nodes_observed) = crate::renderer_accessibility::observe_tree(
adapter,
@ -81,6 +81,57 @@ pub fn build(
})
}
/// Resolves the window that identifies the process to observe. An app-level
/// surface is not owned by a single window, so several open windows must not be
/// reported as ambiguous when one is requested.
pub(crate) fn resolve_window_for_surface(
adapter: &dyn PlatformAdapter,
app_name: Option<&str>,
window_id: Option<&str>,
surface: crate::SnapshotSurface,
deadline: crate::Deadline,
) -> Result<WindowInfo, AppError> {
if window_id.is_some() || matches!(surface, crate::SnapshotSurface::Window) {
return resolve_window(adapter, app_name, window_id, deadline);
}
crate::window_lookup::select_surface_owner(
windows_for_app(adapter, app_name, deadline)?,
crate::AdapterError::new(
crate::ErrorCode::AppNotFound,
format!(
"No window found to identify the application owning surface '{}'",
surface.as_str()
),
),
)
}
fn windows_for_app(
adapter: &dyn PlatformAdapter,
app_name: Option<&str>,
deadline: crate::Deadline,
) -> Result<Vec<WindowInfo>, AppError> {
let filter = WindowFilter {
focused_only: app_name.is_none(),
app: app_name.map(str::to_string),
};
Ok(windows_matching_app(
adapter.list_windows(&filter, deadline)?,
app_name,
))
}
/// The adapter filter is advisory, so the app name is applied again here.
fn windows_matching_app(windows: Vec<WindowInfo>, app_name: Option<&str>) -> Vec<WindowInfo> {
match app_name {
Some(app) => windows
.into_iter()
.filter(|window| window.app.eq_ignore_ascii_case(app))
.collect(),
None => windows,
}
}
pub(crate) fn resolve_window(
adapter: &dyn PlatformAdapter,
app_name: Option<&str>,
@ -105,12 +156,8 @@ pub(crate) fn resolve_window(
)
})
} else if let Some(app) = app_name {
let candidates = windows
.into_iter()
.filter(|window| window.app.eq_ignore_ascii_case(app))
.collect::<Vec<_>>();
crate::window_lookup::select_window(
candidates,
windows_matching_app(windows, app_name),
crate::AdapterError::new(
crate::ErrorCode::AppNotFound,
format!("No window found for app '{app}'"),

24
crates/core/src/window_lookup.rs
View file

@ -47,6 +47,30 @@ pub(crate) fn find_window_for_process(
)
}
/// A menu bar, menu, or alert belongs to the application, not to one of its
/// windows, so several open windows are not an ambiguity for those surfaces —
/// any window of the app names the same process. Prefers a focused or visible
/// window so the caller still gets the most relevant identity.
pub(crate) fn select_surface_owner(
candidates: Vec<WindowInfo>,
empty_error: crate::AdapterError,
) -> Result<WindowInfo, AppError> {
if candidates.is_empty() {
return Err(empty_error.into());
}
let best = candidates
.iter()
.position(|window| window.state.is_focused)
.or_else(|| {
candidates
.iter()
.position(|window| window.state.visible == Some(true))
})
.unwrap_or(0);
let mut candidates = candidates;
Ok(candidates.swap_remove(best))
}
pub(crate) fn select_window(
mut candidates: Vec<WindowInfo>,
empty_error: crate::AdapterError,

7
crates/ffi/include/agent_desktop.h
View file

@ -1269,8 +1269,9 @@ AdResult ad_close_app(const struct AdAdapter *adapter, const char *id, bool forc
/**
* Launches the application identified by `id` (bundle id on macOS,
* executable path on other platforms) and, on success, writes the
* first window that becomes available into `*out`. Waits up to
* `timeout_ms` for the window to appear; zero means "no wait".
* first window that becomes available into `*out`. Waits for the windows
* the launch itself produces, bounded by `timeout_ms`; zero means "no wait".
* An application that presents no window fails with `WINDOW_NOT_FOUND`.
*
* The returned `AdWindowInfo` owns heap-allocated interior strings that
* must be released with `ad_release_window_fields` once done. On error
@ -1420,7 +1421,7 @@ AdResult ad_execute_by_ref_timeout(const struct AdAdapter *adapter,
* to disk, and writes the JSON envelope into `*out`.
*
* The JSON shape matches `agent-desktop snapshot`:
* `{"version":"2.2","ok":true,"command":"snapshot","data":{"app":"...","window":{...},"ref_count":N,"snapshot_id":"...","tree":{...}}}`.
* `{"version":"2.3","ok":true,"command":"snapshot","data":{"app":"...","window":{...},"ref_count":N,"snapshot_id":"...","tree":{...}}}`.
*
* **`*out` ownership and error behaviour:**
* - On success (`AD_RESULT_OK`): `*out` is a heap-allocated JSON string with `"ok":true`.

47
crates/ffi/src/apps/launch.rs
View file

@ -5,12 +5,26 @@ use crate::convert::window::{
use crate::error::{AdResult, set_last_error};
use crate::ffi_try::trap_panic;
use crate::types::{AdExactWindowInfo, AdWindowInfo};
use agent_desktop_core::{AdapterError, ErrorCode, WindowInfo, launch_result::LaunchResult};
use std::os::raw::c_char;
/// The C entry points hand back one window, so a launch that produced none is
/// an error at this boundary even though it is a valid result in core.
fn launched_window(launched: LaunchResult) -> Result<WindowInfo, AdapterError> {
launched.window.ok_or_else(|| {
AdapterError::new(
ErrorCode::WindowNotFound,
"Application is running, but it has presented no window",
)
.with_suggestion("Launch with activation, or wait for the window before reading it.")
})
}
/// Launches the application identified by `id` (bundle id on macOS,
/// executable path on other platforms) and, on success, writes the
/// first window that becomes available into `*out`. Waits up to
/// `timeout_ms` for the window to appear; zero means "no wait".
/// first window that becomes available into `*out`. Waits for the windows
/// the launch itself produces, bounded by `timeout_ms`; zero means "no wait".
/// An application that presents no window fails with `WINDOW_NOT_FOUND`.
///
/// The returned `AdWindowInfo` owns heap-allocated interior strings that
/// must be released with `ad_release_window_fields` once done. On error
@ -57,7 +71,11 @@ pub unsafe extern "C" fn ad_launch_app(
return crate::error::last_error_code();
}
};
match adapter.inner.launch_app(&id_str, &options, &lease) {
match adapter
.inner
.launch_app(&id_str, &options, &lease)
.and_then(launched_window)
{
Ok(win) => {
*out = window_info_to_c(&win);
AdResult::Ok
@ -112,7 +130,11 @@ pub unsafe extern "C" fn ad_launch_app_exact(
return crate::error::last_error_code();
}
};
match adapter.inner.launch_app(&id, &options, &lease) {
match adapter
.inner
.launch_app(&id, &options, &lease)
.and_then(launched_window)
{
Ok(window) => match validate_exact_window_info(&window) {
Ok(()) => {
*out = exact_window_info_to_c(&window);
@ -181,19 +203,24 @@ mod tests {
_id: &str,
options: &LaunchOptions,
_lease: &InteractionLease,
) -> Result<WindowInfo, AdapterError> {
) -> Result<LaunchResult, AdapterError> {
self.probe.calls.fetch_add(1, Ordering::SeqCst);
self.probe
.timeout_ms
.store(options.timeout_ms, Ordering::SeqCst);
Ok(WindowInfo {
id: "w-launch".into(),
title: "Launched".into(),
Ok(LaunchResult {
app: "Fixture".into(),
pid: ProcessId::new(42),
process_instance: Some("fixture-42".into()),
bounds: None,
state: WindowState::default(),
window: Some(WindowInfo {
id: "w-launch".into(),
title: "Launched".into(),
app: "Fixture".into(),
pid: ProcessId::new(42),
process_instance: Some("fixture-42".into()),
bounds: None,
state: WindowState::default(),
}),
})
}
}

2
crates/ffi/src/commands/snapshot.rs
View file

@ -15,7 +15,7 @@ use std::ptr;
/// to disk, and writes the JSON envelope into `*out`.
///
/// The JSON shape matches `agent-desktop snapshot`:
/// `{"version":"2.2","ok":true,"command":"snapshot","data":{"app":"...","window":{...},"ref_count":N,"snapshot_id":"...","tree":{...}}}`.
/// `{"version":"2.3","ok":true,"command":"snapshot","data":{"app":"...","window":{...},"ref_count":N,"snapshot_id":"...","tree":{...}}}`.
///
/// **`*out` ownership and error behaviour:**
/// - On success (`AD_RESULT_OK`): `*out` is a heap-allocated JSON string with `"ok":true`.

1
crates/ffi/src/convert/app.rs
View file

@ -33,6 +33,7 @@ mod tests {
pid: agent_desktop_core::ProcessId::new(42),
bundle_id: Some("com.apple.finder".into()),
process_instance: Some("42:100".into()),
presentation: None,
};
let c = app_info_to_c(&a);
assert_eq!(unsafe { c_to_string(c.name) }.as_deref(), Some("Finder"));

1
crates/ffi/src/observation/find.rs
View file

@ -197,6 +197,7 @@ pub(crate) unsafe fn decode_query(
selection,
deadline,
max_raw_depth: 50,
surface: None,
materialization: LocatorMaterialization::None,
},
))

5
crates/ffi/src/observation/find_abi_tests.rs
View file

@ -37,7 +37,10 @@ impl ObservationOps for CardinalityAdapter {
}
ObservedTree::from_roots(
roots,
ObservationSource::Window(window.clone()),
ObservationSource::Window {
window: window.clone(),
surface: agent_desktop_core::SnapshotSurface::Window,
},
Default::default(),
self.complete,
)

5
crates/ffi/src/observation/is_abi_tests.rs
View file

@ -26,7 +26,10 @@ impl ObservationOps for DuplicateAdapter {
};
ObservedTree::from_roots(
vec![button(Vec::new()), button(vec!["disabled".into()])],
ObservationSource::Window(window.clone()),
ObservationSource::Window {
window: window.clone(),
surface: agent_desktop_core::SnapshotSurface::Window,
},
Default::default(),
true,
)

1
crates/ffi/src/surfaces/list.rs
View file

@ -268,6 +268,7 @@ mod tests {
pid: agent_desktop_core::ProcessId::new(pid),
bundle_id: None,
process_instance: Some(instance.into()),
presentation: None,
}
}

63
crates/macos/src/actions/activate_descendant.rs Normal file
View file

@ -0,0 +1,63 @@
#[cfg(target_os = "macos")]
mod imp {
use agent_desktop_core::{AdapterError, Deadline};
use crate::actions::chain_delivery::DeliveryOutcome;
use crate::tree::AXElement;
const MAX_CANDIDATES: usize = 4;
/// A row often publishes no activation itself while the cell inside it
/// does: Finder's sidebar `treeitem` is inert and its `cell` carries
/// `AXOpen`. Without this the chain falls through to writing selection,
/// which the row accepts and reports back while the application never
/// navigates.
///
/// Descending is a guess about which child owns the row's behaviour, so
/// only an observed effect ends the chain here. A child that merely claims
/// success — Xcode's rows answer that to `AXConfirm` and do nothing — must
/// not consume the selection step that actually works for them.
pub(crate) fn activate_descendant(
element: &AXElement,
deadline: Deadline,
) -> Result<DeliveryOutcome, AdapterError> {
let instant = crate::tree::locator_deadline::from_operation(deadline)?;
let children = crate::tree::attributes::copy_ax_array_prefix_result(
element,
"AXChildren",
MAX_CANDIDATES,
instant,
)
.ok()
.flatten()
.unwrap_or_default();
for child in children {
for action in crate::tree::action_list::PRIMARY_ACTIVATION_ACTIONS {
let outcome =
crate::actions::ax_helpers::perform_observed_action(&child, action, deadline)?;
if outcome.was_verified() {
return Ok(outcome);
}
}
}
Ok(DeliveryOutcome::NotDelivered)
}
}
#[cfg(not(target_os = "macos"))]
mod imp {
use agent_desktop_core::{AdapterError, Deadline};
use crate::actions::chain_delivery::DeliveryOutcome;
use crate::tree::AXElement;
pub(crate) fn activate_descendant(
_element: &AXElement,
_deadline: Deadline,
) -> Result<DeliveryOutcome, AdapterError> {
Ok(DeliveryOutcome::NotDelivered)
}
}
pub(crate) use imp::activate_descendant;

112
crates/macos/src/actions/activation_effect.rs Normal file
View file

@ -0,0 +1,112 @@
#[cfg(target_os = "macos")]
mod imp {
use agent_desktop_core::Deadline;
use crate::tree::AXElement;
const SETTLE_POLL_MS: u64 = 40;
const SETTLE_BUDGET_MS: u64 = 400;
/// The readback for `perform`, which has no written attribute to re-read.
#[derive(Default)]
pub(crate) struct FocusState {
window_title: Option<String>,
focused_element: Option<AXElement>,
}
/// Accessibility hands back a fresh reference for the same element on every
/// read, and reuses a released one for a different element later, so a raw
/// pointer answers neither question this comparison asks. `CFEqual` is the
/// identity the framework defines.
impl PartialEq for FocusState {
fn eq(&self, other: &Self) -> bool {
self.window_title == other.window_title
&& match (&self.focused_element, &other.focused_element) {
(None, None) => true,
(Some(mine), Some(theirs)) => crate::tree::same_element(mine, theirs),
_ => false,
}
}
}
pub(crate) fn focus_state(element: &AXElement, deadline: Deadline) -> FocusState {
let Some(app) = crate::system::app_ops::pid_from_element(element, deadline)
.map(crate::tree::element_for_pid)
else {
return FocusState::default();
};
FocusState {
window_title: read_element(&app, "AXFocusedWindow", deadline).and_then(|window| {
crate::tree::attributes::copy_string_attr_result(&window, "AXTitle", deadline)
.ok()
.flatten()
}),
focused_element: read_element(&app, "AXFocusedUIElement", deadline),
}
}
pub(crate) fn changed_now(
before: &FocusState,
element: &AXElement,
deadline: Deadline,
) -> bool {
focus_state(element, deadline) != *before
}
pub(crate) fn settled_change(
before: &FocusState,
element: &AXElement,
deadline: Deadline,
) -> bool {
let started = std::time::Instant::now();
loop {
if focus_state(element, deadline) != *before {
return true;
}
if started.elapsed() >= std::time::Duration::from_millis(SETTLE_BUDGET_MS)
|| deadline.is_expired()
{
return false;
}
std::thread::sleep(std::time::Duration::from_millis(SETTLE_POLL_MS));
}
}
fn read_element(app: &AXElement, attr: &str, deadline: Deadline) -> Option<AXElement> {
crate::tree::attributes::copy_element_attr_result(app, attr, deadline)
.ok()
.flatten()
}
}
#[cfg(not(target_os = "macos"))]
mod imp {
use agent_desktop_core::Deadline;
use crate::tree::AXElement;
#[derive(Default, PartialEq)]
pub(crate) struct FocusState;
pub(crate) fn focus_state(_element: &AXElement, _deadline: Deadline) -> FocusState {
FocusState
}
pub(crate) fn changed_now(
_before: &FocusState,
_element: &AXElement,
_deadline: Deadline,
) -> bool {
false
}
pub(crate) fn settled_change(
_before: &FocusState,
_element: &AXElement,
_deadline: Deadline,
) -> bool {
false
}
}
pub(crate) use imp::{changed_now, focus_state, settled_change};

58
crates/macos/src/actions/ax_helpers.rs
View file

@ -19,11 +19,63 @@ mod imp {
deadline: agent_desktop_core::Deadline,
) -> Result<bool, AdapterError> {
let action = CFString::new(name);
run_mutation(el, name, "AXUIElementPerformAction", deadline, |deadline| {
run_mutation(el, name, ax_mutation::PERFORM_API, deadline, |deadline| {
crate::tree::ax_ipc::perform_action(el, action.as_concrete_TypeRef(), deadline)
})
}
/// Decides delivery from what the application did, not what it claimed.
/// Unadvertised actions are never performed, because a responder can answer
/// `kAXErrorSuccess` to an action it never published and never ran. Only an
/// uninformative code needs the settle poll to decide delivery at all; a
/// reported success is already delivered, so observing it merely upgrades
/// the report to verified and must not cost the caller a wait.
pub(crate) fn perform_observed_action(
el: &AXElement,
name: &str,
deadline: agent_desktop_core::Deadline,
) -> Result<crate::actions::chain_delivery::DeliveryOutcome, AdapterError> {
use crate::actions::activation_effect;
use crate::actions::chain_delivery::DeliveryOutcome;
use ax_mutation::PerformSignal;
if !advertises_action(el, name, deadline) {
return Ok(DeliveryOutcome::NotDelivered);
}
let before = activation_effect::focus_state(el, deadline);
let action = CFString::new(name);
let error =
crate::tree::ax_ipc::perform_action(el, action.as_concrete_TypeRef(), deadline)?;
match ax_mutation::classify_perform(name, error)? {
PerformSignal::ReportedUnsupported => Ok(DeliveryOutcome::NotDelivered),
PerformSignal::ReportedDelivered => Ok(DeliveryOutcome::from_delivery(
true,
activation_effect::changed_now(&before, el, deadline),
)),
PerformSignal::Uninformative => Ok(DeliveryOutcome::from_delivery(
activation_effect::settled_change(&before, el, deadline),
true,
)),
}
}
fn advertises_action(
el: &AXElement,
name: &str,
deadline: agent_desktop_core::Deadline,
) -> bool {
let mut usage = crate::tree::observation_usage::ObservationUsage::with_defaults();
let read = crate::tree::capabilities::copy_action_names_with_status(
el,
std::time::Instant::now() + deadline.remaining(),
&mut usage,
);
match read.value {
Some(actions) => actions.iter().any(|action| action == name),
None => true,
}
}
pub(crate) fn set_ax_bool_or_err(
el: &AXElement,
attr: &str,
@ -334,6 +386,6 @@ mod imp {
}
pub(crate) use imp::{
ax_focus_or_err, element_role, is_attr_settable, set_ax_bool_or_err, set_ax_string_or_err,
set_ax_value_coerced, try_ax_action_or_err,
ax_focus_or_err, element_role, is_attr_settable, perform_observed_action, set_ax_bool_or_err,
set_ax_string_or_err, set_ax_value_coerced, try_ax_action_or_err,
};

83
crates/macos/src/actions/ax_mutation.rs
View file

@ -1,10 +1,12 @@
use accessibility_sys::{
kAXErrorAPIDisabled, kAXErrorActionUnsupported, kAXErrorAttributeUnsupported,
kAXErrorCannotComplete, kAXErrorIllegalArgument, kAXErrorInvalidUIElement, kAXErrorNoValue,
kAXErrorNotImplemented, kAXErrorSuccess,
kAXErrorCannotComplete, kAXErrorFailure, kAXErrorIllegalArgument, kAXErrorInvalidUIElement,
kAXErrorNoValue, kAXErrorNotImplemented, kAXErrorSuccess,
};
use agent_desktop_core::{AdapterError, DeliverySemantics, ErrorCode};
pub(crate) const PERFORM_API: &str = "AXUIElementPerformAction";
pub(crate) fn classify_result(
_element: &crate::tree::AXElement,
operation: &str,
@ -15,8 +17,21 @@ pub(crate) fn classify_result(
}
fn classify(operation: &str, api: &str, error: i32) -> Result<bool, AdapterError> {
signal_or_error(operation, api, error).map(|signal| signal == PerformSignal::ReportedDelivered)
}
fn signal_or_error(operation: &str, api: &str, error: i32) -> Result<PerformSignal, AdapterError> {
if error == kAXErrorSuccess {
return Ok(true);
return Ok(PerformSignal::ReportedDelivered);
}
if error == kAXErrorActionUnsupported
|| error == kAXErrorNotImplemented
|| error == kAXErrorFailure
{
return Ok(PerformSignal::ReportedUnsupported);
}
if error == kAXErrorAttributeUnsupported || error == kAXErrorNoValue {
return Ok(PerformSignal::Uninformative);
}
if error == kAXErrorAPIDisabled {
return Err(AdapterError::permission_denied()
@ -25,13 +40,6 @@ fn classify(operation: &str, api: &str, error: i32) -> Result<bool, AdapterError
))
.with_disposition(DeliverySemantics::not_delivered()));
}
if error == kAXErrorActionUnsupported
|| error == kAXErrorAttributeUnsupported
|| error == kAXErrorNoValue
|| error == kAXErrorNotImplemented
{
return Ok(false);
}
if error == kAXErrorInvalidUIElement {
return Err(AdapterError::new(
ErrorCode::StaleRef,
@ -77,11 +85,25 @@ fn classify(operation: &str, api: &str, error: i32) -> Result<bool, AdapterError
))
}
/// How much a perform return code proves. Responders break the contract in both
/// directions — a code that reports failure after acting, and success after
/// doing nothing — so delivery is settled by observation, not by the code.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum PerformSignal {
ReportedDelivered,
ReportedUnsupported,
Uninformative,
}
pub(crate) fn classify_perform(operation: &str, error: i32) -> Result<PerformSignal, AdapterError> {
signal_or_error(operation, PERFORM_API, error)
}
#[cfg(test)]
mod tests {
use accessibility_sys::{
kAXErrorAPIDisabled, kAXErrorActionUnsupported, kAXErrorCannotComplete,
kAXErrorInvalidUIElement, kAXErrorSuccess,
kAXErrorAPIDisabled, kAXErrorActionUnsupported, kAXErrorAttributeUnsupported,
kAXErrorCannotComplete, kAXErrorInvalidUIElement, kAXErrorSuccess,
};
use agent_desktop_core::{DeliveryDisposition, ErrorCode, RetryDisposition};
@ -99,6 +121,43 @@ mod tests {
assert!(!result.unwrap());
}
#[test]
fn perform_codes_are_graded_by_how_much_they_prove() {
use super::{PerformSignal, classify_perform};
assert_eq!(
classify_perform("AXPress", kAXErrorSuccess).unwrap(),
PerformSignal::ReportedDelivered
);
assert_eq!(
classify_perform("AXPress", kAXErrorActionUnsupported).unwrap(),
PerformSignal::ReportedUnsupported
);
assert_eq!(
classify_perform("AXOpen", kAXErrorAttributeUnsupported).unwrap(),
PerformSignal::Uninformative
);
}
/// `tree::action_list` reads `kAXErrorFailure` as "this element implements
/// no such action". A perform must not read the same code as a failure, or
/// an element the reader called action-less becomes an error when acted on.
#[test]
fn perform_and_capability_reads_agree_on_the_appkit_absence_code() {
assert_eq!(
super::classify_perform("AXScrollToVisible", accessibility_sys::kAXErrorFailure)
.unwrap(),
super::PerformSignal::ReportedUnsupported
);
}
#[test]
fn perform_keeps_reporting_transport_failures_as_errors() {
let error = super::classify_perform("AXOpen", kAXErrorInvalidUIElement)
.expect_err("a stale element must still fail closed");
assert_eq!(error.code, ErrorCode::StaleRef);
}
#[test]
fn invalid_element_is_stale_and_safe_to_retry_with_a_fresh_ref() {
let error = classify("AXPress", "perform", kAXErrorInvalidUIElement)

30
crates/macos/src/actions/chain_defs.rs
View file

@ -16,11 +16,25 @@ mod imp {
count: 1,
},
ChainStep::Action("AXPress"),
ChainStep::Action("AXOpen"),
ChainStep::CustomWithDeadline {
label: "activate_descendant",
func: crate::actions::activate_descendant::activate_descendant,
},
ChainStep::CustomWithDeadline {
label: "select_within_container",
func: crate::actions::container_select::select_within_container,
},
ChainStep::Action("AXConfirm"),
],
suggestion: "Target an element that advertises Click or use an explicit point click.",
continue_after_unverified_delivery: false,
};
/// Continues past an unverified delivery because an `AXShowMenu` that
/// reports success without opening a menu must not consume the fallbacks
/// behind it. Each step re-checks for an open menu first, so continuing
/// cannot raise a second one.
pub(crate) static RIGHT_CLICK_CHAIN: ChainDef = ChainDef {
steps: &[
ChainStep::CGClick {
@ -49,7 +63,7 @@ mod imp {
},
],
suggestion: "Try 'mouse-click --button right --xy X,Y'.",
continue_after_unverified_delivery: false,
continue_after_unverified_delivery: true,
};
pub(crate) static EXPAND_CHAIN: ChainDef = ChainDef {
@ -89,7 +103,19 @@ mod imp {
};
pub(crate) static SEMANTIC_CLICK_CHAIN: ChainDef = ChainDef {
steps: &[ChainStep::Action("AXPress")],
steps: &[
ChainStep::Action("AXPress"),
ChainStep::Action("AXOpen"),
ChainStep::CustomWithDeadline {
label: "activate_descendant",
func: crate::actions::activate_descendant::activate_descendant,
},
ChainStep::CustomWithDeadline {
label: "select_within_container",
func: crate::actions::container_select::select_within_container,
},
ChainStep::Action("AXConfirm"),
],
suggestion: "Target an element that advertises Click.",
continue_after_unverified_delivery: false,
};

15
crates/macos/src/actions/chain_menu_steps.rs
View file

@ -80,7 +80,7 @@ fn show_menu_on_element(
let Some(pid) = crate::system::app_ops::pid_from_element(element, deadline) else {
return Ok(DeliveryOutcome::NotDelivered);
};
if menu_is_open(pid, deadline)? {
if menu_probe(pid, deadline) {
return Ok(DeliveryOutcome::NotDelivered);
}
prepare(element, deadline)?;
@ -102,7 +102,7 @@ fn show_menu_or_press(
let Some(pid) = crate::system::app_ops::pid_from_element(element, deadline) else {
return Ok(DeliveryOutcome::NotDelivered);
};
if menu_is_open(pid, deadline)? {
if menu_probe(pid, deadline) {
return Ok(DeliveryOutcome::NotDelivered);
}
for action in ["AXShowMenu", "AXPress"] {
@ -120,7 +120,7 @@ fn show_menu_or_press(
fn wait_for_new_menu(pid: i32, deadline: Deadline) -> Result<bool, AdapterError> {
let local_end = std::time::Instant::now() + std::time::Duration::from_millis(600);
loop {
if menu_is_open(pid, deadline)? {
if menu_probe(pid, deadline) {
return Ok(true);
}
if deadline.is_expired() || std::time::Instant::now() >= local_end {
@ -131,8 +131,13 @@ fn wait_for_new_menu(pid: i32, deadline: Deadline) -> Result<bool, AdapterError>
}
}
fn menu_is_open(pid: i32, deadline: Deadline) -> Result<bool, AdapterError> {
crate::tree::surfaces::is_menu_open(pid, instant(deadline)?)
/// An application in menu tracking stops answering child reads, so a failed
/// probe is the expected shape of "a menu is up" rather than an application
/// fault. Verification must never turn a delivered action into an error.
fn menu_probe(pid: i32, deadline: Deadline) -> bool {
instant(deadline)
.and_then(|instant| crate::tree::surfaces::is_menu_open(pid, instant))
.unwrap_or(false)
}
fn select_containing_item(element: &AXElement, deadline: Deadline) -> Result<bool, AdapterError> {

5
crates/macos/src/actions/chain_step_exec.rs
View file

@ -22,10 +22,7 @@ mod imp {
match step {
ChainStep::Action(name) => {
prepare(el, ctx.deadline)?;
Ok(DeliveryOutcome::from_delivery(
ax_helpers::try_ax_action_or_err(el, name, ctx.deadline)?,
false,
))
ax_helpers::perform_observed_action(el, name, ctx.deadline)
}
ChainStep::SetBool { attr, value } => {

145
crates/macos/src/actions/container_select.rs Normal file
View file

@ -0,0 +1,145 @@
pub(crate) const SELECTED: &str = "AXSelected";
/// Kept narrow so the settability probe never runs on generic containers.
pub(crate) fn role_activates_by_selection(role: &str) -> bool {
matches!(
role,
"row" | "treeitem" | "cell" | "listitem" | "option" | "tab"
)
}
#[cfg(target_os = "macos")]
mod imp {
use agent_desktop_core::{AdapterError, Deadline};
use core_foundation::{array::CFArray, base::TCFType};
use crate::actions::chain_delivery::DeliveryOutcome;
use crate::tree::AXElement;
use super::SELECTED;
/// `AXOutline` and `AXTable` make `AXSelectedRows` writable; `AXList` and
/// `AXBrowser` columns make `AXSelectedChildren` writable. Most elements
/// publish both names and accept writes to only one.
const SELECTION_ATTRIBUTES: [&str; 2] = ["AXSelectedRows", "AXSelectedChildren"];
const MAX_ANCESTOR_WALK: usize = 6;
const MAX_SELECTION_READBACK: usize = 64;
/// Climbs from the target because the clickable label is usually a
/// descendant of the selectable row. Every write is verified by readback.
pub(crate) fn select_within_container(
element: &AXElement,
deadline: Deadline,
) -> Result<DeliveryOutcome, AdapterError> {
let mut member = element.clone();
for _ in 0..MAX_ANCESTOR_WALK {
if select_member_directly(&member, deadline)? {
return Ok(DeliveryOutcome::DeliveredVerified);
}
let Some(parent) = parent_of(&member, deadline) else {
return Ok(DeliveryOutcome::NotDelivered);
};
if select_member_in_container(&parent, &member, deadline)? {
return Ok(DeliveryOutcome::DeliveredVerified);
}
member = parent;
}
Ok(DeliveryOutcome::NotDelivered)
}
fn select_member_directly(
member: &AXElement,
deadline: Deadline,
) -> Result<bool, AdapterError> {
if !crate::actions::ax_helpers::is_attr_settable(member, SELECTED, deadline)? {
return Ok(false);
}
crate::actions::ax_helpers::set_ax_bool_or_err(member, SELECTED, true, deadline)?;
Ok(crate::tree::attributes::copy_bool_attr(member, SELECTED, deadline) == Some(true))
}
fn select_member_in_container(
container: &AXElement,
member: &AXElement,
deadline: Deadline,
) -> Result<bool, AdapterError> {
for attribute in SELECTION_ATTRIBUTES {
if !crate::actions::ax_helpers::is_attr_settable(container, attribute, deadline)? {
continue;
}
let members = selection_array(member);
let cf_attr = core_foundation::string::CFString::new(attribute);
let error = crate::tree::ax_ipc::set_attribute_value(
container,
cf_attr.as_concrete_TypeRef(),
members.as_CFTypeRef(),
deadline,
)?;
if error == accessibility_sys::kAXErrorSuccess
&& holds_selection(container, member, attribute, deadline)
{
return Ok(true);
}
}
Ok(false)
}
/// `AXElement` does not implement `TCFType`, so the array is built through
/// the CoreFoundation C API.
fn selection_array(member: &AXElement) -> CFArray {
let values = [member.0 as *const std::ffi::c_void];
unsafe {
let raw = core_foundation_sys::array::CFArrayCreate(
std::ptr::null(),
values.as_ptr(),
1,
&core_foundation_sys::array::kCFTypeArrayCallBacks,
);
CFArray::wrap_under_create_rule(raw)
}
}
fn parent_of(element: &AXElement, deadline: Deadline) -> Option<AXElement> {
crate::tree::attributes::copy_element_attr_result(element, "AXParent", deadline)
.ok()
.flatten()
}
fn holds_selection(
container: &AXElement,
member: &AXElement,
attribute: &str,
deadline: Deadline,
) -> bool {
crate::tree::attributes::copy_ax_array_prefix_result(
container,
attribute,
MAX_SELECTION_READBACK,
deadline,
)
.ok()
.flatten()
.is_some_and(|selected| {
selected
.iter()
.any(|entry| crate::tree::capabilities::same_element(entry, member))
})
}
}
#[cfg(not(target_os = "macos"))]
mod imp {
use agent_desktop_core::{AdapterError, Deadline};
use crate::actions::chain_delivery::DeliveryOutcome;
use crate::tree::AXElement;
pub(crate) fn select_within_container(
_element: &AXElement,
_deadline: Deadline,
) -> Result<DeliveryOutcome, AdapterError> {
Ok(DeliveryOutcome::NotDelivered)
}
}
pub(crate) use imp::select_within_container;

3
crates/macos/src/actions/mod.rs
View file

@ -1,3 +1,5 @@
pub(crate) mod activate_descendant;
pub(crate) mod activation_effect;
mod adapter;
pub(crate) mod ax_helpers;
#[cfg(target_os = "macos")]
@ -13,6 +15,7 @@ mod chain_step;
pub(crate) mod chain_step_exec;
pub(crate) mod chain_value_write;
pub(crate) mod chain_verify;
pub(crate) mod container_select;
pub(crate) mod delivery_tracker;
pub(crate) mod dispatch;
pub(crate) mod extras;

6
crates/macos/src/actions/physical_keyboard.rs
View file

@ -158,17 +158,13 @@ fn focused_element_matches(
pid: i32,
deadline: Deadline,
) -> Result<bool, AdapterError> {
use core_foundation::base::{CFEqual, CFTypeRef};
let app = crate::tree::element_for_pid(pid);
prepare(&app, deadline)?;
let result =
crate::tree::attributes::copy_element_attr_result(&app, "AXFocusedUIElement", deadline);
ensure_budget(deadline)?;
let focused = result.map_err(|error| read_error("AXFocusedUIElement", error))?;
Ok(focused.is_some_and(|focused| unsafe {
CFEqual(focused.0 as CFTypeRef, expected.0 as CFTypeRef) != 0
}))
Ok(focused.is_some_and(|focused| crate::tree::same_element(&focused, expected)))
}
fn prepare(element: &AXElement, deadline: Deadline) -> Result<(), AdapterError> {

29
crates/macos/src/actions/post_state.rs
View file

@ -1,6 +1,6 @@
use agent_desktop_core::{
Action, AdapterError, Deadline, ElementState, ErrorCode, EvidenceRequirements, LiveElement,
LiveIdentity, LocatorField, ObservationBudget, Rect,
LiveIdentity, LocatorField, Rect,
};
use std::time::{Duration, Instant};
@ -138,7 +138,7 @@ fn essential_live_evidence_complete(evidence: &agent_desktop_core::LocatorEviden
}
fn new_usage() -> crate::tree::observation_usage::ObservationUsage {
crate::tree::observation_usage::ObservationUsage::new(ObservationBudget::default())
crate::tree::observation_usage::ObservationUsage::with_defaults()
}
fn owning_window_bounds(
@ -159,17 +159,14 @@ fn owning_window_bounds(
crate::tree::element_bounds::read_bounds_with_deadline(&window, deadline_instant(deadline)?)
}
/// The viewport an element is clipped by. `AXTopLevelUIElement` is not a
/// substitute: for menu content it resolves to the menu bar, a 29-point strip
/// that reports every open menu item as offscreen. An element with no window is
/// drawn on its own surface, so its clipping viewport is simply unknown.
fn first_owning_container(
mut read: impl FnMut(&'static str) -> Result<Option<AXElement>, i32>,
) -> Result<Option<AXElement>, (&'static str, i32)> {
for attribute in ["AXWindow", "AXTopLevelUIElement"] {
match read(attribute) {
Ok(Some(element)) => return Ok(Some(element)),
Ok(None) => {}
Err(error) => return Err((attribute, error)),
}
}
Ok(None)
read("AXWindow").map_err(|error| ("AXWindow", error))
}
fn known_role(role: &LocatorField<String>) -> Result<String, AdapterError> {
@ -212,7 +209,7 @@ fn element_state_from_attrs(
let states = crate::tree::state_reader::states_from_element(element, &attrs, &role, &context);
let enabled = Some(attrs.states.enabled);
let hidden = hidden_state(attrs.states.semantic.hidden);
let offscreen = offscreen(attrs.bounds, window_bounds);
let offscreen = crate::tree::state_reader::offscreen(attrs.bounds, window_bounds);
Ok(ElementState {
role,
states,
@ -227,16 +224,6 @@ fn hidden_state(reported: Option<bool>) -> Option<bool> {
reported
}
fn offscreen(bounds: Option<Rect>, window: Option<Rect>) -> Option<bool> {
let (bounds, window) = bounds.zip(window)?;
Some(
bounds.x + bounds.width <= window.x
|| bounds.x >= window.x + window.width
|| bounds.y + bounds.height <= window.y
|| bounds.y >= window.y + window.height,
)
}
#[cfg(test)]
fn normalized_role(ax_role: Option<&str>, ax_subrole: Option<&str>) -> String {
ax_role

12
crates/macos/src/actions/post_state_tests.rs
View file

@ -111,18 +111,20 @@ fn element_visibility_preserves_live_hidden_evidence_for_every_role() {
assert_eq!(hidden_state(Some(true)), Some(true));
}
/// A menu item's `AXTopLevelUIElement` is the menu bar, whose 29-point height
/// clips every open menu item. A window-less element therefore reports no
/// clipping viewport rather than a wrong one.
#[test]
fn top_level_container_is_used_only_when_window_is_authoritatively_absent() {
fn a_window_less_element_reports_no_clipping_viewport() {
let mut attributes = Vec::new();
let container = first_owning_container(|attribute| {
attributes.push(attribute);
Ok((attribute == "AXTopLevelUIElement")
.then(|| crate::tree::AXElement(std::ptr::null_mut())))
Ok(None)
})
.unwrap();
assert!(container.is_some());
assert_eq!(attributes, ["AXWindow", "AXTopLevelUIElement"]);
assert!(container.is_none());
assert_eq!(attributes, ["AXWindow"]);
}
#[test]

54
crates/macos/src/actions/scroll.rs
View file

@ -6,6 +6,47 @@ use crate::tree::AXElement;
const MAX_SCROLL_AMOUNT: u32 = 1_000;
/// A responder can serve a page action while answering an uninformative code
/// for it, which leaves the return value unable to prove anything. The content
/// position is the observable that can, so the scroll is judged by whether the
/// first child actually moved.
fn paged_scroll_moved_content(
target: &AXElement,
direction: &Direction,
amount: u32,
deadline: Deadline,
) -> Result<bool, AdapterError> {
let Some(before) = first_child_origin(target, deadline)? else {
return Ok(false);
};
for _ in 0..amount.max(1) {
try_action(target, page_action(direction), deadline)?;
}
let Some(after) = first_child_origin(target, deadline)? else {
return Ok(false);
};
Ok((before.0 - after.0).abs() > f64::EPSILON || (before.1 - after.1).abs() > f64::EPSILON)
}
fn first_child_origin(
target: &AXElement,
deadline: Deadline,
) -> Result<Option<(f64, f64)>, AdapterError> {
let instant = crate::tree::locator_deadline::from_operation(deadline)?;
let Some(child) =
crate::tree::attributes::copy_ax_array_prefix_result(target, "AXChildren", 1, instant)
.ok()
.flatten()
.and_then(|children| children.into_iter().next())
else {
return Ok(None);
};
Ok(
crate::tree::element_bounds::read_bounds_with_deadline(&child, instant)?
.map(|bounds| (bounds.x, bounds.y)),
)
}
pub(crate) fn ax_scroll(
element: &AXElement,
direction: &Direction,
@ -37,11 +78,20 @@ pub(crate) fn ax_scroll(
if perform_repeated_action(target, page_action(direction), amount, deadline)? {
return Ok((StepMechanism::SemanticApi, false));
}
if paged_scroll_moved_content(target, direction, amount, deadline)? {
return Ok((StepMechanism::SemanticApi, true));
}
Err(AdapterError::new(
ErrorCode::ActionNotSupported,
"No scroll mechanism found on element",
"Element advertises Scroll but no scroll mechanism moved its content",
)
.with_suggestion("Element may not be scrollable, or try the parent container."))
.with_details(serde_json::json!({
"kind": "scroll_advertised_but_inert",
}))
.with_suggestion(
"The application publishes the scroll actions without implementing them. \
Try the parent container, or use '--headed' for a physical wheel scroll.",
))
}
fn accept_optional_visibility_result(

5
crates/macos/src/actions/scroll_into_view.rs
View file

@ -66,8 +66,9 @@ mod imp {
let instant = crate::tree::locator_deadline::from_operation(deadline)?;
let bounds = crate::tree::element_bounds::read_bounds_with_deadline(element, instant)?
.ok_or_else(|| AdapterError::new(ErrorCode::ActionFailed, "Target has no bounds"))?;
let window = crate::tree::surface_read::element(element, "AXWindow", instant)?
.ok_or_else(|| AdapterError::new(ErrorCode::ActionFailed, "Target has no window"))?;
let Some(window) = crate::tree::surface_read::element(element, "AXWindow", instant)? else {
return Ok(None);
};
let window_bounds = crate::tree::element_bounds::read_bounds_with_deadline(
&window, instant,
)?

2
crates/macos/src/system/adapter.rs
View file

@ -69,7 +69,7 @@ impl SystemOps for MacOSAdapter {
id: &str,
options: &agent_desktop_core::launch_options::LaunchOptions,
lease: &InteractionLease,
) -> Result<WindowInfo, AdapterError> {
) -> Result<agent_desktop_core::launch_result::LaunchResult, AdapterError> {
crate::system::launch::launch_app_impl(id, options, lease.deadline())
}

2
crates/macos/src/system/app_inventory_tests.rs
View file

@ -6,6 +6,7 @@ fn app(name: &str, pid: u32) -> AppInfo {
pid: agent_desktop_core::ProcessId::new(pid),
bundle_id: None,
process_instance: Some(format!("instance-{pid}")),
presentation: None,
}
}
@ -15,6 +16,7 @@ fn app_with_bundle(name: &str, pid: u32, bundle_id: &str) -> AppInfo {
pid: agent_desktop_core::ProcessId::new(pid),
bundle_id: Some(bundle_id.to_string()),
process_instance: Some(format!("instance-{pid}")),
presentation: None,
}
}

18
crates/macos/src/system/appkit_bridge.m
View file

@ -17,6 +17,24 @@ typedef struct {
size_t length;
} AgentDesktopBytesResult;
// Reports whether a running application has finished starting up.
// -1 no such process, 0 still starting, 1 finished.
int32_t agent_desktop_app_finished_launching(int32_t pid) {
@try {
@autoreleasepool {
NSRunningApplication *app =
[NSRunningApplication runningApplicationWithProcessIdentifier:pid];
if (app == nil) {
return -1;
}
return app.isFinishedLaunching ? 1 : 0;
}
} @catch (NSException *exception) {
(void)exception;
return -1;
}
}
AgentDesktopTerminateResult agent_desktop_terminate_application(
int32_t pid,
double expectedLaunchTime,

28
crates/macos/src/system/appkit_bridge.rs
View file

@ -107,6 +107,33 @@ fn bridge_error(operation: &str, status: u8, delivery_started: bool) -> AdapterE
.with_disposition(disposition)
}
/// Where a process is in its startup. `NoRecord` covers both a process that
/// exited and one that has not registered with the window server yet, so it
/// answers neither question on its own and the caller has to ask libproc which
/// of the two it is.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum StartupState {
Starting,
Finished,
NoRecord,
}
/// An application that finished starting up has already created whatever
/// windows its launch produces.
#[cfg(target_os = "macos")]
pub(crate) fn startup_state(pid: i32) -> StartupState {
match unsafe { agent_desktop_app_finished_launching(pid) } {
0 => StartupState::Starting,
1 => StartupState::Finished,
_ => StartupState::NoRecord,
}
}
#[cfg(not(target_os = "macos"))]
pub(crate) fn startup_state(_pid: i32) -> StartupState {
StartupState::NoRecord
}
#[cfg(target_os = "macos")]
unsafe extern "C" {
fn agent_desktop_terminate_application(
@ -114,6 +141,7 @@ unsafe extern "C" {
expected_launch_time: f64,
force: u8,
) -> TerminateResult;
fn agent_desktop_app_finished_launching(pid: i32) -> i32;
fn agent_desktop_ensure_cocoa_multithreaded() -> u8;
fn agent_desktop_copy_workspace_snapshot_json() -> BytesResult;
fn agent_desktop_free_bridge_bytes(bytes: *mut u8);

157
crates/macos/src/system/launch.rs
View file

@ -1,12 +1,13 @@
use agent_desktop_core::{
AdapterError, AppInfo, Deadline, DeliverySemantics, ErrorCode, WindowInfo,
launch_options::LaunchOptions,
launch_options::LaunchOptions, launch_result::LaunchResult,
};
use std::time::{Duration, Instant};
const MAX_ARGUMENT_COUNT: usize = 256;
const MAX_ENVIRONMENT_COUNT: usize = 256;
const MAX_LAUNCH_TEXT_BYTES: usize = 1024 * 1024;
const STARTUP_GRACE: Duration = Duration::from_millis(1500);
enum LaunchTarget {
Existing { pid: i32, process_instance: String },
@ -18,7 +19,7 @@ pub(crate) fn launch_app_impl(
id: &str,
options: &LaunchOptions,
parent_deadline: Deadline,
) -> Result<WindowInfo, AdapterError> {
) -> Result<LaunchResult, AdapterError> {
validate_app_identifier(id).map_err(before_launch)?;
validate_launch_options(options).map_err(before_launch)?;
let deadline = if options.timeout_ms == 0 {
@ -27,45 +28,140 @@ pub(crate) fn launch_app_impl(
parent_deadline.capped(Duration::from_millis(options.timeout_ms))
};
ensure_launch_budget(deadline, id).map_err(before_launch)?;
let timeout_ms = options.timeout_ms;
let initial = matching_apps(id, deadline).map_err(before_launch)?;
if options.attach_if_running && initial.len() == 1 {
let app = &initial[0];
let instance = required_instance(app).map_err(before_launch)?;
if options.attach_if_running && initial.len() == 1 && !options.activate {
let app = initial[0].clone();
let instance = required_instance(&app).map_err(before_launch)?;
let pid = crate::system::process_identity::to_pid_t(app.pid).map_err(before_launch)?;
if let Some(window) = exact_window(pid, &instance, deadline).map_err(before_launch)? {
return Ok(window);
}
let window = exact_window(pid, &instance, deadline).map_err(before_launch)?;
return Ok(result_from_app(&app, window));
}
let target = launch_target(options, initial).map_err(before_launch)?;
let launched = crate::system::launch_workspace::open(id, options, deadline)?;
validate_launched_target(&target, &launched).map_err(after_launch)?;
let mut poll_interval = Duration::from_millis(50);
let window =
settled_window(launched.0, &launched.1, options, deadline).map_err(after_launch)?;
result_from_launched(&launched, window, id).map_err(after_launch)
}
/// Waits only for the windows the launch itself produces. Starting up is what
/// creates them, so once the application reports that it finished starting up,
/// every window it was going to open on its own already exists and further
/// polling can only run out the deadline. An application that opens its first
/// window in response to being brought forward — TextEdit, Preview, any
/// document-based application — reports no window here, and `activate` is how a
/// caller asks for one instead of waiting for one it never requested.
#[cfg(target_os = "macos")]
fn settled_window(
pid: i32,
process_instance: &str,
options: &LaunchOptions,
deadline: Deadline,
) -> Result<Option<WindowInfo>, AdapterError> {
let mut poll_interval = Duration::from_millis(25);
let mut grace_ends_at = None;
loop {
if let Some(window) =
exact_window(launched.0, &launched.1, deadline).map_err(after_launch)?
{
return Ok(window);
if let Some(window) = exact_window(pid, process_instance, deadline)? {
return Ok(Some(window));
}
if !should_poll_after_first_observation(options.timeout_ms) {
return Err(launch_no_window_error(id, timeout_ms, &launched));
if options.timeout_ms == 0
|| (!options.activate && grace_over(grace_ends_at, Instant::now()))
{
return Ok(None);
}
if grace_ends_at.is_none() && startup_finished(pid, process_instance)? {
grace_ends_at = Instant::now().checked_add(STARTUP_GRACE);
}
let remaining = deadline.remaining();
if remaining.is_zero() {
return Err(launch_no_window_error(id, timeout_ms, &launched));
return Ok(None);
}
std::thread::sleep(poll_interval.min(remaining));
poll_interval = (poll_interval * 3 / 2).min(Duration::from_millis(250));
}
}
/// Ends the wait when the application has created whatever windows its launch
/// produces. The window server keeps no record of a process that exited, and
/// none of one that has not registered yet, so libproc decides which happened:
/// a process that is gone ends the launch with an error rather than an answer
/// about windows it will never open.
#[cfg(target_os = "macos")]
fn startup_finished(pid: i32, process_instance: &str) -> Result<bool, AdapterError> {
use crate::system::appkit_bridge::StartupState;
match crate::system::appkit_bridge::startup_state(pid) {
StartupState::Starting => Ok(false),
StartupState::Finished => Ok(true),
StartupState::NoRecord => {
if crate::system::process_identity::matches_instance(pid, process_instance)? {
Ok(true)
} else {
Err(launch_target_gone(pid))
}
}
}
}
#[cfg(target_os = "macos")]
fn launch_target_gone(pid: i32) -> AdapterError {
AdapterError::new(
ErrorCode::AppUnresponsive,
"Launched application exited before it presented a window",
)
.with_details(serde_json::json!({ "pid": pid, "complete": false }))
.with_suggestion("Check the application's own launch requirements, then retry.")
}
/// The grace covers the gap between an application reporting that it started
/// and its first window reaching the window server. Waiting starts running out
/// only once there is a completed startup to measure from.
#[cfg(target_os = "macos")]
fn grace_over(grace_ends_at: Option<Instant>, now: Instant) -> bool {
grace_ends_at.is_some_and(|end| now >= end)
}
/// The attaching path reports the display name, so the launching path has to
/// report the same thing for the same field. A window already carries it; the
/// requested identifier is the fallback when there is no window to ask.
#[cfg(target_os = "macos")]
fn launched_display_name(window: Option<&WindowInfo>, id: &str) -> String {
window
.map(|window| window.app.clone())
.unwrap_or_else(|| id.to_owned())
}
#[cfg(target_os = "macos")]
fn result_from_app(app: &AppInfo, window: Option<WindowInfo>) -> LaunchResult {
LaunchResult {
app: app.name.clone(),
pid: app.pid,
process_instance: app.process_instance.clone(),
window,
}
}
#[cfg(target_os = "macos")]
fn result_from_launched(
launched: &(i32, String),
window: Option<WindowInfo>,
id: &str,
) -> Result<LaunchResult, AdapterError> {
Ok(LaunchResult {
app: launched_display_name(window.as_ref(), id),
pid: agent_desktop_core::ProcessId::try_from(launched.0)
.map_err(|_| AdapterError::internal("Launched process identifier is out of range"))?,
process_instance: Some(launched.1.clone()),
window,
})
}
#[cfg(not(target_os = "macos"))]
pub(crate) fn launch_app_impl(
_id: &str,
_options: &LaunchOptions,
_deadline: Deadline,
) -> Result<WindowInfo, AdapterError> {
) -> Result<LaunchResult, AdapterError> {
Err(AdapterError::not_supported("launch_app"))
}
@ -209,11 +305,6 @@ fn before_launch(error: AdapterError) -> AdapterError {
error.with_disposition(DeliverySemantics::not_delivered())
}
#[cfg(target_os = "macos")]
fn should_poll_after_first_observation(timeout_ms: u64) -> bool {
timeout_ms > 0
}
#[cfg(target_os = "macos")]
fn validate_app_identifier(id: &str) -> Result<(), AdapterError> {
let safe_bundle_id = !looks_like_bundle_id(id)
@ -277,26 +368,6 @@ pub(crate) fn looks_like_bundle_id(id: &str) -> bool {
id.contains('.') && !id.ends_with(".app") && !id.contains(' ')
}
#[cfg(target_os = "macos")]
fn launch_no_window_error(id: &str, timeout_ms: u64, launched: &(i32, String)) -> AdapterError {
AdapterError::new(
ErrorCode::WindowNotFound,
format!(
"Application started, but no exact accessible window appeared within {timeout_ms} ms"
),
)
.with_details(serde_json::json!({
"app_name": id,
"pid": launched.0,
"process_instance": launched.1,
"retry_safe": false,
}))
.with_disposition(DeliverySemantics::delivered_unverified())
.with_suggestion(
"Inspect list-apps and list-windows for the returned process; do not repeat the launch blindly.",
)
}
#[cfg(target_os = "macos")]
fn after_launch(error: AdapterError) -> AdapterError {
error.with_disposition(DeliverySemantics::delivered_unverified())

2
crates/macos/src/system/launch_completion.rs
View file

@ -147,7 +147,7 @@ fn validate_result(
)
})
})?;
if !identity.matches_launch_time(result.launch_time) {
if identity.conflicts_with_launch_time(result.launch_time) {
return Err(callback_error(
result,
AdapterError::new(

22
crates/macos/src/system/launch_tests.rs
View file

@ -45,9 +45,12 @@ fn rejects_paths_and_unsafe_bundle_identifiers() {
}
#[test]
fn zero_wait_still_launches_but_never_polls_after_first_observation() {
assert!(!should_poll_after_first_observation(0));
assert!(should_poll_after_first_observation(1));
fn waiting_ends_only_after_a_completed_startup_plus_its_grace() {
let now = std::time::Instant::now();
assert!(!grace_over(None, now));
assert!(!grace_over(now.checked_add(STARTUP_GRACE), now));
assert!(grace_over(Some(now), now));
}
#[test]
@ -87,16 +90,3 @@ fn launch_options_enforce_a_bounded_text_budget() {
assert_eq!(error.code, ErrorCode::InvalidArgs);
}
#[test]
fn launch_no_window_error_keeps_identifier_in_details_only() {
let marker = "MARKER_APP_ID_9f31c4";
let error = launch_no_window_error(marker, 5000, &(77, "generation".into()));
assert!(!error.message.contains(marker));
assert!(error.message.contains("5000"));
let details = error.details.expect("details");
assert_eq!(details["app_name"], marker);
assert_eq!(details["pid"], 77);
assert_eq!(details["retry_safe"], false);
}

2
crates/macos/src/system/launch_workspace.rs
View file

@ -18,7 +18,7 @@ pub(crate) fn open(
"bundle_id": super::launch::looks_like_bundle_id(id),
"arguments": options.args,
"environment": options.env,
"activates": false,
"activates": options.activate,
"prompts": false,
"substitution": false,
"new_instance": creates_new_instance(options),

1
crates/macos/src/system/process_apps.rs
View file

@ -107,6 +107,7 @@ fn parse_apps(text: &str) -> Result<Vec<AppInfo>, AdapterError> {
pid: crate::system::process_identity::from_pid_t(pid)?,
bundle_id: None,
process_instance: None,
presentation: None,
});
}
}

16
crates/macos/src/system/process_identity.rs
View file

@ -92,6 +92,18 @@ impl ProcessIdentity {
(self.launch_time_seconds() - launch_time).abs() <= MAX_LAUNCH_TIME_DELTA_SECONDS
}
/// Absent evidence is not conflicting evidence. NSWorkspace reports no
/// launch date for a process it did not start — a system application
/// already running at login answers zero — so there is nothing to
/// reconcile against libproc and the caller must rely on its other
/// identity checks instead of failing.
pub(crate) fn conflicts_with_launch_time(self, launch_time: f64) -> bool {
if !launch_time.is_finite() || launch_time <= 0.0 {
return false;
}
!self.matches_launch_time(launch_time)
}
pub(crate) fn still_matches(self) -> Result<bool, AdapterError> {
Ok(Self::capture(self.pid)?.is_some_and(|current| current == self))
}
@ -279,6 +291,10 @@ mod tests {
assert!(identity.matches_launch_time(1_700_000_000.25));
assert!(!identity.matches_launch_time(1_700_000_006.0));
assert!(!identity.matches_launch_time(0.0));
assert!(identity.conflicts_with_launch_time(1_700_000_006.0));
assert!(!identity.conflicts_with_launch_time(1_700_000_000.25));
assert!(!identity.conflicts_with_launch_time(0.0));
assert!(!identity.conflicts_with_launch_time(f64::NAN));
}
#[test]

4
crates/macos/src/system/signals_tests.rs
View file

@ -43,12 +43,14 @@ fn matching_apps_filters_by_name_case_insensitively() {
pid: agent_desktop_core::ProcessId::new(42),
bundle_id: None,
process_instance: None,
presentation: None,
},
AppInfo {
name: "Finder".into(),
pid: agent_desktop_core::ProcessId::new(7),
bundle_id: None,
process_instance: None,
presentation: None,
},
],
);
@ -65,6 +67,7 @@ fn app_filter_with_no_constraints_preserves_the_complete_inventory() {
pid: agent_desktop_core::ProcessId::new(7),
bundle_id: None,
process_instance: None,
presentation: None,
}];
let filtered = filter_apps(&filter, apps);
@ -82,6 +85,7 @@ fn surfaces_for_apps_is_empty_without_an_app_or_pid_filter() {
pid: agent_desktop_core::ProcessId::new(1),
bundle_id: None,
process_instance: None,
presentation: None,
}];
let deadline = Instant::now() + Duration::from_secs(5);
let surfaces =

14
crates/macos/src/system/window_ax_state.rs
View file

@ -52,12 +52,24 @@ pub(crate) fn read_frontmost_until(
})
}
/// Frontmost-ness only selects whether a focused window is reported, and a busy
/// application answering "unknown" is not a reason to fail the command that
/// asked. Permission and API failures still propagate, because those describe
/// the caller's access rather than the application's state.
fn is_frontmost(app: &crate::tree::AXElement, deadline: Instant) -> Result<bool, AdapterError> {
match crate::tree::surface_read::boolean(app, "AXFrontmost", deadline) {
Ok(frontmost) => Ok(frontmost == Some(true)),
Err(error) if error.code == ErrorCode::PermDenied => Err(error),
Err(_) => Ok(false),
}
}
fn focused_identity(
app: &crate::tree::AXElement,
pid: i32,
deadline: Instant,
) -> Result<Option<AxWindowIdentity>, AdapterError> {
if crate::tree::surface_read::boolean(app, "AXFrontmost", deadline)? != Some(true) {
if !is_frontmost(app, deadline)? {
return Ok(None);
}
let Some(focused) = crate::tree::surface_read::element(app, "AXFocusedWindow", deadline)?

Some files were not shown because too many files have changed in this diff Show more