mirror of
https://github.com/lahfir/agent-desktop.git
synced 2026-08-06 22:18:59 +00:00
fix: harden macos stale ref resolution (#62)
* fix: harden macos stale ref resolution * fix: tighten stale ref fallback resolution * fix: fail closed for gone titled windows * refactor: tidy stale ref resolver naming
This commit is contained in:
parent
906deec406
commit
9f144c2caf
11 changed files with 286 additions and 186 deletions
|
|
@ -323,7 +323,7 @@ The `error` object may also carry an optional `details` object (e.g. the actiona
|
|||
- Refs are deterministic within a snapshot but NOT stable across snapshots if UI changed
|
||||
- Snapshot refs are stored by snapshot ID under `~/.agent-desktop/snapshots/{snapshot_id}/refmap.json`, with a `latest_snapshot_id` pointer for commands that omit `--snapshot`
|
||||
- `~/.agent-desktop/last_refmap.json` is written only as a latest-snapshot inspection artifact; command code must use `RefStore`
|
||||
- Action commands use optimistic re-identification: `(pid, role, name, bounds_hash)`. Return `STALE_REF` on mismatch.
|
||||
- Action commands use strict re-identification from platform-neutral `RefEntry` evidence: pid, role, path/source surface, role-conditional stable text identity, and bounds hash. Mutable control values are volatile and must not be treated as stable text identity. Return `STALE_REF` on mismatch and `AMBIGUOUS_TARGET` when multiple plausible live candidates remain.
|
||||
- Progressive traversal: `--skeleton` clamps depth to 3, annotates truncated containers with `children_count`. Named/described containers at boundary receive refs as drill-down targets
|
||||
- Drill-down: `--root @ref` starts from a previously-discovered ref with scoped invalidation (only that ref's subtree refs are replaced on re-drill)
|
||||
- RefMap size check: write-side guard prevents >1MB refmap files
|
||||
|
|
|
|||
|
|
@ -29,6 +29,11 @@ Refs are deterministic inside one snapshot but are not stable across UI changes.
|
|||
### RefMap
|
||||
The persisted mapping from refs to the identity evidence needed to re-identify elements later.
|
||||
|
||||
### Stable Text Identity
|
||||
The role-conditional text evidence used during strict ref resolution.
|
||||
|
||||
Names and descriptions can identify a ref when they are stable labels. Mutable control values, including text field content and value text promoted into an accessibility name, are volatile and do not identify the element by themselves. Core owns this policy so macOS, Windows, Linux, CLI, and FFI consumers share the same semantics.
|
||||
|
||||
### Stale Ref
|
||||
A ref whose stored identity no longer matches a live element strongly enough to act safely.
|
||||
|
||||
|
|
|
|||
|
|
@ -368,7 +368,7 @@ Errors include machine-readable codes and recovery hints:
|
|||
| `PERM_DENIED` | Accessibility permission not granted |
|
||||
| `ELEMENT_NOT_FOUND` | No element matched the ref or query |
|
||||
| `APP_NOT_FOUND` | Application not running or no windows |
|
||||
| `STALE_REF` | Ref is from a previous snapshot |
|
||||
| `STALE_REF` | Ref could not be re-identified in the live UI |
|
||||
| `AMBIGUOUS_TARGET` | Ref recovery matched multiple plausible targets |
|
||||
| `SNAPSHOT_NOT_FOUND` | Snapshot ID is missing or expired |
|
||||
| `POLICY_DENIED` | Physical/headed path blocked by policy |
|
||||
|
|
@ -392,7 +392,8 @@ Static elements (labels, groups, containers) appear in the tree for context but
|
|||
Reliability contract:
|
||||
|
||||
- `--session <id>` scopes the latest snapshot pointer to one caller or agent team; explicit `--snapshot <id>` resolves the saved snapshot directly.
|
||||
- Ref actions re-identify targets at action time: a moved unique target can proceed, while missing or changed identity returns `STALE_REF`.
|
||||
- Ref actions re-identify targets at action time: a moved unique target can proceed, while missing or changed stable identity returns `STALE_REF`.
|
||||
- Mutable value text is not treated as stable identity, so text fields and timers can keep resolving when the saved window, path, role, and bounds evidence still identify the same element.
|
||||
- Multiple plausible targets return `AMBIGUOUS_TARGET` instead of choosing arbitrarily.
|
||||
- Actions run an actionability preflight before dispatch: visibility, stability, enabled state, supported action, policy, and editability.
|
||||
- `wait --element @e3 --predicate actionable` polls until the target can be acted on.
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ pub mod permission_report;
|
|||
pub mod permission_state;
|
||||
pub mod ref_action;
|
||||
pub mod ref_alloc;
|
||||
pub mod ref_identity;
|
||||
pub mod refs;
|
||||
mod refs_lock;
|
||||
pub mod refs_store;
|
||||
|
|
|
|||
107
crates/core/src/ref_identity.rs
Normal file
107
crates/core/src/ref_identity.rs
Normal file
|
|
@ -0,0 +1,107 @@
|
|||
use crate::{adapter::SnapshotSurface, refs::RefEntry, roles::is_mutable_value_role};
|
||||
|
||||
/// Returns true when a saved ref has stable text identity beyond role/path/bounds.
|
||||
pub fn has_meaningful_identity(entry: &RefEntry) -> bool {
|
||||
stable_name(
|
||||
entry.role.as_str(),
|
||||
entry.name.as_deref(),
|
||||
entry.value.as_deref(),
|
||||
)
|
||||
.is_some()
|
||||
|| stable_value(entry.role.as_str(), entry.value.as_deref()).is_some()
|
||||
|| meaningful_text(entry.description.as_deref()).is_some()
|
||||
}
|
||||
|
||||
/// Compares saved ref identity against live text without treating mutable
|
||||
/// control values as stable identity.
|
||||
pub fn identity_matches(
|
||||
entry: &RefEntry,
|
||||
actual_name: Option<&str>,
|
||||
actual_value: Option<&str>,
|
||||
actual_description: Option<&str>,
|
||||
) -> bool {
|
||||
let expected_name = stable_name(
|
||||
entry.role.as_str(),
|
||||
entry.name.as_deref(),
|
||||
entry.value.as_deref(),
|
||||
);
|
||||
let expected_value = stable_value(entry.role.as_str(), entry.value.as_deref());
|
||||
let expected_description = meaningful_text(entry.description.as_deref());
|
||||
let actual_name = stable_name(entry.role.as_str(), actual_name, actual_value);
|
||||
let actual_value = stable_value(entry.role.as_str(), actual_value);
|
||||
let actual_description = meaningful_text(actual_description);
|
||||
|
||||
if let Some(expected) = expected_name {
|
||||
return match_primary_identity(expected, actual_name, actual_value);
|
||||
}
|
||||
if let Some(expected) = expected_value {
|
||||
return match_primary_identity(expected, actual_value, actual_name);
|
||||
}
|
||||
if let Some(expected) = expected_description {
|
||||
return match_primary_identity(expected, actual_description, actual_name);
|
||||
}
|
||||
|
||||
if is_mutable_value_role(entry.role.as_str()) {
|
||||
return true;
|
||||
}
|
||||
|
||||
actual_name.is_none() && actual_value.is_none() && actual_description.is_none()
|
||||
}
|
||||
|
||||
/// Allows a platform adapter to search replacement windows only when the saved
|
||||
/// ref has enough non-text evidence for the shared classifier to fail closed.
|
||||
/// A saved source-window title disables this fallback unless a platform first
|
||||
/// finds that title uniquely; otherwise the old titled window is considered gone.
|
||||
pub fn bounded_window_fallback_allowed(entry: &RefEntry) -> bool {
|
||||
matches!(entry.source_surface, SnapshotSurface::Window)
|
||||
&& entry.source_window_id.is_some()
|
||||
&& entry.source_window_title.is_none()
|
||||
&& entry.bounds_hash.is_some()
|
||||
}
|
||||
|
||||
fn match_primary_identity(
|
||||
expected: &str,
|
||||
actual_primary: Option<&str>,
|
||||
actual_fallback: Option<&str>,
|
||||
) -> bool {
|
||||
match actual_primary {
|
||||
Some(actual) => actual == expected,
|
||||
None => actual_fallback == Some(expected),
|
||||
}
|
||||
}
|
||||
|
||||
fn meaningful_text(value: Option<&str>) -> Option<&str> {
|
||||
value.filter(|text| !text.is_empty())
|
||||
}
|
||||
|
||||
fn stable_name<'a>(role: &str, name: Option<&'a str>, value: Option<&str>) -> Option<&'a str> {
|
||||
let name = meaningful_text(name)?;
|
||||
if is_mutable_value_role(role) && value_matches_name(meaningful_text(value), name) {
|
||||
None
|
||||
} else {
|
||||
Some(name)
|
||||
}
|
||||
}
|
||||
|
||||
fn stable_value<'a>(role: &str, value: Option<&'a str>) -> Option<&'a str> {
|
||||
(!is_mutable_value_role(role))
|
||||
.then(|| meaningful_text(value))
|
||||
.flatten()
|
||||
}
|
||||
|
||||
fn value_matches_name(value: Option<&str>, name: &str) -> bool {
|
||||
value == Some(name)
|
||||
|| numeric_text(value)
|
||||
.zip(numeric_text(Some(name)))
|
||||
.is_some_and(|(value, name)| value == name)
|
||||
}
|
||||
|
||||
fn numeric_text(value: Option<&str>) -> Option<f64> {
|
||||
value
|
||||
.and_then(|text| text.parse::<f64>().ok())
|
||||
.filter(|number| number.is_finite())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "ref_identity_tests.rs"]
|
||||
mod tests;
|
||||
|
|
@ -14,7 +14,7 @@ fn entry() -> RefEntry {
|
|||
source_app: None,
|
||||
source_window_id: None,
|
||||
source_window_title: None,
|
||||
source_surface: agent_desktop_core::adapter::SnapshotSurface::Window,
|
||||
source_surface: SnapshotSurface::Window,
|
||||
root_ref: None,
|
||||
path_is_absolute: false,
|
||||
path: smallvec::SmallVec::new(),
|
||||
|
|
@ -98,6 +98,42 @@ fn mutable_value_role_does_not_go_stale_when_value_changes() {
|
|||
assert!(identity_matches(&entry, None, Some("changed"), None));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unnamed_mutable_value_role_does_not_go_stale_when_content_becomes_name() {
|
||||
let mut entry = entry();
|
||||
entry.role = "textfield".into();
|
||||
|
||||
assert!(!has_meaningful_identity(&entry));
|
||||
assert!(identity_matches(
|
||||
&entry,
|
||||
Some("typed document text"),
|
||||
Some("typed document text"),
|
||||
None
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mutable_value_text_promoted_to_name_is_not_stable_identity() {
|
||||
let mut entry = entry();
|
||||
entry.role = "textfield".into();
|
||||
entry.name = Some("00:01".into());
|
||||
entry.value = Some("00:01".into());
|
||||
|
||||
assert!(!has_meaningful_identity(&entry));
|
||||
assert!(identity_matches(&entry, Some("00:06"), Some("00:06"), None));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn formatted_numeric_mutable_value_promoted_to_name_is_not_stable_identity() {
|
||||
let mut entry = entry();
|
||||
entry.role = "slider".into();
|
||||
entry.name = Some("50".into());
|
||||
entry.value = Some("50.0".into());
|
||||
|
||||
assert!(!has_meaningful_identity(&entry));
|
||||
assert!(identity_matches(&entry, Some("51"), Some("51.0"), None));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn named_mutable_value_role_still_uses_name_identity() {
|
||||
let mut entry = entry();
|
||||
|
|
@ -119,3 +155,35 @@ fn named_mutable_value_role_still_uses_name_identity() {
|
|||
None
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mutable_role_label_different_from_value_remains_stable_identity() {
|
||||
let mut entry = entry();
|
||||
entry.role = "combobox".into();
|
||||
entry.name = Some("Font".into());
|
||||
entry.value = Some("Helvetica".into());
|
||||
|
||||
assert!(has_meaningful_identity(&entry));
|
||||
assert!(identity_matches(&entry, Some("Font"), Some("Arial"), None));
|
||||
assert!(!identity_matches(&entry, Some("Size"), Some("Arial"), None));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bounded_window_fallback_requires_window_source_window_id_and_bounds() {
|
||||
let mut entry = entry();
|
||||
entry.source_window_id = Some("platform-window-1".into());
|
||||
entry.bounds_hash = Some(42);
|
||||
|
||||
assert!(bounded_window_fallback_allowed(&entry));
|
||||
entry.source_window_title = Some("Stale Title".into());
|
||||
assert!(!bounded_window_fallback_allowed(&entry));
|
||||
entry.source_window_title = None;
|
||||
entry.bounds_hash = None;
|
||||
assert!(!bounded_window_fallback_allowed(&entry));
|
||||
entry.bounds_hash = Some(42);
|
||||
entry.source_window_id = None;
|
||||
assert!(!bounded_window_fallback_allowed(&entry));
|
||||
entry.source_window_id = Some("platform-window-1".into());
|
||||
entry.source_surface = SnapshotSurface::Menu;
|
||||
assert!(!bounded_window_fallback_allowed(&entry));
|
||||
}
|
||||
|
|
@ -1,58 +1,3 @@
|
|||
use agent_desktop_core::{refs::RefEntry, roles::is_mutable_value_role};
|
||||
|
||||
pub(super) fn has_meaningful_identity(entry: &RefEntry) -> bool {
|
||||
meaningful_text(entry.name.as_deref()).is_some()
|
||||
|| stable_value(entry.role.as_str(), entry.value.as_deref()).is_some()
|
||||
|| meaningful_text(entry.description.as_deref()).is_some()
|
||||
}
|
||||
|
||||
pub(super) fn identity_matches(
|
||||
entry: &RefEntry,
|
||||
actual_name: Option<&str>,
|
||||
actual_value: Option<&str>,
|
||||
actual_description: Option<&str>,
|
||||
) -> bool {
|
||||
let expected_name = meaningful_text(entry.name.as_deref());
|
||||
let expected_value = stable_value(entry.role.as_str(), entry.value.as_deref());
|
||||
let expected_description = meaningful_text(entry.description.as_deref());
|
||||
let actual_name = meaningful_text(actual_name);
|
||||
let actual_value = stable_value(entry.role.as_str(), actual_value);
|
||||
let actual_description = meaningful_text(actual_description);
|
||||
|
||||
if let Some(expected) = expected_name {
|
||||
return match_primary_identity(expected, actual_name, actual_value);
|
||||
}
|
||||
if let Some(expected) = expected_value {
|
||||
return match_primary_identity(expected, actual_value, actual_name);
|
||||
}
|
||||
if let Some(expected) = expected_description {
|
||||
return match_primary_identity(expected, actual_description, actual_name);
|
||||
}
|
||||
|
||||
actual_name.is_none() && actual_value.is_none() && actual_description.is_none()
|
||||
}
|
||||
|
||||
fn match_primary_identity(
|
||||
expected: &str,
|
||||
actual_primary: Option<&str>,
|
||||
actual_fallback: Option<&str>,
|
||||
) -> bool {
|
||||
match actual_primary {
|
||||
Some(actual) => actual == expected,
|
||||
None => actual_fallback == Some(expected),
|
||||
}
|
||||
}
|
||||
|
||||
fn meaningful_text(value: Option<&str>) -> Option<&str> {
|
||||
value.filter(|text| !text.is_empty())
|
||||
}
|
||||
|
||||
fn stable_value<'a>(role: &str, value: Option<&'a str>) -> Option<&'a str> {
|
||||
(!is_mutable_value_role(role))
|
||||
.then(|| meaningful_text(value))
|
||||
.flatten()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "resolve_identity_tests.rs"]
|
||||
mod tests;
|
||||
pub(super) use agent_desktop_core::ref_identity::{
|
||||
bounded_window_fallback_allowed, has_meaningful_identity, identity_matches,
|
||||
};
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ use super::attributes::{
|
|||
use super::element::element_for_pid;
|
||||
use super::element_dedupe::ElementDedupe;
|
||||
use super::resolve_deadline::{ensure_before_deadline, remaining_before_deadline};
|
||||
use super::resolve_identity::bounded_window_fallback_allowed;
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
pub(super) struct CandidateRoots {
|
||||
|
|
@ -44,7 +45,7 @@ pub(super) fn candidate_roots(
|
|||
let mut roots = Vec::new();
|
||||
let mut dedupe = ElementDedupe;
|
||||
let windows = copy_ax_array(&root, "AXWindows").unwrap_or_default();
|
||||
if let Some(window) = exact_source_window_from_windows(&windows, entry, deadline) {
|
||||
if let Some(window) = exact_source_window_from_windows(&windows, entry, deadline)? {
|
||||
dedupe.push(&mut roots, window);
|
||||
}
|
||||
prepare_for_read(&root, deadline)?;
|
||||
|
|
@ -86,16 +87,24 @@ fn source_window_scoped_roots(
|
|||
scope_verified: false,
|
||||
});
|
||||
};
|
||||
if let Some(window) = window_by_number(&windows, source_window_number(entry), deadline) {
|
||||
if let Some(window) = window_by_number(&windows, source_window_number(entry), deadline)? {
|
||||
return Ok(CandidateRoots {
|
||||
roots: vec![window],
|
||||
scope_verified: true,
|
||||
});
|
||||
}
|
||||
if single_window_fallback_allowed(entry) {
|
||||
if let Some(window) = fallback_source_window_root(&windows, entry, deadline) {
|
||||
if let Some(window) = window_by_title(&windows, entry.source_window_title.as_deref(), deadline)?
|
||||
{
|
||||
return Ok(CandidateRoots {
|
||||
roots: vec![window],
|
||||
scope_verified: false,
|
||||
});
|
||||
}
|
||||
if bounded_window_fallback_allowed(entry) {
|
||||
let roots = fallback_replacement_window_roots(&windows, deadline)?;
|
||||
if !roots.is_empty() {
|
||||
return Ok(CandidateRoots {
|
||||
roots: vec![window],
|
||||
roots,
|
||||
scope_verified: false,
|
||||
});
|
||||
}
|
||||
|
|
@ -135,11 +144,7 @@ fn exact_source_window_number_root(
|
|||
let Some(windows) = windows_for_pid(entry.pid, deadline)? else {
|
||||
return Ok(None);
|
||||
};
|
||||
Ok(window_by_number(
|
||||
&windows,
|
||||
source_window_number(entry),
|
||||
deadline,
|
||||
))
|
||||
window_by_number(&windows, source_window_number(entry), deadline)
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
|
|
@ -150,7 +155,7 @@ fn exact_source_window_root(
|
|||
let Some(windows) = windows_for_pid(entry.pid, deadline)? else {
|
||||
return Ok(None);
|
||||
};
|
||||
Ok(exact_source_window_from_windows(&windows, entry, deadline))
|
||||
exact_source_window_from_windows(&windows, entry, deadline)
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
|
|
@ -158,9 +163,9 @@ fn exact_source_window_from_windows(
|
|||
windows: &[AXElement],
|
||||
entry: &RefEntry,
|
||||
deadline: Instant,
|
||||
) -> Option<AXElement> {
|
||||
if let Some(window) = window_by_number(windows, source_window_number(entry), deadline) {
|
||||
return Some(window);
|
||||
) -> Result<Option<AXElement>, AdapterError> {
|
||||
if let Some(window) = window_by_number(windows, source_window_number(entry), deadline)? {
|
||||
return Ok(Some(window));
|
||||
}
|
||||
window_by_title(windows, entry.source_window_title.as_deref(), deadline)
|
||||
}
|
||||
|
|
@ -177,15 +182,17 @@ fn window_by_number(
|
|||
windows: &[AXElement],
|
||||
source_window_number: Option<i64>,
|
||||
deadline: Instant,
|
||||
) -> Option<AXElement> {
|
||||
let source_window_number = source_window_number?;
|
||||
windows
|
||||
.iter()
|
||||
.find(|win| {
|
||||
prepare_for_read(win, deadline).is_ok()
|
||||
&& copy_i64_attr(win, "AXWindowNumber") == Some(source_window_number)
|
||||
})
|
||||
.cloned()
|
||||
) -> Result<Option<AXElement>, AdapterError> {
|
||||
let Some(source_window_number) = source_window_number else {
|
||||
return Ok(None);
|
||||
};
|
||||
for win in windows {
|
||||
prepare_for_read(win, deadline)?;
|
||||
if copy_i64_attr(win, "AXWindowNumber") == Some(source_window_number) {
|
||||
return Ok(Some(win.clone()));
|
||||
}
|
||||
}
|
||||
Ok(None)
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
|
|
@ -193,34 +200,35 @@ fn window_by_title(
|
|||
windows: &[AXElement],
|
||||
source_window_title: Option<&str>,
|
||||
deadline: Instant,
|
||||
) -> Option<AXElement> {
|
||||
let source_window_title = source_window_title?;
|
||||
let index = unique_fallible_matching_index(windows, |win| {
|
||||
) -> Result<Option<AXElement>, AdapterError> {
|
||||
let Some(source_window_title) = source_window_title else {
|
||||
return Ok(None);
|
||||
};
|
||||
let mut found = None;
|
||||
for win in windows {
|
||||
prepare_for_read(win, deadline)?;
|
||||
Ok::<bool, AdapterError>(
|
||||
copy_string_attr(win, "AXTitle").as_deref() == Some(source_window_title),
|
||||
)
|
||||
})?;
|
||||
windows.get(index).cloned()
|
||||
if copy_string_attr(win, "AXTitle").as_deref() == Some(source_window_title) {
|
||||
if found.is_some() {
|
||||
return Ok(None);
|
||||
}
|
||||
found = Some(win.clone());
|
||||
}
|
||||
}
|
||||
Ok(found)
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn fallback_source_window_root(
|
||||
pub(super) fn fallback_replacement_window_roots(
|
||||
windows: &[AXElement],
|
||||
entry: &RefEntry,
|
||||
deadline: Instant,
|
||||
) -> Option<AXElement> {
|
||||
if let Some(window) = window_by_title(windows, entry.source_window_title.as_deref(), deadline) {
|
||||
return Some(window);
|
||||
}
|
||||
if !sole_source_window_fallback_allowed(entry) {
|
||||
return None;
|
||||
}
|
||||
let index = unique_fallible_matching_index(windows, |win| {
|
||||
) -> Result<Vec<AXElement>, AdapterError> {
|
||||
let mut roots = Vec::new();
|
||||
let mut dedupe = ElementDedupe;
|
||||
for win in windows {
|
||||
prepare_for_read(win, deadline)?;
|
||||
Ok::<bool, AdapterError>(copy_string_attr(win, "AXRole").as_deref() == Some("AXWindow"))
|
||||
})?;
|
||||
windows.get(index).cloned()
|
||||
dedupe.push(&mut roots, win.clone());
|
||||
}
|
||||
Ok(roots)
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
|
|
@ -228,17 +236,6 @@ pub(super) fn source_window_scope_required(entry: &RefEntry) -> bool {
|
|||
matches!(entry.source_surface, SnapshotSurface::Window) && source_window_number(entry).is_some()
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
pub(super) fn single_window_fallback_allowed(entry: &RefEntry) -> bool {
|
||||
source_window_scope_required(entry) && entry.bounds_hash.is_some()
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
pub(super) fn sole_source_window_fallback_allowed(entry: &RefEntry) -> bool {
|
||||
single_window_fallback_allowed(entry) && entry.source_window_title.is_none()
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
pub(super) fn source_window_number(entry: &RefEntry) -> Option<i64> {
|
||||
entry
|
||||
.source_window_id
|
||||
|
|
@ -253,20 +250,3 @@ fn prepare_for_read(element: &AXElement, deadline: Instant) -> Result<(), Adapte
|
|||
set_messaging_timeout(element, remaining_before_deadline(deadline)?);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
pub(super) fn unique_fallible_matching_index<T, E>(
|
||||
items: &[T],
|
||||
mut matches: impl FnMut(&T) -> Result<bool, E>,
|
||||
) -> Option<usize> {
|
||||
let mut first = None;
|
||||
for (index, item) in items.iter().enumerate() {
|
||||
match matches(item) {
|
||||
Ok(true) if first.is_none() => first = Some(index),
|
||||
Ok(true) => return None,
|
||||
Ok(false) => {}
|
||||
Err(_) => return None,
|
||||
}
|
||||
}
|
||||
first
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,10 +1,8 @@
|
|||
use super::*;
|
||||
use crate::tree::AXElement;
|
||||
use crate::tree::resolve_classify::classify_candidates;
|
||||
use crate::tree::resolve_roots::{
|
||||
single_window_fallback_allowed, sole_source_window_fallback_allowed, source_window_number,
|
||||
unique_fallible_matching_index,
|
||||
};
|
||||
use crate::tree::resolve_identity::bounded_window_fallback_allowed;
|
||||
use crate::tree::resolve_roots::{fallback_replacement_window_roots, source_window_number};
|
||||
use crate::tree::resolve_search::should_stop_collecting;
|
||||
use agent_desktop_core::adapter::SnapshotSurface;
|
||||
|
||||
|
|
@ -250,14 +248,20 @@ fn non_window_identity_candidate_without_bounds_fails_closed() {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn single_window_fallback_requires_bounds_hash_not_title() {
|
||||
assert!(single_window_fallback_allowed(&entry(
|
||||
fn bounded_window_fallback_requires_untitled_window_ref_with_bounds_hash() {
|
||||
assert!(bounded_window_fallback_allowed(&entry(
|
||||
Some(42),
|
||||
Some("w-10"),
|
||||
None,
|
||||
None
|
||||
)));
|
||||
assert!(!single_window_fallback_allowed(&entry(
|
||||
assert!(!bounded_window_fallback_allowed(&entry(
|
||||
Some(42),
|
||||
Some("w-10"),
|
||||
Some("Stale Title"),
|
||||
None
|
||||
)));
|
||||
assert!(!bounded_window_fallback_allowed(&entry(
|
||||
None,
|
||||
Some("w-10"),
|
||||
Some("Documents"),
|
||||
|
|
@ -265,50 +269,7 @@ fn single_window_fallback_requires_bounds_hash_not_title() {
|
|||
)));
|
||||
let mut menu_entry = entry(Some(42), Some("w-10"), Some("Documents"), None);
|
||||
menu_entry.source_surface = SnapshotSurface::Menu;
|
||||
assert!(!single_window_fallback_allowed(&menu_entry));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sole_window_fallback_requires_missing_title() {
|
||||
assert!(sole_source_window_fallback_allowed(&entry(
|
||||
Some(42),
|
||||
Some("w-10"),
|
||||
None,
|
||||
None
|
||||
)));
|
||||
assert!(!sole_source_window_fallback_allowed(&entry(
|
||||
Some(42),
|
||||
Some("w-10"),
|
||||
Some("Documents"),
|
||||
None
|
||||
)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unique_fallible_matching_index_fails_closed_on_scan_error() {
|
||||
let values = [1, 2, 3];
|
||||
|
||||
assert_eq!(
|
||||
unique_fallible_matching_index(&values, |value| Ok::<bool, ()>(*value == 2)),
|
||||
Some(1)
|
||||
);
|
||||
assert_eq!(
|
||||
unique_fallible_matching_index(&values, |value| Ok::<bool, ()>(*value > 1)),
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
unique_fallible_matching_index(&values, |value| Ok::<bool, ()>(*value == 4)),
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
unique_fallible_matching_index(&values, |value| {
|
||||
if *value == 3 {
|
||||
return Err(());
|
||||
}
|
||||
Ok(*value == 2)
|
||||
}),
|
||||
None
|
||||
);
|
||||
assert!(!bounded_window_fallback_allowed(&menu_entry));
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
@ -320,6 +281,29 @@ fn bounds_hash_keeps_collecting_to_disambiguate_identity_matches() {
|
|||
assert!(should_stop_collecting(2, &entry(None, None, None, None)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bounded_window_fallback_propagates_expired_deadline() {
|
||||
let err = match fallback_replacement_window_roots(
|
||||
&[AXElement(std::ptr::null_mut())],
|
||||
std::time::Instant::now() - std::time::Duration::from_millis(1),
|
||||
) {
|
||||
Ok(_) => panic!("expected timeout"),
|
||||
Err(err) => err,
|
||||
};
|
||||
|
||||
assert_eq!(err.code, ErrorCode::Timeout);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bounded_window_fallback_must_not_stop_after_first_match() {
|
||||
let mut bounded_entry = entry(Some(42), Some("w-10"), Some("Documents"), None);
|
||||
bounded_entry.role = "textfield".into();
|
||||
bounded_entry.name = Some("00:01".into());
|
||||
bounded_entry.value = Some("00:01".into());
|
||||
|
||||
assert!(!should_stop_collecting(2, &bounded_entry));
|
||||
}
|
||||
|
||||
fn description_entry() -> RefEntry {
|
||||
let mut entry = entry(None, Some("w-10"), Some("Freeform"), None);
|
||||
entry.role = "button".into();
|
||||
|
|
|
|||
|
|
@ -59,6 +59,15 @@ the persisted ref identity still needs bounds evidence. The safe pattern is:
|
|||
Do not let presentation options erase identity evidence. Compact output is a
|
||||
serialization concern, not a weaker ref contract.
|
||||
|
||||
### Treat Mutable Values As Volatile Identity
|
||||
|
||||
Strict ref resolution should separate stable labels from mutable control values.
|
||||
Text field content, selected combobox values, slider values, and incrementor
|
||||
values can change between snapshot and action without changing the target
|
||||
element. Core should own that role-conditional identity policy, and platform
|
||||
adapters should only supply native candidates, live attributes, and primitive
|
||||
actions.
|
||||
|
||||
### Centralize Strict Ref Actions
|
||||
|
||||
Ref actions must pass through the same ladder:
|
||||
|
|
|
|||
|
|
@ -118,7 +118,7 @@ Exit codes: `0` success, `1` structured error, `2` argument error.
|
|||
| `APP_NOT_FOUND` | App not running | Launch it first |
|
||||
| `ACTION_FAILED` | AX action rejected | Try an explicit alternative command |
|
||||
| `ACTION_NOT_SUPPORTED` | Element can't do this | Use different command |
|
||||
| `STALE_REF` | Ref from old snapshot | Use the `snapshot_id` returned with this ref, or re-run `snapshot` / `snapshot --skeleton` to get fresh refs |
|
||||
| `STALE_REF` | Ref could not be re-identified in the live UI | Use the `snapshot_id` returned with this ref; if the UI changed or the target disappeared, re-run `snapshot` / `snapshot --skeleton` to get fresh refs |
|
||||
| `AMBIGUOUS_TARGET` | Multiple elements matched the old ref identity | Re-run snapshot and choose a more specific ref |
|
||||
| `SNAPSHOT_NOT_FOUND` | Snapshot ID is missing or expired | Run `snapshot` again and use the returned ID |
|
||||
| `POLICY_DENIED` | A physical/headed path was blocked | Use an explicit mouse/focus/keyboard command if physical interaction is intended |
|
||||
|
|
|
|||
Loading…
Reference in a new issue