diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 6018d9ca4..96efd5c73 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -57,7 +57,7 @@ body: render: shell placeholder: | $ agent-desktop snapshot --app Finder - {"version":"2.5","ok":false,"command":"snapshot","error":{"code":"APP_NOT_FOUND",...}} + {"version":"2.6","ok":false,"command":"snapshot","error":{"code":"APP_NOT_FOUND",...}} validations: required: true diff --git a/AGENTS.md b/AGENTS.md index 7b2df100a..d10a22f5b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -190,12 +190,12 @@ remain the transitive check. Every command prints one envelope. `docs/json-output.md` holds the full contract. ```json -{ "version": "2.5", "ok": true, "command": "snapshot", +{ "version": "2.6", "ok": true, "command": "snapshot", "data": { "app": "Finder", "snapshot_id": "s8f3k2p9", "ref_count": 14, "complete": true, "tree": { } } } ``` ```json -{ "version": "2.5", "ok": false, "command": "click", +{ "version": "2.6", "ok": false, "command": "click", "error": { "code": "STALE_REF", "message": "...", "suggestion": "...", "recovery": { "strategy": "refresh_snapshot_then_retry_original", "retryable": true, "requires_fresh_snapshot": true }, @@ -221,9 +221,11 @@ Every command prints one envelope. `docs/json-output.md` holds the full contract - An element gets a ref when its role is interactive (`INTERACTIVE_ROLES` in `crates/core/src/tree/roles.rs`) or it advertises a primary action. `SetFocus`, `RightClick` and `ScrollTo` alone do not qualify, because web runtimes advertise them on nearly every node. -- Actions re-identify the element strictly from `RefEntry` evidence: pid, role, path and source surface, stable text - identity and a bounds hash. Control values are volatile and never identity. A mismatch returns `STALE_REF`; several - live candidates return `AMBIGUOUS_TARGET`. +- Actions re-identify the element one of two ways. A genuine element token resolves through the adapter, which owns + re-identification on that platform (macOS). Every other ref resolves from its capture proof: role plus every + recorded stable field, that is native id, stable name and description, counted over one complete observation of the + window it was captured from. A ref that was not unique at capture is refused before the application is read. Zero + live matches is `STALE_REF`, two or more is `AMBIGUOUS_TARGET`, and control values and bounds are never identity. - All state lives under one root, `~/.agent-desktop` by default. `AGENT_DESKTOP_HOME` relocates it (`crates/core/src/storage/state_root.rs`). Refmaps live at `snapshots//refmap.json`, capped at 1 MB each; the store keeps the newest 128 snapshots per namespace and prunes to 96. Command code goes through `RefStore`, which diff --git a/CHANGELOG.md b/CHANGELOG.md index 7067eb216..472281230 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,11 +2,16 @@ ## Unreleased +### ⚠ BREAKING CHANGES + +* The C ABI major version is now 7. `ad_resolve_element_exact` and `ad_execute_ref_action_exact_with_policy` always refuse with `ACTION_NOT_SUPPORTED` (`details.kind: exact_entry_unsupported`): a C caller cannot supply the capture proof a ref now resolves by, so act through a snapshot ref instead. +* The response envelope moves to 2.6. On Windows and Linux, a ref to one of several identical controls, a ref from a window the snapshot could not read whole, and a ref from an earlier snapshot now return `AMBIGUOUS_TARGET` with `details.kind: identity_not_unique` where some used to act. A snapshot that could not read its window whole to prove its refs now says so with `data.identity_unproven: true`. See the version 2.6 migration in `docs/json-output.md`. + ### Bug Fixes * Remove the macOS command host. Every CLI call runs in its own process, as on Windows and Linux. The npm install and every CLI call stop any command host an earlier version left running. After an install that skips npm scripts, or a manual install, run `agent-desktop version` once. -* Refs from snapshots taken before this release return `STALE_REF`. Take a new snapshot. -* On macOS a ref follows its own element. A ref to a row that was removed returns `STALE_REF` instead of acting on the row that moved into its place, and identical controls can each be used by ref. +* Refs from snapshots taken before this release carry no capture proof, so take a new snapshot. On Windows and Linux they return `AMBIGUOUS_TARGET` with `details.kind: identity_not_unique`; on macOS a ref still resolves while the element its token names is alive. +* A ref does not act on a different element after the UI changes. On macOS a ref follows its own element: a ref to a removed row returns `STALE_REF`, and identical controls can each be used by ref. On Windows and Linux a ref resolves only if it was unique when captured and exactly one element matches it now; a ref to one of several identical controls returns `AMBIGUOUS_TARGET`. Identity is the element's role and stable fields, not its instance: an identical element that replaced the original is acted on, and a toolkit that keeps a removed widget listed (GTK4, Qt6 on Linux) can still receive the action. ## [1.0.2](https://github.com/vercel-labs/agent-desktop/compare/v1.0.1...v1.0.2) (2026-10-30) diff --git a/CONCEPTS.md b/CONCEPTS.md index 5afd37f84..e68f22eed 100644 --- a/CONCEPTS.md +++ b/CONCEPTS.md @@ -110,10 +110,12 @@ The fail-closed process of re-identifying a ref from stored identity evidence be Strict ref resolution rejects missing, stale, and ambiguous matches instead of guessing. It is the boundary between an old observation and a live desktop mutation. -### Graded Resolution -The order a resolver follows to re-identify a ref, on the evidence the ref carries: it walks the stored positional path first, then falls back to a bounded broad search. A candidate is confirmed by identifier first, then by role-conditional stable text; only a ref with no meaningful identity can resolve by unique geometry, as a last resort. Each tier is a locator, never an identity by itself: a path or a bounds match is always verified against the stored identity evidence before it resolves. +### Ref Re-identification +The rule a stored ref resolves by, which either lands it on the element it was captured from or refuses. -The geometry promotion is deliberately narrow: it fires only when the ref has no meaningful identity and stored bounds, and it resolves only on a unique live match. On Windows and Linux the stored bounds must also be a positive-area rectangle: a zero-extent stored hash never promotes, because offscreen and virtualized elements collapse to shared degenerate rectangles that are structurally non-unique (A17-7). macOS has no such check. +On macOS the entry carries the element's own accessibility address in `retained_object`, so a later command turns the ref back into that element without searching. The address is accepted only while the element is alive and its process, role and recorded identifier still match, so a removed element or an app that restarted is a stale ref rather than a look-alike, and identical twins each keep their own ref. Where an app yields no address, macOS follows the rule below. + +Every other ref resolves from a capture proof. A ref is usable only when the capture saw its whole window completely and exactly one observed element matched its full predicate: role plus every recorded stable field, that is native id, stable name and description. Resolution re-counts that predicate over the live window and accepts exactly one match. Evidence and presentation are separate: a skeleton, a depth-limited snapshot and a `--root` drill are shallow presentations of one complete window observation, so their refs are counted against that whole window and each presented cut container keeps the count of what it hides. A window that cannot be read completely proves nothing, and its refs are refused rather than counted against a narrower tree. Zero matches is a stale ref, two or more is ambiguous, and a field the live read could not answer stays unknown rather than becoming a match, so incomplete evidence never resolves. Values and bounds are never identity. A ref that was not unique at capture is refused before anything is read from the application, because no search can pick one of two identical elements. ## Read Outcomes diff --git a/Cargo.lock b/Cargo.lock index 432d1d680..f44f9213f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -209,9 +209,9 @@ dependencies = [ [[package]] name = "async-executor" -version = "1.14.0" +version = "1.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c96bf972d85afc50bf5ab8fe2d54d1586b4e0b46c97c50a0c9e71e2f7bcd812a" +checksum = "f426a31126c735126297bc5ea7a521263c9faff4f63ca415267ee625651c4f37" dependencies = [ "async-task", "concurrent-queue", diff --git a/README.md b/README.md index 5413ba78c..a338c3cd1 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ agent-desktop snapshot --app Finder -i # observe again to verify ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "snapshot", "data": { @@ -149,7 +149,7 @@ import ctypes lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib") lib.ad_adapter_create.restype = ctypes.c_void_p lib.ad_adapter_destroy.argtypes = [ctypes.c_void_p] -assert lib.ad_init(6) == 0 # AD_RESULT_OK when the header's AD_ABI_VERSION_MAJOR matches the library +assert lib.ad_init(7) == 0 # AD_RESULT_OK when the header's AD_ABI_VERSION_MAJOR matches the library adapter = lib.ad_adapter_create() lib.ad_adapter_destroy(adapter) ``` diff --git a/apps/docs/content/docs/commands/app-window.mdx b/apps/docs/content/docs/commands/app-window.mdx index 19a780aba..3f56d5327 100644 --- a/apps/docs/content/docs/commands/app-window.mdx +++ b/apps/docs/content/docs/commands/app-window.mdx @@ -45,7 +45,7 @@ agent-desktop launch gtk3-demo ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "launch", "data": { @@ -108,7 +108,7 @@ agent-desktop close-app [--force] ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "close-app", "data": { @@ -158,7 +158,7 @@ agent-desktop list-apps [--app NAME] [--timeout-ms MS] ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "list-apps", "data": { @@ -199,7 +199,7 @@ agent-desktop list-windows [--app NAME] [--timeout-ms MS] ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "list-windows", "data": [ @@ -244,7 +244,7 @@ Pass at least one flag; with none the command fails with `INVALID_ARGS`. `data.f ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "focus-window", "data": { @@ -287,7 +287,7 @@ agent-desktop resize-window --app gedit --width 800 --height 600 ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "resize-window", "data": { @@ -360,7 +360,7 @@ agent-desktop minimize [--app NAME] [--window-id ID] ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "minimize", "data": { @@ -400,7 +400,7 @@ agent-desktop restore [--app NAME] [--window-id ID] ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "restore", "data": { diff --git a/apps/docs/content/docs/commands/batch.mdx b/apps/docs/content/docs/commands/batch.mdx index b8f267117..c4cf0ad31 100644 --- a/apps/docs/content/docs/commands/batch.mdx +++ b/apps/docs/content/docs/commands/batch.mdx @@ -72,7 +72,7 @@ Entries run one at a time through the same dispatch as the CLI, so each entry's ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "batch", "data": { @@ -81,8 +81,8 @@ Entries run one at a time through the same dispatch as the CLI, so each entry's "limits": { "max_entries": 64, "max_input_bytes": 1048576, "max_output_bytes": 4194304 }, "not_started_entries": 0, "results": [ - { "command": "version", "data": { "os": "linux", "target": "x86_64", "version": "1.0.1" }, "execution": "completed", "index": 0, "ok": true, "version": "2.5" }, - { "command": "wait", "data": { "waited_ms": 100 }, "execution": "completed", "index": 1, "ok": true, "version": "2.5" } + { "command": "version", "data": { "os": "linux", "target": "x86_64", "version": "1.0.1" }, "execution": "completed", "index": 0, "ok": true, "version": "2.6" }, + { "command": "wait", "data": { "waited_ms": 100 }, "execution": "completed", "index": 1, "ok": true, "version": "2.6" } ], "semantics": { "atomic": false, diff --git a/apps/docs/content/docs/commands/clipboard.mdx b/apps/docs/content/docs/commands/clipboard.mdx index 08a76b434..eb8cd6497 100644 --- a/apps/docs/content/docs/commands/clipboard.mdx +++ b/apps/docs/content/docs/commands/clipboard.mdx @@ -25,13 +25,13 @@ agent-desktop clipboard-get [--format text|auto|image|file-urls] [--out PATH] ```json -{ "version": "2.5", "ok": true, "command": "clipboard-get", "data": { "text": "hello from agent-desktop", "type": "text" } } +{ "version": "2.6", "ok": true, "command": "clipboard-get", "data": { "text": "hello from agent-desktop", "type": "text" } } ``` An image result is `{ "type": "image", "path", "width", "height" }`, and a file list is `{ "type": "file_urls", "file_urls": [...] }`. Image bytes are never inlined. When the clipboard holds nothing in the requested form, the command still succeeds: ```json -{ "version": "2.5", "ok": true, "command": "clipboard-get", "data": { "found": false, "type": "text" } } +{ "version": "2.6", "ok": true, "command": "clipboard-get", "data": { "found": false, "type": "text" } } ``` Check `found` before you read `text`. Errors: `INVALID_ARGS` for an unknown format or an unwritable `--out`, and `PLATFORM_NOT_SUPPORTED` when a platform helper is missing. @@ -61,7 +61,7 @@ One call writes one representation: `--file-url` wins over `--image`, which wins ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "clipboard-set", "data": { @@ -86,7 +86,7 @@ agent-desktop clipboard-clear ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "clipboard-clear", "data": { diff --git a/apps/docs/content/docs/commands/interaction.mdx b/apps/docs/content/docs/commands/interaction.mdx index 1d8ef6bfd..c6402a4ac 100644 --- a/apps/docs/content/docs/commands/interaction.mdx +++ b/apps/docs/content/docs/commands/interaction.mdx @@ -14,7 +14,7 @@ Every successful ref action returns an action report in `data`. This is a `click ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "click", "data": { diff --git a/apps/docs/content/docs/commands/keyboard.mdx b/apps/docs/content/docs/commands/keyboard.mdx index 5cf796c17..e13914037 100644 --- a/apps/docs/content/docs/commands/keyboard.mdx +++ b/apps/docs/content/docs/commands/keyboard.mdx @@ -34,7 +34,7 @@ Keys are physical keyboard input, and `press` does not need `--headed` for an un ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "press", "data": { @@ -68,7 +68,7 @@ A held key needs a process that can guarantee its release, and every agent-deskt ```json { - "version": "2.5", + "version": "2.6", "ok": false, "command": "key-down", "error": { diff --git a/apps/docs/content/docs/commands/mouse.mdx b/apps/docs/content/docs/commands/mouse.mdx index 1df4800c1..50779c720 100644 --- a/apps/docs/content/docs/commands/mouse.mdx +++ b/apps/docs/content/docs/commands/mouse.mdx @@ -40,7 +40,7 @@ agent-desktop --headed hover --xy 640,400 ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "hover", "data": { @@ -114,7 +114,7 @@ agent-desktop --headed mouse-click --xy 500,300 [--button left|right|middle] [-- ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "mouse-click", "data": { @@ -172,7 +172,7 @@ agent-desktop --headed mouse-wheel --x 700 --y 400 --dy -2 ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "mouse-wheel", "data": { diff --git a/apps/docs/content/docs/commands/notifications.mdx b/apps/docs/content/docs/commands/notifications.mdx index 4b3a19117..200cfbc97 100644 --- a/apps/docs/content/docs/commands/notifications.mdx +++ b/apps/docs/content/docs/commands/notifications.mdx @@ -41,7 +41,7 @@ agent-desktop list-notifications [--app NAME] [--text TEXT] [--limit N] ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "list-notifications", "data": { diff --git a/apps/docs/content/docs/commands/observation.mdx b/apps/docs/content/docs/commands/observation.mdx index 358517ab7..f93023839 100644 --- a/apps/docs/content/docs/commands/observation.mdx +++ b/apps/docs/content/docs/commands/observation.mdx @@ -85,7 +85,7 @@ agent-desktop find --app gnome-calculator --role button --name 7 --exact --first ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "find", "data": { @@ -125,7 +125,7 @@ agent-desktop get [--property text|value|title|bounds|role|states] ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "get", "data": { "property": "value", "ref": "@s30sxlstqzdq8x:e19", "value": "entry" } @@ -160,7 +160,7 @@ agent-desktop is @s30sxlstqzdq8x:e45 --property enabled ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "is", "data": { "applicable": true, "property": "enabled", "ref": "@s30sxlstqzdq8x:e45", "result": false } @@ -195,7 +195,7 @@ agent-desktop screenshot --screen 0 screen.png ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "screenshot", "data": { "format": "png", "height": 900, "path": "/tmp/screen.png", "scale_factor": 1.0, "width": 1440 } @@ -216,7 +216,7 @@ agent-desktop list-displays ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "list-displays", "data": [ @@ -267,7 +267,7 @@ agent-desktop --headed open-system-surface --surface KIND ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "open-system-surface", "data": { diff --git a/apps/docs/content/docs/commands/scroll.mdx b/apps/docs/content/docs/commands/scroll.mdx index 5f19f097d..fec963436 100644 --- a/apps/docs/content/docs/commands/scroll.mdx +++ b/apps/docs/content/docs/commands/scroll.mdx @@ -30,7 +30,7 @@ macOS and Windows scroll through native accessibility paths headless. On Linux, ```json { - "version": "2.5", + "version": "2.6", "ok": false, "command": "scroll", "error": { diff --git a/apps/docs/content/docs/commands/system.mdx b/apps/docs/content/docs/commands/system.mdx index 7548d04d5..58101ce1d 100644 --- a/apps/docs/content/docs/commands/system.mdx +++ b/apps/docs/content/docs/commands/system.mdx @@ -51,7 +51,7 @@ agent-desktop permissions [--request] [--guide] ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "permissions", "data": { @@ -69,7 +69,7 @@ Each state is `granted`, `denied` (with a `suggestion`), `not_required`, or `unk Prints the version, CPU architecture, and OS. It takes no flags. ```json -{ "version": "2.5", "ok": true, "command": "version", "data": { "os": "linux", "target": "x86_64", "version": "1.0.1" } } +{ "version": "2.6", "ok": true, "command": "version", "data": { "os": "linux", "target": "x86_64", "version": "1.0.1" } } ``` The envelope's `version` is the JSON contract version; `data.version` is the release. @@ -144,7 +144,7 @@ agent-desktop --session "$AGENT_DESKTOP_SESSION" cursor-overlay disable ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "cursor-overlay", "data": { @@ -188,7 +188,7 @@ agent-desktop --session "$AGENT_DESKTOP_SESSION" trace export [--out PATH] [--li ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "trace", "data": { diff --git a/apps/docs/content/docs/commands/wait.mdx b/apps/docs/content/docs/commands/wait.mdx index 2a3886a38..b1dcd1aa6 100644 --- a/apps/docs/content/docs/commands/wait.mdx +++ b/apps/docs/content/docs/commands/wait.mdx @@ -57,7 +57,7 @@ agent-desktop wait --element @s2txcl95uvubmn:e6 --predicate value --value 7 --ti ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "wait", "data": { @@ -92,7 +92,7 @@ agent-desktop wait --text Calculator --app gnome-calculator --timeout 3000 ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "wait", "data": { "elapsed_ms": 112, "found": true, "ref": null, "role": "window", "snapshot_id": "s2uiek94hz3vms", "text": "Calculator" } @@ -130,7 +130,7 @@ agent-desktop wait --text Calculator --app gnome-calculator --timeout 3000 "execution": "completed", "index": 1, "ok": true, - "version": "2.5" + "version": "2.6" } ``` @@ -144,7 +144,7 @@ A wait that runs out of time returns `TIMEOUT` with `details.kind: "wait_timeout ```json { - "version": "2.5", + "version": "2.6", "ok": false, "command": "wait", "error": { diff --git a/apps/docs/content/docs/errors-and-recovery.mdx b/apps/docs/content/docs/errors-and-recovery.mdx index 2932b93a7..75d7a990a 100644 --- a/apps/docs/content/docs/errors-and-recovery.mdx +++ b/apps/docs/content/docs/errors-and-recovery.mdx @@ -8,7 +8,7 @@ A failed command prints an envelope with `ok: false` and exits `1` (or `2` for a ```json { - "version": "2.5", + "version": "2.6", "ok": false, "command": "click", "error": { @@ -55,7 +55,7 @@ A failed command prints an envelope with `ok: false` and exits `1` (or `2` for a ACTION_FAILEDThe action was rejected or blocked, or its observed result contradicted the request.Read details and disposition.retry. It never carries recovery. ACTION_NOT_SUPPORTEDThe target, or this platform for this input, does not expose the requested action.Use a different command or element. Repeating it fails the same way. STALE_REFThe ref no longer matches a live element.Take a fresh snapshot and use the new ref. See Snapshots and refs. - AMBIGUOUS_TARGETSeveral live elements, windows, or app instances match, and the command refuses to guess.Narrow the target with find, --root, or --window-id, then use the new ref. It never carries recovery. + AMBIGUOUS_TARGETSeveral live elements, windows, or app instances match, or the ref was not unique when it was captured, and the command refuses to guess.Take a fresh snapshot. When the matches are different elements, narrow the target with find, --root, or --window-id; off macOS, identical twins have no ref path. It never carries recovery. WINDOW_NOT_FOUNDThe requested window does not exist, including a --window-id that matches no window or a closed shell surface.Run list-windows --app <name> and use an id it reports. PLATFORM_NOT_SUPPORTEDThis platform's adapter does not implement the method or surface kind.Use another approach. The platform pages list what each OS supports. TIMEOUTA wait expired, the target never became actionable, or a read stayed incomplete until the deadline.Read details.kind. Retry only when disposition.retry is safe. diff --git a/apps/docs/content/docs/ffi/api.mdx b/apps/docs/content/docs/ffi/api.mdx index 4debe2269..e7b8a6990 100644 --- a/apps/docs/content/docs/ffi/api.mdx +++ b/apps/docs/content/docs/ffi/api.mdx @@ -4,7 +4,7 @@ navTitle: API description: Every function, struct, and enum in agent_desktop.h, grouped by area, with exact C signatures. --- -Signatures on this page are extracted from `crates/ffi/include/agent_desktop.h` (ABI major 6). Every `AdResult` function returns `AD_RESULT_OK` (0) or a negative code; read the details with the [last-error accessors](/desktop/ffi/ownership-and-threading#errors). Which call frees which pointer is in [Ownership](/desktop/ffi/ownership-and-threading#ownership). +Signatures on this page are extracted from `crates/ffi/include/agent_desktop.h`. Every `AdResult` function returns `AD_RESULT_OK` (0) or a negative code; read the details with the [last-error accessors](/desktop/ffi/ownership-and-threading#errors). Which call frees which pointer is in [Ownership](/desktop/ffi/ownership-and-threading#ownership). ## Enums @@ -119,6 +119,8 @@ AdResult ad_trace_export(const struct AdAdapter *adapter, `ad_execute_by_ref` is the primary action entrypoint. `ad_execute_action` and `ad_execute_action_with_policy` act on a native handle and skip strict re-identification and the actionability checks. +The two exact-entry calls, `ad_execute_ref_action_exact_with_policy` and `ad_resolve_element_exact`, fail closed with `AD_RESULT_ERR_ACTION_NOT_SUPPORTED` on every platform. An `AdExactRefEntry` carries no retained element token and no capture proof, so the element it names cannot be re-identified. Act through a snapshot ref with `ad_execute_by_ref`, and get a native handle with `ad_find_exact`. + ```c typedef struct AdNativeHandle { const void *ptr; @@ -274,14 +276,17 @@ AdResult ad_execute_action_with_policy(const struct AdAdapter *adapter, int32_t policy, struct AdActionResult *out); -// Runs the full ref-action pipeline from an exact ref entry. +// Always fails with ACTION_NOT_SUPPORTED: an exact ref entry carries no +// retained element token and no capture proof, so it cannot be +// re-identified. Act through a snapshot ref with ad_execute_by_ref. AdResult ad_execute_ref_action_exact_with_policy(const struct AdAdapter *adapter, const struct AdExactRefEntry *entry, const struct AdAction *action, int32_t policy, struct AdActionResult *out); -// Strictly resolves an exact ref entry to a native handle. +// Always fails with ACTION_NOT_SUPPORTED for the same reason. To get a native +// handle, use ad_find_exact. AdResult ad_resolve_element_exact(const struct AdAdapter *adapter, const struct AdExactRefEntry *entry, struct AdNativeHandle *out); @@ -342,6 +347,7 @@ typedef struct AdFindQuery { ```c // Finds and strictly resolves one element in an exact window; duplicates need a first, last, or nth selection. +// The handle is published only when the landed element still matches the selected entry's recorded identity. AdResult ad_find_exact(const struct AdAdapter *adapter, const struct AdExactWindowInfo *win, const struct AdFindQuery *query, diff --git a/apps/docs/content/docs/ffi/build-and-link.mdx b/apps/docs/content/docs/ffi/build-and-link.mdx index d838d05c3..fd8a10ae8 100644 --- a/apps/docs/content/docs/ffi/build-and-link.mdx +++ b/apps/docs/content/docs/ffi/build-and-link.mdx @@ -86,7 +86,7 @@ cc -std=c11 -I crates/ffi/include main.c \ On Linux against GNOME Calculator it printed this envelope, shown here with `tree` elided: ```json -{"version":"2.5","ok":true,"command":"snapshot","data":{"app":"gnome-calculator","complete":true,"ref_count":2,"snapshot_id":"s18ekeexjf5trb","tree":{"...":"..."}}} +{"version":"2.6","ok":true,"command":"snapshot","data":{"app":"gnome-calculator","complete":true,"ref_count":2,"snapshot_id":"s18ekeexjf5trb","tree":{"...":"..."}}} ``` The macOS install name is `@rpath/libagent_desktop_ffi.dylib`, so a program that links the library needs an rpath to its directory; `dlopen` and `ctypes` callers load by path and need nothing. On Windows, an MSVC program links `agent_desktop_ffi.dll.lib` and loads the DLL at run time. On Linux the program must run inside the user's graphical session to see apps; see [Linux](/desktop/platforms/linux). diff --git a/apps/docs/content/docs/ffi/index.mdx b/apps/docs/content/docs/ffi/index.mdx index 9e92ec5c8..a75cdb5bf 100644 --- a/apps/docs/content/docs/ffi/index.mdx +++ b/apps/docs/content/docs/ffi/index.mdx @@ -35,7 +35,7 @@ The library and the CLI run the same core and the same platform adapter; the `ag ## ABI version -`AD_ABI_VERSION_MAJOR` is 6 in this header. The major increments when a symbol is removed or a struct layout changes; new functions and new result codes are additive. Call `ad_init(AD_ABI_VERSION_MAJOR)` once after loading the library: a mismatch returns `AD_RESULT_ERR_INVALID_ARGS`, and you should call nothing else. +`AD_ABI_VERSION_MAJOR` is 7 in this header. The major increments when a symbol is removed or changes incompatibly: a struct layout change, or a call that now fails where it used to succeed. ABI 7 makes `ad_execute_ref_action_exact_with_policy` and `ad_resolve_element_exact` fail closed with `AD_RESULT_ERR_ACTION_NOT_SUPPORTED`. New functions and new result codes are additive. Call `ad_init(AD_ABI_VERSION_MAJOR)` once after loading the library: a mismatch returns `AD_RESULT_ERR_INVALID_ARGS`, and you should call nothing else. ## Next steps diff --git a/apps/docs/content/docs/ffi/ownership-and-threading.mdx b/apps/docs/content/docs/ffi/ownership-and-threading.mdx index 4e91c9b9e..dc8e07789 100644 --- a/apps/docs/content/docs/ffi/ownership-and-threading.mdx +++ b/apps/docs/content/docs/ffi/ownership-and-threading.mdx @@ -24,8 +24,8 @@ Every pointer the library hands you was allocated by Rust, so C's `free()` canno ad_dismiss_all_notificationsad_dismiss_all_notifications_free(dismissed, failed) ad_launch_app_exact (the window struct's strings)ad_release_exact_window_fields(&out) ad_get_tree_exactad_free_tree(&out) - ad_resolve_element_exact, ad_find_exact (the native handle)ad_free_handle(adapter, &handle) - ad_execute_action, ad_execute_action_with_policy, ad_execute_ref_action_exact_with_policy, ad_notification_actionad_free_action_result(&out) + ad_find_exact (the native handle)ad_free_handle(adapter, &handle) + ad_execute_action, ad_execute_action_with_policy, ad_notification_actionad_free_action_result(&out) ad_screenshot, ad_screenshot_window_exactad_image_buffer_free(buf) @@ -35,6 +35,7 @@ Every pointer the library hands you was allocated by Rust, so C's `free()` canno - `ad_free_handle` zeroes the handle, so a second call on the same struct is a no-op. Free handles before you destroy the adapter that produced them. - Strings inside a struct belong to the struct's owner. Do not free them one by one. - Every fallible function zeroes its out-parameter before validating anything, so calling the paired free function after a failure is safe. +- `ad_resolve_element_exact` and `ad_execute_ref_action_exact_with_policy` allocate nothing. They fail with `AD_RESULT_ERR_ACTION_NOT_SUPPORTED` and leave their out-parameter null or zeroed. ## Threading diff --git a/apps/docs/content/docs/guides/chromium-and-electron.mdx b/apps/docs/content/docs/guides/chromium-and-electron.mdx index c6addb466..141ab1660 100644 --- a/apps/docs/content/docs/guides/chromium-and-electron.mdx +++ b/apps/docs/content/docs/guides/chromium-and-electron.mdx @@ -30,7 +30,7 @@ This is Google Chrome on Linux, with the suggestion and the window object trimme ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "launch", "data": { diff --git a/apps/docs/content/docs/guides/headed-mode.mdx b/apps/docs/content/docs/guides/headed-mode.mdx index 017a22c05..304cdf7b2 100644 --- a/apps/docs/content/docs/guides/headed-mode.mdx +++ b/apps/docs/content/docs/guides/headed-mode.mdx @@ -57,7 +57,7 @@ Depending on the OS: ```json { - "version": "2.5", + "version": "2.6", "ok": false, "command": "hover", "error": { diff --git a/apps/docs/content/docs/installation.mdx b/apps/docs/content/docs/installation.mdx index 634451e3c..3babf6a37 100644 --- a/apps/docs/content/docs/installation.mdx +++ b/apps/docs/content/docs/installation.mdx @@ -84,7 +84,7 @@ agent-desktop status ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "version", "data": { "os": "linux", "target": "x86_64", "version": "1.0.1" } @@ -95,7 +95,7 @@ agent-desktop status ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "status", "data": { diff --git a/apps/docs/content/docs/json-output.mdx b/apps/docs/content/docs/json-output.mdx index 053d63e60..3f7cc7b04 100644 --- a/apps/docs/content/docs/json-output.mdx +++ b/apps/docs/content/docs/json-output.mdx @@ -1,16 +1,16 @@ --- title: JSON output navTitle: JSON output -description: The response envelope every agent-desktop command prints, its omission rules, the OperationReport shape, find results, exit codes, and the contract versions from 2.1 to 2.5. +description: The response envelope every agent-desktop command prints, its omission rules, the OperationReport shape, find results, exit codes, and the contract versions from 2.1 to 2.6. --- -Every command prints exactly one JSON envelope on a single line of standard output. Only `--help`, `--version`, and a bare `agent-desktop` print text instead. The contract version is `2.5`. Log output goes to standard error, so a parser can read standard output as one JSON value. +Every command prints exactly one JSON envelope on a single line of standard output. Only `--help`, `--version`, and a bare `agent-desktop` print text instead. The contract version is `2.6`. Log output goes to standard error, so a parser can read standard output as one JSON value. ## Envelope ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "click", "data": { @@ -25,7 +25,7 @@ Every command prints exactly one JSON envelope on a single line of standard outp FieldTypeMeaning - versionstringContract version. Currently "2.5". + versionstringContract version. Currently "2.6". okbooleantrue when the command returned a result, false when it returned an error. commandstringThe command name, such as click or session. dataobjectThe result. Present only when ok is true. @@ -37,7 +37,7 @@ An error envelope has `error` and no `data`: ```json { - "version": "2.5", + "version": "2.6", "ok": false, "command": "click", "error": { @@ -64,7 +64,7 @@ An error envelope has `error` and no `data`: ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "snapshot", "data": { @@ -145,7 +145,7 @@ A failed `--wait-for` after a delivered operation keeps the operation's own deli ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "find", "data": { @@ -201,6 +201,7 @@ Check `version` before you parse. A consumer written for 2.0 or later can treat 2.3launch returns app, pid, process_instance, and an optional window instead of a bare window object. An application that shows no window is ok: true with window omitted, not WINDOW_NOT_FOUND. 2.4Stateful ref actions (set-value, type, clear, check, uncheck, toggle, expand, collapse) read the state back after one delivery attempt and reject an observed contradiction. A readback failure keeps the adapter's code and adds details.kind: "post_action_verification". Missing evidence stays a successful delivered_unverified with details.verification_scope: "unavailable". 2.5Breaking for the thirteen OperationReport commands. The success flags clicked, moved, scrolled, hovered, dragged, resized, minimized, maximized, restored, closed, cleared, requested (close-app), and the inner ok of clipboard-set are removed. data is the adapter's report with disposition, steps, and target. + 2.6On Windows and Linux, and on macOS for an element with no address, a ref resolves only when one complete read of its window proved it unique at capture and exactly one live element matches it now. A ref to one of two identical controls, a ref from a window that could not be read whole, and every ref in a refmap written before 2.6 fail AMBIGUOUS_TARGET with details.kind: "identity_not_unique". A twin that appears after capture fails with saved_ref_ambiguous. Both carry details.retryable: false. diff --git a/apps/docs/content/docs/platforms/macos.mdx b/apps/docs/content/docs/platforms/macos.mdx index 05c9a3a0f..af9916b06 100644 --- a/apps/docs/content/docs/platforms/macos.mdx +++ b/apps/docs/content/docs/platforms/macos.mdx @@ -24,7 +24,7 @@ The grant belongs to the app that launched agent-desktop, not to the shell. A di ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "permissions", "data": { diff --git a/apps/docs/content/docs/quick-start.mdx b/apps/docs/content/docs/quick-start.mdx index 95c657786..831d4bd1c 100644 --- a/apps/docs/content/docs/quick-start.mdx +++ b/apps/docs/content/docs/quick-start.mdx @@ -27,7 +27,7 @@ agent-desktop snapshot --app gnome-calculator --skeleton -i --compact ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "snapshot", "data": { @@ -74,7 +74,7 @@ agent-desktop click @s2txcl95uvubmn:e11 ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "click", "data": { @@ -108,7 +108,7 @@ agent-desktop wait --element @s2txcl95uvubmn:e6 --predicate value --value 7 --ti ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "wait", "data": { diff --git a/apps/docs/content/docs/sessions.mdx b/apps/docs/content/docs/sessions.mdx index f1e838d9c..f5f28476c 100644 --- a/apps/docs/content/docs/sessions.mdx +++ b/apps/docs/content/docs/sessions.mdx @@ -23,7 +23,7 @@ agent-desktop session gc --ended # delete ended sessions nothin ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "session", "data": { @@ -47,7 +47,7 @@ Ending a session seals the manifest and stops its trace. After that, every other ```json { - "version": "2.5", + "version": "2.6", "ok": false, "command": "snapshot", "error": { diff --git a/apps/docs/content/docs/skeleton-and-drill-down.mdx b/apps/docs/content/docs/skeleton-and-drill-down.mdx index b24f36f4e..262c62669 100644 --- a/apps/docs/content/docs/skeleton-and-drill-down.mdx +++ b/apps/docs/content/docs/skeleton-and-drill-down.mdx @@ -4,7 +4,7 @@ navTitle: Skeleton and drill-down description: Read a dense app in two passes. Take a shallow skeleton, drill into one region by ref, re-drill after a change, and scope searches the same way. --- -A dense app, such as an Electron client or an IDE, can expose thousands of accessibility nodes. A full snapshot of it is slow, costs many tokens, and can run out of budget before it finishes. A skeleton snapshot reads only the top levels. You then drill into the one region you need with `snapshot --root`, so the cost follows the region you act on instead of the whole window. +A dense app, such as an Electron client or an IDE, can expose thousands of accessibility nodes. A full snapshot of it is slow, costs many tokens, and can run out of budget before it finishes. A skeleton snapshot reads only the top levels. You then drill into the one region you need with `snapshot --root`, so the refs you read and act on cover that region instead of the whole window. Only macOS makes the drill read less of the window. Off macOS a drill re-identifies its root over a complete read of the window, as [Drill into a region](#drill-into-a-region) explains. ## The loop @@ -17,6 +17,8 @@ A dense app, such as an Electron client or an IDE, can expose thousands of acces 5. Verify with `wait` or `get` on a ref from the drill, or re-read the region with the same `snapshot --root`. If the action result lists a menu, sheet, or alert in `data.surfaces`, read that [surface](/desktop/surfaces) next. 6. If another region matters, drill it. If the layout changed, or the root ref itself is `STALE_REF`, take a new skeleton. +`-i` keeps only interactive nodes, so a row's static text is dropped. To tell which row a repeated button belongs to, drill that row's container with `--root` and leave out `-i`, then match the button to the label beside it. + Two shortcuts skip the loop: a small app needs only `snapshot -i`, and a known target needs only `find`. See [When to skip the skeleton](#when-to-skip-the-skeleton). ## Take a skeleton @@ -27,7 +29,7 @@ agent-desktop snapshot --app gnome-calculator --skeleton -i --compact ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "snapshot", "data": { @@ -90,7 +92,7 @@ agent-desktop snapshot --root @s2txcl95uvubmn:e1 -i --compact ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "snapshot", "data": { @@ -136,6 +138,8 @@ The tree above is trimmed to the display and two of the 31 buttons and fields th 3. Reads the element's subtree to `--max-depth` (default 10). `--app` and `--window-id` are not used, because the ref already names its window, and `--skeleton` has no effect. 4. Allocates refs for the subtree and writes them into the same snapshot, so `snapshot_id` does not change. +Off macOS the drill re-identifies its root over a complete read of the window and counts its new refs against that same read, so a drill's refs resolve by exactly the rules the skeleton's do. + The result's root is the drilled element, and `ref_count` is the whole snapshot's count: 2 skeleton refs plus 31 drill refs. New refs continue the snapshot's counter, which is why this drill starts at `e3`. The ref you drilled stays valid. ## Re-drill after a change @@ -159,7 +163,7 @@ agent-desktop get @s2txcl95uvubmn:e11 # inside the region: repl ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "get", "data": { "property": "role", "ref": "@s2txcl95uvubmn:e2", "value": "group" } @@ -168,7 +172,7 @@ agent-desktop get @s2txcl95uvubmn:e11 # inside the region: repl ```json { - "version": "2.5", + "version": "2.6", "ok": false, "command": "get", "error": { @@ -207,7 +211,7 @@ A drill is all or nothing. It replaces refs inside an existing snapshot, and a p ```json { - "version": "2.5", + "version": "2.6", "ok": false, "command": "snapshot", "error": { @@ -246,7 +250,7 @@ A `find` that returns matches saves its own new snapshot, and its refs are quali ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "find", "data": { @@ -296,7 +300,7 @@ agent-desktop snapshot --app gtk3-widget-factory -i --compact --timeout-ms 50 ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "snapshot", "data": { diff --git a/apps/docs/content/docs/skills.mdx b/apps/docs/content/docs/skills.mdx index 0a1d77ab6..5af7e7a7d 100644 --- a/apps/docs/content/docs/skills.mdx +++ b/apps/docs/content/docs/skills.mdx @@ -51,7 +51,7 @@ agent-desktop skills get desktop | jq -r '.data.content' ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "skills", "data": { diff --git a/apps/docs/content/docs/snapshots-and-refs.mdx b/apps/docs/content/docs/snapshots-and-refs.mdx index f0118aea8..254fcd155 100644 --- a/apps/docs/content/docs/snapshots-and-refs.mdx +++ b/apps/docs/content/docs/snapshots-and-refs.mdx @@ -12,7 +12,7 @@ agent-desktop snapshot --app gnome-calculator -i --compact ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "snapshot", "data": { @@ -57,6 +57,7 @@ The tree above is trimmed to one button. The real tree nests it under several un data.ref_countThe number of refs the snapshot holds. After a --root drill it counts the whole snapshot, not only the drilled region. data.completePresent on every snapshot. false means the observation ran out of budget. See Partial observations. data.truncated, data.nodes_observedPresent only when complete is false. + data.identity_unprovenPresent and true only when the snapshot needed one whole read of the window to prove its refs and that read failed or was cut short. The tree is still the one presented, but on Windows and Linux, and on macOS for an element with no address, its refs fail with identity_not_unique. Retake the snapshot with a larger --timeout-ms. data.treeThe root node. Every node nests its children. @@ -76,7 +77,7 @@ A node always carries `role` and, when they exist, `ref_id`, `name`, `value`, `d `children_count` and `subtree_truncated` mark where the tree was cut; [Skeleton and drill-down](/desktop/skeleton-and-drill-down) explains both. Three flags shrink the tree, and the [commands reference](/desktop/commands/observation#snapshot) lists every flag: -- `-i` (`--interactive-only`) drops leaves that have no ref. It keeps nodes that lead to refs and nodes marked with `children_count`. +- `-i` (`--interactive-only`) drops leaves that have no ref. It keeps nodes that lead to refs and nodes marked with `children_count`. Static text has no ref, so a row's label is dropped: to see which row a repeated button belongs to, drill that row's container with `--root` and leave out `-i`. - `--compact` collapses a single-child wrapper that has no name, value, description, id, state, or action. - `--max-depth` limits depth. The default is 10 and the maximum is 50. @@ -103,7 +104,7 @@ Every command accepts only this qualified form. A bare `@e6` fails with `INVALID ```json { - "version": "2.5", + "version": "2.6", "ok": false, "command": "click", "error": { @@ -138,11 +139,11 @@ Each ref maps to a `RefEntry`: the evidence needed to find the same element agai Processpid, process_instanceReject a ref whose process exited or whose pid was reused. - Identityrole, name, value, description, native_idMatch the live element. A control's value is never identity for roles whose value changes while you use them, such as textfield, checkbox, and slider. - Geometrybounds, bounds_hash, bounds_spaceTell apart candidates that match equally. For an element with no other identity, a unique matching rectangle can identify it. + Identityretained_object, role, name, value, description, native_id, unique_at_captureRe-identify the element. retained_object is the element's own accessibility address, recorded on macOS. unique_at_capture records whether the entry was the only element in the window its snapshot captured with this role and these stable fields. A skeleton, depth-limited or drilled capture counts against that whole window, so a shallow presentation proves the same thing a full one does. A control's value is never identity for roles whose value changes while you use them, such as textfield, checkbox, and slider. + Geometrybounds, bounds_hash, bounds_spaceNot identity. get --property bounds falls back to them when the live read is unavailable; re-identification never compares them. Capabilitiesstates, available_actionsWhat the element offered at snapshot time. Sourcesource_app, source_window_id, source_window_title, source_window_bounds_hash, source_surfaceTie the ref to its window and surface. - Scopepath, root_ref, path_is_absoluteThe element's position as child indexes, and which drill-down created the ref. + Scopepath, root_ref, path_is_absoluteThe element's child-index position at capture, and which drill-down created the ref. @@ -150,24 +151,27 @@ Each ref maps to a `RefEntry`: the evidence needed to find the same element agai A command never reuses a live handle from the snapshot. Every command that takes a ref, including `click`, `get`, `is`, `snapshot --root`, and `find --root`, re-identifies the element from its stored evidence first. - + 1. The command parses `@s26u7pv9qta5g4:e6` into the snapshot id and the key `@e6`. A bare or malformed ref fails with `INVALID_ARGS`. 2. `RefStore` loads `snapshots/s26u7pv9qta5g4/refmap.json` from the selected session's namespace. A missing snapshot fails with `SNAPSHOT_NOT_FOUND`. 3. The ref map looks up `@e6` and returns its `RefEntry`. A missing key fails with `STALE_REF`. -4. The platform adapter resolves the entry strictly against the live UI: it checks the process and the source window, then searches the window's live tree for elements with the stored role and stable identity. -5. Exactly one match: the command acts on it or reads it. No match: `STALE_REF`. Several matches that nothing separates: `AMBIGUOUS_TARGET`. -6. After `STALE_REF` or `AMBIGUOUS_TARGET`, take a fresh observation and use the new ref. +4. If the entry carries the element's own address, `resolve_element_strict` turns it back into that exact element. If the element is alive and its process, role and recorded id still match, the command acts on it. Otherwise the ref is `STALE_REF` with `details.kind: retained_object_stale`. No search runs, and the ref never lands on a look-alike. +5. If the entry has no address, `unique_at_capture` decides. A ref that was not unique at capture fails with `AMBIGUOUS_TARGET`, `details.kind: identity_not_unique`, before any live read. +6. A ref that was unique is counted again over the live window by role, native id, name and description. The count covers the whole window, which is the scope the capture proved the ref unique in. Exactly one match: the command acts on it. None: `STALE_REF`, `details.kind: saved_ref_absent`. Two or more: `AMBIGUOUS_TARGET`, `details.kind: saved_ref_ambiguous`. A field the live read could not answer leaves the count unproven, which returns `TIMEOUT` with `details.kind: resolution_identity_unknown` instead of a match. +7. After `STALE_REF` or `AMBIGUOUS_TARGET`, take a fresh observation and use the new ref. -The stored position is a hint for where to look, and the identity check decides. Identifier and stable text come first. Geometry identifies an element only when the ref has no identifier and no stable text, and only on a unique rectangle. When several live elements match, the stored bounds can pick one of them; if they cannot, the answer is `AMBIGUOUS_TARGET`. +On macOS a ref carries the element's own address, so identical twins each keep their own ref. When the address could not be taken, macOS follows the rule below. Windows and Linux refs carry no address, so a ref that was not the only element matching its role and stable fields when it was captured is refused. + +The comparison is the identity fields and nothing else: role, native id, name, and description. Control values are volatile and never count. Bounds never count either, so a stored rectangle or a stored path cannot pick between two candidates. A ref that fails these checks is refused rather than guessed at. - - + +
ErrorMeaningWhat to do
STALE_REFThe key is not in the snapshot's ref map, or no live element answers to the stored evidence. The UI changed, the window closed, or a re-drill replaced the ref.Take a fresh snapshot, or re-drill the region, and use the new ref. Nothing was sent to the app.
AMBIGUOUS_TARGETSeveral live elements match the stored evidence, and the stored bounds cannot separate them. The same code reports several windows for one --app.Narrow the target with find --role --name --exact or --root, then use the new ref.
STALE_REFThe key is not in the snapshot's ref map, no live element matches the stored identity (details.kind: saved_ref_absent), or the element the ref names is gone (retained_object_stale). The UI changed, the window closed, or a re-drill replaced the ref.Take a fresh snapshot, or re-drill the region, and use the new ref. Nothing was sent to the app.
AMBIGUOUS_TARGETTwo live elements match the stored identity (saved_ref_ambiguous), the ref was not proven unique when it was captured (identity_not_unique), or several windows matched one --app.For several windows, name the window. A ref that was not proven unique came from identical twins or from a window that could not be read whole within the snapshot timeout; only the second is fixed by a new snapshot, with a larger --timeout-ms. Off macOS, identical twins cannot be picked by ref at all and every snapshot refuses them again: only an explicit coordinate click, agent-desktop --headed mouse-click --xy X,Y, can choose one. It takes the foreground and is unavailable on a Linux Wayland session.
SNAPSHOT_NOT_FOUNDNo saved snapshot has that id: it was pruned, it belongs to another session, or the state root differs.Take a new snapshot under the same --session and state root as the command that uses it.
diff --git a/apps/docs/content/docs/surfaces.mdx b/apps/docs/content/docs/surfaces.mdx index da9de5a77..260275038 100644 --- a/apps/docs/content/docs/surfaces.mdx +++ b/apps/docs/content/docs/surfaces.mdx @@ -88,7 +88,7 @@ To reach one item, `find --surface` is cheaper than a dump: ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "find", "data": { @@ -133,7 +133,7 @@ agent-desktop list-surfaces --app gtk3-widget-factory ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "list-surfaces", "data": { diff --git a/apps/docs/public/desktop/diagrams/ref-resolution/index.html b/apps/docs/public/desktop/diagrams/ref-resolution/index.html index 41647c95b..d78eb6564 100644 --- a/apps/docs/public/desktop/diagrams/ref-resolution/index.html +++ b/apps/docs/public/desktop/diagrams/ref-resolution/index.html @@ -55,66 +55,80 @@ - + How a ref resolves, and when it goes stale - From top to bottom: a command receives a qualified ref. RefStore loads snapshots/snapshot_id/refmap.json, or fails with SNAPSHOT_NOT_FOUND. The ref maps to a RefEntry, or fails with STALE_REF when the key is missing. resolve_element_strict re-identifies the entry in the live tree: exactly one match lets the command act, zero matches is STALE_REF, and two or more is AMBIGUOUS_TARGET. Every failure is fixed by a fresh snapshot and its new ref. + From top to bottom: a command uses a ref such as @s…:e6, and the refmap returns its RefEntry, or STALE_REF when the key is missing. If the entry carries a retained object (on macOS, the element's own address), resolve_element_strict turns it back into that exact element: if it is alive and its pid, role and recorded id still match, the command acts on it, otherwise STALE_REF. No search runs. If the entry has no retained object, unique_at_capture decides: not unique at capture is AMBIGUOUS_TARGET. If it was unique, FullPredicate counts the live elements that match the same predicate: exactly one lets the command act, none is STALE_REF, and two or more is AMBIGUOUS_TARGET. - - - @<snapshot_id>:eN - a qualified ref - - - - RefStore - snapshots/<id>/refmap.json - - gone - - - SNAPSHOT_NOT_FOUND - pruned, or another session - - - - RefEntry - pid, role, path, identity, bounds - - no key - - - STALE_REF - no live element matches - - - - resolve_element_strict - re-identify in the live tree - - 0 matches - - 2+ - - - AMBIGUOUS_TARGET - several elements match - - exactly one match - - - act - on the resolved element - - - - - - fresh snapshot - new snapshot_id, new ref + + + @s…:e6 + a command uses a ref + + + + RefEntry + refmap lookup by key + + no key + + + STALE_REF + key not in the refmap + + + + element_token(entry) + retained object on the entry? + + yes + + + resolve_element_strict + macOS: address back to the same element + + dead or mismatched + + + STALE_REF + dead, or pid, role or id differ + + no retained object + + + unique_at_capture + complete evidence, full predicate + + no + + + AMBIGUOUS_TARGET + not unique at capture, or 2+ live matches + + yes + + + FullPredicate + count live matches in the window + + 2+ matches + + none + + + STALE_REF + no live element matches + + exactly 1 match + + + act + on the resolved element + + alive, fields match diff --git a/crates/core/examples/locator_benchmark/fixture_builder.rs b/crates/core/examples/locator_benchmark/fixture_builder.rs index cf274ed99..d271b2c27 100644 --- a/crates/core/examples/locator_benchmark/fixture_builder.rs +++ b/crates/core/examples/locator_benchmark/fixture_builder.rs @@ -16,6 +16,7 @@ pub(crate) fn live_tree( ObservationSource::Window { window: fixture.window.clone(), surface: agent_desktop_core::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, requirements, ) diff --git a/crates/core/src/actionability/actionability_live_failure_tests.rs b/crates/core/src/actionability/actionability_live_failure_tests.rs index f144e3298..607c4b35f 100644 --- a/crates/core/src/actionability/actionability_live_failure_tests.rs +++ b/crates/core/src/actionability/actionability_live_failure_tests.rs @@ -155,3 +155,66 @@ fn transient_live_read_failure_reports_no_delivery() { assert_eq!(error.disposition, expected); } } + +/// A tokenless ref is held to the whole identity rule capture recorded, not to +/// its identifier alone: a replacement that kept the recorded identifier and +/// changed the recorded text is a different element and must not be dispatched +/// to. +#[test] +fn a_recycled_identifier_with_changed_text_is_stale_before_dispatch() { + struct RecycledIdentifierAdapter; + + impl ObservationOps for RecycledIdentifierAdapter { + fn get_live_element( + &self, + _: &NativeHandle, + _: crate::Deadline, + ) -> Result { + Ok(LiveElement { + identity: crate::LiveIdentity { + name: crate::LocatorField::Known("Delete account".into()), + description: crate::LocatorField::Absent, + identifiers: crate::IdentifierEvidence::typed( + [crate::ElementIdentifier { + kind: crate::IdentifierKind::AxIdentifier, + value: "save-button".into(), + }], + Some(0), + true, + ), + }, + state: ElementState { + role: "button".into(), + states: vec![], + value: None, + enabled: Some(true), + hidden: Some(false), + offscreen: Some(false), + }, + states_complete: true, + bounds: entry().geometry.bounds, + available_actions: vec![capability::CLICK.into()], + }) + } + } + impl ActionOps for RecycledIdentifierAdapter {} + impl InputOps for RecycledIdentifierAdapter {} + impl SystemOps for RecycledIdentifierAdapter {} + + let mut recorded = entry(); + recorded.identity.native_id = Some(crate::ElementIdentifier { + kind: crate::IdentifierKind::AxIdentifier, + value: "save-button".into(), + }); + + let error = check_live( + &recorded, + &NativeHandle::null(), + &RecycledIdentifierAdapter, + &ActionRequest::headless(Action::Click), + ) + .unwrap_err(); + + assert_eq!(error.code, ErrorCode::StaleRef); + assert!(error.message.contains("changed identity")); +} diff --git a/crates/core/src/actionability/actionability_live_tests.rs b/crates/core/src/actionability/actionability_live_tests.rs index 7e0a2676a..2f7089cbf 100644 --- a/crates/core/src/actionability/actionability_live_tests.rs +++ b/crates/core/src/actionability/actionability_live_tests.rs @@ -233,6 +233,7 @@ fn entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -249,6 +250,7 @@ fn entry() -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/actionability/actionability_tests.rs b/crates/core/src/actionability/actionability_tests.rs index 30c727537..085679010 100644 --- a/crates/core/src/actionability/actionability_tests.rs +++ b/crates/core/src/actionability/actionability_tests.rs @@ -24,6 +24,7 @@ fn entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -40,6 +41,7 @@ fn entry() -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/actionability/live.rs b/crates/core/src/actionability/live.rs index 7f3392eab..9edc1089d 100644 --- a/crates/core/src/actionability/live.rs +++ b/crates/core/src/actionability/live.rs @@ -5,11 +5,14 @@ use super::{ stability::StabilityExpectation, }; use crate::{ - AdapterError, DeliverySemantics, ErrorCode, IdentityMatch, LocatorField, + AdapterError, DeliverySemantics, ErrorCode, IdentityMatch, LocatorEvidence, LocatorField, actions::action_request::ActionRequest, adapter::{LiveElement, NativeHandle, PlatformAdapter}, + locator::live_locator::LocatorRefEvidence, refs::RefEntry, - refs::ref_identity::{has_meaningful_identity, identity_match}, + refs::ref_full_predicate::FullPredicate, + refs::ref_identity::{element_token, identity_match}, + refs::ref_live_predicate::LivePredicate, }; /// Groups the read-only coordinates of a live element check: which element, @@ -115,22 +118,7 @@ fn validate_live_identity(entry: &RefEntry, live: &LiveElement) -> Result<(), Ad "Resolved element changed role before dispatch", )); } - if !has_meaningful_identity(entry) { - return Ok(()); - } - let value = live - .state - .value - .clone() - .map(LocatorField::Known) - .unwrap_or(LocatorField::Absent); - match identity_match( - entry, - &live.identity.name, - &value, - &live.identity.description, - &live.identity.identifiers, - ) { + match recorded_identity_match(entry, &live_evidence(live)) { IdentityMatch::Match => Ok(()), IdentityMatch::NoMatch => Err(stale_live_element( "Resolved element changed identity before dispatch", @@ -150,6 +138,49 @@ fn validate_live_identity(entry: &RefEntry, live: &LiveElement) -> Result<(), Ad } } +/// The identity rule the capture recorded. An entry carrying an element token +/// re-identifies itself through that token; every other entry is held to the +/// whole predicate - role, identifier and stable text - that resolution +/// compares, because an identifier alone still matches after the element it +/// named was replaced by one that kept the identifier and changed its text. +fn recorded_identity_match(entry: &RefEntry, live: &LocatorEvidence) -> IdentityMatch { + if element_token(entry).is_some() { + return identity_match( + entry, + &live.name, + &live.value, + &live.description, + &live.identifiers, + ); + } + FullPredicate::of_entry(entry).compare(&LivePredicate::observed(live)) +} + +/// The live read as the evidence a predicate compares: the identity slots it +/// holds, with the geometry and state slots of a snapshot observation left +/// unread because no identity rule reads them. +fn live_evidence(live: &LiveElement) -> LocatorEvidence { + LocatorEvidence { + role: LocatorField::Known(live.state.role.clone()), + name: live.identity.name.clone(), + description: live.identity.description.clone(), + value: live + .state + .value + .clone() + .map(LocatorField::Known) + .unwrap_or(LocatorField::Absent), + identifiers: live.identity.identifiers.clone(), + states: LocatorField::Absent, + ref_evidence: LocatorRefEvidence { + bounds: LocatorField::Absent, + window_bounds: LocatorField::Absent, + available_actions: LocatorField::Known(live.available_actions.clone()), + descriptors: Default::default(), + }, + } +} + fn stale_live_element(message: &'static str) -> AdapterError { AdapterError::new(ErrorCode::StaleRef, message) .with_suggestion( diff --git a/crates/core/src/adapter/hit_test_tests.rs b/crates/core/src/adapter/hit_test_tests.rs index 8c84c2404..e08e3305a 100644 --- a/crates/core/src/adapter/hit_test_tests.rs +++ b/crates/core/src/adapter/hit_test_tests.rs @@ -48,6 +48,7 @@ impl ObservationOps for HitTestAdapter { self.calls.fetch_add(1, Ordering::SeqCst); self.outcome.clone() } + crate::adapter::saved_ref_scope!("button", Some("Save")); } impl ActionOps for HitTestAdapter {} @@ -130,6 +131,7 @@ fn clickable_entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -146,6 +148,7 @@ fn clickable_entry() -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/adapter/mod.rs b/crates/core/src/adapter/mod.rs index 973dd1268..500614da6 100644 --- a/crates/core/src/adapter/mod.rs +++ b/crates/core/src/adapter/mod.rs @@ -6,6 +6,8 @@ pub mod live_element; pub(crate) mod live_identity; pub mod native_handle; pub(crate) mod observation; +#[cfg(test)] +mod saved_ref_test_support; pub(crate) mod system; #[cfg(test)] mod test_support; @@ -14,6 +16,11 @@ pub(crate) use actions::ActionOps; pub(crate) use input::InputOps; pub(crate) use observation::ObservationOps; pub(crate) use observation::optional_live_read; +#[cfg(test)] +pub(crate) use saved_ref_test_support::{ + saved_ref_anchor, saved_ref_element_tree, saved_ref_observation, saved_ref_observation_nodes, + saved_ref_scope, saved_ref_tree, saved_ref_tree_nodes, saved_ref_window_pool, +}; pub(crate) use system::SystemOps; #[cfg(test)] pub(crate) use test_support::{ diff --git a/crates/core/src/adapter/observation.rs b/crates/core/src/adapter/observation.rs index 876fb0373..671282f11 100644 --- a/crates/core/src/adapter/observation.rs +++ b/crates/core/src/adapter/observation.rs @@ -135,6 +135,24 @@ pub trait ObservationOps: Send + Sync { Err(AdapterError::not_supported("resolve_element_strict")) } + /// Whether this adapter records an element token on observed nodes, so a + /// shallow capture can re-identify its refs without a whole-window read. + fn retains_element_tokens(&self) -> bool { + false + } + + /// Checks platform-specific context that the saved ref's identity predicate + /// does not encode, against the token from the same observation that matched + /// it. This must not take a second live read: rooted finds and drills publish + /// refs from that observation. Adapters without extra context accept it. + fn validate_saved_ref_context( + &self, + _entry: &RefEntry, + _observed_token: Option<&str>, + ) -> Result<(), AdapterError> { + Ok(()) + } + fn resolve_locator_anchor( &self, _entry: &RefEntry, @@ -276,14 +294,16 @@ pub trait ObservationOps: Send + Sync { /// Strict receiver proof for a physical pointer move that has no element /// action to carry it. An adapter that can prove who receives the pointer - /// refuses with a `not_delivered` error when the target does not. + /// refuses with a `not_delivered` error when the target does not. `handle` + /// is the element the caller already resolved under the interaction lease, + /// so the adapter never re-identifies the ref by a rule of its own. fn confirm_pointer_receiver( &self, - entry: &RefEntry, + handle: &NativeHandle, point: crate::Point, deadline: Deadline, ) -> Result<(), AdapterError> { - let _ = (entry, point, deadline); + let _ = (handle, point, deadline); Ok(()) } } diff --git a/crates/core/src/adapter/observation_tests.rs b/crates/core/src/adapter/observation_tests.rs index 822cad3ff..6be6e69ad 100644 --- a/crates/core/src/adapter/observation_tests.rs +++ b/crates/core/src/adapter/observation_tests.rs @@ -45,6 +45,7 @@ fn entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -61,6 +62,7 @@ fn entry() -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/adapter/saved_ref_test_support.rs b/crates/core/src/adapter/saved_ref_test_support.rs new file mode 100644 index 000000000..f28660705 --- /dev/null +++ b/crates/core/src/adapter/saved_ref_test_support.rs @@ -0,0 +1,170 @@ +use crate::adapter::test_support::{minimal_accessibility_node, observed_tree}; + +/// The window inventory a saved-ref test searches: one focused window for the +/// lowest process id plus one window per further id test entries use, each +/// named after its process so a recorded window id selects exactly one. +pub(crate) fn saved_ref_window_pool() -> Vec { + [1, 2, 7, 42, 99] + .into_iter() + .map(|pid| { + let mut window = saved_ref_window(&format!("w-{pid}"), pid, "test-instance"); + window.state = crate::WindowState { + is_focused: pid == 1, + visible: Some(true), + ..Default::default() + }; + window + }) + .collect() +} + +/// One window of the pool: the process the test's saved entry belongs to. +pub(crate) fn saved_ref_window(id: &str, pid: u32, instance: &str) -> crate::WindowInfo { + crate::WindowInfo { + id: id.into(), + title: format!("Window {pid}"), + app: "Test".into(), + pid: crate::ProcessId::new(pid), + process_instance: Some(instance.into()), + bounds: None, + state: crate::WindowState::default(), + } +} + +/// A one-node observation carrying the identity a saved-ref test expects its +/// entry to match, so the shared resolver's live count sees exactly one match. +pub(crate) fn saved_ref_tree( + root: crate::locator::live_locator::ObservationRoot<'_>, + role: &str, + name: Option<&str>, +) -> Result { + if matches!( + root, + crate::locator::live_locator::ObservationRoot::Element { .. } + ) { + return saved_ref_element_tree(root); + } + let mut node = minimal_accessibility_node(role, name.unwrap_or_default()); + node.identity.name = name.map(str::to_string); + observed_tree(&root, node) +} + +/// The landed element a resolution re-identifies: the anchored entry's own +/// identity, which is what an element-root observation answers for. +pub(crate) fn saved_ref_element_tree( + root: crate::locator::live_locator::ObservationRoot<'_>, +) -> Result { + let crate::locator::live_locator::ObservationRoot::Element { entry, .. } = root else { + return Err(crate::AdapterError::internal( + "an element observation needs an element root", + )); + }; + let mut node = minimal_accessibility_node(&entry.identity.role, ""); + node.identity.name = entry.identity.name.clone(); + node.identity.value = entry.identity.value.clone(); + node.identity.description = entry.identity.description.clone(); + node.identity.native_id = entry.identity.native_id.clone(); + observed_tree(&root, node) +} + +/// An observation of several nodes, for a stub whose tests save refs for more +/// than one identity: the root itself is a nameless group none of them match. +pub(crate) fn saved_ref_tree_nodes( + root: crate::locator::live_locator::ObservationRoot<'_>, + nodes: &[(&str, Option<&str>, Option<&str>)], +) -> Result { + if matches!( + root, + crate::locator::live_locator::ObservationRoot::Element { .. } + ) { + return saved_ref_element_tree(root); + } + let mut group = minimal_accessibility_node("group", ""); + group.identity.name = None; + group.children = nodes + .iter() + .map(|(role, name, value)| { + let mut node = minimal_accessibility_node(role, name.unwrap_or_default()); + node.identity.name = name.map(str::to_string); + node.identity.value = value.map(str::to_string); + node + }) + .collect(); + observed_tree(&root, group) +} + +/// The anchor landing a saved-ref resolution needs: the stub's own strict +/// resolution answer for the positional anchor built from the live node. +macro_rules! saved_ref_anchor { + () => { + fn resolve_locator_anchor( + &self, + entry: &$crate::RefEntry, + deadline: $crate::Deadline, + ) -> Result<$crate::adapter::NativeHandle, $crate::AdapterError> { + self.resolve_element_strict(entry, deadline) + } + }; +} + +pub(crate) use saved_ref_anchor; + +/// The observation a saved-ref resolution makes: one node carrying the +/// identity the test's entry expects to match, plus its anchor landing. +macro_rules! saved_ref_observation { + ($role:expr, $name:expr) => { + fn observe_tree( + &self, + root: $crate::locator::live_locator::ObservationRoot<'_>, + _request: &$crate::locator::live_locator::ObservationRequest, + ) -> Result<$crate::locator::live_locator::ObservedTree, $crate::AdapterError> { + $crate::adapter::saved_ref_tree(root, $role, $name) + } + + $crate::adapter::saved_ref_anchor!(); + }; +} + +pub(crate) use saved_ref_observation; + +/// [`saved_ref_observation`] for a test stub whose entries name more than one +/// identity. +macro_rules! saved_ref_observation_nodes { + ($nodes:expr) => { + fn observe_tree( + &self, + root: $crate::locator::live_locator::ObservationRoot<'_>, + _request: &$crate::locator::live_locator::ObservationRequest, + ) -> Result<$crate::locator::live_locator::ObservedTree, $crate::AdapterError> { + $crate::adapter::saved_ref_tree_nodes(root, $nodes) + } + + $crate::adapter::saved_ref_anchor!(); + }; +} + +pub(crate) use saved_ref_observation_nodes; + +/// [`saved_ref_observation`] plus the window inventory the resolution +/// searches, for a stub that does not ship one of its own. +macro_rules! saved_ref_scope { + ($role:expr, $name:expr) => { + $crate::adapter::saved_ref_scope!(@inventory); + $crate::adapter::saved_ref_observation!($role, $name); + }; + (nodes: $nodes:expr) => { + $crate::adapter::saved_ref_scope!(@inventory); + $crate::adapter::saved_ref_observation_nodes!($nodes); + }; + (@inventory) => { + fn list_windows( + &self, + _filter: &$crate::adapter::WindowFilter, + _deadline: $crate::Deadline, + ) -> Result, $crate::AdapterError> { + Ok($crate::adapter::saved_ref_window_pool()) + } + }; +} + +pub(crate) use saved_ref_scope; diff --git a/crates/core/src/adapter/test_support.rs b/crates/core/src/adapter/test_support.rs index feed06e8e..db31a9c2f 100644 --- a/crates/core/src/adapter/test_support.rs +++ b/crates/core/src/adapter/test_support.rs @@ -149,6 +149,7 @@ pub(crate) fn minimal_ref_entry(role: &str, name: Option<&str>) -> crate::RefEnt value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -165,6 +166,7 @@ pub(crate) fn minimal_ref_entry(role: &str, name: Option<&str>) -> crate::RefEnt source_window_title: None, source_window_bounds_hash: None, source_surface: crate::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -250,12 +252,17 @@ pub(crate) fn observed_tree( fn subtree(node: crate::AccessibilityNode) -> ObservedSubtree { let native_id = node.identity.native_id.clone(); + let retained = node.identity.retained_object.clone(); + let typed = IdentifierEvidence::typed(native_id, Some(0), true); let evidence = LocatorEvidence { role: LocatorField::Known(node.role), name: field(node.identity.name), description: field(node.identity.description), value: field(node.identity.value), - identifiers: IdentifierEvidence::typed(native_id, Some(0), true), + identifiers: match retained { + Some(token) => typed.with_retained_object(token), + None => typed, + }, states: LocatorField::Known(node.presentation.states), ref_evidence: LocatorRefEvidence { window_bounds: LocatorField::Absent, diff --git a/crates/core/src/commands/drag_occlusion_retry_tests.rs b/crates/core/src/commands/drag_occlusion_retry_tests.rs index 98149bc8d..3f2ffd7fc 100644 --- a/crates/core/src/commands/drag_occlusion_retry_tests.rs +++ b/crates/core/src/commands/drag_occlusion_retry_tests.rs @@ -66,6 +66,7 @@ impl ObservationOps for TransientOcclusionAdapter { Ok(HitTestResult::ReachesTarget) } } + crate::adapter::saved_ref_scope!("button", Some("Item")); } impl ActionOps for TransientOcclusionAdapter {} diff --git a/crates/core/src/commands/drag_retry_tests.rs b/crates/core/src/commands/drag_retry_tests.rs index f5f762785..13c66ba3b 100644 --- a/crates/core/src/commands/drag_retry_tests.rs +++ b/crates/core/src/commands/drag_retry_tests.rs @@ -51,6 +51,7 @@ impl ObservationOps for StaleThenOkAdapter { ) -> Result { Ok(HitTestResult::ReachesTarget) } + crate::adapter::saved_ref_scope!("button", Some("Item")); } impl ActionOps for StaleThenOkAdapter {} @@ -162,6 +163,7 @@ impl ObservationOps for OccludedFromAdapter { bounds: None, }) } + crate::adapter::saved_ref_scope!("button", Some("Item")); } impl ActionOps for OccludedFromAdapter {} @@ -264,6 +266,7 @@ impl ObservationOps for MutuallyOffscreenAdapter { ) -> Result { Ok(HitTestResult::ReachesTarget) } + crate::adapter::saved_ref_scope!("button", Some("Item")); } impl ActionOps for MutuallyOffscreenAdapter { diff --git a/crates/core/src/commands/drag_tests.rs b/crates/core/src/commands/drag_tests.rs index ce2b0d216..236f3f8f3 100644 --- a/crates/core/src/commands/drag_tests.rs +++ b/crates/core/src/commands/drag_tests.rs @@ -92,6 +92,7 @@ impl ObservationOps for DragCaptureAdapter { ) -> Result { Ok(HitTestResult::ReachesTarget) } + crate::adapter::saved_ref_scope!("button", Some("Item")); } impl ActionOps for DragCaptureAdapter {} @@ -273,6 +274,7 @@ fn ref_entry(pid: u32) -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -288,7 +290,7 @@ fn ref_entry(pid: u32) -> RefEntry { source_window_id: Some(format!("w-{pid}")), source_window_title: Some(format!("Window {pid}")), source_window_bounds_hash: None, - source_surface: crate::adapter::SnapshotSurface::Window, + ..Default::default() }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/commands/drag_wait_scope_tests.rs b/crates/core/src/commands/drag_wait_scope_tests.rs index d7fc72b47..caffc92b5 100644 --- a/crates/core/src/commands/drag_wait_scope_tests.rs +++ b/crates/core/src/commands/drag_wait_scope_tests.rs @@ -62,10 +62,10 @@ impl CrossAppWaitAdapter { } fn window_node(window: &WindowInfo) -> AccessibilityNode { - let children = (window.id == "w-2") - .then(Self::dropped_confirmation) - .into_iter() - .collect(); + let mut children = vec![crate::adapter::minimal_accessibility_node("button", "Item")]; + if window.id == "w-2" { + children.push(Self::dropped_confirmation()); + } let mut node = crate::adapter::minimal_accessibility_node("window", &window.title); node.children = children; node @@ -78,6 +78,12 @@ impl ObservationOps for CrossAppWaitAdapter { root: crate::locator::live_locator::ObservationRoot<'_>, _request: &crate::locator::live_locator::ObservationRequest, ) -> Result { + if matches!( + root, + crate::locator::live_locator::ObservationRoot::Element { .. } + ) { + return crate::adapter::saved_ref_element_tree(root); + } let crate::locator::live_locator::ObservationRoot::Window(window) = root else { return Err(AdapterError::internal("expected window root")); }; @@ -127,6 +133,7 @@ impl ObservationOps for CrossAppWaitAdapter { "Item", [crate::actions::capability::CLICK] ); + crate::adapter::saved_ref_anchor!(); } impl ActionOps for CrossAppWaitAdapter {} @@ -247,12 +254,18 @@ fn drag_wait_scoped_to_source_window_misses_destination_confirmation() { assert!(details["after_action"].get("dragged").is_none()); let observed = adapter.observed.lock().unwrap(); assert!( - observed.iter().all(|id| id == "w-1"), - "wait must only inspect the from window: {observed:?}", + observed.iter().filter(|id| *id == "w-1").count() > 1, + "the wait must keep polling the from window: {observed:?}", + ); + assert!( + observed.iter().filter(|id| *id == "w-2").count() <= 2, + "the drop-target window is observed once per endpoint resolution and never by the wait: \ + {observed:?}", ); assert!( - !observed.iter().any(|id| id == "w-2"), - "the drop-target window must never be inspected under from scope: {observed:?}", + observed.iter().filter(|id| *id == "w-1").count() + > observed.iter().filter(|id| *id == "w-2").count(), + "the wait polls the from window, not the drop target: {observed:?}", ); assert!(adapter.captured.lock().unwrap().is_some()); } diff --git a/crates/core/src/commands/execute_by_ref_tests.rs b/crates/core/src/commands/execute_by_ref_tests.rs index ddac972b8..8ae5d3d32 100644 --- a/crates/core/src/commands/execute_by_ref_tests.rs +++ b/crates/core/src/commands/execute_by_ref_tests.rs @@ -32,6 +32,7 @@ fn snapshot_with_ref(role: &str, available_actions: &[&str]) -> String { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -48,6 +49,7 @@ fn snapshot_with_ref(role: &str, available_actions: &[&str]) -> String { source_window_title: Some("Fixture".into()), source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -95,6 +97,7 @@ impl ObservationOps for StaleThenOkAdapter { crate::actions::capability::SET_VALUE ] ); + crate::adapter::saved_ref_scope!(nodes: &[("button", Some("Target"), None), ("textfield", Some("Target"), None)]); } impl ActionOps for StaleThenOkAdapter { @@ -156,6 +159,7 @@ impl ObservationOps for PolicyCaptureAdapter { offscreen: Some(false), } ); + crate::adapter::saved_ref_scope!(nodes: &[("button", Some("Target"), None), ("textfield", Some("Target"), None)]); } impl ActionOps for PolicyCaptureAdapter { diff --git a/crates/core/src/commands/find.rs b/crates/core/src/commands/find.rs index 0ca34b8bb..c98111327 100644 --- a/crates/core/src/commands/find.rs +++ b/crates/core/src/commands/find.rs @@ -199,6 +199,10 @@ mod tests; #[path = "find_live_test_support.rs"] mod test_support; +#[cfg(test)] +#[path = "find_rooted_tests.rs"] +mod rooted_tests; + #[cfg(test)] #[path = "find_live_tests.rs"] mod live_tests; diff --git a/crates/core/src/commands/find_live.rs b/crates/core/src/commands/find_live.rs index b635718b9..aeb5d7abd 100644 --- a/crates/core/src/commands/find_live.rs +++ b/crates/core/src/commands/find_live.rs @@ -5,9 +5,11 @@ use crate::{ context::CommandContext, locator::live_locator::{ LocatorMaterialization, LocatorResolution, LocatorResolveRequest, LocatorSelection, - ObservationRoot, resolve_query, + ObservationRequest, ObservationRoot, ObservedObservation, resolve_query, + selects_tokenless_entry, }, refs::RefMap, + refs::ref_root_read::{RootRead, read_saved_root}, refs::refs_store::RefStore, snapshot, trace::trace_artifacts, @@ -37,21 +39,7 @@ pub(super) fn execute( let deadline = traversal_deadline(args)?; let request = resolve_request(args, deadline); let mut resolution = match args.root.as_deref() { - Some(root_ref) => { - let (_, local_root_ref) = crate::refs::ref_token::resolve_ref_target(root_ref)?; - let entry = crate::commands::helpers::load_ref_entry(root_ref, context)?; - let handle = adapter.resolve_element_strict(&entry, deadline)?; - resolve_query( - adapter, - query, - ObservationRoot::Element { - handle: &handle, - entry: &entry, - root_ref: Some(&local_root_ref), - }, - &request, - )? - } + Some(root_ref) => rooted_resolution(adapter, query, root_ref, &request, context)?, None => { let windows = snapshot::surface_windows( adapter, @@ -86,6 +74,50 @@ pub(super) fn execute( format_response(args, query, resolution, snapshot_id.as_deref()) } +/// A rooted find draws its matches from the root's own subtree. A root with an +/// element token is read as its subtree alone when every ref that subtree could +/// publish carries a token too, since a token re-identifies itself. Any other +/// root is re-identified over one complete whole-window observation - through +/// its predicate - and the query runs scoped to that node: the refs the find +/// publishes are then proven by the one observation it took, never by counts +/// joined from two reads. +fn rooted_resolution( + adapter: &dyn PlatformAdapter, + query: &LocatorQuery, + root_ref: &str, + request: &LocatorResolveRequest, + context: &CommandContext, +) -> Result { + let (_, local_root_ref) = crate::refs::ref_token::resolve_ref_target(root_ref)?; + let entry = crate::commands::helpers::load_ref_entry(root_ref, context)?; + crate::locator::live_locator::resolve_query_from(adapter, query, request, |deadline| { + let read = read_saved_root( + adapter, + &entry, + deadline, + |_, handle| { + let root = ObservationRoot::Element { + handle, + entry: &entry, + root_ref: Some(&local_root_ref), + }; + let observation = + ObservationRequest::locator_for_root(query, request, root, deadline) + .validate()?; + crate::tree::renderer_accessibility::observe_tree(adapter, root, &observation, None) + }, + |tree| !selects_tokenless_entry(tree), + )?; + match read { + RootRead::Subtree(tree) => Ok(ObservedObservation::whole(tree)), + RootRead::Window(window_read) => { + let (_, tree, index) = *window_read; + ObservedObservation::under(tree, index) + } + } + }) +} + fn resolve_request(args: &FindArgs, deadline: crate::Deadline) -> LocatorResolveRequest { let selection = if args.selection.count { LocatorSelection::Count diff --git a/crates/core/src/commands/find_live_test_support.rs b/crates/core/src/commands/find_live_test_support.rs index 074b0557b..28ece857e 100644 --- a/crates/core/src/commands/find_live_test_support.rs +++ b/crates/core/src/commands/find_live_test_support.rs @@ -8,6 +8,8 @@ use crate::{ }; pub(crate) const FRAMED_LABEL: &str = "FramedLabel"; +pub(crate) const TOKEN_ROOT: &str = "axtok1:find-root"; +pub(crate) const TOKEN_ROOT_NAME: &str = "Root"; pub(crate) fn framed_label_screen_bounds() -> crate::Rect { crate::Rect { @@ -30,6 +32,10 @@ fn framed_label_window_bounds() -> crate::Rect { pub(crate) struct LiveFindAdapter { structurally_complete: bool, framed_label: bool, + token_root: bool, + token_matches: bool, + unread_member: bool, + window_reads: std::sync::atomic::AtomicUsize, } impl LiveFindAdapter { @@ -37,23 +43,66 @@ impl LiveFindAdapter { Self { structurally_complete: true, framed_label: false, + token_root: false, + token_matches: false, + unread_member: false, + window_reads: Default::default(), } } pub(crate) fn incomplete() -> Self { Self { structurally_complete: false, - framed_label: false, + ..Self::complete() } } pub(crate) fn with_framed_label() -> Self { Self { - structurally_complete: true, framed_label: true, + ..Self::complete() + } + } + + /// A window whose root element carries an element token while the elements + /// inside it carry none. + pub(crate) fn with_token_root() -> Self { + Self { + token_root: true, + ..Self::complete() + } + } + + /// A token root whose inner elements carry tokens too. + pub(crate) fn with_token_root_and_matches() -> Self { + Self { + token_matches: true, + ..Self::with_token_root() + } + } + + /// A token root whose matches carry tokens beside a tokenless member whose + /// actions are still unread, which hydration could make publishable. + pub(crate) fn with_token_root_and_unread_member() -> Self { + Self { + unread_member: true, + ..Self::with_token_root_and_matches() } } + pub(crate) fn window_reads(&self) -> usize { + self.window_reads.load(std::sync::atomic::Ordering::SeqCst) + } + + fn button(&self, name: &str) -> LocatorEvidence { + let mut evidence = Self::evidence("button", Some(name)); + if self.token_matches { + evidence.identifiers = + IdentifierEvidence::absent().with_retained_object(format!("axtok1:{name}")); + } + evidence + } + fn evidence(role: &str, name: Option<&str>) -> LocatorEvidence { LocatorEvidence { role: LocatorField::Known(role.into()), @@ -85,11 +134,19 @@ impl ObservationOps for LiveFindAdapter { request: &ObservationRequest, ) -> Result { if let ObservationRoot::Element { entry, .. } = &root { + let own = Self::evidence(&entry.identity.role, entry.identity.name.as_deref()); + let mut children = if self.token_root { + vec![self.node(self.button("OnlyInWindowTwo"), Vec::new())] + } else { + Vec::new() + }; + if self.unread_member { + let mut member = Self::evidence("group", Some("Unread")); + member.ref_evidence.available_actions = LocatorField::Unknown; + children.push(self.node(member, Vec::new())); + } return ObservedTree::from_roots( - vec![self.node( - Self::evidence(&entry.identity.role, entry.identity.name.as_deref()), - Vec::new(), - )], + vec![self.node(own, children)], ObservationSource::from_root(&root, request.surface), LocatorStats::default(), self.structurally_complete, @@ -99,18 +156,26 @@ impl ObservationOps for LiveFindAdapter { return Err(AdapterError::internal("expected locator root")); }; let window = *window; + self.window_reads + .fetch_add(1, std::sync::atomic::Ordering::SeqCst); let marker = if window.id == "w-2" { "OnlyInWindowTwo" } else { "OnlyInWindowOne" }; - let mut children = vec![self.node(Self::evidence("button", Some(marker)), Vec::new())]; + let mut children = vec![self.node(self.button(marker), Vec::new())]; if self.framed_label { let mut label = Self::evidence("statictext", Some(FRAMED_LABEL)); label.ref_evidence.bounds = LocatorField::Known(framed_label_screen_bounds()); label.ref_evidence.window_bounds = LocatorField::Known(framed_label_window_bounds()); children.push(self.node(label, Vec::new())); } + if self.token_root { + let mut token_root = Self::evidence("group", Some(TOKEN_ROOT_NAME)); + token_root.identifiers = + IdentifierEvidence::absent().with_retained_object(TOKEN_ROOT.into()); + children = vec![self.node(token_root, children)]; + } let root_node = self.node(Self::evidence("window", Some(&window.title)), children); ObservedTree::from_roots( vec![root_node], @@ -153,6 +218,14 @@ impl ObservationOps for LiveFindAdapter { ]) } + fn resolve_element_strict( + &self, + _entry: &crate::refs::RefEntry, + _deadline: crate::Deadline, + ) -> Result { + Ok(NativeHandle::null()) + } + fn resolve_locator_anchor( &self, _entry: &crate::refs::RefEntry, diff --git a/crates/core/src/commands/find_rooted_tests.rs b/crates/core/src/commands/find_rooted_tests.rs new file mode 100644 index 000000000..df0ba4502 --- /dev/null +++ b/crates/core/src/commands/find_rooted_tests.rs @@ -0,0 +1,180 @@ +use super::test_support::LiveFindAdapter; +use super::*; +use crate::{refs::RefMap, refs::refs_store::RefStore, refs::refs_test_support::HomeGuard}; + +/// find publishes a ref only when the read that produced it counted the element +/// once, and a tokenless root has to be part of that same read: an element-rooted +/// observation of the root carries no census, so the refs a rooted find returned +/// could never be used. Rooting also stops the match set at the root's subtree, +/// so the window above it is not a candidate. +#[test] +fn find_refs_are_proven_by_the_observation_that_produced_them() { + let _guard = HomeGuard::new(); + let adapter = LiveFindAdapter::complete(); + + let unrooted = execute( + find_args(None, Some("OnlyInWindowTwo"), None), + &adapter, + &CommandContext::default(), + ) + .expect("live find should succeed"); + assert_proven_match(&unrooted); + + let mut stored = RefMap::new(); + stored.allocate(stored_button_root()); + let root_snapshot = RefStore::new().unwrap().save_new_snapshot(&stored).unwrap(); + let root_ref = format!("@{root_snapshot}:e1"); + + let found = execute( + find_args(Some(root_ref.clone()), Some("OnlyInWindowTwo"), None), + &adapter, + &CommandContext::default(), + ) + .expect("a rooted find answers from the read that re-identified its root"); + assert_eq!(found["matches"].as_array().map(Vec::len), Some(1)); + assert_proven_match(&found); + + let outside = execute( + find_args(Some(root_ref), None, Some("window")), + &adapter, + &CommandContext::default(), + ) + .expect("a rooted find that matches nothing inside its root still answers"); + assert_eq!( + outside["matches"].as_array().map(Vec::len), + Some(0), + "the window holding the root is outside the subtree the root scopes" + ); +} + +fn find_args(root: Option, name: Option<&str>, role: Option<&str>) -> FindArgs { + FindArgs { + app: None, + window_id: root.is_none().then(|| "w-2".into()), + root, + surface: crate::SnapshotSurface::Window, + filter: FindFilterArgs { + role: role.map(Into::into), + name: name.map(Into::into), + description: None, + native_id: None, + value: None, + text: None, + exact: false, + }, + states: Vec::new(), + timeout_ms: None, + selection: FindSelectionArgs { + count: false, + first: false, + last: false, + nth: None, + limit: None, + }, + } +} + +fn assert_proven_match(response: &serde_json::Value) { + let snapshot_id = response["snapshot_id"] + .as_str() + .expect("find refs carry their snapshot namespace"); + let ref_id = response["matches"][0]["ref_id"] + .as_str() + .expect("the match carries a ref"); + let (_, local_ref) = crate::refs::ref_token::resolve_ref_target(ref_id).unwrap(); + let refmap = RefStore::new().unwrap().load_snapshot(snapshot_id).unwrap(); + let entry = refmap + .get(&local_ref) + .expect("the match's ref is in the snapshot it names"); + assert!( + entry.identity.unique_at_capture, + "a find ref must carry the capture proof of the read that produced it" + ); +} + +/// A root that re-identifies itself by token says nothing about whether the +/// elements inside it carry one, so when one that would be published does not, +/// a rooted find reads the whole window and proves the matches there. The +/// element-rooted read of the same subtree carries no such proof, and the ref +/// it returned could never resolve. +#[test] +fn a_token_root_find_proves_its_tokenless_matches_over_the_window() { + let _guard = HomeGuard::new(); + let adapter = LiveFindAdapter::with_token_root(); + + let mut stored = RefMap::new(); + stored.allocate(stored_token_root()); + let snapshot_id = RefStore::new().unwrap().save_new_snapshot(&stored).unwrap(); + let root_ref = format!("@{snapshot_id}:e1"); + + let found = execute( + find_args(Some(root_ref), Some("OnlyInWindowTwo"), None), + &adapter, + &CommandContext::default(), + ) + .expect("a rooted find answers from the read that re-identified its token root"); + assert_eq!(found["matches"].as_array().map(Vec::len), Some(1)); + assert_proven_match(&found); +} + +/// When every element a token root holds carries a token too, the find reads +/// that subtree alone and never pays for the window around it. +#[test] +fn a_token_root_find_over_token_matches_reads_only_its_subtree() { + let _guard = HomeGuard::new(); + let adapter = LiveFindAdapter::with_token_root_and_matches(); + + let mut stored = RefMap::new(); + stored.allocate(stored_token_root()); + let snapshot_id = RefStore::new().unwrap().save_new_snapshot(&stored).unwrap(); + let root_ref = format!("@{snapshot_id}:e1"); + + let found = execute( + find_args(Some(root_ref), Some("OnlyInWindowTwo"), None), + &adapter, + &CommandContext::default(), + ) + .expect("a token root find answers from its own subtree"); + assert_eq!(found["matches"].as_array().map(Vec::len), Some(1)); + assert_eq!(adapter.window_reads(), 0); +} + +/// A tokenless member whose actions are unread could become publishable once +/// hydrated, so the subtree alone cannot stand and the window proves it. +#[test] +fn a_token_root_with_an_unread_tokenless_member_reads_the_window() { + let _guard = HomeGuard::new(); + let adapter = LiveFindAdapter::with_token_root_and_unread_member(); + + let mut stored = RefMap::new(); + stored.allocate(stored_token_root()); + let snapshot_id = RefStore::new().unwrap().save_new_snapshot(&stored).unwrap(); + let root_ref = format!("@{snapshot_id}:e1"); + + let _ = execute( + find_args(Some(root_ref), Some("OnlyInWindowTwo"), None), + &adapter, + &CommandContext::default(), + ); + assert_eq!(adapter.window_reads(), 1); +} + +fn stored_token_root() -> crate::refs::RefEntry { + let mut entry = + crate::adapter::minimal_ref_entry("group", Some(super::test_support::TOKEN_ROOT_NAME)); + entry.identity.retained_object = Some(super::test_support::TOKEN_ROOT.into()); + entry.process.pid = crate::ProcessId::new(102); + entry.source.source_app = Some("FixtureApp".into()); + entry.source.source_window_id = Some("w-2".into()); + entry.source.source_window_title = Some("Second".into()); + entry +} + +fn stored_button_root() -> crate::refs::RefEntry { + let mut entry = crate::adapter::minimal_ref_entry("button", Some("OnlyInWindowTwo")); + entry.process.pid = crate::ProcessId::new(102); + entry.source.source_app = Some("FixtureApp".into()); + entry.source.source_window_id = Some("w-2".into()); + entry.source.source_window_title = Some("Second".into()); + entry +} diff --git a/crates/core/src/commands/get_consistency_tests.rs b/crates/core/src/commands/get_consistency_tests.rs index 54d5f0f06..2ebf9d3f1 100644 --- a/crates/core/src/commands/get_consistency_tests.rs +++ b/crates/core/src/commands/get_consistency_tests.rs @@ -105,6 +105,7 @@ impl ObservationOps for FailedLiveRead { ) -> Result, AdapterError> { Err(AdapterError::timeout("live read failed")) } + crate::adapter::saved_ref_scope!("checkbox", Some("Target")); } impl ActionOps for FailedLiveRead {} diff --git a/crates/core/src/commands/get_live_value_tests.rs b/crates/core/src/commands/get_live_value_tests.rs index 8d4b68a33..bfde0e157 100644 --- a/crates/core/src/commands/get_live_value_tests.rs +++ b/crates/core/src/commands/get_live_value_tests.rs @@ -18,6 +18,10 @@ impl ObservationOps for LiveValue { ) -> Result, AdapterError> { Ok(self.0.clone()) } + crate::adapter::saved_ref_scope!(nodes: &[ + ("checkbox", Some("Target"), None), + ("textfield", Some("Target"), None), + ]); } impl ActionOps for LiveValue {} diff --git a/crates/core/src/commands/get_tests.rs b/crates/core/src/commands/get_tests.rs index 6447341dd..a5dc6fbfc 100644 --- a/crates/core/src/commands/get_tests.rs +++ b/crates/core/src/commands/get_tests.rs @@ -53,6 +53,12 @@ impl ObservationOps for LiveBoundsAdapter { } Ok(*self.bounds.lock().unwrap()) } + crate::adapter::saved_ref_scope!(nodes: &[ + ("textfield", Some("text-input"), None), + ("textfield", Some("Search"), None), + ("cell", None, Some("42")), + ("button", Some("Close"), None), + ]); } impl ActionOps for LiveBoundsAdapter {} diff --git a/crates/core/src/commands/helpers.rs b/crates/core/src/commands/helpers.rs index c5fb21fe5..25eed7870 100644 --- a/crates/core/src/commands/helpers.rs +++ b/crates/core/src/commands/helpers.rs @@ -51,31 +51,35 @@ fn resolve_ref_within_deadline( context: &CommandContext, ) -> Result<(RefEntry, crate::adapter::NativeHandle), AppError> { let entry = load_ref_entry(ref_id, context)?; - let handle = resolve_handle_within_deadline(adapter, &entry, deadline).inspect_err(|err| { - let _ = context.trace_lazy("ref.resolve.error", || { - json!({ - "ref": ref_id, - "code": err.code.as_str(), - "message": err.message.clone(), - "details": err.details.clone() - }) - }); - })?; + let handle = + resolve_handle_within_deadline(adapter, &entry, deadline, None).inspect_err(|err| { + let _ = context.trace_lazy("ref.resolve.error", || { + json!({ + "ref": ref_id, + "code": err.code.as_str(), + "message": err.message.clone(), + "details": err.details.clone() + }) + }); + })?; tracing::debug!("resolve: {} resolved successfully", ref_id); context.trace_lazy("ref.resolve.ok", || json!({ "ref": ref_id }))?; Ok((entry, handle)) } /// Gives one-shot resolution its full caller budget; polling paths own their slices. +/// `lease` is an interaction lease the caller already holds, so observation +/// activation reuses it instead of queueing behind the same process's lock. pub(crate) fn resolve_handle_within_deadline( adapter: &dyn PlatformAdapter, entry: &RefEntry, deadline: crate::Deadline, + lease: Option<&crate::InteractionLease>, ) -> Result { if deadline.is_expired() { return Err(deadline.timeout_error()); } - adapter.resolve_element_strict(entry, deadline) + crate::refs::resolve_saved_ref::resolve_saved_ref(adapter, entry, deadline, lease) } pub(crate) fn execute_ref_action_with_context( @@ -83,6 +87,28 @@ pub(crate) fn execute_ref_action_with_context( adapter: &dyn PlatformAdapter, request: ActionRequest, context: &CommandContext, +) -> Result { + execute_ref_action_dispatching( + args, + adapter, + request, + context, + crate::ref_action::ref_action_single::dispatch_resolved, + ) +} + +/// Runs a ref action through `dispatch`, for a caller that keeps what the +/// dispatch lands on. +pub(crate) fn execute_ref_action_dispatching( + args: RefArgs, + adapter: &dyn PlatformAdapter, + request: ActionRequest, + context: &CommandContext, + dispatch: impl Fn( + crate::ref_action::ref_action_context::RefActionContext<'_>, + ActionRequest, + &crate::InteractionLease, + ) -> Result, ) -> Result { let request = request.with_timeout_ms(args.timeout_ms); validate_post_action_wait(context)?; @@ -99,7 +125,7 @@ pub(crate) fn execute_ref_action_with_context( context, }, request, - crate::ref_action::dispatch_resolved, + dispatch, ) .map_err(AppError::Adapter)?; let value = serde_json::to_value(result).map_err(|error| { @@ -268,7 +294,7 @@ pub(crate) fn execute_ref_action_result_with_context( context, }, request, - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .map_err(AppError::Adapter)?; Ok((entry, result)) diff --git a/crates/core/src/commands/helpers_pipeline_lease_tests.rs b/crates/core/src/commands/helpers_pipeline_lease_tests.rs index 944c1634e..ec4de60ab 100644 --- a/crates/core/src/commands/helpers_pipeline_lease_tests.rs +++ b/crates/core/src/commands/helpers_pipeline_lease_tests.rs @@ -94,6 +94,7 @@ impl ObservationOps for CountingPipelineAdapter { ) -> Result { Ok(crate::adapter::hit_test::HitTestResult::ReachesTarget) } + crate::adapter::saved_ref_scope!("button", Some("OK")); } impl ActionOps for CountingPipelineAdapter { diff --git a/crates/core/src/commands/helpers_ref_action_dispatch_tests.rs b/crates/core/src/commands/helpers_ref_action_dispatch_tests.rs index 25d9d80dc..15d2523dd 100644 --- a/crates/core/src/commands/helpers_ref_action_dispatch_tests.rs +++ b/crates/core/src/commands/helpers_ref_action_dispatch_tests.rs @@ -49,6 +49,10 @@ impl ObservationOps for RecordingAdapter { offscreen: Some(false), } ); + crate::adapter::saved_ref_scope!(nodes: &[ + ("button", Some("OK"), None), + ("textfield", Some("OK"), None), + ]); } impl ActionOps for RecordingAdapter { @@ -97,6 +101,10 @@ impl ObservationOps for AmbiguousAdapter { ), ) } + crate::adapter::saved_ref_scope!(nodes: &[ + ("button", Some("OK"), None), + ("textfield", Some("OK"), None), + ]); } impl ActionOps for AmbiguousAdapter { diff --git a/crates/core/src/commands/helpers_ref_action_wait_tests.rs b/crates/core/src/commands/helpers_ref_action_wait_tests.rs index 578bc1cbe..6c6e66f26 100644 --- a/crates/core/src/commands/helpers_ref_action_wait_tests.rs +++ b/crates/core/src/commands/helpers_ref_action_wait_tests.rs @@ -44,12 +44,15 @@ impl ObservationOps for ScopedWaitAdapter { root: crate::locator::live_locator::ObservationRoot<'_>, _request: &crate::locator::live_locator::ObservationRequest, ) -> Result { - if self.lease_held.load(Ordering::SeqCst) { - return Err(AdapterError::internal( - "post-action wait must poll with the interaction lease released", - )); + if matches!( + root, + crate::locator::live_locator::ObservationRoot::Element { .. } + ) { + return crate::adapter::saved_ref_element_tree(root); + } + if !self.lease_held.load(Ordering::SeqCst) { + self.lease_free_polls.fetch_add(1, Ordering::SeqCst); } - self.lease_free_polls.fetch_add(1, Ordering::SeqCst); crate::adapter::observed_tree( &root, AccessibilityNode { @@ -63,7 +66,7 @@ impl ObservationOps for ScopedWaitAdapter { presentation: Default::default(), children_count: None, subtree_truncated: false, - children: vec![], + children: vec![crate::adapter::minimal_accessibility_node("button", "OK")], }, ) } @@ -118,6 +121,7 @@ impl ObservationOps for ScopedWaitAdapter { } crate::adapter::complete_live_observation!("button", "OK", [crate::actions::capability::CLICK]); + crate::adapter::saved_ref_anchor!(); } impl ActionOps for ScopedWaitAdapter { @@ -192,25 +196,23 @@ impl ObservationOps for MultiWindowAdapter { root: crate::locator::live_locator::ObservationRoot<'_>, _request: &crate::locator::live_locator::ObservationRequest, ) -> Result { + if matches!( + root, + crate::locator::live_locator::ObservationRoot::Element { .. } + ) { + return crate::adapter::saved_ref_element_tree(root); + } let crate::locator::live_locator::ObservationRoot::Window(window) = root else { return Err(AdapterError::internal("expected window root")); }; - let children = (window.id == "w-target") - .then(|| AccessibilityNode { - ref_id: None, - role: "button".into(), - identity: crate::NodeIdentity { - retained_object: None, - name: Some("Saved!".into()), - ..Default::default() - }, - presentation: Default::default(), - children_count: None, - subtree_truncated: false, - children: vec![], - }) - .into_iter() - .collect(); + let children: Vec = if window.id == "w-target" { + vec![ + crate::adapter::minimal_accessibility_node("button", "Saved!"), + crate::adapter::minimal_accessibility_node("button", "OK"), + ] + } else { + Vec::new() + }; crate::adapter::observed_tree( &crate::locator::live_locator::ObservationRoot::Window(window), AccessibilityNode { @@ -309,6 +311,7 @@ impl ObservationOps for MultiWindowAdapter { } crate::adapter::complete_live_observation!("button", "OK", [crate::actions::capability::CLICK]); + crate::adapter::saved_ref_anchor!(); } impl ActionOps for MultiWindowAdapter { diff --git a/crates/core/src/commands/helpers_resolution_budget_tests.rs b/crates/core/src/commands/helpers_resolution_budget_tests.rs index 40bdf8fda..d86160a2d 100644 --- a/crates/core/src/commands/helpers_resolution_budget_tests.rs +++ b/crates/core/src/commands/helpers_resolution_budget_tests.rs @@ -18,6 +18,7 @@ impl ObservationOps for BudgetAdapter { .push((deadline.timeout_ms(), deadline.was_capped())); Ok(NativeHandle::null()) } + crate::adapter::saved_ref_scope!("button", Some("OK")); } impl ActionOps for BudgetAdapter {} impl InputOps for BudgetAdapter {} @@ -31,6 +32,7 @@ fn single_attempt_resolution_receives_the_callers_budget_without_poll_slicing() &adapter, &entry(), crate::Deadline::detached_after(timeout).unwrap(), + None, ) .unwrap(); } @@ -48,7 +50,7 @@ fn expired_single_attempt_budget_never_calls_the_adapter() { 1, ) .unwrap(); - let Err(error) = resolve_handle_within_deadline(&adapter, &entry(), deadline) else { + let Err(error) = resolve_handle_within_deadline(&adapter, &entry(), deadline, None) else { panic!("expired resolution must fail"); }; assert_eq!(error.code, ErrorCode::Timeout); diff --git a/crates/core/src/commands/helpers_tests.rs b/crates/core/src/commands/helpers_tests.rs index 1b2829cb1..0e0a58f18 100644 --- a/crates/core/src/commands/helpers_tests.rs +++ b/crates/core/src/commands/helpers_tests.rs @@ -33,6 +33,7 @@ impl ObservationOps for DropCountingAdapter { ) -> Result { Ok(NativeHandle::new(DropProbe(Arc::clone(&self.drops)))) } + crate::adapter::saved_ref_scope!("button", Some("OK")); } impl ActionOps for DropCountingAdapter {} diff --git a/crates/core/src/commands/hover.rs b/crates/core/src/commands/hover.rs index 95e10a4d4..5f1e40d4e 100644 --- a/crates/core/src/commands/hover.rs +++ b/crates/core/src/commands/hover.rs @@ -118,10 +118,10 @@ fn confirm_receiver( deadline: crate::Deadline, adapter: &dyn PlatformAdapter, ) -> Result<(), AppError> { - let Some(entry) = resolved.source_entry.as_ref() else { + let Some(handle) = resolved.handle.as_ref() else { return Ok(()); }; - adapter.confirm_pointer_receiver(entry, resolved.point.clone(), deadline)?; + adapter.confirm_pointer_receiver(handle, resolved.point.clone(), deadline)?; Ok(()) } diff --git a/crates/core/src/commands/hover_preflight_tests.rs b/crates/core/src/commands/hover_preflight_tests.rs index 8dcbcf12a..1ebd34b6b 100644 --- a/crates/core/src/commands/hover_preflight_tests.rs +++ b/crates/core/src/commands/hover_preflight_tests.rs @@ -56,6 +56,7 @@ impl ObservationOps for OccludedTargetAdapter { bounds: None, }) } + crate::adapter::saved_ref_scope!("button", Some("Target")); } impl ActionOps for OccludedTargetAdapter {} @@ -226,6 +227,7 @@ impl ObservationOps for BackgroundTargetAdapter { }) } } + crate::adapter::saved_ref_scope!("button", Some("Target")); } impl ActionOps for BackgroundTargetAdapter {} diff --git a/crates/core/src/commands/hover_receiver_tests.rs b/crates/core/src/commands/hover_receiver_tests.rs index 54768a23d..7f719faf2 100644 --- a/crates/core/src/commands/hover_receiver_tests.rs +++ b/crates/core/src/commands/hover_receiver_tests.rs @@ -28,3 +28,25 @@ fn unproven_pointer_receiver_refuses_once_before_any_move() { assert!(adapter.moved_to.lock().unwrap().is_none()); } } + +#[test] +fn the_pointer_receiver_is_confirmed_on_the_element_core_resolved() { + let _guard = HomeGuard::new(); + let snapshot_id = ref_snapshot(42); + let adapter = HoverCaptureAdapter::new(); + + execute( + HoverArgs { + ref_id: Some(format!("@{}:e1", snapshot_id)), + xy: None, + duration_ms: None, + timeout_ms: None, + }, + &adapter, + &CommandContext::default().with_headed(true), + ) + .expect("the receiver check runs on the resolved handle"); + + assert_eq!(adapter.receiver_checks.load(Ordering::SeqCst), 1); + assert!(adapter.moved_to.lock().unwrap().is_some()); +} diff --git a/crates/core/src/commands/hover_tests.rs b/crates/core/src/commands/hover_tests.rs index d7c552fff..f78a1129f 100644 --- a/crates/core/src/commands/hover_tests.rs +++ b/crates/core/src/commands/hover_tests.rs @@ -15,6 +15,8 @@ use std::sync::{ atomic::{AtomicU32, Ordering}, }; +const CORE_RESOLVED: &str = "core-resolved"; + struct HoverCaptureAdapter { moved_to: Mutex>, focused_pids: Mutex>, @@ -51,7 +53,7 @@ impl ObservationOps for HoverCaptureAdapter { _entry: &RefEntry, _deadline: crate::Deadline, ) -> Result { - Ok(NativeHandle::null()) + Ok(NativeHandle::new(CORE_RESOLVED)) } fn get_element_bounds( @@ -83,11 +85,16 @@ impl ObservationOps for HoverCaptureAdapter { fn confirm_pointer_receiver( &self, - _entry: &RefEntry, + handle: &NativeHandle, _point: crate::Point, _deadline: crate::Deadline, ) -> Result<(), AdapterError> { self.receiver_checks.fetch_add(1, Ordering::SeqCst); + if handle.downcast_ref::<&str>() != Some(&CORE_RESOLVED) { + return Err(AdapterError::internal( + "the receiver was confirmed on an element core did not resolve", + )); + } if self.refuses_receiver { return Err(AdapterError::new( crate::ErrorCode::ActionFailed, @@ -98,6 +105,7 @@ impl ObservationOps for HoverCaptureAdapter { } Ok(()) } + crate::adapter::saved_ref_scope!("button", Some("Target")); } impl ActionOps for HoverCaptureAdapter {} @@ -149,6 +157,7 @@ fn ref_snapshot(pid: u32) -> String { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -165,6 +174,7 @@ fn ref_snapshot(pid: u32) -> String { source_window_title: Some(format!("Window {pid}")), source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -307,6 +317,7 @@ impl ObservationOps for StaleThenOkAdapter { ) -> Result { Ok(HitTestResult::ReachesTarget) } + crate::adapter::saved_ref_scope!("button", Some("Target")); } impl ActionOps for StaleThenOkAdapter {} diff --git a/crates/core/src/commands/is_check_applicability_tests.rs b/crates/core/src/commands/is_check_applicability_tests.rs index 09fc50bf8..c8920571f 100644 --- a/crates/core/src/commands/is_check_applicability_tests.rs +++ b/crates/core/src/commands/is_check_applicability_tests.rs @@ -15,6 +15,7 @@ fn action_availability_makes_toggle_and_expand_applicable() { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -31,6 +32,7 @@ fn action_availability_makes_toggle_and_expand_applicable() { source_window_title: None, source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/commands/is_check_tests.rs b/crates/core/src/commands/is_check_tests.rs index 982efbd7a..7aef4c5c4 100644 --- a/crates/core/src/commands/is_check_tests.rs +++ b/crates/core/src/commands/is_check_tests.rs @@ -70,6 +70,10 @@ impl ObservationOps for LiveStateAdapter { } Ok(*self.bounds.lock().unwrap()) } + crate::adapter::saved_ref_scope!(nodes: &[ + ("checkbox", Some("Target"), None), + ("cell", Some("Disclosure"), None), + ]); } impl ActionOps for LiveStateAdapter {} @@ -97,6 +101,7 @@ fn entry(states: Vec, value: Option<&str>, actions: Vec<&str>) -> RefEnt value: value.map(str::to_string), description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -112,7 +117,7 @@ fn entry(states: Vec, value: Option<&str>, actions: Vec<&str>) -> RefEnt source_window_id: None, source_window_title: None, source_window_bounds_hash: None, - source_surface: crate::adapter::SnapshotSurface::Window, + ..Default::default() }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/commands/point_resolve.rs b/crates/core/src/commands/point_resolve.rs index f71d10321..dce1fb550 100644 --- a/crates/core/src/commands/point_resolve.rs +++ b/crates/core/src/commands/point_resolve.rs @@ -16,6 +16,7 @@ pub(crate) struct ResolvedPoint { pub focused: bool, pub source_entry: Option, pub bounds_hash: Option, + pub handle: Option, } pub(crate) fn require_cursor_policy( @@ -64,6 +65,7 @@ pub(crate) fn resolve_point_from_ref_or_xy_with_context( focused: false, source_entry: Some(entry), bounds_hash: bounds.bounds_hash(), + handle: Some(handle), }); } if let Some((x, y)) = args.xy { @@ -72,6 +74,7 @@ pub(crate) fn resolve_point_from_ref_or_xy_with_context( focused: false, source_entry: None, bounds_hash: None, + handle: None, }); } Err(AppError::invalid_input(args.missing_input_message)) diff --git a/crates/core/src/commands/point_resolve_tests.rs b/crates/core/src/commands/point_resolve_tests.rs index 282cc86fc..33c2985fa 100644 --- a/crates/core/src/commands/point_resolve_tests.rs +++ b/crates/core/src/commands/point_resolve_tests.rs @@ -70,6 +70,7 @@ impl ObservationOps for HitTestOutcomeAdapter { ) -> Result { self.outcome.clone() } + crate::adapter::saved_ref_scope!("button", Some("Target")); } impl ActionOps for HitTestOutcomeAdapter {} @@ -91,6 +92,7 @@ fn ref_snapshot(pid: u32) -> String { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -107,6 +109,7 @@ fn ref_snapshot(pid: u32) -> String { source_window_title: None, source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -253,6 +256,7 @@ fn focus_ref_entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -269,6 +273,7 @@ fn focus_ref_entry() -> RefEntry { source_window_title: Some("Fixture".into()), source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/commands/pointer_action.rs b/crates/core/src/commands/pointer_action.rs index 6ec7a8f94..d58f59cd9 100644 --- a/crates/core/src/commands/pointer_action.rs +++ b/crates/core/src/commands/pointer_action.rs @@ -15,6 +15,7 @@ struct EntryPointResolve<'a> { entry: &'a RefEntry, stability: Option>, lease: Option<&'a crate::InteractionLease>, + allow_scroll: bool, verify_receives_events: bool, } @@ -35,10 +36,11 @@ fn resolve_point_from_entry( entry, stability, lease, + allow_scroll, verify_receives_events, } = request; let mut handle = - resolve_handle_within_deadline(adapter, entry, deadline).inspect_err(|err| { + resolve_handle_within_deadline(adapter, entry, deadline, lease).inspect_err(|err| { let _ = context.trace_lazy("ref.resolve.error", || { json!({ "ref": ref_id, @@ -60,7 +62,7 @@ fn resolve_point_from_entry( ) .into()); } - let Some(lease) = lease else { + let Some(lease) = lease.filter(|_| allow_scroll) else { let mut details = json!({ "check": "visible", "requires_scroll": true }); if let Some(hash) = bounds.and_then(|rect| rect.bounds_hash()) && let Some(object) = details.as_object_mut() @@ -75,7 +77,7 @@ fn resolve_point_from_entry( .into()); }; adapter.scroll_into_view(&handle, lease)?; - handle = resolve_handle_within_deadline(adapter, entry, deadline)?; + handle = resolve_handle_within_deadline(adapter, entry, deadline, Some(lease))?; let observation = adapter.observe_pointer(&handle, deadline)?; bounds = observation.bounds; states = observation.states; @@ -131,6 +133,7 @@ fn resolve_point_from_entry( focused: false, source_entry: Some(entry.clone()), bounds_hash: Some(observed_bounds_hash), + handle: Some(handle), }) } @@ -188,6 +191,7 @@ pub(crate) fn wait_for_point_with_deadline<'a>( entry: &entry, stability, lease: None, + allow_scroll: false, verify_receives_events: false, }, deadline, @@ -269,7 +273,8 @@ pub(crate) fn resolve_point_under_lease<'a>( ref_id, entry: &entry, stability, - lease: allow_scroll.then_some(lease), + lease: Some(lease), + allow_scroll, verify_receives_events: true, }, deadline, @@ -310,23 +315,24 @@ fn scroll_point_target( ) -> Result<(), AppError> { let lease = adapter.acquire_interaction_lease(deadline)?; focus_point_under_lease(args, &lease, adapter, context)?; - let handle = resolve_handle_within_deadline(adapter, entry, deadline)?; + let handle = resolve_handle_within_deadline(adapter, entry, deadline, Some(&lease))?; adapter.scroll_into_view(&handle, &lease)?; Ok(()) } fn point_requires_scroll(err: &AppError) -> bool { - let AppError::Adapter(error) = err else { - return false; - }; - error - .details - .as_ref() - .and_then(|details| details.get("requires_scroll")) + point_detail(err, "requires_scroll") .and_then(Value::as_bool) .unwrap_or(false) } +fn point_detail<'a>(err: &'a AppError, key: &str) -> Option<&'a Value> { + let AppError::Adapter(error) = err else { + return None; + }; + error.details.as_ref()?.get(key) +} + fn point_error_report(error: &AppError) -> Value { json!({ "code": error.code(), @@ -349,13 +355,7 @@ pub(crate) fn ensure_point_deadline( } fn point_observed_bounds_hash(err: &AppError) -> Option { - let AppError::Adapter(err) = err else { - return None; - }; - err.details - .as_ref() - .and_then(|details| details.get("observed_bounds_hash")) - .and_then(Value::as_u64) + point_detail(err, "observed_bounds_hash").and_then(Value::as_u64) } fn is_retryable_point_error(err: &AppError) -> bool { diff --git a/crates/core/src/commands/pointer_action_focus_tests.rs b/crates/core/src/commands/pointer_action_focus_tests.rs index c7d7f960c..9ac58c85a 100644 --- a/crates/core/src/commands/pointer_action_focus_tests.rs +++ b/crates/core/src/commands/pointer_action_focus_tests.rs @@ -45,6 +45,7 @@ impl ObservationOps for MinimizedWindowAdapter { states, }) } + crate::adapter::saved_ref_scope!("button", Some("Target")); } impl ActionOps for MinimizedWindowAdapter { diff --git a/crates/core/src/commands/pointer_action_tests.rs b/crates/core/src/commands/pointer_action_tests.rs index 89ffd11da..15838bdec 100644 --- a/crates/core/src/commands/pointer_action_tests.rs +++ b/crates/core/src/commands/pointer_action_tests.rs @@ -58,6 +58,7 @@ fn point_ref_snapshot() -> String { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -74,6 +75,7 @@ fn point_ref_snapshot() -> String { source_window_title: Some("Window".into()), source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -124,6 +126,7 @@ impl ObservationOps for TerminalAfterRetryAdapter { .with_suggestion("Run 'snapshot' to refresh, then retry with the updated ref.") .with_disposition(DeliverySemantics::not_delivered())) } + crate::adapter::saved_ref_scope!("button", Some("Target")); } impl ActionOps for TerminalAfterRetryAdapter {} @@ -192,6 +195,7 @@ impl ObservationOps for UnresponsiveThenReadyAdapter { ) -> Result { Ok(HitTestResult::ReachesTarget) } + crate::adapter::saved_ref_scope!("button", Some("Target")); } impl ActionOps for UnresponsiveThenReadyAdapter {} @@ -245,6 +249,7 @@ impl ObservationOps for ZeroSizeUnscrollableAdapter { height: 0.0, })) } + crate::adapter::saved_ref_scope!("button", Some("Target")); } impl ActionOps for ZeroSizeUnscrollableAdapter { diff --git a/crates/core/src/commands/pointer_single_shot_tests.rs b/crates/core/src/commands/pointer_single_shot_tests.rs index c384f1222..afd25962c 100644 --- a/crates/core/src/commands/pointer_single_shot_tests.rs +++ b/crates/core/src/commands/pointer_single_shot_tests.rs @@ -97,6 +97,7 @@ impl ObservationOps for SingleShotScrollAdapter { ) -> Result { Ok(HitTestResult::ReachesTarget) } + crate::adapter::saved_ref_scope!("button", Some("Target")); } impl ActionOps for SingleShotScrollAdapter { @@ -175,10 +176,11 @@ fn ref_entry(pid: u32) -> RefEntry { identity: crate::RefEntryIdentity { retained_object: None, role: "button".into(), - name: Some(format!("Target {pid}")), + name: Some("Target".into()), value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -195,6 +197,7 @@ fn ref_entry(pid: u32) -> RefEntry { source_window_title: Some(format!("Window {pid}")), source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/commands/ref_policy_tests.rs b/crates/core/src/commands/ref_policy_tests.rs index fa9463e1c..d713802f6 100644 --- a/crates/core/src/commands/ref_policy_tests.rs +++ b/crates/core/src/commands/ref_policy_tests.rs @@ -96,6 +96,7 @@ impl ObservationOps for RecordingAdapter { .count(); Ok(Some(0..len)) } + crate::adapter::saved_ref_scope!("textfield", Some("Target")); } impl ActionOps for RecordingAdapter { @@ -111,7 +112,6 @@ impl ActionOps for RecordingAdapter { } impl InputOps for RecordingAdapter {} - impl SystemOps for RecordingAdapter { crate::adapter::guarded_interaction_lease!(); crate::adapter::exact_window_focus!(); @@ -137,6 +137,7 @@ fn snapshot_id() -> String { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -169,7 +170,7 @@ fn snapshot_id() -> String { source_window_id: Some("w-1".into()), source_window_title: Some("Fixture".into()), source_window_bounds_hash: None, - source_surface: crate::adapter::SnapshotSurface::Window, + ..Default::default() }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/commands/right_click_tests.rs b/crates/core/src/commands/right_click_tests.rs index 4c096024d..c601ecff0 100644 --- a/crates/core/src/commands/right_click_tests.rs +++ b/crates/core/src/commands/right_click_tests.rs @@ -80,6 +80,7 @@ impl ObservationOps for ProbeFailingAdapter { "Open", [crate::actions::capability::RIGHT_CLICK] ); + crate::adapter::saved_ref_observation!("button", Some("Open")); } impl ActionOps for ProbeFailingAdapter { @@ -119,6 +120,7 @@ fn save_refmap(source_app: Option) -> String { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -135,6 +137,7 @@ fn save_refmap(source_app: Option) -> String { source_window_title: None, source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/commands/scroll_tests.rs b/crates/core/src/commands/scroll_tests.rs index adb8d8635..511ee84ab 100644 --- a/crates/core/src/commands/scroll_tests.rs +++ b/crates/core/src/commands/scroll_tests.rs @@ -34,6 +34,7 @@ impl ObservationOps for StaleThenOkAdapter { "Target", [crate::actions::capability::SCROLL] ); + crate::adapter::saved_ref_scope!("scrollarea", Some("Target")); } impl ActionOps for StaleThenOkAdapter { diff --git a/crates/core/src/commands/select_tests.rs b/crates/core/src/commands/select_tests.rs index d45da9620..201743646 100644 --- a/crates/core/src/commands/select_tests.rs +++ b/crates/core/src/commands/select_tests.rs @@ -33,6 +33,7 @@ impl ObservationOps for StaleThenOkAdapter { "Target", [crate::actions::capability::SELECT] ); + crate::adapter::saved_ref_scope!("combobox", Some("Target")); } impl ActionOps for StaleThenOkAdapter { diff --git a/crates/core/src/commands/set_value_tests.rs b/crates/core/src/commands/set_value_tests.rs index 98b824d17..25cec0eb4 100644 --- a/crates/core/src/commands/set_value_tests.rs +++ b/crates/core/src/commands/set_value_tests.rs @@ -44,6 +44,7 @@ impl ObservationOps for StaleThenOkAdapter { offscreen: Some(false), } ); + crate::adapter::saved_ref_scope!("textfield", Some("Target")); } impl ActionOps for StaleThenOkAdapter { diff --git a/crates/core/src/commands/snapshot.rs b/crates/core/src/commands/snapshot.rs index 94f1dcde9..186c36f2d 100644 --- a/crates/core/src/commands/snapshot.rs +++ b/crates/core/src/commands/snapshot.rs @@ -148,6 +148,9 @@ pub(crate) fn format_snapshot_fields( body["truncated"] = json!(true); body["nodes_observed"] = json!(result.nodes_observed); } + if result.identity_unproven.is_some() { + body["identity_unproven"] = json!(true); + } if let Some(ms) = elapsed_ms { body["elapsed_ms"] = json!(ms); } diff --git a/crates/core/src/commands/snapshot_tests.rs b/crates/core/src/commands/snapshot_tests.rs index bee9671bf..0b81b2277 100644 --- a/crates/core/src/commands/snapshot_tests.rs +++ b/crates/core/src/commands/snapshot_tests.rs @@ -361,3 +361,31 @@ fn root_and_wait_for_are_mutually_exclusive() { other => panic!("expected InvalidArgs, got {other:?}"), } } + +#[test] +fn an_unproven_capture_is_flagged_in_the_payload() { + let mut result = crate::snapshot::SnapshotResult { + tree: crate::adapter::minimal_accessibility_node("window", "Fixture"), + refmap: crate::RefMap::new(), + window: WindowInfo { + id: "w-1".into(), + title: "Fixture".into(), + app: "Fixture".into(), + pid: crate::ProcessId::new(1), + process_instance: None, + bounds: None, + state: crate::WindowState::default(), + }, + snapshot_id: None, + complete: true, + nodes_observed: 1, + identity_unproven: None, + }; + let proven = format_snapshot_fields(&result, None, None).unwrap(); + assert!(proven.get("identity_unproven").is_none()); + + result.identity_unproven = Some("the whole-window read was cut short".into()); + let unproven = format_snapshot_fields(&result, None, None).unwrap(); + assert_eq!(unproven["identity_unproven"], true); + assert_eq!(unproven["complete"], true); +} diff --git a/crates/core/src/commands/type_text_interval.rs b/crates/core/src/commands/type_text_interval.rs index 4e94e26eb..f7ac95dd0 100644 --- a/crates/core/src/commands/type_text_interval.rs +++ b/crates/core/src/commands/type_text_interval.rs @@ -4,12 +4,14 @@ //! also focuses the target's window. The burst then keeps that element for the //! remaining units and takes the interaction lease only while each unit is //! delivered, so other callers act between keystrokes and per-keystroke -//! resolution cannot make the rhythm uneven. Each unit still goes through the +//! resolution cannot make the rhythm uneven. Each unit re-runs the live identity +//! check under that lease. Each unit still goes through the //! adapter's own delivery checks, which for physical keystrokes confirm that the //! target's process and focused element are unchanged before a key is posted. //! The result is verified against the field as the last keystroke left it, and //! a `--wait-for` condition is waited on once, after the last keystroke. +use std::cell::Cell; use std::time::{Duration, Instant}; use serde_json::{Value, json}; @@ -18,7 +20,7 @@ use crate::{ AdapterError, AppError, DeliverySemantics, actions::action::Action, adapter::PlatformAdapter, - commands::helpers::{RefArgs, execute_ref_action_with_context}, + commands::helpers::{RefArgs, execute_ref_action_dispatching}, commands::type_text::{TypeArgs, default_timeout_ms}, context::CommandContext, }; @@ -47,11 +49,12 @@ pub(crate) fn execute( for (index, unit) in units.iter().enumerate() { let slot = clock.wait(); let request = context.request_base(Action::TypeText(unit.clone())); + let mut landed = None; let delivered = match &burst { Some(burst) => deliver(adapter, burst, request, budget(unit)).map(|_| { crate::ref_action::presentation::move_caret(adapter, context, &burst.handle); }), - None => execute_ref_action_with_context( + None => execute_landing( RefArgs { ref_id: args.ref_id.clone(), timeout_ms: waits.then(|| budget(unit)), @@ -60,13 +63,16 @@ pub(crate) fn execute( request, &per_keystroke, ) - .map(|value| last = value), + .map(|(value, handle)| { + last = value; + landed = Some(handle); + }), }; delivered.map_err(|error| after_partial_delivery(error, index, units.len()))?; clock.delivered(slot); - if burst.is_none() && index + 1 < units.len() { + if let Some(handle) = landed.filter(|_| index + 1 < units.len()) { burst = Some( - resolve_for_burst(adapter, &args.ref_id, context, budget(unit)) + burst_around(handle, &args.ref_id, context) .map_err(|error| after_partial_delivery(error, index + 1, units.len()))?, ); } @@ -133,29 +139,59 @@ fn next_due(slot: Instant, now: Instant, interval: Duration) -> Instant { (slot + interval).max(now + interval / 2) } -/// The element the first keystroke resolved, and the process instance it -/// belongs to, which every later keystroke must still reach. +/// The saved ref, its resolved element and process instance, which every later +/// keystroke must still reach. struct Burst { + entry: crate::RefEntry, handle: crate::NativeHandle, process: Option, } -fn resolve_for_burst( +/// Runs the first unit as a full ref action and returns the element it landed +/// on beside its result, so the burst keeps that element instead of resolving +/// it again. +fn execute_landing( + args: RefArgs, adapter: &dyn PlatformAdapter, + request: crate::ActionRequest, + context: &CommandContext, +) -> Result<(Value, crate::NativeHandle), AppError> { + let landed = Cell::new(None); + let value = execute_ref_action_dispatching( + args, + adapter, + request, + context, + |target, request, lease| { + let (result, handle) = + crate::ref_action::dispatch_resolved_landing(target, request, lease)?; + landed.set(Some(handle)); + Ok(result) + }, + )?; + let handle = landed + .take() + .ok_or_else(|| AdapterError::internal("the ref action landed no element"))?; + Ok((value, handle)) +} + +fn burst_around( + handle: crate::NativeHandle, ref_id: &str, context: &CommandContext, - timeout_ms: u64, ) -> Result { let entry = crate::commands::helpers::load_ref_entry(ref_id, context)?; - let deadline = crate::Deadline::after(timeout_ms)?; - let handle = adapter.resolve_element_strict(&entry, deadline)?; let process = entry .process .process_instance .as_deref() .filter(|instance| !instance.is_empty()) .map(|instance| crate::ProcessIdentity::new(entry.process.pid, instance)); - Ok(Burst { handle, process }) + Ok(Burst { + entry, + handle, + process, + }) } fn deliver( @@ -164,15 +200,36 @@ fn deliver( request: crate::ActionRequest, timeout_ms: u64, ) -> Result { + let deadline = crate::Deadline::after(timeout_ms)?; + let lease = adapter.acquire_interaction_lease(deadline)?; + confirm_target(adapter, burst, &request, deadline)?; let request = match &burst.process { Some(process) => request.with_expected_process(process.clone()), None => request, }; - let deadline = crate::Deadline::after(timeout_ms)?; - let lease = adapter.acquire_interaction_lease(deadline)?; Ok(adapter.execute_action(&burst.handle, request, &lease)?) } +/// Re-runs the first keystroke's live check so a replacement element is refused. +fn confirm_target( + adapter: &dyn PlatformAdapter, + burst: &Burst, + request: &crate::ActionRequest, + deadline: crate::Deadline, +) -> Result<(), AppError> { + crate::actionability::check_live_with_stability( + &crate::actionability::LiveCheckTarget { + entry: &burst.entry, + handle: &burst.handle, + adapter, + deadline, + }, + request, + crate::actionability::StabilityExpectation::permissive(burst.entry.geometry.bounds_hash), + )?; + Ok(()) +} + /// Splits text into the units one keystroke produces: a character together /// with the combining marks, variation selectors, emoji modifiers and /// zero-width-joined characters that render with it, and CR LF as one unit. diff --git a/crates/core/src/commands/type_text_interval_caret_tests.rs b/crates/core/src/commands/type_text_interval_caret_tests.rs index f07f5e159..5de4ed4af 100644 --- a/crates/core/src/commands/type_text_interval_caret_tests.rs +++ b/crates/core/src/commands/type_text_interval_caret_tests.rs @@ -51,6 +51,7 @@ impl ObservationOps for CaretField { offscreen: Some(false), } ); + crate::adapter::saved_ref_scope!("textfield", Some("Name")); } impl ActionOps for CaretField { @@ -92,7 +93,7 @@ fn field_in_session(session: &str) -> String { entry.capabilities.available_actions = vec![capability::CLICK.into(), capability::TYPE_TEXT.into()]; entry.source.source_app = Some("Fixture".into()); - entry.source.source_window_id = Some("w-42".into()); + entry.source.source_window_id = Some("w-1".into()); entry.source.source_window_title = Some("Fixture".into()); let mut refmap = crate::refs::RefMap::new(); refmap.allocate(entry); diff --git a/crates/core/src/commands/type_text_interval_clock_tests.rs b/crates/core/src/commands/type_text_interval_clock_tests.rs index c223e7fdc..88d5b5730 100644 --- a/crates/core/src/commands/type_text_interval_clock_tests.rs +++ b/crates/core/src/commands/type_text_interval_clock_tests.rs @@ -35,6 +35,7 @@ impl ObservationOps for SlowFirstKeystroke { offscreen: Some(false), } ); + crate::adapter::saved_ref_scope!("textfield", Some("Target")); } impl ActionOps for SlowFirstKeystroke { diff --git a/crates/core/src/commands/type_text_interval_identity_tests.rs b/crates/core/src/commands/type_text_interval_identity_tests.rs new file mode 100644 index 000000000..159805e5a --- /dev/null +++ b/crates/core/src/commands/type_text_interval_identity_tests.rs @@ -0,0 +1,117 @@ +use crate::adapter::{ActionOps, InputOps, NativeHandle, ObservationOps, SystemOps}; +use crate::commands::type_text::{TypeArgs, execute}; +use crate::refs::refs_test_support::HomeGuard; +use crate::{ + ActionRequest, ActionResult, AdapterError, CommandContext, Rect, RefEntry, actions::capability, +}; +use std::sync::Mutex; + +/// A text field that a differently named field replaces at the same location +/// once the first keystroke has landed: the handle the burst keeps reaches a +/// different element from then on. +#[derive(Default)] +struct ReplacedField { + typed: Mutex, +} + +impl ReplacedField { + fn live_name(&self) -> &'static str { + match self.typed.lock().unwrap().is_empty() { + true => "Target", + false => "Replacement", + } + } +} + +impl ObservationOps for ReplacedField { + fn resolve_element_strict( + &self, + _entry: &RefEntry, + _deadline: crate::Deadline, + ) -> Result { + Ok(NativeHandle::null()) + } + + crate::adapter::complete_live_observation!( + "textfield", + adapter.live_name(), + [capability::TYPE_TEXT], + adapter => crate::ElementState { + role: "textfield".into(), + states: Vec::new(), + value: Some(adapter.typed.lock().unwrap().clone()), + enabled: Some(true), + hidden: Some(false), + offscreen: Some(false), + } + ); + crate::adapter::saved_ref_scope!("textfield", Some("Target")); +} + +impl ActionOps for ReplacedField { + fn execute_action( + &self, + _handle: &NativeHandle, + request: ActionRequest, + _lease: &crate::InteractionLease, + ) -> Result { + if let crate::Action::TypeText(text) = request.action { + self.typed.lock().unwrap().push_str(&text); + } + Ok(ActionResult::delivered_unverified("type_text")) + } +} + +impl InputOps for ReplacedField {} + +impl SystemOps for ReplacedField { + crate::adapter::guarded_interaction_lease!(); + crate::adapter::exact_window_focus!(); +} + +pub(super) fn field_snapshot() -> String { + let bounds = Rect { + x: 1.0, + y: 1.0, + width: 20.0, + height: 20.0, + }; + let mut entry = crate::adapter::minimal_ref_entry("textfield", Some("Target")); + entry.geometry.bounds = Some(bounds); + entry.geometry.bounds_hash = bounds.bounds_hash(); + entry.capabilities.available_actions = vec![capability::TYPE_TEXT.into()]; + entry.source.source_app = Some("Fixture".into()); + entry.source.source_window_id = Some("w-1".into()); + entry.source.source_window_title = Some("Fixture".into()); + let mut refmap = crate::refs::RefMap::new(); + refmap.allocate(entry); + crate::refs::refs_store::RefStore::new() + .expect("the ref store") + .save_new_snapshot(&refmap) + .expect("the snapshot") +} + +/// `type --interval-ms` re-identifies the burst's element under every +/// keystroke's lease: once a field with another name occupies the same +/// location, the remaining text is refused instead of typed into it. +#[test] +fn a_field_replaced_mid_burst_refuses_the_remaining_keystrokes() { + let _home = HomeGuard::new(); + let snapshot_id = field_snapshot(); + let adapter = ReplacedField::default(); + + let error = execute( + TypeArgs { + ref_id: format!("@{snapshot_id}:e1"), + text: "abc".into(), + timeout_ms: None, + interval_ms: Some(5), + }, + &adapter, + &CommandContext::default(), + ) + .expect_err("the replacement is refused"); + + assert_eq!(error.code(), "STALE_REF"); + assert_eq!(*adapter.typed.lock().unwrap(), "a"); +} diff --git a/crates/core/src/commands/type_text_interval_observation_tests.rs b/crates/core/src/commands/type_text_interval_observation_tests.rs new file mode 100644 index 000000000..735d4afa2 --- /dev/null +++ b/crates/core/src/commands/type_text_interval_observation_tests.rs @@ -0,0 +1,98 @@ +use crate::adapter::{ActionOps, InputOps, NativeHandle, ObservationOps, SystemOps}; +use crate::commands::type_text::{TypeArgs, execute}; +use crate::locator::live_locator::{ObservationRequest, ObservationRoot, ObservedTree}; +use crate::refs::refs_test_support::HomeGuard; +use crate::{ + ActionRequest, ActionResult, AdapterError, CommandContext, RefEntry, actions::capability, +}; +use std::sync::Mutex; +use std::sync::atomic::{AtomicUsize, Ordering}; + +/// A text field that counts how often its whole window is observed. +#[derive(Default)] +struct CountedField { + typed: Mutex, + window_reads: AtomicUsize, +} + +impl ObservationOps for CountedField { + fn resolve_element_strict( + &self, + _entry: &RefEntry, + _deadline: crate::Deadline, + ) -> Result { + Ok(NativeHandle::null()) + } + + fn observe_tree( + &self, + root: ObservationRoot<'_>, + _request: &ObservationRequest, + ) -> Result { + if matches!(root, ObservationRoot::Window(_)) { + self.window_reads.fetch_add(1, Ordering::SeqCst); + } + crate::adapter::saved_ref_tree(root, "textfield", Some("Target")) + } + + crate::adapter::complete_live_observation!( + "textfield", + "Target", + [capability::TYPE_TEXT], + adapter => crate::ElementState { + role: "textfield".into(), + states: Vec::new(), + value: Some(adapter.typed.lock().unwrap().clone()), + enabled: Some(true), + hidden: Some(false), + offscreen: Some(false), + } + ); + crate::adapter::saved_ref_anchor!(); + crate::adapter::saved_ref_scope!(@inventory); +} + +impl ActionOps for CountedField { + fn execute_action( + &self, + _handle: &NativeHandle, + request: ActionRequest, + _lease: &crate::InteractionLease, + ) -> Result { + if let crate::Action::TypeText(text) = request.action { + self.typed.lock().unwrap().push_str(&text); + } + Ok(ActionResult::delivered_unverified("type_text")) + } +} + +impl InputOps for CountedField {} + +impl SystemOps for CountedField { + crate::adapter::guarded_interaction_lease!(); + crate::adapter::exact_window_focus!(); +} + +/// `type --interval-ms` keeps the element its first keystroke landed on: the +/// whole window is read once for the whole burst, not again to start it. +#[test] +fn a_burst_reads_the_whole_window_once() { + let _home = HomeGuard::new(); + let snapshot_id = super::identity::field_snapshot(); + let adapter = CountedField::default(); + + execute( + TypeArgs { + ref_id: format!("@{snapshot_id}:e1"), + text: "abc".into(), + timeout_ms: None, + interval_ms: Some(5), + }, + &adapter, + &CommandContext::default(), + ) + .expect("typing succeeds"); + + assert_eq!(*adapter.typed.lock().unwrap(), "abc"); + assert_eq!(adapter.window_reads.load(Ordering::SeqCst), 1); +} diff --git a/crates/core/src/commands/type_text_interval_tests.rs b/crates/core/src/commands/type_text_interval_tests.rs index 321858eb0..cd055bea6 100644 --- a/crates/core/src/commands/type_text_interval_tests.rs +++ b/crates/core/src/commands/type_text_interval_tests.rs @@ -53,3 +53,9 @@ mod clock; #[path = "type_text_interval_caret_tests.rs"] mod caret; + +#[path = "type_text_interval_identity_tests.rs"] +mod identity; + +#[path = "type_text_interval_observation_tests.rs"] +mod observation; diff --git a/crates/core/src/commands/type_text_interval_wait_tests.rs b/crates/core/src/commands/type_text_interval_wait_tests.rs index b5de2c245..a2434b88c 100644 --- a/crates/core/src/commands/type_text_interval_wait_tests.rs +++ b/crates/core/src/commands/type_text_interval_wait_tests.rs @@ -27,7 +27,10 @@ impl TitledField { presentation: Default::default(), children_count: None, subtree_truncated: false, - children: vec![], + children: vec![crate::adapter::minimal_accessibility_node( + "textfield", + "Target", + )], } } } @@ -38,6 +41,12 @@ impl ObservationOps for TitledField { root: crate::locator::live_locator::ObservationRoot<'_>, _request: &crate::locator::live_locator::ObservationRequest, ) -> Result { + if matches!( + root, + crate::locator::live_locator::ObservationRoot::Element { .. } + ) { + return crate::adapter::saved_ref_element_tree(root); + } crate::adapter::observed_tree(&root, self.window()) } @@ -90,6 +99,7 @@ impl ObservationOps for TitledField { offscreen: Some(false), } ); + crate::adapter::saved_ref_anchor!(); } impl ActionOps for TitledField { diff --git a/crates/core/src/commands/type_text_tests.rs b/crates/core/src/commands/type_text_tests.rs index 003e33074..ff8766c42 100644 --- a/crates/core/src/commands/type_text_tests.rs +++ b/crates/core/src/commands/type_text_tests.rs @@ -48,6 +48,7 @@ impl ObservationOps for StaleThenOkAdapter { offscreen: Some(false), } ); + crate::adapter::saved_ref_scope!("textfield", Some("Target")); } impl ActionOps for StaleThenOkAdapter { @@ -200,7 +201,7 @@ fn windowed_snapshot_id() -> String { entry.geometry.bounds_hash = bounds.bounds_hash(); entry.capabilities.available_actions = vec!["SetValue".into()]; entry.source.source_app = Some("Fixture".into()); - entry.source.source_window_id = Some("w-42".into()); + entry.source.source_window_id = Some("w-1".into()); entry.source.source_window_title = Some("Fixture".into()); let mut refmap = crate::refs::RefMap::new(); refmap.allocate(entry); @@ -242,8 +243,8 @@ fn a_headed_interval_resolves_the_field_once_then_posts_each_keystroke_to_it() { ); assert_eq!( burst.retry.calls(), - single.retry.calls() + 1, - "only the first keystroke runs the full ref action; the burst resolves once more" + single.retry.calls(), + "only the first keystroke resolves; the burst keeps the element it landed on" ); assert_eq!( *burst.expected_processes.lock().unwrap(), diff --git a/crates/core/src/commands/wait_element.rs b/crates/core/src/commands/wait_element.rs index 3d6dd7f9a..f25a5fe7c 100644 --- a/crates/core/src/commands/wait_element.rs +++ b/crates/core/src/commands/wait_element.rs @@ -32,7 +32,7 @@ pub(crate) fn wait_for_element( let mut last_observed = json!(null); let mut expected_bounds_hash = None; loop { - match resolve_within_deadline(adapter, &entry, deadline) { + match resolve_within_deadline(adapter, &entry, deadline, None) { ResolveAttemptOutcome::DeadlinePassed => { return wait_timeout::element(ref_id, predicate, timeout_ms, last_observed); } diff --git a/crates/core/src/commands/wait_element_tests.rs b/crates/core/src/commands/wait_element_tests.rs index 5f5b57ed8..175007215 100644 --- a/crates/core/src/commands/wait_element_tests.rs +++ b/crates/core/src/commands/wait_element_tests.rs @@ -41,6 +41,7 @@ impl ObservationOps for LiveErrorPredicateAdapter { ) -> Result, AdapterError> { Err(AdapterError::permission_denied()) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for LiveErrorPredicateAdapter {} @@ -108,6 +109,7 @@ impl ObservationOps for LateObservationAdapter { } Ok(Some("ready".into())) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for LateObservationAdapter {} @@ -219,6 +221,7 @@ impl ObservationOps for DeadlineTimeoutPredicateAdapter { self.polls.fetch_add(1, Ordering::SeqCst); Err(deadline.timeout_error()) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for DeadlineTimeoutPredicateAdapter {} diff --git a/crates/core/src/commands/wait_predicate_tests.rs b/crates/core/src/commands/wait_predicate_tests.rs index b823e6d48..2c0c3d706 100644 --- a/crates/core/src/commands/wait_predicate_tests.rs +++ b/crates/core/src/commands/wait_predicate_tests.rs @@ -107,6 +107,7 @@ impl ObservationOps for FlippingPredicateAdapter { ) -> Result { Ok(crate::adapter::hit_test::HitTestResult::ReachesTarget) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for FlippingPredicateAdapter {} diff --git a/crates/core/src/commands/wait_resolution_tests.rs b/crates/core/src/commands/wait_resolution_tests.rs index d1f37a026..5fb6dee6f 100644 --- a/crates/core/src/commands/wait_resolution_tests.rs +++ b/crates/core/src/commands/wait_resolution_tests.rs @@ -24,6 +24,7 @@ impl ObservationOps for AmbiguousResolveAdapter { Err(AdapterError::ambiguous_target("2 candidates matched") .with_details(serde_json::json!({ "retryable": true }))) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for AmbiguousResolveAdapter {} @@ -48,6 +49,7 @@ impl ObservationOps for TransientResolveAdapter { } Ok(NativeHandle::null()) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for TransientResolveAdapter {} @@ -66,6 +68,7 @@ impl ObservationOps for PermissionResolveAdapter { ) -> Result { Err(AdapterError::permission_denied()) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for PermissionResolveAdapter {} @@ -90,6 +93,7 @@ impl ObservationOps for TimeoutCaptureAdapter { .push(deadline.remaining().as_millis()); Ok(NativeHandle::null()) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for TimeoutCaptureAdapter {} @@ -112,6 +116,7 @@ fn snapshot_with_one_ref() -> String { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -128,6 +133,7 @@ fn snapshot_with_one_ref() -> String { source_window_title: None, source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/commands/wait_test_support.rs b/crates/core/src/commands/wait_test_support.rs index cb78bce1a..3c3beb61f 100644 --- a/crates/core/src/commands/wait_test_support.rs +++ b/crates/core/src/commands/wait_test_support.rs @@ -132,6 +132,7 @@ impl ObservationOps for PredicateAdapter { ) -> Result { Ok(crate::adapter::hit_test::HitTestResult::ReachesTarget) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for PredicateAdapter {} @@ -170,6 +171,7 @@ fn save_ref_in_store(store: RefStore, states: Vec) -> String { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -186,6 +188,7 @@ fn save_ref_in_store(store: RefStore, states: Vec) -> String { source_window_title: None, source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/locator/live_locator/evaluate.rs b/crates/core/src/locator/live_locator/evaluate.rs index d7ef3e02f..ca86dbc3b 100644 --- a/crates/core/src/locator/live_locator/evaluate.rs +++ b/crates/core/src/locator/live_locator/evaluate.rs @@ -10,14 +10,27 @@ use super::{ selection_completeness::first_is_authoritative, validate::{validate_query, validate_request}, }; -use crate::{AdapterError, locator::LocatorQuery}; +use crate::{AdapterError, locator::LocatorQuery, refs::RefPath}; use std::collections::BTreeSet; pub fn evaluate_locator_tree( - mut tree: ObservedTree, + tree: ObservedTree, query: &LocatorQuery, request: &LocatorResolveRequest, ) -> Result { + evaluate_locator_tree_in(tree, None, query, request).map(|(resolution, _)| resolution) +} + +/// `evaluate_locator_tree` over one complete observation, with the matches drawn +/// from the subtree at `scope` alone. The observation comes back with the +/// resolution, because a hydrated match is proven against the same read its +/// query was answered from. +pub(super) fn evaluate_locator_tree_in( + mut tree: ObservedTree, + scope: Option<&RefPath>, + query: &LocatorQuery, + request: &LocatorResolveRequest, +) -> Result<(LocatorResolution, ObservedTree), AdapterError> { validate_query(query)?; validate_request(request)?; let normalized = normalize_query(query); @@ -59,6 +72,9 @@ pub fn evaluate_locator_tree( let mut matched_indices = Vec::new(); let mut unknown = false; for index in 0..tree.nodes.len() { + if !in_scope(&tree.nodes[index].path, scope) { + continue; + } match matches[cell(index, root_clause, clauses.len())] { MatchVerdict::Match => matched_indices.push(index), MatchVerdict::Unknown => unknown = true, @@ -83,6 +99,7 @@ pub fn evaluate_locator_tree( Some(refmap) } }; + let proof = crate::refs::ref_capture_proof::CaptureProof::new(&tree); let mut selected_matches = Vec::with_capacity(selected.len()); for index in selected { let data = match_data(&tree, index, &parents) @@ -91,7 +108,8 @@ pub fn evaluate_locator_tree( .nodes .get(index) .ok_or_else(|| AdapterError::internal("selected locator node is out of bounds"))?; - let entry = ref_entry(node, &tree.source, &normalized); + let mut entry = ref_entry(node, &tree.source, &normalized); + entry.identity.unique_at_capture = proof.proves_node(&entry, index); selected_matches.push(LocatorMatch { data, document_order: node.document_order, @@ -106,6 +124,7 @@ pub fn evaluate_locator_tree( let roles_present = if matched_indices.is_empty() { tree.nodes .iter() + .filter(|node| in_scope(&node.path, scope)) .filter_map(|node| match &node.evidence.role { LocatorField::Known(role) => Some(role.clone()), LocatorField::Absent | LocatorField::Unknown => None, @@ -132,18 +151,21 @@ pub fn evaluate_locator_tree( ) }) }; - Ok(LocatorResolution { - matches: selected_matches, - refmap, - stats, - meta: LocatorResolutionMeta { - total_matches, - complete, - selection_complete, - truncated, - roles_present, + Ok(( + LocatorResolution { + matches: selected_matches, + refmap, + stats, + meta: LocatorResolutionMeta { + total_matches, + complete, + selection_complete, + truncated, + roles_present, + }, }, - }) + tree, + )) } fn evaluate_node( @@ -261,3 +283,10 @@ fn aggregate_subtree( fn cell(node: usize, clause: usize, clause_count: usize) -> usize { node * clause_count + clause } + +/// Whether a node sits inside the subtree a rooted query is scoped to. The scope +/// is the matched root's own path in the same observation, so the root itself and +/// its descendants are in and nothing else is. +fn in_scope(path: &RefPath, scope: Option<&RefPath>) -> bool { + scope.is_none_or(|scope| path.starts_with(scope.as_slice())) +} diff --git a/crates/core/src/locator/live_locator/hydrate.rs b/crates/core/src/locator/live_locator/hydrate.rs index f5f7c8a98..c1d88db38 100644 --- a/crates/core/src/locator/live_locator/hydrate.rs +++ b/crates/core/src/locator/live_locator/hydrate.rs @@ -19,13 +19,21 @@ pub(super) fn retryable_error(error: &AdapterError) -> bool { ) } +/// Resolves each selected match to the entry it is published under. +/// +/// The saved entry keeps the identity the whole-window read recorded and proved +/// unique, and the landing is checked against it. Fields hydration read beyond +/// that identity describe the match in the response only: a proof is never +/// carried onto a predicate wider than the one it was measured for. pub(super) fn selected_matches( adapter: &dyn PlatformAdapter, query: &LocatorQuery, request: &super::LocatorResolveRequest, resolution: &mut LocatorResolution, + observed: &ObservedTree, ) -> Result<(), AppError> { let deadline = request.deadline; + let proof = crate::refs::ref_capture_proof::CaptureProof::new(observed); let mut refmap = RefMap::new(); for matched in &mut resolution.matches { ensure_remaining(deadline)?; @@ -33,6 +41,8 @@ pub(super) fn selected_matches( let bounds_space = preliminary.geometry.bounds_space.unwrap_or_default(); let has_identity = crate::refs::ref_identity::has_meaningful_identity(&preliminary); let has_bounds = preliminary.geometry.bounds_hash.is_some(); + let index = matched.document_order as usize; + let proven = proof.proves_node(&preliminary, index); tracing::debug!( role = preliminary.identity.role, path_len = preliminary.scope.path.len(), @@ -40,7 +50,7 @@ pub(super) fn selected_matches( has_bounds, "hydrating selected locator anchor" ); - if !has_identity && !has_bounds { + if !proven && !has_identity && !has_bounds { return Err(anchor_missing(&preliminary).into()); } if matched.fully_observed { @@ -61,6 +71,7 @@ pub(super) fn selected_matches( hydration_root, &ObservationRequest::selected_hydration(query, request, hydration_root, deadline) .validate()?, + None, )?; hydrated.stats.reads.counts.observation_attempts = hydrated.stats.reads.counts.observation_attempts.max(1); @@ -79,22 +90,19 @@ pub(super) fn selected_matches( } if node.evidence.role.known().map(String::as_str) != Some(preliminary.identity.role.as_str()) - || !anchor_matches(&preliminary, node) + || !anchor_matches(&preliminary, node, proven) { return Err(changed_during_hydration().into()); } - let mut entry = super::materialize::ref_entry(node, &hydrated.source, query); - preserve_verified_identity(&preliminary, node, &mut entry); + let entry = RefEntry { + identity: preliminary.identity.clone(), + ..super::materialize::ref_entry(node, &hydrated.source, query) + }; let name = node.evidence.name.meaningful_string().unwrap_or_default(); let value = node.evidence.value.meaningful_string(); let screen_bounds = super::materialize::screen_bounds(&node.evidence); (entry, name, value, node.document_order, screen_bounds) }; - if !crate::refs::ref_identity::has_meaningful_identity(&entry) - && entry.geometry.bounds_hash.is_none() - { - return Err(anchor_missing(&entry).into()); - } let validation = super::evaluate_locator_tree( hydrated, query, @@ -179,7 +187,14 @@ pub(super) fn geometry_complete( } } -fn anchor_matches(preliminary: &RefEntry, node: &ObservedNode) -> bool { +/// Whether the landed element is the one the anchored entry names. +/// +/// An entry the whole-window read proved unique must match the full predicate +/// it was proven with, the same comparison a saved ref resolves under. A proof +/// that records no text identity and no geometry anchors on that predicate +/// alone. Every other entry, and a proven one that also records identity or +/// geometry, keeps those comparisons too. +fn anchor_matches(preliminary: &RefEntry, node: &ObservedNode, proven: bool) -> bool { let geometry = &node.evidence.ref_evidence; let actual_bounds_hash = match preliminary.geometry.bounds_space.unwrap_or_default() { crate::BoundsSpace::Window => &geometry.window_bounds, @@ -187,7 +202,19 @@ fn anchor_matches(preliminary: &RefEntry, node: &ObservedNode) -> bool { } .known() .and_then(crate::Rect::bounds_hash); - if crate::refs::ref_identity::has_meaningful_identity(preliminary) { + let has_identity = crate::refs::ref_identity::has_meaningful_identity(preliminary); + if proven { + let landed = crate::refs::ref_full_predicate::FullPredicate::of_entry(preliminary).compare( + &crate::refs::ref_live_predicate::LivePredicate::observed(&node.evidence), + ); + if landed != IdentityMatch::Match { + return false; + } + if !has_identity && preliminary.geometry.bounds_hash.is_none() { + return true; + } + } + if has_identity { return match crate::refs::ref_identity::identity_match( preliminary, &node.evidence.name, @@ -208,29 +235,6 @@ fn anchor_matches(preliminary: &RefEntry, node: &ObservedNode) -> bool { && preliminary.geometry.bounds_hash == actual_bounds_hash } -fn preserve_verified_identity(preliminary: &RefEntry, node: &ObservedNode, entry: &mut RefEntry) { - if node.evidence.name.is_unknown() { - entry.identity.name.clone_from(&preliminary.identity.name); - } - if node.evidence.description.is_unknown() { - entry - .identity - .description - .clone_from(&preliminary.identity.description); - } - if node.evidence.value.is_unknown() - && !crate::tree::roles::is_mutable_value_role(&entry.identity.role) - { - entry.identity.value.clone_from(&preliminary.identity.value); - } - if !node.evidence.identifiers.is_complete() { - entry - .identity - .native_id - .clone_from(&preliminary.identity.native_id); - } -} - fn anchor_missing(entry: &RefEntry) -> AdapterError { AdapterError::new( ErrorCode::StaleRef, @@ -312,63 +316,5 @@ fn evidence_incomplete( } #[cfg(test)] -mod retained_tests { - use super::*; - use crate::locator::live_locator::test_support::{evidence, node, tree}; - - #[test] - fn selected_hydration_preserves_window_geometry_and_refuses_unread_or_mixed_frames() { - use crate::{BoundsSpace, LocatorField, Rect}; - let bounds = Rect { - x: 10.0, - y: 20.0, - width: 80.0, - height: 30.0, - }; - let query = LocatorQuery::default(); - let mut observed = node(0, evidence("button", None), vec![], &[]); - observed.evidence.ref_evidence.bounds = LocatorField::Unknown; - observed.evidence.ref_evidence.window_bounds = LocatorField::Known(bounds); - let source = tree(vec![observed.clone()], vec![0], true).source; - let mut entry = super::super::materialize::ref_entry(&observed, &source, &query); - assert_eq!(entry.geometry.bounds_space, Some(BoundsSpace::Window)); - assert!(evidence_complete(&query, &observed, BoundsSpace::Window)); - assert!(anchor_matches(&entry, &observed)); - - observed.evidence.ref_evidence.window_bounds = - LocatorField::Known(Rect { x: 110.0, ..bounds }); - observed.evidence.ref_evidence.bounds = LocatorField::Known(bounds); - assert!(!anchor_matches(&entry, &observed)); - observed.evidence.ref_evidence.bounds = LocatorField::Unknown; - for window in [LocatorField::Unknown, LocatorField::Absent] { - observed.evidence.ref_evidence.window_bounds = window; - assert!(!evidence_complete(&query, &observed, BoundsSpace::Window)); - assert!(!anchor_matches(&entry, &observed)); - } - observed.evidence.ref_evidence.window_bounds = LocatorField::Unknown; - observed.evidence.ref_evidence.bounds = LocatorField::Known(bounds); - assert!(!evidence_complete(&query, &observed, BoundsSpace::Window)); - observed.evidence.ref_evidence.window_bounds = LocatorField::Known(bounds); - observed.evidence.ref_evidence.bounds = LocatorField::Unknown; - entry.geometry.bounds_space = Some(BoundsSpace::Screen); - assert!(!evidence_complete(&query, &observed, BoundsSpace::Screen)); - assert!(!anchor_matches(&entry, &observed)); - } - - #[test] - fn missing_retained_identity_cannot_pass_hydration_through_geometry() { - let mut observed = evidence("textfield", None); - observed.identifiers = observed.identifiers.with_retained_object("host:1".into()); - let observed = node(0, observed, vec![], &[]); - let tree = tree(vec![observed.clone()], vec![0], true); - let entry = - super::super::materialize::ref_entry(&observed, &tree.source, &LocatorQuery::default()); - assert!(anchor_matches(&entry, &observed)); - let mut replacement = observed; - replacement.evidence.identifiers = crate::IdentifierEvidence::absent(); - assert!(!anchor_matches(&entry, &replacement)); - replacement.evidence.identifiers = - crate::IdentifierEvidence::absent().with_retained_object("host:2".into()); - assert!(!anchor_matches(&entry, &replacement)); - } -} +#[path = "hydrate_tests.rs"] +mod tests; diff --git a/crates/core/src/locator/live_locator/hydrate_tests.rs b/crates/core/src/locator/live_locator/hydrate_tests.rs new file mode 100644 index 000000000..48bd16bc6 --- /dev/null +++ b/crates/core/src/locator/live_locator/hydrate_tests.rs @@ -0,0 +1,57 @@ +use super::*; +use crate::locator::live_locator::test_support::{evidence, node, tree}; + +#[test] +fn selected_hydration_preserves_window_geometry_and_refuses_unread_or_mixed_frames() { + use crate::{BoundsSpace, LocatorField, Rect}; + let bounds = Rect { + x: 10.0, + y: 20.0, + width: 80.0, + height: 30.0, + }; + let query = LocatorQuery::default(); + let mut observed = node(0, evidence("button", None), vec![], &[]); + observed.evidence.ref_evidence.bounds = LocatorField::Unknown; + observed.evidence.ref_evidence.window_bounds = LocatorField::Known(bounds); + let source = tree(vec![observed.clone()], vec![0], true).source; + let mut entry = super::super::materialize::ref_entry(&observed, &source, &query); + assert_eq!(entry.geometry.bounds_space, Some(BoundsSpace::Window)); + assert!(evidence_complete(&query, &observed, BoundsSpace::Window)); + assert!(anchor_matches(&entry, &observed, false)); + + observed.evidence.ref_evidence.window_bounds = LocatorField::Known(Rect { x: 110.0, ..bounds }); + observed.evidence.ref_evidence.bounds = LocatorField::Known(bounds); + assert!(!anchor_matches(&entry, &observed, false)); + observed.evidence.ref_evidence.bounds = LocatorField::Unknown; + for window in [LocatorField::Unknown, LocatorField::Absent] { + observed.evidence.ref_evidence.window_bounds = window; + assert!(!evidence_complete(&query, &observed, BoundsSpace::Window)); + assert!(!anchor_matches(&entry, &observed, false)); + } + observed.evidence.ref_evidence.window_bounds = LocatorField::Unknown; + observed.evidence.ref_evidence.bounds = LocatorField::Known(bounds); + assert!(!evidence_complete(&query, &observed, BoundsSpace::Window)); + observed.evidence.ref_evidence.window_bounds = LocatorField::Known(bounds); + observed.evidence.ref_evidence.bounds = LocatorField::Unknown; + entry.geometry.bounds_space = Some(BoundsSpace::Screen); + assert!(!evidence_complete(&query, &observed, BoundsSpace::Screen)); + assert!(!anchor_matches(&entry, &observed, false)); +} + +#[test] +fn a_positional_retained_object_is_not_hydration_identity() { + let mut observed = evidence("textfield", None); + observed.identifiers = observed.identifiers.with_retained_object("host:1".into()); + let observed = node(0, observed, vec![], &[]); + let tree = tree(vec![observed.clone()], vec![0], true); + let entry = + super::super::materialize::ref_entry(&observed, &tree.source, &LocatorQuery::default()); + assert!(!anchor_matches(&entry, &observed, false)); + let mut replacement = observed; + replacement.evidence.identifiers = crate::IdentifierEvidence::absent(); + assert!(!anchor_matches(&entry, &replacement, false)); + replacement.evidence.identifiers = + crate::IdentifierEvidence::absent().with_retained_object("host:2".into()); + assert!(!anchor_matches(&entry, &replacement, false)); +} diff --git a/crates/core/src/locator/live_locator/materialize.rs b/crates/core/src/locator/live_locator/materialize.rs index 9be165a4e..9052f74b4 100644 --- a/crates/core/src/locator/live_locator/materialize.rs +++ b/crates/core/src/locator/live_locator/materialize.rs @@ -11,25 +11,31 @@ pub(crate) fn materialize_refmap( query: &LocatorQuery, selected: Option<&[usize]>, ) -> Result<(RefMap, bool), crate::AdapterError> { - let source = &tree.source; - let nodes = &mut tree.nodes; let mut refmap = RefMap::new(); let mut complete = true; - let indices = selected - .map(|indices| indices.to_vec()) - .unwrap_or_else(|| (0..nodes.len()).collect()); - for index in indices { - let node = &nodes[index]; - let (addressable, known) = addressability(&node.evidence); - complete &= known; - if !addressable { - continue; + let indices: Vec = selected + .map(<[usize]>::to_vec) + .unwrap_or_else(|| (0..tree.nodes.len()).collect()); + let mut planned = Vec::with_capacity(indices.len()); + { + let proof = crate::refs::ref_capture_proof::CaptureProof::new(tree); + for index in indices { + let node = &tree.nodes[index]; + let (addressable, known) = addressability(&node.evidence); + complete &= known; + if !addressable { + continue; + } + let mut entry = ref_entry(node, &tree.source, query); + entry.identity.unique_at_capture = proof.proves_node(&entry, index); + planned.push((index, entry)); } - let entry = ref_entry(node, source, query); + } + for (index, entry) in planned { let ref_id = refmap.try_allocate(entry).map_err(|error| { crate::AdapterError::new(crate::ErrorCode::InvalidArgs, error.to_string()) })?; - nodes[index].ref_id = Some(ref_id); + tree.nodes[index].ref_id = Some(ref_id); } Ok((refmap, complete)) } @@ -45,6 +51,23 @@ pub(crate) fn addressability(evidence: &LocatorEvidence) -> (bool, bool) { } } +/// Whether publishing refs from this observation could publish one that names +/// its node without an element token, which only a window-wide capture proof +/// can vouch for. An unread token counts as absent, and an unsettled +/// addressability counts as addressable: hydration can read actions that make +/// such a node publishable. +pub(crate) fn selects_tokenless_entry(tree: &ObservedTree) -> bool { + tree.nodes.iter().any(|node| { + let (addressable, settled) = addressability(&node.evidence); + (addressable || !settled) + && !node + .evidence + .identifiers + .retained_object() + .is_some_and(crate::refs::ref_identity::is_element_token) + }) +} + /// Whether every field a ref entry and a match response carry was read, none /// of them unknown. An observation of that kind leaves nothing for a second /// read of the same node to add. @@ -93,6 +116,7 @@ pub(crate) fn ref_entry( value: node.evidence.value.meaningful_string(), description: node.evidence.description.meaningful_string(), native_id: identifier, + unique_at_capture: false, }, geometry: crate::RefGeometry { bounds, @@ -109,6 +133,7 @@ pub(crate) fn ref_entry( source_window_title: None, source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -122,7 +147,11 @@ pub(crate) fn ref_entry( fn apply_source(entry: &mut RefEntry, source: &ObservationSource, node_path: &RefPath) { match source { - ObservationSource::Window { window, surface } => { + ObservationSource::Window { + window, + surface, + renderer_accessibility_forced, + } => { entry.process.pid = window.pid; entry.source.source_app = Some(window.app.clone()); entry.source.source_window_id = Some(window.id.clone()); @@ -130,6 +159,7 @@ fn apply_source(entry: &mut RefEntry, source: &ObservationSource, node_path: &Re entry.source.source_window_bounds_hash = window.bounds.as_ref().and_then(crate::Rect::bounds_hash); entry.source.source_surface = *surface; + entry.source.renderer_accessibility_forced = *renderer_accessibility_forced; entry.process.process_instance = window.process_instance.clone(); } ObservationSource::Element { diff --git a/crates/core/src/locator/live_locator/materialize_tests.rs b/crates/core/src/locator/live_locator/materialize_tests.rs index baee8b982..ffcc6cfb7 100644 --- a/crates/core/src/locator/live_locator/materialize_tests.rs +++ b/crates/core/src/locator/live_locator/materialize_tests.rs @@ -313,6 +313,7 @@ fn source_entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -329,6 +330,7 @@ fn source_entry() -> RefEntry { source_window_title: Some("Fixture".into()), source_window_bounds_hash: None, source_surface: SnapshotSurface::Menu, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/locator/live_locator/mod.rs b/crates/core/src/locator/live_locator/mod.rs index 7530489fb..0bfda432b 100644 --- a/crates/core/src/locator/live_locator/mod.rs +++ b/crates/core/src/locator/live_locator/mod.rs @@ -37,6 +37,7 @@ mod observation_request; mod observation_root; mod observation_source; mod observed_node; +mod observed_observation; mod observed_subtree; mod observed_tree; mod predicate; @@ -71,16 +72,20 @@ pub use locator_selection::LocatorSelection; pub use locator_semantic_read_stats::LocatorSemanticReadStats; pub use locator_stats::LocatorStats; pub use locator_traversal_stats::LocatorTraversalStats; +pub(crate) use materialize::ref_entry as materialize_ref_entry; +pub(crate) use materialize::selects_tokenless_entry; pub use observation_budget::ObservationBudget; pub(crate) use observation_completeness::ObservationCompleteness; pub use observation_request::{ObservationMode, ObservationRequest}; pub use observation_root::ObservationRoot; pub use observation_source::ObservationSource; pub use observed_node::ObservedNode; +pub(crate) use observed_observation::ObservedObservation; pub use observed_subtree::ObservedSubtree; pub use observed_tree::ObservedTree; pub use ref_evidence_requirements::RefEvidenceRequirements; pub use resolve::resolve_query; +pub(crate) use resolve::resolve_query_from; pub use validate::{validate_query, validate_request}; #[cfg(test)] diff --git a/crates/core/src/locator/live_locator/observation_request.rs b/crates/core/src/locator/live_locator/observation_request.rs index af67d0296..53bf42523 100644 --- a/crates/core/src/locator/live_locator/observation_request.rs +++ b/crates/core/src/locator/live_locator/observation_request.rs @@ -33,6 +33,9 @@ pub struct ObservationMode { /// (via the `--force-electron-a11y` CLI flag), so the adapter should not /// guess at guidance; it returns the tree it observed. pub force_renderer_accessibility: bool, + /// The read only resolves a saved ref: it needs identity evidence and the + /// child structure, never geometry or actions, so an adapter may skip them. + pub identity_only: bool, } impl ObservationRequest { @@ -71,21 +74,28 @@ impl ObservationRequest { Self { deadline, max_raw_depth: 50, - max_logical_depth: if options.skeleton { - options.max_depth.min(3) - } else { - options.max_depth - }, + max_logical_depth: Self::snapshot_logical_depth(options), surface: options.surface, evidence_plan: EvidencePlan::uniform(EvidenceRequirements::snapshot()), budget: ObservationBudget::default(), observation_mode: ObservationMode { skeleton: options.skeleton, force_renderer_accessibility: options.force_renderer_accessibility, + identity_only: false, }, } } + /// The logical depth a snapshot request presents for these options: a + /// skeleton is an overview, so it never presents more than three levels. + pub fn snapshot_logical_depth(options: &TreeOptions) -> u8 { + if options.skeleton { + options.max_depth.min(3) + } else { + options.max_depth + } + } + pub fn locator( query: &LocatorQuery, request: &LocatorResolveRequest, diff --git a/crates/core/src/locator/live_locator/observation_source.rs b/crates/core/src/locator/live_locator/observation_source.rs index 858a6180c..a4ad8fcda 100644 --- a/crates/core/src/locator/live_locator/observation_source.rs +++ b/crates/core/src/locator/live_locator/observation_source.rs @@ -10,6 +10,9 @@ pub enum ObservationSource { /// must record that, or re-resolving it later searches the window and /// reports the element missing. surface: crate::SnapshotSurface, + /// Whether the window was read with Chromium renderer accessibility + /// forced; a ref materialized from it records that for resolution. + renderer_accessibility_forced: bool, }, Element { entry: Box, @@ -23,6 +26,7 @@ impl ObservationSource { ObservationRoot::Window(window) => Self::Window { window: (*window).clone(), surface, + renderer_accessibility_forced: false, }, ObservationRoot::Element { entry, root_ref, .. diff --git a/crates/core/src/locator/live_locator/observed_node.rs b/crates/core/src/locator/live_locator/observed_node.rs index fceb65dd0..cc090c060 100644 --- a/crates/core/src/locator/live_locator/observed_node.rs +++ b/crates/core/src/locator/live_locator/observed_node.rs @@ -12,3 +12,12 @@ pub struct ObservedNode { pub(crate) children_count: Option, pub(crate) ref_id: Option, } + +impl ObservedNode { + /// A container whose descendants were not read cannot support any verdict + /// about the elements it might hide. + pub(crate) fn omits_descendants(&self) -> bool { + !self.completeness.subtree_complete + || (self.children_count.is_some_and(|count| count > 0) && self.children.is_empty()) + } +} diff --git a/crates/core/src/locator/live_locator/observed_observation.rs b/crates/core/src/locator/live_locator/observed_observation.rs new file mode 100644 index 000000000..4fe92c0e9 --- /dev/null +++ b/crates/core/src/locator/live_locator/observed_observation.rs @@ -0,0 +1,36 @@ +use crate::{AdapterError, AppError, refs::RefPath}; + +use super::ObservedTree; + +/// A tree the caller already observed, and the node a query is scoped to. +/// +/// A rooted find arrives here with the whole-window read that re-identified its +/// root plus that root's path, so its matches are drawn from the root's subtree +/// while the capture proof behind their refs runs over every node of the same +/// observation. Nothing is proven across two reads. +pub(crate) struct ObservedObservation { + pub(crate) tree: ObservedTree, + pub(crate) scope: Option, +} + +impl ObservedObservation { + pub(crate) fn whole(tree: ObservedTree) -> Self { + Self { tree, scope: None } + } + + pub(crate) fn under(tree: ObservedTree, root: usize) -> Result { + let mut observed = Self::whole(tree); + let scope = observed + .tree + .nodes + .get(root) + .map(|node| node.path.clone()) + .ok_or_else(|| { + AppError::Adapter(AdapterError::internal( + "rooted observation is missing its root node", + )) + })?; + observed.scope = Some(scope); + Ok(observed) + } +} diff --git a/crates/core/src/locator/live_locator/observed_subtree.rs b/crates/core/src/locator/live_locator/observed_subtree.rs index c2e1442b7..6cac0e5c4 100644 --- a/crates/core/src/locator/live_locator/observed_subtree.rs +++ b/crates/core/src/locator/live_locator/observed_subtree.rs @@ -6,6 +6,7 @@ pub struct ObservedSubtree { pub(crate) completeness: ObservationCompleteness, pub(crate) children_count: Option, pub(crate) source_child_index: Option, + pub(crate) wrapper: bool, } impl ObservedSubtree { @@ -21,6 +22,7 @@ impl ObservedSubtree { completeness: ObservationCompleteness::new(subtree_complete), children_count, source_child_index: None, + wrapper: false, } } @@ -35,6 +37,13 @@ impl ObservedSubtree { self } + /// Marks a layout wrapper: the platform's walks charge it raw depth and no + /// logical depth, so a shallow presentation keeps what a walk would. + pub fn with_wrapper(mut self, wrapper: bool) -> Self { + self.wrapper = wrapper; + self + } + pub fn is_complete(&self) -> bool { self.completeness.subtree_complete } diff --git a/crates/core/src/locator/live_locator/observed_tree.rs b/crates/core/src/locator/live_locator/observed_tree.rs index 3ee874475..ec2b4a4c3 100644 --- a/crates/core/src/locator/live_locator/observed_tree.rs +++ b/crates/core/src/locator/live_locator/observed_tree.rs @@ -11,6 +11,7 @@ pub struct ObservedTree { pub(crate) source: ObservationSource, pub(crate) stats: LocatorStats, pub(crate) structurally_complete: bool, + pub(crate) logical_depths: Vec, } impl ObservedTree { @@ -29,9 +30,10 @@ impl ObservedTree { source, stats, structurally_complete, + logical_depths: Vec::new(), }; for root in roots { - let index = tree.append(root, RefPath::new())?; + let index = tree.append(root, RefPath::new(), 0)?; tree.roots.push(index); } tree.structurally_complete &= tree @@ -45,6 +47,14 @@ impl ObservedTree { self.structurally_complete } + /// Whether this observation read the whole window: complete, and with no + /// container whose descendants it left unread. A capture proof and a + /// tokenless resolution both rest on it, because both count evidence over + /// every node the window holds. + pub(crate) fn is_exhaustive(&self) -> bool { + self.structurally_complete && !self.nodes.iter().any(ObservedNode::omits_descendants) + } + pub fn into_accessibility_tree(mut self) -> Result { if !self.structurally_complete { return Err(AdapterError::new( @@ -74,6 +84,15 @@ impl ObservedTree { /// honest about its own boundaries. pub fn into_accessibility_tree_partial( mut self, + ) -> Result<(AccessibilityNode, bool, usize), AdapterError> { + self.project_partial() + } + + /// `into_accessibility_tree_partial` for a caller that still needs the + /// observation it projected, such as the capture proof behind every ref the + /// presentation allocates. + pub(crate) fn project_partial( + &mut self, ) -> Result<(AccessibilityNode, bool, usize), AdapterError> { if self.roots.len() != 1 { return Err(AdapterError::internal( @@ -86,7 +105,48 @@ impl ObservedTree { Ok((tree, complete, nodes_observed)) } - fn append(&mut self, subtree: ObservedSubtree, path: RefPath) -> Result { + /// Projects the window cut at `max_logical_depth` the way a walk applies + /// its own limit: a container whose children would sit below the limit + /// keeps none, carries their count and stays complete, so it still anchors + /// a skeleton. Every presented node is the node this observation holds, so a + /// capture proof taken over the whole observation describes it exactly. + pub(crate) fn project_window( + &mut self, + max_logical_depth: u8, + ) -> Result<(AccessibilityNode, bool, usize), AdapterError> { + if self.roots.len() != 1 { + return Err(AdapterError::internal( + "accessibility projection requires exactly one root", + )); + } + let complete = self.structurally_complete; + let nodes_observed = self.nodes.len(); + let depths = std::mem::take(&mut self.logical_depths); + let tree = self.project_shallow(self.roots[0] as usize, &depths, max_logical_depth)?; + Ok((tree, complete, nodes_observed)) + } + + /// Projects the subtree rooted at an observed node, cut the same way. + /// + /// The matched node is the subtree's depth zero, as it is for a walk that + /// starts at an element, so the requested limit counts from there rather + /// than from the window the node happened to sit deep inside. + pub(crate) fn project_subtree( + &mut self, + index: usize, + max_logical_depth: u8, + ) -> Result { + let depths = std::mem::take(&mut self.logical_depths); + let root_depth = depths.get(index).copied().unwrap_or(0); + self.project_shallow(index, &depths, root_depth.saturating_add(max_logical_depth)) + } + + fn append( + &mut self, + subtree: ObservedSubtree, + path: RefPath, + logical_depth: u8, + ) -> Result { let index = u32::try_from(self.nodes.len()) .map_err(|_| AdapterError::internal("observed tree exceeds u32"))?; let document_order = index; @@ -96,6 +156,7 @@ impl ObservedTree { completeness, children_count, source_child_index: _, + wrapper, } = subtree; self.nodes.push(ObservedNode { evidence, @@ -106,6 +167,8 @@ impl ObservedTree { children_count, ref_id: None, }); + self.logical_depths.push(logical_depth); + let child_logical_depth = logical_depth.saturating_add(u8::from(!wrapper)); let mut child_indices = Vec::with_capacity(children.len()); let mut complete = completeness.subtree_complete; let mut previous_native_index = None; @@ -119,7 +182,7 @@ impl ObservedTree { previous_native_index = Some(native_index); let mut child_path = path.clone(); child_path.push(native_index); - let child_index = self.append(child, child_path)?; + let child_index = self.append(child, child_path, child_logical_depth)?; complete &= self.nodes[child_index as usize] .completeness .subtree_complete; @@ -135,6 +198,53 @@ impl ObservedTree { } fn project(&mut self, index: usize) -> Result { + let (mut projected, child_indices) = self.projected_node(index)?; + for child in child_indices { + projected.children.push(self.project(child as usize)?); + } + Ok(projected) + } + + /// Projects the node at `index`, cutting every descendant whose logical + /// depth in the whole observation reaches past `max_logical_depth`; a cut + /// node keeps its `children_count` and stays complete. + fn project_shallow( + &mut self, + index: usize, + depths: &[u8], + max_logical_depth: u8, + ) -> Result { + let (mut projected, child_indices) = self.projected_node(index)?; + let below_limit = child_indices + .first() + .and_then(|child| depths.get(*child as usize)) + .is_some_and(|depth| *depth > max_logical_depth); + if below_limit { + projected.children_count = + projected + .children_count + .or(u32::try_from(child_indices.len()) + .ok() + .filter(|count| *count > 0)); + return Ok(projected); + } + for child in child_indices { + projected.children.push(self.project_shallow( + child as usize, + depths, + max_logical_depth, + )?); + } + Ok(projected) + } + + /// The identity evidence is cloned rather than moved, so a capture proof that + /// runs after the presentation was cut still reads the observation it proves + /// each published entry against. + fn projected_node( + &mut self, + index: usize, + ) -> Result<(AccessibilityNode, Vec), AdapterError> { let node = self .nodes .get_mut(index) @@ -142,10 +252,11 @@ impl ObservedTree { let role = node .evidence .role - .take_known() + .known() + .cloned() .unwrap_or_else(|| "unknown".into()); - let child_indices = std::mem::take(&mut node.children); - let mut projected = AccessibilityNode { + let child_indices = node.children.clone(); + let projected = AccessibilityNode { ref_id: node.ref_id.take(), role, identity: crate::NodeIdentity { @@ -154,21 +265,9 @@ impl ObservedTree { .identifiers .retained_object() .map(str::to_owned), - name: node - .evidence - .name - .take_known() - .filter(|value| !value.is_empty()), - value: node - .evidence - .value - .take_known() - .filter(|value| !value.is_empty()), - description: node - .evidence - .description - .take_known() - .filter(|value| !value.is_empty()), + name: node.evidence.name.meaningful_string(), + value: node.evidence.value.meaningful_string(), + description: node.evidence.description.meaningful_string(), native_id: node.evidence.identifiers.preferred_identifier().cloned(), }, presentation: crate::NodePresentation { @@ -188,9 +287,6 @@ impl ObservedTree { subtree_truncated: !node.completeness.subtree_complete, children: Vec::with_capacity(child_indices.len()), }; - for child in child_indices { - projected.children.push(self.project(child as usize)?); - } - Ok(projected) + Ok((projected, child_indices)) } } diff --git a/crates/core/src/locator/live_locator/observed_tree_tests.rs b/crates/core/src/locator/live_locator/observed_tree_tests.rs index 3a69a61c0..8f654e2ad 100644 --- a/crates/core/src/locator/live_locator/observed_tree_tests.rs +++ b/crates/core/src/locator/live_locator/observed_tree_tests.rs @@ -12,6 +12,7 @@ use super::test_support::evidence; fn source() -> ObservationSource { ObservationSource::Window { surface: crate::SnapshotSurface::Window, + renderer_accessibility_forced: false, window: WindowInfo { id: "w-1".into(), title: "Fixture".into(), @@ -129,6 +130,85 @@ fn core_builder_owns_preorder_paths_and_child_indices() { assert_eq!(tree.nodes[2].path.as_slice(), &[1]); } +#[test] +fn a_shallow_presentation_cuts_at_the_logical_depth_of_the_walks() { + let wrapper = subtree( + "group", + "", + vec![subtree( + "group", + "Content", + vec![subtree("button", "Save", Vec::new())], + )], + ) + .with_wrapper(true); + let mut tree = ObservedTree::from_roots( + vec![subtree( + "window", + "Fixture", + vec![subtree("group", "Sidebar", vec![wrapper])], + )], + source(), + Default::default(), + true, + ) + .unwrap(); + + let (shallow, complete, nodes_observed) = tree.clone().project_window(3).unwrap(); + assert!(complete); + assert_eq!( + nodes_observed, 5, + "the projection reports the whole observation" + ); + let button = &shallow.children[0].children[0].children[0].children[0]; + assert_eq!( + (button.role.as_str(), button.identity.name.as_deref()), + ("button", Some("Save")), + "a wrapper costs no logical depth, so the walk reaches past its path depth" + ); + + let (cut, _, _) = tree.project_window(2).unwrap(); + let content = &cut.children[0].children[0].children[0]; + assert_eq!(content.identity.name.as_deref(), Some("Content")); + assert_eq!(content.children_count, Some(1), "a cut keeps its count"); + assert!(content.children.is_empty()); + assert!(!content.subtree_truncated); +} + +#[test] +fn a_drilled_subtree_counts_its_depth_from_the_matched_node() { + let mut node = subtree("button", "Save", Vec::new()); + for (role, name) in [ + ("group", "C"), + ("group", "B"), + ("group", "A"), + ("window", "Fixture"), + ] { + node = subtree(role, name, vec![node]); + } + let mut tree = + ObservedTree::from_roots(vec![node], source(), Default::default(), true).unwrap(); + + let drilled = tree.clone().project_subtree(3, 1).unwrap(); + assert_eq!( + drilled.identity.name.as_deref(), + Some("C"), + "the drilled node is the subtree root" + ); + assert_eq!( + drilled + .children + .first() + .and_then(|child| child.identity.name.as_deref()), + Some("Save"), + "a limit of one reaches one level below a node at window depth 3" + ); + + let cut = tree.project_subtree(3, 0).unwrap(); + assert_eq!(cut.children_count, Some(1), "a cut keeps its count"); + assert!(cut.children.is_empty()); +} + #[test] fn source_child_indices_survive_omitted_siblings() { let target = subtree("scrollarea", "Nested", Vec::new()).with_source_child_index(0); @@ -260,7 +340,7 @@ fn snapshot_projection_and_find_share_the_same_observation() { fn partial_projection_returns_the_observed_subtree_instead_of_discarding_it() { let incomplete = ObservedSubtree::new(evidence("button", Some("Save")), Vec::new(), false, None); - let tree = ObservedTree::from_roots( + let mut tree = ObservedTree::from_roots( vec![subtree("window", "Fixture", vec![incomplete])], source(), Default::default(), @@ -270,7 +350,7 @@ fn partial_projection_returns_the_observed_subtree_instead_of_discarding_it() { assert!(tree.clone().into_accessibility_tree().is_err()); - let (projected, complete, _) = tree.into_accessibility_tree_partial().unwrap(); + let (projected, complete, _) = tree.project_partial().unwrap(); assert!(!complete); assert_eq!(projected.role, "window"); @@ -289,7 +369,7 @@ fn partial_projection_returns_the_observed_subtree_instead_of_discarding_it() { #[test] fn partial_projection_reports_completeness_when_the_walk_finished() { - let tree = ObservedTree::from_roots( + let mut tree = ObservedTree::from_roots( vec![subtree( "window", "Fixture", @@ -302,7 +382,7 @@ fn partial_projection_reports_completeness_when_the_walk_finished() { .unwrap(); let strict = tree.clone().into_accessibility_tree().unwrap(); - let (projected, complete, nodes_observed) = tree.into_accessibility_tree_partial().unwrap(); + let (projected, complete, nodes_observed) = tree.project_partial().unwrap(); assert!(complete); assert_eq!(nodes_observed, 2); diff --git a/crates/core/src/locator/live_locator/resolve.rs b/crates/core/src/locator/live_locator/resolve.rs index a214d5e49..46ac51023 100644 --- a/crates/core/src/locator/live_locator/resolve.rs +++ b/crates/core/src/locator/live_locator/resolve.rs @@ -1,6 +1,6 @@ use super::{ LocatorResolution, LocatorResolveRequest, LocatorStats, ObservationRequest, ObservationRoot, - evaluate_locator_tree, validate_query, validate_request, + ObservedObservation, validate_query, validate_request, }; use crate::{AdapterError, AppError, ErrorCode, adapter::PlatformAdapter, locator::LocatorQuery}; use serde_json::json; @@ -11,6 +11,23 @@ pub fn resolve_query( query: &LocatorQuery, root: ObservationRoot<'_>, request: &LocatorResolveRequest, +) -> Result { + resolve_query_from(adapter, query, request, |deadline| { + let observation_request = + ObservationRequest::locator_for_root(query, request, root, deadline).validate()?; + crate::tree::renderer_accessibility::observe_tree(adapter, root, &observation_request, None) + .map(ObservedObservation::whole) + }) +} + +/// Resolves a query over trees the caller observes. A caller that already holds +/// the read its root resolution took hands that same tree back here, so the refs +/// it publishes are proven by one observation rather than by a second one. +pub(crate) fn resolve_query_from( + adapter: &dyn PlatformAdapter, + query: &LocatorQuery, + request: &LocatorResolveRequest, + mut observe: impl FnMut(crate::Deadline) -> Result, ) -> Result { validate_query(query)?; validate_request(request)?; @@ -24,8 +41,9 @@ pub fn resolve_query( if remaining.is_zero() { return Err(transient_incomplete_timeout(deadline, &aggregate, last_incomplete).into()); } - let attempt_request = LocatorResolveRequest { ..*request }; - match resolve_query_attempt(adapter, query, root, &attempt_request, &mut aggregate) { + match observe(observation_deadline(request)).and_then(|observed| { + resolve_observed_attempt(adapter, query, observed, request, &mut aggregate) + }) { Ok(mut resolution) => { resolution.stats.reads.counts.observation_attempts = resolution.stats.reads.counts.observation_attempts.max(1); @@ -161,19 +179,14 @@ fn transient_incomplete_timeout( .with_disposition(crate::DeliverySemantics::not_delivered()) } -fn resolve_query_attempt( +fn resolve_observed_attempt( adapter: &dyn PlatformAdapter, query: &LocatorQuery, - root: ObservationRoot<'_>, + observed: ObservedObservation, request: &LocatorResolveRequest, aggregate: &mut LocatorStats, ) -> Result { - let observation_request = - ObservationRequest::locator_for_root(query, request, root, observation_deadline(request)) - .validate()?; - let tree = - crate::tree::renderer_accessibility::observe_tree(adapter, root, &observation_request)?; - let mut tree = tree; + let ObservedObservation { mut tree, scope } = observed; tree.stats.reads.counts.observation_attempts = tree.stats.reads.counts.observation_attempts.max(1); let evaluation_request = LocatorResolveRequest { @@ -183,10 +196,15 @@ fn resolve_query_attempt( }, ..*request }; - let mut resolution = evaluate_locator_tree(tree, query, &evaluation_request)?; + let (mut resolution, tree) = super::evaluate::evaluate_locator_tree_in( + tree, + scope.as_ref(), + query, + &evaluation_request, + )?; if request.materialization == super::LocatorMaterialization::SelectedMatches { if let Err(error) = - super::hydrate::selected_matches(adapter, query, request, &mut resolution) + super::hydrate::selected_matches(adapter, query, request, &mut resolution, &tree) { aggregate.merge_attempt(&resolution.stats); return Err(error); diff --git a/crates/core/src/locator/live_locator/retained_identity_tests.rs b/crates/core/src/locator/live_locator/retained_identity_tests.rs index 1418047ab..205f819e1 100644 --- a/crates/core/src/locator/live_locator/retained_identity_tests.rs +++ b/crates/core/src/locator/live_locator/retained_identity_tests.rs @@ -40,10 +40,29 @@ fn matches(entry: &RefEntry, token: Option<&str>, value: &str) -> IdentityMatch } #[test] -fn retained_identity_rejects_replacements_without_freezing_control_values() { +fn a_positional_retained_object_never_decides_identity() { let entry = saved(); + for token in [ + Some("host-generation:object-1"), + Some("host-generation:object-2"), + None, + ] { + assert_eq!( + matches(&entry, token, "Alpha"), + IdentityMatch::Match, + "a positional retained object must not decide identity: {token:?}" + ); + } + assert!(crate::refs::ref_identity::has_meaningful_identity(&entry)); +} + +#[test] +fn an_element_token_decides_identity_on_its_own() { + let mut entry = saved(); + entry.identity.retained_object = Some("axtok1:7f3a".into()); + assert_eq!( - matches(&entry, Some("host-generation:object-1"), "Edited"), + matches(&entry, Some("axtok1:7f3a"), "Edited"), IdentityMatch::Match ); assert_eq!( @@ -86,9 +105,10 @@ fn snapshot_and_find_preserve_identical_private_identity_evidence() { } #[test] -fn empty_retained_token_cannot_downgrade_to_native_id_matching() { +fn an_empty_retained_object_is_not_an_element_token() { let mut entry = saved(); entry.identity.retained_object = Some(String::new()); - assert_eq!(matches(&entry, Some(""), "Alpha"), IdentityMatch::Unknown); - assert_eq!(matches(&entry, None, "Alpha"), IdentityMatch::Unknown); + assert!(crate::refs::ref_identity::element_token(&entry).is_none()); + assert_eq!(matches(&entry, Some(""), "Alpha"), IdentityMatch::Match); + assert_eq!(matches(&entry, None, "Alpha"), IdentityMatch::Match); } diff --git a/crates/core/src/locator/live_locator/selected_hydration_contract_tests.rs b/crates/core/src/locator/live_locator/selected_hydration_contract_tests.rs index 4f2529e0a..662d67bbf 100644 --- a/crates/core/src/locator/live_locator/selected_hydration_contract_tests.rs +++ b/crates/core/src/locator/live_locator/selected_hydration_contract_tests.rs @@ -10,12 +10,15 @@ use crate::{ }; use std::sync::atomic::{AtomicUsize, Ordering}; -#[derive(Clone, Copy)] +#[derive(Clone, Copy, PartialEq)] enum Mode { OptionalIdentityUnknown, MissingAnchor, + UnprovenMissingAnchor, + ProvenIdentity, RequiredNameUnknown, RequiredDescriptionUnknown, + HydrationAddsName, } struct HydrationContractAdapter { @@ -54,11 +57,20 @@ impl ObservationOps for HydrationContractAdapter { ) -> Result { self.strict_resolutions.fetch_add(1, Ordering::SeqCst); assert!( - crate::refs::ref_identity::has_meaningful_identity(entry) + entry.identity.unique_at_capture + || crate::refs::ref_identity::has_meaningful_identity(entry) || entry.geometry.bounds_hash.is_some() ); Ok(NativeHandle::null()) } + + fn list_windows( + &self, + _filter: &crate::adapter::WindowFilter, + _deadline: crate::Deadline, + ) -> Result, AdapterError> { + Ok(vec![window()]) + } } impl ActionOps for HydrationContractAdapter {} @@ -92,10 +104,104 @@ fn role_only_geometry_anchor_accepts_unknown_optional_identity_once() { assert_eq!(entry.geometry.bounds, Some(bounds())); } +/// An entry the whole-window read proved unique needs no text identity and no +/// geometry of its own: its full predicate is the anchor the landing read is +/// checked against, and the published ref carries that proof so it can resolve +/// again. The legacy prerequisites would have refused it before hydrating. #[test] -fn selected_path_without_identity_or_geometry_rejects_without_retrying() { +fn a_capture_proven_path_only_anchor_is_admitted_and_published_proven() { let adapter = adapter(Mode::MissingAnchor); + let resolution = resolve_query( + &adapter, + &role_query(), + ObservationRoot::Window(&window()), + &selected_request(), + ) + .expect("a role-only element counted once by the whole-window read is addressable"); + + assert_eq!(adapter.window_observations.load(Ordering::SeqCst), 1); + assert_eq!(adapter.strict_resolutions.load(Ordering::SeqCst), 1); + assert_eq!(adapter.hydration_observations.load(Ordering::SeqCst), 1); + let entry = resolution.refmap.unwrap().get("@e1").unwrap().clone(); + assert_eq!(entry.identity.name, None); + assert_eq!(entry.geometry.bounds, Some(bounds())); + assert!(entry.identity.unique_at_capture); +} + +/// Hydration re-materializes the entry from a second read, so it must carry the +/// proof the whole-window read already gave the match: an entry whose identity +/// the window answered is published unique, and the published ref resolves +/// again. Losing that proof would hand the caller a ref that fails on its next +/// use, even though the element never left the window. +#[test] +fn a_hydrated_entry_keeps_the_capture_proof_and_its_ref_resolves() { + let adapter = adapter(Mode::ProvenIdentity); + + let resolution = resolve_query( + &adapter, + &role_query(), + ObservationRoot::Window(&window()), + &selected_request(), + ) + .expect("the identity the window read answered survives hydration"); + assert_eq!(adapter.hydration_observations.load(Ordering::SeqCst), 1); + + let entry = resolution.refmap.unwrap().get("@e1").unwrap().clone(); + assert_eq!(entry.identity.name.as_deref(), Some("Save")); + assert!( + entry.identity.unique_at_capture, + "hydration must keep the proof the whole-window read gave the match" + ); + + let anchored = adapter.strict_resolutions.load(Ordering::SeqCst); + crate::refs::resolve_saved_ref::resolve_saved_ref( + &adapter, + &entry, + crate::Deadline::standard().unwrap(), + None, + ) + .expect("the published ref re-identifies the element it was proven from"); + assert_eq!( + adapter.strict_resolutions.load(Ordering::SeqCst), + anchored + 1, + "the ref lands through one positional anchor" + ); +} + +/// Hydration can read a name the window read did not. The match shows it, but +/// the saved ref keeps the predicate the whole-window read proved unique, so +/// the published ref resolves instead of failing as an unproven enriched one. +#[test] +fn a_name_hydration_adds_is_shown_while_the_saved_ref_keeps_its_proof() { + let adapter = adapter(Mode::HydrationAddsName); + + let resolution = resolve_query( + &adapter, + &role_query(), + ObservationRoot::Window(&window()), + &selected_request(), + ) + .expect("a name only hydration read must not invalidate the match"); + assert_eq!(adapter.hydration_observations.load(Ordering::SeqCst), 1); + assert_eq!(resolution.matches[0].data.name, "Save"); + + let entry = resolution.refmap.unwrap().get("@e1").unwrap().clone(); + assert_eq!(entry.identity.name, None); + assert!(entry.identity.unique_at_capture); + crate::refs::resolve_saved_ref::resolve_saved_ref( + &adapter, + &entry, + crate::Deadline::standard().unwrap(), + None, + ) + .expect("the published ref resolves under the predicate it was proven with"); +} + +#[test] +fn selected_path_without_identity_or_geometry_rejects_without_retrying() { + let adapter = adapter(Mode::UnprovenMissingAnchor); + let error = resolve_query( &adapter, &role_query(), @@ -103,7 +209,7 @@ fn selected_path_without_identity_or_geometry_rejects_without_retrying() { &selected_request(), ) .err() - .expect("path-only selected anchors must fail closed"); + .expect("an anchor the whole-window read could not prove must fail closed"); let error = adapter_error(error); assert_eq!(error.code.as_str(), "STALE_REF"); @@ -179,9 +285,19 @@ fn adapter(mode: Mode) -> HydrationContractAdapter { fn initial_tree(mode: Mode) -> ObservedTree { let mut evidence = evidence(mode, false); evidence.ref_evidence.bounds = match mode { - Mode::MissingAnchor => LocatorField::Absent, + Mode::MissingAnchor | Mode::UnprovenMissingAnchor => LocatorField::Absent, _ => LocatorField::Known(bounds()), }; + if mode == Mode::UnprovenMissingAnchor { + return super::test_support::tree( + vec![ + super::test_support::node(0, evidence.clone(), vec![1], &[]), + super::test_support::node(1, evidence, Vec::new(), &[0]), + ], + vec![0], + true, + ); + } super::test_support::tree( vec![super::test_support::node(0, evidence, Vec::new(), &[])], vec![0], @@ -208,17 +324,25 @@ fn evidence(mode: Mode, hydrating: bool) -> super::LocatorEvidence { let mut evidence = super::test_support::evidence("button", None); evidence.name = match mode { Mode::RequiredNameUnknown if !hydrating => LocatorField::Known("Save".into()), - Mode::RequiredNameUnknown | Mode::OptionalIdentityUnknown | Mode::MissingAnchor => { - LocatorField::Unknown + Mode::HydrationAddsName if hydrating => LocatorField::Known("Save".into()), + Mode::RequiredNameUnknown + | Mode::HydrationAddsName + | Mode::OptionalIdentityUnknown + | Mode::MissingAnchor + | Mode::UnprovenMissingAnchor => LocatorField::Unknown, + Mode::ProvenIdentity | Mode::RequiredDescriptionUnknown => { + LocatorField::Known("Save".into()) } - Mode::RequiredDescriptionUnknown => LocatorField::Known("Save".into()), }; evidence.description = match mode { Mode::RequiredDescriptionUnknown if !hydrating => LocatorField::Known("Primary".into()), - Mode::RequiredDescriptionUnknown | Mode::OptionalIdentityUnknown | Mode::MissingAnchor => { - LocatorField::Unknown - } + Mode::RequiredDescriptionUnknown + | Mode::HydrationAddsName + | Mode::OptionalIdentityUnknown + | Mode::MissingAnchor + | Mode::UnprovenMissingAnchor => LocatorField::Unknown, Mode::RequiredNameUnknown => LocatorField::Known("Primary".into()), + Mode::ProvenIdentity => LocatorField::Absent, }; evidence.identifiers = IdentifierEvidence::absent(); evidence.states = LocatorField::Unknown; diff --git a/crates/core/src/locator/live_locator/selected_hydration_tests.rs b/crates/core/src/locator/live_locator/selected_hydration_tests.rs index 29955ba12..d42891658 100644 --- a/crates/core/src/locator/live_locator/selected_hydration_tests.rs +++ b/crates/core/src/locator/live_locator/selected_hydration_tests.rs @@ -137,7 +137,7 @@ fn role_only_selected_hydration_anchors_before_hydrating_without_rewalking_large assert_eq!(matched.data.name, "Primary action"); assert_eq!(matched.data.ref_id.as_deref(), Some("@e1")); let entry = resolution.refmap.as_ref().unwrap().get("@e1").unwrap(); - assert_eq!(entry.identity.name.as_deref(), Some("Primary action")); + assert_eq!(entry.identity.name, None); assert_eq!( entry .identity diff --git a/crates/core/src/locator/live_locator/test_support.rs b/crates/core/src/locator/live_locator/test_support.rs index 0b1901592..e766b3754 100644 --- a/crates/core/src/locator/live_locator/test_support.rs +++ b/crates/core/src/locator/live_locator/test_support.rs @@ -64,14 +64,37 @@ pub(crate) fn tree( assert_eq!(node.document_order as usize, index); assert!(node.children.iter().all(|child| *child as usize > index)); } + let logical_depths = depths_of(&nodes, &roots); ObservedTree { nodes, roots, source: ObservationSource::Window { window: window(), surface: crate::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, stats: LocatorStats::default(), structurally_complete, + logical_depths, } } + +/// The logical depth of every fixture node: none of them is a layout wrapper, +/// so each level costs one. A fixture that needs a wrapper patches the entry. +fn depths_of(nodes: &[ObservedNode], roots: &[u32]) -> Vec { + let mut depths: Vec = vec![0; nodes.len()]; + let mut pending = roots.to_vec(); + while let Some(index) = pending.pop() { + let Some(node) = nodes.get(index as usize) else { + continue; + }; + let below = depths[index as usize].saturating_add(1); + for child in &node.children { + if let Some(depth) = depths.get_mut(*child as usize) { + *depth = below; + pending.push(*child); + } + } + } + depths +} diff --git a/crates/core/src/locator/live_locator/validation_tests.rs b/crates/core/src/locator/live_locator/validation_tests.rs index 25a4e4ffd..8d174e424 100644 --- a/crates/core/src/locator/live_locator/validation_tests.rs +++ b/crates/core/src/locator/live_locator/validation_tests.rs @@ -144,6 +144,7 @@ fn element_root_derives_surface_with_relative_depth_budgets() { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -160,6 +161,7 @@ fn element_root_derives_surface_with_relative_depth_budgets() { source_window_title: Some("Fixture".into()), source_window_bounds_hash: None, source_surface: crate::SnapshotSurface::Menu, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/ref_action/mod.rs b/crates/core/src/ref_action/mod.rs index 84705d1a6..5358369bc 100644 --- a/crates/core/src/ref_action/mod.rs +++ b/crates/core/src/ref_action/mod.rs @@ -5,6 +5,7 @@ use crate::context::CommandContext; use crate::ref_action::ref_action_context::RefActionContext; use crate::ref_action::ref_action_wait_support::{after_scroll, trace_scroll_error}; use crate::refs::RefEntry; +use crate::refs::resolve_saved_ref::resolve_saved_ref; use crate::{AdapterError, AppError, actionability}; use serde_json::json; @@ -57,11 +58,12 @@ pub(crate) fn preflight_resolved( check_actionability_with_trace(target, request, stability) } -pub(crate) fn dispatch_resolved( +/// Dispatches the action and also returns the handle it landed on. +pub(crate) fn dispatch_resolved_landing( target: RefActionContext<'_>, mut request: ActionRequest, lease: &crate::InteractionLease, -) -> Result { +) -> Result<(ActionResult, NativeHandle), AppError> { let process_instance = target .entry .process @@ -70,9 +72,7 @@ pub(crate) fn dispatch_resolved( .filter(|instance| !instance.is_empty()) .ok_or_else(|| AdapterError::stale_ref_because("target process instance is unavailable"))?; let expected_process = crate::ProcessIdentity::new(target.entry.process.pid, process_instance); - let mut handle = target - .adapter - .resolve_element_strict(target.entry, target.deadline) + let mut handle = resolve_saved_ref(target.adapter, target.entry, target.deadline, Some(lease)) .map_err(mark_pre_dispatch_resolution_failure) .inspect_err(|error| { crate::ref_action::ref_action_wait_support::trace_resolve_error( @@ -110,10 +110,10 @@ pub(crate) fn dispatch_resolved( }; handle = match recovered { Some(handle) => handle, - None => target - .adapter - .resolve_element_strict(target.entry, target.deadline) - .map_err(after_scroll)?, + None => { + resolve_saved_ref(target.adapter, target.entry, target.deadline, Some(lease)) + .map_err(after_scroll)? + } }; let scrolled_target = ResolvedRefAction::new(target, &handle); stable_preflight(&scrolled_target, &request) @@ -163,7 +163,7 @@ pub(crate) fn dispatch_resolved( || json!({ "ref": final_target.ref_id, "action": action_name, "result": &result }), ) .map_err(trace_error_after_delivery)?; - Ok(result) + Ok((result, handle)) } /// An action that opens a sheet, menu, or alert leaves the caller staring at a @@ -313,7 +313,7 @@ pub(crate) fn execute_resolved( let stability = actionability::StabilityExpectation::permissive(target.entry.geometry.bounds_hash); preflight_resolved(&target, &request, stability)?; - let result = dispatch_resolved(target.target, request, lease); + let result = ref_action_single::dispatch_resolved(target.target, request, lease); finish_artifacts(target.target, &pre); result } @@ -368,7 +368,7 @@ pub fn execute_entry_with_context( context, }, request, - dispatch_resolved, + ref_action_single::dispatch_resolved, ) } diff --git a/crates/core/src/ref_action/presentation.rs b/crates/core/src/ref_action/presentation.rs index 561d85b76..30a9cd97c 100644 --- a/crates/core/src/ref_action/presentation.rs +++ b/crates/core/src/ref_action/presentation.rs @@ -81,10 +81,12 @@ fn target_center( .flatten() .and_then(center_of); stored.or_else(|| { - let handle = context - .adapter - .resolve_element_strict(context.entry, deadline) - .ok()?; + let handle = crate::refs::resolve_saved_ref::land_saved_ref( + context.adapter, + context.entry, + deadline, + ) + .ok()?; let bounds = context .adapter .get_element_bounds(&handle, deadline) diff --git a/crates/core/src/ref_action/presentation_caret_tests.rs b/crates/core/src/ref_action/presentation_caret_tests.rs index 1fed5a5fb..2c7dce190 100644 --- a/crates/core/src/ref_action/presentation_caret_tests.rs +++ b/crates/core/src/ref_action/presentation_caret_tests.rs @@ -65,6 +65,7 @@ impl ObservationOps for CaretAdapter { offscreen: Some(false), } ); + crate::adapter::saved_ref_scope!("textfield", Some("Name")); } impl ActionOps for CaretAdapter { @@ -109,7 +110,7 @@ fn field() -> RefEntry { capability::SET_VALUE.into(), ]; entry.source.source_app = Some("Fixture".into()); - entry.source.source_window_id = Some("w-42".into()); + entry.source.source_window_id = Some("w-1".into()); entry.source.source_window_title = Some("Fixture".into()); entry } diff --git a/crates/core/src/ref_action/presentation_lease_tests.rs b/crates/core/src/ref_action/presentation_lease_tests.rs index 47d62d653..eee287943 100644 --- a/crates/core/src/ref_action/presentation_lease_tests.rs +++ b/crates/core/src/ref_action/presentation_lease_tests.rs @@ -29,6 +29,7 @@ impl ObservationOps for LeaseOrderAdapter { } crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for LeaseOrderAdapter { @@ -73,7 +74,7 @@ fn button_at(bounds: Rect) -> RefEntry { entry.geometry.bounds_hash = bounds.bounds_hash(); entry.capabilities.available_actions = vec![capability::CLICK.into()]; entry.source.source_app = Some("Fixture".into()); - entry.source.source_window_id = Some("w-42".into()); + entry.source.source_window_id = Some("w-1".into()); entry.source.source_window_title = Some("Fixture".into()); entry } diff --git a/crates/core/src/ref_action/presentation_shared_cursor_tests.rs b/crates/core/src/ref_action/presentation_shared_cursor_tests.rs index d0a2866aa..21fada9fa 100644 --- a/crates/core/src/ref_action/presentation_shared_cursor_tests.rs +++ b/crates/core/src/ref_action/presentation_shared_cursor_tests.rs @@ -47,6 +47,7 @@ impl ObservationOps for TimedAdapter { } crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for TimedAdapter { @@ -96,7 +97,7 @@ fn button() -> RefEntry { entry.geometry.bounds_hash = bounds.bounds_hash(); entry.capabilities.available_actions = vec![capability::CLICK.into()]; entry.source.source_app = Some("Fixture".into()); - entry.source.source_window_id = Some("w-42".into()); + entry.source.source_window_id = Some("w-1".into()); entry.source.source_window_title = Some("Fixture".into()); entry } diff --git a/crates/core/src/ref_action/ref_action_cursor_overlay_tests.rs b/crates/core/src/ref_action/ref_action_cursor_overlay_tests.rs index 24495b45a..e38e697ce 100644 --- a/crates/core/src/ref_action/ref_action_cursor_overlay_tests.rs +++ b/crates/core/src/ref_action/ref_action_cursor_overlay_tests.rs @@ -1,6 +1,4 @@ -use crate::adapter::{ - ActionOps, InputOps, NativeHandle, ObservationOps, SnapshotSurface, SystemOps, -}; +use crate::adapter::{ActionOps, InputOps, NativeHandle, ObservationOps, SystemOps}; use crate::ref_action::*; use crate::{Action, ActionResult, AdapterError, CursorOverlayControl, actions::capability}; use std::sync::Mutex; @@ -52,6 +50,7 @@ impl ObservationOps for CursorAdapter { offscreen: Some(false), } ); + crate::adapter::saved_ref_scope!("textfield", Some("Run")); } impl ActionOps for CursorAdapter { @@ -76,7 +75,6 @@ impl ActionOps for CursorAdapter { } impl InputOps for CursorAdapter {} - impl SystemOps for CursorAdapter { crate::adapter::guarded_interaction_lease!(); crate::adapter::exact_window_focus!(); @@ -114,6 +112,7 @@ fn entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -126,10 +125,10 @@ fn entry() -> RefEntry { }, source: crate::RefSource { source_app: Some("Fixture".into()), - source_window_id: Some("w-42".into()), + source_window_id: Some("w-1".into()), source_window_title: Some("Fixture".into()), source_window_bounds_hash: None, - source_surface: SnapshotSurface::Window, + ..Default::default() }, scope: crate::RefScope { root_ref: None, @@ -306,6 +305,7 @@ impl ObservationOps for NoOverlaySupportAdapter { "Run", [capability::CLICK, capability::SET_VALUE] ); + crate::adapter::saved_ref_scope!("textfield", Some("Run")); } impl ActionOps for NoOverlaySupportAdapter { diff --git a/crates/core/src/ref_action/ref_action_cursor_overlay_unknown_tests.rs b/crates/core/src/ref_action/ref_action_cursor_overlay_unknown_tests.rs index 152753845..c0fbc28b2 100644 --- a/crates/core/src/ref_action/ref_action_cursor_overlay_unknown_tests.rs +++ b/crates/core/src/ref_action/ref_action_cursor_overlay_unknown_tests.rs @@ -61,6 +61,7 @@ impl ObservationOps for UnknownHitCursorAdapter { ) -> Result { Ok(crate::adapter::hit_test::HitTestResult::Unknown) } + crate::adapter::saved_ref_scope!("textfield", Some("Run")); } impl ActionOps for UnknownHitCursorAdapter { diff --git a/crates/core/src/ref_action/ref_action_disabled_offscreen_tests.rs b/crates/core/src/ref_action/ref_action_disabled_offscreen_tests.rs index 383a0759c..1ce189959 100644 --- a/crates/core/src/ref_action/ref_action_disabled_offscreen_tests.rs +++ b/crates/core/src/ref_action/ref_action_disabled_offscreen_tests.rs @@ -28,7 +28,7 @@ impl ObservationOps for DisabledBelowTheFold { ) -> Result { let scrolled = self.scroll_calls.load(Ordering::SeqCst) > 0; Ok(crate::LiveElement { - identity: crate::adapter::live_identity(""), + identity: crate::adapter::live_identity("Field"), state: ElementState { role: "textfield".into(), states: Vec::new(), @@ -42,6 +42,7 @@ impl ObservationOps for DisabledBelowTheFold { available_actions: vec![capability::SET_VALUE.into()], }) } + crate::adapter::saved_ref_scope!("textfield", Some("Field")); } impl ActionOps for DisabledBelowTheFold { @@ -81,7 +82,7 @@ fn bounds() -> crate::Rect { fn entry() -> RefEntry { let bounds = bounds(); - let mut entry = crate::adapter::minimal_ref_entry("textfield", None); + let mut entry = crate::adapter::minimal_ref_entry("textfield", Some("Field")); entry.identity.value = Some("4".into()); entry.geometry = crate::RefGeometry { bounds_space: None, diff --git a/crates/core/src/ref_action/ref_action_exactly_once_tests.rs b/crates/core/src/ref_action/ref_action_exactly_once_tests.rs index 42974efd0..e09ec38bb 100644 --- a/crates/core/src/ref_action/ref_action_exactly_once_tests.rs +++ b/crates/core/src/ref_action/ref_action_exactly_once_tests.rs @@ -67,7 +67,7 @@ impl ObservationOps for DispatchGuardAdapter { return Err(AdapterError::timeout("strict resolution was uncertain") .with_disposition(crate::DeliverySemantics::uncertain())); } - if self.mode == "ambiguous" && call == 1 { + if self.mode == "ambiguous" && call <= 2 { return Err(AdapterError::ambiguous_target("two live candidates") .with_details(serde_json::json!({ "retryable": true }))); } @@ -126,6 +126,7 @@ impl ObservationOps for DispatchGuardAdapter { ) -> Result { Ok(crate::adapter::hit_test::HitTestResult::ReachesTarget) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for DispatchGuardAdapter { @@ -206,6 +207,7 @@ fn entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -222,6 +224,7 @@ fn entry() -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -240,7 +243,7 @@ fn execute(adapter: &DispatchGuardAdapter, timeout_ms: u64) -> Result Result, AdapterError> { Ok(None) } + crate::adapter::saved_ref_scope!("textfield", Some("Field")); } impl ActionOps for KeyboardAdapter { @@ -87,7 +88,8 @@ fn a_headed_type_on_a_session_that_cannot_activate_skips_window_focus() { let adapter = KeyboardAdapter::default(); let entry: RefEntry = serde_json::from_value(serde_json::json!({ "pid": 1, "process_instance": "test-instance", "role": "textfield", "name": "Field", - "source_window_id": "w-test", "source_surface": "window", "path": [0], + "source_window_id": "w-1", "source_surface": "window", "path": [0], + "unique_at_capture": true, "available_actions": ["TypeText"], "states": [] })) .unwrap(); diff --git a/crates/core/src/ref_action/ref_action_moving_incrementor_tests.rs b/crates/core/src/ref_action/ref_action_moving_incrementor_tests.rs index eb75756e1..ab3ab427b 100644 --- a/crates/core/src/ref_action/ref_action_moving_incrementor_tests.rs +++ b/crates/core/src/ref_action/ref_action_moving_incrementor_tests.rs @@ -22,7 +22,7 @@ fn rectangle(scrolled: bool) -> crate::Rect { } fn entry() -> RefEntry { - let mut entry = crate::adapter::minimal_ref_entry("incrementor", None); + let mut entry = crate::adapter::minimal_ref_entry("incrementor", Some("Value")); entry.identity.retained_object = Some("/spin".into()); let bounds = rectangle(false); entry.geometry = RefGeometry { @@ -40,9 +40,15 @@ impl ObservationOps for Incrementor { entry: &RefEntry, _: crate::Deadline, ) -> Result { - if entry.identity.retained_object.as_deref() != Some("/spin") - || (self.replace_on_scroll && self.scrolled.load(Ordering::SeqCst)) - || entry.geometry.bounds != Some(rectangle(self.scrolled.load(Ordering::SeqCst))) + if self.replace_on_scroll && self.scrolled.load(Ordering::SeqCst) { + return Err(AdapterError::stale_ref_because( + "the retained rectangle changed", + )); + } + if entry + .geometry + .bounds + .is_some_and(|bounds| bounds != rectangle(self.scrolled.load(Ordering::SeqCst))) { return Err(AdapterError::stale_ref_because( "the retained rectangle changed", @@ -57,7 +63,7 @@ impl ObservationOps for Incrementor { _: crate::Deadline, ) -> Result { let scrolled = self.scrolled.load(Ordering::SeqCst); - let mut identity = crate::adapter::live_identity(""); + let mut identity = crate::adapter::live_identity("Value"); identity.identifiers = identity.identifiers.with_retained_object("/spin".into()); Ok(LiveElement { identity, @@ -74,6 +80,7 @@ impl ObservationOps for Incrementor { available_actions: vec![capability::SET_VALUE.into()], }) } + crate::adapter::saved_ref_scope!("incrementor", Some("Value")); } impl ActionOps for Incrementor { diff --git a/crates/core/src/ref_action/ref_action_poll.rs b/crates/core/src/ref_action/ref_action_poll.rs index c5f644e73..e99a8b9eb 100644 --- a/crates/core/src/ref_action/ref_action_poll.rs +++ b/crates/core/src/ref_action/ref_action_poll.rs @@ -10,7 +10,7 @@ use crate::{ ref_action::ref_action_wait_evidence::{failed_check, should_scroll_after_preflight}, ref_action::ref_action_wait_support::{trace_resolve_error, trace_resolve_ok}, ref_action::{ResolvedRefAction, into_adapter_error, preflight_resolved}, - refs::ref_resolve_deadline::{POLL_INTERVAL, resolve_within_deadline}, + refs::ref_resolve_deadline::{POLL_INTERVAL, land_within_deadline}, refs::resolve_attempt_outcome::ResolveAttemptOutcome, }; @@ -22,7 +22,9 @@ const STABILITY_POLL_INTERVAL: Duration = Duration::from_millis(16); /// cross-process lock and holding it for the whole wait budget would serialize /// every agent-desktop process on the machine. Anything resolved here was /// therefore observed without exclusivity, so dispatch re-resolves under the -/// lease. The second resolve is the correctness boundary, not redundant work. +/// lease, and that resolve is the correctness boundary. This loop only lands +/// the ref provisionally, which skips the whole-window count in the common +/// case, so an action pays for one whole-window read rather than two. pub(crate) fn execute_poll_loop( context: RefActionWaitContext<'_>, request: &ActionRequest, @@ -38,7 +40,7 @@ pub(crate) fn execute_poll_loop( }; loop { state.resolve_attempts = state.resolve_attempts.saturating_add(1); - match resolve_within_deadline(context.adapter, context.entry, deadline) { + match land_within_deadline(context.adapter, context.entry, deadline) { ResolveAttemptOutcome::DeadlinePassed => return Err(timeout(&state, deadline)), ResolveAttemptOutcome::Failed(error) => { handle_resolve_failure( diff --git a/crates/core/src/ref_action/ref_action_scroll_recovery_tests.rs b/crates/core/src/ref_action/ref_action_scroll_recovery_tests.rs index dd22a7156..4f22ee75a 100644 --- a/crates/core/src/ref_action/ref_action_scroll_recovery_tests.rs +++ b/crates/core/src/ref_action/ref_action_scroll_recovery_tests.rs @@ -60,6 +60,7 @@ impl ObservationOps for ScrollRecoveryAdapter { available_actions: vec![capability::EXPAND.into()], }) } + crate::adapter::saved_ref_scope!("menuitem", Some("File")); } impl ActionOps for ScrollRecoveryAdapter { @@ -121,6 +122,7 @@ fn entry() -> RefEntry { source_window_title: Some("Notepad".into()), source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }; entry } diff --git a/crates/core/src/ref_action/ref_action_single.rs b/crates/core/src/ref_action/ref_action_single.rs index 3e988ff6f..b64f237a9 100644 --- a/crates/core/src/ref_action/ref_action_single.rs +++ b/crates/core/src/ref_action/ref_action_single.rs @@ -4,6 +4,15 @@ use crate::{ ref_action::ref_action_wait_context::RefActionWaitContext, }; +/// The plain dispatch: the handle the action landed on is dropped. +pub(crate) fn dispatch_resolved( + target: RefActionContext<'_>, + request: ActionRequest, + lease: &crate::InteractionLease, +) -> Result { + crate::ref_action::dispatch_resolved_landing(target, request, lease).map(|(result, _)| result) +} + pub(crate) fn execute_single_shot( context: RefActionWaitContext<'_>, request: ActionRequest, diff --git a/crates/core/src/ref_action/ref_action_tests.rs b/crates/core/src/ref_action/ref_action_tests.rs index ead906fb5..f9aa6962d 100644 --- a/crates/core/src/ref_action/ref_action_tests.rs +++ b/crates/core/src/ref_action/ref_action_tests.rs @@ -44,6 +44,7 @@ impl ObservationOps for SuccessfulAdapter { } crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for SuccessfulAdapter { @@ -82,6 +83,7 @@ impl ObservationOps for FailingAdapter { } crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for FailingAdapter { @@ -113,6 +115,7 @@ impl ObservationOps for ProcessReplacingAdapter { } crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for ProcessReplacingAdapter { @@ -158,6 +161,7 @@ impl ObservationOps for TraceFailureAdapter { } crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for TraceFailureAdapter { @@ -204,6 +208,7 @@ fn entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -216,10 +221,11 @@ fn entry() -> RefEntry { }, source: crate::RefSource { source_app: Some("Fixture".into()), - source_window_id: Some("w-42".into()), + source_window_id: Some("w-1".into()), source_window_title: Some("Fixture".into()), source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/ref_action/ref_action_wait.rs b/crates/core/src/ref_action/ref_action_wait.rs index acbba4ad0..fd44bee2c 100644 --- a/crates/core/src/ref_action/ref_action_wait.rs +++ b/crates/core/src/ref_action/ref_action_wait.rs @@ -77,45 +77,97 @@ fn execute_with_deadline( ), AdapterError, > { - let state = if request.timeout_ms.is_some_and(|timeout_ms| timeout_ms > 0) { - execute_poll_loop(context, &request, deadline)? - } else { - RefActionPollState::default() - }; - let pre = crate::ref_action::capture_pre_artifact( - context.context, - context.adapter, - context.entry, - deadline, - ); - let cursor = crate::cursor_overlay::hold_cursor(context.context, deadline); - crate::ref_action::presentation::before_lease(context, &request, deadline, cursor.is_ok()); - let lease = context - .adapter - .acquire_interaction_lease(deadline) - .map_err(|error| state.attach_error_metrics(error))?; - if deadline.is_expired() { - return Err(state.attach_error_metrics(deadline.timeout_error())); + let polls = request.timeout_ms.is_some_and(|timeout_ms| timeout_ms > 0); + let mut saw_ambiguity = false; + let mut refusal: Option = None; + loop { + let mut state = if polls { + execute_poll_loop(context, &request, deadline) + .map_err(|error| with_refusal(error, refusal.as_ref()))? + } else { + RefActionPollState::default() + }; + state.saw_ambiguity |= saw_ambiguity; + let pre = crate::ref_action::capture_pre_artifact( + context.context, + context.adapter, + context.entry, + deadline, + ); + let cursor = crate::cursor_overlay::hold_cursor(context.context, deadline); + crate::ref_action::presentation::before_lease(context, &request, deadline, cursor.is_ok()); + let lease = context + .adapter + .acquire_interaction_lease(deadline) + .map_err(|error| state.attach_error_metrics(error))?; + if deadline.is_expired() { + let error = state.attach_error_metrics(deadline.timeout_error()); + return Err(with_refusal(error, refusal.as_ref())); + } + if polls { + request.timeout_ms = Some(deadline.remaining_ms()); + } + let lease_started = std::time::Instant::now(); + match execute_single_shot(context, request.clone(), deadline, &lease, &dispatch) { + Ok(mut result) => { + state.attach_transient_ambiguity(&mut result); + state.attach_wait_metrics( + &mut result, + &lease, + u64::try_from(lease_started.elapsed().as_millis()).unwrap_or(u64::MAX), + ); + return Ok((result, lease, pre, deadline, lease_started)); + } + Err(error) => { + drop(lease); + crate::ref_action::finish_artifacts(RefActionContext::new(context, deadline), &pre); + if !(polls && resolves_again(&error, deadline)) { + return Err(error); + } + saw_ambiguity |= error.code == crate::ErrorCode::AmbiguousTarget; + refusal = Some(error); + crate::wait::deadline::sleep( + crate::refs::ref_resolve_deadline::POLL_INTERVAL.min(deadline.remaining()), + ); + } + } } - if request.timeout_ms.is_some_and(|timeout_ms| timeout_ms > 0) { - request.timeout_ms = Some(deadline.remaining_ms()); +} + +/// Whether a dispatch-time refusal goes back to the wait: the full resolution +/// under the lease failed in a way that may clear, nothing was sent, and the +/// caller's budget is not spent. +fn resolves_again(error: &AdapterError, deadline: Deadline) -> bool { + error.disposition == crate::DeliverySemantics::not_delivered() + && error.is_retryable_resolution_failure() + && !deadline.is_expired() +} + +/// Carries the dispatch-time refusal that sent the action back to the wait as +/// `details.refusal`, so a budget that runs out there still reports why the +/// ref was not acted on. +fn with_refusal(mut error: AdapterError, refusal: Option<&AdapterError>) -> AdapterError { + let Some(refusal) = refusal else { + return error; + }; + let mut recorded = serde_json::json!({ + "code": refusal.code.as_str(), + "message": refusal.message, + }); + if let (Some(object), Some(details)) = (recorded.as_object_mut(), &refusal.details) { + object.insert("details".into(), details.clone()); } - let lease_started = std::time::Instant::now(); - let mut result = match execute_single_shot(context, request, deadline, &lease, dispatch) { - Ok(result) => result, - Err(error) => { - drop(lease); - crate::ref_action::finish_artifacts(RefActionContext::new(context, deadline), &pre); - return Err(error); + let mut details = error + .details + .take() + .unwrap_or_else(|| serde_json::json!({})); + match details.as_object_mut() { + Some(object) => { + object.insert("refusal".into(), recorded); } - }; - state.attach_transient_ambiguity(&mut result); - state.attach_wait_metrics( - &mut result, - &lease, - u64::try_from(lease_started.elapsed().as_millis()).unwrap_or(u64::MAX), - ); - Ok((result, lease, pre, deadline, lease_started)) + None => details = serde_json::json!({ "error_details": details, "refusal": recorded }), + } + error.with_details(details) } pub(crate) fn operation_deadline(request: &ActionRequest) -> Result { diff --git a/crates/core/src/ref_action/ref_action_wait_app_not_found_tests.rs b/crates/core/src/ref_action/ref_action_wait_app_not_found_tests.rs index d161b9419..841678bb4 100644 --- a/crates/core/src/ref_action/ref_action_wait_app_not_found_tests.rs +++ b/crates/core/src/ref_action/ref_action_wait_app_not_found_tests.rs @@ -23,6 +23,7 @@ fn entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -44,6 +45,7 @@ fn entry() -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: crate::snapshot::snapshot_surface::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -76,6 +78,7 @@ impl ObservationOps for TicksThenAppNotFoundAdapter { Err(AdapterError::new(ErrorCode::AppNotFound, "app gone")) } } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for TicksThenAppNotFoundAdapter { @@ -114,6 +117,7 @@ impl ObservationOps for AppNotFoundExitedProcessAdapter { ) -> Result { Err(AdapterError::new(ErrorCode::AppNotFound, "app gone")) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for AppNotFoundExitedProcessAdapter { @@ -151,7 +155,7 @@ fn terminal_app_not_found_against_exited_process_carries_process_state_detail() context: &CommandContext::default(), }, ActionRequest::headless(Action::Click), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap_err(); @@ -189,6 +193,7 @@ impl ObservationOps for AppNotFoundUnresponsiveProcessAdapter { ) -> Result { Err(AdapterError::new(ErrorCode::AppNotFound, "app gone")) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for AppNotFoundUnresponsiveProcessAdapter { @@ -231,7 +236,7 @@ fn terminal_app_not_found_against_unresponsive_process_surfaces_app_unresponsive context: &CommandContext::default(), }, ActionRequest::headless(Action::Click), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap_err(); @@ -266,7 +271,7 @@ fn probe_call_count_is_independent_of_auto_wait_tick_count() { context: &CommandContext::default(), }, request_with_timeout(5_000), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap_err(); diff --git a/crates/core/src/ref_action/ref_action_wait_dispatch_tests.rs b/crates/core/src/ref_action/ref_action_wait_dispatch_tests.rs new file mode 100644 index 000000000..6ce9a8c9f --- /dev/null +++ b/crates/core/src/ref_action/ref_action_wait_dispatch_tests.rs @@ -0,0 +1,155 @@ +use super::*; +use crate::locator::live_locator::{ObservationRequest, ObservationRoot, ObservedTree}; + +/// A window whose whole-window read differs from what the provisional landing +/// sees: a twin of the target, or one read that comes back cut short. +struct DispatchReadAdapter { + window_reads: AtomicU32, + dispatches: AtomicU32, + twin: bool, + disabled_after_a_read: bool, +} + +impl DispatchReadAdapter { + fn new(twin: bool) -> Self { + Self { + window_reads: AtomicU32::new(0), + dispatches: AtomicU32::new(0), + twin, + disabled_after_a_read: false, + } + } +} + +impl ObservationOps for DispatchReadAdapter { + fn observe_tree( + &self, + root: ObservationRoot<'_>, + _request: &ObservationRequest, + ) -> Result { + if matches!(root, ObservationRoot::Element { .. }) { + return crate::adapter::saved_ref_element_tree(root); + } + let read = self.window_reads.fetch_add(1, Ordering::SeqCst) + 1; + if self.twin { + return crate::adapter::saved_ref_tree_nodes( + root, + &[("button", Some("Run"), None), ("button", Some("Run"), None)], + ); + } + if read == 1 { + let mut unread = crate::adapter::minimal_accessibility_node("scrollarea", "Feed"); + unread.children_count = Some(3); + let mut window = crate::adapter::minimal_accessibility_node("group", ""); + window.identity.name = None; + window.children = vec![ + crate::adapter::minimal_accessibility_node("button", "Run"), + unread, + ]; + return crate::adapter::observed_tree(&root, window); + } + crate::adapter::saved_ref_tree(root, "button", Some("Run")) + } + + fn resolve_locator_anchor( + &self, + _entry: &RefEntry, + _deadline: crate::Deadline, + ) -> Result { + Ok(NativeHandle::null()) + } + + fn list_windows( + &self, + _filter: &crate::adapter::WindowFilter, + _deadline: crate::Deadline, + ) -> Result, AdapterError> { + Ok(crate::adapter::saved_ref_window_pool()) + } + + crate::adapter::complete_live_observation!( + "button", + "Run", + [capability::CLICK], + adapter => crate::ElementState { + role: "button".into(), + states: Vec::new(), + value: None, + enabled: Some( + !(adapter.disabled_after_a_read && adapter.window_reads.load(Ordering::SeqCst) > 0), + ), + hidden: Some(false), + offscreen: Some(false), + } + ); +} + +impl ActionOps for DispatchReadAdapter { + fn execute_action( + &self, + _handle: &NativeHandle, + _request: ActionRequest, + _lease: &crate::InteractionLease, + ) -> Result { + self.dispatches.fetch_add(1, Ordering::SeqCst); + Ok(crate::actions::action_result::ActionResult::delivered_unverified("click")) + } +} + +impl InputOps for DispatchReadAdapter {} + +impl SystemOps for DispatchReadAdapter { + crate::adapter::guarded_interaction_lease!(); +} + +fn click(adapter: &DispatchReadAdapter, timeout_ms: u64) -> Result { + execute_with_auto_wait( + RefActionWaitCtx { + adapter, + entry: &entry(), + ref_id: "@e1", + context: &CommandContext::default(), + }, + request_with_timeout(timeout_ms), + crate::ref_action::ref_action_single::dispatch_resolved, + ) +} + +#[test] +fn a_twin_found_by_the_count_under_the_lease_is_never_dispatched() { + let adapter = DispatchReadAdapter::new(true); + + let error = click(&adapter, 2_000).unwrap_err(); + + assert_eq!(error.code, ErrorCode::AmbiguousTarget); + assert_eq!(error.details.unwrap()["kind"], "saved_ref_ambiguous"); + assert_eq!(adapter.window_reads.load(Ordering::SeqCst), 1); + assert_eq!(adapter.dispatches.load(Ordering::SeqCst), 0); +} + +#[test] +fn a_cut_short_count_under_the_lease_waits_again_within_the_budget() { + let adapter = DispatchReadAdapter::new(false); + + click(&adapter, 5_000).expect("the second whole-window read is complete"); + + assert_eq!(adapter.window_reads.load(Ordering::SeqCst), 2); + assert_eq!(adapter.dispatches.load(Ordering::SeqCst), 1); +} + +#[test] +fn a_budget_spent_after_a_refusal_reports_that_refusal() { + let mut adapter = DispatchReadAdapter::new(false); + adapter.disabled_after_a_read = true; + + let error = click(&adapter, 600).unwrap_err(); + + assert_eq!(error.code, ErrorCode::Timeout); + let details = error.details.expect("timeout details"); + assert_eq!(details["refusal"]["code"], "TIMEOUT"); + assert_eq!( + details["refusal"]["details"]["kind"], + "resolution_identity_unknown" + ); + assert_eq!(adapter.dispatches.load(Ordering::SeqCst), 0); +} diff --git a/crates/core/src/ref_action/ref_action_wait_lease_tests.rs b/crates/core/src/ref_action/ref_action_wait_lease_tests.rs index 86b85ade9..6de4c4229 100644 --- a/crates/core/src/ref_action/ref_action_wait_lease_tests.rs +++ b/crates/core/src/ref_action/ref_action_wait_lease_tests.rs @@ -80,6 +80,7 @@ impl ObservationOps for DoubleCheckAdapter { ) -> Result { Ok(crate::adapter::hit_test::HitTestResult::ReachesTarget) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for DoubleCheckAdapter { @@ -245,6 +246,7 @@ impl ObservationOps for DisabledAfterLeaseAdapter { self.bounds_calls.fetch_add(1, Ordering::SeqCst); Ok(Some(bounds())) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for DisabledAfterLeaseAdapter {} @@ -288,7 +290,7 @@ fn a_disabled_element_fails_closed_without_sampling_even_with_a_stale_hash() { context: &CommandContext::default(), }, ActionRequest::headed(Action::DoubleClick).with_timeout_ms(Some(5_000)), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .err() .unwrap(); @@ -311,7 +313,7 @@ fn run_with_entry( context: &CommandContext::default(), }, request.with_timeout_ms(Some(timeout_ms)), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap() } @@ -329,7 +331,7 @@ fn lease_acquisition_receives_the_exact_original_deadline() { }, ActionRequest::headless(Action::Click), deadline, - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .err() .unwrap(); diff --git a/crates/core/src/ref_action/ref_action_wait_process_state_tests.rs b/crates/core/src/ref_action/ref_action_wait_process_state_tests.rs index 1ce22a818..1444e0de0 100644 --- a/crates/core/src/ref_action/ref_action_wait_process_state_tests.rs +++ b/crates/core/src/ref_action/ref_action_wait_process_state_tests.rs @@ -26,6 +26,7 @@ fn entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -42,6 +43,7 @@ fn entry() -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: crate::snapshot::snapshot_surface::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -69,6 +71,7 @@ impl ObservationOps for UnresponsiveProcessAdapter { ) -> Result { Err(AdapterError::stale_ref("@e1")) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } fn app(name: &str) -> AppInfo { @@ -122,7 +125,7 @@ fn terminal_stale_ref_against_unresponsive_process_surfaces_app_unresponsive() { context: &CommandContext::default(), }, ActionRequest::headless(Action::Click), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap_err(); @@ -163,6 +166,7 @@ impl ObservationOps for RecycledPidAdapter { ) -> Result { Err(AdapterError::stale_ref("@e1")) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for RecycledPidAdapter {} @@ -196,7 +200,7 @@ fn recycled_pid_never_upgrades_stale_ref_to_app_unresponsive() { context: &CommandContext::default(), }, ActionRequest::headless(Action::Click), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap_err(); @@ -241,6 +245,7 @@ impl ObservationOps for TerminatedProcessAdapter { ) -> Result { Err(AdapterError::stale_ref("@e1")) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for TerminatedProcessAdapter {} @@ -278,7 +283,7 @@ fn terminal_stale_ref_preserves_exited_and_crashed_process_details() { context: &CommandContext::default(), }, ActionRequest::headless(Action::Click), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap_err(); diff --git a/crates/core/src/ref_action/ref_action_wait_stale_tests.rs b/crates/core/src/ref_action/ref_action_wait_stale_tests.rs index 4ebc47c15..7e276dfb6 100644 --- a/crates/core/src/ref_action/ref_action_wait_stale_tests.rs +++ b/crates/core/src/ref_action/ref_action_wait_stale_tests.rs @@ -21,6 +21,7 @@ impl ObservationOps for CompleteNoCandidateAdapter { } crate::adapter::complete_live_observation!("menubutton", "Advanced", [capability::EXPAND]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for CompleteNoCandidateAdapter {} @@ -41,12 +42,16 @@ fn expand_complete_no_candidate_fails_immediately_with_stale_ref() { context: &CommandContext::default(), }, ActionRequest::headless(Action::Expand).with_timeout_ms(Some(5_000)), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap_err(); assert_eq!(error.code, ErrorCode::StaleRef); - assert_eq!(adapter.resolve_calls.load(Ordering::SeqCst), 1); + assert_eq!( + adapter.resolve_calls.load(Ordering::SeqCst), + 2, + "one provisional landing and one full resolution, then no retry" + ); assert!(started.elapsed() < std::time::Duration::from_millis(500)); } @@ -63,7 +68,7 @@ fn incomplete_resolution_still_polls_until_success() { context: &CommandContext::default(), }, request_with_timeout(5_000), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap(); diff --git a/crates/core/src/ref_action/ref_action_wait_success_tests.rs b/crates/core/src/ref_action/ref_action_wait_success_tests.rs index 53f3029ec..311920a02 100644 --- a/crates/core/src/ref_action/ref_action_wait_success_tests.rs +++ b/crates/core/src/ref_action/ref_action_wait_success_tests.rs @@ -27,6 +27,7 @@ fn entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -43,6 +44,7 @@ fn entry() -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: crate::snapshot::snapshot_surface::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -66,6 +68,7 @@ impl ObservationOps for SuccessWithUnresponsiveProbeAdapter { } crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for SuccessWithUnresponsiveProbeAdapter { @@ -108,7 +111,7 @@ fn enrichment_never_converts_a_successful_action_into_a_failure() { context: &CommandContext::default(), }, ActionRequest::headless(Action::Click), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap(); assert_eq!(result.action, "click"); diff --git a/crates/core/src/ref_action/ref_action_wait_tests.rs b/crates/core/src/ref_action/ref_action_wait_tests.rs index ef95b9b2d..858515d61 100644 --- a/crates/core/src/ref_action/ref_action_wait_tests.rs +++ b/crates/core/src/ref_action/ref_action_wait_tests.rs @@ -56,6 +56,7 @@ impl ObservationOps for RetryAdapter { } crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for RetryAdapter { @@ -86,7 +87,7 @@ impl ObservationOps for AmbiguousThenOkAdapter { _deadline: crate::Deadline, ) -> Result { self.resolve_calls.fetch_add(1, Ordering::SeqCst); - if self.resolve_calls.load(Ordering::SeqCst) == 1 { + if self.resolve_calls.load(Ordering::SeqCst) <= 2 { return Err(AdapterError::ambiguous_target("2 candidates") .with_details(serde_json::json!({ "retryable": true }))); } @@ -94,6 +95,7 @@ impl ObservationOps for AmbiguousThenOkAdapter { } crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for AmbiguousThenOkAdapter { @@ -132,6 +134,7 @@ fn entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -148,6 +151,7 @@ fn entry() -> RefEntry { source_window_title: Some("Fixture".into()), source_window_bounds_hash: None, source_surface: crate::snapshot::snapshot_surface::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -174,7 +178,7 @@ fn none_timeout_uses_single_resolve_attempt() { context: &CommandContext::default(), }, ActionRequest::headless(Action::Click), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap_err(); assert_eq!(adapter.resolve_calls.load(Ordering::SeqCst), 1); @@ -194,7 +198,7 @@ fn transient_ambiguity_is_recorded_in_result_details() { context: &CommandContext::default(), }, request_with_timeout(5_000), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap(); assert_eq!( @@ -245,6 +249,7 @@ impl ObservationOps for DisabledPolicyBlockedAdapter { available_actions: vec![capability::CLICK.into()], }) } + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for DisabledPolicyBlockedAdapter {} @@ -272,7 +277,7 @@ fn policy_denied_disabled_target_fails_fast_without_exhausting_wait_budget() { context: &CommandContext::default(), }, ActionRequest::headless(Action::RightClick).with_timeout_ms(Some(5_000)), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap_err(); @@ -307,6 +312,7 @@ impl ObservationOps for SettledStaleAdapter { } crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for SettledStaleAdapter { @@ -343,7 +349,7 @@ fn complete_no_candidate_returns_stale_ref_without_exhausting_budget() { context: &CommandContext::default(), }, request_with_timeout(300), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap_err(); @@ -352,7 +358,11 @@ fn complete_no_candidate_returns_stale_ref_without_exhausting_budget() { let details = err.details.expect("settled details"); assert_eq!(details["kind"], "resolve_no_candidate"); assert_eq!(details["complete"], true); - assert_eq!(adapter.resolve_calls.load(Ordering::SeqCst), 1); + assert_eq!( + adapter.resolve_calls.load(Ordering::SeqCst), + 2, + "one provisional landing and one full resolution, then no retry" + ); } #[path = "ref_action_wait_lease_tests.rs"] @@ -363,3 +373,6 @@ mod unresponsive_tests; #[path = "ref_action_wait_stale_tests.rs"] mod stale_tests; + +#[path = "ref_action_wait_dispatch_tests.rs"] +mod dispatch_tests; diff --git a/crates/core/src/ref_action/ref_action_wait_unresponsive_tests.rs b/crates/core/src/ref_action/ref_action_wait_unresponsive_tests.rs index ef9aa46d0..ba14c5676 100644 --- a/crates/core/src/ref_action/ref_action_wait_unresponsive_tests.rs +++ b/crates/core/src/ref_action/ref_action_wait_unresponsive_tests.rs @@ -19,6 +19,7 @@ impl ObservationOps for TransientUnresponsiveAdapter { } crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for TransientUnresponsiveAdapter { @@ -51,7 +52,7 @@ fn auto_wait_retries_a_transiently_unresponsive_accessibility_service() { context: &CommandContext::default(), }, ActionRequest::headless(Action::Click).with_timeout_ms(Some(5_000)), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ) .unwrap(); diff --git a/crates/core/src/refs/mod.rs b/crates/core/src/refs/mod.rs index 956b39b04..07deed4f5 100644 --- a/crates/core/src/refs/mod.rs +++ b/crates/core/src/refs/mod.rs @@ -4,6 +4,7 @@ use serde::{Deserialize, Serialize}; use smallvec::SmallVec; use std::cell::RefCell; use std::collections::HashMap; +use std::collections::HashSet; use std::collections::hash_map::RandomState; use std::hash::BuildHasher; use std::path::{Path, PathBuf}; @@ -103,24 +104,38 @@ impl RefMap { .filter_map(|entry| entry.identity.retained_object.as_deref()) } + /// Removes the refs a drill-down replaces: the entries rooted at `root`, the + /// entries the same capture recorded below the root's own path, and - + /// transitively - everything those entries rooted, so a re-drill of an + /// ancestor leaves no ref of a nested drill behind. The selected root always + /// survives. pub fn remove_by_root_ref(&mut self, root: &str) { - let root_entry = self.inner.get(root).cloned(); - self.inner.retain(|ref_id, entry| { + let root_entry = self.inner.get(root); + let mut removed: HashSet = HashSet::new(); + for (ref_id, entry) in &self.inner { if ref_id == root { - return true; + continue; } let owned = entry.scope.root_ref.as_deref() == Some(root); - let descendant = root_entry.as_ref().is_some_and(|parent| { - entry.process == parent.process - && entry.source == parent.source - && (parent.scope.root_ref.is_none() || parent.scope.path_is_absolute) - && (entry.scope.root_ref.is_none() || entry.scope.path_is_absolute) - && entry.scope.path.starts_with(&parent.scope.path) - && (entry.scope.path.len() > parent.scope.path.len() - || entry.scope.root_ref.is_some()) - }); - !owned && !descendant - }); + let replaced = root_entry.is_some_and(|parent| same_capture_descendant(entry, parent)); + if owned || replaced { + removed.insert(ref_id.clone()); + } + } + let mut pending: Vec = removed.iter().cloned().collect(); + while let Some(owner) = pending.pop() { + for (ref_id, entry) in &self.inner { + if removed.contains(ref_id) + || ref_id == root + || entry.scope.root_ref.as_deref() != Some(owner.as_str()) + { + continue; + } + removed.insert(ref_id.clone()); + pending.push(ref_id.clone()); + } + } + self.inner.retain(|ref_id, _| !removed.contains(ref_id)); } pub fn validate(&self) -> Result<(), AppError> { @@ -182,6 +197,22 @@ impl Default for RefMap { } } +/// Whether `entry` sits below `parent` within one capture's path coordinates. +/// +/// The ref that allocated a set of entries names that capture: the window walk +/// records none, and a drill records the ref it drilled, which also makes its +/// paths absolute. An entry another capture observed - another region's drill, +/// or a node the tree has since moved under these coordinates - never compares +/// equal however its path reads. +fn same_capture_descendant(entry: &RefEntry, parent: &RefEntry) -> bool { + entry.process == parent.process + && entry.source == parent.source + && entry.scope.root_ref == parent.scope.root_ref + && entry.scope.path_is_absolute == parent.scope.path_is_absolute + && entry.scope.path.len() > parent.scope.path.len() + && entry.scope.path.starts_with(&parent.scope.path) +} + fn refmap_path() -> Result { Ok(crate::storage::state_root::resolve_configured_state_root()?.join("last_refmap.json")) } @@ -333,14 +364,19 @@ pub(crate) mod ref_alloc_scope; pub(crate) mod ref_alloc_source; pub(crate) mod ref_allocation; pub(crate) mod ref_capabilities; +pub(crate) mod ref_capture_proof; pub(crate) mod ref_entry; pub(crate) mod ref_entry_identity; +pub(crate) mod ref_full_predicate; pub(crate) mod ref_geometry; pub mod ref_identity; pub(crate) mod ref_identity_match; +pub(crate) mod ref_live_predicate; pub(crate) mod ref_process; pub(crate) mod ref_resolve_deadline; +pub(crate) mod ref_root_read; pub(crate) mod ref_scope; +pub(crate) mod ref_skeleton_anchors; pub(crate) mod ref_source; pub(crate) mod ref_token; pub(crate) mod refs_lock; @@ -350,13 +386,13 @@ pub(crate) mod refs_test_support; pub(crate) mod refs_validate; pub(crate) mod resolve_attempt_outcome; pub mod resolve_errors; -#[cfg(test)] -#[path = "refs_tests.rs"] -mod tests; - +pub mod resolve_saved_ref; #[cfg(test)] #[path = "refs_serde_tests.rs"] mod serde_tests; +#[cfg(test)] +#[path = "refs_tests.rs"] +mod tests; #[cfg(test)] #[path = "refs_size_tests.rs"] diff --git a/crates/core/src/refs/ref_alloc.rs b/crates/core/src/refs/ref_alloc.rs index 19515b79d..72f328042 100644 --- a/crates/core/src/refs/ref_alloc.rs +++ b/crates/core/src/refs/ref_alloc.rs @@ -1,5 +1,7 @@ use crate::AccessibilityNode; +use crate::refs::ref_capture_proof::CaptureProof; use crate::refs::ref_identity::has_meaningful_identity; +use crate::refs::ref_skeleton_anchors::{carries_element_token, plan_skeleton_anchors}; use crate::refs::{RefEntry, RefMap}; use std::collections::HashSet; @@ -33,6 +35,7 @@ pub(crate) fn ref_entry_from_node( .native_id .clone() .filter(|id| !id.value.trim().is_empty()), + unique_at_capture: false, }, geometry: crate::RefGeometry { bounds, @@ -49,6 +52,7 @@ pub(crate) fn ref_entry_from_node( source_window_title: source.window_title.map(str::to_string), source_window_bounds_hash: source.window_bounds_hash, source_surface: source.surface, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref, @@ -170,76 +174,105 @@ pub fn transform_tree( node } -pub(crate) fn allocate_refs( +pub(crate) fn allocate_refs<'a>( node: AccessibilityNode, refmap: &mut RefMap, - config: &RefAllocConfig, + config: &RefAllocConfig<'a>, ) -> Result { - let mut skeleton_anchors = HashSet::new(); - if config.scope.root_ref_id.is_none() { - plan_skeleton_anchors(&node, &mut Vec::new(), &mut skeleton_anchors); - } + let plan = AllocationPlan::new(&node, config); allocate_refs_at_path( node, refmap, config, &mut config.scope.path_prefix.to_vec(), - &skeleton_anchors, + &plan, ) } -fn plan_skeleton_anchors( +/// Whether allocation over this presentation selects an entry that cannot +/// re-identify itself on its own: a ref-able node, or a boundary container +/// offered as a skeleton drill target, whose identity carries no element token. +/// A presentation whose selected entries all carry a token resolves through +/// those tokens and needs no capture census, so a caller can ask this before +/// paying for the observation a census needs. +pub(crate) fn selects_tokenless_entry(node: &AccessibilityNode) -> bool { + let plan = AllocationPlan::candidates(node); + selects_tokenless_at(node, &mut Vec::new(), &plan) +} + +fn selects_tokenless_at( node: &AccessibilityNode, path: &mut Vec, - anchors: &mut HashSet>, + plan: &AllocationPlan<'_>, ) -> bool { - let mut uncovered_boundary = node.children_count.is_some(); - for (index, child) in node.children.iter().enumerate() { + let tokenless = !carries_element_token(node); + if tokenless && (is_ref_able(node) || plan.skeleton_anchors.contains(path.as_slice())) { + return true; + } + node.children.iter().enumerate().any(|(index, child)| { path.push(index); - uncovered_boundary |= plan_skeleton_anchors(child, path, anchors); + let selected = selects_tokenless_at(child, path, plan); path.pop(); - } - if !uncovered_boundary { - return false; - } - if can_receive_observed_ref(node) && is_ref_able(node) { - return false; - } - if !path.is_empty() && is_resolvable_skeleton_anchor(node) { - anchors.insert(path.clone()); - return false; - } - true + selected + }) } -/// A skeleton anchor is offered as a drill target because it carries -/// evidence a later resolve can re-identify it by. When that evidence is -/// geometry alone, the entry must keep its bounds: dropping them because the -/// caller did not ask to *see* bounds leaves a ref this predicate approved -/// and nothing can resolve, which is a drill target that fails on its own -/// snapshot. The ref-able path guards the same strip with the same -/// condition. -fn is_resolvable_skeleton_anchor(node: &AccessibilityNode) -> bool { - can_receive_observed_ref(node) - && (has_meaningful_text(node.identity.name.as_deref()) - || has_meaningful_text(node.identity.description.as_deref()) - || node.identity.native_id.as_ref().is_some_and(|identifier| { - identifier.kind != crate::IdentifierKind::Unknown - && !identifier.value.trim().is_empty() - }) - || node - .presentation - .bounds - .and_then(|bounds| bounds.bounds_hash()) - .is_some()) +/// What one allocation decides once for every node it visits: which boundary +/// nodes anchor a skeleton, and which entries the observation's capture proof +/// found exactly once. +struct AllocationPlan<'a> { + skeleton_anchors: HashSet>, + capture: Option<&'a CaptureProof<'a>>, } -fn can_receive_observed_ref(node: &AccessibilityNode) -> bool { - crate::Role::is_canonical(&node.role) && node.role != "unknown" +impl<'a> AllocationPlan<'a> { + /// Anchors the boundaries a later resolve can re-identify: one carrying an + /// element token or one the capture found exactly once. A boundary that is + /// neither leaves its anchor to the nearest ancestor that is, since geometry + /// is not identity and a twin cannot be re-identified. + fn new(node: &AccessibilityNode, config: &RefAllocConfig<'a>) -> Self { + let capture = config.capture; + let mut skeleton_anchors = HashSet::new(); + if config.scope.root_ref_id.is_none() { + let offered = |boundary: &AccessibilityNode, path: &[usize]| { + carries_element_token(boundary) + || capture.is_some_and(|proof| { + proof.proves(&skeleton_anchor_entry(boundary, &config.source, path), path) + }) + }; + plan_skeleton_anchors(node, &mut Vec::new(), &mut skeleton_anchors, &offered); + } + Self { + skeleton_anchors, + capture, + } + } + + /// Every boundary that could anchor a skeleton, offered or not, because a + /// caller costs the census from the candidates. + fn candidates(node: &AccessibilityNode) -> Self { + let mut skeleton_anchors = HashSet::new(); + plan_skeleton_anchors(node, &mut Vec::new(), &mut skeleton_anchors, &|_, _| true); + Self { + skeleton_anchors, + capture: None, + } + } + + fn proven(&self, entry: &RefEntry, path: &[usize]) -> bool { + self.capture + .is_some_and(|capture| capture.proves(entry, path)) + } } -fn has_meaningful_text(value: Option<&str>) -> bool { - value.is_some_and(|text| !text.trim().is_empty()) +fn skeleton_anchor_entry( + node: &AccessibilityNode, + source: &crate::refs::ref_alloc_source::RefAllocSource<'_>, + path: &[usize], +) -> RefEntry { + let mut entry = ref_entry_from_node(node, source, None, path); + entry.capabilities.available_actions = vec![]; + entry } fn allocate_refs_at_path( @@ -247,7 +280,7 @@ fn allocate_refs_at_path( refmap: &mut RefMap, config: &RefAllocConfig, path: &mut Vec, - skeleton_anchors: &HashSet>, + plan: &AllocationPlan, ) -> Result { let node_is_ref_able = is_ref_able(&node); @@ -259,17 +292,20 @@ fn allocate_refs_at_path( path, ); entry.scope.path_is_absolute = config.scope.root_ref_id.is_some(); + entry.identity.unique_at_capture = plan.proven(&entry, path); + entry.source.renderer_accessibility_forced = config.options.force_renderer_accessibility; if !config.options.include_bounds && has_meaningful_identity(&entry) { entry.geometry.bounds = None; } node.ref_id = allocate_observed_ref(refmap, entry)?; } - let is_skeleton_anchor = !node_is_ref_able && skeleton_anchors.contains(path.as_slice()); + let is_skeleton_anchor = !node_is_ref_able && plan.skeleton_anchors.contains(path.as_slice()); if is_skeleton_anchor { - let mut entry = ref_entry_from_node(&node, &config.source, None, path); - entry.capabilities.available_actions = vec![]; + let mut entry = skeleton_anchor_entry(&node, &config.source, path); + entry.identity.unique_at_capture = plan.proven(&entry, path); + entry.source.renderer_accessibility_forced = config.options.force_renderer_accessibility; if !config.options.include_bounds && has_meaningful_identity(&entry) { entry.geometry.bounds = None; } @@ -284,7 +320,7 @@ fn allocate_refs_at_path( for (idx, child) in node.children.into_iter().enumerate() { let collapsible = config.options.compact && is_collapsible(&child); path.push(idx); - let child = allocate_refs_at_path(child, refmap, config, path, skeleton_anchors)?; + let child = allocate_refs_at_path(child, refmap, config, path, plan)?; path.pop(); if collapsible { if let Some(child) = child.children.into_iter().next() { diff --git a/crates/core/src/refs/ref_alloc_anchor_tests.rs b/crates/core/src/refs/ref_alloc_anchor_tests.rs new file mode 100644 index 000000000..a805e7439 --- /dev/null +++ b/crates/core/src/refs/ref_alloc_anchor_tests.rs @@ -0,0 +1,133 @@ +use super::*; +use crate::AccessibilityNode; + +fn boundary(role: &str, name: Option<&str>) -> AccessibilityNode { + let mut node = node(role, name); + node.children_count = Some(1); + node +} + +/// The whole-window observation a snapshot counts before it presents a cut. +fn uncut(mut tree: AccessibilityNode) -> AccessibilityNode { + tree.children_count = None; + tree.children = tree.children.into_iter().map(uncut).collect(); + tree +} + +fn fixture_window() -> crate::WindowInfo { + crate::WindowInfo { + id: "w-1".into(), + title: "Fixture".into(), + app: "Fixture".into(), + pid: crate::ProcessId::new(7), + process_instance: Some("test-instance".into()), + bounds: None, + state: crate::WindowState::default(), + } +} + +fn capture_config<'a>(capture: &'a CaptureProof<'a>) -> RefAllocConfig<'a> { + RefAllocConfig { + capture: Some(capture), + options: crate::refs::ref_alloc_options::RefAllocOptions { + include_bounds: true, + interactive_only: false, + compact: false, + force_renderer_accessibility: false, + }, + source: source(7), + scope: crate::refs::ref_alloc_scope::RefAllocScope { + root_ref_id: None, + path_prefix: &[], + }, + } +} + +fn allocation(tree: AccessibilityNode) -> RefMap { + let observed = crate::adapter::observed_tree( + &crate::locator::live_locator::ObservationRoot::Window(&fixture_window()), + uncut(tree.clone()), + ) + .expect("the fixture observes"); + let proof = CaptureProof::new(&observed); + let mut refmap = RefMap::new(); + allocate_refs(tree, &mut refmap, &capture_config(&proof)).expect("the tree allocates"); + refmap +} + +/// A boundary container is a drill target only when a later resolve can prove +/// it is the element it names, so an unproven twin is never handed out while a +/// boundary the capture found once still is. +#[test] +fn a_skeleton_anchor_is_offered_only_when_the_capture_proves_it() { + let mut root = node("window", Some("w")); + root.children = vec![boundary("group", Some("Sidebar"))]; + assert_eq!(allocation(root).len(), 1, "a boundary found once is a ref"); + + let mut twinned = node("window", Some("w")); + twinned.children = vec![ + boundary("group", Some("Sidebar")), + boundary("group", Some("Sidebar")), + ]; + assert_eq!( + allocation(twinned).len(), + 0, + "a boundary the capture cannot tell from its twin is not offered" + ); +} + +/// Geometry is not identity: a boundary whose only evidence is its bounds is +/// offered only when the census proves its role and text are its own. +#[test] +fn a_bounds_only_boundary_is_a_drill_target_only_when_proven() { + let mut only = node("window", Some("w")); + only.children = vec![boundary("group", None)]; + assert_eq!(allocation(only).len(), 1, "the only group is proven unique"); + + let mut shared = node("window", Some("w")); + shared.children = vec![boundary("group", None), boundary("group", None)]; + assert_eq!( + allocation(shared).len(), + 0, + "two identical boundaries cannot be told apart" + ); +} + +/// A twin the capture cannot tell apart is never offered, so the boundary it +/// would have covered falls to the nearest ancestor the capture does prove. +#[test] +fn a_unique_ancestor_anchors_a_boundary_whose_twin_is_not_offered() { + let mut panel = node("group", Some("Panel")); + panel.children = vec![ + boundary("group", Some("Sidebar")), + boundary("group", Some("Sidebar")), + ]; + let mut root = node("window", Some("w")); + root.children = vec![panel]; + + let refmap = allocation(root); + assert_eq!(refmap.len(), 1, "only the ancestor is a drill target"); + let anchor = refmap.get("@e1").expect("the ancestor holds the first ref"); + assert_eq!(anchor.identity.name.as_deref(), Some("Panel")); + assert!(anchor.identity.unique_at_capture); +} + +/// A ref-able twin at a boundary is refused when drilled, so it covers nothing: +/// the boundary falls to the nearest ancestor the capture proves. +#[test] +fn a_twin_ref_able_boundary_leaves_its_anchor_to_a_proven_ancestor() { + let mut panel = node("group", Some("Panel")); + panel.children = vec![ + boundary("button", Some("More")), + boundary("button", Some("More")), + ]; + let mut root = node("window", Some("w")); + root.children = vec![panel]; + + let refmap = allocation(root); + let anchor = (1..=refmap.len()) + .filter_map(|index| refmap.get(&format!("@e{index}"))) + .find(|entry| entry.identity.name.as_deref() == Some("Panel")) + .expect("the proven ancestor anchors the boundary"); + assert!(anchor.identity.unique_at_capture); +} diff --git a/crates/core/src/refs/ref_alloc_bounds_tests.rs b/crates/core/src/refs/ref_alloc_bounds_tests.rs index bd5ac7ffc..95c85c7e2 100644 --- a/crates/core/src/refs/ref_alloc_bounds_tests.rs +++ b/crates/core/src/refs/ref_alloc_bounds_tests.rs @@ -10,10 +10,12 @@ fn a_nameless_positive_area_ref_allocated_with_bounds_hidden_keeps_its_only_iden root.children = vec![node("button", None)]; let mut refmap = RefMap::new(); let config = RefAllocConfig { + capture: None, options: crate::refs::ref_alloc_options::RefAllocOptions { include_bounds: false, interactive_only: false, compact: false, + force_renderer_accessibility: false, }, source: crate::refs::ref_alloc_source::RefAllocSource { pid: crate::ProcessId::new(7), @@ -56,10 +58,12 @@ fn a_ref_with_its_own_identity_still_has_its_rect_stripped() { root.children = vec![node("button", Some("Open"))]; let mut refmap = RefMap::new(); let config = RefAllocConfig { + capture: None, options: crate::refs::ref_alloc_options::RefAllocOptions { include_bounds: false, interactive_only: false, compact: false, + force_renderer_accessibility: false, }, source: crate::refs::ref_alloc_source::RefAllocSource { pid: crate::ProcessId::new(7), diff --git a/crates/core/src/refs/ref_alloc_config.rs b/crates/core/src/refs/ref_alloc_config.rs index b282d872d..1e9c23eaf 100644 --- a/crates/core/src/refs/ref_alloc_config.rs +++ b/crates/core/src/refs/ref_alloc_config.rs @@ -1,6 +1,6 @@ use crate::{ refs::ref_alloc_options::RefAllocOptions, refs::ref_alloc_scope::RefAllocScope, - refs::ref_alloc_source::RefAllocSource, + refs::ref_alloc_source::RefAllocSource, refs::ref_capture_proof::CaptureProof, }; #[derive(Clone, Copy)] @@ -8,4 +8,5 @@ pub(crate) struct RefAllocConfig<'a> { pub(crate) options: RefAllocOptions, pub(crate) source: RefAllocSource<'a>, pub(crate) scope: RefAllocScope<'a>, + pub(crate) capture: Option<&'a CaptureProof<'a>>, } diff --git a/crates/core/src/refs/ref_alloc_options.rs b/crates/core/src/refs/ref_alloc_options.rs index d2026cfff..c9ab6308b 100644 --- a/crates/core/src/refs/ref_alloc_options.rs +++ b/crates/core/src/refs/ref_alloc_options.rs @@ -1,6 +1,7 @@ -#[derive(Clone, Copy)] +#[derive(Clone, Copy, Default)] pub(crate) struct RefAllocOptions { pub(crate) include_bounds: bool, pub(crate) interactive_only: bool, pub(crate) compact: bool, + pub(crate) force_renderer_accessibility: bool, } diff --git a/crates/core/src/refs/ref_alloc_ordering_tests.rs b/crates/core/src/refs/ref_alloc_ordering_tests.rs index 7c407138a..c0d5a39c7 100644 --- a/crates/core/src/refs/ref_alloc_ordering_tests.rs +++ b/crates/core/src/refs/ref_alloc_ordering_tests.rs @@ -17,10 +17,12 @@ fn allocate_refs_assigns_refs_in_depth_first_order() { let mut refmap = RefMap::new(); let config = RefAllocConfig { + capture: None, options: crate::refs::ref_alloc_options::RefAllocOptions { include_bounds: false, interactive_only: false, compact: false, + force_renderer_accessibility: false, }, source: crate::refs::ref_alloc_source::RefAllocSource { pid: crate::ProcessId::new(1), @@ -97,10 +99,12 @@ fn allocate_refs_keeps_bounds_in_refmap_when_snapshot_includes_bounds() { root.children = vec![node("button", Some("Open"))]; let mut refmap = RefMap::new(); let config = RefAllocConfig { + capture: None, options: crate::refs::ref_alloc_options::RefAllocOptions { include_bounds: true, interactive_only: false, compact: false, + force_renderer_accessibility: false, }, source: crate::refs::ref_alloc_source::RefAllocSource { pid: crate::ProcessId::new(7), @@ -193,10 +197,12 @@ fn structural_allocation_failure_still_aborts_snapshot() { fn allocation_config(include_bounds: bool) -> RefAllocConfig<'static> { RefAllocConfig { + capture: None, options: crate::refs::ref_alloc_options::RefAllocOptions { include_bounds, interactive_only: false, compact: false, + force_renderer_accessibility: false, }, source: crate::refs::ref_alloc_source::RefAllocSource { pid: crate::ProcessId::new(7), diff --git a/crates/core/src/refs/ref_alloc_tests.rs b/crates/core/src/refs/ref_alloc_tests.rs index dda36cff3..d837b8017 100644 --- a/crates/core/src/refs/ref_alloc_tests.rs +++ b/crates/core/src/refs/ref_alloc_tests.rs @@ -266,10 +266,11 @@ fn allocate_refs_skips_web_container_noise() { let mut refmap = RefMap::new(); let config = RefAllocConfig { + capture: None, options: crate::refs::ref_alloc_options::RefAllocOptions { include_bounds: true, interactive_only: true, - compact: false, + ..Default::default() }, source: source(7), scope: crate::refs::ref_alloc_scope::RefAllocScope { @@ -298,10 +299,11 @@ fn allocate_refs_records_structural_paths() { let mut refmap = RefMap::new(); let config = RefAllocConfig { + capture: None, options: crate::refs::ref_alloc_options::RefAllocOptions { include_bounds: true, interactive_only: false, - compact: false, + ..Default::default() }, source: crate::refs::ref_alloc_source::RefAllocSource { pid: crate::ProcessId::new(7), @@ -340,10 +342,11 @@ fn allocate_refs_keeps_bounds_hash_when_snapshot_hides_bounds() { root.children = vec![node("button", Some("Open"))]; let mut refmap = RefMap::new(); let config = RefAllocConfig { + capture: None, options: crate::refs::ref_alloc_options::RefAllocOptions { include_bounds: false, interactive_only: false, - compact: false, + ..Default::default() }, source: crate::refs::ref_alloc_source::RefAllocSource { pid: crate::ProcessId::new(7), @@ -386,6 +389,8 @@ fn entry_hash() -> u64 { .unwrap() } +#[path = "ref_alloc_anchor_tests.rs"] +mod anchor_tests; #[path = "ref_alloc_bounds_tests.rs"] mod bounds_tests; #[path = "ref_alloc_ordering_tests.rs"] diff --git a/crates/core/src/refs/ref_capture_proof.rs b/crates/core/src/refs/ref_capture_proof.rs new file mode 100644 index 000000000..659363748 --- /dev/null +++ b/crates/core/src/refs/ref_capture_proof.rs @@ -0,0 +1,220 @@ +use std::collections::HashMap; +use std::sync::OnceLock; + +use crate::{ + ElementIdentifier, IdentityMatch, RefEntry, + locator::live_locator::{LocatorEvidence, LocatorField, ObservationSource, ObservedTree}, + refs::ref_full_predicate::{DESCRIPTION, FullPredicate, NAME, NATIVE_ID}, + refs::ref_identity::stable_name, + refs::ref_live_predicate::LivePredicate, + tree::accname::non_blank, +}; + +const ROLE: u8 = DESCRIPTION << 1; +const MASKS: usize = (NATIVE_ID | NAME | DESCRIPTION) as usize + 1; + +/// One whole-window observation, counted once into the field subsets its +/// entries record, so every tokenless entry published from it is proven by the +/// rule `proves_node` applies rather than by a fresh scan per entry. +/// +/// The count is the comparison resolution runs: `FullPredicate::of_entry` of the +/// entry being published, measured against the evidence every node reported, so +/// an identifier chosen by a query or a field that went unread counts the way it +/// will count later. +pub(crate) struct CaptureProof<'a> { + observed: &'a ObservedTree, + start: Option, + buckets: [OnceLock, usize>>; MASKS], + unread: Vec<(u8, LivePredicate<'a>)>, + whole_window: bool, +} + +impl<'a> CaptureProof<'a> { + pub(crate) fn new(observed: &'a ObservedTree) -> Self { + let whole_window = + matches!(observed.source, ObservationSource::Window { .. }) && observed.is_exhaustive(); + Self { + observed, + whole_window, + start: observed.roots.first().map(|root| *root as usize), + buckets: std::array::from_fn(|_| OnceLock::new()), + unread: observed + .nodes + .iter() + .filter_map(|node| { + let bits = unread_bits(&node.evidence); + (bits != 0).then(|| (bits, LivePredicate::observed(&node.evidence))) + }) + .collect(), + } + } + + /// Starts allocation paths at the node whose observed path is `prefix`, as a + /// drill does: the prefix is that node's own path, and the rest of an + /// allocation path counts positions in the projected tree below it. + pub(crate) fn rooted_at(mut self, prefix: &[usize]) -> Self { + self.start = self + .observed + .nodes + .iter() + .position(|node| node.path.as_slice() == prefix); + self + } + + /// Whether `entry`, allocated at `path`, is proven. The path is read the way + /// allocation wrote it: the start node's own path, then child positions in + /// the projection, which skips no child the observation kept even when the + /// native indices have gaps. + pub(crate) fn proves(&self, entry: &RefEntry, path: &[usize]) -> bool { + let Some(mut index) = self.start else { + return false; + }; + let Some(prefix) = self + .observed + .nodes + .get(index) + .map(|node| node.path.as_slice()) + else { + return false; + }; + let Some(steps) = path.strip_prefix(prefix) else { + return false; + }; + for step in steps { + match self + .observed + .nodes + .get(index) + .and_then(|node| node.children.get(*step)) + { + Some(child) => index = *child as usize, + None => return false, + } + } + self.proves_node(entry, index) + } + + /// Whether `entry` was proven unique at the node it was built from: the + /// observation must be a complete whole-window read, exactly one of its nodes + /// must match the entry's full predicate, no node may leave a recorded field + /// unread, and that one node must be this one. + pub(crate) fn proves_node(&self, entry: &RefEntry, index: usize) -> bool { + if !self.whole_window { + return false; + } + let predicate = FullPredicate::of_entry(entry); + let fields = predicate.mask(); + if self.buckets[fields as usize] + .get_or_init(|| count_nodes(self.observed, fields)) + .get(&key_of(&predicate)) + != Some(&index) + { + return false; + } + !self.unread.iter().any(|(bits, live)| { + bits & (fields | ROLE) != 0 && predicate.compare(live) == IdentityMatch::Unknown + }) + } +} + +/// The node indices each full predicate of this field subset matched, as a +/// single index per predicate. A predicate two nodes match is stored with the +/// last of them, so it never equals the node an entry was built from. +fn count_nodes(observed: &ObservedTree, fields: u8) -> HashMap, usize> { + let mut counted: HashMap, (usize, usize)> = HashMap::new(); + for (index, node) in observed.nodes.iter().enumerate() { + for key in keys_of(&node.evidence, fields) { + let entry = counted.entry(key).or_insert((0, index)); + entry.0 += 1; + entry.1 = index; + } + } + counted + .into_iter() + .filter(|(_, (count, _))| *count == 1) + .map(|(key, (_, index))| (key, index)) + .collect() +} + +fn keys_of<'a>(evidence: &'a LocatorEvidence, fields: u8) -> Vec> { + let Some(role) = non_blank(evidence.role.known().map(String::as_str)) else { + return Vec::new(); + }; + let name = match fields & NAME { + 0 => None, + _ => match stable_name( + role, + evidence.name.known().map(String::as_str), + evidence.value.known().map(String::as_str), + ) { + Some(name) => Some(name), + None => return Vec::new(), + }, + }; + let description = match fields & DESCRIPTION { + 0 => None, + _ => match non_blank(evidence.description.known().map(String::as_str)) { + Some(description) => Some(description), + None => return Vec::new(), + }, + }; + let key = |native_id: Option<&'a ElementIdentifier>| LiveKey { + role, + name, + description, + native_id, + }; + if fields & NATIVE_ID == 0 { + return vec![key(None)]; + } + evidence + .identifiers + .identifiers() + .iter() + .map(|native_id| key(Some(native_id))) + .collect() +} + +/// The fields this element could not answer, so a predicate that records one of +/// them reaches `Unknown` here exactly as it will at resolution. +fn unread_bits(evidence: &LocatorEvidence) -> u8 { + let live = LivePredicate::observed(evidence); + let mut bits = 0; + match &live.role { + LocatorField::Known(role) => { + if live.stable_name(role).is_unknown() { + bits |= NAME; + } + } + LocatorField::Unknown => bits |= ROLE, + LocatorField::Absent => {} + } + if live.description.is_unknown() { + bits |= DESCRIPTION; + } + if !live.native_ids_complete { + bits |= NATIVE_ID; + } + bits +} + +fn key_of<'b>(predicate: &FullPredicate<'b>) -> LiveKey<'b> { + LiveKey { + role: predicate.role, + name: predicate.name, + description: predicate.description, + native_id: predicate.native_id, + } +} + +#[derive(Clone, Copy, PartialEq, Eq, Hash)] +struct LiveKey<'a> { + role: &'a str, + name: Option<&'a str>, + description: Option<&'a str>, + native_id: Option<&'a ElementIdentifier>, +} + +#[cfg(test)] +#[path = "ref_capture_proof_tests.rs"] +mod tests; diff --git a/crates/core/src/refs/ref_capture_proof_tests.rs b/crates/core/src/refs/ref_capture_proof_tests.rs new file mode 100644 index 000000000..4234fd167 --- /dev/null +++ b/crates/core/src/refs/ref_capture_proof_tests.rs @@ -0,0 +1,342 @@ +use crate::locator::live_locator::{ + IdentifierEvidence, LocatorEvidence, LocatorField, LocatorRefEvidence, LocatorStats, + ObservationRoot, ObservationSource, ObservedSubtree, ObservedTree, +}; +use crate::refs::ref_alloc::allocate_refs; +use crate::refs::ref_alloc_config::RefAllocConfig; +use crate::refs::ref_alloc_options::RefAllocOptions; +use crate::refs::ref_alloc_scope::RefAllocScope; +use crate::refs::ref_alloc_source::RefAllocSource; +use crate::refs::ref_capture_proof::CaptureProof; +use crate::{AccessibilityNode, ElementIdentifier, IdentifierKind, RefMap, WindowInfo}; + +fn named(role: &str, name: &str) -> AccessibilityNode { + crate::adapter::minimal_accessibility_node(role, name) +} + +fn window(children: Vec) -> AccessibilityNode { + let mut root = named("window", "Fixture"); + root.children = children; + root +} + +fn fixture_window() -> WindowInfo { + WindowInfo { + id: "w-1".into(), + title: "Fixture".into(), + app: "Fixture".into(), + pid: crate::ProcessId::new(1), + process_instance: Some("test-instance".into()), + bounds: None, + state: crate::WindowState::default(), + } +} + +fn config<'a>(capture: &'a CaptureProof<'a>) -> RefAllocConfig<'a> { + RefAllocConfig { + options: RefAllocOptions { + include_bounds: false, + interactive_only: false, + compact: false, + force_renderer_accessibility: false, + }, + source: RefAllocSource { + pid: crate::ProcessId::new(1), + app: Some("Test"), + window_id: None, + window_title: Some("Test Window"), + window_bounds_hash: None, + process_instance: Some("test-instance"), + surface: crate::SnapshotSurface::Window, + }, + scope: RefAllocScope { + root_ref_id: None, + path_prefix: &[], + }, + capture: Some(capture), + } +} + +fn flags(root: AccessibilityNode, observed: &ObservedTree) -> Vec { + let proof = CaptureProof::new(observed); + let mut refmap = RefMap::new(); + allocate_refs(root, &mut refmap, &config(&proof)).unwrap(); + (1..=refmap.len()) + .filter_map(|index| refmap.get(&format!("@e{index}"))) + .map(|entry| entry.identity.unique_at_capture) + .collect() +} + +fn observed_from(root: AccessibilityNode) -> ObservedTree { + let window = fixture_window(); + crate::adapter::observed_tree(&ObservationRoot::Window(&window), root) + .expect("the fixture observes") +} + +fn evidence( + role: &str, + name: &str, + ids: Vec, + preferred: usize, +) -> LocatorEvidence { + LocatorEvidence { + role: LocatorField::Known(role.into()), + name: LocatorField::Known(name.into()), + description: LocatorField::Absent, + value: LocatorField::Absent, + identifiers: IdentifierEvidence::typed(ids, Some(preferred), true), + states: LocatorField::Known(Vec::new()), + ref_evidence: LocatorRefEvidence { + window_bounds: LocatorField::Absent, + bounds: LocatorField::Absent, + available_actions: LocatorField::Known(Vec::new()), + descriptors: Default::default(), + }, + } +} + +fn identifier(kind: IdentifierKind, value: &str) -> ElementIdentifier { + ElementIdentifier { + kind, + value: value.into(), + } +} + +fn twin_carrying_the_identifier_secondhand() -> (AccessibilityNode, ObservedTree) { + let mut captured = named("button", "Send"); + captured.identity.native_id = Some(identifier(IdentifierKind::AxIdentifier, "x-1")); + let mut twin = named("button", "Send"); + twin.identity.native_id = Some(identifier(IdentifierKind::AxDomIdentifier, "dom-2")); + + let observed = ObservedTree::from_roots( + vec![ObservedSubtree::new( + evidence("window", "Fixture", Vec::new(), 0), + vec![ + ObservedSubtree::new( + evidence( + "button", + "Send", + vec![identifier(IdentifierKind::AxIdentifier, "x-1")], + 0, + ), + Vec::new(), + true, + None, + ), + ObservedSubtree::new( + evidence( + "button", + "Send", + vec![ + identifier(IdentifierKind::AxDomIdentifier, "dom-2"), + identifier(IdentifierKind::AxIdentifier, "x-1"), + ], + 0, + ), + Vec::new(), + true, + None, + ), + ], + true, + None, + )], + ObservationSource::Window { + window: fixture_window(), + surface: crate::SnapshotSurface::Window, + renderer_accessibility_forced: false, + }, + LocatorStats::default(), + true, + ) + .expect("the fixture observes"); + + (window(vec![captured, twin]), observed) +} + +#[test] +fn a_complete_observation_marks_the_entry_it_found_once() { + let root = window(vec![ + named("button", "Send"), + named("button", "Send"), + named("button", "Cancel"), + ]); + let observed = observed_from(root.clone()); + + assert_eq!(flags(root, &observed), [false, false, true]); +} + +#[test] +fn an_identifier_a_twin_carries_secondhand_is_not_unique() { + let (root, observed) = twin_carrying_the_identifier_secondhand(); + + assert_eq!(flags(root, &observed), [false, true]); +} + +#[test] +fn find_marks_only_a_whole_window_match_found_once() { + let observed = observed_from(window(vec![ + named("button", "Send"), + named("button", "Send"), + named("button", "Cancel"), + ])); + let proof = CaptureProof::new(&observed); + let proven: Vec<_> = observed + .nodes + .iter() + .enumerate() + .filter(|(_, node)| node.evidence.role.known().map(String::as_str) == Some("button")) + .filter(|(index, node)| { + let entry = crate::locator::live_locator::materialize_ref_entry( + node, + &observed.source, + &crate::locator::LocatorQuery::default(), + ); + proof.proves_node(&entry, *index) + }) + .filter_map(|(_, node)| node.evidence.name.meaningful_string()) + .collect(); + assert_eq!(proven, ["Cancel"]); +} + +fn twin_buttons_sharing_a_secondary_identifier() -> ObservedTree { + let shared = || { + vec![ + identifier(IdentifierKind::AxIdentifier, "shared"), + identifier(IdentifierKind::AxDomIdentifier, "dom-a"), + ] + }; + let mut second = shared(); + second[1] = identifier(IdentifierKind::AxDomIdentifier, "dom-b"); + ObservedTree::from_roots( + vec![ObservedSubtree::new( + evidence("window", "Fixture", Vec::new(), 0), + vec![ + ObservedSubtree::new( + evidence("button", "Send", shared(), 1), + Vec::new(), + true, + None, + ), + ObservedSubtree::new( + evidence("button", "Send", second, 1), + Vec::new(), + true, + None, + ), + ], + true, + None, + )], + ObservationSource::Window { + window: fixture_window(), + surface: crate::SnapshotSurface::Window, + renderer_accessibility_forced: false, + }, + LocatorStats::default(), + true, + ) + .expect("the fixture observes") +} + +fn send_query(native_id: &str) -> crate::locator::LocatorQuery { + crate::locator::LocatorQuery { + identity: crate::locator::IdentityPredicate { + role: Some("button".into()), + name: Some("Send".into()), + native_id: Some(native_id.into()), + ..Default::default() + }, + ..Default::default() + } +} + +fn all_matches() -> crate::locator::live_locator::LocatorResolveRequest { + crate::locator::live_locator::LocatorResolveRequest { + selection: crate::locator::live_locator::LocatorSelection::All { limit: Some(64) }, + deadline: crate::Deadline::standard().expect("a standard deadline"), + max_raw_depth: 50, + surface: None, + materialization: crate::locator::live_locator::LocatorMaterialization::None, + } +} + +/// The proof a published entry carries is the predicate it was built with, not +/// the identifier the tree happens to prefer. Two twins that share a secondary +/// identifier both record it when the query names it, so neither may claim to +/// have been found once; the twin the query names by its preferred identifier is +/// the only one the read can prove. +#[test] +fn a_published_entry_is_proven_by_its_own_predicate_not_the_preferred_identifier() { + let observed = twin_buttons_sharing_a_secondary_identifier(); + + let shared = crate::locator::live_locator::evaluate_locator_tree( + observed.clone(), + &send_query("shared"), + &all_matches(), + ) + .expect("the observation evaluates"); + assert_eq!(shared.matches.len(), 2); + for matched in &shared.matches { + assert_eq!( + matched + .entry + .identity + .native_id + .as_ref() + .map(|id| id.value.as_str()), + Some("shared") + ); + assert!( + !matched.entry.identity.unique_at_capture, + "a predicate both twins carry was not found once" + ); + } + + let named = crate::locator::live_locator::evaluate_locator_tree( + observed, + &send_query("dom-b"), + &all_matches(), + ) + .expect("the observation evaluates"); + assert_eq!(named.matches.len(), 1); + assert!(named.matches[0].entry.identity.unique_at_capture); +} + +#[test] +fn a_skipped_native_sibling_does_not_unprove_the_refs_after_it() { + let observed = ObservedTree::from_roots( + vec![ObservedSubtree::new( + evidence("window", "Fixture", Vec::new(), 0), + vec![ + ObservedSubtree::new( + evidence("button", "Cancel", Vec::new(), 1), + Vec::new(), + true, + None, + ) + .with_source_child_index(0), + ObservedSubtree::new( + evidence("button", "Save", Vec::new(), 1), + Vec::new(), + true, + None, + ) + .with_source_child_index(2), + ], + true, + None, + )], + ObservationSource::Window { + window: fixture_window(), + surface: crate::SnapshotSurface::Window, + renderer_accessibility_forced: false, + }, + LocatorStats::default(), + true, + ) + .expect("the fixture observes"); + let root = window(vec![named("button", "Cancel"), named("button", "Save")]); + + assert_eq!(flags(root, &observed), [true, true]); +} diff --git a/crates/core/src/refs/ref_entry_identity.rs b/crates/core/src/refs/ref_entry_identity.rs index 9c6054cc4..b5a4cef81 100644 --- a/crates/core/src/refs/ref_entry_identity.rs +++ b/crates/core/src/refs/ref_entry_identity.rs @@ -1,6 +1,6 @@ use serde::{Deserialize, Serialize}; -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[derive(Debug, Clone, PartialEq, Default, Serialize, Deserialize)] pub struct RefEntryIdentity { #[serde(default, skip_serializing_if = "Option::is_none")] pub retained_object: Option, @@ -12,4 +12,8 @@ pub struct RefEntryIdentity { pub description: Option, #[serde(default, skip_serializing_if = "Option::is_none")] pub native_id: Option, + /// Set when the capture read complete evidence and exactly one observed + /// element matched this entry's full identity predicate. + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub unique_at_capture: bool, } diff --git a/crates/core/src/refs/ref_full_predicate.rs b/crates/core/src/refs/ref_full_predicate.rs new file mode 100644 index 000000000..196e31d83 --- /dev/null +++ b/crates/core/src/refs/ref_full_predicate.rs @@ -0,0 +1,118 @@ +use crate::{ + ElementIdentifier, IdentityMatch, RefEntry, + locator::live_locator::LocatorField, + refs::ref_identity::{equality_match, meaningful_field, stable_name, stable_name_field}, + refs::ref_live_predicate::LivePredicate, + tree::accname::non_blank, +}; + +pub(crate) const NATIVE_ID: u8 = 0b001; +pub(crate) const NAME: u8 = 0b010; +pub(crate) const DESCRIPTION: u8 = 0b100; + +/// The identity fields capture and resolution both compare: role plus whichever +/// of native identifier, stable name and description an element records. +/// Mutable values and bounds are never identity. +#[derive(Clone, Copy)] +pub(crate) struct FullPredicate<'a> { + pub(crate) role: &'a str, + pub(crate) native_id: Option<&'a ElementIdentifier>, + pub(crate) name: Option<&'a str>, + pub(crate) description: Option<&'a str>, + mask: u8, +} + +impl<'a> FullPredicate<'a> { + pub(crate) fn of_entry(entry: &'a RefEntry) -> Self { + Self::of_fields( + &entry.identity.role, + entry.identity.name.as_deref(), + entry.identity.value.as_deref(), + entry.identity.description.as_deref(), + entry.identity.native_id.as_ref(), + ) + } + + fn of_fields( + role: &'a str, + name: Option<&'a str>, + value: Option<&'a str>, + description: Option<&'a str>, + native_id: Option<&'a ElementIdentifier>, + ) -> Self { + Self { + role, + native_id: native_id.filter(|identifier| !identifier.value.trim().is_empty()), + name: stable_name(role, name, value), + description: non_blank(description), + mask: 0, + } + .recorded() + } + + /// The one rule capture and resolution apply: every field the predicate + /// records must equal the live element's, and a field the live read could + /// not answer is unknown instead of matching. + pub(crate) fn compare(&self, live: &LivePredicate<'_>) -> IdentityMatch { + [ + field_match(Some(self.role), &live.role), + self.native_id_match(live), + field_match( + self.name, + &stable_name_field(self.role, &live.name, &live.value), + ), + field_match(self.description, &meaningful_field(&live.description)), + ] + .into_iter() + .fold(IdentityMatch::Match, conjunction) + } + + fn native_id_match(&self, live: &LivePredicate<'_>) -> IdentityMatch { + let Some(recorded) = self.native_id else { + return IdentityMatch::Match; + }; + if live.native_ids.contains(recorded) { + IdentityMatch::Match + } else if live.native_ids_complete { + IdentityMatch::NoMatch + } else { + IdentityMatch::Unknown + } + } + + /// The fields this predicate records, which is the field subset a capture + /// proof counts this predicate against. + pub(crate) fn mask(&self) -> u8 { + self.mask + } + + fn recorded(mut self) -> Self { + self.mask = bit(self.native_id.is_some(), NATIVE_ID) + | bit(self.name.is_some(), NAME) + | bit(self.description.is_some(), DESCRIPTION); + self + } +} + +fn bit(recorded: bool, field: u8) -> u8 { + if recorded { field } else { 0 } +} + +fn field_match(expected: Option<&str>, actual: &LocatorField<&str>) -> IdentityMatch { + match expected { + Some(expected) => match actual { + LocatorField::Known(actual) => equality_match(expected, actual), + LocatorField::Absent => IdentityMatch::NoMatch, + LocatorField::Unknown => IdentityMatch::Unknown, + }, + None => IdentityMatch::Match, + } +} + +fn conjunction(left: IdentityMatch, right: IdentityMatch) -> IdentityMatch { + match (left, right) { + (IdentityMatch::NoMatch, _) | (_, IdentityMatch::NoMatch) => IdentityMatch::NoMatch, + (IdentityMatch::Unknown, _) | (_, IdentityMatch::Unknown) => IdentityMatch::Unknown, + _ => IdentityMatch::Match, + } +} diff --git a/crates/core/src/refs/ref_identity.rs b/crates/core/src/refs/ref_identity.rs index f1a7c0ddd..a9e4ec6e9 100644 --- a/crates/core/src/refs/ref_identity.rs +++ b/crates/core/src/refs/ref_identity.rs @@ -5,13 +5,35 @@ use crate::{ tree::roles::is_mutable_value_role, }; +const ELEMENT_TOKEN_PREFIX: &str = "axtok1:"; + +/// The retained object that is a genuine element token. A platform that keeps +/// only a positional path or a runtime id retains a location, not a handle, so +/// core never treats that value as identity evidence. +pub fn element_token(entry: &RefEntry) -> Option<&str> { + entry + .identity + .retained_object + .as_deref() + .filter(|token| is_element_token(token)) +} + +/// Whether a retained object value is a genuine element token rather than a +/// retained location. +pub fn is_element_token(token: &str) -> bool { + token.starts_with(ELEMENT_TOKEN_PREFIX) +} + pub fn has_meaningful_identity(entry: &RefEntry) -> bool { - if entry.identity.retained_object.is_some() { - return true; - } - entry.identity.native_id.as_ref().is_some_and(|identifier| { + element_token(entry).is_some() + || recorded_native_id(entry).is_some() + || has_stable_text_identity(entry) +} + +fn recorded_native_id(entry: &RefEntry) -> Option<&crate::ElementIdentifier> { + entry.identity.native_id.as_ref().filter(|identifier| { crate::tree::accname::non_blank(Some(identifier.value.as_str())).is_some() - }) || has_stable_text_identity(entry) + }) } pub fn has_stable_text_identity(entry: &RefEntry) -> bool { @@ -54,7 +76,7 @@ pub fn identity_match( actual_description: &LocatorField, actual_identifiers: &IdentifierEvidence, ) -> IdentityMatch { - if let Some(expected) = entry.identity.retained_object.as_deref() { + if let Some(expected) = element_token(entry) { return match ( crate::tree::accname::non_blank(Some(expected)), actual_identifiers.retained_object(), @@ -63,27 +85,34 @@ pub fn identity_match( _ => IdentityMatch::Unknown, }; } - if let Some(expected) = entry.identity.native_id.as_ref() { - let Some(expected_value) = crate::tree::accname::non_blank(Some(expected.value.as_str())) - else { - return IdentityMatch::Unknown; - }; - if actual_identifiers - .identifiers() - .iter() - .any(|actual| actual.kind == expected.kind && actual.value == expected_value) - { - return IdentityMatch::Match; - } - if !actual_identifiers.is_complete() { - return IdentityMatch::Unknown; - } - return IdentityMatch::NoMatch; + if entry.identity.native_id.is_some() { + return native_id_match(entry, actual_identifiers); } stable_text_match(entry, actual_name, actual_value, actual_description) } +fn native_id_match(entry: &RefEntry, actual_identifiers: &IdentifierEvidence) -> IdentityMatch { + let Some(expected) = entry.identity.native_id.as_ref() else { + return IdentityMatch::Match; + }; + let Some(expected_value) = crate::tree::accname::non_blank(Some(expected.value.as_str())) + else { + return IdentityMatch::Unknown; + }; + if actual_identifiers + .identifiers() + .iter() + .any(|actual| actual.kind == expected.kind && actual.value == expected_value) + { + return IdentityMatch::Match; + } + if !actual_identifiers.is_complete() { + return IdentityMatch::Unknown; + } + IdentityMatch::NoMatch +} + #[cfg(test)] pub(crate) fn identity_matches( entry: &RefEntry, @@ -117,11 +146,11 @@ pub(crate) fn identity_matches( /// Whether one element's stable text carves a live identity that can refute /// or corroborate the stored one. /// -/// Public because the Windows resolver composes it with `identity_match` to -/// keep its AutomationId-first corroboration rule: an id that still resolves +/// Public because the Linux resolver composes it with `identity_match` to keep +/// its identifier-first corroboration rule: an identifier that still resolves /// onto a candidate whose stable text drifted must be refuted, the A7-3 /// wrong-target shape. macOS consumes only `identity_match`, which already -/// routes id-less entries here; Windows runs the corroboration tier itself. +/// routes id-less entries here; Linux runs the corroboration tier itself. pub fn stable_text_match( entry: &RefEntry, actual_name: &LocatorField, @@ -174,7 +203,7 @@ fn match_primary_identity( } } -fn equality_match(expected: &str, actual: &str) -> IdentityMatch { +pub(crate) fn equality_match(expected: &str, actual: &str) -> IdentityMatch { if actual == expected { IdentityMatch::Match } else { @@ -204,9 +233,9 @@ fn option_field(value: Option<&str>) -> LocatorField { .unwrap_or(LocatorField::Absent) } -fn meaningful_field(field: &LocatorField) -> LocatorField<&str> { +pub(crate) fn meaningful_field>(field: &LocatorField) -> LocatorField<&str> { match field { - LocatorField::Known(value) => crate::tree::accname::non_blank(Some(value.as_str())) + LocatorField::Known(value) => crate::tree::accname::non_blank(Some(value.as_ref())) .map(LocatorField::Known) .unwrap_or(LocatorField::Absent), LocatorField::Absent => LocatorField::Absent, @@ -214,10 +243,10 @@ fn meaningful_field(field: &LocatorField) -> LocatorField<&str> { } } -fn stable_name_field<'a>( +pub(crate) fn stable_name_field<'a, T: AsRef>( role: &str, - name: &'a LocatorField, - value: &LocatorField, + name: &'a LocatorField, + value: &LocatorField, ) -> LocatorField<&'a str> { let name = meaningful_field(name); if !is_mutable_value_role(role) { @@ -241,7 +270,11 @@ fn stable_value_field<'a>(role: &str, value: &'a LocatorField) -> Locato } } -fn stable_name<'a>(role: &str, name: Option<&'a str>, value: Option<&str>) -> Option<&'a str> { +pub(crate) fn stable_name<'a>( + role: &str, + name: Option<&'a str>, + value: Option<&str>, +) -> Option<&'a str> { let name = crate::tree::accname::non_blank(name)?; if is_mutable_value_role(role) && value_matches_name(crate::tree::accname::non_blank(value), name) diff --git a/crates/core/src/refs/ref_identity_tests.rs b/crates/core/src/refs/ref_identity_tests.rs index b9c463648..6597599e0 100644 --- a/crates/core/src/refs/ref_identity_tests.rs +++ b/crates/core/src/refs/ref_identity_tests.rs @@ -32,6 +32,7 @@ fn entry() -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -48,6 +49,7 @@ fn entry() -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/refs/ref_live_predicate.rs b/crates/core/src/refs/ref_live_predicate.rs new file mode 100644 index 000000000..681c5dad6 --- /dev/null +++ b/crates/core/src/refs/ref_live_predicate.rs @@ -0,0 +1,34 @@ +use crate::{ + ElementIdentifier, + locator::live_locator::{LocatorEvidence, LocatorField}, + refs::ref_identity::{meaningful_field, stable_name_field}, +}; + +/// What a live element reports for the fields a predicate records, with the +/// identifier read's completeness kept so an unread field answers `Unknown`. +pub(crate) struct LivePredicate<'a> { + pub(crate) role: LocatorField<&'a str>, + pub(crate) name: LocatorField<&'a str>, + pub(crate) value: LocatorField<&'a str>, + pub(crate) description: LocatorField<&'a str>, + pub(crate) native_ids: &'a [ElementIdentifier], + pub(crate) native_ids_complete: bool, +} + +impl<'a> LivePredicate<'a> { + pub(crate) fn observed(evidence: &'a LocatorEvidence) -> Self { + Self { + role: meaningful_field(&evidence.role), + name: meaningful_field(&evidence.name), + value: meaningful_field(&evidence.value), + description: meaningful_field(&evidence.description), + native_ids: evidence.identifiers.identifiers(), + native_ids_complete: evidence.identifiers.is_complete(), + } + } + + /// The stable name a predicate carrying `role` compares this element by. + pub(crate) fn stable_name(&self, role: &str) -> LocatorField<&str> { + stable_name_field(role, &self.name, &self.value) + } +} diff --git a/crates/core/src/refs/ref_resolve_deadline.rs b/crates/core/src/refs/ref_resolve_deadline.rs index 5c932b6fa..843d12112 100644 --- a/crates/core/src/refs/ref_resolve_deadline.rs +++ b/crates/core/src/refs/ref_resolve_deadline.rs @@ -3,7 +3,7 @@ use std::time::Duration; use serde_json::json; use crate::{ - AdapterError, Deadline, ErrorCode, PlatformAdapter, RefEntry, + AdapterError, Deadline, ErrorCode, NativeHandle, PlatformAdapter, RefEntry, refs::resolve_attempt_outcome::ResolveAttemptOutcome, }; @@ -20,11 +20,34 @@ pub(crate) fn resolve_within_deadline( adapter: &dyn PlatformAdapter, entry: &RefEntry, deadline: Deadline, + lease: Option<&crate::InteractionLease>, +) -> ResolveAttemptOutcome { + attempt_within_deadline(deadline, |slice| { + crate::refs::resolve_saved_ref::resolve_saved_ref(adapter, entry, slice, lease) + }) +} + +/// Runs one provisional landing attempt under the same slice as +/// [`resolve_within_deadline`]. Its handle is fit only for a wait that a full +/// resolution under the interaction lease follows before anything is sent. +pub(crate) fn land_within_deadline( + adapter: &dyn PlatformAdapter, + entry: &RefEntry, + deadline: Deadline, +) -> ResolveAttemptOutcome { + attempt_within_deadline(deadline, |slice| { + crate::refs::resolve_saved_ref::land_saved_ref(adapter, entry, slice) + }) +} + +fn attempt_within_deadline( + deadline: Deadline, + attempt: impl FnOnce(Deadline) -> Result, ) -> ResolveAttemptOutcome { if deadline.remaining_slice(RESOLVE_ATTEMPT).is_err() { return ResolveAttemptOutcome::DeadlinePassed; } - match adapter.resolve_element_strict(entry, deadline.capped(RESOLVE_ATTEMPT)) { + match attempt(deadline.capped(RESOLVE_ATTEMPT)) { Ok(_) if deadline.is_expired() => ResolveAttemptOutcome::DeadlinePassed, Ok(handle) => ResolveAttemptOutcome::Resolved(handle), Err(error) => classify_error(error, deadline), diff --git a/crates/core/src/refs/ref_root_read.rs b/crates/core/src/refs/ref_root_read.rs new file mode 100644 index 000000000..be94b1fed --- /dev/null +++ b/crates/core/src/refs/ref_root_read.rs @@ -0,0 +1,38 @@ +use crate::{ + AppError, Deadline, PlatformAdapter, RefEntry, WindowInfo, + adapter::NativeHandle, + refs::ref_identity::element_token, + refs::resolve_saved_ref::{SavedRefMatch, observe_saved_ref, recorded_window}, +}; + +/// How a drill-down or a rooted find read its root: the root's subtree alone, +/// or one complete observation of the root's window with the root's node in it. +pub(crate) enum RootRead { + Subtree(T), + Window(Box), +} + +/// Reads the root a saved ref names. A root carrying an element token is read +/// as its subtree alone through `read_subtree`, and that read stands only when +/// `stands_alone` accepts it: a token re-identifies itself, but a tokenless ref +/// the subtree would publish needs the window-wide capture proof. Every other +/// case locates the root in a complete whole-window observation. +pub(crate) fn read_saved_root( + adapter: &dyn PlatformAdapter, + entry: &RefEntry, + deadline: Deadline, + read_subtree: impl FnOnce(&WindowInfo, &NativeHandle) -> Result, + stands_alone: impl FnOnce(&T) -> bool, +) -> Result, AppError> { + if element_token(entry).is_some() { + let window = recorded_window(adapter, entry, deadline)?; + let handle = adapter.resolve_element_strict(entry, deadline)?; + let subtree = read_subtree(&window, &handle)?; + if stands_alone(&subtree) { + return Ok(RootRead::Subtree(subtree)); + } + } + Ok(RootRead::Window(Box::new(observe_saved_ref( + adapter, entry, deadline, None, false, + )?))) +} diff --git a/crates/core/src/refs/ref_skeleton_anchors.rs b/crates/core/src/refs/ref_skeleton_anchors.rs new file mode 100644 index 000000000..52fe0aaab --- /dev/null +++ b/crates/core/src/refs/ref_skeleton_anchors.rs @@ -0,0 +1,71 @@ +use crate::AccessibilityNode; +use crate::refs::ref_alloc::is_ref_able; +use crate::refs::ref_identity::is_element_token; +use std::collections::HashSet; + +/// Plans the boundary containers that anchor a skeleton, deepest first. A +/// boundary is covered by a ref-able node, or becomes an anchor when it can +/// hold a ref and carries identity evidence, only when `offered` accepts that +/// node; otherwise the uncovered boundary passes to the parent, so the nearest +/// accepted ancestor anchors it instead. Returns whether a boundary below +/// `node` is still uncovered. +pub(crate) fn plan_skeleton_anchors( + node: &AccessibilityNode, + path: &mut Vec, + anchors: &mut HashSet>, + offered: &dyn Fn(&AccessibilityNode, &[usize]) -> bool, +) -> bool { + let mut uncovered_boundary = node.children_count.is_some(); + for (index, child) in node.children.iter().enumerate() { + path.push(index); + uncovered_boundary |= plan_skeleton_anchors(child, path, anchors, offered); + path.pop(); + } + if !uncovered_boundary { + return false; + } + if can_receive_observed_ref(node) && is_ref_able(node) && offered(node, path.as_slice()) { + return false; + } + if !path.is_empty() + && can_receive_observed_ref(node) + && carries_identity_evidence(node) + && offered(node, path.as_slice()) + { + anchors.insert(path.clone()); + return false; + } + true +} + +/// Whether a boundary container carries anything a census could prove it by: an +/// element token, a recorded name, description or native identifier, or bounds. +/// Without one it is no drill target, so it earns neither a ref nor a census. +fn carries_identity_evidence(node: &AccessibilityNode) -> bool { + carries_element_token(node) + || has_meaningful_text(node.identity.name.as_deref()) + || has_meaningful_text(node.identity.description.as_deref()) + || node.identity.native_id.as_ref().is_some_and(|identifier| { + identifier.kind != crate::IdentifierKind::Unknown && !identifier.value.trim().is_empty() + }) + || node + .presentation + .bounds + .and_then(|bounds| bounds.bounds_hash()) + .is_some() +} + +fn has_meaningful_text(value: Option<&str>) -> bool { + value.is_some_and(|text| !text.trim().is_empty()) +} + +pub(crate) fn carries_element_token(node: &AccessibilityNode) -> bool { + node.identity + .retained_object + .as_deref() + .is_some_and(is_element_token) +} + +fn can_receive_observed_ref(node: &AccessibilityNode) -> bool { + crate::Role::is_canonical(&node.role) && node.role != "unknown" +} diff --git a/crates/core/src/refs/ref_source.rs b/crates/core/src/refs/ref_source.rs index e3ce1a037..43465df36 100644 --- a/crates/core/src/refs/ref_source.rs +++ b/crates/core/src/refs/ref_source.rs @@ -1,6 +1,6 @@ use serde::{Deserialize, Serialize}; -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] pub struct RefSource { pub source_app: Option, #[serde(default, skip_serializing_if = "Option::is_none")] @@ -11,4 +11,8 @@ pub struct RefSource { pub source_window_bounds_hash: Option, #[serde(default, skip_serializing_if = "crate::SnapshotSurface::is_window")] pub source_surface: crate::SnapshotSurface, + /// Whether the capture read the window with Chromium renderer accessibility + /// forced, so resolution reads it the same way and counts the same evidence. + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub renderer_accessibility_forced: bool, } diff --git a/crates/core/src/refs/refs_serde_tests.rs b/crates/core/src/refs/refs_serde_tests.rs index 2f1dca806..3b26f8ce1 100644 --- a/crates/core/src/refs/refs_serde_tests.rs +++ b/crates/core/src/refs/refs_serde_tests.rs @@ -14,6 +14,7 @@ fn minimal_entry(role: &str) -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -30,6 +31,7 @@ fn minimal_entry(role: &str) -> RefEntry { source_window_title: None, source_window_bounds_hash: Some(0xA11C_E551), source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -190,6 +192,7 @@ fn ref_entry_full_roundtrip_preserves_all_fields() { kind: crate::IdentifierKind::AxIdentifier, value: "email-field".into(), }), + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -206,6 +209,7 @@ fn ref_entry_full_roundtrip_preserves_all_fields() { source_window_title: Some("Compose".into()), source_window_bounds_hash: None, source_surface: SnapshotSurface::Sheet, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: Some("@e5".into()), diff --git a/crates/core/src/refs/refs_store_tests.rs b/crates/core/src/refs/refs_store_tests.rs index cc0f7fd12..f5de66df5 100644 --- a/crates/core/src/refs/refs_store_tests.rs +++ b/crates/core/src/refs/refs_store_tests.rs @@ -24,6 +24,7 @@ fn entry(name: &str) -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -40,6 +41,7 @@ fn entry(name: &str) -> RefEntry { source_window_title: Some("Test Window".into()), source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/refs/refs_store_transaction_tests.rs b/crates/core/src/refs/refs_store_transaction_tests.rs index 8e352e04d..178b2c9db 100644 --- a/crates/core/src/refs/refs_store_transaction_tests.rs +++ b/crates/core/src/refs/refs_store_transaction_tests.rs @@ -14,6 +14,7 @@ fn entry(name: &str, root_ref: Option<&str>) -> crate::RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: RefGeometry { bounds_space: None, @@ -30,6 +31,7 @@ fn entry(name: &str, root_ref: Option<&str>) -> crate::RefEntry { source_window_title: Some("Fixture".into()), source_window_bounds_hash: None, source_surface: crate::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: RefScope { root_ref: root_ref.map(str::to_string), diff --git a/crates/core/src/refs/refs_tests.rs b/crates/core/src/refs/refs_tests.rs index eb7980e5a..7a3794bb2 100644 --- a/crates/core/src/refs/refs_tests.rs +++ b/crates/core/src/refs/refs_tests.rs @@ -13,6 +13,7 @@ pub(super) fn entry(role: &str, name: Option<&str>) -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -29,6 +30,7 @@ pub(super) fn entry(role: &str, name: Option<&str>) -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -170,6 +172,63 @@ fn drill_replaces_full_snapshot_and_nested_descendants_only() { } } +#[test] +fn a_nested_drill_is_pruned_through_ownership_after_its_root_moved() { + let mut map = RefMap::new(); + let mut allocate = |path: &[usize], owner: Option<&str>| { + let mut value = entry("button", Some("Target")); + value.scope.path = path.iter().copied().collect(); + value.scope.root_ref = owner.map(str::to_string); + value.scope.path_is_absolute = owner.is_some(); + map.allocate(value) + }; + let root = allocate(&[0, 0, 0], None); + let nested_root = allocate(&[9, 1, 4], Some(&root)); + let nested_child = allocate(&[9, 1, 4, 0], Some(&nested_root)); + let sibling = allocate(&[1], None); + map.remove_by_root_ref(&root); + assert!(map.get(&root).is_some()); + assert!(map.get(&sibling).is_some()); + for removed in [nested_root, nested_child] { + assert!( + map.get(&removed).is_none(), + "a nested drill written in another capture's paths was left behind: {removed}" + ); + } +} + +#[test] +fn a_drill_keeps_the_entries_of_another_capture_whose_paths_enclose_it() { + let mut map = RefMap::new(); + let mut allocate = |path: &[usize], owner: Option<&str>| { + let mut value = entry("button", Some("Target")); + value.scope.path = path.iter().copied().collect(); + value.scope.root_ref = owner.map(str::to_string); + value.scope.path_is_absolute = owner.is_some(); + map.allocate(value) + }; + let original = allocate(&[0, 0, 0], None); + let drilled = allocate(&[0, 0], Some(&original)); + let neighbouring = allocate(&[1, 1], None); + let neighbouring_drill = allocate(&[0, 0, 0, 1], Some(&neighbouring)); + + map.remove_by_root_ref(&drilled); + + assert!( + map.get(&original).is_some(), + "an entry of the window walk is not this drill's to remove" + ); + assert!( + map.get(&neighbouring_drill).is_some(), + "another region's drill ref is not this drill's to remove" + ); + assert!( + map.get(&drilled).is_some(), + "the selected root must survive" + ); + assert!(map.get(&neighbouring).is_some()); +} + #[test] fn test_save_load_roundtrip_with_home_override() { let _guard = HomeGuard::new(); @@ -192,6 +251,7 @@ fn test_save_load_roundtrip_with_home_override() { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -208,6 +268,7 @@ fn test_save_load_roundtrip_with_home_override() { source_window_title: Some("Test Window".into()), source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, diff --git a/crates/core/src/refs/resolve_errors.rs b/crates/core/src/refs/resolve_errors.rs index 24047a602..00d7ea172 100644 --- a/crates/core/src/refs/resolve_errors.rs +++ b/crates/core/src/refs/resolve_errors.rs @@ -23,6 +23,26 @@ pub fn identity_unknown_error(entry: &RefEntry) -> AdapterError { })) } +/// A tokenless ref that was not proven unique when it was captured. +pub(crate) fn identity_not_unique(entry: &RefEntry) -> AdapterError { + AdapterError::new( + ErrorCode::AmbiguousTarget, + "Saved ref was not proven unique when it was captured, so it cannot be re-identified safely", + ) + .with_suggestion( + "Either another element in the window matched it, or the window could not be read \ + whole within the snapshot timeout. For a large window, retake the snapshot with a \ + larger --timeout-ms. Identical twins are refused again by every snapshot; where \ + headed pointer input is available, an explicit --headed coordinate click can pick one.", + ) + .with_details(json!({ + "kind": "identity_not_unique", + "retryable": false, + "role": entry.identity.role, + })) + .with_disposition(crate::DeliverySemantics::not_delivered()) +} + /// Stamps the final incomplete diagnosis with `deadline_elapsed` so the /// caller sees why the retries ran out, preserving the incomplete's own /// details rather than discarding them for a bare `TIMEOUT`. Shared by both diff --git a/crates/core/src/refs/resolve_saved_ref.rs b/crates/core/src/refs/resolve_saved_ref.rs new file mode 100644 index 000000000..4d15f8340 --- /dev/null +++ b/crates/core/src/refs/resolve_saved_ref.rs @@ -0,0 +1,398 @@ +use serde_json::json; + +use crate::{ + AdapterError, AppError, Deadline, IdentityMatch, PlatformAdapter, RefEntry, WindowInfo, + adapter::{NativeHandle, TreeOptions}, + locator::live_locator::{ + ObservationRequest, ObservationRoot, ObservedTree, materialize_ref_entry, + }, + refs::ref_full_predicate::FullPredicate, + refs::ref_identity::element_token, + refs::ref_live_predicate::LivePredicate, + refs::resolve_errors::{identity_not_unique, identity_unknown_error}, +}; + +/// The depth a whole-window observation reaches: the limit a snapshot walk +/// supports, so a ref taken from the deepest snapshot node is inside the scope +/// it is captured from and resolved against. +pub(crate) const WHOLE_WINDOW_DEPTH: u8 = 50; + +/// Resolves a saved ref under the reviewed identity rule. +/// +/// A genuine retained element token resolves through the adapter, which owns +/// re-identification on that platform. Every other entry resolves only when it +/// was unique at capture and exactly one live element still matches its full +/// recorded predicate over complete evidence of the window it was captured +/// from; anything less is refused instead of guessed at. +pub fn resolve_saved_ref( + adapter: &dyn PlatformAdapter, + entry: &RefEntry, + deadline: Deadline, + lease: Option<&crate::InteractionLease>, +) -> Result { + if element_token(entry).is_some() { + return adapter.resolve_element_strict(entry, deadline); + } + let (_, observed, found) = observe_saved_ref(adapter, entry, deadline, lease, true)?; + let anchor = positional_anchor(entry, &observed, found)?; + let handle = adapter.resolve_locator_anchor(&anchor, deadline)?; + let root = ObservationRoot::Element { + handle: &handle, + entry: &anchor, + root_ref: None, + }; + confirm_landed(adapter, entry, root, deadline, lease)?; + Ok(handle) +} + +/// Lands a tokenless saved ref at its recorded path and confirms the landed +/// element against the full predicate, skipping the whole-window count. +/// +/// The count is what proves the ref still unique, so this handle may only feed +/// a wait that runs a full [`resolve_saved_ref`] under the interaction lease +/// before acting, or a presentation that never acts, such as placing the cursor +/// overlay. A landing that fails or does not confirm falls back to the full +/// resolution, so a moved, missing or twinned element gets the same verdict +/// either way and only the common case is cheaper. +pub(crate) fn land_saved_ref( + adapter: &dyn PlatformAdapter, + entry: &RefEntry, + deadline: Deadline, +) -> Result { + if element_token(entry).is_none() + && entry.identity.unique_at_capture + && let Ok(handle) = adapter.resolve_locator_anchor(entry, deadline) + { + let root = ObservationRoot::Element { + handle: &handle, + entry, + root_ref: None, + }; + if confirm_landed(adapter, entry, root, deadline, None).is_ok() { + return Ok(handle); + } + } + resolve_saved_ref(adapter, entry, deadline, None) +} + +/// The window a saved ref is captured from, one complete observation of it, and +/// the node in that observation it names. +pub(crate) type SavedRefMatch = (WindowInfo, ObservedTree, usize); + +/// The evidence every tokenless ref is judged by, shared by resolution, by a +/// drill-down and by a rooted find, so a capture proof and a later resolve +/// measure one observation. A ref carrying an element token names its node by +/// that token; every other ref names it by the predicate it was capture-proven +/// with, and an entry that was not unique at capture has no such evidence to +/// gather. `identity_only` marks a read that only resolves and publishes nothing. +pub(crate) fn observe_saved_ref( + adapter: &dyn PlatformAdapter, + entry: &RefEntry, + deadline: Deadline, + lease: Option<&crate::InteractionLease>, + identity_only: bool, +) -> Result { + let token = element_token(entry); + if token.is_none() && !entry.identity.unique_at_capture { + return Err(identity_not_unique(entry)); + } + let window = recorded_window(adapter, entry, deadline)?; + let base = TreeOptions { + surface: entry.source.source_surface, + force_renderer_accessibility: entry.source.renderer_accessibility_forced, + ..TreeOptions::default() + }; + let mut request = whole_window_request(&base, deadline)?; + request.observation_mode.identity_only = identity_only; + let observed = observe_window(adapter, &request, &window, lease)?; + let index = saved_ref_node(entry, token, &observed)?; + let node = observed + .nodes + .get(index) + .ok_or_else(|| AdapterError::internal("saved ref match is missing from the observation"))?; + adapter.validate_saved_ref_context(entry, node.evidence.identifiers.retained_object())?; + Ok((window, observed, index)) +} + +/// Confirms that the element an adapter landed is the one `entry` names, under +/// the same full predicate a saved ref resolves by. A caller that publishes a +/// landed handle - the C find entry point hands one to C callers - validates it +/// here instead of trusting the landing alone. +pub fn confirm_landed_element( + adapter: &dyn PlatformAdapter, + entry: &RefEntry, + handle: &NativeHandle, + deadline: Deadline, +) -> Result<(), AdapterError> { + let root = ObservationRoot::Element { + handle, + entry, + root_ref: None, + }; + confirm_landed(adapter, entry, root, deadline, None) +} + +/// The anchor lands a stored path, which a different element can move into +/// between the live count and the descent, so the handle is re-identified +/// before it is handed back: only the recorded predicate proves the landing. +fn confirm_landed( + adapter: &dyn PlatformAdapter, + entry: &RefEntry, + root: ObservationRoot<'_>, + deadline: Deadline, + lease: Option<&crate::InteractionLease>, +) -> Result<(), AdapterError> { + let options = TreeOptions { + max_depth: 0, + surface: entry.source.source_surface, + ..TreeOptions::default() + }; + let request = ObservationRequest::snapshot(&options, deadline).validate()?; + let observed = + crate::tree::renderer_accessibility::observe_tree(adapter, root, &request, lease) + .map_err(adapter_error)?; + let landed = observed + .roots + .first() + .and_then(|index| observed.nodes.get(*index as usize)) + .ok_or_else(|| AdapterError::internal("the landed element was not observed"))?; + match FullPredicate::of_entry(entry).compare(&LivePredicate::observed(&landed.evidence)) { + IdentityMatch::Match => Ok(()), + IdentityMatch::Unknown => Err(identity_unknown_error(entry)), + IdentityMatch::NoMatch => Err(absent(entry)), + } +} + +/// The window the ref was captured from, never any other window of the +/// process: resolution re-searches the scope the capture proved unique. +/// +/// The surface lookup owns shell surfaces and the focused-window case, and it +/// already resolves a recorded window id by that id; `belongs_to_entry` checks +/// the process and its generation on that answer. Only an entry that records no +/// window id falls back to enumerating the process's windows, where the +/// generation alone still says which one it was. +pub(crate) fn recorded_window( + adapter: &dyn PlatformAdapter, + entry: &RefEntry, + deadline: Deadline, +) -> Result { + let resolved = crate::snapshot::resolve_window_for_surface( + adapter, + entry.source.source_app.as_deref(), + entry.source.source_window_id.as_deref(), + entry.source.source_surface, + deadline, + ); + let mut failure = None; + match resolved { + Ok(window) if belongs_to_entry(&window, entry) => return Ok(window), + Ok(_) => {} + Err(error) => failure = Some(adapter_error(error)), + } + if entry.source.source_window_id.is_some() { + return Err(failure.unwrap_or_else(gone)); + } + let listed = adapter.list_windows( + &crate::adapter::WindowFilter { + focused_only: false, + app: entry.source.source_app.clone(), + }, + deadline, + )?; + listed + .into_iter() + .find(|window| belongs_to_entry(window, entry)) + .ok_or_else(|| failure.unwrap_or_else(gone)) +} + +fn gone() -> AdapterError { + AdapterError::stale_ref_because("The window the ref was captured from is gone") +} + +/// Whether a live window is the one the entry was captured from. Pid and a +/// recorded window id must agree. Process generations must agree when both +/// sides carry one; a window listing that cannot read the generation leaves +/// the pid and window id to decide, since some macOS window records omit it. +fn belongs_to_entry(window: &WindowInfo, entry: &RefEntry) -> bool { + window.pid == entry.process.pid + && entry + .source + .source_window_id + .as_deref() + .is_none_or(|id| window.id == id) + && window + .process_instance + .as_deref() + .zip(entry.process.process_instance.as_deref()) + .is_none_or(|(live, saved)| live == saved) +} + +/// The one observation a tokenless ref is judged by, in the settings resolution +/// uses: the whole window at the deepest limit a walk supports, so a capture +/// proof and a later resolution count the same evidence. `base` supplies the +/// surface and the renderer-accessibility hint; its depth and presentation +/// settings are ignored. +pub(crate) fn observe_whole_window( + adapter: &dyn PlatformAdapter, + base: &TreeOptions, + window: &WindowInfo, + deadline: Deadline, + lease: Option<&crate::InteractionLease>, +) -> Result { + observe_window( + adapter, + &whole_window_request(base, deadline)?, + window, + lease, + ) +} + +fn whole_window_request( + base: &TreeOptions, + deadline: Deadline, +) -> Result { + let options = TreeOptions { + max_depth: WHOLE_WINDOW_DEPTH, + surface: base.surface, + force_renderer_accessibility: base.force_renderer_accessibility, + ..TreeOptions::default() + } + .with_ref_identity_bounds(); + ObservationRequest::snapshot(&options, deadline).validate() +} + +fn observe_window( + adapter: &dyn PlatformAdapter, + request: &ObservationRequest, + window: &WindowInfo, + lease: Option<&crate::InteractionLease>, +) -> Result { + let mut observed = crate::tree::renderer_accessibility::observe_tree( + adapter, + ObservationRoot::Window(window), + request, + lease, + ) + .map_err(adapter_error)?; + if let crate::locator::live_locator::ObservationSource::Window { + renderer_accessibility_forced, + .. + } = &mut observed.source + { + *renderer_accessibility_forced = request.observation_mode.force_renderer_accessibility; + } + Ok(observed) +} + +fn saved_ref_node( + entry: &RefEntry, + token: Option<&str>, + observed: &ObservedTree, +) -> Result { + if let Some(token) = token { + let found = observed + .nodes + .iter() + .position(|node| node.evidence.identifiers.retained_object() == Some(token)); + return match found { + Some(index) => Ok(index), + None if observed.is_exhaustive() => Err(absent(entry)), + None => Err(identity_unknown_error(entry)), + }; + } + unique_live_match(entry, observed) +} + +fn unique_live_match(entry: &RefEntry, observed: &ObservedTree) -> Result { + let predicate = FullPredicate::of_entry(entry); + let mut found = None; + let mut incomplete = !observed.is_exhaustive(); + for (index, node) in observed.nodes.iter().enumerate() { + match predicate.compare(&LivePredicate::observed(&node.evidence)) { + IdentityMatch::Match if found.is_some() => return Err(ambiguous_live_match(entry)), + IdentityMatch::Match => found = Some(index), + IdentityMatch::Unknown => incomplete = true, + IdentityMatch::NoMatch => {} + } + } + if incomplete { + return Err(identity_unknown_error(entry)); + } + found.ok_or_else(|| absent(entry)) +} + +/// A positional anchor at the matching node's path, carrying the saved +/// predicate: the adapter lands that path and its own anchor verification +/// re-checks the recorded identity, so a row that moved under the path is +/// refused rather than acted on. +fn positional_anchor( + entry: &RefEntry, + observed: &ObservedTree, + index: usize, +) -> Result { + let node = observed + .nodes + .get(index) + .ok_or_else(|| AdapterError::internal("saved ref match is missing from the observation"))?; + let mut anchor = materialize_ref_entry( + node, + &observed.source, + &crate::locator::LocatorQuery::default(), + ); + anchor.identity = crate::RefEntryIdentity { + retained_object: entry.identity.retained_object.clone(), + unique_at_capture: false, + ..entry.identity.clone() + }; + Ok(anchor) +} + +fn ambiguous_live_match(entry: &RefEntry) -> AdapterError { + AdapterError::ambiguous_target( + "Two or more live elements match the saved ref's recorded role and stable identity", + ) + .with_suggestion( + "An element identical to the captured one appeared, so neither can be picked by ref and \ + a new snapshot refuses both as twins. Act on a control that is unique in the window, or, \ + where headed pointer input is available, use an explicit --headed coordinate click.", + ) + .with_details(json!({ + "kind": "saved_ref_ambiguous", + "retryable": false, + "complete": true, + "role": entry.identity.role, + })) +} + +fn absent(entry: &RefEntry) -> AdapterError { + AdapterError::stale_ref_because("No live element matches the saved ref's recorded identity") + .with_details(json!({ + "kind": "saved_ref_absent", + "retryable": false, + "complete": true, + "role": entry.identity.role, + })) +} + +fn adapter_error(error: AppError) -> AdapterError { + match error { + AppError::Adapter(error) => error, + other => AdapterError::internal(other.to_string()), + } +} + +#[cfg(test)] +#[path = "resolve_saved_ref_tests.rs"] +mod tests; + +#[cfg(test)] +#[path = "resolve_saved_ref_window_tests.rs"] +mod window_tests; + +#[cfg(test)] +#[path = "resolve_saved_ref_context_tests.rs"] +mod context_tests; + +#[cfg(test)] +#[path = "resolve_saved_ref_test_support.rs"] +mod test_support; diff --git a/crates/core/src/refs/resolve_saved_ref_context_tests.rs b/crates/core/src/refs/resolve_saved_ref_context_tests.rs new file mode 100644 index 000000000..114e730df --- /dev/null +++ b/crates/core/src/refs/resolve_saved_ref_context_tests.rs @@ -0,0 +1,78 @@ +use super::test_support::{RuleAdapter, entry, node, tree_with}; +use crate::{RefMap, refs::refs_test_support::HomeGuard}; + +/// The platform context gate must run on the observation used by both callers, +/// even when a complete node skips locator hydration and no anchor is resolved. +#[test] +fn rooted_find_and_drill_validate_the_observed_menu_context_before_publishing() { + use crate::commands::find::{FindArgs, FindFilterArgs, FindSelectionArgs}; + use crate::context::CommandContext; + use crate::refs::refs_store::RefStore; + + let _guard = HomeGuard::new(); + for token in [Some("captured-popup"), Some("replacement-popup"), None] { + let accepted = token == Some("captured-popup"); + let mut live = node("button", "Run"); + live.identity.retained_object = token.map(str::to_owned); + let adapter = RuleAdapter::new(tree_with(vec![live])); + let mut entry = entry(); + entry.source.source_surface = crate::SnapshotSurface::Menu; + entry.identity.retained_object = Some("captured-popup".into()); + let mut stored = RefMap::new(); + stored.allocate(entry); + let store = RefStore::new().unwrap(); + let snapshot_id = store.save_new_snapshot(&stored).unwrap(); + let root = format!("@{snapshot_id}:e1"); + let found = crate::commands::find::execute( + FindArgs { + app: None, + window_id: None, + root: Some(root.clone()), + surface: crate::SnapshotSurface::Window, + filter: FindFilterArgs { + role: Some("button".into()), + name: Some("Run".into()), + description: None, + native_id: None, + value: None, + text: None, + exact: true, + }, + states: Vec::new(), + selection: FindSelectionArgs { + count: false, + first: true, + last: false, + nth: None, + limit: None, + }, + timeout_ms: Some(1000), + }, + &adapter, + &CommandContext::default(), + ); + assert_eq!(found.is_ok(), accepted, "find: {token:?}"); + let drilled = crate::snapshot::snapshot_ref::run_from_ref_with_context( + &adapter, + &crate::TreeOptions::default(), + &root, + &CommandContext::default(), + 1000, + ); + assert_eq!(drilled.is_ok(), accepted, "drill: {token:?}"); + assert_eq!( + adapter.counts(), + (2, 2, 0, 0), + "no second observation or landing" + ); + if !accepted { + assert_eq!(found.err().unwrap().code(), "STALE_REF"); + assert_eq!(drilled.err().unwrap().code(), "STALE_REF"); + assert_eq!( + serde_json::to_value(store.load_latest().unwrap()).unwrap(), + serde_json::to_value(&stored).unwrap(), + "a refused find or drill must not publish replacement refs", + ); + } + } +} diff --git a/crates/core/src/refs/resolve_saved_ref_test_support.rs b/crates/core/src/refs/resolve_saved_ref_test_support.rs new file mode 100644 index 000000000..70173b037 --- /dev/null +++ b/crates/core/src/refs/resolve_saved_ref_test_support.rs @@ -0,0 +1,208 @@ +use std::sync::{ + Mutex, + atomic::{AtomicU32, Ordering}, +}; + +use crate::adapter::{ActionOps, InputOps, ObservationOps, SystemOps}; +use crate::locator::live_locator::ObservationRoot; +use crate::refs::ref_alloc::allocate_refs; +use crate::refs::ref_alloc_config::RefAllocConfig; +use crate::refs::ref_alloc_options::RefAllocOptions; +use crate::refs::ref_alloc_scope::RefAllocScope; +use crate::refs::ref_alloc_source::RefAllocSource; +use crate::refs::resolve_saved_ref::resolve_saved_ref as resolve_entry; +use crate::{ + AccessibilityNode, AdapterError, Deadline, NativeHandle, RefEntry, RefMap, WindowInfo, +}; + +pub(super) struct RuleAdapter { + pub(super) windows: Vec, + pub(super) shell: Option, + pub(super) observed: AccessibilityNode, + pub(super) landed: Option, + pub(super) windows_listed: AtomicU32, + pub(super) observations: AtomicU32, + pub(super) strict: AtomicU32, + pub(super) anchors: AtomicU32, + pub(super) anchored: Mutex>, +} + +impl RuleAdapter { + pub(super) fn new(observed: AccessibilityNode) -> Self { + Self { + windows: vec![window()], + shell: None, + observed, + landed: None, + windows_listed: AtomicU32::new(0), + observations: AtomicU32::new(0), + strict: AtomicU32::new(0), + anchors: AtomicU32::new(0), + anchored: Mutex::new(None), + } + } + + pub(super) fn counts(&self) -> (u32, u32, u32, u32) { + ( + self.windows_listed.load(Ordering::SeqCst), + self.observations.load(Ordering::SeqCst), + self.strict.load(Ordering::SeqCst), + self.anchors.load(Ordering::SeqCst), + ) + } +} + +impl ObservationOps for RuleAdapter { + fn list_windows( + &self, + _filter: &crate::adapter::WindowFilter, + _deadline: Deadline, + ) -> Result, AdapterError> { + self.windows_listed.fetch_add(1, Ordering::SeqCst); + Ok(self.windows.clone()) + } + + fn observe_tree( + &self, + root: ObservationRoot<'_>, + _request: &crate::locator::live_locator::ObservationRequest, + ) -> Result { + self.observations.fetch_add(1, Ordering::SeqCst); + match (&self.landed, root) { + (Some(landed), ObservationRoot::Element { .. }) => { + crate::adapter::observed_tree(&root, landed.clone()) + } + (None, ObservationRoot::Element { .. }) => crate::adapter::saved_ref_element_tree(root), + _ => crate::adapter::observed_tree(&root, self.observed.clone()), + } + } + + fn validate_saved_ref_context( + &self, + entry: &RefEntry, + observed_token: Option<&str>, + ) -> Result<(), AdapterError> { + if entry.source.source_surface == crate::SnapshotSurface::Menu + && entry.identity.retained_object.as_deref() != observed_token + { + return Err(AdapterError::stale_ref_because("menu context changed") + .with_details(serde_json::json!({ "retryable": false }))); + } + Ok(()) + } + + fn resolve_element_strict( + &self, + _entry: &RefEntry, + _deadline: Deadline, + ) -> Result { + self.strict.fetch_add(1, Ordering::SeqCst); + Ok(NativeHandle::new("strict")) + } + + fn resolve_shell_surface( + &self, + _surface: crate::SnapshotSurface, + _deadline: Deadline, + ) -> Result { + match &self.shell { + Some(window) => Ok(window.clone()), + None => Err(AdapterError::not_supported("resolve_shell_surface")), + } + } + + fn resolve_locator_anchor( + &self, + entry: &RefEntry, + _deadline: Deadline, + ) -> Result { + self.anchors.fetch_add(1, Ordering::SeqCst); + *self.anchored.lock().unwrap() = Some(entry.clone()); + Ok(NativeHandle::new("anchor")) + } +} + +impl ActionOps for RuleAdapter {} +impl InputOps for RuleAdapter {} +impl SystemOps for RuleAdapter { + fn supported_surfaces(&self) -> Vec { + vec![crate::SnapshotSurface::Window, crate::SnapshotSurface::Menu] + } +} + +pub(super) fn window() -> WindowInfo { + WindowInfo { + id: "w-1".into(), + title: "Fixture".into(), + app: "Fixture".into(), + pid: crate::ProcessId::new(1), + process_instance: Some("test-instance".into()), + bounds: None, + state: crate::WindowState::default(), + } +} + +pub(super) fn entry() -> RefEntry { + let mut entry = crate::adapter::minimal_ref_entry("button", Some("Run")); + entry.source.source_app = Some("Fixture".into()); + entry.source.source_window_id = Some("w-1".into()); + entry +} + +pub(super) fn node(role: &str, name: &str) -> AccessibilityNode { + crate::adapter::minimal_accessibility_node(role, name) +} + +pub(super) fn tree_with(children: Vec) -> AccessibilityNode { + let mut root = node("group", "Window"); + root.children = children; + root +} + +pub(super) fn resolve( + adapter: &RuleAdapter, + entry: &RefEntry, +) -> Result { + resolve_entry(adapter, entry, Deadline::standard().unwrap(), None) +} + +pub(super) fn capture(root: AccessibilityNode, name: &str) -> RefEntry { + let mut refmap = RefMap::new(); + let observed = crate::adapter::observed_tree(&ObservationRoot::Window(&window()), root.clone()) + .expect("the fixture observes"); + let proof = crate::refs::ref_capture_proof::CaptureProof::new(&observed); + let config = RefAllocConfig { + capture: Some(&proof), + ..config_without_uniqueness() + }; + allocate_refs(root, &mut refmap, &config).unwrap(); + (1..=refmap.len()) + .filter_map(|index| refmap.get(&format!("@e{index}")).cloned()) + .find(|entry| entry.identity.name.as_deref() == Some(name)) + .expect("the captured element has a ref") +} + +fn config_without_uniqueness() -> RefAllocConfig<'static> { + RefAllocConfig { + options: RefAllocOptions { + include_bounds: false, + interactive_only: false, + compact: false, + force_renderer_accessibility: false, + }, + source: RefAllocSource { + pid: crate::ProcessId::new(1), + app: Some("Fixture"), + window_id: Some("w-1"), + window_title: Some("Fixture"), + window_bounds_hash: None, + process_instance: Some("test-instance"), + surface: crate::SnapshotSurface::Window, + }, + scope: RefAllocScope { + root_ref_id: None, + path_prefix: &[], + }, + capture: None, + } +} diff --git a/crates/core/src/refs/resolve_saved_ref_tests.rs b/crates/core/src/refs/resolve_saved_ref_tests.rs new file mode 100644 index 000000000..dcf264c99 --- /dev/null +++ b/crates/core/src/refs/resolve_saved_ref_tests.rs @@ -0,0 +1,235 @@ +use std::sync::atomic::Ordering; + +use super::test_support::{RuleAdapter, capture, entry, node, resolve, tree_with, window}; +use crate::{ErrorCode, refs::refs_test_support::HomeGuard}; + +#[test] +fn a_unique_live_match_resolves_through_a_positional_anchor() { + let _guard = HomeGuard::new(); + let adapter = RuleAdapter::new(tree_with(vec![ + node("button", "Cancel"), + node("button", "Run"), + ])); + + let Ok(handle) = resolve(&adapter, &entry()) else { + panic!("one live match must resolve"); + }; + + assert_eq!(handle.downcast_ref::<&str>(), Some(&"anchor")); + assert_eq!(adapter.counts(), (1, 2, 0, 1)); + let anchor = adapter + .anchored + .lock() + .unwrap() + .clone() + .expect("anchor entry"); + assert_eq!(anchor.scope.path.as_slice(), [1]); + assert_eq!(anchor.identity.name.as_deref(), Some("Run")); +} + +#[test] +fn a_live_twin_refuses_as_ambiguous() { + let _guard = HomeGuard::new(); + let adapter = RuleAdapter::new(tree_with(vec![ + node("button", "Run"), + node("button", "Run"), + ])); + + let Err(error) = resolve(&adapter, &entry()) else { + panic!("a twin cannot be picked"); + }; + + assert_eq!(error.code, ErrorCode::AmbiguousTarget); + assert_eq!(error.details.unwrap()["kind"], "saved_ref_ambiguous"); + assert_eq!(adapter.anchors.load(Ordering::SeqCst), 0); +} + +#[test] +fn a_landed_element_that_is_not_the_recorded_one_is_stale() { + let _guard = HomeGuard::new(); + let mut adapter = RuleAdapter::new(tree_with(vec![node("button", "Run")])); + adapter.landed = Some(node("button", "Cancel")); + + let Err(error) = resolve(&adapter, &entry()) else { + panic!("the anchor landed a different element"); + }; + + assert_eq!(error.code, ErrorCode::StaleRef); + assert_eq!(error.details.unwrap()["kind"], "saved_ref_absent"); + assert_eq!(adapter.anchors.load(Ordering::SeqCst), 1); +} + +/// A shell container is not in the window inventory, so the recorded window id +/// finds it only through the adapter's shell-surface seam. +#[test] +fn a_shell_surface_ref_reaches_its_container_without_the_window_inventory() { + let _guard = HomeGuard::new(); + let mut adapter = RuleAdapter::new(tree_with(vec![node("button", "Run")])); + adapter.windows.clear(); + adapter.shell = Some(window()); + let mut entry = entry(); + entry.source.source_window_id = Some("w-1".into()); + entry.source.source_surface = crate::SnapshotSurface::SystemTray; + + let Ok(handle) = resolve(&adapter, &entry) else { + panic!("the shell container answers for its recorded id"); + }; + + assert_eq!(handle.downcast_ref::<&str>(), Some(&"anchor")); + assert_eq!(adapter.windows_listed.load(Ordering::SeqCst), 0); +} + +#[test] +fn a_removed_target_is_stale() { + let _guard = HomeGuard::new(); + let adapter = RuleAdapter::new(tree_with(vec![node("button", "Cancel")])); + + let Err(error) = resolve(&adapter, &entry()) else { + panic!("a removed target is stale"); + }; + + assert_eq!(error.code, ErrorCode::StaleRef); + let details = error.details.unwrap(); + assert_eq!(details["kind"], "saved_ref_absent"); + assert_eq!(details["complete"], true); + assert_eq!(adapter.anchors.load(Ordering::SeqCst), 0); +} + +#[test] +fn an_entry_not_unique_at_capture_is_refused_before_any_live_search() { + let _guard = HomeGuard::new(); + let adapter = RuleAdapter::new(tree_with(vec![node("button", "Run")])); + let mut entry = entry(); + entry.identity.unique_at_capture = false; + + let Err(error) = resolve(&adapter, &entry) else { + panic!("an unproven ref must be refused"); + }; + + assert_eq!(error.code, ErrorCode::AmbiguousTarget); + assert_eq!(error.details.unwrap()["kind"], "identity_not_unique"); + assert_eq!(adapter.counts(), (0, 0, 0, 0)); +} + +#[test] +fn an_element_token_goes_straight_to_the_adapter() { + let _guard = HomeGuard::new(); + let adapter = RuleAdapter::new(tree_with(vec![])); + let mut entry = entry(); + entry.identity.unique_at_capture = false; + entry.identity.retained_object = Some("axtok1:page-7:node-3".into()); + + let Ok(handle) = resolve(&adapter, &entry) else { + panic!("a token is re-identified by the adapter"); + }; + + assert_eq!(handle.downcast_ref::<&str>(), Some(&"strict")); + assert_eq!(adapter.counts(), (0, 0, 1, 0)); +} + +#[test] +fn incomplete_evidence_never_resolves() { + let _guard = HomeGuard::new(); + let mut truncated = node("scrollarea", "Feed"); + truncated.children_count = Some(2); + let adapter = RuleAdapter::new(tree_with(vec![node("button", "Run"), truncated])); + + let Err(error) = resolve(&adapter, &entry()) else { + panic!("an unread subtree hides a twin"); + }; + + assert_eq!(error.code, ErrorCode::Timeout); + assert_eq!( + error.details.unwrap()["kind"], + "resolution_identity_unknown" + ); + assert_eq!(adapter.anchors.load(Ordering::SeqCst), 0); +} + +#[test] +fn capture_and_resolution_agree_on_the_same_element() { + let _guard = HomeGuard::new(); + let alone = tree_with(vec![node("button", "Run"), node("button", "Cancel")]); + let captured = capture(alone.clone(), "Run"); + assert!(captured.identity.unique_at_capture); + assert!(resolve(&RuleAdapter::new(alone), &captured).is_ok()); + + let twinned = tree_with(vec![node("button", "Run"), node("button", "Run")]); + let twin_captured = capture(twinned.clone(), "Run"); + assert!(!twin_captured.identity.unique_at_capture); + let refused = resolve(&RuleAdapter::new(twinned), &twin_captured); + assert_eq!( + refused.err().map(|error| error.code), + Some(ErrorCode::AmbiguousTarget) + ); + + let renamed = tree_with(vec![node("button", "Cancel")]); + let stale = resolve(&RuleAdapter::new(renamed), &captured); + assert_eq!( + stale.err().map(|error| error.code), + Some(ErrorCode::StaleRef) + ); +} + +fn land( + adapter: &RuleAdapter, + entry: &crate::RefEntry, +) -> Result { + super::land_saved_ref(adapter, entry, crate::Deadline::standard().unwrap()) +} + +#[test] +fn a_confirmed_landing_skips_the_whole_window_count() { + let _guard = HomeGuard::new(); + let adapter = RuleAdapter::new(tree_with(vec![node("button", "Run")])); + + land(&adapter, &entry()).expect("the recorded path lands the recorded element"); + + assert_eq!(adapter.counts(), (0, 1, 0, 1)); +} + +#[test] +fn a_landing_that_does_not_confirm_falls_back_to_the_full_resolution() { + let _guard = HomeGuard::new(); + let mut adapter = RuleAdapter::new(tree_with(vec![node("button", "Run")])); + adapter.landed = Some(node("button", "Cancel")); + + let Err(error) = land(&adapter, &entry()) else { + panic!("a different element sits at the recorded path"); + }; + + assert_eq!(error.code, ErrorCode::StaleRef); + assert_eq!(error.details.unwrap()["kind"], "saved_ref_absent"); + assert_eq!(adapter.observations.load(Ordering::SeqCst), 3); + assert_eq!(adapter.anchors.load(Ordering::SeqCst), 2); +} + +#[test] +fn a_provisional_landing_leaves_the_twin_verdict_to_the_full_resolution() { + let _guard = HomeGuard::new(); + let adapter = RuleAdapter::new(tree_with(vec![ + node("button", "Run"), + node("button", "Run"), + ])); + + assert!(land(&adapter, &entry()).is_ok()); + let Err(error) = resolve(&adapter, &entry()) else { + panic!("the full resolution counts the twin"); + }; + assert_eq!(error.code, ErrorCode::AmbiguousTarget); +} + +#[test] +fn an_unproven_entry_is_never_landed_provisionally() { + let _guard = HomeGuard::new(); + let adapter = RuleAdapter::new(tree_with(vec![node("button", "Run")])); + let mut entry = entry(); + entry.identity.unique_at_capture = false; + + let Err(error) = land(&adapter, &entry) else { + panic!("an unproven ref is refused"); + }; + + assert_eq!(error.details.unwrap()["kind"], "identity_not_unique"); + assert_eq!(adapter.counts(), (0, 0, 0, 0)); +} diff --git a/crates/core/src/refs/resolve_saved_ref_window_tests.rs b/crates/core/src/refs/resolve_saved_ref_window_tests.rs new file mode 100644 index 000000000..c3f455046 --- /dev/null +++ b/crates/core/src/refs/resolve_saved_ref_window_tests.rs @@ -0,0 +1,118 @@ +use std::sync::atomic::Ordering; + +use super::observe_saved_ref; +use super::test_support::{RuleAdapter, entry, node, resolve, tree_with, window}; +use crate::{AdapterError, Deadline, ErrorCode, WindowInfo, refs::refs_test_support::HomeGuard}; + +fn window_of(id: &str, pid: u32, instance: &str) -> WindowInfo { + WindowInfo { + id: id.into(), + pid: crate::ProcessId::new(pid), + process_instance: Some(instance.into()), + ..window() + } +} + +fn refused(adapter: &RuleAdapter, entry: &crate::RefEntry) -> AdapterError { + match resolve(adapter, entry) { + Ok(_) => panic!("a window the ref was not captured from must not answer for it"), + Err(error) => error, + } +} + +#[test] +fn a_recorded_window_of_another_process_generation_is_refused() { + let _guard = HomeGuard::new(); + let mut adapter = RuleAdapter::new(tree_with(vec![node("button", "Run")])); + adapter.windows = vec![window_of("w-1", 1, "relaunched")]; + + let error = refused(&adapter, &entry()); + + assert_eq!(error.code, ErrorCode::StaleRef); + assert_eq!(adapter.observations.load(Ordering::SeqCst), 0); + assert_eq!(adapter.anchors.load(Ordering::SeqCst), 0); +} + +#[test] +fn a_recorded_window_owned_by_another_pid_is_refused() { + let _guard = HomeGuard::new(); + let mut adapter = RuleAdapter::new(tree_with(vec![node("button", "Run")])); + adapter.windows = vec![window_of("w-1", 2, "test-instance")]; + + let error = refused(&adapter, &entry()); + + assert_eq!(error.code, ErrorCode::StaleRef); + assert_eq!(adapter.observations.load(Ordering::SeqCst), 0); +} + +#[test] +fn a_closed_recorded_window_is_never_replaced_by_another() { + let _guard = HomeGuard::new(); + let mut adapter = RuleAdapter::new(tree_with(vec![node("button", "Run")])); + adapter.windows = vec![window_of("w-2", 1, "test-instance")]; + + let error = refused(&adapter, &entry()); + + assert!( + matches!(error.code, ErrorCode::StaleRef | ErrorCode::WindowNotFound), + "a closed window is gone, not substituted: {error:?}" + ); + assert_eq!(adapter.observations.load(Ordering::SeqCst), 0); +} + +#[test] +fn an_entry_without_a_window_id_finds_the_window_of_its_generation() { + let _guard = HomeGuard::new(); + let mut adapter = RuleAdapter::new(tree_with(vec![node("button", "Run")])); + adapter.windows = vec![ + window_of("w-0", 1, "relaunched"), + window_of("w-1", 1, "test-instance"), + ]; + let mut entry = entry(); + entry.source.source_window_id = None; + + resolve(&adapter, &entry).expect("the window of the captured generation answers"); + + assert_eq!(adapter.anchors.load(Ordering::SeqCst), 1); +} + +#[test] +fn an_entry_without_a_window_id_refuses_windows_of_other_generations() { + let _guard = HomeGuard::new(); + let mut adapter = RuleAdapter::new(tree_with(vec![node("button", "Run")])); + adapter.windows = vec![window_of("w-0", 1, "relaunched")]; + let mut entry = entry(); + entry.source.source_window_id = None; + + let error = refused(&adapter, &entry); + + assert_eq!(error.code, ErrorCode::StaleRef); + assert_eq!(adapter.observations.load(Ordering::SeqCst), 0); +} + +#[test] +fn a_token_missing_from_a_cut_short_read_is_not_proven_absent() { + let _guard = HomeGuard::new(); + let mut unread = node("scrollarea", "Feed"); + unread.children_count = Some(4); + let cut_short = RuleAdapter::new(tree_with(vec![node("button", "Cancel"), unread])); + let whole = RuleAdapter::new(tree_with(vec![node("button", "Cancel")])); + let mut entry = entry(); + entry.identity.retained_object = Some("axtok1:page-7:node-3".into()); + let deadline = Deadline::standard().expect("a standard deadline"); + + let Err(unknown) = observe_saved_ref(&cut_short, &entry, deadline, None, false) else { + panic!("the token may sit in the unread region"); + }; + assert_eq!(unknown.code, ErrorCode::Timeout); + assert_eq!( + unknown.details.expect("details")["kind"], + "resolution_identity_unknown" + ); + + let Err(absent) = observe_saved_ref(&whole, &entry, deadline, None, false) else { + panic!("a whole read without the token proves it gone"); + }; + assert_eq!(absent.code, ErrorCode::StaleRef); + assert_eq!(absent.details.expect("details")["kind"], "saved_ref_absent"); +} diff --git a/crates/core/src/response/output.rs b/crates/core/src/response/output.rs index 549ab6144..c93061692 100644 --- a/crates/core/src/response/output.rs +++ b/crates/core/src/response/output.rs @@ -4,7 +4,7 @@ use serde_json::Value; use crate::response::recovery_hint::RecoveryHint; use crate::{AppError, DeliverySemantics, ErrorCode, RetryDisposition}; -pub const ENVELOPE_VERSION: &str = "2.5"; +pub const ENVELOPE_VERSION: &str = "2.6"; /// Structured output envelope used by the CLI and future programmatic transports. #[derive(Debug, Serialize)] diff --git a/crates/core/src/snapshot/mod.rs b/crates/core/src/snapshot/mod.rs index 96cd82251..f0ed1ebff 100644 --- a/crates/core/src/snapshot/mod.rs +++ b/crates/core/src/snapshot/mod.rs @@ -2,7 +2,6 @@ use crate::{ AccessibilityNode, AppError, WindowInfo, adapter::{PlatformAdapter, TreeOptions, WindowFilter}, context::CommandContext, - locator::live_locator::{ObservationRequest, ObservationRoot}, refs::RefMap, refs::ref_alloc::{self, RefAllocConfig}, refs::ref_alloc_options::RefAllocOptions, @@ -21,6 +20,9 @@ pub struct SnapshotResult { pub snapshot_id: Option, pub complete: bool, pub nodes_observed: usize, + /// Set when the whole-window read that proves tokenless refs failed or was + /// cut short; it says why, and those refs are refused when used. + pub identity_unproven: Option, } impl SnapshotResult { @@ -38,16 +40,12 @@ pub fn build( deadline: crate::Deadline, ) -> Result { let windows = surface_windows(adapter, app_name, window_id, opts.surface, deadline)?; - let observation_options = opts.with_ref_identity_bounds(); - let request = ObservationRequest::snapshot(&observation_options, deadline).validate()?; - let (window, observed) = first_surface_owner(windows, |window| { - crate::tree::renderer_accessibility::observe_tree( - adapter, - ObservationRoot::Window(window), - &request, - ) + let (window, capture) = first_surface_owner(windows, |window| { + window_capture::capture_window(adapter, opts, window, deadline) })?; - let (raw_tree, complete, nodes_observed) = observed.into_accessibility_tree_partial()?; + let identity_unproven = capture.identity_unproven.clone(); + let (raw_tree, complete, nodes_observed, observed) = capture.into_parts()?; + let capture_proof = crate::refs::ref_capture_proof::CaptureProof::new(&observed); let mut refmap = RefMap::new(); let config = RefAllocConfig { @@ -55,6 +53,7 @@ pub fn build( include_bounds: opts.include_bounds, interactive_only: opts.interactive_only, compact: opts.compact, + force_renderer_accessibility: opts.force_renderer_accessibility, }, source: RefAllocSource { pid: window.pid, @@ -69,6 +68,7 @@ pub fn build( root_ref_id: None, path_prefix: &[], }, + capture: Some(&capture_proof), }; let mut tree = ref_alloc::allocate_refs(raw_tree, &mut refmap, &config)?; @@ -81,6 +81,7 @@ pub fn build( snapshot_id: None, complete, nodes_observed, + identity_unproven, }) } @@ -91,10 +92,12 @@ pub fn build( /// Kinds naming OS chrome rather than a window's own sub-surface are routed to /// the adapter's `resolve_shell_surface` seam ahead of the application path, /// because no application owns them and the window inventory cannot identify -/// them. An adapter that answers with the trait default falls through to the -/// application path, so an adapter's shape never changes by upgrading; every +/// them - not even a recorded or requested window id, which a shell container +/// carries too. An adapter that answers with the trait default falls through to +/// the application path, so an adapter's shape never changes by upgrading; every /// other answer - a missing surface, a build refusal, a timeout - is returned -/// as the adapter supplied it. +/// as the adapter supplied it, and a surface that answers with another window +/// leaves the id to the application path. pub(crate) fn resolve_window_for_surface( adapter: &dyn PlatformAdapter, app_name: Option<&str>, @@ -102,18 +105,22 @@ pub(crate) fn resolve_window_for_surface( surface: crate::SnapshotSurface, deadline: crate::Deadline, ) -> Result { - if window_id.is_some() || matches!(surface, crate::SnapshotSurface::Window) { - return resolve_window(adapter, app_name, window_id, deadline); - } if is_shell_surface(surface) { match adapter.resolve_shell_surface(surface, deadline) { - Ok(window) => return Ok(window), - Err(error) if !error.is_default_not_supported("resolve_shell_surface") => { + Ok(window) if window_id.is_none_or(|id| window.id == id) => return Ok(window), + Ok(_) => {} + Err(error) + if window_id.is_none() + && !error.is_default_not_supported("resolve_shell_surface") => + { return Err(error.into()); } Err(_) => {} } } + if window_id.is_some() || matches!(surface, crate::SnapshotSurface::Window) { + return resolve_window(adapter, app_name, window_id, deadline); + } crate::system::window_lookup::select_surface_owner( windows_for_app(adapter, app_name, deadline)?, crate::AdapterError::new( @@ -333,6 +340,9 @@ pub(crate) fn emit_snapshot_saved( if !result.window.app.is_empty() { fields["app"] = json!(result.window.app); } + if let Some(reason) = &result.identity_unproven { + fields["identity_unproven"] = json!(reason); + } fields }) } @@ -343,6 +353,19 @@ pub mod snapshot_surface; #[cfg(test)] #[path = "snapshot_tests.rs"] mod tests; +pub(crate) mod window_capture; + +#[cfg(test)] +#[path = "tokenless_capture_support.rs"] +mod tokenless_capture_support; + +#[cfg(test)] +#[path = "tokenless_capture_tests.rs"] +mod tokenless_capture_tests; + +#[cfg(test)] +#[path = "tokenless_drill_tests.rs"] +mod tokenless_drill_tests; #[cfg(test)] #[path = "snapshot_window_tests.rs"] diff --git a/crates/core/src/snapshot/skeleton_anchor_tests.rs b/crates/core/src/snapshot/skeleton_anchor_tests.rs index ab0ab7d60..72e6d1089 100644 --- a/crates/core/src/snapshot/skeleton_anchor_tests.rs +++ b/crates/core/src/snapshot/skeleton_anchor_tests.rs @@ -17,8 +17,7 @@ fn native_id_ancestor_covers_unnamed_boundaries() { let mut root = node("window"); root.children = vec![split_view]; - let mut refmap = RefMap::new(); - let result = ref_alloc::allocate_refs(root, &mut refmap, &run_config(false, false)).unwrap(); + let (result, refmap) = run_captured(root); let split_view = &result.children[0]; assert!(split_view.ref_id.is_some()); @@ -48,8 +47,7 @@ fn web_boundary_with_only_ubiquitous_affordances_gets_anchor_ref() { let mut root = node("window"); root.children = vec![boundary]; - let mut refmap = RefMap::new(); - let result = ref_alloc::allocate_refs(root, &mut refmap, &run_config(false, false)).unwrap(); + let (result, refmap) = run_captured(root); let boundary = &result.children[0]; let entry = refmap.get(boundary.ref_id.as_deref().unwrap()).unwrap(); @@ -70,8 +68,7 @@ fn unlabeled_bounded_boundary_gets_drill_ref() { let mut root = node("window"); root.children = vec![boundary]; - let mut refmap = RefMap::new(); - let result = ref_alloc::allocate_refs(root, &mut refmap, &run_config(false, false)).unwrap(); + let (result, refmap) = run_captured(root); let boundary = &result.children[0]; assert!(boundary.ref_id.is_some()); @@ -102,8 +99,7 @@ fn a_named_bounded_boundary_still_has_its_rect_stripped() { let mut root = node("window"); root.children = vec![boundary]; - let mut refmap = RefMap::new(); - let result = ref_alloc::allocate_refs(root, &mut refmap, &run_config(false, false)).unwrap(); + let (result, refmap) = run_captured(root); let boundary = &result.children[0]; let entry = refmap.get(boundary.ref_id.as_deref().unwrap()).unwrap(); @@ -114,6 +110,7 @@ fn a_named_bounded_boundary_still_has_its_rect_stripped() { #[test] fn deepest_resolvable_anchor_wins() { let mut boundary = node("group"); + boundary.identity.name = Some("Sidebar".into()); boundary.children_count = Some(5); boundary.presentation.bounds = Some(crate::Rect { x: 10.0, @@ -130,8 +127,7 @@ fn deepest_resolvable_anchor_wins() { let mut root = node("window"); root.children = vec![split_view]; - let mut refmap = RefMap::new(); - let result = ref_alloc::allocate_refs(root, &mut refmap, &run_config(false, false)).unwrap(); + let (result, refmap) = run_captured(root); assert!(result.children[0].ref_id.is_none()); assert!(result.children[0].children[0].ref_id.is_some()); diff --git a/crates/core/src/snapshot/snapshot_ref.rs b/crates/core/src/snapshot/snapshot_ref.rs index 3c7e157ab..2791cd464 100644 --- a/crates/core/src/snapshot/snapshot_ref.rs +++ b/crates/core/src/snapshot/snapshot_ref.rs @@ -1,16 +1,28 @@ use crate::{ - AppError, - adapter::{PlatformAdapter, TreeOptions, WindowFilter}, + AccessibilityNode, AppError, WindowInfo, + adapter::{PlatformAdapter, TreeOptions}, context::CommandContext, locator::live_locator::{ObservationRequest, ObservationRoot}, refs::ref_alloc::{self, RefAllocConfig}, refs::ref_alloc_options::RefAllocOptions, refs::ref_alloc_scope::RefAllocScope, refs::ref_alloc_source::RefAllocSource, + refs::ref_root_read::{RootRead, read_saved_root}, refs::refs_store::RefStore, snapshot::SnapshotResult, }; +/// What one drill-down presents: the window it reports, the subtree that +/// replaces the stored refs, the path coordinates its refs are recorded in, and +/// the whole-window observation that proves them. +struct DrillCapture { + window: WindowInfo, + tree: AccessibilityNode, + nodes_observed: usize, + path_prefix: crate::refs::RefPath, + observed: crate::locator::live_locator::ObservedTree, +} + #[cfg(test)] pub fn run_from_ref( adapter: &dyn PlatformAdapter, @@ -35,6 +47,14 @@ pub fn run_from_ref( /// A drill-down replaces refs inside an existing snapshot, so it must observe /// its whole subtree or leave the stored map untouched. Partial projection is /// only safe for a full snapshot, which writes a fresh map and destroys nothing. +/// +/// A root with an element token is read as its subtree alone when every ref +/// that subtree holds carries a token too, since a token re-identifies itself. +/// Any other drill locates its root in one complete observation of its window, +/// by its token or by its recorded predicate, and projects the subtree from that +/// same observation: the refs it writes were counted against the window they +/// will be resolved in, and the walk that located the root doubles as their +/// evidence. pub fn run_from_ref_with_context( adapter: &dyn PlatformAdapter, opts: &TreeOptions, @@ -53,31 +73,26 @@ pub fn run_from_ref_with_context( .clone(); let deadline = crate::Deadline::after(timeout_ms)?; - let window = resolve_result_window(&entry, adapter, deadline)?; - let handle = adapter.resolve_element_strict(&entry, deadline)?; - - let observation_options = opts.with_ref_identity_bounds(); - let raw_tree = crate::tree::renderer_accessibility::observe_tree( - adapter, - ObservationRoot::Element { - handle: &handle, - entry: &entry, - root_ref: Some(&local_root_ref), - }, - &ObservationRequest::snapshot(&observation_options, deadline).validate()?, - )? - .into_accessibility_tree()?; - let nodes_observed = count_nodes(&raw_tree); + let capture = drill_capture(adapter, opts, &entry, deadline)?; + let DrillCapture { + window, + tree: raw_tree, + nodes_observed, + path_prefix, + observed, + } = capture; + let capture_proof = + crate::refs::ref_capture_proof::CaptureProof::new(&observed).rooted_at(&path_prefix); let source_app = entry.source.source_app.as_deref(); let source_window_id = entry.source.source_window_id.as_deref(); let source_window_title = entry.source.source_window_title.as_deref(); - let path_prefix = entry.scope.path.clone(); let config = RefAllocConfig { options: RefAllocOptions { include_bounds: opts.include_bounds, interactive_only: opts.interactive_only, compact: opts.compact, + force_renderer_accessibility: entry.source.renderer_accessibility_forced, }, source: RefAllocSource { pid: entry.process.pid, @@ -92,6 +107,7 @@ pub fn run_from_ref_with_context( root_ref_id: Some(&local_root_ref), path_prefix: path_prefix.as_slice(), }, + capture: Some(&capture_proof), }; let (mut tree, refmap) = store.update_existing_snapshot( @@ -127,55 +143,99 @@ pub fn run_from_ref_with_context( snapshot_id: Some(active_snapshot_id), complete: true, nodes_observed, + identity_unproven: None, }) } -/// The drill result reports the window the ref was captured from, so it must -/// resolve that exact window — not any window of the process. Resolving before -/// observation keeps a failed drill from persisting refs it never returns. -fn resolve_result_window( - entry: &crate::refs::RefEntry, +fn drill_capture( adapter: &dyn PlatformAdapter, + opts: &TreeOptions, + entry: &crate::refs::RefEntry, deadline: crate::Deadline, -) -> Result { - let Some(window_id) = entry.source.source_window_id.as_deref() else { - let instance = entry.process.process_instance.as_deref().ok_or_else(|| { - AppError::Adapter(crate::AdapterError::stale_ref_because( - "root ref has no process-instance identity", +) -> Result { + ObservationRequest::snapshot(opts, deadline).validate()?; + let (window, mut observed, index) = match read_saved_root( + adapter, + entry, + deadline, + |window, handle| read_token_subtree(adapter, opts, entry, window, handle, deadline), + |subtree| !ref_alloc::selects_tokenless_entry(&subtree.tree), + )? { + RootRead::Subtree(subtree) => return Ok(subtree), + RootRead::Window(window_read) => *window_read, + }; + if !observed.is_exhaustive() { + return Err(drill_incomplete(observed.nodes.len())); + } + let path_prefix = observed + .nodes + .get(index) + .map(|node| node.path.clone()) + .ok_or_else(|| { + AppError::Adapter(crate::AdapterError::internal( + "saved ref match is missing from the observation", )) })?; - return crate::system::window_lookup::find_window_for_process( - crate::ProcessIdentity::new(entry.process.pid, instance), - adapter, - deadline, - ); - }; - let candidates = adapter - .list_windows( - &WindowFilter { - focused_only: false, - app: entry.source.source_app.clone(), - }, - deadline, - )? - .into_iter() - .filter(|window| window.id == window_id && window.pid == entry.process.pid) - .collect(); - crate::system::window_lookup::select_window( - candidates, + let nodes_observed = observed.nodes.len(); + let tree = observed.project_subtree(index, ObservationRequest::snapshot_logical_depth(opts))?; + Ok(DrillCapture { + window, + tree, + nodes_observed, + path_prefix, + observed, + }) +} + +/// A token root's subtree read alone, through the adapter that owns the +/// token. It proves nothing, so it stands only for a drill whose refs all carry +/// tokens. +fn read_token_subtree( + adapter: &dyn PlatformAdapter, + opts: &TreeOptions, + entry: &crate::refs::RefEntry, + window: &WindowInfo, + handle: &crate::adapter::NativeHandle, + deadline: crate::Deadline, +) -> Result { + let options = opts.with_ref_identity_bounds(); + let mut observed = crate::tree::renderer_accessibility::observe_tree( + adapter, + ObservationRoot::Element { + handle, + entry, + root_ref: None, + }, + &ObservationRequest::snapshot(&options, deadline).validate()?, + None, + )?; + let (tree, complete, nodes_observed) = observed.project_partial()?; + if !complete { + return Err(drill_incomplete(nodes_observed)); + } + Ok(DrillCapture { + window: window.clone(), + tree, + nodes_observed, + path_prefix: entry.scope.path.clone(), + observed, + }) +} + +fn drill_incomplete(nodes_observed: usize) -> AppError { + AppError::Adapter( crate::AdapterError::new( - crate::ErrorCode::WindowNotFound, - format!("Source window '{window_id}' was not found"), + crate::ErrorCode::Timeout, + "The drill-down observation was incomplete, so no refs were replaced", ) - .with_suggestion("Run 'snapshot' to refresh, then retry with an updated ref"), - "Multiple windows matched the source window ID", + .with_suggestion("Retry with a larger --timeout-ms, or drill a narrower subtree") + .with_details(serde_json::json!({ + "kind": "observation_incomplete", + "nodes_observed": nodes_observed, + })), ) } -fn count_nodes(node: &crate::AccessibilityNode) -> usize { - 1 + node.children.iter().map(count_nodes).sum::() -} - #[cfg(test)] #[path = "snapshot_ref_tests.rs"] mod tests; diff --git a/crates/core/src/snapshot/snapshot_ref_alloc_tests.rs b/crates/core/src/snapshot/snapshot_ref_alloc_tests.rs index 6188c662a..d0d4039e3 100644 --- a/crates/core/src/snapshot/snapshot_ref_alloc_tests.rs +++ b/crates/core/src/snapshot/snapshot_ref_alloc_tests.rs @@ -22,10 +22,12 @@ fn drill_config<'a>( compact: bool, ) -> RefAllocConfig<'a> { RefAllocConfig { + capture: None, options: crate::refs::ref_alloc_options::RefAllocOptions { include_bounds: false, interactive_only, compact, + force_renderer_accessibility: false, }, source: crate::refs::ref_alloc_source::RefAllocSource { pid: crate::ProcessId::new(pid), diff --git a/crates/core/src/snapshot/snapshot_ref_merge_tests.rs b/crates/core/src/snapshot/snapshot_ref_merge_tests.rs index 657afa586..c1c7d69c8 100644 --- a/crates/core/src/snapshot/snapshot_ref_merge_tests.rs +++ b/crates/core/src/snapshot/snapshot_ref_merge_tests.rs @@ -9,7 +9,8 @@ fn test_run_from_ref_multiple_drill_downs_accumulate() { let first = run_from_ref(&adapter_one, &drill_opts(), "@e1", Some(&snapshot_id)).unwrap(); let from_e1_ref = first.tree.ref_id.clone().expect("first drill ref"); - let adapter_two = StubAdapter::new(named("button", "FromE2")); + let adapter_two = + StubAdapter::new(named("button", "FromE2")).with_token("axtok1:button-Toolbar"); let second = run_from_ref(&adapter_two, &drill_opts(), "@e2", Some(&snapshot_id)).unwrap(); let from_e2_ref = second.tree.ref_id.clone().expect("second drill ref"); @@ -35,13 +36,13 @@ fn test_drilldown_refmap_matches_golden_fixture() { let _guard = HomeGuard::new(); let mut seed = RefMap::new(); seed.allocate(ref_entry_from_node( - &named("group", "Sidebar"), + &drilled("group", "Sidebar"), &source("Fixture"), None, &[0], )); seed.allocate(ref_entry_from_node( - &named("group", "Toolbar"), + &drilled("group", "Toolbar"), &source("Fixture"), None, &[1], @@ -54,7 +55,7 @@ fn test_drilldown_refmap_matches_golden_fixture() { let _ = run_from_ref(&adapter, &drill_opts(), "@e1", Some(&snapshot_id)).unwrap(); let toolbar_subtree = named("button", "Back"); - let adapter = StubAdapter::new(toolbar_subtree); + let adapter = StubAdapter::new(toolbar_subtree).with_token("axtok1:group-Toolbar"); let _ = run_from_ref(&adapter, &drill_opts(), "@e2", Some(&snapshot_id)).unwrap(); let on_disk = load_latest(); diff --git a/crates/core/src/snapshot/snapshot_ref_tests.rs b/crates/core/src/snapshot/snapshot_ref_tests.rs index 027650e22..a472bd5c7 100644 --- a/crates/core/src/snapshot/snapshot_ref_tests.rs +++ b/crates/core/src/snapshot/snapshot_ref_tests.rs @@ -6,18 +6,6 @@ use crate::adapter::{ActionOps, InputOps, NativeHandle, ObservationOps, SystemOp use crate::refs::ref_alloc::ref_entry_from_node; use crate::refs::refs_test_support::HomeGuard; use crate::{refs::RefMap, refs::refs_store::RefStore}; -use std::sync::{ - Arc, - atomic::{AtomicU32, Ordering}, -}; - -struct DropProbe(Arc); - -impl Drop for DropProbe { - fn drop(&mut self) { - self.0.fetch_add(1, Ordering::SeqCst); - } -} fn node(role: &str) -> AccessibilityNode { AccessibilityNode { @@ -37,6 +25,12 @@ fn named(role: &str, name: &str) -> AccessibilityNode { n } +fn drilled(role: &str, name: &str) -> AccessibilityNode { + let mut n = named(role, name); + n.identity.retained_object = Some(format!("axtok1:{role}-{name}")); + n +} + fn source(app: &'static str) -> crate::refs::ref_alloc_source::RefAllocSource<'static> { crate::refs::ref_alloc_source::RefAllocSource { pid: crate::ProcessId::new(42), @@ -52,30 +46,21 @@ fn source(app: &'static str) -> crate::refs::ref_alloc_source::RefAllocSource<'s struct StubAdapter { subtree: AccessibilityNode, subtree_error: Option, - resolve_calls: AtomicU32, - drops: Arc, + token: String, windows: Vec, expected_app: Option, } impl StubAdapter { fn new(subtree: AccessibilityNode) -> Self { - Self { - subtree, - subtree_error: None, - resolve_calls: AtomicU32::new(0), - drops: Arc::new(AtomicU32::new(0)), - windows: vec![window_info("w-42", true)], - expected_app: None, - } + Self::with_windows(subtree, vec![window_info("w-42", true)]) } fn with_windows(subtree: AccessibilityNode, windows: Vec) -> Self { Self { subtree, subtree_error: None, - resolve_calls: AtomicU32::new(0), - drops: Arc::new(AtomicU32::new(0)), + token: "axtok1:group-Sidebar".into(), windows, expected_app: None, } @@ -83,14 +68,15 @@ impl StubAdapter { fn with_subtree_error(error: AdapterError) -> Self { Self { - subtree: node("group"), subtree_error: Some(error), - resolve_calls: AtomicU32::new(0), - drops: Arc::new(AtomicU32::new(0)), - windows: vec![window_info("w-42", true)], - expected_app: None, + ..Self::new(node("group")) } } + + fn with_token(mut self, token: &str) -> Self { + self.token = token.into(); + self + } } fn window_info(id: &str, focused: bool) -> crate::WindowInfo { @@ -117,7 +103,17 @@ impl ObservationOps for StubAdapter { if let Some(error) = &self.subtree_error { return Err(error.clone()); } - crate::adapter::observed_tree(&root, self.subtree.clone()) + if matches!( + root, + crate::locator::live_locator::ObservationRoot::Element { .. } + ) { + return crate::adapter::observed_tree(&root, self.subtree.clone()); + } + let mut drilled = self.subtree.clone(); + drilled.identity.retained_object = Some(self.token.clone()); + let mut window = node("window"); + window.children = vec![drilled]; + crate::adapter::observed_tree(&root, window) } fn list_windows( @@ -136,8 +132,7 @@ impl ObservationOps for StubAdapter { _entry: &crate::refs::RefEntry, _deadline: crate::Deadline, ) -> Result { - self.resolve_calls.fetch_add(1, Ordering::SeqCst); - Ok(NativeHandle::new(DropProbe(Arc::clone(&self.drops)))) + Ok(NativeHandle::null()) } } @@ -192,9 +187,14 @@ fn local_ref(ref_id: &str) -> String { fn seed_skeleton_refmap() -> RefMap { let mut map = RefMap::new(); - let anchor = ref_entry_from_node(&named("group", "Sidebar"), &source("TestApp"), None, &[0]); + let anchor = ref_entry_from_node(&drilled("group", "Sidebar"), &source("TestApp"), None, &[0]); let _ = map.allocate(anchor); - let other = ref_entry_from_node(&named("button", "Toolbar"), &source("TestApp"), None, &[1]); + let other = ref_entry_from_node( + &drilled("button", "Toolbar"), + &source("TestApp"), + None, + &[1], + ); let _ = map.allocate(other); map } @@ -242,8 +242,7 @@ fn test_run_from_ref_returns_subtree_and_persists_refs() { "drilled refs must retain an absolute path for fast, scoped resolution" ); assert_eq!(drill_entry.scope.path.as_slice(), [0, 0]); - assert_eq!(adapter.resolve_calls.load(Ordering::SeqCst), 1); - assert_eq!(adapter.drops.load(Ordering::SeqCst), 1); + assert!(drill_entry.identity.unique_at_capture); } #[test] @@ -280,7 +279,7 @@ fn test_run_from_ref_explicit_session_snapshot_with_matching_context() { } #[test] -fn test_run_from_ref_drops_handle_when_subtree_read_fails() { +fn test_run_from_ref_leaves_stored_refs_when_subtree_read_fails() { let _guard = HomeGuard::new(); let snapshot_id = save_latest(seed_skeleton_refmap()); @@ -291,8 +290,11 @@ fn test_run_from_ref_drops_handle_when_subtree_read_fails() { let result = run_from_ref(&adapter, &drill_opts(), "@e1", Some(&snapshot_id)); assert!(result.is_err()); - assert_eq!(adapter.resolve_calls.load(Ordering::SeqCst), 1); - assert_eq!(adapter.drops.load(Ordering::SeqCst), 1); + assert_eq!( + load_latest().len(), + 2, + "a failed drill must not persist refs" + ); } #[test] @@ -352,6 +354,35 @@ fn test_run_from_ref_re_drill_replaces_drill_refs_only() { ); } +#[test] +fn a_tokenless_drill_refuses_a_depth_no_walk_supports() { + use crate::snapshot::tokenless_capture_support as tokenless; + + let _guard = HomeGuard::new(); + let adapter = tokenless::CaptureAdapter::new(tokenless::anchored_window()); + let captured = tokenless::snapshot(&adapter); + let (anchor_ref, _) = tokenless::named(&captured.refmap, "Sidebar"); + let snapshot_id = tokenless::save(&captured.refmap); + let too_deep = TreeOptions { + max_depth: 51, + ..drill_opts() + }; + let walks_before = adapter.captures(); + + let drill = run_from_ref(&adapter, &too_deep, &anchor_ref, Some(&snapshot_id)); + + match drill { + Ok(_) => panic!("a depth no snapshotted walk reaches must be refused"), + Err(AppError::Adapter(error)) => assert_eq!(error.code, crate::ErrorCode::InvalidArgs), + Err(other) => panic!("expected Adapter(InvalidArgs), got {other:?}"), + } + assert_eq!( + adapter.captures(), + walks_before, + "the refusal must land before the drill observes anything" + ); +} + #[path = "snapshot_ref_window_tests.rs"] mod window_tests; diff --git a/crates/core/src/snapshot/snapshot_ref_window_tests.rs b/crates/core/src/snapshot/snapshot_ref_window_tests.rs index 5efb8cd03..6e311c0e6 100644 --- a/crates/core/src/snapshot/snapshot_ref_window_tests.rs +++ b/crates/core/src/snapshot/snapshot_ref_window_tests.rs @@ -6,9 +6,9 @@ fn seed_skeleton_refmap_on_window(window_id: Option<&'static str>) -> RefMap { ..source("TestApp") }; let mut map = RefMap::new(); - let anchor = ref_entry_from_node(&named("group", "Sidebar"), &src, None, &[0]); + let anchor = ref_entry_from_node(&drilled("group", "Sidebar"), &src, None, &[0]); let _ = map.allocate(anchor); - let other = ref_entry_from_node(&named("button", "Toolbar"), &src, None, &[1]); + let other = ref_entry_from_node(&drilled("button", "Toolbar"), &src, None, &[1]); let _ = map.allocate(other); map } diff --git a/crates/core/src/snapshot/snapshot_surface_window_tests.rs b/crates/core/src/snapshot/snapshot_surface_window_tests.rs index 39c04287f..b3d8f37d8 100644 --- a/crates/core/src/snapshot/snapshot_surface_window_tests.rs +++ b/crates/core/src/snapshot/snapshot_surface_window_tests.rs @@ -1,5 +1,6 @@ use super::*; use crate::adapter::{ActionOps, InputOps, ObservationOps, SystemOps}; +use crate::locator::live_locator::{ObservationRequest, ObservationRoot}; struct SurfaceAdapter { windows: Vec, diff --git a/crates/core/src/snapshot/snapshot_tests.rs b/crates/core/src/snapshot/snapshot_tests.rs index 91765904d..4cb5074eb 100644 --- a/crates/core/src/snapshot/snapshot_tests.rs +++ b/crates/core/src/snapshot/snapshot_tests.rs @@ -15,10 +15,12 @@ fn node(role: &str) -> AccessibilityNode { fn run_config(compact: bool, interactive_only: bool) -> RefAllocConfig<'static> { RefAllocConfig { + capture: None, options: crate::refs::ref_alloc_options::RefAllocOptions { include_bounds: false, interactive_only, compact, + force_renderer_accessibility: false, }, source: crate::refs::ref_alloc_source::RefAllocSource { pid: crate::ProcessId::new(1), @@ -41,6 +43,56 @@ fn run_compact(tree: AccessibilityNode) -> AccessibilityNode { ref_alloc::allocate_refs(tree, &mut refmap, &run_config(true, false)).unwrap() } +fn window_info() -> crate::WindowInfo { + crate::WindowInfo { + id: "w-1".into(), + title: "Test Window".into(), + app: "Fixture".into(), + pid: crate::ProcessId::new(1), + process_instance: Some("test-instance".into()), + bounds: None, + state: crate::WindowState::default(), + } +} + +/// The whole-window observation a snapshot counts before it presents a cut: a +/// boundary no longer hides children from it, so the census sees every node. +fn uncut(mut tree: AccessibilityNode) -> AccessibilityNode { + tree.children_count = None; + tree.children = tree.children.into_iter().map(uncut).collect(); + tree +} + +/// Allocates over the capture proof a snapshot takes before it allocates. An +/// entry that cannot re-identify itself by an element token is published only +/// from such a proof, so a fixture that expects one needs this capture. +fn run_captured(tree: AccessibilityNode) -> (AccessibilityNode, RefMap) { + run_captured_with(tree, run_config(false, false)) +} + +fn run_captured_with( + tree: AccessibilityNode, + config: RefAllocConfig<'_>, +) -> (AccessibilityNode, RefMap) { + let observed = crate::adapter::observed_tree( + &crate::locator::live_locator::ObservationRoot::Window(&window_info()), + uncut(tree.clone()), + ) + .expect("the fixture observes"); + let proof = crate::refs::ref_capture_proof::CaptureProof::new(&observed); + let mut refmap = RefMap::new(); + let result = ref_alloc::allocate_refs( + tree, + &mut refmap, + &RefAllocConfig { + capture: Some(&proof), + ..config + }, + ) + .expect("the tree allocates"); + (result, refmap) +} + fn run_compact_interactive(tree: AccessibilityNode) -> AccessibilityNode { let mut refmap = RefMap::new(); ref_alloc::allocate_refs(tree, &mut refmap, &run_config(true, true)).unwrap() @@ -177,14 +229,14 @@ fn test_skeleton_named_container_gets_ref() { let mut root = node("window"); root.children = vec![container]; - let mut refmap = RefMap::new(); - let result = ref_alloc::allocate_refs(root, &mut refmap, &run_config(false, false)).unwrap(); + let (result, refmap) = run_captured(root); assert!(result.children[0].ref_id.is_some()); assert_eq!(refmap.len(), 1); let entry = refmap .get(result.children[0].ref_id.as_deref().unwrap()) .unwrap(); + assert!(entry.identity.unique_at_capture); assert!(entry.capabilities.available_actions.is_empty()); } @@ -212,10 +264,12 @@ fn test_skeleton_anchor_suppressed_in_drilldown() { let mut refmap = RefMap::new(); let config = RefAllocConfig { + capture: None, options: crate::refs::ref_alloc_options::RefAllocOptions { include_bounds: false, interactive_only: false, compact: false, + force_renderer_accessibility: false, }, source: crate::refs::ref_alloc_source::RefAllocSource { pid: crate::ProcessId::new(1), @@ -248,8 +302,7 @@ fn test_skeleton_described_container_gets_ref() { let mut root = node("window"); root.children = vec![container]; - let mut refmap = RefMap::new(); - let result = ref_alloc::allocate_refs(root, &mut refmap, &run_config(false, false)).unwrap(); + let (result, refmap) = run_captured(root); assert!(result.children[0].ref_id.is_some()); assert_eq!(refmap.len(), 1); @@ -295,12 +348,13 @@ fn test_skeleton_fixture_matches_golden() { root.identity.name = Some("Test Window".into()); root.children = vec![sidebar, described, content]; - let mut refmap = RefMap::new(); let config = RefAllocConfig { + capture: None, options: crate::refs::ref_alloc_options::RefAllocOptions { include_bounds: false, interactive_only: false, compact: false, + force_renderer_accessibility: false, }, source: crate::refs::ref_alloc_source::RefAllocSource { pid: crate::ProcessId::new(42), @@ -316,7 +370,7 @@ fn test_skeleton_fixture_matches_golden() { path_prefix: &[], }, }; - let result = ref_alloc::allocate_refs(root, &mut refmap, &config).unwrap(); + let (result, refmap) = run_captured_with(root, config); assert_eq!(refmap.len(), 4, "should allocate 4 refs total"); let result_value = serde_json::to_value(&result).unwrap(); diff --git a/crates/core/src/snapshot/tokenless_capture_support.rs b/crates/core/src/snapshot/tokenless_capture_support.rs new file mode 100644 index 000000000..d3fe6c9ee --- /dev/null +++ b/crates/core/src/snapshot/tokenless_capture_support.rs @@ -0,0 +1,295 @@ +use std::sync::{ + Mutex, + atomic::{AtomicU32, Ordering}, +}; + +use crate::adapter::{ActionOps, InputOps, ObservationOps, SystemOps, TreeOptions}; +use crate::locator::live_locator::{ObservationRequest, ObservationRoot, ObservedTree}; +use crate::{ + AccessibilityNode, AdapterError, Deadline, NativeHandle, RefEntry, RefMap, WindowInfo, + refs::refs_store::RefStore, +}; + +pub(super) const TOKEN: &str = "axtok1:run"; + +/// Rewrites what the whole-window walk returns, so a test can make that one +/// read come back cut short, fail, or depend on the request it was sent. +pub(super) type WholeReadFault = + fn(&ObservationRequest, AccessibilityNode) -> Result; + +pub(super) struct CaptureAdapter { + tree: Mutex, + window: WindowInfo, + captures: AtomicU32, + pub(super) subtrees: AtomicU32, + pub(super) strict: AtomicU32, + anchors: Mutex>, + whole_read: Mutex>, +} + +impl CaptureAdapter { + pub(super) fn new(tree: AccessibilityNode) -> Self { + Self { + tree: Mutex::new(tree), + window: fixture_window(), + captures: AtomicU32::new(0), + subtrees: AtomicU32::new(0), + strict: AtomicU32::new(0), + anchors: Mutex::new(Vec::new()), + whole_read: Mutex::new(None), + } + } + + pub(super) fn fault_whole_read(&self, fault: WholeReadFault) { + *self.whole_read.lock().expect("the fault locks") = Some(fault); + } + + pub(super) fn show(&self, tree: AccessibilityNode) { + *self.tree.lock().expect("the fixture tree locks") = tree; + } + + pub(super) fn captures(&self) -> u32 { + self.captures.load(Ordering::SeqCst) + } + + pub(super) fn anchored(&self) -> Vec { + self.anchors.lock().expect("the anchors lock").clone() + } +} + +impl ObservationOps for CaptureAdapter { + fn list_windows( + &self, + _filter: &crate::adapter::WindowFilter, + _deadline: Deadline, + ) -> Result, AdapterError> { + Ok(vec![self.window.clone()]) + } + + fn observe_tree( + &self, + root: ObservationRoot<'_>, + request: &ObservationRequest, + ) -> Result { + match root { + ObservationRoot::Window(_) => { + self.captures.fetch_add(1, Ordering::SeqCst); + let tree = self.tree.lock().expect("the fixture tree locks").clone(); + let tree = cut(tree, 0, request.max_logical_depth); + let fault = *self.whole_read.lock().expect("the fault locks"); + let tree = match fault { + Some(fault) + if request.max_logical_depth + >= crate::refs::resolve_saved_ref::WHOLE_WINDOW_DEPTH => + { + fault(request, tree)? + } + _ => tree, + }; + crate::adapter::observed_tree(&root, tree) + } + ObservationRoot::Element { entry, .. } => { + self.subtrees.fetch_add(1, Ordering::SeqCst); + let tree = self.tree.lock().expect("the fixture tree locks").clone(); + match entry + .identity + .retained_object + .as_deref() + .and_then(|token| find_token_subtree(tree, token)) + { + Some(subtree) => crate::adapter::observed_tree( + &root, + cut(subtree, 0, request.max_logical_depth), + ), + None => crate::adapter::saved_ref_element_tree(root), + } + } + } + } + + fn resolve_element_strict( + &self, + _entry: &RefEntry, + _deadline: Deadline, + ) -> Result { + self.strict.fetch_add(1, Ordering::SeqCst); + Ok(NativeHandle::new("strict")) + } + + fn resolve_locator_anchor( + &self, + entry: &RefEntry, + _deadline: Deadline, + ) -> Result { + self.anchors + .lock() + .expect("the anchors lock") + .push(entry.clone()); + Ok(NativeHandle::new("anchor")) + } +} + +impl ActionOps for CaptureAdapter {} +impl InputOps for CaptureAdapter {} +impl SystemOps for CaptureAdapter {} + +pub(super) fn fixture_window() -> WindowInfo { + WindowInfo { + id: "w-1".into(), + title: "Fixture".into(), + app: "Fixture".into(), + pid: crate::ProcessId::new(1), + process_instance: Some("test-instance".into()), + bounds: None, + state: crate::WindowState::default(), + } +} + +pub(super) fn node(role: &str, name: &str) -> AccessibilityNode { + crate::adapter::minimal_accessibility_node(role, name) +} + +pub(super) fn group(children: Vec) -> AccessibilityNode { + let mut group = node("group", ""); + group.children = children; + group +} + +pub(super) fn window(children: Vec) -> AccessibilityNode { + let mut root = node("window", "Fixture"); + root.children = children; + root +} + +/// The subtree whose root carries `token`, as an element read of it returns. +fn find_token_subtree(node: AccessibilityNode, token: &str) -> Option { + if node.identity.retained_object.as_deref() == Some(token) { + return Some(node); + } + node.children + .into_iter() + .find_map(|child| find_token_subtree(child, token)) +} + +/// The window as a walk limited to `max_logical_depth` returns it: a container +/// whose children sit below the limit keeps none and carries their count. +pub(super) fn cut( + mut node: AccessibilityNode, + depth: u8, + max_logical_depth: u8, +) -> AccessibilityNode { + if node.children.is_empty() { + return node; + } + if depth >= max_logical_depth { + node.children_count = node + .children_count + .or(u32::try_from(node.children.len()).ok()); + node.children = Vec::new(); + return node; + } + node.children = node + .children + .into_iter() + .map(|child| cut(child, depth + 1, max_logical_depth)) + .collect(); + node +} + +/// A cut container carrying its own name three levels down, with a button the +/// skeleton cannot reach below it. +pub(super) fn anchored_window() -> AccessibilityNode { + let mut sidebar = node("group", "Sidebar"); + sidebar.children = vec![group(vec![node("button", "Save")])]; + window(vec![group(vec![group(vec![sidebar])])]) +} + +/// `anchored_window` whose anchor re-identifies itself by an element token, so +/// the button below it is the one tokenless ref a drill from it must prove. +pub(super) fn token_anchored_window() -> AccessibilityNode { + let mut sidebar = node("group", "Sidebar"); + sidebar.identity.retained_object = Some(TOKEN.into()); + sidebar.children = vec![group(vec![node("button", "Save")])]; + window(vec![group(vec![group(vec![sidebar])])]) +} + +/// `anchored_window` with a twin of the anchor hidden below the skeleton depth, +/// beside a second anchor that stays unique. +pub(super) fn twinned_window() -> AccessibilityNode { + let mut twin = node("group", "Sidebar"); + twin.children = vec![node("button", "Twin")]; + let mut sidebar = node("group", "Sidebar"); + sidebar.children = vec![group(vec![twin])]; + let mut toolbar = node("group", "Toolbar"); + toolbar.children = vec![node("button", "Open")]; + window(vec![group(vec![group(vec![sidebar, toolbar])])]) +} + +/// `anchored_window` with the skeleton anchor pushed one sibling to the right, +/// so the path the snapshot saved is no longer where the anchor lives. +pub(super) fn shifted_window() -> AccessibilityNode { + let mut sidebar = node("group", "Sidebar"); + sidebar.children = vec![group(vec![node("button", "Save")])]; + window(vec![group(vec![ + node("button", "Inserted"), + group(vec![sidebar]), + ])]) +} + +pub(super) fn skeleton_opts() -> TreeOptions { + TreeOptions { + max_depth: 3, + skeleton: true, + ..Default::default() + } +} + +pub(super) fn drill_opts() -> TreeOptions { + TreeOptions { + max_depth: 10, + ..Default::default() + } +} + +pub(super) fn snapshot(adapter: &CaptureAdapter) -> crate::snapshot::SnapshotResult { + crate::snapshot::build( + adapter, + &skeleton_opts(), + Some("Fixture"), + None, + Deadline::standard().expect("a standard deadline"), + ) + .expect("the window snapshots") +} + +pub(super) fn save(refmap: &RefMap) -> String { + RefStore::new() + .expect("the snapshot store opens") + .save_new_snapshot(refmap) + .expect("the snapshot refmap saves") +} + +pub(super) fn named<'a>(refmap: &'a RefMap, name: &str) -> (String, &'a RefEntry) { + (1..=64) + .filter_map(|index| { + let ref_id = format!("@e{index}"); + refmap + .get(&ref_id) + .filter(|entry| entry.identity.name.as_deref() == Some(name)) + .map(|entry| (ref_id, entry)) + }) + .next() + .unwrap_or_else(|| panic!("no ref names '{name}'")) +} + +pub(super) fn resolve( + adapter: &CaptureAdapter, + entry: &RefEntry, +) -> Result { + crate::refs::resolve_saved_ref::resolve_saved_ref( + adapter, + entry, + Deadline::standard().expect("a standard deadline"), + None, + ) +} diff --git a/crates/core/src/snapshot/tokenless_capture_tests.rs b/crates/core/src/snapshot/tokenless_capture_tests.rs new file mode 100644 index 000000000..aeeb15b0b --- /dev/null +++ b/crates/core/src/snapshot/tokenless_capture_tests.rs @@ -0,0 +1,200 @@ +use std::sync::atomic::Ordering; + +use super::tokenless_capture_support::*; +use crate::locator::live_locator::ObservationRequest; +use crate::refs::refs_test_support::HomeGuard; +use crate::{AccessibilityNode, AdapterError, Deadline, ErrorCode, RefMap}; + +fn entries(refmap: &RefMap) -> Vec<&crate::RefEntry> { + (1..=64) + .filter_map(|index| refmap.get(&format!("@e{index}"))) + .collect() +} + +fn cut_short( + _request: &ObservationRequest, + mut tree: AccessibilityNode, +) -> Result { + let mut unread = node("scrollarea", "Unread"); + unread.children_count = Some(3); + unread.subtree_truncated = true; + tree.children.push(unread); + Ok(tree) +} + +fn faulted( + _request: &ObservationRequest, + _tree: AccessibilityNode, +) -> Result { + Err(AdapterError::new( + ErrorCode::ActionFailed, + "a node below the shallow depth faulted", + )) +} + +fn needs_forced_renderer( + request: &ObservationRequest, + tree: AccessibilityNode, +) -> Result { + if request.observation_mode.force_renderer_accessibility { + return Ok(tree); + } + Err(AdapterError::new( + ErrorCode::ActionFailed, + "the renderer tree is thin until accessibility is forced", + )) +} + +#[test] +fn a_whole_window_read_cut_short_keeps_the_complete_shallow_tree() { + let _guard = HomeGuard::new(); + let adapter = CaptureAdapter::new(anchored_window()); + adapter.fault_whole_read(cut_short); + + let captured = snapshot(&adapter); + assert_eq!(adapter.captures(), 2); + assert!(captured.complete, "the shallow walk finished"); + assert_eq!( + captured.identity_unproven.as_deref(), + Some("the whole-window read was cut short") + ); + let tree = serde_json::to_string(&captured.tree).expect("the tree serializes"); + assert!( + !tree.contains("Unread"), + "the cut-short read is not presented" + ); + assert!( + entries(&captured.refmap).is_empty(), + "a read that was cut short proves nothing, so no anchor is handed out" + ); +} + +#[test] +fn a_failed_whole_window_read_keeps_the_shallow_tree() { + let _guard = HomeGuard::new(); + let adapter = CaptureAdapter::new(anchored_window()); + adapter.fault_whole_read(faulted); + + let captured = snapshot(&adapter); + assert!(captured.complete, "the shallow walk finished"); + assert_eq!( + captured.identity_unproven.as_deref(), + Some("a node below the shallow depth faulted") + ); + assert!(entries(&captured.refmap).is_empty()); +} + +#[test] +fn the_whole_window_read_carries_the_forced_renderer_hint() { + let _guard = HomeGuard::new(); + let adapter = CaptureAdapter::new(anchored_window()); + adapter.fault_whole_read(needs_forced_renderer); + let forced = crate::adapter::TreeOptions { + force_renderer_accessibility: true, + ..skeleton_opts() + }; + + let captured = crate::snapshot::build( + &adapter, + &forced, + Some("Fixture"), + None, + Deadline::standard().expect("a standard deadline"), + ) + .expect("the window snapshots"); + assert!(captured.identity_unproven.is_none()); + let (anchor_ref, anchor) = named(&captured.refmap, "Sidebar"); + assert!( + anchor.identity.unique_at_capture, + "the forced read is adopted and proves the anchor" + ); + assert!(anchor.source.renderer_accessibility_forced); + resolve(&adapter, anchor).expect("resolution reads the window the way the capture did"); + + let snapshot_id = save(&captured.refmap); + let drilled = crate::snapshot::snapshot_ref::run_from_ref( + &adapter, + &drill_opts(), + &anchor_ref, + Some(&snapshot_id), + ) + .expect("a drill reads the window the way the capture did"); + let (_, button) = named(&drilled.refmap, "Save"); + assert!(button.source.renderer_accessibility_forced); + resolve(&adapter, button).expect("a drilled ref reads the window the way the capture did"); +} + +#[test] +fn an_incomplete_window_offers_no_drill_anchor() { + let _guard = HomeGuard::new(); + let mut hidden = node("scrollarea", "Feed"); + hidden.children_count = Some(3); + hidden.subtree_truncated = true; + let mut sidebar = node("group", "Sidebar"); + sidebar.children = vec![group(vec![node("button", "Save")])]; + let adapter = CaptureAdapter::new(window(vec![group(vec![group(vec![sidebar, hidden])])])); + + let captured = snapshot(&adapter); + assert!(!captured.complete, "the window was not read whole"); + assert_eq!( + captured.refmap.len(), + 0, + "an unread region proves nothing, so no anchor is handed out" + ); +} + +#[test] +fn a_shallow_capture_of_tokened_refs_takes_one_walk() { + let _guard = HomeGuard::new(); + let mut run = node("button", "Run"); + run.identity.retained_object = Some(TOKEN.into()); + let mut deep = node("group", ""); + deep.identity.retained_object = Some("axtok1:deep".into()); + deep.children = vec![node("button", "Deep")]; + let adapter = CaptureAdapter::new(window(vec![run, group(vec![group(vec![deep])])])); + + let captured = snapshot(&adapter); + assert_eq!( + adapter.captures(), + 1, + "a ref that re-identifies itself by token needs no second walk" + ); + let (_, ref_entry) = named(&captured.refmap, "Run"); + assert_eq!(ref_entry.identity.retained_object.as_deref(), Some(TOKEN)); + + resolve(&adapter, ref_entry).expect("a token resolves through the adapter"); + assert_eq!(adapter.strict.load(Ordering::SeqCst), 1); + assert!(adapter.anchored().is_empty()); +} + +#[test] +fn a_deep_capture_cut_short_is_flagged_as_unproven() { + let _guard = HomeGuard::new(); + let mut unread = node("scrollarea", "Feed"); + unread.children_count = Some(3); + unread.subtree_truncated = true; + let adapter = CaptureAdapter::new(window(vec![node("button", "Run"), unread])); + let deep = crate::adapter::TreeOptions { + max_depth: crate::refs::resolve_saved_ref::WHOLE_WINDOW_DEPTH, + ..Default::default() + }; + + let captured = crate::snapshot::build( + &adapter, + &deep, + Some("Fixture"), + None, + Deadline::standard().expect("a standard deadline"), + ) + .expect("the window snapshots"); + + assert_eq!( + adapter.captures(), + 1, + "a deep capture is its own whole read" + ); + assert_eq!( + captured.identity_unproven.as_deref(), + Some("the window read was cut short") + ); +} diff --git a/crates/core/src/snapshot/tokenless_drill_tests.rs b/crates/core/src/snapshot/tokenless_drill_tests.rs new file mode 100644 index 000000000..0715fdcce --- /dev/null +++ b/crates/core/src/snapshot/tokenless_drill_tests.rs @@ -0,0 +1,188 @@ +use std::sync::atomic::Ordering; + +use super::tokenless_capture_support::*; +use crate::refs::refs_test_support::HomeGuard; +use crate::snapshot::snapshot_ref::run_from_ref; + +#[test] +fn a_skeleton_anchor_resolves_and_its_drill_resolves() { + let _guard = HomeGuard::new(); + let adapter = CaptureAdapter::new(anchored_window()); + + let captured = snapshot(&adapter); + assert_eq!( + adapter.captures(), + 1, + "an adapter without element tokens reads the whole window once" + ); + let (anchor_ref, anchor) = named(&captured.refmap, "Sidebar"); + assert!( + anchor.identity.unique_at_capture, + "an inert anchor is counted by the capture census too" + ); + assert_eq!(captured.refmap.len(), 1); + let sidebar = &captured.tree.children[0].children[0].children[0]; + assert_eq!(sidebar.children_count, Some(1), "a cut keeps its count"); + + let snapshot_id = save(&captured.refmap); + let before = adapter.captures(); + let drilled = run_from_ref(&adapter, &drill_opts(), &anchor_ref, Some(&snapshot_id)) + .expect("a tokenless drill reuses the walk its root resolved through"); + assert_eq!(adapter.captures(), before + 1, "one walk, not two"); + assert_eq!( + adapter.subtrees.load(Ordering::SeqCst), + 0, + "a tokenless drill never reads the subtree again" + ); + + let (_, button) = named(&drilled.refmap, "Save"); + assert!( + button.identity.unique_at_capture, + "the drill counted its refs against the window it will be resolved in" + ); + assert_eq!(button.identity.role, "button"); + + resolve(&adapter, button).expect("the drilled ref resolves"); + let anchored = adapter.anchored(); + assert_eq!(anchored.len(), 1); + assert_eq!(anchored[0].scope.path.as_slice(), &[0, 0, 0, 0, 0]); +} + +#[test] +fn a_twin_below_the_skeleton_depth_is_not_offered_as_an_anchor() { + let _guard = HomeGuard::new(); + let adapter = CaptureAdapter::new(twinned_window()); + + let captured = snapshot(&adapter); + assert_eq!( + captured.refmap.len(), + 1, + "the twin is never handed out as a drill target, so only the proven anchor remains" + ); + let (_, toolbar) = named(&captured.refmap, "Toolbar"); + assert!( + toolbar.identity.unique_at_capture, + "the census that refuses the twin still proves its neighbour" + ); + + resolve(&adapter, toolbar).expect("a proven anchor resolves"); + assert_eq!(adapter.anchored().len(), 1); + assert_eq!(adapter.anchored()[0].scope.path.as_slice(), &[0, 0, 1]); +} + +#[test] +fn a_drill_under_a_moved_root_records_the_live_path() { + let _guard = HomeGuard::new(); + let adapter = CaptureAdapter::new(anchored_window()); + + let captured = snapshot(&adapter); + let (anchor_ref, _) = named(&captured.refmap, "Sidebar"); + assert_eq!( + captured + .refmap + .get(&anchor_ref) + .expect("anchor") + .scope + .path + .as_slice(), + &[0, 0, 0] + ); + let snapshot_id = save(&captured.refmap); + + adapter.show(shifted_window()); + let first = run_from_ref(&adapter, &drill_opts(), &anchor_ref, Some(&snapshot_id)) + .expect("the moved anchor resolves"); + let (first_button, button) = named(&first.refmap, "Save"); + assert_eq!( + button.scope.path.as_slice(), + &[0, 1, 0, 0, 0], + "a drilled ref is recorded in the path of the node the drill matched" + ); + + let second = run_from_ref(&adapter, &drill_opts(), &anchor_ref, Some(&snapshot_id)) + .expect("the re-drill replaces one subtree"); + let (second_button, _) = named(&second.refmap, "Save"); + assert!( + second.refmap.get(&first_button).is_none(), + "the previous drill's refs are pruned through their root, not by path" + ); + assert_ne!(first_button, second_button); + assert!( + second.refmap.get(&anchor_ref).is_some(), + "the drill never removes its own root" + ); +} + +#[test] +fn a_token_rooted_drill_proves_its_tokenless_descendants() { + let _guard = HomeGuard::new(); + let adapter = CaptureAdapter::new(token_anchored_window()); + + let captured = snapshot(&adapter); + let (anchor_ref, anchor) = named(&captured.refmap, "Sidebar"); + assert_eq!(anchor.identity.retained_object.as_deref(), Some(TOKEN)); + let snapshot_id = save(&captured.refmap); + + let drilled = run_from_ref(&adapter, &drill_opts(), &anchor_ref, Some(&snapshot_id)) + .expect("a token-rooted drill is located in the whole-window read"); + + let (_, button) = named(&drilled.refmap, "Save"); + assert!( + button.identity.unique_at_capture, + "the button below a token root is counted against the whole window" + ); + resolve(&adapter, button).expect("the drilled ref resolves"); +} + +#[test] +fn a_token_rooted_drill_of_tokened_refs_reads_only_its_subtree() { + let _guard = HomeGuard::new(); + let mut tokened = node("button", "Save"); + tokened.identity.retained_object = Some("axtok1:save".into()); + let mut sidebar = node("group", "Sidebar"); + sidebar.identity.retained_object = Some(TOKEN.into()); + sidebar.children = vec![group(vec![tokened])]; + let adapter = CaptureAdapter::new(window(vec![group(vec![group(vec![sidebar])])])); + let captured = snapshot(&adapter); + let (anchor_ref, _) = named(&captured.refmap, "Sidebar"); + let snapshot_id = save(&captured.refmap); + let windows_before = adapter.captures(); + + let drilled = run_from_ref(&adapter, &drill_opts(), &anchor_ref, Some(&snapshot_id)) + .expect("a drill whose refs all carry tokens reads its subtree"); + + assert_eq!(adapter.captures(), windows_before, "no whole-window read"); + assert_eq!(adapter.subtrees.load(Ordering::SeqCst), 1); + let (_, button) = named(&drilled.refmap, "Save"); + assert_eq!( + button.identity.retained_object.as_deref(), + Some("axtok1:save") + ); +} + +#[test] +fn a_token_rooted_drill_whose_window_read_is_cut_short_replaces_nothing() { + let _guard = HomeGuard::new(); + let adapter = CaptureAdapter::new(token_anchored_window()); + adapter.fault_whole_read(|_, mut tree| { + let mut unread = node("scrollarea", "Unread"); + unread.children_count = Some(3); + unread.subtree_truncated = true; + tree.children.push(unread); + Ok(tree) + }); + let captured = snapshot(&adapter); + let (anchor_ref, _) = named(&captured.refmap, "Sidebar"); + let snapshot_id = save(&captured.refmap); + + let Err(error) = run_from_ref(&adapter, &drill_opts(), &anchor_ref, Some(&snapshot_id)) else { + panic!("a drill is all or nothing"); + }; + + assert_eq!(error.code(), "TIMEOUT"); + let stored = crate::refs::refs_store::RefStore::new() + .expect("the snapshot store opens") + .load_snapshot(&snapshot_id) + .expect("the snapshot loads"); + assert_eq!(stored.len(), captured.refmap.len(), "no ref was replaced"); +} diff --git a/crates/core/src/snapshot/window_capture.rs b/crates/core/src/snapshot/window_capture.rs new file mode 100644 index 000000000..d5687200c --- /dev/null +++ b/crates/core/src/snapshot/window_capture.rs @@ -0,0 +1,147 @@ +use crate::{ + AppError, WindowInfo, + adapter::{PlatformAdapter, TreeOptions}, + locator::live_locator::{ObservationRequest, ObservationRoot, ObservedTree}, +}; + +/// One window capture: the observation every ref is counted against, plus the +/// logical depth the presented tree is cut at when that observation reaches +/// deeper than the caller asked for. +pub(crate) struct WindowCapture { + pub(crate) observed: ObservedTree, + presentation_depth: Option, + /// Why the whole-window read that would prove this capture's tokenless + /// refs failed or came back cut short, so those refs stay unproven. + pub(crate) identity_unproven: Option, +} + +impl WindowCapture { + pub(crate) fn into_parts( + self, + ) -> Result<(crate::AccessibilityNode, bool, usize, ObservedTree), AppError> { + let Self { + mut observed, + presentation_depth, + .. + } = self; + let (tree, complete, nodes_observed) = match presentation_depth { + Some(depth) => observed.project_window(depth)?, + None => observed.project_partial()?, + }; + Ok((tree, complete, nodes_observed, observed)) + } +} + +/// Observes one window, and for a presentation whose shallow walk leaves refs +/// that can only re-identify themselves by their recorded identity, observes +/// the whole window instead and presents a cut of that one observation. +/// +/// A shallow walk never sees the nodes below its depth, so the census that +/// proves a ref unique cannot run over it, and every ref it allocates is +/// refused at resolution. A ref carrying an element token re-identifies itself, +/// so a capture made of those keeps its single shallow walk. A whole-window +/// read that fails or comes back incomplete keeps that shallow tree and proves +/// nothing, rather than counting a narrower window or failing a capture whose +/// shallow walk already succeeded. +pub(crate) fn capture_window( + adapter: &dyn PlatformAdapter, + opts: &TreeOptions, + window: &WindowInfo, + deadline: crate::Deadline, +) -> Result { + let depth = ObservationRequest::snapshot_logical_depth(opts); + let shallow = depth < crate::refs::resolve_saved_ref::WHOLE_WINDOW_DEPTH; + let mut unproven = None; + if shallow && !adapter.retains_element_tokens() { + let attempt = deadline.capped(deadline.remaining() * 3 / 4); + match whole_window(adapter, opts, window, attempt) { + Ok(observed) => { + return Ok(WindowCapture { + observed, + presentation_depth: Some(depth), + identity_unproven: None, + }); + } + Err(reason) => unproven = Some(reason), + } + } + let observed = observe_window(adapter, opts, window, deadline)?; + if !needs_whole_window(&observed) { + return Ok(WindowCapture { + observed, + presentation_depth: None, + identity_unproven: None, + }); + } + let unproven = match unproven { + Some(reason) => reason, + None if !shallow => "the window read was cut short".to_owned(), + None => match whole_window(adapter, opts, window, deadline) { + Ok(whole) => { + return Ok(WindowCapture { + observed: whole, + presentation_depth: Some(depth), + identity_unproven: None, + }); + } + Err(reason) => reason, + }, + }; + Ok(WindowCapture { + observed, + presentation_depth: None, + identity_unproven: Some(unproven), + }) +} + +/// The whole-window read that proves tokenless refs, or why it cannot. +fn whole_window( + adapter: &dyn PlatformAdapter, + opts: &TreeOptions, + window: &WindowInfo, + deadline: crate::Deadline, +) -> Result { + match crate::refs::resolve_saved_ref::observe_whole_window( + adapter, opts, window, deadline, None, + ) { + Ok(whole) if whole.is_exhaustive() => Ok(whole), + Ok(_) => Err("the whole-window read was cut short".to_owned()), + Err(error) => Err(error.message), + } +} + +fn observe_window( + adapter: &dyn PlatformAdapter, + opts: &TreeOptions, + window: &WindowInfo, + deadline: crate::Deadline, +) -> Result { + let observation_options = opts.with_ref_identity_bounds(); + let request = ObservationRequest::snapshot(&observation_options, deadline).validate()?; + crate::tree::renderer_accessibility::observe_tree( + adapter, + ObservationRoot::Window(window), + &request, + None, + ) +} + +/// Whether the presentation allocation builds from this observation keeps an +/// entry that can only re-identify itself through the capture proof: a ref or a +/// skeleton drill anchor whose identity carries no element token. Only a whole +/// window proves such an entry unique, so a capture that would present one is +/// re-observed. A presentation whose selected entries all carry a token resolves +/// through the tokens and keeps its single shallow walk. +fn needs_whole_window(observed: &ObservedTree) -> bool { + if observed.is_exhaustive() { + return false; + } + let mut presented = observed.clone(); + presented + .project_partial() + .is_ok_and(|(presented, _, _)| crate::refs::ref_alloc::selects_tokenless_entry(&presented)) +} + +#[cfg(test)] +#[path = "window_capture_tests.rs"] +mod tests; diff --git a/crates/core/src/snapshot/window_capture_tests.rs b/crates/core/src/snapshot/window_capture_tests.rs new file mode 100644 index 000000000..5c6634f3f --- /dev/null +++ b/crates/core/src/snapshot/window_capture_tests.rs @@ -0,0 +1,84 @@ +use super::needs_whole_window; +use crate::locator::live_locator::{ObservationRoot, ObservedTree}; +use crate::{AccessibilityNode, WindowInfo}; + +fn window() -> WindowInfo { + WindowInfo { + id: "w-1".into(), + title: "Fixture".into(), + app: "Fixture".into(), + pid: crate::ProcessId::new(1), + process_instance: Some("test-instance".into()), + bounds: None, + state: crate::WindowState::default(), + } +} + +fn node(role: &str, name: &str) -> AccessibilityNode { + crate::adapter::minimal_accessibility_node(role, name) +} + +fn tokened(role: &str, name: &str, token: &str) -> AccessibilityNode { + let mut node = node(role, name); + node.identity.retained_object = Some(format!("axtok1:{token}")); + node +} + +fn observed(root: AccessibilityNode) -> ObservedTree { + let window = window(); + crate::adapter::observed_tree(&ObservationRoot::Window(&window), root) + .expect("the fixture observes") +} + +/// A window whose walk stopped at a container: the container keeps the count of +/// what it hid, so a shallow presentation can still offer it as a drill target. +fn cut_window(anchor: AccessibilityNode, ref_able: AccessibilityNode) -> AccessibilityNode { + let mut hidden = node("group", "Hidden"); + hidden.children = vec![ref_able]; + let mut anchor = anchor; + anchor.children_count = Some(1); + anchor.children = Vec::new(); + let mut root = node("window", "Fixture"); + root.children = vec![anchor, hidden]; + root +} + +#[test] +fn a_cut_capture_of_tokenless_refs_needs_the_whole_window() { + let observed = observed(cut_window(node("group", "Sidebar"), node("button", "Save"))); + + assert!(needs_whole_window(&observed)); +} + +#[test] +fn a_cut_capture_whose_refs_carry_tokens_needs_no_second_walk() { + let observed = observed(cut_window( + tokened("group", "Sidebar", "sidebar"), + tokened("button", "Save", "save"), + )); + + assert!(!needs_whole_window(&observed)); +} + +/// A cut boundary with nothing a later resolve could re-identify it by is not a +/// drill target, so it is not an entry the census would prove: the capture keeps +/// its single walk even though that boundary omits descendants. +#[test] +fn a_cut_boundary_that_cannot_anchor_a_drill_needs_no_second_walk() { + let observed = observed(cut_window( + node("group", ""), + tokened("button", "Save", "save"), + )); + + assert!(!needs_whole_window(&observed)); +} + +#[test] +fn a_capture_that_reached_the_whole_window_needs_no_second_walk() { + let mut save = node("button", "Save"); + save.children = vec![node("group", "Icon")]; + let mut root = node("window", "Fixture"); + root.children = vec![save]; + + assert!(!needs_whole_window(&observed(root))); +} diff --git a/crates/core/src/trace/trace_artifacts_dispatch_budget_tests.rs b/crates/core/src/trace/trace_artifacts_dispatch_budget_tests.rs index ca66aa8f5..624caf17e 100644 --- a/crates/core/src/trace/trace_artifacts_dispatch_budget_tests.rs +++ b/crates/core/src/trace/trace_artifacts_dispatch_budget_tests.rs @@ -45,6 +45,7 @@ impl ObservationOps for DeadlineConsumingScreenshotAdapter { "Run", [crate::actions::capability::CLICK] ); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for DeadlineConsumingScreenshotAdapter { @@ -106,7 +107,7 @@ fn slow_pre_capture_preserves_dispatch_budget() { context: &context, }, ActionRequest::headless(Action::Click).with_timeout_ms(Some(500)), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ); assert!( @@ -160,7 +161,7 @@ fn contended_artifact_lock_preserves_dispatch_budget() { context: &context, }, ActionRequest::headless(Action::Click).with_timeout_ms(Some(ACTION_BUDGET_MS)), - crate::ref_action::dispatch_resolved, + crate::ref_action::ref_action_single::dispatch_resolved, ); assert!(result.is_ok(), "artifact lock blocked dispatch: {result:?}"); diff --git a/crates/core/src/trace/trace_artifacts_tests.rs b/crates/core/src/trace/trace_artifacts_tests.rs index b12de2964..373f1b1b8 100644 --- a/crates/core/src/trace/trace_artifacts_tests.rs +++ b/crates/core/src/trace/trace_artifacts_tests.rs @@ -43,6 +43,7 @@ pub(super) fn entry(pid: u32) -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: true, }, geometry: crate::RefGeometry { bounds_space: None, @@ -59,6 +60,7 @@ pub(super) fn entry(pid: u32) -> RefEntry { source_window_title: Some("Fixture".into()), source_window_bounds_hash: None, source_surface: crate::adapter::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: crate::RefScope { root_ref: None, @@ -92,6 +94,7 @@ impl ObservationOps for PngAdapter { } crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for PngAdapter { @@ -155,6 +158,7 @@ impl ObservationOps for FailingScreenshotAdapter { } crate::adapter::complete_live_observation!("button", "Run", [capability::CLICK]); + crate::adapter::saved_ref_scope!("button", Some("Run")); } impl ActionOps for FailingScreenshotAdapter { diff --git a/crates/core/src/trace/trace_artifacts_toctou_tests.rs b/crates/core/src/trace/trace_artifacts_toctou_tests.rs index 1fe86ba08..714fa0094 100644 --- a/crates/core/src/trace/trace_artifacts_toctou_tests.rs +++ b/crates/core/src/trace/trace_artifacts_toctou_tests.rs @@ -27,6 +27,8 @@ impl ObservationOps for ScreenshotMutationAdapter { Ok(NativeHandle::null()) } + crate::adapter::saved_ref_scope!("button", Some("Run")); + fn get_element_bounds( &self, _handle: &NativeHandle, diff --git a/crates/core/src/tree/renderer_accessibility.rs b/crates/core/src/tree/renderer_accessibility.rs index 4bc5fe9a5..a3a99b463 100644 --- a/crates/core/src/tree/renderer_accessibility.rs +++ b/crates/core/src/tree/renderer_accessibility.rs @@ -13,10 +13,15 @@ use crate::{ const INITIAL_ACTIVATION_RETRY: std::time::Duration = std::time::Duration::from_millis(25); const MAX_ACTIVATION_RETRY: std::time::Duration = std::time::Duration::from_millis(400); +/// Observes through the adapter, settling a renderer that has not published +/// its accessibility tree yet. `lease` is the interaction lease the caller +/// already holds, if any: activation reuses it rather than queueing behind the +/// same process's non-reentrant lease until the deadline expires. pub(crate) fn observe_tree( adapter: &dyn PlatformAdapter, root: ObservationRoot<'_>, request: &ObservationRequest, + lease: Option<&crate::InteractionLease>, ) -> Result { let mut request = *request; let mut activated = false; @@ -26,7 +31,7 @@ pub(crate) fn observe_tree( Ok(tree) => return Ok(tree), Err(error) if error.requires_renderer_accessibility_activation() => { if !activated { - request.deadline = activate(adapter, root, request.deadline)?; + request.deadline = activate(adapter, root, request.deadline, lease)?; activated = true; } let remaining = request.deadline.remaining(); @@ -45,9 +50,10 @@ fn activate( adapter: &dyn PlatformAdapter, root: ObservationRoot<'_>, observing: crate::Deadline, + lease: Option<&crate::InteractionLease>, ) -> Result { let started = std::time::Instant::now(); - switch_on(adapter, root).map_err(|error| match error { + switch_on(adapter, root, lease).map_err(|error| match error { AppError::Adapter(ref adapter_error) if adapter_error.code == crate::ErrorCode::Timeout => { observing.timeout_error().into() } @@ -56,9 +62,19 @@ fn activate( Ok(observing.extended_by(started.elapsed())) } -fn switch_on(adapter: &dyn PlatformAdapter, root: ObservationRoot<'_>) -> Result<(), AppError> { - let acquired = adapter.acquire_interaction_lease(crate::Deadline::standard()?)?; - adapter.activate_renderer_accessibility(root_process(root)?, &acquired)?; +fn switch_on( + adapter: &dyn PlatformAdapter, + root: ObservationRoot<'_>, + lease: Option<&crate::InteractionLease>, +) -> Result<(), AppError> { + let process = root_process(root)?; + match lease { + Some(lease) => adapter.activate_renderer_accessibility(process, lease)?, + None => { + let acquired = adapter.acquire_interaction_lease(crate::Deadline::standard()?)?; + adapter.activate_renderer_accessibility(process, &acquired)?; + } + } Ok(()) } diff --git a/crates/core/src/tree/renderer_accessibility_tests.rs b/crates/core/src/tree/renderer_accessibility_tests.rs index 5248b59fc..64744439c 100644 --- a/crates/core/src/tree/renderer_accessibility_tests.rs +++ b/crates/core/src/tree/renderer_accessibility_tests.rs @@ -94,7 +94,7 @@ fn activation_lease_is_dropped_before_observation_retry() { .validate() .unwrap(); - observe_tree(&adapter, ObservationRoot::Window(&window), &request).unwrap(); + observe_tree(&adapter, ObservationRoot::Window(&window), &request, None).unwrap(); assert_eq!(adapter.observations.load(Ordering::SeqCst), 2); assert_eq!(adapter.activations.load(Ordering::SeqCst), 1); @@ -159,7 +159,7 @@ fn a_renderer_that_never_activates_is_not_re_walked_continuously() { .validate() .unwrap(); - observe_tree(&adapter, ObservationRoot::Window(&window), &request) + observe_tree(&adapter, ObservationRoot::Window(&window), &request, None) .expect_err("a renderer that never activates must still fail at the deadline"); let attempts = adapter.observations.load(Ordering::SeqCst); @@ -175,6 +175,7 @@ struct SwitchAdapter { takes: std::time::Duration, switched: AtomicBool, offered_ms: std::sync::atomic::AtomicU64, + acquisitions: AtomicU32, walks: std::sync::Mutex>, } @@ -231,6 +232,7 @@ impl SystemOps for SwitchAdapter { &self, deadline: crate::Deadline, ) -> Result { + self.acquisitions.fetch_add(1, Ordering::SeqCst); crate::InteractionLease::guarded(deadline, ()) } @@ -256,6 +258,7 @@ fn switch(needs_ms: u64, takes_ms: u64) -> SwitchAdapter { takes: std::time::Duration::from_millis(takes_ms), switched: AtomicBool::new(false), offered_ms: std::sync::atomic::AtomicU64::new(0), + acquisitions: AtomicU32::new(0), walks: std::sync::Mutex::new(Vec::new()), } } @@ -278,13 +281,36 @@ fn snapshot_request(deadline: crate::Deadline) -> ObservationRequest { .unwrap() } +#[test] +fn a_held_lease_is_reused_instead_of_acquired_again() { + let adapter = switch(0, 0); + let window = renderer_window(); + let deadline = crate::Deadline::after(60_000).unwrap(); + let held = crate::InteractionLease::guarded(deadline, ()).unwrap(); + + observe_tree( + &adapter, + ObservationRoot::Window(&window), + &snapshot_request(deadline), + Some(&held), + ) + .expect("the switch runs under the lease the caller holds"); + + assert!(adapter.switched.load(Ordering::SeqCst)); + assert_eq!( + adapter.acquisitions.load(Ordering::SeqCst), + 0, + "re-acquiring the lease it is already under would queue until the deadline" + ); +} + #[test] fn time_spent_switching_is_given_back_to_the_observation() { let adapter = switch(0, 300); let window = renderer_window(); let request = snapshot_request(crate::Deadline::after(60_000).unwrap()); - observe_tree(&adapter, ObservationRoot::Window(&window), &request) + observe_tree(&adapter, ObservationRoot::Window(&window), &request, None) .expect("the walk after the switch has the budget the first walk left"); let walks = adapter.walks.lock().unwrap(); @@ -318,6 +344,7 @@ fn a_spent_observation_still_switches_so_the_next_one_is_ready() { &adapter, ObservationRoot::Window(&window), &snapshot_request(spent), + None, ) .expect_err("an observation with nothing left does not walk again"); @@ -339,7 +366,7 @@ fn a_refused_switch_reports_the_operation_budget_and_is_not_retried() { let window = renderer_window(); let request = snapshot_request(crate::Deadline::after(3_000).unwrap()); - let error = observe_tree(&adapter, ObservationRoot::Window(&window), &request) + let error = observe_tree(&adapter, ObservationRoot::Window(&window), &request, None) .expect_err("a switch that cannot run fails the observation"); let AppError::Adapter(error) = error else { @@ -361,7 +388,7 @@ fn the_credit_for_switch_time_stays_inside_an_inherited_deadline() { let parent = crate::Deadline::detached_after(1_000).unwrap(); let _scope = crate::wait::deadline::enter_scope(Some(parent)); - observe_tree(&adapter, ObservationRoot::Window(&window), &request) + observe_tree(&adapter, ObservationRoot::Window(&window), &request, None) .expect("the walk after the switch still fits the inherited deadline"); let walks = adapter.walks.lock().unwrap(); diff --git a/crates/ffi/README.md b/crates/ffi/README.md index 6f5e2b8fb..a2610d8d8 100644 --- a/crates/ffi/README.md +++ b/crates/ffi/README.md @@ -47,6 +47,12 @@ carry the process generation and surface ID of the observed identity. ABI 6 and `AD_EXACT_REF_ENTRY_VERSION` 2 add `bounds_space` to `AdRefGeometry`, inside `AdExactRefEntry`: an `AD_BOUNDS_SPACE_*` constant naming the frame of `bounds`, screen-relative on macOS and Windows and window-relative on Linux. + +ABI 7 makes the two exact-entry ref calls, +`ad_execute_ref_action_exact_with_policy` and `ad_resolve_element_exact`, fail +closed with `AD_RESULT_ERR_ACTION_NOT_SUPPORTED`: an exact entry carries no retained +element token and no capture proof, so it cannot be re-identified. Act through +a snapshot ref with `ad_execute_by_ref`, and get a handle with `ad_find_exact`. `ad_execute_by_ref` and `ad_wait` accept only snapshot-qualified refs (`@:eN`). diff --git a/crates/ffi/include/agent_desktop.h b/crates/ffi/include/agent_desktop.h index 0fb7bedbf..b56987013 100644 --- a/crates/ffi/include/agent_desktop.h +++ b/crates/ffi/include/agent_desktop.h @@ -45,14 +45,15 @@ * * Version-bump rule: increment this constant (and update the header via * `scripts/update-ffi-header.sh`) whenever a breaking change is made to the - * C ABI — a removed or incompatibly-changed `ad_*` symbol, or a layout - * change to any `repr(C)` struct. Additive changes (new `ad_*` symbols, new - * error codes) do **not** require a bump. It is recommended to call `ad_init` - * with the major compiled against the header to verify ABI compatibility; a - * mismatch means the header and dylib are incompatible and the consumer should - * refuse to proceed rather than risk undefined behaviour. + * C ABI — a removed or incompatibly-changed `ad_*` symbol, a layout change to + * any `repr(C)` struct, or a call that now fails where it used to succeed. + * Additive changes (new `ad_*` symbols, new error codes) do **not** require a + * bump. It is recommended to call `ad_init` with the major compiled against + * the header to verify ABI compatibility; a mismatch means the header and + * dylib are incompatible and the consumer should refuse to proceed rather than + * risk undefined behaviour. */ -#define AD_ABI_VERSION_MAJOR 6 +#define AD_ABI_VERSION_MAJOR 7 /** * Maximum byte length (excluding the NUL terminator) accepted for any @@ -1103,8 +1104,11 @@ AdResult ad_execute_action_with_policy(const struct AdAdapter *adapter, struct AdActionResult *out); /** - * Executes a struct-based ref action with exact process-generation and typed - * native-id evidence. + * Validates a struct-based ref action and refuses it without acting. An exact + * entry carries no retained element token and no capture proof, so a + * well-formed call fails with `ACTION_NOT_SUPPORTED` + * (`exact_entry_unsupported`) and a malformed one with `INVALID_ARGS`; + * callers act through a snapshot ref with `ad_execute_by_ref` instead. * * # Safety * @@ -1142,7 +1146,11 @@ AdResult ad_execute_ref_action_exact_with_policy(const struct AdAdapter *adapter AdResult ad_free_handle(const struct AdAdapter *adapter, struct AdNativeHandle *handle); /** - * Resolves an element using process-generation and typed native-id evidence. + * Validates an exact entry and refuses it. An exact entry carries no retained + * element token and no capture proof, so a well-formed one fails with + * `ACTION_NOT_SUPPORTED` (`exact_entry_unsupported`) and a malformed one with + * `INVALID_ARGS`; callers act through a snapshot ref with `ad_execute_by_ref` + * or locate the element with `ad_find_exact` instead. * * # Safety * @@ -1365,7 +1373,7 @@ AdResult ad_execute_by_ref_timeout(const struct AdAdapter *adapter, * interactive elements, persists the refmap to disk, and writes the JSON envelope into `*out`. * * The JSON shape matches `agent-desktop snapshot`: - * `{"version":"2.5","ok":true,"command":"snapshot","data":{"app":"...","window":{...},"ref_count":N,"snapshot_id":"...","complete":true,"tree":{...}}}`. + * `{"version":"2.6","ok":true,"command":"snapshot","data":{"app":"...","window":{...},"ref_count":N,"snapshot_id":"...","complete":true,"tree":{...}}}`. * * `data.complete` is always present. A snapshot that exhausts its observation * budget still succeeds with `"complete":false` and the tree it did observe, @@ -1865,9 +1873,11 @@ void ad_notification_list_free(struct AdNotificationList *list); /** * Finds and strictly resolves one element within a generation-pinned window. * `AdFindQuery.control.selection` must explicitly request first, last, or nth - * behavior when duplicate matches are acceptable. The returned native handle - * is adapter-bound and thread-affine; release it with `ad_free_handle` on the - * resolving thread. + * behavior when duplicate matches are acceptable. The handle is published only + * after the landed element is re-identified by the selected entry's full + * predicate, so a control that moved into the anchored path is refused rather + * than returned. The returned native handle is adapter-bound and thread-affine; + * release it with `ad_free_handle` on the resolving thread. * * # Safety * All pointers must be valid and `out_handle` must be writable. diff --git a/crates/ffi/src/abi_version.rs b/crates/ffi/src/abi_version.rs index 8c5182436..d25338304 100644 --- a/crates/ffi/src/abi_version.rs +++ b/crates/ffi/src/abi_version.rs @@ -6,13 +6,14 @@ use std::ffi::CStr; /// /// Version-bump rule: increment this constant (and update the header via /// `scripts/update-ffi-header.sh`) whenever a breaking change is made to the -/// C ABI — a removed or incompatibly-changed `ad_*` symbol, or a layout -/// change to any `repr(C)` struct. Additive changes (new `ad_*` symbols, new -/// error codes) do **not** require a bump. It is recommended to call `ad_init` -/// with the major compiled against the header to verify ABI compatibility; a -/// mismatch means the header and dylib are incompatible and the consumer should -/// refuse to proceed rather than risk undefined behaviour. -pub const AD_ABI_VERSION_MAJOR: u32 = 6; +/// C ABI — a removed or incompatibly-changed `ad_*` symbol, a layout change to +/// any `repr(C)` struct, or a call that now fails where it used to succeed. +/// Additive changes (new `ad_*` symbols, new error codes) do **not** require a +/// bump. It is recommended to call `ad_init` with the major compiled against +/// the header to verify ABI compatibility; a mismatch means the header and +/// dylib are incompatible and the consumer should refuse to proceed rather than +/// risk undefined behaviour. +pub const AD_ABI_VERSION_MAJOR: u32 = 7; static MISMATCH_MESSAGE: &CStr = c"ABI major version mismatch: recompile against the installed header"; diff --git a/crates/ffi/src/abi_version_tests.rs b/crates/ffi/src/abi_version_tests.rs index a38562c9d..6cb1d00da 100644 --- a/crates/ffi/src/abi_version_tests.rs +++ b/crates/ffi/src/abi_version_tests.rs @@ -1,8 +1,8 @@ use super::*; #[test] -fn abi_major_covers_the_ref_geometry_coordinate_frame() { - assert_eq!(AD_ABI_VERSION_MAJOR, 6); +fn abi_major_covers_the_exact_entry_refusal() { + assert_eq!(AD_ABI_VERSION_MAJOR, 7); } #[test] diff --git a/crates/ffi/src/actions/exact_entry.rs b/crates/ffi/src/actions/exact_entry.rs new file mode 100644 index 000000000..7052c2343 --- /dev/null +++ b/crates/ffi/src/actions/exact_entry.rs @@ -0,0 +1,340 @@ +use crate::convert::string::optional_adapter_string; +use crate::convert::surface::snapshot_surface_from_c; +use crate::types::{AdExactRefEntry, AdIdentifierKind, AdRefEntry}; +use agent_desktop_core::{ + AdapterError, ElementIdentifier, ErrorCode, Rect, RefCapabilities, RefEntry as CoreRefEntry, + RefEntryIdentity, RefGeometry, RefProcess, RefScope, RefSource, +}; + +const MAX_REF_FIELD_BYTES: usize = 65_536; +const MAX_REF_TOKEN_BYTES: usize = 256; + +/// The refusal every well-formed exact entry gets. An exact entry carries no +/// retained element token and no capture proof, so the library cannot tell +/// whether the element it describes is the one a caller means. +pub(crate) fn exact_entry_unsupported() -> AdapterError { + AdapterError::new( + ErrorCode::ActionNotSupported, + "An exact entry carries no capture proof, so the element it describes cannot be re-identified", + ) + .with_suggestion( + "Snapshot the window and act through the ref with ad_execute_by_ref, or locate the \ + element with ad_find_exact.", + ) + .with_details(serde_json::json!({ + "kind": "exact_entry_unsupported", + "retryable": false, + })) + .with_disposition(agent_desktop_core::DeliverySemantics::not_delivered()) +} + +pub(crate) unsafe fn core_ref_entry_from_exact( + exact: &AdExactRefEntry, +) -> Result { + if exact.version != crate::types::exact_ref_entry::AD_EXACT_REF_ENTRY_VERSION + || exact.size as usize != crate::types::exact_ref_entry::AD_EXACT_REF_ENTRY_SIZE + { + return Err(AdapterError::new( + ErrorCode::InvalidArgs, + "AdExactRefEntry version or size does not match this library", + )); + } + let process_instance = unsafe { optional_string(exact.process_instance, "process_instance") }? + .filter(|value| !value.is_empty()) + .ok_or_else(|| { + AdapterError::new(ErrorCode::InvalidArgs, "process_instance is null or empty") + })?; + let native_id = + match unsafe { optional_string(exact.entry.identity.native_id, "identity.native_id") }? { + Some(value) if value.trim().is_empty() => { + return Err(AdapterError::new( + ErrorCode::InvalidArgs, + "identity.native_id is empty", + )); + } + Some(value) => { + let kind = AdIdentifierKind::from_c(exact.identifier_kind).ok_or_else(|| { + AdapterError::new( + ErrorCode::InvalidArgs, + "identity.identifier_kind has an invalid discriminant", + ) + })?; + Some(ElementIdentifier { + kind: kind.to_core(), + value, + }) + } + None => None, + }; + unsafe { decode_core_ref_entry(&exact.entry, process_instance, native_id) } +} + +unsafe fn decode_core_ref_entry( + entry: &AdRefEntry, + process_instance: String, + native_id: Option, +) -> Result { + if entry.process.pid == 0 { + return Err(AdapterError::new( + ErrorCode::InvalidArgs, + "identity.pid must be positive", + )); + } + let role = unsafe { optional_string(entry.identity.role, "identity.role") }? + .ok_or_else(|| AdapterError::new(ErrorCode::InvalidArgs, "role is null"))?; + if !agent_desktop_core::Role::is_canonical(&role) || role == "unknown" { + return Err(AdapterError::new( + ErrorCode::InvalidArgs, + "identity.role is not canonical", + )); + } + let name = unsafe { optional_string(entry.identity.name, "identity.name") }?; + let value = unsafe { optional_string(entry.identity.value, "identity.value") }?; + let description = + unsafe { optional_string(entry.identity.description, "identity.description") }?; + let states = unsafe { + string_array( + entry.capabilities.states.items, + entry.capabilities.states.count, + "states", + "AD_MAX_REF_STATES", + crate::types::ref_entry::AD_MAX_REF_STATES, + ) + }?; + let available_actions = unsafe { + string_array( + entry.capabilities.available_actions.items, + entry.capabilities.available_actions.count, + "available_actions", + "AD_MAX_REF_ACTIONS", + crate::types::ref_entry::AD_MAX_REF_ACTIONS, + ) + }?; + let bounds = if entry.geometry.has_bounds { + Some( + Rect { + x: entry.geometry.bounds.x, + y: entry.geometry.bounds.y, + width: entry.geometry.bounds.width, + height: entry.geometry.bounds.height, + } + .validate()?, + ) + } else { + None + }; + let supplied_bounds_hash = if entry.geometry.has_bounds_hash { + Some(entry.geometry.bounds_hash) + } else { + None + }; + let derived_bounds_hash = bounds.and_then(|value| value.bounds_hash()); + let source_surface = snapshot_surface_from_c(entry.source.surface, "source.surface")?; + let path = unsafe { ref_path(entry.scope.path, entry.scope.path_count)? }; + if let (Some(supplied), Some(derived)) = (supplied_bounds_hash, derived_bounds_hash) + && supplied != derived + { + return Err(AdapterError::new( + ErrorCode::InvalidArgs, + "geometry.bounds_hash does not match geometry.bounds", + )); + } + let bounds_hash = supplied_bounds_hash.or(derived_bounds_hash); + let source_app = unsafe { optional_string(entry.source.app, "source.app") }?; + let source_window_id = unsafe { optional_string(entry.source.window_id, "source.window_id") }?; + let source_window_title = + unsafe { optional_string(entry.source.window_title, "source.window_title") }?; + let source_window_bounds_hash = entry + .source + .has_window_bounds_hash + .then_some(entry.source.window_bounds_hash); + let root_ref = unsafe { optional_string(entry.scope.root_ref, "scope.root_ref") }?; + for field in [ + Some(role.as_str()), + Some(process_instance.as_str()), + name.as_deref(), + value.as_deref(), + description.as_deref(), + native_id + .as_ref() + .map(|identifier| identifier.value.as_str()), + source_app.as_deref(), + source_window_id.as_deref(), + source_window_title.as_deref(), + root_ref.as_deref(), + ] + .into_iter() + .flatten() + { + if field.len() > MAX_REF_FIELD_BYTES { + return Err(AdapterError::new( + ErrorCode::InvalidArgs, + "ref identity evidence exceeds the field limit", + )); + } + } + if states + .iter() + .chain(available_actions.iter()) + .any(|value| value.len() > MAX_REF_TOKEN_BYTES) + { + return Err(AdapterError::new( + ErrorCode::InvalidArgs, + "ref state or action evidence exceeds the token limit", + )); + } + + let bounds_space = match entry.geometry.bounds_space { + crate::AD_BOUNDS_SPACE_UNKNOWN => None, + crate::AD_BOUNDS_SPACE_SCREEN => Some(agent_desktop_core::BoundsSpace::Screen), + crate::AD_BOUNDS_SPACE_WINDOW => Some(agent_desktop_core::BoundsSpace::Window), + _ => { + return Err(AdapterError::new( + ErrorCode::InvalidArgs, + "invalid geometry.bounds_space", + )); + } + }; + let required_space = if cfg!(target_os = "linux") { + agent_desktop_core::BoundsSpace::Window + } else { + agent_desktop_core::BoundsSpace::Screen + }; + if bounds_hash.is_some() && bounds_space != Some(required_space) { + let frame = if cfg!(target_os = "linux") { + "window" + } else { + "screen" + }; + return Err(AdapterError::new( + ErrorCode::InvalidArgs, + format!("Ref geometry requires {frame}-relative bounds on this platform"), + )); + } + let decoded = CoreRefEntry { + process: RefProcess { + pid: agent_desktop_core::ProcessId::new(entry.process.pid), + process_instance: Some(process_instance), + }, + identity: RefEntryIdentity { + retained_object: None, + role, + name, + value, + description, + native_id, + unique_at_capture: false, + }, + geometry: RefGeometry { + bounds_space, + bounds, + bounds_hash, + }, + capabilities: RefCapabilities { + states, + available_actions, + }, + source: RefSource { + source_app, + source_window_id, + source_window_title, + source_window_bounds_hash, + source_surface, + renderer_accessibility_forced: false, + }, + scope: RefScope { + root_ref, + path_is_absolute: entry.scope.path_is_absolute, + path, + }, + }; + if !agent_desktop_core::refs::ref_identity::has_meaningful_identity(&decoded) + && (bounds_space != Some(required_space) || bounds.is_none()) + { + return Err(AdapterError::new( + ErrorCode::InvalidArgs, + format!("Geometry-only refs require {required_space:?}-relative bounds"), + )); + } + Ok(decoded) +} + +unsafe fn optional_string( + ptr: *const std::os::raw::c_char, + field: &str, +) -> Result, AdapterError> { + optional_adapter_string(ptr, field) +} + +fn check_array_len( + len: usize, + is_null: bool, + field: &str, + constant: &str, + max: usize, +) -> Result<(), AdapterError> { + if len > max { + return Err(AdapterError::new( + ErrorCode::InvalidArgs, + format!("{field} count {len} exceeds {constant} ({max})"), + )); + } + if is_null { + return Err(AdapterError::new( + ErrorCode::InvalidArgs, + format!("{field} count is nonzero but pointer is null"), + )); + } + Ok(()) +} + +unsafe fn string_array( + ptr: *const *const std::os::raw::c_char, + len: usize, + field: &str, + constant: &str, + max: usize, +) -> Result, AdapterError> { + if len == 0 { + return Ok(Vec::new()); + } + check_array_len(len, ptr.is_null(), field, constant, max)?; + let items = unsafe { std::slice::from_raw_parts(ptr, len) }; + items + .iter() + .enumerate() + .map(|(index, item)| { + let element = format!("{field}[{index}]"); + unsafe { optional_string(*item, &element) }?.ok_or_else(|| { + AdapterError::new(ErrorCode::InvalidArgs, format!("{element} is null")) + }) + }) + .collect() +} + +unsafe fn ref_path( + ptr: *const u32, + len: usize, +) -> Result, AdapterError> { + if len == 0 { + return Ok(smallvec::SmallVec::new()); + } + check_array_len( + len, + ptr.is_null(), + "path", + "AD_MAX_REF_PATH_DEPTH", + crate::types::ref_entry::AD_MAX_REF_PATH_DEPTH, + )?; + let mut path = smallvec::SmallVec::new(); + path.extend( + unsafe { std::slice::from_raw_parts(ptr, len) } + .iter() + .map(|item| *item as usize), + ); + Ok(path) +} + +#[cfg(test)] +#[path = "exact_entry_tests.rs"] +mod tests; diff --git a/crates/ffi/src/actions/resolve_tests.rs b/crates/ffi/src/actions/exact_entry_tests.rs similarity index 100% rename from crates/ffi/src/actions/resolve_tests.rs rename to crates/ffi/src/actions/exact_entry_tests.rs diff --git a/crates/ffi/src/actions/execute.rs b/crates/ffi/src/actions/execute.rs index c7fe2281e..7711e317a 100644 --- a/crates/ffi/src/actions/execute.rs +++ b/crates/ffi/src/actions/execute.rs @@ -1,7 +1,7 @@ use crate::AdAdapter; use crate::actions::conversion::action_from_c; +use crate::actions::exact_entry::{core_ref_entry_from_exact, exact_entry_unsupported}; use crate::actions::result::action_result_to_c; -use crate::commands::app_error_to_adapter; use crate::error::{self, AdResult}; use crate::ffi_try::trap_panic; use crate::types::{AdAction, AdActionResult, AdExactRefEntry, AdNativeHandle, AdPolicyKind}; @@ -117,8 +117,11 @@ pub unsafe extern "C" fn ad_execute_action_with_policy( }) } -/// Executes a struct-based ref action with exact process-generation and typed -/// native-id evidence. +/// Validates a struct-based ref action and refuses it without acting. An exact +/// entry carries no retained element token and no capture proof, so a +/// well-formed call fails with `ACTION_NOT_SUPPORTED` +/// (`exact_entry_unsupported`) and a malformed one with `INVALID_ARGS`; +/// callers act through a snapshot ref with `ad_execute_by_ref` instead. /// /// # Safety /// @@ -138,56 +141,20 @@ pub unsafe extern "C" fn ad_execute_ref_action_exact_with_policy( crate::pointer_guard::guard_non_null!(adapter, c"adapter is null"); crate::pointer_guard::guard_non_null!(entry, c"entry is null"); crate::pointer_guard::guard_non_null!(action, c"action is null"); - let core_entry = match crate::actions::resolve::core_ref_entry_from_exact(&*entry) { - Ok(entry) => entry, - Err(error) => { - error::set_last_error(&error); - return error::last_error_code(); - } - }; - execute_core_ref_action(adapter, core_entry, &*action, policy, out) - }) -} - -unsafe fn execute_core_ref_action( - adapter: *const AdAdapter, - entry: agent_desktop_core::RefEntry, - action: &AdAction, - policy: i32, - out: *mut AdActionResult, -) -> AdResult { - let core_action = match decode_action(action) { - Ok(action) => action, - Err(result) => return result, - }; - let policy = match decode_policy(policy) { - Ok(policy) => policy, - Err(result) => return result, - }; - let adapter_ref = crate::adapter::acquire_adapter!(adapter); - let request = action_request(policy, core_action); - let context = match adapter_ref.command_context() { - Ok(context) => context, - Err(error) => { - error::set_last_error(&app_error_to_adapter(error)); + if let Err(error) = core_ref_entry_from_exact(&*entry) { + error::set_last_error(&error); return error::last_error_code(); } - }; - match agent_desktop_core::ref_action::execute_entry_with_context( - adapter_ref.inner.as_ref(), - &entry, - request, - &context, - ) { - Ok(result) => { - unsafe { *out = action_result_to_c(&result) }; - AdResult::Ok + if let Err(result) = decode_action(&*action) { + return result; } - Err(error) => { - error::set_last_error(&error); - error::last_error_code() + if let Err(result) = decode_policy(policy) { + return result; } - } + let _live = crate::adapter::acquire_adapter!(adapter); + error::set_last_error(&exact_entry_unsupported()); + error::last_error_code() + }) } fn decode_action(action: &AdAction) -> Result { diff --git a/crates/ffi/src/actions/mod.rs b/crates/ffi/src/actions/mod.rs index c2d211989..74809ae42 100644 --- a/crates/ffi/src/actions/mod.rs +++ b/crates/ffi/src/actions/mod.rs @@ -1,4 +1,5 @@ pub(crate) mod conversion; +pub(crate) mod exact_entry; pub(crate) mod execute; pub(crate) mod native_handle; pub(crate) mod resolve; diff --git a/crates/ffi/src/actions/resolve.rs b/crates/ffi/src/actions/resolve.rs index fd5b1350a..4acae8372 100644 --- a/crates/ffi/src/actions/resolve.rs +++ b/crates/ffi/src/actions/resolve.rs @@ -1,18 +1,14 @@ use crate::AdAdapter; -use crate::convert::string::optional_adapter_string; -use crate::convert::surface::snapshot_surface_from_c; +use crate::actions::exact_entry::{core_ref_entry_from_exact, exact_entry_unsupported}; use crate::error::{self, AdResult}; use crate::ffi_try::trap_panic; -use crate::types::{AdExactRefEntry, AdIdentifierKind, AdNativeHandle, AdRefEntry}; -use agent_desktop_core::{ - AdapterError, ElementIdentifier, ErrorCode, Rect, RefCapabilities, RefEntry as CoreRefEntry, - RefEntryIdentity, RefGeometry, RefProcess, RefScope, RefSource, -}; +use crate::types::{AdExactRefEntry, AdNativeHandle}; -const MAX_REF_FIELD_BYTES: usize = 65_536; -const MAX_REF_TOKEN_BYTES: usize = 256; - -/// Resolves an element using process-generation and typed native-id evidence. +/// Validates an exact entry and refuses it. An exact entry carries no retained +/// element token and no capture proof, so a well-formed one fails with +/// `ACTION_NOT_SUPPORTED` (`exact_entry_unsupported`) and a malformed one with +/// `INVALID_ARGS`; callers act through a snapshot ref with `ad_execute_by_ref` +/// or locate the element with `ad_find_exact` instead. /// /// # Safety /// @@ -28,372 +24,12 @@ pub unsafe extern "C" fn ad_resolve_element_exact( (*out).ptr = std::ptr::null(); crate::pointer_guard::guard_non_null!(adapter, c"adapter is null"); crate::pointer_guard::guard_non_null!(entry, c"entry is null"); - let core_entry = match core_ref_entry_from_exact(&*entry) { - Ok(entry) => entry, - Err(error) => { - error::set_last_error(&error); - return error::last_error_code(); - } - }; - resolve_core_entry(adapter, &core_entry, out) - }) -} - -unsafe fn resolve_core_entry( - adapter: *const AdAdapter, - entry: &CoreRefEntry, - out: *mut AdNativeHandle, -) -> AdResult { - let adapter_id = adapter.addr(); - let adapter = crate::adapter::acquire_adapter!(adapter); - let process = match process_identity(entry) { - Ok(process) => process, - Err(error) => { + if let Err(error) = core_ref_entry_from_exact(&*entry) { error::set_last_error(&error); return error::last_error_code(); } - }; - let deadline = crate::operation::operation_deadline!(); - match adapter.inner.resolve_element_strict(entry, deadline) { - Ok(handle) => { - match crate::actions::native_handle::into_ffi_handle(adapter_id, handle, process) { - Ok(token) => { - unsafe { (*out).ptr = token }; - AdResult::Ok - } - Err(error) => { - error::set_last_error(&error); - error::last_error_code() - } - } - } - Err(error) => { - error::set_last_error(&error); - error::last_error_code() - } - } -} - -pub(crate) unsafe fn core_ref_entry_from_exact( - exact: &AdExactRefEntry, -) -> Result { - if exact.version != crate::types::exact_ref_entry::AD_EXACT_REF_ENTRY_VERSION - || exact.size as usize != crate::types::exact_ref_entry::AD_EXACT_REF_ENTRY_SIZE - { - return Err(AdapterError::new( - ErrorCode::InvalidArgs, - "AdExactRefEntry version or size does not match this library", - )); - } - let process_instance = unsafe { optional_string(exact.process_instance, "process_instance") }? - .filter(|value| !value.is_empty()) - .ok_or_else(|| { - AdapterError::new(ErrorCode::InvalidArgs, "process_instance is null or empty") - })?; - let native_id = - match unsafe { optional_string(exact.entry.identity.native_id, "identity.native_id") }? { - Some(value) if value.trim().is_empty() => { - return Err(AdapterError::new( - ErrorCode::InvalidArgs, - "identity.native_id is empty", - )); - } - Some(value) => { - let kind = AdIdentifierKind::from_c(exact.identifier_kind).ok_or_else(|| { - AdapterError::new( - ErrorCode::InvalidArgs, - "identity.identifier_kind has an invalid discriminant", - ) - })?; - Some(ElementIdentifier { - kind: kind.to_core(), - value, - }) - } - None => None, - }; - unsafe { decode_core_ref_entry(&exact.entry, process_instance, native_id) } -} - -unsafe fn decode_core_ref_entry( - entry: &AdRefEntry, - process_instance: String, - native_id: Option, -) -> Result { - if entry.process.pid == 0 { - return Err(AdapterError::new( - ErrorCode::InvalidArgs, - "identity.pid must be positive", - )); - } - let role = unsafe { optional_string(entry.identity.role, "identity.role") }? - .ok_or_else(|| AdapterError::new(ErrorCode::InvalidArgs, "role is null"))?; - if !agent_desktop_core::Role::is_canonical(&role) || role == "unknown" { - return Err(AdapterError::new( - ErrorCode::InvalidArgs, - "identity.role is not canonical", - )); - } - let name = unsafe { optional_string(entry.identity.name, "identity.name") }?; - let value = unsafe { optional_string(entry.identity.value, "identity.value") }?; - let description = - unsafe { optional_string(entry.identity.description, "identity.description") }?; - let states = unsafe { - string_array( - entry.capabilities.states.items, - entry.capabilities.states.count, - "states", - "AD_MAX_REF_STATES", - crate::types::ref_entry::AD_MAX_REF_STATES, - ) - }?; - let available_actions = unsafe { - string_array( - entry.capabilities.available_actions.items, - entry.capabilities.available_actions.count, - "available_actions", - "AD_MAX_REF_ACTIONS", - crate::types::ref_entry::AD_MAX_REF_ACTIONS, - ) - }?; - let bounds = if entry.geometry.has_bounds { - Some( - Rect { - x: entry.geometry.bounds.x, - y: entry.geometry.bounds.y, - width: entry.geometry.bounds.width, - height: entry.geometry.bounds.height, - } - .validate()?, - ) - } else { - None - }; - let supplied_bounds_hash = if entry.geometry.has_bounds_hash { - Some(entry.geometry.bounds_hash) - } else { - None - }; - let derived_bounds_hash = bounds.and_then(|value| value.bounds_hash()); - let source_surface = snapshot_surface_from_c(entry.source.surface, "source.surface")?; - let path = unsafe { ref_path(entry.scope.path, entry.scope.path_count)? }; - if let (Some(supplied), Some(derived)) = (supplied_bounds_hash, derived_bounds_hash) - && supplied != derived - { - return Err(AdapterError::new( - ErrorCode::InvalidArgs, - "geometry.bounds_hash does not match geometry.bounds", - )); - } - let bounds_hash = supplied_bounds_hash.or(derived_bounds_hash); - let source_app = unsafe { optional_string(entry.source.app, "source.app") }?; - let source_window_id = unsafe { optional_string(entry.source.window_id, "source.window_id") }?; - let source_window_title = - unsafe { optional_string(entry.source.window_title, "source.window_title") }?; - let source_window_bounds_hash = entry - .source - .has_window_bounds_hash - .then_some(entry.source.window_bounds_hash); - let root_ref = unsafe { optional_string(entry.scope.root_ref, "scope.root_ref") }?; - for field in [ - Some(role.as_str()), - Some(process_instance.as_str()), - name.as_deref(), - value.as_deref(), - description.as_deref(), - native_id - .as_ref() - .map(|identifier| identifier.value.as_str()), - source_app.as_deref(), - source_window_id.as_deref(), - source_window_title.as_deref(), - root_ref.as_deref(), - ] - .into_iter() - .flatten() - { - if field.len() > MAX_REF_FIELD_BYTES { - return Err(AdapterError::new( - ErrorCode::InvalidArgs, - "ref identity evidence exceeds the field limit", - )); - } - } - if states - .iter() - .chain(available_actions.iter()) - .any(|value| value.len() > MAX_REF_TOKEN_BYTES) - { - return Err(AdapterError::new( - ErrorCode::InvalidArgs, - "ref state or action evidence exceeds the token limit", - )); - } - - let bounds_space = match entry.geometry.bounds_space { - crate::AD_BOUNDS_SPACE_UNKNOWN => None, - crate::AD_BOUNDS_SPACE_SCREEN => Some(agent_desktop_core::BoundsSpace::Screen), - crate::AD_BOUNDS_SPACE_WINDOW => Some(agent_desktop_core::BoundsSpace::Window), - _ => { - return Err(AdapterError::new( - ErrorCode::InvalidArgs, - "invalid geometry.bounds_space", - )); - } - }; - let required_space = if cfg!(target_os = "linux") { - agent_desktop_core::BoundsSpace::Window - } else { - agent_desktop_core::BoundsSpace::Screen - }; - if bounds_hash.is_some() && bounds_space != Some(required_space) { - let frame = if cfg!(target_os = "linux") { - "window" - } else { - "screen" - }; - return Err(AdapterError::new( - ErrorCode::InvalidArgs, - format!("Ref geometry requires {frame}-relative bounds on this platform"), - )); - } - let decoded = CoreRefEntry { - process: RefProcess { - pid: agent_desktop_core::ProcessId::new(entry.process.pid), - process_instance: Some(process_instance), - }, - identity: RefEntryIdentity { - retained_object: None, - role, - name, - value, - description, - native_id, - }, - geometry: RefGeometry { - bounds_space, - bounds, - bounds_hash, - }, - capabilities: RefCapabilities { - states, - available_actions, - }, - source: RefSource { - source_app, - source_window_id, - source_window_title, - source_window_bounds_hash, - source_surface, - }, - scope: RefScope { - root_ref, - path_is_absolute: entry.scope.path_is_absolute, - path, - }, - }; - if !agent_desktop_core::refs::ref_identity::has_meaningful_identity(&decoded) - && (bounds_space != Some(required_space) || bounds.is_none()) - { - return Err(AdapterError::new( - ErrorCode::InvalidArgs, - format!("Geometry-only refs require {required_space:?}-relative bounds"), - )); - } - Ok(decoded) -} - -fn process_identity( - entry: &CoreRefEntry, -) -> Result { - let instance = entry.process.process_instance.clone().ok_or_else(|| { - AdapterError::new( - ErrorCode::InvalidArgs, - "process_instance is required for a native handle", - ) - })?; - Ok(agent_desktop_core::ProcessIdentity::new( - entry.process.pid, - instance, - )) -} - -unsafe fn optional_string( - ptr: *const std::os::raw::c_char, - field: &str, -) -> Result, AdapterError> { - optional_adapter_string(ptr, field) -} - -fn check_array_len( - len: usize, - is_null: bool, - field: &str, - constant: &str, - max: usize, -) -> Result<(), AdapterError> { - if len > max { - return Err(AdapterError::new( - ErrorCode::InvalidArgs, - format!("{field} count {len} exceeds {constant} ({max})"), - )); - } - if is_null { - return Err(AdapterError::new( - ErrorCode::InvalidArgs, - format!("{field} count is nonzero but pointer is null"), - )); - } - Ok(()) -} - -unsafe fn string_array( - ptr: *const *const std::os::raw::c_char, - len: usize, - field: &str, - constant: &str, - max: usize, -) -> Result, AdapterError> { - if len == 0 { - return Ok(Vec::new()); - } - check_array_len(len, ptr.is_null(), field, constant, max)?; - let items = unsafe { std::slice::from_raw_parts(ptr, len) }; - items - .iter() - .enumerate() - .map(|(index, item)| { - let element = format!("{field}[{index}]"); - unsafe { optional_string(*item, &element) }?.ok_or_else(|| { - AdapterError::new(ErrorCode::InvalidArgs, format!("{element} is null")) - }) - }) - .collect() -} - -unsafe fn ref_path( - ptr: *const u32, - len: usize, -) -> Result, AdapterError> { - if len == 0 { - return Ok(smallvec::SmallVec::new()); - } - check_array_len( - len, - ptr.is_null(), - "path", - "AD_MAX_REF_PATH_DEPTH", - crate::types::ref_entry::AD_MAX_REF_PATH_DEPTH, - )?; - let mut path = smallvec::SmallVec::new(); - path.extend( - unsafe { std::slice::from_raw_parts(ptr, len) } - .iter() - .map(|item| *item as usize), - ); - Ok(path) + let _live = crate::adapter::acquire_adapter!(adapter); + error::set_last_error(&exact_entry_unsupported()); + error::last_error_code() + }) } - -#[cfg(test)] -#[path = "resolve_tests.rs"] -mod tests; diff --git a/crates/ffi/src/commands/snapshot.rs b/crates/ffi/src/commands/snapshot.rs index f8f777c4e..271d41a54 100644 --- a/crates/ffi/src/commands/snapshot.rs +++ b/crates/ffi/src/commands/snapshot.rs @@ -15,7 +15,7 @@ use std::ptr; /// interactive elements, persists the refmap to disk, and writes the JSON envelope into `*out`. /// /// The JSON shape matches `agent-desktop snapshot`: -/// `{"version":"2.5","ok":true,"command":"snapshot","data":{"app":"...","window":{...},"ref_count":N,"snapshot_id":"...","complete":true,"tree":{...}}}`. +/// `{"version":"2.6","ok":true,"command":"snapshot","data":{"app":"...","window":{...},"ref_count":N,"snapshot_id":"...","complete":true,"tree":{...}}}`. /// /// `data.complete` is always present. A snapshot that exhausts its observation /// budget still succeeds with `"complete":false` and the tree it did observe, diff --git a/crates/ffi/src/observation/find.rs b/crates/ffi/src/observation/find.rs index 21475a19a..65dc9024a 100644 --- a/crates/ffi/src/observation/find.rs +++ b/crates/ffi/src/observation/find.rs @@ -16,9 +16,11 @@ const MAX_CONTAINMENT_DEPTH: usize = 8; /// Finds and strictly resolves one element within a generation-pinned window. /// `AdFindQuery.control.selection` must explicitly request first, last, or nth -/// behavior when duplicate matches are acceptable. The returned native handle -/// is adapter-bound and thread-affine; release it with `ad_free_handle` on the -/// resolving thread. +/// behavior when duplicate matches are acceptable. The handle is published only +/// after the landed element is re-identified by the selected entry's full +/// predicate, so a control that moved into the anchored path is refused rather +/// than returned. The returned native handle is adapter-bound and thread-affine; +/// release it with `ad_free_handle` on the resolving thread. /// /// # Safety /// All pointers must be valid and `out_handle` must be writable. @@ -113,9 +115,18 @@ unsafe fn find_in_window( let process = agent_desktop_core::ProcessIdentity::new(entry.process.pid, process_instance); match adapter .inner - .resolve_element_strict(&entry, request.deadline) + .resolve_locator_anchor(&entry, request.deadline) { Ok(handle) => { + if let Err(error) = agent_desktop_core::refs::resolve_saved_ref::confirm_landed_element( + adapter.inner.as_ref(), + &entry, + &handle, + request.deadline, + ) { + set_last_error(&error); + return crate::error::last_error_code(); + } match crate::actions::native_handle::into_ffi_handle(adapter_id, handle, process) { Ok(token) => { unsafe { (*out_handle).ptr = token }; diff --git a/crates/ffi/src/observation/find_abi_tests.rs b/crates/ffi/src/observation/find_abi_tests.rs index 8b8e3facf..921cddf6d 100644 --- a/crates/ffi/src/observation/find_abi_tests.rs +++ b/crates/ffi/src/observation/find_abi_tests.rs @@ -14,6 +14,7 @@ const TEST_TIMEOUT_MS: u64 = 1_000; struct CardinalityAdapter { duplicate: bool, complete: bool, + landed: &'static str, } impl ActionOps for CardinalityAdapter {} @@ -26,6 +27,14 @@ impl ObservationOps for CardinalityAdapter { root: ObservationRoot<'_>, _request: &ObservationRequest, ) -> Result { + if let ObservationRoot::Element { .. } = root { + return ObservedTree::from_roots( + vec![subtree_named(self.landed, self.complete)], + ObservationSource::from_root(&root, agent_desktop_core::SnapshotSurface::Window), + Default::default(), + self.complete, + ); + } let ObservationRoot::Window(window) = root else { return Err(agent_desktop_core::AdapterError::internal( "expected window root", @@ -40,18 +49,31 @@ impl ObservationOps for CardinalityAdapter { ObservationSource::Window { window: window.clone(), surface: agent_desktop_core::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, Default::default(), self.complete, ) } + + fn resolve_locator_anchor( + &self, + _entry: &agent_desktop_core::RefEntry, + _deadline: agent_desktop_core::Deadline, + ) -> Result { + Ok(agent_desktop_core::NativeHandle::null()) + } } fn subtree(complete: bool) -> ObservedSubtree { + subtree_named("Save", complete) +} + +fn subtree_named(name: &str, complete: bool) -> ObservedSubtree { ObservedSubtree::new( LocatorEvidence { role: LocatorField::Known("button".into()), - name: LocatorField::Known("Save".into()), + name: LocatorField::Known(name.into()), description: LocatorField::Absent, value: LocatorField::Absent, identifiers: IdentifierEvidence::absent(), @@ -123,6 +145,7 @@ fn strict_duplicate_is_ambiguous_at_the_c_boundary() { call(CardinalityAdapter { duplicate: true, complete: true, + landed: "Save", }), AdResult::ErrAmbiguousTarget ); @@ -134,7 +157,63 @@ fn strict_incomplete_observation_times_out_at_the_c_boundary() { call(CardinalityAdapter { duplicate: false, complete: false, + landed: "Save", }), AdResult::ErrTimeout ); } + +/// A find lands the entry its observation materialized through the adapter's +/// locator anchor, the landing every platform shares, rather than through a +/// saved-ref search. The stub implements the anchor and nothing else, so a +/// regression back to strict resolution fails here instead of on a desktop. +#[test] +fn a_found_element_lands_through_the_locator_anchor() { + let adapter = crate::adapter::register_adapter(AdAdapter { + inner: Box::new(CardinalityAdapter { + duplicate: false, + complete: true, + landed: "Save", + }), + session_id: None, + _session_lease: None, + }) + .unwrap(); + let mut out = AdNativeHandle { + ptr: std::ptr::null(), + }; + + let result = unsafe { ad_find_exact(adapter, &ffi_window(), &query(), &mut out) }; + let freed = unsafe { crate::actions::native_handle::ad_free_handle(adapter, &mut out) }; + + assert_eq!(result, AdResult::Ok); + assert!(out.ptr.is_null(), "freeing the handle must zero it"); + assert_eq!(freed, AdResult::Ok); + unsafe { crate::adapter::ad_adapter_destroy(adapter) }; +} + +/// The handle a C caller receives is the element the query selected, so the +/// landing is re-identified by that entry's full predicate before it is +/// published: a different control occupying the anchored path is refused. +#[test] +fn a_landing_that_is_not_the_selected_element_is_never_published() { + let adapter = crate::adapter::register_adapter(AdAdapter { + inner: Box::new(CardinalityAdapter { + duplicate: false, + complete: true, + landed: "Delete", + }), + session_id: None, + _session_lease: None, + }) + .unwrap(); + let mut out = AdNativeHandle { + ptr: std::ptr::null(), + }; + + let result = unsafe { ad_find_exact(adapter, &ffi_window(), &query(), &mut out) }; + unsafe { crate::adapter::ad_adapter_destroy(adapter) }; + + assert_eq!(result, AdResult::ErrStaleRef); + assert!(out.ptr.is_null(), "a refused landing publishes no handle"); +} diff --git a/crates/ffi/src/observation/is_abi_tests.rs b/crates/ffi/src/observation/is_abi_tests.rs index 5a86c13f3..371062d34 100644 --- a/crates/ffi/src/observation/is_abi_tests.rs +++ b/crates/ffi/src/observation/is_abi_tests.rs @@ -32,6 +32,7 @@ impl ObservationOps for DuplicateAdapter { ObservationSource::Window { window: window.clone(), surface: agent_desktop_core::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, Default::default(), true, diff --git a/crates/ffi/tests/c_abi_actions.rs b/crates/ffi/tests/c_abi_actions.rs index 3ac3b3685..5e3b3d992 100644 --- a/crates/ffi/tests/c_abi_actions.rs +++ b/crates/ffi/tests/c_abi_actions.rs @@ -80,6 +80,35 @@ fn exact_ref_action_without_process_generation_is_rejected() { }); } +#[test] +fn a_well_formed_exact_ref_action_is_refused_without_acting() { + with_adapter(|adapter| unsafe { + let role = CString::new("button").unwrap(); + let name = CString::new("Save").unwrap(); + let process_instance = CString::new("generation-1").unwrap(); + let mut entry = default_exact_ref_entry(); + entry.process_instance = process_instance.as_ptr(); + entry.entry.process.pid = 1; + entry.entry.identity.role = role.as_ptr(); + entry.entry.identity.name = name.as_ptr(); + let action = default_action(); + let mut out: AdActionResult = std::mem::zeroed(); + + let rc = ad_execute_ref_action_exact_with_policy( + adapter, + &entry, + &action, + AdPolicyKind::Headless as i32, + &mut out, + ); + + let message = CStr::from_ptr(ad_last_error_message()).to_string_lossy(); + assert_eq!(rc, AdResult::ErrActionNotSupported, "message={message}"); + assert!(out.steps.is_null()); + assert!(message.contains("no capture proof"), "message={message}"); + }); +} + #[test] fn execute_action_policy_rejects_null_adapter_on_worker_thread() { let rc = std::thread::spawn(|| unsafe { diff --git a/crates/ffi/tests/c_abi_inputs.rs b/crates/ffi/tests/c_abi_inputs.rs index f7800a441..5486cfcd6 100644 --- a/crates/ffi/tests/c_abi_inputs.rs +++ b/crates/ffi/tests/c_abi_inputs.rs @@ -85,6 +85,29 @@ fn exact_resolve_rejects_invalid_utf8_name_at_the_name_boundary() { }); } +#[test] +fn exact_resolve_refuses_an_entry_with_no_capture_proof() { + with_adapter(|adapter| unsafe { + let role = std::ffi::CString::new("button").unwrap(); + let name = std::ffi::CString::new("Save").unwrap(); + let process_instance = std::ffi::CString::new("generation-1").unwrap(); + let mut exact = default_exact_ref_entry(); + exact.process_instance = process_instance.as_ptr(); + exact.entry.process.pid = 1; + exact.entry.identity.role = role.as_ptr(); + exact.entry.identity.name = name.as_ptr(); + let mut out = AdNativeHandle { + ptr: std::ptr::null(), + }; + let rc = ad_resolve_element_exact(adapter, &exact, &mut out); + assert_eq!(rc, AdResult::ErrActionNotSupported); + assert!(out.ptr.is_null()); + let message = CStr::from_ptr(ad_last_error_message()).to_string_lossy(); + assert!(message.contains("no capture proof"), "message={message}"); + assert!(!message.contains("--headed"), "message={message}"); + }); +} + #[test] fn exact_resolve_rejects_invalid_utf8_description_at_the_description_boundary() { with_adapter(|adapter| unsafe { diff --git a/crates/linux/src/adapter.rs b/crates/linux/src/adapter.rs index 1e8acf648..686eae0e1 100644 --- a/crates/linux/src/adapter.rs +++ b/crates/linux/src/adapter.rs @@ -59,6 +59,14 @@ impl ObservationOps for LinuxAdapter { crate::tree::resolve_element_strict(self.bus(), entry, deadline) } + fn validate_saved_ref_context( + &self, + entry: &RefEntry, + observed_token: Option<&str>, + ) -> Result<(), AdapterError> { + crate::tree::menu::menu_resolve::validate_observed_context(entry, observed_token) + } + fn resolve_locator_anchor( &self, entry: &RefEntry, @@ -185,14 +193,13 @@ impl ObservationOps for LinuxAdapter { fn confirm_pointer_receiver( &self, - entry: &RefEntry, + handle: &NativeHandle, point: Point, deadline: Deadline, ) -> Result<(), AdapterError> { - let handle = self.resolve_element_strict(entry, deadline)?; crate::input::physical::target_pointer::confirm_receiver( self.bus(), - &handle, + handle, &point, deadline, ) diff --git a/crates/linux/src/system/tray/fit_tests.rs b/crates/linux/src/system/tray/fit_tests.rs index 0b4d710c3..48189639a 100644 --- a/crates/linux/src/system/tray/fit_tests.rs +++ b/crates/linux/src/system/tray/fit_tests.rs @@ -68,6 +68,7 @@ fn projected() -> AccessibilityNode { ObservationSource::Window { window, surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, Default::default(), true, diff --git a/crates/linux/src/tree/live/test_support.rs b/crates/linux/src/tree/live/test_support.rs index 0e4a1d07f..7eb23f99d 100644 --- a/crates/linux/src/tree/live/test_support.rs +++ b/crates/linux/src/tree/live/test_support.rs @@ -114,6 +114,7 @@ pub(crate) fn ref_entry(role: &str) -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: false, }, geometry: agent_desktop_core::RefGeometry { bounds_space: None, @@ -130,6 +131,7 @@ pub(crate) fn ref_entry(role: &str) -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: agent_desktop_core::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: agent_desktop_core::RefScope { root_ref: None, diff --git a/crates/linux/src/tree/live_read/clipped_row_tests.rs b/crates/linux/src/tree/live_read/clipped_row_tests.rs index d8ea29c6e..2f3fc6c28 100644 --- a/crates/linux/src/tree/live_read/clipped_row_tests.rs +++ b/crates/linux/src/tree/live_read/clipped_row_tests.rs @@ -3,7 +3,8 @@ use std::sync::Mutex; use agent_desktop_core::{ Action, ActionRequest, ActionResult, AdapterError, Deadline, InteractionLease, LiveElement, - NativeHandle, Rect, RefEntry, + NativeHandle, ObservationRequest, ObservationRoot, ObservationSource, ObservedSubtree, + ObservedTree, ProcessId, Rect, RefEntry, WindowFilter, WindowInfo, WindowState, }; use crate::actions::physical_target::tests::fake::FakeWorld; @@ -17,6 +18,18 @@ use crate::tree::placement::Placement; use crate::tree::vocabulary::state_bits::{ENABLED, SHOWING, VISIBLE}; use crate::tree::{AtspiElement, Extents, Field, ObservedRead}; +fn fixture_window() -> WindowInfo { + WindowInfo { + id: "fixture-window".into(), + title: "Settings".into(), + app: "Fixture".into(), + pid: ProcessId::new(1), + process_instance: Some("fixture-instance".into()), + bounds: None, + state: WindowState::default(), + } +} + struct Sidebar { scene: Mutex<(ObservedRead, Placement)>, events: Mutex>, @@ -66,7 +79,11 @@ impl Sidebar { } impl ObservationOps for Sidebar { - fn resolve_element_strict( + fn list_windows(&self, _: &WindowFilter, _: Deadline) -> Result, AdapterError> { + Ok(vec![fixture_window()]) + } + + fn resolve_locator_anchor( &self, _: &RefEntry, _: Deadline, @@ -83,6 +100,21 @@ impl ObservationOps for Sidebar { .into_native_handle()) } + fn observe_tree( + &self, + root: ObservationRoot<'_>, + _: &ObservationRequest, + ) -> Result { + let scene = self.scene.lock().unwrap(); + let evidence = evidence_for(&scene.0, &scene.0.node.element); + ObservedTree::from_roots( + vec![ObservedSubtree::new(evidence, Vec::new(), true, None)], + ObservationSource::from_root(&root, root.surface()), + agent_desktop_core::LocatorStats::default(), + true, + ) + } + fn get_live_element(&self, _: &NativeHandle, _: Deadline) -> Result { let scene = self.scene.lock().unwrap(); let evidence = evidence_for(&scene.0, &scene.0.node.element); @@ -166,6 +198,7 @@ fn a_clipped_row_is_scrolled_into_view_and_clicked_at_its_new_geometry() { entry.process.process_instance = Some("fixture-instance".into()); entry.source.source_window_id = Some("fixture-window".into()); entry.identity.name = Some("Mouse & Touchpad".into()); + entry.identity.unique_at_capture = true; entry.geometry.bounds = Some(Rect { x: 100.0, y: 200.0, diff --git a/crates/linux/src/tree/live_read/live_menu_tests.rs b/crates/linux/src/tree/live_read/live_menu_tests.rs index a83f087e8..26d4ada99 100644 --- a/crates/linux/src/tree/live_read/live_menu_tests.rs +++ b/crates/linux/src/tree/live_read/live_menu_tests.rs @@ -138,17 +138,42 @@ fn menu_live_identity_uses_the_fresh_context_and_actual_live_object_path() { let live = element_via(&mut reader, &menu_handle("/menu/open"), deadline()).expect("live menu item"); - assert_eq!(verdict(&menu_ref(&captured), &live), IdentityMatch::Match); - let expected = captured.encode(":1.1/menu/open"); + let token = live + .identity + .identifiers + .retained_object() + .expect("a menu item reports the popup it was read in"); + let (context, object) = MenuContext::decode(token).expect("a popup context token"); + assert!( + captured.matches(&context), + "the context read now, not the stored one" + ); + assert_eq!(object, ":1.1/menu/open"); + assert_eq!(token, captured.encode(":1.1/menu/open")); assert_eq!( - live.identity.identifiers.retained_object(), - Some(expected.as_str()) + verdict(&menu_ref(&captured), &live), + IdentityMatch::NoMatch, + "a popup context and an object path are a place, never the element's identity" ); } #[test] -fn changed_incomplete_or_unavailable_menu_context_mismatches_the_snapshot() { +fn a_changed_incomplete_or_unavailable_menu_context_changes_the_transport_token_only() { let captured = MenuContext::test("/menu", 42); + let captured_token = captured.encode(":1.1/menu/open"); + let mut baseline = reader(Some(captured.clone())); + let unchanged = + element_via(&mut baseline, &menu_handle("/menu/open"), deadline()).expect("live item"); + assert_eq!( + unchanged.identity.identifiers.retained_object(), + Some(captured_token.as_str()) + ); + assert_eq!( + verdict(&menu_ref(&captured), &unchanged), + IdentityMatch::NoMatch, + "the predicate never separated the popups, so only the token can" + ); + for current in [ Some(MenuContext::test("/replacement", 42)), Some(MenuContext::test_incomplete("/menu", 42)), @@ -157,18 +182,44 @@ fn changed_incomplete_or_unavailable_menu_context_mismatches_the_snapshot() { let mut reader = reader(current); let live = element_via(&mut reader, &menu_handle("/menu/open"), deadline()).expect("live item"); + assert_ne!( + live.identity.identifiers.retained_object(), + Some(captured_token.as_str()), + "the popup context read now is the transport evidence" + ); assert_eq!(verdict(&menu_ref(&captured), &live), IdentityMatch::NoMatch); } } #[test] -fn the_same_context_does_not_hide_a_changed_live_object_path() { +fn a_changed_live_object_path_changes_the_transport_token_under_the_same_context() { let captured = MenuContext::test("/menu", 42); + let captured_token = captured.encode(":1.1/menu/open"); + let mut baseline = reader(Some(captured.clone())); + let unchanged = + element_via(&mut baseline, &menu_handle("/menu/open"), deadline()).expect("live item"); + assert_eq!( + unchanged.identity.identifiers.retained_object(), + Some(captured_token.as_str()) + ); + assert_eq!( + verdict(&menu_ref(&captured), &unchanged), + IdentityMatch::NoMatch, + "the predicate never separated the two paths, so only the token can" + ); + let mut reader = reader(Some(captured.clone())); - let live = + let moved = element_via(&mut reader, &menu_handle("/menu/replacement"), deadline()).expect("live item"); - - assert_eq!(verdict(&menu_ref(&captured), &live), IdentityMatch::NoMatch); + assert_ne!( + moved.identity.identifiers.retained_object(), + Some(captured_token.as_str()), + "the object path read now is the transport evidence" + ); + assert_eq!( + verdict(&menu_ref(&captured), &moved), + IdentityMatch::NoMatch + ); } #[test] diff --git a/crates/linux/src/tree/menu/menu_context.rs b/crates/linux/src/tree/menu/menu_context.rs index 758143ff3..8280429d3 100644 --- a/crates/linux/src/tree/menu/menu_context.rs +++ b/crates/linux/src/tree/menu/menu_context.rs @@ -49,6 +49,9 @@ impl MenuContext { self.complete && other.complete && self == other } + /// The entry scoped to this popup. A menu ref carries the popup context it + /// was taken in; a popup that reads differently, or a ref without one, + /// cannot answer for it. pub(crate) fn entry_for(&self, entry: &RefEntry) -> Option { let token = entry.identity.retained_object.as_deref()?; let (expected, object) = Self::decode(token)?; diff --git a/crates/linux/src/tree/menu/menu_context_tests.rs b/crates/linux/src/tree/menu/menu_context_tests.rs index 6cad47ad0..7a0df797b 100644 --- a/crates/linux/src/tree/menu/menu_context_tests.rs +++ b/crates/linux/src/tree/menu/menu_context_tests.rs @@ -135,3 +135,87 @@ fn a_menu_context_round_trips_utf8_paths_and_item_paths() { assert!(context.matches(&decoded)); assert_eq!(object, "/org/a11y/atspi/accessible/メニュー/0"); } + +#[test] +fn a_menu_ref_without_its_popup_context_cannot_be_scoped_to_any_popup() { + let context = MenuContext::test("/popup", 10); + let entry = ref_entry("menuitem"); + + assert!(entry.identity.retained_object.is_none()); + assert!(context.entry_for(&entry).is_none()); +} + +#[test] +fn observed_menu_context_is_validated_through_the_adapter_without_a_live_read() { + use agent_desktop_core::{ObservationOps, SnapshotSurface}; + + let adapter = crate::LinuxAdapter::new(); + let context = MenuContext::test("/popup", 10); + let complete = context.encode("/item"); + let incomplete = MenuContext::test_incomplete("/popup", 10).encode("/item"); + let mut entry = ref_entry("menuitem"); + entry.source.source_surface = SnapshotSurface::Menu; + entry.identity.retained_object = Some(complete.clone()); + let same_popup = context.encode("/other-item"); + assert!( + adapter + .validate_saved_ref_context(&entry, Some(&same_popup)) + .is_ok() + ); + + for (saved, observed, kind) in [ + ( + Some(complete.clone()), + Some(MenuContext::test("/replacement", 10).encode("/item")), + "menu_context_changed", + ), + ( + Some(complete.clone()), + Some(MenuContext::test("/popup", 20).encode("/item")), + "menu_context_changed", + ), + ( + Some(complete.clone()), + Some(incomplete.clone()), + "menu_context_unknown", + ), + ( + Some(incomplete), + Some(complete.clone()), + "menu_context_unknown", + ), + (None, Some(complete.clone()), "menu_context_unknown"), + (Some(complete.clone()), None, "menu_context_unknown"), + ( + Some(complete.clone()), + Some("malformed".into()), + "menu_context_unknown", + ), + ( + Some("malformed".into()), + Some(complete.clone()), + "menu_context_unknown", + ), + ( + Some(complete.clone()), + Some(context.encode("")), + "menu_context_unknown", + ), + ( + Some(context.encode("")), + Some(complete), + "menu_context_unknown", + ), + ] { + entry.identity.retained_object = saved; + let error = adapter + .validate_saved_ref_context(&entry, observed.as_deref()) + .unwrap_err(); + assert_eq!(error.code, agent_desktop_core::ErrorCode::StaleRef); + let details = error.details.unwrap(); + assert_eq!(details["kind"], kind); + assert_eq!(details["retryable"], kind == "menu_context_unknown"); + } + entry.source.source_surface = SnapshotSurface::Window; + assert!(adapter.validate_saved_ref_context(&entry, None).is_ok()); +} diff --git a/crates/linux/src/tree/menu/menu_resolve.rs b/crates/linux/src/tree/menu/menu_resolve.rs index 23ee02f61..0b3db5739 100644 --- a/crates/linux/src/tree/menu/menu_resolve.rs +++ b/crates/linux/src/tree/menu/menu_resolve.rs @@ -13,12 +13,13 @@ use crate::tree::observe::node_evidence::read_evidence; use crate::tree::placement::popup_position::{PopupPosition, popup_position_via}; use crate::tree::properties::liveness::Liveness; use crate::tree::properties::read_purpose::ReadPurpose; +use crate::tree::resolve::Source; use crate::tree::resolve::path_descent::Levels; +use crate::tree::resolve::resolve_anchor::resolve_anchor; use crate::tree::resolve::resolve_errors::{ambiguous, settle_gone, stopped_app}; use crate::tree::resolve::resolve_gate::stored_window; use crate::tree::resolve::resolve_search::evaluate; use crate::tree::resolve::searched::Searched; -use crate::tree::resolve::{Source, resolve}; use crate::tree::walk::level::Level; use crate::tree::walk::walker::walk_tree; use crate::tree::{AtspiElement, WalkLimits}; @@ -69,7 +70,7 @@ impl Source for MenuSource<'_> { fn evidence(&mut self, element: &AtspiElement) -> Result { let connection = &self.connection; - let mut read = within( + let read = within( self.deadline, read_evidence( connection, @@ -77,10 +78,6 @@ impl Source for MenuSource<'_> { crate::tree::properties::read_purpose::ReadPurpose::Identity, ), )?; - read.evidence.identifiers = read - .evidence - .identifiers - .with_retained_object(element.object_identity()); match read.liveness { Liveness::TimedOut => Err(unanswered()), _ => Ok(read.evidence), @@ -97,6 +94,34 @@ impl Source for MenuSource<'_> { } } +/// Validates the popup binding carried by the matching node in core's shared +/// observation. A unique text match in another popup cannot authorize a rooted +/// find or drill, even though neither operation needs a native handle. +pub(crate) fn validate_observed_context( + entry: &RefEntry, + observed_token: Option<&str>, +) -> Result<(), AdapterError> { + if !is_menu_entry(entry) { + return Ok(()); + } + let captured = entry + .identity + .retained_object + .as_deref() + .and_then(MenuContext::decode) + .filter(|(_, object)| !object.is_empty()); + let observed = observed_token + .and_then(MenuContext::decode) + .filter(|(_, object)| !object.is_empty()); + match (captured, observed) { + (Some((captured, _)), Some((observed, _))) if captured.matches(&observed) => Ok(()), + (captured, observed) => Err(context_changed( + captured.is_some_and(|(context, _)| context.is_complete()) + && observed.is_some_and(|(context, _)| context.is_complete()), + )), + } +} + fn context_changed(complete: bool) -> AdapterError { AdapterError::stale_ref_because("The open menu no longer proves the captured context") .with_suggestion("Keep the menu open, take a fresh menu snapshot, and use the new ref") @@ -138,7 +163,7 @@ pub(crate) fn resolve_menu( levels: None, deadline, }; - resolve(&mut source, scoped_entry, deadline) + resolve_anchor(&mut source, scoped_entry, deadline) }) .and_then(|element| open_handle(element, |element| position_of(bus, element, deadline))) } diff --git a/crates/linux/src/tree/observe/mod.rs b/crates/linux/src/tree/observe/mod.rs index 03e15d276..b4f64749e 100644 --- a/crates/linux/src/tree/observe/mod.rs +++ b/crates/linux/src/tree/observe/mod.rs @@ -86,10 +86,11 @@ type Walker<'a> = dyn FnMut( + 'a; /// Walks until the tree has stopped growing when the window of a web engine -/// is observed, otherwise once. An element a ref already names sits in a tree -/// that has already been observed, so it does not wait for window population. -/// The second value is false when the wait ended on its budget with -/// the tree still changing. A later read that fails because the budget ran +/// is observed, otherwise once. A read that only resolves a saved ref is +/// never waited for: the ref was captured from a settled tree, and one +/// complete read of the live window is what it is judged by. The second value +/// is false when the wait ended on its budget with the tree still changing. +/// A later read that fails because the budget ran /// out returns the walk before it; any other failure is the failure of the /// observation. fn settled_walk( @@ -99,18 +100,22 @@ fn settled_walk( request: &ObservationRequest, host: &mut SettleHost<'_>, ) -> Result<(Walk, bool), AdapterError> { - let web_process = matches!(root, ObservationRoot::Window(_)) && (host.is_web_process)(element)?; + let settles = !request.observation_mode.identity_only; + let web_process = + settles && matches!(root, ObservationRoot::Window(_)) && (host.is_web_process)(element)?; let mut tracker: Option<(Readiness, Instant)> = None; let mut previous: Option> = None; let mut sampling_deadline = request.deadline; + let full = full_read(request); loop { - let purpose = if tracker - .as_ref() - .is_some_and(|(readiness, _)| !readiness.needs_payload()) + let purpose = if full == ReadPurpose::Snapshot + && tracker + .as_ref() + .is_some_and(|(readiness, _)| !readiness.needs_payload()) { ReadPurpose::Sampling } else { - ReadPurpose::Snapshot + full }; let walk = match (host.walker)(element, limits, sampling_deadline, purpose) { Ok(walk) => walk, @@ -127,7 +132,8 @@ fn settled_walk( { return nested_surface(element, walk, limits, request, host); } - let waits = matches!(root, ObservationRoot::Window(_)) + let waits = settles + && matches!(root, ObservationRoot::Window(_)) && (web_process || has_web_document(&walk.root)); if !walk.complete || !waits { return Ok(if purpose == ReadPurpose::Sampling { @@ -164,7 +170,7 @@ fn settled_walk( } Step::Wait(gap) => { (host.pause)(gap); - if purpose == ReadPurpose::Snapshot { + if purpose == full { previous = Some(walk); } } @@ -191,7 +197,7 @@ fn nested_surface( max_logical_depth: whole.max_logical_depth, ..*limits }; - window_walk = (host.walker)(window, &uncut, request.deadline, ReadPurpose::Snapshot)?; + window_walk = (host.walker)(window, &uncut, request.deadline, full_read(request))?; } if !window_walk.complete || is_incomplete(&window_walk.root, request.surface) { return Err(AdapterError::timeout( @@ -213,10 +219,20 @@ fn nested_surface( .with_suggestion("Snapshot the window and select a dialog by ref")); } }; - let walk = (host.walker)(&dialog, limits, request.deadline, ReadPurpose::Snapshot)?; + let walk = (host.walker)(&dialog, limits, request.deadline, full_read(request))?; Ok((walk, true)) } +/// The walk's own read: identity evidence alone when the request only +/// resolves a saved ref, the full snapshot read otherwise. +fn full_read(request: &ObservationRequest) -> ReadPurpose { + if request.observation_mode.identity_only { + ReadPurpose::Count + } else { + ReadPurpose::Snapshot + } +} + /// What the settle loop needs from outside the tree: the walk itself, whether /// a process is a web engine or has stopped, and how to wait. struct SettleHost<'a> { @@ -257,7 +273,9 @@ fn observe_decorated( }; let started = Instant::now(); let (mut walk, settled) = settled_walk(&root, &element, &limits, &request, &mut host)?; - (host.place)(&mut walk)?; + if !request.observation_mode.identity_only { + (host.place)(&mut walk)?; + } let complete = walk.complete && settled; let displays = read_displays(); let (subtree, mut stats) = project_with(walk, &limits, decorate, displays.as_deref()); diff --git a/crates/linux/src/tree/observe/observe_sampling_tests.rs b/crates/linux/src/tree/observe/observe_sampling_tests.rs index 7d3732e95..eff8353a1 100644 --- a/crates/linux/src/tree/observe/observe_sampling_tests.rs +++ b/crates/linux/src/tree/observe/observe_sampling_tests.rs @@ -103,3 +103,34 @@ fn an_interrupted_narrow_sample_returns_the_full_payload_as_incomplete() { assert!(!complete); assert_eq!(root.identity.name.as_deref(), Some("b")); } + +#[test] +fn an_identity_only_read_skips_extents_actions_and_placement() { + let purposes = std::cell::RefCell::new(Vec::new()); + let mut walker = |_: &AtspiElement, _: &WalkLimits, _, purpose| { + purposes.borrow_mut().push(purpose); + walk_of(2, true) + }; + let window = window(); + let mut identity_only = request(5000); + identity_only.observation_mode.identity_only = true; + let result = observe_with( + ObservationRoot::Window(&window), + &identity_only, + SettleHost { + walker: &mut walker, + is_web_process: &|_| Ok(true), + stopped: &|_| false, + pause: &mut |_| {}, + place: &mut |_| panic!("an identity-only read must not place"), + }, + ) + .unwrap(); + assert!(result.is_complete()); + let purposes = purposes.borrow(); + assert!(!purposes.is_empty()); + assert!(purposes.iter().all(|purpose| purpose.identity() + && purpose.children() + && !purpose.bounds() + && !purpose.actions())); +} diff --git a/crates/linux/src/tree/observe/observe_settle_tests.rs b/crates/linux/src/tree/observe/observe_settle_tests.rs index e62e2ad84..1de9e2204 100644 --- a/crates/linux/src/tree/observe/observe_settle_tests.rs +++ b/crates/linux/src/tree/observe/observe_settle_tests.rs @@ -322,3 +322,31 @@ fn web_readiness_depends_on_document_evidence_not_tree_size() { let large_shell = observe(true, false, 1_000, &counts(&[80, 80, 80])); assert!(!large_shell.complete); } + +#[test] +fn an_identity_only_read_of_a_web_window_walks_once_without_waiting() { + let calls = Cell::new(0usize); + let paused = Cell::new(false); + let mut walker = |_: &AtspiElement, _: &WalkLimits, _, _| { + calls.set(calls.get() + 1); + walk_of(calls.get() + 1, true) + }; + let mut pause = |_: Duration| paused.set(true); + let window = window(); + let mut identity_only = request(5000); + identity_only.observation_mode.identity_only = true; + let tree = observe_with( + ObservationRoot::Window(&window), + &identity_only, + SettleHost { + walker: &mut walker, + place: &mut |_| Ok(()), + is_web_process: &|_| panic!("an identity-only read does not ask what the process is"), + stopped: &|_| false, + pause: &mut pause, + }, + ) + .expect("observes"); + assert_eq!((calls.get(), paused.get()), (1, false)); + assert!(tree.is_complete()); +} diff --git a/crates/linux/src/tree/observe/projection.rs b/crates/linux/src/tree/observe/projection.rs index bc728de0e..8bc969418 100644 --- a/crates/linux/src/tree/observe/projection.rs +++ b/crates/linux/src/tree/observe/projection.rs @@ -151,6 +151,7 @@ impl Projector<'_> { ) -> ObservedSubtree { let mut evidence = self.evidence_of(&node.payload, node.in_web_document); visibility.append(&node.payload, &mut evidence); + let wrapper = crate::tree::walk::wrapper::classify(&node.payload).wrapper; let WalkedNode { payload, children, @@ -177,6 +178,7 @@ impl Projector<'_> { } ObservedSubtree::new(evidence, children, !truncated, child_count) .with_source_child_index(source_index) + .with_wrapper(wrapper) } } diff --git a/crates/linux/src/tree/properties/read_purpose.rs b/crates/linux/src/tree/properties/read_purpose.rs index 2debe0e5c..82f4e2e99 100644 --- a/crates/linux/src/tree/properties/read_purpose.rs +++ b/crates/linux/src/tree/properties/read_purpose.rs @@ -3,6 +3,9 @@ pub(crate) enum ReadPurpose { Snapshot, Sampling, + /// A whole-window read that only counts a saved ref's matches: identity + /// and children, without extents or actions. + Count, Identity, Resolve, Element, @@ -20,7 +23,12 @@ impl ReadPurpose { pub(in crate::tree) fn identity(self) -> bool { matches!( self, - Self::Snapshot | Self::Sampling | Self::Identity | Self::Resolve | Self::Element + Self::Snapshot + | Self::Sampling + | Self::Count + | Self::Identity + | Self::Resolve + | Self::Element ) } @@ -33,7 +41,7 @@ impl ReadPurpose { } pub(in crate::tree) fn bounds(self) -> bool { - (self.identity() && self != Self::Sampling) + (self.identity() && !matches!(self, Self::Sampling | Self::Count)) || matches!(self, Self::State | Self::Bounds | Self::Frame) } @@ -49,6 +57,7 @@ impl ReadPurpose { self, Self::Snapshot | Self::Sampling + | Self::Count | Self::Structure | Self::Shape | Self::Gate diff --git a/crates/linux/src/tree/resolve/resolve_candidates.rs b/crates/linux/src/tree/resolve/resolve_candidates.rs index 19e893bab..03adae17f 100644 --- a/crates/linux/src/tree/resolve/resolve_candidates.rs +++ b/crates/linux/src/tree/resolve/resolve_candidates.rs @@ -8,7 +8,7 @@ use crate::tree::vocabulary::roles::may_resolve_to; use crate::tree::walk::walker_expand::read_observed; use crate::tree::{AtspiElement, ObservedRead, Walk, WalkLimits, WalkedNode}; -const NODES_IN_FLIGHT: usize = 8; +const NODES_IN_FLIGHT: usize = 32; fn may_be_target(node: &WalkedNode, entry: &RefEntry) -> bool { node.payload diff --git a/crates/linux/src/tree/resolve/resolve_refusal_tests.rs b/crates/linux/src/tree/resolve/resolve_refusal_tests.rs index d15f8865c..1600650e4 100644 --- a/crates/linux/src/tree/resolve/resolve_refusal_tests.rs +++ b/crates/linux/src/tree/resolve/resolve_refusal_tests.rs @@ -315,13 +315,24 @@ fn a_retained_object_ref_is_found_without_following_its_stored_path() { assert_eq!(fake.reads.searches, 1); } +/// Identity is the element's role and stable fields, so the replacement is +/// refused for its different name; a new object at the old rectangle alone is +/// not. #[test] -fn a_retained_object_ref_refuses_a_replacement_at_the_old_rectangle() { +fn a_retained_object_ref_refuses_a_differently_named_replacement_at_the_old_rectangle() { let mut fake = Fake::new(); - fake.add(FRAME_PATH, "/first", window_field(0, 38, 420, 34)); + let mut first = window_field(0, 38, 420, 34); + first.node.properties.name = Field::Present("First".to_owned()); + fake.add(FRAME_PATH, "/first", first); fake.add(FRAME_PATH, "/second", window_field(0, 80, 420, 34)); let mut entry = fake.entry_of("/first", true); entry.identity.retained_object = Some(":1.2/first".to_owned()); + assert_eq!( + resolve(&mut fake, &entry, deadline()) + .expect("the recorded object resolves") + .path(), + "/first" + ); let children = &mut fake.nodes.get_mut(FRAME_PATH).expect("frame").children; children.remove(0); diff --git a/crates/linux/src/tree/resolve/resolve_search.rs b/crates/linux/src/tree/resolve/resolve_search.rs index 3f75addcf..c38f9354f 100644 --- a/crates/linux/src/tree/resolve/resolve_search.rs +++ b/crates/linux/src/tree/resolve/resolve_search.rs @@ -22,12 +22,7 @@ fn visit( ) { let is_root = path.is_empty(); let limits = WalkLimits::default(); - let mut evidence = project_node(&node.payload, node.in_web_document, &limits); - if entry.identity.retained_object.is_some() { - evidence.identifiers = evidence - .identifiers - .with_retained_object(node.payload.node.element.object_identity()); - } + let evidence = project_node(&node.payload, node.in_web_document, &limits); let admission = admit_node(entry, &evidence); match admission { Admission::Collect => found.candidates.push(candidate( diff --git a/crates/linux/src/tree/resolve/resolve_search_tests.rs b/crates/linux/src/tree/resolve/resolve_search_tests.rs index b1aab50dd..dc81a1314 100644 --- a/crates/linux/src/tree/resolve/resolve_search_tests.rs +++ b/crates/linux/src/tree/resolve/resolve_search_tests.rs @@ -10,7 +10,7 @@ fn a_childless_native_frame_is_conclusively_empty() { } #[test] -fn retained_unnamed_containers_are_not_blocked_by_named_peers() { +fn an_unnamed_container_is_never_admitted_by_its_object_path() { use crate::tree::resolve::resolve_fake::{FRAME_PATH, node}; use crate::tree::vocabulary::role_numbers::PANEL; @@ -20,18 +20,17 @@ fn retained_unnamed_containers_are_not_blocked_by_named_peers() { let mut entry = fake.entry_of("/viewport", false); entry.identity.retained_object = Some(":1.2/viewport".to_owned()); let found = evaluate(&fake.walk(), &entry, false); - assert!(!found.unread_region); - assert_eq!(found.candidates.len(), 1); - assert_eq!(found.candidates[0].element.path(), "/viewport"); + assert!(found.candidates.is_empty()); + assert!(found.unread_region); entry.identity.retained_object = Some(":1.2/replaced".to_owned()); let found = evaluate(&fake.walk(), &entry, false); - assert!(!found.unread_region); assert!(found.candidates.is_empty()); + assert!(found.unread_region); } #[test] -fn a_named_menu_item_resolves_only_its_context_scoped_object_path() { +fn a_named_menu_item_is_counted_by_its_identity_not_its_object_path() { use crate::tree::resolve::resolve_fake::{FRAME_PATH, node}; use crate::tree::vocabulary::role_numbers::MENU_ITEM; @@ -48,11 +47,10 @@ fn a_named_menu_item_resolves_only_its_context_scoped_object_path() { let found = evaluate(&fake.walk(), &entry, false); assert!(!found.unread_region); - assert_eq!(found.candidates.len(), 1); - assert_eq!(found.candidates[0].element.path(), "/preferences"); + assert_eq!(found.candidates.len(), 2); entry.identity.retained_object = Some(":1.2/removed".to_owned()); let found = evaluate(&fake.walk(), &entry, false); assert!(!found.unread_region); - assert!(found.candidates.is_empty()); + assert_eq!(found.candidates.len(), 2); } diff --git a/crates/linux/src/tree/resolve/resolve_web_tests.rs b/crates/linux/src/tree/resolve/resolve_web_tests.rs index 435392e88..d94cd98ff 100644 --- a/crates/linux/src/tree/resolve/resolve_web_tests.rs +++ b/crates/linux/src/tree/resolve/resolve_web_tests.rs @@ -89,29 +89,31 @@ fn remote_panel() -> Fake { fake } +/// Identity is the element's role and stable fields, so the changed name is +/// what refuses the replacement; another owner at the same path alone does not. #[test] -fn a_retained_ref_refuses_the_same_path_and_rectangle_on_another_owner() { +fn a_retained_ref_refuses_a_differently_named_replacement_on_another_owner() { let mut fake = Fake::new(); fake.add( FRAME_PATH, "/field", - node(crate::tree::vocabulary::role_numbers::ENTRY, "", 100, 24), + node( + crate::tree::vocabulary::role_numbers::ENTRY, + "Field", + 100, + 24, + ), ); let mut entry = fake.entry_of("/field", true); - let original = resolve(&mut fake, &entry, deadline()).expect("original field"); - entry.identity.retained_object = fake - .evidence(&original) - .unwrap() - .identifiers - .retained_object() - .map(str::to_owned); - assert!(entry.identity.retained_object.is_some()); + entry.identity.retained_object = Some(":1.2/field".to_owned()); assert_eq!( resolve(&mut fake, &entry, deadline()).unwrap().bus_name(), BUS ); fake.pids.insert(":1.99".to_owned(), 99); - fake.nodes.get_mut("/field").unwrap().bus = ":1.99".to_owned(); + let other = fake.nodes.get_mut("/field").unwrap(); + other.bus = ":1.99".to_owned(); + other.read.node.properties.name = Field::Present("Other".to_owned()); let error = resolve(&mut fake, &entry, deadline()).expect_err("replacement owner"); assert_eq!(error.code, ErrorCode::StaleRef); } diff --git a/crates/linux/src/tree/walk/walker_expand.rs b/crates/linux/src/tree/walk/walker_expand.rs index a09ea285d..86cb162a6 100644 --- a/crates/linux/src/tree/walk/walker_expand.rs +++ b/crates/linux/src/tree/walk/walker_expand.rs @@ -15,7 +15,7 @@ use crate::tree::properties::read_purpose::ReadPurpose; use crate::tree::vocabulary::label_read::read_label_if_needed; use crate::tree::{AtspiElement, Expansion, ObservedRead, RawNode, WalkLimits}; -const NODES_IN_FLIGHT: usize = 8; +const NODES_IN_FLIGHT: usize = 32; const PID_LOOKUPS_IN_FLIGHT: usize = 64; /// Stands in for the process id of a bus name whose lookup failed. Process @@ -113,8 +113,8 @@ impl<'a> LevelReader<'a> { self } - /// Reads one level of the tree with at most `NODES_IN_FLIGHT` nodes (about - /// sixty calls) outstanding at a time. Below the root a node that is gone, + /// Reads one level of the tree with at most `NODES_IN_FLIGHT` nodes + /// outstanding at a time, so a wide level costs one round of calls. Below the root a node that is gone, /// or whose process id could not be found, is kept as an unreadable node /// marked gone. Transport failures abort the walk at every depth. pub(in crate::tree) fn expand( diff --git a/crates/macos/src/system/process/process_tests.rs b/crates/macos/src/system/process/process_tests.rs index 25030fe07..ce7b1584c 100644 --- a/crates/macos/src/system/process/process_tests.rs +++ b/crates/macos/src/system/process/process_tests.rs @@ -80,7 +80,7 @@ fn zombie_only_group_does_not_turn_sigkill_eperm_into_cleanup_failure() { let process_group = i32::try_from(child.id()).expect("child pid fits macOS pid_t"); std::thread::sleep(Duration::from_millis(25)); let mut signals = Vec::new(); - let deadline = Instant::now() + Duration::from_millis(100); + let deadline = Instant::now() + Duration::from_secs(5); let failures = terminate_process_group_with(&mut child, process_group, deadline, |group, signal| { signals.push(signal); diff --git a/crates/macos/src/tree/adapter.rs b/crates/macos/src/tree/adapter.rs index 7dee59cdd..c273ad8ea 100644 --- a/crates/macos/src/tree/adapter.rs +++ b/crates/macos/src/tree/adapter.rs @@ -35,6 +35,10 @@ impl ObservationOps for MacOSAdapter { crate::tree::resolve::resolve_element_with_deadline(entry, deadline) } + fn retains_element_tokens(&self) -> bool { + crate::tree::resolve::retained::tokens_available() + } + fn resolve_locator_anchor( &self, entry: &RefEntry, diff --git a/crates/macos/src/tree/query/mod.rs b/crates/macos/src/tree/query/mod.rs index ecc4e7c74..2e33c361f 100644 --- a/crates/macos/src/tree/query/mod.rs +++ b/crates/macos/src/tree/query/mod.rs @@ -264,6 +264,7 @@ mod tests { value: None, description: None, native_id: None, + unique_at_capture: false, }, geometry: RefGeometry { bounds_space: None, @@ -280,6 +281,7 @@ mod tests { source_window_title: None, source_window_bounds_hash: None, source_surface: Default::default(), + renderer_accessibility_forced: false, }, scope: RefScope { root_ref: None, diff --git a/crates/macos/src/tree/query/traversal.rs b/crates/macos/src/tree/query/traversal.rs index 02f74d1e5..daf4810d2 100644 --- a/crates/macos/src/tree/query/traversal.rs +++ b/crates/macos/src/tree/query/traversal.rs @@ -161,12 +161,10 @@ impl LocatorTraversal { if !subtree_complete { self.arena.mark_incomplete(); } - Ok(Some(ObservedSubtree::new( - read.evidence, - children, - subtree_complete, - children_count, - ))) + Ok(Some( + ObservedSubtree::new(read.evidence, children, subtree_complete, children_count) + .with_wrapper(read.web_wrapper), + )) } fn visit_children( diff --git a/crates/macos/src/tree/resolve/mod.rs b/crates/macos/src/tree/resolve/mod.rs index b795b9271..4461ea553 100644 --- a/crates/macos/src/tree/resolve/mod.rs +++ b/crates/macos/src/tree/resolve/mod.rs @@ -1,27 +1,33 @@ #[cfg(target_os = "macos")] -use agent_desktop_core::refs::ref_identity::has_meaningful_identity; +use agent_desktop_core::refs::ref_identity::element_token; use agent_desktop_core::{AdapterError, ErrorCode, NativeHandle, RefEntry, SnapshotSurface}; use std::time::{Duration, Instant}; #[cfg(target_os = "macos")] use super::resolve::resolve_classify::identity_summary_for_message; #[cfg(target_os = "macos")] +use super::resolve::resolve_errors::element_token_required; +#[cfg(target_os = "macos")] use super::resolve::resolve_read_context::ResolveReadContext; #[cfg(target_os = "macos")] use super::resolve::resolve_roots::{candidate_roots, source_window_scope_required}; #[cfg(target_os = "macos")] -use super::resolve::resolve_search::{find_entry_by_path, find_entry_in_roots}; - -const MAX_RESOLVE_DEPTH: u8 = 50; +use super::resolve::resolve_search::find_entry_by_path; +/// Re-identifies a saved ref by the element token captured with it, and refuses +/// any ref that carries none: a token names the exact element, so no path or +/// search can stand in for it. #[cfg(target_os = "macos")] pub(crate) fn resolve_element_with_deadline( entry: &RefEntry, operation_deadline: agent_desktop_core::Deadline, ) -> Result { verify_process_instance(entry)?; + if element_token(entry).is_none() { + return Err(element_token_required(entry)); + } let deadline = crate::tree::locator_deadline::from_operation(operation_deadline)?; - let result = retry_incomplete_until(deadline, || resolve_once(entry, deadline)); + let result = retry_incomplete_until(deadline, || resolve_token_once(entry, deadline)); match result { Err(error) if error.code == ErrorCode::ElementNotFound => { Err(stale_ref_error(entry, &error)) @@ -67,73 +73,14 @@ fn verify_process_instance(entry: &RefEntry) -> Result<(), AdapterError> { } #[cfg(target_os = "macos")] -fn resolve_once(entry: &RefEntry, deadline: Instant) -> Result { +fn resolve_token_once(entry: &RefEntry, deadline: Instant) -> Result { let started = Instant::now(); crate::tree::locator_deadline::remaining(deadline)?; let mut read_context = ResolveReadContext::new(deadline); - if let Some(result) = token_resolution(entry, deadline, &mut read_context, started) { - return result; + match token_resolution(entry, deadline, &mut read_context, started) { + Some(result) => result, + None => Err(element_token_required(entry)), } - if can_use_path_fast_path(entry) { - let roots = match candidate_roots(entry, &mut read_context) { - Ok(roots) => roots, - Err(error) => { - return finish_resolution( - Err(error), - &mut read_context, - started, - "path_roots", - true, - ); - } - }; - match find_entry_by_path(&roots.roots, entry, roots.scope_verified, &mut read_context) { - Ok(handle) => { - return finish_resolution( - Ok(handle), - &mut read_context, - started, - "path_success", - true, - ); - } - Err(error) if error.code == ErrorCode::ElementNotFound => { - tracing::debug!(stats = ?read_context.stats, "strict resolution path fast-path fell back to broad search"); - } - Err(error) => { - return finish_resolution( - Err(error), - &mut read_context, - started, - "path_error", - true, - ); - } - } - } - if !can_use_broad_search(entry) { - return finish_resolution( - Err(AdapterError::element_not_found("element")), - &mut read_context, - started, - "identity_insufficient", - false, - ); - } - let roots = match candidate_roots(entry, &mut read_context) { - Ok(roots) => roots, - Err(error) => { - return finish_resolution(Err(error), &mut read_context, started, "broad_roots", false); - } - }; - let result = find_entry_in_roots( - &roots.roots, - entry, - MAX_RESOLVE_DEPTH, - roots.scope_verified, - &mut read_context, - ); - finish_resolution(result, &mut read_context, started, "broad_search", false) } #[cfg(target_os = "macos")] @@ -168,7 +115,7 @@ fn resolve_locator_anchor_once( ); } }; - let result = find_entry_by_path(&roots.roots, entry, roots.scope_verified, &mut read_context); + let result = find_entry_by_path(&roots, entry, &mut read_context); finish_resolution( result, &mut read_context, @@ -180,7 +127,7 @@ fn resolve_locator_anchor_once( /// The token answer a resolution pass shares: `Some` when the saved ref carries /// a retained object, whether it resolved or was refused, and `None` when the -/// entry has no token and the path and search passes must decide. +/// entry has no token, which only the locator-anchor pass answers with a path. #[cfg(target_os = "macos")] fn token_resolution( entry: &RefEntry, @@ -299,19 +246,8 @@ fn is_retryable_resolution_error(error: &AdapterError) -> bool { error.is_transient_incomplete_read() } -#[cfg(target_os = "macos")] -fn can_use_path_fast_path(entry: &RefEntry) -> bool { - let source_scoped = if entry.source.source_surface == SnapshotSurface::Window { - source_window_scope_required(entry) - } else { - true - }; - source_scoped - && (entry.scope.root_ref.is_none() || entry.scope.path_is_absolute) - && !entry.scope.path.is_empty() - && (entry.geometry.bounds_hash.is_some() || has_meaningful_identity(entry)) -} - +/// A tokenless anchor lands by position, so the gate asks only that its path be +/// anchored to a recorded window or surface; core confirms identity after landing. #[cfg(target_os = "macos")] fn can_use_locator_anchor_path(entry: &RefEntry) -> bool { let source_scoped = if entry.source.source_surface == SnapshotSurface::Window { @@ -319,18 +255,10 @@ fn can_use_locator_anchor_path(entry: &RefEntry) -> bool { } else { true }; - source_scoped - && (entry.scope.root_ref.is_none() || entry.scope.path_is_absolute) - && (entry.geometry.bounds_hash.is_some() || has_meaningful_identity(entry)) -} - -#[cfg(target_os = "macos")] -fn can_use_broad_search(entry: &RefEntry) -> bool { - entry.geometry.bounds_hash.is_some() || has_meaningful_identity(entry) + source_scoped && (entry.scope.root_ref.is_none() || entry.scope.path_is_absolute) } pub(crate) mod resolve_ax_read; -pub(super) mod resolve_bounds; pub(super) mod resolve_classify; pub(super) mod resolve_errors; pub(super) mod resolve_read_context; diff --git a/crates/macos/src/tree/resolve/resolve_bounds.rs b/crates/macos/src/tree/resolve/resolve_bounds.rs deleted file mode 100644 index 49a86884e..000000000 --- a/crates/macos/src/tree/resolve/resolve_bounds.rs +++ /dev/null @@ -1,38 +0,0 @@ -use agent_desktop_core::{Rect, RefEntry}; - -use crate::tree::AXElement; - -#[cfg(target_os = "macos")] -pub(in crate::tree) fn bounds_match_with_deadline( - element: &AXElement, - entry: &RefEntry, - deadline: std::time::Instant, -) -> Result { - let Some(expected) = entry.geometry.bounds_hash else { - return Ok(true); - }; - Ok(read_bounds_with_deadline(element, deadline)? - .map(|actual| actual.bounds_hash() == Some(expected)) - .unwrap_or(false)) -} - -#[cfg(target_os = "macos")] -fn read_bounds_with_deadline( - element: &AXElement, - deadline: std::time::Instant, -) -> Result, agent_desktop_core::AdapterError> { - #[cfg(test)] - if element.0.is_null() { - return Ok(None); - } - let position = super::resolve_ax_read::read_point( - element, - accessibility_sys::kAXPositionAttribute, - deadline, - )?; - let size = - super::resolve_ax_read::read_size(element, accessibility_sys::kAXSizeAttribute, deadline)?; - Ok(position - .zip(size) - .and_then(|(position, size)| crate::tree::element_bounds::rect_from_parts(position, size))) -} diff --git a/crates/macos/src/tree/resolve/resolve_classify.rs b/crates/macos/src/tree/resolve/resolve_classify.rs index 37420ba82..0cd566d7a 100644 --- a/crates/macos/src/tree/resolve/resolve_classify.rs +++ b/crates/macos/src/tree/resolve/resolve_classify.rs @@ -1,122 +1,21 @@ -use agent_desktop_core::{ - AdapterError, NativeHandle, RefEntry, refs::ref_identity::has_meaningful_identity, -}; +use agent_desktop_core::{AdapterError, NativeHandle, RefEntry}; -#[cfg(target_os = "macos")] -use super::resolve_bounds::bounds_match_with_deadline; use crate::tree::AXElement; +/// Turns the role-matching landings into a handle: one is the element core +/// then confirms by its full predicate, none is not found, more is ambiguous. #[cfg(target_os = "macos")] pub(in crate::tree) fn classify_candidates( mut matches: Vec, entry: &RefEntry, - source_window_verified: bool, - deadline: std::time::Instant, ) -> Result { match matches.len() { 0 => Err(AdapterError::element_not_found("element")), - 1 => { - let candidate = matches.remove(0); - if candidate_is_sufficiently_verified( - &candidate, - entry, - source_window_verified, - deadline, - )? { - retained_handle(candidate) - } else { - Err(AdapterError::element_not_found("element")) - } - } - _ => classify_ambiguous_candidates(matches, entry, deadline), - } -} - -#[cfg(target_os = "macos")] -fn candidate_is_sufficiently_verified( - candidate: &AXElement, - entry: &RefEntry, - source_window_verified: bool, - deadline: std::time::Instant, -) -> Result { - if source_window_verified && !candidate_requires_bounds(entry) { - return Ok(true); - } - verified_bounds_match(candidate, entry, deadline) -} - -fn candidate_requires_bounds(entry: &RefEntry) -> bool { - !has_meaningful_identity(entry) -} - -#[cfg(target_os = "macos")] -fn verified_bounds_match( - candidate: &AXElement, - entry: &RefEntry, - deadline: std::time::Instant, -) -> Result { - if entry.geometry.bounds_hash.is_none() { - return Ok(false); - } - bounds_match_with_deadline(candidate, entry, deadline) -} - -#[cfg(target_os = "macos")] -fn classify_ambiguous_candidates( - matches: Vec, - entry: &RefEntry, - deadline: std::time::Instant, -) -> Result { - let has_bounds_hash = entry.geometry.bounds_hash.is_some(); - let mut bounds_matches = Vec::new(); - if has_bounds_hash { - for candidate in &matches { - if verified_bounds_match(candidate, entry, deadline)? { - bounds_matches.push(candidate.clone()); - } - } - } - match classify_bounds_matches(bounds_matches.len(), has_bounds_hash) { - BoundsMatchOutcome::Resolved => retained_handle(bounds_matches.remove(0)), - BoundsMatchOutcome::Stale => Err(stale_ref_from_bounds_mismatch(entry, matches.len())), - BoundsMatchOutcome::Ambiguous => Err(ambiguous_target_error(&matches, entry)), - } -} - -#[derive(Debug, PartialEq, Eq)] -pub(in crate::tree) enum BoundsMatchOutcome { - Resolved, - Stale, - Ambiguous, -} - -pub(in crate::tree) fn classify_bounds_matches( - bounds_match_count: usize, - has_bounds_hash: bool, -) -> BoundsMatchOutcome { - if !has_bounds_hash { - return BoundsMatchOutcome::Ambiguous; - } - match bounds_match_count { - 0 => BoundsMatchOutcome::Stale, - 1 => BoundsMatchOutcome::Resolved, - _ => BoundsMatchOutcome::Ambiguous, + 1 => retained_handle(matches.remove(0)), + _ => Err(ambiguous_target_error(&matches, entry)), } } -#[cfg(target_os = "macos")] -fn stale_ref_from_bounds_mismatch(entry: &RefEntry, candidate_count: usize) -> AdapterError { - AdapterError::stale_ref_because(format!( - "No live candidate matched the saved element's bounds: \ - {candidate_count} shared its identity, none its geometry" - )) - .with_details(serde_json::json!({ - "kind": "bounds_mismatch", - "candidate_count": candidate_count, - "identity": identity_summary_for_message(entry), - })) -} - #[cfg(target_os = "macos")] fn ambiguous_target_error(matches: &[AXElement], entry: &RefEntry) -> AdapterError { let count = matches.len(); diff --git a/crates/macos/src/tree/resolve/resolve_errors.rs b/crates/macos/src/tree/resolve/resolve_errors.rs index 597cdf967..95cabbfde 100644 --- a/crates/macos/src/tree/resolve/resolve_errors.rs +++ b/crates/macos/src/tree/resolve/resolve_errors.rs @@ -1,25 +1,29 @@ use agent_desktop_core::{AdapterError, DeliverySemantics, ErrorCode, RefEntry}; -pub(in crate::tree) fn native_identifier_role_reuse(entry: &RefEntry) -> AdapterError { +/// Core owns the payload (`agent_desktop_core::refs::resolve_errors::identity_unknown_error`) +/// so both adapters share one definition: an unreadable candidate +/// means the same thing whichever adapter's search produced it. +pub(in crate::tree) use agent_desktop_core::refs::resolve_errors::identity_unknown_error as identity_unknown; + +/// macOS re-identifies a saved ref only through the element token captured with +/// it, so a ref that carries none is refused instead of searched for. +pub(in crate::tree) fn element_token_required(entry: &RefEntry) -> AdapterError { AdapterError::new( - ErrorCode::StaleRef, - "Saved native identifier was found only on a different accessibility role", + ErrorCode::ElementNotFound, + "A saved ref on macOS is re-identified only through the element token captured with it, \ + and this ref carries none", + ) + .with_suggestion( + "Run 'snapshot' to take a ref that carries an element token, then retry with the new ref", ) - .with_suggestion("Run 'snapshot' to refresh, then retry with the updated ref.") .with_disposition(DeliverySemantics::not_delivered()) .with_details(serde_json::json!({ - "kind": "native_identifier_role_reuse", - "expected_role": entry.identity.role, - "complete": true, + "kind": "element_token_required", "retryable": false, + "role": entry.identity.role, })) } -/// Core owns the payload (`agent_desktop_core::refs::resolve_errors::identity_unknown_error`) -/// so both adapters share one definition: an unreadable candidate -/// means the same thing whichever adapter's search produced it. -pub(in crate::tree) use agent_desktop_core::refs::resolve_errors::identity_unknown_error as identity_unknown; - pub(in crate::tree) fn incomplete_traversal(phase: &str, depth: u8) -> AdapterError { AdapterError::new( ErrorCode::AppUnresponsive, diff --git a/crates/macos/src/tree/resolve/resolve_roots.rs b/crates/macos/src/tree/resolve/resolve_roots.rs index 832bfe3dd..8e3cd0c40 100644 --- a/crates/macos/src/tree/resolve/resolve_roots.rs +++ b/crates/macos/src/tree/resolve/resolve_roots.rs @@ -5,17 +5,11 @@ use crate::tree::AXElement; use crate::tree::element_dedupe::push_unique; use crate::tree::properties::element::element_for_pid; -#[cfg(target_os = "macos")] -pub(in crate::tree) struct CandidateRoots { - pub roots: Vec, - pub scope_verified: bool, -} - #[cfg(target_os = "macos")] pub(in crate::tree) fn candidate_roots( entry: &RefEntry, context: &mut ResolveReadContext, -) -> Result { +) -> Result, AdapterError> { let deadline = context.deadline; crate::tree::locator_deadline::remaining(deadline)?; let pid = crate::system::process::process_identity::to_pid_t(entry.process.pid)?; @@ -55,10 +49,7 @@ pub(in crate::tree) fn candidate_roots( read_root_array(&application, "AXChildren", context)?, ); crate::tree::locator_deadline::remaining(deadline)?; - Ok(CandidateRoots { - roots, - scope_verified: false, - }) + Ok(roots) } #[cfg(target_os = "macos")] @@ -66,7 +57,7 @@ fn source_surface_scoped_roots( application: &AXElement, entry: &RefEntry, context: &mut ResolveReadContext, -) -> Result { +) -> Result, AdapterError> { let deadline = context.deadline; let pid = crate::system::process::process_identity::to_pid_t(entry.process.pid)?; let root = match entry.source.source_surface { @@ -91,14 +82,11 @@ fn source_surface_scoped_roots( ) })?; let root = surface_or_bridge_error(root, error)?; - return Ok(CandidateRoots { - scope_verified: true, - roots: vec![root], - }); + return Ok(vec![root]); } Err(error) => return Err(error), }; - match scoped.roots.into_iter().next() { + match scoped.into_iter().next() { Some(window) => crate::tree::surfaces::surface_in_window( &window, entry.source.source_surface, @@ -128,10 +116,7 @@ fn source_surface_scoped_roots( ), ); }; - Ok(CandidateRoots { - roots: vec![root], - scope_verified: true, - }) + Ok(vec![root]) } #[cfg(target_os = "macos")] @@ -139,7 +124,7 @@ fn source_window_scoped_roots( application: &AXElement, entry: &RefEntry, context: &mut ResolveReadContext, -) -> Result { +) -> Result, AdapterError> { let deadline = context.deadline; if let Some(id) = entry.source.source_window_id.as_deref() { if source_window_number(entry).is_none() { @@ -176,19 +161,7 @@ fn source_window_scoped_roots( context, )? }; - let scope_verified = - source_scope_verified(entry.source.source_window_id.as_deref(), window.is_some()); - Ok(CandidateRoots { - scope_verified, - roots: window.into_iter().collect(), - }) -} - -pub(in crate::tree) fn source_scope_verified( - source_window_id: Option<&str>, - matched: bool, -) -> bool { - source_window_id.is_some() && matched + Ok(window.into_iter().collect()) } #[cfg(target_os = "macos")] diff --git a/crates/macos/src/tree/resolve/resolve_search.rs b/crates/macos/src/tree/resolve/resolve_search.rs index 018bdb49b..37d6be165 100644 --- a/crates/macos/src/tree/resolve/resolve_search.rs +++ b/crates/macos/src/tree/resolve/resolve_search.rs @@ -1,24 +1,21 @@ use agent_desktop_core::{ - AdapterError, EvidenceRequirements, IdentityMatch, NativeHandle, RefEntry, - refs::ref_identity::{has_stable_text_identity, identity_match}, + AdapterError, EvidenceRequirements, IdentityMatch, LocatorField, NativeHandle, RefEntry, }; -use rustc_hash::FxHashSet; use super::resolve_classify::classify_candidates; +use super::resolve_errors::identity_unknown as identity_unknown_error; pub(in crate::tree) use super::resolve_errors::incomplete_traversal as incomplete_traversal_error; -use super::resolve_errors::{ - identity_unknown as identity_unknown_error, - native_identifier_role_reuse as native_identifier_role_reuse_error, -}; use super::resolve_read_context::ResolveReadContext; use crate::tree::AXElement; use crate::tree::element_dedupe::push_unique; +/// Lands a tokenless anchor by its path inside the scoped roots. Only the role +/// of the landed element is checked here; core confirms its identity against +/// the full recorded predicate after the landing. #[cfg(target_os = "macos")] pub(in crate::tree) fn find_entry_by_path( roots: &[AXElement], entry: &RefEntry, - source_window_verified: bool, read_context: &mut ResolveReadContext, ) -> Result { crate::tree::locator_deadline::remaining(read_context.deadline)?; @@ -29,8 +26,7 @@ pub(in crate::tree) fn find_entry_by_path( let Some(candidate) = element_at_path(root, &entry.scope.path, read_context)? else { continue; }; - let identity = candidate_identity(&candidate, entry, read_context)?; - match identity { + match landed_role_match(&candidate, entry, read_context)? { IdentityMatch::Match => { push_unique(&mut matches, candidate); } @@ -39,27 +35,14 @@ pub(in crate::tree) fn find_entry_by_path( } IdentityMatch::NoMatch => {} } - if should_stop_collecting(matches.len(), entry) { + if matches.len() > 1 { break; } } - if should_stop_collecting(matches.len(), entry) { - return classify_candidates( - matches, - entry, - source_window_verified, - read_context.deadline, - ); - } - if let Some(error) = incomplete { - return Err(error); + match incomplete { + Some(error) if matches.len() < 2 => Err(error), + _ => classify_candidates(matches, entry), } - classify_candidates( - matches, - entry, - source_window_verified, - read_context.deadline, - ) } #[cfg(target_os = "macos")] @@ -126,209 +109,29 @@ fn record_path_child_read( } #[cfg(target_os = "macos")] -pub(in crate::tree) fn find_entry_in_roots( - roots: &[AXElement], - entry: &RefEntry, - resolve_depth: u8, - source_window_verified: bool, - read_context: &mut ResolveReadContext, -) -> Result { - let mut matches = Vec::new(); - let mut incomplete = None; - let mut identifier_role_reuse = false; - for root in roots { - if should_stop_collecting(matches.len(), entry) { - break; - } - let mut ancestors = FxHashSet::default(); - let mut context = CollectContext { - entry, - max_depth: resolve_depth, - ancestors: &mut ancestors, - matches: &mut matches, - incomplete: &mut incomplete, - identifier_role_reuse: &mut identifier_role_reuse, - read_context, - }; - collect_elements_recursive(root, 0, &mut context)?; - } - if should_stop_collecting(matches.len(), entry) { - return classify_candidates( - matches, - entry, - source_window_verified, - read_context.deadline, - ); - } - if let Some(error) = incomplete { - return Err(error); - } - if identifier_role_reuse && matches.is_empty() { - return Err(native_identifier_role_reuse_error(entry)); - } - classify_candidates( - matches, - entry, - source_window_verified, - read_context.deadline, - ) -} - -#[cfg(target_os = "macos")] -struct CollectContext<'a> { - entry: &'a RefEntry, - max_depth: u8, - ancestors: &'a mut FxHashSet, - matches: &'a mut Vec, - incomplete: &'a mut Option, - identifier_role_reuse: &'a mut bool, - read_context: &'a mut ResolveReadContext, -} - -#[cfg(target_os = "macos")] -fn collect_elements_recursive( - element: &AXElement, - depth: u8, - context: &mut CollectContext<'_>, -) -> Result<(), AdapterError> { - if should_stop_collecting(context.matches.len(), context.entry) { - return Ok(()); - } - crate::tree::locator_deadline::remaining(context.read_context.deadline)?; - let pointer = element.0 as usize; - if !context.ancestors.insert(pointer) { - return Ok(()); - } - - let read = read_node( - element, - identity_requirements(context.entry), - context.read_context, - )?; - if read.invalid_element { - mark_incomplete( - context, - incomplete_traversal_error("invalid_element", depth), - ); - context.ancestors.remove(&pointer); - return Ok(()); - } - - if native_identifier_reused_by_different_role(context.entry, &read.evidence) { - *context.identifier_role_reuse = true; - } - - match read.evidence.role.known() { - Some(role) => { - if role == &context.entry.identity.role { - match super::retained::candidate_match(element, context.entry, &read.evidence)? { - IdentityMatch::Match => { - push_unique(context.matches, element.clone()); - } - IdentityMatch::Unknown => { - if provisional_geometry_candidate(context.entry) { - push_unique(context.matches, element.clone()); - } else { - mark_incomplete(context, identity_unknown_error(context.entry)); - } - } - IdentityMatch::NoMatch => {} - } - } - } - None if read.evidence.role.is_unknown() => { - mark_incomplete(context, incomplete_traversal_error("role", depth)); - } - None => {} - } - - if depth >= context.max_depth { - if !read.child_read.elements.is_empty() - || !read.child_read.complete - || read.child_read.truncated() - { - mark_incomplete(context, incomplete_traversal_error("depth_limit", depth)); - } - } else { - if !read.child_read.complete || read.child_read.truncated() { - mark_incomplete(context, incomplete_traversal_error("children", depth)); - } - for child in &read.child_read.elements { - collect_elements_recursive(child, depth.saturating_add(1), context)?; - if should_stop_collecting(context.matches.len(), context.entry) { - break; - } - } - } - - context.ancestors.remove(&pointer); - Ok(()) -} - -#[cfg(target_os = "macos")] -fn candidate_identity( +fn landed_role_match( element: &AXElement, entry: &RefEntry, context: &mut ResolveReadContext, ) -> Result { - let read = read_node(element, identity_requirements(entry), context)?; + let requirements = EvidenceRequirements { + role: true, + ..EvidenceRequirements::default() + }; + let read = read_node(element, requirements, context)?; if read.invalid_element { return Ok(IdentityMatch::Unknown); } - let matched = match read.evidence.role.known() { - Some(role) if role == &entry.identity.role => { - let identity = super::retained::candidate_match(element, entry, &read.evidence)?; - if identity == IdentityMatch::Unknown && provisional_geometry_candidate(entry) { - IdentityMatch::Match - } else { - identity - } - } - Some(_) => IdentityMatch::NoMatch, - None if read.evidence.role.is_unknown() => IdentityMatch::Unknown, - None => IdentityMatch::NoMatch, - }; - Ok(matched) + Ok(role_match(&read.evidence.role, &entry.identity.role)) } -pub(in crate::tree) fn match_native_or_text_identity( - entry: &RefEntry, - evidence: &agent_desktop_core::LocatorEvidence, -) -> IdentityMatch { - identity_match( - entry, - &evidence.name, - &evidence.value, - &evidence.description, - &evidence.identifiers, - ) -} - -pub(in crate::tree) fn native_identifier_reused_by_different_role( - entry: &RefEntry, - evidence: &agent_desktop_core::LocatorEvidence, -) -> bool { - let Some(expected) = entry.identity.native_id.as_ref() else { - return false; - }; - evidence - .role - .known() - .is_some_and(|role| role != &entry.identity.role) - && evidence.identifiers.is_complete() - && evidence - .identifiers - .identifiers() - .iter() - .any(|identifier| identifier == expected) -} - -/// Whether bounds alone may stand in for an element whose identity is unknown. -/// A ref that recorded a token never qualifies: only that token may match it, -/// which `has_meaningful_identity` already enforces. -fn provisional_geometry_candidate(entry: &RefEntry) -> bool { - entry.geometry.bounds_hash.is_some() - && !agent_desktop_core::refs::ref_identity::has_meaningful_identity(entry) +fn role_match(role: &LocatorField, expected: &str) -> IdentityMatch { + match role.known() { + Some(actual) if actual == expected => IdentityMatch::Match, + Some(_) => IdentityMatch::NoMatch, + None if role.is_unknown() => IdentityMatch::Unknown, + None => IdentityMatch::NoMatch, + } } #[cfg(target_os = "macos")] @@ -361,29 +164,6 @@ fn read_node( Ok(read) } -fn identity_requirements(entry: &RefEntry) -> EvidenceRequirements { - let semantic_text = has_stable_text_identity(entry) || entry.identity.role == "cell"; - EvidenceRequirements { - role: true, - name: semantic_text, - description: semantic_text, - value: semantic_text, - identifiers: entry.identity.native_id.is_some(), - states: false, - ref_evidence: Default::default(), - } -} - -pub(in crate::tree) fn should_stop_collecting(match_count: usize, entry: &RefEntry) -> bool { - match_count > 1 && entry.geometry.bounds_hash.is_none() -} - -fn mark_incomplete(context: &mut CollectContext<'_>, error: AdapterError) { - if context.incomplete.is_none() { - *context.incomplete = Some(error); - } -} - #[cfg(all(test, target_os = "macos"))] #[path = "resolve_search_tests.rs"] mod tests; diff --git a/crates/macos/src/tree/resolve/resolve_search_tests.rs b/crates/macos/src/tree/resolve/resolve_search_tests.rs index 7a4c20818..79d307471 100644 --- a/crates/macos/src/tree/resolve/resolve_search_tests.rs +++ b/crates/macos/src/tree/resolve/resolve_search_tests.rs @@ -1,51 +1,26 @@ -use super::provisional_geometry_candidate; -use agent_desktop_core::{ - ProcessId, RefCapabilities, RefEntry, RefEntryIdentity, RefGeometry, RefProcess, RefScope, - RefSource, SnapshotSurface, -}; +use super::role_match; +use agent_desktop_core::{IdentityMatch, LocatorField}; -fn bounds_only_entry(retained_object: Option<&str>) -> RefEntry { - RefEntry { - process: RefProcess { - pid: ProcessId::new(7), - process_instance: Some("test-instance".into()), - }, - identity: RefEntryIdentity { - retained_object: retained_object.map(String::from), - role: "cell".into(), - name: None, - value: None, - description: None, - native_id: None, - }, - geometry: RefGeometry { - bounds_space: None, - bounds: None, - bounds_hash: Some(42), - }, - capabilities: RefCapabilities { - states: vec![], - available_actions: vec![], - }, - source: RefSource { - source_app: None, - source_window_id: Some("w-10".into()), - source_window_title: None, - source_window_bounds_hash: None, - source_surface: SnapshotSurface::Window, - }, - scope: RefScope { - root_ref: None, - path_is_absolute: false, - path: smallvec::smallvec![0, 1], - }, - } +#[test] +fn only_a_known_role_equal_to_the_saved_role_matches() { + assert_eq!( + role_match(&LocatorField::Known("cell".into()), "cell"), + IdentityMatch::Match + ); + assert_eq!( + role_match(&LocatorField::Known("row".into()), "cell"), + IdentityMatch::NoMatch + ); } #[test] -fn bounds_stand_in_for_an_identityless_ref_but_never_replace_a_token() { - assert!(provisional_geometry_candidate(&bounds_only_entry(None))); - assert!(!provisional_geometry_candidate(&bounds_only_entry(Some( - "axtok1:0000000000000000" - )))); +fn an_unreadable_role_is_unknown_and_an_absent_role_is_a_mismatch() { + assert_eq!( + role_match(&LocatorField::Unknown, "cell"), + IdentityMatch::Unknown + ); + assert_eq!( + role_match(&LocatorField::Absent, "cell"), + IdentityMatch::NoMatch + ); } diff --git a/crates/macos/src/tree/resolve/resolve_tests.rs b/crates/macos/src/tree/resolve/resolve_tests.rs index 34c968efb..7cfd9d99e 100644 --- a/crates/macos/src/tree/resolve/resolve_tests.rs +++ b/crates/macos/src/tree/resolve/resolve_tests.rs @@ -2,12 +2,10 @@ use super::*; use crate::tree::AXElement; use crate::tree::resolve::resolve_classify::classify_candidates; use crate::tree::resolve::resolve_roots::{ - require_unique_window_number_match, source_scope_verified, source_window_number, - source_window_scope_required, + require_unique_window_number_match, source_window_number, source_window_scope_required, }; use crate::tree::resolve::resolve_search::{ - find_entry_by_path, incomplete_traversal_error, match_native_or_text_identity, - native_identifier_reused_by_different_role, walk_finite_path, + find_entry_by_path, incomplete_traversal_error, walk_finite_path, }; use agent_desktop_core::{ RefCapabilities, RefEntryIdentity, RefGeometry, RefProcess, RefScope, RefSource, @@ -32,6 +30,7 @@ pub(super) fn entry( value: None, description: None, native_id: None, + unique_at_capture: false, }, geometry: RefGeometry { bounds_space: None, @@ -48,6 +47,7 @@ pub(super) fn entry( source_window_title: source_window_title.map(String::from), source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: RefScope { root_ref: root_ref.map(String::from), @@ -58,27 +58,7 @@ pub(super) fn entry( } #[test] -fn path_fast_path_requires_a_hard_window_scope() { - assert!(!can_use_path_fast_path(&entry(None, None, None, None))); - assert!(can_use_path_fast_path(&entry( - None, - Some("w-10"), - None, - None - ))); - assert!(can_use_path_fast_path(&entry( - None, - None, - Some("Documents"), - None - ))); - let mut menu = entry(Some(42), Some("w-10"), None, None); - menu.source.source_surface = SnapshotSurface::Menu; - assert!(can_use_path_fast_path(&menu)); -} - -#[test] -fn locator_anchor_path_requires_scope_and_identity_or_geometry() { +fn locator_anchor_path_requires_scope_and_nothing_else() { let mut unscoped = entry(Some(42), None, None, None); assert!(!can_use_locator_anchor_path(&unscoped)); @@ -87,12 +67,12 @@ fn locator_anchor_path_requires_scope_and_identity_or_geometry() { unscoped.geometry.bounds_hash = None; unscoped.identity.name = None; - assert!(!can_use_locator_anchor_path(&unscoped)); + assert!(can_use_locator_anchor_path(&unscoped)); } #[test] fn drill_down_path_must_be_absolute() { - assert!(!can_use_path_fast_path(&entry( + assert!(!can_use_locator_anchor_path(&entry( Some(42), Some("w-10"), None, @@ -100,23 +80,7 @@ fn drill_down_path_must_be_absolute() { ))); let mut absolute = entry(Some(42), Some("w-10"), None, Some("@e1")); absolute.scope.path_is_absolute = true; - assert!(can_use_path_fast_path(&absolute)); -} - -#[test] -fn broad_search_requires_bounds_or_stable_identity() { - let mut blank = entry(None, None, None, None); - blank.identity.name = None; - - assert!(!can_use_broad_search(&blank)); - blank.identity.native_id = Some(agent_desktop_core::ElementIdentifier { - kind: agent_desktop_core::IdentifierKind::AxDomIdentifier, - value: "dom-id".into(), - }); - assert!(can_use_broad_search(&blank)); - blank.identity.native_id = None; - blank.identity.description = Some("Insert Text Box".into()); - assert!(can_use_broad_search(&blank)); + assert!(can_use_locator_anchor_path(&absolute)); } #[test] @@ -155,13 +119,6 @@ fn duplicate_ax_window_bridge_matches_are_ambiguous() { assert_eq!(error.details.unwrap()["candidate_count"], 2); } -#[test] -fn mutable_title_scope_never_claims_exact_verification() { - assert!(!source_scope_verified(None, true)); - assert!(!source_scope_verified(Some("w-42"), false)); - assert!(source_scope_verified(Some("w-42"), true)); -} - #[test] fn strict_resolution_requires_the_exact_process_instance() { let pid = i32::try_from(std::process::id()).expect("test pid fits macOS pid_t"); @@ -188,6 +145,27 @@ fn strict_resolution_requires_the_exact_process_instance() { ); } +#[test] +fn a_saved_ref_without_an_element_token_is_refused_not_searched_for() { + let pid = i32::try_from(std::process::id()).expect("test pid fits macOS pid_t"); + let instance = crate::system::process::process_identity::token_for_pid(pid) + .unwrap() + .expect("current process identity"); + let mut tokenless = entry(Some(42), Some("w-42"), None, None); + tokenless.process.pid = agent_desktop_core::ProcessId::try_from(pid).unwrap(); + tokenless.process.process_instance = Some(instance); + + let error = resolve_element_with_deadline( + &tokenless, + agent_desktop_core::Deadline::after(1_000).unwrap(), + ) + .err() + .expect("a tokenless ref must be refused"); + + assert_eq!(error.code, ErrorCode::ElementNotFound); + assert_eq!(error.details.unwrap()["kind"], "element_token_required"); +} + #[test] fn only_explicitly_retryable_incomplete_errors_retry() { let retryable = incomplete_traversal_error("children", 3); @@ -232,7 +210,6 @@ fn expired_deadline_fails_before_path_reads() { let error = find_entry_by_path( &[], &entry(Some(42), Some("w-42"), None, None), - true, &mut context, ) .err() @@ -274,14 +251,9 @@ fn preserved_native_path_replays_nested_target_in_one_read_per_level() { #[test] fn ambiguity_reports_bounded_candidate_summaries() { let candidates = (0..11).map(|_| AXElement(std::ptr::null_mut())).collect(); - let error = classify_candidates( - candidates, - &entry(None, Some("w-42"), None, None), - true, - Instant::now() + Duration::from_secs(1), - ) - .err() - .expect("duplicate candidates must be ambiguous"); + let error = classify_candidates(candidates, &entry(None, Some("w-42"), None, None)) + .err() + .expect("duplicate candidates must be ambiguous"); assert_eq!(error.code, ErrorCode::AmbiguousTarget); let details = error.details.unwrap(); @@ -295,91 +267,19 @@ fn unique_scoped_candidate_resolves_even_after_bounds_move() { let handle = classify_candidates( vec![AXElement(std::ptr::null_mut())], &entry(Some(42), Some("w-42"), None, None), - true, - Instant::now() + Duration::from_secs(1), ); assert!(handle.is_ok()); } #[test] -fn scoped_role_only_candidate_does_not_bypass_bounds_verification() { +fn a_role_only_candidate_resolves_for_core_to_confirm() { let mut weak = entry(None, Some("w-42"), None, None); weak.identity.name = None; - let error = classify_candidates( - vec![AXElement(std::ptr::null_mut())], - &weak, - true, - Instant::now() + Duration::from_secs(1), - ) - .err() - .expect("window scope alone must not identify a role-only candidate"); - - assert_eq!(error.code, ErrorCode::ElementNotFound); -} - -#[test] -fn authoritatively_absent_native_identifier_is_a_definitive_mismatch() { - use agent_desktop_core::{ - IdentifierEvidence, LocatorEvidence, LocatorField, LocatorRefEvidence, - }; - - let mut stored = entry(Some(42), Some("w-42"), None, None); - stored.identity.native_id = Some(agent_desktop_core::ElementIdentifier { - kind: agent_desktop_core::IdentifierKind::AxIdentifier, - value: "stable-id".into(), - }); - let live = LocatorEvidence { - role: LocatorField::Known(stored.identity.role.clone()), - name: LocatorField::Known("Investors".into()), - description: LocatorField::Absent, - value: LocatorField::Absent, - identifiers: IdentifierEvidence::absent(), - states: LocatorField::Absent, - ref_evidence: LocatorRefEvidence { - window_bounds: LocatorField::Absent, - bounds: LocatorField::Absent, - available_actions: LocatorField::Absent, - descriptors: Default::default(), - }, - }; + let handle = classify_candidates(vec![AXElement(std::ptr::null_mut())], &weak); - assert_eq!( - match_native_or_text_identity(&stored, &live), - agent_desktop_core::IdentityMatch::NoMatch - ); -} - -#[test] -fn native_identifier_reuse_by_a_different_role_is_definitively_stale() { - use agent_desktop_core::{ - ElementIdentifier, IdentifierEvidence, IdentifierKind, LocatorEvidence, LocatorField, - LocatorRefEvidence, - }; - - let mut stored = entry(None, Some("w-42"), None, None); - let identifier = ElementIdentifier { - kind: IdentifierKind::AxDomIdentifier, - value: "compose".into(), - }; - stored.identity.native_id = Some(identifier.clone()); - let live = LocatorEvidence { - role: LocatorField::Known("textfield".into()), - name: LocatorField::Absent, - description: LocatorField::Absent, - value: LocatorField::Absent, - identifiers: IdentifierEvidence::typed([identifier], Some(0), true), - states: LocatorField::Absent, - ref_evidence: LocatorRefEvidence { - window_bounds: LocatorField::Absent, - bounds: LocatorField::Absent, - available_actions: LocatorField::Absent, - descriptors: Default::default(), - }, - }; - - assert!(native_identifier_reused_by_different_role(&stored, &live)); + assert!(handle.is_ok()); } #[test] diff --git a/crates/macos/src/tree/resolve/resolve_tests_more.rs b/crates/macos/src/tree/resolve/resolve_tests_more.rs index ce46b8544..5a4dd6232 100644 --- a/crates/macos/src/tree/resolve/resolve_tests_more.rs +++ b/crates/macos/src/tree/resolve/resolve_tests_more.rs @@ -1,130 +1,5 @@ use super::tests::entry; use super::*; -use crate::tree::AXElement; -use crate::tree::resolve::resolve_classify::{ - BoundsMatchOutcome, classify_bounds_matches, classify_candidates, -}; -use crate::tree::resolve::resolve_search::{match_native_or_text_identity, should_stop_collecting}; - -#[test] -fn identifier_kind_mismatch_is_not_an_exact_match() { - use agent_desktop_core::{ - ElementIdentifier, IdentifierEvidence, IdentifierKind, LocatorEvidence, LocatorField, - LocatorRefEvidence, - }; - - let mut stored = entry(Some(42), Some("w-42"), None, None); - stored.identity.native_id = Some(ElementIdentifier { - kind: IdentifierKind::AxIdentifier, - value: "stable-id".into(), - }); - let live = LocatorEvidence { - role: LocatorField::Known(stored.identity.role.clone()), - name: LocatorField::Known("Investors".into()), - description: LocatorField::Absent, - value: LocatorField::Absent, - identifiers: IdentifierEvidence::typed( - [ElementIdentifier { - kind: IdentifierKind::AxDomIdentifier, - value: "stable-id".into(), - }], - Some(0), - true, - ), - states: LocatorField::Absent, - ref_evidence: LocatorRefEvidence { - window_bounds: LocatorField::Absent, - bounds: LocatorField::Absent, - available_actions: LocatorField::Absent, - descriptors: Default::default(), - }, - }; - - assert_eq!( - match_native_or_text_identity(&stored, &live), - agent_desktop_core::IdentityMatch::NoMatch - ); -} - -#[test] -fn duplicate_identity_candidates_are_stale_when_bounds_drift() { - let error = classify_candidates( - vec![ - AXElement(std::ptr::null_mut()), - AXElement(std::ptr::null_mut()), - ], - &entry(Some(42), Some("w-42"), None, None), - true, - Instant::now() + Duration::from_secs(1), - ) - .err() - .expect("bounds mismatch against every candidate must be stale, not ambiguous"); - - assert_eq!(error.code, ErrorCode::StaleRef); - let details = error.details.unwrap(); - assert_eq!(details["kind"], "bounds_mismatch"); - assert_eq!(details["candidate_count"], 2); -} - -#[test] -fn missing_bounds_hash_always_falls_through_to_ambiguous() { - assert_eq!( - classify_bounds_matches(0, false), - BoundsMatchOutcome::Ambiguous - ); - assert_eq!( - classify_bounds_matches(2, false), - BoundsMatchOutcome::Ambiguous - ); -} - -#[test] -fn zero_bounds_matches_is_stale_not_ambiguous() { - assert_eq!(classify_bounds_matches(0, true), BoundsMatchOutcome::Stale); -} - -#[test] -fn one_bounds_match_resolves_uniquely() { - assert_eq!( - classify_bounds_matches(1, true), - BoundsMatchOutcome::Resolved - ); -} - -#[test] -fn two_or_more_bounds_matches_remain_ambiguous() { - assert_eq!( - classify_bounds_matches(2, true), - BoundsMatchOutcome::Ambiguous - ); - assert_eq!( - classify_bounds_matches(38, true), - BoundsMatchOutcome::Ambiguous - ); -} - -#[test] -fn unscoped_candidate_requires_matching_bounds() { - let error = classify_candidates( - vec![AXElement(std::ptr::null_mut())], - &entry(Some(42), Some("w-42"), None, None), - false, - Instant::now() + Duration::from_secs(1), - ) - .err() - .expect("unscoped candidate without matching bounds must fail"); - - assert_eq!(error.code, ErrorCode::ElementNotFound); -} - -#[test] -fn bounds_search_keeps_collecting_for_disambiguation() { - assert!(!should_stop_collecting( - 2, - &entry(Some(42), None, None, None) - )); - assert!(should_stop_collecting(2, &entry(None, None, None, None))); -} #[test] fn ax_read_errors_distinguish_absent_unknown_and_permission_denied() { diff --git a/crates/macos/src/tree/resolve/retained.rs b/crates/macos/src/tree/resolve/retained.rs index 7018dd6da..7da4d5190 100644 --- a/crates/macos/src/tree/resolve/retained.rs +++ b/crates/macos/src/tree/resolve/retained.rs @@ -1,8 +1,5 @@ use crate::tree::AXElement; -use agent_desktop_core::{ - AdapterError, DeliverySemantics, ErrorCode, IdentifierEvidence, IdentityMatch, LocatorEvidence, - RefEntry, -}; +use agent_desktop_core::{AdapterError, IdentifierEvidence}; use core_foundation::{ base::TCFType, data::{CFData, CFDataRef}, @@ -77,41 +74,9 @@ pub(crate) fn evidence(element: &AXElement, identifiers: IdentifierEvidence) -> } } -/// Refuses a saved ref that carries a token this build cannot resolve at all. -pub(crate) fn validate(token: Option<&str>) -> Result<(), AdapterError> { - match token { - Some(token) if !token.starts_with(TOKEN_PREFIX) => Err(unsupported_token()), - _ => Ok(()), - } -} - -/// Compares the token a candidate carries in `evidence` with the entry's token. -/// The candidate's own token is already in the evidence its read produced. -pub(crate) fn candidate_match( - _element: &AXElement, - entry: &RefEntry, - evidence: &LocatorEvidence, -) -> Result { - let Some(token) = entry.identity.retained_object.as_deref() else { - return Ok(super::resolve_search::match_native_or_text_identity( - entry, evidence, - )); - }; - validate(Some(token))?; - Ok(match evidence.identifiers.retained_object() { - Some(actual) if actual == token => IdentityMatch::Match, - Some(_) => IdentityMatch::NoMatch, - None => super::resolve_search::match_native_or_text_identity(entry, evidence), - }) -} - -fn unsupported_token() -> AdapterError { - AdapterError::new( - ErrorCode::StaleRef, - "The saved ref carries a native identity this build cannot resolve", - ) - .with_suggestion("Run 'snapshot' to refresh, then retry with the updated ref.") - .with_disposition(DeliverySemantics::not_delivered()) +/// Whether this macOS build can create and resolve element tokens. +pub(crate) fn tokens_available() -> bool { + symbols().is_some() } fn symbols() -> Option<&'static Symbols> { @@ -167,7 +132,3 @@ unsafe extern "C" { fn dlsym(handle: *mut c_void, symbol: *const std::ffi::c_char) -> *mut c_void; fn dlopen(path: *const std::ffi::c_char, mode: i32) -> *mut c_void; } - -#[cfg(all(test, target_os = "macos"))] -#[path = "retained_tests.rs"] -mod tests; diff --git a/crates/macos/src/tree/resolve/retained_resolve_tests.rs b/crates/macos/src/tree/resolve/retained_resolve_tests.rs index 79c4f16b2..6216e3af5 100644 --- a/crates/macos/src/tree/resolve/retained_resolve_tests.rs +++ b/crates/macos/src/tree/resolve/retained_resolve_tests.rs @@ -14,6 +14,7 @@ fn entry(role: &str) -> RefEntry { process_instance: Some("test-instance".into()), }, identity: RefEntryIdentity { + unique_at_capture: false, retained_object: None, role: role.into(), name: Some("Investors".into()), @@ -36,6 +37,7 @@ fn entry(role: &str) -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: RefScope { root_ref: None, diff --git a/crates/macos/src/tree/resolve/retained_tests.rs b/crates/macos/src/tree/resolve/retained_tests.rs deleted file mode 100644 index 91acca5ed..000000000 --- a/crates/macos/src/tree/resolve/retained_tests.rs +++ /dev/null @@ -1,85 +0,0 @@ -use super::*; -use agent_desktop_core::{LocatorField, LocatorRefEvidence, SnapshotSurface}; - -fn entry(token: Option<&str>) -> RefEntry { - RefEntry { - process: agent_desktop_core::RefProcess { - pid: agent_desktop_core::ProcessId::new(1), - process_instance: None, - }, - identity: agent_desktop_core::RefEntryIdentity { - retained_object: token.map(str::to_owned), - role: "button".into(), - name: Some("Target".into()), - value: None, - description: None, - native_id: None, - }, - geometry: agent_desktop_core::RefGeometry { - bounds_space: None, - bounds: None, - bounds_hash: None, - }, - capabilities: agent_desktop_core::RefCapabilities { - states: Vec::new(), - available_actions: Vec::new(), - }, - source: agent_desktop_core::RefSource { - source_app: None, - source_window_id: None, - source_window_title: None, - source_window_bounds_hash: None, - source_surface: SnapshotSurface::Window, - }, - scope: agent_desktop_core::RefScope { - root_ref: None, - path_is_absolute: false, - path: Default::default(), - }, - } -} - -fn candidate(token: Option<&str>, name: &str) -> LocatorEvidence { - LocatorEvidence { - role: LocatorField::Known("button".into()), - name: LocatorField::Known(name.into()), - description: LocatorField::Absent, - value: LocatorField::Absent, - identifiers: token.map_or_else(IdentifierEvidence::absent, |token| { - IdentifierEvidence::absent().with_retained_object(token.into()) - }), - states: LocatorField::Absent, - ref_evidence: LocatorRefEvidence { - window_bounds: LocatorField::Absent, - bounds: LocatorField::Absent, - available_actions: LocatorField::Absent, - descriptors: Default::default(), - }, - } -} - -#[test] -fn a_candidate_matches_only_its_own_token() { - let element = AXElement(std::ptr::null_mut()); - let saved = entry(Some("axtok1:00ff")); - let match_of = |saved: &RefEntry, token: Option<&str>| { - candidate_match(&element, saved, &candidate(token, "Target")) - }; - - assert_eq!( - match_of(&saved, Some("axtok1:00ff")).unwrap(), - IdentityMatch::Match - ); - assert_eq!( - match_of(&saved, Some("axtok1:00fe")).unwrap(), - IdentityMatch::NoMatch - ); - assert_eq!(match_of(&saved, None).unwrap(), IdentityMatch::Unknown); - assert_eq!(match_of(&entry(None), None).unwrap(), IdentityMatch::Match); - assert_eq!( - match_of(&entry(Some("host:1")), Some("axtok1:00ff")) - .unwrap_err() - .code, - ErrorCode::StaleRef - ); -} diff --git a/crates/windows/src/adapter.rs b/crates/windows/src/adapter.rs index 418d1c5aa..dae424b7f 100644 --- a/crates/windows/src/adapter.rs +++ b/crates/windows/src/adapter.rs @@ -80,16 +80,6 @@ impl ObservationOps for WindowsAdapter { .into_accessibility_tree() } - /// Re-resolves a stored ref to a live element, fail-closed - the - /// drill-down root `snapshot --root @ref` needs. - fn resolve_element_strict( - &self, - entry: &RefEntry, - deadline: Deadline, - ) -> Result { - crate::tree::resolve::resolve_element_strict(entry, deadline) - } - fn resolve_locator_anchor( &self, entry: &RefEntry, diff --git a/crates/windows/src/tree/chromium.rs b/crates/windows/src/tree/chromium.rs index 6c5af64a4..08a7351c0 100644 --- a/crates/windows/src/tree/chromium.rs +++ b/crates/windows/src/tree/chromium.rs @@ -210,6 +210,7 @@ mod tests { value: None, description: None, native_id: None, + unique_at_capture: false, }, geometry: agent_desktop_core::RefGeometry { bounds_space: None, @@ -226,6 +227,7 @@ mod tests { source_window_title: None, source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: agent_desktop_core::RefScope { root_ref: None, diff --git a/crates/windows/src/tree/element.rs b/crates/windows/src/tree/element.rs index 473e0cc21..f6cec1b61 100644 --- a/crates/windows/src/tree/element.rs +++ b/crates/windows/src/tree/element.rs @@ -77,7 +77,7 @@ impl UIAElement { } /// The verified process identity, if this element resolved through the - /// strict resolver. + /// locator anchor. pub(crate) fn verified_process(&self) -> Option<(u32, &str)> { self.1 .as_ref() diff --git a/crates/windows/src/tree/hit_test/envelope_tests.rs b/crates/windows/src/tree/hit_test/envelope_tests.rs index 53361d6e5..41dd1e2f6 100644 --- a/crates/windows/src/tree/hit_test/envelope_tests.rs +++ b/crates/windows/src/tree/hit_test/envelope_tests.rs @@ -8,6 +8,8 @@ use agent_desktop_core::{ }; use std::sync::atomic::{AtomicU32, Ordering}; +const TOKEN: &str = "axtok1:00"; + struct EnvelopeAdapter { live: LiveElement, hit: HitTestResult, @@ -119,7 +121,6 @@ impl ActionOps for EnvelopeAdapter { } impl InputOps for EnvelopeAdapter {} - impl SystemOps for EnvelopeAdapter { fn acquire_interaction_lease( &self, @@ -150,7 +151,7 @@ fn live_button(enabled: bool, bounds: Rect) -> LiveElement { identity: LiveIdentity { name: LocatorField::Known("Run".into()), description: LocatorField::Absent, - identifiers: IdentifierEvidence::absent(), + identifiers: IdentifierEvidence::absent().with_retained_object(TOKEN.into()), }, state: ElementState { role: "button".into(), @@ -179,12 +180,10 @@ fn entry_for(live: &LiveElement) -> RefEntry { process_instance: Some("test-instance".into()), }, identity: RefEntryIdentity { - retained_object: None, + retained_object: Some(TOKEN.into()), role: live.state.role.clone(), name: Some("Run".into()), - value: None, - description: None, - native_id: None, + ..Default::default() }, geometry: RefGeometry { bounds_space: None, @@ -201,6 +200,7 @@ fn entry_for(live: &LiveElement) -> RefEntry { source_window_title: Some("Fixture".into()), source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: RefScope { root_ref: None, diff --git a/crates/windows/src/tree/live_read_tests.rs b/crates/windows/src/tree/live_read_tests.rs index 8492555e9..a364601bb 100644 --- a/crates/windows/src/tree/live_read_tests.rs +++ b/crates/windows/src/tree/live_read_tests.rs @@ -89,8 +89,8 @@ fn the_secure_control_s_live_value_is_withheld_at_the_reader_path() { let fixture = HostedFixture::spawn().expect("a fixture host starts"); let entry = blank_secure_entry(&fixture); - let handle = crate::tree::resolve::resolve_element_strict(&entry, deadline()) - .expect("the blank secure ref resolves through path and geometry"); + let handle = crate::tree::resolve::resolve_anchor::resolve_locator_anchor(&entry, deadline()) + .expect("the blank secure ref resolves through the locator anchor"); let read = read_live_element(&handle, deadline()).expect("the shared read succeeds"); let value = live_value(&read); assert_eq!( @@ -248,6 +248,7 @@ fn blank_secure_entry(fixture: &HostedFixture) -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: false, }, geometry: agent_desktop_core::RefGeometry { bounds_space: None, @@ -264,6 +265,7 @@ fn blank_secure_entry(fixture: &HostedFixture) -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: agent_desktop_core::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: agent_desktop_core::RefScope { root_ref: None, @@ -297,12 +299,12 @@ fn find_secure( path.extend_from_slice(prefix); return Ok(Some((path, properties, node_evidence, failed))); } - let mut ignored = false; - let children = crate::tree::resolve::resolve_search::enumerate_children( + let children = crate::tree::resolve::resolve_search::descent::read_children( source, element, budget, - &mut ignored, + &crate::tree::resolve::resolve_anchor::ANCHOR_DESCENT, + None, )?; for (index, child) in children.iter().enumerate() { prefix.push(index); diff --git a/crates/windows/src/tree/resolve/mod.rs b/crates/windows/src/tree/resolve/mod.rs index 123d6fdd7..29dddd8b9 100644 --- a/crates/windows/src/tree/resolve/mod.rs +++ b/crates/windows/src/tree/resolve/mod.rs @@ -6,215 +6,9 @@ use super::resolve::resolve_pacing::{FIRST_RETRY_PAUSE, next_retry_pause, sleep_ #[cfg(target_os = "windows")] use super::element::UIAElement; #[cfg(target_os = "windows")] -use super::resolve::resolve_chrome_repair::repair_past_leading_chrome; -#[cfg(all(test, target_os = "windows"))] -use super::resolve::resolve_match::CandidateOutcome; -use super::resolve::resolve_match::stale_ref_error; -#[cfg(target_os = "windows")] -use super::resolve::resolve_match::{ambiguous_target_error, owning_process_exited_error}; -#[cfg(target_os = "windows")] -use super::resolve::resolve_search::descent::PathLanding; -#[cfg(target_os = "windows")] -use super::resolve::resolve_search::{ - SearchContext, accept_path_landing, can_use_path_fast_path, entry_is_unverifiable, - identity_unknown_error, resolve_walk_budget, search_under, should_stop_collecting, - walk_stored_path, -}; -#[cfg(all(test, target_os = "windows"))] -use super::walk::walker::{TreeSource, WalkBudget}; -#[cfg(target_os = "windows")] -use super::walk::walker_source::UiaTreeSource; - -/// Resolves a stored ref to its live element, fail-closed and three-state. -/// -/// The search descends from the stored window's root to a resolve-scoped -/// depth, reading each candidate with the same composition the walk uses -/// (`UiaTreeSource::evidence`), gates on role, and runs core's composed -/// identity rule (`resolve_match::candidate_outcome`). Then: -/// -/// - zero candidates and every decision was readable -> `STALE_REF`, settled -/// - zero candidates but one was **unreadable** -> incomplete-and-retryable -/// (`AppUnresponsive`, the three-state discipline: an `Unknown` verdict is -/// never a `NoMatch`) -/// - two or more candidates that all match -> `AMBIGUOUS_TARGET` -/// - exactly one -> a `NativeHandle` wrapping the live element -/// -/// Anything short of an exact match fails closed rather than guessing, -/// because A7-3 measured Explorer re-resolving 29 of 29 `AutomationId` keys -/// with 5 landing on a different element - the silent-wrong-target shape -/// strictness exists to prevent. -/// -/// The whole resolution runs through a `deadline`-bounded retry loop -/// that retries only what is genuinely incomplete - an `AppUnresponsive` -/// error stamped explicitly retryable (an unreadable candidate, a vanished -/// or transient node mid-descent). A settled answer - `STALE_REF` from a -/// completed search, `AMBIGUOUS_TARGET`, a permission denial - is never -/// retried. Every re-attempt re-verifies process liveness through -/// `resolve_window_root` (A14-4's prescribed cure), so a dead process -/// converts to settled `STALE_REF` on the next attempt instead of burning -/// the deadline, and final expiry stamps `deadline_elapsed` onto the last -/// incomplete diagnosis rather than a bare `TIMEOUT` that discards it. -#[cfg(target_os = "windows")] -pub(crate) fn resolve_element_strict( - entry: &RefEntry, - deadline: Deadline, -) -> Result { - retry_incomplete_until(deadline, || resolve_attempt(entry, deadline)) -} - -/// One resolution attempt: the path tier, then the broad search, then the -/// verdict. -/// -/// The two tiers report what they left unread separately and keep it separate -/// all the way to [`attempt_verdict`], because a landing and a region are -/// different claims. The path tier's landing is decided by -/// [`accept_path_landing`], which is handed the landed element alone; the -/// region each tier left unread only ever bears on the *negative* verdict, and -/// is composed once, at the end, by the function that settles it. -#[cfg(target_os = "windows")] -fn resolve_attempt(entry: &RefEntry, deadline: Deadline) -> Result { - if entry_is_unverifiable(entry) { - return Err(stale_ref_error(entry)); - } - let root = resolve_window_root(entry, deadline)?; - let source = UiaTreeSource::for_root(&root)?; - let prepared = source.prepare_root(&root)?; - - let budget = resolve_walk_budget(deadline); - - let path = if can_use_path_fast_path(entry) { - walk_stored_path(&source, &prepared, &entry.scope.path, &budget)? - } else { - PathLanding::not_walked() - }; - if let Some(accepted) = path - .element - .as_ref() - .and_then(|candidate| accept_path_landing(&source, candidate, entry)) - .or_else(|| repair_past_leading_chrome(&source, &prepared, entry, &budget)) - { - return Ok(into_verified_handle(accepted, entry)); - } - - let mut searched = Vec::new(); - let mut search_unread_region = false; - let search_ctx = SearchContext { - source: &source, - entry, - budget: &budget, - }; - search_under( - &search_ctx, - &prepared, - 0, - &mut searched, - &mut search_unread_region, - )?; - - let candidate_hashes: Vec> = searched - .iter() - .map(|candidate| candidate.bounds_hash) - .collect(); - match attempt_verdict(&candidate_hashes, &path, search_unread_region, entry) { - SearchVerdict::Incomplete => Err(identity_unknown_error(entry)), - SearchVerdict::Stale => Err(stale_ref_error(entry)), - SearchVerdict::Resolved(index) => { - let Some(candidate) = searched.into_iter().nth(index) else { - return Err(stale_ref_error(entry)); - }; - Ok(into_verified_handle(candidate.element, entry)) - } - SearchVerdict::Ambiguous => Err(ambiguous_target_error(entry, searched.len())), - } -} - -/// What the completed (or early-stopped) broad search decided, before a -/// resolved candidate is wired into a live handle. -#[cfg(target_os = "windows")] -enum SearchVerdict { - /// A candidate or an unrelated region could not be read and the - /// verdict is not yet settled - the caller retries. - Incomplete, - /// No candidate matched and nothing was left unread. - Stale, - /// Exactly one candidate settles the match; its index into the - /// searched list. - Resolved(usize), - /// Two or more candidates match and no stored hash disambiguates them. - Ambiguous, -} - -/// Composes every tier's unread region into the one question -/// [`classify_search`] asks - did this attempt read enough of the tree to -/// settle a negative verdict - and classifies the collected candidates -/// against it. -/// -/// Takes the path tier's landing itself rather than a bare flag, so the tier's -/// own report is the only thing that can answer for it. A gap met while -/// walking the stored path is a region this attempt did not read, exactly as -/// one met in the broad search is, and the path walk is the only tier that -/// descends past the search's own depth cap - so a gap there can be the sole -/// signal that the tier able to reach the element never got to look. An -/// attempt that dropped it would settle `STALE_REF` - "the element is gone" - -/// off a walk that never finished. -/// -/// The landing's own `element` is deliberately unread here. Whether the path -/// found something is settled before this point, by -/// [`super::resolve_search::accept_path_landing`]; what survives to the -/// verdict is only what the walk did not see. -#[cfg(target_os = "windows")] -fn attempt_verdict( - candidate_hashes: &[Option], - path: &PathLanding, - search_unread_region: bool, - entry: &RefEntry, -) -> SearchVerdict { - classify_search( - candidate_hashes, - path.unread_region || search_unread_region, - entry, - ) -} - -/// Composes the searched candidates' tie-break hashes with the unread-region -/// flag into a verdict, mirroring macOS's ordering -/// (`crates/macos/src/tree/resolve/resolve_search.rs:43-58`) even though nothing on -/// Windows stops the search itself early: `search_under` always walks the -/// whole depth-bounded subtree, and `should_stop_collecting` is consulted -/// only here, post-hoc, after collection is done. Two or more matches with -/// no stored hash to disambiguate is checked first, because no unread region -/// could turn an already-conclusive ambiguity into anything else. Only below -/// that threshold does an incomplete region withhold the verdict, so a sole -/// match found while another region of the tree stayed unreadable is -/// retried rather than resolved, and a multi-candidate tie-break with a -/// stored hash still waits on the full picture before it classifies. -/// Because the walk itself never stops early, `ambiguous_target_error`'s -/// candidate count is always the full count found across the entire -/// searched subtree, never a partial one. +use super::resolve::resolve_match::owning_process_exited_error; #[cfg(target_os = "windows")] -fn classify_search( - candidate_hashes: &[Option], - unread_region: bool, - entry: &RefEntry, -) -> SearchVerdict { - let stop_collecting = should_stop_collecting(candidate_hashes.len(), entry); - if unread_region && !stop_collecting { - return SearchVerdict::Incomplete; - } - match candidate_hashes.len() { - 0 => SearchVerdict::Stale, - 1 => SearchVerdict::Resolved(0), - _ => match super::resolve::resolve_match::select_by_bounds_hash( - candidate_hashes, - entry.geometry.bounds_hash, - ) { - super::resolve::resolve_match::Selection::Resolved(index) => { - SearchVerdict::Resolved(index) - } - super::resolve::resolve_match::Selection::Ambiguous => SearchVerdict::Ambiguous, - }, - } -} +use super::resolve::resolve_match::stale_ref_error; /// Wraps the resolved element with the ref's verified process identity, so the /// live-read path can corroborate that the provider is still the one @@ -285,18 +79,6 @@ pub(crate) fn is_retryable_resolution_error(error: &AdapterError) -> bool { #[cfg(target_os = "windows")] pub(crate) use agent_desktop_core::refs::resolve_errors::mark_deadline_elapsed; -/// The non-Windows twin. The crate cross-compiles to the Linux lane with the -/// resolver reachable, but there are no UI Automation elements there, so every -/// stored ref fails closed as stale rather than attempting a search that -/// cannot find anything. -#[cfg(not(target_os = "windows"))] -pub(crate) fn resolve_element_strict( - entry: &RefEntry, - _deadline: Deadline, -) -> Result { - Err(stale_ref_error(entry)) -} - /// Reaches the stored window's root element from the ref's source window id. /// /// The fail-closed window gate (the A7-3 wrong-target shape exists to @@ -376,15 +158,10 @@ fn verify_stored_failure_error(entry: &RefEntry, deadline: Deadline) -> AdapterE } pub(crate) mod resolve_anchor; -#[cfg(target_os = "windows")] -pub(crate) mod resolve_chrome_repair; pub(crate) mod resolve_match; pub(crate) mod resolve_pacing; pub(crate) mod resolve_search; + #[cfg(all(test, target_os = "windows"))] #[path = "resolve_tests.rs"] mod windows_only; - -#[cfg(all(test, target_os = "windows"))] -#[path = "resolve_pair_window.rs"] -mod pair_window; diff --git a/crates/windows/src/tree/resolve/resolve_ambiguity_tests.rs b/crates/windows/src/tree/resolve/resolve_ambiguity_tests.rs deleted file mode 100644 index ca89e32ce..000000000 --- a/crates/windows/src/tree/resolve/resolve_ambiguity_tests.rs +++ /dev/null @@ -1,140 +0,0 @@ -use crate::tree::resolve::*; - -use crate::tree::resolve::pair_window::{ - APP_MARKER, DuplicatePairWindow, NAME_MARKER, PairGeometry, TITLE_MARKER, automation_id, - collect_marked, -}; - -/// Fails if any marker survived into any operator-visible slot of the error. -fn assert_redacted(error: &AdapterError, secret: &str, slot: &str) { - let rendered = format!( - "{} | {} | {} | {}", - error.message, - error.suggestion.clone().unwrap_or_default(), - error.platform_detail.clone().unwrap_or_default(), - error - .details - .as_ref() - .map(ToString::to_string) - .unwrap_or_default(), - ); - assert!( - !rendered.to_lowercase().contains(&secret.to_lowercase()), - "the {slot} marker reached an ambiguity error: {rendered}" - ); -} - -/// The `AMBIGUOUS_TARGET` branch, driven end to end against two live -/// controls no tier can separate, with the error's redaction guarantee -/// asserted rather than eyeballed. -/// -/// The pair shares role, name, `AutomationId` and rectangle, so the identity -/// tiers both match and the bounds tie-break is handed two equal hashes plus -/// the stored one - the arm that must stay ambiguous rather than pick a -/// candidate, since picking one is the silent-wrong-target shape A7-3 -/// measured. The second half pins a deliberate divergence from macOS, which -/// embeds entry name, description and window title in its ambiguous details: -/// here the resolver's answer carries shape only - kind and candidate count - -/// with no application-derived text in the message, the suggestion, the -/// platform detail or the details. -#[test] -fn duplicate_evidence_candidates_settle_ambiguous_target_carrying_no_application_text() { - crate::tree::fixture::ensure_test_apartment(); - let hosted = - DuplicatePairWindow::create(PairGeometry::Coincident).expect("a duplicate window hosts"); - let deadline = crate::tree::walk::walker_fake::deadline(); - let root = crate::tree::automation::root_from_hwnd(hosted.handle, deadline) - .expect("the hosted window resolves"); - let source = UiaTreeSource::for_root(&root).expect("a tree source"); - let prepared = source.prepare_root(&root).expect("a prepared root"); - let budget = WalkBudget::new(10, deadline); - - let mut duplicates = Vec::new(); - collect_marked(&source, &prepared, &budget, &mut duplicates); - assert_eq!( - duplicates.len(), - 2, - "the host must present exactly two marked controls" - ); - let (first, second) = (&duplicates[0], &duplicates[1]); - let role = first.role.known().cloned().expect("a read role"); - let name = first.name.known().cloned().expect("a read name"); - let native_id = automation_id(first); - let hash = crate::tree::resolve::resolve_match::bounds_hash_of(first); - assert_eq!(second.role.known(), Some(&role), "the roles are equal"); - assert_eq!(second.name.known(), Some(&name), "the names are equal"); - assert_eq!( - automation_id(second), - native_id, - "the identifiers are equal" - ); - assert_eq!( - crate::tree::resolve::resolve_match::bounds_hash_of(second), - hash, - "the bounds hashes are equal, so the tie-break cannot separate them" - ); - let hash = hash.expect("a positive-area bounds hash"); - let rect = *first - .ref_evidence - .bounds - .known() - .expect("a read bounds rectangle"); - - let pid = agent_desktop_core::ProcessId::from(std::process::id()); - let mut entry = crate::tree::walk::walker_fake::ref_entry(&role); - entry.process = agent_desktop_core::RefProcess { - pid, - process_instance: Some( - crate::system::process_identity::token_for_pid(pid) - .expect("the token read answers") - .expect("a live process has a token"), - ), - }; - entry.identity.name = Some(name); - entry.identity.native_id = native_id; - entry.geometry = agent_desktop_core::RefGeometry { - bounds_space: None, - bounds: Some(rect), - bounds_hash: Some(hash), - }; - entry.source.source_app = Some(APP_MARKER.into()); - entry.source.source_window_id = Some(format!("w-{}", hosted.handle)); - entry.source.source_window_title = Some(TITLE_MARKER.into()); - - for evidence in &duplicates { - assert_eq!( - crate::tree::resolve::resolve_match::candidate_outcome(&entry, evidence), - CandidateOutcome::Matched, - "both live controls satisfy the identity tiers" - ); - } - - let error = resolve_element_strict(&entry, deadline) - .err() - .expect("two indistinguishable candidates must not resolve to either one"); - - assert_eq!( - error.code, - ErrorCode::AmbiguousTarget, - "the resolver settles ambiguity instead of guessing: {error:?}" - ); - assert_eq!( - error - .details - .as_ref() - .and_then(|details| details.get("candidate_count")) - .and_then(serde_json::Value::as_u64), - Some(2), - "the shape-only details report how many candidates matched" - ); - assert_redacted(&error, NAME_MARKER, "control name"); - assert_redacted(&error, TITLE_MARKER, "window title"); - assert_redacted(&error, APP_MARKER, "application file name"); - assert_redacted( - &error, - &automation_id(first) - .expect("the shared control id surfaces as an automation id") - .value, - "automation id", - ); -} diff --git a/crates/windows/src/tree/resolve/resolve_anchor.rs b/crates/windows/src/tree/resolve/resolve_anchor.rs index 3990eff15..25721f902 100644 --- a/crates/windows/src/tree/resolve/resolve_anchor.rs +++ b/crates/windows/src/tree/resolve/resolve_anchor.rs @@ -1,18 +1,20 @@ -//! The locator-anchor resolver: selected-match hydration's path-only -//! variant of `resolve_element_strict`. +//! The locator-anchor resolver: the path-only resolver core's selected-match +//! hydration and stored-ref resolution both land on. //! //! Core's `live_locator::hydrate` re-observes the matched subtree from the //! return handle and enforces evidence completeness itself; Windows only has to //! land the anchor. The anchor's path is exact from the just-walked tree, so -//! there is **no broad-search fallback** - and its classification is the -//! inverse of the strict search's: every step is descent along a stored path -//! that churn makes permanently wrong, so a path step that lands nowhere, a +//! there is **no broad-search fallback**: every step is descent along a stored +//! path that churn makes permanently wrong, so a path step that lands nowhere, a //! role-refuted candidate, or a vanished node **settles the attempt //! immediately** (a completed-search `STALE_REF`, one attempt, the deadline //! intact for core's fresh re-observation). Only the unresponsive/transport //! class runs through the bounded retry loop, which re-walks the same cheap //! O(depth) path rather than replaying an expensive search. //! +//! The landing is not re-identified here: the recorded identity and geometry are +//! core's to compare against, on the same handle this returns. +//! //! An expired deadline is the one failure that must not settle. There was no //! attempt left to settle: a path walk cut short by the deadline lands short //! of the stored index, and reporting that as a miss would tell core the @@ -21,15 +23,10 @@ use agent_desktop_core::{AdapterError, Deadline, NativeHandle, RefEntry}; -#[cfg(target_os = "windows")] -use super::resolve_match::CandidateOutcome; use super::resolve_match::stale_ref_error; -use super::resolve_search::descent::{DescentPolicy, DescentVerdict, ExpiryPolicy}; +use super::resolve_search::descent::{DescentPolicy, DescentVerdict}; #[cfg(target_os = "windows")] -use super::resolve_search::{ - entry_is_locatable, geometry_matches, identity_unknown_error, provisional_geometry_candidate, - resolve_walk_budget, -}; +use super::resolve_search::{identity_unknown_error, resolve_walk_budget, window_rooted}; use crate::system::runtime::hresult::ReadDisposition; use crate::tree::automation::{UiaFailure, uia_failure_disposition}; #[cfg(target_os = "windows")] @@ -58,22 +55,22 @@ pub(crate) fn resolve_locator_anchor( Err(stale_ref_error(entry)) } -/// The pure entry gate runs first, before any of `resolve_window_root`, +/// The one pure gate runs first, before any of `resolve_window_root`, /// `UiaTreeSource::for_root` or `prepare_root` - three cross-process calls a -/// verdict `entry_is_locatable` can already settle from the stored ref alone -/// should never pay for. `resolve.rs`'s `resolve_attempt` runs its own -/// equivalent gate (`entry_is_unverifiable`) in the same first position. +/// verdict the stored ref alone can settle should never pay for. /// -/// The anchor reuses `entry_is_locatable` directly rather than the fast -/// path's `can_use_path_fast_path`: an empty path is a valid anchor - the -/// root element itself can be the stored target - so the anchor omits the -/// fast path's non-empty-path conjunct. +/// That gate is the path's scope: a path recorded relative to a drill-down root +/// is not one the window's own root can descend, and an empty path is a valid +/// anchor because the root element itself can be the stored target. Everything +/// about the element itself - name, identifier, bounds - belongs to core's +/// full-predicate check on the landing, so an element that carries none of them +/// still lands. #[cfg(target_os = "windows")] fn resolve_locator_anchor_once( entry: &RefEntry, deadline: Deadline, ) -> Result { - if !entry_is_locatable(entry) { + if !window_rooted(entry) { return Err(stale_ref_error(entry)); } @@ -89,71 +86,12 @@ fn resolve_locator_anchor_once( &budget, &ANCHOR_DESCENT, )?; - let Some(candidate) = landing.element else { + let Some(candidate) = landing else { return Err(stale_ref_error(entry)); }; let (_, evidence, _) = source.evidence(&candidate); anchor_role_verdict(entry, &evidence)?; - match anchor_geometry_verdict(entry, &evidence) { - AnchorGeometryVerdict::Promote => { - Ok(crate::tree::resolve::into_verified_handle(candidate, entry)) - } - AnchorGeometryVerdict::SettleStale => Err(stale_ref_error(entry)), - AnchorGeometryVerdict::Retry => Err(identity_unknown_error(entry)), - } -} - -/// The three-state verdict `candidate_outcome`'s `Incomplete` answer resolves -/// to once the geometry tier has had its say. -#[cfg(target_os = "windows")] -enum AnchorGeometryVerdict { - /// The candidate is the stored element. - Promote, - /// The candidate is definitively not the stored element: either the - /// identity tier refuted it, or the geometry tier ran its comparison and - /// the hashes disagreed. - SettleStale, - /// A field the matcher needed could not be read; this is the only - /// verdict the caller's retry loop should replay. - Retry, -} - -/// Composes `candidate_outcome` with the geometry promotion tier into the -/// anchor's settle/retry decision. -/// -/// A geometry mismatch is only ever a genuinely transient read when the -/// live bounds themselves came back [`agent_desktop_core::LocatorField::Unknown`] -/// (the provider could not answer this attempt); a bounds read that -/// succeeded and simply disagreed with the stored hash is a definitive -/// answer and settles, the same as a refuted identity tier does. -#[cfg(target_os = "windows")] -fn anchor_geometry_verdict( - entry: &RefEntry, - evidence: &agent_desktop_core::LocatorEvidence, -) -> AnchorGeometryVerdict { - match super::resolve_match::candidate_outcome(entry, evidence) { - CandidateOutcome::Matched => AnchorGeometryVerdict::Promote, - CandidateOutcome::Incomplete if geometry_matches(entry, evidence) => { - AnchorGeometryVerdict::Promote - } - CandidateOutcome::Incomplete if geometry_comparison_settled(entry, evidence) => { - AnchorGeometryVerdict::SettleStale - } - CandidateOutcome::Incomplete => AnchorGeometryVerdict::Retry, - CandidateOutcome::Refuted => AnchorGeometryVerdict::SettleStale, - } -} - -/// Whether a failed geometry promotion is a settled mismatch rather than a -/// transient read: the entry must be eligible for the geometry tier at all -/// (`provisional_geometry_candidate`), and the candidate's live bounds must -/// have actually been read rather than come back unreadable this attempt. -#[cfg(target_os = "windows")] -fn geometry_comparison_settled( - entry: &RefEntry, - evidence: &agent_desktop_core::LocatorEvidence, -) -> bool { - provisional_geometry_candidate(entry) && !evidence.ref_evidence.bounds.is_unknown() + Ok(crate::tree::resolve::into_verified_handle(candidate, entry)) } /// The anchor's role guard: an [`agent_desktop_core::LocatorField::Unknown`] @@ -195,7 +133,6 @@ const ANCHOR_CONTEXT: &str = "walk a locator anchor's path"; /// to trigger. pub(crate) const ANCHOR_DESCENT: DescentPolicy = DescentPolicy { classify: anchor_descent_verdict, - on_expiry: ExpiryPolicy::Surface, descend_context: ANCHOR_CONTEXT, sibling_context: ANCHOR_CONTEXT, }; diff --git a/crates/windows/src/tree/resolve/resolve_anchor_tests.rs b/crates/windows/src/tree/resolve/resolve_anchor_tests.rs index b534f1bf8..1edbe6813 100644 --- a/crates/windows/src/tree/resolve/resolve_anchor_tests.rs +++ b/crates/windows/src/tree/resolve/resolve_anchor_tests.rs @@ -1,10 +1,11 @@ use super::*; use crate::tree::element::UIAElement; use crate::tree::fixture::{HostedFixture, ensure_test_apartment}; +use crate::tree::resolve::resolve_search::descent::descend_path; use crate::tree::walk::walker_fake::{deadline, ref_entry, scan_subtree}; use agent_desktop_core::{ ErrorCode, IdentifierEvidence, LocatorEvidence, LocatorField, LocatorRefEvidence, - NodeDescriptor, ProcessId, Rect, RefEntry, + NodeDescriptor, ProcessId, RefEntry, }; use std::ops::ControlFlow; @@ -90,7 +91,52 @@ fn an_exact_anchor_path_resolves_on_the_unchanged_fixture() { let entry = blank_entry_with_path(&fixture, Vec::new()); let handle = resolve_locator_anchor(&entry, deadline()) .expect("the exact anchor path resolves the secure element"); - assert!(handle.downcast_ref::().is_some()); + assert_verified_stamp(&handle, entry.process.pid.get(), &verified_token(&entry)); +} + +/// The anchor re-identifies nothing: the path and the landing's role are all it +/// checks, and core compares the recorded predicate against the handle the +/// anchor returns. An element that reports no name, no identifier and no usable +/// bounds - the one unnamed textfield a complete observation can still prove +/// unique - therefore has to land instead of being refused before the walk. +#[test] +fn an_entry_with_no_identity_and_no_usable_geometry_still_lands() { + ensure_test_apartment(); + let fixture = HostedFixture::spawn().expect("a fixture host starts"); + let mut entry = blank_entry_with_path(&fixture, Vec::new()); + entry.geometry = agent_desktop_core::RefGeometry { + bounds_space: None, + bounds: None, + bounds_hash: None, + }; + + let handle = resolve_locator_anchor(&entry, deadline()) + .expect("the anchor lands the stored path and leaves identity to core"); + assert_verified_stamp(&handle, entry.process.pid.get(), &verified_token(&entry)); +} + +/// Every handle the anchor hands back carries the ref's verified process +/// identity, so the shared live read can corroborate that the provider +/// answering is still the generation resolution verified. +/// +/// Without the stamp the corroboration is skipped for every resolved handle, +/// and A14-9's finding - a dead provider's reads succeeding empty on some +/// builds - lands as an answer rather than a `STALE_REF`. The live-read tests +/// build their own stamp, so nothing there notices a resolver that stops +/// producing one; this is the assertion that sees it. +fn assert_verified_stamp(handle: &NativeHandle, pid: u32, token: &str) { + let resolved = handle + .downcast_ref::() + .expect("the resolved handle carries a UI Automation element"); + assert_eq!( + resolved.verified_process(), + Some((pid, token)), + "the anchor must stamp the ref's verified process identity onto the handle it returns" + ); +} + +fn verified_token(entry: &RefEntry) -> String { + entry.process.process_instance.clone().unwrap_or_default() } /// A path that points at a sibling beyond the target settles stale in a @@ -133,17 +179,15 @@ fn a_role_refuted_landing_settles_stale() { let source = crate::tree::walk::walker_source::UiaTreeSource::for_root(&root).expect("a source"); let prepared = source.prepare_root(&root).expect("a prepared root"); - let landing = crate::tree::resolve::resolve_search::descent::descend_path( + let landed = descend_path( &source, &prepared, &entry.scope.path, &crate::tree::walk::walker::WalkBudget::new(10, deadline), &ANCHOR_DESCENT, ) - .expect("the stored path descends"); - let landed = landing.element.expect( - "the stored path must land on a live element for the role guard to be what refuses it", - ); + .expect("the stored path descends") + .expect("the stored path must land on a live element for the role guard to be what refuses it"); let (_, evidence, _) = source.evidence(&landed); assert!( evidence @@ -160,68 +204,13 @@ fn a_role_refuted_landing_settles_stale() { assert_eq!(error.code, ErrorCode::StaleRef); } -/// `Rect::bounds_hash` answers `Some` for a zero-extent rectangle - it only -/// rejects an invalid one - so an id-less, text-less entry whose stored hash -/// comes from a zero-extent rectangle is not eligible for the geometry tier -/// at all (`provisional_geometry_candidate` requires positive area). The -/// eligibility gate must reject it before the path is even walked, settling -/// in one attempt instead of landing the path every retry and burning the -/// shared find deadline on a landing that could never verify. -#[test] -fn a_zero_extent_id_less_entry_settles_on_the_first_attempt() { - ensure_test_apartment(); - let fixture = HostedFixture::spawn().expect("a fixture host starts"); - let mut entry = blank_entry_with_path(&fixture, Vec::new()); - let zero_extent = Rect { - x: 0.0, - y: 0.0, - width: 0.0, - height: 0.0, - }; - let hash = zero_extent - .bounds_hash() - .expect("a zero-extent rect is still a valid rect, and still hashes"); - entry.geometry = agent_desktop_core::RefGeometry { - bounds_space: None, - bounds: Some(zero_extent), - bounds_hash: Some(hash), - }; - - let deadline = deadline(); - let attempts = std::cell::Cell::new(0); - let result = crate::tree::resolve::retry_incomplete_until(deadline, || { - attempts.set(attempts.get() + 1); - resolve_locator_anchor_once(&entry, deadline) - }); - - let error = match result { - Err(error) => error, - Ok(_) => panic!("a zero-extent, id-less anchor must never resolve"), - }; - assert_eq!(error.code, ErrorCode::StaleRef); - assert_eq!( - attempts.get(), - 1, - "a zero-extent id-less entry must settle in one attempt, not burn the deadline retrying" - ); -} - -fn locator_entry(role: &str, positive_area_hash: Option) -> RefEntry { - let (bounds, bounds_hash) = match positive_area_hash { - Some(rect) => (Some(rect), rect.bounds_hash()), - None => (None, None), - }; +fn locator_entry(role: &str) -> RefEntry { let mut entry = ref_entry(role); - entry.geometry = agent_desktop_core::RefGeometry { - bounds_space: None, - bounds, - bounds_hash, - }; entry.scope.path_is_absolute = true; entry } -fn locator_evidence(role: &str, bounds: LocatorField) -> LocatorEvidence { +fn locator_evidence(role: &str) -> LocatorEvidence { LocatorEvidence { role: LocatorField::Known(role.to_string()), name: LocatorField::Absent, @@ -231,74 +220,20 @@ fn locator_evidence(role: &str, bounds: LocatorField) -> LocatorEvidence { states: LocatorField::Absent, ref_evidence: LocatorRefEvidence { window_bounds: LocatorField::Absent, - bounds, + bounds: LocatorField::Absent, available_actions: LocatorField::Absent, descriptors: NodeDescriptor::default(), }, } } -/// A geometry-eligible entry whose live bounds were read successfully but -/// disagree with the stored hash has a definitive answer - the comparison -/// ran and failed - and must settle stale rather than retry. -#[test] -fn a_read_geometry_mismatch_settles_stale() { - let stored = Rect { - x: 0.0, - y: 0.0, - width: 10.0, - height: 10.0, - }; - let live = Rect { - x: 50.0, - y: 50.0, - width: 10.0, - height: 10.0, - }; - let entry = locator_entry("button", Some(stored)); - let evidence = locator_evidence("button", LocatorField::Known(live)); - - match anchor_geometry_verdict(&entry, &evidence) { - AnchorGeometryVerdict::SettleStale => {} - AnchorGeometryVerdict::Promote => panic!("a disagreeing hash must never promote"), - AnchorGeometryVerdict::Retry => { - panic!("a read-and-disagreeing hash is a definitive answer, not a transient read") - } - } -} - -/// The same geometry-eligible entry, but this attempt's live bounds read -/// came back unreadable rather than disagreeing. That is the genuinely -/// transient case `identity_unknown_error` stays reserved for, and must -/// still retry. -#[test] -fn an_unread_live_geometry_still_retries() { - let stored = Rect { - x: 0.0, - y: 0.0, - width: 10.0, - height: 10.0, - }; - let entry = locator_entry("button", Some(stored)); - let evidence = locator_evidence("button", LocatorField::Unknown); - - match anchor_geometry_verdict(&entry, &evidence) { - AnchorGeometryVerdict::Retry => {} - AnchorGeometryVerdict::Promote => panic!("an unread bounds field must never promote"), - AnchorGeometryVerdict::SettleStale => { - panic!("an unread bounds field is a transient read, not a settled mismatch") - } - } -} - /// Pins the fix for a tri-state defect where a failed role read was treated /// the same as a settled role mismatch. A role read that comes back /// `Unknown` must stay retryable, never a settled `STALE_REF`. #[test] fn a_role_read_that_comes_back_unknown_is_retryable_not_settled() { - let entry = locator_entry("button", None); - let evidence = locator_evidence("button", LocatorField::Absent); - let mut unknown_role_evidence = evidence; + let entry = locator_entry("button"); + let mut unknown_role_evidence = locator_evidence("button"); unknown_role_evidence.role = LocatorField::Unknown; let error = match anchor_role_verdict(&entry, &unknown_role_evidence) { @@ -326,8 +261,8 @@ fn a_role_read_that_comes_back_unknown_is_retryable_not_settled() { /// read successfully but disagrees with the stored ref settles stale. #[test] fn a_known_but_mismatched_role_settles_stale() { - let entry = locator_entry("button", None); - let evidence = locator_evidence("checkbox", LocatorField::Absent); + let entry = locator_entry("button"); + let evidence = locator_evidence("checkbox"); let error = match anchor_role_verdict(&entry, &evidence) { Err(error) => error, diff --git a/crates/windows/src/tree/resolve/resolve_chrome_repair.rs b/crates/windows/src/tree/resolve/resolve_chrome_repair.rs deleted file mode 100644 index ef9ea057a..000000000 --- a/crates/windows/src/tree/resolve/resolve_chrome_repair.rs +++ /dev/null @@ -1,122 +0,0 @@ -//! The path tier's second reading of a stored path, for the one drift a Win32 -//! container causes on its own: growing scroll bars ahead of its content. -//! -//! A `SysTreeView32` or `SysListView32` whose content outgrows the view gains a -//! vertical scroll bar, a horizontal one and the size grip between them, and -//! the proxy exposes all three as the container's *first* children. Every -//! content child's raw index moves up by the chrome count, so a path recorded -//! before the growth now descends into a scroll bar. Regedit measured it: the -//! outline's children went from `[Computer]` to `[scrollbar, scrollbar, -//! handle, Computer]` the moment `HKEY_CLASSES_ROOT` expanded, and the broad -//! search the declined landing fell through to then read every one of the -//! expanded key's thousands of subkeys and spent the whole deadline. - -use agent_desktop_core::RefEntry; - -use super::resolve_search::{ - NodeAdmission, SEARCH_DESCENT, admit_node, can_use_path_fast_path, descent, -}; -use crate::tree::walk::walker::{TreeSource, WalkBudget}; - -const LEADING_CHROME_ROLES: [&str; 2] = ["scrollbar", "handle"]; - -const MAX_LEADING_CHROME: usize = 3; - -const SIBLING_CHECK_CAP: usize = 64; - -/// Re-walks the stored path allowing for leading chrome that appeared after -/// the ref was recorded, and answers the landing only when a level shifted -/// and the landing is the one sibling that answers to the stored identity. -/// -/// Leading chrome is the two scroll bars and the size grip (at most three -/// children, roles `scrollbar` and `handle`). A level that carries it is -/// shifted past it only when the stored index lands *inside* the chrome, the -/// one reading that cannot be ambiguous: a content ref recorded after the -/// chrome appeared counts it and never lands there. A level that carries -/// chrome but whose stored index lands on content cannot say whether it was -/// recorded before or after the chrome appeared, so the whole repair -/// declines rather than guess - either guess can enter a different container -/// whose child shares the stored identity, and a wrong verified target is -/// worse than leaving that ref to the broad search. Levels without chrome are -/// walked as stored. The identity check reads the landing's own siblings (a -/// list wider than 64 declines), and any read fault declines. -/// -/// The stored bounds are deliberately not consulted. A container that grew -/// scroll bars also re-laid out and usually scrolled, so the stored rectangle -/// names a position that no longer exists. -pub(crate) fn repair_past_leading_chrome( - source: &S, - root: &S::Node, - entry: &RefEntry, - budget: &WalkBudget, -) -> Option { - if !can_use_path_fast_path(entry) { - return None; - } - let mut parent = None; - let mut current = root.clone(); - let mut shifted = false; - for &index in entry.scope.path.iter() { - let take = index.saturating_add(MAX_LEADING_CHROME + 1); - let children = - descent::read_children(source, ¤t, budget, &SEARCH_DESCENT, Some(take)).ok()?; - let chrome = children - .elements - .iter() - .take(MAX_LEADING_CHROME) - .take_while(|child| is_leading_chrome(source, child)) - .count(); - if chrome > 0 && index >= chrome { - return None; - } - shifted |= chrome > 0; - let next = children.elements.get(index + chrome)?.clone(); - parent = Some(std::mem::replace(&mut current, next)); - } - let parent = parent?; - (shifted && is_unique_identity_match(source, &parent, ¤t, entry, budget)) - .then_some(current) -} - -fn is_leading_chrome(source: &S, node: &S::Node) -> bool { - let (_, evidence, _) = source.evidence(node); - evidence - .role - .known() - .is_some_and(|role| LEADING_CHROME_ROLES.contains(&role.as_str())) -} - -fn is_unique_identity_match( - source: &S, - parent: &S::Node, - landing: &S::Node, - entry: &RefEntry, - budget: &WalkBudget, -) -> bool { - let Ok(siblings) = descent::read_children( - source, - parent, - budget, - &SEARCH_DESCENT, - Some(SIBLING_CHECK_CAP + 1), - ) else { - return false; - }; - if !siblings.complete || siblings.elements.len() > SIBLING_CHECK_CAP { - return false; - } - let mut matched = None; - for sibling in &siblings.elements { - let (_, evidence, _) = source.evidence(sibling); - match admit_node(entry, &evidence) { - NodeAdmission::Collect if matched.is_none() => matched = Some(sibling), - NodeAdmission::Collect | NodeAdmission::Unread => return false, - NodeAdmission::Reject => {} - } - } - matched.is_some_and(|only| source.same_element(only, landing)) -} - -#[cfg(test)] -#[path = "resolve_chrome_repair_tests.rs"] -mod tests; diff --git a/crates/windows/src/tree/resolve/resolve_chrome_repair_tests.rs b/crates/windows/src/tree/resolve/resolve_chrome_repair_tests.rs deleted file mode 100644 index 6fc647c6b..000000000 --- a/crates/windows/src/tree/resolve/resolve_chrome_repair_tests.rs +++ /dev/null @@ -1,180 +0,0 @@ -use super::*; -use crate::tree::properties::property_ids::TreeProperty; -use crate::tree::test_support::{number, text}; -use crate::tree::walk::walker_fake::{FakeTree, budget, ref_entry}; - -const SCROLL_BAR: i32 = 50014; -const THUMB: i32 = 50027; -const TREE_ITEM: i32 = 50024; -const OUTLINE: i32 = 1; -const COMPUTER: i32 = 20; -const CLASSES_ROOT: i32 = 30; -const CURRENT_USER: i32 = 31; - -fn tree_item(tree: FakeTree, node: i32, name: &str) -> FakeTree { - tree.reading( - node, - &[ - number(TreeProperty::ControlType, TREE_ITEM), - (TreeProperty::Name, text(name)), - ], - ) -} - -fn chrome(tree: FakeTree, node: i32, control_type: i32) -> FakeTree { - tree.reading(node, &[number(TreeProperty::ControlType, control_type)]) -} - -/// The regedit outline after `HKEY_CLASSES_ROOT` expanded: two scroll bars and -/// the size grip now lead the outline's children, ahead of `Computer`. -fn grown_outline(second_root_name: &str) -> FakeTree { - let tree = FakeTree::default() - .with_children(0, &[OUTLINE]) - .with_children(OUTLINE, &[10, 11, 12, COMPUTER]) - .with_children(COMPUTER, &[CLASSES_ROOT, CURRENT_USER]); - let tree = chrome(tree, 10, SCROLL_BAR); - let tree = chrome(tree, 11, SCROLL_BAR); - let tree = chrome(tree, 12, THUMB); - let tree = tree_item(tree, COMPUTER, "Computer"); - let tree = tree_item(tree, CLASSES_ROOT, "HKEY_CLASSES_ROOT"); - tree_item(tree, CURRENT_USER, second_root_name) -} - -/// The ref the snapshot stored before the expand: `[outline, Computer, -/// HKEY_CLASSES_ROOT]` with the outline's content counted from index 0, and a -/// bounds hash the scrolled view no longer reproduces. -fn classes_root_entry() -> RefEntry { - let mut entry = ref_entry("treeitem"); - entry.identity.name = Some("HKEY_CLASSES_ROOT".into()); - entry.geometry.bounds_hash = Some(7); - entry.scope.path = vec![0, 0, 0].into(); - entry -} - -#[test] -fn a_path_recorded_before_scroll_bars_appeared_lands_past_them() { - let tree = grown_outline("HKEY_CURRENT_USER"); - - let repaired = repair_past_leading_chrome(&tree, &0, &classes_root_entry(), &budget(10)); - - assert_eq!( - repaired, - Some(CLASSES_ROOT), - "the stored index counts from the first content child, so the repair must step past \ - the two scroll bars and the size grip and land on the stored key" - ); -} - -#[test] -fn a_path_with_no_leading_chrome_is_left_to_the_other_tiers() { - let tree = FakeTree::default() - .with_children(0, &[OUTLINE]) - .with_children(OUTLINE, &[COMPUTER]) - .with_children(COMPUTER, &[CLASSES_ROOT, CURRENT_USER]); - let tree = tree_item(tree, COMPUTER, "Computer"); - let tree = tree_item(tree, CLASSES_ROOT, "HKEY_CLASSES_ROOT"); - let tree = tree_item(tree, CURRENT_USER, "HKEY_CURRENT_USER"); - - let repaired = repair_past_leading_chrome(&tree, &0, &classes_root_entry(), &budget(10)); - - assert_eq!( - repaired, None, - "with no chrome to explain a drift, a landing the plain path tier declined must not be \ - re-accepted here without its bounds check" - ); -} - -#[test] -fn a_duplicate_sibling_identity_declines_the_repaired_landing() { - let tree = grown_outline("HKEY_CLASSES_ROOT"); - - let repaired = repair_past_leading_chrome(&tree, &0, &classes_root_entry(), &budget(10)); - - assert_eq!( - repaired, None, - "the repair ignores the stale bounds, so a second sibling answering to the same \ - identity must leave the tie-break to the broad search" - ); -} - -/// A path recorded while the scroll bars already existed counts them, so its -/// index lands on content and must not be shifted a second time into a -/// neighbouring parent that holds a child with the same identity. -#[test] -fn a_path_that_already_counts_the_chrome_is_not_shifted_into_a_neighbour() { - const OTHER: i32 = 21; - const IMPOSTOR: i32 = 40; - let tree = grown_outline("HKEY_CURRENT_USER") - .with_children(OUTLINE, &[10, 11, 12, COMPUTER, 13, 14, OTHER]) - .with_children(OTHER, &[IMPOSTOR]); - let tree = tree_item(tree, 13, "Spacer A"); - let tree = tree_item(tree, 14, "Spacer B"); - let tree = tree_item(tree, OTHER, "Other"); - let tree = tree_item(tree, IMPOSTOR, "HKEY_CLASSES_ROOT"); - let mut entry = classes_root_entry(); - entry.scope.path = vec![0, 3, 0].into(); - - let repaired = repair_past_leading_chrome(&tree, &0, &entry, &budget(10)); - - assert_eq!( - repaired, None, - "index 3 already lands on Computer past the chrome; adding the chrome again reaches \ - the neighbour's same-named child, which must never answer for the ref" - ); -} - -#[test] -fn a_content_index_is_never_shifted_into_another_container() { - const A: i32 = 13; - const B: i32 = 14; - const C: i32 = 15; - const D: i32 = 16; - const WRONG: i32 = 50; - const WANTED: i32 = 51; - let tree = grown_outline("HKEY_CURRENT_USER") - .with_children(OUTLINE, &[10, 11, 12, A, B, C, D]) - .with_children(A, &[WRONG]) - .with_children(D, &[WANTED]); - let tree = [(A, "A"), (B, "B"), (C, "C"), (D, "D"), (WRONG, "Unrelated")] - .into_iter() - .fold(tree, |tree, (node, name)| tree_item(tree, node, name)); - let tree = tree_item(tree, WANTED, "HKEY_CLASSES_ROOT"); - let mut entry = classes_root_entry(); - entry.scope.path = vec![0, 3, 0].into(); - - let repaired = repair_past_leading_chrome(&tree, &0, &entry, &budget(10)); - - assert_eq!( - repaired, None, - "index 3 lands on content, so no level may shift it into D's same-named child" - ); -} - -#[test] -fn an_ambiguous_deeper_level_declines_even_after_an_earlier_shift() { - const P: i32 = 63; - const Q: i32 = 64; - const R: i32 = 65; - const S: i32 = 66; - const IMPOSTOR: i32 = 70; - let tree = grown_outline("HKEY_CURRENT_USER") - .with_children(COMPUTER, &[60, 61, 62, P, Q, R, S]) - .with_children(P, &[IMPOSTOR]); - let tree = chrome(tree, 60, SCROLL_BAR); - let tree = chrome(tree, 61, SCROLL_BAR); - let tree = chrome(tree, 62, THUMB); - let tree = [(P, "P"), (Q, "Q"), (R, "R"), (S, "S")] - .into_iter() - .fold(tree, |tree, (node, name)| tree_item(tree, node, name)); - let tree = tree_item(tree, IMPOSTOR, "HKEY_CLASSES_ROOT"); - let mut entry = classes_root_entry(); - entry.scope.path = vec![0, 0, 3, 0].into(); - - let repaired = repair_past_leading_chrome(&tree, &0, &entry, &budget(10)); - - assert_eq!( - repaired, None, - "Computer's index 3 lands on content beside chrome, so it is ambiguous and the \ - earlier shift must not carry the repair into P's same-named child" - ); -} diff --git a/crates/windows/src/tree/resolve/resolve_descent.rs b/crates/windows/src/tree/resolve/resolve_descent.rs index 49b5efd51..1616fd0e0 100644 --- a/crates/windows/src/tree/resolve/resolve_descent.rs +++ b/crates/windows/src/tree/resolve/resolve_descent.rs @@ -1,11 +1,9 @@ -//! The bounded sibling enumeration and child-index path descent that both -//! resolvers drive. +//! The bounded sibling enumeration and child-index path descent the locator +//! anchor drives. //! -//! The broad search and the locator anchor walk the same two cross-process -//! primitives - one `first_child` followed by a `next_sibling` chain - under -//! the same sibling cap and the same deadline. They differ in exactly two -//! decisions: what a failed enumeration step means, and what an expired -//! deadline means. Those two are the parameters; the loop is written once. +//! Both primitives are the same two cross-process calls - one `first_child` +//! followed by a `next_sibling` chain - under the same sibling cap and the +//! same deadline, and the loop is written once. //! //! The loop consults the deadline on every iteration, mirroring the element //! walk's own enumeration (`walker_enumerate.rs`). Checking it only around the @@ -24,152 +22,65 @@ pub(crate) enum DescentVerdict { /// The node enumerates nothing further and that is a real answer: keep /// what was collected, and the list still counts as whole. Settled, - /// The node enumerates nothing further, but only because a read failed: - /// keep what was collected and record that the list is partial. - Unfinished, /// The failure is the attempt's answer and propagates as an error. Surfaced, } -/// What an expired deadline means to the resolver that asked. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub(crate) enum ExpiryPolicy { - /// Keep the partial list and record that it is partial. For a resolver - /// whose partial answer is already retried, an unfinished enumeration is - /// the signal it acts on. - Unfinish, - /// Surface the timeout. For a resolver that settles in a single attempt, a - /// partial list would be read as a settled absence - "the element is gone" - - /// when the truth is only that time ran out. - Surface, -} - -/// The two decisions that differ between the resolvers, with the shape-only -/// phrase each enumeration axis reports a surfaced failure under. +/// The decision that differs per enumeration axis, with the shape-only phrase +/// each axis reports a surfaced failure under. pub(crate) struct DescentPolicy { pub(crate) classify: fn(UiaFailure) -> DescentVerdict, - pub(crate) on_expiry: ExpiryPolicy, pub(crate) descend_context: &'static str, pub(crate) sibling_context: &'static str, } -/// One node's child list as the enumeration found it. -#[derive(Debug)] -pub(crate) struct ChildList { - pub(crate) elements: Vec, - pub(crate) complete: bool, -} - -/// Where a child-index path walk ended, and what that walk left unread. -/// -/// The two are separate facts about separate parts of the tree, returned -/// unfolded because folding them loses the only property that makes either one -/// usable. -/// -/// `element` is trustworthy on its own terms. [`read_children`] only ever -/// appends, and every exit it can take - the sibling cap, an expired deadline, -/// a failed read - breaks the loop rather than skipping an entry, so the list -/// it returns is always a *prefix* of the node's real child list. An index that -/// prefix reaches therefore names the same child the whole list would name at -/// that index, and [`descend_path`] descends only through indices the prefix -/// reached: a stored index past a truncation reads `None` and lands nowhere. So -/// a landed element is the element the stored path names, and anything an -/// enumeration on the way down failed to read lies strictly *after* the index -/// this walk used at that level. -/// -/// `unread_region` is a fact about that unread remainder and never about the -/// landing. It says this walk did not see the whole tree, so a verdict that -/// depends on having seen the whole of it - above all a negative one, "the -/// stored element is gone" - is not this walk's to settle. -pub(crate) struct PathLanding { - pub(crate) element: Option, - pub(crate) unread_region: bool, -} - -impl PathLanding { - /// The landing of a path walk that was never run, for a caller whose - /// eligibility gate declined the tier. It read nothing, so it lands - /// nowhere and leaves no region unread of its own - the tiers below it - /// still cover the whole tree. - pub(crate) fn not_walked() -> Self { - Self { - element: None, - unread_region: false, - } - } -} - /// Enumerates one element's children, honouring the sibling cap as a hard /// bound on pathological lists and the deadline as a bound on time. /// /// `take` stops the loop after that many siblings: `None` enumerates the -/// whole list (the broad search, which must see every sibling to settle an -/// absence), while `descend_path` passes the wanted index plus one, so -/// reaching the wanted index costs index-plus-one cross-process calls -/// however long the list runs. A take-hit reports whole exactly like a -/// cap-hit, for the same reason: no verdict the caller draws from the -/// returned prefix changes - an index the prefix reaches names the same -/// child the whole list would name - and an unfinished verdict would -/// withhold the entire search's answer over siblings the broad search still -/// walks in full on its own pass. The one shape this does not cover is a -/// ref deeper than the search's own depth cap whose path declines: the -/// broad search cannot reach past its cap to re-cover a take-truncated -/// remainder down there, so that remainder is unseen by every tier. +/// whole list, while `descend_path` passes the wanted index plus one, so +/// reaching the wanted index costs index-plus-one cross-process calls however +/// long the list runs. /// -/// The cap is the other place this enumeration's completeness is not -/// literal: a cap-hit keeps the truncated list and still reports it whole. -/// That is deliberate. The bound exists to make the loop total against a -/// sibling chain that never terminates - the ancestor cycle guard cannot see -/// such a chain, because it revisits no element on any root-to-node path - -/// not to clamp a large but finite list. Measured breadth is nowhere near -/// it: the widest node in the probe corpus sits in a virtualized Explorer -/// file list whose entire tree over `%WINDIR%\System32` is 196 nodes (A1-1), -/// and a WinForms `ListBox` exposes no items at all to a COM client (A17-2). +/// The cap is the other place this enumeration is not literal: a cap-hit +/// keeps the truncated list and answers it rather than erroring. That is +/// deliberate. The bound exists to make the loop total against a sibling +/// chain that never terminates - the ancestor cycle guard cannot see such a +/// chain, because it revisits no element on any root-to-node path - not to +/// clamp a large but finite list. Measured breadth is nowhere near it: the +/// widest node in the probe corpus sits in a virtualized Explorer file list +/// whose entire tree over `%WINDIR%\System32` is 196 nodes (A1-1), and a +/// WinForms `ListBox` exposes no items at all to a COM client (A17-2). /// -/// Reporting the truncation unfinished would cost more honesty than it buys, -/// for two reasons. The bound is shared with the walk that issues refs in the -/// first place: `walk_from_root` enumerates under the same -/// `WalkBudget::max_siblings` and refuses to present a cap-cut child list as -/// whole, so no stored ref can name an element past this point in its parent's -/// sibling list. And an unfinished verdict is not scoped to the node that -/// truncated - it withholds the entire search's answer, so one oversized list -/// anywhere in a window would convert a correct settled miss elsewhere in that -/// window into a retry loop that re-truncates identically on every attempt and -/// expires as `TIMEOUT`. +/// Surfacing the truncation would cost more honesty than it buys: the bound is +/// shared with the walk that issues refs in the first place, and +/// `walk_from_root` enumerates under the same `WalkBudget::max_siblings` and +/// refuses to present a cap-cut child list as whole, so no stored ref can name +/// an element past this point in its parent's sibling list. One oversized list +/// anywhere in a window would otherwise convert every correct settled miss in +/// that window into a retry loop that re-truncates identically on every +/// attempt and expires as `TIMEOUT`. pub(crate) fn read_children( source: &S, element: &S::Node, budget: &WalkBudget, policy: &DescentPolicy, take: Option, -) -> Result, AdapterError> { +) -> Result, AdapterError> { let limit = take .map(|take| take.min(budget.max_siblings)) .unwrap_or(budget.max_siblings); let mut elements = Vec::new(); let mut current = match source.first_child(element) { Ok(first) => first, - Err(failure) if failure.is_exhaustion() => { - return Ok(ChildList { - elements, - complete: true, - }); - } + Err(failure) if failure.is_exhaustion() => return Ok(elements), Err(failure) => return classified(failure, elements, policy, policy.descend_context), }; loop { if elements.len() >= limit { break; } - if let Err(expired) = budget.deadline.ensure_remaining() { - return match policy.on_expiry { - ExpiryPolicy::Unfinish => Ok(ChildList { - elements, - complete: false, - }), - ExpiryPolicy::Surface => Err(expired), - }; - } + budget.deadline.ensure_remaining()?; elements.push(current.clone()); if elements.len() >= limit { break; @@ -180,10 +91,7 @@ pub(crate) fn read_children( Err(failure) => return classified(failure, elements, policy, policy.sibling_context), } } - Ok(ChildList { - elements, - complete: true, - }) + Ok(elements) } /// Walks the stored child-index path from a root, O(depth) child reads. @@ -191,21 +99,27 @@ pub(crate) fn read_children( /// Each level enumerates only to the wanted index plus one and stops: a path /// step needs the one child it names, never the thousand siblings after it /// (a ref to a subkey near the top of an expanded registry hive would -/// otherwise read all of the hive's thousands of subkeys first). A path step that lands nowhere yields no element; -/// what that absence means belongs to the caller, not here. The index is -/// taken against the raw enumeration, the same space the walk that issued -/// the ref recorded its stored index in, so a walk that omitted a sibling -/// from its own output still leaves the stored index pointing at the child -/// it named. +/// otherwise read all of the hive's thousands of subkeys first). +/// +/// The landing is trustworthy on its own terms. [`read_children`] only ever +/// appends, and every exit it can take - the sibling cap, an expired deadline, +/// a failed read - breaks the loop rather than skipping an entry, so the list +/// it returns is always a *prefix* of the node's real child list. An index that +/// prefix reaches therefore names the same child the whole list would name at +/// that index, and this function descends only through indices the prefix +/// reached: a stored index past a truncation yields `None` and lands nowhere. +/// What that absence means belongs to the caller, not here. The index is taken +/// against the raw enumeration, the same space the walk that issued the ref +/// recorded its stored index in, so a walk that omitted a sibling from its own +/// output still leaves the stored index pointing at the child it named. pub(crate) fn descend_path( source: &S, root: &S::Node, path: &[usize], budget: &WalkBudget, policy: &DescentPolicy, -) -> Result, AdapterError> { +) -> Result, AdapterError> { let mut current = root.clone(); - let mut unread_region = false; for &index in path { let read = read_children( source, @@ -214,19 +128,12 @@ pub(crate) fn descend_path( policy, Some(index.saturating_add(1)), )?; - unread_region |= !read.complete; - let Some(child) = read.elements.get(index) else { - return Ok(PathLanding { - element: None, - unread_region, - }); + let Some(child) = read.get(index) else { + return Ok(None); }; current = child.clone(); } - Ok(PathLanding { - element: Some(current), - unread_region, - }) + Ok(Some(current)) } fn classified( @@ -234,16 +141,9 @@ fn classified( elements: Vec, policy: &DescentPolicy, context: &str, -) -> Result, AdapterError> { +) -> Result, AdapterError> { match (policy.classify)(failure) { - DescentVerdict::Settled => Ok(ChildList { - elements, - complete: true, - }), - DescentVerdict::Unfinished => Ok(ChildList { - elements, - complete: false, - }), + DescentVerdict::Settled => Ok(elements), DescentVerdict::Surfaced => Err(uia_failure_error(failure, context)), } } diff --git a/crates/windows/src/tree/resolve/resolve_descent_tests.rs b/crates/windows/src/tree/resolve/resolve_descent_tests.rs index 5bdf16aa8..e342d6981 100644 --- a/crates/windows/src/tree/resolve/resolve_descent_tests.rs +++ b/crates/windows/src/tree/resolve/resolve_descent_tests.rs @@ -8,7 +8,6 @@ use crate::tree::name_evidence::LabelOutcome; use crate::tree::properties::ElementProperties; use crate::tree::resolve::resolve_anchor::ANCHOR_DESCENT; use crate::tree::resolve::resolve_search; -use crate::tree::resolve::resolve_search::SEARCH_DESCENT; use crate::tree::walk::walker::{NodeKey, walk_vocabulary}; use agent_desktop_core::{Deadline, ErrorCode, LocatorEvidence}; @@ -113,37 +112,15 @@ fn expired_budget() -> WalkBudget { WalkBudget::new(10, deadline) } -/// The control for both expiry pins: with time left, each policy enumerates -/// the whole sibling list, so an expiry assertion below can only be produced -/// by the deadline check itself. +/// The control for the expiry pins below: with time left the enumeration +/// reads the whole sibling list, so an expiry assertion below can only be +/// produced by the deadline check itself. #[test] -fn a_live_deadline_enumerates_the_whole_sibling_list_under_both_policies() { +fn a_live_deadline_enumerates_the_whole_sibling_list() { let tree = StubTree::with_children(3); - for policy in [&SEARCH_DESCENT, &ANCHOR_DESCENT] { - let read = read_children(&tree, &0, &live_budget(), policy, None) - .expect("an unfaulting enumeration succeeds"); - assert_eq!(read.elements, vec![1, 2, 3]); - assert!(read.complete); - } -} - -/// The search's expiry policy: the enumeration stops on an expired deadline -/// and reports the list as partial. Unfinished is what the search's own -/// classification already means by retryable, and the search did not finish, -/// so a truncated list must never be presented as a whole one. -#[test] -fn an_expired_deadline_leaves_the_search_enumeration_unfinished() { - let tree = StubTree::with_children(10); - let read = read_children(&tree, &0, &expired_budget(), &SEARCH_DESCENT, None) - .expect("the search keeps its partial list rather than surfacing"); - assert!( - !read.complete, - "an enumeration cut short by the deadline is not a whole child list" - ); - assert!( - read.elements.len() < 10, - "the deadline must be consulted inside the loop, not only around it" - ); + let read = read_children(&tree, &0, &live_budget(), &ANCHOR_DESCENT, None) + .expect("an unfaulting enumeration succeeds"); + assert_eq!(read, vec![1, 2, 3]); } /// The anchor's expiry policy: it settles in one attempt, so an expired @@ -157,7 +134,7 @@ fn an_expired_deadline_surfaces_a_timeout_for_the_anchor_never_a_settled_miss() Err(error) => error, Ok(read) => panic!( "an expired anchor enumeration must surface, it yielded {} children", - read.elements.len() + read.len() ), }; assert_eq!(error.code, ErrorCode::Timeout); @@ -173,19 +150,17 @@ fn an_expired_deadline_never_lands_the_anchor_path_nowhere() { Err(error) => error, Ok(landing) => panic!( "a starved path walk must surface, it landed on {:?}", - landing.element + landing ), }; assert_eq!(error.code, ErrorCode::Timeout); } -/// The inverse pair that gives the two classifiers their reason to exist. A -/// node that vanished mid-descent is not evidence the *target* vanished, so -/// the broad search records the gap and keeps looking; the anchor walks one -/// exact path that churn makes permanently wrong, so the same failure is its -/// settled answer. +/// A node that vanished mid-descent is the anchor's settled answer: it walks +/// one exact path that churn makes permanently wrong, so the absence is what +/// it reports rather than a gap it keeps looking past. #[test] -fn a_vanished_node_leaves_the_search_unfinished_but_settles_the_anchor() { +fn a_vanished_node_settles_the_anchor() { let vanished = UiaFailure::Sentinel(ERR_INVALID_OBJECT); assert_eq!( uia_failure_disposition(vanished), @@ -194,28 +169,15 @@ fn a_vanished_node_leaves_the_search_unfinished_but_settles_the_anchor() { ); let tree = StubTree::with_children(3).failing_on_siblings(vanished); - let searched = read_children(&tree, &0, &live_budget(), &SEARCH_DESCENT, None) - .expect("a vanished sibling never surfaces for the search"); - assert_eq!(searched.elements, vec![1]); - assert!( - !searched.complete, - "a node vanishing mid-descent leaves the search retryable, never a settled absence" - ); - let anchored = read_children(&tree, &0, &live_budget(), &ANCHOR_DESCENT, None) .expect("a vanished sibling never surfaces for the anchor"); - assert_eq!(anchored.elements, vec![1]); - assert!( - anchored.complete, - "a vanished node is the anchor's settled answer, never a retry" - ); + assert_eq!(anchored, vec![1]); } -/// Transport is the other axis of the same inversion: the search absorbs it as -/// a gap it can retry past, while the anchor propagates it so its bounded -/// retry loop replays the cheap path walk rather than swallowing the fault. +/// Transport propagates for the anchor so its bounded retry loop replays the +/// cheap path walk rather than swallowing the fault. #[test] -fn a_transport_failure_leaves_the_search_unfinished_and_surfaces_for_the_anchor() { +fn a_transport_failure_surfaces_for_the_anchor() { let transport = UiaFailure::Sentinel(ERR_TIMEOUT); assert_eq!( uia_failure_disposition(transport), @@ -224,10 +186,6 @@ fn a_transport_failure_leaves_the_search_unfinished_and_surfaces_for_the_anchor( ); let tree = StubTree::with_children(3).failing_on_siblings(transport); - let searched = read_children(&tree, &0, &live_budget(), &SEARCH_DESCENT, None) - .expect("a transport failure never surfaces for the search"); - assert!(!searched.complete); - let error = match read_children(&tree, &0, &live_budget(), &ANCHOR_DESCENT, None) { Err(error) => error, Ok(_) => panic!("a transport failure must surface for the anchor, not settle"), @@ -235,11 +193,10 @@ fn a_transport_failure_leaves_the_search_unfinished_and_surfaces_for_the_anchor( assert!(error.is_explicitly_retryable()); } -/// A settled absence is the one answer both resolvers read the same way: the -/// provider said the enumeration cannot exist, so the empty list is real and -/// neither resolver has anything left to retry for. +/// A settled absence means the provider said the enumeration cannot exist, so +/// the empty list is real and the anchor has nothing left to retry for. #[test] -fn a_settled_absence_ends_both_enumerations_whole() { +fn a_settled_absence_ends_the_enumeration_whole() { let absent = UiaFailure::Sentinel(ERR_INVALID_ARG); assert_eq!( uia_failure_disposition(absent), @@ -248,21 +205,15 @@ fn a_settled_absence_ends_both_enumerations_whole() { ); let tree = StubTree::with_children(3).failing_to_descend(absent); - for policy in [&SEARCH_DESCENT, &ANCHOR_DESCENT] { - let read = read_children(&tree, &0, &live_budget(), policy, None) - .expect("a settled absence never surfaces"); - assert!(read.elements.is_empty()); - assert!( - read.complete, - "a settled absence is a real answer, never an unfinished read" - ); - } + let read = read_children(&tree, &0, &live_budget(), &ANCHOR_DESCENT, None) + .expect("a settled absence never surfaces"); + assert!(read.is_empty()); } -/// A terminal failure has no defined recovery, so neither resolver may present +/// A terminal failure has no defined recovery, so the anchor may not present /// a truncated list as an answer. #[test] -fn a_terminal_failure_surfaces_for_both() { +fn a_terminal_failure_surfaces() { let terminal = UiaFailure::Sentinel(ERR_NULL_PTR); assert_eq!( uia_failure_disposition(terminal), @@ -271,39 +222,30 @@ fn a_terminal_failure_surfaces_for_both() { ); let tree = StubTree::with_children(3).failing_to_descend(terminal); - for policy in [&SEARCH_DESCENT, &ANCHOR_DESCENT] { - let error = match read_children(&tree, &0, &live_budget(), policy, None) { - Err(error) => error, - Ok(_) => panic!("a terminal failure must surface, never yield a child list"), - }; - assert!(!error.is_explicitly_retryable()); - } + let error = match read_children(&tree, &0, &live_budget(), &ANCHOR_DESCENT, None) { + Err(error) => error, + Ok(_) => panic!("a terminal failure must surface, never yield a child list"), + }; + assert!(!error.is_explicitly_retryable()); } -/// The sibling cap truncates the list and still reports it whole, under both -/// policies. That verdict is the search's `incomplete` input, and -/// `classify_search` settles `STALE_REF` from a candidate-less search whose -/// `incomplete` flag is clear, so this pin is the link between the bound and -/// the settled answer. Changing it decides the whole resolution contract for -/// an oversized list and must never be an incidental edit. +/// The sibling cap truncates the list and answers it rather than erroring. A +/// stored index past the cap therefore lands nowhere - the anchor's +/// settled-miss signal - rather than surfacing a read failure the retry loop +/// would replay to the same truncation. Changing it decides the whole +/// resolution contract for an oversized list and must never be an incidental +/// edit. #[test] fn the_sibling_cap_truncates_and_reports_the_truncated_list_as_whole() { let tree = StubTree::with_children(12); let capped = live_budget().with_max_siblings(4); - for policy in [&SEARCH_DESCENT, &ANCHOR_DESCENT] { - let read = read_children(&tree, &0, &capped, policy, None) - .expect("a cap-hit never surfaces as an error"); - assert_eq!( - read.elements, - vec![1, 2, 3, 4], - "the cap bounds the list at exactly its own value" - ); - assert!( - read.complete, - "a cap-cut list is reported whole, so the search settles rather than retrying a \ - truncation that would repeat identically" - ); - } + let read = read_children(&tree, &0, &capped, &ANCHOR_DESCENT, None) + .expect("a cap-hit never surfaces as an error"); + assert_eq!( + read, + vec![1, 2, 3, 4], + "the cap bounds the list at exactly its own value" + ); } /// The bound every resolver enumerates under is at least the bound the walk @@ -327,10 +269,9 @@ fn every_resolver_enumerates_at_least_as_far_as_the_walk_that_issued_the_ref() { /// The anchor's reading of the same truncation, one level up. A stored index /// past the cap lands nowhere, and `descend_path` reports that as a `None` -/// landing on a walk it still calls whole - the anchor's settled-miss signal, -/// reached through the missing element rather than through completeness, which -/// the anchor never consults. So the cap-hit settles the anchor in one attempt -/// instead of replaying a path walk that would truncate at the same place. +/// landing - the anchor's settled-miss signal. So the cap-hit settles the +/// anchor in one attempt instead of replaying a path walk that would truncate +/// at the same place. #[test] fn a_stored_index_past_the_sibling_cap_settles_the_anchor_rather_than_retrying() { let tree = StubTree::with_children(12); @@ -340,58 +281,20 @@ fn a_stored_index_past_the_sibling_cap_settles_the_anchor_rather_than_retrying() .expect("a cap-hit never surfaces for the anchor"); assert!( - landing.element.is_none(), + landing.is_none(), "an index the truncated list does not reach lands nowhere" ); - assert!(!landing.unread_region); -} - -/// The seam that carries a path walk's own gap out to the resolution that -/// asked for it. Landing nowhere has two causes that must never be conflated: -/// the stored index is genuinely absent, or the enumeration that would have -/// reached it faulted. Only the second leaves a region unread, and -/// `walk_stored_path` is where the caller learns which it was. Losing that -/// distinction settles `STALE_REF` - "the element is gone" - off a walk that -/// never finished, and the path walk is the one tier descending past the broad -/// search's depth cap, so its gap can be the only evidence left to read. -#[test] -fn a_faulted_path_step_lands_nowhere_and_reports_the_walk_unfinished() { - let walk = |tree: &StubTree| { - let landing = resolve_search::walk_stored_path(tree, &0, &[2], &live_budget()) - .expect("a transport fault never surfaces for the search"); - (landing.element, landing.unread_region) - }; - - assert_eq!( - walk(&StubTree::with_children(3).failing_on_siblings(UiaFailure::Sentinel(ERR_TIMEOUT))), - (None, true), - "a step that could not be read lands nowhere and is a gap the resolution must see" - ); - assert_eq!( - walk(&StubTree::with_children(3)), - (Some(3), false), - "the control reaches the stored index on a whole walk, so the pin above is not vacuous" - ); } -/// The shape-only phrases each axis reports under are part of the two -/// policies, not of the shared loop, so a caller cannot lose its own wording -/// by reusing the enumeration. +/// The shape-only phrase both enumeration axes report under belongs to the +/// policy, not to the shared loop: the anchor descends one exact path, so +/// descent and sibling walk are the same act to a reader of the error. #[test] -fn each_policy_keeps_its_own_axis_wording() { +fn the_anchor_keeps_one_axis_wording() { let terminal = UiaFailure::Sentinel(ERR_NULL_PTR); let descend = StubTree::with_children(3).failing_to_descend(terminal); let siblings = StubTree::with_children(3).failing_on_siblings(terminal); - let descend_error = read_children(&descend, &0, &live_budget(), &SEARCH_DESCENT, None) - .expect_err("a terminal descent surfaces"); - let sibling_error = read_children(&siblings, &0, &live_budget(), &SEARCH_DESCENT, None) - .expect_err("a terminal sibling walk surfaces"); - assert_ne!( - descend_error.message, sibling_error.message, - "the search reports its two enumeration axes distinctly" - ); - let anchor_descend = read_children(&descend, &0, &live_budget(), &ANCHOR_DESCENT, None) .expect_err("a terminal descent surfaces"); let anchor_sibling = read_children(&siblings, &0, &live_budget(), &ANCHOR_DESCENT, None) diff --git a/crates/windows/src/tree/resolve/resolve_landing_tests.rs b/crates/windows/src/tree/resolve/resolve_landing_tests.rs index 12c500d12..11446a8aa 100644 --- a/crates/windows/src/tree/resolve/resolve_landing_tests.rs +++ b/crates/windows/src/tree/resolve/resolve_landing_tests.rs @@ -1,14 +1,14 @@ -//! The property every stored-path landing rests on. +//! The property every stored-path landing rests on, and what reaching one +//! costs. //! //! `read_children` only ever appends, and each of its exits breaks the loop //! rather than skipping an entry, so a truncated enumeration is a *prefix* of -//! the real child list. `descend_path` applies that one level at a time, so the -//! pins below are stated at one level and compose down the path by the loop -//! that walks it. +//! the real child list. `descend_path` applies that one level at a time, so an +//! index the prefix reaches names the same child the whole list would name +//! there - and reaching it costs only the reads up to that index. use super::*; -use crate::tree::automation::UiaFailure; -use crate::tree::resolve::resolve_search::{SEARCH_DESCENT, walk_stored_path}; +use crate::tree::resolve::resolve_anchor::ANCHOR_DESCENT; use crate::tree::walk::walker::TreeSource; use agent_desktop_core::Deadline; @@ -18,8 +18,7 @@ fn budget() -> WalkBudget { WalkBudget::new(10, Deadline::standard().expect("a standard deadline")) } -/// The lemma the path tier trusts its landing on, and the reason a landing does -/// not have to wait on the rest of the walk: what a truncation removes is a +/// The lemma the anchor trusts its landing on: what a truncation removes is a /// suffix, so every index below it still names the child a whole walk would /// name there. A mid-list gap - the one shape that would make a landing a /// guess - is unconstructible, because the enumeration has no way to skip an @@ -27,86 +26,49 @@ fn budget() -> WalkBudget { /// /// A take stop is a truncation the walk chose: indices below it land on the /// same child a whole walk names, and a fault past it is masked - the walk -/// never got there. A fault within take still surfaces beside the landing, -/// so this pin covers both halves: identical landings with no unread claimed -/// where nothing faulted, and the unread region surviving where it did. +/// never got there. #[test] fn an_index_a_truncated_walk_reaches_names_the_child_a_whole_walk_names() { let short = StubTree::with_children(6).faulting_after(3); let whole = StubTree::with_children(6); for index in 0..2 { - let landing = walk_stored_path(&short, &0, &[index], &budget()) - .expect("a transport fault never surfaces for the search"); - let control = - walk_stored_path(&whole, &0, &[index], &budget()).expect("an unfaulting walk answers"); + let landing = descend_path(&short, &0, &[index], &budget(), &ANCHOR_DESCENT) + .expect("a fault past the take stop never surfaces"); + let control = descend_path(&whole, &0, &[index], &budget(), &ANCHOR_DESCENT) + .expect("an unfaulting walk answers"); assert_eq!( - landing.element, control.element, + landing, control, "index {index} sits below the take stop, so the short walk must land on the same \ child the whole walk lands on" ); - assert!( - !landing.unread_region, - "the short walk stopped before the fault it never reached, so it claims nothing \ - unread" - ); } - - let faulty = StubTree::with_children(6).faulting_after(1); - let landing = walk_stored_path(&faulty, &0, &[2], &budget()) - .expect("a transport fault never surfaces for the search"); - - assert_eq!( - landing.element, None, - "the fault inside take leaves index two unreached" - ); - assert!( - landing.unread_region, - "a fault the walk did reach still withholds the verdict" - ); } -/// The other half of the same property: the truncation is never crossed. An -/// index the read prefix does not reach lands nowhere rather than sliding onto -/// whatever sibling the enumeration did manage to read, so the fall-through to -/// the broad search - not a wrong element - is what a stored index past a gap -/// produces. +/// A take-hit keeps the truncated prefix and answers it, like a cap-hit: +/// every index the prefix reaches names the same child the whole list names +/// there. #[test] -fn an_index_past_a_truncation_lands_nowhere_rather_than_on_a_neighbour() { - let truncated = StubTree::with_children(6).faulting_after(3); - for index in 3..6 { - let landing = walk_stored_path(&truncated, &0, &[index], &budget()) - .expect("a transport fault never surfaces for the search"); - assert_eq!( - landing.element, None, - "index {index} is past the truncation and must land nowhere" - ); - assert!(landing.unread_region); - } +fn take_reports_a_truncated_prefix_whole() { + let tree = StubTree::with_children(100); + + let read = read_children(&tree, &0, &budget(), &ANCHOR_DESCENT, Some(2)) + .expect("a bounded prefix answers"); - let control = walk_stored_path(&StubTree::with_children(6), &0, &[5], &budget()) - .expect("an unfaulting walk answers"); - assert_eq!( - control.element, - Some(6), - "the same indices are reachable on a whole walk, so the pin above is not vacuous" - ); + assert_eq!(read.len(), 2); } -/// A tier whose eligibility gate declined to run it read nothing, so it has no -/// region of its own to withhold a verdict for. Reporting one would make every -/// ref that skips the path tier - anything with an empty stored path - retry -/// against a gap that no attempt can ever close. +/// An index past the end still lands nowhere, take or no take: exhaustion is +/// a complete answer about a list that simply ends. #[test] -fn a_path_tier_that_never_walked_lands_nowhere_and_claims_no_unread_region() { - let landing = PathLanding::::not_walked(); - - assert!(landing.element.is_none()); - assert!( - !landing.unread_region, - "a walk that never ran left nothing unread; the tiers below it still cover the tree" - ); +fn a_miss_past_the_end_still_lands_nowhere() { + let tree = StubTree::with_children(3); + + let landing = + descend_path(&tree, &0, &[5], &budget(), &ANCHOR_DESCENT).expect("exhaustion answers"); + + assert!(landing.is_none()); } struct Counting<'a> { @@ -176,9 +138,10 @@ fn descend_path_reaches_index_zero_with_a_single_call() { let tree = StubTree::with_children(100); let counting = Counting::over(&tree); - let landing = walk_stored_path(&counting, &0, &[0], &budget()).expect("a whole prefix answers"); + let landing = descend_path(&counting, &0, &[0], &budget(), &ANCHOR_DESCENT) + .expect("a whole prefix answers"); - assert_eq!(landing.element, Some(1)); + assert_eq!(landing, Some(1)); assert_eq!(counting.calls(), 1); } @@ -189,34 +152,9 @@ fn descend_path_costs_index_plus_one_calls_mid_list() { let tree = StubTree::with_children(100); let counting = Counting::over(&tree); - let landing = - walk_stored_path(&counting, &0, &[50], &budget()).expect("a whole prefix answers"); + let landing = descend_path(&counting, &0, &[50], &budget(), &ANCHOR_DESCENT) + .expect("a whole prefix answers"); - assert_eq!(landing.element, Some(51)); + assert_eq!(landing, Some(51)); assert_eq!(counting.calls(), 51); } - -/// A take-hit keeps the truncated prefix and still reports it whole, like a -/// cap-hit: the broad search re-covers the unseen remainder on its own pass. -#[test] -fn take_reports_a_truncated_prefix_whole() { - let tree = StubTree::with_children(100); - - let read = read_children(&tree, &0, &budget(), &SEARCH_DESCENT, Some(2)) - .expect("a bounded prefix answers"); - - assert_eq!(read.elements.len(), 2); - assert!(read.complete); -} - -/// An index past the end still lands nowhere, take or no take: exhaustion is -/// a complete answer about a list that simply ends. -#[test] -fn a_miss_past_the_end_still_lands_nowhere() { - let tree = StubTree::with_children(3); - - let landing = walk_stored_path(&tree, &0, &[5], &budget()).expect("exhaustion answers"); - - assert!(landing.element.is_none()); - assert!(!landing.unread_region); -} diff --git a/crates/windows/src/tree/resolve/resolve_match.rs b/crates/windows/src/tree/resolve/resolve_match.rs index 3587473ad..1612a1aef 100644 --- a/crates/windows/src/tree/resolve/resolve_match.rs +++ b/crates/windows/src/tree/resolve/resolve_match.rs @@ -1,127 +1,6 @@ -use agent_desktop_core::{ - AdapterError, LocatorEvidence, LocatorField, RefEntry, - refs::ref_identity::{has_stable_text_identity, identity_match, stable_text_match}, -}; +use agent_desktop_core::{AdapterError, RefEntry}; use serde_json::json; -use crate::tree::element::UIAElement; - -/// A matched live element plus the tie-break hash read off its evidence. -#[derive(Clone)] -pub(crate) struct Candidate { - pub(crate) element: UIAElement, - pub(crate) bounds_hash: Option, -} - -/// The three-state outcome the composed rule demands: a candidate either -/// matched, was refuted, or **could not be read** - and an unreadable -/// candidate makes the attempt incomplete-and-retryable rather than silently -/// non-matching. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub(crate) enum CandidateOutcome { - /// The candidate is the stored element. - Matched, - /// A read field contradicts the stored evidence; the candidate is not it. - Refuted, - /// A field that decides the question could not be read. Never a `NoMatch`. - Incomplete, -} - -/// The composed rule over core's verdicts. -/// -/// `identity_match` settles the id tier for an id-bearing ref, and by itself -/// would settle on an id hit without consulting text - the exact behaviour -/// A7-3 measured as silently wrong on Windows (Explorer re-resolving 5 of 29 -/// index-keyed refs onto a different element). The corroboration tier runs -/// only when the stored ref carries a stable text identity -/// (`has_stable_text_identity`): an id `Match` + stable-text `NoMatch` is -/// refuted, and a `NoMatch` never collapses an `Unknown`. The tier keys the -/// incomplete flag on the verdict itself: an `IdentityMatch::Unknown`, which -/// core produces when a decisive field reads `LocatorField::Unknown` (a read -/// that failed), becomes `Incomplete`. A structurally absent field is not -/// `Unknown` and settles `NoMatch` instead, because `match_primary_identity` -/// returns `NoMatch` when both the primary and fallback fields are `Absent`. An -/// id `Match` on a ref with no stored text identity (an icon-only button) -/// resolves without corroboration, preserving the previous behaviour. An id-less ref -/// is routed through `identity_match`, which performs the text match itself. -pub(crate) fn candidate_outcome(entry: &RefEntry, evidence: &LocatorEvidence) -> CandidateOutcome { - let id_tier = identity_match( - entry, - &evidence.name, - &evidence.value, - &evidence.description, - &evidence.identifiers, - ); - if entry.identity.native_id.is_some() - && id_tier == agent_desktop_core::IdentityMatch::Match - && has_stable_text_identity(entry) - { - return outcome_of(stable_text_match( - entry, - &evidence.name, - &evidence.value, - &evidence.description, - )); - } - outcome_of(id_tier) -} - -/// The one mapping from a core verdict to this resolver's outcome. Written -/// once so the two tiers cannot drift apart: an unreadable field is always -/// `Incomplete`, never a refutation. -fn outcome_of(verdict: agent_desktop_core::IdentityMatch) -> CandidateOutcome { - match verdict { - agent_desktop_core::IdentityMatch::Match => CandidateOutcome::Matched, - agent_desktop_core::IdentityMatch::NoMatch => CandidateOutcome::Refuted, - agent_desktop_core::IdentityMatch::Unknown => CandidateOutcome::Incomplete, - } -} - -/// Derives the tie-break hash from the walk-shaped evidence slot. -pub(crate) fn bounds_hash_of(evidence: &LocatorEvidence) -> Option { - match evidence.ref_evidence.bounds { - LocatorField::Known(rect) => rect.bounds_hash(), - LocatorField::Absent | LocatorField::Unknown => None, - } -} - -/// What the 2+-candidate branch decided. -pub(crate) enum Selection { - /// Exactly one candidate matched the stored bounds hash; its index in the - /// searched candidate list. - Resolved(usize), - /// No stored hash, no candidate matched it, or more than one did. - Ambiguous, -} - -/// Breaks a tie among several evidence-equal candidates by bounds hash. -/// -/// Pure over the hashes alone, not the elements, so the decision is testable -/// without a live COM element in scope. The bounds hash never refutes an -/// exact match; it only ever picks among several - a stored ref with no hash -/// (bounds failed at capture, or hidden by the requester) cannot be -/// disambiguated this way, so it stays ambiguous rather than guessing. -pub(crate) fn select_by_bounds_hash( - candidate_hashes: &[Option], - expected_hash: Option, -) -> Selection { - let Some(expected_hash) = expected_hash else { - return Selection::Ambiguous; - }; - let mut sole = None; - let mut matches = 0; - for (index, hash) in candidate_hashes.iter().enumerate() { - if *hash == Some(expected_hash) { - matches += 1; - sole = Some(index); - } - } - match (matches, sole) { - (1, Some(index)) => Selection::Resolved(index), - _ => Selection::Ambiguous, - } -} - pub(crate) fn stale_ref_error(_entry: &RefEntry) -> AdapterError { let error = AdapterError::stale_ref_because("Stored ref does not match any live element"); let default_retryable = error.permits_retry_by_default(); @@ -150,32 +29,6 @@ pub(crate) fn owning_process_exited_error(_entry: &RefEntry) -> AdapterError { })) } -/// The settled `AMBIGUOUS_TARGET`, stamped on both retry axes like every -/// other resolver verdict. -/// -/// `complete` is true because every path that reaches here has a conclusive -/// answer: `classify_search` withholds the verdict as `Incomplete` whenever an -/// unread region could still change it, and the one case it lets through with -/// an unread region (`should_stop_collecting`: two or more matches and no -/// stored bounds hash to separate them) cannot be resolved by any further -/// candidate, only made more ambiguous. -/// -/// `retryable` is false because nothing about a re-attempt differs: the stored -/// evidence is fixed, and the search that produced several equally-matching -/// candidates is deterministic over the same tree. The stamp keeps that -/// verdict out of core's poll loop, which retries a resolution failure only -/// when it is explicitly retryable. -pub(crate) fn ambiguous_target_error(_entry: &RefEntry, count: usize) -> AdapterError { - AdapterError::ambiguous_target("Multiple live elements match the stored identity") - .with_suggestion("Take a fresh snapshot and use its new ref") - .with_details(json!({ - "kind": "resolve_ambiguous", - "candidate_count": count, - "complete": true, - "retryable": false, - })) -} - #[cfg(test)] #[path = "resolve_match_tests.rs"] mod tests; diff --git a/crates/windows/src/tree/resolve/resolve_match_tests.rs b/crates/windows/src/tree/resolve/resolve_match_tests.rs index e23338d09..d94ba3ae2 100644 --- a/crates/windows/src/tree/resolve/resolve_match_tests.rs +++ b/crates/windows/src/tree/resolve/resolve_match_tests.rs @@ -1,202 +1,19 @@ use super::*; -use agent_desktop_core::{ - DeliverySemantics, ElementIdentifier, ErrorCode, IdentifierEvidence, IdentifierKind, - NodeDescriptor, -}; +use agent_desktop_core::{DeliverySemantics, ElementIdentifier, ErrorCode, IdentifierKind}; -fn field(value: Option<&str>) -> LocatorField { - match value { - Some(value) => LocatorField::Known(value.to_string()), - None => LocatorField::Absent, - } -} - -fn evidence( - name: Option<&str>, - value: Option<&str>, - description: Option<&str>, - native_id: Option<&str>, -) -> LocatorEvidence { - LocatorEvidence { - role: LocatorField::Known("button".to_string()), - name: field(name), - value: field(value), - description: field(description), - identifiers: IdentifierEvidence::typed( - native_id.into_iter().map(|value| ElementIdentifier { - kind: IdentifierKind::AutomationId, - value: value.to_string(), - }), - None, - true, - ), - states: LocatorField::Absent, - ref_evidence: agent_desktop_core::LocatorRefEvidence { - window_bounds: LocatorField::Absent, - bounds: LocatorField::Absent, - available_actions: LocatorField::Absent, - descriptors: NodeDescriptor::default(), - }, - } -} - -fn entry( - native: Option<&str>, - role: &str, - name: Option<&str>, - value: Option<&str>, - description: Option<&str>, -) -> RefEntry { - let mut entry = crate::tree::walk::walker_fake::ref_entry(role); - entry.identity.name = name.map(str::to_string); - entry.identity.value = value.map(str::to_string); - entry.identity.description = description.map(str::to_string); - entry.identity.native_id = native.map(|value| ElementIdentifier { +fn entry() -> RefEntry { + let mut entry = crate::tree::walk::walker_fake::ref_entry("button"); + entry.identity.name = Some("Label".into()); + entry.identity.native_id = Some(ElementIdentifier { kind: IdentifierKind::AutomationId, - value: value.to_string(), + value: "btn-1".into(), }); entry } -/// The A7-3 silent-wrong-target pin under the composed matcher: a -/// candidate sharing the stored `native_id` and role whose stable name -/// drifted is refuted, not matched. Weakening the corroboration to -/// id-only would let the wrong element resolve - the exact shape A7-3 -/// measured on Explorer. -#[test] -fn a_matching_native_id_with_a_stable_name_drift_is_refuted() { - let stored = entry(Some("row-77"), "row", Some("Alpha"), None, None); - let drifted = evidence(Some("Beta"), None, None, Some("row-77")); - - assert_eq!( - candidate_outcome(&stored, &drifted), - CandidateOutcome::Refuted - ); - let matching = evidence(Some("Alpha"), None, None, Some("row-77")); - assert_eq!( - candidate_outcome(&stored, &matching), - CandidateOutcome::Matched - ); -} - -/// A candidate whose stable-text field could not be read is **incomplete**, -/// never refuted - the test fails if `Unknown` is collapsed into `NoMatch`. -#[test] -fn an_unreadable_name_is_incomplete_not_a_mismatch() { - let stored = entry(Some("btn-1"), "button", Some("Old Label"), None, None); - let unreadable = LocatorEvidence { - name: LocatorField::Unknown, - ..evidence(None, None, None, Some("btn-1")) - }; - - assert_eq!( - candidate_outcome(&stored, &unreadable), - CandidateOutcome::Incomplete - ); -} - -/// An id `Match` on a ref with no stored text identity (an icon-only -/// button) resolves without corroboration - the previous behaviour preserved. -#[test] -fn an_id_match_without_stored_text_resolves_without_corroboration() { - let stored = entry(Some("icon-1"), "button", None, None, None); - let live = evidence(None, None, None, Some("icon-1")); - - assert_eq!(candidate_outcome(&stored, &live), CandidateOutcome::Matched); -} - -/// A mutable-value role whose name mirrors its current value has no stable -/// text identity by core's own `stable_name` rule (name == value is -/// dropped as volatile), so an id `Match` resolves without corroboration - -/// the prior pin, carried through core rather than re-implemented on Windows. -/// The realistic textfield shape: stored name and value agree, the live -/// candidate's drifted name mirrors its own new value. -#[test] -fn a_mutable_value_role_with_a_drifted_live_name_still_matches_on_native_id() { - let stored = entry( - Some("field-1"), - "textfield", - Some("Old Value"), - Some("Old Value"), - None, - ); - let live = evidence(Some("New Value"), Some("New Value"), None, Some("field-1")); - - assert_eq!(candidate_outcome(&stored, &live), CandidateOutcome::Matched); -} - -/// The same mutable-value role with a **stable** name (a labelled field -/// whose name does not mirror the value) keeps the corroboration tier: a -/// drifted name refutes even with a matching id, exactly as a stable-role -/// name would. The composition never special-cases mutable roles itself. -#[test] -fn a_mutable_value_role_with_a_stable_label_refutes_a_drifted_name() { - let stored = entry( - Some("field-2"), - "textfield", - Some("First Name"), - Some("Alice"), - None, - ); - let drifted = evidence( - Some("First Name Is Drifted"), - Some("Alice"), - None, - Some("field-2"), - ); - - assert_eq!( - candidate_outcome(&stored, &drifted), - CandidateOutcome::Refuted - ); -} - -/// A ref with no id and no stored text, against a blank live candidate, -/// is not-known rather than a confident non-match: core's -/// `empty_identity_match` reports `Unknown` (incomplete) instead of -/// inventing a `NoMatch`. The corroboration tier must survive the switch. -#[test] -fn a_blank_ref_and_a_blank_candidate_are_incomplete_not_refuted() { - let stored = entry(None, "", None, None, None); - let blank_live = evidence(None, None, None, None); - - assert_eq!( - candidate_outcome(&stored, &blank_live), - CandidateOutcome::Incomplete - ); -} - -/// A ref whose stored name was blank cannot be refuted by a blank live -/// candidate when the id still matches - blank-cannot-refute, pinned on -/// the id-bearing side. -#[test] -fn a_blank_stored_name_does_not_refute_a_matching_id() { - let stored = entry(Some("row-1"), "listitem", None, None, None); - let live = evidence(None, None, None, Some("row-1")); - - assert_eq!(candidate_outcome(&stored, &live), CandidateOutcome::Matched); -} - -/// A candidate whose native identifier cannot be read is incomplete: the -/// id tier cannot decide it, and an unreadable field is never a `NoMatch`. -#[test] -fn an_unreadable_native_id_is_incomplete() { - let stored = entry(Some("btn-1"), "button", Some("Label"), None, None); - let unreadable_id = LocatorEvidence { - identifiers: IdentifierEvidence::unknown(), - ..evidence(Some("Label"), None, None, None) - }; - - assert_eq!( - candidate_outcome(&stored, &unreadable_id), - CandidateOutcome::Incomplete - ); -} - #[test] fn a_zero_candidate_resolution_is_stale_windows() { - let stored = entry(Some("btn-1"), "button", Some("Label"), None, None); - assert_eq!(stale_ref_error(&stored).code, ErrorCode::StaleRef); + assert_eq!(stale_ref_error(&entry()).code, ErrorCode::StaleRef); } /// The settled stale ref sits on two different retry axes at once, and a @@ -208,8 +25,7 @@ fn a_zero_candidate_resolution_is_stale_windows() { #[cfg(target_os = "windows")] #[test] fn a_settled_stale_ref_is_query_retryable_complete_and_not_adapter_loop_retryable() { - let stored = entry(Some("btn-1"), "button", Some("Label"), None, None); - let error = stale_ref_error(&stored); + let error = stale_ref_error(&entry()); assert!( error.is_explicitly_retryable(), @@ -240,8 +56,7 @@ fn a_settled_stale_ref_is_query_retryable_complete_and_not_adapter_loop_retryabl /// either is invisible from the message alone. #[test] fn the_settled_stale_ref_message_names_the_evidence_mismatch_not_a_missing_refmap_entry() { - let stored = entry(Some("btn-1"), "button", Some("Label"), None, None); - let error = stale_ref_error(&stored); + let error = stale_ref_error(&entry()); assert_eq!(error.message, "Stored ref does not match any live element"); assert!( @@ -259,70 +74,3 @@ fn the_settled_stale_ref_message_names_the_evidence_mismatch_not_a_missing_refma "the derived retryability must survive the construction change" ); } - -/// The ambiguous verdict sits on the same two retry axes as its stale sibling -/// and used to stamp neither, so a regression on either was invisible. It is -/// complete because `classify_search` only reaches this arm with a conclusive -/// answer, and non-retryable because the same search over the same stored -/// evidence cannot separate the candidates on a second attempt - the stamp is -/// what keeps it out of core's resolution poll loop, which retries only an -/// explicitly retryable failure. -#[test] -fn an_ambiguous_target_is_complete_and_settled_rather_than_retried() { - let stored = entry(Some("btn-1"), "button", Some("Label"), None, None); - let error = ambiguous_target_error(&stored, 2); - - assert_eq!(error.code, ErrorCode::AmbiguousTarget); - assert_eq!(flag(&error, "complete"), Some(true)); - assert_eq!(flag(&error, "retryable"), Some(false)); - assert!(!error.is_explicitly_retryable()); - assert!(!error.permits_retry_by_default()); - assert_eq!( - error - .details - .as_ref() - .and_then(|details| details.get("candidate_count")) - .and_then(serde_json::Value::as_u64), - Some(2), - "the shape-only candidate count must survive the added stamps" - ); -} - -fn flag(error: &agent_desktop_core::AdapterError, key: &str) -> Option { - error - .details - .as_ref() - .and_then(|details| details.get(key)) - .and_then(serde_json::Value::as_bool) -} - -/// Two candidates carrying the same evidence, neither distinguished by -/// hash, must stay ambiguous - the caller's `AMBIGUOUS_TARGET`, not a -/// guess at either one. -#[test] -fn two_identical_evidence_candidates_stay_ambiguous() { - let selection = select_by_bounds_hash(&[Some(1), Some(1)], Some(1)); - - assert!(matches!(selection, Selection::Ambiguous)); -} - -/// Exactly one of several candidates carries the stored bounds hash: that -/// one resolves. The expected hash sits at index 1 on purpose, so a -/// first-match-wins regression (picking index 0 regardless of hash) would -/// fail this assertion rather than pass it by accident. -#[test] -fn the_sole_candidate_matching_the_stored_hash_resolves() { - let selection = select_by_bounds_hash(&[Some(2), Some(1)], Some(1)); - - assert!(matches!(selection, Selection::Resolved(1))); -} - -/// A stored ref with no bounds hash cannot be disambiguated by hash at -/// all, so several candidates stay ambiguous rather than one being picked -/// by a `None == None` coincidence. -#[test] -fn no_stored_hash_cannot_disambiguate() { - let selection = select_by_bounds_hash(&[None, None], None); - - assert!(matches!(selection, Selection::Ambiguous)); -} diff --git a/crates/windows/src/tree/resolve/resolve_pair_window.rs b/crates/windows/src/tree/resolve/resolve_pair_window.rs deleted file mode 100644 index 635ca40fd..000000000 --- a/crates/windows/src/tree/resolve/resolve_pair_window.rs +++ /dev/null @@ -1,232 +0,0 @@ -//! A hosted Win32 window presenting two evidence-equal children, shared by -//! the resolver's multi-candidate tests. -//! -//! The pair is the reproducible way an owned fixture presents duplicate -//! evidence (A17-3): both children carry the same class, the same text and the -//! same control id, so the UI Automation bridge publishes the same role, the -//! same `Name` and the same `AutomationId` for both. Only their rectangles are -//! the caller's to choose, which is what decides whether the bounds tie-break -//! has two equal hashes it cannot separate or two distinct ones it must pick -//! between. - -use std::ffi::c_void; -use std::sync::mpsc::{Sender, channel}; -use std::thread::JoinHandle; -use std::time::{Duration, Instant}; - -use agent_desktop_core::{ElementIdentifier, IdentifierKind, LocatorEvidence}; -use windows_sys::Win32::Foundation::HWND; -use windows_sys::Win32::System::LibraryLoader::GetModuleHandleW; -use windows_sys::Win32::System::Threading::GetCurrentThreadId; -use windows_sys::Win32::UI::WindowsAndMessaging::{ - CreateWindowExW, DestroyWindow, DispatchMessageW, GetMessageW, MSG, PostThreadMessageW, - SW_SHOWNOACTIVATE, ShowWindow, TranslateMessage, WM_QUIT, WS_CHILD, WS_OVERLAPPEDWINDOW, - WS_VISIBLE, -}; - -use crate::system::cursor_overlay::wide::wide; -use crate::tree::element::UIAElement; -use crate::tree::walk::walker::WalkBudget; -use crate::tree::walk::walker_source::UiaTreeSource; - -/// Application-derived text the host puts on the wire so a redaction -/// regression has something to leak. A marker reaching an error's message, -/// suggestion, platform detail or details is app text the resolver had no -/// right to echo. -pub(crate) const NAME_MARKER: &str = "zzduplicatenamezz"; -pub(crate) const TITLE_MARKER: &str = "zzduplicatetitlezz"; -pub(crate) const APP_MARKER: &str = "zzduplicateappzz.exe"; - -/// The control id both children share, so the UI Automation bridge hands -/// both the same `AutomationId`. -const SHARED_CONTROL_ID: usize = 31337; -const CONTROL_BORDER: u32 = 0x0080_0000; -const CONTROL_LEFT: i32 = 8; -const CONTROL_TOP: i32 = 8; -const CONTROL_WIDTH: i32 = 200; -const CONTROL_HEIGHT: i32 = 24; -const CONTROL_GAP: i32 = 40; -const PAIR_WINDOW_WIDTH: i32 = 420; -const PAIR_WINDOW_HEIGHT: i32 = 220; -const READY_TIMEOUT: Duration = Duration::from_secs(30); -const WALKABLE_TIMEOUT: Duration = Duration::from_secs(20); -const WALKABLE_POLL: Duration = Duration::from_millis(100); - -/// Where the pair's second child sits relative to the first. -#[derive(Clone, Copy)] -pub(crate) enum PairGeometry { - /// One rectangle for both: the bounds tie-break is handed two equal - /// hashes and no tier can separate the pair. - Coincident, - /// Two rectangles: the bounds tie-break is the only tier that separates - /// the pair, and it must pick the stored one rather than the first. - Separated, -} - -impl PairGeometry { - fn second_top(self) -> i32 { - match self { - Self::Coincident => CONTROL_TOP, - Self::Separated => CONTROL_TOP + CONTROL_HEIGHT + CONTROL_GAP, - } - } -} - -/// A window whose two children are indistinguishable to every identity tier: -/// same class, same text, same shared control id. -pub(crate) struct DuplicatePairWindow { - pub(crate) handle: isize, - thread_id: u32, - host: Option>, -} - -impl DuplicatePairWindow { - pub(crate) fn create(geometry: PairGeometry) -> Result { - let (sender, receiver) = channel(); - let host = std::thread::spawn(move || host_duplicate_pair(sender, geometry)); - let (handle, thread_id) = receiver - .recv_timeout(READY_TIMEOUT) - .map_err(|_| String::from("the duplicate-pair host never reported a window"))?; - let hosted = Self { - handle, - thread_id, - host: Some(host), - }; - await_walkable(handle)?; - Ok(hosted) - } -} - -impl Drop for DuplicatePairWindow { - fn drop(&mut self) { - unsafe { PostThreadMessageW(self.thread_id, WM_QUIT, 0, 0) }; - if let Some(host) = self.host.take() { - let _ = host.join(); - } - } -} - -/// Creates the window and its duplicate pair on this thread and pumps until -/// told to quit. -/// -/// The pump has to own the window and do nothing else: `ElementFromHandle` -/// sends `WM_GETOBJECT`, and a cross-thread `SendMessage` blocks until the -/// owning thread dispatches it, so a thread that both hosts the window and -/// waits on a UI Automation result deadlocks. Teardown posts `WM_QUIT` to the -/// thread rather than the window, so the window is destroyed by the thread -/// that created it. -fn host_duplicate_pair(ready: Sender<(isize, u32)>, geometry: PairGeometry) { - let class = wide("#32770"); - let title = wide(TITLE_MARKER); - let stage = - crate::tree::fixture::offscreen_origin::stage(None, PAIR_WINDOW_WIDTH, PAIR_WINDOW_HEIGHT); - let (left, top) = stage.origin(); - let window = unsafe { - CreateWindowExW( - 0, - class.as_ptr(), - title.as_ptr(), - WS_OVERLAPPEDWINDOW, - left, - top, - PAIR_WINDOW_WIDTH, - PAIR_WINDOW_HEIGHT, - std::ptr::null_mut(), - std::ptr::null_mut(), - GetModuleHandleW(std::ptr::null()), - std::ptr::null(), - ) - }; - if window.is_null() { - return; - } - duplicate_control(window, CONTROL_TOP); - duplicate_control(window, geometry.second_top()); - unsafe { ShowWindow(window, SW_SHOWNOACTIVATE) }; - if ready - .send((window as isize, unsafe { GetCurrentThreadId() })) - .is_err() - { - unsafe { DestroyWindow(window) }; - return; - } - let mut message = MSG::default(); - while unsafe { GetMessageW(&mut message, std::ptr::null_mut(), 0, 0) } > 0 { - unsafe { TranslateMessage(&message) }; - unsafe { DispatchMessageW(&message) }; - } - unsafe { DestroyWindow(window) }; -} - -fn duplicate_control(parent: HWND, top: i32) { - let class = wide("BUTTON"); - let text = wide(NAME_MARKER); - unsafe { - CreateWindowExW( - 0, - class.as_ptr(), - text.as_ptr(), - WS_CHILD | WS_VISIBLE | CONTROL_BORDER, - CONTROL_LEFT, - top, - CONTROL_WIDTH, - CONTROL_HEIGHT, - parent, - SHARED_CONTROL_ID as *mut c_void, - GetModuleHandleW(std::ptr::null()), - std::ptr::null(), - ) - }; -} - -/// Blocks until the window resolves to a UI Automation root, so a loaded -/// runner's first `WM_GETOBJECT` timing out is a wait rather than a failure. -fn await_walkable(handle: isize) -> Result<(), String> { - let expiry = Instant::now() + WALKABLE_TIMEOUT; - let mut last = String::from("the duplicate-pair window never resolved"); - while Instant::now() < expiry { - let deadline = - agent_desktop_core::Deadline::standard().map_err(|error| error.message.clone())?; - match crate::tree::automation::root_from_hwnd(handle, deadline) { - Ok(_) => return Ok(()), - Err(error) => last = error.message.clone(), - } - std::thread::sleep(WALKABLE_POLL); - } - Err(last) -} - -/// Collects the marked controls in the order the resolver's own search visits -/// them - same enumeration, same recursion - so a candidate's index here is -/// the index the search verdict will report. -pub(crate) fn collect_marked( - source: &UiaTreeSource, - element: &UIAElement, - budget: &WalkBudget, - out: &mut Vec, -) { - crate::tree::walk::walker_fake::walk_all( - source, - element, - budget, - 8, - &mut |_prefix, evidence| { - if evidence - .name - .known() - .is_some_and(|name| name == NAME_MARKER) - { - out.push(evidence); - } - }, - ); -} - -pub(crate) fn automation_id(evidence: &LocatorEvidence) -> Option { - evidence - .identifiers - .identifiers() - .iter() - .find(|identifier| matches!(identifier.kind, IdentifierKind::AutomationId)) - .cloned() -} diff --git a/crates/windows/src/tree/resolve/resolve_path_fallback_tests.rs b/crates/windows/src/tree/resolve/resolve_path_fallback_tests.rs deleted file mode 100644 index 1c77de1c9..000000000 --- a/crates/windows/src/tree/resolve/resolve_path_fallback_tests.rs +++ /dev/null @@ -1,229 +0,0 @@ -use crate::tree::resolve::*; -use agent_desktop_core::{LocatorEvidence, refs::RefPath}; - -/// The fixture control the stored ref names. Its role is shared with the -/// window's title-bar buttons, which is what makes a same-role decoy -/// available without the role gate deciding the case first. -const TARGET_NAME: &str = "fixture-button"; - -fn catalogue( - source: &UiaTreeSource, - element: &UIAElement, - budget: &WalkBudget, - out: &mut Vec<(RefPath, LocatorEvidence)>, -) { - crate::tree::walk::walker_fake::walk_all( - source, - element, - budget, - 8, - &mut |prefix, evidence| { - let mut path = RefPath::default(); - path.extend_from_slice(prefix); - out.push((path, evidence)); - }, - ); -} - -fn stored_entry( - fixture_handle: isize, - pid: u32, - evidence: &LocatorEvidence, - path: RefPath, -) -> RefEntry { - let pid = agent_desktop_core::ProcessId::from(pid); - let mut entry = crate::tree::walk::walker_fake::ref_entry( - &evidence.role.known().cloned().unwrap_or_default(), - ); - entry.process = agent_desktop_core::RefProcess { - pid, - process_instance: Some( - crate::system::process_identity::token_for_pid(pid) - .expect("the token read answers") - .expect("a live fixture process has a token"), - ), - }; - entry.identity.name = evidence.name.known().cloned(); - entry.source.source_app = Some("fixture.exe".into()); - entry.source.source_window_id = Some(format!("w-{fixture_handle}")); - entry.scope.path_is_absolute = true; - entry.scope.path = path; - entry -} - -/// The path tier's positive half, driven against a live tree, and the shape a -/// static reading of the resolver mistook for unsound: the tier settles on the -/// element the stored path names, from that element's own evidence and nothing -/// else. No completeness of the surrounding walk is in scope for it - the -/// signature has no room for one - because a landing is the child the stored -/// index names whatever else an enumeration failed to read. What a truncation -/// removes is a suffix, and it lies past the index the walk already used. -/// -/// Both directions are pinned on the same live elements: the stored ref's own -/// landing settles, and a same-role landing the stored identity refutes does -/// not. A tier that accepted everything, or nothing, fails one of the two. -#[test] -fn the_path_tier_settles_on_the_element_the_stored_path_names_and_on_no_other() { - crate::tree::fixture::ensure_test_apartment(); - let fixture = crate::tree::fixture::HostedFixture::spawn().expect("a fixture host starts"); - let deadline = crate::tree::walk::walker_fake::deadline(); - let root = crate::tree::automation::root_from_hwnd(fixture.handle(), deadline) - .expect("the fixture window resolves"); - let source = UiaTreeSource::for_root(&root).expect("a tree source"); - let prepared = source.prepare_root(&root).expect("a prepared root"); - let budget = WalkBudget::new(10, deadline); - - let mut catalogued = Vec::new(); - catalogue(&source, &prepared, &budget, &mut catalogued); - - let (target_path, target) = catalogued - .iter() - .find(|(_, evidence)| { - evidence - .name - .known() - .is_some_and(|name| name == TARGET_NAME) - }) - .cloned() - .expect("the fixture exposes its named button"); - let (decoy_path, _) = catalogued - .iter() - .find(|(path, evidence)| { - path != &target_path - && evidence.role.known() == target.role.known() - && evidence.name.known() != target.name.known() - }) - .cloned() - .expect("the fixture exposes a same-role control with a different identity"); - - let entry = stored_entry( - fixture.handle(), - fixture.process_id(), - &target, - target_path.clone(), - ); - let land = |path: &RefPath| { - crate::tree::resolve::resolve_search::walk_stored_path(&source, &prepared, path, &budget) - .expect("the path walk answers") - .element - .expect("the stored path still lands on a live element") - }; - - assert!( - crate::tree::resolve::resolve_search::accept_path_landing( - &source, - &land(&target_path), - &entry - ) - .is_some(), - "the tier must settle on the element the stored path names" - ); - assert!( - crate::tree::resolve::resolve_search::accept_path_landing( - &source, - &land(&decoy_path), - &entry - ) - .is_none(), - "a same-role landing the stored identity refutes is never this tier's answer" - ); -} - -/// The path fast-path's fall-through, driven against a live tree. -/// -/// The stored child-index path still lands on a live element - so this is -/// not the path-missed case - but the element now occupying that index is a -/// different one of the same role, which the identity evidence refutes. A -/// refutation at the stored index is evidence about that element only, never -/// about the stored one, so resolution must continue into the broad search -/// and find the real target elsewhere in the tree. Settling `STALE_REF` there -/// instead would strand every ref whose siblings were reordered. -#[test] -fn a_refuted_element_at_the_stored_path_falls_through_to_the_broad_search() { - crate::tree::fixture::ensure_test_apartment(); - let fixture = crate::tree::fixture::HostedFixture::spawn().expect("a fixture host starts"); - let deadline = crate::tree::walk::walker_fake::deadline(); - let root = crate::tree::automation::root_from_hwnd(fixture.handle(), deadline) - .expect("the fixture window resolves"); - let source = UiaTreeSource::for_root(&root).expect("a tree source"); - let prepared = source.prepare_root(&root).expect("a prepared root"); - let budget = WalkBudget::new(10, deadline); - - let mut catalogued = Vec::new(); - catalogue(&source, &prepared, &budget, &mut catalogued); - - let (target_path, target) = catalogued - .iter() - .find(|(_, evidence)| { - evidence - .name - .known() - .is_some_and(|name| name == TARGET_NAME) - }) - .cloned() - .expect("the fixture exposes its named button"); - let (decoy_path, decoy) = catalogued - .iter() - .find(|(path, evidence)| { - path != &target_path - && evidence.role.known() == target.role.known() - && evidence.name.known() != target.name.known() - }) - .cloned() - .expect("the fixture exposes a same-role control with a different identity"); - - let entry = stored_entry( - fixture.handle(), - fixture.process_id(), - &target, - decoy_path.clone(), - ); - assert!( - crate::tree::resolve::resolve_search::can_use_path_fast_path(&entry), - "the fast path must actually run, or the fall-through is never reached" - ); - - let landed = crate::tree::resolve::resolve_search::walk_stored_path( - &source, - &prepared, - &entry.scope.path, - &budget, - ) - .expect("the path walk answers") - .element - .expect("the stored path still lands on a live element"); - let (_, landed_evidence, _) = source.evidence(&landed); - assert_eq!( - landed_evidence.role.known(), - target.role.known(), - "the landing shares the stored role, so the role gate is not what rejects it" - ); - assert_eq!( - crate::tree::resolve::resolve_match::candidate_outcome(&entry, &landed_evidence), - CandidateOutcome::Refuted, - "the element at the stored index is refuted by the stored identity" - ); - - let handle = resolve_element_strict(&entry, deadline) - .expect("a refutation at the stored path must not settle the resolution"); - let resolved = handle - .downcast_ref::() - .expect("the resolved handle carries a UI Automation element"); - let (_, resolved_evidence, _) = source.evidence(resolved); - - assert_eq!( - resolved_evidence.name.known(), - target.name.known(), - "the broad search found the stored element" - ); - assert_eq!( - resolved_evidence.ref_evidence.bounds.known(), - target.ref_evidence.bounds.known(), - "the resolved element is the stored one, not the decoy at the stored path" - ); - assert_ne!( - resolved_evidence.ref_evidence.bounds.known(), - decoy.ref_evidence.bounds.known(), - "the decoy and the target are genuinely different elements" - ); -} diff --git a/crates/windows/src/tree/resolve/resolve_retry_tests.rs b/crates/windows/src/tree/resolve/resolve_retry_tests.rs index 075865bd5..4ed38a291 100644 --- a/crates/windows/src/tree/resolve/resolve_retry_tests.rs +++ b/crates/windows/src/tree/resolve/resolve_retry_tests.rs @@ -187,127 +187,3 @@ fn the_deadline_stamp_leaks_no_marker_into_message_or_platform_detail() { Some(true) ); } - -fn entry_with_hash(bounds_hash: Option) -> RefEntry { - let mut entry = crate::tree::walk::walker_fake::ref_entry("button"); - entry.geometry.bounds_hash = bounds_hash; - entry.scope.path_is_absolute = true; - entry -} - -/// A sole match found while another region of the tree was unreadable must -/// not resolve - the search is not yet conclusive, and the true target (or a -/// second candidate that would make it ambiguous) could be sitting in the -/// unread part. Fails against the pre-fix ordering, which consulted -/// `incomplete` only in the zero-candidate arm. -#[test] -fn one_match_with_an_incomplete_region_is_retried_not_resolved() { - let entry = entry_with_hash(None); - let verdict = classify_search(&[None], true, &entry); - assert!(matches!(verdict, SearchVerdict::Incomplete)); -} - -/// The same sole match resolves once the search completed without anything -/// left unread. -#[test] -fn one_match_with_a_complete_search_resolves() { - let entry = entry_with_hash(None); - let verdict = classify_search(&[None], false, &entry); - assert!(matches!(verdict, SearchVerdict::Resolved(0))); -} - -/// Two or more matches with no stored hash to disambiguate is conclusively -/// ambiguous already, so an incomplete region elsewhere must not withhold -/// that verdict: nothing left unread could turn it into anything but -/// `AMBIGUOUS_TARGET`. Guards that gating resolution on `incomplete` does -/// not also gate this already-settled case. -#[test] -fn two_matches_with_no_hash_classify_as_ambiguous_even_when_incomplete() { - let entry = entry_with_hash(None); - let verdict = classify_search(&[None, None], true, &entry); - assert!(matches!(verdict, SearchVerdict::Ambiguous)); -} - -/// A 2+-candidate case with a stored hash never reaches the early-stop -/// threshold (`should_stop_collecting` requires no stored hash), so an -/// incomplete region still withholds the verdict rather than tie-breaking -/// on a partial candidate set. -#[test] -fn two_matches_with_a_stored_hash_and_an_incomplete_region_are_retried() { - let entry = entry_with_hash(Some(0xAAAA)); - let verdict = classify_search(&[Some(0xAAAA), Some(0xBBBB)], true, &entry); - assert!(matches!(verdict, SearchVerdict::Incomplete)); -} - -/// The 2+-candidate tie-break still classifies normally - resolving on the -/// sole hash match - once the search is complete, even though a stored hash -/// was present and the early-stop threshold was never reached. -#[test] -fn two_matches_with_a_stored_hash_and_a_complete_search_still_classify() { - let entry = entry_with_hash(Some(0xAAAA)); - let resolved = classify_search(&[Some(0xAAAA), Some(0xBBBB)], false, &entry); - assert!(matches!(resolved, SearchVerdict::Resolved(0))); - - let ambiguous = classify_search(&[Some(0xAAAA), Some(0xAAAA)], false, &entry); - assert!(matches!(ambiguous, SearchVerdict::Ambiguous)); -} - -/// An entry with no id, no stable text, and no positive-area bounds hash can -/// never be verified by any tier. The fixture's secure password control -/// structurally withholds its name, value, and description even on a fully -/// successful read - a genuine live candidate that core's identity matcher -/// can only ever report `Unknown` against. A stored ref that also carries no -/// distinguishing geometry can never settle a match against it, so -/// `resolve_attempt` must settle `STALE_REF` in exactly one attempt rather -/// than searching this tree and burning the retry budget on a candidate -/// that reads back `Unknown` every time. -#[test] -fn a_structurally_unverifiable_entry_settles_stale_in_one_attempt() { - crate::tree::fixture::ensure_test_apartment(); - let fixture = crate::tree::fixture::HostedFixture::spawn().expect("a fixture host starts"); - let deadline = crate::tree::walk::walker_fake::deadline(); - let root = crate::tree::automation::root_from_hwnd(fixture.handle(), deadline) - .expect("the fixture window resolves"); - let source = UiaTreeSource::for_root(&root).expect("a tree source"); - let prepared = source.prepare_root(&root).expect("a prepared root"); - let budget = WalkBudget::new(10, deadline); - let found = find_password(&source, &prepared, &budget) - .expect("the fixture walk succeeds") - .expect("a secure password element exists"); - let (_, _, evidence, _) = found; - let role = evidence.role.known().cloned().unwrap_or_default(); - - let token = crate::system::process_identity::token_for_pid( - agent_desktop_core::ProcessId::from(fixture.process_id()), - ) - .unwrap() - .expect("a live fixture process has a token"); - - let mut entry = crate::tree::walk::walker_fake::ref_entry(&role); - entry.process = agent_desktop_core::RefProcess { - pid: agent_desktop_core::ProcessId::from(fixture.process_id()), - process_instance: Some(token), - }; - entry.source.source_app = Some("fixture.exe".into()); - entry.source.source_window_id = Some(format!("w-{}", fixture.handle())); - - assert!( - crate::tree::resolve::resolve_search::entry_is_unverifiable(&entry), - "id-less, textless, geometry-less entry must be structurally unverifiable" - ); - - let mut attempts = 0; - let result = retry_incomplete_until(deadline, || { - attempts += 1; - resolve_attempt(&entry, deadline) - }); - - assert_eq!( - attempts, 1, - "a structurally unverifiable entry must settle in one attempt, never retry-burn" - ); - match result { - Err(error) => assert_eq!(error.code, agent_desktop_core::ErrorCode::StaleRef), - Ok(_) => panic!("an unverifiable entry must not resolve to a live element"), - } -} diff --git a/crates/windows/src/tree/resolve/resolve_search.rs b/crates/windows/src/tree/resolve/resolve_search.rs index bba7ab9c5..6808fae76 100644 --- a/crates/windows/src/tree/resolve/resolve_search.rs +++ b/crates/windows/src/tree/resolve/resolve_search.rs @@ -1,36 +1,23 @@ -//! The resolution search family: the bounded broad search, the path -//! fast-path, and the geometry promotion. +//! The stored-path landing family: the scope gate the locator anchor applies and +//! the bounded child enumeration it descends through (in `descent`). //! -//! Split from `resolve.rs` so that neither file crosses the 400-line cap. -//! Everything here mirrors a shape the macOS adapter already settled: the -//! search's role gate and its incomplete tracking, the path fast-path behind -//! `can_use_path_fast_path` (`crates/macos/src/tree/resolve/mod.rs`), and the -//! geometry promotion predicate `provisional_geometry_candidate` -//! (`crates/macos/src/tree/resolve/resolve_search.rs`) - with one Windows-measured -//! addition, that the stored hash must come from a positive-area rectangle -//! (A17-7). +//! Identity is not judged here: the landing is re-checked by core's full +//! predicate (`agent_desktop_core::refs::resolve_saved_ref`), which owns +//! identity for every platform. -use agent_desktop_core::{ - AdapterError, Deadline, LocatorEvidence, RefEntry, refs::ref_identity::has_meaningful_identity, -}; +use agent_desktop_core::{Deadline, RefEntry}; -use super::resolve_match::{Candidate, CandidateOutcome, bounds_hash_of}; -use crate::system::runtime::hresult::ReadDisposition; -use crate::tree::automation::{UiaFailure, uia_failure_disposition}; -use crate::tree::element::UIAElement; -use crate::tree::walk::walker::{DEFAULT_MAX_SIBLINGS, TreeSource, WalkBudget}; -use crate::tree::walk::walker_source::UiaTreeSource; -use descent::{DescentPolicy, DescentVerdict, ExpiryPolicy}; +use crate::tree::walk::walker::{DEFAULT_MAX_SIBLINGS, WalkBudget}; #[path = "resolve_descent.rs"] pub(crate) mod descent; /// The resolve-scoped depth cap. /// -/// Independently bounded from the walk's own ceiling, mirroring macOS's -/// `MAX_RESOLVE_DEPTH` (`crates/macos/src/tree/resolve/mod.rs:17`). Electron -/// elements commonly sit at depth 25+, so the cap is the search bound rather -/// than the walk bound. +/// Independently bounded from the walk's own ceiling, and at least as deep as +/// core's whole-window observation, so every node a capture can count is one +/// the anchor can land. Electron elements commonly sit at depth 25+, so the cap +/// is the resolution bound rather than the walk bound. pub(crate) const MAX_RESOLVE_DEPTH: u8 = 50; /// The budget a resolution attempt enumerates under. @@ -58,343 +45,12 @@ pub(crate) use agent_desktop_core::refs::resolve_errors::identity_unknown_error; /// Whether the stored path is scoped to the window root rather than to a /// drill-down ancestor: no `root_ref` at all, or an absolute path that -/// overrides one. Shared by every eligibility gate below - the fast path, -/// the anchor - so window-rootedness is decided in exactly one place. +/// overrides one. The anchor descends from the window's own root, so a path +/// relative to some other node is not one it can walk. pub(crate) fn window_rooted(entry: &RefEntry) -> bool { entry.scope.root_ref.is_none() || entry.scope.path_is_absolute } -/// Whether the stored ref is locatable by any resolution tier at all: -/// window-rooted, and not [`entry_is_unverifiable`] - an id, stable text, or -/// a positive-area bounds hash eligible for the geometry tier -/// ([`provisional_geometry_candidate`] is the exact predicate that tier -/// promotes on, reused rather than re-derived). -/// -/// This is the anchor's entire eligibility gate, and the fast path's -/// eligibility gate before its own non-empty-path conjunct. A bare -/// `RefEntry::geometry.bounds_hash` is not this signal: a zero-extent -/// rectangle still hashes (`Rect::bounds_hash` only rejects an invalid -/// rectangle, not an empty one), so a hash-only check would pass an id-less, -/// text-less entry the geometry tier can never actually promote, land the -/// path, and only then discover it cannot verify - burning the retry budget -/// on a landing that was never eligible. -pub(crate) fn entry_is_locatable(entry: &RefEntry) -> bool { - window_rooted(entry) && !entry_is_unverifiable(entry) -} - -/// Whether the stored child-index path may be walked as a fast path. -/// -/// Mirrors macOS's `can_use_path_fast_path`: [`entry_is_locatable`] plus a -/// non-empty path. -pub(crate) fn can_use_path_fast_path(entry: &RefEntry) -> bool { - entry_is_locatable(entry) && !entry.scope.path.is_empty() -} - -/// Walks the stored child-index path from a root, O(depth) child reads. -/// -/// Hands back both of [`descent::PathLanding`]'s facts unfolded, which is the -/// whole reason it returns a landing rather than an element and an -/// incompleteness output parameter. The landing is what -/// [`accept_path_landing`] decides on; the unread region is what an attempt's -/// *negative* verdict is withheld by. Merging them into one flag makes an -/// accepted landing look like it ignored an obligation, and makes a dropped -/// region look like a tidy scope. -/// -/// A path step that lands nowhere yields no element; the caller treats that as -/// a miss and falls back to the broad search, never as a verdict that the -/// target is gone. -/// -/// Generic over the tree source so the two are separable: what a gap on this -/// walk means to the resolution as a whole can be exercised without a live UI -/// Automation provider to fault on demand. -pub(crate) fn walk_stored_path( - source: &S, - root: &S::Node, - path: &[usize], - budget: &WalkBudget, -) -> Result, AdapterError> { - descent::descend_path(source, root, path, budget, &SEARCH_DESCENT) -} - -/// Decides the element the stored path landed on, from that element's own -/// evidence and nothing else. -/// -/// The signature is the argument. This tier is handed the element the path -/// names and the ref it is matched against, and no completeness of the -/// surrounding walk is in scope for it to consult. That is a property, not an -/// omission: a landing is the element the stored path names whatever else the -/// walk failed to read (see [`descent::PathLanding`]), and the claim this tier -/// makes is positional - the ref named *this* location and the element there -/// still answers to the stored identity - never a uniqueness claim over the -/// tree. Uniqueness is the broad search's claim to make, and a ref this tier -/// declines falls through to it. -/// -/// Declining is therefore never a verdict. `None` says only that this tier -/// could not settle the match, and the attempt continues. -/// -/// The admission rule is [`admit_node`] itself rather than a second copy of -/// the role gate, the composed identity rule and the geometry promotion, so -/// the two tiers cannot drift into accepting different elements. One check -/// sits ahead of it: [`geometry_contradicts`] refutes a landing whose live -/// bounds hash disagrees with a known stored one even when `admit_node`'s -/// identity tiers alone would call it a match - role and name survive a list -/// reordering, a stored bounds hash does not, so this tier must not return -/// early on identity alone and pre-empt the broad search's own bounds -/// tie-break. -pub(crate) fn accept_path_landing( - source: &UiaTreeSource, - candidate: &UIAElement, - entry: &RefEntry, -) -> Option { - let (_, evidence, _) = source.evidence(candidate); - if geometry_contradicts(entry, &evidence) { - return None; - } - match admit_node(entry, &evidence) { - NodeAdmission::Collect => Some(candidate.clone()), - NodeAdmission::Unread | NodeAdmission::Reject => None, - } -} - -/// The geometry promotion eligibility predicate. -/// -/// Mirrors macOS's `provisional_geometry_candidate` with one -/// Windows-measured addition: the stored bounds hash must come from a -/// positive-area rectangle (A17-7 - offscreen and virtualized elements -/// collapse to shared zero-extent bounds that are structurally non-unique), -/// and the stored bounds must agree. A ref with no meaningful text identity -/// and a zero-extent stored hash is unresolvable by design and never promotes. -pub(crate) fn provisional_geometry_candidate(entry: &RefEntry) -> bool { - entry.geometry.bounds_hash.is_some() - && entry - .geometry - .bounds - .is_some_and(|rect| rect.width > 0.0 && rect.height > 0.0) - && !has_meaningful_identity(entry) -} - -/// Whether a candidate's live bounds hash matches the stored one under the -/// promotion's eligibility rules - the unique-geometry-match that promotes an -/// otherwise-unreadable candidate to resolved. -pub(crate) fn geometry_matches(entry: &RefEntry, evidence: &LocatorEvidence) -> bool { - provisional_geometry_candidate(entry) && bounds_hash_of(evidence) == entry.geometry.bounds_hash -} - -/// Whether the stored bounds hash and a candidate's live bounds hash are both -/// known and disagree. -/// -/// Deliberately not [`geometry_matches`]: that predicate is gated by -/// [`provisional_geometry_candidate`], which requires -/// `!has_meaningful_identity(entry)` - it exists to promote an otherwise -/// unreadable candidate, not to refute an identity-matched one, so it always -/// answers `false` for a ref that carries a name or an id, which is exactly -/// the shape (duplicate-identity siblings sharing role, name and -/// `AutomationId`) this refutation exists to catch. An unknown hash on either -/// side answers `false` here, never a refutation - the resolver's tri-state -/// discipline treats an unread field as `Unread`, not as evidence of absence. -pub(crate) fn geometry_contradicts(entry: &RefEntry, evidence: &LocatorEvidence) -> bool { - match (entry.geometry.bounds_hash, bounds_hash_of(evidence)) { - (Some(stored), Some(live)) => stored != live, - _ => false, - } -} - -/// Whether the stored ref carries nothing any resolution tier could ever -/// verify a live candidate against: no id, no stable text, and no -/// positive-area bounds hash for the geometry tier to promote on. -/// -/// Meant to be checked once over the entry before the walk runs, because no -/// amount of successful reading changes the answer for this class: with no -/// stored name, value or description, core's `stable_text_match` has no -/// `expected` text to equality-match against, so it can only ever fall -/// through to `empty_identity_match` over the live candidate's own fields - -/// which resolves `NoMatch` the instant any of them reads back `Known`, and -/// `Unknown` only when none of them do. `IdentityMatch::Match` is therefore -/// unreachable for this class no matter what the walk reads, so an unbounded -/// walk would spend a full attempt - refuting some candidates, marking -/// others incomplete - reaching a verdict the entry alone already settles. -pub(crate) fn entry_is_unverifiable(entry: &RefEntry) -> bool { - !has_meaningful_identity(entry) && !provisional_geometry_candidate(entry) -} - -/// Whether an already-collected match count settles `AMBIGUOUS_TARGET` on -/// its own, with nothing left to read that could change the answer. -/// -/// Despite the name, nothing here stops `search_under`'s own walk early - -/// Windows's search always collects the whole depth-bounded subtree, and -/// this predicate is consulted only after collection finishes, inside -/// `classify_search`. Two or more matches with no stored hash to -/// disambiguate is conclusively ambiguous already: nothing left unread could -/// turn it into anything other than `AMBIGUOUS_TARGET`, so an incomplete -/// region elsewhere in the tree must not withhold that verdict. Mirrors -/// macOS's `should_stop_collecting` -/// (`crates/macos/src/tree/resolve/resolve_search.rs:374-376`), which does stop its -/// walk early on this signal - Windows only reuses its name and its -/// classification rule, not its early exit. -pub(crate) fn should_stop_collecting(match_count: usize, entry: &RefEntry) -> bool { - match_count > 1 && entry.geometry.bounds_hash.is_none() -} - -/// Bundles the three values that stay invariant across `search_under`'s -/// recursive descent - the tree source, the stored ref being resolved -/// against, and the walk budget - so the recursive signature threads one -/// reference instead of carrying each individually at every call. -pub(crate) struct SearchContext<'a> { - pub(crate) source: &'a UiaTreeSource, - pub(crate) entry: &'a RefEntry, - pub(crate) budget: &'a WalkBudget, -} - -/// Whether a child at `depth + 1` would still be within the resolve depth -/// cap and therefore actually get searched. -/// -/// Enumeration is gated on this rather than run unconditionally: at -/// `depth + 1 == MAX_RESOLVE_DEPTH`, every child the enumeration would -/// return is immediately discarded by the depth guard on entry to the next -/// call, so running the enumeration first pays a real, cross-process -/// sibling walk (up to the sibling cap) purely for children the search will -/// never evaluate. Skipping the enumeration also means a transport failure -/// among those never-searched children can no longer flag the search -/// incomplete - correctly so: nothing this search will ever look at can be -/// the reason to retry it, so the retry would buy nothing. -fn child_search_is_reachable(depth: u8) -> bool { - depth + 1 < MAX_RESOLVE_DEPTH -} - -/// What the search does with one node it has read. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub(crate) enum NodeAdmission { - /// The node is a candidate the verdict counts. - Collect, - /// The node's own evidence could not settle the question, so this region - /// of the tree was not read conclusively and the verdict is withheld. - Unread, - /// The node is refuted, or its role rules it out. Nothing is withheld. - Reject, -} - -/// Decides one node's admission from the stored ref and the node's evidence: -/// the role gate, the composed identity rule, and the geometry promotion. -/// -/// Pure over evidence rather than over a live element, so each arm is -/// decidable without a UI Automation provider that can be made to withhold a -/// property on demand - the separation `select_by_bounds_hash` already draws -/// for the tie-break. -/// -/// The promotion arm is reachable only on an `Incomplete` identity, and for -/// the class it exists to serve that verdict is structural rather than -/// transient: a ref whose role carries a mutable value and whose stored text -/// is blank - the shape a secure edit stores - makes core's -/// `stable_text_match` answer `Unknown` on every attempt, whatever the read -/// succeeds at. Geometry is the only tier that class has, and only from a -/// positive-area rectangle: offscreen and virtualized elements collapse to -/// shared zero-extent bounds that are structurally non-unique (A17-7), so a -/// zero-extent stored rectangle leaves the node unread rather than promoting -/// it. An unpromoted `Incomplete` is never a refutation - the tri-state -/// discipline the whole resolver is built on. -pub(crate) fn admit_node(entry: &RefEntry, evidence: &LocatorEvidence) -> NodeAdmission { - let role_matches = evidence - .role - .known() - .is_some_and(|role| role == &entry.identity.role); - if !role_matches { - if evidence.role.is_unknown() { - return NodeAdmission::Unread; - } - return NodeAdmission::Reject; - } - match super::resolve_match::candidate_outcome(entry, evidence) { - CandidateOutcome::Matched => NodeAdmission::Collect, - CandidateOutcome::Incomplete if geometry_matches(entry, evidence) => NodeAdmission::Collect, - CandidateOutcome::Incomplete => NodeAdmission::Unread, - CandidateOutcome::Refuted => NodeAdmission::Reject, - } -} - -/// Searches the subtree under `element` to the resolve depth, collecting the -/// candidates the composed matcher accepted and recording every part of the -/// subtree it could not read. -pub(crate) fn search_under( - ctx: &SearchContext<'_>, - element: &UIAElement, - depth: u8, - out: &mut Vec, - unread_region: &mut bool, -) -> Result<(), AdapterError> { - if depth >= MAX_RESOLVE_DEPTH { - return Ok(()); - } - ctx.budget.deadline.ensure_remaining()?; - - let (_, evidence, _) = ctx.source.evidence(element); - match admit_node(ctx.entry, &evidence) { - NodeAdmission::Collect => out.push(build_candidate(element, &evidence)), - NodeAdmission::Unread => *unread_region = true, - NodeAdmission::Reject => {} - } - - if child_search_is_reachable(depth) { - let children = enumerate_children(ctx.source, element, ctx.budget, unread_region)?; - for child in children { - search_under(ctx, &child, depth + 1, out, unread_region)?; - } - } - Ok(()) -} - -/// Enumerates one element's children for the search, honouring the sibling cap -/// as a hard bound on pathological lists. -pub(crate) fn enumerate_children( - source: &UiaTreeSource, - element: &UIAElement, - budget: &WalkBudget, - unread_region: &mut bool, -) -> Result, AdapterError> { - let read = descent::read_children(source, element, budget, &SEARCH_DESCENT, None)?; - *unread_region |= !read.complete; - Ok(read.elements) -} - -/// The search's descent policy. -/// -/// An expired deadline leaves the search unfinished rather than surfacing: -/// unfinished is exactly what the search's own classification already means by -/// retryable, and the search did not finish, so a partial collection must not -/// be classified as a settled absence. -pub(crate) const SEARCH_DESCENT: DescentPolicy = DescentPolicy { - classify: search_descent_verdict, - on_expiry: ExpiryPolicy::Unfinish, - descend_context: "descend to a stored ref", - sibling_context: "walk a stored ref's siblings", -}; - -/// Classifies a descent failure under the read disposition: a settled -/// absence means the node enumerates nothing (a real answer, not -/// incomplete); a transport failure or vanished node marks the search -/// incomplete and the descent continues - a non-target node dying -/// mid-descent under live churn is not evidence the target is gone; a -/// terminal failure propagates. -fn search_descent_verdict(failure: UiaFailure) -> DescentVerdict { - match uia_failure_disposition(failure) { - ReadDisposition::SettledAbsence => DescentVerdict::Settled, - ReadDisposition::Retryable | ReadDisposition::Unavailable => DescentVerdict::Unfinished, - ReadDisposition::Terminal => DescentVerdict::Surfaced, - } -} - -/// Builds the candidate from the walk-composed evidence the search already -/// read, projecting the tie-break hash off the same evidence slot. -pub(crate) fn build_candidate(element: &UIAElement, evidence: &LocatorEvidence) -> Candidate { - Candidate { - element: element.clone(), - bounds_hash: bounds_hash_of(evidence), - } -} - #[cfg(all(test, target_os = "windows"))] #[path = "resolve_search_tests.rs"] mod tests; - -#[cfg(all(test, target_os = "windows"))] -#[path = "resolve_search_admission_tests.rs"] -mod admission_tests; diff --git a/crates/windows/src/tree/resolve/resolve_search_admission_tests.rs b/crates/windows/src/tree/resolve/resolve_search_admission_tests.rs deleted file mode 100644 index f64a854b5..000000000 --- a/crates/windows/src/tree/resolve/resolve_search_admission_tests.rs +++ /dev/null @@ -1,176 +0,0 @@ -use super::*; -use crate::tree::resolve::resolve_match::candidate_outcome; -use agent_desktop_core::{IdentifierEvidence, LocatorField, NodeDescriptor, Rect}; - -/// A role whose value is mutable, which is what makes `candidate_outcome` -/// answer `Incomplete` for a blank ref on every attempt rather than only when -/// a read happens to fail. That is the entry into the promotion arm. -const PROMOTABLE_ROLE: &str = "textfield"; - -/// A role whose value is stable, so a stored name settles the match outright -/// and the control arms below are decided by identity rather than geometry. -const NAMED_ROLE: &str = "button"; - -fn rect(width: f64, height: f64) -> Rect { - Rect { - x: 12.0, - y: 24.0, - width, - height, - } -} - -fn stored(role: &str, name: Option<&str>, bounds: Option) -> RefEntry { - let mut entry = crate::tree::walk::walker_fake::ref_entry(role); - entry.identity.name = name.map(str::to_string); - entry.geometry = agent_desktop_core::RefGeometry { - bounds_space: None, - bounds, - bounds_hash: bounds.and_then(|rect| rect.bounds_hash()), - }; - entry.scope.path_is_absolute = true; - entry -} - -fn live( - role: LocatorField, - name: LocatorField, - bounds: LocatorField, -) -> LocatorEvidence { - LocatorEvidence { - role, - name, - value: LocatorField::Absent, - description: LocatorField::Absent, - identifiers: IdentifierEvidence::absent(), - states: LocatorField::Absent, - ref_evidence: agent_desktop_core::LocatorRefEvidence { - window_bounds: LocatorField::Absent, - bounds, - available_actions: LocatorField::Absent, - descriptors: NodeDescriptor::default(), - }, - } -} - -fn role_known(role: &str) -> LocatorField { - LocatorField::Known(role.to_string()) -} - -/// The promotion arm itself, on the shape it exists for: a ref with no text -/// identity whose role carries a mutable value can never be settled by text, -/// so an `Incomplete` identity corroborated by a positive-area geometry match -/// is collected as a candidate rather than withheld as an unread region. Take -/// this arm away and the class has no tier left at all. -#[test] -fn an_unsettleable_identity_is_collected_when_positive_area_geometry_agrees() { - let bounds = rect(40.0, 20.0); - let entry = stored(PROMOTABLE_ROLE, None, Some(bounds)); - let evidence = live( - role_known(PROMOTABLE_ROLE), - LocatorField::Absent, - LocatorField::Known(bounds), - ); - - assert_eq!( - candidate_outcome(&entry, &evidence), - CandidateOutcome::Incomplete, - "the promotion arm is only reachable through an incomplete identity" - ); - assert_eq!(admit_node(&entry, &evidence), NodeAdmission::Collect); -} - -/// The other half of the same rule (A17-7): offscreen and virtualized -/// elements collapse to shared zero-extent bounds, so a zero-extent stored -/// rectangle identifies nothing even when its hash agrees exactly. The stored -/// and live hashes are asserted equal here, so the zero extent is the only -/// thing left that can refuse the promotion. -#[test] -fn a_zero_extent_stored_rectangle_never_promotes_even_on_an_exact_hash_agreement() { - let flattened = rect(0.0, 0.0); - let entry = stored(PROMOTABLE_ROLE, None, Some(flattened)); - let evidence = live( - role_known(PROMOTABLE_ROLE), - LocatorField::Absent, - LocatorField::Known(flattened), - ); - - assert!( - entry.geometry.bounds_hash.is_some(), - "a zero-extent rectangle still hashes, so the hash alone cannot be the gate" - ); - assert_eq!( - bounds_hash_of(&evidence), - entry.geometry.bounds_hash, - "the live and stored hashes agree, so only the zero extent can refuse this" - ); - assert_eq!(admit_node(&entry, &evidence), NodeAdmission::Unread); -} - -/// An unpromoted incomplete is withheld, never refuted. Geometry that -/// disagrees - or that could not be read - says nothing about a candidate -/// whose identity was never settleable, so the region stays unread and the -/// resolution retries rather than counting this node as ruled out. -#[test] -fn an_incomplete_identity_the_geometry_does_not_corroborate_stays_unread() { - let entry = stored(PROMOTABLE_ROLE, None, Some(rect(40.0, 20.0))); - - let elsewhere = live( - role_known(PROMOTABLE_ROLE), - LocatorField::Absent, - LocatorField::Known(rect(80.0, 20.0)), - ); - assert_eq!(admit_node(&entry, &elsewhere), NodeAdmission::Unread); - - let unreadable = live( - role_known(PROMOTABLE_ROLE), - LocatorField::Absent, - LocatorField::Unknown, - ); - assert_eq!(admit_node(&entry, &unreadable), NodeAdmission::Unread); -} - -/// The two settled identity arms, so the promotion above is not the only -/// behaviour this decision has: a matching identity is collected and a -/// contradicted one is rejected outright, neither of them withheld. -#[test] -fn a_settled_identity_is_collected_or_rejected_and_never_withheld() { - let entry = stored(NAMED_ROLE, Some("Save"), None); - - let same = live( - role_known(NAMED_ROLE), - LocatorField::Known("Save".to_string()), - LocatorField::Absent, - ); - assert_eq!(admit_node(&entry, &same), NodeAdmission::Collect); - - let other = live( - role_known(NAMED_ROLE), - LocatorField::Known("Cancel".to_string()), - LocatorField::Absent, - ); - assert_eq!(admit_node(&entry, &other), NodeAdmission::Reject); -} - -/// The role gate, both directions. A role that read back as something else is -/// a real refusal, while a role that could not be read at all leaves the -/// region unread - collapsing the second into the first would settle -/// `STALE_REF` off a tree the search never managed to classify. -#[test] -fn a_different_role_is_rejected_but_an_unreadable_one_is_only_unread() { - let entry = stored(NAMED_ROLE, Some("Save"), None); - let name = LocatorField::Known("Save".to_string()); - - let different = live( - role_known(PROMOTABLE_ROLE), - name.clone(), - LocatorField::Absent, - ); - assert_eq!(admit_node(&entry, &different), NodeAdmission::Reject); - - let absent = live(LocatorField::Absent, name.clone(), LocatorField::Absent); - assert_eq!(admit_node(&entry, &absent), NodeAdmission::Reject); - - let unreadable = live(LocatorField::Unknown, name, LocatorField::Absent); - assert_eq!(admit_node(&entry, &unreadable), NodeAdmission::Unread); -} diff --git a/crates/windows/src/tree/resolve/resolve_search_tests.rs b/crates/windows/src/tree/resolve/resolve_search_tests.rs index a9f7667a2..3f989d785 100644 --- a/crates/windows/src/tree/resolve/resolve_search_tests.rs +++ b/crates/windows/src/tree/resolve/resolve_search_tests.rs @@ -1,59 +1,17 @@ use super::*; -use crate::tree::fixture::ensure_test_apartment; -use agent_desktop_core::{ - ElementIdentifier, IdentifierEvidence, IdentifierKind, LocatorField, NodeDescriptor, -}; +use agent_desktop_core::ElementIdentifier; -fn entry( - bounds: Option, - hash: Option, - name: Option<&str>, - native: Option<&str>, -) -> RefEntry { +fn entry() -> RefEntry { let mut entry = crate::tree::walk::walker_fake::ref_entry("button"); - entry.identity.name = name.map(str::to_string); - entry.identity.native_id = native.map(|value| ElementIdentifier { - kind: IdentifierKind::AutomationId, - value: value.to_string(), + entry.identity.name = Some("name".into()); + entry.identity.native_id = Some(ElementIdentifier { + kind: agent_desktop_core::IdentifierKind::AutomationId, + value: "id".into(), }); - entry.geometry = agent_desktop_core::RefGeometry { - bounds_space: None, - bounds, - bounds_hash: hash, - }; entry.scope.path_is_absolute = true; entry } -fn rect(width: f64, height: f64) -> agent_desktop_core::Rect { - agent_desktop_core::Rect { - x: 10.0, - y: 10.0, - width, - height, - } -} - -fn evidence(bounds_hash: Option) -> LocatorEvidence { - LocatorEvidence { - role: LocatorField::Known("button".to_string()), - name: LocatorField::Absent, - value: LocatorField::Absent, - description: LocatorField::Absent, - identifiers: IdentifierEvidence::absent(), - states: LocatorField::Absent, - ref_evidence: agent_desktop_core::LocatorRefEvidence { - window_bounds: LocatorField::Absent, - bounds: match bounds_hash { - Some(_) => LocatorField::Known(rect(40.0, 20.0)), - None => LocatorField::Absent, - }, - available_actions: LocatorField::Absent, - descriptors: NodeDescriptor::default(), - }, - } -} - /// Window-rootedness is a disjunction and every other builder in this file /// leaves `root_ref` unset, which satisfies it through the first disjunct /// alone. Both disjuncts are pinned here, in both directions: dropping the @@ -61,7 +19,7 @@ fn evidence(bounds_hash: Option) -> LocatorEvidence { /// the rest of this suite stayed green. #[test] fn a_drill_down_ref_is_window_rooted_only_when_its_path_is_absolute() { - let mut absolute_under_a_root = entry(None, Some(1), Some("name"), None); + let mut absolute_under_a_root = entry(); absolute_under_a_root.scope.root_ref = Some("root".to_string()); absolute_under_a_root.scope.path_is_absolute = true; assert!(window_rooted(&absolute_under_a_root)); @@ -75,7 +33,7 @@ fn a_drill_down_ref_is_window_rooted_only_when_its_path_is_absolute() { /// all is window-rooted whatever its path flag says. #[test] fn a_ref_with_no_drill_down_root_is_window_rooted_whatever_its_path_flag() { - let mut rootless = entry(None, Some(1), Some("name"), None); + let mut rootless = entry(); rootless.scope.root_ref = None; rootless.scope.path_is_absolute = false; assert!(window_rooted(&rootless)); @@ -83,222 +41,3 @@ fn a_ref_with_no_drill_down_root_is_window_rooted_whatever_its_path_flag() { rootless.scope.path_is_absolute = true; assert!(window_rooted(&rootless)); } - -#[test] -fn a_window_rooted_non_empty_path_with_identity_qualifies_for_the_fast_path() { - let mut absolute = entry(None, Some(1), Some("name"), Some("id")); - absolute.scope.path.push(2); - assert!(can_use_path_fast_path(&absolute)); -} - -#[test] -fn a_relative_drill_down_path_skips_the_fast_path() { - let mut relative = entry(None, Some(1), None, Some("id")); - relative.scope.root_ref = Some("root".to_string()); - relative.scope.path_is_absolute = false; - assert!(!can_use_path_fast_path(&relative)); -} - -#[test] -fn an_empty_path_skips_the_fast_path() { - assert!(!can_use_path_fast_path(&entry(None, Some(1), None, None))); -} - -#[test] -fn a_ref_with_no_id_text_or_hash_skips_the_fast_path() { - assert!(!can_use_path_fast_path(&entry(None, None, None, None))); -} - -/// `can_use_path_fast_path` must apply the same positive-area rule -/// `provisional_geometry_candidate` promotes on, not a bare -/// `bounds_hash.is_some()`: a zero-extent stored rectangle still hashes -/// (`Rect::bounds_hash` only rejects an invalid rectangle), so an id-less, -/// text-less entry backed by one is not eligible for the geometry tier and -/// must not qualify for the fast path either. -#[test] -fn a_zero_extent_bounds_hash_does_not_qualify_for_the_fast_path() { - let mut zero_extent = entry(Some(rect(0.0, 0.0)), Some(1), None, None); - zero_extent.scope.path.push(0); - assert!(!can_use_path_fast_path(&zero_extent)); -} - -#[test] -fn promotion_requires_a_positive_area_stored_bounds() { - let positive = entry(Some(rect(40.0, 20.0)), Some(1), None, None); - assert!(provisional_geometry_candidate(&positive)); - - let zero_extent = entry(Some(rect(0.0, 0.0)), Some(1), None, None); - assert!(!provisional_geometry_candidate(&zero_extent)); - - let only_hash = entry(None, Some(1), None, None); - assert!(!provisional_geometry_candidate(&only_hash)); -} - -#[test] -fn promotion_never_fires_when_the_entry_has_a_text_identity() { - let named = entry(Some(rect(40.0, 20.0)), Some(1), Some("name"), None); - assert!(!provisional_geometry_candidate(&named)); - - let native = entry(Some(rect(40.0, 20.0)), Some(1), None, Some("id")); - assert!(!provisional_geometry_candidate(&native)); -} - -#[test] -fn geometry_matches_only_on_the_live_bounds_hash() { - let live_hash = rect(40.0, 20.0) - .bounds_hash() - .expect("a positive-area hash"); - let stored = entry(Some(rect(40.0, 20.0)), Some(live_hash), None, None); - assert!(geometry_matches(&stored, &evidence(Some(live_hash)))); - assert!(!geometry_matches(&stored, &evidence(None))); -} - -#[test] -fn geometry_matches_never_promotes_a_zero_extent_stored_hash() { - let stored = entry(None, Some(0x1234), None, None); - assert!(!geometry_matches(&stored, &evidence(Some(0x1234)))); -} - -#[test] -fn an_id_less_textless_zero_extent_ref_is_unverifiable() { - let zero_extent = entry(Some(rect(0.0, 0.0)), Some(0x1234), None, None); - assert!(entry_is_unverifiable(&zero_extent)); - - let no_bounds_at_all = entry(None, None, None, None); - assert!(entry_is_unverifiable(&no_bounds_at_all)); -} - -#[test] -fn a_ref_verifiable_by_any_single_tier_is_not_unverifiable() { - let named = entry(None, None, Some("name"), None); - assert!(!entry_is_unverifiable(&named)); - - let identified = entry(None, None, None, Some("id")); - assert!(!entry_is_unverifiable(&identified)); - - let geometry_only = entry(Some(rect(40.0, 20.0)), Some(0x1234), None, None); - assert!(!entry_is_unverifiable(&geometry_only)); -} - -/// The shape allocation emits for nameless positive-area content once it -/// keeps the rect that is the entry's only identity: locatable by the -/// geometry tier rather than failed closed by `entry_is_unverifiable` -/// before any candidate search runs - the A24-11 stale-rate mechanism. -#[test] -fn a_nameless_positive_area_ref_is_locatable_by_the_geometry_tier() { - let geometry_only = entry(Some(rect(40.0, 20.0)), Some(0x1234), None, None); - assert!(!has_meaningful_identity(&geometry_only)); - assert!(provisional_geometry_candidate(&geometry_only)); - assert!(!entry_is_unverifiable(&geometry_only)); - assert!(entry_is_locatable(&geometry_only)); -} - -#[test] -fn should_stop_collecting_fires_only_past_one_match_with_no_stored_hash() { - let no_hash = entry(None, None, None, None); - assert!(!should_stop_collecting(0, &no_hash)); - assert!(!should_stop_collecting(1, &no_hash)); - assert!(should_stop_collecting(2, &no_hash)); - - let with_hash = entry(None, Some(0x1234), None, None); - assert!(!should_stop_collecting(2, &with_hash)); -} - -/// The geometry tier's promotion decision, taken against a live control rather -/// than a hand-built one. -/// -/// The fixture's password edit is the shape the tier exists for: a ref stored -/// for secure content carries no text identity, so positive-area geometry is -/// the only evidence left to verify it against. What is asserted is the -/// promotion itself - `geometry_matches` over the evidence the walk actually -/// read - and each condition that produces it is withdrawn in turn, so -/// weakening either one of them shows up here. Asserting only that the fixture -/// has a secure element would survive inverting the promotion predicate -/// outright, pinning the fixture instead of the tier. -#[test] -fn a_ref_stored_for_the_live_secure_edit_promotes_on_geometry_alone() { - ensure_test_apartment(); - let fixture = crate::tree::fixture::HostedFixture::spawn().expect("a fixture host starts"); - let source = UiaTreeSource::for_root( - &crate::tree::automation::root_from_hwnd( - fixture.handle(), - crate::tree::walk::walker_fake::deadline(), - ) - .expect("the fixture resolves"), - ) - .expect("a tree source"); - let prepared = source - .prepare_root( - &crate::tree::automation::root_from_hwnd( - fixture.handle(), - crate::tree::walk::walker_fake::deadline(), - ) - .expect("the fixture resolves"), - ) - .expect("a prepared root"); - let budget = WalkBudget::new(10, crate::tree::walk::walker_fake::deadline()); - let found = find_secure(&source, &prepared, &budget) - .expect("the fixture walk succeeds") - .expect("a secure element exists"); - let (_, properties, live, _) = found; - assert!( - properties.is_secure(), - "the control under test is the secure edit, not some other element" - ); - let live_bounds = *live.ref_evidence.bounds.known().expect("live bounds"); - let live_hash = live_bounds - .bounds_hash() - .expect("the live secure edit occupies a positive-area rectangle"); - - let stored = entry(Some(live_bounds), Some(live_hash), None, None); - assert!( - provisional_geometry_candidate(&stored), - "a ref with no text identity and a positive-area hash is what the tier promotes" - ); - assert!( - geometry_matches(&stored, &live), - "the stored geometry must promote against the evidence the walk read" - ); - - let named = entry(Some(live_bounds), Some(live_hash), Some("Password"), None); - assert!( - !geometry_matches(&named, &live), - "a ref with a text identity has a tier of its own and must never promote on geometry" - ); - - let flattened = entry(Some(rect(0.0, 0.0)), Some(live_hash), None, None); - assert!( - !geometry_matches(&flattened, &live), - "a zero-extent stored rectangle is structurally non-unique and must never promote" - ); -} - -fn find_secure( - source: &UiaTreeSource, - element: &UIAElement, - budget: &WalkBudget, -) -> Result< - Option<( - agent_desktop_core::refs::RefPath, - crate::tree::properties::ElementProperties, - LocatorEvidence, - u64, - )>, - AdapterError, -> { - crate::tree::walk::walker_fake::scan_subtree( - source, - element, - budget, - 10, - &mut |prefix, properties, evidence, failed| { - if properties.is_secure() { - let mut path = agent_desktop_core::refs::RefPath::default(); - path.extend_from_slice(prefix); - std::ops::ControlFlow::Break((path, properties, evidence, failed)) - } else { - std::ops::ControlFlow::Continue(()) - } - }, - ) -} diff --git a/crates/windows/src/tree/resolve/resolve_tests.rs b/crates/windows/src/tree/resolve/resolve_tests.rs index 92d821bab..2bc182043 100644 --- a/crates/windows/src/tree/resolve/resolve_tests.rs +++ b/crates/windows/src/tree/resolve/resolve_tests.rs @@ -1,196 +1,16 @@ -use super::pair_window::automation_id; use super::*; -use agent_desktop_core::{ElementIdentifier, LocatorEvidence}; +use crate::tree::walk::walker::WalkBudget; +use crate::tree::walk::walker_source::UiaTreeSource; +use agent_desktop_core::LocatorEvidence; use std::ops::ControlFlow; -fn capture_identified( - root: &UIAElement, - deadline: agent_desktop_core::Deadline, -) -> Option<(Option, Option, Option)> { - let source = UiaTreeSource::for_root(root).ok()?; - let prepared = source.prepare_root(root).ok()?; - let budget = WalkBudget::new(10, deadline); - walk_for_identity(&source, &prepared, &budget) -} - -fn walk_for_identity( - source: &UiaTreeSource, - element: &UIAElement, - budget: &WalkBudget, -) -> Option<(Option, Option, Option)> { - crate::tree::walk::walker_fake::scan_subtree( - source, - element, - budget, - 10, - &mut |_, _, evidence, _| { - let native_id = automation_id(&evidence); - if native_id.is_some() { - ControlFlow::Break(( - native_id, - evidence.role.known().cloned(), - evidence.name.known().cloned(), - )) - } else { - ControlFlow::Continue(()) - } - }, - ) - .ok() - .flatten() -} - -#[test] -fn a_fixture_ref_resolves_to_the_same_element_end_to_end() { - crate::tree::fixture::ensure_test_apartment(); - let fixture = crate::tree::fixture::HostedFixture::spawn().expect("a fixture host starts"); - let window = crate::tree::test_support::fixture_window(&fixture); - let deadline = crate::tree::walk::walker_fake::deadline(); - let root = crate::tree::automation::root_from_hwnd(fixture.handle(), deadline) - .expect("the fixture window resolves"); - let token = window.process_instance.clone().unwrap(); - let expected_stamp = (window.pid.get(), token.clone()); - - let captured = capture_identified(&root, deadline).expect("a fixture element has an id"); - - let mut entry = - crate::tree::walk::walker_fake::ref_entry(&captured.1.clone().unwrap_or_default()); - entry.process = agent_desktop_core::RefProcess { - pid: window.pid, - process_instance: Some(token), - }; - entry.identity.name = captured.2.clone(); - entry.identity.native_id = captured.0.clone(); - entry.source.source_app = Some("fixture.exe".into()); - entry.source.source_window_id = Some(window.id.clone()); - - let handle = resolve_element_strict(&entry, deadline) - .expect("the stored identity re-resolves to a live element"); - - assert_verified_stamp(&handle, &expected_stamp, "the broad search"); -} - -/// Every handle the resolver hands back carries the ref's verified process -/// identity, so the shared live read can corroborate that the provider -/// answering is still the generation resolution verified. -/// -/// Without the stamp the corroboration is skipped for every resolved handle, -/// and A14-9's finding - a dead provider's reads succeeding empty on some -/// builds - lands as an answer rather than a `STALE_REF`. The live-read tests -/// build their own stamp, so nothing there notices a resolver that stops -/// producing one; this is the assertion that sees it. -fn assert_verified_stamp( - handle: &agent_desktop_core::NativeHandle, - expected: &(u32, String), - tier: &str, -) { - let resolved = handle - .downcast_ref::() - .expect("the resolved handle carries a UI Automation element"); - assert_eq!( - resolved.verified_process(), - Some((expected.0, expected.1.as_str())), - "{tier} must stamp the ref's verified process identity onto the handle it returns" - ); -} - -/// A ref taken from the fixture's password control - no text identity, -/// positive-area bounds, secure content withheld - resolves through the -/// path fast-path and the geometry tier on an unchanged tree, and the -/// secure value reaches no error or detail. -#[test] -fn a_blank_secure_ref_resolves_through_the_path_and_geometry_tier() { - crate::tree::fixture::ensure_test_apartment(); - let fixture = crate::tree::fixture::HostedFixture::spawn().expect("a fixture host starts"); - let deadline = crate::tree::walk::walker_fake::deadline(); - let root = crate::tree::automation::root_from_hwnd(fixture.handle(), deadline) - .expect("the fixture window resolves"); - let source = UiaTreeSource::for_root(&root).expect("a tree source"); - let prepared = source.prepare_root(&root).expect("a prepared root"); - let budget = WalkBudget::new(10, deadline); - - let found = find_password(&source, &prepared, &budget) - .expect("the fixture exposes a password edit") - .expect("a password element"); - let (path, _, evidence, _) = found; - let role = evidence.role.known().cloned(); - let rect = evidence.ref_evidence.bounds.known().expect("a bounds"); - let hash = rect.bounds_hash().expect("a positive-area hash"); - - let pid = agent_desktop_core::ProcessId::from(fixture.process_id()); - let token = crate::system::process_identity::token_for_pid(pid) - .unwrap() - .expect("a live fixture process has a token"); - let expected_stamp = (pid.get(), token.clone()); - - let mut entry = crate::tree::walk::walker_fake::ref_entry(&role.clone().unwrap_or_default()); - entry.process = agent_desktop_core::RefProcess { - pid, - process_instance: Some(token), - }; - entry.geometry = agent_desktop_core::RefGeometry { - bounds_space: None, - bounds: Some(*rect), - bounds_hash: Some(hash), - }; - entry.source.source_app = Some("fixture.exe".into()); - entry.source.source_window_id = Some(format!("w-{}", fixture.handle())); - entry.scope.path_is_absolute = true; - entry.scope.path = path; - - assert!( - crate::tree::resolve::resolve_search::can_use_path_fast_path(&entry), - "a window-rooted path with a positive-area hash qualifies" - ); - assert!( - crate::tree::resolve::resolve_search::provisional_geometry_candidate(&entry), - "no meaningful identity plus a positive-area hash is promotion-eligible" - ); - - let handle = resolve_element_strict(&entry, deadline) - .expect("the blank secure ref resolves through path and geometry"); - - assert_verified_stamp(&handle, &expected_stamp, "the path and geometry tier"); -} - -fn find_password( - source: &UiaTreeSource, - element: &UIAElement, - budget: &WalkBudget, -) -> Result< - Option<( - agent_desktop_core::refs::RefPath, - crate::tree::properties::ElementProperties, - LocatorEvidence, - u64, - )>, - AdapterError, -> { - crate::tree::walk::walker_fake::scan_subtree( - source, - element, - budget, - 10, - &mut |prefix, properties, evidence, failed| { - if properties.is_secure() { - let mut path = agent_desktop_core::refs::RefPath::default(); - path.extend_from_slice(prefix); - ControlFlow::Break((path, properties, evidence, failed)) - } else { - ControlFlow::Continue(()) - } - }, - ) -} - /// The cross-process takeover shape, end to end. The stored evidence here /// genuinely describes a live element - captured from that very tree moments -/// earlier, and proven resolvable by the test above - while the stored -/// process is a *different* process that is alive and of exactly the stored -/// generation. Only the handle's live owner can tell the two apart, so every -/// resolver reaching a window root through `resolve_window_root` must settle -/// `STALE_REF` rather than find the element in an application the ref never -/// named. +/// earlier - while the stored process is a *different* process that is alive +/// and of exactly the stored generation. Only the handle's live owner can tell +/// the two apart, so every resolver reaching a window root through +/// `resolve_window_root` must settle `STALE_REF` rather than find the element +/// in an application the ref never named. #[test] fn a_ref_whose_window_belongs_to_another_process_never_resolves_into_it() { crate::tree::fixture::ensure_test_apartment(); @@ -244,10 +64,6 @@ fn a_ref_whose_window_belongs_to_another_process_never_resolves_into_it() { "the window gate itself", resolve_window_root(&entry, deadline).err().map(code_of), ), - ( - "the strict resolver", - resolve_element_strict(&entry, deadline).err().map(code_of), - ), ( "the locator anchor", crate::tree::resolve::resolve_anchor::resolve_locator_anchor(&entry, deadline) @@ -259,13 +75,42 @@ fn a_ref_whose_window_belongs_to_another_process_never_resolves_into_it() { verdicts.to_vec(), vec![ ("the window gate itself", Some(ErrorCode::StaleRef)), - ("the strict resolver", Some(ErrorCode::StaleRef)), ("the locator anchor", Some(ErrorCode::StaleRef)), ], "no resolver may reach into a window another process owns" ); } +fn find_password( + source: &UiaTreeSource, + element: &UIAElement, + budget: &WalkBudget, +) -> Result< + Option<( + agent_desktop_core::refs::RefPath, + crate::tree::properties::ElementProperties, + LocatorEvidence, + u64, + )>, + AdapterError, +> { + crate::tree::walk::walker_fake::scan_subtree( + source, + element, + budget, + 10, + &mut |prefix, properties, evidence, failed| { + if properties.is_secure() { + let mut path = agent_desktop_core::refs::RefPath::default(); + path.extend_from_slice(prefix); + ControlFlow::Break((path, properties, evidence, failed)) + } else { + ControlFlow::Continue(()) + } + }, + ) +} + fn code_of(error: AdapterError) -> ErrorCode { error.code } @@ -275,15 +120,3 @@ mod retry; #[path = "resolve_retry_classification_tests.rs"] mod retry_classification; - -#[path = "resolve_verdict_tests.rs"] -mod verdict; - -#[path = "resolve_ambiguity_tests.rs"] -mod ambiguity; - -#[path = "resolve_tie_break_tests.rs"] -mod tie_break; - -#[path = "resolve_path_fallback_tests.rs"] -mod path_fallback; diff --git a/crates/windows/src/tree/resolve/resolve_tie_break_tests.rs b/crates/windows/src/tree/resolve/resolve_tie_break_tests.rs deleted file mode 100644 index 3ff7e157e..000000000 --- a/crates/windows/src/tree/resolve/resolve_tie_break_tests.rs +++ /dev/null @@ -1,331 +0,0 @@ -use crate::tree::resolve::*; -use agent_desktop_core::{LocatorEvidence, LocatorField, refs::RefPath}; - -use crate::tree::resolve::pair_window::{ - DuplicatePairWindow, NAME_MARKER, PairGeometry, TITLE_MARKER, automation_id, collect_marked, -}; -use crate::tree::resolve::resolve_search::{ - NodeAdmission, admit_node, enumerate_children, geometry_contradicts, -}; - -/// The bounds tie-break returns the stored candidate, not the first one it -/// collected. -/// -/// Two live controls share role, `Name` and `AutomationId`, so every identity -/// tier matches both and the bounds hash is the only tier that separates them. -/// The stored ref carries the hash of the candidate the search collects -/// **second**, so a resolver that took the head of its candidate list instead -/// of the selected index would return the other control and report success - -/// the silent-wrong-target class A7-3 measured, where Explorer re-resolved 5 -/// of 29 refs onto a different element without an error. A first-match-wins -/// regression therefore cannot pass here by luck: the correct answer is never -/// at index zero. -#[test] -fn the_bounds_tie_break_resolves_the_stored_candidate_not_the_first_collected() { - crate::tree::fixture::ensure_test_apartment(); - let hosted = - DuplicatePairWindow::create(PairGeometry::Separated).expect("a duplicate window hosts"); - let deadline = crate::tree::walk::walker_fake::deadline(); - let root = crate::tree::automation::root_from_hwnd(hosted.handle, deadline) - .expect("the hosted window resolves"); - let source = UiaTreeSource::for_root(&root).expect("a tree source"); - let prepared = source.prepare_root(&root).expect("a prepared root"); - let budget = WalkBudget::new(10, deadline); - - let mut duplicates = Vec::new(); - collect_marked(&source, &prepared, &budget, &mut duplicates); - assert_eq!( - duplicates.len(), - 2, - "the host must present exactly two marked controls" - ); - let (head, target) = (&duplicates[0], &duplicates[1]); - let role = head.role.known().cloned().expect("a read role"); - let name = head.name.known().cloned().expect("a read name"); - let native_id = automation_id(head); - assert_eq!(target.role.known(), Some(&role), "the roles are equal"); - assert_eq!(target.name.known(), Some(&name), "the names are equal"); - assert_eq!( - automation_id(target), - native_id, - "the identifiers are equal, so no identity tier separates the pair" - ); - - let head_hash = crate::tree::resolve::resolve_match::bounds_hash_of(head) - .expect("the head candidate has a positive-area bounds hash"); - let target_hash = crate::tree::resolve::resolve_match::bounds_hash_of(target) - .expect("the stored candidate has a positive-area bounds hash"); - assert_ne!( - head_hash, target_hash, - "the two controls must occupy distinct rectangles, or the tie-break has nothing to pick by" - ); - let rect = *target - .ref_evidence - .bounds - .known() - .expect("a read bounds rectangle"); - - let pid = agent_desktop_core::ProcessId::from(std::process::id()); - let mut entry = crate::tree::walk::walker_fake::ref_entry(&role); - entry.process = agent_desktop_core::RefProcess { - pid, - process_instance: Some( - crate::system::process_identity::token_for_pid(pid) - .expect("the token read answers") - .expect("a live process has a token"), - ), - }; - entry.identity.name = Some(name); - entry.identity.native_id = native_id; - entry.geometry = agent_desktop_core::RefGeometry { - bounds_space: None, - bounds: Some(rect), - bounds_hash: Some(target_hash), - }; - entry.source.source_window_id = Some(format!("w-{}", hosted.handle)); - entry.source.source_window_title = Some(TITLE_MARKER.into()); - - for evidence in &duplicates { - assert_eq!( - crate::tree::resolve::resolve_match::candidate_outcome(&entry, evidence), - CandidateOutcome::Matched, - "both live controls satisfy every identity tier" - ); - } - assert!( - matches!( - classify_search(&[Some(head_hash), Some(target_hash)], false, &entry), - SearchVerdict::Resolved(1) - ), - "the stored hash selects the second candidate, so index zero is the wrong answer" - ); - - let handle = resolve_element_strict(&entry, deadline) - .expect("the stored hash separates the pair, so resolution settles on one of them"); - - let resolved = handle - .downcast_ref::() - .expect("the resolved handle carries a UI Automation element"); - let (_, resolved_evidence, _) = source.evidence(resolved); - assert_eq!( - crate::tree::resolve::resolve_match::bounds_hash_of(&resolved_evidence), - Some(target_hash), - "the resolver returned the candidate the stored hash selected, not the first collected" - ); -} - -/// Collects the marked controls' evidence together with the resolver's own -/// DFS path to each - the fixture root down to a marked node, one child -/// index per hop - so a test can build a stored ref whose `scope.path` names -/// a specific duplicate rather than only checking what the broad search -/// collects. Mirrors `collect_marked`'s own enumeration -/// (`enumerate_children`), so a path recorded here is the exact path the -/// fast path itself would walk. -fn catalogue_marked( - source: &UiaTreeSource, - element: &UIAElement, - depth: u8, - budget: &WalkBudget, - prefix: &mut Vec, - out: &mut Vec<(RefPath, LocatorEvidence)>, -) { - if depth >= 8 { - return; - } - let (_, evidence, _) = source.evidence(element); - if evidence - .name - .known() - .is_some_and(|name| name == NAME_MARKER) - { - let mut path = RefPath::default(); - path.extend_from_slice(prefix); - out.push((path, evidence)); - } - let mut ignored = false; - let Ok(children) = enumerate_children(source, element, budget, &mut ignored) else { - return; - }; - for (index, child) in children.iter().enumerate() { - prefix.push(index); - catalogue_marked(source, child, depth + 1, budget, prefix, out); - prefix.pop(); - } -} - -/// Builds the stored ref the duplicate-pair fast-path tests share, differing -/// only in the path and bounds hash under test. -fn duplicate_entry( - window_handle: isize, - role: String, - name: String, - native_id: Option, - rect: agent_desktop_core::Rect, - bounds_hash: Option, - path: RefPath, -) -> RefEntry { - let pid = agent_desktop_core::ProcessId::from(std::process::id()); - let mut entry = crate::tree::walk::walker_fake::ref_entry(&role); - entry.process = agent_desktop_core::RefProcess { - pid, - process_instance: Some( - crate::system::process_identity::token_for_pid(pid) - .expect("the token read answers") - .expect("a live process has a token"), - ), - }; - entry.identity.name = Some(name); - entry.identity.native_id = native_id; - entry.geometry = agent_desktop_core::RefGeometry { - bounds_space: None, - bounds: Some(rect), - bounds_hash, - }; - entry.source.source_window_id = Some(format!("w-{window_handle}")); - entry.source.source_window_title = Some(TITLE_MARKER.into()); - entry.scope.path = path; - entry -} - -/// The concrete failure the fast path's early return created, and the -/// unchanged case beside it. A ref stored against one duplicate control lands, -/// at its own stored path, on the *other* one - same role, name and -/// `AutomationId` - because the index the ref names now holds a different -/// sibling. Every identity tier agrees on the impostor, so only the stored -/// bounds hash can refuse it. Before the fix, `accept_path_landing` returned -/// the impostor immediately and the broad search - the only tier that applies -/// the bounds hash - never ran; after it, the contradiction falls through to -/// the broad search, which finds both duplicates and tie-breaks onto the one -/// the stored hash actually names. The same catalogue drives the unchanged -/// case too: a landing whose live hash agrees with the stored one is still -/// accepted directly, so the refutation is not a rejection of every mismatch. -#[test] -fn the_fast_path_refuses_a_contradicting_hash_and_still_accepts_an_agreeing_one() { - crate::tree::fixture::ensure_test_apartment(); - let hosted = - DuplicatePairWindow::create(PairGeometry::Separated).expect("a duplicate window hosts"); - let deadline = crate::tree::walk::walker_fake::deadline(); - let root = crate::tree::automation::root_from_hwnd(hosted.handle, deadline) - .expect("the hosted window resolves"); - let source = UiaTreeSource::for_root(&root).expect("a tree source"); - let prepared = source.prepare_root(&root).expect("a prepared root"); - let budget = WalkBudget::new(10, deadline); - - let mut prefix = Vec::new(); - let mut catalogued = Vec::new(); - catalogue_marked(&source, &prepared, 0, &budget, &mut prefix, &mut catalogued); - assert_eq!( - catalogued.len(), - 2, - "the host must present exactly two marked controls" - ); - let (_, stored) = &catalogued[0]; - let (impostor_path, impostor) = &catalogued[1]; - let role = stored.role.known().cloned().expect("a read role"); - let name = stored.name.known().cloned().expect("a read name"); - let native_id = automation_id(stored); - assert_eq!( - automation_id(impostor), - native_id, - "the pair must share identity, or the contradiction has nothing to disprove" - ); - - let stored_hash = crate::tree::resolve::resolve_match::bounds_hash_of(stored) - .expect("the stored candidate has a positive-area bounds hash"); - let impostor_hash = crate::tree::resolve::resolve_match::bounds_hash_of(impostor) - .expect("the impostor candidate has a positive-area bounds hash"); - assert_ne!( - stored_hash, impostor_hash, - "the two controls must occupy distinct rectangles, or the fast path has nothing to refute" - ); - let rect = *stored - .ref_evidence - .bounds - .known() - .expect("a read bounds rectangle"); - - let contradicting = duplicate_entry( - hosted.handle, - role.clone(), - name.clone(), - native_id.clone(), - rect, - Some(stored_hash), - impostor_path.clone(), - ); - let landed_on_impostor = - walk_stored_path(&source, &prepared, &contradicting.scope.path, &budget) - .expect("the path walk answers") - .element - .expect("the stored path still lands on a live element"); - assert!( - accept_path_landing(&source, &landed_on_impostor, &contradicting).is_none(), - "a live bounds hash that contradicts the stored one must not be accepted by the fast path" - ); - - let handle = resolve_element_strict(&contradicting, deadline) - .expect("the broad search still finds the stored candidate by its bounds hash"); - let resolved = handle - .downcast_ref::() - .expect("the resolved handle carries a UI Automation element"); - let (_, resolved_evidence, _) = source.evidence(resolved); - assert_eq!( - crate::tree::resolve::resolve_match::bounds_hash_of(&resolved_evidence), - Some(stored_hash), - "resolution must return the element the stored hash names, not the impostor at the stored path" - ); - - let equal_hash_entry = duplicate_entry( - hosted.handle, - role, - name, - native_id, - rect, - Some(impostor_hash), - impostor_path.clone(), - ); - assert!( - accept_path_landing(&source, &landed_on_impostor, &equal_hash_entry).is_some(), - "equal hashes must not be refused" - ); -} - -/// The guard against the fix over-tightening: an unread live bounds hash is -/// never a refutation. Pinned directly over evidence, no live window needed, -/// because a live UI Automation button never fails a bounds read - the -/// tri-state discipline this predicate honours exists for providers that do, -/// the same discipline `resolve_search_admission_tests.rs` pins for -/// `admit_node`. Without this direction, treating any live/stored -/// disagreement as a refutation - including an unread one - would strand -/// every ref whose bounds momentarily failed to read. -#[test] -fn a_path_landing_with_an_unreadable_live_bounds_hash_is_still_accepted() { - let mut entry = crate::tree::walk::walker_fake::ref_entry("button"); - entry.identity.name = Some("Save".into()); - entry.geometry.bounds_hash = Some(0xdead_beef); - entry.scope.path_is_absolute = true; - let unreadable = LocatorEvidence { - role: LocatorField::Known("button".into()), - name: LocatorField::Known("Save".into()), - value: LocatorField::Absent, - description: LocatorField::Absent, - identifiers: agent_desktop_core::IdentifierEvidence::absent(), - states: LocatorField::Absent, - ref_evidence: agent_desktop_core::LocatorRefEvidence { - window_bounds: LocatorField::Absent, - bounds: LocatorField::Unknown, - available_actions: LocatorField::Absent, - descriptors: agent_desktop_core::NodeDescriptor::default(), - }, - }; - - assert!( - !geometry_contradicts(&entry, &unreadable), - "an unreadable live bounds hash must never contradict the stored one" - ); - assert_eq!( - admit_node(&entry, &unreadable), - NodeAdmission::Collect, - "the identity tiers alone still settle the match, so the landing is still collected" - ); -} diff --git a/crates/windows/src/tree/resolve/resolve_verdict_tests.rs b/crates/windows/src/tree/resolve/resolve_verdict_tests.rs deleted file mode 100644 index d1f816fb4..000000000 --- a/crates/windows/src/tree/resolve/resolve_verdict_tests.rs +++ /dev/null @@ -1,74 +0,0 @@ -//! How an attempt's two tiers compose into its verdict: either tier's unread -//! region alone withholds a negative one, and only a walk that read everything -//! may settle `STALE_REF`. - -use crate::tree::resolve::*; - -fn entry() -> RefEntry { - let mut entry = crate::tree::walk::walker_fake::ref_entry("button"); - entry.identity.name = Some("Save".to_string()); - entry.scope.path_is_absolute = true; - entry -} - -fn landing(unread_region: bool) -> PathLanding { - PathLanding { - element: None, - unread_region, - } -} - -/// The path tier is the only tier that descends past the broad search's depth -/// cap, so a gap it met can be the sole evidence that the tier able to reach -/// the stored element never got to look. An attempt that let that gap die with -/// the tier would answer `STALE_REF` - "the element is gone" - off a walk that -/// never finished. -#[test] -fn a_gap_on_the_stored_path_alone_withholds_the_negative_verdict() { - assert!( - matches!( - attempt_verdict(&[], &landing(true), false, &entry()), - SearchVerdict::Incomplete - ), - "a candidate-less search must not settle stale while the path walk left a region unread" - ); -} - -/// The search's own gap withholds it by the same rule, so neither tier is -/// privileged and neither can be dropped in favour of the other. -#[test] -fn a_gap_in_the_broad_search_alone_withholds_the_negative_verdict() { - assert!(matches!( - attempt_verdict(&[], &landing(false), true, &entry()), - SearchVerdict::Incomplete - )); -} - -/// The control both pins above depend on: with every region read and no -/// candidate found, the attempt does settle. Without this, withholding on an -/// unread region would be indistinguishable from never settling at all. -#[test] -fn a_walk_that_read_everything_and_found_nothing_settles_stale() { - assert!(matches!( - attempt_verdict(&[], &landing(false), false, &entry()), - SearchVerdict::Stale - )); -} - -/// The positive verdict is withheld on the same evidence: a sole match found -/// while the path walk left a region unread could be the wrong one of two, and -/// the unread part is exactly where the second would be. -#[test] -fn a_sole_match_is_retried_while_the_path_walk_left_a_region_unread() { - assert!(matches!( - attempt_verdict(&[None], &landing(true), false, &entry()), - SearchVerdict::Incomplete - )); - assert!( - matches!( - attempt_verdict(&[None], &landing(false), false, &entry()), - SearchVerdict::Resolved(0) - ), - "the same sole match resolves once nothing is left unread" - ); -} diff --git a/crates/windows/src/tree/resolve_dead_process_tests.rs b/crates/windows/src/tree/resolve_dead_process_tests.rs index 23f9db239..86b8cdc0b 100644 --- a/crates/windows/src/tree/resolve_dead_process_tests.rs +++ b/crates/windows/src/tree/resolve_dead_process_tests.rs @@ -9,7 +9,7 @@ use agent_desktop_core::{Deadline, DeliverySemantics, ErrorCode, ProcessId, RefEntry}; use crate::tree::fixture::{HostedFixture, ensure_test_apartment}; -use crate::tree::resolve::resolve_element_strict; +use crate::tree::resolve::resolve_anchor::resolve_locator_anchor; fn entry_for(fixture: &HostedFixture, token: String, window_handle: isize) -> RefEntry { let mut entry = crate::tree::walk::walker_fake::ref_entry("button"); @@ -49,7 +49,7 @@ fn a_dead_owning_process_settles_terminally_on_the_first_attempt() { let deadline = Deadline::after(5_000).expect("a deadline"); let started = std::time::Instant::now(); - let result = resolve_element_strict(&entry, deadline); + let result = resolve_locator_anchor(&entry, deadline); let elapsed = started.elapsed(); let error = match result { @@ -88,7 +88,7 @@ fn a_live_owner_with_a_reassigned_handle_stays_on_the_retryable_path() { let entry = entry_for(&fixture_a, token_a, fixture_b.handle()); let deadline = Deadline::after(5_000).expect("a deadline"); - let result = resolve_element_strict(&entry, deadline); + let result = resolve_locator_anchor(&entry, deadline); let error = match result { Err(error) => error, diff --git a/crates/windows/src/tree/walk/walker_enumerate.rs b/crates/windows/src/tree/walk/walker_enumerate.rs index 13ebc9a94..db8d039c7 100644 --- a/crates/windows/src/tree/walk/walker_enumerate.rs +++ b/crates/windows/src/tree/walk/walker_enumerate.rs @@ -184,12 +184,10 @@ impl<'a, S: TreeSource> TreeWalk<'a, S> { if !subtree_complete { self.complete = false; } - Some(ObservedSubtree::new( - evidence, - children, - subtree_complete, - children_count, - )) + Some( + ObservedSubtree::new(evidence, children, subtree_complete, children_count) + .with_wrapper(web_wrapper), + ) } /// The child enumeration loop. diff --git a/crates/windows/src/tree/walk/walker_fake.rs b/crates/windows/src/tree/walk/walker_fake.rs index 10b8e0760..10e4a2dce 100644 --- a/crates/windows/src/tree/walk/walker_fake.rs +++ b/crates/windows/src/tree/walk/walker_fake.rs @@ -12,10 +12,7 @@ use crate::tree::walk::walker::{ NodeKey, TreeSource, WalkBudget, WalkOutcome, walk_from_root, walk_vocabulary, }; #[cfg(target_os = "windows")] -use crate::tree::{ - element::UIAElement, resolve::resolve_search::enumerate_children, - walk::walker_source::UiaTreeSource, -}; +use crate::tree::{element::UIAElement, walk::walker_source::UiaTreeSource}; #[cfg(target_os = "windows")] use std::ops::ControlFlow; @@ -195,6 +192,7 @@ pub(crate) fn ref_entry(role: &str) -> RefEntry { value: None, description: None, native_id: None, + unique_at_capture: false, }, geometry: agent_desktop_core::RefGeometry { bounds_space: None, @@ -211,6 +209,7 @@ pub(crate) fn ref_entry(role: &str) -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: agent_desktop_core::SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: agent_desktop_core::RefScope { root_ref: None, @@ -265,8 +264,8 @@ struct ScanLimits<'a> { /// re-implemented this recursion by hand; only the predicate and the value /// built on a match differ between them, so both live here once. A child /// enumeration failure anywhere aborts the whole search, as it does in the -/// production search this mirrors (`resolve_search::enumerate_children` -/// propagates the error with `?`). +/// production path walk this mirrors ([`fake_children`] propagates the error +/// with `?`). #[cfg(target_os = "windows")] pub(crate) fn scan_subtree( source: &UiaTreeSource, @@ -299,8 +298,7 @@ fn scan_at( if let ControlFlow::Break(value) = visit(prefix, properties, evidence, failed) { return Ok(Some(value)); } - let mut ignored = false; - let children = enumerate_children(source, element, limits.budget, &mut ignored)?; + let children = fake_children(source, element, limits.budget)?; for (index, child) in children.iter().enumerate() { prefix.push(index); let found = scan_at(source, child, limits, prefix, visit)?; @@ -312,50 +310,20 @@ fn scan_at( Ok(None) } -/// Depth-first visit of every node in the subtree, index path included, -/// tolerating a child-enumeration failure by simply not descending past it - -/// the rest of the tree is still worth cataloguing. -/// -/// The counterpart to `scan_subtree` for the tests that build a catalogue or -/// a collection instead of stopping at a first match: `visit` decides what, -/// if anything, to keep by closing over its own accumulator. +/// The fake walk's child list, enumerated under the same bounded loop and +/// policy the anchor descends with: a settled absence is a whole empty list, +/// and any other read failure aborts the scan rather than inventing children. #[cfg(target_os = "windows")] -pub(crate) fn walk_all( +fn fake_children( source: &UiaTreeSource, - root: &UIAElement, + element: &UIAElement, budget: &WalkBudget, - max_depth: u8, - visit: &mut impl FnMut(&[usize], LocatorEvidence), -) { - walk_all_at( +) -> Result, AdapterError> { + crate::tree::resolve::resolve_search::descent::read_children( source, - root, - ScanLimits { budget, max_depth }, - &mut Vec::new(), - visit, - ); -} - -#[cfg(target_os = "windows")] -fn walk_all_at( - source: &UiaTreeSource, - element: &UIAElement, - limits: ScanLimits<'_>, - prefix: &mut Vec, - visit: &mut impl FnMut(&[usize], LocatorEvidence), -) { - if prefix.len() >= limits.max_depth as usize { - return; - } - let (_, evidence, _) = source.evidence(element); - visit(prefix, evidence); - let mut ignored = false; - let Ok(children) = enumerate_children(source, element, limits.budget, &mut ignored) else { - return; - }; - for (index, child) in children.iter().enumerate() { - prefix.push(index); - walk_all_at(source, child, limits, prefix, visit); - prefix.pop(); - } + element, + budget, + &crate::tree::resolve::resolve_anchor::ANCHOR_DESCENT, + None, + ) } diff --git a/docs/json-output.md b/docs/json-output.md index 945814017..665541be6 100644 --- a/docs/json-output.md +++ b/docs/json-output.md @@ -4,7 +4,7 @@ Every command returns structured JSON: ```json { - "version": "2.5", + "version": "2.6", "ok": true, "command": "click", "data": { @@ -18,7 +18,7 @@ Errors include machine-readable codes and recovery hints: ```json { - "version": "2.5", + "version": "2.6", "ok": false, "command": "click", "error": { @@ -110,6 +110,18 @@ Each of these commands now returns the adapter's own report inside `data`: - The Rust `ActionResult` and `OperationReport` readers reject a payload without an explicit `disposition`; a missing disposition no longer reads back as delivered-unverified. - The C ABI is unchanged. Status-only functions such as `ad_close_app` and `ad_drag` still cannot expose the disposition, steps or target of a successful call. +## Version 2.6 migration + +Version 2.6 changes which refs resolve on Windows and Linux, and on macOS for an element that yields no address. Such a ref resolves only when one complete read of its window proved it unique when it was captured and exactly one live element matches its role and stable fields now. The only new payload fields are `data.identity_unproven` and, on a ref action whose budget ran out after a refusal, `error.details.refusal`. + +- A ref to one of two identical controls, which earlier resolved by position or geometry, now fails `AMBIGUOUS_TARGET` with `details.kind: "identity_not_unique"` before anything is read. So does a ref captured from a window that could not be read whole within the snapshot timeout. +- A refmap written before 2.6 records no capture proof, so all of its refs on these platforms fail the same way. Take a new snapshot. +- An identical twin that appears after capture fails `AMBIGUOUS_TARGET` with `details.kind: "saved_ref_ambiguous"`. +- All of these carry `details.retryable: false` and a `not_delivered` disposition. `disposition.retry` stays `safe` because nothing was sent, but repeating the same ref returns the same refusal. +- When a ref action's full resolution under the lease is refused in a way that may clear, the action waits again within its `--timeout-ms`. If that budget then runs out, the error carries the refusal in `details.refusal`: its `code`, `message`, and `details` when the refusal had any. + +A snapshot that needed one complete read of its window to prove its refs, and could not get it, says so with `"identity_unproven": true` in `data`. `complete` still describes the presented tree. On Windows and Linux, and on macOS for an element with no address, every ref of that snapshot fails `AMBIGUOUS_TARGET` with `details.kind: "identity_not_unique"`; retake the snapshot with a larger `--timeout-ms`. The field is omitted otherwise. + ## Headed action verification - Headed ref interactions activate the exact source app/window before final pointer hit testing. Read-only readiness polling must not prevent a background target from reaching that focus phase; final hit testing remains mandatory before delivery. @@ -127,6 +139,21 @@ Earlier output synthesized a display name, falling back to `value` or `name` as always present must tolerate its absence and apply their own fallback to `role` or `value`. +## Ref re-identification details + +Every ref is re-identified from its stored entry or refused, and `details.kind` names which. The core resolver refuses a saved ref with one of these four kinds: + +| `details.kind` | Code | Meaning | +|------|------|---------| +| `retained_object_stale` | `STALE_REF` | macOS: the element's recorded address is dead, or its process, role or recorded identifier no longer matches. `details.reason` is `dead`, `pid`, `role` or `native_identifier`. No search runs, so the ref cannot land on a look-alike. Take a fresh snapshot. | +| `saved_ref_absent` | `STALE_REF` | No live element matches the stored role and stable fields over complete evidence, so the element is gone. Take a fresh snapshot. | +| `saved_ref_ambiguous` | `AMBIGUOUS_TARGET` | An element identical to the captured one appeared, so two or more live elements match the stored role and stable fields. A fresh snapshot refuses both as twins; act on a control that is unique in the window. | +| `identity_not_unique` | `AMBIGUOUS_TARGET` | The ref was not proven the only element matching its role and stable fields when it was captured, so nothing is searched. Either an identical twin existed, or the window could not be read whole within the snapshot timeout (retake the snapshot with a larger `--timeout-ms`). It is reached on Windows, on Linux, and on macOS when the element yields no address. A twin is refused by every snapshot; only an explicit coordinate click (`--headed mouse-click --xy`) can pick one, it takes the foreground, and it is unavailable on a Linux Wayland session. | + +The fifth outcome is `resolution_identity_unknown`, which carries `TIMEOUT`: a live field could not be read, so the identity count stayed unproven. It is retryable. Linux menu and tray refs report the same kind as `STALE_REF`. + +After the core resolver chooses the element, the platform adapter lands it, and a landing failure keeps the adapter's own `details.kind` (for example `resolve_no_candidate` on Linux and Windows, or `resolution_complete_absence` on macOS). The platform skills list those kinds. `saved_ref_absent`, `saved_ref_ambiguous` and `identity_not_unique` carry `details.retryable: false`, because waiting cannot change the answer. Every one of these is `not_delivered`. + ## Error codes | Code | Meaning | diff --git a/docs/solutions/best-practices/identity-fingerprint-against-os-reorder-2026-04-16.md b/docs/solutions/best-practices/identity-fingerprint-against-os-reorder-2026-04-16.md index 5ba059266..7c1c58a12 100644 --- a/docs/solutions/best-practices/identity-fingerprint-against-os-reorder-2026-04-16.md +++ b/docs/solutions/best-practices/identity-fingerprint-against-os-reorder-2026-04-16.md @@ -79,7 +79,7 @@ discards the bytes rather than attributing them to the wrong surface Backend choice (Modern vs Legacy) is orthogonal: the fingerprint applies regardless of which capture path produces the PNG. -**Element identity inside repeated collections.** A repeated native identifier does not identify a row. After a filter, a different row can take the original position with the same identifier and bounds, and disabling the path optimization does not help, because a broad search still matches them. Do not freeze mutable values to fix this, since ordinary text inputs must stay editable through the same ref. On macOS, refs carry no retained-object token, so `ref_identity::identity_match` in `crates/core/src/refs/ref_identity.rs` returns `Match` on the shared identifier and the aliasing stays reachable there. +**Element identity inside repeated collections.** A repeated native identifier does not identify a row. After a filter, a different row can take the original position with the same identifier and bounds, and disabling the path optimization does not help, because a broad search still matches them. Do not freeze mutable values to fix this, since ordinary text inputs must stay editable through the same ref. macOS keeps the element's own accessibility address, so `ref_identity::identity_match` in `crates/core/src/refs/ref_identity.rs` refuses a removed row or a twin instead of aliasing it. Every other platform resolves a ref from its capture proof: it was unique at capture and exactly one live element matches its full predicate now, so a twin is refused as `AMBIGUOUS_TARGET` before the window is read, and a unique element that was destroyed and recreated is acted on as its replacement. ## Prevention diff --git a/docs/solutions/best-practices/playwright-grade-desktop-reliability-2026-06-02.md b/docs/solutions/best-practices/playwright-grade-desktop-reliability-2026-06-02.md index 6af14cadc..005d07240 100644 --- a/docs/solutions/best-practices/playwright-grade-desktop-reliability-2026-06-02.md +++ b/docs/solutions/best-practices/playwright-grade-desktop-reliability-2026-06-02.md @@ -34,10 +34,15 @@ selected namespace has no match. ### 2. Re-resolve strictly at action time -Load the saved `RefEntry`, then let the adapter re-identify the live element -from role, process generation, source, stable text identity, geometry, path, -and native evidence. Zero matches is `STALE_REF`; more than one plausible match -is `AMBIGUOUS_TARGET`. Mutable field values are not stable identity. +Load the saved `RefEntry` and re-identify the live element before acting. An +entry carrying an element token (macOS) resolves through the adapter, which owns +re-identification on that platform. Every other entry resolves in core +(`resolve_saved_ref`) from its capture proof: it must have been unique in one +complete read of its window when captured, and exactly one live element must +match its role, native id, stable name and description now. Zero matches is +`STALE_REF`; two or more, or a ref never proven unique, is `AMBIGUOUS_TARGET`. +Mutable values, geometry and path are never identity; the adapter only lands +the element core chose, and core re-checks the landing. ### 3. Separate actionability from dispatch @@ -80,10 +85,12 @@ and its timeout variant run that core command path. Route a new language-binding equivalent of a CLI command through the high-level path, and test both the observed effect and the reported mechanism for headed and headless input. -`ad_execute_action` and the struct-based direct action entrypoints are -deliberately lower-level escape hatches. They act on a caller-held native handle -or entry, may apply the supplied policy verbatim, and do not claim RefStore-backed -CLI parity. Their safety docs must say which boundary they skip: strict +`ad_execute_action` and `ad_execute_action_with_policy` are deliberately +lower-level escape hatches. They act on a caller-held native handle, may apply +the supplied policy verbatim, and do not claim RefStore-backed CLI parity. The +exact-entry calls (`ad_resolve_element_exact`, +`ad_execute_ref_action_exact_with_policy`) refuse every well-formed entry from +ABI 7, because an entry a caller builds carries no capture proof. Their safety docs must say which boundary they skip: strict resolution, preflight, or base-policy elevation. ABI structs are versioned and layout-pinned alongside the committed C header. diff --git a/probes/linux/FINDINGS.md b/probes/linux/FINDINGS.md index 7047f4b9b..1a73d4240 100644 --- a/probes/linux/FINDINGS.md +++ b/probes/linux/FINDINGS.md @@ -980,6 +980,21 @@ Pre-code probes for menus first, then notifications (L32-1 to L32-16), then the +## Area L42 — ref re-identification and capture uniqueness + +Measured on a second rig, not the machine in Environment: Docker on macOS, an `ubuntu:24.04` session with `Xvfb :99` (1400x900x24), a private AT-SPI bus, Openbox and `GSETTINGS_BACKEND=keyfile` with `toolkit-accessibility` on. Three fixture apps were written for the run, one per toolkit (GTK3 through `python3-gi`, GTK4, Qt6 through `python3-pyqt6`); package versions were not recorded this pass. Each app shows four rows, a label `Alpha` to `Delta` plus one identical `Delete` button per row, one unique `Remove Beta`, `Rerender` (destroys and recreates every row), and `Trap`, `Drop Trap` and `Hide Trap`. Every row and button carries a creation-time instance id and the app logs `create` and `fire` to a file outside the accessibility channel, so an effect is proved by which widget instance ran its handler, never by `ok`. Release CLI, headless semantic actions only; no `--headed` pass. + +| id | probe | stack | scope | phases.md expectation | observed | verdict | action | +| --- | --- | --- | --- | --- | --- | --- | --- | +| L42-1 | Unique ref `Remove Beta` clicked once per toolkit after `snapshot --app idv- -i --compact` | dbus | api-contract | none: measures the shipped rule in `crates/core/src/refs/resolve_saved_ref.rs` | GTK3 1/1 resolved and fired `Beta#2`; GTK4 1/1 `Beta#2`; Qt6 1/1 `Beta#2`. The effector is the captured row's own button instance | CONFIRMS | The single live full-predicate match resolves and lands on the element it was captured from, on all three toolkits | +| L42-2 | Four identical `Delete` refs, one per row, each clicked once per toolkit | dbus | api-contract | none: same shipped rule | GTK3 4/4 refused `AMBIGUOUS_TARGET`, `details.kind: identity_not_unique`, 0 wrong rows, 0 effects; GTK4 4/4 refused, 0 wrong, 0 effects; Qt6 4/4 refused, 0 wrong, 0 effects. A refused ref costs 5.0-16.8 ms against 33.0-127.3 ms for a resolved one (L42-6), so no click reached the tree | CONFIRMS | Twins are refused at capture, before any live read. Report it as a capture verdict, not a failed search | +| L42-3 | Unique ref clicked twice, with the element removed between the clicks; then refs captured before `Rerender` | dbus | api-contract | none: same shipped rule | First click `ok` and fired `Beta#10`; second click `STALE_REF`, `details.kind: saved_ref_absent`, no effect, identical on all three toolkits. After `Rerender`, the four `Delete` refs were refused 4/4 (`identity_not_unique`) and the unique ref 1/1 resolved onto the **recreated** widget (`Beta#18`) | CONFIRMS | Zero live matches is stale. A unique name re-resolves onto a replacement element, so the rule protects twins only; do not read resolution as "the same instance is still there" | +| L42-4 | Ref captured, widget destroyed, click issued immediately, 12 samples per toolkit (GTK4 unparent, Qt6 `hide()` plus `setParent(None)` with the reference kept) | dbus | api-contract | none: same shipped rule | GTK3 `destroy()` drops the accessible at once: 7/7 `STALE_REF`, no effect, absent from the listing. GTK4 `unparent()` keeps it listed (3/3 immediately, gone after 0.5 s): 11/12 `ok` with `delivered_unverified` and **no effect**, 1/12 `ACTION_FAILED`, 0 `STALE_REF`. Qt6 keeps it listed and live: 6/6 `ok` and the dead widget's handler fired (`Trap#1`) | NEW-EDGE | A toolkit that keeps a detached accessible listed lets a ref resolve onto a control the user can no longer see. GTK4 reports success with no effect, Qt6 delivers to the zombie. Both are platform limits no identity rule reaches; stated in the Linux skill | +| L42-5 | Same destruction with the widget only hidden instead of removed, 3 samples per toolkit | dbus | api-contract | none: same shipped rule | GTK3 3/3 `ACTION_FAILED`, no effect. GTK4 3/3 `ok`, handler fired on an invisible widget. Qt6 3/3 `ok`, handler fired, still listed | NEW-EDGE | Hidden is not proven invisible by the identity check. The action gate is what must refuse, and it does on GTK3 only | +| L42-6 | `get --property text`, median of 30 runs, warm process | dbus | box | none | GTK3 33.0 ms (24.9-152.3), GTK4 127.3 ms (68.6-244.3), Qt6 43.7 ms (34.5-260.2); the same call on a refused twin ref took 5.0 / 5.7 / 16.8 ms | NEW-EDGE | A resolved ref costs one window observation plus a path landing; a refused ref costs neither. Budget the observation, not the search, when sizing a ref-action timeout | + +The rig could not read its own log label through `get`: `statictext` gets no ref, and a non-editable GTK3 entry came back as role `document` with `value: null`. Label text was read from a fresh snapshot through `find --text 'deleted:'` instead, and every effect was cross-checked against the application-side instance log. Not exercised: `--headed` physical clicks, Windows and macOS, and an app that crashes or hangs mid-run. + ## Contradictions index | row | phases.md section contradicted | phases.md correction | diff --git a/skills/agent-desktop-ffi/SKILL.md b/skills/agent-desktop-ffi/SKILL.md index 8ce2527b3..100d4e4d1 100644 --- a/skills/agent-desktop-ffi/SKILL.md +++ b/skills/agent-desktop-ffi/SKILL.md @@ -88,7 +88,9 @@ are not exposed over the ABI. A bare `@eN` ref returns including `TypeText`, defaults to `headless`. Pass `AD_POLICY_KIND_FOCUS_FALLBACK` (1) to allow focus without cursor movement, or `AD_POLICY_KIND_HEADED` (2) to allow physical cursor and keyboard delivery. `ad_execute_ref_action_exact_with_policy` - runs the same preflight from an `AdExactRefEntry`. + always fails with `AD_RESULT_ERR_ACTION_NOT_SUPPORTED`: an exact entry has no + retained element token and no capture proof, so it cannot be re-identified. + Use `ad_execute_by_ref` instead. - **Generation-safe direct APIs.** Every direct targeting function takes process-generation evidence: `AdExactRefEntry`, `AdExactWindowInfo`, @@ -100,9 +102,14 @@ are not exposed over the ABI. A bare `@eN` ref returns is null. A missing `sheet` or `menu` next to a listed kind is unknown, not absent. Only the Windows adapter fills it; macOS and Linux return null. -- **Native handles.** `ad_resolve_element_exact` and `ad_find_exact` return an - `AdNativeHandle`. `ad_execute_action` (headless, no preflight) and - `ad_execute_action_with_policy` act on it directly, bypassing the ref-action +- **Native handles.** `ad_find_exact` returns an `AdNativeHandle`. It re-identifies the + element before publishing it: it raises the landed handle against the selected entry's + recorded role, native id, name and description, and returns `AD_RESULT_ERR_STALE_REF` + instead of a handle when a different control occupies the anchored path. + `ad_resolve_element_exact` takes an exact ref entry and always fails with + `AD_RESULT_ERR_ACTION_NOT_SUPPORTED` for the same reason as the exact action call. + `ad_execute_action` (headless, no preflight) and + `ad_execute_action_with_policy` act on a handle directly, bypassing the ref-action pipeline. Release every handle with `ad_free_handle(adapter, &handle)` on the same adapter and thread, before `ad_adapter_destroy`, on every platform. It zeroes `handle.ptr`, so a second call is a no-op. diff --git a/skills/agent-desktop-ffi/references/build-and-link.md b/skills/agent-desktop-ffi/references/build-and-link.md index 1f58adb56..c94556033 100644 --- a/skills/agent-desktop-ffi/references/build-and-link.md +++ b/skills/agent-desktop-ffi/references/build-and-link.md @@ -245,7 +245,7 @@ lib = ctypes.CDLL("./target/release-ffi/libagent_desktop_ffi.dylib") # ABI handshake lib.ad_init.restype = c_int32 lib.ad_init.argtypes = [ctypes.c_uint32] -AD_ABI_VERSION_MAJOR = 6 # sync with header macro +AD_ABI_VERSION_MAJOR = 7 # sync with header macro rc = lib.ad_init(AD_ABI_VERSION_MAJOR) assert rc == 0, f"ABI mismatch: rc={rc}" diff --git a/skills/agent-desktop-ffi/references/ownership.md b/skills/agent-desktop-ffi/references/ownership.md index a0bf67e72..89ed21cd4 100644 --- a/skills/agent-desktop-ffi/references/ownership.md +++ b/skills/agent-desktop-ffi/references/ownership.md @@ -53,8 +53,10 @@ Free it with `ad_free_string`. On an infrastructure error `*out` stays null | Allocates | Frees with | |----------------------------------------------------------------------------------------|-----------------------------------------| | `ad_get_tree_exact(adapter, win, opts, &out)` | `ad_free_tree(&out)` | -| `ad_resolve_element_exact(adapter, entry, &handle)` | `ad_free_handle(adapter, &handle)`: zeroes `handle.ptr` so a follow-up call is a no-op | -| `ad_find_exact(adapter, win, query, &handle)` | same as `ad_resolve_element_exact` | +| `ad_find_exact(adapter, win, query, &handle)` | `ad_free_handle(adapter, &handle)`: zeroes `handle.ptr` so a follow-up call is a no-op | + +`ad_resolve_element_exact` allocates nothing: it always fails with +`AD_RESULT_ERR_ACTION_NOT_SUPPORTED` and leaves `handle.ptr` null. ### Action results @@ -62,9 +64,11 @@ Free it with `ad_free_string`. On an infrastructure error `*out` stays null |----------------------------------------------------------------------------------------|------------------------------| | `ad_execute_action(adapter, handle, action, &out)` | `ad_free_action_result(&out)` | | `ad_execute_action_with_policy(adapter, handle, action, policy, &out)` | `ad_free_action_result(&out)` | -| `ad_execute_ref_action_exact_with_policy(adapter, entry, action, policy, &out)` | `ad_free_action_result(&out)` | | `ad_notification_action(adapter, &request, &out)`: set `request.identity` from `ad_list_notifications` and choose an explicit non-headless policy; reorder mismatches fail closed | `ad_free_action_result(&out)` | +`ad_execute_ref_action_exact_with_policy` allocates nothing: it always fails with +`AD_RESULT_ERR_ACTION_NOT_SUPPORTED` and leaves `out` zeroed. + ### Clipboard and image buffers | Allocates | Frees with | @@ -130,5 +134,5 @@ In particular: `*out = NULL` before any guard, so on infrastructure failures no allocation is made and `ad_free_string(NULL)` is a safe no-op. -- `ad_*_list` and `ad_resolve_element_exact` / `ad_find_exact` all apply the same +- `ad_*_list` and `ad_find_exact` all apply the same pattern to their handle / list out-params. diff --git a/skills/agent-desktop-ffi/references/threading.md b/skills/agent-desktop-ffi/references/threading.md index 11a8f8347..2aca2846b 100644 --- a/skills/agent-desktop-ffi/references/threading.md +++ b/skills/agent-desktop-ffi/references/threading.md @@ -61,8 +61,9 @@ the owner adapter token no longer exists. ## Native-handle thread ownership -`ad_resolve_element_exact` and `ad_find_exact` return an opaque -`AdNativeHandle`. Native handles are adapter-bound and thread-affine: +`ad_find_exact` returns an opaque `AdNativeHandle`. `ad_resolve_element_exact` +never produces one: it fails with `AD_RESULT_ERR_ACTION_NOT_SUPPORTED`. Native handles +are adapter-bound and thread-affine: - resolve, use, and release a handle on the same thread; - pass the same adapter that produced the handle; diff --git a/skills/agent-desktop-linux/SKILL.md b/skills/agent-desktop-linux/SKILL.md index ececffb2b..014fae694 100644 --- a/skills/agent-desktop-linux/SKILL.md +++ b/skills/agent-desktop-linux/SKILL.md @@ -81,14 +81,21 @@ Observation and semantic actions need no helper tool. requested depth cut. A shell with no document yet stays incomplete when the budget ends. A `--timeout` shorter than the first read returns `TIMEOUT`, never a partial tree. -- **Geometry-only refs require unchanged geometry.** Position and size must - both match, in the ref's recorded coordinate frame; a one-pixel change breaks - identity. After a layout change or a window move, take a fresh snapshot - instead of retrying the old ref. Indistinguishable siblings stay - `AMBIGUOUS_TARGET`: in a Text Editor window with three tabs, the text views - share one rectangle and one state, so a ref to any of them answers - `AMBIGUOUS_TARGET` with `candidate_count` 3 until the extra tabs close. A ref - belongs to the window it was taken in, so two windows with the same title each +- **A ref must have been unique.** Linux stores no element address, so a ref + resolves only when it was the only element in the window its snapshot + captured with its role, native id, name and description, and exactly one live + element still matches. Uniqueness is counted over the whole window, not over + the shallow tree a `--skeleton` or `--max-depth` snapshot presents, so a twin + the shallow tree never showed still refuses the ref. + Position, size and value never count as identity, so a layout change does not + by itself stale a ref, but a twin does: identical buttons, or views that share + a role and carry no other field, refuse with `AMBIGUOUS_TARGET`, + `details.kind: identity_not_unique`, before anything reaches the app, and an + element that is gone is `STALE_REF`, `saved_ref_absent`. Take a + fresh snapshot after a list changes. To pick one of two identical controls on + X11, use `--headed mouse-click --xy`; a Wayland session refuses headed pointer + input, so there the twin has neither a ref nor a pointer path. A ref belongs + to the window it was taken in, so two windows with the same title each resolve their own refs. - **Chromium and Qt apps need a launch flag or variable** to appear on the bus; Electron apps need none, only `--no-sandbox` where `chrome-sandbox` is not @@ -205,9 +212,17 @@ cursor; on Wayland none is drawn. an X11 session. Headless scoped `press` requires live target focus on either display server. Focusing a window on another workspace can switch the user's current desktop. -- Linux exact FFI refs carrying bounds or a bounds hash require window-relative - geometry, even when the ref also carries a name or identifier. Geometry-free - identity refs are supported. +- The exact FFI ref calls (`ad_resolve_element_exact` and + `ad_execute_ref_action_exact_with_policy`) fail with `ACTION_NOT_SUPPORTED` on every + platform; their bounds rule applies first: bounds must be window-relative. +- A ref re-identifies a unique element, not a widget instance. A toolkit that + destroys and recreates its rows (GTK4, Qt6) resolves a ref with a unique name + onto the replacement, so the action lands on the new element. Identical + controls are refused as `AMBIGUOUS_TARGET` instead. +- A toolkit that keeps a removed widget listed (GTK4 unparented, Qt6 + `setParent(None)`) lets a ref resolve onto a control the user can no longer + see. GTK4 then answers `ok` with `delivered_unverified` and no effect. No + identity rule reaches this; check the application's own effect. ## References diff --git a/skills/agent-desktop-linux/references/troubleshooting.md b/skills/agent-desktop-linux/references/troubleshooting.md index eda04708e..a51075a6f 100644 --- a/skills/agent-desktop-linux/references/troubleshooting.md +++ b/skills/agent-desktop-linux/references/troubleshooting.md @@ -27,7 +27,7 @@ Start with `agent-desktop status` and `agent-desktop list-apps`. | `ACTION_NOT_SUPPORTED`, `native_window_unproven` | The display server did not answer the native window read while the listing ran. `launch` reads again until its timeout before it answers this. Run `list-windows` again once the display has settled. A window of another process whose native surfaces cannot be classified stays listed and never refuses the listing | | `STALE_REF` with `kind: resolution_identity_unknown`, or a click timeout with that `last_report` | Strict resolution could not prove a menu ref's live identity. Do not infer a hung app from a readable menu; see the shell reference | | `APP_UNRESPONSIVE` | The app is stopped (`SIGSTOP`) or its main loop is blocked. Resume it; `close-app --force` ends it | -| `STALE_REF` | The UI changed or the window closed. See `details.kind`: `resolve_no_candidate` means no live element matched and is retryable (an action waits its budget before it reports it); `resolve_target_destroyed` needs a replacement target; `menu_closed` needs its opener reopened before a fresh menu snapshot. The last two forbid retrying the old ref | +| `STALE_REF` | The element is gone or no longer matches. `details.kind: saved_ref_absent` means no live element matched the stored role and stable fields, so take a fresh snapshot; `retained_object_stale` never appears on Linux. `menu_closed` needs its opener reopened before a fresh menu snapshot; `resolve_target_destroyed` needs a replacement target; a menu or tray `resolve_no_candidate` is retryable | | `ACTION_NOT_SUPPORTED`, `selection_index_unproven` | The toolkit's selection indices cannot identify accessible rows safely. A headless select cannot map the row and refreshing cannot help. On X11 run `--headed select` with the same ref; see "Toolkit limits" | | `INTERNAL` "private file parent is accessible by group or other users" | The state root is group-writable or world-accessible. `chmod 700` it; see `references/bus-and-permissions.md` | diff --git a/skills/agent-desktop-macos/SKILL.md b/skills/agent-desktop-macos/SKILL.md index eb732c5fc..8047b3a61 100644 --- a/skills/agent-desktop-macos/SKILL.md +++ b/skills/agent-desktop-macos/SKILL.md @@ -49,7 +49,7 @@ Read the core skill first (`agent-desktop skills get desktop`) for the observe, **Identity.** `launch` accepts a display name (`"System Settings"`) or a bundle ID (`com.apple.systempreferences`). Other commands take the display name through `--app`. Window IDs such as `w-4521` stay valid for the life of the window. Find them with `list-windows`. -**Refs follow their element.** A ref stores the element's own accessibility address. If the element is removed or the app restarts, the ref returns `STALE_REF`; it never lands on a look-alike. Identical controls, such as several `Delete` buttons, each keep their own ref. +**Refs carry each element's own address.** A ref turns back into the exact element it was captured from, so identical twins, such as several identical `Delete` buttons, each act on their own row. If the element was removed or the app restarted, the ref is `STALE_REF`, `details.kind: retained_object_stale`, and it never falls back to a search for a look-alike. When an app yields no address for an element, that ref follows the cross-platform rule instead and must have been the only element in its window with its role, native id, name and description. Take a fresh snapshot after a list changes. **Keys.** The primary modifier is `cmd`: `press cmd+c`, `press cmd+s`. Quit shortcuts such as `cmd+q`, `cmd+shift+q`, `cmd+alt+esc`, `ctrl+cmd+q` and `cmd+shift+delete` return `POLICY_DENIED` unless you pass `--force`. To quit an app, use `close-app`. Punctuation keys work in combos, as the symbol (`,` `.` `/` `;` `'` `[` `]` `\` `-` `=` `` ` ``) or by name (`comma`, `period`, `slash`, `semicolon`, `quote`, `leftbracket`, `rightbracket`, `backslash`, `minus`, `equal`, `grave`). Each one is sent with the key that types it in the active keyboard layout, so `press cmd+,` opens Settings on ANSI and JIS keyboards alike. diff --git a/skills/agent-desktop-windows/SKILL.md b/skills/agent-desktop-windows/SKILL.md index e693da8d3..947c629df 100644 --- a/skills/agent-desktop-windows/SKILL.md +++ b/skills/agent-desktop-windows/SKILL.md @@ -91,6 +91,7 @@ observation, actions and capture there fail or return nothing. ## What differs on Windows - Ref actions are semantic UI Automation calls in headless and `--headed` mode, except `right-click`, `double-click`, `triple-click`, `type` and `focus`, which are physical or take focus and need `--headed`. +- A ref resolves only when it was the only element in the window its snapshot captured with its role, native id, name and description, and exactly one live element still matches. Uniqueness counts the whole window, not the shallow tree a `--skeleton` or `--max-depth` snapshot presents, so a twin below the cut still refuses the ref. Identical twins, such as several identical `Delete` buttons, are refused with `AMBIGUOUS_TARGET`, `details.kind: identity_not_unique`, before anything reaches the app; a live second match is `saved_ref_ambiguous`, and a removed element is `STALE_REF`, `saved_ref_absent`. Take a fresh snapshot after a list changes, and use `--headed mouse-click --xy` when you must pick one of two identical controls. - The cursor overlay plays its per-action travel only for headless semantic actions. A `--headed` action moves the real cursor instead. ## References diff --git a/skills/agent-desktop-windows/references/chromium-and-electron.md b/skills/agent-desktop-windows/references/chromium-and-electron.md index 5b0bead77..284830b0e 100644 --- a/skills/agent-desktop-windows/references/chromium-and-electron.md +++ b/skills/agent-desktop-windows/references/chromium-and-electron.md @@ -55,15 +55,11 @@ Take a new snapshot, so the adapter resolves from a new handle. Retrying the sam read cannot fix it. After you launch a WPF application, use `wait --event window-opened` or a short `wait` before the first snapshot. -## Unnamed Web Content Resolves When Its Bounds Are Its Only Identity - -A web content element with **no accessible name** resolves through strict ref -resolution when its rectangle has positive area: the ref keeps its bounds as its -identity, and resolution accepts the single live candidate whose bounds hash and -role match. Named elements resolve normally. Two cases still need headed -coordinate input from a parent's bounds, or a CDP client (`launch --cdp`): - -- an unnamed element with a zero-area rectangle (offscreen or virtualized - content shares zero-extent bounds, which cannot be unique); -- an unnamed element whose geometry matches two or more live candidates, which - returns `AMBIGUOUS_TARGET` instead of guessing. +## Unnamed Web Content + +A ref to web content with **no accessible name** is identified by its role plus +any AutomationId or description it carries, and resolves only if that +combination was unique in the window when you took it and still is. Bounds are +never identity. A ref to one of several unnamed elements that share role, +AutomationId and description returns `AMBIGUOUS_TARGET`; use headed coordinate +input from a parent's bounds, or a CDP client (`launch --cdp`). diff --git a/skills/agent-desktop/SKILL.md b/skills/agent-desktop/SKILL.md index c24e35dfd..5e369dfe0 100644 --- a/skills/agent-desktop/SKILL.md +++ b/skills/agent-desktop/SKILL.md @@ -41,6 +41,8 @@ agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact # verify: re-read ``` - A small app needs no skeleton. Take a full `snapshot -i`. +- `-i` keeps interactive nodes only, so a row's static-text label is dropped. To tell which row a repeated button belongs to, drill that row's container with `--root` and leave out `-i`. On Windows and Linux that reads the row but does not make its button actionable: a button repeated in other rows is a twin in the window and its ref is refused (see Refs). +- A `--root` drill adds its refs to the same snapshot id. Refs outside the drilled subtree, and the drilled root itself, stay valid; refs that were already inside it are replaced, so an old ref there returns `STALE_REF`. - Never skeleton a surface (`--surface menu`, `sheet`, `alert`). A surface is already focused. - An action that opens a menu, sheet or alert lists it in `data.surfaces`. Target that overlay next. - After an async step, use `wait` (see `commands-system.md`). A fixed sleep is a race. @@ -50,8 +52,10 @@ agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact # verify: re-read - Output refs are qualified: `@:e`, for example `@s8f3k2p9:e1`. Pass them to later commands as they are. - Refs are valid for one snapshot. After any UI change, re-drill the affected region or take a new snapshot. - Only addressable elements get refs. Static text and inert containers do not. -- `STALE_REF`: the ref no longer matches the live UI. Take a fresh snapshot and use the new ref. -- `AMBIGUOUS_TARGET`: more than one live element matches. Narrow the target with `find --role --name --exact` or `--root`, then use the new ref. +- A command re-identifies the element before it acts. On macOS the ref carries the element's own address, so identical twins each keep their ref. On Windows and Linux a ref resolves only when it was the only element in the window its snapshot captured with its role, native id, name and description, counted even below a skeleton or depth cut, and exactly one live element still matches. Identical twins there have no ref path: only `--headed mouse-click --xy X,Y` can choose one, it takes the foreground, and a Linux Wayland session refuses it. +- `STALE_REF`: the element is gone or no longer matches. `details.kind` is `saved_ref_absent`, or `retained_object_stale` on macOS. Take a fresh snapshot and use the new ref. +- `AMBIGUOUS_TARGET`: an identical element appeared (`saved_ref_ambiguous`), or the ref was not proven unique at capture (`identity_not_unique`). Repeating the ref or re-snapshotting identical twins gives the same refusal; act on a control that is unique in the window. `identity_not_unique` also follows a snapshot that could not read the whole window in time: retake it with a larger `--timeout-ms`. +- `TIMEOUT` with `details.kind: resolution_identity_unknown`: a live read did not answer, so identity is unproven. Nothing was sent; retry. - `SNAPSHOT_NOT_FOUND`: the snapshot id expired. Snapshot again. ## Delivery and retry diff --git a/skills/agent-desktop/references/commands-interaction.md b/skills/agent-desktop/references/commands-interaction.md index 2f17a40e3..a5c3dc983 100644 --- a/skills/agent-desktop/references/commands-interaction.md +++ b/skills/agent-desktop/references/commands-interaction.md @@ -44,7 +44,7 @@ Exit codes: `0` success, `1` structured error, `2` argument error. The `error` o | `PERM_DENIED` | A required OS permission is missing | Read the platform skill | | `ELEMENT_NOT_FOUND` | Ref cannot be resolved in the live UI | Snapshot again | | `STALE_REF` | Ref no longer matches the live UI | Snapshot again, use the new ref | -| `AMBIGUOUS_TARGET` | Several live elements match the old identity | Narrow with `find`, then use the new ref | +| `AMBIGUOUS_TARGET` | Several live elements match the ref's identity, or it was not proven unique at capture | Act on a control unique in the window; identical twins need `--headed mouse-click --xy` (see `commands-observation.md`) | | `SNAPSHOT_NOT_FOUND` | Snapshot id missing or expired | Snapshot again | | `APP_NOT_FOUND` | App not running | `launch` it | | `WINDOW_NOT_FOUND` | No matching window | Check the name, use `list-windows` | diff --git a/skills/agent-desktop/references/commands-observation.md b/skills/agent-desktop/references/commands-observation.md index b5966f5f2..877654583 100644 --- a/skills/agent-desktop/references/commands-observation.md +++ b/skills/agent-desktop/references/commands-observation.md @@ -29,8 +29,9 @@ Each ref names the snapshot that owns it, so a ref keeps resolving against its o Choose the scope: - `-i --compact` keeps interactive elements only and drops empty wrappers. Use both by default. +- `-i` keeps interactive nodes and cut containers, so a row's static-text label is dropped. To tie a repeated button to its row, drill that row's container with `--root` and leave out `-i`. On Windows and Linux this identifies the row but a button repeated across rows stays a twin, so its ref is refused. - `--skeleton` clamps depth to 3 and adds `children_count` to cut containers. Each cut branch exposes its deepest safely resolvable drill target, or its nearest resolvable ancestor when the boundary has no stable identity. -- `--root ` starts from a ref found earlier. Re-drilling a root replaces only the refs from that root's previous drill. Other regions and the skeleton keep their refs. `--root` cannot combine with `--surface` or `--wait-for`. +- `--root ` starts from a ref found earlier. Re-drilling a root replaces the refs from that root's previous drill, and those of any drill made under it. Refs outside the drilled subtree keep working, skeleton refs inside it are replaced, and the new refs join the same snapshot id. A drill inherits the whole-window identity count of its snapshot, so its refs resolve like the skeleton's. `--root` cannot combine with `--surface` or `--wait-for`. - `--max-depth` (default 10) limits depth. Reach deeper subtrees with `--root`. - Use `--include-bounds` only when you need coordinates. - With `--app` and no `--window-id`, `snapshot` and `find` take the app's only visible window, else its only focused window, and answer `AMBIGUOUS_TARGET` with the candidates when neither settles it. Pass `--window-id` when an app has several windows. @@ -39,12 +40,24 @@ Choose the scope: `data.complete` is on every snapshot. When the observation budget runs out, the command still succeeds with `"complete": false`, `"truncated": true`, `"nodes_observed"`, and the tree it saw. Each node with cut descendants carries `"subtree_truncated": true`. Read `complete`; do not wait for a `TIMEOUT`. Raise `--timeout-ms` (default 3000) or drill with `--root` to finish the tree. A `--root` drill is all or nothing: an incomplete read returns `TIMEOUT`. +`"identity_unproven": true` means the snapshot could not read the whole window in time to prove its refs. On Windows and Linux, and on macOS for an element with no address, those refs fail with `identity_not_unique`, so retake the snapshot with a larger `--timeout-ms` before acting. + A cold Chromium or Electron app can take 10 to 25 seconds to expose its web tree. If a fresh snapshot comes back thin, raise `--timeout-ms` and snapshot again. ### Which refs exist An element gets a ref when its role is interactive or it advertises a primary action. `scrollarea` and `disclosure` get refs because `scroll`, `expand` and `collapse` need them. Static text and inert containers stay in the tree without refs. Web content advertises focus, right-click and scroll-to on almost every node, so those alone do not earn a ref. `find` returns `bounds` for such matches. Drive them with `--xy` (right-click, hover, drag) or scroll the container. +### Ref re-identification + +A command re-identifies the element from the ref before it acts. On macOS the ref carries the element's own address, so identical twins each keep their ref. On Windows and Linux a ref resolves only when it was the only element in the window its snapshot captured with its role, native id, name and description, and exactly one live element still matches. A skeleton, depth-limited or drilled capture counts against the whole window, so a shallow tree proves the same thing a full one does. Values and bounds never count as identity. + +- `STALE_REF`, `details.kind: saved_ref_absent`: no live element matches the stored identity. Take a fresh snapshot. On macOS `retained_object_stale` means the recorded address is dead or no longer matches. +- `AMBIGUOUS_TARGET`, `details.kind: saved_ref_ambiguous`: an identical element appeared, so two live elements match. `identity_not_unique`: the ref was not proven unique at capture, so nothing was searched. That is an identical twin, which every snapshot refuses again, or a window the snapshot could not read whole in time, which a new snapshot with a larger `--timeout-ms` fixes. +- `TIMEOUT`, `details.kind: resolution_identity_unknown`: a live read did not answer, so identity is unproven. Nothing was sent; retry. + +Off macOS, identical twins have no ref path: only `--headed mouse-click --xy X,Y` can choose one. It takes the foreground, and a Linux Wayland session refuses it. + ### Surfaces Surfaces are overlays: `menu`, `menubar`, `sheet`, `popover`, `alert`, `focused`, plus OS shell kinds on some platforms. Read `supported_surfaces` from `status` before you ask for one. An unsupported kind returns `PLATFORM_NOT_SUPPORTED` with the supported list in `details`. A shell surface needs no `--app`. When it is closed you get `WINDOW_NOT_FOUND`; raise it with `open-system-surface`. diff --git a/skills/agent-desktop/references/commands-system.md b/skills/agent-desktop/references/commands-system.md index 4e0b5ce72..d654f8826 100644 --- a/skills/agent-desktop/references/commands-system.md +++ b/skills/agent-desktop/references/commands-system.md @@ -163,6 +163,8 @@ Batch runs entries in order in one process. It is not a transaction. Entries use Wait entries use flat JSON keys matching the wait argument fields. For event waits, use `event` and optional `app`, `timeout` (milliseconds; defaults to 30000), `window`, and `window_id`. For example, the batch above waits up to five seconds for a new TextEdit window. `window` and `window_id` narrow only window lifecycle events (`window-opened`, `window-closed`, `focus-changed`); they do not scope text waits. In an action-then-event sequence, the batch captures the event baseline before the action, and does not start the action if that baseline read fails. +Keys are the command's own field names, so a positional argument is named after itself rather than after its placeholder: `press COMBO` is `{"combo":"cmd+s"}`, `click REF` is `{"ref_id":"@s8f3k2p9:e1"}`, and `type REF TEXT` takes `ref_id` and `text`. A flag drops its leading `--` and writes `-` as `_`, so `--timeout-ms` is `timeout_ms`. + ## Sessions and traces A session is an on-disk container under `/sessions//` with a manifest, snapshot refmaps and, when tracing is on, a `trace/` directory. The state root is `~/.agent-desktop`. `AGENT_DESKTOP_HOME` relocates it. It must be an absolute path, or the command fails with `INVALID_ARGS`. `status` reports it as `state_root`. diff --git a/skills/agent-desktop/references/workflows.md b/skills/agent-desktop/references/workflows.md index b476f2810..22157384a 100644 --- a/skills/agent-desktop/references/workflows.md +++ b/skills/agent-desktop/references/workflows.md @@ -29,7 +29,7 @@ agent-desktop click @s8f3k2p9:e18 agent-desktop snapshot --root @s8f3k2p9:e2 -i --compact ``` -- Re-drilling a root replaces only that root's refs. Refs from the skeleton and from other drills stay valid. +- Re-drilling a root replaces that root's refs and the refs of drills made under it. Skeleton refs elsewhere, and drills in other regions, stay valid. - Interactive elements inside skeleton depth already have normal refs. - If you know the role or exact name, skip the map and run `find` with `--role --name --exact`. diff --git a/src/tests/conformance.rs b/src/tests/conformance.rs index 5ceb0c6f3..2f8cd4ee1 100644 --- a/src/tests/conformance.rs +++ b/src/tests/conformance.rs @@ -49,7 +49,8 @@ impl ObservationOps for ContractAdapter { identity: LiveIdentity { name: LocatorField::Known("OK".into()), description: LocatorField::Absent, - identifiers: IdentifierEvidence::absent(), + identifiers: IdentifierEvidence::absent() + .with_retained_object("axtok1:contract-node".into()), }, state: ElementState { role: "button".into(), @@ -168,12 +169,13 @@ fn entry(bounds: Rect) -> RefEntry { process_instance: Some("contract-process".into()), }, identity: RefEntryIdentity { - retained_object: None, + retained_object: Some("axtok1:contract-node".into()), role: "button".into(), name: Some("OK".into()), value: None, description: None, native_id: None, + unique_at_capture: false, }, geometry: RefGeometry { bounds_space: None, @@ -190,6 +192,7 @@ fn entry(bounds: Rect) -> RefEntry { source_window_title: None, source_window_bounds_hash: None, source_surface: SnapshotSurface::Window, + renderer_accessibility_forced: false, }, scope: RefScope { root_ref: None, diff --git a/src/visual_debug/capture.rs b/src/visual_debug/capture.rs index d849196fb..d22bacdb6 100644 --- a/src/visual_debug/capture.rs +++ b/src/visual_debug/capture.rs @@ -45,7 +45,9 @@ pub(super) fn click_before( ) -> Result<(WindowInfo, Value), AppError> { let deadline = Deadline::standard()?; let entry = RefStore::for_session(context.session_id())?.load_ref(&args.ref_id)?; - let handle = adapter.resolve_element_strict(&entry, deadline)?; + let handle = agent_desktop_core::refs::resolve_saved_ref::resolve_saved_ref( + adapter, &entry, deadline, None, + )?; let bounds = adapter.get_element_bounds(&handle, deadline)?; let window = window_for_entry(&entry, adapter, deadline)?; let node = json!({ diff --git a/src/visual_debug/capture_tests.rs b/src/visual_debug/capture_tests.rs index d84ee1845..f702fbc53 100644 --- a/src/visual_debug/capture_tests.rs +++ b/src/visual_debug/capture_tests.rs @@ -154,6 +154,7 @@ fn saved_snapshot_and_click_capture_use_exact_source_evidence() { let window = window(); let entry: RefEntry = serde_json::from_value(json!({ "pid": 5, "process_instance": "test-instance", "role": "button", "name": "Save", + "retained_object": "axtok1:page-1:node-0", "path": [0], "source_surface": "window", "source_window_id": "w-test", "source_window_bounds_hash": window.bounds.unwrap().bounds_hash(), "states": [], "available_actions": ["Click"], "bounds": window.bounds, diff --git a/tests/e2e-windows/scenarios/Observation.ps1 b/tests/e2e-windows/scenarios/Observation.ps1 index bfa174644..7c436c857 100644 --- a/tests/e2e-windows/scenarios/Observation.ps1 +++ b/tests/e2e-windows/scenarios/Observation.ps1 @@ -7,7 +7,7 @@ annotates truncated containers, --root replaces only that subtree's refs, find resolves by --native-id/--name/--role, bounds are present with --include-bounds and absent in compact mode, and the twin controls - prove both arms of bounds-hash disambiguation. + prove an identical pair is refused rather than guessed at. #> Set-StrictMode -Version 2.0 @@ -67,7 +67,7 @@ function Invoke-ObservationScenario { Register-Legs -Names @( 'role-coverage-one-snapshot', 'scroll-container-refable', 'skeleton-depth-clamp', 'root-drilldown-scoped-refs', 'find-by-name', 'find-by-role', 'bounds-present-with-flag', - 'bounds-absent-compact', 'twins-ambiguous-after-move', 'twins-resolve-without-move' + 'bounds-absent-compact', 'twins-ambiguous-after-move', 'twins-refused-without-move' ) Enter-Stage -Lock DesktopLease -Body { @@ -140,15 +140,15 @@ function Invoke-ObservationScenario { inside the caller's block just because the caller happens to hold the lock at runtime. #> Enter-Stage -Lock DesktopLease -Body { - <# select_by_bounds_hash resolves whenever exactly one live - candidate still matches the stored hash - the only way to reach - SearchVerdict::Ambiguous is to move BOTH twins so the stored - hash matches neither, which move-twins does. A second leg - proves the tie-break also works in the non-moved direction. #> + <# A twin is never unique at capture, so acting through either ref is + refused as AMBIGUOUS_TARGET before the window is read + (resolve_saved_ref::identity_not_unique), whether or not the card + changed since the find that published the ref. move-twins rebuilds + the card; the second leg leaves it alone. #> $twins = Find-TargetMatches -App $App -NativeId 'twin-control' -TimeoutSeconds 10 if (-not $twins -or $twins.Count -ne 2) { Add-Fail -Leg 'twins-ambiguous-after-move' -Reason "expected exactly 2 twin-control hits, saw $(if ($twins) { $twins.Count } else { 0 })" - Add-Fail -Leg 'twins-resolve-without-move' -Reason 'skipped: twin discovery itself failed' + Add-Fail -Leg 'twins-refused-without-move' -Reason 'skipped: twin discovery itself failed' return } $staleTwin = $twins.Targets[0] @@ -160,7 +160,7 @@ function Invoke-ObservationScenario { exactly this refusal-tier assertion. #> $movedEnvelope = Invoke-Target -Target $staleTwin -Action 'click' try { - Assert-Envelope -Envelope $movedEnvelope -ErrorCode 'AMBIGUOUS_TARGET' -Delivery 'not_delivered' -Retry 'safe' + Assert-Envelope -Envelope $movedEnvelope -ErrorCode 'AMBIGUOUS_TARGET' -Delivery 'not_delivered' -Retry 'safe' -Details @{ kind = 'identity_not_unique'; retryable = $false } Add-Pass -Leg 'twins-ambiguous-after-move' } catch { Add-Fail -Leg 'twins-ambiguous-after-move' -Reason $_.Exception.Message @@ -170,13 +170,13 @@ function Invoke-ObservationScenario { if ($freshTwins -and $freshTwins.Count -eq 2) { $unmovedEnvelope = Invoke-Target -Target $freshTwins.Targets[0] -Action 'click' try { - Assert-Envelope -Envelope $unmovedEnvelope -Ok - Add-Pass -Leg 'twins-resolve-without-move' + Assert-Envelope -Envelope $unmovedEnvelope -ErrorCode 'AMBIGUOUS_TARGET' -Delivery 'not_delivered' -Retry 'safe' -Details @{ kind = 'identity_not_unique'; retryable = $false } + Add-Pass -Leg 'twins-refused-without-move' } catch { - Add-Fail -Leg 'twins-resolve-without-move' -Reason $_.Exception.Message + Add-Fail -Leg 'twins-refused-without-move' -Reason $_.Exception.Message } } else { - Add-Fail -Leg 'twins-resolve-without-move' -Reason 'twin-control did not re-resolve to 2 hits after the ambiguity leg' + Add-Fail -Leg 'twins-refused-without-move' -Reason 'twin-control did not re-resolve to 2 hits after the ambiguity leg' } } } diff --git a/tests/fixture-app-windows/FixtureCardsClicks.cs b/tests/fixture-app-windows/FixtureCardsClicks.cs index 166abef5f..a24cdfc13 100644 --- a/tests/fixture-app-windows/FixtureCardsClicks.cs +++ b/tests/fixture-app-windows/FixtureCardsClicks.cs @@ -79,20 +79,10 @@ private static void BuildTwinControls(GroupBox card, int y) card.Controls.Add(twinB); Button moveTwins = NewButton(card, "move-twins", "Move twins", 210, y, 110); - // Translating Location alone does not defeat a stored ref for - // either twin: agent-desktop's Windows resolver re-identifies a - // ref by its stored child-index tree path before it ever reaches - // the bounds-hash tie-break (can_use_path_fast_path, - // crates/windows/src/tree/resolve/resolve_search.rs), and a bare location - // change leaves both twins at their original Controls indices - - // the path lands on the exact original element, which is a - // correct re-identification, not an ambiguity. Removing and - // re-adding both twins shifts every later sibling's index - // (moveTwins included), so the stored path for the stale twin now - // lands on a control whose identity does not match and the - // resolver falls through to the broad search. Only then do two - // evidence-identical candidates reach the bounds-hash tier, and - // only then does moving both defeat that tie-break too. + // Rebuilding the card after the refs are published covers a UI + // change between capture and act. Both twins stay refused either + // way: a twin is never unique at capture, so no live read and no + // path or geometry tie-break is reached (resolve_saved_ref.rs). moveTwins.Click += delegate { int dx = 14;