Skip to content

feat(identity): resolve Linux and Windows refs through a capture rule - #442

Merged
lahfir merged 12 commits into
laf/identity-leanfrom
laf/identity-linux-windows
Nov 2, 2026
Merged

lahfir merged 12 commits into
laf/identity-leanfrom
laf/identity-linux-windows

Conversation

@lahfir

@lahfir lahfir commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator
  1. Linux and Windows expose no durable element address, so after fix: remove the macOS command host #414 a ref there can still act on the row that moves into a removed row's place.
  2. A Linux or Windows ref now resolves only if one complete window read proved it unique when taken and exactly one live element matches it now; otherwise it is refused.
  3. Stacked on feat(macos): re-identify refs through the element's own accessibility address #441. Includes fix(identity): validate popup context before rooted reads publish refs #445 and fix(identity): keep shallow captures and read the window once per action #447.
  • Applies the same proof wherever a ref is published: snapshot, skeleton, drill, find and hydration.
  • Checks a positional landing against the ref's full predicate before acting, in the CLI and in FFI find.
  • Linux menu refs keep their popup binding, checked on every rooted read.
  • Deletes the Windows strict resolver and the macOS path search, which lose their last callers.
#441 this PR
wrong row hit, Linux GTK3 1 0
identical twin refs acted on, Linux GTK3/GTK4/Qt6 — 0 / 12
unique-looking refs proven, 6 Linux apps — 167 / 167

Linux cost (ms, median of 7, arm64 container)

app get #441 get this PR is #441 is this PR skeleton #441 skeleton this PR
GTK3 18 50 17 36 35 36
GTK4 51 97 56 107 121 130
gnome-calculator 40 86 51 110 120 138
gedit 23 79 26 79 59 86
zenity 30 93 38 97 85 104

Every tokenless action re-reads its window to prove the ref is still unique. That read fetches identity only (no extents, actions or placement), and a skeleton reads the window once. Full snapshots are unchanged. macOS is unaffected: its refs carry element addresses (VM A/B against #441 within ±6%).

Linux and Windows expose no durable element address, so a ref there resolves only when one complete whole-window read proved it unique at capture and exactly one live element matches its full predicate now. Every place that publishes a ref proves it the same way, positional landings are confirmed against the predicate, FFI find re-identifies a landing before publishing it, and Linux menu refs keep their popup binding. The Windows strict resolver and the macOS path search lose their last callers and are deleted.
@vercel

vercel Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
agent-desktop Ignored Ignored Preview, v0 Nov 2, 2026 9:53pm UTC

Comment thread crates/core/src/refs/ref_alloc.rs Resolved
Rooted find and snapshot drills accepted a matching predicate from a replacement Linux popup. The shared saved-ref observation now checks the captured popup binding against the same observation before refs are published.
Snapshots keep a finished shallow read and flag refs they could not prove with identity_unproven. A ref action reads the window once and still runs the full rule under the lease. Capture and resolution read the same evidence, including the forced renderer flag. Hydration keeps the proven predicate, skeleton anchors fall back to the nearest proven ancestor, token-rooted drills read only their subtree unless it holds a tokenless ref, macOS tokenless anchors land by position, interval typing reuses its first handle, and exact FFI entries refuse with ACTION_NOT_SUPPORTED.
Comment thread crates/core/src/refs/ref_alloc.rs Resolved

@Railly Railly left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • p1. +1 ctate's path note
  • p2. "known open findings" looks mostly fixed by 443a316/ d29aa1d, maybe refresh the body
  • p3. a linux/windows timing table like #441's might help, every action reads the whole window and skeleton reads it twice
  • p4. one unreadable node with the same role refuses every ref of that role, a count on real apps might help

…d cost

The capture proof looked refs up by native child index while allocation writes positions in the projected tree, so after a walker skipped a sibling a unique ref was published unproven. The proof now walks the allocation path through the observation's own child lists, and projection keeps those lists.

A capture on an adapter without element tokens now reads the whole window once instead of a shallow walk followed by a whole-window read, and the Linux walker keeps up to 32 nodes in flight per level instead of 8, so a wide level costs one round of calls.
@lahfir
lahfir force-pushed the laf/identity-linux-windows branch from f2d84dd to 1f875cc Compare November 2, 2026 08:00
Resolving a tokenless ref re-reads its whole window to prove it is still unique. That read now asks for identity evidence only, and the Linux walker skips extents, actions and the GTK4 placement pass for it, which the count never uses. Capture, drill and rooted find reads keep their full evidence. Measured on Linux against the previous head, get and is drop by 9 to 40 ms per call with every run still resolving.
lahfir added a commit that referenced this pull request Nov 2, 2026
Stack the windowless-app work on the identity rework and keep #442 where
the two overlap: resolution by element token or capture proof replaces the
bounds, fast-path and retained-object pieces this branch had added, and
drill-down goes through recorded_window. On top of that, background apps
with no window still resolve to their process owner, the process-rooted
anchor gate stays, and belongs_to_entry now requires the owner kind to
match so a process-owned ref never lands on a window that appears later.
Comment thread skills/agent-desktop/SKILL.md Resolved
…read runs long

On adapters without element tokens a capture reads the whole window first. That read could spend the whole deadline, so the shallow fallback started expired and returned TIMEOUT instead of the shallow snapshot. The whole-window attempt now gets three quarters of the remaining time. The skills also no longer claim that an unnamed control is identified by role alone or that a re-drill keeps refs inside the drilled subtree.
@lahfir
lahfir requested review from Railly and ctate November 2, 2026 18:40
find --root on a root carrying an element token observed the whole window
to re-identify it, though a token re-identifies itself. Drill-down already
reads such a root's subtree directly. Both now go through one helper,
read_saved_root: a token root is read as its subtree and stands when the
subtree publishes no tokenless ref; otherwise, and for every tokenless
root, the whole-window capture proof still applies.
…saved ref

Resolving a tokenless ref re-reads its whole window to prove it is still unique. For a Chromium or Electron window that read ran the settle loop: three equal node counts 200 ms apart, so every action paid about 400 ms of sleep. On a page that kept changing the loop gave up, the observation was marked incomplete, and the ref was refused as unproven even though the element was identifiable.

Settling belongs to capture, where refs are published and proven unique. A read that only resolves a saved ref now walks once. Ambiguity, an absent element and a walk cut short by its budget are still refused.
…ooted find

The subtree fast path counted only nodes already known to be ref-able, so a
tokenless member whose actions were still unread passed the check, and
hydration could then publish it without a window-wide capture proof; acting
on it later failed with identity_not_unique. An unsettled node now counts as
addressable, as an unread token already counts as absent, so such a subtree
falls back to the window proof.
cargo deny fails the audit on a yanked crate. 1.14.1 is the latest
compatible release; nothing else in the lock file moves.
…t exhaust

The 100 ms budget only bounded how fast an unrelated /usr/bin/true exits, so
a busy machine skipped the final group probe and failed the test. The reap
poll returns as soon as the child exits, so a 5 s ceiling keeps the test as
fast while removing the flake.
@lahfir
lahfir merged commit 0dd484f into main Nov 2, 2026
38 checks passed
lahfir added a commit that referenced this pull request Nov 2, 2026
lucamaraschi added a commit to lucamaraschi/agent-desktop that referenced this pull request Nov 3, 2026
Main removed the macOS command host (vercel-labs#414) and re-identifies tokenless
refs through a capture rule (vercel-labs#441, vercel-labs#442).

- Every command is now its own process on macOS too, so the policy layer
  drops the host forwarding of AGENT_DESKTOP_POLICY_DIR, reads it
  directly, and the one-shot checkpoint test runs on macOS as well.
- A ref action now finds the ref's app and window before it reads the
  element, so the policy scenario desk lists each seeded ref's app and
  window and counts the element read at observe_tree.
- The live window_at_point grid test accepts the owner's name over
  another user's window when the hit test can read that owner's identity
  (root's loginwindow), as well as the refusal.
justcarlson added a commit to justcarlson/agent-desktop that referenced this pull request Nov 3, 2026
…as exited

Red: on Linux, once the owner exits, list_windows with the app filter answers APP_NOT_FOUND, and recorded_window passes that through. The e2e contract (tests/e2e-linux/README.md, launch.recovery_strategy_on_error and observation.stale_ref_after_app_killed) requires STALE_REF with process_state=exited for a dead owner's ref.

candidate vercel-labs#442 fix
justcarlson added a commit to justcarlson/agent-desktop that referenced this pull request Nov 3, 2026
recorded_window looks the window up by application. Once the owner exits, the Linux lookup answers APP_NOT_FOUND, and the ref was refused as an unknown app. Map that answer to STALE_REF, so enrichment adds process_state=exited and the exited-process suggestion, as the e2e contract requires.

candidate vercel-labs#442 fix
justcarlson added a commit to justcarlson/agent-desktop that referenced this pull request Nov 3, 2026
…n the ref contract

A ref that was not unique at capture is refused with AMBIGUOUS_TARGET identity_not_unique and not_delivered. Nothing reaches the app, so the contract row expecting a refusal holds. Other AMBIGUOUS_TARGET kinds still do not count.

candidate vercel-labs#442 fix
justcarlson added a commit to justcarlson/agent-desktop that referenced this pull request Nov 3, 2026
…able

The Linux leg launch.recovery_strategy_on_error requires details.retryable=false
and no recovery object when the ref's owner has exited, the shape the adapters
give with kind resolve_owning_process_exited. The saved-ref resolver's
APP_NOT_FOUND mapping gave a bare stale ref, so the envelope gained a
retryable recovery hint.

candidate vercel-labs#442 fix
justcarlson added a commit to justcarlson/agent-desktop that referenced this pull request Nov 3, 2026
The saved-ref resolver turns the window lookup's APP_NOT_FOUND into STALE_REF.
Without kind resolve_owning_process_exited and retryable=false the envelope
offered a retryable recovery hint for an owner that is gone, which the Linux
contract leg launch.recovery_strategy_on_error refuses.

candidate vercel-labs#442 fix
lucamaraschi added a commit to lucamaraschi/agent-desktop that referenced this pull request Nov 4, 2026
Main removed the macOS command host (vercel-labs#414) and re-identifies tokenless
refs through a capture rule (vercel-labs#441, vercel-labs#442).

- Every command is now its own process on macOS too, so the policy layer
  drops the host forwarding of AGENT_DESKTOP_POLICY_DIR, reads it
  directly, and the one-shot checkpoint test runs on macOS as well.
- A ref action now finds the ref's app and window before it reads the
  element, so the policy scenario desk lists each seeded ref's app and
  window and counts the element read at observe_tree.
- The live window_at_point grid test accepts the owner's name over
  another user's window when the hit test can read that owner's identity
  (root's loginwindow), as well as the refusal.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants