Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ body:
render: shell
placeholder: |
$ agent-desktop snapshot --app Finder
{"version":"2.5","ok":false,"command":"snapshot","error":{"code":"APP_NOT_FOUND",...}}
{"version":"2.6","ok":false,"command":"snapshot","error":{"code":"APP_NOT_FOUND",...}}
validations:
required: true

Expand Down
12 changes: 7 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,12 +190,12 @@ remain the transitive check.
Every command prints one envelope. `docs/json-output.md` holds the full contract.

```json
{ "version": "2.5", "ok": true, "command": "snapshot",
{ "version": "2.6", "ok": true, "command": "snapshot",
"data": { "app": "Finder", "snapshot_id": "s8f3k2p9", "ref_count": 14, "complete": true, "tree": { } } }
```

```json
{ "version": "2.5", "ok": false, "command": "click",
{ "version": "2.6", "ok": false, "command": "click",
"error": { "code": "STALE_REF", "message": "...", "suggestion": "...",
"recovery": { "strategy": "refresh_snapshot_then_retry_original", "retryable": true,
"requires_fresh_snapshot": true },
Expand All @@ -221,9 +221,11 @@ Every command prints one envelope. `docs/json-output.md` holds the full contract
- An element gets a ref when its role is interactive (`INTERACTIVE_ROLES` in `crates/core/src/tree/roles.rs`) or it
advertises a primary action. `SetFocus`, `RightClick` and `ScrollTo` alone do not qualify, because web runtimes
advertise them on nearly every node.
- Actions re-identify the element strictly from `RefEntry` evidence: pid, role, path and source surface, stable text
identity and a bounds hash. Control values are volatile and never identity. A mismatch returns `STALE_REF`; several
live candidates return `AMBIGUOUS_TARGET`.
- Actions re-identify the element one of two ways. A genuine element token resolves through the adapter, which owns
re-identification on that platform (macOS). Every other ref resolves from its capture proof: role plus every
recorded stable field, that is native id, stable name and description, counted over one complete observation of the
window it was captured from. A ref that was not unique at capture is refused before the application is read. Zero
live matches is `STALE_REF`, two or more is `AMBIGUOUS_TARGET`, and control values and bounds are never identity.
- All state lives under one root, `~/.agent-desktop` by default. `AGENT_DESKTOP_HOME` relocates it
(`crates/core/src/storage/state_root.rs`). Refmaps live at `snapshots/<snapshot_id>/refmap.json`, capped at 1 MB each; the
store keeps the newest 128 snapshots per namespace and prunes to 96. Command code goes through `RefStore`, which
Expand Down
9 changes: 7 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,16 @@

## Unreleased

### ⚠ BREAKING CHANGES

* The C ABI major version is now 7. `ad_resolve_element_exact` and `ad_execute_ref_action_exact_with_policy` always refuse with `ACTION_NOT_SUPPORTED` (`details.kind: exact_entry_unsupported`): a C caller cannot supply the capture proof a ref now resolves by, so act through a snapshot ref instead.
* The response envelope moves to 2.6. On Windows and Linux, a ref to one of several identical controls, a ref from a window the snapshot could not read whole, and a ref from an earlier snapshot now return `AMBIGUOUS_TARGET` with `details.kind: identity_not_unique` where some used to act. A snapshot that could not read its window whole to prove its refs now says so with `data.identity_unproven: true`. See the version 2.6 migration in `docs/json-output.md`.

### Bug Fixes

* Remove the macOS command host. Every CLI call runs in its own process, as on Windows and Linux. The npm install and every CLI call stop any command host an earlier version left running. After an install that skips npm scripts, or a manual install, run `agent-desktop version` once.
* Refs from snapshots taken before this release return `STALE_REF`. Take a new snapshot.
* On macOS a ref follows its own element. A ref to a row that was removed returns `STALE_REF` instead of acting on the row that moved into its place, and identical controls can each be used by ref.
* Refs from snapshots taken before this release carry no capture proof, so take a new snapshot. On Windows and Linux they return `AMBIGUOUS_TARGET` with `details.kind: identity_not_unique`; on macOS a ref still resolves while the element its token names is alive.
* A ref does not act on a different element after the UI changes. On macOS a ref follows its own element: a ref to a removed row returns `STALE_REF`, and identical controls can each be used by ref. On Windows and Linux a ref resolves only if it was unique when captured and exactly one element matches it now; a ref to one of several identical controls returns `AMBIGUOUS_TARGET`. Identity is the element's role and stable fields, not its instance: an identical element that replaced the original is acted on, and a toolkit that keeps a removed widget listed (GTK4, Qt6 on Linux) can still receive the action.

## [1.0.2](https://github.com/vercel-labs/agent-desktop/compare/v1.0.1...v1.0.2) (2026-10-30)

Expand Down
8 changes: 5 additions & 3 deletions CONCEPTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,10 +110,12 @@ The fail-closed process of re-identifying a ref from stored identity evidence be

Strict ref resolution rejects missing, stale, and ambiguous matches instead of guessing. It is the boundary between an old observation and a live desktop mutation.

### Graded Resolution
The order a resolver follows to re-identify a ref, on the evidence the ref carries: it walks the stored positional path first, then falls back to a bounded broad search. A candidate is confirmed by identifier first, then by role-conditional stable text; only a ref with no meaningful identity can resolve by unique geometry, as a last resort. Each tier is a locator, never an identity by itself: a path or a bounds match is always verified against the stored identity evidence before it resolves.
### Ref Re-identification
The rule a stored ref resolves by, which either lands it on the element it was captured from or refuses.

The geometry promotion is deliberately narrow: it fires only when the ref has no meaningful identity and stored bounds, and it resolves only on a unique live match. On Windows and Linux the stored bounds must also be a positive-area rectangle: a zero-extent stored hash never promotes, because offscreen and virtualized elements collapse to shared degenerate rectangles that are structurally non-unique (A17-7). macOS has no such check.
On macOS the entry carries the element's own accessibility address in `retained_object`, so a later command turns the ref back into that element without searching. The address is accepted only while the element is alive and its process, role and recorded identifier still match, so a removed element or an app that restarted is a stale ref rather than a look-alike, and identical twins each keep their own ref. Where an app yields no address, macOS follows the rule below.

Every other ref resolves from a capture proof. A ref is usable only when the capture saw its whole window completely and exactly one observed element matched its full predicate: role plus every recorded stable field, that is native id, stable name and description. Resolution re-counts that predicate over the live window and accepts exactly one match. Evidence and presentation are separate: a skeleton, a depth-limited snapshot and a `--root` drill are shallow presentations of one complete window observation, so their refs are counted against that whole window and each presented cut container keeps the count of what it hides. A window that cannot be read completely proves nothing, and its refs are refused rather than counted against a narrower tree. Zero matches is a stale ref, two or more is ambiguous, and a field the live read could not answer stays unknown rather than becoming a match, so incomplete evidence never resolves. Values and bounds are never identity. A ref that was not unique at capture is refused before anything is read from the application, because no search can pick one of two identical elements.

## Read Outcomes

Expand Down
4 changes: 2 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ agent-desktop snapshot --app Finder -i # observe again to verify

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "snapshot",
"data": {
Expand Down Expand Up @@ -149,7 +149,7 @@ import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_adapter_create.restype = ctypes.c_void_p
lib.ad_adapter_destroy.argtypes = [ctypes.c_void_p]
assert lib.ad_init(6) == 0 # AD_RESULT_OK when the header's AD_ABI_VERSION_MAJOR matches the library
assert lib.ad_init(7) == 0 # AD_RESULT_OK when the header's AD_ABI_VERSION_MAJOR matches the library
adapter = lib.ad_adapter_create()
lib.ad_adapter_destroy(adapter)
```
Expand Down
16 changes: 8 additions & 8 deletions apps/docs/content/docs/commands/app-window.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ agent-desktop launch gtk3-demo
```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "launch",
"data": {
Expand Down Expand Up @@ -108,7 +108,7 @@ agent-desktop close-app <APP> [--force]
```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "close-app",
"data": {
Expand Down Expand Up @@ -158,7 +158,7 @@ agent-desktop list-apps [--app NAME] [--timeout-ms MS]
```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "list-apps",
"data": {
Expand Down Expand Up @@ -199,7 +199,7 @@ agent-desktop list-windows [--app NAME] [--timeout-ms MS]
```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "list-windows",
"data": [
Expand Down Expand Up @@ -244,7 +244,7 @@ Pass at least one flag; with none the command fails with `INVALID_ARGS`. `data.f
```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "focus-window",
"data": {
Expand Down Expand Up @@ -287,7 +287,7 @@ agent-desktop resize-window --app gedit --width 800 --height 600
```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "resize-window",
"data": {
Expand Down Expand Up @@ -360,7 +360,7 @@ agent-desktop minimize [--app NAME] [--window-id ID]
```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "minimize",
"data": {
Expand Down Expand Up @@ -400,7 +400,7 @@ agent-desktop restore [--app NAME] [--window-id ID]
```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "restore",
"data": {
Expand Down
6 changes: 3 additions & 3 deletions apps/docs/content/docs/commands/batch.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Entries run one at a time through the same dispatch as the CLI, so each entry's

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "batch",
"data": {
Expand All @@ -81,8 +81,8 @@ Entries run one at a time through the same dispatch as the CLI, so each entry's
"limits": { "max_entries": 64, "max_input_bytes": 1048576, "max_output_bytes": 4194304 },
"not_started_entries": 0,
"results": [
{ "command": "version", "data": { "os": "linux", "target": "x86_64", "version": "1.0.1" }, "execution": "completed", "index": 0, "ok": true, "version": "2.5" },
{ "command": "wait", "data": { "waited_ms": 100 }, "execution": "completed", "index": 1, "ok": true, "version": "2.5" }
{ "command": "version", "data": { "os": "linux", "target": "x86_64", "version": "1.0.1" }, "execution": "completed", "index": 0, "ok": true, "version": "2.6" },
{ "command": "wait", "data": { "waited_ms": 100 }, "execution": "completed", "index": 1, "ok": true, "version": "2.6" }
],
"semantics": {
"atomic": false,
Expand Down
8 changes: 4 additions & 4 deletions apps/docs/content/docs/commands/clipboard.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,13 @@ agent-desktop clipboard-get [--format text|auto|image|file-urls] [--out PATH]
</table>

```json
{ "version": "2.5", "ok": true, "command": "clipboard-get", "data": { "text": "hello from agent-desktop", "type": "text" } }
{ "version": "2.6", "ok": true, "command": "clipboard-get", "data": { "text": "hello from agent-desktop", "type": "text" } }
```

An image result is `{ "type": "image", "path", "width", "height" }`, and a file list is `{ "type": "file_urls", "file_urls": [...] }`. Image bytes are never inlined. When the clipboard holds nothing in the requested form, the command still succeeds:

```json
{ "version": "2.5", "ok": true, "command": "clipboard-get", "data": { "found": false, "type": "text" } }
{ "version": "2.6", "ok": true, "command": "clipboard-get", "data": { "found": false, "type": "text" } }
```

Check `found` before you read `text`. Errors: `INVALID_ARGS` for an unknown format or an unwritable `--out`, and `PLATFORM_NOT_SUPPORTED` when a platform helper is missing.
Expand Down Expand Up @@ -61,7 +61,7 @@ One call writes one representation: `--file-url` wins over `--image`, which wins

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "clipboard-set",
"data": {
Expand All @@ -86,7 +86,7 @@ agent-desktop clipboard-clear

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "clipboard-clear",
"data": {
Expand Down
2 changes: 1 addition & 1 deletion apps/docs/content/docs/commands/interaction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Every successful ref action returns an action report in `data`. This is a `click

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "click",
"data": {
Expand Down
4 changes: 2 additions & 2 deletions apps/docs/content/docs/commands/keyboard.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Keys are physical keyboard input, and `press` does not need `--headed` for an un

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "press",
"data": {
Expand Down Expand Up @@ -68,7 +68,7 @@ A held key needs a process that can guarantee its release, and every agent-deskt

```json
{
"version": "2.5",
"version": "2.6",
"ok": false,
"command": "key-down",
"error": {
Expand Down
6 changes: 3 additions & 3 deletions apps/docs/content/docs/commands/mouse.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ agent-desktop --headed hover --xy 640,400

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "hover",
"data": {
Expand Down Expand Up @@ -114,7 +114,7 @@ agent-desktop --headed mouse-click --xy 500,300 [--button left|right|middle] [--

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "mouse-click",
"data": {
Expand Down Expand Up @@ -172,7 +172,7 @@ agent-desktop --headed mouse-wheel --x 700 --y 400 --dy -2

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "mouse-wheel",
"data": {
Expand Down
2 changes: 1 addition & 1 deletion apps/docs/content/docs/commands/notifications.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ agent-desktop list-notifications [--app NAME] [--text TEXT] [--limit N]

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "list-notifications",
"data": {
Expand Down
12 changes: 6 additions & 6 deletions apps/docs/content/docs/commands/observation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ agent-desktop find --app gnome-calculator --role button --name 7 --exact --first

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "find",
"data": {
Expand Down Expand Up @@ -125,7 +125,7 @@ agent-desktop get <REF> [--property text|value|title|bounds|role|states]

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "get",
"data": { "property": "value", "ref": "@s30sxlstqzdq8x:e19", "value": "entry" }
Expand Down Expand Up @@ -160,7 +160,7 @@ agent-desktop is @s30sxlstqzdq8x:e45 --property enabled

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "is",
"data": { "applicable": true, "property": "enabled", "ref": "@s30sxlstqzdq8x:e45", "result": false }
Expand Down Expand Up @@ -195,7 +195,7 @@ agent-desktop screenshot --screen 0 screen.png

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "screenshot",
"data": { "format": "png", "height": 900, "path": "/tmp/screen.png", "scale_factor": 1.0, "width": 1440 }
Expand All @@ -216,7 +216,7 @@ agent-desktop list-displays

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "list-displays",
"data": [
Expand Down Expand Up @@ -267,7 +267,7 @@ agent-desktop --headed open-system-surface --surface KIND

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "open-system-surface",
"data": {
Expand Down
2 changes: 1 addition & 1 deletion apps/docs/content/docs/commands/scroll.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ macOS and Windows scroll through native accessibility paths headless. On Linux,

```json
{
"version": "2.5",
"version": "2.6",
"ok": false,
"command": "scroll",
"error": {
Expand Down
8 changes: 4 additions & 4 deletions apps/docs/content/docs/commands/system.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ agent-desktop permissions [--request] [--guide]

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "permissions",
"data": {
Expand All @@ -69,7 +69,7 @@ Each state is `granted`, `denied` (with a `suggestion`), `not_required`, or `unk
Prints the version, CPU architecture, and OS. It takes no flags.

```json
{ "version": "2.5", "ok": true, "command": "version", "data": { "os": "linux", "target": "x86_64", "version": "1.0.1" } }
{ "version": "2.6", "ok": true, "command": "version", "data": { "os": "linux", "target": "x86_64", "version": "1.0.1" } }
```

The envelope's `version` is the JSON contract version; `data.version` is the release.
Expand Down Expand Up @@ -144,7 +144,7 @@ agent-desktop --session "$AGENT_DESKTOP_SESSION" cursor-overlay disable

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "cursor-overlay",
"data": {
Expand Down Expand Up @@ -188,7 +188,7 @@ agent-desktop --session "$AGENT_DESKTOP_SESSION" trace export [--out PATH] [--li

```json
{
"version": "2.5",
"version": "2.6",
"ok": true,
"command": "trace",
"data": {
Expand Down
Loading
Loading