Skip to content

Clarify developer workflow progress and delivery messages - #489

Draft
Shahzaib (shahzaibj) wants to merge 9 commits into
masterfrom
shjameel-microsoft-workflow-messaging
Draft

Shahzaib (shahzaibj) wants to merge 9 commits into
masterfrom
shjameel-microsoft-workflow-messaging

Conversation

@shahzaibj

@shahzaibj Shahzaib (shahzaibj) commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Follow-up to merged #488, from a new branch based on master. Developer-workflow tooling only; no Android authentication behavior changes.

  • Replace ambiguous progress wording with recorded facts and next steps. Emit builder/repair start receipts at actual dispatch, not cached replay.
  • Separate local verification, independent source review, publication, exact-head CI, external blockers and human approvals without repeating generic disclaimers.
  • Retry transient Windows atomic-outbox replacement failures asynchronously and serialize commits across runs. Keep durable-before-send/acknowledgement ordering, cancellation and at-least-once delivery semantics.
  • Coalesce notification failures by operation and underlying cause, including fresh Error objects and wrappers. Report a changed cause and corresponding recovery, not every retry. Keep full details in ignored, rotating local diagnostics; unrelated successes do not clear another operation's incident. Coalescing is process-local, not an exactly-once UI guarantee.
  • Reuse an accepted waiting classification while CI/review evidence is unchanged, including at the wait bound. In-place feedback edits, policies, diagnostics, conflicts and source/merge/build bindings invalidate it; collection metadata does not. Stable durable milestone fingerprints, repair/agent ceilings and wait timing remain unchanged.
  • Preserve failed/deferred local reasons across repeated CI description updates without changing the compact marker schema. Cap the diagnostic summary at 500 characters while keeping full verification evidence. Report zero source-repair allowance as disabled repairs, not an exhausted budget.
  • Recheck disposal after an awaited terminal-outbox commit, before starting its send. A notice committed during disposal stays pending for a later instance instead of being sent and acknowledged after shutdown.
  • Preserve the recorded stopping/failure/defer sentence before trimming long command narration in local notices and terminal evidence, including the actual 12-minute-cap case. Keep the 500-character bound and full receipts unchanged.
  • Report tactical work as an assessment, not automatically authored source. Distinguish a confirmed no-op push from a new source head; an unavailable resulting head remains explicit. Repair classification, steps, raw receipts, identities and budgets stay unchanged.
  • Expose permission-gated code_feature_workflow_pause through the joined owner session so a requested stop can be enforced without private transport or process termination. Guard run identity/name/state and confirm settlement/live-agent evidence; never resume, replace or change budgets. User-requested pauses are not prerequisite questions.

Message examples

Before After
Implementation and execution verification are still pending. Evidence complete; implementation is next.
No distinct builder-start receipt. Implementation in progress. (only at actual dispatch)
Local verification - deferred, followed by generic disclaimers. Local verification is incomplete; remaining checks need CI, plus the specific recorded reason.
Current PR validation is pending. CI checks are queued. / CI checks are running. / Waiting for CI results for the current PR head.
A full-path delivery error on every retry. Some workflow updates could not reach this chat. Retrying automatically. Details are in the local workflow diagnostics.
No distinct recovery notice. Workflow updates can reach this chat again.
Long command narration cut before the stopping reason. Builder reported: Stopped verification at the 12-minute cap; remaining checks need CI.
Repair pushed, despite an unchanged source head. Push completed; the source head is unchanged. CI observation continues.
Repair budget exhausted after 0 repair(s); remaining failures... Source repairs are disabled for this run. A developer must review the remaining findings.

Real build/test/CI failures remain explicit; publication and source review do not become verification or merge approval.

Verification

  • node --test --test-reporter=spec .github\extensions\CodeFeature\*.test.mjs .github\extensions\DevDesignDoc\*.test.mjs: 269 passed, 0 failed, 0 skipped on Windows.
  • Pause-control prerequisite focused completion and registered-extension suites: 84 passed on d4a573958; the current combined suite includes the additional terminal-cause regression.
  • Regressions include one observer classification across 60 unchanged CI waits, edited feedback with unchanged IDs/counts, changed policy/head/build evidence, new failed tasks and CI completion. Delivery regressions cover fresh/wrapped failures, shared storage incidents, changed causes, operation-specific recovery, private diagnostics/fallback/rotation, and genuine failures after silent admission-window delays.
  • Native Windows sharing-lock tests cover non-blocking transient recovery, bounded persistent failure, preserved previous JSON and no false acknowledgement. Existing dispatch/replay/cancellation, registered-extension and supervision tests remain passing.
  • Review-fix regressions cover repeated CI updates for failed/deferred local reasons, the 500-character summary bound, admitted/pending-outbox prompts below 4,000 characters for verbose evidence, full-receipt preservation, disabled-repair wording, and correct I/O incident scope when late catch-up workers join an in-flight send. Both inline review findings have fix/test replies and are resolved.
  • The later disposal finding is reproduced by both a native Windows deny-delete sharing lock during replacement and a deferred asynchronous store. Both tests fail before the guard and pass afterward, verifying no post-disposal send/acknowledgement, a retained pending marker, and later-instance replay. This third finding also has a fix/test reply and is resolved.
  • Pause-control regressions cover owner/name/state/input guards, normal permission gating, unchanged accounting, cooperative cancellation, already-settled no-op reads, unsupported APIs, provider failures, unconfirmed acknowledgements/live agents and user-pause messaging. The production extension registration includes the control.
  • JavaScript syntax checks and git diff --check passed.
  • Exact-head CodeQL analyzers (actions and JavaScript/TypeScript), aggregate CodeQL and CLA passed for review-fix head 79082e4770ff099c82fe3e692f8a5f5fd1e06f8b. AB-ID skipped under the existing exemption.

Base: fdf05b51820b7a5fe369845e5c228419eb42ae44

Unit-tested/current PR head: 79082e4770ff099c82fe3e692f8a5f5fd1e06f8b

Live validation boundary

A first cold-start read-only observer check on f238ab8679029b8aae10f59e3d4b466efea1026a (run 16091cf0-a586-4298-a46f-d127cfe52ce1) completed with one classifier, no repairs, exact-head CI passing and no observed completion-delivery failures. With repairs disabled, it handed off the two unresolved review findings rather than claiming readiness. Both findings were verified against the source, fixed and tested in 7f1efbf01f3a69fe2905bb96fdb3386dcb28ed68.

A second fresh session validated the then-current head 7f1efbf01f3a69fe2905bb96fdb3386dcb28ed68 through the registered pr-observer, again with repairs, pushes and status publication disabled and a finite two-poll CI wait allowance.

Most recent read-only observer run f2227c88-3fb6-495b-ae09-d42cd7851ec6, attempt 1, settled completed with result awaiting-human-review: "Current-head CI passed; repository review/merge policies require human action." Local source, pre/post PR head and observed CI head all matched 7f1efbf01f3a69fe2905bb96fdb3386dcb28ed68. It spawned 0 agents, made 0 repairs, and required no continuation.

The owning transcript/outbox recorded two delivered milestones and one delivered terminal notice, with no pending notice, delivery error or diagnostic file. Actual chat wording included: "The observation of PR #489 finished with CI passing for the current head. A human reviewer must complete the required review, approvals, and merge decision; the PR remains draft." A milestone was admitted while active, but its assistant reply occurred after this short run settled; no pre-settlement visible-reply timing is claimed.

Separate host/MCP discovery/startup warnings did not block the observation and were not completion-delivery failures. No authentication cause was established for those warnings.

The post-live disposal guard at e2d0f1e5f90adc5d5f4ddf85ef55b380cd216bd8 has passing native/async regressions and exact-head CI. Its last production-observation receipt remains bound to 7f1efbf01f3a69fe2905bb96fdb3386dcb28ed68.

The later pause-control change is required for the user's new stop-on-error E2E policy. It was verified against an actual registered code-feature attempt before starting a separate, genuinely fresh full PBI implementation session. This control test is not completed implementation E2E coverage.

Actual control probe f49d23e9-cd30-47ca-b924-f7766684a0c0 returned paused, pauseInfo.type=user, pauseRequested=true, liveAgentCount=0 and canResume=true. A subsequent independent read confirmed settlement, no live phase/agents and retained journal. Active accounting accrued normally from 36,502 to 51,222 ms before settlement, then froze; it was not reset. No evidence agent, builder, implementation or publication was admitted.

The probe passed createPullRequests=false and observePullRequests=false, but omitted allowLocalOnly=true; production policy can override those flags when the session has a UI. Therefore no guaranteed local-only policy is claimed. The verified pause occurred before implementation/publication, and no arguments were changed, probe resumed or second probe launched. The unused authored control fixture was not executed after inspection selected the production registration.

The clean full PBI attempt f7ba14b6-5de3-4f31-ab5a-c19a1a2a12bd ran in a separate fresh caller on d4a573958a8187e4210c0c11993cbb73fa0ec11a, with the exact prompt "Use the coding workflow to implement PBI 3804406" and no options or limit overrides. Evidence completed and the actual builder dispatch emitted "Implementation in progress". The attached wait then failed with an external-request timeout; the owner paused this same run under the stop-on-error policy. It is an interrupted/failed E2E attempt, not completed implementation coverage: two agents spawned, zero live direct agents, builder cancelled, and no confirmed publication/CI receipt. No resume, requeue or allowance change occurred. Partial target files and earlier runs/PRs are preserved.

Read-only Windows System events establish host sleep at 2026-11-07T19:10:13Z, wake around 22:51:04Z, and resume-time clock correction. The external timeout/late rejected response and an independent coordinator held-call return were recorded around wake. This explains the unexplained wall-clock gap; logs do not identify a particular blocked internal SDK read or establish the unbounded-read candidates as the incident cause. No speculative wait-timing patch or replacement was made. A subsequent public read confirmed the exact worker verification shell completed with exit code 1 and is not running; no process stop was needed. Its output remains preserved, and no local verification pass is claimed.

Exactly one fresh replacement, run da3f9598-f473-4e06-b195-20d5c42f55db, was admitted at 2026-11-07T23:46:36Z in a new owning process pinned to d4a573958a8187e4210c0c11993cbb73fa0ec11a. It used only "Use the coding workflow to implement PBI 3804406", with no options, model or limit overrides. No OS/power changes or awake-host promise were inferred. Evidence, actual builder dispatch, source review, publication, normal CI handling and two tactical repair rounds ran under the same-ID attached wait.

It settled completed / needs-owner-decision, not merge-ready or green-CI verified, and published the separate Authenticator PR: https://msazure.visualstudio.com/One/_git/AD-MFA-phonefactor-phoneApp-android/pullrequest/17856207 . Final observed source: f2194b226ad8b1c7b19440a363367ee4088aaaa1. Public owner status confirmed 9 direct agents spawned, 0 live, no live phase and canResume=false; raw consumption is {activeMs:2982429,subagents:9,nanoAiu:142999080000} (not converted to credits).

The original combined local Gradle attempt stopped at the 12-minute cap, with zero retries and no successful combined exit. The 41 ECS test passes are builder/XML evidence on the original 03e2905da35f56d80c26b74028306db654ad53e8 source, not a full local pass for the repaired head; the final local receipt is unverified because it predates that head. No connected/device tests ran. Current-head Authenticator PR Checks build 190023203 remained in progress at settlement; stale build validation and Build 0 metadata unavailable: (404) Not Found are explicit caveats.

Round 1 worker/reviewer accepted a no-source-change disposition after CI checkout failed before compilation/tests. Subsequent read-only inspection of the actual target commits corrects the earlier no-op characterization: deterministic commit 1e20ecb38c14d900cfa81eb89f52685589618437 added root local.properties. The push was not a no-op. Round 2 worker/reviewer confirmed local deletion, but published f2194b226ad8b1c7b19440a363367ee4088aaaa1 changed only .gitignore; the SDK file remained tracked and its deletion unstaged. Round 2 addressed a normal review finding, but the final snapshot still requested deletion of tracked root local.properties and an ignore rule. The two-repair allowance is used up. The human owner/reviewers must address remaining configuration/review policies and CI; a zero-repair observer continuation is available, not running and was not launched. On noticing the final metadata error, the owner invoked the supported pause on the same run; it was already completed (pauseRequested=false, live agents 0), so no control was applied and accounting was preserved. No caller wait/transport interruption or sleep recurrence was observed.

The E2E exposed two messaging gaps: long command narration hid the recorded stopping reason under the existing 500-character cap, and no-change repair work was described as authored/pushed source. Both are fixed separately at c5d5692ac8d9229854aa112409a43f9a66caf31b with failing-before/passing-after regressions for the actual cap, terminal/head-change evidence, unchanged/changed/unconfirmed push heads, bounds and full-receipt preservation. The pinned E2E remains unchanged. No new full workflow was run on this later message-only commit; its verification is the 255-test combined suite and exact-head CI.
The stop/control receipt is not a full-E2E success claim, and transcript receipts do not independently verify desktop rendering.

These short read-only checks are not another full implementation-to-CI E2E, do not manufacture persistent errors or long CI waits, and do not independently verify desktop rendering.

The original full E2E on db0aa9a5a38ddbdcb7e6c563f69989eed925ac77 remains a failed/limit-bound test: it stopped at maxTotalSubagents=50 during observation. Its journal accounts for 45 observer classifications (44 waiting results and one repair), plus five evidence/build/review agents. One repair was reviewed and pushed; later independent CI success does not change the original exhausted-run outcome. That run was never hot-patched with the Windows persistence fix or these follow-ups, resumed, replaced, requeued or given more budget.

Already-loaded extensions must be refreshed to adopt the completion-delivery changes. Queue admission/transcript receipts are not proof of UI rendering. Human review and merge remain required; this PR stays draft.

Review guard follow-up

Four later inline findings are fixed in 79082e4770ff099c82fe3e692f8a5f5fd1e06f8b, with fix/test replies in their original threads and all four resolved:

  • Stop confirmation requires explicit numeric liveAgentCount === 0. Missing, null, negative, nonfinite or malformed counts return an unconfirmed-stop error, including already-settled reads; permission and accounting behavior is preserved.
  • Checkout-only recovery examines failed tasks from every current-head non-review build, including unfinished builds with a null result. Known compiler/test failures prevent transport recovery, while retry targets remain only failed builds; advisory and stale builds are not source-failure evidence.
  • The checkout-retry caller explicitly selects stage-only recovery. Missing failed-stage identifiers, denied PATCHes or partial stage acknowledgement cause a pipeline-owner handoff, never a whole-build policy requeue. The separate legacy fallback for other callers is unchanged.
  • The existing index is checked before changed-file discovery, retaining the post-staging check. No-change work with only a staged SDK-config addition/edit blocks before commit/push and preserves the local file and index. Approved tracked SDK-file deletion still works.

All four defects were reproduced by failing-before regressions. Native Git/PowerShell and SDK-boundary tests now pass, and the full combined suite is 269 passed, 0 failed, 0 skipped, with exact-head CodeQL/CLA success. The branch was pushed fast-forward with the required co-author trailer and remains draft. These fixes did not start/resume any workflow, alter budgets, merge, or modify the preserved Authenticator implementation work. Full-E2E receipts remain pinned to their recorded earlier source heads.

Subsequent workflow-logic fixes

The read-only E2E investigation identified functional defects beyond messaging. They are fixed at 0ac16569303f1f3a1da67c07c457525513970f5b:

  • Remove the embedded observer's executor-identity git add -A fallback. Native Git worktrees with a forwarded executor now stage explicit repair files only; a no-change assessment creates no commit. Returned-head and remote verification use the actual supplied executor. The historical E2E commit 1e20ecb38c14d900cfa81eb89f52685589618437 did add root local.properties; that earlier attempt remains unchanged, not relabeled as a no-op.
  • Permit deletion of tracked root/nested local.properties while continuing to exclude additions/edits. Reject explicitly requested SDK-file additions and pre-staged SDK additions/edits before publication, preserving the local file/index rather than discarding user work. This fixes the case where the worker/reviewer approved deletion but deterministic commit f2194b226ad8b1c7b19440a363367ee4088aaaa1 published only .gitignore.
  • Reject absent, zero, negative, fractional and malformed policy build IDs before metadata lookup. Missing required builds remain pending/unknown; real positive-ID metadata errors remain explicit. No build-zero request or fabricated CI pass.
  • Preserve checkout connection-reset/timeout diagnostics and route a proven current-head checkout-only failure to one journaled remote retry per source head, inside the existing CI observation bound and before source-repair agents. No tactical source-repair slot is spent. Durable replay on a different build for the same head cannot request another retry. Successful stages are untouched; unconfirmed/repeated/disabled retry paths hand off to the pipeline owner. Genuine compiler/test failures retain precedence, and cancellation propagates.

Native Git and PowerShell regressions reproduce the prior bugs before the fixes and pass afterward, including no-change/real-change staging, SDK cleanup and pre-staged protection, invalid/positive build IDs, actual checkout log collection, failed-stage-only retry, boolean acknowledgement, cancellation, mixed compiler defects and durable replay. The full existing CodeFeature + DevDesignDoc suite passes 265 tests, 0 failures/skips. Related README/builder guidance is updated; syntax and diff checks pass. The commit is fast-forward and contains the required co-author trailer.

No new full E2E, resumed attempt, observer continuation, target checkout change, allowance increase or merge was performed for these fixes. A future run must cold-load this head; the completed/paused E2E receipts remain pinned to their actual earlier source revisions. The existing Authenticator PR's SDK configuration/review/CI handoff is not silently repaired by this tooling commit.
No new ADO work item was supplied for this follow-up; #488's associated item is already resolved. The existing skip AB ID validation label is retained rather than attributing another fix to that resolved item.

Replace ambiguous evidence, verification, CI and terminal wording with recorded facts and next responsible actions. Report builder dispatch only at the journaled execution boundary and retain raw statuses, recovery and delivery safeguards.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@shahzaibj
Shahzaib (shahzaibj) requested a review from a team as a code owner November 6, 2026 22:33
@shahzaibj Shahzaib (shahzaibj) added the skip AB ID validation This PR does not require an AB ID item. label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

❌ Work item link check failed. Description does not contain AB#{ID}.

Click here to Learn more.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Observer refreshes discard local-verification failure reasons, while completion formatting inserts unbounded build diagnostics into prompts and the durable outbox.

2 open issues
What changed in this PR

This follow-up to #488 clarifies CodeFeature and PR-observer progress and delivery messages without changing Android authentication behavior.

Changes:

  • Adds implementation and tactical-repair start receipts at agent invocation, with replay and cancellation coverage.
  • Separates local verification, source review, publication, CI state, and required human actions.
  • Updates completion messages, managed PR-description wording, documentation, and regression tests.
File Description
.github/​skills/​code-feature/​SKILL.md Defines plain-language progress and handoff guidance.
.github/​extensions/​CodeFeature/​workflow.test.mjs Tests dispatch timing, replay, cancellation, and milestone wording.
.github/​extensions/​CodeFeature/​workflow.mjs Adds builder-start receipts and revises workflow summaries.
.github/​extensions/​CodeFeature/​verification.test.mjs Tests verification wording, CI states, blockers, and repeated description updates.
.github/​extensions/​CodeFeature/​verification.mjs Adds readable local and CI verification summaries.
.github/​extensions/​CodeFeature/​supervision.mjs Refines progress relay and silent-wait instructions.
.github/​extensions/​CodeFeature/​status.mjs Maps delivery states and blockers to summaries and next actions.
.github/​extensions/​CodeFeature/​README.md Documents message examples and dispatch receipt semantics.
.github/​extensions/​CodeFeature/​observer.mjs Revises CI messages, repair receipts, and managed description updates.
.github/​extensions/​CodeFeature/​milestones.mjs Presents milestone summaries separately from recovery metadata.
.github/​extensions/​CodeFeature/​extension.test.mjs Updates integration assertions for start receipts and wording.
.github/​extensions/​CodeFeature/​completion.test.mjs Tests terminal messages, diagnostics, and qualified readiness claims.
.github/​extensions/​CodeFeature/​completion.mjs Formats settled outcomes with verification details and responsible owners.
.github/​agents/​pr-observer.agent.md Guides evidence-based CI summaries and owner actions.

Review effort: Balanced

Shahzaib (shahzaibj) and others added 3 commits November 6, 2026 17:38
Reproduce transient and persistent Windows sharing violations against the real outbox store. Retry atomic replacement asynchronously, serialize commits across runs, and retain durable-before-send and acknowledgement guarantees.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Fix the two review findings surfaced by live observation, clarify disabled source repairs, and keep notification incidents scoped to the failing I/O operation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An awaited outbox save can let disposal occur before the terminal send, causing a post-disposal send that removes the pending receipt; recheck disposed immediately before sending to preserve it for later delivery.

1 open issue
2 resolved since last review

Review effort: Balanced

Shahzaib (shahzaibj) and others added 4 commits November 6, 2026 20:17
Recheck disposal after the awaited outbox commit before starting a terminal send. Regressions reproduce the async-save and native Windows replacement boundaries and verify retained-receipt replay.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Confirm supported run identity and settlement through the joined SDK session without starting, resuming or changing budgets. User-requested pauses no longer fabricate prerequisite questions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The stop-confirmation guard can falsely report a successful pause when liveAgentCount is missing or null, without confirming that no live work remains.

4 open issues
1 resolved since last review

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip AB ID validation This PR does not require an AB ID item.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants