Skip to content

fix: preserve timeliness metrics when Cloud Run integration leads - #25103

Merged
Git-on-my-level merged 6 commits into
mainfrom
lane/timeliness-export
Nov 11, 2026
Merged

Git-on-my-level merged 6 commits into
mainfrom
lane/timeliness-export

Conversation

@Git-on-my-level

@Git-on-my-level Git-on-my-level commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Cloud Run backend-integration shares the timeliness lease with GKE. Its collector and bridge must retain all sampler families, and alerts must reject a superseded leader's replayed gauges.

Head: 264a2ebd46062b032a406a9571f20e465994f272

Failure-Class: FC-metric-dropped-by-scrape-allowlist

Keep all producer-registered timeliness families in the sync GMP profile and dev/prod bridge. Each alert value joins its own observation with ignoring(leg), retaining every source identity label, then selects sources tied for the newest numeric observation before aggregation. Election and sampling behavior are unchanged.

Alert expressions

Source Rule UID / refs
alerts/batch-timeliness.json omi-batch-unit-unpublished-2h: A/B/D; omi-batch-timeliness-sample-stale: A/D; omi-batch-timeliness-scan-blind: A
alerts/batch-input-retention.json omi-batch-staged-input-age: A
alerts/pipeline-drain.json omi-pipeline-projection-drain-stall: C; omi-pipeline-held-row-age: A
alerts/critical-failures.json omi-batch-held-backlog-growing: A, including growth subquery and saturation

The combined import and migration inventory mirror these expressions. Per-source max_over_time(observed_at[20m]) bridges sparse-point lookback gaps; 1200s freshness covers the 600s maximum cadence + 60s collection grace + about 180s delivery delay, with 360s jitter margin. The stale rule retains its 5m pending period. All threshold annotations and import-disposition expectations agree with the executable 1200s conditions.

The four timeliness prefixes are pinned in both exporter contracts; the prefix-overlap checks remain intact. Producer names drive export and consumer coverage. Pure-Python CI tests check every consuming expression's own-source ignoring(leg) pairing and newest-observation selector, then mutate every consumer to prove missing selection, wrong source identity and unpaired values are rejected. Existing test-selection coverage and full-suite fallback are preserved. No .github changes.

Focused validation

Documented backend/test.sh runner through omi-heavy, with promtool unset as in CI: 196 passed, 1 skipped across exporter telemetry (56), production import disposition (7), structural/export guard (52 plus the intentional local-tool skip), monitoring alert contracts (75), and notes prompt-version checks (6). The pure-Python structural and mutation tests run in CI. The promtool fixture is explicitly local-only, opts in through PROMTOOL_BIN, and its skip reason states that CI does not run it.

Explicit local check, promtool 2.55.1 installed outside the repository:

PROMTOOL_BIN=/Volumes/Ephemeral/scratch/tokenlease-promtool/promtool \
PYTEST_ADDOPTS='-s -k test_prometheus_replayed_handoffs_and_sustained_leadership' \
BACKEND_UNIT_TEST_FILE_LIST=<one-file-list> bash backend/test.sh

Passing output from the committed fixture on this head:

1 passed, 52 deselected in 2.83s

The fixture invoked promtool test rules on its generated YAML on this merge head (exit 0):

  SUCCESS

This evaluates both replayed handoff directions, every consuming split expression, sustained 600s observations with 180s Cloud Run delay, and a stopped sampler that still pages.

Main synchronization: merged current origin/main normally, without conflicts. This includes the notes prompt-version fix verified by green main Backend Unit Tests run 43000370042. No rebase or force-push. The previous notes-digest blocker is resolved by upstream; the focused suite and explicit local promtool fixture pass on this merge head. Backend Unit Tests run 43006342683 passed every shard (1/8, 2/8, 3/8, 4/8, 5/8, 6/8, 7/8, 8/8) and backend guardrails on this exact head. The previous shard-3 notes-digest failure is cleared.

Post-deploy qualification requires a sustained Cloud Run-led window, covering multiple sample intervals and pending windows in GMP and the Prometheus alert datasource, followed by both handoff directions. A first visible sample is insufficient. No merge, deployment, GCP tier or production-account access.

Product invariants affected

none

View guided diff Turn on auto-fix

🤖 Generated with Claude Code

Git-on-my-level and others added 6 commits November 11, 2026 13:55
Keep every timeliness sampler family in the sync GMP profile and dev/prod Prometheus bridges. Derive export coverage from QueueMetrics registrations and guard cross-plane alert selectors.

Failure-Class: FC-metric-dropped-by-scrape-allowlist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Match each alert input to its own newest numeric observation before aggregation so the Cloud Run bridge cannot replay a superseded breach. Retain observation history and allow 1200 seconds for bounded sampling and delivery delay. Guard every producer-derived family and wire the guard into changed-file selection.

Failure-Class: FC-metric-dropped-by-scrape-allowlist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the sync keep regex by retaining both record-token telemetry from main and all timeliness sampler families.

Failure-Class: FC-metric-dropped-by-scrape-allowlist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Preserve the full-suite fallback for previously unmapped producer and deploy changes while explicitly selecting the timeliness guard. Monitoring paths retain their existing area selection with the new guard added.

Failure-Class: FC-metric-dropped-by-scrape-allowlist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pin all four exported timeliness prefixes without weakening overlap checks. Align 1200-second freshness annotations and import contracts, and mutation-test every consuming expression without promtool. Keep Prometheus semantics an explicit local check with visible passing output.

Failure-Class: FC-metric-dropped-by-scrape-allowlist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bring in the notes prompt version fix so the full backend unit suite can pass while preserving the timeliness export and newest-source alert guards.

Failure-Class: FC-metric-dropped-by-scrape-allowlist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Git-on-my-level
Git-on-my-level merged commit 62935e7 into main Nov 11, 2026
66 checks passed
@Git-on-my-level
Git-on-my-level deleted the lane/timeliness-export branch November 11, 2026 11:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant