Repository navigation
fix: preserve timeliness metrics when Cloud Run integration leads - #25103
Merged
Merged
Conversation
Keep every timeliness sampler family in the sync GMP profile and dev/prod Prometheus bridges. Derive export coverage from QueueMetrics registrations and guard cross-plane alert selectors. Failure-Class: FC-metric-dropped-by-scrape-allowlist Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Match each alert input to its own newest numeric observation before aggregation so the Cloud Run bridge cannot replay a superseded breach. Retain observation history and allow 1200 seconds for bounded sampling and delivery delay. Guard every producer-derived family and wire the guard into changed-file selection. Failure-Class: FC-metric-dropped-by-scrape-allowlist Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the sync keep regex by retaining both record-token telemetry from main and all timeliness sampler families. Failure-Class: FC-metric-dropped-by-scrape-allowlist Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Preserve the full-suite fallback for previously unmapped producer and deploy changes while explicitly selecting the timeliness guard. Monitoring paths retain their existing area selection with the new guard added. Failure-Class: FC-metric-dropped-by-scrape-allowlist Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pin all four exported timeliness prefixes without weakening overlap checks. Align 1200-second freshness annotations and import contracts, and mutation-test every consuming expression without promtool. Keep Prometheus semantics an explicit local check with visible passing output. Failure-Class: FC-metric-dropped-by-scrape-allowlist Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bring in the notes prompt version fix so the full backend unit suite can pass while preserving the timeliness export and newest-source alert guards. Failure-Class: FC-metric-dropped-by-scrape-allowlist Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cloud Run backend-integration shares the timeliness lease with GKE. Its collector and bridge must retain all sampler families, and alerts must reject a superseded leader's replayed gauges.
Head:
264a2ebd46062b032a406a9571f20e465994f272Failure-Class: FC-metric-dropped-by-scrape-allowlist
Keep all producer-registered timeliness families in the sync GMP profile and dev/prod bridge. Each alert value joins its own observation with
ignoring(leg), retaining every source identity label, then selects sources tied for the newest numeric observation before aggregation. Election and sampling behavior are unchanged.Alert expressions
alerts/batch-timeliness.jsonomi-batch-unit-unpublished-2h: A/B/D;omi-batch-timeliness-sample-stale: A/D;omi-batch-timeliness-scan-blind: Aalerts/batch-input-retention.jsonomi-batch-staged-input-age: Aalerts/pipeline-drain.jsonomi-pipeline-projection-drain-stall: C;omi-pipeline-held-row-age: Aalerts/critical-failures.jsonomi-batch-held-backlog-growing: A, including growth subquery and saturationThe combined import and migration inventory mirror these expressions. Per-source
max_over_time(observed_at[20m])bridges sparse-point lookback gaps; 1200s freshness covers the 600s maximum cadence + 60s collection grace + about 180s delivery delay, with 360s jitter margin. The stale rule retains its 5m pending period. All threshold annotations and import-disposition expectations agree with the executable 1200s conditions.The four timeliness prefixes are pinned in both exporter contracts; the prefix-overlap checks remain intact. Producer names drive export and consumer coverage. Pure-Python CI tests check every consuming expression's own-source
ignoring(leg)pairing and newest-observation selector, then mutate every consumer to prove missing selection, wrong source identity and unpaired values are rejected. Existing test-selection coverage and full-suite fallback are preserved. No.githubchanges.Focused validation
Documented
backend/test.shrunner throughomi-heavy, with promtool unset as in CI: 196 passed, 1 skipped across exporter telemetry (56), production import disposition (7), structural/export guard (52 plus the intentional local-tool skip), monitoring alert contracts (75), and notes prompt-version checks (6). The pure-Python structural and mutation tests run in CI. The promtool fixture is explicitly local-only, opts in throughPROMTOOL_BIN, and its skip reason states that CI does not run it.Explicit local check, promtool 2.55.1 installed outside the repository:
Passing output from the committed fixture on this head:
The fixture invoked
promtool test ruleson its generated YAML on this merge head (exit 0):This evaluates both replayed handoff directions, every consuming split expression, sustained 600s observations with 180s Cloud Run delay, and a stopped sampler that still pages.
Main synchronization: merged current
origin/mainnormally, without conflicts. This includes the notes prompt-version fix verified by green main Backend Unit Tests run 43000370042. No rebase or force-push. The previous notes-digest blocker is resolved by upstream; the focused suite and explicit local promtool fixture pass on this merge head. Backend Unit Tests run 43006342683 passed every shard (1/8, 2/8, 3/8, 4/8, 5/8, 6/8, 7/8, 8/8) and backend guardrails on this exact head. The previous shard-3 notes-digest failure is cleared.Post-deploy qualification requires a sustained Cloud Run-led window, covering multiple sample intervals and pending windows in GMP and the Prometheus alert datasource, followed by both handoff directions. A first visible sample is insufficient. No merge, deployment, GCP tier or production-account access.
Product invariants affected
none
🤖 Generated with Claude Code