[wdconinc]

Briefly, what does this PR introduce?

Two GitLab CI robustness fixes around the per-pipeline BuildKit builder (BUILDER_NAME).

1. Race on docker buildx create (eicweb job 8767659)

Two jobs of pipeline 152034 started at the same moment on eic-n0 (base-cuda: [cuda_runtime] and base: [debian_stable_base]):

cuda_runtime 22:11:10.37 buildx create → OK (no container yet → node inactive, no connection)
cuda_runtime 22:11:11.14 bootstrap: creating container buildx_buildkit_builder-1520340
debian       22:11:11.33 buildx create → container running, buildkitd not yet listening
                         "failed to initialize builder ... listing workers"
                         → buildx drops the builder from this job's config
debian       22:11:12.69 fallback `buildx use` → "failed to find instance" → exit 1
cuda_runtime 22:11:12.98 container done

• buildx create connects to a running builder container to list workers, so the race shows up there, not at bootstrap as the comment assumed.
• When that check fails, buildx removes the builder entry it just wrote. Each job starts with an empty buildx config, so the || docker buildx use fallback can't recover, and the job exits before the bootstrap retry loop runs.
• The window is about 2 s, but every pipeline opens it because all base and base-cuda jobs start together.

Fix: retry create || use together with inspect --bootstrap (5 attempts, 10 s apart). After a successful create, a later attempt fails on the existing name and use selects it. I also updated the comment.

2. clean_builder aborted by a bad API response (eicweb job 8746912)

One pipelines-API curl took 5 min and returned non-JSON. jq then failed ("Invalid numeric literal") and the job exited (code 5), so it skipped the remaining stale builders. Now curl --fail --max-time 30 is used, and a failed or unparseable response leaves the status empty, so that builder is kept and the loop continues.

Testing

• YAML parses, and the folded before_script line is valid bash.
• Ran the loop under bash -eo pipefail with a stubbed docker:
  • create fails once, then succeeds: recovers, exit 0.
  • create always fails: 5 attempts, exit 1.
• The jq line with a non-JSON input gives an empty status and doesn't exit.
• Not yet run on GitLab CI.

What kind of change does this PR introduce?

• Bug fix
• New feature
• Documentation update
• Other

---

Generated by Claude Code

[Copilot]

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🔵 Needs a closer look

The change alters GitLab CI builder creation/cleanup on shared runners and, because its correctness depends on real GitLab runner concurrency and Docker Buildx/BuildKit timing/API behavior, it should be validated by an actual pipeline run before proceeding.

No open issues 🎉

What changed in this PR

Improves GitLab CI robustness for per-pipeline BuildKit builders by mitigating a buildx startup race and making stale-builder cleanup resilient to flaky GitLab API responses.

Changes:

• Retry docker buildx create || docker buildx use together with docker buildx inspect --bootstrap to handle transient builder readiness/concurrency races.
• Harden clean_builder by bounding/failing curl safely and tolerating non-JSON/failed API responses so the cleanup loop continues.

File	Description
.gitlab-ci.yml	Adds retry logic around Buildx builder selection/bootstrapping and hardens the stale builder cleanup loop against slow/failed/non-JSON GitLab API responses.

_{Review effort: Balanced}

[github-actions]

Capybara summary for PR 504

• rec_dis_10x100_minQ2=100_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• rec_dis_10x100_minQ2=1_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• rec_dis_18x275_minQ2=100_epic_craterlake_amd64 (Δ: 290) (new artifact) (ref artifact)
• rec_dis_18x275_minQ2=1_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• rec_dis_5x41_minQ2=1_epic_craterlake_amd64 (Δ: 270) (new artifact) (ref artifact)
• rec_dis_5x41_minQ2=1_epic_craterlake_amd64_vs_arm64 (Δ: 4917)
• rec_dis_5x41_minQ2=1_epic_craterlake_arm64 (Δ: 232) (new artifact) (ref artifact)
• rec_e_1GeV_20GeV_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• rec_e_1GeV_20GeV_epic_craterlake_amd64_vs_arm64 (Δ: 3730)
• rec_e_1GeV_20GeV_epic_craterlake_arm64 (Δ: 1) (new artifact) (ref artifact)
• rec_pi_1GeV_20GeV_epic_craterlake_amd64 (Δ: 84) (new artifact) (ref artifact)
• rec_pi_1GeV_20GeV_epic_craterlake_amd64_vs_arm64 (Δ: 3453)
• rec_pi_1GeV_20GeV_epic_craterlake_arm64 (Δ: 227) (new artifact) (ref artifact)
• sim_dis_10x100_minQ2=100_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• sim_dis_10x100_minQ2=1_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• sim_dis_18x275_minQ2=100_epic_craterlake_amd64 (Δ: 13) (new artifact) (ref artifact)
• sim_dis_18x275_minQ2=1_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• sim_dis_5x41_minQ2=1_epic_craterlake_amd64 (Δ: 13) (new artifact) (ref artifact)
• sim_dis_5x41_minQ2=1_epic_craterlake_amd64_vs_arm64 (Δ: 408)
• sim_dis_5x41_minQ2=1_epic_craterlake_arm64 (Δ: 12) (new artifact) (ref artifact)
• sim_e_1GeV_20GeV_epic_craterlake_amd64 (Δ: 1) (new artifact) (ref artifact)
• sim_e_1GeV_20GeV_epic_craterlake_amd64_vs_arm64 (Δ: 369)
• sim_e_1GeV_20GeV_epic_craterlake_arm64 (Δ: 1) (new artifact) (ref artifact)
• sim_pi_1GeV_20GeV_epic_craterlake_amd64 (Δ: 12) (new artifact) (ref artifact)
• sim_pi_1GeV_20GeV_epic_craterlake_amd64_vs_arm64 (Δ: 346)
• sim_pi_1GeV_20GeV_epic_craterlake_arm64 (Δ: 23) (new artifact) (ref artifact)
  ^{_{Last updated 2026-11-13T03:14-05:00 3021657}}