Skip to content

Latest commit

 

History

History
1877 lines (1595 loc) · 114 KB

ci.md

File metadata and controls

1877 lines (1595 loc) · 114 KB

Continuous integration

This page explains what each GitHub Actions workflow checks, how the runtime batches are planned and run, and how to investigate a failing batch. How to run tests locally and how to add them is described in Testing.

Workflows

Workflow File Triggers What it checks
Code checks code.yml maintained branch push or PR, manual; weekly (Monday 05:17 UTC) for the vulnerability check only Unit tests, race detector, vet, lint, tag compilation, CI tooling tests, compiled tag coverage, Go vulnerability check
Docker checks docker.yml development push or PR, manual The environment category: Docker bridge behaviour without Ceph
Short short.yml development push or PR, manual The short category on Tentacle 20.2.4
Topology topology.yml development PR, manual The topology category on Tentacle 20.2.4
Multicluster multicluster.yml development PR, manual The multicluster category on Tentacle 20.2.4
Recovery recovery.yml development PR, manual The recovery category on Tentacle 20.2.4
Periodic periodic.yml daily at 04:41 UTC; manual Random flaky samples on supported releases or historical patches; explicit full matrices and repeated investigations
Release matrix release-matrix.yml PR targeting main, release/* or support/*, push to main or support/*; manual Environment and every required runtime category on every supported Ceph release
Structural review evidence structural-collect.yml pull request Report-only structure and lint deltas of library changes, published as one PR comment by structural-review.yml; not a required check
Experiments experiments.yml manual only One known-defect case with go test: every result line REPRODUCED on its default image, then, when a fixed image is given, the same results NOT_REPRODUCED with it injected
Native regressions native-regressions.yml manual only One optional batch chosen from the workflow's batch input, taking only its pass or failure: an experiment case, another optional native check or the failure_archive proof (see Native regressions for how to list and dispatch the choices)
ARM64 arm64.yml weekly (Wednesday 05:37 UTC); pull request touching its planner, image, storage, runtime template, bootstrap or helper paths; manual The source-owned arm64 cohort on a native GitHub-hosted ARM64 runner (see Native ARM64 cohort); not a required check
Block fault probe block-fault-probe.yml pull request touching its scripts or workflow, manual A research probe of delay and I/O-error tables on a scratch loop and device-mapper mapping, without Ceph; manual public_api also qualifies the public API on every supported release using a source-pinned helper; not a required check
CI command ci-command.yml pull request comment /ci run ..., /ci require ..., /ci retry ... or /ci drop ... The named runtime batches on the pull request's head commit (see Running chosen batches from a comment); the last three also change the pull request's required requested CI
Requested CI gate requested-ci-gate.yml every pull request run of Code checks, when it is requested; manual Runs no test. Sets the commit status Required requested CI on the head of every open pull request (see Requiring requested CI before a merge)

The ordinary runtime workflows and explicit full periodic runs call the reusable template tagged-runtime.yml (workflow_call only). The workflows are independent: none waits for another, and each can be re-run on its own.

Displayed check names are layered as <workflow> / <caller job> / <batch>, for example Multicluster / Scenarios / mirror_initial_daemons or Periodic / Squid topology / empty_bootstrap. Each runtime workflow also has a Plan job.

Pull request scope

A feature pull request to develop runs, on Tentacle 20.2.4 role images and in each batch's default network mode:

  • Code checks, for every change;
  • Docker checks and every short batch. With the ci:selected-batches label, Short runs only the batches that declaration selection selects (see Opting in to declaration selection);
  • of the topology, multicluster and recovery categories, only the batches whose own test files the pull request changes, and the batches of an experiment case any of whose files (helpers and manual included) it changes. A change to a shared helper or a public package selects none of them; Periodic runs them on the default branch.

The ci:full label (adding it starts a run) selects every batch of those categories instead. Each Plan job's tag-plan.json artifact lists the changed files and the selected batches.

Own-test-file selection compares proven commits. The caller passes the event's base commit and the requested head; the planner records both, the checked-out commit and its parents, and the compared range in changed_selection. A two-parent test merge is accepted only when its second parent is the requested head and the base is an ancestor of its first parent. Missing commits, ambiguous merge bases or unrelated history fail planning instead of selecting nothing. To inspect a selection manually:

python3 .github/scripts/tag_scenarios.py plan --category recovery \
  --changed-since BASE_SHA --changed-head PR_HEAD_SHA --output plan.json

Without --changed-head the planner uses a plain merge-base comparison (recorded as legacy-merge-base). --changed-files FILE is a separate manual input.

Pushes to develop run Code checks, Docker checks and Short. PRs targeting main, release/* or support/* and pushes to those branches receive release candidate validation. Development continues under ordinary CI while a candidate is frozen.

To run any other batch on a pull request, such as one batch of another category or a batch on Squid, use /ci run <batch>. The command adds a run and never changes the checks above. A reviewer who needs that run to pass before the merge uses /ci require <batch> instead.

The workflows set GOPROXY=https://proxy.golang.org|direct, so a failed proxy request falls back to the module's origin. The Go jobs share one module cache keyed by go.sum, and the runtime jobs share one build cache keyed by the installed Go version and go.sum. Each runtime Plan job fills a missing module cache with tag_scenarios.py modules and a missing build cache with tag_scenarios.py buildcache (one batch per package); runtime jobs and Code checks only restore them. A comment run passes save_caches: false and saves neither: it belongs to the default branch, whose caches every later run restores, but checks out an unmerged head. setup-go's own cache is off. A cache only speeds up a build: each job still compiles, records and verifies its own binary.

Documentation-only changes

Ordinary development runtime workflows use paths-ignore for docs/**, top-level *.md, LICENSE and internal/integration/goceph/**, so a change to only those paths runs no runtime batch. A Markdown file anywhere else, for example under a package's testdata/, can be test input and runs the batches.

Code checks always starts, because its checks are required status checks of the maintained branch rules. Its Detect documentation-only changes job runs .github/scripts/docs_only.py. When that reports docs_only=true, Unit tests, Race detector, Vet, compile and CI tooling and Compiled tag coverage and helpers are skipped as whole jobs (a skipped job reports success), and the Vulnerability check matrix jobs run with every step skipped so that each keeps its expanded check name. The skip is by policy, not a pass of those checks.

A pull request to develop is documentation-only when every changed file is on this allowlist:

  • README.md, any other top-level *.md file and LICENSE;
  • Markdown files directly inside docs/contributing/, docs/guides/ and docs/internals/, except the files a check reads: docs/contributing/api-naming.md and docs/contributing/configuration-composition.md (the review note that the API catalogue receipts name, and the design note that ceph doc comments link to), docs/guides/messenger-client-fixtures.md and docs/guides/bug-compatibility.md.

Every other path runs the full Code checks, including a package's README.md (Go tests read it), any non-Markdown or nested file under docs/, and a changed path whose old or new Git mode is not an ordinary file. test_docs_only.py fails when a Go source or tag_scenarios.py names a documentation path that is on the allowlist. Add a newly read file to READ_DOC_FILES in docs_only.py.

The detection fails open. Release candidates, maintained branch pushes, the weekly schedule, manual runs, pull requests from forks, and any failure to determine the change set (no base, failed fetch or diff, empty list, script error) run every check. The dependent jobs use !cancelled(), so a failed detection job runs them. The script compares the pull request's merge commit with the event's base and lists renames as a deletion and an addition.

Concurrency and cancellation

Workflows Concurrency group Effect
Code checks, Docker checks, Short, Topology, Multicluster, Recovery, Release matrix workflow name, event, then the PR number (pull requests), the ref (pushes) or the run ID (manual runs) A new commit on a PR cancels that PR's previous run of the same workflow. A new push to main cancels the previous main push run. Manual runs are never cancelled.
Periodic workflow name and ref; explicit representative investigations add -campaign Rotate, discovery, targeted, full and storage runs wait in one group. They never cancel an executing run. The representative group is separate and bounded; it has no schedule. GitHub retains only one waiting run per group, so a later queued request can replace an earlier queued request.
Topology, Multicluster, Recovery as above, plus the run ID for a label other than ci:full An unrelated label starts a run that skips its jobs and cancels nothing.
Short as above, plus the run ID for a label other than ci:selected-batches Adding or removing ci:selected-batches replaces the run in progress.
CI command per caller job: PR number, head commit, selected profiles and release An identical command waits for the one in progress; a third identical command replaces the waiting one, which ends cancelled. Different selections run independently.
CI command (Register, Gate), Requested CI gate the fixed name required-requested-ci, per job Every job that publishes Required requested CI waits its turn and none is cancelled while it runs. A waiting job is replaced by a newer one (see How the status is kept).
Structural review evidence workflow name and PR number A new commit on a PR cancels that PR's previous run.
Native regressions none Runs are independent.
ARM64 workflow name, then the PR number or the run ID A new commit on a PR cancels that PR's previous run. Scheduled and manual runs are never cancelled.
Block fault probe workflow name, then the PR number or the run ID A new commit on a PR cancels that PR's previous probe.

Consequences:

  • Different PRs never cancel each other. In a stack, base each PR on the head branch of the previous one.
  • Retargeting a PR requires checks for the new target. Confirm that the run identifies the intended base and commit before using its evidence. GitHub also emits edited for title and body changes. Those edits rerun candidate validation, so prepare the description before opening the promotion PR.
  • Push runs are scoped to their own branch. A change on develop cannot cancel a release candidate on another branch.

Code checks

The local commands for these jobs are in Local checks without Docker.

Job Timeout Runs
Unit tests 15 min make test
Race detector 15 min make race
Vet, compile and CI tooling 25 min make vet, golangci-lint, make tag-compile, the Python tests in .github/scripts (test_*.py), actionlint and the go-ceph harness test
Compiled tag coverage and helpers 25 min tag_scenarios.py plan --verify, then tag_scenarios.py code (the Docker-free code batch); uploads code-tag-coverage
Vulnerability check (Go 1.26.x and 1.27.x) 20 min make vulncheck; uploads vulncheck-go-<version>

golangci-lint is the release that code.yml pins by version and archive SHA-256. The golangci-lint action installs it for this job and for the structural review collector: it verifies the archive restored from the Actions cache, or downloads it with up to five attempts that retry only transport failures and 429/5xx answers, and the lint step itself downloads nothing. To move to another release, change both callers' version and sha256 (from the release's checksums.txt) and the Makefile comment; test_golangci_lint_install.py checks that they agree.

The Python tests include test_tag_scenarios.py for the planner and test_scenario_shards.py, which checks the workflow files: one workflow per category, no test-name lists in workflows, budgets taken from the source, compile before Docker, cleanup on the same runner, and failure reporting that never hides the test step's status.

Vulnerability check

make vulncheck builds govulncheck v1.9.0 and scans the module with -test for linux/amd64 with no tags, all and all,goceph,native_regression, and for linux/arm64 and darwin/arm64 with no tags. The nested go-ceph probe module needs cgo, so it gets a module scan only.

.github/scripts/check_vulns.py decides the result, because govulncheck's JSON mode exits 0 on findings. Per vulnerability it takes the most precise level: symbol and package findings fail the check, module-only findings are listed and pass, and a malformed or incomplete stream fails as a broken scan.

The job runs with the module's minimum Go (1.26.x) and the current release (1.27.x), because standard-library findings depend on the toolchain. It restores the shared module cache and the build cache for its Go version; a successful default-branch scan saves a missing build cache. Caches never skip a scan or retain the vulnerability database. Every setup-go step sets check-latest: true, and go.mod names a toolchain line, so an older patch release with known standard-library vulnerabilities is not used. The weekly scheduled run catches new advisories; the other Code checks jobs skip it. Dependabot targets develop for version updates. Its security alerts on the default branch also need the maintained-line patch procedure. .github/dependabot.yml proposes Go module updates daily and GitHub Actions updates weekly.

How batches are planned

.github/scripts/tag_scenarios.py builds the CI matrix from the test source. Nothing in the workflows names a test.

  1. Catalog. internal/tools/tagcatalog parses every *_test.go file (skipping hidden directories, artifacts, vendor and testdata) and records its //go:build expression, tags, Test... functions and //ci: metadata.
  2. Discovery. Every file that declares tests and has a constraint belongs to exactly one category (ci_<category>) and one batch (ci_batch_<name>), compiles under all, and carries one valid //ci: line. One (category, batch, package) triple is one profile; its files agree on budget and image roles. Optional (ci_optional) and SDK (ci_sdk) files stay out of the required set and need a tag beyond all.
  3. Profile tags. A batch compiles with every capability tag found in the source plus ci, ci_<category>, ci_batch and ci_batch_<name>. No file of another batch may compile into the profile.
  4. Selection. plan --category <category> writes that category's profiles as the matrix; plan --optional --batch <name> selects one optional batch; plan --representative selects the campaign set.
  5. Verification. plan --verify compiles the all, category and single-batch builds, including every optional batch, with go test -list and compares the test names with the catalog. Code checks runs it; the runtime workflows do not.

Each profile has a stable ID, <category>-<batch>-<8 hex digits>, such as short-default-a488da21, used in artifact names, summaries and annotations.

The planner, compiler and runner always use CGO_ENABLED=0, GOWORK=off and GOFLAGS=-mod=readonly, and pass only build tags and the batch timeout to go test, with no -short or name filters; the only -run is the list of mode-declaring tests of a network-mode entry other than the default. To move a test between batches, change its build constraint.

Checker batches (CephFS process quiescence and process recovery per receipt kind, and RBD receiver readiness) declare checker= and case= in //ci:. They run in one network mode per entry, and afterwards check_scenario_quiescence.py, check_scenario_recovery.py or check_scenario_receivers.py verifies that mode's evidence in the log. The planner requires the batches of a checker to cover every mode and kind.

Bug compatibility selection

A ceph_bug_<id> tag selects the compatibility path of one Ceph bug (see the policy); it never selects tests. The planner never collects it as a capability, and all or native_regression never enable it. Only --bug-compatibility TAGS, accepted by plan, verify, compile, run, code, optional and buildcache, selects one:

python3 .github/scripts/tag_scenarios.py plan --bug-compatibility ceph_bug_<id>

The selection is refused when a tag is not registered in bug-compatibility-tags.json, is retired, is repeated or conflicts with another selected tag. The planner also refuses a test file that names an unregistered or retired bug tag, and a registered tag that no source file names. With no selection every ordinary path compiles.

Both paths of a bug must keep the same tests in the same batch: for every profile and each package's all build the planner toggles each bug tag the package names and requires identical test names. Toggles that would activate conflicting tags are skipped. The selected tags apply to every profile of the plan, appear in its tags and bug_compatibility, and so change the build key, compile receipt and history configuration, but not the profile ID. run --binary refuses a binary compiled under another selection.

tagged-runtime.yml takes the selection as the bug_compatibility input (empty by default). Plan passes it to plan and buildcache, so an invalid selection fails before any runtime job. Each runtime job passes it to compile, run or optional with --profile-tags set to its matrix entry's tags, and fails before building when the recomputed profile differs. The selection reaches commands only as an argument; tag_scenarios.py reads none from the environment. No caller sets the input, so every workflow runs the empty selection.

Advisory declaration selection

internal/tools/batchselect narrows each integration test package (internal/integration and its rbd, cephfs and rgw packages) at the level of declarations, because its *_helpers_test.go files and the shared internal/integration/harness compile into every batch. By default it is only reported:

python3 .github/scripts/tag_scenarios.py plan --category short \
  --select-declarations origin/develop --apply-declarations \
  --summary /dev/stdout --output artifacts/tag-plan.json

Without --apply-declarations, the plan gains a declaration_selection object (selected and excluded reasons per profile, fallback, changed_declarations) and keeps the full matrix. With it, the matrix keeps only the selected profiles and selected_scope is declarations. --summary FILE appends the decisions as a Markdown table.

The analysis has no default platform: the planner passes -goos linux and its --goarch (amd64 unless a plan names another), and the plan records both, so a selection never describes another architecture's source files than the plan's. The tool diffs the checkout against its merge base with the base, type-checks the package's all build (and each batch build not covered by it) with go/types, together with the harness from source (-with), and builds one graph over the top-level declarations of both. Edges are resolved identifiers, interface method calls to every implementing type of the two packages, and each type to all its methods. A test file of another package is never part of the build and is ignored. A batch is selected when one of its roots (Test, Fuzz and Example functions, init functions and package-level var initializers it compiles) reaches a changed declaration. An added file counts as a change to each of its declarations.

Every batch is selected, with the reason, when the change touches anything the graph cannot attribute: a line outside every declaration (file header, //go: directive), an import of a package that no other file of the build imports, a deleted, untracked or non-Go file in the package, .github/, go.mod, go.sum, any other path outside the package and the harness other than documentation, a removed init or initializer, or anything that fails to parse or type-check. Each package's fallback selects only that package's batches. Profiles outside the integration packages are always selected, and a tool failure selects every profile.

Opting in to declaration selection

Short is the only workflow that narrows by declarations, and only for a development pull request with the ci:selected-batches label. The caller then passes the base commit as the select_declarations input of tagged-runtime.yml, and Plan runs plan --select-declarations BASE --apply-declarations --summary "$GITHUB_STEP_SUMMARY". The Plan summary lists every batch with yes or no and its reason.

Adding or removing the label replaces the run in progress; later commits run selected while the label stays. Every other trigger and workflow runs all its batches, and the develop push after a feature merge runs every Short batch, so a selection miss fails there. Record a miss with its plan, failed job and selection evidence in a new issue. The label only helps for changes inside the integration test packages and the harness.

Running chosen batches from a comment

A collaborator can run chosen runtime batches on a pull request by commenting:

/ci run <batch|profile>... [on <release|codename|all>...]

For example, /ci run rbd_mirror_bandwidth network_faults on squid runs two batches on Squid 19.2.5 role images. The run is added beside the usual checks.

  • Names. A batch (network_faults), category and batch (short/network_faults), profile name (short/network_faults/internal-integration) or profile ID, separated by blanks or commas. Any runtime category and the optional batches are allowed; code batches are refused. Duplicates run once.
  • Releases. After on: a release (20.2.4), a codename (tentacle, squid) or all. Without on, the current release. Each batch runs in its default network mode.
  • Validation. Names are checked against the head commit's planner (tag_scenarios.py vocabulary). One unknown name refuses the whole command, and the reply lists every unknown name.
  • Format. The command must start its line and appear once per comment; quoted or fenced lines are ignored. Editing a comment does not run it.
  • Authorization. Only authors with association OWNER, MEMBER or COLLABORATOR run commands; others get a refusal reply. Fork and closed pull requests are refused; issue comments and bots are ignored.

The workflow records the pull request's head commit when it reads the pull request, and every job runs that commit. issue_comment runs use the workflow and scripts from the protected default branch:

Job Permissions What it does
Command contents: read, pull-requests: read Runs ci_command.py authorize on the default-branch checkout. No pull request code runs.
Validate contents: read Checks out the default branch and the recorded head side by side without credentials; the head's planner lists batches and releases, the default branch's ci_command.py select resolves the names.
Register contents: read, pull-requests: write, statuses: write Only for /ci require, /ci retry and /ci drop: runs required_ci.py register on the default-branch checkout (see Requiring requested CI before a merge).
Acknowledge statuses: write Sets a pending commit status on the head.
Scenarios <release> contents: read Calls tagged-runtime.yml with category selected and the resolved profile IDs, once per release.
Report contents: read, pull-requests: write, statuses: write Sets the final status and replies with the result, run link or refusal.
Gate contents: read, pull-requests: write, statuses: write Only after the run of a required selection: runs required_ci.py evaluate on the default-branch checkout.

The result appears on the pull request as the commit status CI command <comment-id>: <names> on <releases> (success, failure, or error for a cancelled run). A context longer than 100 characters is cut and ends with a 12-character digest of the selection. Refused commands set no status. A cancelled run is never re-run automatically; comment again or re-run the workflow run.

To check a command locally on saved event and pull request JSON:

python3 .github/scripts/tag_scenarios.py vocabulary --output artifacts/vocabulary.json
python3 .github/scripts/ci_command.py authorize --event event.json --pull pull.json
python3 .github/scripts/ci_command.py select \
  --request '{"names": ["network_faults"], "selectors": ["squid"], "duplicates": [], "sha": "<head>", "pull": 1, "comment_id": 1}' \
  --vocabulary artifacts/vocabulary.json

Requiring requested CI before a merge

/ci run is advisory: no merge depends on its result. To make a selection part of the merge decision of one pull request, a reviewer requires it:

/ci require <batch|profile>... [on <release|codename|all>...]
/ci retry <batch|profile>... [on <release|codename|all>...]
/ci drop <batch|profile>... [on <release|codename|all>...]

The names, releases, format and authorization are those of /ci run, and so are the refusals: an unknown name, a fork, a closed pull request or an unauthorized author changes no requirement.

Command Effect
/ci require Adds the selection to the pull request's requirements and runs it on the head commit. Requiring a selection that is already required changes nothing and runs nothing. Different selections add up, and each must pass.
/ci retry Runs a required selection again on the current head commit, after a failure, a cancelled run or a new push. That run replaces the earlier one as the selection's evidence. Its names are resolved again on the new head; if they no longer resolve, the selection is recorded as blocked and fails until the pull request restores the names or the selection is dropped.
/ci drop Removes the selection from the requirements. Its names are not resolved, so a selection whose batch is gone can be dropped.

A selection is its names and release selectors as typed, in any order. /ci require network_faults and /ci require network_faults on tentacle are two selections even while Tentacle is the current release, so retry and drop repeat the selection as it was required. The summary comment lists the exact commands.

The status

The commit status Required requested CI is set on the head commit of every open pull request, with or without requirements:

State Meaning
success The pull request requires nothing, or every required selection passed on this head commit.
pending A required selection is running, or has no run on this head commit.
failure A required selection failed, its run was cancelled, skipped or ended without a known result, or its names no longer resolve.
error The requirements are not known: a comment of the ledger was edited or cannot be trusted, the comments of a pull request with this head could not be read, or the job that publishes the status did not finish.

A pull request with requirements has one summary comment, which the status links to. It lists every required selection with its state, the commit that was tested, the command, run and result that are its evidence, who required it, and the retry and drop commands for it. The comment is updated in place.

What satisfies a requirement is narrow on purpose:

  • only a successful run of the selection's latest accepted require or retry command counts. The result of a command that a later retry replaced, or of a requirement that was dropped and required again, is ignored whenever it arrives;
  • the run must have tested the pull request's current head commit. After a push the requirement stays and its evidence does not: the new head is pending until the selection is retried on it. Nothing starts that run automatically;
  • the run tests the head commit itself, not a merge with the base, so a change of the base branch or of the pull request's target neither satisfies nor invalidates a requirement;
  • a result is recorded on the pull request whose command started the run. The same commit tested for another pull request is no evidence;
  • a run without a recorded result is no evidence. A selection that stays running after its run ended lost its result; retry it.

A commit status belongs to a commit, and GitHub evaluates a required status on the head commit of the pull request. Pull requests that share a head commit therefore share the status, and it is the worst of theirs: none of them passes until the requirements of each are met. The description names the pull request that decides it.

Reviewer steps

  1. Request. Comment /ci require <selection> on the pull request.

  2. Wait for the acknowledgment. The comment is processed by a workflow run, seconds to minutes later. Until then nothing is required and the status can still be green, so hold the merge. The acknowledgment is a reply of github-actions that starts with Required:; the status of the head is set pending before that reply is written. A reply that says the command is not confirmed, or no reply at all, means that the command may not have been recorded: read the summary comment and repeat the command.

  3. Read the result. Report replies with the outcome and the run link, and the status and the summary follow. A failed selection is diagnosed like any failed batch (see Reading a failing batch).

  4. Retry or drop. /ci retry <selection> after a fix was pushed or when the failure was not the change's; /ci drop <selection> when the selection is no longer wanted. Both are acknowledged the same way (Retrying the required selection:, No longer required:), and both stay on the pull request with their author.

  5. Verify before the merge. Required requested CI is success on the exact commit to merge, and the summary comment names that commit as its head and lists every selection that was asked for as passed. A summary for another commit means the status was not evaluated for this pull request (see How the status is kept):

    gh api repos/<owner>/<repository>/commits/<head-sha>/status \
      --jq '.statuses[] | select(.context == "Required requested CI") | {state, description, target_url}'

    The status blocks a merge by itself only on a branch whose rule requires the context (see Enabling the branch rule). On any other branch this step is the reviewer's.

Do not edit or delete the replies of github-actions that record a requirement. An edited one turns the status into error; the summary then links to it, and deleting it and repeating the commands it stood for restores the ledger. GitHub shows the deletion in the pull request's timeline.

This is separate from the release gate. A release candidate's own retry rule (see Release candidate validation) decides what a candidate may contain; a required selection is a check of one reviewed commit and of exactly the coverage the reviewer asked for, and it has no automatic retry.

How the status is kept

The requirements of a pull request are a ledger of comments on it. A record is the first line of a comment written with the workflow token, an HTML comment that Register (require, retry, blocked, drop) and Report (result) write and never change. required_ci.py reads the ledgers again on every evaluation and derives every status from them; nothing else stores a requirement. A comment by anyone else that copies a record is not one.

Three jobs publish the status, all with required_ci.py from the default branch, and none checks out a pull request:

Job When What it does
Register of ci-command.yml an accepted require, retry or drop Reads the pull request's head and ledger, sets the head pending (not for drop), writes the record with the acknowledgment, then evaluates. A failure before the record is written records and acknowledges nothing; a failure to write it leaves the head pending.
Gate of ci-command.yml after Report recorded the result of a required selection's run Evaluates.
Evaluate of requested-ci-gate.yml every pull request run of Code checks, when it is requested; manual dispatch Evaluates.

Code checks starts for every opened, pushed, reopened or retargeted pull request of a maintained branch, documentation-only ones included, so each new head gets its status without a command. An evaluation covers every open pull request and sets a status only where it changed.

The three jobs share the concurrency group required-requested-ci, so no evaluation publishes from a ledger that another job is changing. GitHub keeps one waiting job per group. A waiting evaluation that a newer one replaces loses nothing; a waiting Register that is replaced has recorded nothing, and Report answers that the command is not confirmed.

A failed job does not fail a commit status: a commit keeps the status it has until another one is set. The publishing jobs therefore turn what they cannot establish into error and never leave a head as it was:

  • a pull request whose comments cannot be read after two repeated attempts, or cannot be evaluated, makes its head error (The requirements of #N could not be read), for every pull request that has that head. The others are published and the job fails;
  • a job whose script does not report every head as published, because it failed, was cancelled or timed out, or because the checkout before it failed, ends with a step that sets error (The evaluation did not finish) on the commit that started it: the head of the Code checks run for Evaluate, the commit of the command for Register and Gate. That step needs only the token and gh;
  • a summary comment is not written for a head whose status could not be set, so it never says more than the status.

error is cleared by the next evaluation that succeeds: any pull request event, any requirement command, or gh workflow run "Requested CI gate". A manual dispatch has no commit of its own, so a failed one withdraws nothing.

One case is outside what these jobs can do. When the job cannot reach the API to list the pull requests or to set a status, or never starts, every commit keeps the status it has. That matters for a commit that is already green: a pull request pushed to, or opened at, a commit that another pull request made green shows that green until an evaluation succeeds. The failed Requested CI gate or CI command run is the signal, and step 5 of the reviewer steps catches it: the summary comment of a pull request with requirements names the head it was written for, and a summary that names another commit, or that is missing although selections were required, means that the status is not this pull request's. Run the gate again before merging.

The jobs that compile and run the pull request (Validate and Scenarios, and every job of the pull request workflows) hold a read-only token and can write neither a status nor a comment. Pull requests from forks cannot be given requirements, and their heads get the success of a pull request without any. As for every check of this repository, someone with write access can change a workflow on a branch and so set any status; the rule that only GitHub Actions may provide the context excludes statuses set with a personal token, not that.

Enabling the branch rule

The rule is the ruleset Git Flow / requested CI, which requires the context from GitHub Actions on main, develop and every release and support line. Those are the branches whose pull requests start Code checks, and that request is what makes the gate publish the status on a new head. The ruleset is optional: scripts/release/git_flow.py creates it only when asked, beside the other rulesets and without changing them, and verify checks it only where it is installed. Until then the status is advisory and step 5 of the reviewer steps is the reviewer's.

A branch rule must require the context only when every pull request it covers gets it, otherwise that pull request waits for a status that no event will bring. Install it in this order:

  1. Confirm on a Dependabot pull request that Required requested CI appears on its head by itself. Its Code checks run is requested by another actor than a maintainer's, and the gate must publish for it as for any other pull request.

  2. Dispatch the gate and check that it succeeded: gh workflow run "Requested CI gate".

  3. Plan. The plan prints the ruleset and names every open pull request to a maintained branch whose head has no status yet:

    python3 scripts/release/git_flow.py plan --requested-ci
  4. Apply and verify. apply refuses while the plan still names a pull request, and after any drift of the existing rulesets:

    python3 scripts/release/git_flow.py apply --requested-ci
    python3 scripts/release/git_flow.py verify

After that, a pull request opened or pushed later gets the status from the Code checks request of that event.

How a batch runs

Each matrix entry is one job on one runner of the plan's target architecture: ubuntu-24.04 for amd64, the default, and ubuntu-24.04-arm for the arm64 cohort. The Plan job runs on the same architecture, because it warms the build cache that the jobs restore.

Step Phase in summary What happens
Restore the Go module and build caches n/a Restores what a Plan job saved. A build-cache miss only slows compilation.
Prepare Go modules Go module preparation tag_scenarios.py modules runs go mod download within 300 seconds.
Compile selected tagged suite Compilation tag_scenarios.py compile first refuses a runner whose Go toolchain is not natively the matrix entry's goarch (host and target), then runs go test -c within 180 seconds with GOPROXY=off, writes compile/<package>.test, and records in compile-report.json the binary's sha256, its build key (package, tags, Go environment, version, GOOS, GOARCH), the sha256 of every source file read, and the listed tests. It first refuses a recomputed profile whose tags differ from the matrix entry's.
Prepare runner storage Runner storage preparation Ceph batches only. runner_storage.py prepare skips the existing fixed-directory cleanup only when the exact workload, on its qualified release, roles and platform, and the actual storage layout have a qualification and sufficient headroom. Otherwise it removes the unused runner toolchains. Writes runner-storage/storage-prepare.json and, when reclamation ran, storage-free.json. Cleanup remains GitHub-hosted-only and never removes Docker resources.
Docker baseline and engine identity Docker baseline Records the engine identity and every resource labelled org.testcontainers=true.
Resolve native published Ceph roles Published image preparation Ceph batches only. Pulls the declared role images from ghcr.io/jsyoo5b/ceph-testcontainers-images by their official-<release>-<role> tags (see Role images of a run), for the entry's linux/<goarch>, refusing a Docker engine of another platform, records the image ID and the platform child that Docker selected from each index and exports the selected CEPH_TEST_*_IMAGE variables, which name the tags. A pull that fails on a registry transport error (TLS handshake or DNS timeout, reset, EOF, any 5xx answer to a manifest, blob or token request) is retried twice, after 10 and 30 seconds; a registry refusal (unknown manifest, denied, unauthorized, rate limit, any 4xx answer) is never retried. Never builds.
Prepare Ryuk and the test images Ryuk and test image preparation Every batch, Docker-only ones included. in_test_images.py pulls, with the same bounded retries, the Ryuk image of the go.mod version of testcontainers-go (its ReaperDefaultImage) and every image that the batch's compiled tests name besides their roles, so that no test pulls an image itself. batchselect -images finds those in the source, for the GOOS and GOARCH that the compile receipt records, and the step refuses a Docker engine of another platform: each image reference among the constants of the declarations that the tests reach (constants of other packages included), and a reference prefix that a test completes with the Ceph version at run time (the release's official image, pulled by tag). A published role tag that a test names itself, such as both releases of the rolling upgrade, is one of those constants and is pulled by its tag. With prepared role images, the harness fallback to a published role tag (ceph.DefaultImage) is skipped. Writes test-images/summary.json and test-images/pull-<name>[-<attempt>].log.
Record runner storage with the images n/a Ceph batches only. Writes runner-storage/storage-images.json. Its failure never changes the job result.
Verify post-image runner headroom Post-image runner headroom Ceph batches only. runner_storage.py ready refuses the job before the tests when the pulled images left too little headroom for the workload. Writes runner-storage/storage-ready.json.
Native assertions and completion proof Native assertions and completion tag_scenarios.py run (or optional) with --binary runs the compiled binary and applies the pass rules below.
Record runner storage after the tests n/a Ceph batches only, also after a failure. Writes runner-storage/storage-after.json.
Release the qualification pressure file n/a pressure storage policies only, also after a failure. Deletes the pressure file only when its device and inode match storage-pressure.json.
Verify owned Docker resource cleanup Docker resource cleanup Runs whenever the baseline succeeded, also after a failure. Waits up to 30 seconds for Ryuk, then fails if new labelled resources remain. Never deletes anything.
Show actual failure phases in PR check annotations and summary n/a Writes the phase table to the summary and phases.json, and annotates failed or cancelled phases.
Normalize the scenario history row n/a Writes history-row.json (see Scenario history). Its failure never changes the job result.
Save compile, native and phase results n/a Uploads tag-scenario-<id>-<release>.
Record the failure archive upload n/a After a native failure, writes the number of local archive manifests and the upload's outcome and artifact to the summary.
Read the scenario history summary n/a On failure, reads summary.json from the scenario-history branch for the next step; a missing branch leaves it out.
Identify completed failing tests n/a On failure, annotates each failed test or subtest found in native.log.

The job sets CGO_ENABLED=0, TESTCONTAINERS_HOST_OVERRIDE=127.0.0.1 and CEPH_TEST_PARALLEL=2, and uses the latest Go 1.26.x.

Reusing the compiled binary

run --binary <compile-directory> (and optional --binary) executes the binary that compile wrote. Before starting anything it requires a successful receipt for the same profile and tags, a matching binary sha256, a matching recomputed build key, unchanged source hashes, and a test inventory (recorded and from -test.list=.) equal to the batch's declared tests. A failed check never falls back to go test. The binary runs in the package directory with -test.v -test.count=1 and the batch's -test.timeout; a network-mode entry adds -test.run for its --tests. native/report.json records execution: test-binary, the sha256 and the build key.

Without --binary, run builds and runs with go test. To follow the CI path locally:

python3 .github/scripts/tag_scenarios.py compile --category code --batch pure --package ./internal/integration --directory artifacts/c/compile
python3 .github/scripts/tag_scenarios.py run --category code --batch pure --package ./internal/integration --directory artifacts/c/run --binary artifacts/c/compile

Image roles

Ceph batches declare their roles in //ci: metadata: essential, client, cephfs, rbd-aux and rgw. Ordinary RADOS and RBD use essential,client; CephFS adds cephfs, RBD mirroring and cache add rbd-aux, and gateways add rgw. An undeclared Ceph profile gets all five; non-Ceph profiles get none. Only declared roles are prepared, and a missing role is never substituted: ceph.Run, or the feature's own operation, refuses a missing role (harness.IntegrationImages itself only requires the essential image). The optional role_routing and role_mirror_refresh batches cover constrained routing and can be requested with /ci run.

Network modes

A test that can run on the Docker bridge and on the host network declares the modes with harness.NetworkModes, its default first, and runs one subtest per mode:

for _, mode := range harness.NetworkModes(t, "host", "bridge") {
	host, name := mode.Host, mode.Name
	t.Run(name, func(t *testing.T) { /* ceph.WithHostNetwork() when host */ })
}

The default is host for the RADOS products (ceph, rbd, cephfs), whose native clients must reach the advertised daemon addresses, and bridge for RGW. A test using a bridge-only feature (NetworkConfig.WithSeparateClusterNetwork, InterruptNetwork, ApplyNetworkProfile) declares bridge alone or keeps that part in the bridge subtest.

The planner reads the modes from the call, so they are string literals in the Test function itself, and the files of one batch declare the same modes. A matrix entry sets CEPH_TEST_NETWORK:

Scope Entries of a batch
Development PRs and pushes, CI command (networks: default) One, in the default mode, with all tests
Periodic and the release matrix (networks: all) The default entry, plus one per other mode that runs only the mode-declaring tests (--tests), named <batch> (<mode>)

A test without harness.NetworkModes runs once and ignores CEPH_TEST_NETWORK. Run locally with an undeclared mode, a test skips; with the variable unset, it runs every declared mode.

How a batch passes

The native phase succeeds only if native.log shows:

  • exit status 0;
  • exactly one === RUN and one --- PASS for every compiled top-level test, and no other top-level test;
  • exactly one --- PASS for every started subtest;
  • no --- FAIL, FAIL, panic: or --- SKIP line at any level;
  • exactly one PASS line after all results, followed by one ok <package> line under go test, or as the last line under --binary;
  • for checker batches, a passing strict checker.

A skipped test therefore fails CI. Tests assert release differences instead of skipping (see Writing the test body).

Runner time budgets

Job Limit
Code checks jobs 5 to 25 minutes, fixed in code.yml
Plan job of each runtime workflow 10 minutes
Each batch job job-timeout from the batch's //ci: line
The go test process of a batch timeout from the batch's //ci: line

The planner accepts a process timeout up to 6 hours and a job timeout of 1 to 360 minutes, and requires the job timeout to exceed the process timeout; batches leave 5 to 15 minutes for setup and cleanup. Run tag_scenarios.py plan to see every budget. Runners are GitHub-hosted ubuntu-24.04 machines with 16 GB of memory (ubuntu-24.04-arm for the arm64 cohort). CEPH_TEST_PARALLEL=2 lets two opted-in tests run at once in a batch; the rest run sequentially within the process timeout.

Runner budget

The account runs at most 40 hosted jobs at once, across all of its repositories: over three days with 4524 executed jobs in its active repositories, 40 jobs ran at once for 48 minutes in total and never more. tag_scenarios.py allocates them by what a run is for:

Run Allocation Reason
Release matrix Every runner except OTHER_REPOSITORY_RUNNERS It gates a release during a merge window, when no other work of this repository should be in flight.
Periodic At most PERIODIC_RUNNERS, and nothing while a Release matrix run is in progress It searches for flaky tests: slow is fine, taking the account is not.
Pull request No reserved share; each runtime caller is bounded by PULL_REQUEST_MAX_PARALLEL A pull request validates its own change, and several are open at once.
Constant Value Meaning
RUNNER_LIMIT 40 The measured limit of the account.
OTHER_REPOSITORY_RUNNERS 8 Runners that the Release matrix leaves to the account's other repositories.
RELEASE_CALLER_RUNNER_MINUTES short 325, topology 335, multicluster 195, recovery 230 Measured runner time of one release caller of each category.
ENVIRONMENT_MAX_PARALLEL 1 max-parallel of an environment caller.
PERIODIC_RUNNERS 8 Runners that every mode of Periodic holds together at most.
PERIODIC_CALLERS 4 Release callers of a full periodic matrix that run at once (max-parallel of the caller matrix).
PERIODIC_MAX_PARALLEL 1 max-parallel of each release caller in Periodic.
CAMPAIGN_MAX_PARALLEL 3 max-parallel of the representative campaign.
STORAGE_MAX_PARALLEL 5 max-parallel of a storage qualification dispatch.
PULL_REQUEST_MAX_PARALLEL short 8, topology 3, multicluster 2, recovery 2, environment 1 max-parallel of a pull request's runtime caller of each category.
PULL_REQUESTS_AT_ONCE 2 Pull requests that can each run every category beside the other repositories' runners.

Release matrix. release_max_parallel takes the share of 32 runners less the environment caller, splits it evenly between the supported releases, and splits a release's part between its four callers in proportion to RELEASE_CALLER_RUNNER_MINUTES, whole runners by largest remainder. With three releases that is short 3, topology 3, multicluster 2 and recovery 2 for each release: 31 runners with the environment caller. A new supported release changes the allocation without an edit here. The runner minutes come from a Release matrix run in which every job executed (506 jobs, about 3250 runner minutes): one release's callers took about 325 (short, 51 jobs), 335 (topology, 59), 195 (multicluster, 23) and 230 (recovery, 34) minutes. That run was bounded at short 3 and one job for each other caller, held at most 19 runners and took 5.86 hours, the serial time of one topology caller. With the allocation above the callers need about 108, 112, 98 and 115 minutes of runtime jobs, so a run is expected to take about two hours. The longest job was 19 minutes, so more runners would shorten it little.

The Release matrix leaves no runner for this repository's pull requests. A pull request during a merge window is either a mistake or the fix of a defect that already blocks the candidate; in the second case the candidate's run is cancelled (see Candidate failures), which frees its runners.

Periodic. The daily sample is one job. A full matrix runs PERIODIC_CALLERS callers at once with one runtime job each beside the environment caller, five runners; a storage qualification dispatch and the sample wait in the same concurrency group. The representative campaign has a group of its own and can run beside a full matrix, so the two add up to eight. Every run of Periodic first asks whether a Release matrix run is queued or in progress (periodic_gate.py --yield-to release-matrix.yml) and, if one is, starts no native work; the skipped run is not repeated. A Release matrix run that starts after Periodic did runs beside it.

Pull requests. The entry points pass their category's bound, so a pull request that runs every category holds at most 16 runners, and two such pull requests fit beside the other repositories' eight.

tag_scenarios.py releases --allocation release fails when the Release matrix callers could exceed RUNNER_LIMIT - OTHER_REPOSITORY_RUNNERS, and --allocation periodic fails when the modes of Periodic together could exceed PERIODIC_RUNNERS. A bound only delays jobs: every entry still runs with fail-fast: false. Batches are not packed into shared jobs.

The order of matrix entries is not a start order. GitHub creates a matrix's jobs in an order of its own, and while every runner of the account is busy, queued jobs start roughly in that creation order, so reordering the planner's entries does not move a long batch forward. A release caller is bound by its summed runner time: replaying measured durations onto a caller's slots longest-first shortens the makespan by under 1% and delays the mean completion by several minutes. A pull request run ends with its longest batch or, when its callers' bounds hold jobs back, with a caller's summed runner time over its bound; the batch layout sets its length, not the dispatch order (issue 388). A long batch whose tests are independent opts them into harness.ParallelWhenEnabled rather than adding a job.

To measure runner use, fill a cache from the GitHub API and report offline:

python3 .github/scripts/ci_measurements.py fetch --cache artifacts/ci-cache --repo <owner/repo> \
  --since <date> --marker <name>=<commit> [--cache-logs N]
python3 .github/scripts/ci_measurements.py report --cache artifacts/ci-cache --marker <name>=<commit>
python3 .github/scripts/ci_measurements.py cohorts --cache artifacts/ci-cache \
  --history <scenario-history checkout> --repo <owner/repo> --output cohorts.json
python3 .github/scripts/ci_measurements.py caches --cache artifacts/ci-cache --repo <owner/repo> --output caches.json

The runtime workflows were renamed (Ceph short became Short, and so on for the other six). report matches runs by the current names, so a run that started under an old name is not counted; choose --since after the rename.

report groups executions into cohorts by the newest marker commit their head contains and gives job counts, runner minutes, wall time, runner acquisition delay, phase times and simulated makespans. cohorts joins jobs to history rows by repository, run, attempt and job ID and groups them by configuration (see Durable history and classification); rows that disagree about one job are listed as conflicts, never merged. caches lists each cache restore, save, import or export with an outcome taken only from log lines (--cache-logs fetches them); a green step is never read as a hit or a write, and a missing log is unknown.

Periodic runs

Periodic primarily searches for flaky tests and patch-dependent differences. The daily 04:41 UTC firing runs one bounded native allocation. Tuesday and Friday use discovery; the other five days use rotate. The scheduled run uses the default branch independently of candidate validation. No mode starts native work while a Release matrix run is queued or in progress (Runner budget); that firing is skipped.

Mode Selection Invocation
rotate Random 2–3 supported releases and 2–3 native batches/network selections Five scheduled slots per week; manual default
discovery Random 2–3 historical/new patches and 2–3 compact parents, plus applicable directed characterization Two scheduled slots per week; manual
targeted Explicit source ref, catalogued patch, profile IDs and optional parent tests/network Manual failure comparison
full Every required profile/network on every supported release, plus environment checks Manual only
representative Source-owned representative batches with explicit repetitions Manual only; default one repetition
storage Existing runner-storage qualification Manual only

Frozen plan and bounded execution

Sample plan freezes the exact checkout SHA, the Ceph image references (the tags of the selected release), parent test names, tags, network modes and the random seed. It resolves no image: the worker pulls the tags, and Ryuk and the images the tests name themselves, when it runs. The run identity seeds the draw, so later runs can select a different combination even at the same source SHA. The plan is uploaded before Sequential sample starts. Every native case consumes that plan and the compiled binary's existing source/inventory receipt; an unknown test, expanded timeout or changed source/image selection is refused. The worker records the actual native build string and local image/platform IDs before tests.

The worker executes batches sequentially, with independent test-owned fixtures. There is no shared-cluster manager. Module/image preparation is amortized; verified binaries can be reused only for the same source profile in this run. Only inadequate disk admission permits the existing standard hosted cleanup; sufficient headroom causes no deletion. Every actual Docker storage filesystem must pass admission, including post-image and per-batch checks. A dirty cleanup stops further native admission; the executor never prunes other resources.

All selected versions share one 110-minute deadline; the executor reserves cleanup time, and does not start a case without its full bounded native window. Core selections have a ten-minute cap and extensions at most twenty minutes, each no greater than the source timeout. A case that times out, skips, panics or fails its existing completion/checker assertions remains failed or incomplete. The source timeout and complete compiled inventory of ordinary CI are unchanged.

Workflow limits include preparation and reporting: plan 15 minutes, worker 120 minutes (including upload margin), history 15 minutes; at most 150 summed runner-minutes and three allocations per scheduled execution. This is an upper bound, not a latency measurement. Runner queue/dependency delay is separate. Full and repeated manual investigations retain their existing larger bounds.

Core and extension inventory

The core reuses these existing parent tests in their declared default network:

  • TestClusterLifecycle: identity/readiness, RADOS bytes and topology/removal.
  • TestRBDLifecycle: native RBD operations and data.
  • TestCephFSFilesystem: native filesystem I/O and topology/removal.
  • TestRGWS3: real S3 operations through RGW.
  • TestClientIdentities: authentication/capability refusal and working controls.
  • TestCephFSOSDRecovery: OSD interruption/recovery and retained exact bytes.

rotate draws from required native batches; discovery draws from the six compact parents above. Each draw uses the same selected batches/parents across its 2–3 versions, with the declared network selection frozen before execution. Alternate network modes keep the existing network-test scope, and checker profiles retain their complete parent set. Environment/code checks remain in their existing workflows. There are no cost bins, exhaustive rotation ledger, or fixed coverage/revisit guarantees. A random sample can repeat or miss a version, batch or network across successive executions.

Selection, evidence and failure routing

A Go-process start checkpoint distinguishes an actual native attempt from a preparation failure or cancellation before execution. Native failures, timeouts, incomplete work, unavailable artifacts and ineligible environments remain separate outcomes. The first failure remains recorded; there is no automatic retry until green. Dirty cleanup stops the remaining versions too.

Frozen plans and packed receipts live under periodic/ on the existing scenario-history branch, keyed by run, attempt and release, with a separate bundle receipt retaining unstarted versions. They retain the source and build identities, the image references with what the engine reported for them, the selected tests and the preparation/native/cleanup outcomes. They are not ingested as full-profile passes by the ordinary same-configuration classifier. Every execution uses its selected source; an old SHA is not held while waiting for coverage. A sample is never proof that current main passed every version.

Historical discovery uses a separate inventory in .github/ceph-discovery-images.json: official aggregate images for Squid 19.2.0–19.2.7, Tentacle 20.2.0–20.2.5 and Umbrella 21.2.0, each named by its release tag. The 20.2.5 row is the upstream aggregate (quay.io/ceph/ceph:v20.2.5) until a role aggregate is published; other patches use the published official-<release>-all tag, and the inventory refuses a row whose reference is anything else. Each cell keeps one release for every role/client. Only linux/amd64 executes by default; arm64 manifest availability is not native arm64 validation. Umbrella's official amd64 images require x86-64-v3; a failed CPU/platform preflight is an environment result, not a Ceph failure. Missing artifacts and inapplicable extra-image contracts remain preparation/eligibility outcomes rather than native passes.

Discovery does not add historical patches to SUPPORTED_RELEASES or change support policy. Add a newly published patch to the discovery inventory to make it eligible for random selection; use targeted for an immediate check. The discovery_schedule characterization on 19.2.2/19.2.3 (rbd_schedule_remove) records malformed schedule removal with empty and populated schedules for both rbd_support schedule handlers, trash purge and mirror snapshot, which share one removal path. It asserts the refusal with a schedule present and preserves native exit codes and unchanged-state controls in the durable receipt: characterizing an old success on invalid input is not claiming that behavior is correct. Discovery also samples the existing RGW multisite bootstrap contract on 19.2.5/19.2.6/19.2.7, preserving the documented 19.2.6 failure as a control.

Investigate meaningful differences with scoped issues. Preserve the first failure, then compare the same test at the recorded source with an adjacent patch or control. No automatic full-matrix expansion, retry-until-green, support-policy change or compatibility workaround follows a discovery result.

# Profile IDs come from the plan or the source-owned vocabulary.
gh workflow run "Periodic" -f mode=targeted -f ref=<original-sha> \
  -f patch=19.2.2 -f profiles=<exact-profile-id> -f tests=<parent-test> -f network=host
gh workflow run "Periodic" -f mode=targeted -f ref=<same-original-sha> \
  -f patch=19.2.3 -f profiles=<same-profile-id> -f tests=<same-parent-test> -f network=host
# Complete release work is explicit:
gh workflow run "Periodic" -f mode=full

The requested source must contain these planning/execution tools. Local periodic_execute.py execute --trial records working-tree provenance rather than claiming a verified Git checkout. Measure downloaded packed receipts with the existing tooling:

python3 .github/scripts/ci_measurements.py periodic --receipts <artifact-directory> --output <report.json>

This reports observed preparation/compile/native times per version; shared runner allocation and queue timing remain unknown in these receipts.

Supported releases

SUPPORTED_RELEASES in .github/scripts/run_image_matrix.py lists each supported release with its codename and official image tag (Tentacle 20.2.4, Umbrella 21.2.0 and Squid 19.2.5). The manual-full periodic and candidate Releases jobs generate the callers:

python3 .github/scripts/tag_scenarios.py releases --scope all --allocation periodic --check-prefix 'Periodic'
python3 .github/scripts/tag_scenarios.py releases --scope all --allocation release --check-prefix 'Release matrix'

Each release and category becomes a <Codename> <category> call of tagged-runtime.yml; environment needs no release and stays a separate job. The command fails unless every required profile runs on every release of the scope exactly once, and test_scenario_shards.py checks the same offline.

To add a release, add its row once ceph-testcontainers-images publishes its role images under the <variant>-<release>-<role> tags, add it to the release options of periodic.yml and the public-api matrix of block-fault-probe.yml, which cannot read the table (test_scenario_shards.py fails until they match), run the CI tooling tests, and for a new major release add it to every release-limited row of the test expectation table (see Release differences). A release that is not a row is refused. The default release (RELEASE, the release defaults of tagged-runtime.yml, periodic.yml and the scenario-images action, and ceph.DefaultImage) changes only with the current release.

Role images of a run

CI is a consumer of the published role images like any other: it selects them by tag, and it keeps no digest of them (Image references). A compatible rebuild that ceph-testcontainers-images publishes under the same tag needs no change here. The next job that pulls the tag uses it.

  • image_references in .github/scripts/run_image_matrix.py names the images: ghcr.io/jsyoo5b/ceph-testcontainers-images:<variant>-<release>-<role>, with all for the aggregate. ceph.DefaultImage is official-20.2.4-all.
  • Each runtime job pulls the tags of the roles that its batch declares (image_roles) and runs the tests by those tags. A role that no selected batch uses, such as the optional mgr-modules when its tag is unavailable, is never asked of the registry and cannot stop the run, and a Docker-only batch asks nothing.
  • Nothing resolves a tag to a digest before the jobs start, and no workflow input carries one. A tag republished while a run is in progress therefore reaches the jobs that pull it afterwards, a job that is run again later included. The jobs of one run can so differ in what they ran, and each says what that was.
  • Each job records what it ran: scenario-images-*/summary.json has, for every role, the tag it prepared and what the engine reported for it (the local image ID, any registry digests and the platform child). test-images/summary.json has the same for Ryuk and for every image a test names itself. These are observations of that job. A history row takes the image IDs from them, so attempts on different builds of one tag count as different configurations (see Durable history and classification); nothing else reads them, and an image without a registry digest (a tag pulled through a mirror, a local build) is recorded with an empty list and runs like any other.
  • A periodic sample (rotate, targeted) references the role tags that its selected entries declare, with essential always among them, pulls each once, in its single job, and records the same observations in its images/summary.json.
  • The module takes no part in this: it resolves no tag, enforces no freshness and reports no identity.

Release candidate validation

The release matrix runs for each PR targeting main, release/* or support/* and for a push to main or support/*; a push to main that is the merge of a validated promotion PR reuses that PR's run. A push to release/* runs none: a release line follows develop, and its candidate is validated once, by its promotion PR to main, instead of a second time on the same tree. It runs the environment profile and the required release-bound profiles on every supported Ceph release. Neither documentation-only selection nor an ordinary development PR changes this matrix. The target-specific check (for example, Release candidate validated (release/1.2)) gives each branch rule a single result for its own complete matrix. A manual dispatch emits a different check name. The six Code checks also apply to each candidate.

For publication, verify the push event, protected branch, candidate SHA, workflow identity and successful required jobs. A PR test-merge run qualifies that proposed merge. Tag the protected commit only after its own validation succeeds, by its own matrix or by a verified promotion reuse. See the cutover and branch rules.

Promotion reuse

A promotion PR to main runs the full matrix on GitHub's test merge. The protected merge that follows usually has the same tree and the same two parents, and a second matrix on its push would test the same content again with hundreds of native jobs. The push run therefore asks first whether the promotion run already qualifies the pushed commit:

  1. Releases plans the matrix as always and writes its coverage: the required cells, the image references, and one digest of both.
  2. On a promotion PR to main, once Release candidate validated (main) has passed, the Candidate receipt job writes the artifact release-candidate-receipt-<attempt> with two files. The receipt names the commit that the workflow checked out with its tree and parents, the PR, its head and target, the run and attempt, the coverage, the image references and the jobs whose first failure a release retry passed. The evidence file holds the history row of every runtime job (what it checked out, built and pulled, the image IDs and digests it resolved, each phase's result, its cleanup check and the hashes of its logs and reports) and the report of each retry. The jobs' own artifacts expire after 7 to 14 days; the receipt and this evidence are kept for 90. The job fetches the rows through the API, page by page, because a release run writes more artifacts than actions/download-artifact lists, and it fails when the record does not account for every runtime job or the checked-out commit is not a merge of two parents. The gate's result stays as it is; the failed job says before the merge that the push will run the full matrix.
  3. On a push to main, Promotion reuse runs scripts/release/promotion_reuse.py verify and writes promotion-reuse-receipt-<attempt> with its decision and, on a reuse, its own copy of the evidence file.

The push reuses the promotion run only when all of this holds:

Check Source
The pushed commit is a merge of two parents, the merge commit of exactly one merged PR that targeted this branch from a branch of this repository, and its second parent is that PR's head. Repository and PR API
A pull request run of release-matrix.yml for that head, of that head branch and repository, passed in its latest attempt, with the target's Release candidate validated job and every other job successful. Actions API
That attempt's candidate receipt exists, is not expired, and names the same run, attempt, PR, target and head. Artifact
The receipt's tested commit, read from the repository, has the tree and parents that the receipt records, and they equal the pushed merge's tree and parents. Repository API
The receipt's coverage digest equals the digest that Releases computed for the pushed tree. This run
The evidence file is the one that the receipt records, and it has exactly one unaltered history row for every runtime job of the run. Each row is of this run and the tested commit, has complete records and a passed cleanup check, names the images it ran if it needed Ceph, and either passed or failed only its native phase and passed its one retry, by the retry's own report and the job's retry step. Artifact and Actions API

A live receipt therefore never qualifies a run by itself. Without the evidence file, or with an entry that is missing, foreign, altered, failed or uncleaned, the full matrix runs. The run's shorter-lived artifacts are not read, so their expiry changes nothing.

The run's reported head alone proves nothing about what was tested; the receipt's tested commit, checked against the repository, does. The receipt is trusted because the workflow that wrote it came from the merged PR's head merged into the protected branch. A failed retry, a cancelled or incomplete run and a skipped job never qualify.

When the push reuses the run, the runtime and environment callers are skipped and Release candidate validated (main) passes on the reuse receipt. In every other case, including an unreadable fact, an expired receipt or a failure of the Promotion reuse job itself, the full matrix runs and the gate needs it to pass; the reuse receipt then lists the reasons. On a reuse the receipt also summarizes what the jobs ran: every image reference with the image IDs and digests it resolved to, and the Go toolchains. The full native logs are not kept beyond their own artifacts' retention; the evidence holds their hashes.

Rehearsing a promotion

The reuse path only runs for a protected merge, so it cannot be tried on a feature branch, and trying it first on a real promotion would make a release the test. The branch rehearsal/promotion exists for that: release-matrix.yml treats a PR to it and a push to it as it treats main, with the gate Release candidate validated (rehearsal/promotion), and promotion_reuse.py applies the same checks. A receipt names its target, so a rehearsal never qualifies a push to main, nor the reverse. Nothing is tagged or published from the branch, and no other workflow runs for it.

A rehearsal tries the candidate record and the reuse decision, not the coverage of a release, so its matrix is small: tag_scenarios.py releases --scope rehearsal plans one caller, REHEARSAL_CATEGORY on the current release, beside the environment caller. A run takes minutes instead of hours. What only the size of a release run shows (its number of artifacts, its runner demand) is not rehearsed: for the runner demand dispatch Release matrix by hand, and check the record of a large run by recording it again from a depth-1 clone of its tested commit with promotion_reuse.py record.

The branch is not kept. To rehearse a change of the receipts or the reuse decision:

  1. Create rehearsal/promotion at the base to promote onto, with a ruleset for it alone that requires a pull request and forbids deletion and non-fast-forward pushes.
  2. Push the commit under test to a branch of its own and open a PR from it to rehearsal/promotion, before that commit is the head of any other open PR. Two open PRs with one head commit make the checks that publish to the PR of a commit (the structural review comment) refuse. When the gate has passed, check that Candidate receipt succeeded and wrote release-candidate-receipt-<attempt>.
  3. Merge the PR with a merge commit. The push run must end with Promotion reuse successful, the runtime and environment callers skipped and the gate passed, and its promotion-reuse-receipt-<attempt> must say reused and name the PR's run.
  4. For the other path, merge a second PR whose Release matrix has not passed (cancel it first). The push run's receipt must say full-matrix with the reason, and the callers must start; cancel that run once they have.
  5. Delete the ruleset and the branch.

A push to support/* always runs its full matrix. The six Code checks run on the pushed commit in either case.

Release retry

A release candidate is judged by more than five hundred native jobs, and the validation passes only when every one of them does. A rare intermittent failure in any of them would block every candidate. Each job of a release matrix (a caller with require_profiles) therefore runs the parent tests that failed once more, in the same job and on the same runner, and the job's result is that second run's:

  • tag_scenarios.py retry reads the first run's native/report.json and native.log. It retries only a failure of named tests (native-test-failure or native-registry-transport) whose process ended as Go ends failed tests: exit status 1 and a final FAIL. It refuses one after a build or process failure, a skipped test, a panic, a fatal runtime error or a signal (any of which can end the binary or crash it beyond its failed tests), or a parent that neither failed nor passed. A checker profile retries its whole run, because its checker validates one complete run.
  • Before it starts it waits up to 120 s for the first run's testcontainers containers, Ryuk aside, to be gone.
  • Its timeout is the profile's process timeout, bounded by the second budget that the job keeps: a release job's timeout-minutes is retry_job_timeout, twice the source's job-timeout up to 360 minutes, and the retry uses what lies beyond job-timeout, less five minutes for the steps after it.
  • The retry writes native-retry/report.json and its run under native-retry/attempt/. The first run's receipts stay in native/, the history row records the first failure, the native phase annotation becomes a warning, the failure reports still run, and the artifact is kept for 14 days.
  • A refused or failed retry fails the job. The owned-resource cleanup check runs after the retry and covers both runs.

A first failure that its retry passed is still a failure of that test. It is recorded in an issue like any other (see Releasing), and it does not block the candidate. Development workflows, Periodic and the CI command never retry.

Representative campaign

The campaign repeats a source-owned set of batches on one pinned commit to build same-configuration pass and fail records. The set is every batch whose //ci: line declares representative=<theme> (see Testing); plan --representative fails if a theme (topology, auth, mirroring, recovery, flaky) has no batch.

It runs only through an explicit dispatch with mode representative:

  1. Pin resolves the commit (the ref input or the run's commit).
  2. Representative calls the template with category: representative. Each entry repeats repetitions times (default 1, at most 10) as separate jobs named <batch> #<n>, artifacts ending in -r<n>, each checking out the pinned commit and pulling the role tags of the release. History rows carry repetition_id <n> and the image IDs each attempt ran, so a tag rebuilt during a campaign splits its attempts into two configurations instead of mixing them.
  3. Every attempt keeps the normal completion, no-skip and cleanup proofs; nothing is retried.

A campaign is bounded at 100 jobs and 3000 summed job-timeout minutes, does not count as a matrix execution for the gate, and has its own concurrency group.

gh workflow run "Periodic" -f mode=representative -f repetitions=5 -f release=19.2.5
gh workflow run "Periodic" -f mode=representative -f ref=<commit>

ref must contain the campaign tooling, because every job runs the pinned commit's scripts.

Native ARM64 cohort

The ARM64 workflow runs a small source-owned cohort on a GitHub-hosted native ARM64 runner (ubuntu-24.04-arm) with the linux/arm64 children of the published role images. A batch joins it with architecture=arm64 on its //ci: line (see Testing). The cohort is default and network_faults of ./internal/integration: cluster startup and bootstrap failure cleanup, the Ceph tools in the role containers, pool and placement observation, and the netem helper image. The cgroup, device-mapper and clock-interposition recipes stay out of it until their host prerequisites are established on that runner.

python3 .github/scripts/tag_scenarios.py plan --architecture-cohort --goarch arm64 --output plan.json
gh workflow run "ARM64" [-f ref=<commit>]

The workflow calls the template with category: architecture and goarch: arm64. The cohort plan claims only its batches (coverage_scope architecture); every category workflow still runs amd64, the default, and none of them depends on the ARM64 workflow, so amd64 acceptance stays attributable on its own. plan --goarch arm64 without --architecture-cohort is refused, so no caller can run an ordinary category on ARM64 by accident. The ARM64 workflow runs weekly, on a pull request that changes the planner, image or storage preparation, the runtime template, the bootstrap scripts or the netem helper, and manually. It is not a required check, and its rows are not written to the scenario history.

The architectures are recorded separately, each by the phase that observes it:

Fact Receipt
Planned architecture tag-plan.json (goos, goarch) and each matrix entry's goarch
Go toolchain host and target compile-report.json platform (planned, host, target) and the binary's build key
Docker engine runtime-cleanup engine identity and scenario-images/summary.json docker
Selected image each image's selected_platform (the child Docker chose) beside its reference and image_id in scenario-images/summary.json and test-images/summary.json

Compilation refuses a runner whose Go toolchain is not natively the planned architecture, host and target alike, so a cross-compiled binary is never built. Image preparation and test image preparation refuse an engine of another platform than the plan's and the binary's: emulation is separate evidence, not qualification.

Runner storage on ARM64 has no measured qualification: runner_storage.QUALIFIED_STORAGE_PLATFORMS holds linux/amd64 only. On another engine the layout is unverified (docker-platform-not-qualified), the job keeps the standard cleanup and the conservative minimum headroom guard, and a cohort budget of .github/runner-storage.json measured on another platform is refused (budget_refused: engine-platform-differs) instead of granting its allowance. The storage-*.json receipts of each run record the ARM64 runner's actual filesystems and layout reasons; a skip shortcut or a pressure policy on ARM64 needs a linux/arm64 budget measured from them.

Read a run by its phases, not its colour. An unavailable runner leaves the job queued or failed before any step; a compile, storage or image refusal is an environment result; a native failure, timeout or --- SKIP fails the native phase as on amd64; cleanup is checked on the same runner. A skipped or cancelled job is not qualification, and a pass qualifies only the cohort's batches on the recorded release, images and runner.

Native regressions

Optional (ci_optional) batches are outside every required check. Run one with tag_scenarios.py optional --batch <name>, with /ci run <name>, or through Native regressions, which runs the one batch its batch input names and takes only its pass or failure. The result of a known-defect reproducer is checked by Experiments (see Experiment case tiers).

The options of the workflow's batch input are the batches it accepts, and the source decides what each one runs. To list them, and to find the case manual of one:

# The choices of the dispatch, failure_archive included.
sed -n '/^      batch:/,/^      rgw_image:/p' .github/workflows/native-regressions.yml
# The test files, and with them the case directory and its README.md, of one batch.
grep -lw ci_batch_rgw_empty_sse_header internal/integration/experiments/*/*_test.go
# Dispatch one.
gh workflow run "Native regressions" --ref main -f batch=rgw_empty_sse_header

Most choices are experiment cases: each case's README.md gives its trigger, controls, command and the meaning of each outcome. A batch outside internal/integration/experiments is described with its feature below. A new optional batch joins the dispatch when it is added to the batch options; test_scenario_shards.py keeps the expected list and checks that every choice is an optional batch of the source.

native_rgw_translation also tests a patched RGW image; an empty rgw_image uses the published role image, and the workflow never builds one:

gh workflow run "Native regressions" --ref main -f batch=native_rgw_translation \
  -f rgw_image=ghcr.io/example/ceph-rgw:patched

cephfs_fscrypt (TestCephFSFscrypt) is another optional experiment case, cephfs_fscrypt, that passes on the published images. Its client containers mount CephFS with the engine's kernel and with ceph-fuse and get CAP_SYS_ADMIN and apparmor=unconfined; the kernel subtest also needs a host kernel that has the ceph filesystem type registered. It refuses, naming the missing requirement, and never loads a module or skips. Outside the tag runner, build it with go test -tags all,cephfs_fscrypt.

The following optional batches are not experiment cases. They are a selection: the source declares every optional batch (see Optional tests), and the timing and fault batches are described with their features, for example in Daemon timing and the fault/blockio README.

cephfs_fuse (TestCephFSFuseMount) mounts CephFS with ceph-fuse in client containers given /dev/fuse, CAP_SYS_ADMIN and apparmor=unconfined. It needs a rootful engine whose kernel has FUSE: a Linux Docker Engine, or Docker Desktop (it passed with kernel 7.0.14-linuxkit). On an engine that refuses the mount it fails naming the missing requirement and never skips. Outside the tag runner, build it with go test -tags all,cephfs_fuse.

rolling_upgrade (TestRollingUpgradeSquidToTentacle) upgrades a 19.2.5 cluster to 20.2.4 one daemon at a time with the published official role images of both releases, which it names by tag; see Rolling upgrade from Squid to Tentacle. It pulls the images of both releases whatever release the profile names.

desktop_capabilities (TestDesktopRBDOverNBD) maps an RBD image through rbd-nbd in a privileged helper from the official Ceph image and checks the bytes through the device and through librbd. It skips when the engine kernel has no loaded NBD driver or no /dev/nbd nodes, and never loads a module. It has been run on Docker Desktop, whose kernel has NBD built in.

zone_boundary (TestZoneBoundaryRegionalCluster, TestZoneBoundaryTwoSubnetRouting) splits one cluster into two logical availability zones behind caller-owned routers on its public and cluster bridges, qualifies the boundary's round trips with netem, partitions it and reads the data back after recovery, and records whether the engine forwards between two Docker bridges through a container. Its helpers and routers run with NET_ADMIN and namespaced sysctls. It is the research proof behind ceph.WithZoneConfig, not the way a test builds zones; see Hand-built routers for what it does and Regional zones for the public fixture.

osd_encrypted_store (TestEncryptedBlockDeviceOSD, TestEncryptedBlockStoreKeepsACollidingMapping) runs one BlueStore OSD on a linear device-mapper node over a test-owned LUKS2 dm-crypt mapping, locks and unlocks the store with a key only the test holds, and requires the original OSD, store and bytes after the unlock. A privileged keeper in the client role image binds the engine host's /dev; its cleanup removes a mapping or loop only while it still has the DM UUID, device number or backing inode the test recorded, and keeps a mapping that only shares the test's name. The engine host's kernel must have the device-mapper crypt target registered; the test fails naming it and never loads a module, except on a GitHub-hosted runner. Outside the tag runner, build it with go test -tags all,osd_encrypted_store (see Testing).

failure_archive is described in Controlled failed-native download proof.

Experiment case tiers

An experiment case runs where its cost and determinism allow, through the existing triggers only. The tiers are listed in Experiment cases. Each case's README.md records its tier, its batch and budget (the //ci: timeout), its measured runs and the triggers that apply to it, so adding a case changes nothing here.

  • A cheap deterministic case is a required batch of a non-short category, so a pull request runs it only when it changes a file of the case's directory (see Pull request scope), and periodic runs keep it covered on the default branch. A change to a public package or the harness selects none of these batches; the fixture behaviour they use is covered by the ordinary batches of its package (rbd_schedule_remove calls no fixture API beyond starting the cluster).
  • A known-defect reproducer prints REPRODUCED or NOT_REPRODUCED and passes either way; only its given and when steps fail it. An expensive one stays an optional native_regression batch that runs only on request. Experiments runs one case with go test: on the case's default image every result line must be REPRODUCED. When a fixed image is given, the case then runs again with it injected and every result the default run reproduced must be printed again, by the same name, as NOT_REPRODUCED. The default run stays as the control, so a trigger that stopped eliciting the defect on every image fails the dispatch, and so does a fixed run that never evaluates the reproduced defect. A result only the fixed run reaches (rbd_mirror_demote_restart/resumed-bytes, evaluated once the copy resumes) is printed and not judged. A case whose defect has no fixed release has only the default run. The cluster and every client a case starts run its one image, whatever role images the batch environment names. Native regressions and /ci run <batch> on <release> still run the batch, but take only its pass or failure.
  • An expensive case that passes on the published images would be scheduled through the periodic discovery selection (periodic_selection.py), not through pull requests. No current case is in that tier.

Reading a failing batch

  1. Find the batch. The check name gives the category and batch, for example Recovery / Scenarios / container_pause.

  2. Read the annotations. A phase annotation names the failed step and the profile ID; Go test result annotations list each failed test as <profile id>: TestName/subtest, or say that none could be identified (for example after a kill).

    gh run view <run-id> --log-failed
    gh api 'repos/{owner}/{repo}/check-runs/<job-id>/annotations'
  3. Check the job summary. It shows each phase as success, failure, cancelled, skipped, not run or not required. A phase shows where execution stopped, not the root cause.

  4. Download the artifacts.

    gh run download <run-id> -n tag-scenario-<profile-id>-<release> -D artifacts/ci-<run-id>

<id> is the matrix entry's profile ID, <release> the Ceph release such as 20.2.4, and a repeated campaign entry adds -r<n>.

Artifact Contents
tag-scenario-<id>-<release> modules/ and compile/ logs and reports (the binary is not uploaded); native/native.log; native/report.json; native/test-artifacts/ (retained test artifacts such as failure archives); test-images/ (the Ryuk and test image receipt: the analysis, each image with the tests and source positions that named it, skipped references, pull attempts); runner-storage/ (see below); phases.json; history-row.json
runtime-cleanup-<id>-<release> before.json and after.json: engine identity and labelled resources before and after
runtime-cleanup-<id>-<release>-baseline Only when the baseline itself failed
scenario-images-<id>-<release> summary.json with each role's tag and what the engine reported for it (image ID, any registry digests, platform), every pull attempt under pulls, and the pull-<role>[-<attempt>].log files
plan-<category>-<release> tag-plan.json (selected profiles) and tag-dispatch.json (the expanded matrix with network scope, repetitions, run, attempt, release and category)
code-tag-coverage tag-plan.json with compiled coverage and the code batch results

Uploads are retained for 7 days, or 14 days when the job (or the uploading composite action) has already failed at upload time.

native/report.json:

  • classification: passed; native-test-failure (a test failed, panicked or hit the Go timeout); native-registry-transport (every failed test failed on a Docker daemon registry transport error in its own output, such as testcontainers-go pulling Ryuk or a test's own image, so no Ceph assertion of it ran; scenario history counts it as an image failure); incomplete-skipped-fixture (a skip); process-or-build-failure (non-zero exit without a failure line); incomplete-native-proof (exit 0 but a pass rule or strict checker failed);
  • completion.errors, completion.failed_tests, completion.skipped_tests;
  • registry_transport: each failed test with such an error, and its stage (reaper or image), reason, registry phase (resolve, token, manifest, blob, registry for another /v2/ request) and endpoint without query or user information; phase and endpoint are unknown for a 5xx answer that the daemon reports without its request;
  • compiled_tests, command, elapsed_seconds, exit_code, strict_completion;
  • test_artifacts: files left in native/test-artifacts/ with sizes and the number of failure archive manifests.

modules-report.json and compile-report.json record exit_code, timed_out, timeout_seconds, elapsed_seconds and a classification of prepared/compiled, modules-failure/compile-failure, modules-timeout/compile-timeout, or compile-receipt-failure (the build succeeded but the receipt was not recorded, so the binary is not reused). A timed-out command is killed with its process group and has exit_code: null.

runner-storage/storage-prepare.json records the observed Docker storage roots, their independent filesystems, required and available bytes/inodes, the skip/reclaim decision, cleanup duration and post-check. Paths on the same device share one observation; space on different filesystems is never added. Every candidate store filesystem must retain the default MON 30% available share after the preparation and runtime allowances. Containerd writable snapshots can hold the MON store on a different device from Docker's metadata root, so the guard conservatively keeps this floor on every candidate device. The receipt's docker_root marker identifies the metadata filesystem for the controlled pressure file; it does not locate the MON store. storage-ready.json repeats the check after actual image pulls, retaining the runtime allowance. Failure of either guard stops the job before native assertions. A qualification names no image digest: the role images are the release's published tags, the image-preparation allowance bounds what their pull may take before it, and this post-image check is what refuses a job whose images, for example after a republished tag grew, left less than the runtime allowance.

The qualifications in .github/runner-storage.json cover these Tentacle AMD64 profiles. Each has an 8GiB image-preparation allowance and 100,000 free inodes, additional to the per-filesystem MON share and its runtime allowance.

Profile Maximum simultaneously live logical OSD backing Runtime allowance
rados_fixtures_watchers Two 1GiB OSDs 4GiB
cephfs_fixtures_data Two parallel fixtures with up to three 1GiB OSDs each 8GiB
cephfs_fixtures_admin One sequential fixture with up to two 1GiB OSDs 8GiB
rbd_fixtures_mirror Two clusters with two 1GiB OSDs each 8GiB
RBD default Two parallel fixtures: lifecycle with up to three 1GiB OSDs and namespaces with two 1GiB OSDs 8GiB

The data profile retains the three OSDs required by its erasure-coded pool assertions; its host-network topology still adds and removes an OSD. Qualified startup paths retain 1GiB stores and skip calibration. The RBD default profile pins its startup and topology helpers, including the default OSD size and count; its lifecycle assertions still add a third OSD. Logical capacity is a conservative source bound, not reserved physical space or a measured runtime peak. Control stores, client data and logs are additional; the qualification evidence records sampled filesystem consumption separately.

The catalog selects exactly one entry for the compiled profile. Its workload sources, listed startup/data helpers, dependency manifests, tags, selected tests, release, declared roles and platform must match. Changes to a relevant helper expire that workload's qualification. Unrelated profiles cannot inherit its allowance, and an ambiguous or missing entry retains conservative cleanup. The data and RBD default qualifications also require CEPH_TEST_PARALLEL to be at most two; higher or invalid values retain conservative preparation instead of inheriting their two-fixture bounds. Other profiles and releases remain unqualified. Image and native comparison evidence belongs in the implementation pull request.

Only verified local classic-overlay2 or socket-linked containerd-overlayfs layouts may skip. Existing local volume mounts and nested mounts are inspected. Containerd must have one running process whose CLI/configuration root and socket match Docker's reported socket. The built-in defaults are not inferred, so a package-default containerd whose root and grpc.address are left unset always retains cleanup; imported/proxy configuration and snapshotter root overrides keep conservative preparation. Unlinked roots are recorded as candidates and cannot authorize skipping. Remote endpoints, unknown roots, volume plugins/options and missing evidence keep conservative cleanup. The post-cleanup guard for an unqualified workload is explicitly a minimum safety check, not a claim to know its peak.

storage-free.json records the ordinary fixed allowlist removal when needed: Android, .NET, Haskell, CodeQL and Boost. Only if its post-cleanup headroom still fails does a second fixed list remove unused Swift, PowerShell and vcpkg installations. storage-free-supplemental.json records that additional action; storage-prepare.json retains the standard-cleanup observation and both action durations. Qualified sufficient headroom skips all removal; sufficient standard cleanup skips supplemental removal. Go, the system Python interpreter, runner files and Docker/containerd data are retained. A smaller hosted filesystem uses the same MON reserve and workload allowances: it must recover enough actual space from these directories or fail admission. Neither the runner label nor successful directory removal guarantees sufficient space; the post-cleanup and post-image checks still decide from each filesystem's observed bytes and inodes. storage-images.json and storage-after.json retain the Docker-root filesystem and docker system df observations after pulls and tests. These endpoints separate image and run allocation from runner baseline; they do not establish a transient peak or reclaim historical preparation time.

For a controlled hosted comparison, dispatch Periodic with mode=storage and the same commit ref, release and storage_policy=auto, always, pressure or pressure-supplemental. The storage_profiles input selects comma-separated source-owned profile IDs, defaulting to the watcher profile. Up to five selected jobs run concurrently, retaining ordinary assertions and cleanup checks; repeat dispatches for each policy on the same source and selection. Pressure refuses an unqualified profile before allocating a file. always is the reclamation control. pressure creates one exclusive owned file on the Docker filesystem to cross the admission boundary, retaining the MON floor, runtime reserve and 2GiB extra; it refuses allocations outside 1–40GiB or any non-hosted/unqualified layout. The file stays through native assertions and an always-run step removes it only when its device/inode receipt matches. The pressure file is also excluded from artifact uploads. pressure-supplemental first performs standard cleanup, then holds a pressure file up to 64GiB to leave a 4GiB admission shortfall while retaining the same MON floor, runtime reserve and 2GiB extra. Standard cleanup cannot recover the same SDKs twice, so the extra list must supply the missing headroom. If standard cleanup already leaves a shortfall, no pressure file is allocated. Its ordinary, supplemental and pressure receipts remain separate. This mode is an explicit hosted qualification control; normal auto jobs never allocate pressure files or preclean before their admission decision. This qualification never fills a shared host or intentionally approaches ENOSPC. For an end-to-end comparison, dispatch Short at one fixed commit with storage_policy=auto or always and the same ref. It executes the complete short selection with the usual network modes and concurrency, including unqualified workloads. Use its measured start/end times and final job to assess workflow completion, rather than summing avoided preparation seconds. The manual control cannot change the policy or checkout of push/PR events. Normal schedules, workload coverage and runner concurrency limits are unchanged. A storage dispatch of Periodic shares the matrix's concurrency group (see Concurrency and cancellation).

Typical causes by phase

Failed phase Look at Common causes
Go module preparation modules/modules.log, modules/modules-report.json Proxy and origin both failed, or go.sum lacks an entry.
Compilation compile/compile.log, compile/compile-report.json A test calls a helper that only another batch compiles; run plan --verify (see Testing).
Docker baseline the step log, runtime-cleanup-<id>-<release>-baseline Docker was not ready within 60 seconds.
Published image preparation scenario-images-<id>-<release> A role image is missing from the registry or could not be pulled. classification: registry-transport and registry_failure (image, reference, reason, phase, endpoint, attempts) mean the registry was not reached after the bounded retries; nothing about the image is established.
Ryuk and test image preparation test-images/summary.json The same for Ryuk or a test's own image. An error that a test completes an image prefix this preparation cannot name means a test builds an image reference that the analysis does not know; give it a constant or extend plan_images.
Native assertions and completion native/native.log, native/report.json An assertion failed, a Ceph wait or the batch timeout expired, or a test skipped. A panic: test timed out block lists the tests still running. A health wait that sees only MON_DISK_LOW points at runner storage: read runner-storage/. native-registry-transport points at a registry that a test's own pull could not reach: testcontainers-go does not retry the daemon's HTTP 500 for it, and no Ceph assertion of that test ran.
Docker resource cleanup runtime-cleanup-<id>-<release>/after.json A test left a container, network or volume behind, for example by not registering cleanup when Run returned an error.

Failed integration tests print the last 16 KB (8 KB in the multicluster and host-network helpers) of every daemon's log into native.log before the cluster is removed; search for container ... logs:.

Dispatch join

dispatch_join.py joins a downloaded tag-dispatch.json offline to a saved jobs-API response of the same run and attempt, optionally with history-row.json receipts:

python3 .github/scripts/dispatch_join.py --dispatch tag-dispatch.json \
  --jobs jobs.json [--receipts downloaded-artifacts] [--caller "Tentacle short"]

Jobs match entries by name. Every entry, job and receipt ends in exactly one bucket (matched, conflict, ambiguous, unmatched, invalid or out of scope); exit 3 means the accounting failed and exit 2 that the input was incomplete. A match proves neither runner allocation nor native execution, and the tool is not wired into any workflow.

Failure archives

The runner passes Go's artifact flags in both execution modes (-artifacts -outputdir=<native>/test-artifacts, or the -test. forms for the binary). A test that finalizes its cluster with finalize.TerminateOnCleanup and Archive writes its failure archive under native/test-artifacts/_artifacts/<package>/<test>/<random>/ceph-<cluster>-<execution>/ with diagnostics.json, crash-metadata.json and manifest.json. A passing test leaves no directory.

Creation, upload and availability are separate facts. The test log and the manifest record what each collector wrote, and test_artifacts lists the files present after the run. For a failed native step, the job summary's Failure archive section records the manifest count, the upload outcome and the artifact ID. Only downloading the artifact and reading its manifest shows that the archive was retained. An archive is a snapshot taken before termination. A test that is still running near the profile's timeout writes a before-deadline archive before Go's timeout panic, which runs no cleanup; its manifest is never final. Nothing is archived when the runner is lost, the job is cancelled or the process is killed: the test binary exits on the cancellation's SIGINT without cleanups, and Ryuk removes the containers about ten seconds later, so a later step would race it and is not a supported collector.

archive_audit.py audits a downloaded archive directory without changing any verdict:

python3 .github/scripts/archive_audit.py <archive-directory> --output audit.json

It reports per file and per archive the containment, recorded write outcome, JSON readability and size agreement, and joins each capture_files record to its file entry. evidence_complete is true only when the producer declared the archive complete, the manifest is final with a termination, and every file and capture copy is verified. A producer-declared failure is never reported as download corruption.

Controlled failed-native download proof

The optional failure_archive batch runs two deliberately failing native subtests (an ordinary assertion and a finalization check) through the real testing.T, each producing an archive with its trigger, snapshot and debug restore receipts. Run it only on explicit request:

gh workflow run native-regressions.yml --ref <branch> -f batch=failure_archive

The native step and the workflow must fail. Then download the attempt's artifacts by their exact names and audit them from a checkout of that head:

gh run download <run-id> -n <tag-scenario-artifact> -D artifacts/archive-proof/scenario
gh run download <run-id> -n <scenario-images-artifact> -D artifacts/archive-proof/images
gh run download <run-id> -n <runtime-cleanup-artifact> -D artifacts/archive-proof/cleanup
python3 .github/scripts/failed_archive_proof.py \
  --scenario artifacts/archive-proof/scenario \
  --images artifacts/archive-proof/images --cleanup artifacts/archive-proof/cleanup \
  --root . --revision <full-checked-out-commit> --run-id <run-id> --attempt <attempt> \
  --output artifacts/archive-proof/audit.json

Exit 0 means archive_verification: verified with native_outcome: failed: both failed cases, matching binary and source hashes, the prepared image's ID, both archives with their listed files, debug receipts, successful restoration and termination, no leaked resources, and the history row's identity. Do not mix artifacts of different attempts. The audit does not prove the upload itself; keep the upload step and artifact ID as separate evidence.

Scenario history

scenario_history.py turns one job's receipts into one row of schema ceph-scenario-history/v1. It reads files only and never infers a cause. The workflow writes it as history-row.json; to build one from downloads:

python3 .github/scripts/scenario_history.py normalize \
  --scenario a/scenario --cleanup a/cleanup --images a/images \
  --repository <owner/repo> --run-id <run-id> --run-attempt <attempt> --job-id <job-id> \
  --output row.json

A row holds:

  • execution_key: repository, workflow, run, attempt, job, repetition, profile ID, checker, case and network mode;
  • source: the requested head and the checked-out commit and tree;
  • build: build key, binary sha256, a digest of the source hashes, and the native phase's binary identity (native_binary says whether they match);
  • profile: profile identity, tags and bug_compatibility. Receipts that disagree on tags, or whose selection differs from --bug-compatibility, write no row;
  • runtime: release, layout, platform and, for every role, the reference and the image ID that the job's engine reported (and any registry digests it listed, which may be none), or not_applicable;
  • phases: per phase the step outcome, the receipt and their join. A pass needs both; a missing receipt is missing, never passed;
  • outcome.status (failed, cancelled, passed or incomplete) and first_failure;
  • tests: every RUN, PASS, FAIL and SKIP line with its duration;
  • comparability: complete, or incomplete with what is missing;
  • evidence: the sha256 of every input and row_sha256.
python3 .github/scripts/scenario_history.py ingest --history history.jsonl row.json
python3 .github/scripts/scenario_history.py summary --history history.jsonl

ingest skips identical rows and refuses a different row under the same execution key or a row without run identity. summary counts outcomes per profile and release, failures by phase and classification, and p50/p90 native durations; cancelled rows are left out of the failure rate.

Durable history and classification

The History job of every Periodic run that executes the matrix or a campaign runs scenario_history.py collect on a checkout of the scenario-history branch. It appends the run's rows to rows/<run-id>-<run-attempt>.jsonl, rewrites summary.json and pushes. A conflicting row fails the job and changes nothing.

summary.json (schema ceph-scenario-history/v1/classification) has, per profile and release, the last 20 attempts and the 10 most recent configurations. A configuration is the profile, network mode, release, checked-out tree, build key (which includes the bug compatibility selection), platform and every role image ID; rows without a complete identity count as incomparable_rows. Each configuration records passes, failures, excluded attempts by kind, the first failure, per-test failed/ran counts (only attempts whose log has the test's result) and a classification.

passed, assertion_failure, timeout and cleanup_failure are scenario outcomes. runner_failure, image_failure, cancelled and incomplete are excluded.

Classification Scenario outcomes of the configuration
insufficient_samples fewer than two
confirmed_pass two or more, all passed
deterministic_failure two or more, all failed with the same phase, test and classification
confirmed_fail two or more, all failed, in different ways
suspected_intermittent at least one pass and one failure
git fetch origin scenario-history
git worktree add artifacts/history origin/scenario-history
python3 .github/scripts/scenario_history.py classify --history-dir artifacts/history

When a runtime job fails, report_test_failures.py reads summary.json and adds Scenario history notices beside the failed tests, such as TestClientRevocation/aes failed 2/10 earlier same-config attempts that ran it, with the configuration's classification. The notices never decide the job's result.

Reproducing and re-running

A release matrix job already retries its failed tests once (see Release retry). Re-run only the failed jobs when the failure looks unrelated to the change:

gh run rerun <run-id> --failed

To reproduce a batch locally with the same tags, budget and pass rules, run the tag runner; for identical images, first follow Reproducing CI's role images:

python3 .github/scripts/tag_scenarios.py run \
  --category recovery --batch container_pause --package ./internal/integration \
  --directory artifacts/repro-container-pause

Local runs are sequential unless CEPH_TEST_PARALLEL is set. To isolate one test, use go test -tags=all -run (see Testing).

Structural review reports

The Structural review evidence workflow collects report-only type and lint deltas for pull requests that change library code, starting with the pull request event. Its job summary and structural-review-RUN_ID-ATTEMPT artifact retain named base/head evidence. A trusted workflow_run publisher updates one current-head PR comment as soon as that run completes, without waiting for Code checks. Collection/upload are advisory and do not change required checks, documentation-only policy or native selection. See Structural review for identity checks, limits, artifact contents, exclusions and local commands.