This page explains what each GitHub Actions workflow checks, how the runtime batches are planned and run, and how to investigate a failing batch. How to run tests locally and how to add them is described in Testing.
| Workflow | File | Triggers | What it checks |
|---|---|---|---|
| Code checks | code.yml |
maintained branch push or PR, manual; weekly (Monday 05:17 UTC) for the vulnerability check only | Unit tests, race detector, vet, lint, tag compilation, CI tooling tests, compiled tag coverage, Go vulnerability check |
| Docker checks | docker.yml |
development push or PR, manual | The environment category: Docker bridge behaviour without Ceph |
| Short | short.yml |
development push or PR, manual | The short category on Tentacle 20.2.4 |
| Topology | topology.yml |
development PR, manual | The topology category on Tentacle 20.2.4 |
| Multicluster | multicluster.yml |
development PR, manual | The multicluster category on Tentacle 20.2.4 |
| Recovery | recovery.yml |
development PR, manual | The recovery category on Tentacle 20.2.4 |
| Periodic | periodic.yml |
daily at 04:41 UTC; manual | Random flaky samples on supported releases or historical patches; explicit full matrices and repeated investigations |
| Release matrix | release-matrix.yml |
PR targeting main, release/* or support/*, push to main or support/*; manual |
Environment and every required runtime category on every supported Ceph release |
| Structural review evidence | structural-collect.yml |
pull request | Report-only structure and lint deltas of library changes, published as one PR comment by structural-review.yml; not a required check |
| Experiments | experiments.yml |
manual only | One known-defect case with go test: every result line REPRODUCED on its default image, then, when a fixed image is given, the same results NOT_REPRODUCED with it injected |
| Native regressions | native-regressions.yml |
manual only | One optional batch chosen from the workflow's batch input, taking only its pass or failure: an experiment case, another optional native check or the failure_archive proof (see Native regressions for how to list and dispatch the choices) |
| ARM64 | arm64.yml |
weekly (Wednesday 05:37 UTC); pull request touching its planner, image, storage, runtime template, bootstrap or helper paths; manual | The source-owned arm64 cohort on a native GitHub-hosted ARM64 runner (see Native ARM64 cohort); not a required check |
| Block fault probe | block-fault-probe.yml |
pull request touching its scripts or workflow, manual | A research probe of delay and I/O-error tables on a scratch loop and device-mapper mapping, without Ceph; manual public_api also qualifies the public API on every supported release using a source-pinned helper; not a required check |
| CI command | ci-command.yml |
pull request comment /ci run ..., /ci require ..., /ci retry ... or /ci drop ... |
The named runtime batches on the pull request's head commit (see Running chosen batches from a comment); the last three also change the pull request's required requested CI |
| Requested CI gate | requested-ci-gate.yml |
every pull request run of Code checks, when it is requested; manual | Runs no test. Sets the commit status Required requested CI on the head of every open pull request (see Requiring requested CI before a merge) |
The ordinary runtime workflows and explicit full periodic runs call the reusable template tagged-runtime.yml
(workflow_call only). The workflows are independent: none waits for
another, and each can be re-run on its own.
Displayed check names are layered as <workflow> / <caller job> / <batch>,
for example Multicluster / Scenarios / mirror_initial_daemons or
Periodic / Squid topology / empty_bootstrap. Each runtime workflow
also has a Plan job.
A feature pull request to develop runs, on Tentacle 20.2.4 role images and in each batch's
default network mode:
- Code checks, for every change;
- Docker checks and every
shortbatch. With theci:selected-batcheslabel, Short runs only the batches that declaration selection selects (see Opting in to declaration selection); - of the
topology,multiclusterandrecoverycategories, only the batches whose own test files the pull request changes, and the batches of an experiment case any of whose files (helpers and manual included) it changes. A change to a shared helper or a public package selects none of them; Periodic runs them on the default branch.
The ci:full label (adding it starts a run) selects every
batch of those categories instead. Each Plan job's tag-plan.json artifact
lists the changed files and the selected batches.
Own-test-file selection compares proven commits. The caller passes the
event's base commit and the requested head; the planner records both, the
checked-out commit and its parents, and the compared range in
changed_selection. A two-parent test merge is accepted only when its second
parent is the requested head and the base is an ancestor of its first parent.
Missing commits, ambiguous merge bases or unrelated history fail planning
instead of selecting nothing. To inspect a selection manually:
python3 .github/scripts/tag_scenarios.py plan --category recovery \
--changed-since BASE_SHA --changed-head PR_HEAD_SHA --output plan.jsonWithout --changed-head the planner uses a plain merge-base comparison
(recorded as legacy-merge-base). --changed-files FILE is a separate
manual input.
Pushes to develop run Code checks, Docker checks and Short. PRs targeting
main, release/* or support/* and pushes to those branches receive
release candidate validation. Development
continues under ordinary CI while a candidate is frozen.
To run any other batch on a pull request, such as one batch of another
category or a batch on Squid, use /ci run <batch>. The command adds a run
and never changes the checks above. A reviewer who needs that run to pass
before the merge uses /ci require <batch>
instead.
The workflows set GOPROXY=https://proxy.golang.org|direct, so a failed
proxy request falls back to the module's origin. The Go jobs share one module
cache keyed by go.sum, and the runtime jobs share one build cache keyed by
the installed Go version and go.sum. Each runtime Plan job fills a missing
module cache with tag_scenarios.py modules and a missing build cache with
tag_scenarios.py buildcache (one batch per package); runtime jobs and Code
checks only restore them. A comment run
passes save_caches: false and saves neither: it belongs to the default
branch, whose caches every later run restores, but checks out an unmerged
head. setup-go's own cache is off. A cache only speeds up a build: each job
still compiles, records and verifies its own binary.
Ordinary development runtime workflows use
paths-ignore for docs/**, top-level *.md, LICENSE and
internal/integration/goceph/**, so a change to only those paths runs no
runtime batch. A Markdown file anywhere else, for example under a package's
testdata/, can be test input and runs the batches.
Code checks always starts, because its checks are required status checks of
the maintained branch rules. Its Detect documentation-only changes job runs
.github/scripts/docs_only.py. When that reports docs_only=true, Unit
tests, Race detector, Vet, compile and CI tooling and Compiled tag coverage
and helpers are skipped as whole jobs (a skipped job reports success), and the
Vulnerability check matrix jobs run with every step skipped so that each keeps
its expanded check name. The skip is by policy, not a pass of those checks.
A pull request to develop is documentation-only when every changed file is on this
allowlist:
README.md, any other top-level*.mdfile andLICENSE;- Markdown files directly inside
docs/contributing/,docs/guides/anddocs/internals/, except the files a check reads:docs/contributing/api-naming.mdanddocs/contributing/configuration-composition.md(the review note that the API catalogue receipts name, and the design note thatcephdoc comments link to),docs/guides/messenger-client-fixtures.mdanddocs/guides/bug-compatibility.md.
Every other path runs the full Code checks, including a package's
README.md (Go tests read it), any non-Markdown or nested file
under docs/, and a changed path whose old or new Git mode is not an ordinary
file. test_docs_only.py fails when a Go source or tag_scenarios.py names a
documentation path that is on the allowlist. Add a newly read file to
READ_DOC_FILES in docs_only.py.
The detection fails open. Release candidates, maintained branch pushes, the weekly schedule, manual runs,
pull requests from forks, and any failure to determine the change set (no
base, failed fetch or diff, empty list, script error) run every check. The
dependent jobs use !cancelled(), so a failed detection job runs them. The
script compares the pull request's merge commit with the event's base and
lists renames as a deletion and an addition.
| Workflows | Concurrency group | Effect |
|---|---|---|
| Code checks, Docker checks, Short, Topology, Multicluster, Recovery, Release matrix | workflow name, event, then the PR number (pull requests), the ref (pushes) or the run ID (manual runs) | A new commit on a PR cancels that PR's previous run of the same workflow. A new push to main cancels the previous main push run. Manual runs are never cancelled. |
| Periodic | workflow name and ref; explicit representative investigations add -campaign |
Rotate, discovery, targeted, full and storage runs wait in one group. They never cancel an executing run. The representative group is separate and bounded; it has no schedule. GitHub retains only one waiting run per group, so a later queued request can replace an earlier queued request. |
| Topology, Multicluster, Recovery | as above, plus the run ID for a label other than ci:full |
An unrelated label starts a run that skips its jobs and cancels nothing. |
| Short | as above, plus the run ID for a label other than ci:selected-batches |
Adding or removing ci:selected-batches replaces the run in progress. |
| CI command | per caller job: PR number, head commit, selected profiles and release | An identical command waits for the one in progress; a third identical command replaces the waiting one, which ends cancelled. Different selections run independently. |
CI command (Register, Gate), Requested CI gate |
the fixed name required-requested-ci, per job |
Every job that publishes Required requested CI waits its turn and none is cancelled while it runs. A waiting job is replaced by a newer one (see How the status is kept). |
| Structural review evidence | workflow name and PR number | A new commit on a PR cancels that PR's previous run. |
| Native regressions | none | Runs are independent. |
| ARM64 | workflow name, then the PR number or the run ID | A new commit on a PR cancels that PR's previous run. Scheduled and manual runs are never cancelled. |
| Block fault probe | workflow name, then the PR number or the run ID | A new commit on a PR cancels that PR's previous probe. |
Consequences:
- Different PRs never cancel each other. In a stack, base each PR on the head branch of the previous one.
- Retargeting a PR requires checks for the new target. Confirm that the run
identifies the intended base and commit before using its evidence. GitHub
also emits
editedfor title and body changes. Those edits rerun candidate validation, so prepare the description before opening the promotion PR. - Push runs are scoped to their own branch. A change on
developcannot cancel a release candidate on another branch.
The local commands for these jobs are in Local checks without Docker.
| Job | Timeout | Runs |
|---|---|---|
| Unit tests | 15 min | make test |
| Race detector | 15 min | make race |
| Vet, compile and CI tooling | 25 min | make vet, golangci-lint, make tag-compile, the Python tests in .github/scripts (test_*.py), actionlint and the go-ceph harness test |
| Compiled tag coverage and helpers | 25 min | tag_scenarios.py plan --verify, then tag_scenarios.py code (the Docker-free code batch); uploads code-tag-coverage |
| Vulnerability check (Go 1.26.x and 1.27.x) | 20 min | make vulncheck; uploads vulncheck-go-<version> |
golangci-lint is the release that code.yml pins by version and archive
SHA-256. The golangci-lint action installs it for this job and for the
structural review collector: it verifies the archive restored from the
Actions cache, or downloads it with up to five attempts that retry only
transport failures and 429/5xx answers, and the lint step itself downloads
nothing. To move to another release, change both callers' version and
sha256 (from the release's checksums.txt) and the Makefile comment;
test_golangci_lint_install.py checks that they agree.
The Python tests include test_tag_scenarios.py for the planner and
test_scenario_shards.py, which checks the workflow files: one workflow per
category, no test-name lists in workflows, budgets taken from the source,
compile before Docker, cleanup on the same runner, and failure reporting that
never hides the test step's status.
make vulncheck builds govulncheck v1.9.0 and scans the module with -test
for linux/amd64 with no tags, all and all,goceph,native_regression, and
for linux/arm64 and darwin/arm64 with no tags. The nested go-ceph probe module
needs cgo, so it gets a module scan only.
.github/scripts/check_vulns.py decides the result, because govulncheck's
JSON mode exits 0 on findings. Per vulnerability it takes the most precise
level: symbol and package findings fail the check, module-only findings are
listed and pass, and a malformed or incomplete stream fails as a broken scan.
The job runs with the module's minimum Go (1.26.x) and the current release
(1.27.x), because standard-library findings depend on the toolchain. It
restores the shared module cache and the build cache for its Go version; a
successful default-branch scan saves a missing build cache. Caches never skip
a scan or retain the vulnerability database. Every setup-go step sets
check-latest: true, and go.mod names a toolchain line, so an older
patch release with known standard-library vulnerabilities is not used. The
weekly scheduled run catches new advisories; the other Code checks jobs skip
it. Dependabot targets develop for version updates. Its security alerts on
the default branch also need the maintained-line patch procedure.
.github/dependabot.yml proposes Go module updates daily and GitHub
Actions updates weekly.
.github/scripts/tag_scenarios.py builds the CI matrix from the test source.
Nothing in the workflows names a test.
- Catalog.
internal/tools/tagcatalogparses every*_test.gofile (skipping hidden directories,artifacts,vendorandtestdata) and records its//go:buildexpression, tags,Test...functions and//ci:metadata. - Discovery. Every file that declares tests and has a constraint belongs
to exactly one category (
ci_<category>) and one batch (ci_batch_<name>), compiles underall, and carries one valid//ci:line. One(category, batch, package)triple is one profile; its files agree on budget and image roles. Optional (ci_optional) and SDK (ci_sdk) files stay out of the required set and need a tag beyondall. - Profile tags. A batch compiles with every capability tag found in the
source plus
ci,ci_<category>,ci_batchandci_batch_<name>. No file of another batch may compile into the profile. - Selection.
plan --category <category>writes that category's profiles as the matrix;plan --optional --batch <name>selects one optional batch;plan --representativeselects the campaign set. - Verification.
plan --verifycompiles theall, category and single-batch builds, including every optional batch, withgo test -listand compares the test names with the catalog. Code checks runs it; the runtime workflows do not.
Each profile has a stable ID, <category>-<batch>-<8 hex digits>, such as
short-default-a488da21, used in artifact names, summaries and annotations.
The planner, compiler and runner always use CGO_ENABLED=0, GOWORK=off and
GOFLAGS=-mod=readonly, and pass only build tags and the batch timeout to
go test, with no -short or name filters; the only -run is the list of
mode-declaring tests of a network-mode entry other than the default. To move a test between
batches, change its build constraint.
Checker batches (CephFS process quiescence and process recovery per receipt
kind, and RBD receiver readiness) declare checker= and case= in //ci:.
They run in one network mode per entry, and afterwards
check_scenario_quiescence.py, check_scenario_recovery.py or
check_scenario_receivers.py verifies that mode's evidence in the log. The
planner requires the batches of a checker to cover every mode and kind.
A ceph_bug_<id> tag selects the compatibility path of one Ceph bug (see the
policy); it never selects tests. The planner never
collects it as a capability, and all or native_regression never enable
it. Only --bug-compatibility TAGS, accepted by plan, verify, compile,
run, code, optional and buildcache, selects one:
python3 .github/scripts/tag_scenarios.py plan --bug-compatibility ceph_bug_<id>The selection is refused when a tag is not registered in
bug-compatibility-tags.json, is
retired, is repeated or conflicts with another selected tag. The planner also
refuses a test file that names an unregistered or retired bug tag, and a
registered tag that no source file names. With no selection every ordinary
path compiles.
Both paths of a bug must keep the same tests in the same batch: for every
profile and each package's all build the planner toggles each bug tag the
package names and requires identical test names. Toggles that would activate
conflicting tags are skipped. The selected tags apply to every profile of the
plan, appear in its tags and bug_compatibility, and so change the build
key, compile receipt and history configuration, but not the profile ID.
run --binary refuses a binary compiled under another selection.
tagged-runtime.yml takes the selection as the bug_compatibility input
(empty by default). Plan passes it to plan and buildcache, so an invalid
selection fails before any runtime job. Each runtime job passes it to
compile, run or optional with --profile-tags set to its matrix
entry's tags, and fails before building when the recomputed profile differs.
The selection reaches commands only as an argument; tag_scenarios.py reads
none from the environment. No caller sets the input, so every workflow runs
the empty selection.
internal/tools/batchselect narrows each integration test package
(internal/integration and its rbd, cephfs and rgw packages) at the
level of declarations, because its *_helpers_test.go files and the shared
internal/integration/harness compile into every batch.
By default it is only reported:
python3 .github/scripts/tag_scenarios.py plan --category short \
--select-declarations origin/develop --apply-declarations \
--summary /dev/stdout --output artifacts/tag-plan.jsonWithout --apply-declarations, the plan gains a declaration_selection
object (selected and excluded reasons per profile, fallback,
changed_declarations) and keeps the full matrix. With it, the matrix keeps
only the selected profiles and selected_scope is declarations.
--summary FILE appends the decisions as a Markdown table.
The analysis has no default platform: the planner passes -goos linux and its
--goarch (amd64 unless a plan names another), and the plan records both, so
a selection never describes another architecture's source files than the
plan's. The tool diffs the checkout against its merge base with the base, type-checks
the package's all build (and each batch build not covered by it) with
go/types, together with the harness from source (-with), and builds one
graph over the top-level declarations of both. Edges are resolved
identifiers, interface method calls to every implementing type of the two
packages, and each type to all its methods. A test file of another package
is never part of the build and is ignored. A batch is selected when one of its roots
(Test, Fuzz and Example functions, init functions and package-level
var initializers it compiles) reaches a changed declaration. An added file
counts as a change to each of its declarations.
Every batch is selected, with the reason, when the change touches anything
the graph cannot attribute: a line outside every declaration (file header,
//go: directive), an import of a package that no other file of the build
imports, a deleted, untracked or non-Go file in the package, .github/,
go.mod, go.sum, any other path outside the package and the harness other
than documentation, a removed init or initializer, or anything that fails to
parse or type-check. Each package's fallback selects only that package's
batches. Profiles outside the integration packages are always selected, and a
tool failure selects every profile.
Short is the only workflow that narrows by declarations, and only for a
development pull request with the ci:selected-batches label. The caller then passes the
base commit as the select_declarations input of tagged-runtime.yml, and
Plan runs plan --select-declarations BASE --apply-declarations --summary "$GITHUB_STEP_SUMMARY". The Plan summary lists every batch with yes or
no and its reason.
Adding or removing the label replaces the run in progress; later commits run
selected while the label stays. Every other trigger and workflow runs all its
batches, and the develop push after a feature merge runs every Short batch, so a
selection miss fails there. Record a miss with its plan, failed job and
selection evidence in a new issue. The label only helps for changes inside the
integration test packages and the harness.
A collaborator can run chosen runtime batches on a pull request by commenting:
/ci run <batch|profile>... [on <release|codename|all>...]
For example, /ci run rbd_mirror_bandwidth network_faults on squid runs two
batches on Squid 19.2.5 role images. The run is added beside the usual checks.
- Names. A batch (
network_faults), category and batch (short/network_faults), profile name (short/network_faults/internal-integration) or profile ID, separated by blanks or commas. Any runtime category and the optional batches are allowed;codebatches are refused. Duplicates run once. - Releases. After
on: a release (20.2.4), a codename (tentacle,squid) orall. Withouton, the current release. Each batch runs in its default network mode. - Validation. Names are checked against the head commit's planner
(
tag_scenarios.py vocabulary). One unknown name refuses the whole command, and the reply lists every unknown name. - Format. The command must start its line and appear once per comment; quoted or fenced lines are ignored. Editing a comment does not run it.
- Authorization. Only authors with association
OWNER,MEMBERorCOLLABORATORrun commands; others get a refusal reply. Fork and closed pull requests are refused; issue comments and bots are ignored.
The workflow records the pull request's head commit when it reads the pull
request, and every job runs that commit. issue_comment runs use the
workflow and scripts from the protected default branch:
| Job | Permissions | What it does |
|---|---|---|
Command |
contents: read, pull-requests: read |
Runs ci_command.py authorize on the default-branch checkout. No pull request code runs. |
Validate |
contents: read |
Checks out the default branch and the recorded head side by side without credentials; the head's planner lists batches and releases, the default branch's ci_command.py select resolves the names. |
Register |
contents: read, pull-requests: write, statuses: write |
Only for /ci require, /ci retry and /ci drop: runs required_ci.py register on the default-branch checkout (see Requiring requested CI before a merge). |
Acknowledge |
statuses: write |
Sets a pending commit status on the head. |
Scenarios <release> |
contents: read |
Calls tagged-runtime.yml with category selected and the resolved profile IDs, once per release. |
Report |
contents: read, pull-requests: write, statuses: write |
Sets the final status and replies with the result, run link or refusal. |
Gate |
contents: read, pull-requests: write, statuses: write |
Only after the run of a required selection: runs required_ci.py evaluate on the default-branch checkout. |
The result appears on the pull request as the commit status
CI command <comment-id>: <names> on <releases> (success, failure, or
error for a cancelled run). A context longer than 100 characters is cut and
ends with a 12-character digest of the selection. Refused commands set no
status. A cancelled run is never re-run automatically; comment again or
re-run the workflow run.
To check a command locally on saved event and pull request JSON:
python3 .github/scripts/tag_scenarios.py vocabulary --output artifacts/vocabulary.json
python3 .github/scripts/ci_command.py authorize --event event.json --pull pull.json
python3 .github/scripts/ci_command.py select \
--request '{"names": ["network_faults"], "selectors": ["squid"], "duplicates": [], "sha": "<head>", "pull": 1, "comment_id": 1}' \
--vocabulary artifacts/vocabulary.json/ci run is advisory: no merge depends on its result. To make a selection
part of the merge decision of one pull request, a reviewer requires it:
/ci require <batch|profile>... [on <release|codename|all>...]
/ci retry <batch|profile>... [on <release|codename|all>...]
/ci drop <batch|profile>... [on <release|codename|all>...]
The names, releases, format and authorization are those of /ci run, and so
are the refusals: an unknown name, a fork, a closed pull request or an
unauthorized author changes no requirement.
| Command | Effect |
|---|---|
/ci require |
Adds the selection to the pull request's requirements and runs it on the head commit. Requiring a selection that is already required changes nothing and runs nothing. Different selections add up, and each must pass. |
/ci retry |
Runs a required selection again on the current head commit, after a failure, a cancelled run or a new push. That run replaces the earlier one as the selection's evidence. Its names are resolved again on the new head; if they no longer resolve, the selection is recorded as blocked and fails until the pull request restores the names or the selection is dropped. |
/ci drop |
Removes the selection from the requirements. Its names are not resolved, so a selection whose batch is gone can be dropped. |
A selection is its names and release selectors as typed, in any order.
/ci require network_faults and /ci require network_faults on tentacle
are two selections even while Tentacle is the current release, so retry and
drop repeat the selection as it was required. The summary comment lists the
exact commands.
The commit status Required requested CI is set on the head commit of
every open pull request, with or without requirements:
| State | Meaning |
|---|---|
success |
The pull request requires nothing, or every required selection passed on this head commit. |
pending |
A required selection is running, or has no run on this head commit. |
failure |
A required selection failed, its run was cancelled, skipped or ended without a known result, or its names no longer resolve. |
error |
The requirements are not known: a comment of the ledger was edited or cannot be trusted, the comments of a pull request with this head could not be read, or the job that publishes the status did not finish. |
A pull request with requirements has one summary comment, which the status
links to. It lists every required selection with its state, the commit that
was tested, the command, run and result that are its evidence, who required
it, and the retry and drop commands for it. The comment is updated in
place.
What satisfies a requirement is narrow on purpose:
- only a successful run of the selection's latest accepted
requireorretrycommand counts. The result of a command that a laterretryreplaced, or of a requirement that was dropped and required again, is ignored whenever it arrives; - the run must have tested the pull request's current head commit. After a
push the requirement stays and its evidence does not: the new head is
pendinguntil the selection is retried on it. Nothing starts that run automatically; - the run tests the head commit itself, not a merge with the base, so a change of the base branch or of the pull request's target neither satisfies nor invalidates a requirement;
- a result is recorded on the pull request whose command started the run. The same commit tested for another pull request is no evidence;
- a run without a recorded result is no evidence. A selection that stays
runningafter its run ended lost its result; retry it.
A commit status belongs to a commit, and GitHub evaluates a required status on the head commit of the pull request. Pull requests that share a head commit therefore share the status, and it is the worst of theirs: none of them passes until the requirements of each are met. The description names the pull request that decides it.
-
Request. Comment
/ci require <selection>on the pull request. -
Wait for the acknowledgment. The comment is processed by a workflow run, seconds to minutes later. Until then nothing is required and the status can still be green, so hold the merge. The acknowledgment is a reply of
github-actionsthat starts withRequired:; the status of the head is setpendingbefore that reply is written. A reply that says the command is not confirmed, or no reply at all, means that the command may not have been recorded: read the summary comment and repeat the command. -
Read the result.
Reportreplies with the outcome and the run link, and the status and the summary follow. A failed selection is diagnosed like any failed batch (see Reading a failing batch). -
Retry or drop.
/ci retry <selection>after a fix was pushed or when the failure was not the change's;/ci drop <selection>when the selection is no longer wanted. Both are acknowledged the same way (Retrying the required selection:,No longer required:), and both stay on the pull request with their author. -
Verify before the merge.
Required requested CIissuccesson the exact commit to merge, and the summary comment names that commit as its head and lists every selection that was asked for aspassed. A summary for another commit means the status was not evaluated for this pull request (see How the status is kept):gh api repos/<owner>/<repository>/commits/<head-sha>/status \ --jq '.statuses[] | select(.context == "Required requested CI") | {state, description, target_url}'
The status blocks a merge by itself only on a branch whose rule requires the context (see Enabling the branch rule). On any other branch this step is the reviewer's.
Do not edit or delete the replies of github-actions that record a
requirement. An edited one turns the status into error; the summary then
links to it, and deleting it and repeating the commands it stood for
restores the ledger. GitHub shows the deletion in the pull request's
timeline.
This is separate from the release gate. A release candidate's own retry rule (see Release candidate validation) decides what a candidate may contain; a required selection is a check of one reviewed commit and of exactly the coverage the reviewer asked for, and it has no automatic retry.
The requirements of a pull request are a ledger of comments on it. A record
is the first line of a comment written with the workflow token, an HTML
comment that Register (require, retry, blocked, drop) and Report
(result) write and never change. required_ci.py reads the ledgers again on
every evaluation and derives every status from them; nothing else stores a
requirement. A comment by anyone else that copies a record is not one.
Three jobs publish the status, all with required_ci.py from the default
branch, and none checks out a pull request:
| Job | When | What it does |
|---|---|---|
Register of ci-command.yml |
an accepted require, retry or drop |
Reads the pull request's head and ledger, sets the head pending (not for drop), writes the record with the acknowledgment, then evaluates. A failure before the record is written records and acknowledges nothing; a failure to write it leaves the head pending. |
Gate of ci-command.yml |
after Report recorded the result of a required selection's run |
Evaluates. |
Evaluate of requested-ci-gate.yml |
every pull request run of Code checks, when it is requested; manual dispatch | Evaluates. |
Code checks starts for every opened, pushed, reopened or retargeted pull request of a maintained branch, documentation-only ones included, so each new head gets its status without a command. An evaluation covers every open pull request and sets a status only where it changed.
The three jobs share the concurrency group required-requested-ci, so no
evaluation publishes from a ledger that another job is changing. GitHub
keeps one waiting job per group. A waiting evaluation that a newer one
replaces loses nothing; a waiting Register that is replaced has recorded
nothing, and Report answers that the command is not confirmed.
A failed job does not fail a commit status: a commit keeps the status it
has until another one is set. The publishing jobs therefore turn what they
cannot establish into error and never leave a head as it was:
- a pull request whose comments cannot be read after two repeated attempts,
or cannot be evaluated, makes its head
error(The requirements of #N could not be read), for every pull request that has that head. The others are published and the job fails; - a job whose script does not report every head as published, because it
failed, was cancelled or timed out, or because the checkout before it
failed, ends with a step that sets
error(The evaluation did not finish) on the commit that started it: the head of the Code checks run forEvaluate, the commit of the command forRegisterandGate. That step needs only the token andgh; - a summary comment is not written for a head whose status could not be set, so it never says more than the status.
error is cleared by the next evaluation that succeeds: any pull request
event, any requirement command, or
gh workflow run "Requested CI gate". A manual dispatch has no commit of
its own, so a failed one withdraws nothing.
One case is outside what these jobs can do. When the job cannot reach the
API to list the pull requests or to set a status, or never starts, every
commit keeps the status it has. That matters for a commit that is already
green: a pull request pushed to, or opened at, a commit that another pull
request made green shows that green until an evaluation succeeds. The
failed Requested CI gate or CI command run is the signal, and
step 5 of the reviewer steps catches it: the summary
comment of a pull request with requirements names the head it was written
for, and a summary that names another commit, or that is missing although
selections were required, means that the status is not this pull request's.
Run the gate again before merging.
The jobs that compile and run the pull request (Validate and
Scenarios, and every job of the pull request workflows) hold a read-only
token and can write neither a status nor a comment. Pull requests from forks
cannot be given requirements, and their heads get the success of a pull
request without any. As for every check of this repository, someone with
write access can change a workflow on a branch and so set any status; the
rule that only GitHub Actions may provide the context excludes statuses set
with a personal token, not that.
The rule is the ruleset Git Flow / requested CI, which requires the
context from GitHub Actions on main, develop and every release and
support line. Those are the branches whose pull requests start Code checks,
and that request is what makes the gate publish the status on a new head.
The ruleset is optional: scripts/release/git_flow.py creates it only when
asked, beside the other rulesets and without changing them, and verify
checks it only where it is installed. Until then the status is advisory and
step 5 of the reviewer steps is the reviewer's.
A branch rule must require the context only when every pull request it covers gets it, otherwise that pull request waits for a status that no event will bring. Install it in this order:
-
Confirm on a Dependabot pull request that
Required requested CIappears on its head by itself. Its Code checks run is requested by another actor than a maintainer's, and the gate must publish for it as for any other pull request. -
Dispatch the gate and check that it succeeded:
gh workflow run "Requested CI gate". -
Plan. The plan prints the ruleset and names every open pull request to a maintained branch whose head has no status yet:
python3 scripts/release/git_flow.py plan --requested-ci
-
Apply and verify.
applyrefuses while the plan still names a pull request, and after any drift of the existing rulesets:python3 scripts/release/git_flow.py apply --requested-ci python3 scripts/release/git_flow.py verify
After that, a pull request opened or pushed later gets the status from the Code checks request of that event.
Each matrix entry is one job on one runner of the plan's target architecture:
ubuntu-24.04 for amd64, the default, and ubuntu-24.04-arm for the
arm64 cohort. The Plan job runs on the same
architecture, because it warms the build cache that the jobs restore.
| Step | Phase in summary | What happens |
|---|---|---|
| Restore the Go module and build caches | n/a | Restores what a Plan job saved. A build-cache miss only slows compilation. |
| Prepare Go modules | Go module preparation | tag_scenarios.py modules runs go mod download within 300 seconds. |
| Compile selected tagged suite | Compilation | tag_scenarios.py compile first refuses a runner whose Go toolchain is not natively the matrix entry's goarch (host and target), then runs go test -c within 180 seconds with GOPROXY=off, writes compile/<package>.test, and records in compile-report.json the binary's sha256, its build key (package, tags, Go environment, version, GOOS, GOARCH), the sha256 of every source file read, and the listed tests. It first refuses a recomputed profile whose tags differ from the matrix entry's. |
| Prepare runner storage | Runner storage preparation | Ceph batches only. runner_storage.py prepare skips the existing fixed-directory cleanup only when the exact workload, on its qualified release, roles and platform, and the actual storage layout have a qualification and sufficient headroom. Otherwise it removes the unused runner toolchains. Writes runner-storage/storage-prepare.json and, when reclamation ran, storage-free.json. Cleanup remains GitHub-hosted-only and never removes Docker resources. |
| Docker baseline and engine identity | Docker baseline | Records the engine identity and every resource labelled org.testcontainers=true. |
| Resolve native published Ceph roles | Published image preparation | Ceph batches only. Pulls the declared role images from ghcr.io/jsyoo5b/ceph-testcontainers-images by their official-<release>-<role> tags (see Role images of a run), for the entry's linux/<goarch>, refusing a Docker engine of another platform, records the image ID and the platform child that Docker selected from each index and exports the selected CEPH_TEST_*_IMAGE variables, which name the tags. A pull that fails on a registry transport error (TLS handshake or DNS timeout, reset, EOF, any 5xx answer to a manifest, blob or token request) is retried twice, after 10 and 30 seconds; a registry refusal (unknown manifest, denied, unauthorized, rate limit, any 4xx answer) is never retried. Never builds. |
| Prepare Ryuk and the test images | Ryuk and test image preparation | Every batch, Docker-only ones included. in_test_images.py pulls, with the same bounded retries, the Ryuk image of the go.mod version of testcontainers-go (its ReaperDefaultImage) and every image that the batch's compiled tests name besides their roles, so that no test pulls an image itself. batchselect -images finds those in the source, for the GOOS and GOARCH that the compile receipt records, and the step refuses a Docker engine of another platform: each image reference among the constants of the declarations that the tests reach (constants of other packages included), and a reference prefix that a test completes with the Ceph version at run time (the release's official image, pulled by tag). A published role tag that a test names itself, such as both releases of the rolling upgrade, is one of those constants and is pulled by its tag. With prepared role images, the harness fallback to a published role tag (ceph.DefaultImage) is skipped. Writes test-images/summary.json and test-images/pull-<name>[-<attempt>].log. |
| Record runner storage with the images | n/a | Ceph batches only. Writes runner-storage/storage-images.json. Its failure never changes the job result. |
| Verify post-image runner headroom | Post-image runner headroom | Ceph batches only. runner_storage.py ready refuses the job before the tests when the pulled images left too little headroom for the workload. Writes runner-storage/storage-ready.json. |
| Native assertions and completion proof | Native assertions and completion | tag_scenarios.py run (or optional) with --binary runs the compiled binary and applies the pass rules below. |
| Record runner storage after the tests | n/a | Ceph batches only, also after a failure. Writes runner-storage/storage-after.json. |
| Release the qualification pressure file | n/a | pressure storage policies only, also after a failure. Deletes the pressure file only when its device and inode match storage-pressure.json. |
| Verify owned Docker resource cleanup | Docker resource cleanup | Runs whenever the baseline succeeded, also after a failure. Waits up to 30 seconds for Ryuk, then fails if new labelled resources remain. Never deletes anything. |
| Show actual failure phases in PR check annotations and summary | n/a | Writes the phase table to the summary and phases.json, and annotates failed or cancelled phases. |
| Normalize the scenario history row | n/a | Writes history-row.json (see Scenario history). Its failure never changes the job result. |
| Save compile, native and phase results | n/a | Uploads tag-scenario-<id>-<release>. |
| Record the failure archive upload | n/a | After a native failure, writes the number of local archive manifests and the upload's outcome and artifact to the summary. |
| Read the scenario history summary | n/a | On failure, reads summary.json from the scenario-history branch for the next step; a missing branch leaves it out. |
| Identify completed failing tests | n/a | On failure, annotates each failed test or subtest found in native.log. |
The job sets CGO_ENABLED=0, TESTCONTAINERS_HOST_OVERRIDE=127.0.0.1 and
CEPH_TEST_PARALLEL=2, and uses the latest Go 1.26.x.
run --binary <compile-directory> (and optional --binary) executes the
binary that compile wrote. Before starting anything it requires a
successful receipt for the same profile and tags, a matching binary sha256, a
matching recomputed build key, unchanged source hashes, and a test inventory
(recorded and from -test.list=.) equal to the batch's declared tests. A
failed check never falls back to go test. The binary runs in the package
directory with -test.v -test.count=1 and the batch's -test.timeout; a
network-mode entry adds -test.run for its --tests. native/report.json
records execution: test-binary, the sha256 and the build key.
Without --binary, run builds and runs with go test. To follow the CI
path locally:
python3 .github/scripts/tag_scenarios.py compile --category code --batch pure --package ./internal/integration --directory artifacts/c/compile
python3 .github/scripts/tag_scenarios.py run --category code --batch pure --package ./internal/integration --directory artifacts/c/run --binary artifacts/c/compileCeph batches declare their roles in //ci: metadata: essential, client,
cephfs, rbd-aux and rgw. Ordinary RADOS and RBD use essential,client;
CephFS adds cephfs, RBD mirroring and cache add rbd-aux, and gateways add
rgw. An undeclared Ceph profile gets all five; non-Ceph profiles get none.
Only declared roles are prepared, and a missing role is never substituted:
ceph.Run, or the feature's own operation, refuses a missing role
(harness.IntegrationImages itself only requires the essential image). The optional
role_routing and role_mirror_refresh batches cover constrained routing
and can be requested with /ci run.
A test that can run on the Docker bridge and on the host network declares the
modes with harness.NetworkModes, its default first, and runs one subtest per mode:
for _, mode := range harness.NetworkModes(t, "host", "bridge") {
host, name := mode.Host, mode.Name
t.Run(name, func(t *testing.T) { /* ceph.WithHostNetwork() when host */ })
}The default is host for the RADOS products (ceph, rbd, cephfs),
whose native clients must reach the advertised daemon addresses, and bridge
for RGW. A test using a bridge-only feature
(NetworkConfig.WithSeparateClusterNetwork, InterruptNetwork,
ApplyNetworkProfile) declares bridge alone or keeps that part in the
bridge subtest.
The planner reads the modes from the call, so they are string literals in the
Test function itself, and the files of one batch declare the same modes. A
matrix entry sets CEPH_TEST_NETWORK:
| Scope | Entries of a batch |
|---|---|
Development PRs and pushes, CI command (networks: default) |
One, in the default mode, with all tests |
Periodic and the release matrix (networks: all) |
The default entry, plus one per other mode that runs only the mode-declaring tests (--tests), named <batch> (<mode>) |
A test without harness.NetworkModes runs once and ignores CEPH_TEST_NETWORK. Run
locally with an undeclared mode, a test skips; with the variable unset, it
runs every declared mode.
The native phase succeeds only if native.log shows:
- exit status 0;
- exactly one
=== RUNand one--- PASSfor every compiled top-level test, and no other top-level test; - exactly one
--- PASSfor every started subtest; - no
--- FAIL,FAIL,panic:or--- SKIPline at any level; - exactly one
PASSline after all results, followed by oneok <package>line undergo test, or as the last line under--binary; - for checker batches, a passing strict checker.
A skipped test therefore fails CI. Tests assert release differences instead of skipping (see Writing the test body).
| Job | Limit |
|---|---|
| Code checks jobs | 5 to 25 minutes, fixed in code.yml |
| Plan job of each runtime workflow | 10 minutes |
| Each batch job | job-timeout from the batch's //ci: line |
The go test process of a batch |
timeout from the batch's //ci: line |
The planner accepts a process timeout up to 6 hours and a job timeout of 1 to
360 minutes, and requires the job timeout to exceed the process timeout;
batches leave 5 to 15 minutes for setup and cleanup. Run
tag_scenarios.py plan to see every budget. Runners are GitHub-hosted
ubuntu-24.04 machines with 16 GB of memory (ubuntu-24.04-arm for the
arm64 cohort). CEPH_TEST_PARALLEL=2 lets two
opted-in tests run at once in a batch; the rest run sequentially within the
process timeout.
The account runs at most 40 hosted jobs at once, across all of its
repositories: over three days with 4524 executed jobs in its active
repositories, 40 jobs ran at once for 48 minutes in total and never more.
tag_scenarios.py allocates them by what a run is for:
| Run | Allocation | Reason |
|---|---|---|
| Release matrix | Every runner except OTHER_REPOSITORY_RUNNERS |
It gates a release during a merge window, when no other work of this repository should be in flight. |
| Periodic | At most PERIODIC_RUNNERS, and nothing while a Release matrix run is in progress |
It searches for flaky tests: slow is fine, taking the account is not. |
| Pull request | No reserved share; each runtime caller is bounded by PULL_REQUEST_MAX_PARALLEL |
A pull request validates its own change, and several are open at once. |
| Constant | Value | Meaning |
|---|---|---|
RUNNER_LIMIT |
40 | The measured limit of the account. |
OTHER_REPOSITORY_RUNNERS |
8 | Runners that the Release matrix leaves to the account's other repositories. |
RELEASE_CALLER_RUNNER_MINUTES |
short 325, topology 335, multicluster 195, recovery 230 | Measured runner time of one release caller of each category. |
ENVIRONMENT_MAX_PARALLEL |
1 | max-parallel of an environment caller. |
PERIODIC_RUNNERS |
8 | Runners that every mode of Periodic holds together at most. |
PERIODIC_CALLERS |
4 | Release callers of a full periodic matrix that run at once (max-parallel of the caller matrix). |
PERIODIC_MAX_PARALLEL |
1 | max-parallel of each release caller in Periodic. |
CAMPAIGN_MAX_PARALLEL |
3 | max-parallel of the representative campaign. |
STORAGE_MAX_PARALLEL |
5 | max-parallel of a storage qualification dispatch. |
PULL_REQUEST_MAX_PARALLEL |
short 8, topology 3, multicluster 2, recovery 2, environment 1 | max-parallel of a pull request's runtime caller of each category. |
PULL_REQUESTS_AT_ONCE |
2 | Pull requests that can each run every category beside the other repositories' runners. |
Release matrix. release_max_parallel takes the share of 32 runners less
the environment caller, splits it evenly between the supported releases, and
splits a release's part between its four callers in proportion to
RELEASE_CALLER_RUNNER_MINUTES, whole runners by largest remainder. With
three releases that is short 3, topology 3, multicluster 2 and recovery 2 for
each release: 31 runners with the environment caller. A new supported release
changes the allocation without an edit here. The runner minutes come from a
Release matrix run in which every job executed (506 jobs, about 3250 runner
minutes): one release's callers took about 325 (short, 51 jobs), 335
(topology, 59), 195 (multicluster, 23) and 230 (recovery, 34) minutes. That
run was bounded at short 3 and one job for each other caller, held at most 19
runners and took 5.86 hours, the serial time of one topology caller. With the
allocation above the callers need about 108, 112, 98 and 115 minutes of
runtime jobs, so a run is expected to take about two hours. The longest job
was 19 minutes, so more runners would shorten it little.
The Release matrix leaves no runner for this repository's pull requests. A pull request during a merge window is either a mistake or the fix of a defect that already blocks the candidate; in the second case the candidate's run is cancelled (see Candidate failures), which frees its runners.
Periodic. The daily sample is one job. A full matrix runs
PERIODIC_CALLERS callers at once with one runtime job each beside the
environment caller, five runners; a storage qualification dispatch and the
sample wait in the same concurrency group. The representative campaign has a
group of its own and can run beside a full matrix, so the two add up to
eight. Every run of Periodic first asks whether a Release matrix run is
queued or in progress (periodic_gate.py --yield-to release-matrix.yml) and,
if one is, starts no native work; the skipped run is not repeated. A Release
matrix run that starts after Periodic did runs beside it.
Pull requests. The entry points pass their category's bound, so a pull request that runs every category holds at most 16 runners, and two such pull requests fit beside the other repositories' eight.
tag_scenarios.py releases --allocation release fails when the Release
matrix callers could exceed RUNNER_LIMIT - OTHER_REPOSITORY_RUNNERS, and
--allocation periodic fails when the modes of Periodic together could
exceed PERIODIC_RUNNERS. A bound only delays jobs: every entry still runs
with fail-fast: false. Batches are not packed into shared jobs.
The order of matrix entries is not a start order. GitHub creates a matrix's
jobs in an order of its own, and while every runner of the account is busy,
queued jobs start roughly in that creation order, so reordering the planner's
entries does not move a long batch forward. A release caller is bound by its
summed runner time: replaying measured durations onto a caller's slots
longest-first shortens the makespan by under 1% and delays the mean
completion by several minutes. A pull request run ends with its longest batch
or, when its callers' bounds hold jobs back, with a caller's summed runner time
over its bound; the batch layout sets its length, not the dispatch order
(issue 388). A
long batch whose tests are independent opts them into
harness.ParallelWhenEnabled rather than adding a job.
To measure runner use, fill a cache from the GitHub API and report offline:
python3 .github/scripts/ci_measurements.py fetch --cache artifacts/ci-cache --repo <owner/repo> \
--since <date> --marker <name>=<commit> [--cache-logs N]
python3 .github/scripts/ci_measurements.py report --cache artifacts/ci-cache --marker <name>=<commit>
python3 .github/scripts/ci_measurements.py cohorts --cache artifacts/ci-cache \
--history <scenario-history checkout> --repo <owner/repo> --output cohorts.json
python3 .github/scripts/ci_measurements.py caches --cache artifacts/ci-cache --repo <owner/repo> --output caches.jsonThe runtime workflows were renamed (Ceph short became Short, and so on
for the other six). report matches runs by the current names, so a run that
started under an old name is not counted; choose --since after the rename.
report groups executions into cohorts by the newest marker commit their
head contains and gives job counts, runner minutes, wall time, runner
acquisition delay, phase times and simulated makespans. cohorts joins jobs
to history rows by repository, run, attempt and job ID and groups them by
configuration (see Durable history and classification);
rows that disagree about one job are listed as conflicts, never merged.
caches lists each cache restore, save, import or export with an outcome
taken only from log lines (--cache-logs fetches them); a green step is never
read as a hit or a write, and a missing log is unknown.
Periodic primarily searches for flaky tests and patch-dependent differences.
The daily 04:41 UTC firing runs one bounded native allocation. Tuesday and
Friday use discovery; the other five days use rotate. The scheduled run uses
the default branch independently of candidate validation. No mode starts
native work while a Release matrix run is queued or in progress
(Runner budget); that firing is skipped.
| Mode | Selection | Invocation |
|---|---|---|
rotate |
Random 2–3 supported releases and 2–3 native batches/network selections | Five scheduled slots per week; manual default |
discovery |
Random 2–3 historical/new patches and 2–3 compact parents, plus applicable directed characterization | Two scheduled slots per week; manual |
targeted |
Explicit source ref, catalogued patch, profile IDs and optional parent tests/network | Manual failure comparison |
full |
Every required profile/network on every supported release, plus environment checks | Manual only |
representative |
Source-owned representative batches with explicit repetitions | Manual only; default one repetition |
storage |
Existing runner-storage qualification | Manual only |
Sample plan freezes the exact checkout SHA, the Ceph image references (the
tags of the selected release), parent test names, tags, network modes and the
random seed. It resolves no image: the worker pulls the tags, and Ryuk and
the images the tests name themselves, when it runs. The run identity seeds the draw, so later
runs can select a different combination even at the same source SHA. The plan is uploaded before Sequential sample starts. Every native case consumes that plan and the compiled binary's
existing source/inventory receipt; an unknown test, expanded timeout or changed
source/image selection is refused. The worker records the actual native build
string and local image/platform IDs before tests.
The worker executes batches sequentially, with independent test-owned fixtures. There is no shared-cluster manager. Module/image preparation is amortized; verified binaries can be reused only for the same source profile in this run. Only inadequate disk admission permits the existing standard hosted cleanup; sufficient headroom causes no deletion. Every actual Docker storage filesystem must pass admission, including post-image and per-batch checks. A dirty cleanup stops further native admission; the executor never prunes other resources.
All selected versions share one 110-minute deadline; the executor reserves cleanup time, and does not start a case without its full bounded native window. Core selections have a ten-minute cap and extensions at most twenty minutes, each no greater than the source timeout. A case that times out, skips, panics or fails its existing completion/checker assertions remains failed or incomplete. The source timeout and complete compiled inventory of ordinary CI are unchanged.
Workflow limits include preparation and reporting: plan 15 minutes, worker 120 minutes (including upload margin), history 15 minutes; at most 150 summed runner-minutes and three allocations per scheduled execution. This is an upper bound, not a latency measurement. Runner queue/dependency delay is separate. Full and repeated manual investigations retain their existing larger bounds.
The core reuses these existing parent tests in their declared default network:
TestClusterLifecycle: identity/readiness, RADOS bytes and topology/removal.TestRBDLifecycle: native RBD operations and data.TestCephFSFilesystem: native filesystem I/O and topology/removal.TestRGWS3: real S3 operations through RGW.TestClientIdentities: authentication/capability refusal and working controls.TestCephFSOSDRecovery: OSD interruption/recovery and retained exact bytes.
rotate draws from required native batches; discovery draws from the six
compact parents above. Each draw uses the same selected batches/parents across
its 2–3 versions, with the declared network selection frozen before execution.
Alternate network modes keep the existing network-test scope, and checker
profiles retain their complete parent set. Environment/code checks remain in
their existing workflows. There are no cost bins, exhaustive rotation ledger,
or fixed coverage/revisit guarantees. A random sample can repeat or miss a
version, batch or network across successive executions.
A Go-process start checkpoint distinguishes an actual native attempt from a preparation failure or cancellation before execution. Native failures, timeouts, incomplete work, unavailable artifacts and ineligible environments remain separate outcomes. The first failure remains recorded; there is no automatic retry until green. Dirty cleanup stops the remaining versions too.
Frozen plans and packed receipts live under periodic/ on the existing
scenario-history branch, keyed by run, attempt and release, with a separate
bundle receipt retaining unstarted versions. They retain the source and build
identities, the image references with what the engine reported for them, the
selected tests and the preparation/native/cleanup outcomes. They are not
ingested as full-profile passes by the ordinary same-configuration classifier.
Every execution uses its selected source; an old SHA is not held while waiting
for coverage. A sample is never proof that current main passed every version.
Historical discovery uses a separate inventory in
.github/ceph-discovery-images.json: official aggregate images for Squid
19.2.0–19.2.7, Tentacle 20.2.0–20.2.5 and Umbrella 21.2.0, each named by its
release tag. The 20.2.5 row is the upstream aggregate
(quay.io/ceph/ceph:v20.2.5) until a role aggregate is published; other
patches use the published official-<release>-all tag, and the inventory
refuses a row whose reference is anything else. Each cell keeps one
release for every role/client.
Only linux/amd64 executes by default; arm64 manifest availability is not native
arm64 validation. Umbrella's official amd64 images require x86-64-v3; a failed
CPU/platform preflight is an environment result, not a Ceph failure. Missing
artifacts and inapplicable extra-image contracts remain
preparation/eligibility outcomes rather than native passes.
Discovery does not add historical patches to SUPPORTED_RELEASES or change
support policy. Add a newly published patch to the discovery inventory to make
it eligible for random selection; use targeted for an immediate check.
The discovery_schedule characterization on 19.2.2/19.2.3
(rbd_schedule_remove) records
malformed schedule removal with empty and populated schedules for both
rbd_support schedule handlers, trash purge and mirror snapshot, which share
one removal path. It asserts the refusal with a schedule present and preserves
native exit codes and unchanged-state controls in the durable receipt:
characterizing an old success on invalid input is not claiming that behavior is
correct. Discovery also samples the existing RGW multisite bootstrap contract on
19.2.5/19.2.6/19.2.7, preserving the documented 19.2.6 failure as a control.
Investigate meaningful differences with scoped issues. Preserve the first failure, then compare the same test at the recorded source with an adjacent patch or control. No automatic full-matrix expansion, retry-until-green, support-policy change or compatibility workaround follows a discovery result.
# Profile IDs come from the plan or the source-owned vocabulary.
gh workflow run "Periodic" -f mode=targeted -f ref=<original-sha> \
-f patch=19.2.2 -f profiles=<exact-profile-id> -f tests=<parent-test> -f network=host
gh workflow run "Periodic" -f mode=targeted -f ref=<same-original-sha> \
-f patch=19.2.3 -f profiles=<same-profile-id> -f tests=<same-parent-test> -f network=host
# Complete release work is explicit:
gh workflow run "Periodic" -f mode=fullThe requested source must contain these planning/execution tools. Local
periodic_execute.py execute --trial records working-tree provenance rather
than claiming a verified Git checkout. Measure downloaded packed receipts with
the existing tooling:
python3 .github/scripts/ci_measurements.py periodic --receipts <artifact-directory> --output <report.json>This reports observed preparation/compile/native times per version; shared runner allocation and queue timing remain unknown in these receipts.
SUPPORTED_RELEASES in .github/scripts/run_image_matrix.py lists each
supported release with its codename and official image tag (Tentacle
20.2.4, Umbrella 21.2.0 and Squid 19.2.5). The manual-full periodic and candidate Releases jobs generate the callers:
python3 .github/scripts/tag_scenarios.py releases --scope all --allocation periodic --check-prefix 'Periodic'
python3 .github/scripts/tag_scenarios.py releases --scope all --allocation release --check-prefix 'Release matrix'Each release and category becomes a <Codename> <category> call of
tagged-runtime.yml; environment needs no release and stays a separate
job. The command fails unless every required profile runs on every
release of the scope exactly once, and test_scenario_shards.py checks the
same offline.
To add a release, add its row once ceph-testcontainers-images publishes its
role images under the <variant>-<release>-<role> tags, add it to the
release options of periodic.yml and the
public-api matrix of block-fault-probe.yml, which cannot read the table
(test_scenario_shards.py fails until they match), run the CI tooling tests,
and for a new major release add it to
every release-limited row of the test expectation table (see
Release differences). A release that is not
a row is refused. The default release (RELEASE, the release defaults of
tagged-runtime.yml, periodic.yml and the scenario-images action, and
ceph.DefaultImage) changes only with the current release.
CI is a consumer of the published role images like any other: it selects them by tag, and it keeps no digest of them (Image references). A compatible rebuild that ceph-testcontainers-images publishes under the same tag needs no change here. The next job that pulls the tag uses it.
image_referencesin.github/scripts/run_image_matrix.pynames the images:ghcr.io/jsyoo5b/ceph-testcontainers-images:<variant>-<release>-<role>, withallfor the aggregate.ceph.DefaultImageisofficial-20.2.4-all.- Each runtime job pulls the tags of the roles that its batch declares
(
image_roles) and runs the tests by those tags. A role that no selected batch uses, such as the optionalmgr-moduleswhen its tag is unavailable, is never asked of the registry and cannot stop the run, and a Docker-only batch asks nothing. - Nothing resolves a tag to a digest before the jobs start, and no workflow input carries one. A tag republished while a run is in progress therefore reaches the jobs that pull it afterwards, a job that is run again later included. The jobs of one run can so differ in what they ran, and each says what that was.
- Each job records what it ran:
scenario-images-*/summary.jsonhas, for every role, the tag it prepared and what the engine reported for it (the local image ID, any registry digests and the platform child).test-images/summary.jsonhas the same for Ryuk and for every image a test names itself. These are observations of that job. A history row takes the image IDs from them, so attempts on different builds of one tag count as different configurations (see Durable history and classification); nothing else reads them, and an image without a registry digest (a tag pulled through a mirror, a local build) is recorded with an empty list and runs like any other. - A periodic sample (
rotate,targeted) references the role tags that its selected entries declare, with essential always among them, pulls each once, in its single job, and records the same observations in itsimages/summary.json. - The module takes no part in this: it resolves no tag, enforces no freshness and reports no identity.
The release matrix runs for each PR targeting main, release/* or
support/* and for a push to main or support/*; a push to main that
is the merge of a validated promotion PR reuses that PR's run.
A push to release/* runs none: a release line follows develop, and
its candidate is validated once, by its promotion PR to main, instead of a
second time on the same tree.
It runs the environment profile and the required release-bound profiles on
every supported Ceph release. Neither documentation-only selection nor an
ordinary development PR changes this matrix. The target-specific check (for example,
Release candidate validated (release/1.2)) gives each branch rule a single
result for its own complete matrix. A manual dispatch emits a different
check name. The six Code checks also apply to each candidate.
For publication, verify the push event, protected branch, candidate SHA, workflow identity and successful required jobs. A PR test-merge run qualifies that proposed merge. Tag the protected commit only after its own validation succeeds, by its own matrix or by a verified promotion reuse. See the cutover and branch rules.
A promotion PR to main runs the full matrix on GitHub's test merge. The
protected merge that follows usually has the same tree and the same two
parents, and a second matrix on its push would test the same content again
with hundreds of native jobs. The push run therefore asks first whether the
promotion run already qualifies the pushed commit:
Releasesplans the matrix as always and writes its coverage: the required cells, the image references, and one digest of both.- On a promotion PR to
main, onceRelease candidate validated (main)has passed, theCandidate receiptjob writes the artifactrelease-candidate-receipt-<attempt>with two files. The receipt names the commit that the workflow checked out with its tree and parents, the PR, its head and target, the run and attempt, the coverage, the image references and the jobs whose first failure a release retry passed. The evidence file holds the history row of every runtime job (what it checked out, built and pulled, the image IDs and digests it resolved, each phase's result, its cleanup check and the hashes of its logs and reports) and the report of each retry. The jobs' own artifacts expire after 7 to 14 days; the receipt and this evidence are kept for 90. The job fetches the rows through the API, page by page, because a release run writes more artifacts thanactions/download-artifactlists, and it fails when the record does not account for every runtime job or the checked-out commit is not a merge of two parents. The gate's result stays as it is; the failed job says before the merge that the push will run the full matrix. - On a push to
main,Promotion reuserunsscripts/release/promotion_reuse.py verifyand writespromotion-reuse-receipt-<attempt>with its decision and, on a reuse, its own copy of the evidence file.
The push reuses the promotion run only when all of this holds:
| Check | Source |
|---|---|
| The pushed commit is a merge of two parents, the merge commit of exactly one merged PR that targeted this branch from a branch of this repository, and its second parent is that PR's head. | Repository and PR API |
A pull request run of release-matrix.yml for that head, of that head branch and repository, passed in its latest attempt, with the target's Release candidate validated job and every other job successful. |
Actions API |
| That attempt's candidate receipt exists, is not expired, and names the same run, attempt, PR, target and head. | Artifact |
| The receipt's tested commit, read from the repository, has the tree and parents that the receipt records, and they equal the pushed merge's tree and parents. | Repository API |
The receipt's coverage digest equals the digest that Releases computed for the pushed tree. |
This run |
| The evidence file is the one that the receipt records, and it has exactly one unaltered history row for every runtime job of the run. Each row is of this run and the tested commit, has complete records and a passed cleanup check, names the images it ran if it needed Ceph, and either passed or failed only its native phase and passed its one retry, by the retry's own report and the job's retry step. | Artifact and Actions API |
A live receipt therefore never qualifies a run by itself. Without the evidence file, or with an entry that is missing, foreign, altered, failed or uncleaned, the full matrix runs. The run's shorter-lived artifacts are not read, so their expiry changes nothing.
The run's reported head alone proves nothing about what was tested; the receipt's tested commit, checked against the repository, does. The receipt is trusted because the workflow that wrote it came from the merged PR's head merged into the protected branch. A failed retry, a cancelled or incomplete run and a skipped job never qualify.
When the push reuses the run, the runtime and environment callers are
skipped and Release candidate validated (main) passes on the reuse receipt.
In every other case, including an unreadable fact, an expired receipt or a
failure of the Promotion reuse job itself, the full matrix runs and the
gate needs it to pass; the reuse receipt then lists the reasons. On a reuse
the receipt also summarizes what the jobs ran: every image reference with
the image IDs and digests it resolved to, and the Go toolchains. The full
native logs are not kept beyond their own artifacts' retention; the evidence
holds their hashes.
The reuse path only runs for a protected merge, so it cannot be tried on a
feature branch, and trying it first on a real promotion would make a release
the test. The branch rehearsal/promotion exists for that: release-matrix.yml
treats a PR to it and a push to it as it treats main, with the gate
Release candidate validated (rehearsal/promotion), and
promotion_reuse.py applies the same checks. A receipt names its target, so a
rehearsal never qualifies a push to main, nor the reverse. Nothing is tagged
or published from the branch, and no other workflow runs for it.
A rehearsal tries the candidate record and the reuse decision, not the
coverage of a release, so its matrix is small: tag_scenarios.py releases --scope rehearsal plans one caller, REHEARSAL_CATEGORY on the current
release, beside the environment caller. A run takes minutes instead of
hours. What only the size of a release run shows (its number of artifacts,
its runner demand) is not rehearsed: for the runner demand dispatch
Release matrix by hand, and check the record of a large run by recording
it again from a depth-1 clone of its tested commit with
promotion_reuse.py record.
The branch is not kept. To rehearse a change of the receipts or the reuse decision:
- Create
rehearsal/promotionat the base to promote onto, with a ruleset for it alone that requires a pull request and forbids deletion and non-fast-forward pushes. - Push the commit under test to a branch of its own and open a PR from it
to
rehearsal/promotion, before that commit is the head of any other open PR. Two open PRs with one head commit make the checks that publish to the PR of a commit (the structural review comment) refuse. When the gate has passed, check thatCandidate receiptsucceeded and wroterelease-candidate-receipt-<attempt>. - Merge the PR with a merge commit. The push run must end with
Promotion reusesuccessful, the runtime and environment callers skipped and the gate passed, and itspromotion-reuse-receipt-<attempt>must sayreusedand name the PR's run. - For the other path, merge a second PR whose Release matrix has not passed
(cancel it first). The push run's receipt must say
full-matrixwith the reason, and the callers must start; cancel that run once they have. - Delete the ruleset and the branch.
A push to support/* always runs its full matrix. The six Code checks run
on the pushed commit in either case.
A release candidate is judged by more than five hundred native jobs, and the
validation passes only when every one of them does. A rare intermittent
failure in any of them would block every candidate. Each job of a release
matrix (a caller with require_profiles) therefore runs the parent tests
that failed once more, in the same job and on the same runner, and the job's
result is that second run's:
tag_scenarios.py retryreads the first run'snative/report.jsonandnative.log. It retries only a failure of named tests (native-test-failureornative-registry-transport) whose process ended as Go ends failed tests: exit status 1 and a finalFAIL. It refuses one after a build or process failure, a skipped test, a panic, a fatal runtime error or a signal (any of which can end the binary or crash it beyond its failed tests), or a parent that neither failed nor passed. A checker profile retries its whole run, because its checker validates one complete run.- Before it starts it waits up to 120 s for the first run's testcontainers containers, Ryuk aside, to be gone.
- Its timeout is the profile's process timeout, bounded by the second budget
that the job keeps: a release job's
timeout-minutesisretry_job_timeout, twice the source'sjob-timeoutup to 360 minutes, and the retry uses what lies beyondjob-timeout, less five minutes for the steps after it. - The retry writes
native-retry/report.jsonand its run undernative-retry/attempt/. The first run's receipts stay innative/, the history row records the first failure, the native phase annotation becomes a warning, the failure reports still run, and the artifact is kept for 14 days. - A refused or failed retry fails the job. The owned-resource cleanup check runs after the retry and covers both runs.
A first failure that its retry passed is still a failure of that test. It is recorded in an issue like any other (see Releasing), and it does not block the candidate. Development workflows, Periodic and the CI command never retry.
The campaign repeats a source-owned set of batches on one pinned commit to
build same-configuration pass and fail records. The set is
every batch whose //ci: line declares representative=<theme> (see
Testing); plan --representative fails
if a theme (topology, auth, mirroring, recovery, flaky) has no
batch.
It runs only through an explicit dispatch with mode representative:
Pinresolves the commit (therefinput or the run's commit).Representativecalls the template withcategory: representative. Each entry repeatsrepetitionstimes (default 1, at most 10) as separate jobs named<batch> #<n>, artifacts ending in-r<n>, each checking out the pinned commit and pulling the role tags of the release. History rows carryrepetition_id<n>and the image IDs each attempt ran, so a tag rebuilt during a campaign splits its attempts into two configurations instead of mixing them.- Every attempt keeps the normal completion, no-skip and cleanup proofs; nothing is retried.
A campaign is bounded at 100 jobs and 3000 summed job-timeout minutes, does
not count as a matrix execution for the gate, and has its own concurrency
group.
gh workflow run "Periodic" -f mode=representative -f repetitions=5 -f release=19.2.5
gh workflow run "Periodic" -f mode=representative -f ref=<commit>ref must contain the campaign tooling, because every job runs the pinned
commit's scripts.
The ARM64 workflow runs a small source-owned cohort on a GitHub-hosted native
ARM64 runner (ubuntu-24.04-arm) with the linux/arm64 children of the
published role images. A batch joins it with architecture=arm64 on its //ci: line
(see Testing). The cohort is
default and network_faults of ./internal/integration: cluster startup and
bootstrap failure cleanup, the Ceph tools in the role containers, pool and
placement observation, and the netem helper image. The cgroup, device-mapper
and clock-interposition recipes stay out of it until their host
prerequisites are established on that runner.
python3 .github/scripts/tag_scenarios.py plan --architecture-cohort --goarch arm64 --output plan.json
gh workflow run "ARM64" [-f ref=<commit>]The workflow calls the template with category: architecture and
goarch: arm64. The cohort plan claims only its batches (coverage_scope
architecture); every category workflow still runs amd64, the default, and
none of them depends on the ARM64 workflow, so amd64 acceptance stays
attributable on its own. plan --goarch arm64 without --architecture-cohort
is refused, so no caller can run an ordinary category on ARM64 by accident.
The ARM64 workflow runs weekly, on a pull request that changes the planner, image or storage
preparation, the runtime template, the bootstrap scripts or the netem helper,
and manually. It is not a required check, and its rows are not written to
the scenario history.
The architectures are recorded separately, each by the phase that observes it:
| Fact | Receipt |
|---|---|
| Planned architecture | tag-plan.json (goos, goarch) and each matrix entry's goarch |
| Go toolchain host and target | compile-report.json platform (planned, host, target) and the binary's build key |
| Docker engine | runtime-cleanup engine identity and scenario-images/summary.json docker |
| Selected image | each image's selected_platform (the child Docker chose) beside its reference and image_id in scenario-images/summary.json and test-images/summary.json |
Compilation refuses a runner whose Go toolchain is not natively the planned architecture, host and target alike, so a cross-compiled binary is never built. Image preparation and test image preparation refuse an engine of another platform than the plan's and the binary's: emulation is separate evidence, not qualification.
Runner storage on ARM64 has no measured qualification:
runner_storage.QUALIFIED_STORAGE_PLATFORMS holds linux/amd64 only. On
another engine the layout is unverified (docker-platform-not-qualified), the
job keeps the standard cleanup and the conservative minimum headroom guard,
and a cohort budget of .github/runner-storage.json measured on another
platform is refused (budget_refused: engine-platform-differs) instead of
granting its allowance. The storage-*.json receipts of each run record the
ARM64 runner's actual filesystems and layout reasons; a skip shortcut or a
pressure policy on ARM64 needs a linux/arm64 budget measured from them.
Read a run by its phases, not its colour. An unavailable runner leaves the
job queued or failed before any step; a compile, storage or image refusal is
an environment result; a native failure, timeout or --- SKIP fails the
native phase as on amd64; cleanup is checked on the same runner. A skipped or
cancelled job is not qualification, and a pass qualifies only the cohort's
batches on the recorded release, images and runner.
Optional (ci_optional) batches are outside every required check. Run one
with tag_scenarios.py optional --batch <name>, with /ci run <name>, or
through Native regressions, which runs the one batch its batch input names
and takes only its pass or failure. The result of a known-defect reproducer is
checked by Experiments (see Experiment case tiers).
The options of the workflow's batch input are the batches it accepts, and
the source decides what each one runs. To list them, and to find the case
manual of one:
# The choices of the dispatch, failure_archive included.
sed -n '/^ batch:/,/^ rgw_image:/p' .github/workflows/native-regressions.yml
# The test files, and with them the case directory and its README.md, of one batch.
grep -lw ci_batch_rgw_empty_sse_header internal/integration/experiments/*/*_test.go
# Dispatch one.
gh workflow run "Native regressions" --ref main -f batch=rgw_empty_sse_headerMost choices are experiment cases: each case's
README.md gives its trigger, controls, command and the meaning of each
outcome. A batch outside internal/integration/experiments is described with
its feature below. A new optional batch joins the dispatch when it is added to
the batch options; test_scenario_shards.py keeps the expected list and
checks that every choice is an optional batch of the source.
native_rgw_translation also tests a patched RGW image; an empty
rgw_image uses the published role image, and the workflow never builds one:
gh workflow run "Native regressions" --ref main -f batch=native_rgw_translation \
-f rgw_image=ghcr.io/example/ceph-rgw:patchedcephfs_fscrypt (TestCephFSFscrypt) is another optional experiment case,
cephfs_fscrypt,
that passes on the published images. Its client containers mount CephFS
with the engine's kernel and with ceph-fuse and get CAP_SYS_ADMIN and
apparmor=unconfined; the kernel subtest also needs a host kernel that has
the ceph filesystem type registered. It refuses, naming the missing
requirement, and never loads a module or skips. Outside the tag runner,
build it with go test -tags all,cephfs_fscrypt.
The following optional batches are not experiment cases. They are a
selection: the source declares every optional batch (see
Optional tests), and the timing and fault
batches are described with their features, for example in
Daemon timing and the
fault/blockio README.
cephfs_fuse (TestCephFSFuseMount) mounts CephFS with ceph-fuse in
client containers given /dev/fuse, CAP_SYS_ADMIN and
apparmor=unconfined. It needs a rootful engine whose kernel has FUSE: a
Linux Docker Engine, or Docker Desktop (it passed with kernel
7.0.14-linuxkit). On an engine that refuses the mount it fails naming the
missing requirement and never skips. Outside the tag runner, build it with
go test -tags all,cephfs_fuse.
rolling_upgrade (TestRollingUpgradeSquidToTentacle) upgrades a 19.2.5
cluster to 20.2.4 one daemon at a time with the published official role
images of both releases, which it names by tag; see
Rolling upgrade from Squid to Tentacle.
It pulls the images of both releases whatever release the profile names.
desktop_capabilities (TestDesktopRBDOverNBD) maps an RBD image through
rbd-nbd in a privileged helper from the official Ceph image and checks the
bytes through the device and through librbd. It skips when the engine kernel
has no loaded NBD driver or no /dev/nbd nodes, and never loads a module. It
has been run on Docker Desktop, whose kernel has NBD built in.
zone_boundary (TestZoneBoundaryRegionalCluster,
TestZoneBoundaryTwoSubnetRouting) splits one cluster into two logical
availability zones behind caller-owned routers on its public and cluster
bridges, qualifies the boundary's round trips with netem, partitions it and
reads the data back after recovery, and records whether the engine forwards
between two Docker bridges through a container. Its helpers and routers run
with NET_ADMIN and namespaced sysctls. It is the research proof behind
ceph.WithZoneConfig, not the way a test builds zones; see
Hand-built routers
for what it does and Regional zones for the
public fixture.
osd_encrypted_store (TestEncryptedBlockDeviceOSD,
TestEncryptedBlockStoreKeepsACollidingMapping) runs one BlueStore OSD on a
linear device-mapper node over a test-owned LUKS2 dm-crypt mapping, locks and
unlocks the store with a key only the test holds, and requires the original
OSD, store and bytes after the unlock. A privileged keeper in the client role
image binds the engine host's /dev; its cleanup removes a mapping or loop
only while it still has the DM UUID, device number or backing inode the test
recorded, and keeps a mapping that only shares the test's name. The engine
host's kernel must have the device-mapper crypt target registered; the test
fails naming it and never loads a module, except on a GitHub-hosted runner.
Outside the tag runner, build it with go test -tags all,osd_encrypted_store
(see Testing).
failure_archive is described in
Controlled failed-native download proof.
An experiment case runs where its cost and determinism allow, through the
existing triggers only. The tiers are listed in
Experiment cases. Each case's README.md
records its tier, its batch and budget (the //ci: timeout), its measured
runs and the triggers that apply to it, so adding a case changes nothing
here.
- A cheap deterministic case is a required batch of a non-
shortcategory, so a pull request runs it only when it changes a file of the case's directory (see Pull request scope), and periodic runs keep it covered on the default branch. A change to a public package or the harness selects none of these batches; the fixture behaviour they use is covered by the ordinary batches of its package (rbd_schedule_removecalls no fixture API beyond starting the cluster). - A known-defect reproducer prints
REPRODUCEDorNOT_REPRODUCEDand passes either way; only its given and when steps fail it. An expensive one stays an optionalnative_regressionbatch that runs only on request.Experimentsruns one case withgo test: on the case's default image every result line must beREPRODUCED. When a fixed image is given, the case then runs again with it injected and every result the default run reproduced must be printed again, by the same name, asNOT_REPRODUCED. The default run stays as the control, so a trigger that stopped eliciting the defect on every image fails the dispatch, and so does a fixed run that never evaluates the reproduced defect. A result only the fixed run reaches (rbd_mirror_demote_restart/resumed-bytes, evaluated once the copy resumes) is printed and not judged. A case whose defect has no fixed release has only the default run. The cluster and every client a case starts run its one image, whatever role images the batch environment names.Native regressionsand/ci run <batch> on <release>still run the batch, but take only its pass or failure. - An expensive case that passes on the published images would be scheduled
through the periodic
discoveryselection (periodic_selection.py), not through pull requests. No current case is in that tier.
-
Find the batch. The check name gives the category and batch, for example
Recovery / Scenarios / container_pause. -
Read the annotations. A phase annotation names the failed step and the profile ID;
Go test resultannotations list each failed test as<profile id>: TestName/subtest, or say that none could be identified (for example after a kill).gh run view <run-id> --log-failed gh api 'repos/{owner}/{repo}/check-runs/<job-id>/annotations'
-
Check the job summary. It shows each phase as
success,failure,cancelled,skipped,not runornot required. A phase shows where execution stopped, not the root cause. -
Download the artifacts.
gh run download <run-id> -n tag-scenario-<profile-id>-<release> -D artifacts/ci-<run-id>
<id> is the matrix entry's profile ID, <release> the Ceph release such as
20.2.4, and a repeated campaign entry adds -r<n>.
| Artifact | Contents |
|---|---|
tag-scenario-<id>-<release> |
modules/ and compile/ logs and reports (the binary is not uploaded); native/native.log; native/report.json; native/test-artifacts/ (retained test artifacts such as failure archives); test-images/ (the Ryuk and test image receipt: the analysis, each image with the tests and source positions that named it, skipped references, pull attempts); runner-storage/ (see below); phases.json; history-row.json |
runtime-cleanup-<id>-<release> |
before.json and after.json: engine identity and labelled resources before and after |
runtime-cleanup-<id>-<release>-baseline |
Only when the baseline itself failed |
scenario-images-<id>-<release> |
summary.json with each role's tag and what the engine reported for it (image ID, any registry digests, platform), every pull attempt under pulls, and the pull-<role>[-<attempt>].log files |
plan-<category>-<release> |
tag-plan.json (selected profiles) and tag-dispatch.json (the expanded matrix with network scope, repetitions, run, attempt, release and category) |
code-tag-coverage |
tag-plan.json with compiled coverage and the code batch results |
Uploads are retained for 7 days, or 14 days when the job (or the uploading composite action) has already failed at upload time.
native/report.json:
classification:passed;native-test-failure(a test failed, panicked or hit the Go timeout);native-registry-transport(every failed test failed on a Docker daemon registry transport error in its own output, such as testcontainers-go pulling Ryuk or a test's own image, so no Ceph assertion of it ran; scenario history counts it as an image failure);incomplete-skipped-fixture(a skip);process-or-build-failure(non-zero exit without a failure line);incomplete-native-proof(exit 0 but a pass rule or strict checker failed);completion.errors,completion.failed_tests,completion.skipped_tests;registry_transport: each failed test with such an error, and its stage (reaperorimage), reason, registry phase (resolve,token,manifest,blob,registryfor another/v2/request) and endpoint without query or user information; phase and endpoint areunknownfor a 5xx answer that the daemon reports without its request;compiled_tests,command,elapsed_seconds,exit_code,strict_completion;test_artifacts: files left innative/test-artifacts/with sizes and the number of failure archive manifests.
modules-report.json and compile-report.json record exit_code,
timed_out, timeout_seconds, elapsed_seconds and a classification of
prepared/compiled, modules-failure/compile-failure,
modules-timeout/compile-timeout, or compile-receipt-failure (the build
succeeded but the receipt was not recorded, so the binary is not reused). A
timed-out command is killed with its process group and has exit_code: null.
runner-storage/storage-prepare.json records the observed Docker storage roots,
their independent filesystems, required and available bytes/inodes, the
skip/reclaim decision, cleanup duration and post-check. Paths on the same
device share one observation; space on different filesystems is never added.
Every candidate store filesystem must retain the default MON 30% available
share after the preparation and runtime allowances. Containerd writable
snapshots can hold the MON store on a different device from Docker's metadata
root, so the guard conservatively keeps this floor on every candidate device.
The receipt's docker_root marker identifies the metadata filesystem for the
controlled pressure file; it does not locate the MON store.
storage-ready.json repeats the check after actual image pulls, retaining the
runtime allowance. Failure of
either guard stops the job before native assertions. A qualification names
no image digest: the role images are the release's published tags, the
image-preparation allowance bounds what their pull may take before it, and
this post-image check is what refuses a job whose images, for example after
a republished tag grew, left less than the runtime allowance.
The qualifications in .github/runner-storage.json cover these Tentacle AMD64
profiles. Each has an 8GiB image-preparation allowance and 100,000 free inodes,
additional to the per-filesystem MON share and its runtime allowance.
| Profile | Maximum simultaneously live logical OSD backing | Runtime allowance |
|---|---|---|
rados_fixtures_watchers |
Two 1GiB OSDs | 4GiB |
cephfs_fixtures_data |
Two parallel fixtures with up to three 1GiB OSDs each | 8GiB |
cephfs_fixtures_admin |
One sequential fixture with up to two 1GiB OSDs | 8GiB |
rbd_fixtures_mirror |
Two clusters with two 1GiB OSDs each | 8GiB |
RBD default |
Two parallel fixtures: lifecycle with up to three 1GiB OSDs and namespaces with two 1GiB OSDs | 8GiB |
The data profile retains the three OSDs required by its erasure-coded pool assertions; its host-network topology still adds and removes an OSD. Qualified startup paths retain 1GiB stores and skip calibration. The RBD default profile pins its startup and topology helpers, including the default OSD size and count; its lifecycle assertions still add a third OSD. Logical capacity is a conservative source bound, not reserved physical space or a measured runtime peak. Control stores, client data and logs are additional; the qualification evidence records sampled filesystem consumption separately.
The catalog selects exactly one entry for the compiled profile. Its workload
sources, listed startup/data helpers, dependency manifests, tags, selected
tests, release, declared roles and platform must match. Changes to a relevant helper expire that workload's
qualification. Unrelated profiles cannot inherit its allowance, and an ambiguous
or missing entry retains conservative cleanup. The data and RBD default qualifications
also require CEPH_TEST_PARALLEL to be at most two; higher or invalid values
retain conservative preparation instead of inheriting their two-fixture bounds.
Other profiles and releases remain unqualified. Image and native comparison
evidence belongs in the implementation pull request.
Only verified local classic-overlay2 or socket-linked containerd-overlayfs
layouts may skip. Existing local volume mounts and nested mounts are inspected.
Containerd must have one running process whose CLI/configuration root and socket
match Docker's reported socket. The built-in defaults are not inferred, so a
package-default containerd whose root and grpc.address are left unset always
retains cleanup; imported/proxy configuration and snapshotter
root overrides keep conservative preparation. Unlinked roots are recorded as
candidates and cannot authorize skipping. Remote endpoints,
unknown roots, volume plugins/options and missing evidence keep conservative
cleanup. The post-cleanup guard for an unqualified workload is explicitly a
minimum safety check, not a claim to know its peak.
storage-free.json records the ordinary fixed allowlist removal when needed:
Android, .NET, Haskell, CodeQL and Boost. Only if its post-cleanup headroom
still fails does a second fixed list remove unused Swift, PowerShell and vcpkg
installations. storage-free-supplemental.json records that additional action;
storage-prepare.json retains the standard-cleanup observation and both action
durations. Qualified sufficient headroom skips all removal; sufficient standard cleanup
skips supplemental removal. Go, the system Python interpreter, runner files and
Docker/containerd data are retained. A smaller hosted filesystem uses the same
MON reserve and workload allowances: it must recover enough actual space from
these directories or fail admission. Neither the runner label nor successful
directory removal guarantees sufficient space; the post-cleanup and post-image
checks still decide from each filesystem's observed bytes and inodes.
storage-images.json and storage-after.json retain the Docker-root filesystem
and docker system df observations after pulls and tests. These endpoints
separate image and run allocation from runner baseline; they do not establish
a transient peak or reclaim historical preparation time.
For a controlled hosted comparison, dispatch Periodic with mode=storage
and the same commit ref, release and storage_policy=auto, always,
pressure or pressure-supplemental. The storage_profiles input selects comma-separated source-owned profile IDs,
defaulting to the watcher profile. Up to five selected jobs run concurrently,
retaining ordinary assertions and cleanup checks; repeat dispatches for each
policy on the same source and selection. Pressure refuses an unqualified
profile before allocating a file.
always is the reclamation control. pressure creates one exclusive owned file
on the Docker filesystem to cross the admission boundary, retaining the MON
floor, runtime reserve and 2GiB extra; it refuses allocations outside 1–40GiB
or any non-hosted/unqualified layout. The file stays through native assertions
and an always-run step removes it only when its device/inode receipt matches.
The pressure file is also excluded from artifact uploads.
pressure-supplemental first performs standard cleanup, then holds a pressure
file up to 64GiB to leave a 4GiB admission shortfall while retaining the same
MON floor, runtime reserve and 2GiB extra. Standard cleanup cannot recover the
same SDKs twice, so the extra list must supply the missing headroom. If standard
cleanup already leaves a shortfall, no pressure file is allocated. Its ordinary,
supplemental and pressure receipts remain separate. This mode is an explicit
hosted qualification control; normal auto jobs never allocate pressure files or
preclean before their admission decision.
This qualification never fills a shared host or intentionally approaches ENOSPC.
For an end-to-end comparison, dispatch Short at one fixed commit with
storage_policy=auto or always and the same ref. It executes the complete
short selection with the usual network modes and concurrency, including
unqualified workloads. Use its measured start/end times and final job to assess
workflow completion, rather than summing avoided preparation seconds. The
manual control cannot change the policy or checkout of push/PR events.
Normal schedules, workload coverage and runner concurrency limits are unchanged.
A storage dispatch of Periodic shares the matrix's concurrency group (see
Concurrency and cancellation).
| Failed phase | Look at | Common causes |
|---|---|---|
| Go module preparation | modules/modules.log, modules/modules-report.json |
Proxy and origin both failed, or go.sum lacks an entry. |
| Compilation | compile/compile.log, compile/compile-report.json |
A test calls a helper that only another batch compiles; run plan --verify (see Testing). |
| Docker baseline | the step log, runtime-cleanup-<id>-<release>-baseline |
Docker was not ready within 60 seconds. |
| Published image preparation | scenario-images-<id>-<release> |
A role image is missing from the registry or could not be pulled. classification: registry-transport and registry_failure (image, reference, reason, phase, endpoint, attempts) mean the registry was not reached after the bounded retries; nothing about the image is established. |
| Ryuk and test image preparation | test-images/summary.json |
The same for Ryuk or a test's own image. An error that a test completes an image prefix this preparation cannot name means a test builds an image reference that the analysis does not know; give it a constant or extend plan_images. |
| Native assertions and completion | native/native.log, native/report.json |
An assertion failed, a Ceph wait or the batch timeout expired, or a test skipped. A panic: test timed out block lists the tests still running. A health wait that sees only MON_DISK_LOW points at runner storage: read runner-storage/. native-registry-transport points at a registry that a test's own pull could not reach: testcontainers-go does not retry the daemon's HTTP 500 for it, and no Ceph assertion of that test ran. |
| Docker resource cleanup | runtime-cleanup-<id>-<release>/after.json |
A test left a container, network or volume behind, for example by not registering cleanup when Run returned an error. |
Failed integration tests print the last 16 KB (8 KB in the multicluster and
host-network helpers) of every daemon's log into
native.log before the cluster is removed; search for container ... logs:.
dispatch_join.py joins a downloaded tag-dispatch.json offline to a saved
jobs-API response of the same run and attempt, optionally with
history-row.json receipts:
python3 .github/scripts/dispatch_join.py --dispatch tag-dispatch.json \
--jobs jobs.json [--receipts downloaded-artifacts] [--caller "Tentacle short"]Jobs match entries by name. Every entry, job and receipt ends in exactly one bucket (matched, conflict, ambiguous, unmatched, invalid or out of scope); exit 3 means the accounting failed and exit 2 that the input was incomplete. A match proves neither runner allocation nor native execution, and the tool is not wired into any workflow.
The runner passes Go's artifact flags in both execution modes
(-artifacts -outputdir=<native>/test-artifacts, or the -test. forms for
the binary). A test that finalizes its cluster with
finalize.TerminateOnCleanup and Archive
writes its failure archive under
native/test-artifacts/_artifacts/<package>/<test>/<random>/ceph-<cluster>-<execution>/
with diagnostics.json, crash-metadata.json and manifest.json. A passing
test leaves no directory.
Creation, upload and availability are separate facts. The test log and the
manifest record what each collector wrote, and test_artifacts lists the
files present after the run. For a failed native step, the job summary's
Failure archive section records the manifest count, the upload outcome and
the artifact ID. Only downloading the artifact and reading its manifest shows
that the archive was retained. An archive is a snapshot taken before
termination. A test that is still running near the profile's timeout
writes a before-deadline archive before Go's timeout panic, which runs no
cleanup; its manifest is never final. Nothing is archived when the runner is
lost, the job is cancelled or the process is killed: the test binary exits
on the cancellation's SIGINT without cleanups, and Ryuk removes the
containers about ten seconds later, so a later step would race it and is not
a supported collector.
archive_audit.py audits a downloaded archive directory without changing
any verdict:
python3 .github/scripts/archive_audit.py <archive-directory> --output audit.jsonIt reports per file and per archive the containment, recorded write outcome,
JSON readability and size agreement, and joins each capture_files record to
its file entry. evidence_complete is true only when the producer declared
the archive complete, the manifest is final with a termination, and every
file and capture copy is verified. A producer-declared failure is never
reported as download corruption.
The optional failure_archive batch runs two deliberately failing native
subtests (an ordinary assertion and a finalization check) through the real
testing.T, each producing an archive with its trigger, snapshot and debug
restore receipts. Run it only on explicit request:
gh workflow run native-regressions.yml --ref <branch> -f batch=failure_archiveThe native step and the workflow must fail. Then download the attempt's artifacts by their exact names and audit them from a checkout of that head:
gh run download <run-id> -n <tag-scenario-artifact> -D artifacts/archive-proof/scenario
gh run download <run-id> -n <scenario-images-artifact> -D artifacts/archive-proof/images
gh run download <run-id> -n <runtime-cleanup-artifact> -D artifacts/archive-proof/cleanup
python3 .github/scripts/failed_archive_proof.py \
--scenario artifacts/archive-proof/scenario \
--images artifacts/archive-proof/images --cleanup artifacts/archive-proof/cleanup \
--root . --revision <full-checked-out-commit> --run-id <run-id> --attempt <attempt> \
--output artifacts/archive-proof/audit.jsonExit 0 means archive_verification: verified with native_outcome: failed:
both failed cases, matching binary and source hashes, the prepared image's ID, both
archives with their listed files, debug receipts, successful restoration and
termination, no leaked resources, and the history row's identity. Do not mix
artifacts of different attempts. The audit does not prove the upload itself;
keep the upload step and artifact ID as separate evidence.
scenario_history.py turns one job's receipts into one row of schema
ceph-scenario-history/v1. It reads files only and never infers a cause. The
workflow writes it as history-row.json; to build one from downloads:
python3 .github/scripts/scenario_history.py normalize \
--scenario a/scenario --cleanup a/cleanup --images a/images \
--repository <owner/repo> --run-id <run-id> --run-attempt <attempt> --job-id <job-id> \
--output row.jsonA row holds:
execution_key: repository, workflow, run, attempt, job, repetition, profile ID, checker, case and network mode;source: the requested head and the checked-out commit and tree;build: build key, binary sha256, a digest of the source hashes, and the native phase's binary identity (native_binarysays whether they match);profile: profile identity, tags andbug_compatibility. Receipts that disagree on tags, or whose selection differs from--bug-compatibility, write no row;runtime: release, layout, platform and, for every role, the reference and the image ID that the job's engine reported (and any registry digests it listed, which may be none), ornot_applicable;phases: per phase the step outcome, the receipt and their join. A pass needs both; a missing receipt ismissing, neverpassed;outcome.status(failed,cancelled,passedorincomplete) andfirst_failure;tests: every RUN, PASS, FAIL and SKIP line with its duration;comparability:complete, orincompletewith what is missing;evidence: the sha256 of every input androw_sha256.
python3 .github/scripts/scenario_history.py ingest --history history.jsonl row.json
python3 .github/scripts/scenario_history.py summary --history history.jsonlingest skips identical rows and refuses a different row under the same
execution key or a row without run identity. summary counts outcomes per
profile and release, failures by phase and classification, and p50/p90 native
durations; cancelled rows are left out of the failure rate.
The History job of every Periodic run that executes the matrix or a
campaign runs scenario_history.py collect on a checkout of the
scenario-history branch. It appends the run's rows to
rows/<run-id>-<run-attempt>.jsonl, rewrites summary.json and pushes. A
conflicting row fails the job and changes nothing.
summary.json (schema ceph-scenario-history/v1/classification) has, per
profile and release, the last 20 attempts and the 10 most recent
configurations. A configuration is the profile, network mode, release,
checked-out tree, build key (which includes the bug compatibility selection),
platform and every role image ID; rows without a complete identity count as
incomparable_rows. Each configuration records passes, failures, excluded
attempts by kind, the first failure, per-test failed/ran counts (only
attempts whose log has the test's result) and a classification.
passed, assertion_failure, timeout and cleanup_failure are scenario
outcomes. runner_failure, image_failure, cancelled and incomplete are
excluded.
| Classification | Scenario outcomes of the configuration |
|---|---|
insufficient_samples |
fewer than two |
confirmed_pass |
two or more, all passed |
deterministic_failure |
two or more, all failed with the same phase, test and classification |
confirmed_fail |
two or more, all failed, in different ways |
suspected_intermittent |
at least one pass and one failure |
git fetch origin scenario-history
git worktree add artifacts/history origin/scenario-history
python3 .github/scripts/scenario_history.py classify --history-dir artifacts/historyWhen a runtime job fails, report_test_failures.py reads summary.json and
adds Scenario history notices beside the failed tests, such as
TestClientRevocation/aes failed 2/10 earlier same-config attempts that ran it, with the configuration's classification. The notices never decide the
job's result.
A release matrix job already retries its failed tests once (see Release retry). Re-run only the failed jobs when the failure looks unrelated to the change:
gh run rerun <run-id> --failedTo reproduce a batch locally with the same tags, budget and pass rules, run the tag runner; for identical images, first follow Reproducing CI's role images:
python3 .github/scripts/tag_scenarios.py run \
--category recovery --batch container_pause --package ./internal/integration \
--directory artifacts/repro-container-pauseLocal runs are sequential unless CEPH_TEST_PARALLEL is set. To isolate one
test, use go test -tags=all -run (see Testing).
The Structural review evidence workflow collects report-only type and lint
deltas for pull requests that change library code, starting with the pull
request event. Its job summary and structural-review-RUN_ID-ATTEMPT artifact
retain named base/head evidence. A trusted workflow_run publisher updates
one current-head PR comment as soon as that run completes, without waiting
for Code checks. Collection/upload are advisory and do not change required
checks, documentation-only policy or native selection. See
Structural review for identity checks, limits, artifact
contents, exclusions and local commands.