Repository navigation
Prototype: re-render the build workflows from the recipes (conda-forge style) - #229
Draft
EzraBrooks wants to merge 9 commits into
Draft
EzraBrooks wants to merge 9 commits into
EzraBrooks wants to merge 9 commits into
Conversation
The PR build of each distribution and platform is no longer one job that needs re-running after six hours: a plan job lists vinca-gha's stages (--json), pr-stages.yaml runs them in fixed stage slots, one matrix job per batch of recipes, and the packages pass between the stages as artifacts, read as a local channel. A collect job saves them as the PR's build cache and runs the ABI check. pr-stages.yaml is generated by tools/pr_stages.py (rs check checks it). vinca is pinned to the --json change on top of the previous pin.
…requests and releases staged-build.yaml (tools/staged_build.py) has 64 stage slots shared by all distributions and platforms instead of 64 per distribution and platform. Each job reads its recipes from the schedule, downloads only the packages of the jobs it needs (vinca-gha's needs), and fails when one of them failed, so the other distributions and platforms go on. Jobs that succeeded in an earlier attempt skip their build when later stages are re-run. release.yaml runs the same workflow with uploads, by hand while the build branches still build the releases.
download-artifact's pattern only sees a run's first 1,000 artifacts: the stage jobs and collect download the artifacts they need by name (tools/download_artifacts.cjs), and the build log is an artifact only after a failure. A failed recipe now holds back only the recipes that need it (vinca-gha's requires): the job builds the rest with --continue-on-failure=skip-dependents, and status/<job>.json records each recipe's outcome for the later jobs. Releases upload the packages of the recipes that built.
…b isn't built either rattler-build's skip-dependents skips the outputs that need a failed one to build, not the ones that only run with it, e.g. ros-<distro>-<name> of ros2-<name>.
…itted tools/rerender.py renders .github/workflows/build-<distro>-<platform>.yaml from every recipe of the distribution (generate-recipes --all): one job per content-defined batch, named after its first recipe, after the jobs it needs directly, sorted by name, so a re-render changes only the jobs a change touches. staged-plan.yaml picks the jobs with a recipe to build at run time; the others are skipped. testpr.yaml re-renders the changed distributions and commits the result to the pull request (fails for forks until `pixi run rs rerender`). Replaces the stage slots (staged-build.yaml).
This was referenced Nov 16, 2026
GitHub doesn't start a run that calls all 20 rendered workflows (startup_failure with 3 or more), so each build-<distro>-<platform>.yaml is a workflow of its own, triggered by pull_request and workflow_dispatch, with inline jobs running the composite action .github/actions/stage-job. staged-plan.yaml's select job decides whether a pull request builds the distribution and platform; build-status.yaml (tools/build_status.py) sums the workflows up in the "All builds" commit status. release.yaml dispatches the build workflows; the ABI comment is posted per distribution from staged-collect.yaml.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prototype, stacked on #216 (its commits are included; review from
a0f1394bon).Conda-forge style re-rendering: the build workflows are generated from the recipes' dependency graph and committed, instead of being computed at run time (#216's stage slots) or pushed to build branches.
What's generated
pixi run rs rerender [<distro>...]generates every recipe of each distribution and platform (generate-recipes --all, so uploads don't change it) and writes.github/workflows/build-<distro>-<platform>.yaml: one job per batch of recipes, each after the jobs that build its requirements, plusbuild-distros.yaml, which calls the selected ones.staged-plan.yamlkeeps the recipes missing from the channel and the PR cache. Only the jobs with one of them run; the rest are skipped and take no runner. A recipe to build that's in no job fails the plan ("re-render").Stable diffs
needs: a rename doesn't ripple into every dependent.Keeping it up to date
testpr.yamlre-renders the changed distributions (rendermatrix) and commits the result to pull requests from this repository as robostack-bot, so CI runs on that commit. Pull requests from forks fail until they commitpixi run rs rerender <distro>. A bot re-render commit that is still stale fails rather than looping.rs checkchecks the rendered files against the templates offline.Before this could merge
.github/workflows/stage-job.yamlforrelease.yaml.