Skip to content

Prototype: re-render the build workflows from the recipes (conda-forge style) - #229

Draft
EzraBrooks wants to merge 9 commits into
RoboStack:mainfrom
EzraBrooks:rerender-prototype
Draft

EzraBrooks wants to merge 9 commits into
RoboStack:mainfrom
EzraBrooks:rerender-prototype

Conversation

@EzraBrooks

Copy link
Copy Markdown
Contributor

Prototype, stacked on #216 (its commits are included; review from a0f1394b on).

Conda-forge style re-rendering: the build workflows are generated from the recipes' dependency graph and committed, instead of being computed at run time (#216's stage slots) or pushed to build branches.

What's generated

  • pixi run rs rerender [<distro>...] generates every recipe of each distribution and platform (generate-recipes --all, so uploads don't change it) and writes .github/workflows/build-<distro>-<platform>.yaml: one job per batch of recipes, each after the jobs that build its requirements, plus build-distros.yaml, which calls the selected ones.
  • At run time, staged-plan.yaml keeps the recipes missing from the channel and the PR cache. Only the jobs with one of them run; the rest are skipped and take no runner. A recipe to build that's in no job fails the plan ("re-render").
  • The jobs follow the real dependency graph (no lockstep stages); 127–166 jobs per distribution and platform, at most 190 KB per file.

Stable diffs

  • A recipe's level is its longest chain of requirements. Each level's recipes, sorted by name, start a new batch where a recipe's name hashes to 0 (mod 12), capped at 50 recipes: content-defined boundaries, so one change doesn't shift the batches after it.
  • A job is named after its first recipe, jobs are sorted by name, and a job lists only its nearest needs: a rename doesn't ripple into every dependent.
  • Two renders are byte-identical. On lyrical win-64's real graph: removing a leaf package changes 2 lines, adding a package 47 (its job split), a dependency that moves a package two levels deeper 220 (its whole subtree changes level).

Keeping it up to date

testpr.yaml re-renders the changed distributions (render matrix) and commits the result to pull requests from this repository as robostack-bot, so CI runs on that commit. Pull requests from forks fail until they commit pixi run rs rerender <distro>. A bot re-render commit that is still stale fails rather than looping. rs check checks the rendered files against the templates offline.

Before this could merge

  • Drop the DO NOT MERGE commit (lyrical only, linux-64 + win-64, build number 31).
  • Possibly move the batching into vinca-gha as an option.
  • prefix.dev trusted publishing must allow .github/workflows/stage-job.yaml for release.yaml.

The PR build of each distribution and platform is no longer one job that needs re-running after six hours: a plan job lists vinca-gha's stages (--json), pr-stages.yaml runs them in fixed stage slots, one matrix job per batch of recipes, and the packages pass between the stages as artifacts, read as a local channel. A collect job saves them as the PR's build cache and runs the ABI check.

pr-stages.yaml is generated by tools/pr_stages.py (rs check checks it). vinca is pinned to the --json change on top of the previous pin.
…requests and releases

staged-build.yaml (tools/staged_build.py) has 64 stage slots shared by all distributions and platforms instead of 64 per distribution and platform. Each job reads its recipes from the schedule, downloads only the packages of the jobs it needs (vinca-gha's needs), and fails when one of them failed, so the other distributions and platforms go on. Jobs that succeeded in an earlier attempt skip their build when later stages are re-run.

release.yaml runs the same workflow with uploads, by hand while the build branches still build the releases.
download-artifact's pattern only sees a run's first 1,000 artifacts: the stage jobs
and collect download the artifacts they need by name (tools/download_artifacts.cjs),
and the build log is an artifact only after a failure.

A failed recipe now holds back only the recipes that need it (vinca-gha's requires):
the job builds the rest with --continue-on-failure=skip-dependents, and
status/<job>.json records each recipe's outcome for the later jobs. Releases upload
the packages of the recipes that built.
…b isn't built either

rattler-build's skip-dependents skips the outputs that need a failed one to build, not
the ones that only run with it, e.g. ros-<distro>-<name> of ros2-<name>.
…itted

tools/rerender.py renders .github/workflows/build-<distro>-<platform>.yaml from every
recipe of the distribution (generate-recipes --all): one job per content-defined batch,
named after its first recipe, after the jobs it needs directly, sorted by name, so a
re-render changes only the jobs a change touches. staged-plan.yaml picks the jobs with a
recipe to build at run time; the others are skipped. testpr.yaml re-renders the changed
distributions and commits the result to the pull request (fails for forks until
`pixi run rs rerender`). Replaces the stage slots (staged-build.yaml).
GitHub doesn't start a run that calls all 20 rendered workflows (startup_failure with 3
or more), so each build-<distro>-<platform>.yaml is a workflow of its own, triggered by
pull_request and workflow_dispatch, with inline jobs running the composite action
.github/actions/stage-job. staged-plan.yaml's select job decides whether a pull request
builds the distribution and platform; build-status.yaml (tools/build_status.py) sums the
workflows up in the "All builds" commit status. release.yaml dispatches the build
workflows; the ABI comment is posted per distribution from staged-collect.yaml.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant