Skip to content

Design note: Cloud Pathways patterns for the simulation Jobs refactor - #163

Open
anchapin wants to merge 1 commit into
developfrom
docs/cloud-pathways-patterns
Open

anchapin wants to merge 1 commit into
developfrom
docs/cloud-pathways-patterns

Conversation

@anchapin

@anchapin anchapin commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Draft design note only — no implementation, no chart changes.

Google Cloud Pathways went GA on 9 Nov 2026. This note evaluates it for OpenStudio Server deployment and concludes:

  • Do not adopt Cloud Pathways itself — it is a JAX runtime for Google Cloud TPUs on GKE, and OpenStudio / EnergyPlus workloads are CPU-based, embarrassingly parallel simulations.
  • Do adopt three of its patterns for the planned refactor of simulations into Kubernetes Jobs with advanced scaling (e.g. Karpenter):
    1. Single-controller orchestration — each datapoint as an idempotent Job, controller as sole writer of run state, manifest in object storage.
    2. Elastic N-1 fault tolerance — datapoint-granularity checkpointing with completion markers, so a lost node / spot interruption costs only in-flight datapoints.
    3. Colocated sidecars — init/sidecar in the datapoint Job template for staging, hash verification, result upload, and completion markers.

Adds docs/cloud-pathways-patterns-design.md.

Companion PR

The application-side half of this design (run-controller logic, completion and staging contracts) is in NatLabRockies/OpenStudio-server#896: NatLabRockies/OpenStudio-server#896 — adding doc/cloud_pathways_patterns_design.md there. The two documents also cross-reference each other by repo and path.

Test plan

None — documentation only. The note itself proposes a three-check gate (kill a worker node mid-run, restart the controller mid-run, wall-clock/cost comparison) for any future implementation PR.

Please do not merge on my behalf — opening for review/discussion only.

Also in this series: the OSimFlow mapping/gap note — anchapin/OSimFlow#2133: anchapin/OSimFlow#2133

Draft design note (no implementation): do not adopt Cloud Pathways
(TPU/JAX/GKE runtime), adopt three of its patterns for the Kubernetes
Jobs refactor - single-controller orchestration, N-1 elastic fault
tolerance via datapoint completion markers, and colocated sidecars.
Companion application-side doc is proposed for OpenStudio-server.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant