Repository navigation
Conversation
…h remote-only reads - benchmarks/common/remote_cache.smk: when SNAKEMAKE_OUTPUT_CACHE is a pelican://, osdf:// or root:// URL, keep the --cache entries in that storage through the pelican or xrootd storage plugin, without setting --default-storage-provider (which would move every workflow file). Cache keys and entry names are unchanged. - BENCHMARKS_CACHE_REMOTE_ONLY=1: outputs of rules in REMOTE_CACHE_READY_RULES become stub files holding the entry's read URL (BENCHMARKS_CACHE_READ_PREFIX, e.g. root:// for entries written through pelican://); consumers open them with cache_resolve. - Remote lookup/download failures are cache misses; upload failures keep the output local. - femc_electron: simulate/recon outputs read through cache_resolve (eicrecon, uproot). - doc/cache-policy.md section 9, README pointer. Inactive unless SNAKEMAKE_OUTPUT_CACHE is a URL; CI is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qEz4t7nCHhQCcoTqrvKZM
Each failed remote operation waits through the storage client's timeouts and the plugin's retries (minutes with XRootD defaults); skip the remote after the first failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qEz4t7nCHhQCcoTqrvKZM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Proof of concept: keep the
--cacheentries in Pelican/OSDF or XRootD storage. Downstream consumers can then read the cached outputs overroot://and skip the download.Inactive unless
SNAKEMAKE_OUTPUT_CACHEis a URL. CI is unchanged.Why not plain Snakemake
snakemake/caching/storage.py), but it only turns on with--default-storage-provider. That flag would move every workflow file to the remote storage.What this does
benchmarks/common/remote_cache.smkpelican://orosdf://→ pelican plugin,root[s]://→ xrootd plugin. Override withBENCHMARKS_CACHE_STORAGE_PROVIDER.StorageOutputFileCacheand installs it in place ofsnakemake.workflow.LocalOutputFileCache. This is a Snakemake internal, so it is the fragile part. Checked on 9.6.3 (CI) and 9.27.2, whose cache method signatures differ.BENCHMARKS_CACHE_REMOTE_ONLY=1: for rules inREMOTE_CACHE_READY_RULES, the output path holds a stub file instead of the data. This happens both on a hit and after a miss has been uploaded.BENCHMARKS_CACHE_READ_PREFIX+ entry name. This lets you write throughpelican://and read overroot://.Snakefile:cache_resolveshell function inshell.prefix. The prefix is not part of the provenance hash.femc_electronas the migrated example:eicreconreads the sim file throughcache_resolve.uproot.doc/cache-policy.mdsection 9, plus a pointer in the README.Tested (toy workflow; the
fsplugin stands in for remote storage; Snakemake 9.6.3 and 9.27.2)root://127.0.0.1:1(dead endpoint): pipeline completes with local outputs. Wall time 11 s with the circuit breaker. Without it, about 70 s per operation even with tightenedXRD_*timeouts.Snakefiledry-run renderseicrecon $(cache_resolve sim_output/...).pelican:///osdf://cache, the module loads the pelican plugin and stops only at the network step (no outbound access in the test environment).Not tested / open issues
snakemake-storage-plugin-pelicanorsnakemake-storage-plugin-xrootd). For remote-onlyuprootreads,fsspec-xrootdis also needed.existsreports it as a hit..gitlab-ci.yml(mkdir, symlink cleanup,garbage-collect.py) assume a local path.XRD_CONNECTIONRETRY/XRD_CONNECTIONWINDOW.fetch_epic(remote → cache copy) could be replaced by direct reads of the source URL.Generated by Claude Code