Skip to content

Proof of concept: remote Snakemake output cache (Pelican/OSDF, XRootD) with remote-only reads - #373

Draft
wdconinc wants to merge 2 commits into
masterfrom
ccr-a06ed11d-172954
Draft

wdconinc wants to merge 2 commits into
masterfrom
ccr-a06ed11d-172954

Conversation

@wdconinc

Copy link
Copy Markdown
Contributor

Summary

Proof of concept: keep the --cache entries in Pelican/OSDF or XRootD storage. Downstream consumers can then read the cached outputs over root:// and skip the download.

Inactive unless SNAKEMAKE_OUTPUT_CACHE is a URL. CI is unchanged.

producer job ──(miss: run, upload)──▶ <SNAKEMAKE_OUTPUT_CACHE>/<provenance hash><suffix>
     │                                         ▲
     └─ output path = stub file ──────────────┘  (remote-only mode)
           "#snakemake-remote-cache-entry\n<read URL>\n"
consumer: eicrecon $(cache_resolve {input.SIM_FILE})  →  root://…/<hash>.root

Why not plain Snakemake

  • Snakemake has a storage-backed cache (snakemake/caching/storage.py), but it only turns on with --default-storage-provider. That flag would move every workflow file to the remote storage.
  • On a hit, Snakemake always downloads the entry to the output path. It has no way to leave the data remote.

What this does

  • benchmarks/common/remote_cache.smk
    • Picks the plugin from the URL scheme: pelican:// or osdf:// → pelican plugin, root[s]:// → xrootd plugin. Override with BENCHMARKS_CACHE_STORAGE_PROVIDER.
    • Wraps Snakemake's StorageOutputFileCache and installs it in place of snakemake.workflow.LocalOutputFileCache. This is a Snakemake internal, so it is the fragile part. Checked on 9.6.3 (CI) and 9.27.2, whose cache method signatures differ.
    • Cache keys and entry names stay the same as in the local cache.
    • BENCHMARKS_CACHE_REMOTE_ONLY=1: for rules in REMOTE_CACHE_READY_RULES, the output path holds a stub file instead of the data. This happens both on a hit and after a miss has been uploaded.
    • Read URL = BENCHMARKS_CACHE_READ_PREFIX + entry name. This lets you write through pelican:// and read over root://.
    • Failure handling:
      • A failed lookup or download counts as a cache miss and the job runs.
      • A failed upload leaves the output local.
      • After the first failure, the remote is skipped for the rest of the run.
      • A failed upload of an entry that already exists (concurrent writer) is accepted.
  • Snakefile: cache_resolve shell function in shell.prefix. The prefix is not part of the provenance hash.
  • femc_electron as the migrated example:
    • eicrecon reads the sim file through cache_resolve.
    • The plotting script resolves each file before opening it with uproot.
  • doc/cache-policy.md section 9, plus a pointer in the README.

Tested (toy workflow; the fs plugin stands in for remote storage; Snakemake 9.6.3 and 9.27.2)

  • Remote-only, miss: job runs, entry is uploaded, output becomes a stub, consumer reads the remote copy.
  • Remote-only, hit: output is a stub, consumer reads the remote copy.
  • Default mode, hit: full file is downloaded, consumer reads the local path.
  • root://127.0.0.1:1 (dead endpoint): pipeline completes with local outputs. Wall time 11 s with the circuit breaker. Without it, about 70 s per operation even with tightened XRD_* timeouts.
  • Real Snakefile dry-run renders eicrecon $(cache_resolve sim_output/...).
  • With a pelican:///osdf:// cache, the module loads the pelican plugin and stops only at the network step (no outbound access in the test environment).

Not tested / open issues

  • No real Pelican origin or XRootD endpoint was used. Not verified:
    • writing needs a token with write permission for the namespace;
    • which host and namespace to use for reads.
  • The storage plugin must be installed in the CI environment (snakemake-storage-plugin-pelican or snakemake-storage-plugin-xrootd). For remote-only uproot reads, fsspec-xrootd is also needed.
  • Uploads are not atomic: a reader can see a partly written entry, and exists reports it as a hit.
  • Remote entries are not garbage collected; that is left to the storage (e.g. a volatile namespace).
  • The CI cache steps in .gitlab-ci.yml (mkdir, symlink cleanup, garbage-collect.py) assume a local path.
  • Detecting an unreachable host still costs one round of the storage client's timeouts. With XRootD defaults this is more than 5 min, so set XRD_CONNECTIONRETRY/XRD_CONNECTIONWINDOW.
  • With a remote cache, fetch_epic (remote → cache copy) could be replaced by direct reads of the source URL.

Generated by Claude Code

…h remote-only reads

- benchmarks/common/remote_cache.smk: when SNAKEMAKE_OUTPUT_CACHE is a
  pelican://, osdf:// or root:// URL, keep the --cache entries in that
  storage through the pelican or xrootd storage plugin, without setting
  --default-storage-provider (which would move every workflow file).
  Cache keys and entry names are unchanged.
- BENCHMARKS_CACHE_REMOTE_ONLY=1: outputs of rules in
  REMOTE_CACHE_READY_RULES become stub files holding the entry's read URL
  (BENCHMARKS_CACHE_READ_PREFIX, e.g. root:// for entries written through
  pelican://); consumers open them with cache_resolve.
- Remote lookup/download failures are cache misses; upload failures keep
  the output local.
- femc_electron: simulate/recon outputs read through cache_resolve
  (eicrecon, uproot).
- doc/cache-policy.md section 9, README pointer.

Inactive unless SNAKEMAKE_OUTPUT_CACHE is a URL; CI is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019qEz4t7nCHhQCcoTqrvKZM
Each failed remote operation waits through the storage client's timeouts
and the plugin's retries (minutes with XRootD defaults); skip the remote
after the first failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019qEz4t7nCHhQCcoTqrvKZM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants