Repository navigation
Conversation
The cache-only free-space pass could not reach --min-free when most of /scratch was held by scratch directories of failed pipelines (and the cache entries they keep alive). garbage-collect.py now: - scans the scratch directories (--scratch-pattern) and measures their last activity as the newest mtime in the tree; bin/env.sh touches $LOCAL_DATA_PATH at the start of every job, so job (re)starts count, - removes those inactive for more than --scratch-max-age (72h), replacing the find -mmin +4320 on the top-level directory mtime, - when free space is still short after removing all removable cache entries, removes scratch directories inactive for more than --scratch-grace-period (24h), least recently active first; cache entries that only they kept alive become removable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qEz4ubnxM1akeVWt23Hv2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qEz4ubnxM1akeVWt23Hv2
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
4 open issues
os.lstat()can raisePermissionError/OSErrorwhile traversing (not justFileNotFoundError).… · NewPath.lstat()can raisePermissionError(and otherOSErrors) on unreadable entries under… · Newbisect.insort()does not support akey=parameter on older Python versions. Since your tuples… · New The error message is inaccurate when only one of--scratch-max-ageor--scratch-grace-periodis… · New
What changed in this PR
Updates garbage-collect.py to also manage pipeline scratch directories (in addition to the Snakemake output cache) so that inactive scratch trees don’t prevent reaching the filesystem free-space minimum.
Changes:
- Add scratch-directory scanning, activity detection, and eviction logic driven by
--scratch-*flags. - Replace CI
find ... -mminscratch cleanup withgarbage-collect.pyflags and document the new policy. - Touch
$LOCAL_DATA_PATHat job start to keep active scratch directories from being considered stale.
| File | Description |
|---|---|
| garbage-collect.py | Adds scratch eviction and “pins” model so scratch symlinks only protect cache entries while the scratch dir exists. |
| doc/cache-policy.md | Documents scratch “activity” definition and the new eviction flow when --min-free can’t be met. |
| bin/env.sh | Touches $LOCAL_DATA_PATH to reflect job activity for scratch staleness tracking. |
| .gitlab-ci.yml | Switches from find -mmin to garbage-collect.py --scratch-* in common:setup. |
Review effort: Lite
- catch OSError (not only FileNotFoundError) when stat-ing or listing entries under --symlink-roots-dir, so one unreadable entry does not abort garbage collection - bisect.insort without key= (Python < 3.10); the tuples sort by score - clearer error when a --scratch-* age is given without its prerequisites Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qEz4ubnxM1akeVWt23Hv2
|
The failed Generated by Claude Code |
|
eicweb pipeline 152529 (b406925) failed with 17 failed jobs. Comparison with the last 25 failed
So most of these failures are not specific to this PR. Generated by Claude Code |


Problem
A recent garbage collection run could not reach the 200 GiB free-space minimum:
Scratch directories of failed pipelines (
cleanup:scratchonly runs on success) occupy space themselves and keep cache entries alive through their symlinks. Before this change, the only thing that removed them was afind -mmin +4320on the top-level directory mtime. That timestamp barely changes while jobs are running, and the check did not depend on how much space was free.Change
Stale scratch handling moves into
garbage-collect.py, which already holds the lock:lstatmtime of the scratch directory or anything below it.bin/env.shnow touches$LOCAL_DATA_PATHat the start of every job, so a job start or retry counts as activity.--scratch-max-age 72h: always remove scratch directories inactive for longer than this (replaces thefind).--scratch-grace-period 24h: if free space is still below--min-freeafter all deletable cache entries are removed, remove scratch directories inactive for longer than this, least recently active first. Cache entries that only the removed directory pointed to become deletable and are removed (same score order) before the next directory is considered.--scratch-pattern(repeatable):detector_benchmarks_*,physics_benchmarks_*,generator_benchmarks_*.doc/cache-policy.mdsection 7 and thecleanup:scratchcomment are updated.Testing
Synthetic
/scratchtree with directories inactive for 100 h, 40 h and 30 h, one directory that recently had a file written, and one non-pipeline directory:On eicweb (
common:setupof pipeline 152529, b406925, Python 3.14):generator_benchmarks_*), least recently active first (48.0 h → 24.0 h); none active within 24 h.Benchmark job failures in pipelines 152527/152529 mostly also occur on
master(see comment).Points for review
/scratchis short of space, loses its scratch data (the longest job timeout in this repository is 4 h).bin/env.sh, so their directories count as active only through files their jobs write.generator_benchmarks_*directories were not removed by the previous 72 hfindeither; with this change they are expired for the first time.🤖 Generated with Claude Code
https://claude.ai/code/session_019qEz4ubnxM1akeVWt23Hv2