Skip to content

garbage-collect.py: evict inactive pipeline scratch directories when short of free space - #374

Open
wdconinc wants to merge 3 commits into
masterfrom
ccr-6cc78246-2s0g57
Open

wdconinc wants to merge 3 commits into
masterfrom
ccr-6cc78246-2s0g57

Conversation

@wdconinc

@wdconinc wdconinc commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem

A recent garbage collection run could not reach the 200 GiB free-space minimum:

Free space after pruning: 0.0 GiB (minimum 200.0 GiB)
... 529.1 GiB in 8753 entries that symbolic links under /scratch point to (alive).
Outside the cache, about 432.0 GiB of the 1006.9 GiB filesystem is used (pipeline scratch directories, ...)

Scratch directories of failed pipelines (cleanup:scratch only runs on success) occupy space themselves and keep cache entries alive through their symlinks. Before this change, the only thing that removed them was a find -mmin +4320 on the top-level directory mtime. That timestamp barely changes while jobs are running, and the check did not depend on how much space was free.

Change

Stale scratch handling moves into garbage-collect.py, which already holds the lock:

  • Activity measure: the newest lstat mtime of the scratch directory or anything below it. bin/env.sh now touches $LOCAL_DATA_PATH at the start of every job, so a job start or retry counts as activity.
  • --scratch-max-age 72h: always remove scratch directories inactive for longer than this (replaces the find).
  • --scratch-grace-period 24h: if free space is still below --min-free after all deletable cache entries are removed, remove scratch directories inactive for longer than this, least recently active first. Cache entries that only the removed directory pointed to become deletable and are removed (same score order) before the next directory is considered.
  • --scratch-pattern (repeatable): detector_benchmarks_*, physics_benchmarks_*, generator_benchmarks_*.
  • The warning when the minimum cannot be reached now also reports the scratch directories that were kept.
  • Without the new flags the behaviour is unchanged.
free < min? ─► deletable cache entry? ─yes─► delete it ─────────────────────┐
                    │ no                                                      │
                    ▼                                                         │
         scratch dir inactive > 24h? ─yes─► delete it; release its pins ─────┤
                    │ no                                                      │
                    ▼                                                         │
              WARNING, stop ◄─────────────────────────────────────────────────┘

doc/cache-policy.md section 7 and the cleanup:scratch comment are updated.

Testing

Synthetic /scratch tree with directories inactive for 100 h, 40 h and 30 h, one directory that recently had a file written, and one non-pipeline directory:

  • The 100 h directory was always removed.
  • With the free-space minimum unreachable, the 40 h directory was removed before the 30 h one, and the cache entries they alone pointed to were then deleted.
  • Cache entries still linked from the recently active directory or from the non-pipeline directory were kept.
  • With enough free space, only the 72 h expiry applied.
  • Without the scratch flags, the output was the same as before.

On eicweb (common:setup of pipeline 152529, b406925, Python 3.14):

  • Free space 15.0 → 91.7 GiB; cache 614.8 → 538.4 GiB.
  • 23 scratch directories removed (incl. 2 generator_benchmarks_*), least recently active first (48.0 h → 24.0 h); none active within 24 h.
  • After the first 5, the 1137 cache entries (76.4 GiB) that only they pointed to were removed.
  • Minimum not reached: 110 scratch directories active within 24 h hold 275.3 GiB, 148.8 GiB of cache is within its 6 h grace period, 389.6 GiB is alive.

Benchmark job failures in pipelines 152527/152529 mostly also occur on master (see comment).

Points for review

  • 24 h grace: a pipeline with no job started for more than 24 h, then retried by hand while /scratch is short of space, loses its scratch data (the longest job timeout in this repository is 4 h).
  • physics_benchmarks and generator_benchmarks do not run this repository's bin/env.sh, so their directories count as active only through files their jobs write.
  • generator_benchmarks_* directories were not removed by the previous 72 h find either; with this change they are expired for the first time.

🤖 Generated with Claude Code

https://claude.ai/code/session_019qEz4ubnxM1akeVWt23Hv2

The cache-only free-space pass could not reach --min-free when most of
/scratch was held by scratch directories of failed pipelines (and the
cache entries they keep alive). garbage-collect.py now:

- scans the scratch directories (--scratch-pattern) and measures their
  last activity as the newest mtime in the tree; bin/env.sh touches
  $LOCAL_DATA_PATH at the start of every job, so job (re)starts count,
- removes those inactive for more than --scratch-max-age (72h),
  replacing the find -mmin +4320 on the top-level directory mtime,
- when free space is still short after removing all removable cache
  entries, removes scratch directories inactive for more than
  --scratch-grace-period (24h), least recently active first; cache
  entries that only they kept alive become removable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019qEz4ubnxM1akeVWt23Hv2
Copilot AI balanced review requested due to automatic review settings November 15, 2026 20:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open issues
What changed in this PR

Updates garbage-collect.py to also manage pipeline scratch directories (in addition to the Snakemake output cache) so that inactive scratch trees don’t prevent reaching the filesystem free-space minimum.

Changes:

  • Add scratch-directory scanning, activity detection, and eviction logic driven by --scratch-* flags.
  • Replace CI find ... -mmin scratch cleanup with garbage-collect.py flags and document the new policy.
  • Touch $LOCAL_DATA_PATH at job start to keep active scratch directories from being considered stale.
File Description
garbage-collect.py Adds scratch eviction and “pins” model so scratch symlinks only protect cache entries while the scratch dir exists.
doc/​cache-policy.md Documents scratch “activity” definition and the new eviction flow when --min-free can’t be met.
bin/​env.sh Touches $LOCAL_DATA_PATH to reflect job activity for scratch staleness tracking.
.gitlab-ci.yml Switches from find -mmin to garbage-collect.py --scratch-* in common:setup.

Review effort: Lite

Comment thread garbage-collect.py Resolved
Comment thread garbage-collect.py Resolved
Comment thread garbage-collect.py Resolved
Comment thread garbage-collect.py Resolved
- catch OSError (not only FileNotFoundError) when stat-ing or listing
  entries under --symlink-roots-dir, so one unreadable entry does not
  abort garbage collection
- bisect.insort without key= (Python < 3.10); the tuples sort by score
- clearer error when a --scratch-* age is given without its prerequisites

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019qEz4ubnxM1akeVWt23Hv2

Copy link
Copy Markdown
Contributor Author

The failed copilot-pull-request-reviewer check (on the earlier head 99754c4) is not a failure of this PR's code. Copilot's agentic review did not start before its timeout because no runner was available, as its own review says. The review findings it did post have been addressed in b406925.


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

eicweb pipeline 152529 (b406925) failed with 17 failed jobs. Comparison with the last 25 failed master pipelines on eicweb (6d1192d, which do not contain this PR):

Failed in 152529 Also failed on master
sim:ecal_gaps, sim:backwards_ecal: [pi-, 500MeV], sim:nhcal_pion_rejection: [8 9] 18, 18, 12 of 25
bench:backwards_ecal, bench:insert_neutron, bench:insert_tau, bench:lowq2_reconstruction, bench:zdc_lambda 20, 18, 7, 3, 1 of 25
collect_results:{ecal_gaps,femc_photon,insert_neutron,zdc_lambda,zdc_sigma} yes
bench:zdc_sigma, bench:femc_photon, bench:emcal_barrel_pi0, bench:nhcal_basic_distribution_analysis_full not in this sample (on master their upstream jobs often failed first)

So most of these failures are not specific to this PR. /scratch had 15 GiB free when this pipeline's common:setup ran, which is a likely common cause. Garbage collection in this pipeline only removed scratch directories inactive for more than 24 h, and only cache entries no symbolic link pointed to anymore, so it did not touch this or any other active pipeline's data. The four jobs in the last row are unconfirmed until their logs are checked.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants