Skip to content

feat: Resume an interrupted Hessian fit - #617

Merged
luciaquirke merged 10 commits into
EleutherAI:mainfrom
BryanBradfo:feature/hessian-fit-resume
Nov 17, 2026
Merged

luciaquirke merged 10 commits into
EleutherAI:mainfrom
BryanBradfo:feature/hessian-fit-resume

Conversation

@BryanBradfo

Copy link
Copy Markdown
Contributor

Closes #453, following the design discussed there. Picks up from #470.

--checkpoint_interval N saves a Hessian fit's running sums every N documents. Running the same command with --resume true continues from the last save.

How it works

  • After each phase (covariance, eigendecomposition, eigenvalue correction), rank 0 writes a marker and each rank saves its RNG state. A resumed fit skips the finished phases.
  • Within a data pass, each rank saves its collector's sums through state_dict() / load_state_dict(), and a resumed pass skips the batches already summed. KFAC, TKFAC, Shampoo and the correction collector support it.
  • Resuming with another model, method, hessian_dtype, use_dataset_labels or dataset is an error.
  • Everything is saved under checkpoint/, which is removed once the fit is done. With the default checkpoint_interval=0, nothing changes.

Tests

  • tests/ekfac_tests/test_fit_resume.py interrupts the fit in each phase, in one process and with 2 gloo ranks. The resumed factors must equal the uninterrupted ones exactly, and only the batches after the last save may run again.
  • With the CLI on pythia-14m (8000 documents), I killed the fit with SIGKILL in each phase and with --module_partitions 2. Each resumed fit matched an uninterrupted one as closely as two uninterrupted fits match each other.
  • I only have one GPU, so I have not run it on several GPUs with NCCL.

Limitations

  • autocorrelation can't be checkpointed; setting checkpoint_interval with it is an error.
  • Only the influence pipeline resumes a fit. hessian, trackstar and trak still clear the partial output.
  • A crash between the end of a pass and its phase marker reruns that pass.
  • With sampled labels, a resumed fit only matches the uninterrupted one when the seed is fixed (--debug).

@BryanBradfo

Copy link
Copy Markdown
Contributor Author

@luciaquirke this implements the design from #453. One thing I could not check: I only have a single GPU, so the multi-rank path is tested with gloo on CPU, not with NCCL. Happy to adjust anything! 😄

Comment thread docs/performance.rst Outdated

Hessian fitting can be batched over modules as well as documents. Set ``hessian_cfg.module_partitions`` to fit the factors in that many groups of modules, one pass over the data per group, when they don't all fit in memory at once.

Set ``hessian_cfg.checkpoint_interval`` to save the fit's running sums every that many documents. An interrupted fit then resumes with ``--resume``, repeating only the documents since the last save and skipping the phases it finished.

@luciaquirke luciaquirke Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't a performance consideration I don't think. Also I'm trying to keep the non-boilerplate parts of the documents human-written

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise LGTM

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed, thank you @luciaquirke for the review!

@luciaquirke
luciaquirke merged commit 99fc121 into EleutherAI:main Nov 17, 2026
10 checks passed
@luciaquirke

Copy link
Copy Markdown
Collaborator

Thank you very much for your contribution!!

@BryanBradfo

Copy link
Copy Markdown
Contributor Author

Thanks for the review and the merge! I'd like to follow up on what #617 left out (the autocorrelation fit). I'll open an issue with the design first! 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Checkpoint Hessian fit

2 participants