Repository navigation
feat: Resume an interrupted Hessian fit - #617
Merged
luciaquirke merged 10 commits intoNov 17, 2026
Merged
Conversation
Contributor
Author
|
@luciaquirke this implements the design from #453. One thing I could not check: I only have a single GPU, so the multi-rank path is tested with gloo on CPU, not with NCCL. Happy to adjust anything! 😄 |
luciaquirke
reviewed
Nov 12, 2026
|
|
||
| Hessian fitting can be batched over modules as well as documents. Set ``hessian_cfg.module_partitions`` to fit the factors in that many groups of modules, one pass over the data per group, when they don't all fit in memory at once. | ||
|
|
||
| Set ``hessian_cfg.checkpoint_interval`` to save the fit's running sums every that many documents. An interrupted fit then resumes with ``--resume``, repeating only the documents since the last save and skipping the phases it finished. |
Collaborator
There was a problem hiding this comment.
This isn't a performance consideration I don't think. Also I'm trying to keep the non-boilerplate parts of the documents human-written
Contributor
Author
There was a problem hiding this comment.
Removed, thank you @luciaquirke for the review!
Collaborator
|
Thank you very much for your contribution!! |
Contributor
Author
|
Thanks for the review and the merge! I'd like to follow up on what #617 left out (the autocorrelation fit). I'll open an issue with the design first! 😄 |
This was referenced Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #453, following the design discussed there. Picks up from #470.
--checkpoint_interval Nsaves a Hessian fit's running sums every N documents. Running the same command with--resume truecontinues from the last save.How it works
state_dict()/load_state_dict(), and a resumed pass skips the batches already summed. KFAC, TKFAC, Shampoo and the correction collector support it.hessian_dtype,use_dataset_labelsor dataset is an error.checkpoint/, which is removed once the fit is done. With the defaultcheckpoint_interval=0, nothing changes.Tests
tests/ekfac_tests/test_fit_resume.pyinterrupts the fit in each phase, in one process and with 2 gloo ranks. The resumed factors must equal the uninterrupted ones exactly, and only the batches after the last save may run again.--module_partitions 2. Each resumed fit matched an uninterrupted one as closely as two uninterrupted fits match each other.Limitations
autocorrelationcan't be checkpointed; settingcheckpoint_intervalwith it is an error.influencepipeline resumes a fit.hessian,trackstarandtrakstill clear the partial output.--debug).