Skip to content

fix: Apply a dense Gram after a global projection again - #622

Open
BryanBradfo wants to merge 1 commit into
EleutherAI:mainfrom
BryanBradfo:fix/trak-dense-gram
Open

BryanBradfo wants to merge 1 commit into
EleutherAI:mainfrom
BryanBradfo:fix/trak-dense-gram

Conversation

@BryanBradfo

Copy link
Copy Markdown
Contributor

bergson trak stops at step 2/4 on main:

ValueError: A global projection is preconditioned with a factored Hessian, but .../train_hessian is dense.

Since #612, GradientCollector sends every Hessian given with a global projection to load_projection_preconditioner, which applies H^-1/2 before the projection and only accepts factored Hessians. TRAK's Gram is dense and over the projected gradients, so before #612 the builder applied it after the projection.

This only takes the pre-projection path for factored Hessians. A dense Gram goes back to the builder, as before #612. P-SIFT is unchanged.

Tests

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant