Skip to content

perf[next]: select instead of concatenate in embedded concat_where under torch.compile - #3026

Draft
havogt wants to merge 6 commits into
GridTools:mainfrom
havogt:concat-where-select
Draft

havogt wants to merge 6 commits into
GridTools:mainfrom
havogt:concat-where-select

Conversation

@havogt

@havogt havogt commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Under torch.compile, embedded concat_where along one dimension now evaluates as an elementwise select over the result range whenever both (broadcast) branches cover that range, instead of slicing the branches and concatenating the pieces. Inductor fuses the select into its consumers, while the concatenation materializes the result. Eager execution and JAX keep the concatenation: executed eagerly the select reads both branches over the whole result and is slower, and under jax.jit it lost on the largest grid measured. The select is the concat_where entry of TorchOps.builtin_overrides; outside a trace and where a branch does not cover the result it uses the generic implementation.

…en both branches cover the result

When both branches are defined on the whole result range along the
condition's dimension, the result is a masked select of the two, which
avoids slicing and concatenating copies. Other cases keep the
concatenation.
@havogt

havogt commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Written by an agent on behalf of @havogt

return _concat(*f_slices, *t_slices, dim=domain_dim)


def _where_on_range(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's make it a torch-only override as the performance results suggest..

Eagerly the select reads both branches over the whole result and is
slower than the concatenation; under jax.jit it was a loss on the largest
measured grid. The choice is the ArrayOps hook concat_where_selects,
which only TorchOps answers for traced tensors.
@havogt havogt changed the title perf[next]: select instead of concatenate in embedded concat_where when both branches cover the result perf[next]: select instead of concatenate in embedded concat_where under torch.compile Nov 20, 2026
@havogt

havogt commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Measured on santis (one GH200) with icon4py's DiffusionGlobal on global grids with 80 levels, median time per call. Results agree with the dace backend to 4.4e-16.

R02B05 R02B06 R02B07
torch.compile, before → after 1.418 → 1.171 ms 5.18 → 4.28 ms 20.17 → 16.65 ms
jax.jit (tables as jit arguments), after/before 0.967 0.953 1.068

Caveat: both rows were measured with an earlier revision of this PR that selected under every backend, on a torch.compile development branch that already contained #2989 and #2895, not on main itself. For torch.compile that is the code path kept here; the jax.jit row is why JAX keeps the concatenation.

🤖 Written by an agent on behalf of @havogt

…erride of TorchOps

Replaces the ArrayOps.concat_where_selects switch: the generic concat_where
is main's again, with its slicing split out for the torch override.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant