Skip to content

feat[eve]: lazy_import_module utility for lazily importing modules and subpackages - #3020

Draft
egparedes wants to merge 18 commits into
mainfrom
eve-lazy-imports
Draft

egparedes wants to merge 18 commits into
mainfrom
eve-lazy-imports

Conversation

@egparedes

@egparedes egparedes commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

Follows #3019 (merged), which turned eve.utils into a subpackage.

Adds the gt4py.eve.utils.lazy_imports submodule with lazy_import_module() (re-exported as gt4py.eve.utils.lazy_import_module()), which returns a module stand-in and only imports the module on first attribute access. It follows the idea of lazy_loader.load() but also works for subpackages (e.g. "cupyx.profiler", "jax.numpy") without importing their parents up front.

  • Availability check without running code. It walks the parent chain using the sys.meta_path finders. Imported parents are searched through their runtime __path__. If a child isn't found under a parent that isn't imported yet, the parent is imported for real and the lookup is retried.
    • on_missing="raise" raises ModuleNotFoundError for a library that isn't installed; "none" returns None. This covers libraries that aren't installed; an installed library that fails to import raises on first access.
  • Normal import semantics. The stand-in is not registered in sys.modules; it resolves with a plain importlib.import_module(). Import order, locking, import-time side effects and rollback on failure therefore behave exactly as with a normal import.
  • Helpers (also re-exported from eve.utils):
    • resolve_lazy_module() returns the real module, for code that needs module identity (is, ==, dict keys).
    • is_lazy_module() tells whether an object is one of these stand-ins.

Towards Python 3.15 lazy imports (PEP 810)

This is a temporary, interoperable step towards the explicit lazy import statements of Python 3.15 (PEP 810), which GT4Py can't use while it supports 3.12–3.14.

  • Shared semantics: the module is not in sys.modules until its first use, it is then imported with a plain import, and a failed import is retried on the next use.
  • Migration: call sites can switch to lazy import ... once 3.15 is the minimum version. The module docstring lists the differences to handle until then.

Why not lazy_loader.load() / importlib.util.LazyLoader

The eve.utils.lazy_imports module docstring has the rationale. In short:

  • With lazy.load()/importlib.util.LazyLoader, a subpackage either imports its parent eagerly or runs before its parent.
  • A lazy module registered in sys.modules gets loaded by any scan of sys.modules. gt4py itself scans it while importing (inspect.getmodule() via eve.codegen).
  • A registered lazy module for a broken library (e.g. cupy without CUDA) also raises its ImportError inside unrelated code, e.g. array_api_compat.array_namespace().

The design went through four rounds of adversarial review, with experiments on CPython 3.12–3.14.

Not in this PR

Migrating existing try: import x / except ImportError: x = None sites. These module-identity sites need to be checked when that happens:

  • _core/ndarray_utils.py
  • next/field_utils.py
  • next/embedded/array_ops.py
  • next.fingerprinting.strict_fingerprinter

Requirements

  • All fixes and/or new features come with corresponding tests.
  • Important design decisions have been documented in the appropriate ADR inside the docs/development/ADRs/ folder. N/A: following review, the design notes live in the eve.utils.lazy_imports module docstring instead.

Move the contents of the 1900-line `eve/utils.py` module, unchanged, into
thematic submodules of a new `eve.utils` subpackage:

- `caching`: `lru_cache`, `optional_lru_cache`
- `classes`: `get_fully_qualified_name`, `register_subclasses`, `noninstantiable`, ...
- `containers`: `CustomDefaultDictBase`, `CustomMapping`, `Namespace`, `FrozenNamespace`
- `dispatching`: `singledispatcher`, `merge_dispatchers`
- `functional`: attribute/item getters and checkers, `IndexerCallable`, `fluid_partial`
- `hashing`: `HashableBy`, `content_hash`, `ddiff`, `dhash`, ...
- `iterators`: `XIterable`, `xiter`, `xenumerate`, `first`, ...
- `strings`: `CaseStyleConverter`, `SequentialIDGenerator`

The package `__init__` re-exports every public name (including the
`boltons` and `toolz` re-exports), so `gt4py.eve.utils.<name>` keeps working
unchanged and the existing tests are untouched.
…d subpackages

Add the `eve.utils.lazy_imports` submodule with `lazy_import_module()`,
`is_lazy_module()` and `resolve_lazy_module()`, re-exported from `eve.utils`.
The returned proxy is not registered in `sys.modules` and resolves itself
with a plain `importlib.import_module()` on first attribute access;
availability is checked without executing any module code, also for
subpackages. Design rationale and rejected alternatives in
docs/development/ADRs/eve/0001-Lazy_Module_Imports.md.
…alIDGenerator into eve.utils.classes

Address review: the custom mappings and namespaces (formerly
`eve.utils.containers`), `IndexerCallable` (from `functional`) and
`SequentialIDGenerator` (from `strings`) now live in `eve.utils.classes`,
whose module docstring is updated accordingly.
# Conflicts:
#	src/gt4py/eve/utils/__init__.py
…e.utils module

Re-export `deepdiff`, `xxhash`, `xtyping`, `NOTHING`, `NothingType` and
`ArgsOnlyCallable` from `eve.utils` for backwards compatibility, and pin
the public names of the package in a test.
….utils imports

Address review: `deepdiff`, `xxhash`, `xtyping`, `NOTHING`, `NothingType`
and `ArgsOnlyCallable` are no longer re-exported from `eve.utils`.
# Conflicts:
#	tests/eve_tests/unit_tests/test_utils.py
…re-exporting TypeVars

Address review: the `toolz` re-export now precedes the internal imports, and
the `T`, `S`, `K` and `P` TypeVars are no longer re-exported from
`eve.utils`.
# Conflicts:
#	tests/eve_tests/unit_tests/test_utils.py

@egparedes egparedes left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial thoughts

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drop the ADR and add the really important information into docstrings

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3720e5e: the ADR (and the docs/development/ADRs/eve/ folder, which only held it) is removed. The important parts now live in docstrings:

  • the eve.utils.lazy_imports module docstring: why the proxy is kept out of sys.modules (vs lazy_loader.load() / importlib.util.LazyLoader), the semantics and the caveats;
  • the resolve_lazy_module() docstring: what to check when migrating a call site (identity uses).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add mentions to the 3.15 lazy imports in comments or in the module docstring. The goal is to make the current implementation just an intermediate temporary interoperable step towards 3.15 lazy imports features, so this implementation should follow a similar looking approach, and should be easy to drop in the future without larger changes in the rest of the codebase.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 3720e5e. The module docstring now presents the module as a temporary stand-in for PEP 810 explicit lazy imports (Python 3.15), and the lazy_import_module() docstring points to it.

  • Shared semantics: the module stays out of sys.modules until first use, it is then imported with a plain import, and a failed import is retried on the next use. These match PEP 810, so call sites can move to lazy import ... once 3.15 is the minimum version.
  • Differences call sites must handle:
    • the proxy is not rebound to the real module, so identity checks need resolve_lazy_module();
    • attribute access goes through the proxy, and dir() imports the module;
    • availability is checked eagerly. PEP 810 forbids lazy import inside try blocks, so optional dependencies will still need an explicit availability check.
  • Intermediate step: the docstring also mentions the __lazy_modules__ global for modules that are always installed.

…me them as a step towards PEP 810

Address review: drop the ADR, and document in the `eve.utils.lazy_imports`
module docstring that it is a temporary stand-in for Python 3.15 explicit
lazy imports (PEP 810), with the differences and the migration path.
…zy import availability check

Address review findings:
- Finders raising because the parent package is not imported (e.g.
  `PathFinder` for namespace portions: `KeyError` on Python <= 3.14) no
  longer crash `lazy_import_module()`; the parent is imported and the
  lookup retried, as for other modules not found statically.
- Submodules registered in `sys.modules` by non-package parents are found,
  as with plain imports.
- Docstrings no longer claim that the availability check never executes
  module code, and no longer list `dir()` as a difference with PEP 810.
- Tests for namespace packages, aliases from non-package parents,
  non-`ImportError` failures, success after failure, `importlib.reload()`
  and concurrent calls; the thread test now really jitters each thread.
…ty check imports modules

Address review findings:
- Docstrings state that finding a module may import a parent package (and
  with it, possibly the module itself, which is then returned directly),
  and name `importlib.import_module()` as the import mechanism.
- Single `sys.modules` lookups, robust to concurrent modifications.
- Empty, relative or malformed names raise `ValueError`, as plain imports do.
- The namespace package test looks up the deepest name first, so it covers
  the fallback; new tests for a target imported by its parent, proxies with
  `on_missing="none"` and invalid names.
Base automatically changed from eve-utils-subpackage to main November 20, 2026 14:19
egparedes added a commit that referenced this pull request Nov 20, 2026
## Description

Turns the 1900-line `eve/utils.py` module into an `eve.utils` subpackage. Every function, class and constant is moved **unchanged**: their ASTs are identical to the originals (see the verification comment below). The TypeVars (`T`, `P`, `_P`, `_T`, …) are copied verbatim into each submodule that uses them. Submodules:

| Submodule | Contents |
|---|---|
| `caching` | `lru_cache`, `optional_lru_cache` |
| `classes` | `get_fully_qualified_name`, `register_subclasses`, `noninstantiable`, `is_noninstantiable`, `IndexerCallable`, `CustomDefaultDictBase`, `CustomMapping`, `Namespace`, `FrozenNamespace`, `SequentialIDGenerator` |
| `dispatching` | `singledispatcher`, `merge_dispatchers` |
| `functional` | `isinstancechecker`, `attrchecker`, `attrgetter_`, `getitem_`, `itemgetter_`, `fluid_partial`, `with_fluid_partial` |
| `hashing` | `HashableBy`, `hashable_by`, `hashable_by_id`, `cached_hash`, `EqualityBy`, `content_hash`, `ddiff`, `dhash`, `pprint_ddiff` |
| `iterators` | `XIterable`, `xiter`, `xenumerate`, `as_xiter`, `first` |
| `strings` | `AnyWordsIterable`, `CaseStyleConverter` |

The package `__init__` re-exports the public functions, classes and constants of the old module, plus the `boltons` helpers and `toolz`, so `gt4py.eve.utils.<name>` keeps working for all of them. No call site changes, and the tests are unchanged.

**Compatibility:**
- **Still importable from `eve.utils`:** every public function, class and constant of the old module, the `boltons` helpers and `toolz`.
- **No longer reachable through `eve.utils`:**
  - names the old module only imported for its own use: `deepdiff`, `xxhash`, `xtyping`, `NOTHING`, `NothingType`, `ArgsOnlyCallable`;
  - TypeVars (`T`, `S`, `K`, `P` and the private ones), which are now declared privately in each submodule that uses them;
  - private names: `_CONTENT_HASH_PICKLE_PROTOCOL`, …

  Nothing in the repo uses any of them through `eve.utils`.

Objects now report the submodule in `__module__` (e.g. `gt4py.eve.utils.classes.FrozenNamespace`). Pickles created before this refactor still load, because the old paths resolve through the re-exports (verified in a comment below).

Next in the stack: #3020 adds the lazy import utilities as a new `eve.utils.lazy_imports` submodule.

## Requirements

- [x] All fixes and/or new features come with corresponding tests. Pure refactor: the existing tests and doctests cover it.
- [ ] Important design decisions have been documented in the appropriate ADR inside the [docs/development/ADRs/](docs/development/ADRs/README.md) folder. N/A.
# Conflicts:
#	src/gt4py/eve/utils/__init__.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant