Repository navigation
Conversation
Move the contents of the 1900-line `eve/utils.py` module, unchanged, into thematic submodules of a new `eve.utils` subpackage: - `caching`: `lru_cache`, `optional_lru_cache` - `classes`: `get_fully_qualified_name`, `register_subclasses`, `noninstantiable`, ... - `containers`: `CustomDefaultDictBase`, `CustomMapping`, `Namespace`, `FrozenNamespace` - `dispatching`: `singledispatcher`, `merge_dispatchers` - `functional`: attribute/item getters and checkers, `IndexerCallable`, `fluid_partial` - `hashing`: `HashableBy`, `content_hash`, `ddiff`, `dhash`, ... - `iterators`: `XIterable`, `xiter`, `xenumerate`, `first`, ... - `strings`: `CaseStyleConverter`, `SequentialIDGenerator` The package `__init__` re-exports every public name (including the `boltons` and `toolz` re-exports), so `gt4py.eve.utils.<name>` keeps working unchanged and the existing tests are untouched.
…d subpackages Add the `eve.utils.lazy_imports` submodule with `lazy_import_module()`, `is_lazy_module()` and `resolve_lazy_module()`, re-exported from `eve.utils`. The returned proxy is not registered in `sys.modules` and resolves itself with a plain `importlib.import_module()` on first attribute access; availability is checked without executing any module code, also for subpackages. Design rationale and rejected alternatives in docs/development/ADRs/eve/0001-Lazy_Module_Imports.md.
…alIDGenerator into eve.utils.classes Address review: the custom mappings and namespaces (formerly `eve.utils.containers`), `IndexerCallable` (from `functional`) and `SequentialIDGenerator` (from `strings`) now live in `eve.utils.classes`, whose module docstring is updated accordingly.
# Conflicts: # src/gt4py/eve/utils/__init__.py
…e.utils module Re-export `deepdiff`, `xxhash`, `xtyping`, `NOTHING`, `NothingType` and `ArgsOnlyCallable` from `eve.utils` for backwards compatibility, and pin the public names of the package in a test.
….utils imports Address review: `deepdiff`, `xxhash`, `xtyping`, `NOTHING`, `NothingType` and `ArgsOnlyCallable` are no longer re-exported from `eve.utils`.
# Conflicts: # tests/eve_tests/unit_tests/test_utils.py
…re-exporting TypeVars Address review: the `toolz` re-export now precedes the internal imports, and the `T`, `S`, `K` and `P` TypeVars are no longer re-exported from `eve.utils`.
Address review.
# Conflicts: # tests/eve_tests/unit_tests/test_utils.py
egparedes
left a comment
There was a problem hiding this comment.
Initial thoughts
There was a problem hiding this comment.
Drop the ADR and add the really important information into docstrings
There was a problem hiding this comment.
Done in 3720e5e: the ADR (and the docs/development/ADRs/eve/ folder, which only held it) is removed. The important parts now live in docstrings:
- the
eve.utils.lazy_importsmodule docstring: why the proxy is kept out ofsys.modules(vslazy_loader.load()/importlib.util.LazyLoader), the semantics and the caveats; - the
resolve_lazy_module()docstring: what to check when migrating a call site (identity uses).
There was a problem hiding this comment.
Add mentions to the 3.15 lazy imports in comments or in the module docstring. The goal is to make the current implementation just an intermediate temporary interoperable step towards 3.15 lazy imports features, so this implementation should follow a similar looking approach, and should be easy to drop in the future without larger changes in the rest of the codebase.
There was a problem hiding this comment.
Done in 3720e5e. The module docstring now presents the module as a temporary stand-in for PEP 810 explicit lazy imports (Python 3.15), and the lazy_import_module() docstring points to it.
- Shared semantics: the module stays out of
sys.modulesuntil first use, it is then imported with a plain import, and a failed import is retried on the next use. These match PEP 810, so call sites can move tolazy import ...once 3.15 is the minimum version. - Differences call sites must handle:
- the proxy is not rebound to the real module, so identity checks need
resolve_lazy_module(); - attribute access goes through the proxy, and
dir()imports the module; - availability is checked eagerly. PEP 810 forbids
lazy importinsidetryblocks, so optional dependencies will still need an explicit availability check.
- the proxy is not rebound to the real module, so identity checks need
- Intermediate step: the docstring also mentions the
__lazy_modules__global for modules that are always installed.
…me them as a step towards PEP 810 Address review: drop the ADR, and document in the `eve.utils.lazy_imports` module docstring that it is a temporary stand-in for Python 3.15 explicit lazy imports (PEP 810), with the differences and the migration path.
…zy import availability check Address review findings: - Finders raising because the parent package is not imported (e.g. `PathFinder` for namespace portions: `KeyError` on Python <= 3.14) no longer crash `lazy_import_module()`; the parent is imported and the lookup retried, as for other modules not found statically. - Submodules registered in `sys.modules` by non-package parents are found, as with plain imports. - Docstrings no longer claim that the availability check never executes module code, and no longer list `dir()` as a difference with PEP 810. - Tests for namespace packages, aliases from non-package parents, non-`ImportError` failures, success after failure, `importlib.reload()` and concurrent calls; the thread test now really jitters each thread.
…ty check imports modules Address review findings: - Docstrings state that finding a module may import a parent package (and with it, possibly the module itself, which is then returned directly), and name `importlib.import_module()` as the import mechanism. - Single `sys.modules` lookups, robust to concurrent modifications. - Empty, relative or malformed names raise `ValueError`, as plain imports do. - The namespace package test looks up the deepest name first, so it covers the fallback; new tests for a target imported by its parent, proxies with `on_missing="none"` and invalid names.
## Description Turns the 1900-line `eve/utils.py` module into an `eve.utils` subpackage. Every function, class and constant is moved **unchanged**: their ASTs are identical to the originals (see the verification comment below). The TypeVars (`T`, `P`, `_P`, `_T`, …) are copied verbatim into each submodule that uses them. Submodules: | Submodule | Contents | |---|---| | `caching` | `lru_cache`, `optional_lru_cache` | | `classes` | `get_fully_qualified_name`, `register_subclasses`, `noninstantiable`, `is_noninstantiable`, `IndexerCallable`, `CustomDefaultDictBase`, `CustomMapping`, `Namespace`, `FrozenNamespace`, `SequentialIDGenerator` | | `dispatching` | `singledispatcher`, `merge_dispatchers` | | `functional` | `isinstancechecker`, `attrchecker`, `attrgetter_`, `getitem_`, `itemgetter_`, `fluid_partial`, `with_fluid_partial` | | `hashing` | `HashableBy`, `hashable_by`, `hashable_by_id`, `cached_hash`, `EqualityBy`, `content_hash`, `ddiff`, `dhash`, `pprint_ddiff` | | `iterators` | `XIterable`, `xiter`, `xenumerate`, `as_xiter`, `first` | | `strings` | `AnyWordsIterable`, `CaseStyleConverter` | The package `__init__` re-exports the public functions, classes and constants of the old module, plus the `boltons` helpers and `toolz`, so `gt4py.eve.utils.<name>` keeps working for all of them. No call site changes, and the tests are unchanged. **Compatibility:** - **Still importable from `eve.utils`:** every public function, class and constant of the old module, the `boltons` helpers and `toolz`. - **No longer reachable through `eve.utils`:** - names the old module only imported for its own use: `deepdiff`, `xxhash`, `xtyping`, `NOTHING`, `NothingType`, `ArgsOnlyCallable`; - TypeVars (`T`, `S`, `K`, `P` and the private ones), which are now declared privately in each submodule that uses them; - private names: `_CONTENT_HASH_PICKLE_PROTOCOL`, … Nothing in the repo uses any of them through `eve.utils`. Objects now report the submodule in `__module__` (e.g. `gt4py.eve.utils.classes.FrozenNamespace`). Pickles created before this refactor still load, because the old paths resolve through the re-exports (verified in a comment below). Next in the stack: #3020 adds the lazy import utilities as a new `eve.utils.lazy_imports` submodule. ## Requirements - [x] All fixes and/or new features come with corresponding tests. Pure refactor: the existing tests and doctests cover it. - [ ] Important design decisions have been documented in the appropriate ADR inside the [docs/development/ADRs/](docs/development/ADRs/README.md) folder. N/A.
# Conflicts: # src/gt4py/eve/utils/__init__.py
Description
Follows #3019 (merged), which turned
eve.utilsinto a subpackage.Adds the
gt4py.eve.utils.lazy_importssubmodule withlazy_import_module()(re-exported asgt4py.eve.utils.lazy_import_module()), which returns a module stand-in and only imports the module on first attribute access. It follows the idea oflazy_loader.load()but also works for subpackages (e.g."cupyx.profiler","jax.numpy") without importing their parents up front.sys.meta_pathfinders. Imported parents are searched through their runtime__path__. If a child isn't found under a parent that isn't imported yet, the parent is imported for real and the lookup is retried.on_missing="raise"raisesModuleNotFoundErrorfor a library that isn't installed;"none"returnsNone. This covers libraries that aren't installed; an installed library that fails to import raises on first access.sys.modules; it resolves with a plainimportlib.import_module(). Import order, locking, import-time side effects and rollback on failure therefore behave exactly as with a normal import.eve.utils):resolve_lazy_module()returns the real module, for code that needs module identity (is,==, dict keys).is_lazy_module()tells whether an object is one of these stand-ins.Towards Python 3.15 lazy imports (PEP 810)
This is a temporary, interoperable step towards the explicit
lazy importstatements of Python 3.15 (PEP 810), which GT4Py can't use while it supports 3.12–3.14.sys.modulesuntil its first use, it is then imported with a plain import, and a failed import is retried on the next use.lazy import ...once 3.15 is the minimum version. The module docstring lists the differences to handle until then.Why not
lazy_loader.load()/importlib.util.LazyLoaderThe
eve.utils.lazy_importsmodule docstring has the rationale. In short:lazy.load()/importlib.util.LazyLoader, a subpackage either imports its parent eagerly or runs before its parent.sys.modulesgets loaded by any scan ofsys.modules. gt4py itself scans it while importing (inspect.getmodule()viaeve.codegen).ImportErrorinside unrelated code, e.g.array_api_compat.array_namespace().The design went through four rounds of adversarial review, with experiments on CPython 3.12–3.14.
Not in this PR
Migrating existing
try: import x / except ImportError: x = Nonesites. These module-identity sites need to be checked when that happens:_core/ndarray_utils.pynext/field_utils.pynext/embedded/array_ops.pynext.fingerprinting.strict_fingerprinterRequirements
eve.utils.lazy_importsmodule docstring instead.