Example

Say we have a modulespam.py:

# simulate some work
importtime
time.sleep(10)
print("spam loaded")

And a moduleeggs.pywhich imports it:

importspam
print("imports done")

If we runPython-Leggs.py,thespammodule will never be imported (because it is never referenced after the import),"spamloaded "will never be printed, and there will be no 10 second delay.

But ifeggs.pysimply references the namespamafter importing it, that will be enough to trigger the import ofspam.py:

importspam
print("imports done")
spam

Now if we runPython-Leggs.py,we will see the output"importsdone " printed first, then a 10 second delay, and then"spamloaded "printed after that.

Of course, in real use cases (especially with lazy imports), it’s not recommended to rely on import side effects like this to trigger real work. This example is just to clarify the behavior of lazy imports.

Another way to explain the effect of lazy imports is that it is as if each lazy import statement had instead been written inline in the source code immediately before each use of the imported name. So one can think of lazy imports as similar to transforming this code:

importfoo

deffunc1():
returnfoo.bar()

deffunc2():
returnfoo.baz()

To this:

deffunc1():
importfoo
returnfoo.bar()

deffunc2():
importfoo
returnfoo.baz()

This gives a good sense of when the import offoowill occur under lazy imports, but lazy import is not really equivalent to this code transformation. There are several notable differences:

• Unlike in the latter code, under lazy imports the namefoostill does exist in the module’s global namespace, and can be imported or referenced by other modules that import this one. (Such references would also trigger the import.)
• The runtime overhead of lazy imports is much lower than the latter code; after the first reference to the namefoowhich triggers the import, subsequent references will have zero import system overhead; they are indistinguishable from a normal name reference.

In a sense, lazy imports turn the import statement into just a declaration of an imported name or names, to later be fully resolved when referenced.

An import in the stylefromfooimportbarcan also be made lazy. When the import occurs, the namebarwill be added to the module namespace as a lazy import. The first reference tobarwill importfooand resolvebar tofoo.bar.

Intended usage

Since lazy imports are a potentially-breaking semantic change, they should be enabled only by the author or maintainer of a Python application, who is prepared to thoroughly test the application under the new semantics, ensure it behaves as expected, and opt-out any specific imports as needed (see below). Lazy imports should not be enabled speculatively by the end user of a Python application with any expectation of success.

It is the responsibility of the application developer enabling lazy imports for their application to opt-out any library imports that turn out to need to be eager for their application to work correctly; it is not the responsibility of library authors to ensure that their library behaves exactly the same under lazy imports.

The documentation of the feature, the-Lflag, and the newimportlib APIs will be clear about the intended usage and the risks of adoption without testing.

Implementation

Lazy imports are represented internally by a “lazy import” object. When a lazy import occurs (sayimportfooorfromfooimportbar), the key"foo" or"bar"is immediately added to the module namespace dictionary, but with its value set to an internal-only “lazy import” object that preserves all the necessary metadata to execute the import later.

A new boolean flag inPyDictKeysObject(dk_lazy_imports) is set to signal that this particular dictionary may contain lazy import objects. This flag is only used to efficiently resolve all lazy objects in “bulk” operations, when a dictionary may contain lazy objects.

Anytime a key is looked up in a dictionary to extract its value, the value is checked to see if it is a lazy import object. If so, the lazy object is immediately resolved, the relevant imported modules executed, the lazy import object is replaced in the dictionary (if possible) by the actual imported value, and the resolved value is returned from the lookup function. A dictionary could mutate as part of an import side effect while resolving a lazy import object. In this case it is not possible to efficiently replace the key value with the resolved object. In this case, the lazy import object will gain a cached pointer to the resolved object. On next access that cached reference will be returned and the lazy import object will be replaced in the dict with the resolved value.

Because this is all handled internally by the dictionary implementation, lazy import objects can never escape from the module namespace to become visible to Python code; they are always resolved at their first reference. No stub, dummy or thunk objects are ever visible to Python code or placed insys.modules. If a module is imported lazily, no entry for it will appear insys.modules at all until it is actually imported on first reference.

If two different modules (modaandmodb) both contain a lazyimport foo,each module’s namespace dictionary will have an independent lazy import object under the key"foo",delaying import of the samefoomodule. This is not a problem. When there is first a reference to, say,moda.foo,the modulefoowill be imported and placed insys.modulesas usual, and the lazy object under the keymoda.__dict__[ "foo" ]will be replaced by the actual modulefoo.At this pointmodb.__dict__[ "foo" ]will remain a lazy import object. Whenmodb.foois later referenced, it will also try to importfoo.This import will find the module already present in sys.modules,as is normal for subsequent imports of the same module in Python, and at this point will replace the lazy import object at modb.__dict__[ "foo" ]with the actual modulefoo.

There are two cases in which a lazy import object can escape a dictionary:

• Into another dictionary: to preserve the performance of bulk-copy operations likedict.update()anddict.copy(),they do not check for or resolve lazy import objects. However, if the source dict has thedk_lazy_imports flag set that indicates it may contain lazy objects, that flag will be passed on to the updated/copied dictionary. This still ensures that the lazy import object can’t escape into Python code without being resolved.
• Through the garbage collector: lazy imported objects are still Python objects and live within the garbage collector; as such, they can be collected and seen via e.g.gc.get_objects().If a lazy object becomes visible to Python code in this way, it is opaque and inert; it has no useful methods or attributes. Arepr()of it would be shown as something like: <lazy_object'fully.qualified.name'>.

When a lazy object is added to a dictionary, the flagdk_lazy_importsis set. Once set, the flag is only cleared ifalllazy import objects in the dictionary are resolved, e.g. prior to dictionary iteration.

All dictionary iteration methods involving values (such asdict.items(), dict.values(),PyDict_Next()etc.) will attempt to resolvealllazy import objects in the dictionary prior to starting the iteration. Since only (some) module namespace dictionaries will ever havedk_lazy_importsset, the extra overhead of resolving all lazy import objects inside a dictionary is only paid by those dictionaries that need it. Minimizing the overhead on normal non-lazy dictionaries is the sole purpose of thedk_lazy_importsflag.

PyDict_Nextwill attempt to resolve all lazy import objects the first time position0is accessed, and those imports could fail with exceptions. Since PyDict_Nextcannot set an exception,PyDict_Nextwill return0 immediately in this case, and any exception will be printed to stderr as an unraisable exception.

For this reason, this PEP introducesPyDict_NextWithError,which works in the same way asPyDict_Next,but which can set an error when returning0 and this should be checked viaPyErr_Occurred()after the call.

The eagerness of imports withintry/except/withblocks or within class or function bodies is handled in the compiler via a new EAGER_IMPORT_NAMEopcode that always imports eagerly. Top-level imports use IMPORT_NAME,which may be lazy or eager depending on-Land/or importlib.set_lazy_imports().

Debugging

Debug logging fromPython-vwill include logging whenever an import statement has been encountered but execution of the import will be deferred.

Python’s-Ximporttimefeature for profiling import costs adapts naturally to lazy imports; the profiled time is the time spent actually importing.

Although lazy import objects are not generally visible to Python code, in some debugging cases it may be useful to check from Python code whether the value at a given key in a given dictionary is a lazy import object, without triggering its resolution. For this purpose,importlib.is_lazy_import()can be used:

fromimportlibimportis_lazy_import

importfoo

is_lazy_import(globals(),"foo")

foo

is_lazy_import(globals(),"foo")

In this example, if lazy imports have been enabled the first call to is_lazy_importwill returnTrueand the second will returnFalse.

Per-module opt-out

Due to the backwards compatibility issues mentioned below, it may be necessary for an application using lazy imports to force some imports to be eager.

In first-party code, since imports inside atryorwithblock are never lazy, this can be easily accomplished:

try:# force these imports to be eager
importfoo
importbar
finally:
pass

This PEP proposes to add a newimportlib.eager_imports()context manager, so the above technique can be less verbose and doesn’t require comments to clarify its intent:

fromimportlibimporteager_imports

witheager_imports():
importfoo
importbar

Since imports within context managers are always eager, theeager_imports() context manager can just be an alias to a null context manager. The context manager’s effect is not transitive:fooandbarwill be imported eagerly, but imports within those modules will still follow the usual laziness rules.

The more difficult case can occur if an import in third-party code that can’t easily be modified must be forced to be eager. For this purpose, importlib.set_lazy_imports()takes a second optional keyword-only excludingargument, which can be set to a container of module names within which all imports will be eager:

fromimportlibimportset_lazy_imports

set_lazy_imports(excluding=["one.mod","another"])

The effect of this is also shallow: all imports withinone.modwill be eager, but not imports in all modules imported byone.mod.

Theexcludingparameter ofset_lazy_imports()can be a container of any kind that will be checked to see whether it contains a module name. If the module name is contained in the object, imports within it will be eager. Thus, arbitrary opt-out logic can be encoded in a__contains__method:

importre
fromimportlibimportset_lazy_imports

classChecker:
def__contains__(self,name):
returnre.match(r"foo\.[^.]+\.logger",name)

set_lazy_imports(excluding=Checker())

If Python was executed with the-Lflag, then lazy imports will already be globally enabled, and the only effect of callingset_lazy_imports(True, excluding=...)will be to globally set the eager module names/callback. If set_lazy_imports(True)is called with noexcludingargument, the exclusion list/callback will be cleared and all eligible imports (module-level imports not intry/except/with,and notimport*) will be lazy from that point forward.

This opt-out system is designed to maintain the possibility of local reasoning about the laziness of an import. You only need to see the code of one module, and theexcludingargument toset_lazy_imports,if any, to know whether a given import will be eager or lazy.

Testing

The CPython test suite will pass with lazy imports enabled (with some tests skipped). One buildbot should run the test suite with lazy imports enabled.

C API

For authors of C extension modules, the proposed public C API is as follows:

• voidPyDict_ResolveLazyImports(PyObject*dict)resolves all lazy objects in a dictionary, if any. To be used prior callingPyDict_NextWithError() orPyDict_Next().
• PyDict_NextWithError(),works the same way asPyDict_Next(),with the exception it propagates any errors to the caller by returning0and setting an exception. The caller should usePyErr_Occurred()to check for any errors.

Import Side Effects

Import side effects that would otherwise be produced by the execution of imported modules during the execution of import statements will be deferred until the imported objects are used.

These import side effects may include:

• code executing any side-effecting logic during import;
• relying on imported submodules being set as attributes in the parent module.

A relevant and typical affected case is theclicklibrary for building Python command-line interfaces. If e.g.cli=click.group()is defined inmain.py,and sub.pyimportsclifrommainand adds subcommands to it via decorator (@cli mand(...)), but the actualcli()call is in main.py,then lazy imports may prevent the subcommands from being registered, since in this case Click is depending on side effects of the import ofsub.py.In this case the fix is to ensure the import ofsub.pyis eager, e.g. by using theimportlib.eager_imports()context manager.

Dynamic Paths

There could be issues related to dynamic Python import paths; particularly, adding (and then removing after the import) paths fromsys.path:

sys.path.insert(0,"/path/to/foo/module")
importfoo
delsys.path[0]
foo.Bar()

In this case, with lazy imports enabled, the import offoowill not actually occur while the addition tosys.pathis present.

An easy fix for this (which also improves the code style and ensures cleanup) would be to place thesys.pathmodifications in a context manager. This resolves the issue, since imports inside awithblock are always eager.

Deferred Exceptions

Exceptions that occur during a lazy import bubble up and erase the partially-constructed module(s) fromsys.modules,just as exceptions during normal import do.

Since errors raised during a lazy import will occur later than they would if the import were eager (i.e. wherever the name is first referenced), it is also possible that they could be accidentally caught by exception handlers that did not expect the import to be running within theirtryblock, leading to confusion.

Wrapping deferred exceptions

To reduce the potential for confusion, exceptions raised in the course of executing a lazy import could be replaced by aLazyImportError exception (a subclass ofImportError), with a__cause__set to the original exception.

Ensuring that all lazy import errors are raised asLazyImportErrorwould reduce the likelihood that they would be accidentally caught and mistaken for a different expected exception. However, in practice we have seen cases, e.g. inside tests, where failing modules raiseunittest.SkipTestexception and this too would end up being wrapped inLazyImportError,making such tests fail because the true exception type is hidden. The drawbacks here seem to outweigh the hypothetical case where unexpected deferred exceptions are caught by mistake.

Per-module opt-in

A per-module opt-in using future imports (i.e. from__future__importlazy_imports) does not make sense because __future__imports are not feature flags, they are for transition to behaviors which will become default in the future. It is not clear if lazy imports will ever make sense as the default behavior, so we should not promise this with a__future__import.

There are other cases where a library might desire to locally opt-in to lazy imports for a particular module; e.g. a lazy top-level__init__.pyfor a large library, to make its subcomponents accessible as lazy attributes. For now, to keep the feature simpler, this PEP chooses to focus on the “application” use case and does not address the library use case. The underlying laziness mechanism introduced in this PEP could be used in the future to address this use case as well.

Explicit syntax for individual lazy imports

If the primary objective of lazy imports were solely to work around import cycles and forward references, an explicitly-marked syntax for particular targeted imports to be lazy would make a lot of sense. But in practice it would be very hard to get robust startup time or memory use benefits from this approach, since it would require converting most imports within your code base (and in third-party dependencies) to use the lazy import syntax.

It would be possible to aim for a “shallow” laziness where only the top-level imports of subsystems from the main module are made explicitly lazy, but then imports within the subsystems are all eager. This is extremely fragile, though – it only takes one mis-placed import to undo the carefully constructed shallow laziness. Globally enabling lazy imports, on the other hand, provides in-depth robust laziness where you always pay only for the imports you use.

There may be use cases (e.g. for static typing) where individually-marked lazy imports are desirable to avoid forward references, but the perf/memory benefits of globally lazy imports are not needed. Since this is a different set of motivating use cases and requires new syntax, we prefer not to include it in this PEP. Another PEP could build on top of this implementation and propose the additional syntax.

Environment variable to enable lazy imports

Providing an environment variable opt-in lends itself too easily to abuse of the feature. It may seem tempting for a Python user to, for instance, globally set the environment variable in their shell in the hopes of speeding up all the Python programs they run. This usage with untested programs is likely to lead to spurious bug reports and maintenance burden for the authors of those tools. To avoid this, we choose not to provide an environment variable opt-in at all.

Removing the-Lflag

We do provide the-LCLI flag, which could in theory be abused in a similar way by an end user running an individual Python program that is run with Pythonsomescript.pyorPython-msomescript(rather than distributed via Python packaging tools). But the potential scope for misuse is much less with-Lthan an environment variable, and-Lis valuable for some applications to maximize startup time benefits by ensuring that all imports from the start of a process will be lazy, so we choose to keep it.

It is already the case that running arbitrary Python programs with command line flags they weren’t intended to be used with (e.g.-s,-S,-E,or -I) can have unexpected and breaking results.-Lis nothing new in this regard.

Half-lazy imports

It would be possible to eagerly run the import loader to the point of finding the module source, but then defer the actual execution of the module and creation of the module object. The advantage of this would be that certain classes of import errors (e.g. a simple typo in the module name) would be caught eagerly instead of being deferred to the use of an imported name.

The disadvantage would be that the startup time benefits of lazy imports would be significantly reduced, since unused imports would still require a filesystem stat()call, at least. It would also introduce a possibly non-obvious split betweenwhichimport errors are raised eagerly and which are delayed, when lazy imports are enabled.

This idea is rejected for now on the basis that in practice, confusion about import typos has not been an observed problem with the reference implementation. Generally delayed imports are not delayed forever, and errors show up soon enough to be caught and fixed (unless the import is truly unused.)

Another possible motivation for half-lazy imports would be to allow modules themselves to control via some flag whether they are imported lazily or eagerly. This is rejected both on the basis that it requires half-lazy imports, giving up some of the performance benefits of import laziness, and because in general modules do not decide how or when they are imported, the module importing them decides that. There isn’t clear rationale for this PEP to invert that control; instead it just provides more options for the importing code to make the decision.

Lazy dynamic imports

It would be possible to add alazy=Trueor similar option to __import__()and/orimportlib.import_module(),to enable them to perform lazy imports. That idea is rejected in this PEP for lack of a clear use case. Dynamic imports are already far outside thePEP 8code style recommendations for imports, and can easily be made precisely as lazy as desired by placing them at the desired point in the code flow. These aren’t commonly used at module top level, which is where lazy imports applies.

Deep eager-imports override

The proposedimportlib.eager_imports()context manager and excluded modules in theimportlib.set_lazy_imports(excluding=...)override all have shallow effects: they only force eagerness for the location they are applied to, not transitively. It would be possible to provide a deep/transitive version of one or both. That idea is rejected in this PEP because the implementation would be complex (taking into account threads and async code), experience with the reference implementation has not shown it to be necessary, and because it prevents local reasoning about laziness of imports.

A deep override can lead to confusing behavior because the transitively-imported modules may be imported from multiple locations, some of which use the “deep eager override” and some of which don’t. Thus those modules may still be imported lazily initially, if they are first imported from a location that doesn’t have the override.

With deep overrides it is not possible to locally reason about whether a given import will be lazy or eager. With the behavior specified in this PEP, such local reasoning is possible.

Making lazy imports the default behavior

Making lazy imports the default/sole behavior of Python imports, instead of opt-in, would have some long-term benefits, in that library authors would (eventually) no longer need to consider the possibility of both semantics.

However, the backwards-incompatibilies are such that this could only be considered over a long time frame, with a__future__import. It is not at all clear that lazy imports should become the default import semantics for Python.

This PEP takes the position that the Python community needs more experience with lazy imports before considering making it the default behavior, so that is entirely left to a possible future PEP.