gh-154904: Speed up import of shutil by probing compression extensions

[maxday]

Speed up import shutil by probing compression extensions instead of importing the wrappers

shutil imports bz2, lzma and compression.zstd at module scope only to set the _*_SUPPORTED booleans, then discards them, also dragging incompression._common[._streams] and compression.zstd._zstdfile. Every program importing shutil pays for it even if it never touches an archive. They are pure Python wrappers whose only dependency that can be missing is the extension they wrap, so probe that directly:

-    import bz2
-    del bz2
+    import _bz2
+    del _bz2

Same for lzma→_lzma, compression.zstd→_zstd. zlib is left alone: it is the extension. tarfile/zipfile still import the wrappers when an archive is created or extracted.

Net import cost (median of 40 fresh interpreters, minus -c pass, one core):

	before	after	delta
import shutil	12.75 ms	10.25 ms	−2.50 ms (−19.6%)
import urllib.request	36.64 ms	34.30 ms	−2.34 ms (−6.4%)

urllib.request benefits via tempfile→shutil. Motivation is short-lived processes on constrained hardware (CLI tools, serverless cold starts).

No API change: flags, both registries and get_{archive,unpack}_formats() are unchanged, including on builds missing a compression library. The probe is a real import, so ImportError semantics are exact, unlike find_spec('bz2'), which resolves the always-present wrapper and reports success where _bz2 failed to compile (the case on my machine). PEP 810 lazy import also won't do: the flags are read at module scope (shutil.py:1145, :1354), so must be final at import time.

Tests

Added to TestArchives: test_compression_supported_flags pins the flags to real import semantics, test_compression_wrappers_not_imported_by_shutil prevents the eager imports returning. Existing test_unpack_archive_* coverage unchanged; patchcheck clean.

$ ./python -m test test_shutil -v -m "test_compression*"   ok, ok
$ ./python -m test test_shutil test_zipfile test_tarfile test_importlib \
    test_site test_venv test_zipimport test_bz2 test_lzma test_zstd \
    test_gzip test_urllib2 test_logging -j6
  SUCCESS — 14 OK, run=3,959 skipped=314 (test_bz2 skips on main too)

negative controls, each test fails without the half it covers:
  find_spec variant:  ..._supported_flags FAIL: True != False
  unpatched main:     ..._wrappers_not_imported FAIL:
                      "['lzma', 'compression', 'compression.zstd']" != '[]'

fixes: #154904

• Issue: shutil: avoid importing bz2, lzma and compression.zstd just to probe availability #154904

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[aisk]

Maybe we can use ensure_lazy_imports to make it cleaner:

Suggested change

	def test_compression_wrappers_not_imported_by_shutil(self):
	# Importing shutil must not pull in the compression wrappers: they are
	# only needed once an archive is actually created or extracted, and
	# importing them measurably slows down every process that uses shutil.
	wrappers = ('bz2', 'lzma', 'compression', 'compression.zstd')
	script = (
	'import sys, shutil; '
	f'print([m for m in {wrappers!r} if m in sys.modules])'
	)
	# -I so that a sitecustomize/usercustomize importing one of these
	# cannot make the test fail spuriously.
	rc, stdout, stderr = assert_python_ok('-I', '-c', script)
	self.assertEqual(stdout.decode().strip(), '[]', stderr)
	def test_compression_wrappers_not_imported_by_shutil(self):
	# gh-154904: Importing shutil must not pull in the compression wrappers: they are
	# only needed once an archive is actually created or extracted, and
	# importing them measurably slows down every process that uses shutil.
	ensure_lazy_imports("shutil", {"bz2", "lzma", "compression", "compression.zstd"})

I'm not test it, and you need to add the import statement at the top of this file:

from test.support.import_helper import ensure_lazy_imports

[aisk]

I'm not quite sure if test_compression_supported_flags is needed here. The wrappers already import the extension modules at the top level, so this test can hardly fail. For example, the zlib subtest just compares the result of import zlib with itself.

[maxday]

Thanks for the review @aisk, both applied.

One open question: I left off @support.cpython_only. Nearly all existing ensure_lazy_imports call sites carry it (test_argparse.py is the exception), so happy to add it back if that's the preference :)

[aisk]

I guess it's the behavior of don't add a placeholder object to the module, when lazy importing, is the implement detail of CPython, so added the decorator to let other implementation to skip this.

In this PR, we don't use lazy import, so we should not add it.

[maxday]

Thanks @aisk leaving it as is then! Let me know if you have other comments

[hugovk]

cc @emmatyping who may be interested in this.

---

Another idea, replace the three try/excepts with:

from importlib.util import find_spec

_BZ2_SUPPORTED = find_spec('_bz2') is not None
_LZMA_SUPPORTED = find_spec('_lzma') is not None
_ZSTD_SUPPORTED = find_spec('_zstd') is not None

Gives:

Before: 8 ms	PR: 6 ms	find_spec: 5 ms

[maxday]

Thanks @hugovk! I did try that first, and it's what pushed me to the import probe. As fas as I understandfind_spec answers a different question: it asks whether a module can be located, not whether it can be initialised. For an extension module those come apart, and when they do find_spec reports success where the import fails.

find_spec('_bz2') returns a spec as soon as ExtensionFileLoader finds the .so on the path. Everything that happens after that: dlopen, symbol resolution, PyInit__bz2, module exec, is where a broken extension actually fails, and find_spec never gets there.

Let's say the extension is compiled against a library that isn't present at runtime (ie: missing/mismatched libbz2). The .so is on disk, dlopen fails:

find_spec('_bz2')  -> True
import _bz2        -> ImportError: libbz2.so.1: cannot open shared object file

Let me know if I'm missing something, happy to use find_spec if you think that scenario can not happen but it might feels a bit safer to use import?

[hugovk]

Ah yes, it is a change in behaviour. With main/this PR if the .so is corrupt, and we try and use it, we get:

ValueError: unknown archive format 'bztar'

With find_spec:

tarfile.CompressionError: bz2 module is not available

Which is maybe more informative? Anyway, it's probably not worth changing the behaviour for this small extra speedup, especially as shutil.get_archive_formats() then also claims "bztar"is available.

[maxday]

Thanks for confirming!