diff --git a/peps/pep-0842.rst b/peps/pep-0842.rst index 9fb923db2da..4396efa8519 100644 --- a/peps/pep-0842.rst +++ b/peps/pep-0842.rst @@ -14,7 +14,7 @@ Abstract ======== This PEP proposes an ``__export__`` variable that modules can define to -limit visibility and access to variables from outside the module. +express intent about the visibility of variables from outside the module. For example: @@ -33,25 +33,37 @@ For example: .. code-block:: pycon >>> import spam + >>> 'Public' in dir(spam) + True + >>> 'Private' in dir(spam) + False >>> spam.Public >>> spam.Private - Traceback (most recent call last): - File "", line 1, in - spam.Private - ImportError: 'Private' is not exported by 'spam' + :1: RuntimeWarning: 'Private' is not exported by 'spam' + + + +This is **not** intended to be an access modifier for Python; see +:ref:`pep-842-not-an-access-modifier`. Motivation ========== -Private names can be difficult to disambiguate on their own ------------------------------------------------------------ +Module-level names need privacy +------------------------------- + +A developer is writing a Python module. The module is intended to have one +"public" class -- a class that is intended for users of the module -- called +``PublicAPI``. As part of implementing ``PublicAPI``, the developer wants to +create another class, called ``Helper``. However, ``Helper`` is not meant to +be public in the same way that ``PublicAPI`` is public. ``Helper`` is supposed +to only be used by the developer of the module -- a "private" API. + +Nonetheless, the developer declares the two classes as such: -Imagine that a developer wants to define a private class in -their module. Their first instinct might be to simply define their class -as such: .. code-block:: python @@ -59,18 +71,22 @@ as such: class Helper: ... -However, this comes with no indication that ``Helper`` is supposed to be -internal to the module, so users may accidentally begin using and relying -on it. In fact, Python's interactive :func:`help` function will even include -``Helper`` in its output next to everything else in the module. + class PublicAPI: + ... + + +The problem with this is that ``Helper`` comes with no indication that it's not +a public API. It shows up in autocomplete by language servers, the :func:`dir` +function, Python's interactive :func:`help` function, and every other API meant +for introspection. How are users supposed to know that they aren't supposed to +use this? -Prefixed names are less maintainable ------------------------------------- +Prefixed names aren't necessarily a great solution +-------------------------------------------------- -When developing a Python module, it is common to prefix a name with ``_`` -to denote that it is private. So, as a solution to the above problem, -the developer prefixes the name with ``_``: +In Python, the convention for declaring private names is to prefix it +with ``_``. So, the developer changes ``Helper`` into ``_Helper``: .. code-block:: python @@ -78,13 +94,29 @@ the developer prefixes the name with ``_``: class _Helper: ... +This is generally the standard for Python libraries today, but it's not clear +that this is the best long term solution. This works (with some caveats; see the +sections below), but this is (subjectively) less readable, and does require +more keystrokes by the maintainer. Ideally, users shouldn't be tempted to +reach for private names from modules in the first place. + +However, it is acknowledged that this idea is going against 30 years of +convention; even if this PEP is accepted, it's expected that "underscored" +names (names prefixed with a leading ``_``) will remain a staple of Python +for years to come. The purpose of this PEP is not to eliminate the need for +``_`` in module-level names, but instead to clear up corner cases where a +private name is ambiguous or tempting. In other words, this PEP is intended +to improve expressiveness and clarity with private APIs, *not* to add brand +new functionality. + -Now, it's clear to users that the name is internal, at the expense of the name -being (subjectively) less readable and requiring more keystrokes by the maintainer. +It's not always clear where names need prefixing +************************************************ -In addition, it can be difficult to remember where names need to be prefixed. -To put this issue into perspective, imagine that a developer wants to import -some other modules in their code: +Python defines names through many different constructs, some of which are not +always clear or intuitive to the developer. As a result, it can be difficult to +remember where names need to be prefixed. To put this issue into perspective, +imagine that a developer wants to import some other modules in their code: .. code-block:: python @@ -112,12 +144,95 @@ The solution to this is to also prefix every imported name with ``_``: import asyncio as _asyncio import tabnanny as _tabnanny -This brings us back to the original problem: this sprinkles the -code with extra underscores, and puts mental overhead on the developer -by requiring them to remember to prefix their imports with ``_``. -Ideally, users shouldn't be tempted to reach for private names from modules in -the first place. +But, again, this sprinkles the code with even more underscored names. + + +.. _pep-842-prefixed-public: + +Prefixed names are not a universal rule +*************************************** + +As modules evolve, some underscored names are made public, either because users +did not clearly understand that an underscore indicated instability, or because +users found useful functionality in a module's private API, and nothing was +discouraging them from using it. + +In the standard library, a prime example of this is the :mod:`ctypes` module. +``ctypes`` is full of public APIs that are subject to Python's backwards +compatibility policy, but contain a leading underscore. For example: + +1. :class:`ctypes._CFuncPtr` +2. :class:`ctypes._CData` +3. :class:`ctypes._Pointer` + +This sends the wrong message to consumers of the API. When seeing things like +this in a codebase, it makes it seem like the code is opting out of backwards +compatibility, or that an underscored name does not mean "private" in the +module. In both cases, consumers are inclined to reach for more private names +(because there's no apparent consequence for doing so), making this problem worse. + + +We want to be nice to users, not shrug them away +------------------------------------------------ + +When a user decides to use a private API, accidentally or not, they will +inevitably be broken by the library author. In many cases, this results +in a bug report asking for the API to be fixed or restored to prevent +downstream breakage. In this case, the library maintainer has to make a decision: + +1. Tell the user that they're in the wrong for using it, and allow the breakage + to take place. +2. Commit to maintaining the private API as public, increasing the burden on + themselves and encountering some of the problems described in + :ref:`pep-842-prefixed-public`. + +This PEP is not intended to solve this problem entirely, but instead is meant +to mitigate it by making it much clearer that a user is accessing a private name; +in other words, this PEP wants to decrease (or eliminate) the amount of accidental +private API usage in practice. By accessing a private API, the user must make a +conscious decision to do so. + + +Library consumers use runtime introspection for documentation +************************************************************* + +A counterargument to the above section is that a library should clearly +document what is private and what is public. In theory, yes, but in practice, +users don't read the documentation in full. + +A common practice when designing APIs is to design for intuition. If an API +is named and placed well, then a user often won't need to reach for the +documentation. Python is no exception to this. + +When prototyping, it's typical for someone to use :func:`dir` or :func:`help` +in Python's interactive :term:`REPL` to look for attributes that are useful to +them. In this case, if something is intuitive enough for the user, they will +simply reach for it without checking the documentation first. In a language +as dynamic as Python, the way people consume APIs is also dynamic. + + +``__all__`` is only a convention +-------------------------------- + +The fundamental issue here is that Python has no way to express which names +in a module are "private" or "public". Prefixing is an option, but given the +reasons above, it's not always a bulletproof solution for library authors. + +Currently, the other convention for expressing which names are public is +done through a module's ``__all__`` variable. This has two major downsides: + +1. ``__all__`` often gets out of sync, because as developers add, change, or + remove names from their module, there is often nothing pushing them towards + changing ``__all__``, because again, using it to list public names is only + a convention and not enforced by anything. +2. ``__all__`` is not always exhaustive. See the :ref:`rejected ideas + ` for examples on where the items in ``__all__`` + might only be a subset of the "public" names in a module. In short, it can + be difficult to control namespace pollution and declare all public names in + ``__all__`` simultaneously. + +This PEP intends to solve both of these problems with a new ``__export__`` variable. Specification @@ -129,6 +244,7 @@ Specification ``__export__`` rules -------------------- + Object requirements ******************* @@ -180,8 +296,8 @@ Module attribute access When ``__export__`` is present in a module's globals, all access to attributes present on the module object will also check if the attribute name is present in ``__export__`` (via ``__contains__`` or through iteration, as specified previously). -If the attribute name is not present in ``__export__``, then an :exc:`ImportError` -is raised. For example: +If the attribute name is not present in ``__export__``, then a :exc:`RuntimeWarning` +is emitted. For example: .. code-block:: python @@ -197,10 +313,14 @@ is raised. For example: >>> spam.a 42 >>> spam.b - Traceback (most recent call last): - File "", line 1, in - spam.b - ImportError: 'b' is not exported by 'spam' + :1: RuntimeWarning: 'b' is not exported by 'spam' + 24 + + +.. note:: + + This also affects ``from`` imports, because those use the same attribute + access mechanism. Dunder names @@ -223,32 +343,6 @@ For example: 'spam' -Lazy imports -************ - -:ref:`Lazy imports ` that are not listed in ``__export__`` -will not be reified upon being accessed outside the module. For example: - -.. code-block:: python - - # spam.py - lazy import json - - __export__ = [] - -.. code-block:: pycon - - >>> import spam, sys - >>> assert 'json' in sys.lazy_modules - >>> spam.json - Traceback (most recent call last): - File "", line 1, in - spam.json - ImportError: 'json' is not exported by 'spam' - >>> # json is still lazy and has not been resolved - >>> assert 'json' in sys.lazy_modules - - Module ``__getattr__`` functions -------------------------------- @@ -337,16 +431,6 @@ names that are not present in ``__export__``. For example: >>> dir(spam) [..., 'a', 'b'] -It is worth noting that there are real consequences for including unexported -names in custom ``__dir__`` functions. For example, :func:`help` can no longer -be used with the above module: - -.. code-block:: pycon - - >>> import spam - >>> help(spam) - 'b' is not exported by 'spam' - .. _pep-842-implicit-all: @@ -410,11 +494,13 @@ following code: .. code-block:: python - __all__ = __export__ + if "__all__" not in globals(): + __all__ = __export__ def _is_dunder_name(name): return (len(name) > 4) and name.startswith("__") and name.endswith("__") + # Attributes not in the __dict__ fall back to the normal lookup def __getattribute__(name): try: value = globals()[name] @@ -425,7 +511,7 @@ following code: return value if name not in __export__: - raise ImportError(f"{name!r} is not exported by {__name__!r}") + __import__("warnings").warn(f"{name!r} is not exported by {__name__!r}", RuntimeWarning, stacklevel=1) return value @@ -444,14 +530,15 @@ following code: Rationale ========= +.. _pep-842-not-an-access-modifier: -``__export__`` is not a secure access modifier ----------------------------------------------- +``__export__`` is not an access modifier +---------------------------------------- -This PEP does not aim to be a secure mechanism for preventing access to -private attributes in modules. In fact, bypassing ``__export__`` is trivial; -simply access ``mod.__dict__['attr_name']`` instead of ``mod.attr_name`` at -runtime. +This PEP does not aim to be a mechanism for preventing access to private +attributes in modules. The :exc:`RuntimeWarning` can be filtered away, +disabled, or bypassed (such as by accessing attributes through the module's +``__dict__``). This is by design. Python does not include access modifiers as a language feature for a reason. To `quote `__ Eric Smith: @@ -500,11 +587,21 @@ Reference Implementation A reference implementation of this PEP can be found `here `__. +Performance +----------- + +The reference implementation does not currently implement any optimizations +to reduce the overhead of the ``__export__`` lookup or iteration, meaning +that there is likely some overhead. However, if this PEP is accepted, +optimizations will be implemented before the feature lands in :term:`CPython`. + Rejected Ideas ============== +.. _pep-842-all-for-exports: + Reuse ``__all__`` for exports ----------------------------- @@ -549,6 +646,29 @@ syntax. Third-party solutions and widespread adoption would make it much clearer that new syntax is the best choice for Python in the long run. +Raising an exception upon accessing unexported attributes +--------------------------------------------------------- + +This PEP initially proposed raising an :exc:`ImportError` upon accessing +module attributes that were not listed in ``__export__``. For example: + +.. code-block:: pycon + + >>> import module + >>> module.unexported + Traceback (most recent call last): + File "", line 1, in + module.unexported + ImportError: 'unexported' is not exported by 'module' + + +This caused a lot of concern, as many were fundamentally uncomfortable with +the idea of introducing any notion of "private attributes" in Python. The purpose +of this proposal is to improve *expression* of private variables, not *security*. +As such, this proposal switched to emitting warnings when accessing unexported +names. + + Open Issues =========== @@ -566,7 +686,11 @@ behind this PEP. Change History ============== -TBD. +* 01-Aug-2026 + + - Accessing an unexported attribute now emits a :exc:`RuntimeWarning` instead + of raising an :exc:`ImportError`. + - Significantly overhauled the motivation section. Copyright