Permalink
Cannot retrieve contributors at this time
| PEP: 3149 | |
| Title: ABI version tagged .so files | |
| Version: $Revision$ | |
| Last-Modified: $Date$ | |
| Author: Barry Warsaw <barry@python.org> | |
| Status: Final | |
| Type: Standards Track | |
| Content-Type: text/x-rst | |
| Created: 09-Jul-2010 | |
| Python-Version: 3.2 | |
| Post-History: 2010-07-14, 2010-07-22 | |
| Resolution: https://mail.python.org/pipermail/python-dev/2010-September/103408.html | |
| Abstract | |
| ======== | |
| PEP 3147 [1]_ described an extension to Python's import machinery that | |
| improved the sharing of Python source code, by allowing more than one | |
| byte compilation file (.pyc) to be co-located with each source file. | |
| This PEP defines an adjunct feature which allows the co-location of | |
| extension module files (.so) in a similar manner. This optional, | |
| build-time feature will enable downstream distributions of Python to | |
| more easily provide more than one Python major version at a time. | |
| Background | |
| ========== | |
| PEP 3147 defined the file system layout for a pure-Python package, | |
| where multiple versions of Python are available on the system. For | |
| example, where the `alpha` package containing source modules `one.py` | |
| and `two.py` exist on a system with Python 3.2 and 3.3, the post-byte | |
| compilation file system layout would be:: | |
| alpha/ | |
| __pycache__/ | |
| __init__.cpython-32.pyc | |
| __init__.cpython-33.pyc | |
| one.cpython-32.pyc | |
| one.cpython-33.pyc | |
| two.cpython-32.pyc | |
| two.cpython-33.pyc | |
| __init__.py | |
| one.py | |
| two.py | |
| For packages with extension modules, a similar differentiation is | |
| needed for the module's .so files. Extension modules compiled for | |
| different Python major versions are incompatible with each other due | |
| to changes in the ABI. Different configuration/compilation options | |
| for the same Python version can result in different ABIs | |
| (e.g. --with-wide-unicode). | |
| While PEP 384 [2]_ defines a stable ABI, it will minimize, but not | |
| eliminate extension module incompatibilities between Python builds or | |
| major versions. Thus a mechanism for discriminating extension module | |
| file names is proposed. | |
| Rationale | |
| ========= | |
| Linux distributions such as Ubuntu [3]_ and Debian [4]_ provide more | |
| than one Python version at the same time to their users. For example, | |
| Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1, | |
| with Python 2.6 being the default. | |
| In order to share as much as possible between the available Python | |
| versions, these distributions install third party package modules | |
| (``.pyc`` and ``.so`` files) into `/usr/share/pyshared` and symlink to | |
| them from `/usr/lib/pythonX.Y/dist-packages`. The symlinks exist | |
| because in a pre-PEP 3147 world (i.e < Python 3.2), the `.pyc` files | |
| resulting from byte compilation by the various installed Pythons will | |
| name collide with each other. For Python versions >= 3.2, all | |
| pure-Python packages can be shared, because the `.pyc` files will no | |
| longer cause file system naming conflicts. Eliminating these symlinks | |
| makes for a simpler, more robust Python distribution. | |
| A similar situation arises with shared library extensions. Because | |
| extension modules are typically named `foo.so` for a `foo` extension | |
| module, these would also name collide if `foo` was provided for more | |
| than one Python version. | |
| In addition, because different configuration/compilation options for | |
| the same Python version can cause different ABIs to be presented to | |
| extension modules. On POSIX systems for example, the configure | |
| options ``--with-pydebug``, ``--with-pymalloc``, and | |
| ``--with-wide-unicode`` all change the ABI. This PEP proposes to | |
| encode build-time options in the file name of the ``.so`` extension | |
| module files. | |
| PyPy [5]_ can also benefit from this PEP, allowing it to avoid name | |
| collisions in extension modules built for its API, but with a | |
| different `.so` tag. | |
| Proposal | |
| ======== | |
| The configure/compilation options chosen at Python interpreter | |
| build-time will be encoded in the shared library file name for | |
| extension modules. This "tag" will appear between the module base | |
| name and the operation file system extension for shared libraries. | |
| The following information *MUST* be included in the shared library | |
| file name: | |
| * The Python implementation (e.g. cpython, pypy, jython, etc.) | |
| * The interpreter's major and minor version numbers | |
| These two fields are separated by a hyphen and no dots are to appear | |
| between the major and minor version numbers. E.g. ``cpython-32``. | |
| Python implementations *MAY* include additional flags in the file name | |
| tag as appropriate. For example, on POSIX systems these flags will | |
| also contribute to the file name: | |
| * ``--with-pydebug`` (flag: ``d``) | |
| * ``--with-pymalloc`` (flag: ``m``) | |
| * ``--with-wide-unicode`` (flag: ``u``) | |
| By default in Python 3.2, ``configure`` enables ``--with-pymalloc`` so | |
| shared library file names would appear as ``foo.cpython-32m.so``. | |
| When the other two flags are also enabled, the file names would be | |
| ``foo.cpython-32dmu.so``. | |
| The shared library file name tag is used unconditionally; it cannot be | |
| changed. The tag and extension module suffix are available through | |
| the ``sysconfig`` modules via the following variables:: | |
| >>> sysconfig.get_config_var('EXT_SUFFIX') | |
| '.cpython-32mu.so' | |
| >>> sysconfig.get_config_var('SOABI') | |
| 'cpython-32mu' | |
| Note that ``$SOABI`` contains just the tag, while ``$EXT_SUFFIX`` includes the | |
| platform extension for shared library files, and is the exact suffix | |
| added to the extension module name. | |
| For an arbitrary package `foo`, you might see these files when the | |
| distribution package was installed:: | |
| /usr/lib/python/foo.cpython-32m.so | |
| /usr/lib/python/foo.cpython-33m.so | |
| (These paths are for example purposes only. Distributions are free to | |
| use whatever filesystem layout they choose, and nothing in this PEP | |
| changes the locations where from-source builds of Python are | |
| installed.) | |
| Python's dynamic module loader will recognize and import shared | |
| library extension modules with a tag that matches its build-time | |
| options. For backward compatibility, Python will also continue to | |
| import untagged extension modules, e.g. ``foo.so``. | |
| This shared library tag would be used globally for all distutils-based | |
| extension modules, regardless of where on the file system they are | |
| built. Extension modules built by means other than distutils would | |
| either have to calculate the tag manually, or fallback to the | |
| non-tagged `.so` file name. | |
| Proven approach | |
| =============== | |
| The approach described here is already proven, in a sense, on Debian | |
| and Ubuntu system where different extensions are used for debug builds | |
| of Python and extension modules. Debug builds on Windows also already | |
| use a different file extension for dynamic libraries, and in fact | |
| encoded (in a different way than proposed in this PEP) the Python | |
| major and minor version in the `.dll` file name. | |
| Windows | |
| ======= | |
| This PEP only addresses build issues on POSIX systems that use the | |
| ``configure`` script. While Windows or other platform support is not | |
| explicitly disallowed under this PEP, platform expertise is needed in | |
| order to evaluate, describe, and implement support on such platforms. | |
| It is not currently clear that the facilities in this PEP are even | |
| useful for Windows. | |
| PEP 384 | |
| ======= | |
| PEP 384 defines a stable ABI for extension modules. In theory, | |
| universal adoption of PEP 384 would eliminate the need for this PEP | |
| because all extension modules could be compatible with any Python | |
| version. In practice of course, it will be impossible to achieve | |
| universal adoption, and as described above, different built-time flags | |
| still affect the ABI. Thus even with a stable ABI, this PEP may still | |
| be necessary. While a complete specification is reserved for PEP 384, | |
| here is a discussion of the relevant issues. | |
| PEP 384 describes a change to ``PyModule_Create()`` where ``3`` is | |
| passed as the API version if the extension was complied with | |
| ``Py_LIMITED_API``. This should be formalized into an official macro | |
| called ``PYTHON_ABI_VERSION`` to mirror ``PYTHON_API_VERSION``. If | |
| and when the ABI changes in an incompatible way, this version number | |
| would be bumped. To facilitate sharing, Python would be extended to | |
| search for extension modules with the ``PYTHON_ABI_VERSION`` number in | |
| its name. The prefix ``abi`` is reserved for Python's use. | |
| Thus, an initial implementation of PEP 384, when Python is configured | |
| with the default set of flags, would search for the following file | |
| names when extension module `foo` is imported (in this order):: | |
| foo.cpython-XYm.so | |
| foo.abi3.so | |
| foo.so | |
| The distutils [6]_ ``build_ext`` command would also have to be | |
| extended to compile to shared library files with the ``abi3`` tag, | |
| when the module author indicates that their extension supports that | |
| version of the ABI. This could be done in a backward compatible way | |
| by adding a keyword argument to the ``Extension`` class, such as:: | |
| Extension('foo', ['foo.c'], abi=3) | |
| Martin v. Löwis describes his thoughts [7]_ about the applicability of this | |
| PEP to PEP 384. In summary: | |
| * ``--with-pydebug`` would not be supported by the stable ABI because | |
| this changes the layout of ``PyObject``, which is an exposed | |
| structure. | |
| * ``--with-pymalloc`` has no bearing on the issue. | |
| * ``--with-wide-unicode`` is trickier, though Martin's inclination is | |
| to force the stable ABI to use a ``Py_UNICODE`` that matches the | |
| platform's ``wchar_t``. | |
| Alternatives | |
| ============ | |
| In the initial python-dev thread [8]_ where this idea was first | |
| introduced, several alternatives were suggested. For completeness | |
| they are listed here, along with the reasons for not adopting them. | |
| Independent directories or symlinks | |
| ----------------------------------- | |
| Debian and Ubuntu could simply add a version-specific directory to | |
| ``sys.path`` that would contain just the extension modules for that | |
| version of Python. Or the symlink trick eliminated in PEP 3147 could | |
| be retained for just shared libraries. This approach is rejected | |
| because it propagates the essential complexity that PEP 3147 tries to | |
| avoid, and adds potentially several additional directories to search | |
| for all modules, even when the number of extension modules is much | |
| fewer than the total number of Python packages. For example, builds | |
| were made available both with and without wide unicode, with and | |
| without pydebug, and with and without pymalloc, the total number of | |
| directories search increases substantially. | |
| Don't share packages with extension modules | |
| ------------------------------------------- | |
| It has been suggested that Python packages with extension modules not | |
| be shared among all supported Python versions on a distribution. Even | |
| with adoption of PEP 3149, extension modules will have to be compiled | |
| for every supported Python version, so perhaps sharing of such | |
| packages isn't useful anyway. Not sharing packages with extensions | |
| though is infeasible for several reasons. | |
| If a pure-Python package is shared in one version, should it suddenly | |
| be not-shared if the next release adds an extension module for speed? | |
| Also, even though all extension shared libraries will be compiled and | |
| distributed once for every supported Python, there's a big difference | |
| between duplicating the `.so` files and duplicating all `.py` files. | |
| The extra size increases the download time for such packages, and more | |
| immediately, increases the space pressures on already constrained | |
| distribution CD-ROMs. | |
| Reference implementation | |
| ======================== | |
| Work on this code is tracked in a Bazaar branch on Launchpad [9]_ | |
| until it's ready for merge into Python 3.2. The work-in-progress diff | |
| can also be viewed [10]_ and is updated automatically as new changes | |
| are uploaded. | |
| References | |
| ========== | |
| .. [1] PEP 3147 | |
| .. [2] PEP 384 | |
| .. [3] Ubuntu: <http://www.ubuntu.com> | |
| .. [4] Debian: <http://www.debian.org> | |
| .. [5] http://codespeak.net/pypy/dist/pypy/doc/ | |
| .. [6] http://docs.python.org/py3k/distutils/index.html | |
| .. [7] https://mail.python.org/pipermail/python-dev/2010-August/103330.html | |
| .. [8] https://mail.python.org/pipermail/python-dev/2010-June/100998.html | |
| .. [9] https://code.edge.launchpad.net/~barry/python/sovers | |
| .. [10] https://code.edge.launchpad.net/~barry/python/sovers/+merge/29411 | |
| Copyright | |
| ========= | |
| This document has been placed in the public domain. | |
| .. | |
| Local Variables: | |
| mode: indented-text | |
| indent-tabs-mode: nil | |
| sentence-end-double-space: t | |
| fill-column: 70 | |
| coding: utf-8 | |
| End: |