Permalink
Cannot retrieve contributors at this time
| PEP: 460 | |
| Title: Add binary interpolation and formatting | |
| Version: $Revision$ | |
| Last-Modified: $Date$ | |
| Author: Antoine Pitrou <solipsis@pitrou.net> | |
| Status: Withdrawn | |
| Type: Standards Track | |
| Content-Type: text/x-rst | |
| Created: 06-Jan-2014 | |
| Python-Version: 3.5 | |
| Abstract | |
| ======== | |
| This PEP proposes to add minimal formatting operations to bytes and | |
| bytearray objects. The proposed additions are: | |
| * ``bytes % ...`` and ``bytearray % ...`` for percent-formatting, | |
| similar in syntax to percent-formatting on ``str`` objects | |
| (accepting a single object, a tuple or a dict). | |
| * ``bytes.format(...)`` and ``bytearray.format(...)`` for a formatting | |
| similar in syntax to ``str.format()`` (accepting positional as well as | |
| keyword arguments). | |
| * ``bytes.format_map(...)`` and ``bytearray.format_map(...)`` for an | |
| API similar to ``str.format_map(...)``, with the same formatting | |
| syntax and semantics as ``bytes.format()`` and ``bytearray.format()``. | |
| Rationale | |
| ========= | |
| In Python 2, ``str % args`` and ``str.format(args)`` allow the formatting | |
| and interpolation of bytestrings. This feature has commonly been used | |
| for the assembling of protocol messages when protocols are known to use | |
| a fixed encoding. | |
| Python 3 generally mandates that text be stored and manipulated as unicode | |
| (i.e. ``str`` objects, not ``bytes``). In some cases, though, it makes | |
| sense to manipulate ``bytes`` objects directly. Typical usage is binary | |
| network protocols, where you can want to interpolate and assemble several | |
| bytes object (some of them literals, some of them compute) to produce | |
| complete protocol messages. For example, protocols such as HTTP or SIP | |
| have headers with ASCII names and opaque "textual" values using a varying | |
| and/or sometimes ill-defined encoding. Moreover, those headers can be | |
| followed by a binary body... which can be chunked and decorated with ASCII | |
| headers and trailers! | |
| While there are reasonably efficient ways to accumulate binary data | |
| (such as using a ``bytearray`` object, the ``bytes.join`` method or | |
| even ``io.BytesIO``), none of them leads to the kind of readable and | |
| intuitive code that is produced by a %-formatted or {}-formatted template | |
| and a formatting operation. | |
| Binary formatting features | |
| ========================== | |
| Supported features | |
| ------------------ | |
| In this proposal, percent-formatting for ``bytes`` and ``bytearray`` | |
| supports the following features: | |
| * Looking up formatting arguments by position as well as by name (i.e., | |
| ``%s`` as well as ``%(name)s``). | |
| * ``%s`` will try to get a ``Py_buffer`` on the given value, and fallback | |
| on calling ``__bytes__``. The resulting binary data is inserted at | |
| the given point in the string. This is expected to work with bytes, | |
| bytearray and memoryview objects (as well as a couple others such | |
| as pathlib's path objects). | |
| * ``%c`` will accept an integer between 0 and 255, and insert a byte of the | |
| given value. | |
| Braces-formatting for ``bytes`` and ``bytearray`` supports the following | |
| features: | |
| * All the kinds of argument lookup supported by ``str.format()`` (explicit | |
| positional lookup, auto-incremented positional lookup, keyword lookup, | |
| attribute lookup, etc.) | |
| * Insertion of binary data when no modifier or layout is specified | |
| (e.g. ``{}``, ``{0}``, ``{name}``). This has the same semantics as | |
| ``%s`` for percent-formatting (see above). | |
| * The ``c`` modifier will accept an integer between 0 and 255, and insert a | |
| byte of the given value (same as ``%c`` above). | |
| Unsupported features | |
| -------------------- | |
| All other features present in formatting of ``str`` objects (either | |
| through the percent operator or the ``str.format()`` method) are | |
| unsupported. Those features imply treating the recipient of the | |
| operator or method as text, which goes counter to the text / bytes | |
| separation (for example, accepting ``%d`` as a format code would imply | |
| that the bytes object really is an ASCII-compatible text string). | |
| Amongst those unsupported features are not only most type-specific | |
| format codes, but also the various layout specifiers such as padding | |
| or alignment. Besides, ``str`` objects are not acceptable as arguments | |
| to the formatting operations, even when using e.g. the ``%s`` format code. | |
| ``__format__`` isn't called. | |
| Criticisms | |
| ========== | |
| * The development cost and maintenance cost. | |
| * In 3.3 encoding to ASCII or latin-1 is as fast as memcpy (but it still | |
| creates a separate object). | |
| * Developers will have to work around the lack of binary formatting anyway, | |
| if they want to support Python 3.4 and earlier. | |
| * bytes.join() is consistently faster than format to join bytes strings | |
| (XXX *is it?*). | |
| * Formatting functions could be implemented in a third party module, | |
| rather than added to builtin types. | |
| Other proposals | |
| =============== | |
| A new type datatype | |
| ------------------- | |
| It was proposed to create a new datatype specialized for "network | |
| programming". The authors of this PEP believe this is counter-productive. | |
| Python 3 already has several major types dedicated to manipulation of | |
| binary data: ``bytes``, ``bytearray``, ``memoryview``, ``io.BytesIO``. | |
| Adding yet another type would make things more confusing for users, and | |
| interoperability between libraries more painful (also potentially | |
| sub-optimal, due to the necessary conversions). | |
| Moreover, not one type would be needed, but two: one immutable type (to | |
| allow for hashing), and one mutable type (as efficient accumulation is | |
| often necessary when working with network messages). | |
| Resolution | |
| ========== | |
| This PEP is made obsolete by the `acceptance | |
| <https://mail.python.org/pipermail/python-dev/2014-March/133621.html>`_ | |
| of :pep:`461`, which introduces a more extended formatting language for | |
| bytes objects in conjunction with the modulo operator. | |
| References | |
| ========== | |
| * `Issue #3982: support .format for bytes | |
| <http://bugs.python.org/issue3982>`_ | |
| * `Mercurial project | |
| <http://mercurial.selenic.com/>`_ | |
| * `Twisted project | |
| <http://twistedmatrix.com/trac/>`_ | |
| * `Documentation of Python 2 formatting (str % args) | |
| <http://docs.python.org/2/library/stdtypes.html#string-formatting>`_ | |
| * `Documentation of Python 2 formatting (str.format) | |
| <http://docs.python.org/2/library/string.html#formatstrings>`_ | |
| Copyright | |
| ========= | |
| This document has been placed in the public domain. | |
| .. | |
| Local Variables: | |
| mode: indented-text | |
| indent-tabs-mode: nil | |
| sentence-end-double-space: t | |
| fill-column: 70 | |
| coding: utf-8 | |
| End: |