Classes and types in the Python typing module

This article brought to you by LWN subscribers

Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

Mark Shannon is concerned that the Python core developers may be replaying a mistake: treating two distinct things as being the same. Treating byte strings and Unicode text-strings interchangeably is part of what led to Python 3, so he would rather not see that happen again with types and classes. The Python typing module, which is meant to support type hints, currently implements types as classes. That leads to several kinds of problems, as Shannon described in his session at the 2017 Python Language Summit.

He wanted to convince people that the typing module is "heading in the wrong direction". He is not opposed to type hints or variable annotations, but is concerned that the typing module is conflating types and classes in a way that is detrimental. Classes are for object-oriented programming, while types declare what something is. A class can be a subclass of another without being a subtype of it. List[int] and List[float] (lists of integers and floating point numbers, respectively) are distinct types, he said, but are both implemented by the list class. In the current typing module, types are implemented as classes.

This has happened before, with bytes and Unicode in the Python 2 days, Shannon said. He would rather see this get addressed now, before the core developers (and the language) get to that point again.

Practical problems

Using classes for types has some concrete negative effects. Classes are "large and bad" in CPython, but are much worse for MicroPython. A namedtuple-based implementation of List[int] is around 1/60 the size of the class-based one.

There are also some oddities. He showed two class definitions:

    class MyList(Sequence[int], list): pass

    class MyList(list, Sequence[int]): pass

It turns out that the method resolution order (MRO) comes into play. MRO determines which method actually gets called when multiple inheritance is used; Python tracks that on the __mro__ attribute. For a class that inherits builtins.list, the MRO has three items, but for List[int] it has 17.

Types and type constructors are already hard enough to understand, he said. Turning them into classes and metaclasses just makes that worse. In addition, since types in typing already have a custom metaclass, it makes it difficult to define a type for a class that has its own custom metaclass.

When adopting type hints, the core developers made a few promises, Shannon said. Type hints would allow programs to be checked for type errors, they would always be optional, and using them should not slow your program down. The first two of those have been kept, but the last has not. Every time you run a program with type hints, it pulls in large chunk of code that slows things down.

Options

He presented three options. The first was to continue using types as classes, but to painfully check that an instance of Iterable(int) actually produces integers for each entry. Then hope that things don't get as bad as they did for bytes and Unicode. Another was "the status quo"; much the same as the first, but to ignore the checks that seem expensive. The option that he prefers is to keep types and classes distinct, which will remove the "conceptual muddle" and reduce the run-time overhead of using types. He has a minimal prototype implementation on GitHub to demonstrate what he means.

Attendees were generally supportive of his ideas; Guido van Rossum filed a bug for typing on some of the issues he raised. There were also suggestions on ways to reduce the overhead for code that uses type hints. Łukasz Langa noted that Instagram had reduced the size of compiled Python (i.e. bytecode) by 1.5% just by removing the docstrings; perhaps something similar could be done to remove the type annotations to reduce the size of the code.

[I would like to thank the Linux Foundation for travel assistance to Portland for the summit.]