Real Python

Async IO in Python: A Complete Walkthrough

2019-01-16T14:00:00+00:00

Async IO is a concurrent programming design that has received dedicated support in Python, evolving rapidly from Python 3.4 through 3.7, and probably beyond.

You may be thinking with dread, “Concurrency, parallelism, threading, multiprocessing. That’s a lot to grasp already. Where does async IO fit in?”

This tutorial is built to help you answer that question, giving you a firmer grasp of Python’s approach to async IO.

Here’s what you’ll cover:

Asynchronous IO (async IO): a language-agnostic paradigm (model) that has implementations across a host of programming languages
async/await: two new Python keywords that are used to define coroutines
asyncio: the Python package that provides a foundation and API for running and managing coroutines

Coroutines (specialized generator functions) are the heart of async IO in Python, and we’ll dive into them later on.

Note: In this article, I use the term async IO to denote the language-agnostic design of asynchronous IO, while asyncio refers to the Python package.

Before you get started, you’ll need to make sure you’re set up to use asyncio and other libraries found in this tutorial.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Setting Up Your Environment

You’ll need Python 3.7 or above to follow this article in its entirety, as well as the aiohttp and aiofiles packages:

$ python3.7 -m venv ./py37async
$ source ./py37async/bin/activate  # Windows: .\py37async\Scripts\activate.bat
$ pip install --upgrade pip aiohttp aiofiles  # Optional: aiodns

For help with installing Python 3.7 and setting up a virtual environment, check out Python 3 Installation & Setup Guide or Virtual Environments Primer.

With that, let’s jump in.

The 10,000-Foot View of Async IO

Async IO is a bit lesser known than its tried-and-true cousins, multiprocessing and threading. This section will give you a fuller picture of what async IO is and how it fits into its surrounding landscape.

Where Does Async IO Fit In?

Concurrency and parallelism are expansive subjects that are not easy to wade into. While this article focuses on async IO and its implementation in Python, it’s worth taking a minute to compare async IO to its counterparts in order to have context about how async IO fits into the larger, sometimes dizzying puzzle.

Parallelism consists of performing multiple operations at the same time. Multiprocessing is a means to effect parallelism, and it entails spreading tasks over a computer’s central processing units (CPUs, or cores). Multiprocessing is well-suited for CPU-bound tasks: tightly bound for loops and mathematical computations usually fall into this category.

Concurrency is a slightly broader term than parallelism. It suggests that multiple tasks have the ability to run in an overlapping manner. (There’s a saying that concurrency does not imply parallelism.)

Threading is a concurrent execution model whereby multiple threads take turns executing tasks. One process can contain multiple threads. Python has a complicated relationship with threading thanks to its GIL, but that’s beyond the scope of this article.

What’s important to know about threading is that it’s better for IO-bound tasks. While a CPU-bound task is characterized by the computer’s cores continually working hard from start to finish, an IO-bound job is dominated by a lot of waiting on input/output to complete.

To recap the above, concurrency encompasses both multiprocessing (ideal for CPU-bound tasks) and threading (suited for IO-bound tasks). Multiprocessing is a form of parallelism, with parallelism being a specific type (subset) of concurrency. The Python standard library has offered longstanding support for both of these through its multiprocessing, threading, and concurrent.futures packages.

Now it’s time to bring a new member to the mix. Over the last few years, a separate design has been more comprehensively built into CPython: asynchronous IO, enabled through the standard library’s asyncio package and the new async and await language keywords. To be clear, async IO is not a newly invented concept, and it has existed or is being built into other languages and runtime environments, such as Go, C#, or Scala.

The asyncio package is billed by the Python documentation as a library to write concurrent code. However, async IO is not threading, nor is it multiprocessing. It is not built on top of either of these.

In fact, async IO is a single-threaded, single-process design: it uses cooperative multitasking, a term that you’ll flesh out by the end of this tutorial. It has been said in other words that async IO gives a feeling of concurrency despite using a single thread in a single process. Coroutines (a central feature of async IO) can be scheduled concurrently, but they are not inherently concurrent.

To reiterate, async IO is a style of concurrent programming, but it is not parallelism. It’s more closely aligned with threading than with multiprocessing but is very much distinct from both of these and is a standalone member in concurrency’s bag of tricks.

That leaves one more term. What does it mean for something to be asynchronous? This isn’t a rigorous definition, but for our purposes here, I can think of two properties:

Asynchronous routines are able to “pause” while waiting on their ultimate result and let other routines run in the meantime.
Asynchronous code, through the mechanism above, facilitates concurrent execution. To put it differently, asynchronous code gives the look and feel of concurrency.

Here’s a diagram to put it all together. The white terms represent concepts, and the green terms represent ways in which they are implemented or effected:

I’ll stop there on the comparisons between concurrent programming models. This tutorial is focused on the subcomponent that is async IO, how to use it, and the APIs that have sprung up around it. For a thorough exploration of threading versus multiprocessing versus async IO, pause here and check out Jim Anderson’s overview of concurrency in Python. Jim is way funnier than me and has sat in more meetings than me, to boot.

Async IO Explained

Async IO may at first seem counterintuitive and paradoxical. How does something that facilitates concurrent code use a single thread and a single CPU core? I’ve never been very good at conjuring up examples, so I’d like to paraphrase one from Miguel Grinberg’s 2017 PyCon talk, which explains everything quite beautifully:

Chess master Judit Polgár hosts a chess exhibition in which she plays multiple amateur players. She has two ways of conducting the exhibition: synchronously and asynchronously.

Assumptions:

24 opponents

Judit makes each chess move in 5 seconds

Opponents each take 55 seconds to make a move

Games average 30 pair-moves (60 moves total)

Synchronous version: Judit plays one game at a time, never two at the same time, until the game is complete. Each game takes (55 + 5) * 30 == 1800 seconds, or 30 minutes. The entire exhibition takes 24 * 30 == 720 minutes, or 12 hours.

Asynchronous version: Judit moves from table to table, making one move at each table. She leaves the table and lets the opponent make their next move during the wait time. One move on all 24 games takes Judit 24 * 5 == 120 seconds, or 2 minutes. The entire exhibition is now cut down to 120 * 30 == 3600 seconds, or just 1 hour. (Source)

There is only one Judit Polgár, who has only two hands and makes only one move at a time by herself. But playing asynchronously cuts the exhibition time down from 12 hours to one. So, cooperative multitasking is a fancy way of saying that a program’s event loop (more on that later) communicates with multiple tasks to let each take turns running at the optimal time.

Async IO takes long waiting periods in which functions would otherwise be blocking and allows other functions to run during that downtime. (A function that blocks effectively forbids others from running from the time that it starts until the time that it returns.)

Async IO Is Not Easy

I’ve heard it said, “Use async IO when you can; use threading when you must.” The truth is that building durable multithreaded code can be hard and error-prone. Async IO avoids some of the potential speedbumps that you might otherwise encounter with a threaded design.

But that’s not to say that async IO in Python is easy. Be warned: when you venture a bit below the surface level, async programming can be difficult too! Python’s async model is built around concepts such as callbacks, events, transports, protocols, and futures—just the terminology can be intimidating. The fact that its API has been changing continually makes it no easier.

Luckily, asyncio has matured to a point where most of its features are no longer provisional, while its documentation has received a huge overhaul and some quality resources on the subject are starting to emerge as well.

The `asyncio` Package and `async`/`await`

Now that you have some background on async IO as a design, let’s explore Python’s implementation. Python’s asyncio package (introduced in Python 3.4) and its two keywords, async and await, serve different purposes but come together to help you declare, build, execute, and manage asynchronous code.

The `async`/`await` Syntax and Native Coroutines

A Word of Caution: Be careful what you read out there on the Internet. Python’s async IO API has evolved rapidly from Python 3.4 to Python 3.7. Some old patterns are no longer used, and some things that were at first disallowed are now allowed through new introductions. For all I know, this tutorial will join the club of the outdated soon too.

At the heart of async IO are coroutines. A coroutine is a specialized version of a Python generator function. Let’s start with a baseline definition and then build off of it as you progress here: a coroutine is a function that can suspend its execution before reaching return, and it can indirectly pass control to another coroutine for some time.

Later, you’ll dive a lot deeper into how exactly the traditional generator is repurposed into a coroutine. For now, the easiest way to pick up how coroutines work is to start making some.

Let’s take the immersive approach and write some async IO code. This short program is the Hello World of async IO but goes a long way towards illustrating its core functionality:

#!/usr/bin/env python3
# countasync.py

import asyncio

async def count():
    print("One")
    await asyncio.sleep(1)
    print("Two")

async def main():
    await asyncio.gather(count(), count(), count())

if __name__ == "__main__":
    import time
    s = time.perf_counter()
    asyncio.run(main())
    elapsed = time.perf_counter() - s
    print(f"{__file__} executed in {elapsed:0.2f} seconds.")

When you execute this file, take note of what looks different than if you were to define the functions with just def and time.sleep():

$ python3 countasync.py
One
One
One
Two
Two
Two
countasync.py executed in 1.01 seconds.

The order of this output is the heart of async IO. Talking to each of the calls to count() is a single event loop, or coordinator. When each task reaches await asyncio.sleep(1), the function yells up to the event loop and gives control back to it, saying, “I’m going to be sleeping for 1 second. Go ahead and let something else meaningful be done in the meantime.”

Contrast this to the synchronous version:

#!/usr/bin/env python3
# countsync.py

import time

def count():
    print("One")
    time.sleep(1)
    print("Two")

def main():
    for _ in range(3):
        count()

if __name__ == "__main__":
    s = time.perf_counter()
    main()
    elapsed = time.perf_counter() - s
    print(f"{__file__} executed in {elapsed:0.2f} seconds.")

When executed, there is a slight but critical change in order and execution time:

$ python3 countsync.py
One
Two
One
Two
One
Two
countsync.py executed in 3.01 seconds.

While using time.sleep() and asyncio.sleep() may seem banal, they are used as stand-ins for any time-intensive processes that involve wait time. (The most mundane thing you can wait on is a sleep() call that does basically nothing.) That is, time.sleep() can represent any time-consuming blocking function call, while asyncio.sleep() is used to stand in for a non-blocking call (but one that also takes some time to complete).

As you’ll see in the next section, the benefit of awaiting something, including asyncio.sleep(), is that the surrounding function can temporarily cede control to another function that’s more readily able to do something immediately. In contrast, time.sleep() or any other blocking call is incompatible with asynchronous Python code, because it will stop everything in its tracks for the duration of the sleep time.

The Rules of Async IO

At this point, a more formal definition of async, await, and the coroutine functions that they create are in order. This section is a little dense, but getting a hold of async/await is instrumental, so come back to this if you need to:

The syntax async def introduces either a native coroutine or an asynchronous generator. The expressions async with and async for are also valid, and you’ll see them later on.
The keyword await passes function control back to the event loop. (It suspends the execution of the surrounding coroutine.) If Python encounters an await f() expression in the scope of g(), this is how await tells the event loop, “Suspend execution of g() until whatever I’m waiting on—the result of f()—is returned. In the meantime, go let something else run.”

In code, that second bullet point looks roughly like this:

async def g():
    # Pause here and come back to g() when f() is ready
    r = await f()
    return r

There’s also a strict set of rules around when and how you can and cannot use async/await. These can be handy whether you are still picking up the syntax or already have exposure to using async/await:

A function that you introduce with async def is a coroutine. It may use await, return, or yield, but all of these are optional. Declaring async def noop(): pass is valid:
- Using await and/or return creates a coroutine function. To call a coroutine function, you must await it to get its results.
- It is less common (and only recently legal in Python) to use yield in an async def block. This creates an asynchronous generator, which you iterate over with async for. Forget about async generators for the time being and focus on getting down the syntax for coroutine functions, which use await and/or return.
- Anything defined with async def may not use yield from, which will raise a SyntaxError.
Just like it’s a SyntaxError to use yield outside of a def function, it is a SyntaxError to use await outside of an async def coroutine. You can only use await in the body of coroutines.

Here are some terse examples meant to summarize the above few rules:

async def f(x):
    y = await z(x)  # OK - `await` and `return` allowed in coroutines
    return y

async def g(x):
    yield x  # OK - this is an async generator

async def m(x):
    yield from gen(x)  # No - SyntaxError

def m(x):
    y = await z(x)  # Still no - SyntaxError (no `async def` here)
    return y

Finally, when you use await f(), it’s required that f() be an object that is awaitable. Well, that’s not very helpful, is it? For now, just know that an awaitable object is either (1) another coroutine or (2) an object defining an .__await__() dunder method that returns an iterator. If you’re writing a program, for the large majority of purposes, you should only need to worry about case #1.

That brings us to one more technical distinction that you may see pop up: an older way of marking a function as a coroutine is to decorate a normal def function with @asyncio.coroutine. The result is a generator-based coroutine. This construction has been outdated since the async/await syntax was put in place in Python 3.5.

These two coroutines are essentially equivalent (both are awaitable), but the first is generator-based, while the second is a native coroutine:

import asyncio

@asyncio.coroutine
def py34_coro():
    """Generator-based coroutine, older syntax"""
    yield from stuff()

async def py35_coro():
    """Native coroutine, modern syntax"""
    await stuff()

If you’re writing any code yourself, prefer native coroutines for the sake of being explicit rather than implicit. Generator-based coroutines will be removed in Python 3.10.

Towards the latter half of this tutorial, we’ll touch on generator-based coroutines for explanation’s sake only. The reason that async/await were introduced is to make coroutines a standalone feature of Python that can be easily differentiated from a normal generator function, thus reducing ambiguity.

Don’t get bogged down in generator-based coroutines, which have been deliberately outdated by async/await. They have their own small set of rules (for instance, await cannot be used in a generator-based coroutine) that are largely irrelevant if you stick to the async/await syntax.

Without further ado, let’s take on a few more involved examples.

Here’s one example of how async IO cuts down on wait time: given a coroutine makerandom() that keeps producing random integers in the range [0, 10], until one of them exceeds a threshold, you want to let multiple calls of this coroutine not need to wait for each other to complete in succession. You can largely follow the patterns from the two scripts above, with slight changes:

#!/usr/bin/env python3
# rand.py

import asyncio
import random

# ANSI colors
c = (
    "\033[0m",   # End of color
    "\033[36m",  # Cyan
    "\033[91m",  # Red
    "\033[35m",  # Magenta
)

async def randint(a: int, b: int) -> int:
    return random.randint(a, b)

async def makerandom(idx: int, threshold: int = 6) -> int:
    print(c[idx + 1] + f"Initiated makerandom({idx}).")
    i = await randint(0, 10)
    while i <= threshold:
        print(c[idx + 1] + f"makerandom({idx}) == {i} too low; retrying.")
        await asyncio.sleep(idx + 1)
        i = await randint(0, 10)
    print(c[idx + 1] + f"---> Finished: makerandom({idx}) == {i}" + c[0])
    return i

async def main():
    res = await asyncio.gather(*(makerandom(i, 10 - i - 1) for i in range(3)))
    return res

if __name__ == "__main__":
    random.seed(444)
    r1, r2, r3 = asyncio.run(main())
    print()
    print(f"r1: {r1}, r2: {r2}, r3: {r3}")

The colorized output says a lot more than I can and gives you a sense for how this script is carried out:

rand.py execution

This program uses one main coroutine, makerandom(), and runs it concurrently across 3 different inputs. Most programs will contain small, modular coroutines and one wrapper function that serves to chain each of the smaller coroutines together. main() is then used to gather tasks (futures) by mapping the central coroutine across some iterable or pool.

In this miniature example, the pool is range(3). In a fuller example presented later, it is a set of URLs that need to be requested, parsed, and processed concurrently, and main() encapsulates that entire routine for each URL.

While “making random integers” (which is CPU-bound more than anything) is maybe not the greatest choice as a candidate for asyncio, it’s the presence of asyncio.sleep() in the example that is designed to mimic an IO-bound process where there is uncertain wait time involved. For example, the asyncio.sleep() call might represent sending and receiving not-so-random integers between two clients in a message application.

Async IO Design Patterns

Async IO comes with its own set of possible script designs, which you’ll get introduced to in this section.

Chaining Coroutines

A key feature of coroutines is that they can be chained together. (Remember, a coroutine object is awaitable, so another coroutine can await it.) This allows you to break programs into smaller, manageable, recyclable coroutine:

#!/usr/bin/env python3
# chained.py

import asyncio
import random
import time

async def randint(a: int, b: int) -> int:
    return random.randint(a, b)

async def part1(n: int) -> str:
    i = await randint(0, 10)
    print(f"part1({n}) sleeping for {i} seconds.")
    await asyncio.sleep(i)
    result = f"result{n}-1"
    print(f"Returning part1({n}) == {result}.")
    return result

async def part2(n: int, arg: str) -> str:
    i = await randint(0, 10)
    print(f"part2{n, arg} sleeping for {i} seconds.")
    await asyncio.sleep(i)
    result = f"result{n}-2 derived from {arg}"
    print(f"Returning part2{n, arg} == {result}.")
    return result

async def chain(n: int) -> None:
    start = time.perf_counter()
    p1 = await part1(n)
    p2 = await part2(n, p1)
    end = time.perf_counter() - start
    print(f"-->Chained result{n} => {p2} (took {end:0.2f} seconds).")

async def main(*args):
    await asyncio.gather(*(chain(n) for n in args))

if __name__ == "__main__":
    import sys
    random.seed(444)
    args = [1, 2, 3] if len(sys.argv) == 1 else map(int, sys.argv[1:])
    start = time.perf_counter()
    asyncio.run(main(*args))
    end = time.perf_counter() - start
    print(f"Program finished in {end:0.2f} seconds.")

Pay careful attention to the output, where part1() sleeps for a variable amount of time, and part2() begins working with the results as they become available:

$ python3 chained.py 9 6 3
part1(9) sleeping for 4 seconds.
part1(6) sleeping for 4 seconds.
part1(3) sleeping for 0 seconds.
Returning part1(3) == result3-1.
part2(3, 'result3-1') sleeping for 4 seconds.
Returning part1(9) == result9-1.
part2(9, 'result9-1') sleeping for 7 seconds.
Returning part1(6) == result6-1.
part2(6, 'result6-1') sleeping for 4 seconds.
Returning part2(3, 'result3-1') == result3-2 derived from result3-1.
-->Chained result3 => result3-2 derived from result3-1 (took 4.00 seconds).
Returning part2(6, 'result6-1') == result6-2 derived from result6-1.
-->Chained result6 => result6-2 derived from result6-1 (took 8.01 seconds).
Returning part2(9, 'result9-1') == result9-2 derived from result9-1.
-->Chained result9 => result9-2 derived from result9-1 (took 11.01 seconds).
Program finished in 11.01 seconds.

In this setup, the runtime of main() will be equal to the maximum runtime of the tasks that it gathers together and schedules.

Using a Queue

The asyncio package provides queue classes that are designed to be similar to classes of the queue module. In our examples so far, we haven’t really had a need for a queue structure. In chained.py, each task (future) is composed of a set of coroutines that explicitly await each other and pass through a single input per chain.

There is an alternative structure that can also work with async IO: a number of producers, which are not associated with each other, add items to a queue. Each producer may add multiple items to the queue at staggered, random, unannounced times. A group of consumers pull items from the queue as they show up, greedily and without waiting for any other signal.

In this design, there is no chaining of any individual consumer to a producer. The consumers don’t know the number of producers, or even the cumulative number of items that will be added to the queue, in advance.

It takes an individual producer or consumer a variable amount of time to put and extract items from the queue, respectively. The queue serves as a throughput that can communicate with the producers and consumers without them talking to each other directly.

Note: While queues are often used in threaded programs because of the thread-safety of queue.Queue(), you shouldn’t need to concern yourself with thread safety when it comes to async IO. (The exception is when you’re combining the two, but that isn’t done in this tutorial.)

One use-case for queues (as is the case here) is for the queue to act as a transmitter for producers and consumers that aren’t otherwise directly chained or associated with each other.

The synchronous version of this program would look pretty dismal: a group of blocking producers serially add items to the queue, one producer at a time. Only after all producers are done can the queue be processed, by one consumer at a time processing item-by-item. There is a ton of latency in this design. Items may sit idly in the queue rather than be picked up and processed immediately.

An asynchronous version, asyncq.py, is below. The challenging part of this workflow is that there needs to be a signal to the consumers that production is done. Otherwise, await q.get() will hang indefinitely, because the queue will have been fully processed, but consumers won’t have any idea that production is complete.

(Big thanks for some help from a StackOverflow user for helping to straighten out main(): the key is to await q.join(), which blocks until all items in the queue have been received and processed, and then to cancel the consumer tasks, which would otherwise hang up and wait endlessly for additional queue items to appear.)

Here is the full script:

#!/usr/bin/env python3
# asyncq.py

import asyncio
import itertools as it
import os
import random
import time

async def makeitem(size: int = 5) -> str:
    return os.urandom(size).hex()

async def randint(a: int, b: int) -> int:
    return random.randint(a, b)

async def randsleep(a: int = 1, b: int = 5, caller=None) -> None:
    i = await randint(a, b)
    if caller:
        print(f"{caller} sleeping for {i} seconds.")
    await asyncio.sleep(i)

async def produce(name: int, q: asyncio.Queue) -> None:
    n = await randint(1, 5)
    for _ in it.repeat(None, n):  # Synchronous loop for each single producer
        await randsleep(caller=f"Producer {name}")
        i = await makeitem()
        t = time.perf_counter()
        await q.put((i, t))
        print(f"Producer {name} added <{i}> to queue.")

async def consume(name: int, q: asyncio.Queue) -> None:
    while True:
        await randsleep(caller=f"Consumer {name}")
        i, t = await q.get()
        now = time.perf_counter()
        print(f"Consumer {name} got element <{i}>"
              f" in {now-t:0.5f} seconds.")
        q.task_done()

async def main(nprod: int, ncon: int):
    q = asyncio.Queue()
    producers = [asyncio.create_task(produce(n, q)) for n in range(nprod)]
    consumers = [asyncio.create_task(consume(n, q)) for n in range(ncon)]
    await asyncio.gather(*producers)
    await q.join()  # Implicitly awaits consumers, too
    for c in consumers:
        c.cancel()

if __name__ == "__main__":
    import argparse
    random.seed(444)
    parser = argparse.ArgumentParser()
    parser.add_argument("-p", "--nprod", type=int, default=5)
    parser.add_argument("-c", "--ncon", type=int, default=10)
    ns = parser.parse_args()
    start = time.perf_counter()
    asyncio.run(main(**ns.__dict__))
    elapsed = time.perf_counter() - start
    print(f"Program completed in {elapsed:0.5f} seconds.")

The first few coroutines are helper functions that return a random string, a fractional-second performance counter, and a random integer. A producer puts anywhere from 1 to 5 items into the queue. Each item is a tuple of (i, t) where i is a random string and t is the time at which the producer attempts to put the tuple into the queue.

When a consumer pulls an item out, it simply calculates the elapsed time that the item sat in the queue using the timestamp that the item was put in with.

Keep in mind that asyncio.sleep() is used to mimic some other, more complex coroutine that would eat up time and block all other execution if it were a regular blocking function.

Here is a test run with two producers and five consumers:

$ python3 asyncq.py -p 2 -c 5
Producer 0 sleeping for 3 seconds.
Producer 1 sleeping for 3 seconds.
Consumer 0 sleeping for 4 seconds.
Consumer 1 sleeping for 3 seconds.
Consumer 2 sleeping for 3 seconds.
Consumer 3 sleeping for 5 seconds.
Consumer 4 sleeping for 4 seconds.
Producer 0 added <377b1e8f82> to queue.
Producer 0 sleeping for 5 seconds.
Producer 1 added <413b8802f8> to queue.
Consumer 1 got element <377b1e8f82> in 0.00013 seconds.
Consumer 1 sleeping for 3 seconds.
Consumer 2 got element <413b8802f8> in 0.00009 seconds.
Consumer 2 sleeping for 4 seconds.
Producer 0 added <06c055b3ab> to queue.
Producer 0 sleeping for 1 seconds.
Consumer 0 got element <06c055b3ab> in 0.00021 seconds.
Consumer 0 sleeping for 4 seconds.
Producer 0 added <17a8613276> to queue.
Consumer 4 got element <17a8613276> in 0.00022 seconds.
Consumer 4 sleeping for 5 seconds.
Program completed in 9.00954 seconds.

In this case, the items process in fractions of a second. A delay can be due to two reasons:

Standard, largely unavoidable overhead
Situations where all consumers are sleeping when an item appears in the queue

With regards to the second reason, luckily, it is perfectly normal to scale to hundreds or thousands of consumers. You should have no problem with python3 asyncq.py -p 5 -c 100. The point here is that, theoretically, you could have different users on different systems controlling the management of producers and consumers, with the queue serving as the central throughput.

So far, you’ve been thrown right into the fire and seen three related examples of asyncio calling coroutines defined with async and await. If you’re not completely following or just want to get deeper into the mechanics of how modern coroutines came to be in Python, you’ll start from square one with the next section.

Async IO’s Roots in Generators

Earlier, you saw an example of the old-style generator-based coroutines, which have been outdated by more explicit native coroutines. The example is worth re-showing with a small tweak:

import asyncio

@asyncio.coroutine
def py34_coro():
    """Generator-based coroutine"""
    # No need to build these yourself, but be aware of what they are
    s = yield from stuff()
    return s

async def py35_coro():
    """Native coroutine, modern syntax"""
    s = await stuff()
    return s

async def stuff():
    return 0x10, 0x20, 0x30

As an experiment, what happens if you call py34_coro() or py35_coro() on its own, without await, or without any calls to asyncio.run() or other asyncio “porcelain” functions? Calling a coroutine in isolation returns a coroutine object:

>>>

>>> py35_coro()
<coroutine object py35_coro at 0x10126dcc8>

This isn’t very interesting on its surface. The result of calling a coroutine on its own is an awaitable coroutine object.

Time for a quiz: what other feature of Python looks like this? (What feature of Python doesn’t actually “do much” when it’s called on its own?)

Hopefully you’re thinking of generators as an answer to this question, because coroutines are enhanced generators under the hood. The behavior is similar in this regard:

>>>

>>> def gen():
...     yield 0x10, 0x20, 0x30
...
>>> g = gen()
>>> g  # Nothing much happens - need to iterate with `.__next__()`
<generator object gen at 0x1012705e8>
>>> next(g)
(16, 32, 48)

Generator functions are, as it so happens, the foundation of async IO (regardless of whether you declare coroutines with async def rather than the older @asyncio.coroutine wrapper). Technically, await is more closely analogous to yield from than it is to yield. (But remember that yield from x() is just syntactic sugar to replace for i in x(): yield i.)

One critical feature of generators as it pertains to async IO is that they can effectively be stopped and restarted at will. For example, you can break out of iterating over a generator object and then resume iteration on the remaining values later. When a generator function reaches yield, it yields that value, but then it sits idle until it is told to yield its subsequent value.

This can be fleshed out through an example:

>>>

>>> from itertools import cycle
>>> def endless():
...     """Yields 9, 8, 7, 6, 9, 8, 7, 6, ... forever"""
...     yield from cycle((9, 8, 7, 6))

>>> e = endless()
>>> total = 0
>>> for i in e:
...     if total < 30:
...         print(i, end=" ")
...         total += i
...     else:
...         print()
...         # Pause execution. We can resume later.
...         break
9 8 7 6 9 8 7 6 9 8 7 6 9 8

>>> # Resume
>>> next(e), next(e), next(e)
(6, 9, 8)

The await keyword behaves similarly, marking a break point at which the coroutine suspends itself and lets other coroutines work. “Suspended,” in this case, means a coroutine that has temporarily ceded control but not totally exited or finished. Keep in mind that yield, and by extension yield from and await, mark a break point in a generator’s execution.

This is the fundamental difference between functions and generators. A function is all-or-nothing. Once it starts, it won’t stop until it hits a return, then pushes that value to the caller (the function that calls it). A generator, on the other hand, pauses each time it hits a yield and goes no further. Not only can it push this value to calling stack, but it can keep a hold of its local variables when you resume it by calling next() on it.

There’s a second and lesser-known feature of generators that also matters. You can send a value into a generator as well through its .send() method. This allows generators (and coroutines) to call (await) each other without blocking. I won’t get any further into the nuts and bolts of this feature, because it matters mainly for the implementation of coroutines behind the scenes, but you shouldn’t ever really need to use it directly yourself.

If you’re interested in exploring more, you can start at PEP 342, where coroutines were formally introduced. Brett Cannon’s How the Heck Does Async-Await Work in Python is also a good read, as is the PYMOTW writeup on asyncio. Lastly, there’s David Beazley’s Curious Course on Coroutines and Concurrency, which dives deep into the mechanism by which coroutines run.

Let’s try to condense all of the above articles into a few sentences: there is a particularly unconventional mechanism by which these coroutines actually get run. Their result is an attribute of the exception object that gets thrown when their .send() method is called. There’s some more wonky detail to all of this, but it probably won’t help you use this part of the language in practice, so let’s move on for now.

To tie things together, here are some key points on the topic of coroutines as generators:

Coroutines are repurposed generators that take advantage of the peculiarities of generator methods.
Old generator-based coroutines use yield from to wait for a coroutine result. Modern Python syntax in native coroutines simply replaces yield from with await as the means of waiting on a coroutine result. The await is analogous to yield from, and it often helps to think of it as such.
The use of await is a signal that marks a break point. It lets a coroutine temporarily suspend execution and permits the program to come back to it later.

Other Features: `async for` and Async Generators + Comprehensions

Along with plain async/await, Python also enables async for to iterate over an asynchronous iterator. The purpose of an asynchronous iterator is for it to be able to call asynchronous code at each stage when it is iterated over.

A natural extension of this concept is an asynchronous generator. Recall that you can use await, return, or yield in a native coroutine. Using yield within a coroutine became possible in Python 3.6 (via PEP 525), which introduced asynchronous generators with the purpose of allowing await and yield to be used in the same coroutine function body:

>>>

>>> async def mygen(u: int = 10):
...     """Yield powers of 2."""
...     i = 0
...     while i < u:
...         yield 2 ** i
...         i += 1
...         await asyncio.sleep(0.1)

Last but not least, Python enables asynchronous comprehension with async for. Like its synchronous cousin, this is largely syntactic sugar:

>>>

>>> async def main():
...     # This does *not* introduce concurrent execution
...     # It is meant to show syntax only
...     g = [i async for i in mygen()]
...     f = [j async for j in mygen() if not (j // 3 % 5)]
...     return g, f
...
>>> g, f = asyncio.run(main())
>>> g
[1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
>>> f
[1, 2, 16, 32, 256, 512]

This is a crucial distinction: neither asynchronous generators nor comprehensions make the iteration concurrent. All that they do is provide the look-and-feel of their synchronous counterparts, but with the ability for the loop in question to give up control to the event loop for some other coroutine to run.

In other words, asynchronous iterators and asynchronous generators are not designed to concurrently map some function over a sequence or iterator. They’re merely designed to let the enclosing coroutine allow other tasks to take their turn. The async for and async with statements are only needed to the extent that using plain for or with would “break” the nature of await in the coroutine. This distinction between asynchronicity and concurrency is a key one to grasp.

The Event Loop and `asyncio.run()`

You can think of an event loop as something like a while True loop that monitors coroutines, taking feedback on what’s idle, and looking around for things that can be executed in the meantime. It is able to wake up an idle coroutine when whatever that coroutine is waiting on becomes available.

Thus far, the entire management of the event loop has been implicitly handled by one function call:

asyncio.run(main())  # Python 3.7+

asyncio.run(), introduced in Python 3.7, is responsible for getting the event loop, running tasks until they are marked as complete, and then closing the event loop.

There’s a more long-winded way of managing the asyncio event loop, with get_event_loop(). The typical pattern looks like this:

loop = asyncio.get_event_loop()
try:
    loop.run_until_complete(main())
finally:
    loop.close()

You’ll probably see loop.get_event_loop() floating around in older examples, but unless you have a specific need to fine-tune control over the event loop management, asyncio.run() should be sufficient for most programs.

If you do need to interact with the event loop within a Python program, loop is a good-old-fashioned Python object that supports introspection with loop.is_running() and loop.is_closed(). You can manipulate it if you need to get more fine-tuned control, such as in scheduling a callback by passing the loop as an argument.

What is more crucial is understanding a bit beneath the surface about the mechanics of the event loop. Here are a few points worth stressing about the event loop.

#1: Coroutines don’t do much on their own until they are tied to the event loop.

You saw this point before in the explanation on generators, but it’s worth restating. If you have a main coroutine that awaits others, simply calling it in isolation has little effect:

>>>

>>> import asyncio

>>> async def main():
...     print("Hello ...")
...     await asyncio.sleep(1)
...     print("World!")

>>> routine = main()
>>> routine
<coroutine object main at 0x1027a6150>

Remember to use asyncio.run() to actually force execution by scheduling the main() coroutine (future object) for execution on the event loop:

>>>

>>> asyncio.run(routine)
Hello ...
World!

(Other coroutines can be executed with await. It is typical to wrap just main() in asyncio.run(), and chained coroutines with await will be called from there.)

#2: By default, an async IO event loop runs in a single thread and on a single CPU core. Usually, running one single-threaded event loop in one CPU core is more than sufficient. It is also possible to run event loops across multiple cores. Check out this talk by John Reese for more, and be warned that your laptop may spontaneously combust.

#3. Event loops are pluggable. That is, you could, if you really wanted, write your own event loop implementation and have it run tasks just the same. This is wonderfully demonstrated in the uvloop package, which is an implementation of the event loop in Cython.

That is what is meant by the term “pluggable event loop”: you can use any working implementation of an event loop, unrelated to the structure of the coroutines themselves. The asyncio package itself ships with two different event loop implementations, with the default being based on the selectors module. (The second implementation is built for Windows only.)

A Full Program: Asynchronous Requests

You’ve made it this far, and now it’s time for the fun and painless part. In this section, you’ll build a web-scraping URL collector, areq.py, using aiohttp, a blazingly fast async HTTP client/server framework. (We just need the client part.) Such a tool could be used to map connections between a cluster of sites, with the links forming a directed graph.

Note: You may be wondering why Python’s requests package isn’t compatible with async IO. requests is built on top of urrlib3, which in turn uses Python’s http and socket modules.

By default, socket operations are blocking. This means that Python won’t like await requests.get(url) because .get() is not awaitable. In contrast, almost everything in aiohttp is an awaitable coroutine, such as session.request() and response.text(). It’s a great package otherwise, but you’re doing yourself a disservice by using requests in asynchronous code.

The high-level program structure will look like this:

Read a sequence of URLs from a local file, urls.txt.
Send GET requests for the URLs and decode the resulting content. If this fails, stop there for a URL.
Search for the URLs within href tags in the HTML of the responses.
Write the results to foundurls.txt.
Do all of the above as asynchronously and concurrently as possible. (Use aiohttp for the requests, and aiofiles for the file-appends. These are two primary examples of IO that are well-suited for the async IO model.)

Here are the contents of urls.txt. It’s not huge, and contains mostly highly trafficked sites:

$ cat urls.txt
https://regex101.com/
https://docs.python.org/3/this-url-will-404.html
https://www.nytimes.com/guides/
https://www.mediamatters.org/
https://1.1.1.1/
https://www.politico.com/tipsheets/morning-money
https://www.bloomberg.com/markets/economics
https://www.ietf.org/rfc/rfc2616.txt

The second URL in the list should return a 404 response, which you’ll need to handle gracefully. If you’re running an expanded version of this program, you’ll probably need to deal with much hairier problems than this, such a server disconnections and endless redirects.

The requests themselves should be made using a single session, to take advantage of reusage of the session’s internal connection pool.

Let’s take a look at the full program. We’ll walk through things step-by-step after:

#!/usr/bin/env python3
# areq.py

"""Asynchronously get links embedded in multiple pages' HMTL."""

import asyncio
import logging
import re
import sys
from typing import IO
import urllib.error
import urllib.parse

import aiofiles
import aiohttp
from aiohttp import ClientSession

logging.basicConfig(
    format="%(asctime)s %(levelname)s:%(name)s: %(message)s",
    level=logging.DEBUG,
    datefmt="%H:%M:%S",
    stream=sys.stderr,
)
logger = logging.getLogger("areq")
logging.getLogger("chardet.charsetprober").disabled = True

HREF_RE = re.compile(r'href="(.*?)"')

async def fetch_html(url: str, session: ClientSession, **kwargs) -> str:
    """GET request wrapper to fetch page HTML.

    kwargs are passed to `session.request()`.
    """

    resp = await session.request(method="GET", url=url, **kwargs)
    resp.raise_for_status()
    logger.info("Got response [%s] for URL: %s", resp.status, url)
    html = await resp.text()
    return html

async def parse(url: str, session: ClientSession, **kwargs) -> set:
    """Find HREFs in the HTML of `url`."""
    found = set()
    try:
        html = await fetch_html(url=url, session=session, **kwargs)
    except (
        aiohttp.ClientError,
        aiohttp.http_exceptions.HttpProcessingError,
    ) as e:
        logger.error(
            "aiohttp exception for %s [%s]: %s",
            url,
            getattr(e, "status", None),
            getattr(e, "message", None),
        )
        return found
    except Exception as e:
        logger.exception(
            "Non-aiohttp exception occured:  %s", getattr(e, "__dict__", {})
        )
        return found
    else:
        for link in HREF_RE.findall(html):
            try:
                abslink = urllib.parse.urljoin(url, link)
            except (urllib.error.URLError, ValueError):
                logger.exception("Error parsing URL: %s", link)
                pass
            else:
                found.add(abslink)
        logger.info("Found %d links for %s", len(found), url)
        return found

async def write_one(file: IO, url: str, **kwargs) -> None:
    """Write the found HREFs from `url` to `file`."""
    res = await parse(url=url, **kwargs)
    if not res:
        return None
    async with aiofiles.open(file, "a") as f:
        for p in res:
            await f.write(f"{url}\t{p}\n")
        logger.info("Wrote results for source URL: %s", url)

async def bulk_crawl_and_write(file: IO, urls: set, **kwargs) -> None:
    """Crawl & write concurrently to `file` for multiple `urls`."""
    async with ClientSession() as session:
        tasks = []
        for url in urls:
            tasks.append(
                write_one(file=file, url=url, session=session, **kwargs)
            )
        await asyncio.gather(*tasks)

if __name__ == "__main__":
    import pathlib
    import sys

    assert sys.version_info >= (3, 7), "Script requires Python 3.7+."
    here = pathlib.Path(__file__).parent

    with open(here.joinpath("urls.txt")) as infile:
        urls = set(map(str.strip, infile))

    outpath = here.joinpath("foundurls.txt")
    with open(outpath, "w") as outfile:
        outfile.write("source_url\tparsed_url\n")

    asyncio.run(bulk_crawl_and_write(file=outpath, urls=urls))

This script is longer than our initial toy programs, so let’s break it down.

The constant HREF_RE is a regular expression to extract what we’re ultimately searching for, href tags within HTML:

>>>

>>> HREF_RE.search('Go to <a href="https://realpython.com/">Real Python</a>')
<re.Match object; span=(15, 45), match='href="https://realpython.com/"'>

The coroutine fetch_html() is a wrapper around a GET request to make the request and decode the resulting page HTML. It makes the request, awaits the response, and raises right away in the case of a non-200 status:

resp = await session.request(method="GET", url=url, **kwargs)
resp.raise_for_status()

If the status is okay, fetch_html() returns the page HTML (a str). Notably, there is no exception handling done in this function. The logic is to propagate that exception to the caller and let it be handled there:

html = await resp.text()

We await session.request() and resp.text() because they’re awaitable coroutines. The request/response cycle would otherwise be the long-tailed, time-hogging portion of the application, but with async IO, fetch_html() lets the event loop work on other readily available jobs such as parsing and writing URLs that have already been fetched.

Next in the chain of coroutines comes parse(), which waits on fetch_html() for a given URL, and then extracts all of the href tags from that page’s HTML, making sure that each is valid and formatting it as an absolute path.

Admittedly, the second portion of parse() is blocking, but it consists of a quick regex match and ensuring that the links discovered are made into absolute paths.

In this specific case, this synchronous code should be quick and inconspicuous. But just remember that any line within a given coroutine will block other coroutines unless that line uses yield, await, or return. If the parsing was a more intensive process, you might want to consider running this portion in its own process with loop.run_in_executor().

Next, the coroutine write() takes a file object and a single URL, and waits on parse() to return a set of the parsed URLs, writing each to the file asynchronously along with its source URL through use of aiofiles, a package for async file IO.

Lastly, bulk_crawl_and_write() serves as the main entry point into the script’s chain of coroutines. It uses a single session, and a task is created for each URL that is ultimately read from urls.txt.

Here are a few additional points that deserve mention:

The default ClientSession has an adapter with a maximum of 100 open connections. To change that, pass an instance of asyncio.connector.TCPConnector to ClientSession. You can also specify limits on a per-host basis.
You can specify max timeouts for both the session as a whole and for individual requests.
This script also uses async with, which works with an asynchronous context manager. I haven’t devoted a whole section to this concept because the transition from synchronous to asynchronous context managers is fairly straightforward. The latter has to define .__aenter__() and .__aexit__() rather than .__exit__() and .__enter__(). As you might expect, async with can only be used inside a coroutine function declared with async def.

If you’d like to explore a bit more, the companion files for this tutorial up at GitHub have comments and docstrings attached as well.

Here’s the execution in all of its glory, as areq.py gets, parses, and saves results for 9 URLs in under a second:

$ python3 areq.py
21:33:22 DEBUG:asyncio: Using selector: KqueueSelector
21:33:22 INFO:areq: Got response [200] for URL: https://www.mediamatters.org/
21:33:22 INFO:areq: Found 115 links for https://www.mediamatters.org/
21:33:22 INFO:areq: Got response [200] for URL: https://www.nytimes.com/guides/
21:33:22 INFO:areq: Got response [200] for URL: https://www.politico.com/tipsheets/morning-money
21:33:22 INFO:areq: Got response [200] for URL: https://www.ietf.org/rfc/rfc2616.txt
21:33:22 ERROR:areq: aiohttp exception for https://docs.python.org/3/this-url-will-404.html [404]: Not Found
21:33:22 INFO:areq: Found 120 links for https://www.nytimes.com/guides/
21:33:22 INFO:areq: Found 143 links for https://www.politico.com/tipsheets/morning-money
21:33:22 INFO:areq: Wrote results for source URL: https://www.mediamatters.org/
21:33:22 INFO:areq: Found 0 links for https://www.ietf.org/rfc/rfc2616.txt
21:33:22 INFO:areq: Got response [200] for URL: https://1.1.1.1/
21:33:22 INFO:areq: Wrote results for source URL: https://www.nytimes.com/guides/
21:33:22 INFO:areq: Wrote results for source URL: https://www.politico.com/tipsheets/morning-money
21:33:22 INFO:areq: Got response [200] for URL: https://www.bloomberg.com/markets/economics
21:33:22 INFO:areq: Found 3 links for https://www.bloomberg.com/markets/economics
21:33:22 INFO:areq: Wrote results for source URL: https://www.bloomberg.com/markets/economics
21:33:23 INFO:areq: Found 36 links for https://1.1.1.1/
21:33:23 INFO:areq: Got response [200] for URL: https://regex101.com/
21:33:23 INFO:areq: Found 23 links for https://regex101.com/
21:33:23 INFO:areq: Wrote results for source URL: https://regex101.com/
21:33:23 INFO:areq: Wrote results for source URL: https://1.1.1.1/

That’s not too shabby! As a sanity check, you can check the line-count on the output. In my case, it’s 626, though keep in mind this may fluctuate:

$ wc -l foundurls.txt
     626 foundurls.txt

$ head -n 3 foundurls.txt
source_url  parsed_url
https://www.bloomberg.com/markets/economics https://www.bloomberg.com/feedback
https://www.bloomberg.com/markets/economics https://www.bloomberg.com/notices/tos

Next Steps: If you’d like to up the ante, make this webcrawler recursive. You can use aio-redis to keep track of which URLs have been crawled within the tree to avoid requesting them twice, and connect links with Python’s networkx library.

Remember to be nice. Sending 1000 concurrent requests to a small, unsuspecting website is bad, bad, bad. There are ways to limit how many concurrent requests you’re making in one batch, such as in using the sempahore objects of asyncio or using a pattern like this one. If you don’t heed this warning, you may get a massive batch of TimeoutError exceptions and only end up hurting your own program.

Async IO in Context

Now that you’ve seen a healthy dose of code, let’s step back for a minute and consider when async IO is an ideal option and how you can make the comparison to arrive at that conclusion or otherwise choose a different model of concurrency.

When and Why Is Async IO the Right Choice?

This tutorial is no place for an extended treatise on async IO versus threading versus multiprocessing. However, it’s useful to have an idea of when async IO is probably the best candidate of the three.

The battle over async IO versus multiprocessing is not really a battle at all. In fact, they can be used in concert. If you have multiple, fairly uniform CPU-bound tasks (a great example is a grid search in libraries such as scikit-learn or keras), multiprocessing should be an obvious choice.

Simply putting async before every function is a bad idea if all of the functions use blocking calls. (This can actually slow down your code.) But as mentioned previously, there are places where async IO and multiprocessing can live in harmony.

The contest between async IO and threading is a little bit more direct. I mentioned in the introduction that “threading is hard.” The full story is that, even in cases where threading seems easy to implement, it can still lead to infamous impossible-to-trace bugs due to race conditions and memory usage, among other things.

Threading also tends to scale less elegantly than async IO, because threads are a system resource with a finite availability. Creating thousands of threads will fail on many machines, and I don’t recommend trying it in the first place. Creating thousands of async IO tasks is completely feasible.

Async IO shines when you have multiple IO-bound tasks where the tasks would otherwise be dominated by blocking IO-bound wait time, such as:

Network IO, whether your program is the server or the client side
Serverless designs, such as a peer-to-peer, multi-user network like a group chatroom
Read/write operations where you want to mimic a “fire-and-forget” style but worry less about holding a lock on whatever you’re reading and writing to

The biggest reason not to use it is that await only supports a specific set of objects that define a specific set of methods. If you want to do async read operations with a certain DBMS, you’ll need to find not just a Python wrapper for that DBMS, but one that supports the async/await syntax. Coroutines that contain synchronous calls block other coroutines and tasks from running.

For a shortlist of libraries that work with async/await, see the list at the end of this tutorial.

Async IO It Is, but Which One?

This tutorial focuses on async IO, the async/await syntax, and using asyncio for event-loop management and specifying tasks. asyncio certainly isn’t the only async IO library out there. This observation from Nathaniel J. Smith says a lot:

[In] a few years, asyncio might find itself relegated to becoming one of those stdlib libraries that savvy developers avoid, like urllib2.

…

What I’m arguing, in effect, is that asyncio is a victim of its own success: when it was designed, it used the best approach possible; but since then, work inspired by asyncio – like the addition of async/await – has shifted the landscape so that we can do even better, and now asyncio is hamstrung by its earlier commitments. (Source)

To that end, a few big-name alternatives that do what asyncio does, albeit with different APIs and different approaches, are curio and trio. Personally, I think that if you’re building a moderately sized, straightforward program, just using asyncio is plenty sufficient and understandable, and lets you avoid adding yet another large dependency outside of Python’s standard library.

But by all means, check out curio and trio, and you might find that they get the same thing done in a way that’s more intuitive for you as the user. Many of the package-agnostic concepts presented here should permeate to alternative async IO packages as well.

Odds and Ends

In these next few sections, you’ll cover some miscellaneous parts of asyncio and async/await that haven’t fit neatly into the tutorial thus far, but are still important for building and understanding a full program.

Other Top-Level `asyncio` Functions

In addition to asyncio.run(), you’ve seen a few other package-level functions such as asyncio.create_task() and asyncio.gather().

You can use create_task() to schedule the execution of a coroutine object, followed by asyncio.run():

>>>

>>> import asyncio

>>> async def coro(seq) -> list:
...     """'IO' wait time is proportional to the max element."""
...     await asyncio.sleep(max(seq))
...     return list(reversed(seq))
...
>>> async def main():
...     # This is a bit redundant in the case of one task
...     # We could use `await coro([3, 2, 1])` on its own
...     t = asyncio.create_task(coro([3, 2, 1]))  # Python 3.7+
...     await t
...     print(f't: type {type(t)}')
...     print(f't done: {t.done()}')
...
>>> t = asyncio.run(main())
t: type <class '_asyncio.Task'>
t done: True

There’s a subtlety to this pattern: if you don’t await t within main(), it may finish before main() itself signals that it is complete. Because asyncio.run(main()) calls loop.run_until_complete(main()), the event loop is only concerned (without await t present) that main() is done, not that the tasks that get created within main() are done. Without await t, the loop’s other tasks will be cancelled, possibly before they are completed. If you need to get a list of currently pending tasks, you can use asyncio.Task.all_tasks().

Note: asyncio.create_task() was introduced in Python 3.7. In Python 3.6 or lower, use asyncio.ensure_future() in place of create_task().

Separately, there’s asyncio.gather(). While it doesn’t do anything tremendously special, gather() is meant to neatly put a collection of coroutines (futures) into a single future. As a result, it returns a single future object, and, if you await asyncio.gather() and specify multiple tasks or coroutines, you’re waiting for all of them to be completed. (This somewhat parallels queue.join() from our earlier example.) The result of gather() will be a list of the results across the inputs:

>>>

>>> import time
>>> async def main():
...     t = asyncio.create_task(coro([3, 2, 1]))
...     t2 = asyncio.create_task(coro([10, 5, 0]))  # Python 3.7+
...     print('Start:', time.strftime('%X'))
...     a = await asyncio.gather(t, t2)
...     print('End:', time.strftime('%X'))  # Should be 10 seconds
...     print(f'Both tasks done: {all((t.done(), t2.done()))}')
...     return a
...
>>> a = asyncio.run(main())
Start: 16:20:11
End: 16:20:21
Both tasks done: True
>>> a
[[1, 2, 3], [0, 5, 10]]

You probably noticed that gather() waits on the entire result set of the Futures or coroutines that you pass it. Alternatively, you can loop over asyncio.as_completed() to get tasks as they are completed, in the order of completion. The function returns an iterator that yields tasks as they finish. Below, the result of coro([3, 2, 1]) will be available before coro([10, 5, 0]) is complete, which is not the case with gather():

>>>

>>> async def main():
...     t = asyncio.create_task(coro([3, 2, 1]))
...     t2 = asyncio.create_task(coro([10, 5, 0]))
...     print('Start:', time.strftime('%X'))
...     for res in asyncio.as_completed((t, t2)):
...         compl = await res
...         print(f'res: {compl} completed at {time.strftime("%X")}')
...     print('End:', time.strftime('%X'))
...     print(f'Both tasks done: {all((t.done(), t2.done()))}')
...
>>> a = asyncio.run(main())
Start: 09:49:07
res: [1, 2, 3] completed at 09:49:10
res: [0, 5, 10] completed at 09:49:17
End: 09:49:17
Both tasks done: True

Lastly, you may also see asyncio.ensure_future(). You should rarely need it, because it’s a lower-level plumbing API and largely replaced by create_task(), which was introduced later.

The Precedence of `await`

While they behave somewhat similarly, the await keyword has significantly higher precedence than yield. This means that, because it is more tightly bound, there are a number of instances where you’d need parentheses in a yield from statement that are not required in an analogous await statement. For more information, see examples of await expressions from PEP 492.

Conclusion

You’re now equipped to use async/await and the libraries built off of it. Here’s a recap of what you’ve covered:

Asynchronous IO as a language-agnostic model and a way to effect concurrency by letting coroutines indirectly communicate with each other
The specifics of Python’s new async and await keywords, used to mark and define coroutines
asyncio, the Python package that provides the API to run and manage coroutines

Resources

Python Version Specifics

Async IO in Python has evolved swiftly, and it can be hard to keep track of what came when. Here’s a list of Python minor-version changes and introductions related to asyncio:

3.3: The yield from expression allows for generator delegation.
3.4: asyncio was introduced in the Python standard library with provisional API status.
3.5: async and await became a part of the Python grammar, used to signify and wait on coroutines. They were not yet reserved keywords. (You could still define functions or variables named async and await.)
3.6: Asynchronous generators and asynchronous comprehensions were introduced. The API of asyncio was declared stable rather than provisional.
3.7: async and await became reserved keywords. (They cannot be used as identifiers.) They are intended to replace the asyncio.coroutine() decorator. asyncio.run() was introduced to the asyncio package, among a bunch of other features.

If you want to be safe (and be able to use asyncio.run()), go with Python 3.7 or above to get the full set of features.

Articles

Here’s a curated list of additional resources:

Real Python: Speed up your Python Program with Concurrency
Real Python: What is the Python Global Interpreter Lock?
CPython: The asyncio package source
Python docs: Data model > Coroutines
TalkPython: Async Techniques and Examples in Python
Brett Cannon: How the Heck Does Async-Await Work in Python 3.5?
PYMOTW: asyncio
A. Jesse Jiryu Davis and Guido van Rossum: A Web Crawler With asyncio Coroutines
Andy Pearce: The State of Python Coroutines: yield from
Nathaniel J. Smith: Some Thoughts on Asynchronous API Design in a Post-async/await World
Armin Ronacher: I don’t understand Python’s Asyncio
Andy Balaam: series on asyncio (4 posts)
Stack Overflow: Python asyncio.semaphore in async-await function
Yeray Diaz:
- AsyncIO for the Working Python Developer
- Asyncio Coroutine Patterns: Beyond await

A few Python What’s New sections explain the motivation behind language changes in more detail:

What’s New in Python 3.3 (yield from and PEP 380)
What’s New in Python 3.6 (PEP 525 & 530)

From David Beazley:

YouTube talks:

PEP	Date Created
PEP 342 – Coroutines via Enhanced Generators	2005-05
PEP 380 – Syntax for Delegating to a Subgenerator	2009-02
PEP 3153 – Asynchronous IO support	2011-05
PEP 3156 – Asynchronous IO Support Rebooted: the “asyncio” Module	2012-12
PEP 492 – Coroutines with async and await syntax	2015-04
PEP 525 – Asynchronous Generators	2016-07
PEP 530 – Asynchronous Comprehensions	2016-09

Libraries That Work With `async`/`await`

From aio-libs:

aiohttp: Asynchronous HTTP client/server framework
aioredis: Async IO Redis support
aiopg: Async IO PostgreSQL support
aiomcache: Async IO memcached client
aiokafka: Async IO Kafka client
aiozmq: Async IO ZeroMQ support
aiojobs: Jobs scheduler for managing background tasks
async_lru: Simple LRU cache for async IO

From magicstack:

uvloop: Ultra fast async IO event loop
asyncpg: (Also very fast) async IO PostgreSQL support

From other hosts:

trio: Friendlier asyncio intended to showcase a radically simpler design
aiofiles: Async file IO
asks: Async requests-like http library
asyncio-redis: Async IO Redis support
aioprocessing: Integrates multiprocessing module with asyncio
umongo: Async IO MongoDB client
unsync: Unsynchronize asyncio
aiostream: Like itertools, but async

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Speed Up Your Python Program With Concurrency

2019-01-14T14:00:00+00:00

If you’ve heard lots of talk about asyncio being added to Python but are curious how it compares to other concurrency methods or are wondering what concurrency is and how it might speed up your program, you’ve come to the right place.

In this article, you’ll learn the following:

What concurrency is
What parallelism is
How some of Python’s concurrency methods compare, including threading, asyncio, and multiprocessing
When to use concurrency in your program and which module to use

This article assumes that you have a basic understanding of Python and that you’re using at least version 3.6 to run the examples. You can download the examples from the Real Python GitHub repo.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

What Is Concurrency?

The dictionary definition of concurrency is simultaneous occurrence. In Python, the things that are occurring simultaneously are called by different names (thread, task, process) but at a high level, they all refer to a sequence of instructions that run in order.

I like to think of them as different trains of thought. Each one can be stopped at certain points, and the CPU or brain that is processing them can switch to a different one. The state of each one is saved so it can be restarted right where it was interrupted.

You might wonder why Python uses different words for the same concept. It turns out that threads, tasks, and processes are only the same if you view them from a high level. Once you start digging into the details, they all represent slightly different things. You’ll see more of how they are different as you progress through the examples.

Now let’s talk about the simultaneous part of that definition. You have to be a little careful because, when you get down to the details, only multiprocessing actually runs these trains of thought at literally the same time. Threading and asyncio both run on a single processor and therefore only run one at a time. They just cleverly find ways to take turns to speed up the overall process. Even though they don’t run different trains of thought simultaneously, we still call this concurrency.

The way the threads or tasks take turns is the big difference between threading and asyncio. In threading, the operating system actually knows about each thread and can interrupt it at any time to start running a different thread. This is called pre-emptive multitasking since the operating system can pre-empt your thread to make the switch.

Pre-emptive multitasking is handy in that the code in the thread doesn’t need to do anything to make the switch. It can also be difficult because of that “at any time” phrase. This switch can happen in the middle of a single Python statement, even a trivial one like x = x + 1.

Asyncio, on the other hand, uses cooperative multitasking. The tasks must cooperate by announcing when they are ready to be switched out. That means that the code in the task has to change slightly to make this happen.

The benefit of doing this extra work up front is that you always know where your task will be swapped out. It will not be swapped out in the middle of a Python statement unless that statement is marked. You’ll see later how this can simplify parts of your design.

What Is Parallelism?

So far, you’ve looked at concurrency that happens on a single processor. What about all of those CPU cores your cool, new laptop has? How can you make use of them? multiprocessing is the answer.

With multiprocessing, Python creates new processes. A process here can be thought of as almost a completely different program, though technically they’re usually defined as a collection of resources where the resources include memory, file handles and things like that. One way to think about it is that each process runs in its own Python interpreter.

Because they are different processes, each of your trains of thought in a multiprocessing program can run on a different core. Running on a different core means that they actually can run at the same time, which is fabulous. There are some complications that arise from doing this, but Python does a pretty good job of smoothing them over most of the time.

Now that you have an idea of what concurrency and parallelism are, let’s review their differences, and then we can look at why they can be useful:

Concurrency Type	Switching Decision	Number of Processors
Pre-emptive multitasking (`threading`)	The operating system decides when to switch tasks external to Python.	1
Cooperative multitasking (`asyncio`)	The tasks decide when to give up control.	1
Multiprocessing (`multiprocessing`)	The processes all run at the same time on different processors.	Many

Each of these types of concurrency can be useful. Let’s take a look at what types of programs they can help you speed up.

When Is Concurrency Useful?

Concurrency can make a big difference for two types of problems. These are generally called CPU-bound and I/O-bound.

I/O-bound problems cause your program to slow down because it frequently must wait for input/output (I/O) from some external resource. They arise frequently when your program is working with things that are much slower than your CPU.

Examples of things that are slower than your CPU are legion, but your program thankfully does not interact with most of them. The slow things your program will interact with most frequently are the file system and network connections.

Let’s see what that looks like:

In the diagram above, the blue boxes show time when your program is doing work, and the red boxes are time spent waiting for an I/O operation to complete. This diagram is not to scale because requests on the internet can take several orders of magnitude longer than CPU instructions, so your program can end up spending most of its time waiting. This is what your browser is doing most of the time.

On the flip side, there are classes of programs that do significant computation without talking to the network or accessing a file. These are the CPU-bound programs, because the resource limiting the speed of your program is the CPU, not the network or the file system.

Here’s a corresponding diagram for a CPU-bound program:

As you work through the examples in the following section, you’ll see that different forms of concurrency work better or worse with CPU-bound and I/O-bound programs. Adding concurrency to your program adds extra code and complications, so you’ll need to decide if the potential speed up is worth the extra effort. By the end of this article, you should have enough info to start making that decision.

Here’s a quick summary to clarify this concept:

I/O-Bound Process	CPU-Bound Process
Your program spends most of its time talking to a slow device, like a network connection, a hard drive, or a printer.	You program spends most of its time doing CPU operations.
Speeding it up involves overlapping the times spent waiting for these devices.	Speeding it up involves finding ways to do more computations in the same amount of time.

You’ll look at I/O-bound programs first. Then, you’ll get to see some code dealing with CPU-bound programs.

How to Speed Up an I/O-Bound Program

Let’s start by focusing on I/O-bound programs and a common problem: downloading content over the network. For our example, you will be downloading web pages from a few sites, but it really could be any network traffic. It’s just easier to visualize and set up with web pages.

Synchronous Version

We’ll start with a non-concurrent version of this task. Note that this program requires the requests module. You should run pip install requests before running it, probably using a virtualenv. This version does not use concurrency at all:

import requests
import time


def download_site(url, session):
    with session.get(url) as response:
        print(f"Read {len(response.content)} from {url}")


def download_all_sites(sites):
    with requests.Session() as session:
        for url in sites:
            download_site(url, session)


if __name__ == "__main__":
    sites = [
        "http://www.jython.org",
        "http://olympus.realpython.org/dice",
    ] * 80
    start_time = time.time()
    download_all_sites(sites)
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} in {duration} seconds")

As you can see, this is a fairly short program. download_site() just downloads the contents from a URL and prints the size. One small thing to point out is that we’re using a Session object from requests.

It is possible to simply use get() from requests directly, but creating a Session object allows requests to do some fancy networking tricks and really speed things up.

download_all_sites() creates the Session and then walks through the list of sites, downloading each one in turn. Finally, it prints out how long this process took so you can have the satisfaction of seeing how much concurrency has helped us in the following examples.

The processing diagram for this program will look much like the I/O-bound diagram in the last section.

Note: Network traffic is dependent on many factors that can vary from second to second. I’ve seen the times of these tests double from one run to another due to network issues.

Why the Synchronous Version Rocks

The great thing about this version of code is that, well, it’s easy. It was comparatively easy to write and debug. It’s also more straight-forward to think about. There’s only one train of thought running through it, so you can predict what the next step is and how it will behave.

The Problems With the Synchronous Version

The big problem here is that it’s relatively slow compared to the other solutions we’ll provide. Here’s an example of what the final output gave on my machine:

$ ./io_non_concurrent.py
   [most output skipped]
Downloaded 160 in 14.289619207382202 seconds

Note: Your results may vary significantly. When running this script, I saw the times vary from 14.2 to 21.9 seconds. For this article, I took the fastest of three runs as the time. The differences between the methods will still be clear.

Being slower isn’t always a big issue, however. If the program you’re running takes only 2 seconds with a synchronous version and is only run rarely, it’s probably not worth adding concurrency. You can stop here.

What if your program is run frequently? What if it takes hours to run? Let’s move on to concurrency by rewriting this program using threading.

`threading` Version

As you probably guessed, writing a threaded program takes more effort. You might be surprised at how little extra effort it takes for simple cases, however. Here’s what the same program looks like with threading:

import concurrent.futures
import requests
import threading
import time


thread_local = threading.local()


def get_session():
    if not getattr(thread_local, "session", None):
        thread_local.session = requests.Session()
    return thread_local.session


def download_site(url):
    session = get_session()
    with session.get(url) as response:
        print(f"Read {len(response.content)} from {url}")


def download_all_sites(sites):
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        executor.map(download_site, sites)


if __name__ == "__main__":
    sites = [
        "http://www.jython.org",
        "http://olympus.realpython.org/dice",
    ] * 80
    start_time = time.time()
    download_all_sites(sites)
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} in {duration} seconds")

When you add threading, the overall structure is the same and you only needed to make a few changes. download_all_sites() changed from calling the function once per site to a more complex structure.

In this version, you’re creating a ThreadPoolExecutor, which seems like a complicated thing. Let’s break that down: ThreadPoolExecutor = Thread + Pool + Executor.

You already know about the Thread part. That’s just a train of thought we mentioned earlier. The Pool portion is where it starts to get interesting. This object is going to create a pool of threads, each of which can run concurrently. Finally, the Executor is the part that’s going to control how and when each of the threads in the pool will run. It will execute the request in the pool.

Helpfully, the standard library implements ThreadPoolExecutor as a context manager so you can use the with syntax to manage creating and freeing the pool of Threads.

Once you have a ThreadPoolExecutor, you can use its handy .map() method. This method runs the passed-in function on each of the sites in the list. The great part is that it automatically runs them concurrently using the pool of threads it is managing.

Those of you coming from other languages, or even Python 2, are probably wondering where the usual objects and functions are that manage the details you’re used to when dealing with threading, things like Thread.start(), Thread.join(), and Queue.

These are all still there, and you can use them to achieve fine-grained control of how your threads are run. But, starting with Python 3.2, the standard library added a higher-level abstraction called Executors that manage many of the details for you if you don’t need that fine-grained control.

The other interesting change in our example is that each thread needs to create its own requests.Session() object. When you’re looking at the documentation for requests, it’s not necessarily easy to tell, but reading this issue, it seems fairly clear that you need a separate Session for each thread.

This is one of the interesting and difficult issues with threading. Because the operating system is in control of when your task gets interrupted and another task starts, any data that is shared between the threads needs to be protected, or thread-safe. Unfortunately requests.Session() is not thread-safe.

There are several strategies for making data accesses thread-safe depending on what the data is and how you’re using it. One of them is to use thread-safe data structures like Queue from Python’s queue module.

These objects use low-level primitives like threading.Lock to ensure that only one thread can access a block of code or a bit of memory at the same time. You are using this strategy indirectly by way of the ThreadPoolExecutor object.

Another strategy to use here is something called thread local storage. Threading.local() creates an object that look like a global but is specific to each individual thread. In your example, this is done with threadLocal and get_session():

threadLocal = threading.local()


def get_session():
    if getattr(threadLocal, "session", None) is None:
        threadLocal.session = requests.Session()
    return threadLocal.session

ThreadLocal is in the threading module to specifically solve this problem. It looks a little odd, but you only want to create one of these objects, not one for each thread. The object itself takes care of separating accesses from different threads to different data.

When get_session() is called, the session it looks up is specific to the particular thread on which it’s running. So each thread will create a single session the first time it calls get_session() and then will simply use that session on each subsequent call throughout its lifetime.

Finally, a quick note about picking the number of threads. You can see that the example code uses 5 threads. Feel free to play around with this number and see how the overall time changes. You might expect that having one thread per download would be the fastest but, at least on my system it was not. I found the fastest results somewhere between 5 and 10 threads. If you go any higher than that, then the extra overhead of creating and destroying the threads erases any time savings.

The difficult answer here is that the correct number of threads is not a constant from one task to another. Some experimentation is required.

Why the threading Version Rocks

It’s fast! Here’s the fastest run of my tests. Remember that the non-concurrent version took more than 14 seconds:

$ ./io_threading.py
   [most output skipped]
Downloaded 160 in 3.7238826751708984 seconds

Here’s what its execution timing diagram looks like:

It uses multiple threads to have multiple open requests out to web sites at the same time, allowing your program to overlap the waiting times and get the final result faster! Yippee! That was the goal.

The Problems with the threading Version

Well, as you can see from the example, it takes a little more code to make this happen, and you really have to give some thought to what data is shared between threads.

Threads can interact in ways that are subtle and hard to detect. These interactions can cause race conditions that frequently result in random, intermittent bugs that can be quite difficult to find. Those of you who are unfamiliar with the concept of race conditions might want to expand and read the section below.

Race conditions are an entire class of subtle bugs that can and frequently do happen in multi-threaded code. Race conditions happen because the programmer has not sufficiently protected data accesses to prevent threads from interfering with each other. You need to take extra steps when writing threaded code to ensure things are thread-safe.

What’s going on here is that the operating system is controlling when your thread runs and when it gets swapped out to let another thread run. This thread swapping can occur at any point, even while doing sub-steps of a Python statement. As a quick example, look at this function:

import concurrent.futures


counter = 0


def increment_counter(fake_value):
    global counter
    for _ in range(100):
        counter += 1


if __name__ == "__main__":
    fake_data = [x for x in range(5000)]
    counter = 0
    with concurrent.futures.ThreadPoolExecutor(max_workers=5000) as executor:
        executor.map(increment_counter, fake_data)

This code is quite similar to the structure you used in the threading example above. The difference is that each of the threads is accessing the same global variable counter and incrementing it. Counter is not protected in any way, so it is not thread-safe.

In order to increment counter, each of the threads needs to read the current value, add one to it, and the save that value back to the variable. That happens in this line: counter += 1.

Because the operating system knows nothing about your code and can swap threads at any point in the execution, it’s possible for this swap to happen after a thread has read the value but before it has had the chance to write it back. If the new code that is running modifies counter as well, then the first thread has a stale copy of the data and trouble will ensue.

As you can imagine, hitting this exact situation is fairly rare. You can run this program thousands of times and never see the problem. That’s what makes this type of problem quite difficult to debug as it can be quite hard to reproduce and can cause random-looking errors to show up.

As a further example, I want to remind you that requests.Session() is not thread-safe. This means that there are places where the type of interaction described above could happen if multiple threads use the same Session. I bring this up not to cast aspersions on requests but rather to point out that these are difficult problems to resolve.

`asyncio` Version

Before you jump into examining the asyncio example code, let’s talk more about how asyncio works.

asyncio Basics

This will be a simplified version of asycio. There are many details that are glossed over here, but it still conveys the idea of how it works.

The general concept of asyncio is that a single Python object, called the event loop, controls how and when each task gets run. The event loop is aware of each task and knows what state it’s in. In reality, there are many states that tasks could be in, but for now let’s imagine a simplified event loop that just has two states.

The ready state will indicate that a task has work to do and is ready to be run, and the waiting state means that the task is waiting for some external thing to finish, such as a network operation.

Your simplified event loop maintains two lists of tasks, one for each of these states. It selects one of the ready tasks and starts it back to running. That task is in complete control until it cooperatively hands the control back to the event loop.

When the running task gives control back to the event loop, the event loop places that task into either the ready or waiting list and then goes through each of the tasks in the waiting list to see if it has become ready by an I/O operation completing. It knows that the tasks in the ready list are still ready because it knows they haven’t run yet.

Once all of the tasks have been sorted into the right list again, the event loop picks the next task to run, and the process repeats. Your simplified event loop picks the task that has been waiting the longest and runs that. This process repeats until the event loop is finished.

An important point of asyncio is that the tasks never give up control without intentionally doing so. They never get interrupted in the middle of an operation. This allows us to share resources a bit more easily in asyncio than in threading. You don’t have to worry about making your code thread-safe.

That’s a high-level view of what’s happening with asyncio. If you want more detail, this StackOverflow answer provides some good details if you want to dig deeper.

async and await

Now let’s talk about two new keywords that were added to Python: async and await. In light of the discussion above, you can view await as the magic that allows the task to hand control back to the event loop. When your code awaits a function call, it’s a signal that the call is likely to be something that takes a while and that the task should give up control.

It’s easiest to think of async as a flag to Python telling it that the function about to be defined uses await. There are some cases where this is not strictly true, like asynchronous generators, but it holds for many cases and gives you a simple model while you’re getting started.

One exception to this that you’ll see in the next code is the async with statement, which creates a context manager from an object you would normally await. While the semantics are a little different, the idea is the same: to flag this context manager as something that can get swapped out.

As I’m sure you can imagine, there’s some complexity in managing the interaction between the event loop and the tasks. For developers starting out with asyncio, these details aren’t important, but you do need to remember that any function that calls await needs to be marked with async. You’ll get a syntax error otherwise.

Back to Code

Now that you’ve got a basic understanding of what asyncio is, let’s walk through the asyncio version of the example code and figure out how it works. Note that this version adds aiohttp. You should run pip install aiohttp before running it:

import asyncio
import time
import aiohttp


async def download_site(session, url):
    async with session.get(url) as response:
        print("Read {0} from {1}".format(response.content_length, url))


async def download_all_sites(sites):
    async with aiohttp.ClientSession() as session:
        tasks = []
        for url in sites:
            task = asyncio.ensure_future(download_site(session, url))
            tasks.append(task)
        await asyncio.gather(*tasks, return_exceptions=True)


if __name__ == "__main__":
    sites = [
        "http://www.jython.org",
        "http://olympus.realpython.org/dice",
    ] * 80
    start_time = time.time()
    asyncio.get_event_loop().run_until_complete(download_all_sites(sites))
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} sites in {duration} seconds")

This version is a bit more complex than the previous two. It has a similar structure, but there’s a bit more work setting up the tasks than there was creating the ThreadPoolExecutor. Let’s start at the top of the example.

download_site()

download_site() at the top is almost identical to the threading version with the exception of the async keyword on the function definition line and the async with keywords when you actually call session.get(). You’ll see later why Session can be passed in here rather than using thread-local storage.

download_all_sites()

download_all_sites() is where you will see the biggest change from the threading example.

You can share the session across all tasks, so the session is created here as a context manager. The tasks can share the session because they are all running on the same thread. There is no way one task could interrupt another while the session is in a bad state.

Inside that context manager, it creates a list of tasks using asyncio.ensure_future(), which also takes care of starting them. Once all the tasks are created, this function uses asyncio.gather() to keep the session context alive until all of the tasks have completed.

The threading code does something similar to this, but the details are conveniently handled in the ThreadPoolExecutor. There currently is not an AsyncioPoolExecutor class.

There is one small but important change buried in the details here, however. Remember how we talked about the number of threads to create? It wasn’t obvious in the threading example what the optimal number of threads was.

One of the cool advantages of asyncio is that it scales far better than threading. Each task takes far fewer resources and less time to create than a thread, so creating and running more of them works well. This example just creates a separate task for each site to download, which works out quite well.

__main__

Finally, the nature of asyncio means that you have to start up the event loop and tell it which tasks to run. The __main__ section at the bottom of the file contains the code to get_event_loop() and then run_until_complete(). If nothing else, they’ve done an excellent job in naming those functions.

If you’ve updated to Python 3.7, the Python core developers simplified this syntax for you. Instead of the asyncio.get_event_loop().run_until_complete() tongue-twister, you can just use asyncio.run().

Why the asyncio Version Rocks

It’s really fast! In the tests on my machine, this was the fastest version of the code by a good margin:

$ ./io_asyncio.py
   [most output skipped]
Downloaded 160 in 2.5727896690368652 seconds

The execution timing diagram looks quite similar to what’s happening in the threading example. It’s just that the I/O requests are all done by the same thread:

The lack of a nice wrapper like the ThreadPoolExecutor makes this code a bit more complex than the threading example. This is a case where you have to do a little extra work to get much better performance.

Also there’s a common argument that having to add async and await in the proper locations is an extra complication. To a small extent, that is true. The flip side of this argument is that it forces you to think about when a given task will get swapped out, which can help you create a better, faster, design.

The scaling issue also looms large here. Running the threading example above with a thread for each site is noticeably slower than running it with a handful of threads. Running the asyncio example with hundreds of tasks didn’t slow it down at all.

The Problems With the asyncio Version

There are a couple of issues with asyncio at this point. You need special async versions of libraries to gain the full advantage of asycio. Had you just used requests for downloading the sites, it would have been much slower because requests is not designed to notify the event loop that it’s blocked. This issue is getting smaller and smaller as time goes on and more libraries embrace asyncio.

Another, more subtle, issue is that all of the advantages of cooperative multitasking get thrown away if one of the tasks doesn’t cooperate. A minor mistake in code can cause a task to run off and hold the processor for a long time, starving other tasks that need running. There is no way for the event loop to break in if a task does not hand control back to it.

With that in mind, let’s step up to a radically different approach to concurrency, multiprocessing.

`multiprocessing` Version

Unlike the previous approaches, the multiprocessing version of the code takes full advantage of the multiple CPUs that your cool, new computer has. Or, in my case, that my clunky, old laptop has. Let’s start with the code:

import requests
import multiprocessing
import time

session = None


def set_global_session():
    global session
    if not session:
        session = requests.Session()


def download_site(url):
    with session.get(url) as response:
        name = multiprocessing.current_process().name
        print(f"{name}:Read {len(response.content)} from {url}")


def download_all_sites(sites):
    with multiprocessing.Pool(initializer=set_global_session) as pool:
        pool.map(download_site, sites)


if __name__ == "__main__":
    sites = [
        "http://www.jython.org",
        "http://olympus.realpython.org/dice",
    ] * 80
    start_time = time.time()
    download_all_sites(sites)
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} in {duration} seconds")

This is much shorter than the asyncio example and actually looks quite similar to the threading example, but before we dive into the code, let’s take a quick tour of what multiprocessing does for you.

multiprocessing in a Nutshell

Up until this point, all of the examples of concurrency in this article run only on a single CPU or core in your computer. The reasons for this have to do with the current design of CPython and something called the Global Interpreter Lock, or GIL.

This article won’t dive into the hows and whys of the GIL. It’s enough for now to know that the synchronous, threading, and asyncio versions of this example all run on a single CPU.

multiprocessing in the standard library was designed to break down that barrier and run your code across multiple CPUs. At a high level, it does this by creating a new instance of the Python interpreter to run on each CPU and then farming out part of your program to run on it.

As you can imagine, bringing up a separate Python interpreter is not as fast as starting a new thread in the current Python interpreter. It’s a heavyweight operation and comes with some restrictions and difficulties, but for the correct problem, it can make a huge difference.

multiprocessing Code

The code has a few small changes from our synchronous version. The first one is in download_all_sites(). Instead of simply calling download_site() repeatedly, it creates a multiprocessing.Pool object and has it map download_site to the iterable sites. This should look familiar from the threading example.

What happens here is that the Pool creates a number of separate Python interpreter processes and has each one run the specified function on some of the items in the iterable, which in our case is the list of sites. The communication between the main process and the other processes is handled by the multiprocessing module for you.

The line that creates Pool is worth your attention. First off, it does not specify how many processes to create in the Pool, although that is an optional parameter. By default, multiprocessing.Pool() will determine the number of CPUs in your computer and match that. This is frequently the best answer, and it is in our case.

For this problem, increasing the number of processes did not make things faster. It actually slowed things down because the cost for setting up and tearing down all those processes was larger than the benefit of doing the I/O requests in parallel.

Next we have the initializer=set_global_session part of that call. Remember that each process in our Pool has its own memory space. That means that they cannot share things like a Session object. You don’t want to create a new Session each time the function is called, you want to create one for each process.

The initializer function parameter is built for just this case. There is not a way to pass a return value back from the initializer to the function called by the process download_site(), but you can initialize a global session variable to hold the single session for each process. Because each process has its own memory space, the global for each one will be different.

That’s really all there is to it. The rest of the code is quite similar to what you’ve seen before.

Why the multiprocessing Version Rocks

The multiprocessing version of this example is great because it’s relatively easy to set up and requires little extra code. It also takes full advantage of the CPU power in your computer. The execution timing diagram for this code looks like this:

The Problems With the multiprocessing Version

This version of the example does require some extra setup, and the global session object is strange. You have to spend some time thinking about which variables will be accessed in each process.

Finally, it is clearly slower than the asyncio and threading versions in this example:

$ ./io_mp.py
    [most output skipped]
Downloaded 160 in 5.718175172805786 seconds

That’s not surprising, as I/O-bound problems are not really why multiprocessing exists. You’ll see more as you step into the next section and look at CPU-bound examples.

How to Speed Up a CPU-Bound Program

Let’s shift gears here a little bit. The examples so far have all dealt with an I/O-bound problem. Now, you’ll look into a CPU-bound problem. As you saw, an I/O-bound problem spends most of its time waiting for external operations, like a network call, to complete. A CPU-bound problem, on the other hand, does few I/O operations, and its overall execution time is a factor of how fast it can process the required data.

For the purposes of our example, we’ll use a somewhat silly function to create something that takes a long time to run on the CPU. This function computes the square of each number from 1 to the passed-in value:

def cpu_bound(number):
    return sum(i * i for i in range(number))

You’ll be passing in large numbers, so this will take a while. Remember, this is just a placeholder for your code that actually does something useful and requires significant processing time, like computing the roots of equations or sorting a large data structure.

CPU-Bound Synchronous Version

Now let’s look at the non-concurrent version of the example:

import time


def cpu_bound(number):
    return sum(i * i for i in range(number))


def find_sums(numbers):
    for number in numbers:
        cpu_bound(number)


if __name__ == "__main__":
    numbers = [5_000_000 + x for x in range(20)]

    start_time = time.time()
    find_sums(numbers)
    duration = time.time() - start_time
    print(f"Duration {duration} seconds")

This code calls cpu_bound() 20 times with a different large number each time. It does all of this on a single thread in a single process on a single CPU. The execution timing diagram looks like this:

Unlike the I/O-bound examples, the CPU-bound examples are usually fairly consistent in their run times. This one takes about 7.8 seconds on my machine:

$ ./cpu_non_concurrent.py
Duration 7.834432125091553 seconds

Clearly we can do better than this. This is all running on a single CPU with no concurrency. Let’s see what we can do to make it better.

`threading` and `asyncio` Versions

How much do you think rewriting this code using threading or asyncio will speed this up?

If you answered “Not at all,” give yourself a cookie. If you answered, “It will slow it down,” give yourself two cookies.

Here’s why: In your I/O-bound example above, much of the overall time was spent waiting for slow operations to finish. threading and asyncio sped this up by allowing you to overlap the times you were waiting instead of doing them sequentially.

On a CPU-bound problem, however, there is no waiting. The CPU is cranking away as fast as it can to finish the problem. In Python, both threads and tasks run on the same CPU in the same process. That means that the one CPU is doing all of the work of the non-concurrent code plus the extra work of setting up threads or tasks. It takes more than 10 seconds:

$ ./cpu_threading.py
Duration 10.407078266143799 seconds

I’ve written up a threading version of this code and placed it with the other example code in the GitHub repo so you can go test this yourself. Let’s not look at that just yet, however.

CPU-Bound `multiprocessing` Version

Now you’ve finally reached where multiprocessing really shines. Unlike the other concurrency libraries, multiprocessing is explicitly designed to share heavy CPU workloads across multiple CPUs. Here’s what its execution timing diagram looks like:

Here’s what the code looks like:

import multiprocessing
import time


def cpu_bound(number):
    return sum(i * i for i in range(number))


def find_sums(numbers):
    with multiprocessing.Pool() as pool:
        pool.map(cpu_bound, numbers)


if __name__ == "__main__":
    numbers = [5_000_000 + x for x in range(20)]

    start_time = time.time()
    find_sums(numbers)
    duration = time.time() - start_time
    print(f"Duration {duration} seconds")

Little of this code had to change from the non-concurrent version. You had to import multiprocessing and then just change from looping through the numbers to creating a multiprocessing.Pool object and using its .map() method to send individual numbers to worker-processes as they become free.

This was just what you did for the I/O-bound multiprocessing code, but here you don’t need to worry about the Session object.

As mentioned above, the processes optional parameter to the multiprocessing.Pool() constructor deserves some attention. You can specify how many Process objects you want created and managed in the Pool. By default, it will determine how many CPUs are in your machine and create a process for each one. While this works great for our simple example, you might want to have a little more control in a production environment.

Also, as we mentioned in the first section about threading, the multiprocessing.Pool code is built upon building blocks like Queue and Semaphore that will be familiar to those of you who have done multithreaded and multiprocessing code in other languages.

Why the multiprocessing Version Rocks

The multiprocessing version of this example is great because it’s relatively easy to set up and requires little extra code. It also takes full advantage of the CPU power in your computer.

Hey, that’s exactly what I said the last time we looked at multiprocessing. The big difference is that this time it is clearly the best option. It takes 2.5 seconds on my machine:

$ ./cpu_mp.py
Duration 2.5175397396087646 seconds

That’s much better than we saw with the other options.

The Problems With the multiprocessing Version

There are some drawbacks to using multiprocessing. They don’t really show up in this simple example, but splitting your problem up so each processor can work independently can sometimes be difficult.

Also, many solutions require more communication between the processes. This can add some complexity to your solution that a non-concurrent program would not need to deal with.

When to Use Concurrency

You’ve covered a lot of ground here, so let’s review some of the key ideas and then discuss some decision points that will help you determine which, if any, concurrency module you want to use in your project.

The first step of this process is deciding if you should use a concurrency module. While the examples here make each of the libraries look pretty simple, concurrency always comes with extra complexity and can often result in bugs that are difficult to find.

Hold out on adding concurrency until you have a known performance issue and then determine which type of concurrency you need. As Donald Knuth has said, “Premature optimization is the root of all evil (or at least most of it) in programming.”

Once you’ve decided that you should optimize your program, figuring out if your program is CPU-bound or I/O-bound is a great next step. Remember that I/O-bound programs are those that spend most of their time waiting for something to happen while CPU-bound programs spend their time processing data or crunching numbers as fast as they can.

As you saw, CPU-bound problems only really gain from using multiprocessing. threading and asyncio did not help this type of problem at all.

For I/O-bound problems, there’s a general rule of thumb in the Python community: “Use asyncio when you can, threading when you must.” asyncio can provide the best speed up for this type of program, but sometimes you will require critical libraries that have not been ported to take advantage of asyncio. Remember that any task that doesn’t give up control to the event loop will block all of the other tasks.

Conclusion

You’ve now seen the basic types of concurrency available in Python:

threading
asyncio
multiprocessing

You’ve got the understanding to decide which concurrency method you should use for a given problem, or if you should use any at all! In addition, you’ve achieved a better understanding of some of the problems that can arise when you’re using concurrency.

I hope you’ve learned a lot from this article and that you find a great use for concurrency in your own projects!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Django Migrations: A Primer

2019-01-09T12:10:21+00:00

Since version 1.7, Django has come with built-in support for database migrations. In Django, database migrations usually go hand in hand with models: whenever you code up a new model, you also generate a migration to create the necessary table in the database. However, migrations can do much more.

You are going to learn how Django Migrations work and how you can get the most out of them over the course of four articles and one video:

Part 1: Django Migrations: A Primer (current article)
Part 2: Digging Deeper into Migrations
Part 3: Data Migrations
Video: Django 1.7 Migrations - primer

In this article, you’ll get comfortable with Django migrations and learn the following:

How to create database tables without writing any SQL
How to automatically modify your database after you changed your models
How to revert changes made to your database

Free Bonus: Click here to get access to a free Django Learning Resources Guide (PDF) that shows you tips and tricks as well as common pitfalls to avoid when building Python + Django web applications.

The Problems That Migrations Solve

If you are new to Django or web development in general, you might not be familiar with the concept of database migrations, and it might not seem obvious why they’re a good idea.

First, let’s quickly define a couple of terms to make sure everybody is on the same page. Django is designed to work with a relational database, stored in a relational database management system like PostgreSQL, MySQL, or SQLite.

In a relational database, data is organized in tables. A database table has a certain number of columns, but it can have any number of rows. Each column has a specific datatype, like a string of a certain maximum length or a positive integer. The description of all tables with their columns and their respective datatypes is called a database schema.

All database systems supported by Django use the language SQL to create, read, update and delete data in a relational database. SQL is also used to create, change, and delete the database tables themselves.

Working directly with SQL can be quite cumbersome, so to make your life easier, Django comes with an object-relational mapper, or ORM for short. The ORM maps the relational database to the world of object oriented programming. Instead of defining database tables in SQL, you write Django models in Python. Your models define database fields, which correspond to the columns in their database tables.

Here’s an example of how a Django model class is mapped to a database table:

But just defining a model class in a Python file does not make a database table magically appear out of nowhere. Creating the database tables to store your Django models is the job of a database migration. Additionally, whenever you make a change to your models, like adding a field, the database has to be changed too. Migrations handle that as well.

Here are a few ways Django migrations make your life easier.

Making Database Changes Without SQL

Without migrations, you would have to connect to your database and type in a bunch of SQL commands or use a graphical tool like PHPMyAdmin to modify the database schema every time you wanted to change your model definition.

In Django, migrations are primarily written in Python, so you don’t have to know any SQL unless you have really advanced use cases.

Avoiding Repetition

Creating a model and then writing SQL to create the database tables for it would be repetitive.

Migrations are generated from your models, making sure you don’t repeat yourself.

Ensuring Model Definitions and the Database Schema in Sync

Usually, you have multiple instances of your database, for example one database for each developer in your team, a database for testing and a database with live data.

Without migrations, you will have to perform any schema changes on each one of your database, and you will have to keep track which changes have already been made to which database.

With Django Migrations, you can easily keep multiple databases in sync with your models.

Tracking Database Schema Change in Version Control

A version control system, like Git is excellent for code, but not so much for database schemas.

As migrations are plain Python in Django, you can put them in a version control system just like any other piece of code.

By now, you’re hopefully convinced that migrations are a useful and powerful tool. Let’s start learning how to unleash that power.

Setting Up a Django Project

Throughout this tutorial, you are going to work on a simple Bitcoin tracker app as an example project.

The first step is to install Django. Here is how you do that on Linux or macOS X using a virtual environment:

$ python3 -m venv env
$ source env/bin/activate
(env) $ $ pip install "Django==2.1.*"
...
Successfully installed Django-2.1.3

Now you have created a new virtual environment and activated it, as well as installed Django in that virtual environment.

Note that on Windows, you would run env/bin/activate.bat instead of source env/bin/activate to activate your virtual environment.

For easier readability, console examples won’t include the (env) part of the prompt from now on.

With Django installed, you can create the project using the following commands:

$ django-admin.py startproject bitcoin_tracker
$ cd bitcoin_tracker
$ python manage.py startapp historical_data

This gives you a simple project and an app called historical_data. You should now have this directory structure:

bitcoin_tracker/
|
├── bitcoin_tracker/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
|
├── historical_data/
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── migrations/
│   │   └── __init__.py
|   |
│   ├── models.py
│   ├── tests.py
│   └── views.py
|
└── manage.py

Within the bitcoin_tracker directory, there are two sub-directories: bitcoin_tracker for project-wide files and historical_data containing files for the app you created.

Now, to create a model, add this class in historical_data/models.py:

class PriceHistory(models.Model):
    date = models.DateTimeField(auto_now_add=True)
    price = models.DecimalField(max_digits=7, decimal_places=2)
    volume = models.PositiveIntegerField()

This is the basic model to keep track of Bitcoin prices.

Also, don’t forget to add the newly created app to settings.INSTALLED_APPS. Open bitcoin_tracker/settings.py and append historical_data to the list INSTALLED_APPS, like this:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'historical_data',
]

The other settings are fine for this project. This tutorial assumes that your project is configured to use an SQLite database, which is the default.

Creating Migrations

With the model created, the first thing you need to do is create a migration for it. You can do this with the following command:

$ python manage.py makemigrations historical_data
Migrations for 'historical_data':
  historical_data/migrations/0001_initial.py
    - Create model PriceHistory

Note: Specifying the name of the application, historical_data, is optional. Leaving it off creates migrations for all apps.

This creates the migrations file that instructs Django on how to create the database tables for the models defined in your application. Let’s have another look at the directory tree:

bitcoin_tracker/
|
├── bitcoin_tracker/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
|
├── historical_data/
│   ├── migrations/
│   │   ├── 0001_initial.py
│   │   └── __init__.py
|   |
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
|
├── db.sqlite3
└── manage.py

As you can see, the migrations directory now contains a new file: 0001_initial.py.

Note: You might notice that running the makemigrations command also created the file db.sqlite3, which contains your SQLite database.

When you try to access a non-existing SQLite3 database file, it will automatically be created.

This behavior is unique to SQLite3. If you use any other database backend like PostgreSQL or MySQL, you have to create the database yourself before running makemigrations.

You can take a peek at the database with the dbshell management command. In SQLite, the command to list all tables is simply .tables:

$ python manage.py dbshell
SQLite version 3.19.3 2017-06-27 16:48:08
Enter ".help" for usage hints.
sqlite> .tables
sqlite>

The database is still empty. That will change when you apply the migration. Type .quit to exit the SQLite shell.

Applying Migrations

You have now created the migration, but to actually make any changes in the database, you have to apply it with the management command migrate:

$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, historical_data, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying historical_data.0001_initial... OK
  Applying sessions.0001_initial... OK

There is a lot going on here! According to the output, your migration has been successfully applied. But where do all the other migrations come from?

Remember the setting INSTALLED_APPS? Some of the other apps listed there also come with migrations, and the migrate management command applies the migrations for all installed apps by default.

Have another look at the database:

$ python manage.py dbshell
SQLite version 3.19.3 2017-06-27 16:48:08
Enter ".help" for usage hints.
sqlite> .tables
auth_group                    django_admin_log
auth_group_permissions        django_content_type
auth_permission               django_migrations
auth_user                     django_session
auth_user_groups              historical_data_pricehistory
auth_user_user_permissions
sqlite>

Now there are multiple tables. Their names give you an idea of their purpose. The migration that you generated in the previous step has created the historical_data_pricehistory table. Let’s inspect it using the .schema command:

sqlite> .schema --indent historical_data_pricehistory
CREATE TABLE IF NOT EXISTS "historical_data_pricehistory"(
  "id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
  "date" datetime NOT NULL,
  "price" decimal NOT NULL,
  "volume" integer unsigned NOT NULL
);

The .schema command prints out the CREATE statement that you would execute to create the table. The parameter --indent formats it nicely. Even if you are not familiar with SQL syntax, you can see that the schema of the historical_data_pricehistory table reflects the fields of the PriceHistory model.

There is a column for each field and an additional column id for the primary key, which Django creates automatically unless you explicitly specify a primary key in your model.

Here’s what happens if you run the migrate command again:

$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, historical_data, sessions
Running migrations:
  No migrations to apply.

Nothing! Django remembers which migrations have already been applied and does not try to rerun them.

It is worth noting that you can also limit the migrate management command to a single app:

$ python manage.py migrate historical_data
Operations to perform:
 Apply all migrations: historical_data
Running migrations:
 No migrations to apply.

As you can see, Django now only applies migrations for the historical_data app.

When you are running the migrations for the first time, it is a good idea to apply all migrations to ensure your database contains the necessary tables for the features you might take for granted, like user authentication and sessions.

Changing Models

Your models are not set in stone. Your models will change as your Django project gains more features. You might add or remove fields or change their types and options.

When you change the definition of a model, the database tables used to store these models have to be changed too. If your model definitions don’t match your current database schema, you will most likely run into a django.db.utils.OperationalError.

So how do you change the database tables? By creating and applying a migration.

While testing your Bitcoin tracker, you realize that you made a mistake. People are selling fractions of a Bitcoin, so the field volume should be of the type DecimalField instead of PositiveIntegerField.

Let’s change the model to look like this:

class PriceHistory(models.Model):
    date = models.DateTimeField(auto_now_add=True)
    price = models.DecimalField(max_digits=7, decimal_places=2)
    volume = models.DecimalField(max_digits=7, decimal_places=3)

Without migrations, you would have to figure out the SQL syntax to turn a PositiveIntegerField into a DecimalField. Luckily, Django will handle that for you. Just tell it to make migrations:

$ python manage.py makemigrations
Migrations for 'historical_data':
  historical_data/migrations/0002_auto_20181112_1950.py
    - Alter field volume on pricehistory

Note: The name of the migration file (0002_auto_20181112_1950.py) is based on the current time and will be different if you follow along on your system.

Now you apply this migration to your database:

$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, historical_data, sessions
Running migrations:
  Applying historical_data.0002_auto_20181112_1950... OK

The migration has been applied successfully, so you can use dbshell to verify that the changes had an effect:

$ python manage.py dbshell
SQLite version 3.19.3 2017-06-27 16:48:08
Enter ".help" for usage hints.
sqlite> .schema --indent historical_data_pricehistory
CREATE TABLE IF NOT EXISTS "historical_data_pricehistory" (
  "id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
  "date" datetime NOT NULL,
  "price" decimal NOT NULL,
  "volume" decimal NOT NULL
);

If you compare the new schema with the schema you saw earlier, you will notice that the type of the volume column has changed from integer to decimal to reflect the change of the volume field in the model from PositiveIntegerField to DecimalField.

Listing Out Migrations

If you want to know what migrations exist in a Django project, you don’t have to dig trough the migrations directories of your installed apps. You can use the showmigrations command:

$ ./manage.py showmigrations
admin
 [X] 0001_initial
 [X] 0002_logentry_remove_auto_add
 [X] 0003_logentry_add_action_flag_choices
auth
 [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [X] 0007_alter_validators_add_error_messages
 [X] 0008_alter_user_username_max_length
 [X] 0009_alter_user_last_name_max_length
contenttypes
 [X] 0001_initial
 [X] 0002_remove_content_type_name
historical_data
 [X] 0001_initial
 [X] 0002_auto_20181112_1950
sessions
 [X] 0001_initial

This lists all apps in the project and the migrations associated with each app. Also, it will put a big X next to the migrations that have already been applied.

For our little example, the showmigrations command is not particularly exciting, but it comes in handy when you start working on an existing code base or work in a team where you are not the only person adding migrations.

Unapplying Migrations

Now you know how to make changes to your database schema by creating and applying migrations. At some point, you might want to undo changes and switch back to an earlier database schema because you:

Want to test a migration a colleague wrote
Realize that a change you made was a bad idea
Work on multiple features with different database changes in parallel
Want to restore a backup that was created when the database still had an older schema

Luckily, migrations don’t have to be a one-way street. In many cases, the effects of a migration can be undone by unapplying a migration. To unapply a migration, you have to call migrate with the name of the app and the name of the migration before the migration you want to unapply.

If you want to revert the migration 0002_auto_20181112_1950 in your historical_data app, you have to pass 0001_initial as an argument to the migrate command:

$ python manage.py migrate historical_data 0001_initial
Operations to perform:
  Target specific migration: 0001_initial, from historical_data
Running migrations:
  Rendering model states... DONE
  Unapplying historical_data.0002_auto_20181112_1950... OK

The migration has been unapplied, meaning that the changes to the database have been reversed.

Unapplying a migration does not remove its migration file. The next time you run the migrate command, the migration will be applied again.

Caution: Don’t confuse unapplying migrations with the undo operation you are used to from your favorite text editor.

Not all database operations can be completely reverted. If you remove a field from a model, create a migration, and apply it, Django will remove the respective column from the database.

Unapplying that migration will re-create the column, but it won’t bring back the data that was stored in that column!

When you’re dealing with migration names, Django saves you a few keystrokes by not forcing you to spell out the whole name of the migration. It needs just enough of the name to identify it uniquely.

In the previous example, it would have been enough to run python manage.py migrate historical_data 0001.

Naming Migrations

In the above example, Django came up with a name for the migration based on the timestamp—something like *0002_auto_20181112_1950. If you’re not happy with that, then you can use the --name parameter to provide a custom name (without the .py extension).

To try that out, you first have to remove the old migration. You have already unapplied it, so you can safely delete the file:

$ rm historical_data/migrations/0002_auto_20181112_1950.py

Now you can recreate it with a more descriptive name:

$ ./manage.py makemigrations historical_data --name 0002_switch_to_decimals

This will create the same migration as before except with the new name of 0002_switch_to_decimals.

Conclusion

You covered quite a bit of ground in this tutorial and learned the fundamentals of Django migrations.

To recap, the basic steps to use Django migrations look like this:

Create or update a model
Run ./manage.py makemigrations <app_name>
Run ./manage.py migrate to migrate everything or ./manage.py migrate <app_name> to migrate an individual app
Repeat as necessary

That’s it! This workflow will work the majority of the time, but if things don’t work out as expected, you also know how to list and unapply migrations.

If you previously created and modified your database tables with hand-written SQL, you have now become much more efficient by delegating this work to Django migrations.

In the next tutorial in this series, you will dig deeper into the topic and learn how Django Migrations work under the hood.

Cheers!

Video

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Ultimate Guide to Python Type Checking

2019-01-07T14:00:00+00:00

In this guide, you will get a look into Python type checking. Traditionally, types have been handled by the Python interpreter in a flexible but implicit way. Recent versions of Python allow you to specify explicit type hints that can be used by different tools to help you develop your code more efficiently.

In this tutorial, you’ll learn about the following:

Type annotations and type hints
Adding static types to code, both your code and the code of others
Running a static type checker
Enforcing types at runtime

This is a comprehensive guide that will cover a lot of ground. If you want to just get a quick glimpse of how type hints work in Python, and see whether type checking is something you would include in your code, you don’t need to read all of it. The two sections Hello Types and Pros and Cons will give you a taste of how type checking works and recommendations about when it’ll be useful.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Type Systems

All programming languages include some kind of type system that formalizes which categories of objects it can work with and how those categories are treated. For instance, a type system can define a numerical type, with 42 as one example of an object of numerical type.

Dynamic Typing

Python is a dynamically typed language. This means that the Python interpreter does type checking only as code runs, and that the type of a variable is allowed to change over its lifetime. The following dummy examples demonstrate that Python has dynamic typing:

>>>

>>> if False:
...     1 + "two"  # This line never runs, so no TypeError is raised
... else:
...     1 + 2
...
3

>>> 1 + "two"  # Now this is type checked, and a TypeError is raised
TypeError: unsupported operand type(s) for +: 'int' and 'str'

In the first example, the branch 1 + "two" never runs so it’s never type checked. The second example shows that when 1 + "two" is evaluated it raises a TypeError since you can’t add an integer and a string in Python.

Next, let’s see if variables can change type:

>>>

>>> thing = "Hello"
>>> type(thing)
<class 'str'>

>>> thing = 28.1
>>> type(thing)
<class 'float'>

type() returns the type of an object. These examples confirm that the type of thing is allowed to change, and Python correctly infers the type as it changes.

Static Typing

The opposite of dynamic typing is static typing. Static type checks are performed without running the program. In most statically typed languages, for instance C and Java, this is done as your program is compiled.

With static typing, variables generally are not allowed to change types, although mechanisms for casting a variable to a different type may exist.

Let’s look at a quick example from a statically typed language. Consider the following Java snippet:

String thing;
thing = "Hello";

The first line declares that the variable name thing is bound to the String type at compile time. The name can never be rebound to another type. In the second line, thing is assigned a value. It can never be assigned a value that is not a String object. For instance, if you were to later say thing = 28.1f the compiler would raise an error because of incompatible types.

Python will always remain a dynamically typed language. However, PEP 484 introduced type hints, which make it possible to also do static type checking of Python code.

Unlike how types work in most other statically typed languages, type hints by themselves don’t cause Python to enforce types. As the name says, type hints just suggest types. There are other tools, which you’ll see later, that perform static type checking using type hints.

Duck Typing

Another term that is often used when talking about Python is duck typing. This moniker comes from the phrase “if it walks like a duck and it quacks like a duck, then it must be a duck” (or any of its variations).

Duck typing is a concept related to dynamic typing, where the type or the class of an object is less important than the methods it defines. Using duck typing you do not check types at all. Instead you check for the presence of a given method or attribute.

As an example, you can call len() on any Python object that defines a .__len__() method:

>>>

>>> class TheHobbit:
...     def __len__(self):
...         return 95022
...
>>> the_hobbit = TheHobbit()
>>> len(the_hobbit)
95022

Note that the call to len() gives the return value of the .__len__() method. In fact, the implementation of len() is essentially equivalent to the following:

def len(obj):
    return obj.__len__()

In order to call len(obj), the only real constraint on obj is that it must define a .__len__() method. Otherwise, the object can be of types as different as str, list, dict, or TheHobbit.

Duck typing is somewhat supported when doing static type checking of Python code, using structural subtyping. You’ll learn more about duck typing later.

Hello Types

In this section you’ll see how to add type hints to a function. The following function turns a text string into a headline by adding proper capitalization and a decorative line:

def headline(text, align=True):
    if align:
        return f"{text.title()}\n{'-' * len(text)}"
    else:
        return f" {text.title()} ".center(50, "o")

By default the function returns the headline left aligned with an underline. By setting the align flag to False you can alternatively have the headline be centered with a surrounding line of o:

>>>

>>> print(headline("python type checking"))
Python Type Checking
--------------------

>>> print(headline("python type checking", align=False))
oooooooooooooo Python Type Checking oooooooooooooo

It’s time for our first type hints! To add information about types to the function, you simply annotate its arguments and return value as follows:

def headline(text: str, align: bool = True) -> str:
    ...

The text: str syntax says that the text argument should be of type str. Similarly, the optional align argument should have type bool with the default value True. Finally, the -> str notation specifies that headline() will return a string.

In terms of style, PEP 8 recommends the following:

Use normal rules for colons, that is, no space before and one space after a colon: text: str.
Use spaces around the = sign when combining an argument annotation with a default value: align: bool = True.
Use spaces around the -> arrow: def headline(...) -> str.

Adding type hints like this has no runtime effect: they are only hints and are not enforced on their own. For instance, if we use a wrong type for the (admittedly badly named) align argument, the code still runs without any problems or warnings:

>>>

>>> print(headline("python type checking", align="left"))
Python Type Checking
--------------------

Note: The reason this seemingly works is that the string "left" compares as truthy. Using align="center" would not have the desired effect as "center" is also truthy.

To catch this kind of error you can use a static type checker. That is, a tool that checks the types of your code without actually running it in the traditional sense.

You might already have such a type checker built into your editor. For instance PyCharm immediately gives you a warning:

The most common tool for doing type checking is Mypy though. You’ll get a short introduction to Mypy in a moment, while you can learn much more about how it works later.

If you don’t already have Mypy on your system, you can install it using pip:

$ pip install mypy

Put the following code in a file called headlines.py:

 1 # headlines.py
 2 
 3 def headline(text: str, align: bool = True) -> str:
 4     if align:
 5         return f"{text.title()}\n{'-' * len(text)}"
 6     else:
 7         return f" {text.title()} ".center(50, "o")
 8 
 9 print(headline("python type checking"))
10 print(headline("use mypy", align="center"))

This is essentially the same code you saw earlier: the definition of headline() and two examples that are using it.

Now run Mypy on this code:

$ mypy headlines.py
headlines.py:10: error: Argument "align" to "headline" has incompatible
                        type "str"; expected "bool"

Based on the type hints, Mypy is able to tell us that we are using the wrong type on line 10.

To fix the issue in the code you should change the value of the align argument you are passing in. You might also rename the align flag to something less confusing:

 1 # headlines.py
 2 
 3 def headline(text: str, centered: bool = False):
 4     if not centered:
 5         return f"{text.title()}\n{'-' * len(text)}"
 6     else:
 7         return f" {text.title()} ".center(50, "o")
 8 
 9 print(headline("python type checking"))
10 print(headline("use mypy", centered=True))

Here you’ve changed align to centered, and correctly used a boolean value for centered when calling headline(). The code now passes Mypy:

$ mypy headlines.py
$

No output from Mypy means that no type errors were detected. Furthermore, when you run the code you see the expected output:

$ python headlines.py
Python Type Checking
--------------------
oooooooooooooooooooo Use Mypy oooooooooooooooooooo

The first headline is aligned to the left, while the second one is centered.

Pros and Cons

The previous section gave you a little taste of what type checking in Python looks like. You also saw an example of one of the advantages of adding types to your code: type hints help catch certain errors. Other advantages include:

Type hints help document your code. Traditionally, you would use docstrings if you wanted to document the expected types of a function’s arguments. This works, but as there is no standard for docstrings (despite PEP 257 they can’t be easily used for automatic checks.
Type hints improve IDEs and linters. They make it much easier to statically reason about your code. This in turn allows IDEs to offer better code completion and similar features. With the type annotation, PyCharm knows that text is a string, and can give specific suggestions based on this:
Type hints help you build and maintain a cleaner architecture. The act of writing type hints forces you to think about the types in your program. While the dynamic nature of Python is one of its great assets, being conscious about relying on duck typing, overloaded methods, or multiple return types is a good thing.

Of course, static type checking is not all peaches and cream. There are also some downsides you should consider:

Type hints take developer time and effort to add. Even though it probably pays off in spending less time debugging, you will spend more time entering code.
Type hints work best in modern Pythons. Annotations were introduced in Python 3.0, and it’s possible to use type comments in Python 2.7. Still, improvements like variable annotations and postponed evaluation of type hints mean that you’ll have a better experience doing type checks using Python 3.6 or even Python 3.7.
Type hints introduce a slight penalty in start-up time. If you need to use the typing module the import time may be significant, especially in short scripts.

You’ll later learn about the typing module, and how it’s necessary in most cases when you add type hints. Importing modules necessarily take some time, but how much?

To get some idea about this, create two files: empty_file.py should be an empty file, while import_typing.py should contain the following line:

import typing

On Linux it’s quite easy to check how much time the typing import takes using the perf utility:

$ perf stat -r 1000 python3.6 import_typing.py

 Performance counter stats for 'python3.6 import_typing.py' (1000 runs):

 [ ... extra information hidden for brevity ... ]

       0.045161650 seconds time elapsed    ( +-  0.77% )

So running the import typing.py script takes about 45 milliseconds. Of course this is not all time spent on importing typing. Some of this is overhead in starting the Python interpreter, so let’s compare to running Python on an empty file:

$ perf stat -r 1000 python3.6 empty_file.py

 Performance counter stats for 'python3.6 empty_file.py' (1000 runs):

 [ ... extra information hidden for brevity ... ]

       0.028077845 seconds time elapsed    ( +-  0.49% )

Based on this test, the import of the typing module takes around 17 milliseconds on Python 3.6.

One of the advertised improvements in Python 3.7 is faster startup. Let’s see if the results are different:

$ perf stat -r 1000 python3.7 import_typing.py
 [...]
       0.025979806 seconds time elapsed    ( +-  0.31% )

$ perf stat -r 1000 python3.7 empty_file.py
 [...]
       0.020002505 seconds time elapsed    ( +-  0.30% )

Indeed, the general startup time is reduced by about 8 milliseconds, and the time to import typing is down from 17 to around 6 milliseconds—almost 3 times faster.

Using timeit

There are similar tools on other platforms. Python itself comes with the timeit module in the standard library. Typically, we would directly use timeit for the timings above. However, timeit struggles to time imports reliably because Python is clever about importing modules only once. Consider the following example:

$ python3.6 -m timeit "import typing"
10000000 loops, best of 3: 0.134 usec per loop

While you get a result, you should be suspicious about the result: 0.1 microsecond is more than 100000 times faster than what perf measured! What timeit has actually done is to run the import typing statement 30 million times, with Python actually only importing typing once.

To get reasonable results you can tell timeit to only run once:

$ python3.6 -m timeit -n 1 -r 1 "import typing"
1 loops, best of 1: 9.77 msec per loop
$ python3.7 -m timeit -n 1 -r 1 "import typing"
1 loop, best of 1: 1.97 msec per loop

These results are on the same scale as the results from perf above. However, since these are based on only one execution of the code, they are not as reliable as those based on multiple runs.

The conclusion in both these cases is that importing typing takes a few milliseconds. For the majority of programs and scripts you write this will probably not be an issue.

The New importtime Option

In Python 3.7 there is also a new command line option that can be used to figure out how much time imports take. Using -X importtime you’ll get a report about all imports that are made:

$ python3.7 -X importtime import_typing.py
import time: self [us] | cumulative | imported package
[ ... some information hidden for brevity ... ]
import time:       358 |        358 | zipimport
import time:      2107 |      14610 | site
import time:       272 |        272 |   collections.abc
import time:       664 |       3058 |   re
import time:      3044 |       6373 | typing

This shows a similar result. Importing typing takes about 6 milliseconds. If you’ll read the report closely you can notice that around half of this time is spent on importing the collections.abc and re modules which typing depends on.

So, should you use static type checking in your own code? Well, it’s not an all-or-nothing question. Luckily, Python supports the concept of gradual typing. This means that you can gradually introduce types into your code. Code without type hints will be ignored by the static type checker. Therefore, you can start adding types to critical components, and continue as long as it adds value to you.

Looking at the lists above of pros and cons you’ll notice that adding types will have no effect on your running program or the users of your program. Type checking is meant to make your life as a developer better and more convenient.

A few rules of thumb on whether to add types to your project are:

If you are just beginning to learn Python, you can safely wait with type hints until you have more experience.
Type hints add little value in short throw-away scripts.
In libraries that will be used by others, especially ones published on PyPI, type hints add a lot of value. Other code using your libraries need these type hints to be properly type checked itself. For examples of projects using type hints see cursive_re, black, our own Real Python Reader, and Mypy itself.
In bigger projects, type hints help you understand how types flow through your code, and are highly recommended. Even more so in projects where you cooperate with others.

In his excellent article The State of Type Hints in Python Bernát Gábor recommends that “type hints should be used whenever unit tests are worth writing.” Indeed, type hints play a similar role as tests in your code: they help you as a developer write better code.

Hopefully you now have an idea about how type checking works in Python and whether it’s something you would like to employ in your own projects.

In the rest of this guide, we’ll go into more detail about the Python type system, including how you run static type checkers (with particular focus on Mypy), how you type check code that uses libraries without type hints, and how you use annotations at runtime.

Annotations

Annotations were introduced in Python 3.0, originally without any specific purpose. They were simply a way to associate arbitrary expressions to function arguments and return values.

Years later, PEP 484 defined how to add type hints to your Python code, based off work that Jukka Lehtosalo had done on his Ph.D. project—Mypy. The main way to add type hints is using annotations. As type checking is becoming more and more common, this also means that annotations should mainly be reserved for type hints.

The next sections explain how annotations work in the context of type hints.

Function Annotations

For functions, you can annotate arguments and the return value. This is done as follows:

def func(arg: arg_type, optarg: arg_type = default) -> return_type:
    ...

For arguments the syntax is argument: annotation, while the return type is annotated using -> annotation. Note that the annotation must be a valid Python expression.

The following simple example adds annotations to a function that calculates the circumference of a circle:

import math

def circumference(radius: float) -> float:
    return 2 * math.pi * radius

When running the code, you can also inspect the annotations. They are stored in a special .__annotations__ attribute on the function:

>>>

>>> circumference(1.23)
7.728317927830891

>>> circumference.__annotations__
{'radius': <class 'float'>, 'return': <class 'float'>}

Sometimes you might be confused by how Mypy is interpreting your type hints. For those cases there are special Mypy expressions: reveal_type() and reveal_locals(). You can add these to your code before running Mypy, and Mypy will dutifully report which types it has inferred. As an example, save the following code to reveal.py:

 1 # reveal.py
 2 
 3 import math
 4 reveal_type(math.pi)
 5 
 6 radius = 1
 7 circumference = 2 * math.pi * radius
 8 reveal_locals()

Next, run this code through Mypy:

$ mypy reveal.py
reveal.py:4: error: Revealed type is 'builtins.float'

reveal.py:8: error: Revealed local types are:
reveal.py:8: error: circumference: builtins.float
reveal.py:8: error: radius: builtins.int

Even without any annotations Mypy has correctly inferred the types of the built-in math.pi, as well as our local variables radius and circumference.

Note: The reveal expressions are only meant as a tool helping you add types and debug your type hints. If you try to run the reveal.py file as a Python script it will crash with a NameError since reveal_type() is not a function known to the Python interpreter.

If Mypy says that “Name ‘reveal_locals‘ is not defined” you might need to update your Mypy installation. The reveal_locals() expression is available in Mypy version 0.610 and later.

Variable Annotations

In the definition of circumference() in the previous section, you only annotated the arguments and the return value. You did not add any annotations inside the function body. More often than not, this is enough.

However, sometimes the type checker needs help in figuring out the types of variables as well. Variable annotations were defined in PEP 526 and introduced in Python 3.6. The syntax is the same as for function argument annotations:

pi: float = 3.142

def circumference(radius: float) -> float:
    return 2 * pi * radius

The variable pi has been annotated with the float type hint.

Note: Static type checkers are more than able to figure out that 3.142 is a float, so in this example the annotation of pi is not necessary. As you learn more about the Python type system, you’ll see more relevant examples of variable annotations.

Annotations of variables are stored in the module level __annotations__ dictionary:

>>>

>>> circumference(1)
6.284

>>> __annotations__
{'pi': <class 'float'>}

You’re allowed to annotate a variable without giving it a value. This adds the annotation to the __annotations__ dictionary, while the variable remains undefined:

>>>

>>> nothing: str
>>> nothing
NameError: name 'nothing' is not defined

>>> __annotations__
{'nothing': <class 'str'>}

Since no value was assigned to nothing, the name nothing is not yet defined.

Type Comments

As mentioned, annotations were introduced in Python 3, and they’ve not been backported to Python 2. This means that if you’re writing code that needs to support legacy Python, you can’t use annotations.

Instead, you can use type comments. These are specially formatted comments that can be used to add type hints compatible with older code. To add type comments to a function you do something like this:

import math

def circumference(radius):
    # type: (float) -> float
    return 2 * math.pi * radius

The type comments are just comments, so they can be used in any version of Python.

Type comments are handled directly by the type checker, so these types are not available in the __annotations__ dictionary:

>>>

>>> circumference.__annotations__
{}

A type comment must start with the type: literal, and be on the same or the following line as the function definition. If you want to annotate a function with several arguments, you write each type separated by comma:

def headline(text, width=80, fill_char="-"):
    # type: (str, int, str) -> str
    return f" {text.title()} ".center(width, fill_char)

print(headline("type comments work", width=40))

You are also allowed to write each argument on a separate line with its own annotation:

 1 # headlines.py
 2 
 3 def headline(
 4     text,           # type: str
 5     width=80,       # type: int
 6     fill_char="-",  # type: str
 7 ):                  # type: (...) -> str
 8     return f" {text.title()} ".center(width, fill_char)
 9 
10 print(headline("type comments work", width=40))

Run the example through Python and Mypy:

$  python headlines.py
---------- Type Comments Work ----------

$ mypy headline.py
$

If you have errors, for instance if you happened to call headline() with width="full" on line 10, Mypy will tell you:

$ mypy headline.py
headline.py:10: error: Argument "width" to "headline" has incompatible
                       type "str"; expected "int"

You can also add type comments to variables. This is done similarly to how you add type comments to arguments:

pi = 3.142  # type: float

In this example, pi will be type checked as a float variable.

So, Type Annotations or Type Comments?

Should you use annotations or type comments when adding type hints to your own code? In short: Use annotations if you can, use type comments if you must.

Annotations provide a cleaner syntax keeping type information closer to your code. They are also the officially recommended way of writing type hints, and will be further developed and properly maintained in the future.

Type comments are more verbose and might conflict with other kinds of comments in your code like linter directives. However, they can be used in code bases that don’t support annotations.

There is also hidden option number three: stub files. You will learn about these later, when we discuss adding types to third party libraries.

Stub files will work in any version of Python, at the expense of having to maintain a second set of files. In general, you only want to use stub files if you can’t change the original source code.

Playing With Python Types, Part 1

Up until now you’ve only used basic types like str, float, and bool in your type hints. The Python type system is quite powerful, and supports many kinds of more complex types. This is necessary as it needs to be able to reasonably model Python’s dynamic duck typing nature.

In this section you will learn more about this type system, while implementing a simple card game. You will see how to specify:

The type of sequences and mappings like tuples, lists and dictionaries
Type aliases that make code easier to read
That functions and methods do not return anything
Objects that may be of any type

After a short detour into some type theory you will then see even more ways to specify types in Python. You can find the code examples from this section here.

Example: A Deck of Cards

The following example shows an implementation of a regular (French) deck of cards:

 1 # game.py
 2 
 3 import random
 4 
 5 SUITS = "♠ ♡ ♢ ♣".split()
 6 RANKS = "2 3 4 5 6 7 8 9 10 J Q K A".split()
 7 
 8 def create_deck(shuffle=False):
 9     """Create a new deck of 52 cards"""
10     deck = [(s, r) for r in RANKS for s in SUITS]
11     if shuffle:
12         random.shuffle(deck)
13     return deck
14 
15 def deal_hands(deck):
16     """Deal the cards in the deck into four hands"""
17     return (deck[0::4], deck[1::4], deck[2::4], deck[3::4])
18 
19 def play():
20     """Play a 4-player card game"""
21     deck = create_deck(shuffle=True)
22     names = "P1 P2 P3 P4".split()
23     hands = {n: h for n, h in zip(names, deal_hands(deck))}
24 
25     for name, cards in hands.items():
26         card_str = " ".join(f"{s}{r}" for (s, r) in cards)
27         print(f"{name}: {card_str}")
28 
29 if __name__ == "__main__":
30     play()

Each card is represented as a tuple of strings denoting the suit and rank. The deck is represented as a list of cards. create_deck() creates a regular deck of 52 playing cards, and optionally shuffles the cards. deal_hands() deals the deck of cards to four players.

Finally, play() plays the game. As of now, it only prepares for a card game by constructing a shuffled deck and dealing cards to each player. The following is a typical output:

$ python game.py
P4: ♣9 ♢9 ♡2 ♢7 ♡7 ♣A ♠6 ♡K ♡5 ♢6 ♢3 ♣3 ♣Q
P1: ♡A ♠2 ♠10 ♢J ♣10 ♣4 ♠5 ♡Q ♢5 ♣6 ♠A ♣5 ♢4
P2: ♢2 ♠7 ♡8 ♢K ♠3 ♡3 ♣K ♠J ♢A ♣7 ♡6 ♡10 ♠K
P3: ♣2 ♣8 ♠8 ♣J ♢Q ♡9 ♡J ♠4 ♢8 ♢10 ♠9 ♡4 ♠Q

You will see how to extend this example into a more interesting game as we move along.

Sequences and Mappings

Let’s add type hints to our card game. In other words, let’s annotate the functions create_deck(), deal_hands(), and play(). The first challenge is that you need to annotate composite types like the list used to represent the deck of cards and the tuples used to represent the cards themselves.

With simple types like str, float, and bool, adding type hints is as easy as using the type itself:

>>>

>>> name: str = "Guido"
>>> pi: float = 3.142
>>> centered: bool = False

With composite types, you are allowed to do the same:

>>>

>>> names: list = ["Guido", "Jukka", "Ivan"]
>>> version: tuple = (3, 7, 1)
>>> options: dict = {"centered": False, "capitalize": True}

However, this does not really tell the full story. What will be the types of names[2], version[0], and options["centered"]? In this concrete case you can see that they are str, int, and bool, respectively. However, the type hints themselves give no information about this.

Instead, you should use the special types defined in the typing module. These types add syntax for specifying the types of elements of composite types. You can write the following:

>>>

>>> from typing import Dict, List, Tuple

>>> names: List[str] = ["Guido", "Jukka", "Ivan"]
>>> version: Tuple[int, int, int] = (3, 7, 1)
>>> options: Dict[str, bool] = {"centered": False, "capitalize": True}

Note that each of these types start with a capital letter and that they all use square brackets to define item types:

names is a list of strings
version is a 3-tuple consisting of three integers
options is a dictionary mapping strings to boolean values

The typing module contains many more composite types, including Counter, Deque, FrozenSet, NamedTuple, and Set. In addition, the module includes other kinds of types that you’ll see in later sections.

Let’s return to the card game. A card is represented by a tuple of two strings. You can write this as Tuple[str, str], so the type of the deck of cards becomes List[Tuple[str, str]]. Therefore you can annotate create_deck() as follows:

 8 def create_deck(shuffle: bool = False) -> List[Tuple[str, str]]:
 9     """Create a new deck of 52 cards"""
10     deck = [(s, r) for r in RANKS for s in SUITS]
11     if shuffle:
12         random.shuffle(deck)
13     return deck

In addition to the return value, you’ve also added the bool type to the optional shuffle argument.

Note: Tuples and lists are annotated differently.

A tuple is an immutable sequence, and typically consists of a fixed number of possibly differently typed elements. For example, we represent a card as a tuple of suit and rank. In general, you write Tuple[t_1, t_2, ..., t_n] for an n-tuple.

A list is a mutable sequence and usually consists of an unknown number of elements of the same type, for instance a list of cards. No matter how many elements are in the list there is only one type in the annotation: List[t].

In many cases your functions will expect some kind of sequence, and not really care whether it is a list or a tuple. In these cases you should use typing.Sequence when annotating the function argument:

from typing import List, Sequence

def square(elems: Sequence[float]) -> List[float]:
    return [x**2 for x in elems]

Using Sequence is an example of using duck typing. A Sequence is anything that supports len() and .__getitem__(), independent of its actual type.

Type Aliases

The type hints might become quite oblique when working with nested types like the deck of cards. You may need to stare at List[Tuple[str, str]] a bit before figuring out that it matches our representation of a deck of cards.

Now consider how you would annotate deal_hands():

15 def deal_hands(
16     deck: List[Tuple[str, str]]
17 ) -> Tuple[
18     List[Tuple[str, str]],
19     List[Tuple[str, str]],
20     List[Tuple[str, str]],
21     List[Tuple[str, str]],
22 ]:
23     """Deal the cards in the deck into four hands"""
24     return (deck[0::4], deck[1::4], deck[2::4], deck[3::4])

That’s just terrible!

Recall that type annotations are regular Python expressions. That means that you can define your own type aliases by assigning them to new variables. You can for instance create Card and Deck type aliases:

from typing import List, Tuple

Card = Tuple[str, str]
Deck = List[Card]

Card can now be used in type hints or in the definition of new type aliases, like Deck in the example above.

Using these aliases, the annotations of deal_hands() become much more readable:

15 def deal_hands(deck: Deck) -> Tuple[Deck, Deck, Deck, Deck]:
16     """Deal the cards in the deck into four hands"""
17     return (deck[0::4], deck[1::4], deck[2::4], deck[3::4])

Type aliases are great for making your code and its intent clearer. At the same time, these aliases can be inspected to see what they represent:

>>>

>>> from typing import List, Tuple
>>> Card = Tuple[str, str]
>>> Deck = List[Card]

>>> Deck
typing.List[typing.Tuple[str, str]]

Note that when printing Deck, it shows that it’s an alias for a list of 2-tuples of strings.

Functions Without Return Values

You may know that functions without an explicit return still return None:

>>>

>>> def play(player_name):
...     print(f"{player_name} plays")
...

>>> ret_val = play("Jacob")
Jacob plays

>>> print(ret_val)
None

While such functions technically return something, that return value is not useful. You should add type hints saying as much by using None also as the return type:

 1 # play.py
 2 
 3 def play(player_name: str) -> None:
 4     print(f"{player_name} plays")
 5 
 6 ret_val = play("Filip")

The annotations help catch the kinds of subtle bugs where you are trying to use a meaningless return value. Mypy will give you a helpful warning:

$ mypy play.py
play.py:6: error: "play" does not return a value

Note that being explicit about a function not returning anything is different from not adding a type hint about the return value:

# play.py

def play(player_name: str):
    print(f"{player_name} plays")

ret_val = play("Henrik")

In this latter case Mypy has no information about the return value so it will not generate any warning:

$ mypy play.py
$

As a more exotic case, note that you can also annotate functions that are never expected to return normally. This is done using NoReturn:

from typing import NoReturn

def black_hole() -> NoReturn:
    raise Exception("There is no going back ...")

Since black_hole() always raises an exception, it will never return properly.

Example: Play Some Cards

Let’s return to our card game example. In this second version of the game, we deal a hand of cards to each player as before. Then a start player is chosen and the players take turns playing their cards. There are not really any rules in the game though, so the players will just play random cards:

 1 # game.py
 2 
 3 import random
 4 from typing import List, Tuple
 5 
 6 SUITS = "♠ ♡ ♢ ♣".split()
 7 RANKS = "2 3 4 5 6 7 8 9 10 J Q K A".split()
 8 
 9 Card = Tuple[str, str]
10 Deck = List[Card]
11 
12 def create_deck(shuffle: bool = False) -> Deck:
13     """Create a new deck of 52 cards"""
14     deck = [(s, r) for r in RANKS for s in SUITS]
15     if shuffle:
16         random.shuffle(deck)
17     return deck
18 
19 def deal_hands(deck: Deck) -> Tuple[Deck, Deck, Deck, Deck]:
20     """Deal the cards in the deck into four hands"""
21     return (deck[0::4], deck[1::4], deck[2::4], deck[3::4])
22 
23 def choose(items):
24     """Choose and return a random item"""
25     return random.choice(items)
26 
27 def player_order(names, start=None):
28     """Rotate player order so that start goes first"""
29     if start is None:
30         start = choose(names)
31     start_idx = names.index(start)
32     return names[start_idx:] + names[:start_idx]
33 
34 def play() -> None:
35     """Play a 4-player card game"""
36     deck = create_deck(shuffle=True)
37     names = "P1 P2 P3 P4".split()
38     hands = {n: h for n, h in zip(names, deal_hands(deck))}
39     start_player = choose(names)
40     turn_order = player_order(names, start=start_player)
41 
42     # Randomly play cards from each player's hand until empty
43     while hands[start_player]:
44         for name in turn_order:
45             card = choose(hands[name])
46             hands[name].remove(card)
47             print(f"{name}: {card[0] + card[1]:<3}  ", end="")
48         print()
49 
50 if __name__ == "__main__":
51     play()

Note that in addition to changing play(), we have added two new functions that need type hints: choose() and player_order(). Before discussing how we’ll add type hints to them, here is an example output from running the game:

$ python game.py
P3: ♢10  P4: ♣4   P1: ♡8   P2: ♡Q
P3: ♣8   P4: ♠6   P1: ♠5   P2: ♡K
P3: ♢9   P4: ♡J   P1: ♣A   P2: ♡A
P3: ♠Q   P4: ♠3   P1: ♠7   P2: ♠A
P3: ♡4   P4: ♡6   P1: ♣2   P2: ♠K
P3: ♣K   P4: ♣7   P1: ♡7   P2: ♠2
P3: ♣10  P4: ♠4   P1: ♢5   P2: ♡3
P3: ♣Q   P4: ♢K   P1: ♣J   P2: ♡9
P3: ♢2   P4: ♢4   P1: ♠9   P2: ♠10
P3: ♢A   P4: ♡5   P1: ♠J   P2: ♢Q
P3: ♠8   P4: ♢7   P1: ♢3   P2: ♢J
P3: ♣3   P4: ♡10  P1: ♣9   P2: ♡2
P3: ♢6   P4: ♣6   P1: ♣5   P2: ♢8

In this example, player P3 was randomly chosen as the starting player. In turn, each player plays a card: first P3, then P4, then P1, and finally P2. The players keep playing cards as long as they have any left in their hand.

The `Any` Type

choose() works for both lists of names and lists of cards (and any other sequence for that matter). One way to add type hints for this would be the following:

import random
from typing import Any, Sequence

def choose(items: Sequence[Any]) -> Any:
    return random.choice(items)

This means more or less what it says: items is a sequence that can contain items of any type and choose() will return one such item of any type. Unfortunately, this is not that useful. Consider the following example:

 1 # choose.py
 2 
 3 import random
 4 from typing import Any, Sequence
 5 
 6 def choose(items: Sequence[Any]) -> Any:
 7     return random.choice(items)
 8 
 9 names = ["Guido", "Jukka", "Ivan"]
10 reveal_type(names)
11 
12 name = choose(names)
13 reveal_type(name)

While Mypy will correctly infer that names is a list of strings, that information is lost after the call to choose() because of the use of the Any type:

$ mypy choose.py
choose.py:10: error: Revealed type is 'builtins.list[builtins.str*]'
choose.py:13: error: Revealed type is 'Any'

You’ll see a better way shortly. First though, let’s have a more theoretical look at the Python type system, and the special role Any plays.

Type Theory

This tutorial is mainly a practical guide and we will only scratch the surface of the theory underpinning Python type hints. For more details PEP 483 is a good starting point. If you want to get back to the practical examples, feel free to skip to the next section.

Subtypes

One important concept is that of subtypes. Formally, we say that a type T is a subtype of U if the following two conditions hold:

Every value from T is also in the set of values of U type.
Every function from U type is also in the set of functions of T type.

These two conditions guarantees that even if type T is different from U, variables of type T can always pretend to be U.

For a concrete example, consider T = bool and U = int. The bool type takes only two values. Usually these are denoted True and False, but these names are just aliases for the integer values 1 and 0, respectively:

>>>

>>> int(False)
0

>>> int(True)
1

>>> True + True
2

>>> issubclass(bool, int)
True

Since 0 and 1 are both integers, the first condition holds. Above you can see that booleans can be added together, but they can also do anything else integers can. This is the second condition above. In other words, bool is a subtype of int.

The importance of subtypes is that a subtype can always pretend to be its supertype. For instance, the following code type checks as correct:

def double(number: int) -> int:
    return number * 2

print(double(True))  # Passing in bool instead of int

Subtypes are somewhat related to subclasses. In fact all subclasses corresponds to subtypes, and bool is a subtype of int because bool is a subclass of int. However, there are also subtypes that do not correspond to subclasses. For instance int is a subtype of float, but int is not a subclass of float.

Covariant, Contravariant, and Invariant

What happens when you use subtypes inside composite types? For instance, is Tuple[bool] a subtype of Tuple[int]? The answer depends on the composite type, and whether that type is covariant, contravariant, or invariant. This gets technical fast, so let’s just give a few examples:

Tuple is covariant. This means that it preserves the type hierarchy of its item types: Tuple[bool] is a subtype of Tuple[int] because bool is a subtype of int.
List is invariant. Invariant types give no guarantee about subtypes. While all values of List[bool] are values of List[int], you can append an int to List[int] and not to List[bool]. In other words, the second condition for subtypes does not hold, and List[bool] is not a subtype of List[int].
Callable is contravariant in its arguments. This means that it reverses the type hierarchy. You will see how Callable works later, but for now think of Callable[[T], ...] as a function with its only argument being of type T. An example of a Callable[[int], ...] is the double() function defined above. Being contravariant means that if a function operating on a bool is expected, then a function operating on an int would be acceptable.

In general, you don’t need to keep these expression straight. However, you should be aware that subtypes and composite types may not be simple and intuitive.

Gradual Typing and Consistent Types

Earlier we mentioned that Python supports gradual typing, where you can gradually add type hints to your Python code. Gradual typing is essentially made possible by the Any type.

Somehow Any sits both at the top and at the bottom of the type hierarchy of subtypes. Any type behaves as if it is a subtype of Any, and Any behaves as if it is a subtype of any other type. Looking at the definition of subtypes above this is not really possible. Instead we talk about consistent types.

The type T is consistent with the type U if T is a subtype of U or either T or U is Any.

The type checker only complains about inconsistent types. The takeaway is therefore that you will never see type errors arising from the Any type.

This means that you can use Any to explicitly fall back to dynamic typing, describe types that are too complex to describe in the Python type system, or describe items in composite types. For instance, a dictionary with string keys that can take any type as its values can be annotated Dict[str, Any].

Do remember, though, if you use Any the static type checker will effectively not do any type any checking.

Playing With Python Types, Part 2

Let’s return to our practical examples. Recall that you were trying to annotate the general choose() function:

import random
from typing import Any, Sequence

def choose(items: Sequence[Any]) -> Any:
    return random.choice(items)

The problem with using Any is that you are needlessly losing type information. You know that if you pass a list of strings to choose(), it will return a string. Below you’ll see how to express this using type variables, as well as how to work with:

Duck types and protocols
Arguments with None as default value
Class methods
The type of your own classes
Variable number of arguments

Type Variables

A type variable is a special variable that can take on any type, depending on the situation.

Let’s create a type variable that will effectively encapsulate the behavior of choose():

 1 # choose.py
 2 
 3 import random
 4 from typing import Sequence, TypeVar
 5 
 6 Choosable = TypeVar("Chooseable")
 7 
 8 def choose(items: Sequence[Choosable]) -> Choosable:
 9     return random.choice(items)
10 
11 names = ["Guido", "Jukka", "Ivan"]
12 reveal_type(names)
13 
14 name = choose(names)
15 reveal_type(name)

A type variable must be defined using TypeVar from the typing module. When used, a type variable ranges over all possible types and takes the most specific type possible. In the example, name is now a str:

$ mypy choose.py
choose.py:12: error: Revealed type is 'builtins.list[builtins.str*]'
choose.py:15: error: Revealed type is 'builtins.str*'

Consider a few other examples:

 1 # choose_examples.py
 2 
 3 from choose import choose
 4 
 5 reveal_type(choose(["Guido", "Jukka", "Ivan"]))
 6 reveal_type(choose([1, 2, 3]))
 7 reveal_type(choose([True, 42, 3.14]))
 8 reveal_type(choose(["Python", 3, 7])

The first two examples should have type str and int, but what about the last two? The individual list items have different types, and in that case the Choosable type variable does its best to accommodate:

$ mypy choose_examples.py
choose_examples.py:5: error: Revealed type is 'builtins.str*'
choose_examples.py:6: error: Revealed type is 'builtins.int*'
choose_examples.py:7: error: Revealed type is 'builtins.float*'
choose_examples.py:8: error: Revealed type is 'builtins.object*'

As you’ve already seen bool is a subtype of int, which again is a subtype of float. So in the third example the return value of choose() is guaranteed to be something that can be thought of as a float. In the last example, there is no subtype relationship between str and int, so the best that can be said about the return value is that it is an object.

Note that none of these examples raised a type error. Is there a way to tell the type checker that choose() should accept both strings and numbers, but not both at the same time?

You can constrain type variables by listing the acceptable types:

 1 # choose.py
 2 
 3 import random
 4 from typing import Sequence, TypeVar
 5 
 6 Choosable = TypeVar("Choosable", str, float)
 7 
 8 def choose(items: Sequence[Choosable]) -> Choosable:
 9     return random.choice(items)
10 
11 reveal_type(choose(["Guido", "Jukka", "Ivan"]))
12 reveal_type(choose([1, 2, 3]))
13 reveal_type(choose([True, 42, 3.14]))
14 reveal_type(choose(["Python", 3, 7]))

Now Choosable can only be either str or float, and Mypy will note that the last example is an error:

$ mypy choose.py
choose.py:11: error: Revealed type is 'builtins.str*'
choose.py:12: error: Revealed type is 'builtins.float*'
choose.py:13: error: Revealed type is 'builtins.float*'
choose.py:14: error: Revealed type is 'builtins.object*'
choose.py:14: error: Value of type variable "Choosable" of "choose"
                     cannot be "object"

Also note that in the second example the type is considered float even though the input list only contains int objects. This is because Choosable was restricted to strings and floats and int is a subtype of float.

In our card game we want to restrict choose() to be used for str and Card:

Choosable = TypeVar("Choosable", str, Card)

def choose(items: Sequence[Choosable]) -> Choosable:
    ...

We briefly mentioned that Sequence represents both lists and tuples. As we noted, a Sequence can be thought of as a duck type, since it can be any object with .__len__() and .__getitem__() implemented.

Duck Types and Protocols

Recall the following example from the introduction:

def len(obj):
    return obj.__len__()

len() can return the length of any object that has implemented the .__len__() method. How can we add type hints to len(), and in particular the obj argument?

The answer hides behind the academic sounding term structural subtyping. One way to categorize type systems is by whether they are nominal or structural:

In a nominal system, comparisons between types are based on names and declarations. The Python type system is mostly nominal, where an int can be used in place of a float because of their subtype relationship.
In a structural system, comparisons between types are based on structure. You could define a structural type Sized that includes all instances that define .__len__(), irrespective of their nominal type.

There is ongoing work to bring a full-fledged structural type system to Python through PEP 544 which aims at adding a concept called protocols. Most of PEP 544 is already implemented in Mypy though.

A protocol specifies one or more methods that must be implemented. For example, all classes defining .__len__() fulfill the typing.Sized protocol. We can therefore annotate len() as follows:

from typing import Sized

def len(obj: Sized) -> int:
    return obj.__len__()

Other examples of protocols defined in the typing module include Container, Iterable, Awaitable, and ContextManager.

You can also define your own protocols. This is done by inheriting from Protocol and defining the function signatures (with empty function bodies) that the protocol expects. The following example shows how len() and Sized could have been implemented:

from typing_extensions import Protocol

class Sized(Protocol):
    def __len__(self) -> int: ...

def len(obj: Sized) -> int:
    return obj.__len__()

At the time of writing the support for self-defined protocols is still experimental and only available through the typing_extensions module. This module must be explicitly installed from PyPI by doing pip install typing-extensions.

The `Optional` Type

A common pattern in Python is to use None as a default value for an argument. This is usually done either to avoid problems with mutable default values or to have a sentinel value flagging special behavior.

In the card example, the player_order() function uses None as a sentinel value for start saying that if no start player is given it should be chosen randomly:

27 def player_order(names, start=None):
28     """Rotate player order so that start goes first"""
29     if start is None:
30         start = choose(names)
31     start_idx = names.index(start)
32     return names[start_idx:] + names[:start_idx]

The challenge this creates for type hinting is that in general start should be a string. However, it may also take the special non-string value None.

In order to annotate such arguments you can use the Optional type:

from typing import Sequence, Optional

def player_order(
    names: Sequence[str], start: Optional[str] = None
) -> Sequence[str]:
    ...

The Optional type simply says that a variable either has the type specified or is None. An equivalent way of specifying the same would be using the Union type: Union[None, str]

Note that when using either Optional or Union you must take care that the variable has the correct type when you operate on it. This is done in the example by testing whether start is None. Not doing so would cause both static type errors as well as possible runtime errors:

 1 # player_order.py
 2 
 3 from typing import Sequence, Optional
 4 
 5 def player_order(
 6     names: Sequence[str], start: Optional[str] = None
 7 ) -> Sequence[str]:
 8     start_idx = names.index(start)
9     return names[start_idx:] + names[:start_idx]

Mypy tells you that you have not taken care of the case where start is None:

$ mypy player_order.py
player_order.py:8: error: Argument 1 to "index" of "list" has incompatible
                          type "Optional[str]"; expected "str"

Note: The use of None for optional arguments is so common that Mypy handles it automatically. Mypy assumes that a default argument of None indicates an optional argument even if the type hint does not explicitly say so. You could have used the following:

def player_order(names: Sequence[str], start: str = None) -> Sequence[str]:
    ...

If you don’t want Mypy to make this assumption you can turn it off with the --no-implicit-optional command line option.

Example: The Object(ive) of the Game

Let’s rewrite the card game to be more object-oriented. This will allow us to discuss how to properly annotate classes and methods.

A more or less direct translation of our card game into code that uses classes for Card, Deck, Player, and Game looks something like the following:

 1 # game.py
 2 
 3 import random
 4 import sys
 5 
 6 class Card:
 7     SUITS = "♠ ♡ ♢ ♣".split()
 8     RANKS = "2 3 4 5 6 7 8 9 10 J Q K A".split()
 9 
10     def __init__(self, suit, rank):
11         self.suit = suit
12         self.rank = rank
13 
14     def __repr__(self):
15         return f"{self.suit}{self.rank}"
16 
17 class Deck:
18     def __init__(self, cards):
19         self.cards = cards
20 
21     @classmethod
22     def create(cls, shuffle=False):
23         """Create a new deck of 52 cards"""
24         cards = [Card(s, r) for r in Card.RANKS for s in Card.SUITS]
25         if shuffle:
26             random.shuffle(cards)
27         return cls(cards)
28 
29     def deal(self, num_hands):
30         """Deal the cards in the deck into a number of hands"""
31         cls = self.__class__
32         return tuple(cls(self.cards[i::num_hands]) for i in range(num_hands))
33 
34 class Player:
35     def __init__(self, name, hand):
36         self.name = name
37         self.hand = hand
38 
39     def play_card(self):
40         """Play a card from the player's hand"""
41         card = random.choice(self.hand.cards)
42         self.hand.cards.remove(card)
43         print(f"{self.name}: {card!r:<3}  ", end="")
44         return card
45 
46 class Game:
47     def __init__(self, *names):
48         """Set up the deck and deal cards to 4 players"""
49         deck = Deck.create(shuffle=True)
50         self.names = (list(names) + "P1 P2 P3 P4".split())[:4]
51         self.hands = {
52             n: Player(n, h) for n, h in zip(self.names, deck.deal(4))
53         }
54 
55     def play(self):
56         """Play a card game"""
57         start_player = random.choice(self.names)
58         turn_order = self.player_order(start=start_player)
59 
60         # Play cards from each player's hand until empty
61         while self.hands[start_player].hand.cards:
62             for name in turn_order:
63                 self.hands[name].play_card()
64             print()
65 
66     def player_order(self, start=None):
67         """Rotate player order so that start goes first"""
68         if start is None:
69             start = random.choice(self.names)
70         start_idx = self.names.index(start)
71         return self.names[start_idx:] + self.names[:start_idx]
72 
73 if __name__ == "__main__":
74     # Read player names from command line
75     player_names = sys.argv[1:]
76     game = Game(*player_names)
77     game.play()

Now let’s add types to this code.

Type Hints for Methods

First of all type hints for methods work much the same as type hints for functions. The only difference is that the self argument need not be annotated, as it always will be a class instance. The types of the Card class are easy to add:

 6 class Card:
 7     SUITS = "♠ ♡ ♢ ♣".split()
 8     RANKS = "2 3 4 5 6 7 8 9 10 J Q K A".split()
 9 
10     def __init__(self, suit: str, rank: str) -> None:
11         self.suit = suit
12         self.rank = rank
13 
14     def __repr__(self) -> str:
15         return f"{self.suit}{self.rank}"

Note that the .__init__() method always should have None as its return type.

Classes as Types

There is a correspondence between classes and types. For example, all instances of the Card class together form the Card type. To use classes as types you simply use the name of the class.

For example, a Deck essentially consists of a list of Card objects. You can annotate this as follows:

17 class Deck:
18     def __init__(self, cards: List[Card]) -> None:
19         self.cards = cards

Mypy is able to connect your use of Card in the annotation with the definition of the Card class.

This doesn’t work as cleanly though when you need to refer to the class currently being defined. For example, the Deck.create() class method returns an object with type Deck. However, you can’t simply add -> Deck as the Deck class is not yet fully defined.

Instead, you are allowed to use string literals in annotations. These strings will only be evaluated by the type checker later, and can therefore contain self and forward references. The .create() method should use such string literals for its types:

20 class Deck:
21     @classmethod
22     def create(cls, shuffle: bool = False) -> "Deck":
23         """Create a new deck of 52 cards"""
24         cards = [Card(s, r) for r in Card.RANKS for s in Card.SUITS]
25         if shuffle:
26             random.shuffle(cards)
27         return cls(cards)

Note that the Player class also will reference the Deck class. This is however no problem, since Deck is defined before Player:

34 class Player:
35     def __init__(self, name: str, hand: Deck) -> None:
36         self.name = name
37         self.hand = hand

Usually annotations are not used at runtime. This has given wings to the idea of postponing the evaluation of annotations. Instead of evaluating annotations as Python expressions and storing their value, the proposal is to store the string representation of the annotation and only evaluate it when needed.

Such functionality is planned to become standard in the still mythical Python 4.0. However, in Python 3.7 and later, forward references are available through a __future__ import:

from __future__ import annotations

class Deck:
    @classmethod
    def create(cls, shuffle: bool = False) -> Deck:
        ...

With the __future__ import you can use Deck instead of "Deck" even before Deck is defined.

Returning `self` or `cls`

As noted, you should typically not annotate the self or cls arguments. Partly, this is not necessary as self points to an instance of the class, so it will have the type of the class. In the Card example, self has the implicit type Card. Also, adding this type explicitly would be cumbersome since the class is not defined yet. You would have to use the string literal syntax, self: "Card".

There is one case where you might want to annotate self or cls, though. Consider what happens if you have a superclass that other classes inherit from, and which has methods that return self or cls:

 1 # dogs.py
 2 
 3 from datetime import date
 4 
 5 class Animal:
 6     def __init__(self, name: str, birthday: date) -> None:
 7         self.name = name
 8         self.birthday = birthday
 9 
10     @classmethod
11     def newborn(cls, name: str) -> "Animal":
12         return cls(name, date.today())
13 
14     def twin(self, name: str) -> "Animal":
15         cls = self.__class__
16         return cls(name, self.birthday)
17 
18 class Dog(Animal):
19     def bark(self) -> None:
20         print(f"{self.name} says woof!")
21 
22 fido = Dog.newborn("Fido")
23 pluto = fido.twin("Pluto")
24 fido.bark()
25 pluto.bark()

While the code runs without problems, Mypy will flag a problem:

$ mypy dogs.py
dogs.py:24: error: "Animal" has no attribute "bark"
dogs.py:25: error: "Animal" has no attribute "bark"

The issue is that even though the inherited Dog.newborn() and Dog.twin() methods will return a Dog the annotation says that they return an Animal.

In cases like this you want to be more careful to make sure the annotation is correct. The return type should match the type of self or the instance type of cls. This can be done using type variables that keep track of what is actually passed to self and cls:

# dogs.py

from datetime import date
from typing import Type, TypeVar

TAnimal = TypeVar("TAnimal", bound="Animal")

class Animal:
    def __init__(self, name: str, birthday: date) -> None:
        self.name = name
        self.birthday = birthday

    @classmethod
    def newborn(cls: Type[TAnimal], name: str) -> TAnimal:
        return cls(name, date.today())

    def twin(self: TAnimal, name: str) -> TAnimal:
        cls = self.__class__
        return cls(name, self.birthday)

class Dog(Animal):
    def bark(self) -> None:
        print(f"{self.name} says woof!")

fido = Dog.newborn("Fido")
pluto = fido.twin("Pluto")
fido.bark()
pluto.bark()

There are a few things to note in this example:

The type variable TAnimal is used to denote that return values might be instances of subclasses of Animal.
We specify that Animal is an upper bound for TAnimal. Specifying bound means that TAnimal will only be Animal or one of its subclasses. This is needed to properly restrict the types that are allowed.
The typing.Type[] construct is the typing equivalent of type(). You need it to note that the class method expects a class and returns an instance of that class.

Annotating `*args` and `**kwargs`

In the object oriented version of the game, we added the option to name the players on the command line. This is done by listing player names after the name of the program:

$ python game.py GeirArne Dan Joanna
Dan: ♢A   Joanna: ♡9   P1: ♣A   GeirArne: ♣2
Dan: ♡A   Joanna: ♡6   P1: ♠4   GeirArne: ♢8
Dan: ♢K   Joanna: ♢Q   P1: ♣K   GeirArne: ♠5
Dan: ♡2   Joanna: ♡J   P1: ♠7   GeirArne: ♡K
Dan: ♢10  Joanna: ♣3   P1: ♢4   GeirArne: ♠8
Dan: ♣6   Joanna: ♡Q   P1: ♣Q   GeirArne: ♢J
Dan: ♢2   Joanna: ♡4   P1: ♣8   GeirArne: ♡7
Dan: ♡10  Joanna: ♢3   P1: ♡3   GeirArne: ♠2
Dan: ♠K   Joanna: ♣5   P1: ♣7   GeirArne: ♠J
Dan: ♠6   Joanna: ♢9   P1: ♣J   GeirArne: ♣10
Dan: ♠3   Joanna: ♡5   P1: ♣9   GeirArne: ♠Q
Dan: ♠A   Joanna: ♠9   P1: ♠10  GeirArne: ♡8
Dan: ♢6   Joanna: ♢5   P1: ♢7   GeirArne: ♣4

This is implemented by unpacking and passing in sys.argv to Game() when it’s instantiated. The .__init__() method uses *names to pack the given names into a tuple.

Regarding type annotations: even though names will be a tuple of strings, you should only annotate the type of each name. In other words, you should use str and not Tuple[str]:

46 class Game:
47     def __init__(self, *names: str) -> None:
48         """Set up the deck and deal cards to 4 players"""
49         deck = Deck.create(shuffle=True)
50         self.names = (list(names) + "P1 P2 P3 P4".split())[:4]
51         self.hands = {
52             n: Player(n, h) for n, h in zip(self.names, deck.deal(4))
53         }

Similarly, if you have a function or method accepting **kwargs you should only annotate the type of each possible keyword argument.

Callables

Functions are first-class objects in Python. This means that you can use functions as arguments to other functions. That also means that you need to be able to add type hints representing functions.

Functions, as well as lambdas, methods and classes, are represented by typing.Callable. The types of the arguments and the return value are usually also represented. For instance, Callable[[A1, A2, A3], Rt] represents a function with three arguments with types A1, A2, and A3, respectively. The return type of the function is Rt.

In the following example, the function do_twice() calls a given function twice and prints the return values:

 1 # do_twice.py
 2 
 3 from typing import Callable
 4 
 5 def do_twice(func: Callable[[str], str], argument: str) -> None:
 6     print(func(argument))
 7     print(func(argument))
 8 
 9 def create_greeting(name: str) -> str:
10     return f"Hello {name}"
11 
12 do_twice(create_greeting, "Jekyll")

Note the annotation of the func argument to do_twice() on line 5. It says that func should be a callable with one string argument, that also returns a string. One example of such a callable is create_greeting() defined on line 9.

Most callable types can be annotated in a similar manner. However, if you need more flexibility, check out callback protocols and extended callable types.

Example: Hearts

Let’s end with a full example of the game of Hearts. You might already know this game from other computer simulations. Here is a quick recap of the rules:

Four players play with a hand of 13 cards each.
The player holding the ♣2 starts the first round, and must play ♣2.
Players take turns playing cards, following the leading suit if possible.
The player playing the highest card in the leading suit wins the trick, and becomes start player in the next turn.
A player can not lead with a ♡ until a ♡ has already been played in an earlier trick.
After all cards are played, players get points if they take certain cards:
- 13 points for the ♠Q
- 1 point for each ♡
A game lasts several rounds, until one player has 100 points or more. The player with the least points wins.

More details can be found found online.

There are not many new typing concepts in this example that you have not already seen. We’ll therefore not go through this code in detail, but leave it as an example of annotated code.

You can download this code and other examples from GitHub:

# hearts.py

from collections import Counter
import random
import sys
from typing import Any, Dict, List, Optional, Sequence, Tuple, Union
from typing import overload

class Card:
    SUITS = "♠ ♡ ♢ ♣".split()
    RANKS = "2 3 4 5 6 7 8 9 10 J Q K A".split()

    def __init__(self, suit: str, rank: str) -> None:
        self.suit = suit
        self.rank = rank

    @property
    def value(self) -> int:
        """The value of a card is rank as a number"""
        return self.RANKS.index(self.rank)

    @property
    def points(self) -> int:
        """Points this card is worth"""
        if self.suit == "♠" and self.rank == "Q":
            return 13
        if self.suit == "♡":
            return 1
        return 0

    def __eq__(self, other: Any) -> Any:
        return self.suit == other.suit and self.rank == other.rank

    def __lt__(self, other: Any) -> Any:
        return self.value < other.value

    def __repr__(self) -> str:
        return f"{self.suit}{self.rank}"

class Deck(Sequence[Card]):
    def __init__(self, cards: List[Card]) -> None:
        self.cards = cards

    @classmethod
    def create(cls, shuffle: bool = False) -> "Deck":
        """Create a new deck of 52 cards"""
        cards = [Card(s, r) for r in Card.RANKS for s in Card.SUITS]
        if shuffle:
            random.shuffle(cards)
        return cls(cards)

    def play(self, card: Card) -> None:
        """Play one card by removing it from the deck"""
        self.cards.remove(card)

    def deal(self, num_hands: int) -> Tuple["Deck", ...]:
        """Deal the cards in the deck into a number of hands"""
        return tuple(self[i::num_hands] for i in range(num_hands))

    def add_cards(self, cards: List[Card]) -> None:
        """Add a list of cards to the deck"""
        self.cards += cards

    def __len__(self) -> int:
        return len(self.cards)

    @overload
    def __getitem__(self, key: int) -> Card: ...

    @overload
    def __getitem__(self, key: slice) -> "Deck": ...

    def __getitem__(self, key: Union[int, slice]) -> Union[Card, "Deck"]:
        if isinstance(key, int):
            return self.cards[key]
        elif isinstance(key, slice):
            cls = self.__class__
            return cls(self.cards[key])
        else:
            raise TypeError("Indices must be integers or slices")

    def __repr__(self) -> str:
        return " ".join(repr(c) for c in self.cards)

class Player:
    def __init__(self, name: str, hand: Optional[Deck] = None) -> None:
        self.name = name
        self.hand = Deck([]) if hand is None else hand

    def playable_cards(self, played: List[Card], hearts_broken: bool) -> Deck:
        """List which cards in hand are playable this round"""
        if Card("♣", "2") in self.hand:
            return Deck([Card("♣", "2")])

        lead = played[0].suit if played else None
        playable = Deck([c for c in self.hand if c.suit == lead]) or self.hand
        if lead is None and not hearts_broken:
            playable = Deck([c for c in playable if c.suit != "♡"])
        return playable or Deck(self.hand.cards)

    def non_winning_cards(self, played: List[Card], playable: Deck) -> Deck:
        """List playable cards that are guaranteed to not win the trick"""
        if not played:
            return Deck([])

        lead = played[0].suit
        best_card = max(c for c in played if c.suit == lead)
        return Deck([c for c in playable if c < best_card or c.suit != lead])

    def play_card(self, played: List[Card], hearts_broken: bool) -> Card:
        """Play a card from a cpu player's hand"""
        playable = self.playable_cards(played, hearts_broken)
        non_winning = self.non_winning_cards(played, playable)

        # Strategy
        if non_winning:
            # Highest card not winning the trick, prefer points
            card = max(non_winning, key=lambda c: (c.points, c.value))
        elif len(played) < 3:
            # Lowest card maybe winning, avoid points
            card = min(playable, key=lambda c: (c.points, c.value))
        else:
            # Highest card guaranteed winning, avoid points
            card = max(playable, key=lambda c: (-c.points, c.value))
        self.hand.cards.remove(card)
        print(f"{self.name} -> {card}")
        return card

    def has_card(self, card: Card) -> bool:
        return card in self.hand

    def __repr__(self) -> str:
        return f"{self.__class__.__name__}({self.name!r}, {self.hand})"

class HumanPlayer(Player):
    def play_card(self, played: List[Card], hearts_broken: bool) -> Card:
        """Play a card from a human player's hand"""
        playable = sorted(self.playable_cards(played, hearts_broken))
        p_str = "  ".join(f"{n}: {c}" for n, c in enumerate(playable))
        np_str = " ".join(repr(c) for c in self.hand if c not in playable)
        print(f"  {p_str}  (Rest: {np_str})")
        while True:
            try:
                card_num = int(input(f"  {self.name}, choose card: "))
                card = playable[card_num]
            except (ValueError, IndexError):
                pass
            else:
                break
        self.hand.play(card)
        print(f"{self.name} => {card}")
        return card

class HeartsGame:
    def __init__(self, *names: str) -> None:
        self.names = (list(names) + "P1 P2 P3 P4".split())[:4]
        self.players = [Player(n) for n in self.names[1:]]
        self.players.append(HumanPlayer(self.names[0]))

    def play(self) -> None:
        """Play a game of Hearts until one player go bust"""
        score = Counter({n: 0 for n in self.names})
        while all(s < 100 for s in score.values()):
            print("\nStarting new round:")
            round_score = self.play_round()
            score.update(Counter(round_score))
            print("Scores:")
            for name, total_score in score.most_common(4):
                print(f"{name:<15} {round_score[name]:>3} {total_score:>3}")

        winners = [n for n in self.names if score[n] == min(score.values())]
        print(f"\n{' and '.join(winners)} won the game")

    def play_round(self) -> Dict[str, int]:
        """Play a round of the Hearts card game"""
        deck = Deck.create(shuffle=True)
        for player, hand in zip(self.players, deck.deal(4)):
            player.hand.add_cards(hand.cards)
        start_player = next(
            p for p in self.players if p.has_card(Card("♣", "2"))
        )
        tricks = {p.name: Deck([]) for p in self.players}
        hearts = False

        # Play cards from each player's hand until empty
        while start_player.hand:
            played: List[Card] = []
            turn_order = self.player_order(start=start_player)
            for player in turn_order:
                card = player.play_card(played, hearts_broken=hearts)
                played.append(card)
            start_player = self.trick_winner(played, turn_order)
            tricks[start_player.name].add_cards(played)
            print(f"{start_player.name} wins the trick\n")
            hearts = hearts or any(c.suit == "♡" for c in played)
        return self.count_points(tricks)

    def player_order(self, start: Optional[Player] = None) -> List[Player]:
        """Rotate player order so that start goes first"""
        if start is None:
            start = random.choice(self.players)
        start_idx = self.players.index(start)
        return self.players[start_idx:] + self.players[:start_idx]

    @staticmethod
    def trick_winner(trick: List[Card], players: List[Player]) -> Player:
        lead = trick[0].suit
        valid = [
            (c.value, p) for c, p in zip(trick, players) if c.suit == lead
        ]
        return max(valid)[1]

    @staticmethod
    def count_points(tricks: Dict[str, Deck]) -> Dict[str, int]:
        return {n: sum(c.points for c in cards) for n, cards in tricks.items()}

if __name__ == "__main__":
    # Read player names from the command line
    player_names = sys.argv[1:]
    game = HeartsGame(*player_names)
    game.play()

Here are a few points to note in the code:

For type relationships that are hard to express using Union or type variables, you can use the @overload decorator. See Deck.__getitem__() for an example and the documentation for more information.
Subclasses correspond to subtypes, so that a HumanPlayer can be used wherever a Player is expected.
When a subclass reimplements a method from a superclass, the type annotations must match. See HumanPlayer.play_card() for an example.

When starting the game, you control the first player. Enter numbers to choose which cards to play. The following is an example of game play, with the highlighted lines showing where the player made a choice:

$ python hearts.py GeirArne Aldren Joanna Brad

Starting new round:
Brad -> ♣2
  0: ♣5  1: ♣Q  2: ♣K  (Rest: ♢6 ♡10 ♡6 ♠J ♡3 ♡9 ♢10 ♠7 ♠K ♠4)
  GeirArne, choose card: 2
GeirArne => ♣K
Aldren -> ♣10
Joanna -> ♣9
GeirArne wins the trick

  0: ♠4  1: ♣5  2: ♢6  3: ♠7  4: ♢10  5: ♠J  6: ♣Q  7: ♠K  (Rest: ♡10 ♡6 ♡3 ♡9)
  GeirArne, choose card: 0
GeirArne => ♠4
Aldren -> ♠5
Joanna -> ♠3
Brad -> ♠2
Aldren wins the trick

...

Joanna -> ♡J
Brad -> ♡2
  0: ♡6  1: ♡9  (Rest: )
  GeirArne, choose card: 1
GeirArne => ♡9
Aldren -> ♡A
Aldren wins the trick

Aldren -> ♣A
Joanna -> ♡Q
Brad -> ♣J
  0: ♡6  (Rest: )
  GeirArne, choose card: 0
GeirArne => ♡6
Aldren wins the trick

Scores:
Brad             14  14
Aldren           10  10
GeirArne          1   1
Joanna            1   1

Static Type Checking

So far you have seen how to add type hints to your code. In this section you’ll learn more about how to actually perform static type checking of Python code.

The Mypy Project

Mypy was started by Jukka Lehtosalo during his Ph.D. studies at Cambridge around 2012. Mypy was originally envisioned as a Python variant with seamless dynamic and static typing. See Jukka’s slides from PyCon Finland 2012 for examples of the original vision of Mypy.

Most of those original ideas still play a big part in the Mypy project. In fact, the slogan “Seamless dynamic and static typing” is still prominently visible on Mypy’s home page and describes the motivation for using type hints in Python well.

The biggest change since 2012 is that Mypy is no longer a variant of Python. In its first versions Mypy was a stand-alone language that was compatible with Python except for its type declarations. Following a suggestion by Guido van Rossum, Mypy was rewritten to use annotations instead. Today Mypy is a static type checker for regular Python code.

Running Mypy

Before running Mypy for the first time, you must install the program. This is most easily done using pip:

$ pip install mypy

With Mypy installed, you can run it as a regular command line program:

$ mypy my_program.py

Running Mypy on your my_program.py Python file will check it for type errors without actually executing the code.

There are many available options when type checking your code. As Mypy is still under very active development, command line options are liable to change between versions. You should refer to Mypy’s help to see which settings are default on your version:

$ mypy --help
usage: mypy [-h] [-v] [-V] [more options; see below]
            [-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...]

Mypy is a program that will type check your Python code.

[... The rest of the help hidden for brevity ...]

Additionally, the Mypy command line documentation online has a lot of information.

Let’s look at some of the most common options. First of all, if you are using third-party packages without type hints, you may want to silence Mypy’s warnings about these. This can be done with the --ignore-missing-imports option.

The following example uses Numpy to calculate and print the cosine of several numbers:

 1 # cosine.py
 2 
 3 import numpy as np
 4 
 5 def print_cosine(x: np.ndarray) -> None:
 6     with np.printoptions(precision=3, suppress=True):
 7         print(np.cos(x))
 8 
 9 x = np.linspace(0, 2 * np.pi, 9)
10 print_cosine(x)

Note that np.printoptions() is only available in version 1.15 and later of Numpy. Running this example prints some numbers to the console:

$ python cosine.py
[ 1.     0.707  0.    -0.707 -1.    -0.707 -0.     0.707  1.   ]

The actual output of this example is not important. However, you should note that the argument x is annotated with np.ndarray on line 5, as we want to print the cosine of a full array of numbers.

You can run Mypy on this file as usual:

$ mypy cosine.py 
cosine.py:3: error: No library stub file for module 'numpy'
cosine.py:3: note: (Stub files are from https://github.com/python/typeshed)

These warnings may not immediately make much sense to you, but you’ll learn about stubs and typeshed soon. You can essentially read the warnings as Mypy saying that the Numpy package does not contain type hints.

In most cases, missing type hints in third-party packages is not something you want to be bothered with so you can silence these messages:

$ mypy --ignore-missing-imports cosine.py 
$

If you use the --ignore-missing-import command line option, Mypy will not try to follow or warn about any missing imports. This might be a bit heavy-handed though, as it also ignores actual mistakes, like misspelling the name of a package.

Two less intrusive ways of handling third-party packages are using type comments or configuration files.

In a simple example as the one above, you can silence the numpy warning by adding a type comment to the line containing the import:

 3 import numpy as np  # type: ignore

The literal # type: ignore tells Mypy to ignore the import of Numpy.

If you have several files, it might be easier to keep track of which imports to ignore in a configuration file. Mypy reads a file called mypy.ini in the current directory if it is present. This configuration file must contain a section called [mypy] and may contain module specific sections of the form [mypy-module].

The following configuration file will ignore that Numpy is missing type hints:

# mypy.ini

[mypy]

[mypy-numpy]
ignore_missing_imports = True

There are many options that can be specified in the configuration file. It is also possible to specify a global configuration file. See the documentation for more information.

Adding Stubs

Type hints are available for all the packages in the Python standard library. However, if you are using third-party packages you’ve already seen that the situation can be different.

The following example uses the Parse package to do simple text parsing. To follow along you should first install Parse:

$ pip install parse

Parse can be used to recognize simple patterns. Here is a small program that tries its best to figure out your name:

 1 # parse_name.py
 2 
 3 import parse
 4 
 5 def parse_name(text: str) -> str:
 6     patterns = (
 7         "my name is {name}",
 8         "i'm {name}",
 9         "i am {name}",
10         "call me {name}",
11         "{name}",
12     )
13     for pattern in patterns:
14         result = parse.parse(pattern, text)
15         if result:
16             return result["name"]
17     return ""
18 
19 answer = input("What is your name? ")
20 name = parse_name(answer)
21 print(f"Hi {name}, nice to meet you!")

The main flow is defined in the last three lines: ask for your name, parse the answer, and print a greeting. The parse package is called on line 14 in order to try to find a name based on one of the patterns listed on lines 7-11.

The program can be used as follows:

$ python parse_name.py
What is your name? I am Geir Arne
Hi Geir Arne, nice to meet you!

Note that even though I answer I am Geir Arne, the program figures out that I am is not part of my name.

Let’s add a small bug to the program, and see if Mypy is able to help us detect it. Change line 16 from return result["name"] to return result. This will return a parse.Result object instead of the string containing the name.

Next run Mypy on the program:

$ mypy parse_name.py 
parse_name.py:3: error: Cannot find module named 'parse'
parse_name.py:3: note: (Perhaps setting MYPYPATH or using the
                       "--ignore-missing-imports" flag would help)

Mypy prints a similar error to the one you saw in the previous section: It doesn’t know about the parse package. You could try to ignore the import:

$ mypy parse_name.py --ignore-missing-imports
$

Unfortunately, ignoring the import means that Mypy has no way of discovering the bug in our program. A better solution would be to add type hints to the Parse package itself. As Parse is open source you can actually add types to the source code and send a pull request.

Alternatively, you can add the types in a stub file. A stub file is a text file that contains the signatures of methods and functions, but not their implementations. Their main function is to add type hints to code that you for some reason can’t change. To show how this works, we will add some stubs for the Parse package.

First of all, you should put all your stub files inside one common directory, and set the MYPYPATH environment variable to point to this directory. On Mac and Linux you can set MYPYPATH as follows:

$ export MYPYPATH=/home/gahjelle/python/stubs

You can set the variable permanently by adding the line to your .bashrc file. On Windows you can click the start menu and search for environment variables to set MYPYPATH.

Next, create a file inside your stubs directory that you call parse.pyi. It must be named for the package that you are adding type hints for, with a .pyi suffix. Leave this file empty for now. Then run Mypy again:

$ mypy parse_name.py
parse_name.py:14: error: Module has no attribute "parse"

If you have set everything up correctly, you should see this new error message. Mypy uses the new parse.pyi file to figure out which functions are available in the parse package. Since the stub file is empty, Mypy assumes that parse.parse() does not exist, and then gives the error you see above.

The following example does not add types for the whole parse package. Instead it shows the type hints you need to add in order for Mypy to type check your use of parse.parse():

# parse.pyi

from typing import Any, Mapping, Optional, Sequence, Tuple, Union

class Result:
    def __init__(
        self,
        fixed: Sequence[str],
        named: Mapping[str, str],
        spans: Mapping[int, Tuple[int, int]],
    ) -> None: ...
    def __getitem__(self, item: Union[int, str]) -> str: ...
    def __repr__(self) -> str: ...

def parse(
    format: str,
    string: str,
    evaluate_result: bool = ...,
    case_sensitive: bool = ...,
) -> Optional[Result]: ...

The ellipsis ... are part of the file, and should be written exactly as above. The stub file should only contain type hints for variables, attributes, functions, and methods, so the implementations should be left out and replaced by ... markers.

Finally Mypy is able to spot the bug we introduced:

$ mypy parse_name.py
parse_name.py:16: error: Incompatible return value type (got
                         "Result", expected "str")

This points straight to line 16 and the fact that we return a Result object and not the name string. Change return result back to return result["name"], and run Mypy again to see that it’s happy.

Typeshed

You’ve seen how to use stubs to add type hints without changing the source code itself. In the previous section we added some type hints to the third-party Parse package. Now, it wouldn’t be very effective if everybody needs to create their own stubs files for all third-party packages they are using.

Typeshed is a Github repository that contains type hints for the Python standard library, as well as many third-party packages. Typeshed comes included with Mypy so if you are using a package that already has type hints defined in Typeshed, the type checking will just work.

You can also contribute type hints to Typeshed. Make sure to get the permission of the owner of the package first though, especially because they might be working on adding type hints into the source code itself—which is the preferred approach.

Other Static Type Checkers

In this tutorial, we have mainly focused on type checking using Mypy. However, there are other static type checkers in the Python ecosystem.

The PyCharm editor comes with its own type checker included. If you are using PyCharm to write your Python code, it will be automatically type checked.

Facebook has developed Pyre. One of its stated goals is to be fast and performant. While there are some differences, Pyre functions mostly similar to Mypy. See the documentation if you’re interested in trying out Pyre.

Furthermore, Google has created Pytype. This type checker also works mostly the same as Mypy. In addition to checking annotated code, Pytype has some support for running type checks on unannotated code and even adding annotations to code automatically. See the quickstart document for more information.

Using Types at Runtime

As a final note, it’s possible to use type hints also at runtime during execution of your Python program. Runtime type checking will probably never be natively supported in Python.

However, the type hints are available at runtime in the __annotations__ dictionary, and you can use those to do type checks if you desire. Before you run off and write your own package for enforcing types, you should know that there are already several packages doing this for you. Have a look at Enforce, Pydantic, or Pytypes for some examples.

Another use of type hints is for translating your Python code to C and compiling it for optimization. The popular Cython project uses a hybrid C/Python language to write statically typed Python code. However, since version 0.27 Cython has also supported type annotations. Recently, the Mypyc project has become available. While not yet ready for general use, it can compile some type annotated Python code to C extensions.

Conclusion

Type hinting in Python is a very useful feature that you can happily live without. Type hints don’t make you capable of writing any code you can’t write without using type hints. Instead, using type hints makes it easier for you to reason about code, find subtle bugs, and maintain a clean architecture.

In this tutorial you have learned how type hinting works in Python, and how gradual typing make type checks in Python more flexible than in many other languages. You’ve seen some of the pros and cons of using type hints, and how they can be added to code using annotations or type comments. Finally you saw many of the different types that Python supports, as well as how to perform static type checking.

There are many resources to learn more about static type checking in Python. PEP 483 and PEP 484 give a lot of background about how type checking is implemented in Python. The Mypy documentation has a great reference section detailing all the different types available.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Modeling Polymorphism in Django With Python

2019-01-02T14:00:00+00:00

Modeling polymorphism in relational databases is a challenging task. In this article, we present several modeling techniques to represent polymorphic objects in a relational database using the Django object-relational mapping (ORM).

This intermediate-level tutorial is designed for readers who are already familiar with the fundamental design of Django.

Free Bonus: Click here to get the most popular Django tutorials and resources on Real Python and improve your Django + Python web development skills.

What Is Polymorphism?

Polymorphism is the ability of an object to take on many forms. Common examples of polymorphic objects include event streams, different types of users, and products in an e-commerce website. A polymorphic model is used when a single entity requires different functionality or information.

In the examples above, all events are logged for future use, but they can contain different data. All users need be able to log in, but they might have different profile structures. In every e-commerce website, a user wants to put different products in their shopping cart.

Why Is Modeling Polymorphism Challenging?

There are many ways to model polymorphism. Some approaches use standard features of the Django ORM, and some use special features of the Django ORM. The main challenges you’ll encounter when modeling polymorphic objects are the following:

How to represent a single polymorphic object: Polymorphic objects have different attributes. The Django ORM maps attributes to columns in the database. In that case, how should the Django ORM map attributes to the columns in the table? Should different objects reside in the same table? Should you have multiple tables?
How to reference instances of a polymorphic model: To utilize database and Django ORM features, you need to reference objects using foreign keys. How you decide to represent a single polymorphic object is crucial to your ability to reference it.

To truly understand the challenges of modeling polymorphism, you are going to take a small bookstore from its first online website to a big online shop selling all sorts of products. Along the way, you’ll experience and analyze different approaches for modeling polymorphism using the Django ORM.

Note: To follow this tutorial, it is recommended that you use a PostgreSQL backend, Django 2.x, and Python 3.

It’s possible to follow along with other database backends as well. In places where features unique to PostgreSQL are used, an alternative will be presented for other databases.

Naive Implementation

You have a bookstore in a nice part of town right next to a coffee shop, and you want to start selling books online.

You sell only one type of product: books. In your online store, you want to show details about the books, like name and price. You want your users to browse around the website and collect many books, so you also need a cart. You eventually need to ship the books to the user, so you need to know the weight of each book to calculate the delivery fee.

Let’s create a simple model for your new book store:

from django.contrib.auth import get_user_model
from django.db import models


class Book(models.Model):
    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )
    weight = models.PositiveIntegerField(
        help_text='in grams',
    )

    def __str__(self) -> str:
        return self.name


class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    books = models.ManyToManyField(Book)

To create a new book, you provide a name, price, and weight:

>>>

>>> from naive.models import Book
>>> book = Book.objects.create(name='Python Tricks', price=1000, weight=200)
>>> book
<Product: Python Tricks>

To create a cart, you first need to associate it with a user:

>>>

>>> from django.contrib.auth import get_user_model
>>> haki = get_user_model().create_user('haki')

>>> from naive.models import Cart
>>> cart = Cart.objects.create(user=haki)

Then the user can start adding items to it:

>>>

>>> cart.products.add(book)
>>> cart.products.all()
<QuerySet [<Book: Python Tricks>]>

Pro

Easy to understand and maintain: It’s sufficient for a single type of product.

Con

Restricted to homogeneous products: It only supports products with the same set of attributes. Polymorphism is not captured or permitted at all.

Sparse Model

With the success of your online bookstore, users started to ask if you also sell e-books. E-books are a great product for your online store, and you want to start selling them right away.

A physical book is different from an e-book:

An e-book has no weight. It’s a virtual product.
An e-book does not require shipment. Users download it from the website.

To make your existing model support the additional information for selling e-books, you add some fields to the existing Book model:

from django.contrib.auth import get_user_model
from django.db import models


class Book(models.Model):
    TYPE_PHYSICAL = 'physical'
    TYPE_VIRTUAL = 'virtual'
    TYPE_CHOICES = (
        (TYPE_PHYSICAL, 'Physical'),
        (TYPE_VIRTUAL, 'Virtual'),
    )
    type = models.CharField(
        max_length=20,
        choices=TYPE_CHOICES,
    )

    # Common attributes
    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )

    # Specific attributes
    weight = models.PositiveIntegerField(
        help_text='in grams',
    )
    download_link = models.URLField(
        null=True, blank=True,
    )

    def __str__(self) -> str:
        return f'[{self.get_type_display()}] {self.name}'


class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    books = models.ManyToManyField(
        Book,
    )

First, you added a type field to indicate what type of book it is. Then, you added a URL field to store the download link of the e-book.

To add a physical book to your bookstore, do the following:

>>>

>>> from sparse.models import Book
>>> physical_book = Book.objects.create(
...     type=Book.TYPE_PHYSICAL,
...     name='Python Tricks',
...     price=1000,
...     weight=200,
...     download_link=None,
... )
>>> physical_book
<Book: [Physical] Python Tricks>

To add a new e-book, you do the following:

>>>

>>> virtual_book = Book.objects.create(
...     type=Book.TYPE_VIRTUAL,
...     name='The Old Man and the Sea',
...     price=1500,
...     weight=0,
...     download_link='https://books.com/12345',
... )
>>> virtual_book
<Book: [Virtual] The Old Man and the Sea>

Your users can now add both books and e-books to the cart:

>>>

>>> from sparse.models import Cart
>>> cart = Cart.objects.create(user=user)
>>> cart.books.add(physical_book, virtual_book)
>>> cart.books.all()
<QuerySet [<Book: [Physical] Python Tricks>, <Book: [Virtual] The Old Man and the Sea>]>

The virtual books are a big hit, and you decide to hire employees. The new employees are apparently not so tech savvy, and you start seeing weird things in the database:

>>>

>>> Book.objects.create(
...     type=Book.TYPE_PHYSICAL,
...     name='Python Tricks',
...     price=1000,
...     weight=0,
...     download_link='http://books.com/54321',
... )

That book apparently weighs 0 pounds and has a download link.

This e-book apparently weighs 100g and has no download link:

>>>

>>> Book.objects.create(
...     type=Book.TYPE_VIRTUAL,
...     name='Python Tricks',
...     price=1000,
...     weight=100,
...     download_link=None,
... )

This doesn’t make any sense. You have a data integrity problem.

To overcome integrity problems, you add validations to the model:

from django.core.exceptions import ValidationError


class Book(models.Model):

    # ...

    def clean(self) -> None:
        if self.type == Book.TYPE_VIRTUAL:
            if self.weight != 0:
                raise ValidationError(
                    'A virtual product weight cannot exceed zero.'
                )

            if self.download_link is None:
                raise ValidationError(
                    'A virtual product must have a download link.'
                )

        elif self.type == Book.TYPE_PHYSICAL:
            if self.weight == 0:
                raise ValidationError(
                    'A physical product weight must exceed zero.'
                )

            if self.download_link is not None:
                raise ValidationError(
                    'A physical product cannot have a download link.'
                )

        else:
            assert False, f'Unknown product type "{self.type}"'

You used Django’s built-in validation mechanism to enforce data integrity rules. clean() is only called automatically by Django forms. For objects that are not created by a Django form, you need to make sure to explicitly validate the object.

To keep the integrity of the Book model intact, you need to make a little change to the way you create books:

>>>

>>> book = Book(
...    type=Book.TYPE_PHYSICAL,
...    name='Python Tricks',
...    price=1000,
...    weight=0,
...    download_link='http://books.com/54321',
... )
>>> book.full_clean()
ValidationError: {'__all__': ['A physical product weight must exceed zero.']}

>>> book = Book(
...    type=Book.TYPE_VIRTUAL,
...    name='Python Tricks',
...    price=1000,
...    weight=100,
...    download_link=None,
... )
>>> book.full_clean()
ValidationError: {'__all__': ['A virtual product weight cannot exceed zero.']}

When creating objects using the default manager (Book.objects.create(...)), Django will create an object and immediately persist it to the database.

In your case, you want to validate the object before saving if to the database. You first create the object (Book(...)), validate it (book.full_clean()), and only then save it (book.save()).

Denormalization:

A sparse model is a product of denormalization. In a denormalization process, you inline attributes from multiple normalized models into a single table for better performance. A denormalized table will usually have a lot of nullable columns.

Denormalizing is often used in decision support systems such as data warehouses where read performance is most important. Unlike OLTP systems, data warehouses are usually not required to enforce data integrity rules, which makes denormalization ideal.

Pro

Easy to understand and maintain: The sparse model is usually the first step we take when certain types of objects need more information. It’s very intuitive and easy to understand.

Cons

Unable to utilize NOT NULL database constraints: Null values are used for attributes that are not defined for all types of objects.
Complex validation logic: Complex validation logic is required to enforce data integrity rules. The complex logic also requires more tests.
Many Null fields create clutter: Representing multiple types of products in a single model makes it harder to understand and maintain.
New types require schema changes: New types of products require additional fields and validations.

Use Case

The sparse model is ideal when you’re representing heterogeneous objects that share most attributes, and when new items are not added very often.

Semi-Structured Model

Your bookstore is now a huge success, and you are selling more and more books. You have books from different genres and publishers, e-books with different formats, books with odd shapes and sizes, and so on.

In the sparse model approach, you added fields for every new type of product. The model now has a lot of nullable fields, and new developers and employees are having trouble keeping up.

To address the clutter, you decide to keep only the common fields (name and price) on the model. You store the rest of the fields in a single JSONField:

from django.contrib.auth import get_user_model
from django.contrib.postgres.fields import JSONField
from django.db import models

class Book(models.Model):
    TYPE_PHYSICAL = 'physical'
    TYPE_VIRTUAL = 'virtual'
    TYPE_CHOICES = (
        (TYPE_PHYSICAL, 'Physical'),
        (TYPE_VIRTUAL, 'Virtual'),
    )
    type = models.CharField(
        max_length=20,
        choices=TYPE_CHOICES,
    )

    # Common attributes
    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )

    extra = JSONField()

    def __str__(self) -> str:
        return f'[{self.get_type_display()}] {self.name}'


class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    books = models.ManyToManyField(
        Book,
        related_name='+',
    )

JSONField:

In this example, you use PostgreSQL as a database backend. Django provides a built-in JSON field for PostgreSQL in django.contrib.postgres.fields.

For other databases, such as SQLite and MySQL, there are packages that provide similar functionality.

Your Book model is now clutter-free. Common attributes are modeled as fields. Attributes that are not common to all types of products are stored in the extra JSON field:

>>>

>>> from semi_structured.models import Book
>>> physical_book = Book(
...     type=Book.TYPE_PHYSICAL,
...     name='Python Tricks',
...     price=1000,
...     extra={'weight': 200},
... )
>>> physical_book.full_clean()
>>> physical_book.save()
<Book: [Physical] Python Tricks>

>>> virtual_book = Book(
...     type=Book.TYPE_VIRTUAL,
...     name='The Old Man and the Sea',
...     price=1500,
...     extra={'download_link': 'http://books.com/12345'},
... )
>>> virtual_book.full_clean()
>>> virtual_book.save()
<Book: [Virtual] The Old Man and the Sea>

>>> from semi_structured.models import Cart
>>> cart = Cart.objects.create(user=user)
>>> cart.books.add(physical_book, virtual_book)
>>> cart.books.all()
<QuerySet [<Book: [Physical] Python Tricks>, <Book: [Virtual] The Old Man and the Sea>]>

Clearing up the clutter is important, but it comes with a cost. The validation logic is a lot more complicated:

from django.core.exceptions import ValidationError
from django.core.validators import URLValidator

class Book(models.Model):

    # ...

    def clean(self) -> None:

        if self.type == Book.TYPE_VIRTUAL:

            try:
                weight = int(self.extra['weight'])
            except ValueError:
                raise ValidationError(
                    'Weight must be a number'
                )
            except KeyError:
                pass
            else:
                if weight != 0:
                    raise ValidationError(
                        'A virtual product weight cannot exceed zero.'
                    )

            try:
                download_link = self.extra['download_link']
            except KeyError:
                pass
            else:
                # Will raise a validation error
                URLValidator()(download_link)

        elif self.type == Book.TYPE_PHYSICAL:

            try:
                weight = int(self.extra['weight'])
            except ValueError:
                raise ValidationError(
                    'Weight must be a number'
                 )
            except KeyError:
                pass
            else:
                if weight == 0:
                    raise ValidationError(
                        'A physical product weight must exceed zero.'
                     )

            try:
                download_link = self.extra['download_link']
            except KeyError:
                pass
            else:
                if download_link is not None:
                    raise ValidationError(
                        'A physical product cannot have a download link.'
                    )

        else:
            raise ValidationError(f'Unknown product type "{self.type}"')

The benefit of using a proper field is that it validates the type. Both Django and the Django ORM can perform checks to make sure the right type is used for the field. When using a JSONField, you need to validate both the type and the value:

>>>

>>> book = Book.objects.create(
...     type=Book.TYPE_VIRTUAL,
...     name='Python Tricks',
...     price=1000,
...     extra={'weight': 100},
... )
>>> book.full_clean()
ValidationError: {'__all__': ['A virtual product weight cannot exceed zero.']}

Another issue with using JSON is that not all databases have proper support for querying and indexing values in JSON fields.

In PostgreSQL for example, you can query all the books that weigh more than 100:

>>>

>>> Book.objects.filter(extra__weight__gt=100)
<QuerySet [<Book: [Physical] Python Tricks>]>

However, not all database vendors support that.

Another restriction imposed when using JSON is that you are unable to use database constraints such as not null, unique, and foreign keys. You will have to implement these constraints in the application.

This semi-structured approach resembles NoSQL architecture and has many of its advantages and disadvantages. The JSON field is a way to get around the strict schema of a relational database. This hybrid approach provides us with the flexibility to squash many object types into a single table while still maintaining some of the benefits of a relational, strictly and strongly typed database. For many common NoSQL use cases, this approach might actually be more suitable.

Pros

Reduce clutter: Common fields are stored on the model. Other fields are stored in a single JSON field.
Easier to add new types: New types of products don’t require schema changes.

Cons

Complicated and ad hoc validation logic: Validating a JSON field requires validating types as well as values. This challenge can be addressed by using other solutions to validate JSON data such as JSON schema.
Unable to utilize database constraints: Database constraints such as null null, unique and foreign key constraints, which enforce type and data integrity at the database level, cannot be used.
Restricted by database support for JSON: Not all database vendors support querying and indexing JSON fields.
Schema is not enforced by the database system: Schema changes might require backward compatibility or ad hoc migrations. Data can “rot.”
No deep integration with the database metadata system: Metadata about the fields is not stored in the database. Schema is only enforced at the application level.

Use Case

A semi-structured model is ideal when you need to represent heterogeneous objects that don’t share many common attributes, and when new items are added often.

A classic use case for the semi-structured approach is storing events (like logs, analytics, and event stores). Most events have a timestamp, type and metadata like device, user agent, user, and so on. The data for each type is stored in a JSON field. For analytics and log events, it’s important to be able to add new types of events with minimal effort, so this approach is ideal.

Abstract Base Model

So far, you’ve worked around the problem of actually treating your products as heterogeneous. You worked under the assumption that the differences between the products is minimal, so it made sense to maintain them in the same model. This assumption can take you only so far.

Your little store is growing fast, and you want to start selling entirely different types of products, such as e-readers, pens, and notebooks.

A book and an e-book are both products. A product is defined using common attributes such as name and price. In an object-oriented environment, you could look at a Product as a base class or an interface. Every new type of product you add must implement the Product class and extend it with its own attributes.

Django offers the ability to create abstract base classes. Let’s define a Product abstract base class and add two models for Book and EBook:

from django.contrib.auth import get_user_model
from django.db import models


class Product(models.Model):
    class Meta:
        abstract = True

    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )

    def __str__(self) -> str:
        return self.name


class Book(Product):
    weight = models.PositiveIntegerField(
        help_text='in grams',
    )


class EBook(Product):
    download_link = models.URLField()

Notice that both Book and EBook inherit from Product. The fields defined in the base class Product are inherited, so the derived models Book and Ebook don’t need to repeat them.

To add new products, you use the derived classes:

>>>

>>> from abstract_base_model.models import Book
>>> book = Book.objects.create(name='Python Tricks', price=1000, weight=200)
>>> book
<Book: Python Tricks>

>>> ebook = EBook.objects.create(
...     name='The Old Man and the Sea',
...     price=1500,
...     download_link='http://books.com/12345',
... )
>>> ebook
<Book: The Old Man and the Sea>

You might have noticed that the Cart model is missing. You can try to create a Cart model with a ManyToMany field to Product:

class Cart(models.Model):
    user = models.OneToOneField(
       get_user_model(),
       primary_key=True,
       on_delete=models.CASCADE,
    )
    items = models.ManyToManyField(Product)

If you try to reference a ManyToMany field to an abstract model, you will get the following error:

abstract_base_model.Cart.items: (fields.E300) Field defines a relation with model 'Product', which is either not installed, or is abstract.

A foreign key constraint can only point to a concrete table. The abstract base model Product only exists in the code, so there is no products table in the database. The Django ORM will only create tables for the derived models Book and EBook.

Given that you can’t reference the abstract base class Product, you need to reference books and e-books directly:

class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    books = models.ManyToManyField(Book)
    ebooks = models.ManyToManyField(EBook)

You can now add both books and e-books to the cart:

>>>

>>> user = get_user_model().objects.first()
>>> cart = Cart.objects.create(user=user)
>>> cart.books.add(book)
>>> cart.ebooks.add(ebook)

This model is a bit more complicated now. Let’s query the total price of the items in the cart:

>>>

>>> from django.db.models import Sum
>>> from django.db.models.functions import Coalesce
>>> (
...     Cart.objects
...     .filter(pk=cart.pk)
...     .aggregate(total_price=Sum(
...         Coalesce('books__price', 'ebooks__price')
...     ))
... )
{'total_price': 1000}

Because you have more than one type of book, you use Coalesce to fetch either the price of the book or the price of the e-book for each row.

Pro

Easier to implement specific logic: A separate model for each product makes it easier to implement, test, and maintain specific logic.

Cons

Require multiple foreign keys: To reference all types of products, each type needs a foreign key.
Harder to implement and maintain: Operations on all types of products require checking all foreign keys. This adds complexity to the code and makes maintenance and testing harder.
Very hard to scale: New types of products require additional models. Managing many models can be tedious and very hard to scale.

Use Case

An abstract base model is a good choice when there are very few types of objects that required very distinct logic.

An intuitive example is modeling a payment process for your online shop. You want to accept payments with credit cards, PayPal, and store credit. Each payment method goes through a very different process that requires very distinct logic. Adding a new type of payment is not very common, and you don’t plan on adding new payment methods in the near future.

You create a payment process base class with derived classes for credit card payment process, PayPal payment process, and store credit payment process. For each of the derived classes, you implement the payment process in a very different way that cannot be easily shared. In this case, it might make sense to handle each payment process specifically.

Concrete Base Model

Django offers another way to implement inheritance in models. Instead of using an abstract base class that only exists in the code, you can make the base class concrete. “Concrete” means that the base class exists in the database as a table, unlike in the abstract base class solution, where the base class only exists in the code.

Using the abstract base model, you were unable to reference multiple type of products. You were forced to create a many-to-many relation for each type of product. This made it harder to perform tasks on the common fields such as getting the total price of all the items in the cart.

Using a concrete base class, Django will create a table in the database for the Product model. The Product model will have all the common fields you defined in the base model. Derived models such as Book and EBook will reference the Product table using a one-to-one field. To reference a product, you create a foreign key to the base model:

from django.contrib.auth import get_user_model
from django.db import models


class Product(models.Model):
    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )

    def __str__(self) -> str:
        return self.name


class Book(Product):
    weight = models.PositiveIntegerField()


class EBook(Product):
    download_link = models.URLField()

The only difference between this example and the previous one is that the Product model is not defined with abstract=True.

To create new products, you use derived Book and EBook models directly:

>>>

>>> from concrete_base_model.models import Book, EBook
>>> book = Book.objects.create(
...     name='Python Tricks',
...     price=1000,
...     weight=200,
... )
>>> book
<Book: Python Tricks>

>>> ebook = EBook.objects.create(
...     name='The Old Man and the Sea',
...     price=1500,
...     download_link='http://books.com/12345',
... )
>>> ebook
<Book: The Old Man and the Sea>

In the case of concrete base class, it’s interesting to see what’s happening in the underlying database. Let’s look at the tables created by Django in the database:

> \d concrete_base_model_product

Column |          Type          |                         Default
--------+-----------------------+---------------------------------------------------------
id     | integer                | nextval('concrete_base_model_product_id_seq'::regclass)
name   | character varying(100) |
price  | integer                |

Indexes:
   "concrete_base_model_product_pkey" PRIMARY KEY, btree (id)

Referenced by:
   TABLE "concrete_base_model_cart_items" CONSTRAINT "..." FOREIGN KEY (product_id) 
   REFERENCES concrete_base_model_product(id) DEFERRABLE INITIALLY DEFERRED

   TABLE "concrete_base_model_book" CONSTRAINT "..." FOREIGN KEY (product_ptr_id) 
   REFERENCES concrete_base_model_product(id) DEFERRABLE INITIALLY DEFERRED

   TABLE "concrete_base_model_ebook" CONSTRAINT "..." FOREIGN KEY (product_ptr_id) 
   REFERENCES concrete_base_model_product(id) DEFERRABLE INITIALLY DEFERRED

The product table has two familiar fields: name and price. These are the common fields you defined in the Product model. Django also created an ID primary key for you.

In the constraints section, you see multiple tables that are referencing the product table. Two tables that stand out are concrete_base_model_book and concrete_base_model_ebook:

> \d concrete_base_model_book

    Column     |  Type
---------------+---------
product_ptr_id | integer
weight         | integer

Indexes:
   "concrete_base_model_book_pkey" PRIMARY KEY, btree (product_ptr_id)

Foreign-key constraints:
   "..." FOREIGN KEY (product_ptr_id) REFERENCES concrete_base_model_product(id) 
   DEFERRABLE INITIALLY DEFERRED

The Book model has only two fields:

weight is the field you added in the derived Book model.
product_ptr_id is both the primary of the table and a foreign key to the base product model.

Behind the scenes, Django created a base table for product. Then, for each derived model, Django created another table that includes the additional fields, and a field that acts both as a primary key and a foreign key to the product table.

Let’s take a look at a query generated by Django to fetch a single book. Here are the results of print(Book.objects.filter(pk=1).query):

SELECT
    "concrete_base_model_product"."id",
    "concrete_base_model_product"."name",
    "concrete_base_model_product"."price",
    "concrete_base_model_book"."product_ptr_id",
    "concrete_base_model_book"."weight"
FROM
    "concrete_base_model_book"
    INNER JOIN "concrete_base_model_product" ON
        "concrete_base_model_book"."product_ptr_id" = "concrete_base_model_product"."id"
WHERE
    "concrete_base_model_book"."product_ptr_id" = 1

To fetch a single book, Django joined concrete_base_model_product and concrete_base_model_book on the product_ptr_id field. The name and price are in the product table and the weight is in the book table.

Since all the products are managed in the Product table, you can now reference it in a foreign key from the Cart model:

class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    items = models.ManyToManyField(Product)

Adding items to the cart is the same as before:

>>>

>>> from concrete_base_model.models import Cart
>>> cart = Cart.objects.create(user=user)
>>> cart.items.add(book, ebook)
>>> cart.items.all()
<QuerySet [<Book: Python Tricks>, <Book: The Old Man and the Sea>]>

Working with common fields is also simple:

>>>

>>> from django.db.models import Sum
>>> cart.items.aggregate(total_price=Sum('price'))
{'total_price': 2500}

Migrating base classes in Django:

When a derived model is created, Django adds a bases attribute to the migration:

  migrations.CreateModel(
      name='Book',
      fields=[...],
      bases=('concrete_base_model.product',),
  ),

If in the future you remove or change the base class, Django might not be able to perform the migration automatically. You might get this error:

TypeError: metaclass conflict: the metaclass of a derived class must 
be a (non-strict) subclass of the metaclasses of all its bases

This is a known issue in Django (#23818, #23521, #26488). To work around it, you must edit the original migration manually and adjust the bases attribute.

Pros

Primary key is consistent across all types: The product is issued by a single sequence in the base table. This restriction can be easily resolved by using a UUID instead of a sequence.
Common attributes can be queried from a single table: Common queries such as total price, list of product names, and prices can be fetched directly from the base table.

Cons

New product types require schema changes: A new type requires a new model.
Can produce inefficient queries: The data for a single item is in two database tables. Fetching a product requires a join with the base table.
Cannot access extended data from base class instance: A type field is required to downcast an item. This adds complexity to the code. django-polymorphic is a popular module that might eliminate some of these challenges.

Use Case

The concrete base model approach is useful when common fields in the base class are sufficient to satisfy most common queries.

For example, if you often need to query for the cart total price, show a list of items in the cart, or run ad hoc analytic queries on the cart model, you can benefit from having all the common attributes in a single database table.

Generic Foreign Key

Inheritance can sometimes be a nasty business. It forces you to create (possibly premature) abstractions, and it doesn’t always fit nicely into the ORM.

The main problem you have is referencing different products from the cart model. You first tried to squash all the product types into one model (sparse model, semi-structured model), and you got clutter. Then you tried splitting products into separate models and providing a unified interface using a concrete base model. You got a complicated schema and a lot of joins.

Django offers a special way of referencing any model in the project called GenericForeignKey. Generic foreign keys are part of the Content Types framework built into Django. The content type framework is used by Django itself to keep track of models. This is necessary for some core capabilities such as migrations and permissions.

To better understand what content types are and how they facilitate generic foreign keys, let’s look at the content type related to the Book model:

>>>

>>> from django.contrib.contenttypes.models import ContentType
>>> ct = ContentType.objects.get_for_model(Book)
>>> vars(ct)
{'_state': <django.db.models.base.ModelState at 0x7f1c9ea64400>,
'id': 22,
'app_label': 'concrete_base_model',
'model': 'book'}

Each model has a unique identifier. If you want to reference a book with PK 54, you can say, “Get object with PK 54 in the model represented by content type 22.”

GenericForeignKey is implemented exactly like that. To create a generic foreign key, you define two fields:

A reference to a content type (the model)
The primary key of the referenced object (the model instance’s pk attribute)

To implement a many-to-many relation using GenericForeignKey, you need to manually create a model to connect carts with items.

The Cart model remains roughly similar to what you have seen so far:

from django.db import models
from django.contrib.auth import get_user_model


class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )

Unlike previous Cart models, this Cart no longer includes a ManyToMany field. You are going need to do that yourself.

To represent a single item in the cart, you need to reference both the cart and any product:

from django.contrib.contenttypes.fields import GenericForeignKey
from django.contrib.contenttypes.models import ContentType


class CartItem(models.Model):
    cart = models.ForeignKey(
        Cart,
        on_delete=models.CASCADE,
        related_name='items',
    )
    product_object_id = models.IntegerField()
    product_content_type = models.ForeignKey(
        ContentType,
        on_delete=models.PROTECT,
    )
    product = GenericForeignKey(
        'product_content_type',
        'product_object_id',
    )

To add a new item in the Cart, you provide the content type and the primary key:

>>>

>>> book = Book.objects.first()

>>> CartItem.objects.create(
...     product_content_type=ContentType.objects.get_for_model(book),
...     product_object_id=book.pk,
... )
>>> ebook = EBook.objects.first()

>>> CartItem.objects.create(
...    product_content_type=ContentType.objects.get_for_model(ebook),
...    product_object_id=ebook.pk,
... )

Adding an item to a cart is a common task. You can add a method on the cart to add any product to the cart:

class Cart(models.Model):

    # ...

    def add_item(self, product) -> 'CartItem':
        product_content_type = ContentType.objects.get_for_model(product)

        return CartItem.objects.create(
            cart=self,
            product_content_type=product_content_type,
            product_object_id=product.pk,
        )

Adding a new item to a cart is now much shorter:

>>>

>>> cart.add_item(book)
>>> cart.add_item(ebook)

Getting information about the items in the cart is also possible:

>>>

>>> cart.items.all()
<QuerySet [<CartItem: CartItem object (1)>, <CartItem: CartItem object (2)>]

>>> item = cart.items.first()
>>> item.product
<Book: Python Tricks>

>>> item.product.price
1000

So far so good. Where’s the catch?

Let’s try to calculate the total price of the products in the cart:

>>>

>>> from django.db.models import Sum
>>> cart.items.aggregate(total=Sum('product__price'))

FieldError: Field 'product' does not generate an automatic reverse 
relation and therefore cannot be used for reverse querying. 
If it is a GenericForeignKey, consider adding a GenericRelation.

Django tells us it isn’t possible to traverse the generic relation from the generic model to the referenced model. The reason for that is that Django has no idea which table to join to. Remember, the Item model can point to any ContentType.

The error message does mention a GenericRelation. Using a GenericRelation, you can define a reverse relation from the referenced model to the Item model. For example, you can define a reverse relation from the Book model to items of books:

from django.contrib.contenttypes.fields import GenericRelation

class Book(model.Model):
    # ...
    cart_items = GenericRelation(
        'CartItem',
        'product_object_id',
        'product_content_type_id',
        related_query_name='books',
    )

Using the reverse relation, you can answer questions like how many carts include a specific book:

>>>

>>> book.cart_items.count()
4

>>> CartItem.objects.filter(books__id=book.id).count()
4

The two statement are identical.

You still need to know the price of the entire cart. You already saw that fetching the price from each product table is impossible using the ORM. To do that, you have to iterate the items, fetch each item separately, and aggregate:

>>>

>>> sum(item.product.price for item in cart.items.all())
2500

This is one of the major disadvantages of generic foreign keys. The flexibility comes with a great performance cost. It’s very hard to optimize for performance using just the Django ORM.

Structural Subtyping

In the abstract and concrete base class approaches, you used nominal subtyping, which is based on a class hierarchy. Mypy is able to detect this form of relation between two classes and infer types from it.

In the generic relation approach, you used structural subtyping. Structural subtyping exists when a class implements all the methods and attributes of another class. This form of subtyping is very useful when you wish to avoid direct dependency between modules.

Mypy provides a way to utilize structural subtyping using Protocols.

You already identified a product entity with common methods and attributes. You can define a Protocol:

from typing_extensions import Protocol

class Product(Protocol):
    pk: int
    name: str
    price: int

    def __str__(self) -> str:
        ...

Note: The use of class attributes and ellipses (...) in the method definition are new features in Python 3.7. In earlier versions of Python, it isn’t possible to define a Protocol using this syntax. Instead of an ellipsis, methods should have pass in the body. Class attributes such as pk and name can be defined using the @attribute decorator, but it will not work with Django models.

You can now use the Product protocol to add type information. For example, in add_item(), you accept an instance of a product and add it to the cart:

def add_item(
    self,
    product: Product,
) -> 'CartItem':
    product_content_type = ContentType.objects.get_for_model(product)

    return CartItem.objects.create(
        cart=self,
        product_content_type=product_content_type,
        product_object_id=product.pk,
    )

Running mypy on this function will not yield any warnings. Let’s say you change product.pk to product.id, which is not defined in the Product protocol:

def add_item(
    self,
    product: Product,
) -> 'CartItem':
    product_content_type = ContentType.objects.get_for_model(product)

    return CartItem.objects.create(
        cart=self,
        product_content_type=product_content_type,
        product_object_id=product.id,
    )

You will get the following warning from Mypy:

$ mypy
models.py:62: error: "Product" has no attribute "id"

Note: Protocol is not yet a part of Mypy. It’s part of the complementary package called mypy_extentions. The package is developed by the Mypy team and includes features that they thought weren’t ready for the main Mypy package yet.

Pros

Migrations are not needed to add product types: The generic foreign key can reference any model. Adding a new type of product does not require migrations.
Any model can be used as an item: Using generic foreign key, any model can be referenced by the Item model.
Built-in admin support: Django has built-in support for generic foreign keys in the admin. It can inline, for example, information about the referenced models in the detail page.
Self-contained module: There is no direct dependency between the products module and the cart module. This makes this approach ideal for existing projects and pluggable modules.

Cons

Can produce inefficient queries: The ORM cannot determine in advance what models are referenced by the generic foreign key. This makes it very difficult for it to optimize queries that fetch multiple types of products.
Harder to understand and maintain: Generic foreign key eliminates some Django ORM features that require access to specific product models. Accessing information from the product models requires writing more code.
Typing requires Protocol: Mypy is unable to provide type checking for generic models. A Protocol is required.

Use Case

Generic foreign keys are a great choice for pluggable modules or existing projects. The use of GenericForeignKey and structural subtyping abstract any direct dependency between the modules.

In the bookstore example, the book and e-book models can exist in a separate app and new products can be added without changing the cart module. For existing projects, a Cart module can be added with minimal changes to existing code.

The patterns presented in this article play nicely together. Using a mixture of patterns, you can eliminate some of the disadvantages and optimize the schema for your use case.

For example, in the generic foreign key approach, you were unable to get the price of the entire cart quickly. You had to fetch each item separately and aggregate. You can address this specific concern by inlining the price of the product on the Item model (the sparse model approach). This will allow you to query only the Item model to get the total price very quickly.

Conclusion

In this article, you started with a small town bookstore and grew it to a big e-commerce website. You tackled different types of problems and adjusted your model to accommodate the changes. You learned that problems such as complex code and difficulty adding new programmers to the team are often symptoms of a larger problem. You learned how to identify these problems and solve them.

You now know how to plan and implement a polymorphic model using the Django ORM. You’re familiar with multiple approaches, and you understand their pros and cons. You’re able to analyze your use case and decide on the best course of action.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Corey Schafer

2018-12-31T14:00:00+00:00

For this week’s community interview, I am joined by Corey Schafer, of YouTube fame.

Corey is a full-time content creator publishing regular Python tutorials on YouTube. In this interview, we talk to Corey about his YouTube channel and his advice for budding YouTubers and content creators, getting his first developer job, and his passion for woodworking.

Ricky: Welcome to Real Python! We might as well start at the beginning. How’d you get into programming, and when did you start using Python?

Corey: Thanks for having me. I actually got started programming a little later in life than most people you’ve probably interviewed. I used to be a little self-conscious about getting started later, but now I try to be upfront about it so that others aren’t as intimidated if they are just getting started at an “older” age.

I suppose I technically got started in college when I began my degree in Computer Science, but even at that time, I wasn’t taking it very seriously. I would do enough to pass tests, but I wasn’t absorbing any of the information. I definitely wasn’t doing any side projects or using coding in any real-world applications.

It wasn’t until my mid-20s that I randomly applied to an internship for NASA at Kennedy Space Center and started to take programming seriously. To my surprise, I was chosen for the internship. Imposter syndrome set in full-force on day 1. I was definitely out of my league.

But the longer I worked there, the more I realized that these people weren’t superhuman. These were people just like me, with the exception that they’ve spent much more time putting in the work to master their skill.

I thought to myself, “If they can do it, there’s no reason I can’t.” So I left Cape Canaveral with a newfound motivation to really dive into programming and learn as much as I could. I really didn’t feel like I could call myself an actual programmer until my late 20s. I am in my early 30s now, so I feel like I haven’t even scratched the surface in terms of the skills I would like the master.

As far as learning Python, I didn’t start using it until about 4 years ago. I was a full-time front-end JavaScript developer at the West Virginia University GIS Technical Center doing some mapping work. We were using Python for some of the backend scripts, and I was assigned to maintain/update some of those. I found that I enjoyed that much more than the front-end JavaScript work and began using Python on a daily basis.

Ricky: You mentioned that you came to programming a little later in life (though still young). With that context in mind, what was your experience getting your first junior developer job, and do you have any advice for anyone looking for their first developer job later in life?

Corey: I got my first development job in my mid-20s, after my internship in Cape Canaveral. That job was at a small research company in West Virginia (WV). The competition isn’t as fierce in WV as it is in places like San Francisco or Silicon Valley, so I was able to land a position without much prior experience.

I was extremely intimidated when I first started there. I never wanted anyone to see any code that I had written out of fear that it would expose me as not knowing nearly as much as anyone else. I later came to find out that this is a common fear within the community.

Actually working as a developer full-time taught me much more than anything I had learned in school or self-study. Knowing the fundamentals is definitely helpful, but there’s no replacement for actually writing real-world applications and having your code critiqued by people who have been doing this for years.

It’s definitely uncomfortable for your mistakes to be on display for your coworkers to see, but once you get over that discomfort, you come out on the other side less likely to make the same mistakes in the future. Many of my viewers ask me, “How do you just know how to solve these problems?” Well, the truth is, many problems in programming are very similar.

Once you solve problems incorrectly over the years and are shown better and better ways, you eventually learn to recognize certain patterns and use the most efficient methods from the start. This isn’t a skill that most people have naturally. It is developed over years of trial and error.

Ricky: Most of our readers might recognize you from your very popular YouTube channel. I certainly heard of you through it. Not only is it on our Ultimate List of Python YouTube Channels, but I regularly watch your videos and even built one of my most recent Flask apps as a direct result of your Flask course. How have you found YouTube as a platform from a teaching point of view? Any surprising lessons learned?

Corey: Thanks. I was honored to have my channel featured in your article. YouTube is a terrific platform for learning, and I’m excited to see where online learning evolves in the future. Websites like YouTube have definitely lowered the barrier-to-entry for anyone who wants to create content.

Many people think you need a nice recording studio or some financial backing before you get started, but that is no longer the case. As long as you have a computer and a cell phone then you basically have all of the tools you need to get started. I have upgraded my equipment over time, but I first started on YouTube by doing screen recordings on a cheap laptop using the built-in laptop microphone.

For anyone who is thinking about starting a YouTube channel or creating content in general, I do have some lessons I’ve learned over time. I believe the most important lesson I’ve learned is that you should make content for yourself. I try not to create tutorials for topics for the sole purpose of them being popular or what will get the most views… I instead try to create the lessons that I wish I had when learning that topic.

Take notes while you learn certain subjects and keep track of what you found difficult to digest and why. If you get stuck on something, then most likely others are getting stuck on it as well. And once you find solutions to these problems, then you can look back and see if there is a way anyone could have explained it to you that would have helped you understand more easily. If there is, then be sure to pass that on to others.

I really believe that advice carries over into other fields. If you’re an educator, then make content that you personally would have found helpful. If you are a musician, then make music that you personally enjoy. If you’re a comedian, then tell jokes that you think are funny. If you do that, then most likely there are many people out there like yourself who will share your same mindset and love your content.

Ricky: What’s next for the YouTube channel going forward? Any plans to branch out into paid courses or other forms of teaching?

Corey: At the moment, I am sticking to YouTube courses. I feel very fortunate to have a platform that allows me to put my content out there for free and be funded by ad revenue. This allows people who can’t afford courses or who can’t pay for content at the moment to get access to all of my content.

Ideally, my content will always remain free as long as I have enough coming in to live on. It also wouldn’t be possible without generous supporters contributing through sites like Patreon and YouTube Channel Memberships. They support me financially on a monthly basis so that the content can remain free for those who can’t afford it. I would love to continue with this model into the future for as long as I can.

As far as branching out into other forms of teaching, I have thought about creating an online learning platform of some kind. I have used a lot of tools in my personal life that have helped me learn topics quickly. Tools such as spaced-repetition-learning applications and daily coding challenges.

I would love to tackle a project that brought these concepts to a learning platform in a way that would help students absorb material more quickly and fully. Any type of project like that will be something I wouldn’t work on for some time though. For now, I am focused on creating video content.

Ricky: You, sir, are a talented woodworker! Your stuff looks impressive. Is this a new hobby or a long-time passion of yours? How did you get started?

Corey: Oh, thank you for saying that. It is a hobby that I wish I had more time to explore. It is something I have been toying with for many years now. Personally, it is a fantastic way to clear my mind and drain any stress that’s built up.

Woodworking and programming can be similar in many ways. Sometimes I will start a woodworking project and have an idea of what I want the finished product to look like, but not a detailed understanding of where to begin. So you start by jotting down some outlines, then knock out the smaller components, and then after many hours of work you can combine all of that together into the result you were hoping for. There’s a lot of pride that comes with finishing those types of projects.

Ricky: Now for my last question. What else do you get up to in your spare time? What other hobbies and interests do you have, aside from Python? Any you’d like to share and/or plug?

Corey: Since I spend my professional career at a computer, I like to spend my spare time outside as much as possible. This includes hiking, kayaking, camping, swimming, and trips to the dog park. Now that I am working from home and have a more flexible schedule, my girlfriend and I would love to start traveling more. I believe I might mark some international PyCons on my calendar and use those as an excuse to visit some places we have wanted to see for a very long time, with the added bonus of meeting more folks from the Python community.

Thank you, Corey, for joining me this week. You can find Corey’s YouTube channel here. You can get in touch with him via Twitter or his website. And if you’ve benefited from Corey’s videos, consider supporting his efforts with via Patreon.

If there’s someone you’d like me to interview from the Python community, then reach out to me in the comments below, or send me a message on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Make a Location-Based Web App With Django and GeoDjango

2018-12-24T14:00:00+00:00

Throughout this tutorial, you’ll learn how to use Django and GeoDjango to build a location-based web application from scratch. You’ll be building a simple nearby shops application that lists the shops closest to a user’s location.

By the end of this tutorial, you’ll be able to:

Use Django to build a simple web application from scratch
Use the GeoDjango sub-framework to implement geolocation features in your Django application
Use a spatial database (PostgreSQL and PostGIS) to get benefits from the spatial features and easily implement location-aware web apps

Free Bonus: Click here to get free access to additional Django tutorials and resources you can use to deepen your Python web development skills.

Example App Source Code: The full source code for the app you’ll be building in this tutorial is available on the realpython/materials repository on GitHub.

The Tools You Will Be Using

You’ll be using the following tools to develop your nearby shops web application:

The Python programming language
The Django web framework
The PostgreSQL database for persisting data
The PostGIS extension for supporting spatial features in the PostgreSQL database
pip for installing dependencies
The venv module for managing a virtual environment
Docker for installing PostgreSQL and PostGIS

Before diving into practical steps, let’s first start by introducing the frameworks you’ll be using.

Django is the most popular Python framework for building web apps. It makes it easy for developers to quickly build prototypes and meet their project deadlines by providing a plethora of built-in APIs and sub-frameworks such as GeoDjango.

GeoDjango is a built-in application that is included as a contrib module in Django. It’s actually a complete framework itself that can also be used separately from Django. It provides a toolbox of utilities for building GIS web applications.

GIS stands for Geographic Information System. It’s an information system (an organized system for the collection, organization, storage, and communication of information) designed for processing and manipulating data that have geographic or spatial characteristics.

GeoDjango also provides Python bindings to popular spatial libraries such as GEOS, GDAL, and GeoIP, which can be used separately without Django in any Python application or interactively in the shell.

GeoDjango aims to provide a world-class geographic web framework. It has been refactored over the years with the goal of making it easier to work with geospatial data, in other words data that identifies the geographic location of natural or artificial features on Earth and is stored as coordinates and topologies.

GeoDjango integrates very well with the Django ORM and provides a set of geometry fields defined by the Open Geospatial Consortium (OGS) that can be used to map to different types of geometries in geospatial databases:

GeometryField is the base class for all geometric fields in GeoDjango.
PointField is used for storing GEOS Point objects.
PolygonField is used for storing GEOS Polygon objects and so on.

GeoDjango is a very powerful framework for storing and working with geographic data using the Django ORM. It provides an easy-to-use API to find the distances between two points on a map, areas of polygons, the points within a polygon, and so on.

To be able to work with GeoDjango, you’ll need to have two things: a spatial database and geospatial libraries. A spatial database is a database that is optimized for storing and querying data that represents objects defined in a geometric space.

To fully use all features of GeoDjango, you’ll need to install the following open-source geospatial libraries:

GEOS stands for Geometry Engine Open Source. It’s a C++ port of the JTS (Java Topology Suite) that implements the OCG Simple Feature for SQL specification.
GDAL stands for Geospatial Data Abstraction Library. It’s an open-source library for working with raster and vector geospatial data formats.
PROJ.4 is for Cartographic Projections library. It’s an open-source GIS library for easily working with spatial reference systems and projections.
GeoIP is a library that helps users find geographical information based on an IP address.

This tutorial makes use of an Ubuntu 18.04 system for installing prerequisites and a Unix bash for running the commands, but this shouldn’t be a problem for you if you’re using any other system, particularly Unix-based systems like macOS.

For most installation instructions, you’ll be using the aptitude package manager, so you should simply replace that with the equivalent package manager for your system.

Prerequisites

In this section, you’ll be installing the prerequisites needed before you can bootstrap your project, such as Python 3 and GeoDjango dependencies (GEOS, GDAL, and PROJ.4). You’ll also use Docker to set up a PostgreSQL and PostGIS database for your project.

Installing Python 3

There is a big chance that you already have Python 3 installed on your system. If you don’t, you can simply head to the official website and download the binaries for your operating system.

Depending on your system, you may also be able to install Python 3 or upgrade it to the latest version if it’s already installed by using the official package manager.

If you have a problem installing Python 3 or want more information, you can check the Python 3 Installation & Setup Guide, which provides different ways to install Python 3 on your system.

Finally, you can check if you have Python 3 installed by running the following command:

$ python3 --version
Python 3.6.5

Installing GeoDjango Dependencies (GEOS, GDAL, and PROJ.4)

GeoDjango requires a spatial database and a set of open-source geospatial libraries:

GEOS is an open-source geometry engine and a C++ port of the JTS (Java Topology Suite). It’s required by GeoDjango for performing geometric operations.
PROJ.4 is an open-source GIS library for easily working with spatial reference systems and projections. You need it because you’ll be using PostGIS as the spatial database.
GDAL is an open-source geospatial data abstraction library for working with raster and vector data formats. It’s needed for many utilities used by GeoDjango.

You can refer to the docs for more information about spatial databases and the required libraries.

GDAL 2.2 is included in Ubuntu 18.04 so you can simply run the following command to install it:

$ sudo aptitude install gdal-bin libgdal-dev
$ sudo aptitude install python3-gdal

Note: python3-gdal is the Python 3 binding for GDAL.

Next, you can install the other libraries using the following:

$ sudo aptitude install binutils libproj-dev

Note: Since you’re using a binary package for GEOS, you also need to install binutils.

Refer to the docs for detailed instructions about how to install these dependencies on macOS and Windows.

For more information about PROJ.4, you can refer to its official docs.

Setting up a Spatial Database With PostgreSQL and PostGIS

You’ll use PostgreSQL, the most commonly used database with Django. It’s not a spatial database, but thanks to PostGIS you can power up your database with powerful geospatial features.

PostGIS is a spatial database extension that needs to be installed on a PostgreSQL database, which gives it the capability to store and work with spatial data and perform spatial operations. It adds support for geographic objects allowing location queries to be run in SQL.

You can either install PostgreSQL on your system, create a database, and then add the PostGIS extension, or better yet use Docker to quickly create a database using the kartoza postgis image, which provides a container with PostgreSQL and PostGIS already installed:

$ docker run --name=postgis -d -e POSTGRES_USER=user001 -e POSTGRES_PASS=123456789 -e POSTGRES_DBNAME=gis -p 5432:5432 kartoza/postgis:9.6-2.4

After running the command, you’ll have a PostgreSQL server listening on the 5432 port with a database called gis. The database uses the user001 username and the 123456789 password.

Note: You need to have Docker installed on your system. For instructions, you can simply refer to the official docs.

Setting up Your Project

Now that you have a spatial database set up and ready, you can go ahead and setup your Django project. In this section, you’ll use venv to create an isolated virtual environment for your project and install all the required packages such as Django.

Creating a Virtual Environment

A virtual environment allows you to create an isolated environment for the dependencies of your current project. This will allow you to avoid conflicts between the same packages that have different versions.

In Python 3, you can create virtual environments using virtualenv or the venv module.

For more information about Python virtual environments, check out Python Virtual Environments: A Primer.

Now, head over to your terminal and run the following command to create a virtual environment based on Python 3:

$ python3 -m venv env
(env) $

Next, you need to activate the following command:

$ source env/bin/activate

That’s it. You now have your virtual environment activated, and you can install the packages for your project.

Installing Django

The first step after creating and activating a virtual environment is to install Django. The Django package is available from the Python Package Index (PyPI) so you can simply use pip to install it by running the following command in your terminal:

$ pip install django

Creating a Django Project

The project you’ll be creating is a web application that lists shops sorted by distance so your users will be able to discover the shops that are close to their location.

The web application makes use of GeoDjango for easily implementing location requirements like calculating the distances of shops from the user’s location and ordering the shops by distance.

Using GeoDjango, you can get and display the nearest shops that are stored in a PostgreSQL database configured with the PostGIS extension to enable spatial operations.

Now, you’re ready to create a Django project using the django-admin.py script. Simply run the following command:

$ django-admin.py startproject nearbyshops

This will create a project named nearbyshops.

Configuring the PostgreSQL Database

Now that you’ve created a project, let’s continue by configuring the connection to the PostgreSQL and PostGIS spatial database. Open the settings.py file and add django.contrib.gis.db.backends.postgis as the engine with the credentials for the PostGIS database you configured earlier:

DATABASES = {
    'default': {
        'ENGINE': 'django.contrib.gis.db.backends.postgis',
        'NAME': 'gis',
        'USER': 'user001',
        'PASSWORD': '123456789',
        'HOST': 'localhost',
        'PORT': '5432'
    }
}

Note: You need to change the database credentials accordingly if you didn’t specify the same credentials when running the Docker container.

If you try to run your Django server at this point, you’ll get the ImportError: No module named 'psycopg2' error related to psycopg2, which is the most popular PostgreSQL adapter for Python. To solve the error, you simply need to install the psycopg2-binary in your virtual environment using the following:

$ pip install psycopg2-binary

Adding GeoDjango

GeoDjango is a framework that makes it as easy as possible to build GIS and location aware web applications. You can add it by simply including the gis contrib module in the list of installed apps.

Open the settings.py file and locate the INSTALLED_APPS array. Then add the 'django.contrib.gis' module:

INSTALLED_APPS = [
    # [...]
    'django.contrib.gis'
]

Creating a Django Application

A Django project is made up of applications. By default, it contains several core or built-in apps like django.contrib.admin, but you will usually add at least one app that contains your custom project’s code.

Note: For simple projects, you may only need one app, but once your project becomes bigger and has different requirements, you can organize your code in multiple separate apps.

Now that you have created a Django project, configured the connection with the spatial database, and added GeoDjango to the project, you need to create a Django application that you may call shops.

The shops application will contain the code for creating and displaying the shops closest to a user’s location. In the next steps, you are going to perform the following tasks:

Create the app
Add a Shop model
Add a data migration for loading initial demo data (shops)
Add a view function
Add a template

First run the following command to create the app:

$ python manage.py startapp shops

Next, you need to add it to the list of installed apps in the settings.py file, which will make Django recognize it as a part of your project:

INSTALLED_APPS = [
    # [...]
    'shops'
]

Creating a Django Model

After creating the shops application, which will contain the actual code of your project, you need to add models in your app. Django uses an ORM (Object Relational Mapper), which is an abstraction layer between Django and the database that transforms Python objects (or models) into database tables.

In this case, you need one model that represents a shop in the database. You’ll create a Shop model that has the following fields:

name: the name of the shop
location: the location of the shop in latitude and longitude coordinates
address: the address of the shop
city: the city the shop is in

Open the shops/models.py file and add the following code:

from django.contrib.gis.db import models

class Shop(models.Model):
    name = models.CharField(max_length=100)
    location = models.PointField()
    address = models.CharField(max_length=100)
    city = models.CharField(max_length=50)

For the location, you are using the PointField, a GeoDjango-specific geometric field for storing a GEOS Point object that represents a pair of longitude and latitude coordinates.

The other fields are normal Django fields of type CharField that can be used to store strings of small and large size.

Note: Please note that the models module is imported from django.contrib.gis.db and not the usual django.db module.

Creating the Database Tables

With Django, you don’t need to use SQL to create the database tables thanks to its ORM. Let’s create the database tables by using the makemigrations and migrate commands. Head back to your terminal and run the following:

$ python manage.py makemigrations
$ python manage.py migrate

For more information about these commands, check out Django Migrations – A Primer.

Adding a Super User

You need to create a super user so you can access the admin interface. This can be done using the following command:

$ python manage.py createsuperuser

The prompt will ask you for the username, email, and password you want to use for accessing the user account. Enter them and hit Enter.

Registering the Model in the Admin Interface

Django’s admin application provides a complete CRUD interface for managing data.

GeoDjango extends the admin application to add support for working with geometry fields.

Before you can access your models from Django admin, you need to register them.

Open the shops/admin.py file and add the following code:

from django.contrib.gis.admin import OSMGeoAdmin
from .models import Shop

@admin.register(Shop)
class ShopAdmin(OSMGeoAdmin):
    list_display = ('name', 'location')

You are using the @admin.register decorator to register the Shop model in the admin application. The decorated class is a representation of the Shop model in the admin interface and allows you to customize different aspects such as the Shop fields that you want to display. (In your case, it’s the name and location.) For more information about decorators, you can read Primer on Python Decorators.

Since the Shop model includes a GeoDjango field, you need to use the special OSMGeoAdmin class that’s available from the django.contrib.gis.admin package .

You can either use GeoModelAdmin or OSMGeoAdmin, which is a subclass of GeoModelAdmin that uses an Open Street Map layer in the admin to display geometric fields. This provides more information like street and thoroughfare details than would be available with the GeoModelAdmin class, which uses Vector Map Level 0.

You can now run the Django server:

$ python manage.py runserver

You application will be running from localhost:8000, and you can access the admin interface from localhost:8000/admin.

This is a screenshot from the Add shop interface:

You can see that the Location geometric field is displayed as an interactive map. You can zoom the map in and out, and you can choose different selectors at the top right corner of the map to select a location, which is marked by the green circle.

Adding Initial Data

You need some initial demo data for your application, but instead of manually adding data, you can use a data migration.

Data migrations can be used for multiple scenarios, including adding initial data in your database. For more information, check out Data Migrations.

Before creating a migration, let’s first get some real-world data from OpenStreetMap using overpass turbo, a web-based data filtering tool for OpenStreetMap. You can run Overpass API queries and analyze the resulting data interactively on the map.

You can also use the integrated Wizard, which makes it easy to create queries.

In your case, you want to get all the shops in a city. Simply click on the Wizard button. A small window will pop up. In the text field, write a query like “shop in Miami” and click on build and run query.

Next, click on the export button and click on download/copy as raw OSM data to download a JSON file that contains the raw OSM data. Save the file as data.json in your project’s root folder:

This is a screenshot of example data from the file:

You need to get the objects in the elements array. Specifically the lat, lon, and tags (name) fields for each shop.

You can find more details on how you can write Overpass queries from this wiki.

Now, let’s create an empty migration for importing the content of the data.json file in the database, using the following command:

$ python manage.py makemigrations shops --empty

Open the migration file. It has the following code:

from django.db import migrations

class Migration(migrations.Migration):
    dependencies = [
        ('shops', '0001_initial'),
    ]
    operations = [
    ]

You next need to create load_data() to be executed by RunPython(). First, in the import area, add the following imports:

from django.db import migrations
import json
from django.contrib.gis.geos import fromstr
from pathlib import Path

You are importing the Path class from the pathlib package for accessing low-level system functions, the json package for working with JSON, the Django built-in migrations API, and fromstr(), part of the geos package.

Next, add load_data():

DATA_FILENAME = 'data.json'
def load_data(apps, schema_editor):
    Shop = apps.get_model('shops', 'Shop')
    jsonfile = Path(__file__).parents[2] / DATA_FILENAME

    with open(str(jsonfile)) as datafile:
        objects = json.load(datafile)
        for obj in objects['elements']:
            try:
                objType = obj['type']
                if objType == 'node':
                    tags = obj['tags']
                    name = tags.get('name','no-name')
                    longitude = obj.get('lon', 0)
                    latitude = obj.get('lat', 0)
                    location = fromstr(f'POINT({longitude} {latitude})', srid=4326)
                    Shop(name=name, location = location).save()
            except KeyError:
                pass

Let’s explain the code you’ve just added. You first construct the absolute path using the Path class of the pathlib library and open the data.json file. Next, you parse the JSON file into a Python object.

You loop through the elements object containing the locations and tags of shops. Inside the loop, you extract the name and the longitude and latitude coordinates. Then you use formstr() to return a valid GEOSGeometry object corresponding to the spatial data in the string that can be assigned to the location field of the Shop model. Finally, you create and save the instance of the Shop model corresponding to the extracted data.

You are using the with statement, so you don’t have to explicitly close the file, and an f-string for formatting the argument of fromstr().

fromstr() takes a srid as the second parameter. srid stands for Spatial Reference System Identifier. It’s a unique value to identify spatial reference systems (projection systems used for interpreting the data in the spatial database).

The 4326 srid is the most popular system used with PostGIS. It’s also known as WGS84, where units are specified in degrees of longitude and latitude. You can refer to spatialreference.org for a Django-powered database of spatial reference systems.

Next, add the migration class to execute the above function when you run the migrate command:

class Migration(migrations.Migration):

    dependencies = [
        ('shops', '0005_auto_20181018_2050'),
    ]

    operations = [
        migrations.RunPython(load_data)
    ]

That’s it. You can now return to your terminal and run the following:

$ python manage.py migrate

The data from the data.json file will be loaded on your database. Run your Django server and head to your admin interface. You should see your data in the table. In my case, this is a screenshot of a portion of the table:

Displaying Nearby Shops

At this point of the tutorial, you have created:

The shops application, which encapsulates the code for creating and getting nearby shops in your project
The Shop model and the corresponding tables in the database
The initial admin user for accessing the admin interface
The initial demo data for loading real-world shops in the database that you can play with without manually entering a lot of fake data

You also registered the Shop model in the admin application so you can create, update, delete, and list shops from the admin interface.

Next, you’ll add a view function using the generic ListView class that you can use to display a list of nearby shops. You also create an HTML template that will be used by the view function to render the shops and add the URL that will be used to display the shops.

Let’s start by adding a template and a view function that will be used to display the nearby shops from a user’s location.

Open the shops/views.py file and start by importing the necessary APIs:

from django.views import generic
from django.contrib.gis.geos import fromstr
from django.contrib.gis.db.models.functions import Distance
from .models import Shop

Next add a user_location variable where you can hard code a user location:

longitude = -80.191788
latitude = 25.761681

user_location = Point(longitude, latitude, srid=4326)

In this part, you’ll simply hard code the user’s location (the coordinates of Miami in the USA), but this ideally should be specified by the user or retrieved automatically from the user’s browser with their permission using JavaScript and the HTML5 GeoLocation API. You can scroll down to the middle of that page to see a live example of implementing the Geolocation API.

Finally, add the following view class:

class Home(generic.ListView):
    model = Shop
    context_object_name = 'shops'
    queryset = Shop.objects.annotate(distance=Distance('location',
    user_location)
    ).order_by('distance')[0:6]
    template_name = 'shops/index.html'

You are using the generic class-based ListView to create a view.

Class-based views are an alternative way to implement views as Python classes instead of functions. They are used to handle common use cases in web development without re-inventing the wheel. In this example, you’ve just sub-classed the ListView generic view and overridden the model, context_object_name, queryset, and template_name attributes to create a list view that handles HTTP requests without any extra code.

Now let’s focus on the queryset attribute. To get the nearby shops, you simply use .annotate() to annotate each object on the returned queryset with a distance annotation that’s calculated using Distance(), available from GeoDjango, between the location of each shop and the user’s location. You also order the returned queryset by the distance annotation and take only the nearest six shops.

You can learn more about class-based views from the official docs.

Next let’s add the shops/index.html template with the following content:

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <title>Nearby Shops</title>
    </head>
<body>
    <h1>Nearby Shops</h1>
    {% if shops %}
    <ul>
    {% for shop in shops %}
        <li>
        {{ shop.name }}: {{shop.distance}}
        </li>
    {% endfor %}
    </ul>
    {% endif %}
</body>
</html>

The nearest shops are available from the shops context object that you specified as the context_object_name in the class-based view. You loop through the shops object and you display the name and distance from the user’s location for each shop.

Finally, let’s add a URL to our urls.py file:

from django.urls import path
from shops import views

urlpatterns = [
    # [...]
    path('', views.ShopList.as_view())
]

You use .as_view() to return a callable view that takes a request and returns a response, which can be passed as the second parameter for path() that maps paths to views.

Now you can run your Django server. The home page will display a simple un-styled list with the nearest shops from the hard-coded user’s location. This is an example screenshot:

In the screenshot, each item in the list displays the name of the shop (before the colon) and the distance in meters from the user’s location (after the colon). The letter m refers to meters.

Note: The view function you used to display the results in the screenshot is just for testing the queryset annotated by Distance().

In the next part of the series, you’ll use a RESTful endpoint to return the nearest shops. Instead of a Django template, you’ll use Vue.js with some CSS styling for a better look, so stay tuned!

Wrap-Up

Congratulations on creating your location based web application using GeoDjango, which aims to become a world-class geographic framework for implementing GIS apps. You now have the basic skills that you can use to either add simple geolocation stuff to your applications or create GIS apps. You can read the GeoDjango docs for a complete resource of the available APIs and what you can do with them.

You also learned to use Docker to quickly pull and launch a PostgreSQL and PostGIS server. Docker can be used for more than that. It’s a containerization tool for spinning up isolated, reproducible application environments. You can read Django Development with Docker Compose and Machine if you want to learn how to containerize your Django project.

Nowadays, location aware apps (apps that know your location and help you discover nearby objects and services by offering you results based on your location) are all the rage. Using the knowledge you gained in this tutorial, you’ll be able to incorporate this modern feature in your apps developed with Django.

The only requirement, besides the dependencies of GeoDjango, is to use a spatial database (a database that’s capable of storing and manipulating spatial data). For PostgreSQL, one of the most popular database management systems used with Django, you can simply install the PostGIS extension with your database to turn it into a spatial database. Other popular databases like Oracle and MySQL have built-in support for spatial data.

Conclusion

Throughout this tutorial, you used PostgreSQL, PostGIS, Django, and GeoDjango to build a simple nearby shops web application.

Example App Source Code: The full source code for the app you’ll be building in this tutorial is available on the realpython/materials repository on GitHub.

Free Bonus: Click here to get free access to additional Django tutorials and resources you can use to deepen your Python web development skills.

In the next tutorial, you’ll continue building the application with Django REST framework to expose a RESTful API, and you’ll use the JavaScript Vue.js library with the Axios HTTP client to consume the API and render the data.

You’ll also use the HTML5 Geolocation API to automatically get the user’s location from the user’s browser instead of hard-coding it like you did in this tutorial, so stay tuned!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Write Beautiful Python Code With PEP 8

2018-12-19T14:00:00+00:00

PEP 8, sometimes spelled PEP8 or PEP-8, is a document that provides guidelines and best practices on how to write Python code. It was written in 2001 by Guido van Rossum, Barry Warsaw, and Nick Coghlan. The primary focus of PEP 8 is to improve the readability and consistency of Python code.

PEP stands for Python Enhancement Proposal, and there are several of them. A PEP is a document that describes new features proposed for Python and documents aspects of Python, like design and style, for the community.

This tutorial outlines the key guidelines laid out in PEP 8. It’s aimed at beginner to intermediate programmers, and as such I have not covered some of the most advanced topics. You can learn about these by reading the full PEP 8 documentation.

By the end of this tutorial, you’ll be able to:

Write Python code that conforms to PEP 8
Understand the reasoning behind the guidelines laid out in PEP 8
Set up your development environment so that you can start writing PEP 8 compliant Python code

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Why We Need PEP 8

“Readability counts.”

— The Zen of Python

PEP 8 exists to improve the readability of Python code. But why is readability so important? Why is writing readable code one of the guiding principles of the Python language?

As Guido van Rossum said, “Code is read much more often than it is written.” You may spend a few minutes, or a whole day, writing a piece of code to process user authentication. Once you’ve written it, you’re never going to write it again. But you’ll definitely have to read it again. That piece of code might remain part of a project you’re working on. Every time you go back to that file, you’ll have to remember what that code does and why you wrote it, so readability matters.

If you’re new to Python, it can be difficult to remember what a piece of code does a few days, or weeks, after you wrote it. If you follow PEP 8, you can be sure that you’ve named your variables well. You’ll know that you’ve added enough whitespace so it’s easier to follow logical steps in your code. You’ll also have commented your code well. All this will mean your code is more readable and easier to come back to. As a beginner, following the rules of PEP 8 can make learning Python a much more pleasant task.

Following PEP 8 is particularly important if you’re looking for a development job. Writing clear, readable code shows professionalism. It’ll tell an employer that you understand how to structure your code well.

If you have more experience writing Python code, then you may need to collaborate with others. Writing readable code here is crucial. Other people, who may have never met you or seen your coding style before, will have to read and understand your code. Having guidelines that you follow and recognize will make it easier for others to read your code.

Naming Conventions

“Explicit is better than implicit.”

— The Zen of Python

When you write Python code, you have to name a lot of things: variables, functions, classes, packages, and so on. Choosing sensible names will save you time and energy later. You’ll be able to figure out, from the name, what a certain variable, function, or class represents. You’ll also avoid using inappropriate names that might result in errors that are difficult to debug.

Note: Never use l, O, or I single letter names as these can be mistaken for 1 and 0, depending on typeface:

O = 2  # This may look like you're trying to reassign 2 to zero

Naming Styles

The table below outlines some of the common naming styles in Python code and when you should use them:

Type	Naming Convention	Examples
Function	Use a lowercase word or words. Separate words by underscores to improve readability.	`function`, `my_function`
Variable	Use a lowercase single letter, word, or words. Separate words with underscores to improve readability.	`x`, `var`, `my_variable`
Class	Start each word with a capital letter. Do not separate words with underscores. This style is called camel case.	`Model`, `MyClass`
Method	Use a lowercase word or words. Separate words with underscores to improve readability.	`class_method`, `method`
Constant	Use an uppercase single letter, word, or words. Separate words with underscores to improve readability.	`CONSTANT`, `MY_CONSTANT`, `MY_LONG_CONSTANT`
Module	Use a short, lowercase word or words. Separate words with underscores to improve readability.	`module.py`, `my_module.py`
Package	Use a short, lowercase word or words. Do not separate words with underscores.	`package`, `mypackage`

These are some of the common naming conventions and examples of how to use them. But in order to write readable code, you still have to be careful with your choice of letters and words. In addition to choosing the correct naming styles in your code, you also have to choose the names carefully. Below are a few pointers on how to do this as effectively as possible.

How to Choose Names

Choosing names for your variables, functions, classes, and so forth can be challenging. You should put a fair amount of thought into your naming choices when writing code as it will make your code more readable. The best way to name your objects in Python is to use descriptive names to make it clear what the object represents.

When naming variables, you may be tempted to choose simple, single-letter lowercase names, like x. But, unless you’re using x as the argument of a mathematical function, it’s not clear what x represents. Imagine you are storing a person’s name as a string, and you want to use string slicing to format their name differently. You could end up with something like this:

>>>

>>> # Not recommended
>>> x = 'John Smith'
>>> y, z = x.split()
>>> print(z, y, sep=', ')
'Smith, John'

This will work, but you’ll have to keep track of what x, y, and z represent. It may also be confusing for collaborators. A much clearer choice of names would be something like this:

>>>

>>> # Recommended
>>> name = 'John Smith'
>>> first_name, last_name = name.split()
>>> print(last_name, first_name, sep=', ')
'Smith, John'

Similarly, to reduce the amount of typing you do, it can be tempting to use abbreviations when choosing names. In the example below, I have defined a function db() that takes a single argument x and doubles it:

# Not recommended
def db(x):
    return x * 2

At first glance, this could seem like a sensible choice. db() could easily be an abbreviation for double. But imagine coming back to this code in a few days. You may have forgotten what you were trying to achieve with this function, and that would make guessing how you abbreviated it difficult.

The following example is much clearer. If you come back to this code a couple of days after writing it, you’ll still be able to read and understand the purpose of this function:

# Recommended
def multiply_by_two(x):
    return x * 2

The same philosophy applies to all other data types and objects in Python. Always try to use the most concise but descriptive names possible.

Code Layout

“Beautiful is better than ugly.”

— The Zen of Python

How you lay out your code has a huge role in how readable it is. In this section, you’ll learn how to add vertical whitespace to improve the readability of your code. You’ll also learn how to handle the 79 character line limit recommended in PEP 8.

Blank Lines

Vertical whitespace, or blank lines, can greatly improve the readability of your code. Code that’s bunched up together can be overwhelming and hard to read. Similarly, too many blank lines in your code makes it look very sparse, and the reader might need to scroll more than necessary. Below are three key guidelines on how to use vertical whitespace.

Surround top-level functions and classes with two blank lines. Top-level functions and classes should be fairly self-contained and handle separate functionality. It makes sense to put extra vertical space around them, so that it’s clear they are separate:

class MyFirstClass:
    pass


class MySecondClass:
    pass


def top_level_function():
    return None

Surround method definitions inside classes with a single blank line. Inside a class, functions are all related to one another. It’s good practice to leave only a single line between them:

class MyClass:
    def first_method(self):
        return None

    def second_method(self):
        return None

Use blank lines sparingly inside functions to show clear steps. Sometimes, a complicated function has to complete several steps before the return statement. To help the reader understand the logic inside the function, it can be helpful to leave a blank line between each step.

In the example below, there is a function to calculate the variance of a list. This is two-step problem, so I have indicated each step by leaving a blank line between them. There is also a blank line before the return statement. This helps the reader clearly see what’s returned:

def calculate_variance(number_list):
    sum_list = 0
    for number in number_list:
        sum_list = sum_list + number
    mean = sum_list / len(number_list)

    sum_squares = 0
    for number in number_list:
        sum_squares = sum_squares + number**2
    mean_squares = sum_squares / len(number_list)

    return mean_squares - mean**2

If you use vertical whitespace carefully, it can greatly improved the readability of your code. It helps the reader visually understand how your code splits up into sections, and how those sections relate to one another.

Maximum Line Length and Line Breaking

PEP 8 suggests lines should be limited to 79 characters. This is because it allows you to have multiple files open next to one another, while also avoiding line wrapping.

Of course, keeping statements to 79 characters or less is not always possible. PEP 8 outlines ways to allow statements to run over several lines.

Python will assume line continuation if code is contained within parentheses, brackets, or braces:

def function(arg_one, arg_two,
             arg_three, arg_four):
    return arg_one

If it is impossible to use implied continuation, then you can use backslashes to break lines instead:

from mypkg import example1, \
    example2, example3

However, if you can use implied continuation, then you should do so.

If line breaking needs to occur around binary operators, like + and *, it should occur before the operator. This rule stems from mathematics. Mathematicians agree that breaking before binary operators improves readability. Compare the following two examples.

Below is an example of breaking before a binary operator:

# Recommended
total = (first_variable
         + second_variable
         - third_variable)

You can immediately see which variable is being added or subtracted, as the operator is right next to the variable being operated on.

Now, let’s see an example of breaking after a binary operator:

# Not Recommended
total = (first_variable +
         second_variable -
         third_variable)

Here, it’s harder to see which variable is being added and which is subtracted.

Breaking before binary operators produces more readable code, so PEP 8 encourages it. Code that consistently breaks after a binary operator is still PEP 8 compliant. However, you’re encouraged to break before a binary operator.

Indentation

“There should be one—and preferably only one—obvious way to do it.”

— The Zen of Python

Indentation, or leading whitespace, is extremely important in Python. The indentation level of lines of code in Python determines how statements are grouped together.

Consider the following example:

x = 3
if x > 5:
    print('x is larger than 5')

The indented print statement lets Python know that it should only be executed if the if statement returns True. The same indentation applies to tell Python what code to execute when a function is called or what code belongs to a given class.

The key indentation rules laid out by PEP 8 are the following:

Use 4 consecutive spaces to indicate indentation.
Prefer spaces over tabs.

Tabs vs. Spaces

As mentioned above, you should use spaces instead of tabs when indenting code. You can adjust the settings in your text editor to output 4 spaces instead of a tab character, when you press the Tab key.

If you’re using Python 2 and have used a mixture of tabs and spaces to indent your code, you won’t see errors when trying to run it. To help you to check consistency, you can add a -t flag when running Python 2 code from the command line. The interpreter will issue warnings when you are inconsistent with your use of tabs and spaces:

$ python2 -t code.py
code.py: inconsistent use of tabs and spaces in indentation

If, instead, you use the -tt flag, the interpreter will issue errors instead of warnings, and your code will not run. The benefit of using this method is that the interpreter tells you where the inconsistencies are:

$ python2 -tt code.py
  File "code.py", line 3
    print(i, j)
             ^
TabError: inconsistent use of tabs and spaces in indentation

Python 3 does not allow mixing of tabs and spaces. Therefore, if you are using Python 3, then these errors are issued automatically:

$ python3 code.py
  File "code.py", line 3
    print(i, j)
              ^
TabError: inconsistent use of tabs and spaces in indentation

You can write Python code with either tabs or spaces indicating indentation. But, if you’re using Python 3, you must be consistent with your choice. Otherwise, your code will not run. PEP 8 recommends that you always use 4 consecutive spaces to indicate indentation.

Indentation Following Line Breaks

When you’re using line continuations to keep lines to under 79 characters, it is useful to use indentation to improve readability. It allows the reader to distinguish between two lines of code and a single line of code that spans two lines. There are two styles of indentation you can use.

The first of these is to align the indented block with the opening delimiter:

def function(arg_one, arg_two,
             arg_three, arg_four):
    return arg_one

Sometimes you can find that only 4 spaces are needed to align with the opening delimiter. This will often occur in if statements that span multiple lines as the if, space, and opening bracket make up 4 characters. In this case, it can be difficult to determine where the nested code block inside the if statement begins:

x = 5
if (x > 3 and
    x < 10):
    print(x)

In this case, PEP 8 provides two alternatives to help improve readability:

Add a comment after the final condition. Due to syntax highlighting in most editors, this will separate the conditions from the nested code:
```
x = 5
if (x > 3 and
    x < 10):
    # Both conditions satisfied
    print(x)
```

Add extra indentation on the line continuation:

x = 5
if (x > 3 and
        x < 10):
    print(x)

An alternative style of indentation following a line break is a hanging indent. This is a typographical term meaning that every line but the first in a paragraph or statement is indented. You can use a hanging indent to visually represent a continuation of a line of code. Here’s an example:

var = function(
    arg_one, arg_two,
    arg_three, arg_four)

Note: When you’re using a hanging indent, there must not be any arguments on the first line. The following example is not PEP 8 compliant:

# Not Recommended
var = function(arg_one, arg_two,
    arg_three, arg_four)

When using a hanging indent, add extra indentation to distinguish the continued line from code contained inside the function. The following example is difficult to read because the code inside the function is at the same indentation level as the continued lines:

# Not Recommended
def function(
    arg_one, arg_two,
    arg_three, arg_four):
    return arg_one

Instead, it’s better to use a double indent on the line continuation. This helps you to distinguish between function arguments and the function body, improving readability:

def function(
        arg_one, arg_two,
        arg_three, arg_four):
    return arg_one

When you write PEP 8 compliant code, the 79 character line limit forces you to add line breaks in your code. To improve readability, you should indent a continued line to show that it is a continued line. There are two ways of doing this. The first is to align the indented block with the opening delimiter. The second is to use a hanging indent. You are free to chose which method of indentation you use following a line break.

Where to Put the Closing Brace

Line continuations allow you to break lines inside parentheses, brackets, or braces. It’s easy to forget about the closing brace, but it’s important to put it somewhere sensible. Otherwise, it can confuse the reader. PEP 8 provides two options for the position of the closing brace in implied line continuations:

Line up the closing brace with the first non-whitespace character of the previous line:

list_of_numbers = [
    1, 2, 3,
    4, 5, 6,
    7, 8, 9
    ]

Line up the closing brace with the first character of the line that starts the construct:
```
list_of_numbers = [
    1, 2, 3,
    4, 5, 6,
    7, 8, 9
]
```

You are free to chose which option you use. But, as always, consistency is key, so try to stick to one of the above methods.

Comments

“If the implementation is hard to explain, it’s a bad idea.”

— The Zen of Python

You should use comments to document code as it’s written. It is important to document your code so that you, and any collaborators, can understand it. When you or someone else reads a comment, they should be able to easily understand the code the comment applies to and how it fits in with the rest of your code.

Here are some key points to remember when adding comments to your code:

Limit the line length of comments and docstrings to 72 characters.
Use complete sentences, starting with a capital letter.
Make sure to update comments if you change your code.

Block Comments

Use block comments to document a small section of code. They are useful when you have to write several lines of code to perform a single action, such as importing data from a file or updating a database entry. They are important as they help others understand the purpose and functionality of a given code block.

PEP 8 provides the following rules for writing block comments:

Indent block comments to the same level as the code they describe.
Start each line with a # followed by a single space.
Separate paragraphs by a line containing a single #.

Here is a block comment explaining the function of a for loop. Note that the sentence wraps to a new line to preserve the 79 character line limit:

for i in range(0, 10):
    # Loop over i ten times and print out the value of i, followed by a
    # new line character
    print(i, '\n')

Sometimes, if the code is very technical, then it is necessary to use more than one paragraph in a block comment:

def quadratic(a, b, c, x):
    # Calculate the solution to a quadratic equation using the quadratic
    # formula.
    #
    # There are always two solutions to a quadratic equation, x_1 and x_2.
    x_1 = (- b+(b**2-4*a*c)**(1/2)) / (2*a)
    x_2 = (- b-(b**2-4*a*c)**(1/2)) / (2*a)
    return x_1, x_2

If you’re ever in doubt as to what comment type is suitable, then block comments are often the way to go. Use them as much as possible throughout your code, but make sure to update them if you make changes to your code!

Inline Comments

Inline comments explain a single statement in a piece of code. They are useful to remind you, or explain to others, why a certain line of code is necessary. Here’s what PEP 8 has to say about them:

Use inline comments sparingly.
Write inline comments on the same line as the statement they refer to.
Separate inline comments by two or more spaces from the statement.
Start inline comments with a # and a single space, like block comments.
Don’t use them to explain the obvious.

Below is an example of an inline comment:

x = 5  # This is an inline comment

Sometimes, inline comments can seem necessary, but you can use better naming conventions instead. Here’s an example:

x = 'John Smith'  # Student Name

Here, the inline comment does give extra information. However using x as a variable name for a person’s name is bad practice. There’s no need for the inline comment if you rename your variable:

student_name = 'John Smith'

Finally, inline comments such as these are bad practice as they state the obvious and clutter code:

empty_list = []  # Initialize empty list

x = 5
x = x * 5  # Multiply x by 5

Inline comments are more specific than block comments, and it’s easy to add them when they’re not necessary, which leads to clutter. You could get away with only using block comments so, unless you are sure you need an inline comment, your code is more likely to be PEP 8 compliant if you stick to block comments.

Documentation Strings

Documentation strings, or docstrings, are strings enclosed in double (""") or single (''') quotation marks that appear on the first line of any function, class, method, or module. You can use them to explain and document a specific block of code. There is an entire PEP, PEP 257, that covers docstrings, but you’ll get a summary in this section.

The most important rules applying to docstrings are the following:

Surround docstrings with three double quotes on either side, as in """This is a docstring""".
Write them for all public modules, functions, classes, and methods.

Put the """ that ends a multiline docstring on a line by itself:

def quadratic(a, b, c, x):
    """Solve quadratic equation via the quadratic formula.

    A quadratic equation has the following form:
    ax**2 + bx + c = 0

    There always two solutions to a quadratic equation: x_1 & x_2.
    """
    x_1 = (- b+(b**2-4*a*c)**(1/2)) / (2*a)
    x_2 = (- b-(b**2-4*a*c)**(1/2)) / (2*a)

    return x_1, x_2

For one-line docstrings, keep the """ on the same line:

def quadratic(a, b, c, x):
    """Use the quadratic formula"""
    x_1 = (- b+(b**2-4*a*c)**(1/2)) / (2*a)
    x_2 = (- b-(b**2-4*a*c)**(1/2)) / (2*a)

    return x_1, x_2

For a more detailed article on documenting Python code, see Documenting Python Code: A Complete Guide by James Mertz.

Whitespace in Expressions and Statements

“Sparse is better than dense.”

— The Zen of Python

Whitespace can be very helpful in expressions and statements when used properly. If there is not enough whitespace, then code can be difficult to read, as it’s all bunched together. If there’s too much whitespace, then it can be difficult to visually combine related terms in a statement.

Whitespace Around Binary Operators

Surround the following binary operators with a single space on either side:

Assignment operators (=, +=, -=, and so forth)
Comparisons (==, !=, >, <. >=, <=) and (is, is not, in, not in)
Booleans (and, not, or)

Note: When = is used to assign a default value to a function argument, do not surround it with spaces.

# Recommended
def function(default_parameter=5):
    # ...


# Not recommended
def function(default_parameter = 5):
    # ...

When there’s more than one operator in a statement, adding a single space before and after each operator can look confusing. Instead, it is better to only add whitespace around the operators with the lowest priority, especially when performing mathematical manipulation. Here are a couple examples:

# Recommended
y = x**2 + 5
z = (x+y) * (x-y)

# Not Recommended
y = x ** 2 + 5
z = (x + y) * (x - y)

You can also apply this to if statements where there are multiple conditions:

# Not recommended
if x > 5 and x % 2 == 0:
    print('x is larger than 5 and divisible by 2!')

In the above example, the and operator has lowest priority. It may therefore be clearer to express the if statement as below:

# Recommended
if x>5 and x%2==0:
    print('x is larger than 5 and divisible by 2!')

You are free to choose which is clearer, with the caveat that you must use the same amount of whitespace either side of the operator.

The following is not acceptable:

# Definitely do not do this!
if x >5 and x% 2== 0:
    print('x is larger than 5 and divisible by 2!')

In slices, colons act as a binary operators. Therefore, the rules outlined in the previous section apply, and there should be the same amount of whitespace either side. The following examples of list slices are valid:

list[3:4]

# Treat the colon as the operator with lowest priority
list[x+1 : x+2]

# In an extended slice, both colons must be
# surrounded by the same amount of whitespace
list[3:4:5]
list[x+1 : x+2 : x+3]

# The space is omitted if a slice parameter is omitted
list[x+1 : x+2 :]

In summary, you should surround most operators with whitespace. However, there are some caveats to this rule, such as in function arguments or when you’re combining multiple operators in one statement.

When to Avoid Adding Whitespace

In some cases, adding whitespace can make code harder to read. Too much whitespace can make code overly sparse and difficult to follow. PEP 8 outlines very clear examples where whitespace is inappropriate.

The most important place to avoid adding whitespace is at the end of a line. This is known as trailing whitespace. It is invisible and can produce errors that are difficult to trace.

The following list outlines some cases where you should avoid adding whitespace:

Immediately inside parentheses, brackets, or braces:

# Recommended
my_list = [1, 2, 3]

# Not recommended
my_list = [ 1, 2, 3, ]

Before a comma, semicolon, or colon:

x = 5
y = 6

# Recommended
print(x, y)

# Not recommended
print(x , y)

Before the open parenthesis that starts the argument list of a function call:

def double(x):
    return x * 2

# Recommended
double(3)

# Not recommended
double (3)

Before the open bracket that starts an index or slice:

# Recommended
list[3]

# Not recommended
list [3]

Between a trailing comma and a closing parenthesis:

# Recommended
tuple = (1,)

# Not recommended
tuple = (1, )

To align assignment operators:

# Recommended
var1 = 5
var2 = 6
some_long_var = 7

# Not recommended
var1          = 5
var2          = 6
some_long_var = 7

Make sure that there is no trailing whitespace anywhere in your code. There are other cases where PEP 8 discourages adding extra whitespace, such as immediately inside brackets, as well as before commas and colons. You should also never add extra whitespace in order to align operators.

Programming Recommendations

“Simple is better than complex.”

— The Zen of Python

You will often find that there are several ways to perform a similar action in Python (and any other programming language for that matter). In this section, you’ll see some of the suggestions PEP 8 provides to remove that ambiguity and preserve consistency.

Don’t compare boolean values to True or False using the equivalence operator. You’ll often need to check if a boolean value is True or False. When doing so, it is intuitive to do this with a statement like the one below:

# Not recommended
my_bool = 6 > 5
if my_bool == True:
    return '6 is bigger than 5'

The use of the equivalence operator, ==, is unnecessary here. bool can only take values True or False. It is enough to write the following:

# Recommended
if my_bool:
    return '6 is bigger than 5'

This way of performing an if statement with a boolean requires less code and is simpler, so PEP 8 encourages it.

Use the fact that empty sequences are falsy in if statements. If you want to check whether a list is empty, you might be tempted to check the length of the list. If the list is empty, it’s length is 0 which is equivalent to False when used in an if statement. Here’s an example:

# Not recommended
my_list = []
if not len(my_list):
    print('List is empty!')

However, in Python any empty list, string, or tuple is falsy. We can therefore come up with a simpler alternative to the above:

# Recommended
my_list = []
if not my_list:
    print('List is empty!')

While both examples will print out List is empty!, the second option is simpler, so PEP 8 encourages it.

Use is not rather than not ... is in if statements. If you are trying to check whether a variable has a defined value, there are two options. The first is to evaluate an if statement with x is not None, as in the example below:

# Recommended
if x is not None:
    return 'x exists!'

A second option would be to evaluate x is None and then have an if statement based on not the outcome:

# Not recommended
if not x is None:
    return 'x exists!'

While both options will be evaluated correctly, the first is simpler, so PEP 8 encourages it.

Don’t use if x: when you mean if x is not None:. Sometimes, you may have a function with arguments that are None by default. A common mistake when checking if such an argument, arg, has been given a different value is to use the following:

# Not Recommended
if arg:
    # Do something with arg...

This code checks that arg is truthy. Instead, you want to check that arg is not None, so it would be better to use the following:

# Recommended
if arg is not None:
    # Do something with arg...

The mistake being made here is assuming that not None and truthy are equivalent. You could have set arg = []. As we saw above, empty lists are evaluated as falsy in Python. So, even though the argument arg has been assigned, the condition is not met, and so the code in the body of the if statement will not be executed.

Use .startswith() and .endswith() instead of slicing. If you were trying to check if a string word was prefixed, or suffixed, with the word cat, it might seem sensible to use list slicing. However, list slicing is prone to error, and you have to hardcode the number of characters in the prefix or suffix. It is also not clear to someone less familiar with Python list slicing what you are trying to achieve:

# Not recommended
if word[:3] == 'cat':
    print('The word starts with "cat"')

However, this is not as readable as using .startswith():

# Recommended
if word.startswith('cat'):
    print('The word starts with "cat"')

Similarly, the same principle applies when you’re checking for suffixes. The example below outlines how you might check whether a string ends in jpg:

# Not recommended
if file_name[-3:] == 'jpg':
    print('The file is a JPEG')

While the outcome is correct, the notation is a bit clunky and hard to read. Instead, you could use .endswith() as in the example below:

# Recommended
if file_name.endswith('jpg'):
    print('The file is a JPEG')

As with most of these programming recommendations, the goal is readability and simplicity. In Python, there are many different ways to perform the same action, so guidelines on which methods to chose are helpful.

When to Ignore PEP 8

The short answer to this question is never. If you follow PEP 8 to the letter, you can guarantee that you’ll have clean, professional, and readable code. This will benefit you as well as collaborators and potential employers.

However, some guidelines in PEP 8 are inconvenient in the following instances:

If complying with PEP 8 would break compatibility with existing software
If code surrounding what you’re working on is inconsistent with PEP 8
If code needs to remain compatible with older versions of Python

Tips and Tricks to Help Ensure Your Code Follows PEP 8

There is a lot to remember to make sure your code is PEP 8 compliant. It can be a tall order to remember all these rules when you’re developing code. It’s particularly time consuming to update past projects to be PEP 8 compliant. Luckily, there are tools that can help speed up this process. There are two classes of tools that you can use to enforce PEP 8 compliance: linters and autoformatters.

Linters

Linters are programs that analyze code and flag errors. They provide suggestions on how to fix the error. Linters are particularly useful when installed as extensions to your text editor, as they flag errors and stylistic problems while you write. In this section, you’ll see an outline of how the linters work, with links to the text editor extensions at the end.

The best linters for Python code are the following:

pycodestyle is a tool to check your Python code against some of the style conventions in PEP 8.

Install pycodestyle using pip:

$ pip install pycodestyle

You can run pycodestyle from the terminal using the following command:

$ pycodestyle code.py
code.py:1:17: E231 missing whitespace after ','
code.py:2:21: E231 missing whitespace after ','
code.py:6:19: E711 comparison to None should be 'if cond is None:'

flake8 is a tool that combines a debugger, pyflakes, with pycodestyle.

Install flake8 using pip:

$ pip install flake8

Run flake8 from the terminal using the following command:

$ flake8 code.py
code.py:1:17: E231 missing whitespace after ','
code.py:2:21: E231 missing whitespace after ','
code.py:3:17: E999 SyntaxError: invalid syntax
code.py:6:19: E711 comparison to None should be 'if cond is None:'

An example of the output is also shown.

Note: The extra line of output indicates a syntax error.

These are also available as extensions for Atom, Sublime Text, Visual Studio Code, and VIM. You can also find guides on setting up Sublime Text and VIM for Python development, as well as an overview of some popular text editors at Real Python.

Autoformatters

Autoformatters are programs that refactor your code to conform with PEP 8 automatically. Once such program is black, which autoformats code following most of the rules in PEP 8. One big difference is that it limits line length to 88 characters, rather than 79. However, you can overwrite this by adding a command line flag, as you’ll see in an example below.

Install black using pip. It requires Python 3.6+ to run:

$ pip install black

It can be run via the command line, as with the linters. Let’s say you start with the following code that isn’t PEP 8 compliant in a file called code.py:

for i in range(0,3):
    for j in range(0,3):
        if (i==2):
            print(i,j)

You can then run the following command via the command line:

$ black code.py
reformatted code.py
All done! ✨ 🍰 ✨

code.py will be automatically reformatted to look like this:

for i in range(0, 3):
    for j in range(0, 3):
        if i == 2:
            print(i, j)

If you want to alter the line length limit, then you can use the --line-length flag:

$ black --line-length=79 code.py
reformatted code.py
All done! ✨ 🍰 ✨

Two other autoformatters, autopep8 and yapf, perform actions that are similar to what black does.

Another Real Python tutorial, Python Code Quality: Tools & Best Practices by Alexander van Tol, gives a thorough explanation of how to use these tools.

Conclusion

You now know how to write high-quality, readable Python code by using the guidelines laid out in PEP 8. While the guidelines can seem pedantic, following them can really improve your code, especially when it comes to sharing your code with potential employers or collaborators.

In this tutorial, you learned:

What PEP 8 is and why it exists
Why you should aim to write PEP 8 compliant code
How to write code that is PEP 8 compliant

On top of all this, you also saw how to use linters and autoformatters to check your code against PEP 8 guidelines.

If you want to learn more about PEP 8, then you can read the full documentation, or visit pep8.org, which contains the same information but has been nicely formatted. In these documents, you will find the rest of the PEP 8 guidelines not covered in this tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

A Pythonista's Holiday Wish List

2018-12-17T14:00:00+00:00

It’s that time of year again when everyone is looking to get last minute gifts. Whether you’re a friend of a Python developer or one yourself, I’ve got the perfect wish list if you’re looking to get a little something special. I’ve combed through the interwebs and polled my fellow Real Python authors to find the best presents for Pythonistas.

I’ve broken up these suggestions into five categories:

Good causes
Learning resources
Python swag
Hardware
Gadgets

Donating to a Good Cause

The greatest gift that anyone can receive is the feeling of giving to someone else. Whether you want to make a donation yourself or make one in someone’s name, here are some great Python related organizations that could use a donation:

Python Software Foundation (PSF): the official organization that maintains Python and hosts the Python Conference each year
Django Software Foundation (DSF): the official organization that maintains Django, our favorite web app framework
PyLadies: an organization dedicated to helping more women become participants and leaders in the Python open-source community

Donating your money isn’t the only thing that you can do to help the Python Community. You can also donate your time as a volunteer.

Python Learning Resources

I think that one of the greatest gifts that can be given is knowledge. Want to get a good read that’s Python related? Check out The Best Python Books for a great collection of Python related books. Maybe you want something a little more interactive? Check out the courses at Real Python!

One of the best ways to learn about Python is to learn from the masters themselves. A ticket to PyCon 2019 would be the greatest learning gift of all: three days of classes, demonstrations, coding challenges, and rubbing shoulders with the people who dream in Python code! This would be the ultimate gift that would keep on giving. You’ll also get swag bags that will last you all year round.

Python Stickers, T-Shirts, and Coffee Mugs… Oh My!

We’re Pythonistas, so why not show off the programming language we love most! I love showing off my love for Python on my laptop.

Nerdlettering has got your back for all your sticker needs, and if stickers aren’t your thing, then perhaps a sweatshirt or t-shirt might be better. Also don’t forget about that old coffee mug of yours. It could probably use an upgrade as well.

Want something that sports the Real Python logo? They’ve got you covered as well.

Hardware

With the way that tech is advancing every day, every programmer needs an upgrade of their hardware. This may be because you’re running out of hard drive space, have worn out keyboards, or just want something fun. (Or maybe you want something more advanced. Everyone loves a good challenge right?) This is my recommended hardware to check out this season.

External HDD

Whether you have a massive code base or are running several hundred Docker images, you can quickly run out of space to do all your work. Good thing the cost of hard drive capacity is almost constantly dropping! Plus, with USB 3.0, it’s almost as fast or even faster than an internal hard drive. You can get 2 TB of portable space for less than $60! Need something bigger? For a little more than double the price, you can get 8 TB! I recommend sticking to the name brands like Seagate and Western Digital if you can.

A New Mechanical Keyboard

As developers, we’re using the keyboard 99% of the time we’re on the computer. That means that it’s important to have the right keyboard that feels right for you. Don’t know what to get? The folks over at the MechanicalKeyboards subreddit have created a great guide to all things mechanical keyboards.

Recommending a keyboard is like recommending underwear: you never know how it’s going to feel for anyone else, so I highly recommend that you look at the guide above. But if you want my opinion, the keyboard that I use and think feels the best is the Microsoft Sculpt Ergo Keyboard.

Headphones

Let’s face it: there’s nothing like getting into the zone with your favorite music set and crunching away at your codebase. So why not make sure that you get a good set of headphones? I love my Beyerdynamic DT 770 Pro headphones and can’t imagine coding without them. When you’re buying headphones, I recommend using this useful guide created by the helpful folks over at the headphones subreddit.

Gadgets

It’s always fun to have new toys to play with, and us Python developers have a lot to choose from. Here, we’ve collected our favorite gadgets that use Python.

Raspberry Pi

Let’s start off with a little pie—Raspberry Pi that is. There are so many things that you can do with a Raspberry Pi, including the following:

With its starting price of only $35, it’s amazing that you’re getting a multi-core, Bluetooth and WiFi enabled device! I highly recommend getting a kit to get started.

The Open Source Rover

Speaking of Raspberry Pi, how about creating your own Mars Science Lab! NASA’s Jet Propulsion Lab (JPL), the creators of the Mars Rovers, has created an Open Source replica of the Mars Science Laboratory rover that uses the Raspberry Pi.

This comes with a hefty price tag (~$2,500) and isn’t for the faint of heart as it requires purchasing each part separately and assembling the entire rover. Don’t worry about programming the rover though. That was already done by JPL in Python! If the price and difficulty don’t scare you off, then this is a great gift for anyone wanting to learn about robotics.

Vector and Cozmo Robots From Anki

If making your own robot seems a little daunting, then check out Vector and Cozmo. Vector is “the good robot” who can help out with multiple things such as a kitchen timer, taking a selfie, or telling you what the weather is going to be like. If that’s not enough, there is the Python Software Development Kit (SDK) that enables you to make Vector fully customizable!

If you’re new to programming or want to teach someone about programming robots, then Cozmo’s got you covered. With both a full Python SDK and a more simple option available, this robot is a good choice for the beginner and the seasoned coder.

DJI Drones

A little tired of being stuck on the ground? How about flying? DJI has multiple versions of their drones that you can take flight with either manually or automated using their software. If they don’t have the feature you’re looking for, how about checking out the SDK and doing it yourself! DJI has a lot of drone options available, but I’ve got my eye on the Tello, since it’s on the cheaper end and has an easier-to-use Python Library.

Happy Pythonic Holidays

Whatever you decide to get you or your Python developer friends, be sure to enjoy this holiday season and keep on Pythoning! I’m about to push checkout on my Amazon cart now and tell my wife that #PythonMadeMeDoIt.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Thonny: The Beginner-Friendly Python Editor

2018-12-12T14:00:00+00:00

Are you a Python beginner looking for a tool that can support your learning? This article is for you! Every programmer needs a place to write their code. This article will discuss an awesome tool called Thonny that will enable you to start working with Python in a beginner-friendly environment.

In this article, you’ll learn:

How to install Thonny on your computer
How to navigate Thonny’s user interface to use its built-in features
How to use Thonny to write and run your code
How to use Thonny to debug your code

By the end of this article, you’ll be comfortable with the development workflow in Thonny and ready to use it for your Python learning.

So what is Thonny? Great question!

Thonny is a free Python Integrated Development Environment (IDE) that was especially designed with the beginner Pythonista in mind. Specifically, it has a built-in debugger that can help when you run into nasty bugs, and it offers the ability to do step through expression evaluation, among other really awesome features.

Free Sample Chapter: Download a free sample chapter from the Real Python course and gain practical Python programming skills.

Installing Thonny

This article assumes that you have Python 3 installed on your computer. If not, please review Python 3 Installation & Setup.

Web Download

The web download can be accessed via a web browser by visiting the Thonny website. Once on the page, you will see a light gray box in the top right corner like this:

Once you’ve found the gray box, click the appropriate link for your operating system. This tutorial assumes you’ve downloaded version 3.0.1.

Command Line Download

You can also install Thonny via your system’s command line. On Windows, you can do this by starting a program called Command Prompt, while on macOS and Linux you start a program called Terminal. Once you’ve done that, enter the following command:

$ pip install thonny

The User Interface

Let’s make sure you understand what Thonny has to offer. Think of Thonny as the workroom in which you will create amazing Python projects. Your workroom contains a toolbox containing many tools that will enable you to be a rock star Pythonista. In this section, you’ll learn about each of the features of the UI that’ll help you use each of the tools in your Thonny toolbox.

The Code Editor and Shell

Now that you have Thonny installed, open the application. You should see a window with several icons across the top, and two white areas:

Notice the two main sections of the window. The top section is your code editor, where you will write all of your code. The bottom half is your Shell, where you will see outputs from your code.

The Icons

Across the top you’ll see several icons. Let’s explore what each of them does. You’ll see an image of the icons below, with a letter above each one. We will use these letters to talk about each of the icons:

Working our way from left to right, below is a description of each of the icons in the image.

A: The paper icon allows you to create a new file. Typically in Python you want to separate your programs into separate files. You’ll use this button later in the tutorial to create your first program in Thonny!

B: The open folder icon allows you to open a file that already exists on your computer. This might be useful if you come back to a program that you worked on previously.

C: The floppy disk icon allows you to save your code. Press this early and often. You’ll use this later to save your first Thonny Python program.

D: The play icon allows you to run your code. Remember that the code you write is meant to be executed. Running your code means you’re telling Python, “Do what I told you to do!” (In other words, “Read through my code and execute what I wrote.”)

E: The bug icon allows you to debug your code. It’s inevitable that you will encounter bugs when you’re writing code. A bug is another word for a problem. Bugs can come in many forms, sometimes appearing when you use inappropriate syntax and sometimes when your logic is incorrect.

Thonny’s bug button is typically used to spot and investigate bugs. You’ll work with this later in the tutorial. By the way, if you’re wondering why they’re called bugs, there’s also a fun story of how it came about!

F-H: The arrow icons allow you to run your programs step by step. This can be very useful when you’re debugging or, in other words, trying to find those nasty bugs in your code. These icons are used after you press the bug icon. You’ll notice as you hit each arrow, a yellow highlighted bar will indicate which line or section Python is currently evaluating:

The F arrow tells Python to take a big step, meaning jumping to the next line or block of code.
The G arrow tells Python to take a small step, meaning diving deep into each component of an expression.
The H arrow tells Python to exit out of the debugger.

I: The resume icon allows you to return to play mode from debug mode. This is useful in the instance when you no longer want to go step by step through the code, and instead want your program to finish running.

J: The stop icon allows you to stop running your code. This can be particularly useful if, let’s say, your code runs a program that opens a new window, and you want to stop that program. You’ll use the stop icon later in the tutorial.

Let’s Try It!

Get ready to write your first official Python program in Thonny:

Enter the following code into the code editor:
```
print("Hello World")
```
Click the play button to run your program.
See the output in the Shell window.
Click the play button again to see that it says hello one more time.

Congratulations! You’ve now completed your first program in Thonny! You should see Hello world! printed inside the Shell, also known as the console. This is because your program told Python to print this phrase, and the console is where you see the output of this execution.

Other UI Features

To see more of the other features that Thonny has to offer, navigate to the menu bar and select the View dropdown. You should see that Shell has a check mark next to it, which is why you see the Shell section in Thonny’s application window:

Let’s explore some of the other offerings, specifically those that will be useful to a beginning Pythonista:

Help: You’ll select the Help view if you want more information about working with Thonny. Currently this section offers more reading on the following topics: Running Programs Step-wise, how to install 3rd Party Packages, or using Scientific Python Packages.
Variables: This feature can be very valuable. A variable in Python is a value that you define in code. Variables can be numbers, strings, or other complex data structures. This section allows you to see the values assigned to all of the variables in your program.
Assistant: The Assistant is there to give you helpful hints when you hit Exceptions or other types of errors.

The other features will become useful as you advance your skills. Check them out once you get more comfortable with Thonny!

The Code Editor

Now that you have an understanding of the UI, let’s use Thonny to write another little program. In this section, you’ll go through the features of Thonny that will help guide you through your development workflow.

Write Some Code

In the code editor (top portion of the UI), add the following function:

def factorial(num):
    if num == 1:
        return 1
    else:
        return num * factorial(num - 1)

print(factorial(3))

Save Your Code

Before we move on, let’s save your program. Last time, you were prompted to do this after pressing the play button. You can also do this by clicking the blue floppy disk icon or by going to the menu bar and selecting File > Save. Let’s call the program factorial.py.

Run Your Code

In order to run your code, find and press the play icon. The output should look like this:

Debug Your Code

To truly understand what this function is doing, try the step feature. Take a few large and small steps through the function to see what is happening. Remember you can do this by pressing the arrow icons:

As you can see, the steps will show how the computer is evaluating each part of the code. Each pop up window is like a piece of scratch paper that the computer is using to compute each portion of the code. Without this awesome feature, this may have been hard to conceptualize—but now you’ve got it!

Stop Running Your Code

So far, there hasn’t been a need to hit the stop icon for this program, particularly because it exits as soon as it has executed print(). Try increasing the number being passed to the factorial function to 100:

def factorial(num):
    if num == 1:
        return 1
    else:
        return num * factorial(num - 1)

print(factorial(100))

Then step through the function. After a while, you will notice that you will be clicking for a long time to reach the end. This is a good time to use the stop button. The stop button can be really useful to stop a program that is either intentionally or unintentionally running continuously.

Find Syntax Errors in Your Code

Now that you have a simple program that works, let’s break it! By intentionally creating an error in your factorial program, you’ll be able to see how Thonny handles these types of issues.

We will be creating what is called a syntax error. A syntax error is an error that indicates that your code is syntactically incorrect. In other words, your code does not follow the proper way to write Python. When Python notices the error, it will display a syntax error to complain about your invalid code.

Above the print statement, let’s add another print statement that says print("The factorial of 100 is:"). Now let’s go ahead and create syntax errors. In the first print statement, remove the second quotation mark, and in the other remove the second parenthesis.

As you do this, you should see that Thonny will highlight your SyntaxErrors. Missing quotations are highlighted in green, and missing parenthesis are in gray:

For beginners, this is a great resource that will allow you to help spot any typos while you’re writing. Some of the most common and frustrating errors when you start programming are missing quotes and mismatched parentheses.

If you have your Assistant View turned on, you will also notice that it will give you a helpful message to guide you in the right direction when you are debugging:

As you get more comfortable with Thonny, the Assistant can be a useful tool to help you get unstuck!

The Package Manager

As you continue to learn Python, it can be quite useful to download a Python package to use inside of your code. This allows you to use code that someone else has written inside of your program.

Consider an example where you want to do some calculations in your code. Instead of writing your own calculator, you might want to use a third-party package called simplecalculator. In order to do this, you’ll use Thonny’s package manager.

The package manager will allow you to install packages that you will need to use with your program. Specifically, it allows you to add more tools to your toolbox. Thonny has the built-in benefit of handling any conflicts with other Python interpreters.

To access the package manager, go to the menu bar and select Tools > Manage Packages… This should pop open a new window with a search field. Type simplecalculator into that field and click the Search button.

The output should look similar to this:

Go ahead and click Install to install this package. You will see a small window pop up showing the system’s logs while it installs the package. Once it completes, you are ready to use simplecalculator in your code!

In the next section, you will use the simplecalculator package along with some of the other skills you’ve learned in this tutorial to create a simple calculator program.

Check Your Understanding

You’ve learned so much about Thonny so far! Here’s what you’ve learned:

Where to write your code
How to save your code
How to run your code
How to stop your code from running
Where to see your code execute
How to spot SyntaxErrors
How to install third party packages

Let’s check your understanding of these concepts.

Now that you have simplecalculator installed, let’s create a simple program that will use this package. You’ll also use this as an opportunity to check that you understand some of the UI and development features that you’ve learned thus far in the tutorial.

Part 1: Create a File, Add Some Code, and Understand the Code

In Part 1, you will create a file, and add some code to it! Do your best to try to dig into what the code is actually doing. If you get stuck, check out the Take a Deeper Look window. Let’s get started:

Start a new file.
Add the following code into your Thonny code editor:

 1 from calculator.simple import SimpleCalculator
 2 
 3 my_calculator = SimpleCalculator()  
 4 my_calculator.run('2 * 2')
 5 print(my_calculator.lcd)

This code will print out the result of 2 * 2 to the Thonny Shell in the main UI. To understand what each part of the code is doing, check out the Take a Deeper Look section below.

Line 1: This code imports the library calculator inside of the package called simplecalculator. From this library, we import the class called SimpleCalculator from a file called simple.py. You can see the code here.
Lines 2: This is a blank line behind code blocks, which is generally a preferred style. Read more about Python Code Quality in this article.
Line 3: Here we create an instance of the class SimpleCalculator and assign it to a variable called my_calculator. This can be used to run different calculators. If you’re new to classes, you can learn more about object-oriented programming here.
Line 4: Here we have the calculator run the operation 2 * 2 by calling run() and passing in the expression as a string.
Line 5: Here we print the result of the calculation. You’ll notice in order to get the most recent calculation result, we must access the attribute called lcd.

Great! Now that you know exactly what your calculator code is doing, let’s move on to running this code!

Part 2: Save the File, View the Variables, and Run Your Code

Now it’s time to save and run your code. In this section, you’ll make use of two of the icons we reviewed earlier:

Save your new file as calculations.py.
Open the Variables window and make note of the two variables listed. You should see SimpleCalculator and my_calculator. This section also gives you insight into the value that each variable is pointing to.
Run your code! You should see 4.0 in the output:

Great job! Next you’ll explore how Thonny’s debugger can help you to better understand this code.

Other Great Beginner Features

As you get more comfortable with Thonny, the features in this section will come in quite handy.

Debugging

Using your calculations.py script, you’re going to use the debugger to investigate what is happening. Update your code in calculations.py to the following:

from calculator.simple import SimpleCalculator

def create_add_string(x, y):
    '''Returns a string containing an addition expression.'''
    return 'x + y'

my_calculator = SimpleCalculator()  
my_calculator.run(create_add_string(2, 2))
print(my_calculator.lcd)

Hit the save icon to save this version.

You’ll notice the code has a new function called create_add_string(). If you’re unfamiliar with Python functions, learn more in this awesome Real Python course!

As you inspect the function, you may notice why this script will not work as expected. If not, that’s okay! Thonny is going to help you see exactly what is going on, and squash that bug! Go ahead and run your program and see what happens. The Shell output should be the following:

>>> %Run calculations.py
0

Oh no! Now you can see there is a bug in your program. The answer should be 4! Next, you’ll use Thonny’s debugger to find the bug.

Let’s Try It!

Now that we have a bug in our program, this is a great chance to use Thonny’s debugging features:

Click the bug icon at the top of the window. This enters debugger mode.
You should see the import statements highlighted. Click the small step arrow icon, the yellow arrow in the middle. Keep pressing this to see how the debugger works. You should notice that it highlights each step that Python takes to evaluate your program. Once it hits create_add_string(), you should see a new window pop up.
Examine the pop up window carefully. You should see that it shows the values for x and y. Keep pressing the small step icon until you see the value that Python will return to your program. It will be enclosed in a light-blue box: Oh no! There’s the bug! It looks like Python will return a string containing the letters x and y (meaning 'x + y' and not a string containing the values of those variables, like '2 + 2', which is what the calculator is expecting.) Each time you see a light-blue box, you can think of this as Python replacing subexpressions with their values, step by step. The pop up window can be thought of as a piece of scratch paper that Python uses to figure out those values. Continue to step through the program to see how this bug results in a calculation of 0.
The bug here has to do with string formatting. If you are unfamiliar with string formatting, check out this article on Python String Formatting Best Practices. Inside create_add_string(), the f-string formatting method should be used. Update this function to the following:
```
def create_add_string(x, y):
    '''Returns a string containing an addition expression.'''
    return f'{x} + {y}'
```
Run your program again. You should see the following output:
```
>>> %Run calculations.py
4.0
```

Success! You have just demonstrated how the step-by-step debugger can help you find a problem in your code! Next you’ll learn about some other fun Thonny features.

Variable Scope Highlighting

Thonny offers variable highlighting to remind you that the same name doesn’t always mean the same variable. In order for this feature to work, on the menu bar, go to Thonny > Preferences and ensure that Highlight matching names is checked.

Notice in the code snippet below, that create_add_string() now has a new variable called my_calculator, though this is not the same as the my_calculator on lines 10 and 11. You should be able to tell because Thonny highlights the variables that reference the same thing. This my_calculator inside the function only exists within the scope of that function, which is why it is not highlighted when the cursor is on the other my_calculator variable on line 10:

This feature can really help you avoid typos and understand the scope of your variables.

Code Completion

Thonny also offers code completion for APIs. Notice in the snapshot below how pressing the Tab key shows the methods available from the random library:

This can be very useful when you’re working with libraries and don’t want to look at the documentation to find a method or attribute name.

Working on a Pre-Existing Project

Now that you’ve learned the basic features of Thonny, let’s explore how you can use it to work on a pre-existing project.

Find a File on Your Computer

Opening a file on your computer is as easy as going to the menu bar, selecting File > Open, and using your browser to navigate to the file. You can also use the open folder icon at the top of the screen to do this as well.

If you have a requirements.txt file and pip locally installed, you can pip install these from the Thonny system Shell. If you don’t have pip installed, remember you can use the Package Manager to install it:

$ pip install -r requirements.txt

Work on a Project From Github

Now that you are a Thonny expert, you can use it to work on the exercises from Real Python Course 1: Introduction to Python:

Navigate to the Real Python GitHub repo called book1-exercises.
Click the green button labeled Clone or download and select Download Zip.
Click the opening folder icon to navigate and find the downloaded files. You should find a folder called book1-exercises1.
Open one of the files and start working!

This is useful because there are tons of cool projects available on GitHub!

Conclusion

Awesome job getting through this tutorial on Thonny!

You can now start using Thonny to write, debug, and run Python code! If you like Thonny, you might also like some of the other IDEs we’ve listed in Python IDEs and Code Editors (Guide).

Thonny is actively maintained, and new features are being added all the time. There are several awesome new features that are currently in beta that can be found on the Thonny Blog. Thonny’s main development takes place at the Institute of Computer Science of the University of Tartu, Estonia, as well as by contributors around the world.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Brian Peterson

2018-12-10T14:00:00+00:00

To date, I’ve interviewed people you’ve likely heard of before from the Python community. But this column isn’t just about interviewing the rock stars and core devs. It’s also a means to shine light on the huge contributions to the community that can often go unthanked and overlooked. As such, I present to you Brian Peterson.

Brian is a project manager by day, and by night he’s one of the moderators of the Pythonista Café, a peer-to-peer learning community for Pythonistas. In our interview, we talk about how Python helps him in his role as a project manager, and how moderating a large forum for Python enthusiasts has impacted his coding chops. Let’s dig in!

Ricky: Welcome to Real Python! Let’s get started with the question I ask everyone. How did you get into programming, and when did you start using Python?

Brian: Well… thinking back, my first real exposure to programming was testing near object detection system prototypes using an old HP-87. The program was used to move and change target and rotating antenna positions along test tracks inside an automotive laboratory anechoic chamber, all while collecting and processing data from a spectrum analyzer. Seeing the code translate into motion did it for me—something real, and tangible. I was hooked on using programming as an R&D tool to make things come alive.

Over the years, I started spending more time with Linux automation, control systems, data collection and analysis, which naturally led to spending more time writing C code, and then it was like, “Hey, Python is already on the system. Why not take it for a spin?”

I think it was Python 2.3 at that time. I liked Python because it felt natural, and I didn’t have to throw away all my C knowledge. It had some functional programming sprinkled in, and an impressive set of engineering and scientific libraries. To top it off, I could use the same language seamlessly across scripting, interactive data analysis, and writing code. And Python syntax, well, it can be so darned readable that personally, I think it’s hard not to like it. I was hooked.

Ricky: Working in project management and using Python might not seem a natural fit for some people. How do you use Python to help you in the project management office, and what tools and libraries are the most helpful for you on a day-to-day basis?

Brian: Yeah, at first glance Python may not seem like a natural fit. This is in large part due to the way project management is commonly taught—like cooking from a recipe book rather than learning and understanding the culinary arts.

And cookie-cutter project management tools are like the microwave ovens. Yeah, sure, use ‘em for the prepacked, boxed, canned stuff, but for every meal? Seriously? Python also starts making sense when you’re considering different flavors of project management:

Traditional project managers use template-driven approaches.
Agile practitioners use story-driven approaches.
Adaptive practitioners blend existing, new, emerging approaches and creativity to develop the best fit for each unique project or group of projects.

So while off-the-shelf project management tools and services are an excellent fit for traditional projects, and a good fit for some Agile projects, adaptive project management calls for a different breed. In other words, rather than force all projects to fit a tool, why not use Python to construct a toolbox filled with best-of-breed tools to solve different types of problems for different kinds of projects?

It’s also worth considering that modern day Project Management Offices (PMO) don’t just manage projects: there’s ideation, portfolio management, road mapping, strategic planning, KPI monitoring, communications, governance, dashboards, stakeholder portals, working closely with DevOps, business analysts, architects, developers, UX designers, CX folks, subject matter experts, other technologists, and the list goes on and on.

Now imagine a vendor who knows nothing about your particular projects or PMO services with a big box of Legos gluing pieces into place and selling it to you. Yeah, it’s probably a nice contraption, or it wouldn’t be on the market for long.

But compare it to owning your own big box of Legos (Python and goodies), where you can add Legos to pre-built contraptions, use Legos to bridge multiple contraptions together, use them with existing Legos already lying around the organization, or build your own and mix and match any or all of the above.

And to answer the second part of the question, I currently use Python and RESTful APIs to interact with a variety of flexible SaaS tools, for example, ProductPlan, Smartsheet (with the dashboard extension), Airtable, and a commercial ticketing system, and enterprise visualization software. Python makes it easy to wire these tools together in creative ways and fill gaps in functionality to solve many different kinds of problems.

Ricky: How have those tools changed the way you work? And how do you see them changing going forward? Will Python continue to play a role in the future of project management offices?

Brian: I think the most significant change is a sense of freedom, the ability to creatively solve problems, focus on what’s essential, and leverage modern solutions rather than being chained to tools, falling into tool ruts, or suffering from tool rot. As an added bonus, Python makes project management fun and refreshing.

I’m not sure that Python will become mainstream in project management, but I do think it can give project managers who learn it a unique advantage in the automation of tedious tasks, solving complex problems, adapting to the evolution of organizations, and providing valuable outside-the-box PMO services.

Ricky: Do you have any Python or tech related side projects you’re working on right now?

Brian: Yes, Road Maps to the Future is a project in partnership with a network of volunteers and in cooperation with agencies to help pilot it. With over 80,000 local governments in the United States alone, I believe there needs to be a better way to map out where we are, where we are going in the future, and the best ways to get there.

In Python, we have an amazing assortment of tools for doing fantastic things with data, and we’re getting better at collecting discrete data.

But there are opportunities in the exploration and non-linear collection of ideas, the identification of unseen problems, gaps in services, areas for improvement in the overall customer experience, the development of creative and viable alternative solutions to these problems, and then translating them into successful projects with demonstrable benefits.

The road maps to the future are not just compilations of random ideas with new ways to collect information, but cohesive collections of core elements, processes, and partnerships necessary to propel the ideas into action through the organization of projects.

To make it sustainable, it has to be discoverable, simple, quick and fun, so a lot of extra focus is being put into the UX side of the project. The current design uses Python, Vue.js, Azure Cosmos DB, Service Bus and Cognitive Services. It’s still in early development. Project information, details, and code will be released into the public domain next year.

Ricky: One of the reasons I was keen to interview you is because of your contribution to the Python community. You are one of two moderators in the Pythonista Café, a peer-to-peer learning community for Python enthusiasts. How has your journey been from being a member to a moderator, and how has it influenced your Python chops?

Brian: It’s been great! Dan Bader has done such a fantastic job, along with you, Jennifer (co-moderator), the Real Python team, and especially all the amazing contributors from a wide range of backgrounds, geographical locations, and levels of experience who have made the Café into the lively forum it is today.

It really is truly community-driven from the ground up. It has a fun, productive, open-source vibe without fear of public shaming. It’s okay to make mistakes and learn. You can post stuff without having to worry about how the whole world will interpret it or misinterpret it.

I initially joined the Café for several reasons. It’s like a non-cliquey 24/7/365 Python meetup that you can drop in and out of at any time. As forum members have mentioned, it has a much better signal-to-noise ratio than the public forums, not to mention freedom from ads and social media algorithms that manipulate behaviors.

It’s very welcoming and ultra-Python friendly. The passion for programming is seriously contagious, which leads to your question on how it has influenced my Python chops. The answer is that it has made Python even more fun, and I’m now even more committed and focused to continuous learning in Python and its ecosystem. It has also increased my excitement about the possibilities of what can be done with Python in the future.

Ricky: Now for my last question. What other hobbies and interests do you have, aside from Python? Any you’d like to share and/or plug?

Brian: Outside of work and Python, I like to mix it up a lot. I currently work in the Panhandle and spend my off-time on the First Coast, so lately one of my favorite things to do is wandering around the south kayaking the many rivers and swamps; under certain conditions, it’s absolutely surreal. You can read and watch all you want about swamps, but there’s no substitute for the real thing.

Each year I also pick out one or two new subjects to research and study for fun. I am an amateur radio operator, I love music, and I’m an avid reader of books, particularly history, survival stories, and science fiction.

I’m also blessed to have an amazing wife and son. Outside of my projects, my wife keeps things lively with home projects, going to festivals and other events, helping animal rescues, along with exploring new areas and hiking with the dogs. Boredom is an alien concept—it definitely keeps life fun.

Thank you, Brian, for joining me this week. If you’re looking for a friendly Python place to hang your metaphorical hat, you can find the Pythonista Café here. Just be sure to say hi to Brian when you get there.

Do you know an unsung hero in the Python community? If they’d like me to interview them in the future, I can be reached in the comments below, or you can send me a message on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Sending Emails With Python

2018-12-05T14:00:00+00:00

You probably found this tutorial because you want to send emails using Python. Perhaps you want to receive email reminders from your code, send a confirmation email to users when they create an account, or send emails to members of your organization to remind them to pay their dues. Sending emails manually is a time-consuming and error-prone task, but it’s easy to automate with Python.

In this tutorial you’ll learn how to:

Set up a secure connection using SMTP_SSL() and .starttls()
Use Python’s built-in smtplib library to send basic emails
Send emails with HTML content and attachments using the email package
Send multiple personalized emails using a CSV file with contact data
Use the Yagmail package to send email through your Gmail account using only a few lines of code

You’ll find a few transactional email services at the end of this tutorial, which will come in useful when you want to send a large number of emails.

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

Getting Started

Python comes with the built-in smtplib module for sending emails using the Simple Mail Transfer Protocol (SMTP). smtplib uses the RFC 821 protocol for SMTP. The examples in this tutorial will use the Gmail SMTP server to send emails, but the same principles apply to other email services. Although the majority of email providers use the same connection ports as the ones in this tutorial, you can run a quick Google search to confirm yours.

To get started with this tutorial, set up a Gmail account for development, or set up an SMTP debugging server that discards emails you send and prints them to the command prompt instead. Both options are laid out for you below. A local SMTP debugging server can be useful for fixing any issues with email functionality and ensuring your email functions are bug-free before sending out any emails.

Option 1: Setting up a Gmail Account for Development

If you decide to use a Gmail account to send your emails, I highly recommend setting up a throwaway account for the development of your code. This is because you’ll have to adjust your Gmail account’s security settings to allow access from your Python code, and because there’s a chance you might accidentally expose your login details. Also, I found that the inbox of my testing account rapidly filled up with test emails, which is reason enough to set up a new Gmail account for development.

A nice feature of Gmail is that you can use the + sign to add any modifiers to your email address, right before the @ sign. For example, mail sent to my+person1@gmail.com and my+person2@gmail.com will both arrive at my@gmail.com. When testing email functionality, you can use this to emulate multiple addresses that all point to the same inbox.

To set up a Gmail address for testing your code, do the following:

Create a new Google account.
Turn Allow less secure apps to ON. Be aware that this makes it easier for others to gain access to your account.

If you don’t want to lower the security settings of your Gmail account, check out Google’s documentation on how to gain access credentials for your Python script, using the OAuth2 authorization framework.

Option 2: Setting up a Local SMTP Server

You can test email functionality by running a local SMTP debugging server, using the smptd module that comes pre-installed with Python. Rather than sending emails to the specified address, it discards them and prints their content to the console. Running a local debugging server means it’s not necessary to deal with encryption of messages or use credentials to log in to an email server.

You can start a local SMTP debugging server by typing the following in Command Prompt:

$ python -m smtpd -c DebuggingServer -n localhost:1025

On Linux, use the same command preceded by sudo.

Any emails sent through this server will be discarded and shown in the terminal window as a bytes object for each line:

---------- MESSAGE FOLLOWS ----------
b'X-Peer: ::1'
b''
b'From: my@address.com'
b'To: your@address.com'
b'Subject: a local test mail'
b''
b'Hello there, here is a test email'
------------ END MESSAGE ------------

For the rest of the tutorial, I’ll assume you’re using a Gmail account, but if you’re using a local debugging server, just make sure to use localhost as your SMTP server and use port 1025 rather than port 465 or 587. Besides this, you won’t need to use login() or encrypt the communication using SSL/TLS.

Sending a Plain-Text Email

Before we dive into sending emails with HTML content and attachments, you’ll learn to send plain-text emails using Python. These are emails that you could write up in a simple text editor. There’s no fancy stuff like text formatting or hyperlinks. You’ll learn that a bit later.

Starting a Secure SMTP Connection

When you send emails through Python, you should make sure that your SMTP connection is encrypted, so that your message and login credentials are not easily accessed by others. SSL (Secure Sockets Layer) and TLS (Transport Layer Security) are two protocols that can be used to encrypt an SMTP connection. It’s not necessary to use either of these when using a local debugging server.

There are two ways to start a secure connection with your email server:

Start an SMTP connection that is secured from the beginning using SMTP_SSL().
Start an unsecured SMTP connection that can then be encrypted using .starttls().

In both instances, Gmail will encrypt emails using TLS, as this is the more secure successor of SSL. As per Python’s Security considerations, it is highly recommended that you use create_default_context() from the ssl module. This will load the system’s trusted CA certificates, enable host name checking and certificate validation, and try to choose reasonably secure protocol and cipher settings.

If you want to check the encryption for an email in your Gmail inbox, go to More → Show original to see the encryption type listed under the Received header.

smtplib is Python’s built-in module for sending emails to any Internet machine with an SMTP or ESMTP listener daemon.

I’ll show you how to use SMTP_SSL() first, as it instantiates a connection that is secure from the outset and is slightly more concise than the .starttls() alternative. Keep in mind that Gmail requires that you connect to port 465 if using SMTP_SSL(), and to port 587 when using .starttls().

Option 1: Using `SMTP_SSL()`

The code example below creates a secure connection with Gmail’s SMTP server, using the SMTP_SSL() of smtplib to initiate a TLS-encrypted connection. The default context of ssl validates the host name and its certificates and optimizes the security of the connection. Make sure to fill in your own email address instead of my@gmail.com:

import smtplib, ssl

port = 465  # For SSL
password = input("Type your password and press enter: ")

# Create a secure SSL context
context = ssl.create_default_context()

with smtplib.SMTP_SSL("smtp.gmail.com", port, context=context) as server:
    server.login("my@gmail.com", password)
    # TODO: Send email here

Using with smtplib.SMTP_SSL() as server: makes sure that the connection is automatically closed at the end of the indented code block. If port is zero, or not specified, .SMTP_SSL() will use the standard port for SMTP over SSL (port 465).

It’s not safe practice to store your email password in your code, especially if you intend to share it with others. Instead, use input() to let the user type in their password when running the script, as in the example above. If you don’t want your password to show on your screen when you type it, you can import the getpass module and use .getpass() instead for blind input of your password.

Option 2: Using `.starttls()`

Instead of using .SMTP_SSL() to create a connection that is secure from the outset, we can create an unsecured SMTP connection and encrypt it using .starttls().

To do this, create an instance of smtplib.SMTP, which encapsulates an SMTP connection and allows you access to its methods. I recommend defining your SMTP server and port at the beginning of your script to configure them easily.

The code snippet below uses the construction server = SMTP(), rather than the format with SMTP() as server: which we used in the previous example. To make sure that your code doesn’t crash when something goes wrong, put your main code in a try block, and let an except block print any error messages to stdout:

import smtplib, ssl

smtp_server = "smtp.gmail.com"
port = 587  # For starttls
sender_email = "my@gmail.com"
password = input("Type your password and press enter: ")

# Create a secure SSL context
context = ssl.create_default_context()

# Try to log in to server and send email
try:
    server = smtplib.SMTP(smtp_server,port)
    server.ehlo() # Can be omitted
    server.starttls(context=context) # Secure the connection
    server.ehlo() # Can be omitted
    server.login(sender_email, password)
    # TODO: Send email here
except Exception as e:
    # Print any error messages to stdout
    print(e)
finally:
    server.quit()

To identify yourself to the server, .helo() (SMTP) or .ehlo() (ESMTP) should be called after creating an .SMTP() object, and again after .starttls(). This function is implicitly called by .starttls() and .sendmail() if needed, so unless you want to check the SMTP service extensions of the server, it is not necessary to use .helo() or .ehlo() explicitly.

Sending Your Plain-text Email

After you initiated a secure SMTP connection using either of the above methods, you can send your email using .sendmail(), which pretty much does what it says on the tin:

server.sendmail(sender_email, receiver_email, message)

I recommend defining the email addresses and message content at the top of your script, after the imports, so you can change them easily:

sender_email = "my@gmail.com"
receiver_email = "your@gmail.com"
message = """\
Subject: Hi there

This message is sent from Python."""

# Send email here

The message string starts with "Subject: Hi there" followed by two newlines (\n). This ensures Hi there shows up as the subject of the email, and the text following the newlines will be treated as the message body.

The code example below sends a plain-text email using SMTP_SSL():

import smtplib, ssl

port = 465  # For SSL
smtp_server = "smtp.gmail.com"
sender_email = "my@gmail.com"  # Enter your address
receiver_email = "your@gmail.com"  # Enter receiver address
password = input("Type your password and press enter: ")
message = """\
Subject: Hi there

This message is sent from Python."""

context = ssl.create_default_context()
with smtplib.SMTP_SSL(smtp_server, port, context=context) as server:
    server.login(sender_email, password)
    server.sendmail(sender_email, receiver_email, message)

For comparison, here is a code example that sends a plain-text email over an SMTP connection secured with .starttls(). The server.ehlo() lines may be omitted, as they are called implicitly by .starttls() and .sendmail(), if required:

import smtplib, ssl

port = 587  # For starttls
smtp_server = "smtp.gmail.com"
sender_email = "my@gmail.com"
receiver_email = "your@gmail.com"
password = input("Type your password and press enter:")
message = """\
Subject: Hi there

This message is sent from Python."""

context = ssl.create_default_context()
with smtplib.SMTP(smtp_server, port) as server:
    server.ehlo()  # Can be omitted
    server.starttls(context=context)
    server.ehlo()  # Can be omitted
    server.login(sender_email, password)
    server.sendmail(sender_email, receiver_email, message)

Sending Fancy Emails

Python’s built-in email package allows you to structure more fancy emails, which can then be transferred with smptlib as you have done already. Below, you’ll learn how use the email package to send emails with HTML content and attachments.

Including HTML Content

If you want to format the text in your email (bold, italics, and so on), or if you want to add any images, hyperlinks, or responsive content, then HTML comes in very handy. Today’s most common type of email is the MIME (Multipurpose Internet Mail Extensions) Multipart email, combining HTML and plain-text. MIME messages are handled by Python’s email.mime module. For a detailed description, check the documentation.

As not all email clients display HTML content by default, and some people choose only to receive plain-text emails for security reasons, it is important to include a plain-text alternative for HTML messages. As the email client will render the last multipart attachment first, make sure to add the HTML message after the plain-text version.

In the example below, our MIMEText() objects will contain the HTML and plain-text versions of our message, and the MIMEMultipart("alternative") instance combines these into a single message with two alternative rendering options:

import smtplib, ssl
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

sender_email = "my@gmail.com"
receiver_email = "your@gmail.com"
password = input("Type your password and press enter:")

message = MIMEMultipart("alternative")
message["Subject"] = "multipart test"
message["From"] = sender_email
message["To"] = receiver_email

# Create the plain-text and HTML version of your message
text = """\
Hi,
How are you?
Real Python has many great tutorials:
www.realpython.com"""
html = """\
<html>
  <body>
    <p>Hi,<br>
       How are you?<br>
       <a href="http://www.realpython.com">Real Python</a> 
       has many great tutorials.
    </p>
  </body>
</html>
"""

# Turn these into plain/html MIMEText objects
part1 = MIMEText(text, "plain")
part2 = MIMEText(html, "html")

# Add HTML/plain-text parts to MIMEMultipart message
# The email client will try to render the last part first
message.attach(part1)
message.attach(part2)

# Create secure connection with server and send email
context = ssl.create_default_context()
with smtplib.SMTP_SSL("smtp.gmail.com", 465, context=context) as server:
    server.login(sender_email, password)
    server.sendmail(
        sender_email, receiver_email, message.as_string()
    )

In this example, you first define the plain-text and HTML message as string literals, and then store them as plain/html MIMEText objects. These can then be added in this order to the MIMEMultipart("alternative") message and sent through your secure connection with the email server. Remember to add the HTML message after the plain-text alternative, as email clients will try to render the last subpart first.

Adding Attachments Using the `email` Package

In order to send binary files to an email server that is designed to work with textual data, they need to be encoded before transport. This is most commonly done using base64, which encodes binary data into printable ASCII characters.

The code example below shows how to send an email with a PDF file as an attachment:

import email, smtplib, ssl

from email import encoders
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

subject = "An email with attachment from Python"
body = "This is an email with attachment sent from Python"
sender_email = "my@gmail.com"
receiver_email = "your@gmail.com"
password = input("Type your password and press enter:")

# Create a multipart message and set headers
message = MIMEMultipart()
message["From"] = sender_email
message["To"] = receiver_email
message["Subject"] = subject
message["Bcc"] = receiver_email  # Recommended for mass emails

# Add body to email
message.attach(MIMEText(body, "plain"))

filename = "document.pdf"  # In same directory as script

# Open PDF file in binary mode
with open(filename, "rb") as attachment:
    # Add file as application/octet-stream
    # Email client can usually download this automatically as attachment
    part = MIMEBase("application", "octet-stream")
    part.set_payload(attachment.read())

# Encode file in ASCII characters to send by email    
encoders.encode_base64(part)

# Add header as key/value pair to attachment part
part.add_header(
    "Content-Disposition",
    f"attachment; filename= {filename}",
)

# Add attachment to message and convert message to string
message.attach(part)
text = message.as_string()

# Log in to server using secure context and send email
context = ssl.create_default_context()
with smtplib.SMTP_SSL("smtp.gmail.com", 465, context=context) as server:
    server.login(sender_email, password)
    server.sendmail(sender_email, receiver_email, text)

The MIMEultipart() message accepts parameters in the form of RFC5233-style key/value pairs, which are stored in a dictionary and passed to the .add_header method of the Message base class.

Check out the documentation for Python’s email.mime module to learn more about using MIME classes.

Sending Multiple Personalized Emails

Imagine you want to send emails to members of your organization, to remind them to pay their contribution fees. Or maybe you want to send students in your class personalized emails with the grades for their recent assignment. These tasks are a breeze in Python.

Make a CSV File With Relevant Personal Info

An easy starting point for sending multiple personalized emails is to create a CSV (comma-separated values) file that contains all the required personal information. (Make sure not to share other people’s private information without their consent.) A CSV file can be thought of as a simple table, where the first line often contains the column headers.

Below are the contents of the file contacts_file.csv, which I saved in the same folder as my Python code. It contains the names, addresses, and grades for a set of fictional people. I used my+modifier@gmail.com constructions to make sure all emails end up in my own inbox, which in this example is my@gmail.com:

name,email,grade
Ron Obvious,my+ovious@gmail.com,B+
Killer Rabbit of Caerbannog,my+rabbit@gmail.com,A
Brian Cohen,my+brian@gmail.com,C

When creating a CSV file, make sure to separate your values by a comma, without any surrounding whitespaces.

Loop Over Rows to Send Multiple Emails

The code example below shows you how to open a CSV file and loop over its lines of content (skipping the header row). To make sure that the code works correctly before you send emails to all your contacts, I’ve printed Sending email to ... for each contact, which we can later replace with functionality that actually sends out emails:

import csv

with open("contacts_file.csv") as file:
    reader = csv.reader(file)
    next(reader)  # Skip header row
    for name, email, grade in reader:
        print(f"Sending email to {name}")
        # Send email here

In the example above, using with open(filename) as file:makes sure that your file closes at the end of the code block. csv.reader() makes it easy to read a CSV file line by line and extract its values. The next(reader) line skips the header row, so that the following line for name, email, grade in reader: splits subsequent rows at each comma, and stores the resulting values in the strings name, email and grade for the current contact.

If the values in your CSV file contain whitespaces on either or both sides, you can remove them using the .strip() method.

Personalized Content

You can put personalized content in a message by using str.format() to fill in curly-bracket placeholders. For example, "hi {name}, you {result} your assignment".format(name="John", result="passed") will give you "hi John, you passed your assignment".

As of Python 3.6, string formatting can be done more elegantly using f-strings, but these require the placeholders to be defined before the f-string itself. In order to define the email message at the beginning of the script, and fill in placeholders for each contact when looping over the CSV file, the older .format() method is used.

With this in mind, you can set up a general message body, with placeholders that can be tailored to individuals.

Code Example

The following code example lets you send personalized emails to multiple contacts. It loops over a CSV file with name,email,grade for each contact, as in the example above.

The general message is defined in the beginning of the script, and for each contact in the CSV file its {name} and {grade} placeholders are filled in, and a personalized email is sent out through a secure connection with the Gmail server, as you saw before:

import csv, smtplib, ssl

message = """Subject: Your grade

Hi {name}, your grade is {grade}"""
from_address = "my@gmail.com"
password = input("Type your password and press enter: ")

context = ssl.create_default_context()
with smtplib.SMTP_SSL("smtp.gmail.com", 465, context=context) as server:
    server.login(from_address, password)
    with open("contacts_file.csv") as file:
        reader = csv.reader(file)
        next(reader)  # Skip header row
        for name, email, grade in reader:
            server.sendmail(
                from_address,
                email,
                message.format(name=name,grade=grade),
            )

Yagmail

There are multiple libraries designed to make sending emails easier, such as Envelopes, Flanker and Yagmail. Yagmail is designed to work specifically with Gmail, and it greatly simplifies the process of sending emails through a friendly API, as you can see in the code example below:

import yagmail

receiver = "your@gmail.com"
body = "Hello there from Yagmail"
filename = "document.pdf"

yag = yagmail.SMTP("my@gmail.com")
yag.send(
    to=receiver,
    subject="Yagmail test with attachment",
    contents=body, 
    attachments=filename,
)

This code example sends an email with a PDF attachment in a fraction of the lines needed for our example using email and smtplib.

When setting up Yagmail, you can add your Gmail validations to the keyring of your OS, as described in the documentation. If you don’t do this, Yagmail will prompt you to enter your password when required and store it in the keyring automatically.

Transactional Email Services

If you plan to send a large volume of emails, want to see email statistics, and want to ensure reliable delivery, it may be worth looking into transactional email services. Although all of the following services have paid plans for sending large volumes of emails, they also come with a free plan so you can try them out. Some of these free plans are valid indefinitely and may be sufficient for your email needs.

Below is an overview of the free plans for some of the major transactional email services. Clicking on the provider name will take you to the pricing section of their website.

Provider	Free plan
Sendgrid	40,000 emails for your first 30 days, then 100/day
Sendinblue	300 emails/day
Mailgun	First 10,000 emails free
Mailjet	200 emails/day
Amazon SES	62,000 emails/month

You can run a Google search to see which provider best fits your needs, or just try out a few of the free plans to see which API you like working with most.

Sendgrid Code Example

Here’s a code example for sending emails with Sendgrid to give you a flavor of how to use a transactional email service with Python:

import os
import sendgrid
from sendgrid.helpers.mail import Content, Email, Mail

sg = sendgrid.SendGridAPIClient(
    apikey=os.environ.get("SENDGRID_API_KEY")
)
from_email = Email("my@gmail.com")
to_email = Email("your@gmail.com")
subject = "A test email from Sendgrid"
content = Content(
    "text/plain", "Here's a test email sent through Python"
)
mail = Mail(from_email, subject, to_email, content)
response = sg.client.mail.send.post(request_body=mail.get())

# The statements below can be included for debugging purposes
print(response.status_code)
print(response.body)
print(response.headers)

To run this code, you must first:

Sign up for a (free) Sendgrid account
Request an API key for user validation
Add your API key by typing setx SENDGRID_API_KEY "YOUR_API_KEY" in Command Prompt (to store this API key permanently) or set SENDGRID_API_KEY YOUR_API_KEY to store it only for the current client session

More information on how to set up Sendgrid for Mac and Windows can be found in the repository’s README on Github.

Conclusion

You can now start a secure SMTP connection and send multiple personalized emails to the people in your contacts list!

You’ve learned how to send an HTML email with a plain-text alternative and attach files to your emails. The Yagmail package simplifies all these tasks when you’re using a Gmail account. If you plan to send large volumes of email, it is worth looking into transactional email services.

Enjoy sending emails with Python, and remember: no spam please!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Building Serverless Python Apps Using AWS Chalice

2018-12-03T14:00:00+00:00

Shipping a web application usually involves having your code up and running on single or multiple servers. In this model, you end up setting up processes for monitoring, provisioning, and scaling your servers up or down. Although this seems to work well, having all the logistics around a web application handled in an automated manner reduces a lot of manual overhead. Enter Serverless.

With Serverless Architecture, you don’t manage servers. Instead, you only need to ship the code or the executable package to the platform that executes it. It’s not really serverless. The servers do exist, but the developer doesn’t need to worry about them.

AWS introduced Lambda Services, a platform that enables developers to simply have their code executed in a particular runtime environment. To make the platform easy to use, many communities have come up with some really good frameworks around it in order to make the serverless apps a working solution.

By the end of this tutorial, you’ll be able to:

Discuss the benefits of a serverless architecture
Explore Chalice, a Python serverless framework
Build a full blown serverless app for a real world use case
Deploy to Amazon Web Services (AWS) Lambda
Compare Pure and Lambda functions

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Getting Started With AWS Chalice

Chalice, a Python Serverless Microframework developed by AWS, enables you to quickly spin up and deploy a working serverless app that scales up and down on its own as required using AWS Lambda.

Why Chalice?

For Python developers accustomed to the Flask web framework, Chalice should be a breeze in terms of building and shipping your first app. Highly inspired by Flask, Chalice keeps it pretty minimalist in terms of defining what the service should be like and finally making an executable package of the same.

Enough theory! Let’s start with a basic hello-world app and kick-start our serverless journey.

Project Setup

Before diving into Chalice, you’ll set up a working environment on your local machine, which will set you up for the rest of the tutorial.

First, create and activate a virtual environment and install Chalice:

$ python3.6 -m venv env
$ source env/bin/activate
(env)$ pip install chalice

Follow our comprehensive guide on the Pipenv packaging tool.

Note: Chalice comes with a user-friendly CLI that makes it easy to play around with your serverless app.

Now that you have Chalice installed on your virtual environment, let’s use the Chalice CLI to generate some boilerplate code:

(env)$ chalice new-project

Enter the name of the project when prompted and hit return. A new directory is created with that name:

<project-name>/
|
├── .chalice/
│   └── config.json
|
├── .gitignore
├── app.py
└── requirements.txt

See how minimalist the Chalice codebase is. A .chalice directory, app.py, and requirements.txt is all that it requires to have a serverless app up and running. Let’s quickly run the app on our local machine.

Chalice CLI consists of really great utility functions allowing you to perform a number of operations from running locally to deploying in a Lambda environment.

Build and Run Locally

You can simulate the app by running it locally using the local utility of Chalice:

(env)$ chalice local
Serving on 127.0.0.1:8000

By default, Chalice runs on port 8000. We can now check the index route by making a curl request to http://localhost:8000/:

$ curl -X GET http://localhost:8000/
{"hello": "world"}

Now if we look at app.py, we can appreciate the simplicity with which Chalice allows you to build a serverless service. All the complex stuff is handled by the decorators:

from chalice import Chalice
app = Chalice(app_name='serverless-sms-service')

@app.route('/')
def index():
    return {'hello': 'world'}

Note: We haven’t named our app hello-world, as we will build our SMS service on the same app.

Now, let’s move on to deploying our app on the AWS Lambda.

Deploy on AWS Lambda

Chalice makes deploying your serverless app completely effortless. Using the deploy utility, you can simply instruct Chalice to deploy and create a Lambda function that can be accessible via a REST API.

Before we begin deployment, we need to make sure we have our AWS credentials in place, usually located at ~/.aws/config. The contents of the file look as follows:

[default]
aws_access_key_id=<your-access-key-id>
aws_secret_access_key=<your-secret-access-key>
region=<your-region>

With AWS credentials in place, let’s begin our deployment process with just a single command:

(env)$ chalice deploy
Creating deployment package.
Updating policy for IAM role: hello-world-dev
Creating lambda function: hello-world-dev
Creating Rest API
Resources deployed:
  - Lambda ARN: arn:aws:lambda:ap-south-1:679337104153:function:hello-world-dev
  - Rest API URL: https://fqcdyzvytc.execute-api.ap-south-1.amazonaws.com/api/

Note: The generated ARN and API URL in the above snippet will vary from user to user.

Wow! Yes, it is really this easy to get your serverless app up and running. To verify simply make a curl request on the generated Rest API URL:

$ curl -X GET https://fqcdyzvytc.execute-api.ap-south-1.amazonaws.com/api/
{"hello": "world"}

Typically, this is all that you need to get your serverless app up and running. You can also go to your AWS console and see the Lambda function created under the Lambda service section. Each Lambda service has a unique REST API endpoint that can be consumed in any web application.

Next, you will begin building your Serverless SMS Sender service using Twilio as an SMS service provider.

Building a Serverless SMS Sender Service

With a basic hello-world app deployed, let’s move on to building a more real-world application that can be used along with everyday web apps. In this section, you’ll build a completely serverless SMS-sending app that can be plugged into any system and work as expected as long as the input parameters are correct.

In order to send SMS, we will be using Twilio, a developer-friendly SMS service. Before we begin using Twilio, we need to take care of a few prerequisites:

Create an account and acquire ACCOUNT_SID and AUTH_TOKEN.
Get a mobile phone number, which is available for free at Twilio for minor testing stuff.
Install the twilio package in our virtual environment using pip install twilio.

With all the above prerequisites checked, you can start building your SMS service client using Twilio’s Python library. Let’s begin by cloning the repository and creating a new feature branch:

$ git clone <project-url>
$ cd <project-dir>
$ git checkout tags/1.0 -b twilio-support

Now make the following changes to app.py to evolve it from a simple hello-world app to enable support for Twilio service too.

First, let’s include all the import statements:

from os import environ as env

# 3rd party imports
from chalice import Chalice, Response
from twilio.rest import Client
from twilio.base.exceptions import TwilioRestException

# Twilio Config
ACCOUNT_SID = env.get('ACCOUNT_SID')
AUTH_TOKEN = env.get('AUTH_TOKEN')
FROM_NUMBER = env.get('FROM_NUMBER')
TO_NUMBER = env.get('TO_NUMBER')

Next, you’ll encapsulate the Twilio API and use it to send SMS:

app = Chalice(app_name='sms-shooter')

# Create a Twilio client using account_sid and auth token
tw_client = Client(ACCOUNT_SID, AUTH_TOKEN)

@app.route('/service/sms/send', methods=['POST'])
def send_sms():
    request_body = app.current_request.json_body
    if request_body:
        try:
            msg = tw_client.messages.create(
                from_=FROM_NUMBER,
                body=request_body['msg'],
                to=TO_NUMBER)

            if msg.sid:
                return Response(status_code=201,
                                headers={'Content-Type': 'application/json'},
                                body={'status': 'success',
                                      'data': msg.sid,
                                      'message': 'SMS successfully sent'})
            else:
                return Response(status_code=200,
                                headers={'Content-Type': 'application/json'},
                                body={'status': 'failure',
                                      'message': 'Please try again!!!'})
        except TwilioRestException as exc:
            return Response(status_code=400,
                            headers={'Content-Type': 'application/json'},
                            body={'status': 'failure',
                                  'message': exc.msg})

In the above snippet, you simply create a Twilio client object using ACCOUNT_SID and AUTH_TOKEN and use it to send messages under the send_sms view. send_sms is a bare bones function that uses the Twilio client’s API to send the SMS to the specified destination. Before proceeding further, let’s give it a try and run it on our local machine.

Build and Run Locally

Now you can run your app on your machine using the local utility and verify that everything is working fine:

(env)$ chalice local

Now make a curl POST request to http://localhost:8000/service/sms/send with a specific payload and test the app locally:

$ curl -H "Content-Type: application/json" -X POST -d '{"msg": "hey mate!!!"}' http://localhost:8000/service/sms/send

The above request responds as follows:

{
  "status": "success",
  "data": "SM60f11033de4f4e39b1c193025bcd5cd8",
  "message": "SMS successfully sent"
}

The response indicates that the message was successfully sent. Now, let’s move on to deploying the app on AWS Lambda.

Deploy on AWS Lambda

As suggested in the previous deployment section, you just need to issue the following command:

(env)$ chalice deploy
Creating deployment package.
Updating policy for IAM role: sms-shooter-dev
Creating lambda function: sms-shooter-dev
Creating Rest API
Resources deployed:
  - Lambda ARN: arn:aws:lambda:ap-south-1:679337104153:function:sms-shooter-dev
  - Rest API URL: https://qtvndnjdyc.execute-api.ap-south-1.amazonaws.com/api/

Note: The above command succeeds, and you have your API URL in the output as expected. Now on testing the URL, the API throws an error message. What went wrong?

As per AWS Lambda logs, twilio package is not found or installed, so you need to tell the Lambda service to install the dependencies. To do so, you need to add twilio as a dependency to requirements.txt:

twilio==6.18.1

Other packages such as Chalice and its dependencies should not be included in requirements.txt, as they are not a part of Python’s WSGI runtime. Instead, we should maintain a requirements-dev.txt, which is applicable to only the development environment and contains all Chalice-related dependencies. To learn more, check out this GitHub issue.

Once all the package dependencies are sorted, you need to make sure all the environment variables are also shipped along and set correctly during the Lambda runtime. To do so, you have to add all the environment variables in .chalice/config.json in the following manner:

{
  "version": "2.0",
  "app_name": "sms-shooter",
  "stages": {
    "dev": {
      "api_gateway_stage": "api",
      "environment_variables": {
          "ACCOUNT_SID": "<your-account-sid>",
          "AUTH_TOKEN": "<your-auth-token>",
          "FROM_NUMBER": "<source-number>",
          "TO_NUMBER": "<destination-number>"
      }
    }
  }
}

Now we’re good to deploy:

Creating deployment package.
Updating policy for IAM role: sms-shooter-dev
Updating lambda function: sms-shooter-dev
Updating rest API
Resources deployed:
  - Lambda ARN: arn:aws:lambda:ap-south-1:679337104153:function:sms-shooter-dev
  - Rest API URL: https://fqcdyzvytc.execute-api.ap-south-1.amazonaws.com/api/

Do a sanity check by making a curl request to the generated API endpoint:

$ curl -H "Content-Type: application/json" -X POST -d '{"msg": "hey mate!!!"}' https://fqcdyzvytc.execute-api.ap-south-1.amazonaws.com/api/service/sms/send

The above request responds as expected:

{
  "status": "success",
  "data": "SM60f11033de4f4e39b1c193025bcd5cd8",
  "message": "SMS successfully sent"
}

Now, you have a completely serverless SMS sending service up and running. With the front end of this service being a REST API, it can be used in other applications as a plug-and-play feature that is scalable, secure, and reliable.

Refactoring

Finally, we will refactor our SMS app to not contain all the business logic in app.py completely. Instead, we will follow the Chalice prescribed best practices and abstract the business logic under the chalicelib/ directory.

Let’s begin by creating a new branch:

$ git checkout tags/2.0 -b sms-app-refactor

First, create a new directory in the root directory of the project named chalicelib/ and create a new file named sms.py:

(env)$ mkdir chalicelib
(env)$ touch chalicelib/sms.py

Update the above created chalicelib/sms.py with the SMS sending logic by abstracting things from app.py:

from os import environ as env
from twilio.rest import Client

# Twilio Config
ACCOUNT_SID = env.get('ACCOUNT_SID')
AUTH_TOKEN = env.get('AUTH_TOKEN')
FROM_NUMBER = env.get('FROM_NUMBER')
TO_NUMBER = env.get('TO_NUMBER')

# Create a twilio client using account_sid and auth token
tw_client = Client(ACCOUNT_SID, AUTH_TOKEN)


def send(payload_params=None):
    """ send sms to the specified number """
    msg = tw_client.messages.create(
        from_=FROM_NUMBER,
        body=payload_params['msg'],
        to=TO_NUMBER)

    if msg.sid:
        return msg

The above snippet only accepts the input params and responds as required. Now to make this work, we need to make changes to app.py as well:

# Core imports
from chalice import Chalice, Response
from twilio.base.exceptions import TwilioRestException

# App level imports
from chalicelib import sms

app = Chalice(app_name='sms-shooter')


@app.route('/')
def index():
    return {'hello': 'world'}


@app.route('/service/sms/send', methods=['POST'])
def send_sms():
    request_body = app.current_request.json_body
    if request_body:
        try:
            resp = sms.send(request_body)
            if resp:
                return Response(status_code=201,
                                headers={'Content-Type': 'application/json'},
                                body={'status': 'success',
                                      'data': resp.sid,
                                      'message': 'SMS successfully sent'})
            else:
                return Response(status_code=200,
                                headers={'Content-Type': 'application/json'},
                                body={'status': 'failure',
                                      'message': 'Please try again!!!'})
        except TwilioRestException as exc:
            return Response(status_code=400,
                            headers={'Content-Type': 'application/json'},
                            body={'status': 'failure',
                                  'message': exc.msg})

In the above snippet, all the SMS sending logic is invoked from the chalicelib.sms module, making the view layer a lot cleaner in terms of readability. This abstraction lets you add much more complex business logic and customize the functionality as required.

Sanity Check

After refactoring our code, let’s ensure it is running as expected.

Build and Run Locally

Run the app once again using the local utility:

(env)$ chalice local

Make a curl request and verify. Once that’s done, move on to deployment.

Deploy on AWS Lambda

Once you are sure everything is working as expected, you can now finally deploy your app:

(env)$ chalice deploy

As usual, the command executes successfully and you can verify the endpoint.

Conclusion

You now know how to do the following:

Build a serverless application using AWS Chalice in accordance with best practices
Deploy your working app on the Lambda runtime environment

Lambda services under the hood are analogous to pure functions, which have a certain behavior on a set of input/output. Developing precise Lambda services allows for better testing, readability, and atomicity. Since Chalice is a minimalist framework, you can just focus on the business logic, and the rest is taken care of, from deployment to IAM policy generation. This is all with just a single command deployment!

Moreover, Lambda services are mostly focused on heavy CPU bound processing and scale in a self-governed manner, as per the number of requests in a unit of time. Using serverless architecture allows your codebase to be more like SOA (Service Oriented Architecture). Using AWS’s other products in their ecosystem that plug in well with Lambda functions is even more powerful.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Emily Morehouse

2018-11-28T14:00:00+00:00

I’m very pleased to be joined this week by Emily Morehouse.

Emily is one of the newest additions to the CPython core developer team, and the founder and director of engineering of Cuttlesoft. Emily and I talk about the recent CPython core developer sprint and the fact that she completed three majors in college at the same time! We’ll also get into her passion for compilers and abstract syntax trees.

Ricky: Let’s start with the obvious: how did you get into programming, and when did you start using Python?

Emily: My path to programming started with falling in love with Enigma machines. Really, I stumbled into programming during college. I had recently switched one of my majors from Biochemistry to Criminology, and the department had just launched a Computer Criminology program.

I was encouraged to try out the Intro To Programming course to see how I liked it. One of our final projects was building an Enigma machine simulator (in C++, mind you), and I was hooked. I decided to add a third major to take on a full Computer Science degree (more on that later!).

Since the CS program was highly theoretical and focused on languages like C and C++, I started to find ways outside of coursework to learn different things. I picked up Python working on web scrapers on the weekends and was eventually hired as a researcher where we used Python to scrape and analyze public data from various sites.

For me, programming spans this wide range of challenging logic and technical problems to more abstract concepts of how humans think and interact with machines and how technology can enhance our daily lives. It fills a gap between academics and art that I didn’t know I needed to, or could, fill.

Ricky: As you’ve already alluded to, you attended Florida State University, where you completed your CS degree. And a degree in Criminology. And another one in Theater… Did you ever sleep? One degree is hard, but three at once? I’m really curious to know your secrets and any time management hacks for studying and learning to code when you have so much else going on.

Emily: I definitely did not sleep much. On top of all of my schoolwork, I worked a nearly full-time job and even worked as an overnight manager at our local coffee shop while still participating in theater rehearsals and performances.

I was able to get a research position in the CS department to eliminate some of that pressure. I was lucky to have started college with a lot of credits and tested out of a few courses, so I was technically already a year ahead, which gave me more freedom to try out courses like Programming.

It’s all very much how I was raised. From a very young age, I knew that my day started around 7 a.m. I went straight from school to rehearsals and dance classes, then had to do homework until I fell asleep. I had to learn how to retain information and figure things out quickly—and I had to stay organized, so I made a lot of lists.

I’ve asked my parents how I came to be this way, and they just shrug! I’ve always felt very in control of how I spend my time to ensure it’s what I want to be doing, and I think that’s important when staying so busy. You have to want to be doing everything, or else things will fall by the wayside.

I definitely suggest finding a manner of keeping to-do lists and prioritizing your time. I use an app called Bear (like a simplified Evernote, but with programmer-friendly themes and markdown support) along with a lot of task prioritization.

I also figured out that I learn things quickly by writing them down multiple times. I used this method to memorize lines for shows. I’d white-out my lines then go back and physically write them down from memory on a separate piece of paper, rinse, and repeat. I got to the point where if I wrote something down 1 to 2 times, it’d stick.

Ricky: You are the co-founder and director of engineering at Cuttlesoft. It looks as if you started the company before finishing college. What was your motivation for starting your own business instead of applying for junior software developer jobs straight out of college?

Emily: Cuttlesoft was a matter of circumstance. I never imagined I’d run my own company, especially not alongside my now-husband Frank. I was in a weird timing limbo where I’d finished my undergraduate degrees earlier than expected which meant that I missed all of the grad school deadlines.

FSU agreed to let me jump in and start my Masters there, and my intention was to stay for a year then transfer elsewhere where I could continue working with parsers, compilers, and formal verification. I was also getting recruited by huge tech companies, and I was a bit enamored with the idea of living in San Francisco or Boston. (I’d only ever lived in Florida at the time.)

But then Frank and I had found our way into this budding entrepreneurship ecosystem in Tallahassee. We met a few people who became great mentors, and before we could even get our Is dotted, we had our first couple of clients. I thought, “Why do I want to leave all of these people who are invested in my future and success to go be one of the thousands somewhere else?”

I figured that I should take the chance on starting something of my own and continuing on this rapid growth path. I knew I’d learn a lot more in a shorter amount of time than I would almost anywhere else. So I dropped out of graduate school after my first semester and put all of my time into Cuttlesoft.

Looking back, I can’t imagine a different path for me. Soon after I turned down those job offers, Susan Fowler’s story came to light. I couldn’t help but think, “That could have been me.” I truly believe that a company’s culture is top-down, and I’m grateful to get to contribute to a company where I can make a huge impact in a positive manner.

Ricky: This year, you got to fulfill a dream and speak at PyCon, with your talk titled The AST and Me. I’ll admit, some of it went over my head, but I’m still learning. I got the impression that language internals fascinate you. What advice would you give to someone who is at the start of their coding journey and wants to know more about how the sausage is made? What resources would you recommend?

Emily: Yes! I was the weird kid in university who loved the classes that most everyone else hated (Programming Languages, Compilers, Theory of Computation…). I would spend hours drawing out non-deterministic finite automata and state machines for my course notes.

I’m a huge fan of Seven Languages in Seven Weeks: A Pragmatic Guide to Learning Programming Languages by Bruce A. Tate to get a feel for different programming language paradigms. The Dragon Book (Compilers: Principles, Techniques, and Tools) is a classic and is the backbone of so much we still use today. (Python’s compiler is based on this.) Philip Guo’s video series on CPython internals is also awesome and helped me in my journey diving into how Python works under the hood.

Ricky: Huge congratulations are in order, as you have just been promoted to a CPython core developer! You must be so thrilled. How was the initiation at the recent CPython sprints? Anything exciting to share or any stories to tell? Don’t worry, we can keep a secret…

Emily: Thank you! The CPython Sprint was a lot of fun. It’s rare that we get so many core developers in the same room working together. We’re all incredibly grateful for the PSF and this year’s sprint sponsor, Microsoft, for supporting CPython.

I was able to attend sprints and Language Summits at PyCons past and had the chance to get to know a lot of the group previously, so this sprint felt surprisingly normal, but it was super cool to see and work with everyone in person.

I spent most of the sprint implementing PEP 572, the (in)famous assignment expressions PEP, with Guido’s guidance. No matter which side of the fence you fell on with assignment expressions (or I as lovingly now call it, the walrus operator), it’s been incredibly cool to add new syntax to the language and deep dive into the internals to get variable scoping to work as intended. It will be in the alpha versions of 3.8 early next year, so keep an eye out!

One of my favorite parts of the sprint was getting to know more about the history of CPython. Since the beginning of my path in core development, I’ve found that it’s really interesting to hear the stories of how others became core developers, so I pose that question to everyone I can.

Understanding everyone’s journey and motivations for devoting so much of their time and energy to a project (especially those who have been involved since the very early days) is an important step to understanding how to continue growing the group and increasing diversity.

Ricky: Now for my last question: what other hobbies and interests do you have, aside from Python? Any you’d like to share and/or plug?

Emily: I try to take advantage of everything Colorado has to offer in my spare time—coming from Florida, I’m still totally enamored with the Rocky Mountains and love hiking. Denver is also a great foodie city.

When I make the time for it, I also really enjoy yoga, reading, listening to podcasts, playing video games (though I’m still slowly working through the most recent God of War), and trying to keep my houseplants alive. I also enjoy spending time with my husband and our dog—they’re my world.

Thank you, Emily, for joining me this week. You can follow Emily’s work on Twitter or Github. Find out more about her company, Cuttlesoft, here.

If there’s someone you’d like me to interview in the future, reach out to me in the comments below, or send me a message on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Continuous Integration With Python: An Introduction

2018-11-26T14:00:00+00:00

When writing code on your own, the only priority is making it work. However, working in a team of professional software developers brings a plethora of challenges. One of those challenges is coordinating many people working on the same code.

How do professional teams make dozens of changes per day while making sure everyone is coordinated and nothing is broken? Enter continuous integration!

In this tutorial you’ll:

Learn the core concepts behind continuous integration
Understand the benefits of continuous integration
Set up a basic continuous integration system
Create a simple Python example and connect it to the continuous integration system

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

What Is Continuous Integration?

Continuous integration (CI) is the practice of frequently building and testing each change done to your code automatically and as early as possible. Prolific developer and author Martin Fowler defines CI as follows:

“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.” (Source)

Let’s unpack this.

Programming is iterative. The source code lives in a repository that is shared by all members of the team. If you want to work on that product, you must obtain a copy. You will make changes, test them, and integrate them back into the main repo. Rinse and repeat.

Not so long ago, these integrations were big and weeks (or months) apart, causing headaches, wasting time, and losing money. Armed with experience, developers started making minor changes and integrating them more frequently. This reduces the chances of introducing conflicts that you need to resolve later.

After every integration, you need to build the source code. Building means transforming your high-level code into a format your computer knows how to run. Finally, the result is systematically tested to ensure your changes did not introduce errors.

Why Should I Care?

On a personal level, continuous integration is really about how you and your colleagues spend your time.

Using CI, you’ll spend less time:

Worrying about introducing a bug every time you make changes
Fixing the mess someone else made so you can integrate your code
Making sure the code works on every machine, operating system, and browser

Conversely, you’ll spend more time:

Solving interesting problems
Writing awesome code with your team
Co-creating amazing products that provide value to users

How does that sound?

On a team level, it allows for a better engineering culture, where you deliver value early and often. Collaboration is encouraged, and bugs are caught much sooner. Continuous integration will:

Make you and your team faster
Give you confidence that you’re building stable software with fewer bugs
Ensure that your product works on other machines, not just your laptop
Eliminate a lot of tedious overhead and let you focus on what matters
Reduce the time spent resolving conflicts (when different people modify the same code)

Core Concepts

There are several key ideas and practices that you need to understand to work effectively with continuous integration. Also, there might be some words and phrases you aren’t familiar with but are used often when you’re talking about CI. This chapter will introduce you to these concepts and the jargon that comes with them.

Single Source Repository

If you are collaborating with others on a single code base, it’s typical to have a shared repository of source code. Every developer working on the project creates a local copy and makes changes. Once they are satisfied with the changes, they merge them back into the central repository.

It has become a standard to use version control systems (VCS) like Git to handle this workflow for you. Teams typically use an external service to host their source code and handle all the moving parts. The most popular are GitHub, BitBucket, and GitLab.

Git allows you to create multiple branches of a repository. Each branch is an independent copy of the source code and can be modified without affecting other branches. This is an essential feature, and most teams have a mainline branch (often called a master branch) that represents the current state of the project.

If you want to add or modify code, you should create a copy of the main branch and work in your new, development branch. Once you are done, merge those changes back into the master branch.

Git branching

Version control holds more than just code. Documentation and test scripts are usually stored along with the source code. Some programs look for external files used to configure their parameters and initial settings. Other applications need a database schema. All these files should go into your repository.

If you have never used Git or need a refresher, check out our Introduction to Git and GitHub for Python Developers.

Automating the Build

As previously mentioned, building your code means taking the raw source code, and everything necessary for its execution, and translating it into a format that computers can run directly. Python is an interpreted language, so its “build” mainly revolves around test execution rather than compilation.

Running those steps manually after every small change is tedious and takes valuable time and attention from the actual problem-solving you’re trying to do. A big part of continuous integration is automating that process and moving it out of sight (and out of mind).

What does that mean for Python? Think about a more complicated piece of code you have written. If you used a library, package, or framework that doesn’t come with the Python standard library (think anything you needed to install with pip or conda), Python needs to know about that, so the program knows where to look when it finds commands that it doesn’t recognize.

You store a list of those packages in requirements.txt or a Pipfile. These are the dependencies of your code and are necessary for a successful build.

You will often hear the phrase “breaking the build.” When you break the build, it means you introduced a change that rendered the final product unusable. Don’t worry. It happens to everyone, even battle-hardened senior developers. You want to avoid this primarily because it will block everyone else from working.

The whole point of CI is to have everyone working on a known stable base. If they clone a repository that is breaking the build, they will work with a broken version of the code and won’t be able to introduce or test their changes. When you break the build, the top priority is fixing it so everyone can resume work.

Introducing a breaking change to the master branch

When the build is automated, you are encouraged to commit frequently, usually multiple times per day. It allows people to quickly find out about changes and notice if there’s a conflict between two developers. If there are numerous small changes instead of a few massive updates, it’s much easier to locate where the error originated. It will also encourage you to break your work down into smaller chunks, which is easier to track and test.

Automated Testing

Since everyone is committing changes multiple times per day, it’s important to know that your change didn’t break anything else in the code or introduce bugs. In many companies, testing is now a responsibility of every developer. If you write code, you should write tests. At a bare minimum, you should cover every new function with a unit test.

Running tests automatically, with every change committed, is a great way to catch bugs. A failing test automatically causes the build to fail. It will draw your attention to the problems revealed by testing, and the failed build will make you fix the bug you introduced. Tests don’t guarantee that your code is free of bugs, but it does guard against a lot of careless changes.

Automating test execution gives you some peace of mind because you know the server will test your code every time you commit, even if you forgot to do it locally.

Using an External Continuous Integration Service

If something works on your computer, will it work on every computer? Probably not. It’s a cliché excuse and a sort of inside joke among developers to say, “Well, it worked on my machine!” Making the code work locally is not the end of your responsibility.

To tackle this problem, most companies use an external service to handle integration, much like using GitHub for hosting your source code repository. External services have servers where they build code and run tests. They act as monitors for your repository and stop anyone from merging to the master branch if their changes break the build.

Merging changes triggers the CI server

There are many such services out there, with various features and pricing. Most have a free tier so that you can experiment with one of your repositories. You will use a service called CircleCI in an example later in the tutorial.

Testing in a Staging Environment

A production environment is where your software will ultimately run. Even after successfully building and testing your application, you can’t be sure that your code will work on the target computer. That’s why teams deploy the final product in an environment that mimics the production environment. Once you are sure everything works, the application is deployed in the production environment.

Note: This step is more relevant to application code than library code. Any Python libraries you write still need to be tested on a build server, to ensure they work in environments different from your local computer.

You will hear people talking about this clone of the production environment using terms like development environment, staging environment, or testing environment. It’s common to use abbreviations like DEV for the development environment and PROD for the production environment.

The development environment should replicate production conditions as closely as possible. This setup is often called DEV/PROD parity. Keep the environment on your local computer as similar as possible to the DEV and PROD environments to minimize anomalies when deploying applications.

Test in a clone of the production environment

We mention this to introduce you to the vocabulary, but continuously deploying software to DEV and PROD is a whole other topic. The process is called, unsurprisingly, continuous deployment (CD). You can find more resources about it in the Next Steps section of this article.

Your Turn!

The best way to learn is by doing. You now understand all the essential practices of continuous integration, so it’s time to get your hands dirty and create the whole chain of steps necessary to use CI. This chain is often called a CI pipeline.

This is a hands-on tutorial, so fire up your editor and get ready to work through these steps as you read!

We assume that you know the basics of Python and Git. We will use Github as our hosting service and CircleCI as our external continuous integration service. If you don’t have accounts with these services, go ahead and register. Both of these have free tiers!

Problem Definition

Remember, your focus here is adding a new tool to your utility belt, continuous integration. For this example, the Python code itself will be straightforward. You want to spend the bulk of your time internalizing the steps of building a pipeline, instead of writing complicated code.

Imagine your team is working on a simple calculator app. Your task is to write a library of basic mathematical functions: addition, subtraction, multiplication, and division. You don’t care about the actual application, because that’s what your peers will be developing, using functions from your library.

Create a Repo

Log in to your GitHub account, create a new repository and call it CalculatorLibrary. Add a README and .gitignore, then clone the repository to your local machine. If you need more help with this process, have a look at GitHub’s walkthrough on creating a new repository.

Set Up a Working Environment

For others (and the CI server) to replicate your working conditions, you need to set up an environment. Create a virtual environment somewhere outside your repo and activate it:

$ # Create virtual environment
$ python3 -m venv calculator

$ # Activate virtual environment (Mac and Linux)
$ . calculator/bin/activate

The previous commands work on macOS and Linux. If you are a Windows user, check the Platforms table in the official documentation. This will create a directory that contains a Python installation and tell the interpreter to use it. Now we can install packages knowing that it will not influence your system’s default Python installation.

Write a Simple Python Example

Create a new file called calculator.py in the top-level directory of your repository, and copy the following code:

"""
Calculator library containing basic math operations.
"""

def add(first_term, second_term):
    return first_term + second_term

def subtract(first_term, second_term):
    return first_term - second_term

This is a bare-bones example containing two of the four functions we will be writing. Once we have our CI pipeline up and running, you will add the remaining two functions.

Go ahead and commit those changes:

$ # Make sure you are in the correct directory
$ cd CalculatorLibrary
$ git add calculator.py
$ git commit -m "Add functions for addition and subtraction"

Your CalculatorLibrary folder should have the following files right now:

CalculatorLibrary/
|
├── .git
├── .gitignore
├── README.md
└── calculator.py

Great, you have completed one part of the required functionality. The next step is adding tests to make sure your code works the way it’s supposed to.

Write Unit Tests

You will test your code in two steps.

The first step involves linting—running a program, called a linter, to analyze code for potential errors. flake8 is commonly used to check if your code conforms to the standard Python coding style. Linting makes sure your code is easy to read for the rest of the Python community.

The second step is unit testing. A unit test is designed to check a single function, or unit, of code. Python comes with a standard unit testing library, but other libraries exist and are very popular. This example uses pytest.

A standard practice that goes hand in hand with testing is calculating code coverage. Code coverage is the percentage of source code that is “covered” by your tests. pytest has an extension, pytest-cov, that helps you understand your code coverage.

These are external dependencies, and you need to install them:

$ pip install flake8 pytest pytest-cov

These are the only external packages you will use. Make sure to store those dependencies in a requirements.txt file so others can replicate your environment:

$ pip freeze > requirements.txt

To run your linter, execute the following:

$ flake8 --statistics
./calculator.py:3:1: E302 expected 2 blank lines, found 1
./calculator.py:6:1: E302 expected 2 blank lines, found 1
2     E302 expected 2 blank lines, found 1

The --statistics option gives you an overview of how many times a particular error happened. Here we have two PEP 8 violations, because flake8 expects two blank lines before a function definition instead of one. Go ahead and add an empty line before each functions definition. Run flake8 again to check that the error messages no longer appear.

Now it’s time to write the tests. Create a file called test_calculator.py in the top-level directory of your repository and copy the following code:

"""
Unit tests for the calculator library
"""

import calculator


class TestCalculator:

    def test_addition(self):
        assert 4 == calculator.add(2, 2)

    def test_subtraction(self):
        assert 2 == calculator.subtract(4, 2)

These tests make sure that our code works as expected. It is far from extensive because you haven’t tested for potential misuse of your code, but keep it simple for now.

The following command runs your test:

$ pytest -v --cov
collected 2 items

test_calculator.py::TestCalculator::test_addition PASSED [50%]

test_calculator.py::TestCalculator::test_subtraction PASSED [100%]

---------- coverage: platform darwin, python 3.6.6-final-0 -----------
Name                                              Stmts   Miss  Cover
---------------------------------------------------------------------
calculator.py                                         4      0   100%
test_calculator.py                                    6      0   100%
/Users/kristijan.ivancic/code/learn/__init__.py       0      0   100%
---------------------------------------------------------------------
TOTAL                                                10      0   100%

pytest is excellent at test discovery. Because you have a file with the prefix test, pytest knows it will contain unit tests for it to run. The same principles apply to the class and method names inside the file.

The -v flag gives you a nicer output, telling you which tests passed and which failed. In our case, both tests passed. The --cov flag makes sure pytest-cov runs and gives you a code coverage report for calculator.py.

You have completed the preparations. Commit the test file and push all those changes to the master branch:

$ git add test_calculator.py
$ git commit -m "Add unit tests for calculator"
$ git push

At the end of this section, your CalculatorLibrary folder should have the following files:

CalculatorLibrary/
|
├── .git
├── .gitignore
├── README.md
├── calculator.py
├── requirements.txt
└── test_calculator.py

Excellent, both your functions are tested and work correctly.

Connect to CircleCI

At last, you are ready to set up your continuous integration pipeline!

CircleCI needs to know how to run your build and expects that information to be supplied in a particular format. It requires a .circleci folder within your repo and a configuration file inside it. A configuration file contains instructions for all the steps that the build server needs to execute. CircleCI expects this file to be called config.yml.

A .yml file uses a data serialization language, YAML, and it has its own specification. The goal of YAML is to be human readable and to work well with modern programming languages for common, everyday tasks.

In a YAML file, there are three basic ways to represent data:

Mappings (key-value pairs)
Sequences (lists)
Scalars (strings or numbers)

It is very simple to read:

Indentation may be used for structure.
Colons separate key-value pairs.
Dashes are used to create lists.

Create the .circleci folder in your repo and a config.yml file with the following content:

# Python CircleCI 2.0 configuration file
version: 2
jobs:
  build:
    docker:
      - image: circleci/python:3.7

    working_directory: ~/repo

    steps:
      # Step 1: obtain repo from GitHub
      - checkout
      # Step 2: create virtual env and install dependencies
      - run:
          name: install dependencies
          command: |
            python3 -m venv venv
            . venv/bin/activate
            pip install -r requirements.txt
      # Step 3: run linter and tests
      - run:
          name: run tests
          command: |
            . venv/bin/activate
            flake8 --exclude=venv* --statistics
            pytest -v --cov=calculator

Some of these words and concepts might be unfamiliar to you. For example, what is Docker, and what are images? Let’s go back in time a bit.

Remember the problem programmers face when something works on their laptop but nowhere else? Before, developers used to create a program that isolates a part of the computer’s physical resources (memory, hard drive, and so on) and turns them into a virtual machine.

A virtual machine pretends to be a whole computer on its own. It would even have its own operating system. On that operating system, you deploy your application or install your library and test it.

Virtual machines take up a lot of resources, which sparked the invention of containers. The idea is analogous to shipping containers. Before shipping containers were invented, manufacturers had to ship goods in a wide variety of sizes, packaging, and modes (trucks, trains, ships).

By standardizing the shipping container, these goods could be transferred between different shipping methods without any modification. The same idea applies to software containers.

Containers are a lightweight unit of code and its runtime dependencies, packaged in a standardized way, so they can quickly be plugged in and run on the Linux OS. You don’t need to create a whole virtual operating system, as you would with a virtual machine.

Containers only replicate parts of the operating system they need in order to work. This reduces their size and gives them a big performance boost.

Docker is currently the leading container platform, and it’s even able to run Linux containers on Windows and macOS. To create a Docker container, you need a Docker image. Images provide blueprints for containers much like classes provide blueprints for objects. You can read more about Docker in their Get Started guide.

CircleCI maintains pre-built Docker images for several programming languages. In the above configuration file, you have specified a Linux image that has Python already installed. That image will create a container in which everything else happens.

Let’s look at each line of the configuration file in turn:

version: Every config.yml starts with the CircleCI version number, used to issue warnings about breaking changes.
jobs: Jobs represent a single execution of the build and are defined by a collection of steps. If you have only one job, it must be called build.
build: As mentioned before, build is the name of your job. You can have multiple jobs, in which case they need to have unique names.
docker: The steps of a job occur in an environment called an executor. The common executor in CircleCI is a Docker container. It is a cloud-hosted execution environment but other options exist, like a macOS environment.
image: A Docker image is a file used to create a running Docker container. We are using an image that has Python 3.7 preinstalled.
working_directory: Your repository has to be checked out somewhere on the build server. The working directory represents the file path where the repository will be stored.
steps: This key marks the start of a list of steps to be performed by the build server.
checkout: The first step the server needs to do is check the source code out to the working directory. This is performed by a special step called checkout.
run: Executing command-line programs or commands is done inside the command key. The actual shell commands will be nested within.
name: The CircleCI user interface shows you every build step in the form of an expandable section. The title of the section is taken from the value associated with the name key.
command: This key represents the command to run via the shell. The | symbol specifices that what follows is a literal set of commands, one per line, exactly like you’d see in a shell/bash script.

You can read the CircleCI configuration reference document for more information.

Our pipeline is very simple and consists of 3 steps:

Checking out the repository
Installing the dependencies in a virtual environment
Running the linter and tests while inside the virtual environment

We now have everything we need to start our pipeline. Log in to your CircleCI account and click on Add Projects. Find your CalculatorLibrary repo and click Set Up Project. Select Python as your language. Since we already have a config.yml, we can skip the next steps and click Start building.

CircleCI will take you to the execution dashboard for your job. If you followed all the steps correctly, you should see your job succeed.

The final version of your CalculatorLibrary folder should look like this:

CalculatorRepository/
|
├── .circleci
├── .git
├── .gitignore
├── README.md
├── calculator.py
├── requirements.txt
└── test_calculator.py

Congratulations! You have created your first continuous integration pipeline. Now, every time you push to the master branch, a job will be triggered. You can see a list of your current and past jobs by clicking on Jobs in the CircleCI sidebar.

Make Changes

Time to add multiplication to our calculator library.

This time, we will first add a unit test without writing the function. Without the code, the test will fail, which will also fail the CircleCI job. Add the following code to the end of your test_calculator.py:

def test_multiplication(self):
    assert 100 == calculator.multiply(10, 10)

Push the code to the master branch and see the job fail in CircleCI. This shows that continuous integration works and watches your back if you make a mistake.

Now add the code to calculator.py that will make the test pass:

def multiply(first_term, second_term):
    return first_term * second_term

Make sure there are two empty spaces between the multiplication function and the previous one, or else your code will fail the linter check.

The job should be successful this time. This workflow of writing a failing test first and then adding the code to pass the test is called test driven development (TDD). It’s a great way to work because it makes you think about your code structure in advance.

Now try it on your own. Add a test for the division function, see it fail, and write the function to make the test pass.

Notifications

When working on big applications that have a lot of moving parts, it can take a while for the continuous integration job to run. Most teams set up a notification procedure to let them know if one of their jobs fail. They can continue working while waiting for the job to run.

The most popular options are:

Sending an email for each failed build
Sending failure notifications to a Slack channel
Displaying failures on a dashboard visible to everyone

By default, CircleCI should send you an email when a job fails.

Next Steps

You have understood the basics of continuous integration and practiced setting up a pipeline for a simple Python program. This is a big step forward in your journey as a developer. You might be asking yourself, “What now?”

To keep things simple, this tutorial skimmed over some big topics. You can grow your skill set immensely by spending some time going more in-depth into each subject. Here are some topics you can look into further.

Git Workflows

There is much more to Git than what you used here. Each developer team has a workflow tailored to their specific needs. Most of them include branching strategies and something called peer review. They make changes on branches separate from the master branch. When you want to merge those changes with master, other developers must first look at your changes and approve them before you’re allowed to merge.

Note: If you want to learn more about different workflows teams use, have a look at the tutorials on GitHub and BitBucket.

If you want to sharpen your Git skills, we have an article called Advanced Git Tips for Python Developers.

Dependency Management and Virtual Environments

Apart from virtualenv, there are other popular package and environment managers. Some of them deal with just virtual environments, while some handle both package installation and environment management. One of them is Conda:

“Conda is an open source package management system and environment management system that runs on Windows, macOS, and Linux. Conda quickly installs, runs and updates packages and their dependencies. Conda easily creates, saves, loads and switches between environments on your local computer. It was designed for Python programs, but it can package and distribute software for any language.” (Source)

Another option is Pipenv, a younger contender that is rising in popularity among application developers. Pipenv brings together pip and virtualenv into a single tool and uses a Pipfile instead of requirements.txt. Pipfiles offer deterministic environments and more security. This introduction doesn’t do it justice, so check out Pipenv: A Guide to the New Python Packaging Tool.

Testing

Simple unit tests with pytest are only the tip of the iceberg. There’s a whole world out there to explore! Software can be tested on many levels, including integration testing, acceptance testing, regression testing, and so forth. To take your knowledge of testing Python code to the next level, head over to Getting Started With Testing in Python.

Packaging

In this tutorial, you started to build a library of functions for other developers to use in their project. You need to package that library into a format that is easy to distribute and install using, for example pip.

Creating an installable package requires a different layout and some additional files like __init__.py and setup.py. Read Python Application Layouts: A Reference for more information on structuring your code.

To learn how to turn your repository into an installable Python package, read Packaging Python Projects by the Python Packaging Authority.

Continuous Integration

You covered all the basics of CI in this tutorial, using a simple example of Python code. It’s common for the final step of a CI pipeline to create a deployable artifact. An artifact represents a finished, packaged unit of work that is ready to be deployed to users or included in complex products.

For example, to turn your calculator library into a deployable artifact, you would organize it into an installable package. Finally, you would add a step in CircleCI to package the library and store that artifact where other processes can pick it up.

For more complex applications, you can create a workflow to schedule and connect multiple CI jobs into a single execution. Feel free to explore the CircleCI documentation.

Continuous Deployment

You can think of continuous deployment as an extension of CI. Once your code is tested and built into a deployable artifact, it is deployed to production, meaning the live application is updated with your changes. One of the goals is to minimize lead time, the time elapsed between writing a new line of code and putting it in front of users.

Note: To add a bit of confusion to the mix, the acronym CD is not unique. It can also mean Continuous Delivery, which is almost the same as continuous deployment but has a manual verification step between integration and deployment. You can integrate your code at any time but have to push a button to release it to the live application.

Most companies use CI/CD in tandem, so it’s worth your time to learn more about Continuous Delivery/Deployment.

Overview of Continuous Integration Services

You have used CircleCI, one of the most popular continuous integration services. However, this is a big market with a lot of strong contenders. CI products fall into two basic categories: remote and self-hosted services.

Jenkins is the most popular self-hosted solution. It is open-source and flexible, and the community has developed a lot of extensions.

In terms of remote services, there are many popular options like TravisCI, CodeShip, and Semaphore. Big enterprises often have their custom solutions, and they sell them as a service, such as AWS CodePipeline, Microsoft Team Foundation Server, and Oracle’s Hudson.

Which option you choose depends on the platform and features you and your team need. For a more detailed breakdown, have a look at Best CI Software by G2Crowd.

Conclusion

With the knowledge from this tutorial under your belt, you can now answer the following questions:

What is continuous integration?
Why is continuous integration important?
What are the core practices of continuous integration?
How can I set up continuous integration for my Python project?

You have acquired a programming superpower! Understanding the philosophy and practice of continuous integration will make you a valuable member of any team. Awesome work!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Memory Management in Python

2018-11-21T14:00:00+00:00

Ever wonder how Python handles your data behind the scenes? How are your variables stored in memory? When do they get deleted?

In this article, we’re going to do a deep dive into the internals of Python to understand how it handles memory management.

By the end of this article, you’ll:

Learn more about low-level computing, specifically as relates to memory
Understand how Python abstracts lower-level operations
Learn about Python’s internal memory management algorithms

Understanding Python’s internals will also give you better insight into some of Python’s behaviors. Hopefully, you’ll gain a new appreciation for Python as well. So much logic is happening behind the scenes to ensure your program works the way you expect.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Memory Is an Empty Book

You can begin by thinking of a computer’s memory as an empty book intended for short stories. There’s nothing written on the pages yet. Eventually, different authors will come along. Each author wants some space to write their story in.

Since they aren’t allowed to write over each other, they must be careful about which pages they write in. Before they begin writing, they consult the manager of the book. The manager then decides where in the book they’re allowed to write.

Since this book is around for a long time, many of the stories in it are no longer relevant. When no one reads or references the stories, they are removed to make room for new stories.

In essence, computer memory is like that empty book. In fact, it’s common to call fixed-length contiguous blocks of memory pages, so this analogy holds pretty well.

The authors are like different applications or processes that need to store data in memory. The manager, who decides where the authors can write in the book, plays the role of a memory manager of sorts. The person who removed the old stories to make room for new ones is a garbage collector.

Memory Management: From Hardware to Software

Memory management is the process by which applications read and write data. A memory manager determines where to put an application’s data. Since there’s a finite chunk of memory, like the pages in our book analogy, the manager has to find some free space and provide it to the application. This process of providing memory is generally called memory allocation.

On the flip side, when data is no longer needed, it can be deleted, or freed. But freed to where? Where did this “memory” come from?

Somewhere in your computer, there’s a physical device storing data when you’re running your Python programs. There are many layers of abstraction that the Python code goes through before the objects actually get to the hardware though.

One of the main layers above the hardware (such as RAM or a hard drive) is the operating system (OS). It carries out (or denies) requests to read and write memory.

Above the OS, there are applications, one of which is the default Python implementation (included in your OS or downloaded from python.org.) Memory management for your Python code is handled by the Python application. The algorithms and structures that the Python application uses for memory management is the focus of this article.

The Default Python Implementation

The default Python implementation, CPython, is actually written in the C programming language.

When I first heard this, it blew my mind. A language that’s written in another language?! Well, not really, but sort of.

The Python language is defined in a reference manual written in English. However, that manual isn’t all that useful by itself. You still need something to interpret written code based on the rules in the manual.

You also need something to actually execute interpreted code on a computer. The default Python implementation fulfills both of those requirements. It converts your Python code into instructions that it then runs on a virtual machine.

Note: Virtual machines are like physical computers, but they are implemented in software. They typically process basic instructions similar to Assembly instructions.

Python is an interpreted programming language. Your Python code actually gets compiled down to more computer-readable instructions called bytecode. These instructions get interpreted by a virtual machine when you run your code.

Have you ever seen a .pyc file or a __pycache__ folder? That’s the bytecode that gets interpreted by the virtual machine.

It’s important to note that there are implementations other than CPython. IronPython compiles down to run on Microsoft’s Common Language Runtime. Jython compiles down to Java bytecode to run on the Java Virtual Machine. Then there’s PyPy, but that deserves its own entire article, so I’ll just mention it in passing.

For the purposes of this article, I’ll focus on the memory management done by the default implementation of Python, CPython.

Disclaimer: While a lot of this information will carry through to new versions of Python, things may change in the future. Note that the referenced version for this article is the current latest version of Python, 3.7.

Okay, so CPython is written in C, and it interprets Python bytecode. What does this have to do with memory management? Well, the memory management algorithms and structures exist in the CPython code, in C. To understand the memory management of Python, you have to get a basic understanding of CPython itself.

CPython is written in C, which does not natively support object-oriented programming. Because of that, there are quite a bit of interesting designs in the CPython code.

You may have heard that everything in Python is an object, even types such as int and str. Well, it’s true on an implementation level in CPython. There is a struct called a PyObject, which every other object in CPython uses.

Note: A struct, or structure, in C is a custom data type that groups together different data types. To compare to object-oriented languages, it’s like a class with attributes and no methods.

The PyObject, the grand-daddy of all objects in Python, contains only two things:

ob_refcnt: reference count
ob_type: pointer to another type

The reference count is used for garbage collection. Then you have a pointer to the actual object type. That object type is just another struct that describes a Python object (such as a dict or int).

Each object has its own object-specific memory allocator that knows how to get the memory to store that object. Each object also has an object-specific memory deallocator that “frees” the memory once it’s no longer needed.

However, there’s an important factor in all this talk about allocating and freeing memory. Memory is a shared resource on the computer, and bad things can happen if two different processes try to write to the same location at the same time.

The Global Interpreter Lock (GIL)

The GIL is a solution to the common problem of dealing with shared resources, like memory in a computer. When two threads try to modify the same resource at the same time, they can step on each other’s toes. The end result can be a garbled mess where neither of the threads ends up with what they wanted.

Consider the book analogy again. Suppose that two authors stubbornly decide that it’s their turn to write. Not only that, but they both need to write on the same page of the book at the same time.

They each ignore the other’s attempt to craft a story and begin writing on the page. The end result is two stories on top of each other, which makes the whole page completely unreadable.

One solution to this problem is a single, global lock on the interpreter when a thread is interacting with the shared resource (the page in the book). In other words, only one author can write at a time.

Python’s GIL accomplishes this by locking the entire interpreter, meaning that it’s not possible for another thread to step on the current one. When CPython handles memory, it uses the GIL to ensure that it does so safely.

There are pros and cons to this approach, and the GIL is heavily debated in the Python community. To read more about the GIL, I suggest checking out What is the Python Global Interpreter Lock (GIL)?.

Garbage Collection

Let’s revisit the book analogy and assume that some of the stories in the book are getting very old. No one is reading or referencing those stories anymore. If no one is reading something or referencing it in their own work, you could get rid of it to make room for new writing.

That old, unreferenced writing could be compared to an object in Python whose reference count has dropped to 0. Remember that every object in Python has a reference count and a pointer to a type.

The reference count gets increased for a few different reasons. For example, the reference count will increase if you assign it to another variable:

numbers = [1, 2, 3]
# Reference count = 1
more_numbers = numbers
# Reference count = 2

It will also increase if you pass the object as an argument:

total = sum(numbers)

As a final example, the reference count will increase if you include the object in a list:

matrix = [numbers, numbers, numbers]

Python allows you to inspect the current reference count of an object with the sys module. You can use sys.getrefcount(numbers), but keep in mind that passing in the object to getrefcount() increases the reference count by 1.

In any case, if the object is still required to hang around in your code, its reference count is greater than 0. Once it drops to 0, the object has a specific deallocation function that is called which “frees” the memory so that other objects can use it.

But what does it mean to “free” the memory, and how do other objects use it? Let’s jump right into CPython’s memory management.

CPython’s Memory Management

We’re going to dive deep into CPython’s memory architecture and algorithms, so buckle up.

As mentioned before, there are layers of abstraction from the physical hardware to CPython. The operating system (OS) abstracts the physical memory and creates a virtual memory layer that applications (including Python) can access.

An OS-specific virtual memory manager carves out a chunk of memory for the Python process. The darker gray boxes in the image below are now owned by the Python process.

Python uses a portion of the memory for internal use and non-object memory. The other portion is dedicated to object storage (your int, dict, and the like). Note that this was somewhat simplified. If you want the full picture, you can check out the CPython source code, where all this memory management happens.

CPython has an object allocator that is responsible for allocating memory within the object memory area. This object allocator is where most of the magic happens. It gets called every time a new object needs space allocated or deleted.

Typically, the adding and removing of data for Python objects like list and int doesn’t involve too much data at a time. So the design of the allocator is tuned to work well with small amounts of data at a time. It also tries not to allocate memory until it’s absolutely required.

The comments in the source code describe the allocator as “a fast, special-purpose memory allocator for small blocks, to be used on top of a general-purpose malloc.” In this case, malloc is C’s library function for memory allocation.

Now we’ll look at CPython’s memory allocation strategy. First, we’ll talk about the 3 main pieces and how they relate to each other.

Arenas are the largest chunks of memory and are aligned on a page boundary in memory. A page boundary is the edge of a fixed-length contiguous chunk of memory that the OS uses. Python assumes the system’s page size is 256 kilobytes.

Within the arenas are pools, which are one virtual memory page (4 kilobytes). These are like the pages in our book analogy. These pools are fragmented into smaller blocks of memory.

All the blocks in a given pool are of the same “size class.” A size class defines a specific block size, given some amount of requested data. The chart below is taken directly from the source code comments:

Request in bytes	Size of allocated block	Size class idx
1-8	8	0
9-16	16	1
17-24	24	2
25-32	32	3
33-40	40	4
41-48	48	5
49-56	56	6
57-64	64	7
65-72	72	8
…	…	…
497-504	504	62
505-512	512	63

For example, if 42 bytes are requested, the data would be placed into a size 48-byte block.

Pools

Pools are composed of blocks from a single size class. Each pool maintains a double-linked list to other pools of the same size class. In that way, the algorithm can easily find available space for a given block size, even across different pools.

A usedpools list tracks all the pools that have some space available for data for each size class. When a given block size is requested, the algorithm checks this usedpools list for the list of pools for that block size.

Pools themselves must be in one of 3 states: used, full, or empty. A used pool has available blocks for data to be stored. A full pool’s blocks are all allocated and contain data. An empty pool has no data stored and can be assigned any size class for blocks when needed.

A freepools list keeps track of all the pools in the empty state. But when do empty pools get used?

Assume your code needs an 8-byte chunk of memory. If there are no pools in usedpools of the 8-byte size class, a fresh empty pool is initialized to store 8-byte blocks. This new pool then gets added to the usedpools list so it can be used for future requests.

Say a full pool frees some of its blocks because the memory is no longer needed. That pool would get added back to the usedpools list for its size class.

You can see now how pools can move between these states (and even memory size classes) freely with this algorithm.

Blocks

As seen in the diagram above, pools contain a pointer to their “free” blocks of memory. There’s a slight nuance to the way this works. This allocator “strives at all levels (arena, pool, and block) never to touch a piece of memory until it’s actually needed,” according to the comments in the source code.

That means that a pool can have blocks in 3 states. These states can be defined as follows:

untouched: a portion of memory that has not been allocated
free: a portion of memory that was allocated but later made “free” by CPython and that no longer contains relevant data
allocated: a portion of memory that actually contains relevant data

The freeblock pointer points to a singly linked list of free blocks of memory. In other words, a list of available places to put data. If more than the available free blocks are needed, the allocator will get some untouched blocks in the pool.

As the memory manager makes blocks “free,” those now free blocks get added to the front of the freeblock list. The actual list may not be contiguous blocks of memory, like the first nice diagram. It may look something like the diagram below:

Arenas

Arenas contain pools. Those pools can be used, full, or empty. Arenas themselves don’t have as explicit states as pools do though.

Arenas are instead organized into a doubly linked list called usable_arenas. The list is sorted by the number of free pools available. The fewer free pools, the closer the arena is to the front of the list.

This means that the arena that is the most full of data will be selected to place new data into. But why not the opposite? Why not place data where there’s the most available space?

This brings us to the idea of truly freeing memory. You’ll notice that I’ve been saying “free” in quotes quite a bit. The reason is that when a block is deemed “free”, that memory is not actually freed back to the operating system. The Python process keeps it allocated and will use it later for new data. Truly freeing memory returns it to the operating system to use.

Arenas are the only things that can truly be freed. So, it stands to reason that those arenas that are closer to being empty should be allowed to become empty. That way, that chunk of memory can be truly freed, reducing the overall memory footprint of your Python program.

Conclusion

Memory management is an integral part of working with computers. Python handles nearly all of it behind the scenes, for better or for worse.

In this article, you learned:

What memory management is and why it’s important
How the default Python implementation, CPython, is written in the C programming language
How the data structures and algorithms work together in CPython’s memory management to handle your data

Python abstracts away a lot of the gritty details of working with computers. This gives you the power to work on a higher level to develop your code without the headache of worrying about how and where all those bytes are getting stored.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Interactive Data Visualization in Python With Bokeh

2018-11-19T14:00:00+00:00

Bokeh prides itself on being a library for interactive data visualization.

Unlike popular counterparts in the Python visualization space, like Matplotlib and Seaborn, Bokeh renders its graphics using HTML and JavaScript. This makes it a great candidate for building web-based dashboards and applications. However, it’s an equally powerful tool for exploring and understanding your data or creating beautiful custom charts for a project or report.

Using a number of examples on a real-world dataset, the goal of this tutorial is to get you up and running with Bokeh.

You’ll learn how to:

Transform your data into visualizations, using Bokeh
Customize and organize your visualizations
Add interactivity to your visualizations

So let’s jump in.

From Data to Visualization

Building a visualization with Bokeh involves the following steps:

Prepare the data
Determine where the visualization will be rendered
Set up the figure(s)
Connect to and draw your data
Organize the layout
Preview and save your beautiful data creation

Let’s explore each step in more detail.

Prepare the Data

Any good data visualization starts with—you guessed it—data. If you need a quick refresher on handling data in Python, definitely check out the growing number of excellent Real Python tutorials on the subject.

This step commonly involves data handling libraries like Pandas and Numpy and is all about taking the required steps to transform it into a form that is best suited for your intended visualization.

Determine Where the Visualization Will Be Rendered

At this step, you’ll determine how you want to generate and ultimately view your visualization. In this tutorial, you’ll learn about two common options that Bokeh provides: generating a static HTML file and rendering your visualization inline in a Jupyter Notebook.

Set up the Figure(s)

From here, you’ll assemble your figure, preparing the canvas for your visualization. In this step, you can customize everything from the titles to the tick marks. You can also set up a suite of tools that can enable various user interactions with your visualization.

Connect to and Draw Your Data

Next, you’ll use Bokeh’s multitude of renderers to give shape to your data. Here, you have the flexibility to draw your data from scratch using the many available marker and shape options, all of which are easily customizable. This functionality gives you incredible creative freedom in representing your data.

Additionally, Bokeh has some built-in functionality for building things like stacked bar charts and plenty of examples for creating more advanced visualizations like network graphs and maps.

Organize the Layout

If you need more than one figure to express your data, Bokeh’s got you covered. Not only does Bokeh offer the standard grid-like layout options, but it also allows you to easily organize your visualizations into a tabbed layout in just a few lines of code.

In addition, your plots can be quickly linked together, so a selection on one will be reflected on any combination of the others.

Preview and Save Your Beautiful Data Creation

Finally, it’s time to see what you created.

Whether you’re viewing your visualization in a browser or notebook, you’ll be able to explore your visualization, examine your customizations, and play with any interactions that were added.

If you like what you see, you can save your visualization to an image file. Otherwise, you can revisit the steps above as needed to bring your data vision to reality.

That’s it! Those six steps are the building blocks for a tidy, flexible template that can be used to take your data from the table to the big screen:

"""Bokeh Visualization Template

This template is a general outline for turning your data into a 
visualization using Bokeh.
"""
# Data handling
import pandas as pd
import numpy as np

# Bokeh libraries
from bokeh.io import output_file, output_notebook
from bokeh.plotting import figure, show
from bokeh.models import ColumnDataSource
from bokeh.layouts import row, column, gridplot
from bokeh.models.widgets import Tabs, Panel

# Prepare the data

# Determine where the visualization will be rendered
output_file('filename.html')  # Render to static HTML, or 
output_notebook()  # Render inline in a Jupyter Notebook

# Set up the figure(s)
fig = figure()  # Instantiate a figure() object

# Connect to and draw the data

# Organize the layout

# Preview and save 
show(fig)  # See what I made, and save if I like it

Some common code snippets that are found in each step are previewed above, and you’ll see how to fill out the rest as you move through the rest of the tutorial!

Generating Your First Figure

There are multiple ways to output your visualization in Bokeh. In this tutorial, you’ll see these two options:

output_file('filename.html') will write the visualization to a static HTML file.
output_notebook() will render your visualization directly in a Jupyter Notebook.

It’s important to note that neither function will actually show you the visualization. That doesn’t happen until show() is called. However, they will ensure that, when show() is called, the visualization appears where you intend it to.

By calling both output_file() and output_notebook() in the same execution, the visualization will be rendered both to a static HTML file and inline in the notebook. However, if for whatever reason you run multiple output_file() commands in the same execution, only the last one will be used for rendering.

This is a great opportunity to give you your first glimpse at a default Bokeh figure() using output_file():

# Bokeh Libraries
from bokeh.io import output_file
from bokeh.plotting import figure, show

# The figure will be rendered in a static HTML file called output_file_test.html
output_file('output_file_test.html', 
            title='Empty Bokeh Figure')

# Set up a generic figure() object
fig = figure()

# See what it looks like
show(fig)

As you can see, a new browser window opened with a tab called Empty Bokeh Figure and an empty figure. Not shown is the file generated with the name output_file_test.html in your current working directory.

If you were to run the same code snippet with output_notebook() in place of output_file(), assuming you have a Jupyter Notebook fired up and ready to go, you will get the following:

# Bokeh Libraries
from bokeh.io import output_notebook
from bokeh.plotting import figure, show

# The figure will be right in my Jupyter Notebook
output_notebook()

# Set up a generic figure() object
fig = figure()

# See what it looks like
show(fig)

As you can see, the result is the same, just rendered in a different location.

More information about both output_file() and output_notebook() can be found in the Bokeh official docs.

Note: Sometimes, when rendering multiple visualizations sequentially, you’ll see that past renders are not being cleared with each execution. If you experience this, import and run the following between executions:

# Import reset_output (only needed once) 
from bokeh.plotting import reset_output

# Use reset_output() between subsequent show() calls, as needed
reset_output()

Before moving on, you may have noticed that the default Bokeh figure comes pre-loaded with a toolbar. This is an important sneak preview into the interactive elements of Bokeh that come right out of the box. You’ll find out more about the toolbar and how to configure it in the Adding Interaction section at the end of this tutorial.

Getting Your Figure Ready for Data

Now that you know how to create and view a generic Bokeh figure either in a browser or Jupyter Notebook, it’s time to learn more about how to configure the figure() object.

The figure() object is not only the foundation of your data visualization but also the object that unlocks all of Bokeh’s available tools for visualizing data. The Bokeh figure is a subclass of the Bokeh Plot object, which provides many of the parameters that make it possible to configure the aesthetic elements of your figure.

To show you just a glimpse into the customization options available, let’s create the ugliest figure ever:

# Bokeh Libraries
from bokeh.io import output_notebook
from bokeh.plotting import figure, show

# The figure will be rendered inline in my Jupyter Notebook
output_notebook()

# Example figure
fig = figure(background_fill_color='gray',
             background_fill_alpha=0.5,
             border_fill_color='blue',
             border_fill_alpha=0.25,
             plot_height=300,
             plot_width=500,
             h_symmetry=True,
             x_axis_label='X Label',
             x_axis_type='datetime',
             x_axis_location='above',
             x_range=('2018-01-01', '2018-06-30'),
             y_axis_label='Y Label',
             y_axis_type='linear',
             y_axis_location='left',
             y_range=(0, 100),
             title='Example Figure',
             title_location='right',
             toolbar_location='below',
             tools='save')

# See what it looks like
show(fig)

Once the figure() object is instantiated, you can still configure it after the fact. Let’s say you want to get rid of the gridlines:

# Remove the gridlines from the figure() object
fig.grid.grid_line_color = None

# See what it looks like 
show(fig)

The gridline properties are accessible via the figure’s grid attribute. In this case, setting grid_line_color to None effectively removes the gridlines altogether. More details about figure attributes can be found below the fold in the Plot class documentation.

Note: If you’re working in a notebook or IDE with auto-complete functionality, this feature can definitely be your friend! With so many customizable elements, it can be very helpful in discovering the available options:

Otherwise, doing a quick web search, with the keyword bokeh and what you are trying to do, will generally point you in the right direction.

There is tons more I could touch on here, but don’t feel like you’re missing out. I’ll make sure to introduce different figure tweaks as the tutorial progresses. Here are some other helpful links on the topic:

The Bokeh Plot Class is the superclass of the figure() object, from which figures inherit a lot of their attributes.
The Figure Class documentation is a good place to find more detail about the arguments of the figure() object.

Here are a few specific customization options worth checking out:

Text Properties covers all the attributes related to changing font styles, sizes, colors, and so forth.
TickFormatters are built-in objects specifically for formatting your axes using Python-like string formatting syntax.

Sometimes, it isn’t clear how your figure needs to be customized until it actually has some data visualized in it, so next you’ll learn how to make that happen.

Drawing Data With Glyphs

An empty figure isn’t all that exciting, so let’s look at glyphs: the building blocks of Bokeh visualizations. A glyph is a vectorized graphical shape or marker that is used to represent your data, like a circle or square. More examples can be found in the Bokeh gallery. After you create your figure, you are given access to a bevy of configurable glyph methods.

Let’s start with a very basic example, drawing some points on an x-y coordinate grid:

# Bokeh Libraries
from bokeh.io import output_file
from bokeh.plotting import figure, show

# My x-y coordinate data
x = [1, 2, 1]
y = [1, 1, 2]

# Output the visualization directly in the notebook
output_file('first_glyphs.html', title='First Glyphs')

# Create a figure with no toolbar and axis ranges of [0,3]
fig = figure(title='My Coordinates',
             plot_height=300, plot_width=300,
             x_range=(0, 3), y_range=(0, 3),
             toolbar_location=None)

# Draw the coordinates as circles
fig.circle(x=x, y=y,
           color='green', size=10, alpha=0.5)

# Show plot
show(fig)

Once your figure is instantiated, you can see how it can be used to draw the x-y coordinate data using customized circle glyphs.

Here are a few categories of glyphs:

Marker includes shapes like circles, diamonds, squares, and triangles and is effective for creating visualizations like scatter and bubble charts.
Line covers things like single, step, and multi-line shapes that can be used to build line charts.
Bar/Rectangle shapes can be used to create traditional or stacked bar (hbar) and column (vbar) charts as well as waterfall or gantt charts.

Information about the glyphs above, as well as others, can be found in Bokeh’s Reference Guide.

These glyphs can be combined as needed to fit your visualization needs. Let’s say I want to create a visualization that shows how many words I wrote per day to make this tutorial, with an overlaid trend line of the cumulative word count:

import numpy as np

# Bokeh libraries
from bokeh.io import output_notebook
from bokeh.plotting import figure, show

# My word count data
day_num = np.linspace(1, 10, 10)
daily_words = [450, 628, 488, 210, 287, 791, 508, 639, 397, 943]
cumulative_words = np.cumsum(daily_words)

# Output the visualization directly in the notebook
output_notebook()

# Create a figure with a datetime type x-axis
fig = figure(title='My Tutorial Progress',
             plot_height=400, plot_width=700,
             x_axis_label='Day Number', y_axis_label='Words Written',
             x_minor_ticks=2, y_range=(0, 6000),
             toolbar_location=None)

# The daily words will be represented as vertical bars (columns)
fig.vbar(x=day_num, bottom=0, top=daily_words, 
         color='blue', width=0.75, 
         legend='Daily')

# The cumulative sum will be a trend line
fig.line(x=day_num, y=cumulative_words, 
         color='gray', line_width=1,
         legend='Cumulative')

# Put the legend in the upper left corner
fig.legend.location = 'top_left'

# Let's check it out
show(fig)

To combine the columns and lines on the figure, they are simply created using the same figure() object.

Additionally, you can see above how seamlessly a legend can be created by setting the legend property for each glyph. The legend was then moved to the upper left corner of the plot by assigning 'top_left' to fig.legend.location.

You can check out much more info about styling legends. Teaser: they will show up again later in the tutorial when we start digging into interactive elements of the visualization.

A Quick Aside About Data

Anytime you are exploring a new visualization library, it’s a good idea to start with some data in a domain you are familiar with. The beauty of Bokeh is that nearly any idea you have should be possible. It’s just a matter of how you want to leverage the available tools to do so.

The remaining examples will use publicly available data from Kaggle, which has information about the National Basketball Association’s (NBA) 2017-18 season, specifically:

2017-18_playerBoxScore.csv: game-by-game snapshots of player statistics
2017-18_teamBoxScore.csv: game-by-game snapshots of team statistics
2017-18_standings.csv: daily team standings and rankings

This data has nothing to do with what I do for work, but I love basketball and enjoy thinking about ways to visualize the ever-growing amount of data associated with it.

If you don’t have data to play with from school or work, think about something you’re interested in and try to find some data related to that. It will go a long way in making both the learning and the creative process faster and more enjoyable!

To follow along with the examples in the tutorial, you can download the datasets from the links above and read them into a Pandas DataFrame using the following commands:

import pandas as pd

# Read the csv files
player_stats = pd.read_csv('2017-18_playerBoxScore.csv', parse_dates=['gmDate'])
team_stats = pd.read_csv('2017-18_teamBoxScore.csv', parse_dates=['gmDate'])
standings = pd.read_csv('2017-18_standings.csv', parse_dates=['stDate'])

This code snippet reads the data from the three CSV files and automatically interprets the date columns as datetime objects.

It’s now time to get your hands on some real data.

Using the `ColumnDataSource` Object

The examples above used Python lists and Numpy arrays to represent the data, and Bokeh is well equipped to handle these datatypes. However, when it comes to data in Python, you are most likely going to come across Python dictionaries and Pandas DataFrames, especially if you’re reading in data from a file or external data source.

Bokeh is well equipped to work with these more complex data structures and even has built-in functionality to handle them, namely the ColumnDataSource.

You may be asking yourself, “Why use a ColumnDataSource when Bokeh can interface with other data types directly?”

For one, whether you reference a list, array, dictionary, or DataFrame directly, Bokeh is going to turn it into a ColumnDataSource behind the scenes anyway. More importantly, the ColumnDataSource makes it much easier to implement Bokeh’s interactive affordances.

The ColumnDataSource is foundational in passing the data to the glyphs you are using to visualize. Its primary functionality is to map names to the columns of your data. This makes it easier for you to reference elements of your data when building your visualization. It also makes it easier for Bokeh to do the same when building your visualization.

The ColumnDataSource can interpret three types of data objects:

Python dict: The keys are names associated with the respective value sequences (lists, arrays, and so forth).
Pandas DataFrame: The columns of the DataFrame become the reference names for the ColumnDataSource.
Pandas groupby: The columns of the ColumnDataSource reference the columns as seen by calling groupby.describe().

Let’s start by visualizing the race for first place in the NBA’s Western Conference in 2017-18 between the defending champion Golden State Warriors and the challenger Houston Rockets. The daily win-loss records of these two teams is stored in a DataFrame named west_top_2:

>>>

>>> west_top_2 = (standings[(standings['teamAbbr'] == 'HOU') | (standings['teamAbbr'] == 'GS')]
...               .loc[:, ['stDate', 'teamAbbr', 'gameWon']]
...               .sort_values(['teamAbbr','stDate']))
>>> west_top_2.head()
        stDate teamAbbr  gameWon
9   2017-10-17       GS        0
39  2017-10-18       GS        0
69  2017-10-19       GS        0
99  2017-10-20       GS        1
129 2017-10-21       GS        1

From here, you can load this DataFrame into two ColumnDataSource objects and visualize the race:

# Bokeh libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource

# Output to file
output_file('west-top-2-standings-race.html', 
            title='Western Conference Top 2 Teams Wins Race')

# Isolate the data for the Rockets and Warriors
rockets_data = west_top_2[west_top_2['teamAbbr'] == 'HOU']
warriors_data = west_top_2[west_top_2['teamAbbr'] == 'GS']

# Create a ColumnDataSource object for each team
rockets_cds = ColumnDataSource(rockets_data)
warriors_cds = ColumnDataSource(warriors_data)

# Create and configure the figure
fig = figure(x_axis_type='datetime',
             plot_height=300, plot_width=600,
             title='Western Conference Top 2 Teams Wins Race, 2017-18',
             x_axis_label='Date', y_axis_label='Wins',
             toolbar_location=None)

# Render the race as step lines
fig.step('stDate', 'gameWon', 
         color='#CE1141', legend='Rockets', 
         source=rockets_cds)
fig.step('stDate', 'gameWon', 
         color='#006BB6', legend='Warriors', 
         source=warriors_cds)

# Move the legend to the upper left corner
fig.legend.location = 'top_left'

# Show the plot
show(fig)

Notice how the respective ColumnDataSource objects are referenced when creating the two lines. You simply pass the original column names as input parameters and specify which ColumnDataSource to use via the source property.

The visualization shows the tight race throughout the season, with the Warriors building a pretty big cushion around the middle of the season. However, a bit of a late-season slide allowed the Rockets to catch up and ultimately surpass the defending champs to finish the season as the Western Conference number-one seed.

Note: In Bokeh, you can specify colors either by name, hex value, or RGB color code.

For the visualization above, a color is being specified for the respective lines representing the two teams. Instead of using CSS color names like 'red' for the Rockets and 'blue' for the Warriors, you might have wanted to add a nice visual touch by using the official team colors in the form of hex color codes. Alternatively, you could have used tuples representing RGB color codes: (206, 17, 65) for the Rockets, (0, 107, 182) for the Warriors.

Bokeh provides a helpful list of CSS color names categorized by their general hue. Also, htmlcolorcodes.com is a great site for finding CSS, hex, and RGB color codes.

ColumnDataSource objects can do more than just serve as an easy way to reference DataFrame columns. The ColumnDataSource object has three built-in filters that can be used to create views on your data using a CDSView object:

GroupFilter selects rows from a ColumnDataSource based on a categorical reference value
IndexFilter filters the ColumnDataSource via a list of integer indices
BooleanFilter allows you to use a list of boolean values, with True rows being selected

In the previous example, two ColumnDataSource objects were created, one each from a subset of the west_top_2 DataFrame. The next example will recreate the same output from one ColumnDataSource based on all of west_top_2 using a GroupFilter that creates a view on the data:

# Bokeh libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource, CDSView, GroupFilter

# Output to file
output_file('west-top-2-standings-race.html', 
            title='Western Conference Top 2 Teams Wins Race')

# Create a ColumnDataSource
west_cds = ColumnDataSource(west_top_2)

# Create views for each team
rockets_view = CDSView(source=west_cds,
                       filters=[GroupFilter(column_name='teamAbbr', group='HOU')])
warriors_view = CDSView(source=west_cds,
                        filters=[GroupFilter(column_name='teamAbbr', group='GS')])

# Create and configure the figure
west_fig = figure(x_axis_type='datetime',
                  plot_height=300, plot_width=600,
                  title='Western Conference Top 2 Teams Wins Race, 2017-18',
                  x_axis_label='Date', y_axis_label='Wins',
                  toolbar_location=None)

# Render the race as step lines
west_fig.step('stDate', 'gameWon',
              source=west_cds, view=rockets_view,
              color='#CE1141', legend='Rockets')
west_fig.step('stDate', 'gameWon',
              source=west_cds, view=warriors_view,
              color='#006BB6', legend='Warriors')

# Move the legend to the upper left corner
west_fig.legend.location = 'top_left'

# Show the plot
show(west_fig)

Notice how the GroupFilter is passed to CDSView in a list. This allows you to combine multiple filters together to isolate the data you need from the ColumnDataSource as needed.

For information about integrating data sources, check out the Bokeh user guide’s post on the ColumnDataSource and other source objects available.

The Western Conference ended up being an exciting race, but say you want to see if the Eastern Conference was just as tight. Not only that, but you’d like to view them in a single visualization. This is a perfect segue to the next topic: layouts.

Organizing Multiple Visualizations With Layouts

The Eastern Conference standings came down to two rivals in the Atlantic Division: the Boston Celtics and the Toronto Raptors. Before replicating the steps used to create west_top_2, let’s try to put the ColumnDataSource to the test one more time using what you learned above.

In this example, you’ll see how to feed an entire DataFrame into a ColumnDataSource and create views to isolate the relevant data:

# Bokeh libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource, CDSView, GroupFilter

# Output to file
output_file('east-top-2-standings-race.html', 
            title='Eastern Conference Top 2 Teams Wins Race')

# Create a ColumnDataSource
standings_cds = ColumnDataSource(standings)

# Create views for each team
celtics_view = CDSView(source=standings_cds,
                      filters=[GroupFilter(column_name='teamAbbr', 
                                           group='BOS')])
raptors_view = CDSView(source=standings_cds,
                      filters=[GroupFilter(column_name='teamAbbr', 
                                           group='TOR')])

# Create and configure the figure
east_fig = figure(x_axis_type='datetime',
           plot_height=300, plot_width=600,
           title='Eastern Conference Top 2 Teams Wins Race, 2017-18',
           x_axis_label='Date', y_axis_label='Wins',
           toolbar_location=None)

# Render the race as step lines
east_fig.step('stDate', 'gameWon', 
              color='#007A33', legend='Celtics',
              source=standings_cds, view=celtics_view)
east_fig.step('stDate', 'gameWon', 
              color='#CE1141', legend='Raptors',
              source=standings_cds, view=raptors_view)

# Move the legend to the upper left corner
east_fig.legend.location = 'top_left'

# Show the plot
show(east_fig)

The ColumnDataSource was able to isolate the relevant data within a 5,040-by-39 DataFrame without breaking a sweat, saving a few lines of Pandas code in the process.

Looking at the visualization, you can see that the Eastern Conference race was no slouch. After the Celtics roared out of the gate, the Raptors clawed all the way back to overtake their division rival and finish the regular season with five more wins.

With our two visualizations ready, it’s time to put them together.

Similar to the functionality of Matplotlib’s subplot, Bokeh offers the column, row, and gridplot functions in its bokeh.layouts module. These functions can more generally be classified as layouts.

The usage is very straightforward. If you want to put two visualizations in a vertical configuration, you can do so with the following:

# Bokeh library
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.layouts import column

# Output to file
output_file('east-west-top-2-standings-race.html', 
            title='Conference Top 2 Teams Wins Race')

# Plot the two visualizations in a vertical configuration
show(column(west_fig, east_fig))

I’ll save you the two lines of code, but rest assured that swapping column for row in the snippet above will similarly configure the two plots in a horizontal configuration.

Note: If you’re trying out the code snippets as you go through the tutorial, I want to take a quick detour to address an error you may see accessing west_fig and east_fig in the following examples. In doing so, you may receive an error like this:

WARNING:bokeh.core.validation.check:W-1004 (BOTH_CHILD_AND_ROOT): Models should not be a document root...

This is one of many errors that are part of Bokeh’s validation module, where w-1004 in particular is warning about the re-use of west_fig and east_fig in a new layout.

To avoid this error as you test the examples, preface the code snippet illustrating each layout with the following:

# Bokeh libraries
from bokeh.plotting import figure, show
from bokeh.models import ColumnDataSource, CDSView, GroupFilter

# Create a ColumnDataSource
standings_cds = ColumnDataSource(standings)

# Create the views for each team
celtics_view = CDSView(source=standings_cds,
                      filters=[GroupFilter(column_name='teamAbbr', 
                                           group='BOS')])

raptors_view = CDSView(source=standings_cds,
                      filters=[GroupFilter(column_name='teamAbbr', 
                                           group='TOR')])

rockets_view = CDSView(source=standings_cds,
                      filters=[GroupFilter(column_name='teamAbbr', 
                                           group='HOU')])
warriors_view = CDSView(source=standings_cds,
                      filters=[GroupFilter(column_name='teamAbbr', 
                                           group='GS')])

# Create and configure the figure
east_fig = figure(x_axis_type='datetime',
                  plot_height=300,
                  x_axis_label='Date',
                  y_axis_label='Wins',
                  toolbar_location=None)

west_fig = figure(x_axis_type='datetime',
                  plot_height=300,
                  x_axis_label='Date',
                  y_axis_label='Wins',
                  toolbar_location=None)

# Configure the figures for each conference
east_fig.step('stDate', 'gameWon', 
              color='#007A33', legend='Celtics',
              source=standings_cds, view=celtics_view)
east_fig.step('stDate', 'gameWon', 
              color='#CE1141', legend='Raptors',
              source=standings_cds, view=raptors_view)

west_fig.step('stDate', 'gameWon', color='#CE1141', legend='Rockets',
              source=standings_cds, view=rockets_view)
west_fig.step('stDate', 'gameWon', color='#006BB6', legend='Warriors',
              source=standings_cds, view=warriors_view)

# Move the legend to the upper left corner
east_fig.legend.location = 'top_left'
west_fig.legend.location = 'top_left'

# Layout code snippet goes here!

Doing so will renew the relevant components to render the visualization, ensuring that no warning is needed.

Instead of using column or row, you may want to use a gridplot instead.

One key difference of gridplot is that it will automatically consolidate the toolbar across all of its children figures. The two visualizations above do not have a toolbar, but if they did, then each figure would have its own when using column or row. With that, it also has its own toolbar_location property, seen below set to 'right'.

Syntactically, you’ll also notice below that gridplot differs in that, instead of being passed a tuple as input, it requires a list of lists, where each sub-list represents a row in the grid:

# Bokeh libraries
from bokeh.io import output_file
from bokeh.layouts import gridplot

# Output to file
output_file('east-west-top-2-gridplot.html', 
            title='Conference Top 2 Teams Wins Race')

# Reduce the width of both figures
east_fig.plot_width = west_fig.plot_width = 300

# Edit the titles
east_fig.title.text = 'Eastern Conference'
west_fig.title.text = 'Western Conference'

# Configure the gridplot
east_west_gridplot = gridplot([[west_fig, east_fig]], 
                              toolbar_location='right')

# Plot the two visualizations in a horizontal configuration
show(east_west_gridplot)

Lastly, gridplot allows the passing of None values, which are interpreted as blank subplots. Therefore, if you wanted to leave a placeholder for two additional plots, then you could do something like this:

# Bokeh libraries
from bokeh.io import output_file
from bokeh.layouts import gridplot

# Output to file
output_file('east-west-top-2-gridplot.html', 
            title='Conference Top 2 Teams Wins Race')

# Reduce the width of both figures
east_fig.plot_width = west_fig.plot_width = 300

# Edit the titles
east_fig.title.text = 'Eastern Conference'
west_fig.title.text = 'Western Conference'

# Plot the two visualizations with placeholders
east_west_gridplot = gridplot([[west_fig, None], [None, east_fig]], 
                              toolbar_location='right')

# Plot the two visualizations in a horizontal configuration
show(east_west_gridplot)

If you’d rather toggle between both visualizations at their full size without having to squash them down to fit next to or on top of each other, a good option is a tabbed layout.

A tabbed layout consists of two Bokeh widget functions: Tab() and Panel() from the bokeh.models.widgets sub-module. Like using gridplot(), making a tabbed layout is pretty straightforward:

# Bokeh Library
from bokeh.io import output_file
from bokeh.models.widgets import Tabs, Panel

# Output to file
output_file('east-west-top-2-tabbed_layout.html', 
            title='Conference Top 2 Teams Wins Race')

# Increase the plot widths
east_fig.plot_width = west_fig.plot_width = 800

# Create two panels, one for each conference
east_panel = Panel(child=east_fig, title='Eastern Conference')
west_panel = Panel(child=west_fig, title='Western Conference')

# Assign the panels to Tabs
tabs = Tabs(tabs=[west_panel, east_panel])

# Show the tabbed layout
show(tabs)

The first step is to create a Panel() for each tab. That may sound a little confusing, but think of the Tabs() function as the mechanism that organizes the individual tabs created with Panel().

Each Panel() takes as input a child, which can either be a single figure() or a layout. (Remember that a layout is a general name for a column, row, or gridplot.) Once your panels are assembled, they can be passed as input to Tabs() in a list.

Now that you understand how to access, draw, and organize your data, it’s time to move on to the real magic of Bokeh: interaction! As always, check out Bokeh’s User Guide for more information on layouts.

Adding Interaction

The feature that sets Bokeh apart is its ability to easily implement interactivity in your visualization. Bokeh even goes as far as describing itself as an interactive visualization library:

Bokeh is an interactive visualization library that targets modern web browsers for presentation. (Source)

In this section, we’ll touch on five ways that you can add interactivity:

Configuring the toolbar
Selecting data points
Adding hover actions
Linking axes and selections
Highlighting data using the legend

Implementing these interactive elements open up possibilities for exploring your data that static visualizations just can’t do by themselves.

As you saw all the way back in Generating Your First Figure, the default Bokeh figure() comes with a toolbar right out of the box. The default toolbar comes with the following tools (from left to right):

Pan
Box Zoom
Wheel Zoom
Save
Reset
A link to Bokeh’s user guide for Configuring Plot Tools
A link to the Bokeh homepage

The toolbar can be removed by passing toolbar_location=None when instantiating a figure() object, or relocated by passing any of 'above', 'below', 'left', or 'right'.

Additionally, the toolbar can be configured to include any combination of tools you desire. Bokeh offers 18 specific tools across five categories:

Pan/Drag: box_select, box_zoom, lasso_select, pan, xpan, ypan, resize_select
Click/Tap: poly_select, tap
Scroll/Pinch: wheel_zoom, xwheel_zoom, ywheel_zoom
Actions: undo, redo, reset, save
Inspectors: crosshair, hover

To geek out on tools , make sure to visit Specifying Tools. Otherwise, they’ll be illustrated in covering the various interactions covered herein.

Selecting Data Points

Implementing selection behavior is as easy as adding a few specific keywords when declaring your glyphs.

The next example will create a scatter plot that relates a player’s total number of three-point shot attempts to the percentage made (for players with at least 100 three-point shot attempts).

The data can be aggregated from the player_stats DataFrame:

# Find players who took at least 1 three-point shot during the season
three_takers = player_stats[player_stats['play3PA'] > 0]

# Clean up the player names, placing them in a single column
three_takers['name'] = [f'{p["playFNm"]} {p["playLNm"]}' 
                        for _, p in three_takers.iterrows()]

# Aggregate the total three-point attempts and makes for each player
three_takers = (three_takers.groupby('name')
                            .sum()
                            .loc[:,['play3PA', 'play3PM']]
                            .sort_values('play3PA', ascending=False))

# Filter out anyone who didn't take at least 100 three-point shots
three_takers = three_takers[three_takers['play3PA'] >= 100].reset_index()

# Add a column with a calculated three-point percentage (made/attempted)
three_takers['pct3PM'] = three_takers['play3PM'] / three_takers['play3PA']

Here’s a sample of the resulting DataFrame:

>>>

>>> three_takers.sample(5)
                   name  play3PA  play3PM    pct3PM
229        Corey Brewer      110       31  0.281818
78           Marc Gasol      320      109  0.340625
126      Raymond Felton      230       81  0.352174
127  Kristaps Porziņģis      229       90  0.393013
66      Josh Richardson      336      127  0.377976

Let’s say you want to select a groups of players in the distribution, and in doing so mute the color of the glyphs representing the non-selected players:

# Bokeh Libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource, NumeralTickFormatter

# Output to file
output_file('three-point-att-vs-pct.html',
            title='Three-Point Attempts vs. Percentage')

# Store the data in a ColumnDataSource
three_takers_cds = ColumnDataSource(three_takers)

# Specify the selection tools to be made available
select_tools = ['box_select', 'lasso_select', 'poly_select', 'tap', 'reset']

# Create the figure
fig = figure(plot_height=400,
             plot_width=600,
             x_axis_label='Three-Point Shots Attempted',
             y_axis_label='Percentage Made',
             title='3PT Shots Attempted vs. Percentage Made (min. 100 3PA), 2017-18',
             toolbar_location='below',
             tools=select_tools)

# Format the y-axis tick labels as percentages
fig.yaxis[0].formatter = NumeralTickFormatter(format='00.0%')

# Add square representing each player
fig.square(x='play3PA',
           y='pct3PM',
           source=three_takers_cds,
           color='royalblue',
           selection_color='deepskyblue',
           nonselection_color='lightgray',
           nonselection_alpha=0.3)

# Visualize
show(fig)

First, specify the selection tools you want to make available. In the example above, 'box_select', 'lasso_select', 'poly_select', and 'tap' (plus a reset button) were specified in a list called select_tools. When the figure is instantiated, the toolbar is positioned 'below' the plot, and the list is passed to tools to make the tools selected above available.

Each player is initially represented by a royal blue square glyph, but the following configurations are set for when a player or group of players is selected:

Turn the selected player(s) to deepskyblue
Change all non-selected players’ glyphs to a lightgray color with 0.3 opacity

That’s it! With just a few quick additions, the visualization now looks like this:

For even more information about what you can do upon selection, check out Selected and Unselected Glyphs.

Adding Hover Actions

So the ability to select specific player data points that seem of interest in my scatter plot is implemented, but what if you want to quickly see what individual players a glyph represents? One option is to use Bokeh’s HoverTool() to show a tooltip when the cursor crosses paths with a glyph. All you need to do is append the following to the code snippet above:

# Bokeh Library
from bokeh.models import HoverTool

# Format the tooltip
tooltips = [
            ('Player','@name'),
            ('Three-Pointers Made', '@play3PM'),
            ('Three-Pointers Attempted', '@play3PA'),
            ('Three-Point Percentage','@pct3PM{00.0%}'),
           ]

# Add the HoverTool to the figure
fig.add_tools(HoverTool(tooltips=tooltips))

# Visualize
show(fig)

The HoverTool() is slightly different than the selection tools you saw above in that it has properties, specifically tooltips.

First, you can configure a formatted tooltip by creating a list of tuples containing a description and reference to the ColumnDataSource. This list was passed as input to the HoverTool() and then simply added to the figure using add_tools(). Here’s what happened:

Notice the addition of the Hover button to the toolbar, which can be toggled on and off.

If you want to even further emphasize the players on hover, Bokeh makes that possible with hover inspections. Here is a slightly modified version of the code snippet that added the tooltip:

# Format the tooltip
tooltips = [
            ('Player','@name'),
            ('Three-Pointers Made', '@play3PM'),
            ('Three-Pointers Attempted', '@play3PA'),
            ('Three-Point Percentage','@pct3PM{00.0%}'),
           ]

# Configure a renderer to be used upon hover
hover_glyph = fig.circle(x='play3PA', y='pct3PM', source=three_takers_cds,
                         size=15, alpha=0,
                         hover_fill_color='black', hover_alpha=0.5)

# Add the HoverTool to the figure
fig.add_tools(HoverTool(tooltips=tooltips, renderers=[hover_glyph]))

# Visualize
show(fig)

This is done by creating a completely new glyph, in this case circles instead of squares, and assigning it to hover_glyph. Note that the initial opacity is set to zero so that it is invisible until the cursor is touching it. The properties that appear upon hover are captured by setting hover_alpha to 0.5 along with the hover_fill_color.

Now you will see a small black circle appear over the original square when hovering over the various markers:

To further explore the capabilities of the HoverTool(), see the HoverTool and Hover Inspections guides.

Linking Axes and Selections

Linking is the process of syncing elements of different visualizations within a layout. For instance, maybe you want to link the axes of multiple plots to ensure that if you zoom in on one it is reflected on another. Let’s see how it is done.

For this example, the visualization will be able to pan to different segments of a team’s schedule and examine various game stats. Each stat will be represented by its own plot in a two-by-two gridplot() .

The data can be collected from the team_stats DataFrame, selecting the Philadelphia 76ers as the team of interest:

# Isolate relevant data
phi_gm_stats = (team_stats[(team_stats['teamAbbr'] == 'PHI') & 
                           (team_stats['seasTyp'] == 'Regular')]
                .loc[:, ['gmDate', 
                         'teamPTS', 
                         'teamTRB', 
                         'teamAST', 
                         'teamTO', 
                         'opptPTS',]]
                .sort_values('gmDate'))

# Add game number
phi_gm_stats['game_num'] = range(1, len(phi_gm_stats)+1)

# Derive a win_loss column
win_loss = []
for _, row in phi_gm_stats.iterrows():

    # If the 76ers score more points, it's a win
    if row['teamPTS'] > row['opptPTS']:
        win_loss.append('W')
    else:
        win_loss.append('L')

# Add the win_loss data to the DataFrame
phi_gm_stats['winLoss'] = win_loss

Here are the results of the 76ers’ first 5 games:

>>>

>>> phi_gm_stats.head()
        gmDate  teamPTS  teamTRB  teamAST  teamTO  opptPTS  game_num winLoss
10  2017-10-18      115       48       25      17      120         1       L
39  2017-10-20       92       47       20      17      102         2       L
52  2017-10-21       94       41       18      20      128         3       L
80  2017-10-23       97       49       25      21       86         4       W
113 2017-10-25      104       43       29      16      105         5       L

Start by importing the necessary Bokeh libraries, specifying the output parameters, and reading the data into a ColumnDataSource:

# Bokeh Libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource, CategoricalColorMapper, Div
from bokeh.layouts import gridplot, column

# Output to file
output_file('phi-gm-linked-stats.html',
                title='76ers Game Log')

# Store the data in a ColumnDataSource
gm_stats_cds = ColumnDataSource(phi_gm_stats)

Each game is represented by a column, and will be colored green if the result was a win and red for a loss. To accomplish this, Bokeh’s CategoricalColorMapper can be used to map the data values to specified colors:

# Create a CategoricalColorMapper that assigns a color to wins and losses
win_loss_mapper = CategoricalColorMapper(factors = ['W', 'L'], 
                                         palette=['green', 'red'])

For this use case, a list specifying the categorical data values to be mapped is passed to factors and a list with the intended colors to palette. For more on the CategoricalColorMapper, see the Colors section of Handling Categorical Data on Bokeh’s User Guide.

There are four stats to visualize in the two-by-two gridplot: points, assists, rebounds, and turnovers. In creating the four figures and configuring their respective charts, there is a lot of redundancy in the properties. So to streamline the code a for loop can be used:

# Create a dict with the stat name and its corresponding column in the data
stat_names = {'Points': 'teamPTS',
              'Assists': 'teamAST',
              'Rebounds': 'teamTRB',
              'Turnovers': 'teamTO',}

# The figure for each stat will be held in this dict
stat_figs = {}

# For each stat in the dict
for stat_label, stat_col in stat_names.items():

    # Create a figure
    fig = figure(y_axis_label=stat_label, 
                 plot_height=200, plot_width=400,
                 x_range=(1, 10), tools=['xpan', 'reset', 'save'])

    # Configure vbar
    fig.vbar(x='game_num', top=stat_col, source=gm_stats_cds, width=0.9, 
             color=dict(field='winLoss', transform=win_loss_mapper))

    # Add the figure to stat_figs dict
    stat_figs[stat_label] = fig

As you can see, the only parameters that needed to be adjusted were the y-axis-label of the figure and the data that will dictate top in the vbar. These values were easily stored in a dict that was iterated through to create the figures for each stat.

You can also see the implementation of the CategoricalColorMapper in the configuration of the vbar glyph. The color property is passed a dict with the field in the ColumnDataSource to be mapped and the name of the CategoricalColorMapper created above.

The initial view will only show the first 10 games of the 76ers’ season, so there needs to be a way to pan horizontally to navigate through the rest of the games in the season. Thus configuring the toolbar to have an xpan tool allows panning throughout the plot without having to worry about accidentally skewing the view along the vertical axis.

Now that the figures are created, gridplot can be setup by referencing the figures from the dict created above:

# Create layout
grid = gridplot([[stat_figs['Points'], stat_figs['Assists']], 
                [stat_figs['Rebounds'], stat_figs['Turnovers']]])

Linking the axes of the four plots is as simple as setting the x_range of each figure equal to one another:

# Link together the x-axes
stat_figs['Points'].x_range = \
    stat_figs['Assists'].x_range = \
    stat_figs['Rebounds'].x_range = \
    stat_figs['Turnovers'].x_range

To add a title bar to the visualization, you could have tried to do this on the points figure, but it would have been limited to the space of that figure. Therefore, a nice trick is to use Bokeh’s ability to interpret HTML to insert a Div element that contains the title information. Once that is created, simply combine that with the gridplot() in a column layout:

# Add a title for the entire visualization using Div
html = """<h3>Philadelphia 76ers Game Log</h3>
<b><i>2017-18 Regular Season</i>
<br>
</b><i>Wins in green, losses in red</i>
"""
sup_title = Div(text=html)

# Visualize
show(column(sup_title, grid))

Putting all the pieces together results in the following:

Similarly you can easily implement linked selections, where a selection on one plot will be reflected on others.

To see how this works, the next visualization will contain two scatter plots: one that shows the 76ers’ two-point versus three-point field goal percentage and the other showing the 76ers’ team points versus opponent points on a game-by-game basis.

The goal is to be able to select data points on the left-side scatter plot and quickly be able to recognize if the corresponding datapoint on the right scatter plot is a win or loss.

The DataFrame for this visualization is very similar to that from the first example:

# Isolate relevant data
phi_gm_stats_2 = (team_stats[(team_stats['teamAbbr'] == 'PHI') & 
                             (team_stats['seasTyp'] == 'Regular')]
                  .loc[:, ['gmDate', 
                           'team2P%', 
                           'team3P%', 
                           'teamPTS', 
                           'opptPTS']]
                  .sort_values('gmDate'))

# Add game number
phi_gm_stats_2['game_num'] = range(1, len(phi_gm_stats_2) + 1)

# Derive a win_loss column
win_loss = []
for _, row in phi_gm_stats_2.iterrows():

    # If the 76ers score more points, it's a win
    if row['teamPTS'] > row['opptPTS']:
        win_loss.append('W')
    else:
        win_loss.append('L')

# Add the win_loss data to the DataFrame
phi_gm_stats_2['winLoss'] = win_loss

Here’s what the data looks like:

>>>

>>> phi_gm_stats_2.head()
        gmDate  team2P%  team3P%  teamPTS  opptPTS  game_num winLoss
10  2017-10-18   0.4746   0.4286      115      120         1       L
39  2017-10-20   0.4167   0.3125       92      102         2       L
52  2017-10-21   0.4138   0.3333       94      128         3       L
80  2017-10-23   0.5098   0.3750       97       86         4       W
113 2017-10-25   0.5082   0.3333      104      105         5       L

The code to create the visualization is as follows:

# Bokeh Libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource, CategoricalColorMapper, NumeralTickFormatter
from bokeh.layouts import gridplot

# Output inline in the notebook
output_file('phi-gm-linked-selections.html',
            title='76ers Percentages vs. Win-Loss')

# Store the data in a ColumnDataSource
gm_stats_cds = ColumnDataSource(phi_gm_stats_2)

# Create a CategoricalColorMapper that assigns specific colors to wins and losses
win_loss_mapper = CategoricalColorMapper(factors = ['W', 'L'], palette=['Green', 'Red'])

# Specify the tools
toolList = ['lasso_select', 'tap', 'reset', 'save']

# Create a figure relating the percentages
pctFig = figure(title='2PT FG % vs 3PT FG %, 2017-18 Regular Season',
                plot_height=400, plot_width=400, tools=toolList,
                x_axis_label='2PT FG%', y_axis_label='3PT FG%')

# Draw with circle markers
pctFig.circle(x='team2P%', y='team3P%', source=gm_stats_cds, 
              size=12, color='black')

# Format the y-axis tick labels as percenages
pctFig.xaxis[0].formatter = NumeralTickFormatter(format='00.0%')
pctFig.yaxis[0].formatter = NumeralTickFormatter(format='00.0%')

# Create a figure relating the totals
totFig = figure(title='Team Points vs Opponent Points, 2017-18 Regular Season',
                plot_height=400, plot_width=400, tools=toolList,
                x_axis_label='Team Points', y_axis_label='Opponent Points')

# Draw with square markers
totFig.square(x='teamPTS', y='opptPTS', source=gm_stats_cds, size=10,
              color=dict(field='winLoss', transform=win_loss_mapper))

# Create layout
grid = gridplot([[pctFig, totFig]])

# Visualize
show(grid)

This is a great illustration of the power in using a ColumnDataSource. As long as the glyph renderers (in this case, the circle glyphs for the percentages, and square glyphs for the wins and losses) share the same ColumnDataSource, then the selections will be linked by default.

Here’s how it looks in action, where you can see selections made on either figure will be reflected on the other:

By selecting a random sample of data points in the upper right quadrant of the left scatter plot, those corresponding to both high two-point and three-point field goal percentage, the data points on the right scatter plot are highlighted.

Similarly, selecting data points on the right scatter plot that correspond to losses tend to be further to the lower left, lower shooting percentages, on the left scatter plot.

All the details on linking plots can be found at Linking Plots in the Bokeh User Guide.

Highlighting Data Using the Legend

That brings us to the final interactivity example in this tutorial: interactive legends.

In the Drawing Data With Glyphs section, you saw how easy it is to implement a legend when creating your plot. With the legend in place, adding interactivity is merely a matter of assigning a click_policy. Using a single line of code, you can quickly add the ability to either hide or mute data using the legend.

In this example, you’ll see two identical scatter plots comparing the game-by-game points and rebounds of LeBron James and Kevin Durant. The only difference will be that one will use a hide as its click_policy, while the other uses mute.

The first step is to configure the output and set up the data, creating a view for each player from the player_stats DataFrame:

# Bokeh Libraries
from bokeh.plotting import figure, show
from bokeh.io import output_file
from bokeh.models import ColumnDataSource, CDSView, GroupFilter
from bokeh.layouts import row

# Output inline in the notebook
output_file('lebron-vs-durant.html',
            title='LeBron James vs. Kevin Durant')

# Store the data in a ColumnDataSource
player_gm_stats = ColumnDataSource(player_stats)

# Create a view for each player
lebron_filters = [GroupFilter(column_name='playFNm', group='LeBron'),
                  GroupFilter(column_name='playLNm', group='James')]
lebron_view = CDSView(source=player_gm_stats,
                      filters=lebron_filters)

durant_filters = [GroupFilter(column_name='playFNm', group='Kevin'),
                  GroupFilter(column_name='playLNm', group='Durant')]
durant_view = CDSView(source=player_gm_stats,
                      filters=durant_filters)

Before creating the figures, the common parameters across the figure, markers, and data can be consolidated into dictionaries and reused. Not only does this save redundancy in the next step, but it provides an easy way to tweak these parameters later if need be:

# Consolidate the common keyword arguments in dicts
common_figure_kwargs = {
    'plot_width': 400,
    'x_axis_label': 'Points',
    'toolbar_location': None,
}
common_circle_kwargs = {
    'x': 'playPTS',
    'y': 'playTRB',
    'source': player_gm_stats,
    'size': 12,
    'alpha': 0.7,
}
common_lebron_kwargs = {
    'view': lebron_view,
    'color': '#002859',
    'legend': 'LeBron James'
}
common_durant_kwargs = {
    'view': durant_view,
    'color': '#FFC324',
    'legend': 'Kevin Durant'
}

Now that the various properties are set, the two scatter plots can be built in a much more concise fashion:

# Create the two figures and draw the data
hide_fig = figure(**common_figure_kwargs,
                  title='Click Legend to HIDE Data', 
                  y_axis_label='Rebounds')
hide_fig.circle(**common_circle_kwargs, **common_lebron_kwargs)
hide_fig.circle(**common_circle_kwargs, **common_durant_kwargs)

mute_fig = figure(**common_figure_kwargs, title='Click Legend to MUTE Data')
mute_fig.circle(**common_circle_kwargs, **common_lebron_kwargs,
                muted_alpha=0.1)
mute_fig.circle(**common_circle_kwargs, **common_durant_kwargs,
                muted_alpha=0.1)

Note that mute_fig has an extra parameter called muted_alpha. This parameter controls the opacity of the markers when mute is used as the click_policy.

Finally, the click_policy for each figure is set, and they are shown in a horizontal configuration:

# Add interactivity to the legend
hide_fig.legend.click_policy = 'hide'
mute_fig.legend.click_policy = 'mute'

# Visualize
show(row(hide_fig, mute_fig))

Once the legend is in place, all you have to do is assign either hide or mute to the figure’s click_policy property. This will automatically turn your basic legend into an interactive legend.

Also note that, specifically for mute, the additional property of muted_alpha was set in the respective circle glyphs for LeBron James and Kevin Durant. This dictates the visual effect driven by the legend interaction.

For more on all things interaction in Bokeh, Adding Interactions in the Bokeh User Guide is a great place to start.

Summary and Next Steps

Congratulations! You’ve made it to the end of this tutorial.

You should now have a great set of tools to start turning your data into beautiful interactive visualizations using Bokeh.

You learned how to:

Configure your script to render to either a static HTML file or Jupyter Notebook
Instantiate and customize the figure() object
Build your visualization using glyphs
Access and filter your data with the ColumnDataSource
Organize multiple plots in grid and tabbed layouts
Add different forms of interaction, including selections, hover actions, linking, and interactive legends

To explore even more of what Bokeh is capable of, the official Bokeh User Guide is an excellent place to dig into some more advanced topics. I’d also recommend checking out Bokeh’s Gallery for tons of examples and inspiration.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Kenneth Reitz

2018-11-14T14:00:00+00:00

This week, I’m excited to be interviewing the prolific Kenneth Reitz!

Kenneth is the author of the extremely popular requests and pipenv libraries. Join us as we talk about his latest projects and the most challenging code he’s written to date.

Ricky: Let’s start at the beginning… How did you get into programming, and when did you start using Python?

Kenneth: I started programming at a young age. My dad was a programmer, and I taught myself BASIC and C (with this help) at the age of 9. I started using Python in college, when I took my first CS course. Shortly after, I dropped out and learned many other programming languages, but I always kept coming back to Python.

Ricky: Congratulations on your new job with Digital Ocean. You’re the senior member of the Developer Relations team. How are you finding the change in role from your previous job at Heroku, and what can we expect from Digital Ocean moving forward in the Python space?

Kenneth: Thanks! I’m really enjoying the new role and the opportunity to serve the entire development community, not just the Python community. However, my latest work, Responder, has been a Digital Ocean project, so there’s room to expect more from us in the Python space 😊

Ricky: You are, of course, most famous for writing the extremely popular requests library and the new pipenv library. Python.org now recommends the use of pipenv for dependency management. How has the community received pipenv? Have you seen much resistance from the community with developers preferring to stick to venv or older methods of dependency management?

Kenneth: The community has received pipenv very well, and even companies like GitHub are using its standards for security vulnerability scanning. I haven’t seen much resistance from the community at all, aside from some hatred on reddit. It took me a while to realize that /r/python doesn’t represent the Python community as much as it represents redditors who use Python.

Ricky: Now hitting 300 million downloads on your requests library is cool and all, but as a guitarist, what I’m more excited about is your latest project PyTheory. Can you tell us a little about it and your plans for the project going forward?

Kenneth: PyTheory is a very interesting library that attempts to encapsulate all known musical systems into a library. Currently, there is one system: Western. It can render all the different scales for the Western system, programmatically, and tell you the pitches of the notes (either in decimal or symbolic notation). In addition, there are fretboards and chord charts, so you can specify a custom tuning for your guitar, and generate chord charts with it. It’s very abstract.

Definitely the most challenging thing I’ve ever written.

Ricky: So after recently ditching your Mac for a PC, and turning to Microsoft’s (amazing) VS Code for your Python development, are you happy and proud to be a Windows user? For those who may be reading who haven’t used Windows since Windows ‘95, what are they missing?

Kenneth: I love the Mac and prefer it to Windows. I’m just currently bored with my setup and decided to challenge myself by running Windows. I’m very happy and productive, though. Feels right at home.

Windows isn’t what it used to be. It’s a real solid piece of operating system now. I’m running it on my iMac Pro, and it hums like a dream.

Ricky: I know you’re a keen photographer. How long have you been into it, and what’s your favorite photo you’ve taken? Do you have any other hobbies and interests, aside from Python?

Kenneth: I’ve been into photography seriously for about 10 years. My favorite photo I’ve ever taken is probably this one. It was taken on a film camera, after I had a migraine for several weeks, and it was my first time being able to walk outside.

Ricky: Finally, any parting words of wisdom? Anything you’d like to share and/or plug?

Kenneth: Responder! My new web service framework for Python. It’s ASGI, it’s familiar looking, it’s fast, and it’s easier to use than Flask! Great for building APIs. Check it out!

Thank you, Kenneth, for joining me this week. It’s been great to watch the development of Responder happen in realtime on Twitter. You can follow its development or raise an issue here.

As always, if there is someone you would like me to interview in the future, reach out to me in the comments below, or send me a message on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Publish an Open-Source Python Package to PyPI

2018-11-12T14:00:00+00:00

Python is famous for coming with batteries included. Sophisticated capabilities are available in the standard library. You can find modules for working with sockets, parsing CSV, JSON, and XML files, and working with files and file paths.

However great the packages included with Python are, there are many fantastic projects available outside the standard library. These are most often hosted at the Python Packaging Index (PyPI), historically known as the Cheese Shop. At PyPI, you can find everything from Hello World to advanced deep learning libraries.

In this tutorial, you’ll cover how to upload your own package to PyPI. While getting your project published is easier than it used to be, there are still a few steps involved.

You’ll learn how to:

Prepare your Python package for publication
Think about versioning
Upload your package to PyPI

Throughout this tutorial, we’ll use a simple example project: a reader package that can be used to read Real Python tutorials. The first section introduces this project.

A Small Python Package

This section will describe a small Python package that we’ll use as an example that can be published to PyPI. If you already have a package that you want to publish, feel free to skim this section and join up again at the beginning of the next section.

The package that we’ll use is called reader and is an application that can be used to download and read Real Python articles. If you want to follow along, you can get the full source code from our GitHub repository.

Note: The source code as shown and explained below is a simplified, but fully functional, version of the Real Python feed reader. Compared to the package published on PyPI and GitHub, this version lacks some error handling and extra options.

First, have a look at the directory structure of reader. The package lives completely inside a directory that is also named reader:

reader/
│
├── reader/
│   ├── config.txt
│   ├── feed.py
│   ├── __init__.py
│   ├── __main__.py
│   └── viewer.py
│
├── tests/
│   ├── test_feed.py
│   └── test_viewer.py
│
├── MANIFEST.in
├── README.md
└── setup.py

The source code of the package is in a reader subdirectory together with a configuration file. There are a few tests in a separate subdirectory. The tests will not be covered here, but you can find them in the GitHub repository. To learn more about testing, see Anthony Shaw’s great tutorial on Getting Started With Testing in Python.

If you’re working with your own package, you may use a different structure or have other files in your package directory. Our Python Application Layouts reference discusses several different options. The instructions in this guide will work independently of the layout you use.

In the rest of this section, you’ll see how the reader package works. In the next section, you’ll get a closer look at the special files, including setup.py, README.md, and MANIFEST.in, that are needed to publish your package.

Using the Real Python Reader

reader is a very basic web feed reader that can download the latest Real Python articles from the Real Python feed.

Here is an example of using the reader to get the list of the latest articles:

$ python -m reader
The latest tutorials from Real Python (https://realpython.com/)
  0 How to Publish an Open-Source Python Package to PyPI
  1 Python "while" Loops (Indefinite Iteration)
  2 Writing Comments in Python (Guide)
  3 Setting Up Python for Machine Learning on Windows
  4 Python Community Interview With Michael Kennedy
  5 Practical Text Classification With Python and Keras
  6 Getting Started With Testing in Python
  7 Python, Boto3, and AWS S3: Demystified
  8 Python's range() Function (Guide)
  9 Python Community Interview With Mike Grouchy
 10 How to Round Numbers in Python
 11 Building and Documenting Python REST APIs With Flask and Connexion – Part 2
 12 Splitting, Concatenating, and Joining Strings in Python
 13 Image Segmentation Using Color Spaces in OpenCV + Python
 14 Python Community Interview With Mahdi Yusuf
 15 Absolute vs Relative Imports in Python
 16 Top 10 Must-Watch PyCon Talks
 17 Logging in Python
 18 The Best Python Books
 19 Conditional Statements in Python

Notice that each article is numbered. To read one particular article, you use the same command but include the number of the article as well. For instance, to read How to Publish an Open-Source Python Package to PyPI, you add 0 to the command:

$ python -m reader 0
# How to Publish an Open-Source Python Package to PyPI

Python is famous for coming with batteries included. Sophisticated
capabilities are available in the standard library. You can find modules
for working with sockets, parsing CSV, JSON, and XML files, and
working with files and file paths.

However great the packages included with Python are, there are many
fantastic projects available outside the standard library. These are
most often hosted at the Python Packaging Index (PyPI), historically
known as the Cheese Shop. At PyPI, you can find everything from Hello
World to advanced deep learning libraries.

[... The full text of the article ...]

This prints the full article to the console using the Markdown text format.

Note: python -m is used to run a library module or package instead of a script. If you run a package, the contents of the file __main__.py will be executed. See Different Ways of Calling a Package for more info.

By changing the article number, you can read any of the available articles.

A Quick Look at the Code

The details of how reader works are not important for the purpose of this tutorial. However, if you are interested in seeing the implementation, you can expand the sections below. The package consists of five files:

config.txt is a configuration file used to specify the URL of the feed of Real Python tutorials. It’s a text file that can be read by the configparser standard library:

# config.txt

[feed]
url = https://realpython.com/atom.xml

In general, such a config file contains key-value pairs separated into sections. This particular file contains only one section (feed) and one key (url).

Note: A configuration file is probably overkill for this simple package. We include it here for demonstration purposes.

The first source code file we’ll look at is __main__.py. The double underscores indicate that this file has a special meaning in Python. Indeed, when running a package as a script with -m as above, Python executes the contents of the __main__.py file.

In other words, __main__.py acts as the entry point of our program and takes care of the main flow, calling other parts as needed:

# __main__.py

from configparser import ConfigParser
from importlib import resources  # Python 3.7+
import sys

from reader import feed
from reader import viewer

def main():
    """Read the Real Python article feed"""
    # Read URL of the Real Python feed from config file
    cfg = ConfigParser()
    cfg.read_string(resources.read_text("reader", "config.txt"))
    url = cfg.get("feed", "url")

    # If an article ID is given, show the article
    if len(sys.argv) > 1:
        article = feed.get_article(url, sys.argv[1])
        viewer.show(article)

    # If no ID is given, show a list of all articles
    else:
        site = feed.get_site(url)
        titles = feed.get_titles(url)
        viewer.show_list(site, titles)

if __name__ == "__main__":
    main()

Notice that main() is called on the last line. If we do not call main(), then our program would not do anything. As you saw earlier, the program can either list all articles or print one specific article. This is handled by the if-else inside main().

To read the URL to the feed from the configuration file, we use configparser and importlib.resources. The latter is used to import non-code (or resource) files from a package without having to worry about the full file path. It is especially helpful when publishing packages to PyPI where resource files might end up inside binary archives.

importlib.resources became a part of the standard library in Python 3.7. If you are using an older version of Python, you can use importlib_resources instead. This is a backport compatible with Python 2.7, and 3.4 and above. importlib_resources can be installed from PyPI:

$ pip install importlib_resources

See Barry Warzaw’s presentation at PyCon 2018 for more information.

The next file is __init__.py. Again, the double underscores in the filename tell us that this is a special file. __init__.py represents the root of your package. It should usually be kept quite simple, but it’s a good place to put package constants, documentation, and so on:

# __init__.py

# Version of the realpython-reader package
__version__ = "1.0.0"

The special variable __version__ is a convention in Python for adding version numbers to your package. It was introduced in PEP 396. We’ll talk more about versioning later.

Variables defined in __init__.py become available as variables in the package namespace:

>>>

>>> import reader
>>> reader.__version__
'1.0.0'

You should define the __version__ variable in your own packages as well.

Looking at __main__.py, you’ll see that two modules, feed and viewer, are imported and used to read from the feed and show the results. These modules do most of the actual work.

First consider feed.py. This file contains functions for reading from a web feed and parsing the result. Luckily there are already great libraries available to do this. feed.py depends on two modules that are already available on PyPI: feedparser and html2text.

feed.py contains several functions. We’ll discuss them one at a time.

To avoid reading from the web feed more than necessary, we first create a function that remembers the feed the first time it’s read:

 1 # feed.py
 2 
 3 import feedparser
 4 import html2text
 5 
 6 _CACHED_FEEDS = dict()
 7 
 8 def _feed(url):
 9     """Only read a feed once, by caching its contents"""
10     if url not in _CACHED_FEEDS:
11         _CACHED_FEEDS[url] = feedparser.parse(url)
12     return _CACHED_FEEDS[url]

feedparser.parse() reads a feed from the web and returns it in a structure that looks like a dictionary. To avoid downloading the feed more than once, it’s stored in _CACHED_FEEDS and reused for later calls to _feed(). Both _CACHED_FEEDS and _feed() are prefixed by an underscore to indicate that they are support objects not meant to be used directly.

We can get some basic information about the feed by looking in the .feed metadata. The following function picks out the title and link to the web site containing the feed:

14 def get_site(url):
15     """Get name and link to web site of the feed"""
16     info = _feed(url).feed
17     return f"{info.title} ({info.link})"

In addition to .title and .link, attributes like .subtitle, .updated, and .id are also available.

The articles available in the feed can be found inside the .entries list. Article titles can be found with a list comprehension:

19 def get_titles(url):
20     """List titles in feed"""
21     articles = _feed(url).entries
22     return [a.title for a in articles]

.entries lists the articles in the feed sorted chronologically, so that the newest article is .entries[0].

In order to get the contents of one article, we use its index in the .entries list as an article ID:

24 def get_article(url, article_id):
25     """Get article from feed with the given ID"""
26     articles = _feed(url).entries
27     article = articles[int(article_id)]
28     html = article.content[0].value
29     text = html2text.html2text(html)
30     return f"# {article.title}\n\n{text}"

After picking the correct article out of the .entries list, we find the text of the article as HTML on line 28. Next, html2text does a decent job of translating the HTML into much more readable text. As the HTML doesn’t contain the title of the article, the title is added before returning.

The final module is viewer.py. At the moment, it consists of two very simple functions. In practice, we could have used print() directly in __main__.py instead of calling viewer functions. However, having the functionality split off makes it easier to replace it later with something more advanced. Maybe we could add a GUI interface in a later version?

viewer.py contains two functions:

# viewer.py

def show(article):
    """Show one article"""
    print(article)

def show_list(site, titles):
    """Show list of articles"""
    print(f"The latest tutorials from {site}")
    for article_id, title in enumerate(titles):
        print(f"{article_id:>3} {title}")

show() simply prints one article to the console, while show_list() prints a list of titles. The latter also creates article IDs that are used when choosing to read one particular article.

Different Ways of Calling a Package

One challenge when your projects grow in complexity is communicating to the user how to use your project. Since the package consists of four different source code files, how does the user know which file to call to run reader?

The python interpreter program has an -m option that allows you to specify a module name instead of a file name. For instance, if you have a script called hello.py, the following two commands are equivalent:

$ python hello.py
Hi there!

$ python -m hello
Hi there!

One advantage of the latter is that it allows you to call modules that are built into Python as well. One example is calling antigravity:

$ python -m antigravity
Created new window in existing browser session.

Another advantage of using -m is that it works for packages as well as modules. As you saw earlier, you can call the reader package with -m:

$ python -m reader
[...]

Since reader is a package, the name only refers to a directory. How does Python decide which code inside that directory to run? It looks for a file named __main__.py. If such a file exists, it is executed. If __main__.py does not exist, then an error message is printed:

$ python -m math
python: No code object available for math

In this example, you see that the math standard library has not defined a __main__.py file.

If you are creating a package that is supposed to be executed, you should include a __main__.py file. Later, you’ll see how you can also create entry points to your package that will behave like regular programs.

Preparing Your Package for Publication

Now you’ve got a package you want to publish, or maybe you copied our package. Which steps are necessary before you can upload the package to PyPI?

Naming Your Package

The first—and possibly the hardest—step is to come up with a good name for your package. All packages on PyPI need to have unique names. With more than 150,000 packages already on PyPI, chances are that your favorite name is already taken.

You might need to brainstorm and do some research to find the perfect name. Use the PyPI search to check if a name is already taken. The name that you come up with will be visible on PyPI.

To make the reader package easier to find on PyPI, we give it a more descriptive name and call it realpython-reader. The same name will be used to install the package using pip:

$ pip install realpython-reader

Even though we use realpython-reader as the PyPI name, the package is still called reader when it’s imported:

>>>

>>> import reader
>>> help(reader)

>>> from reader import feed
>>> feed.get_titles()
['How to Publish an Open-Source Python Package to PyPI', ...]

As you see, you can use different names for your package on PyPI and when importing. However, if you use the same name or very similar names, then it will be easier for your users.

Configuring Your Package

In order for your package to be uploaded to PyPI, you need to provide some basic information about it. This information is typically provided in the form of a setup.py file. There are initiatives that try to simplify this collection of information. At the moment though, setup.py is the only fully supported way of providing information about your package.

The setup.py file should be placed in the top folder of your package. A fairly minimal setup.py for reader looks like this:

import pathlib
from setuptools import setup

# The directory containing this file
HERE = pathlib.Path(__file__).parent

# The text of the README file
README = (HERE / "README.md").read_text()

# This call to setup() does all the work
setup(
    name="realpython-reader",
    version="1.0.0",
    description="Read the latest Real Python tutorials",
    long_description=README,
    long_description_content_type="text/markdown",
    url="https://github.com/realpython/reader",
    author="Real Python",
    author_email="office@realpython.com",
    license="MIT",
    classifiers=[
        "License :: OSI Approved :: MIT License",
        "Programming Language :: Python :: 3",
        "Programming Language :: Python :: 3.7",
    ],
    packages=["reader"],
    include_package_data=True,
    install_requires=["feedparser", "html2text"],
    entry_points={
        "console_scripts": [
            "realpython=reader.__main__:main",
        ]
    },
)

We will only cover some of the options available in setuptools here. The documentation does a good job of going into all the detail.

The parameters that are 100% necessary in the call to setup() are the following:

name: the name of your package as it will appear on PyPI
version: the current version of your package
packages: the packages and subpackages containing your source code

We will talk more about versions later. The packages parameter takes a list of packages. In our example, there is only one package: reader.

You also need to specify any subpackages. In more complicated projects, there might be many packages to list. To simplify this job, setuptools includes find_packages(), which does a good job of discovering all your subpackages. You could have used find_packages() in the reader project as follows:

from setuptools import find_packages, setup

setup(
    ...
    packages=find_packages(exclude=("tests",)),
    ...
)

While only name, version, and packages are required, your package becomes much easier to find on PyPI if you add some more information. Have a look at the realpython-reader page on PyPI and compare the information with setup.py above. All the information comes from setup.py and README.md.

realpython-reader package at PyPI"/>

The last two parameters to setup() deserve special mention:

install_requires is used to list any dependencies your package has to third party libraries. The reader depends on feedparser and html2text, so they should be listed here.
entry_points is used to create scripts that call a function within your package. In our example, we create a new script realpython that calls main() within the reader/__main__.py file.

For another example of a typical setup file, see Kenneth Reitz’s setup.py repository on GitHub.

Documenting Your Package

Before releasing your package to the world, you should add some documentation. Depending on your package, the documentation can be as small as a simple README file, or as big as a full web page with tutorials, example galleries, and an API reference.

At a minimum, you should include a README file with your project. A good README should quickly describe your project, as well as tell your users how to install and use your package. Typically, you want to include your README as the long_description argument to setup(). This will display your README on PyPI.

Traditionally, PyPI has used reStructuredText for package documentation. However, since March 2018 Markdown has also been supported.

Outside of PyPI, Markdown is more widely supported than reStructuredText. If you don’t need any of the special features of reStructuredText, you’ll be better off keeping your README in Markdown. Note that you should use the setup() parameter long_description_content_type to tell PyPI which format you are using. Valid values are text/markdown, text/x-rst, and text/plain.

For bigger projects, you might want to offer more documentation than can reasonably fit in a single file. In that case, you can use sites like GitHub or Read the Docs, and link to the documentation using the url parameter. In the setup.py example above, url is used to link to the reader GitHub repository.

Versioning Your Package

Your package needs to have a version, and PyPI will only let you do one upload of a particular version for a package. In other words, if you want to update your package on PyPI, you need to increase the version number first. This is a good thing, as it guarantees reproducibility: two systems with the same version of a package should behave the same.

There are many different schemes that can be used for your version number. For Python projects, PEP 440 gives some recommendations. However, in order to be flexible, that PEP is complicated. For a simple project, stick with a simple versioning scheme.

Semantic versioning is a good default scheme to use. The version number is given as three numerical components, for instance 0.1.2. The components are called MAJOR, MINOR, and PATCH, and there are simple rules about when to increment each component:

Increment the MAJOR version when you make incompatible API changes.

Increment the MINOR version when you add functionality in a backwards-compatible manner.

Increment the PATCH version when you make backwards-compatible bug fixes. (Source)

You may need to specify the version in different files inside your project. In the reader project, we specified the version both in setup.py and in reader/__init__.py. To make sure the version numbers are kept consistent, you can use a tool called Bumpversion.

You can install Bumpversion from PyPI:

$ pip install bumpversion

To increment the MINOR version of reader, you would do something like this:

$ bumpversion --current-version 1.0.0 minor setup.py reader/__init__.py

This would change the version number from 1.0.0 to 1.1.0 in both setup.py and reader/__init__.py. To simplify the command, you can also give most of the information in a configuration file. See the Bumpversion documentation for details.

Adding Files to Your Package

Sometimes, you’ll have files inside your package that are not source code files. Examples include data files, binaries, documentation, and—as we have in this project—configuration files.

To tell setup() to include such files, you use a manifest file. For many projects, you don’t need to worry about the manifest, as setup() creates one that includes all code files as well as README files.

If you need to change the manifest, you create a manifest template which must be named MANIFEST.in. This file specifies rules for what to include and exclude:

include reader/*.txt

This example will include all .txt files in the reader directory, which in effect is the configuration file. See the documentation for a list of available rules.

In addition to creating MANIFEST.in, you also need to tell setup() to copy these non-code files. This is done by setting the include_package_data argument to True:

setup(
    ...
    include_package_data=True,
    ...
)

The include_package_data argument controls whether non-code files are copied when your package is installed.

Publishing to PyPI

Your package is finally ready to meet the world outside your computer! In this section, you’ll see how to actually upload your package to PyPI.

If you don’t already have an account on PyPI, now is the time to create one: register your account on PyPI. While you’re at it, you should also register an account on TestPyPI. TestPyPI is very useful, as you can try all the steps of publishing a package without any consequences if you mess up.

To upload your package to PyPI, you’ll use a tool called Twine. You can install Twine using Pip as usual:

$ pip install twine

Using Twine is quite simple, and you will soon see how to use it to check and publish your package.

Building Your Package

Packages on PyPI are not distributed as plain source code. Instead, they are wrapped into distribution packages. The most common formats for distribution packages are source archives and Python wheels.

A source archive consists of your source code and any supporting files wrapped into one tar file. Similarly, a wheel is essentially a zip archive containing your code. In contrast to the source archive, the wheel includes any extensions ready to use.

To create a source archive and a wheel for your package, you can run the following command:

$ python setup.py sdist bdist_wheel

This will create two files in a newly created dist directory, a source archive and a wheel:

reader/
│
└── dist/
    ├── realpython_reader-1.0.0-py3-none-any.whl
    └── realpython-reader-1.0.0.tar.gz

Note: On Windows, the source archive will be a .zip file by default. You can choose the format of the source archive with the --format command line option.

You might wonder how setup.py knows what to do with the sdist and bdist_wheel arguments. If you look back to how setup.py was implemented, there is no mention of sdist, bdist_wheel, or any other command line arguments.

All the command line arguments are instead implemented in the upstream distutils standard library. You can list all available arguments by adding the --help-commands option:

$ python setup.py --help-commands
Standard commands:
  build             build everything needed to install
  build_py          "build" pure Python modules (copy to build directory)
  build_ext         build C/C++ and Cython extensions (compile/link to build directory)
< ... many more commands ...>

For information about one particular command, you can do something like python setup.py sdist --help.

Testing Your Package

First, you should check that the newly built distribution packages contain the files you expect. On Linux and macOS, you should be able to list the contents of the tar source archive as follows:

$ tar tzf realpython-reader-1.0.0.tar.gz
realpython-reader-1.0.0/
realpython-reader-1.0.0/setup.cfg
realpython-reader-1.0.0/README.md
realpython-reader-1.0.0/reader/
realpython-reader-1.0.0/reader/feed.py
realpython-reader-1.0.0/reader/__init__.py
realpython-reader-1.0.0/reader/viewer.py
realpython-reader-1.0.0/reader/__main__.py
realpython-reader-1.0.0/reader/config.txt
realpython-reader-1.0.0/PKG-INFO
realpython-reader-1.0.0/setup.py
realpython-reader-1.0.0/MANIFEST.in
realpython-reader-1.0.0/realpython_reader.egg-info/
realpython-reader-1.0.0/realpython_reader.egg-info/SOURCES.txt
realpython-reader-1.0.0/realpython_reader.egg-info/requires.txt
realpython-reader-1.0.0/realpython_reader.egg-info/dependency_links.txt
realpython-reader-1.0.0/realpython_reader.egg-info/PKG-INFO
realpython-reader-1.0.0/realpython_reader.egg-info/entry_points.txt
realpython-reader-1.0.0/realpython_reader.egg-info/top_level.txt

On Windows, you can use a utility like 7-zip to look inside the corresponding zip file.

You should see all your source code listed, as well as a few new files that have been created containing information you provided in setup.py. In particular, make sure that all subpackages and supporting files are included.

You can also have a look inside the wheel by unzipping it as if it were a zip file. However, if your source archive contains the files you expect, the wheel should be fine as well.

Newer versions of Twine (1.12.0 and above) can also check that your package description will render properly on PyPI. You can run twine check on the files created in dist:

$ twine check dist/*
Checking distribution dist/realpython_reader-1.0.0-py3-none-any.whl: Passed
Checking distribution dist/realpython-reader-1.0.0.tar.gz: Passed

While it won’t catch all problems you might run into, it will for instance let you know if you are using the wrong content type.

Uploading Your Package

Now you’re ready to actually upload your package to PyPI. For this, you’ll again use the Twine tool, telling it to upload the distribution packages you have built. First, you should upload to TestPyPI to make sure everything works as expected:

$ twine upload --repository-url https://test.pypi.org/legacy/ dist/*

Twine will ask you for your username and password.

Note: If you’ve followed the tutorial using the reader package as an example, the previous command will probably fail with a message saying you are not allowed to upload to the realpython-reader project.

You can change the name in setup.py to something unique, for example test-your-username. Then build the project again and upload the newly built files to TestPyPI.

If the upload succeeds, you can quickly head over to TestPyPI, scroll down, and look at your project being proudly displayed among the new releases! Click on your package and make sure everything looks okay.

If you have been following along using the reader package, the tutorial ends here! While you can play with TestPyPI as much as you want, you shouldn’t upload dummy packages to PyPI just for testing.

However, if you have your own package to publish, then the moment has finally arrived! With all the preparations taken care of, this final step is short:

$ twine upload dist/*

Provide your username and password when requested. That’s it!

Head over to PyPI and look up your package. You can find it either by searching, by looking at the Your projects page, or by going directly to the URL of your project: pypi.org/project/your-package-name/.

Congratulations! Your package is published on PyPI!

`pip install` Your Package

Take a moment to bask in the blue glow of the PyPI web page and (of course) brag to your friends.

Then open up a terminal again. There is one more great pay off!

With your package uploaded to PyPI, you can install it with pip as well:

$ pip install your-package-name

Replace your-package-name with the name you chose for your package. For instance, to install the reader package, you would do the following:

$ pip install realpython-reader

Seeing your own code installed by pip is a wonderful feeling!

Other Useful Tools

Before wrapping up, there are a few other tools that are useful to know about when creating and publishing Python packages.

Virtual Environments

In this guide, we haven’t talked about virtual environments. Virtual environments are very useful when working with different projects, each with their own differing requirements and dependencies.

See the following guides for more information:

In particular, it’s useful to test your package inside a minimal virtual environment to make sure you’re including all necessary dependencies in your setup.py file.

Cookiecutter

One great way to get started with your project is to use Cookiecutter. It sets up your project by asking you a few questions based on a template. Many different templates are available.

First, make sure you have Cookiecutter installed on your system. You can install it from PyPI:

$ pip install cookiecutter

As an example, we’ll use the pypackage-minimal template. To use a template, give Cookiecutter a link to the template:

$ cookiecutter https://github.com/kragniz/cookiecutter-pypackage-minimal
author_name [Louis Taylor]: Real Python
author_email [louis@kragniz.eu]: office@realpython.com
package_name [cookiecutter_pypackage_minimal]: realpython-reader
package_version [0.1.0]:
package_description [...]: Read Real Python tutorials
package_url [...]: https://github.com/realpython/reader
readme_pypi_badge [True]:
readme_travis_badge [True]: False
readme_travis_url [...]:

After you have answered a series of questions, Cookiecutter sets up your project. In this example, the template created the following files and directories:

realpython-reader/
│
├── realpython-reader/
│   └── __init__.py
│
├── tests/
│   ├── __init__.py
│   └── test_sample.py
│
├── README.rst
├── setup.py
└── tox.ini

Cookiecutter’s documentation is extensive and includes a long list of available cookiecutters, as well as tutorials on how to create your own template.

Flit

The history of packaging in Python is quite messy. One common criticism is that using an executable file like setup.py for configuration information is not ideal.

PEP 518 defines an alternative: using a file called pyproject.toml instead. The TOML format is a simple configuration file format:

[…] it is human-usable (unlike JSON), it is flexible enough (unlike configparser), stems from a standard (also unlike configparser), and it is not overly complex (unlike YAML). (Source)

While PEP 518 is already a few years old, the pyproject.toml configuration file is not yet fully supported in the standard tools.

However, there are a few new tools that can publish to PyPI based on pyproject.toml. One such tool is Flit, a great little project that makes it easy to publish simple Python packages. Flit doesn’t support advanced packages like those creating C extensions.

You can pip install flit, and then start using it as follows:

$ flit init
Module name [reader]:
Author []: Real Python
Author email []: office@realpython.com
Home page []: https://github.com/realpython/reader
Choose a license (see http://choosealicense.com/ for more info)
1. MIT - simple and permissive
2. Apache - explicitly grants patent rights
3. GPL - ensures that code based on this is shared with the same terms
4. Skip - choose a license later
Enter 1-4 [1]:

Written pyproject.toml; edit that file to add optional extra info.

The flit init command will create a pyproject.toml file based on the answers you give to a few questions. You might need to edit this file slightly before using it. For the reader project, the pyproject.toml file for Flit ends up looking as follows:

[build-system]
requires = ["flit"]
build-backend = "flit.buildapi"

[tool.flit.metadata]
module = "reader"
dist-name = "realpython-reader"
description-file = "README.md"
author = "Real Python"
author-email = "office@realpython.com"
home-page = "https://github.com/realpython/reader"
classifiers = [
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.7",
]
requires-python = ">=3.7"
requires = ["feedparser", "html2text"]

[tool.flit.scripts]
realpython = "reader.__main__:main"

You should recognize most of the items from our original setup.py. One thing to note though is that version and description are missing. This is not a mistake. Flit actually figures these out itself by using __version__ and the docstring defined in the __init__.py file. Flit’s documentation explains everything about the pyproject.toml file.

Flit can build your package and even publish it to PyPI. To build your package, simply do the following:

$ flit build

This creates a source archive and a wheel, exactly like python setup.py sdist bdist_wheel did earlier. To upload your package to PyPI, you can use Twine as earlier. However, you can also use Flit directly:

$ flit publish

The publish command will build your package if necessary, and then upload the files to PyPI, prompting you for your username and password if necessary.

To see Flit in action, have a look at the 2 minute lightning talk from EuroSciPy 2017. The Flit documentation is a great resource for more information. Brett Cannon’s tutorial on packaging up your Python code for PyPI includes a section about Flit.

Poetry

Poetry is another tool that can be used to build and upload your package. It’s quite similar to Flit, especially for the things we’re looking at here.

Before you use Poetry, you need to install it. It’s possible to pip install poetry as well. However, the author recommends that you use a custom installation script to avoid potential dependency conflicts. See the documentation for installation instructions.

With Poetry installed, you start using it with an init command:

$ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [code]: realpython-reader
Version [0.1.0]: 1.0.0
Description []: Read the latest Real Python tutorials
...

This will create a pyproject.toml file based on your answers to questions about your package. Unfortunately, the actual specifications inside the pyproject.toml differ between Flit and Poetry. For Poetry, the pyproject.toml file ends up looking like the following:

[tool.poetry]
name = "realpython-reader"
version = "1.0.0"
description = "Read the latest Real Python tutorials"
readme = "README.md"
homepage = "https://github.com/realpython/reader"
authors = ["Real Python <office@realpython.com>"]
license = "MIT"
packages = [{include = "reader"}]
include = ["reader/*.txt"]

[tool.poetry.dependencies]
python = ">=3.7"
feedparser = ">=5.2"
html2text = ">=2018.1"

[tool.poetry.scripts]
realpython = "reader.__main__:main"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Again, you should recognize all these items from the earlier discussion of setup.py. One thing to note is that Poetry will automatically add classifiers based on the license and the version of Python you specify. Poetry also requires you to be explicit about versions of your dependencies. In fact, dependency management is one of the strong points of Poetry.

Just like Flit, Poetry can build and upload packages to PyPI. The build command creates a source archive and a wheel:

$ poetry build

This will create the two usual files in the dist subdirectory, which you can upload using Twine as earlier. You can also use Poetry to publish to PyPI:

$ poetry publish

This will upload your package to PyPI. In addition to building and publishing, Poetry can help you earlier in the process. Similar to Cookiecutter, Poetry can help you start a new project with the new command. It also supports working with virtual environments. See Poetry’s documentation for all the details.

Apart from the slightly different configuration files, Flit and Poetry work very similarly. Poetry is broader in scope as it also aims to help with dependency management, while Flit has been around a little longer. Andrew Pinkham’s article Python’s New Package Landscape covers both Flit and Poetry. Poetry was one of the topics at the special 100th episode of the Python Bytes podcast.

Conclusion

You now know how to prepare your project and upload it to PyPI, so that it can be installed and used by other people. While there are a few steps you need to go through, seeing your own package on PyPI is a great pay off. Having others find your project useful is even better!

In this tutorial, you’ve seen the steps necessary to publish your own package:

Find a good name for your package
Configure your package using setup.py
Build your package
Upload your package to PyPI

In addition, you’ve also seen a few new tools for publishing packages that use the new pyproject.toml configuration file to simplify the process.

If you still have questions, feel free to reach out in the comments section below. Also, the Python Packaging Authority has a lot of information with more detail than we covered here.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python "while" Loops (Indefinite Iteration)

2018-11-07T14:00:00+00:00

Iteration means executing the same block of code over and over, potentially many times. A programming structure that implements iteration is called a loop.

In programming, there are two types of iteration, indefinite and definite:

With indefinite iteration, the number of times the loop is executed isn’t specified explicitly in advance. Rather, the designated block is executed repeatedly as long as some condition is met.
With definite iteration, the number of times the designated block will be executed is specified explicitly at the time the loop starts.

In this tutorial, you’ll:

Learn about the while loop, the Python control structure used for indefinite iteration
See how to break out of a loop or loop iteration prematurely
Explore infinite loops

When you’re finished, you should have a good grasp of how to use indefinite iteration in Python.

Free Bonus: Click here to get our free Python Cheat Sheet that shows you the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

The `while` Loop

Let’s see how Python’s while statement is used to construct loops. We’ll start simple and embellish as we go.

The format of a rudimentary while loop is shown below:

while <expr>:
    <statement(s)>

<statement(s)> represents the block to be repeatedly executed, often referred to as the body of the loop. This is denoted with indentation, just as in an if statement.

Remember: All control structures in Python use indentation to define blocks. See the discussion on grouping statements in the previous tutorial to review.

The controlling expression, <expr>, typically involves one or more variables that are initialized prior to starting the loop and then modified somewhere in the loop body.

When a while loop is encountered, <expr> is first evaluated in Boolean context. If it is true, the loop body is executed. Then <expr> is checked again, and if still true, the body is executed again. This continues until <expr> becomes false, at which point program execution proceeds to the first statement beyond the loop body.

Consider this loop:

>>>

 1 >>> n = 5
 2 >>> while n > 0:
 3 ...     n -= 1
 4 ...     print(n)
 5 ...
 6 4
 7 3
 8 2
 9 1
10 0

Here’s what’s happening in this example:

n is initially 5. The expression in the while statement header on line 2 is n > 0, which is true, so the loop body executes. Inside the loop body on line 3, n is decremented by 1 to 4, and then printed.
When the body of the loop has finished, program execution returns to the top of the loop at line 2, and the expression is evaluated again. It is still true, so the body executes again, and 3 is printed.
This continues until n becomes 0. At that point, when the expression is tested, it is false, and the loop terminates. Execution would resume at the first statement following the loop body, but there isn’t one in this case.

Note that the controlling expression of the while loop is tested first, before anything else happens. If it’s false to start with, the loop body will never be executed at all:

>>>

>>> n = 0
>>> while n > 0:
...     n -= 1
...     print(n)
...

In the example above, when the loop is encountered, n is 0. The controlling expression n > 0 is already false, so the loop body never executes.

Here’s another while loop involving a list, rather than a numeric comparison:

>>>

>>> a = ['foo', 'bar', 'baz']
>>> while a:
...     print(a.pop(-1))
...
baz
bar
foo

When a list is evaluated in Boolean context, it is truthy if it has elements in it and falsy if it is empty. In this example, a is true as long as it has elements in it. Once all the items have been removed with the .pop() method and the list is empty, a is false, and the loop terminates.

Interruption of Loop Iteration

In each example you have seen so far, the entire body of the while loop is executed on each iteration. Python provides two keywords that terminate a loop iteration prematurely:

break immediately terminates a loop entirely. Program execution proceeds to the first statement following the loop body.
continue immediately terminates the current loop iteration. Execution jumps to the top of the loop, and the controlling expression is re-evaluated to determine whether the loop will execute again or terminate.

The distinction between break and continue is demonstrated in the following diagram:

break and continue

Here’s a script file called break.py that demonstrates the break statement:

 1 n = 5
 2 while n > 0:
 3     n -= 1
 4     if n == 2:
 5         break
 6     print(n)
 7 print('Loop ended.')

Running break.py from a command-line interpreter produces the following output:

C:\Users\john\Documents>python break.py
4
3
Loop ended.

When n becomes 2, the break statement is executed. The loop is terminated completely, and program execution jumps to the print() statement on line 7.

The next script, continue.py, is identical except for a continue statement in place of the break:

 1 n = 5
 2 while n > 0:
 3     n -= 1
 4     if n == 2:
 5         continue
 6     print(n)
 7 print('Loop ended.')

The output of continue.py looks like this:

C:\Users\john\Documents>python continue.py
4
3
1
0
Loop ended.

This time, when n is 2, the continue statement causes termination of that iteration. Thus, 2 isn’t printed. Execution returns to the top of the loop, the condition is re-evaluated, and it is still true. The loop resumes, terminating when n becomes 0, as previously.

The `else` Clause

Python allows an optional else clause at the end of a while loop. This is a unique feature of Python, not found in most other programming languages. The syntax is shown below:

while <expr>:
    <statement(s)>
else:
    <additional_statement(s)>

The <additional_statement(s)> specified in the else clause will be executed when the while loop terminates.

About now, you may be thinking, “How is that useful?” You could accomplish the same thing by putting those statements immediately after the while loop, without the else:

while <expr>:
    <statement(s)>
<additional_statement(s)>

What’s the difference?

In the latter case, without the else clause, <additional_statement(s)> will be executed after the while loop terminates, no matter what.

When <additional_statement(s)> are placed in an else clause, they will be executed only if the loop terminates “by exhaustion”—that is, if the loop iterates until the controlling condition becomes false. If the loop is exited by a break statement, the else clause won’t be executed.

Consider the following example:

>>>

>>> n = 5
>>> while n > 0:
...     n -= 1
...     print(n)
... else:
...     print('Loop done.')
...
4
3
2
1
0
Loop done.

In this case, the loop repeated until the condition was exhausted: n became 0, so n > 0 became false. Because the loop lived out its natural life, so to speak, the else clause was executed. Now observe the difference here:

>>>

>>> n = 5
>>> while n > 0:
...     n -= 1
...     print(n)
...     if n == 2:
...         break
... else:
...     print('Loop done.')
...
4
3
2

This loop is terminated prematurely with break, so the else clause isn’t executed.

It may seem as if the meaning of the word else doesn’t quite fit the while loop as well as it does the if statement. Guido van Rossum, the creator of Python, has actually said that, if he had it to do over again, he’d leave the while loop’s else clause out of the language.

One of the following interpretations might help to make it more intuitive:

Think of the header of the loop (while n > 0) as an if statement (if n > 0) that gets executed over and over, with the else clause finally being executed when the condition becomes false.
Think of else as though it were nobreak, in that the block that follows gets executed if there wasn’t a break.

If you don’t find either of these interpretations helpful, then feel free to ignore them.

When might an else clause on a while loop be useful? One common situation is if you are searching a list for a specific item. You can use break to exit the loop if the item is found, and the else clause can contain code that is meant to be executed if the item isn’t found:

>>>

>>> a = ['foo', 'bar', 'baz', 'qux']
>>> s = 'corge'

>>> i = 0
>>> while i < len(a):
...     if a[i] == s:
...         # Processing for item found
...         break
...     i += 1
... else:
...     # Processing for item not found
...     print(s, 'not found in list.')
...
corge not found in list.

Note: The code shown above is useful to illustrate the concept, but you’d actually be very unlikely to search a list that way.

First of all, lists are usually processed with definite iteration, not a while loop. Definite iteration is covered in the next tutorial in this series.

Secondly, Python provides built-in ways to search for an item in a list. You can use the in operator:

>>>

>>> if s in a:
...     print(s, 'found in list.')
... else:
...     print(s, 'not found in list.')
...
corge not found in list.

The list.index() method would also work. This method raises a ValueError exception if the item isn’t found in the list, so you need to understand exception handling to use it. In Python, you use a try statement to handle an exception. An example is given below:

>>>

>>> try:
...     print(a.index('corge'))
... except ValueError:
...     print(s, 'not found in list.')
...
corge not found in list.

You will learn about exception handling later in this series.

An else clause with a while loop is a bit of an oddity, not often seen. But don’t shy away from it if you find a situation in which you feel it adds clarity to your code!

Infinite Loops

Suppose you write a while loop that theoretically never ends. Sounds weird, right?

Consider this example:

>>>

>>> while True:
...     print('foo')
...
foo
foo
foo
  .
  .
  .
foo
foo
foo
KeyboardInterrupt
Traceback (most recent call last):
  File "<pyshell#2>", line 2, in <module>
    print('foo')

This code was terminated by Ctrl+C, which generates an interrupt from the keyboard. Otherwise, it would have gone on unendingly. Many foo output lines have been removed and replaced by the vertical ellipsis in the output shown.

Clearly, True will never be false, or we’re all in very big trouble. Thus, while True: initiates an infinite loop that will theoretically run forever.

Maybe that doesn’t sound like something you’d want to do, but this pattern is actually quite common. For example, you might write code for a service that starts up and runs forever accepting service requests. “Forever” in this context means until you shut it down, or until the heat death of the universe, whichever comes first.

More prosaically, remember that loops can be broken out of with the break statement. It may be more straightforward to terminate a loop based on conditions recognized within the loop body, rather than on a condition evaluated at the top.

Here’s another variant of the loop shown above that successively removes items from a list using .pop() until it is empty:

>>>

>>> a = ['foo', 'bar', 'baz']
>>> while True:
...     if not a:
...         break
...     print(a.pop(-1))
...
baz
bar
foo

When a becomes empty, not a becomes true, and the break statement exits the loop.

You can also specify multiple break statements in a loop:

while True:
    if <expr1>:  # One condition for loop termination
        break
    ...
    if <expr2>:  # Another termination condition
        break
    ...
    if <expr3>:  # Yet another
        break

In cases like this, where there are multiple reasons to end the loop, it is often cleaner to break out from several different locations, rather than try to specify all the termination conditions in the loop header.

Infinite loops can be very useful. Just remember that you must ensure the loop gets broken out of at some point, so it doesn’t truly become infinite.

Nested `while` Loops

In general, Python control structures can be nested within one another. For example, if/elif/else conditional statements can be nested:

if age < 18:
    if gender == 'M':
        print('son')
    else:
        print('daughter')
elif age >= 18 and age < 65:
    if gender == 'M':
        print('father')
    else:
        print('mother')
else:
    if gender == 'M':
        print('grandfather')
    else:
        print('grandmother')

Similarly, a while loop can be contained within another while loop, as shown here:

>>>

>>> a = ['foo', 'bar']
>>> while len(a):
...     print(a.pop(0))
...     b = ['baz', 'qux']
...     while len(b):
...         print('>', b.pop(0))
...
foo
> baz
> qux
bar
> baz
> qux

A break or continue statement found within nested loops applies to the nearest enclosing loop:

while <expr1>:
    statement
    statement

    while <expr2>:
        statement
        statement
        break  # Applies to while <expr2>: loop

    break  # Applies to while <expr1>: loop

Additionally, while loops can be nested inside if/elif/else statements, and vice versa:

if <expr>:
    statement
    while <expr>:
        statement
        statement
else:
    while <expr>:
        statement
        statement
    statement

while <expr>:
    if <expr>:
        statement
    elif <expr>:
        statement
    else:
        statement

    if <expr>:
        statement

In fact, all the Python control structures can be intermingled with one another to whatever extent you need. That is as it should be. Imagine how frustrating it would be if there were unexpected restrictions like “A while loop can’t be contained within an if statement” or “while loops can only be nested inside one another at most four deep.” You’d have a very difficult time remembering them all.

Seemingly arbitrary numeric or logical limitations are considered a sign of poor program language design. Happily, you won’t find many in Python.

One-Line `while` Loops

As with an if statement, a while loop can be specified on one line. If there are multiple statements in the block that makes up the loop body, they can be separated by semicolons (;):

>>>

>>> n = 5
>>> while n > 0: n -= 1; print(n)

4
3
2
1
0

This only works with simple statements though. You can’t combine two compound statements into one line. Thus, you can specify a while loop all on one line as above, and you write an if statement on one line:

>>>

>>> if True: print('foo')

foo

But you can’t do this:

>>>

>>> while n > 0: n -= 1; if True: print('foo')
SyntaxError: invalid syntax

Remember that PEP 8 discourages multiple statements on one line. So you probably shouldn’t be doing any of this very often anyhow.

Conclusion

In this tutorial, you learned about indefinite iteration using the Python while loop. You’re now able to:

Construct basic and complex while loops
Interrupt loop execution with break and continue
Use the else clause with a while loop
Deal with infinite loops

You should now have a good grasp of how to execute a piece of code repetitively.

The next tutorial in this series covers definite iteration with for loops—recurrent execution where the number of repetitions is specified explicitly.

« Conditional Statements in Python

Python "while" Loops

Python "for" Loops »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

Async IO in Python: A Complete Walkthrough

Setting Up Your Environment

The 10,000-Foot View of Async IO

Where Does Async IO Fit In?

Async IO Explained

Async IO Is Not Easy

The asyncio Package and async/await

The async/await Syntax and Native Coroutines

The Rules of Async IO

Async IO Design Patterns

Chaining Coroutines

Using a Queue

Async IO’s Roots in Generators

Other Features: async for and Async Generators + Comprehensions

The Event Loop and asyncio.run()

A Full Program: Asynchronous Requests

Async IO in Context

When and Why Is Async IO the Right Choice?

Async IO It Is, but Which One?

Odds and Ends

Other Top-Level asyncio Functions

The Precedence of await

Conclusion

Resources

Python Version Specifics

Articles

Related PEPs

Libraries That Work With async/await

Speed Up Your Python Program With Concurrency

What Is Concurrency?

What Is Parallelism?

When Is Concurrency Useful?

How to Speed Up an I/O-Bound Program

Synchronous Version

threading Version

asyncio Version

multiprocessing Version

How to Speed Up a CPU-Bound Program

CPU-Bound Synchronous Version

threading and asyncio Versions

CPU-Bound multiprocessing Version

When to Use Concurrency

Conclusion

Django Migrations: A Primer

The Problems That Migrations Solve

Making Database Changes Without SQL

Avoiding Repetition

Ensuring Model Definitions and the Database Schema in Sync

Tracking Database Schema Change in Version Control

Setting Up a Django Project

Creating Migrations

Applying Migrations

Changing Models

Listing Out Migrations

Unapplying Migrations

Naming Migrations

Conclusion

Video

The Ultimate Guide to Python Type Checking

Type Systems

Dynamic Typing

Static Typing

Duck Typing

Hello Types

Pros and Cons

Annotations

Function Annotations

Variable Annotations

Type Comments

So, Type Annotations or Type Comments?

Playing With Python Types, Part 1

Example: A Deck of Cards

Sequences and Mappings

Type Aliases

Functions Without Return Values

Example: Play Some Cards

The Any Type

Type Theory

Subtypes

The `asyncio` Package and `async`/`await`

The `async`/`await` Syntax and Native Coroutines

Other Features: `async for` and Async Generators + Comprehensions

The Event Loop and `asyncio.run()`

Other Top-Level `asyncio` Functions

The Precedence of `await`

Libraries That Work With `async`/`await`

`threading` Version

`asyncio` Version

`multiprocessing` Version

`threading` and `asyncio` Versions

CPU-Bound `multiprocessing` Version

The `Any` Type

The `Optional` Type

Returning `self` or `cls`

Annotating `*args` and `**kwargs`