Real Python

The Ultimate List of Data Science Podcasts

2019-02-06T06:00:00+00:00

Podcasts are a great way to immerse yourself in an industry, especially when it comes to data science. The field moves extremely quickly, and it can be difficult to keep up with all the new developments happening each week!

Take advantage of those times in the day when your body is busy, but your mind is free: when you’re commuting to work, exercising at the gym, or cleaning up around the house. These are optimal times to engage your brain in learning something new and ensure you’re staying at the top of your field.

There are dozens of data science podcasts out there, covering everything from machine learning and artificial intelligence to big data analytics. We hope this will be a great resource for you to find useful, informative, and engaging shows.

Get ready to dive in!

Free Bonus: Click here to get access to a free NumPy Resources Guide that points you to the best tutorials, videos, and books for improving your NumPy skills.

Active Data Science Podcasts

As of this writing, these data science podcasts are active and still in production. Start deep in the archives and work your way up, or jump right into the latest episode!

Data Skeptic

Website: https://dataskeptic.com/
Twitter: @dataskeptic
Listen: RSS ⋅ iTunes ⋅ Podbean ⋅ Player FM

Data Skeptic is one of the best-known data science podcasts. This weekly show explores topics in data science, statistics, machine learning and artificial intelligence.

Hosted by Kyle Polich, the show is going strong with over 200 episodes for listeners to dive into. Recently, the show has released series of themed episodes that revolve around a larger topic in the data science world, like fake news.

The episodes alternate between interviews with industry professionals and minisodes that explain high-level data science concepts.

The minisodes are co-hosted by Linh Da Tran, who talks with Kyle about data science topics, like natural language processing and k-means clustering. Listeners gain a better understanding of the topic as the hosts talk through it.

Linear Digressions

Website: http://lineardigressions.com
Twitter: @LinDigressions
Listen: RSS ⋅ iTunes ⋅ Podbean ⋅ Player FM

Katie Malone and Ben Jaffe host Linear Digressions, a weekly podcast that explores recent developments in data science, machine learning, and artificial intelligence. The hosts are good friends, and their rapport makes each episode very accessible and easy to understand.

As of this writing, there are over 100 episodes for listeners to dive into. Each episode clocks in at around half an hour, making it a breeze to gain a quick understanding of the topic at hand.

Katie and Ben do a great job at distilling a complex technical topic down to its fundamentals. In just a few short minutes, they demystify neural networks, autoencoders, the Fourier transform, and more.

Talking Machines

Website: https://www.thetalkingmachines.com/
Twitter: @TlkngMchns
Listen: RSS ⋅ iTunes ⋅ Player FM

Former public radio producer Katherine Gorman believes that continuing the public conversation about data science, AI, and machine learning is absolutely essential to preventing another AI winter.

She believes that data science podcasts are a great venue for that discussion. To this end, she hosts Talking Machines along with Professor Neil Lawrence.

The podcast aims to introduce machine learning to a wide audience and help industry professionals, business leaders, and interested laypeople better understand these tools and technologies.

The episodes generally follow a simple format: the hosts chat about industry news, interview a guest, and in the end may answer a listener question. Episodes are released in seasons and tend to be on the longer side at around 40 minutes.

This is where Katherine’s history as a radio host comes in handy: she keeps the show engaging and informative, and works hard to make sure the it presents an accurate picture of the machine learning industry.

O’Reilly Data Show

Website: https://www.oreilly.com/topics/oreilly-data-show-podcast
Twitter: @OReillyMedia
Listen: RSS ⋅ iTunes ⋅ Podbean ⋅ Player FM

Ben Lorica is the Chief Data Scientist at O’Reilly Media. In each episode, he is joined by an industry professional to discuss topics in big data and data science. The episodes run anywhere from 30 to 40 minutes and are very accessible to listen to.

At the beginning of each episode, the host promotes an event series that listeners can attend to learn more about the topics covered in the podcast. The ones mentioned in the intro are the Strata Data Conference and the Artificial Intelligence Conference, but you can find more of the O’Reilly conferences on their event page.

Not So Standard Deviations

Website: http://nssdeviations.com/
Twitter: @NSSDeviations
Listen: RSS ⋅ iTunes ⋅ Podbean ⋅ Player FM

Roger Peng (of the Johns Hopkins Bloomberg School of Public Health) and Hilary Parker (of Stitch Fix) co-host this podcast. They discuss industry news as well as their personal experiences working with data.

Episodes air two or three times a month and can run on the longer side. Most episodes are at least 60 minutes, with some clocking in at almost an hour and a half. These are great for when you have a long commute or spend an evening at home doing chores, so you can really get into the discussion!

Data Stories

Website: http://datastori.es/
Twitter: @datastories
Listen: RSS ⋅ iTunes ⋅ Player FM

This podcast on data visualization focuses on a very specific subset of the data analysis pipeline—a rare gem among data science podcasts. Data viz specialists Enrico Bertini and Moritz Stefaner sit down with a guest every other week to discuss data analysis and visualization.

The show has quite a conversational tone. The hosts bounce ideas off of one another, ask great questions of their guests, and generally keep the conversation flowing. With around 40 minutes of runtime, listeners can settle in to really learn about how we can better visualize our data, as well as the role that data plays in our everyday lives.

SuperDataScience

Website: https://www.superdatascience.com/podcast/
Twitter: @superdatasci
Listen: RSS ⋅ iTunes ⋅ Podbean ⋅ Player FM

Kirill Eremenko is a data science coach and lifestyle entrepreneur, and he brings his experience as an influencer to the SuperDataScience podcast. In his interview episodes, he talks with data scientists and data analysts to learn more about their career paths and how they were able to succeed in the data industry.

In addition to interviewing industry experts, the host airs minisodes that are purely inspirational! Called Five Minute Friday, these minisodes aim to inspire listeners to improve themselves as data scientists, and to offer advice on how to advance in a data science career. This is definitely one of the most motivational data science podcasts out there!

Data Science at Home

Website: https://datascienceathome.com/
Twitter: @ThisIsFrag
Listen: RSS ⋅ iTunes ⋅ Podbean ⋅ Player FM

Francesco Gadaleta wants to make machine learning easy for everyone. In this podcast, he alternates between interview episodes with industry experts, and solo episodes where he discusses a topic on his own.

The show doesn’t seem to be on a fixed schedule, and the episode length varies as well, but in general the interview episodes run closer to an hour, while his solo episodes clock in at around twenty minutes.

The host is pretty opinionated, so it can be interesting to hear his perspective on topics like AI winter, optimization, and the minimum requirements you need to become a data scientist.

This Week in Machine Learning & Artificial Intelligence (TWiML&AI)

Website: https://twimlai.com/
Twitter: @twimlai
Listen: RSS ⋅ iTunes ⋅ Podbean ⋅ Player FM

TWiML&AI is a weekly podcast that discusses the latest developments in data science, machine learning, and artificial intelligence. The host Sam Charrington interviews leading researchers and industry experts to inform a growing community of academics, engineers, business leaders, and other machine learning and AI enthusiasts.

The show caters to a highly targeted audience, and can be pretty technical at times. Listeners who are not industry professionals may need to brush up on background knowledge in order to get the most out of each episode.

There are over two hundred hour-long episodes to listen to. Because the podcast discusses recent developments in this tech space, you can jump right into the latest episode, or head back in the archives and check in on some historical developments in machine learning and AI.

DataFramed

Website: https://www.datacamp.com/community/podcast
Twitter: @DataCamp
Listen: iTunes ⋅ Podbean ⋅ Player FM

Data scientist, writer, and educator Hugo Bowne-Anderson hosts this podcast sponsored by DataCamp.

Each week, the host sits down with industry professionals and academic experts to discuss how the data science industry is impacting the world. The host asks great questions and invites guests who discuss interesting developments in the field as well as their own personal projects.

DataFramed also has short segments spaced throughout the episodes that give the listener more information on certain topics. For instance, in Freelance Data Science, Hugo and Susan Sun talk about how to navigate the data science space as an independent contractor. Justin Boyce gives practical advice on improving workflow in Data Science Best Practices.

Because it is sponsored by DataCamp, their products are pitched a lot, so it can feel a bit sales-y at times. Still, the show is interesting and informative, and Hugo does a great job of drawing in the listener.

Learning Machines 101

Website: https://www.learningmachines101.com/
Twitter: @lm101talk
Listen: RSS ⋅ iTunes ⋅ Podbean ⋅ Player FM

Dr. Richard Golden, Professor of Cognitive Science and Electrical Engineering, hosts Learning Machines 101. The podcast aims to explain advanced concepts in machine learning and artificial intelligence to a wide audience.

Still, the episodes can get pretty technical, covering topics such as knowledge representation, expectation maximization, and spectral clustering.

Listeners might need to listen more than once to really grasp the topic at hand. This shouldn’t be too hard, as the episodes are no more than half an hour long and aren’t released too often. (Only 74 episodes have been released since April 2014.)

Listeners can use this podcast as a jumping-off point into more advanced machine learning topics.

Artificial Intelligence in Industry

Website: http://techemergence.libsyn.com/
Twitter: @Emerj
Subscribe: RSS ⋅ iTunes ⋅ Podbean ⋅ Player FM

This weekly podcast is focused on the practical applications of artificial intelligence in business settings. The episodes are short, insightful, and easy to understand. Within half an hour, host Dan Faggella interviews AI professionals to see how the technology is used in industries from finance and government to retail and education.

Together, Dan and his guests answer questions like “How can you use AI to hire employees?” and “When should you upgrade your AI hardware?” They touch on each topic just long enough to pique the listener’s interest and encourage them to dive deeper on their own later.

Archived Data Science Podcasts

As of this writing, these data science podcasts have run their course. The archives are still available for you to dive into, and are chock-full of useful information, so don’t hesitate to dive right in!

Partially Derivative

Website: http://partiallyderivative.com/
Twitter: @partiallyd
Listen: iTunes ⋅ Podbean ⋅ Player FM

If you like heading to the bar and chatting about industry news with your fellow data scientists, then this is one of the best data science podcasts for you! Jonathan Morgan, Vidya Spandana, and Chris Albon get together to down a few drinks and discuss the latest in data science.

The episodes can run anywhere from 20 minutes to an hour, but generally clock in at around 30 to 40 minutes. While the show is no longer running, there are over one hundred episodes in the archive.

Listeners can delve into the backlog and learn about data scraping, bias models, and pair-programming in Python, as well as review some of the trending news stories of years past.

Machine Learning Guide / Machine Learning Applied

Website: http://ocdevel.com/mlg
Twitter: @lefnire
Listen: RSS ⋅ iTunes ⋅ Podbean ⋅ Player FM

These data science podcasts are both run by Tyler Renelli, and each has a slightly different approach to machine learning and AI.

Machine Learning Guide (MLG) aims to gently introduce listeners to the world of machine learning by explaining topics from the ground up, from the classic algorithms (linear and logistic regression) on up to reinforcement learning and hyperparameters.

The episodes run anywhere from 45 minutes to an hour, but it’s easy to become engrossed in Tyler’s explanations. It’s the perfect podcast to complement other activities, like commuting, exercising, or cleaning up around the house.

One of the best parts of this podcast is the curated learning resources that the host provides at the end of each episode. After listening to a high-lever overview, you can dive deeper into the topic by taking a recommended course or reading a suggested textbook.

His episode on languages and frameworks includes a link to a primer on Python deep learning frameworks. If you follow the episodes in order from beginning to end, and complete the supplemental resources, you will have quite a detailed foundation in machine learning.

As of this writing, MLG has run its course at 29 full-length episodes.

A second podcast called Machine Learning Applied is currently airing, where Tyler focuses on the more practical aspects of machine learning. He answers questions such as what kind of salary one can expect, the best way to store data, and how to get the most out of Jupyter notebooks. Listeners can gain access to Machine Learning Applied by becoming a supporter on Patroen.

Becoming a Data Scientist

Website: https://www.becomingadatascientist.com/category/podcast/
Twitter: @BecomingDataSci
Listen: RSS ⋅ iTunes ⋅ Podbean ⋅ Player FM

This podcast does exactly what its title says. The host, Renee Teate, sits down each week with someone who is on their way to “becoming a data scientist.”

She interviews other data science professionals to see exactly how they were able to carve a path for themselves into the industry. In the very first episode, Renee talks about her own journey into becoming a data scientist as she transitions from her previous role as a data analyst.

As of this writing, the podcast is not currently active. The last episodes came out in early 2017. Still, there is a wealth of information contained in the twenty hour-long episodes that have aired.

If you’re just beginning your foray into the data science world, take a weekend to blast through the archive and see where the possibilities lie!

Conclusion

This list is not exhaustive! There are new podcasts airing all the time, and we can only expect the number of data science podcasts to grow as the field continues to explode in popularity.

Don’t see your favorite show on this list? Leave us a comment down below and let us know your favorite data science podcasts!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Development in Visual Studio Code

2019-02-04T06:00:00+00:00

One of the coolest code editors available to programmers, Visual Studio Code, is an open-source, extensible, light-weight editor available on all platforms. It’s these qualities that make Visual Studio Code from Microsoft very popular, and a great platform for Python development.

In this article, you’ll learn about Python development in Visual Studio Code, including how to:

Install Visual Studio Code
Discover and install extensions that make Python development easy
Write a simple Python application
Learn how to run and debug existing Python programs in VS Code
Connect Visual Studio Code to Git and GitHub to share your code with the world

We assume you are familiar with Python development and already have some form of Python installed on your system (Python 2.7, Python 3.6/3.7, Anaconda, or others). Screenshots and demos for Ubuntu and Windows are provided. Because Visual Studio Code runs on all major platforms, you may see slightly different UI elements and may need to modify certain commands.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Installing and Configuring Visual Studio Code for Python Development

Installing Visual Studio Code is very accessible on any platform. Full instructions for Windows, Mac, and Linux are available, and the editor is updated monthly with new features and bug fixes. You can find everything at the Visual Studio Code website:

In case you were wondering, Visual Studio Code (or VS Code for short) shares almost nothing other than a name with its larger Windows-based namesake, Visual Studio.

Visual Studio Code has built-in support for multiple languages and an extension model with a rich ecosystem of support for others. VS Code is updated monthly, and you can keep up to date at the Microsoft Python blog. Microsoft even makes the VS Code GitHub repo available for anyone to clone and contribute. (Cue the PR flood.)

The VS Code UI is well documented, so I won’t rehash it here:

Extensions for Python Development

As stated above, VS Code supports development in multiple programming languages through a well-documented extension model. The Python extension enables Python development in Visual Studio Code, with the following features:

Support for Python 3.4 and higher, as well as Python 2.7
Code completion with IntelliSense
Linting
Debugging support
Code snippets
Unit testing support
Automatic use of conda and virtual environments
Code editing in Jupyter environments and Jupyter Notebooks

Visual Studio Code extensions cover more than just programming language capabilities:

Keymaps allow users already familiar with Atom, Sublime Text, Emacs, Vim, PyCharm, or other environments to feel at home.
Themes customize the UI whether you like coding in the light, dark, or something more colorful.
Language packs provide a localized experience.

Here are some other extensions and settings I find useful:

GitLens provides tons of useful Git features directly in your editing window, including blame annotations and repository exploration features.
Auto save is easily turned on by selecting File, Auto Save from the menu. The default delay time is 1000 milliseconds, which is also configurable.
Settings Sync allows you to synchronize your VS Code settings across different installations using GitHub. If you work on different machines, this helps keep your environment consistent across them.
Docker lets you quickly and easily work with Docker, helping author Dockerfile and docker-compose.yml, package and deploy your projects, and even generate the proper Docker files for your project.

Of course, you may discover other useful extensions as you use VS Code. Please share your discoveries and settings in the comments!

Discovering and installing new extensions and themes is accessible by clicking on the Extensions icon on the Activity Bar. You can search for extensions using keywords, sort the results numerous ways, and install extensions quickly and easily. For this article, install the Python extension by typing python in the Extensions item on the Activity Bar, and clicking Install:

You can find and install any of the extensions mentioned above in the same manner.

Visual Studio Code Configuration Files

One important thing to mention is that Visual Studio Code is highly configurable through user and workspace settings.

User settings are global across all Visual Studio Code instances, while workspace settings are local to the specific folder or project workspace. Workspace settings give VS Code tons of flexibility, and I call out workspace settings throughout this article. Workspace settings are stored as .json files in a folder local to the project workspace called .vscode.

Start a New Python Program

Let’s start our exploration of Python development in Visual Studio Code with a new Python program. In VS Code, type Ctrl+N to open a new File. (You can also select File, New from the menu.)

Note: The Visual Studio Code UI provides the Command Palette, from which you can search and execute any command without leaving the keyboard. Open the Command Palette using Ctrl+Shift+P, type File: New File, and hit Enter to open a new file.

No matter how you get there, you should see a VS Code window that looks similar to the following:

Once a new file is opened, you can being entering code.

Entering Python Code

For our test code, let’s quickly code up the Sieve of Eratosthenes (which finds all primes less than a given number). Begin typing the following code in the new tab you just opened:

sieve = [True] * 101
for i in range(2, 100):

You should see something similar to this:

Wait, what’s going on? Why isn’t Visual Studio Code doing any keyword highlighting, any auto-formatting, or anything really helpful? What gives?

The answer is that, right now, VS Code doesn’t know what kind of file it’s dealing with. The buffer is called Untitled-1, and if you look in the lower right corner of the window, you’ll see the words Plain Text.

To activate the Python extension, save the file (by selecting File, Save from the menu, File:Save File from the Command Palette, or just using Ctrl+S) as sieve.py. VS Code will see the .py extension and correctly interpret the file as Python code. Now your window should look like this:

That’s much better! VS Code automatically reformats the file as Python, which you can verify by inspecting the language mode in the lower left corner.

If you have multiple Python installations (like Python 2.7, Python 3.x, or Anaconda), you can change which Python interpreter VS Code uses by clicking the language mode indicator, or selecting Python: Select Interpreter from the Command Palette. VS Code supports formatting using pep8 by default, but you can select black or yapf if you wish.

Let’s add the rest of the Sieve code now. To see IntelliSense at work, type this code directly rather than cut and paste, and you should see something like this:

Here’s the full code for a basic Sieve of Eratosthenes:

sieve = [True] * 101
for i in range(2, 100):
    if sieve[i]:
        print(i)
        for j in range(i*i, 100, i):
            sieve[j] = False

As you type this code, VS Code automatically indents the lines under for and if statements for you properly, adds closing parentheses, and makes suggestions for you. That’s the power of IntelliSense working for you.

Running Python Code

Now that the code is complete, you can run it. There is no need to leave the editor to do this: Visual Studio Code can run this program directly in the editor. Save the file (using Ctrl+S), then right-click in the editor window and select Run Python File in Terminal:

You should see the Terminal pane appear at the bottom of the window, with your code output showing.

Python Linting Support

You may have seen a pop up appear while you were typing, stating that linting was not available. You can quickly install linting support from that pop up, which defaults to PyLint. VS Code also supports other linters. Here’s the complete list at the time of this writing:

pylint
flake8
mypy
pydocstyle
pep8
prospector
pyllama
bandit

The Python linting page has complete details on how to setup each linter.

Note: The choice of linter is a project workspace setting, and not a global user setting.

Editing an Existing Python Project

In the Sieve of Eratosthenes example, you created a single Python file. That’s great as an example, but many times, you’ll create larger projects and work on them over a longer period of time. A typical new project work flow might look like this:

Create a folder to hold the project (which may include a new GitHub project)
Change to the new folder
Create the initial Python code using the command code filename.py

Using Visual Studio Code on a Python project (as opposed to a single Python file) opens up tons more functionality that lets VS Code truly shine. Let’s take a look at how it works with a larger project.

Late in the previous millennium, when I was a much younger programmer, I wrote a calculator program that parsed equations written in infix notation, using an adaptation of Edsger Dijkstra’s shunting yard algorithm.

To demonstrate the project-focused features of Visual Studio Code, I began recreating the shunting yard algorithm as an equation evaluation library in Python. To continue following along, feel free to clone the repo locally.

Once the folder is created locally, you can open the entire folder in VS Code quickly. My preferred method (as mentioned above) is modified as follows, since I already have the folder and basic files created:

cd /path/to/project
code .

VS Code understands, and will use, any virtualenv, pipenv, or conda environments it sees when opened this way. You don’t even need to start the virtual environment first! You can even open a folder from the UI, using File, Open Folder from the menu, Ctrl+K, Ctrl+O from the keyboard, or File:Open Folder from the Command Palette.

For my equation eval library project, here’s what I see:

When Visual Studio Code opens the folder, it also opens the files you last had opened. (This is configurable.) You can open, edit, run, and debug any file listed. The Explorer view in the Activity Bar on the left gives you a view of all the files in the folder and shows how many unsaved files exist in the current set of tabs.

Testing Support

VS Code can automatically recognize existing Python tests written in the unittest framework, or the pytest or Nose frameworks if those frameworks are installed in the current environment. I have a unit test written in unittest for the equation eval library, which you can use for this example.

To run your existing unit tests, from any Python file in the project, right-click and select Run Current Unit Test File. You’ll be prompted to specify the test framework, where in the project to search for tests, and the filename pattern your tests utilize.

All of these are saved as workspace settings in your local .vscode/settings.json file and can be modified there. For this equation project, you select unittest, the current folder, and the pattern *_test.py.

Once the test framework is set up and the tests have been discovered, you can run all your tests by clicking Run Tests on the Status Bar and selecting an option from the Command Palette:

You can even run individual tests by opening the test file in VS Code, clicking Run Tests on the Status Bar, and selecting the Run Unit Test Method… and the specific test to run. This makes it trivial to address individual test failures and re-run only failed tests, which is a huge time-saver! Test results are shown in the Output pane under Python Test Log.

Debugging Support

Even though VS Code is a code editor, debugging Python directly within VS Code is possible. VS Code offers many of the features you would expect from a good code debugger, including:

Automatic variable tracking
Watch expressions
Breakpoints
Call stack inspection

You can see them all as part of the Debug view on the Activity Bar:

The debugger can control Python apps running in the built-in terminal or an external terminal instance. It can attach to an already running Python instances, and can even debug Django and Flask apps.

Debugging code in a single Python file is as simple as starting the debugger using F5. You use F10 and F11 to step over and into functions respectively, and Shift+F5 to exit the debugger. Breakpoints are set using F9, or using the mouse by clicking in the left margin in the editor window.

Before you start debugging more complicated projects, including Django or Flask applications, you need to setup and then select a debug configuration. Setting up the debug configuration is relatively straightforward. From the Debug view, select the Configuration drop-down, then Add Configuration, and select Python:

Visual Studio Code will create a debug configuration file under the current folder called .vscode/launch.json, which allows you to setup specific Python configurations as well as settings for debugging specific apps, like Django and Flask.

You can even perform remote debugging, and debug Jinja and Django templates. Close the launch.json file in the editor and select the proper configuration for your application from the Configuration drop-down.

Git Integration

VS Code has built-in support for source control management, and ships with support for Git and GitHub right out of the box. You can install support for other SCM’s in VS Code, and use them side by side. Source control is accessible from the Source Control view:

If your project folder contains a .git folder, VS Code automatically turns on the full range of Git/GitHub functionality. Here are some of the many tasks you can perform:

Commit files to Git
Push changes to, and pull changes from, remote repos
Check-out existing or create new branches and tags
View and resolve merge conflicts
View diffs

All of this functionality is available directly from the VS Code UI:

VS Code will also recognize changes made outside the editor and behave appropriately.

Committing your recent changes within VS Code is a fairly straightforward process. Modified files are shown in the Source Control view with an M marker, while new untracked files are marked with a U. Stage your changes by hovering over the file and then clicking the plus sign (+). Add a commit message at the top of the view, and then click the check mark to commit the changes:

You can push local commits to GitHub from within VS Code as well. Select Sync from the Source Control view menu, or click Synchronize Changes on the status bar next to the branch indicator.

Conclusion

Visual Studio Code is one of the coolest general purpose editors and a great candidate for Python development. In this article, you learned:

How to install VS Code on any platform
How to find and install extensions to enable Python-specific features
How VS Code makes writing a simple Python application easier
How to run and debug existing Python programs within VS Code
How to work with Git and GitHub repositories from VS Code

Visual Studio Code has become my default editor for Python and other tasks, and I hope you give it a chance to become yours as well.

If you have questions or comments, please reach out in the comments below. There is also a lot more information at the Visual Studio Code website than we could cover here.

The author sends thanks to Dan Taylor from the Visual Studio Code team at Microsoft for his time and invaluable input in this article.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python "for" Loops (Definite Iteration)

2019-01-30T06:00:00+00:00

This tutorial will show you how to perform definite iteration with a Python for loop.

In the previous tutorial in this introductory series, you learned the following:

Repetitive execution of the same block of code over and over is referred to as iteration.
There are two types of iteration:
- Definite iteration, in which the number of repetitions is specified explicitly in advance
- Indefinite iteration, in which the code block executes until some condition is met
In Python, indefinite iteration is performed with a while loop.

Here’s what you’ll cover in this tutorial:

You’ll start with a comparison of some different paradigms used by programming languages to implement definite iteration.
Then you will learn about iterables and iterators, two concepts that form the basis of definite iteration in Python.
Finally, you’ll tie it all together and learn about Python’s for loops.

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

A Survey of Definite Iteration in Programming

Definite iteration loops are frequently referred to as for loops because for is the keyword that is used to introduce them in nearly all programming languages, including Python.

Historically, programming languages have offered a few assorted flavors of for loop. These are briefly described in the following sections.

Numeric Range Loop

The most basic for loop is a simple numeric range statement with start and end values. The exact format varies depending on the language but typically looks something like this:

for i = 1 to 10
    <loop body>

Here, the body of the loop is executed ten times. The variable i assumes the value 1 on the first iteration, 2 on the second, and so on. This sort of for loop is used in the languages BASIC, Algol, and Pascal.

Three-Expression Loop

Another form of for loop popularized by the C programming language contains three parts:

An initialization
An expression specifying an ending condition
An action to be performed at the end of each iteration.

This type of has the following form:

for (i = 1; i <= 10; i++)
    <loop body>

Technical Note: In the C programming language, i++ increments the variable i. It is roughly equivalent to i += 1 in Python.

This loop is interpreted as follows:

Initialize i to 1.
Continue looping as long as i <= 10.
Increment i by 1 after each loop iteration.

Three-expression for loops are popular because the expressions specified for the three parts can be nearly anything, so this has quite a bit more flexibility than the simpler numeric range form shown above. These for loops are also featured in the C++, Java, PHP, and Perl languages.

Collection-Based or Iterator-Based Loop

This type of loop iterates over a collection of objects, rather than specifying numeric values or conditions:

for i in <collection>
    <loop body>

Each time through the loop, the variable i takes on the value of the next object in <collection>. This type of for loop is arguably the most generalized and abstract. Perl and PHP also support this type of loop, but it is introduced by the keyword foreach instead of for.

Further Reading: See the For loop Wikipedia page for an in-depth look at the implementation of definite iteration across programming languages.

The Python `for` Loop

Of the loop types listed above, Python only implements the last: collection-based iteration. At first blush, that may seem like a raw deal, but rest assured that Python’s implementation of definite iteration is so versatile that you won’t end up feeling cheated!

Shortly, you’ll dig into the guts of Python’s for loop in detail. But for now, let’s start with a quick prototype and example, just to get acquainted.

Python’s for loop looks like this:

for <var> in <iterable>:
    <statement(s)>

<iterable> is a collection of objects—for example, a list or tuple. The <statement(s)> in the loop body are denoted by indentation, as with all Python control structures, and are executed once for each item in <iterable>. The loop variable <var> takes on the value of the next element in <iterable> each time through the loop.

Here is a representative example:

>>>

>>> a = ['foo', 'bar', 'baz']
>>> for i in a:
...     print(i)
...
foo
bar
baz

In this example, <iterable> is the list a, and <var> is the variable i. Each time through the loop, i takes on a successive item in a, so print() displays the values 'foo', 'bar', and 'baz', respectively. A for loop like this is the Pythonic way to process the items in an iterable.

But what exactly is an iterable? Before examining for loops further, it will be beneficial to delve more deeply into what iterables are in Python.

Iterables

In Python, iterable means an object can be used in iteration. The term is used as:

An adjective: An object may be described as iterable.
A noun: An object may be characterized as an iterable.

If an object is iterable, it can be passed to the built-in Python function iter(), which returns something called an iterator. Yes, the terminology gets a bit repetitive. Hang in there. It all works out in the end.

Each of the objects in the following example is an iterable and returns some type of iterator when passed to iter():

>>>

>>> iter('foobar')                             # String
<str_iterator object at 0x036E2750>

>>> iter(['foo', 'bar', 'baz'])                # List
<list_iterator object at 0x036E27D0>

>>> iter(('foo', 'bar', 'baz'))                # Tuple
<tuple_iterator object at 0x036E27F0>

>>> iter({'foo', 'bar', 'baz'})                # Set
<set_iterator object at 0x036DEA08>

>>> iter({'foo': 1, 'bar': 2, 'baz': 3})       # Dict
<dict_keyiterator object at 0x036DD990>

These object types, on the other hand, aren’t iterable:

>>>

>>> iter(42)                                   # Integer
Traceback (most recent call last):
  File "<pyshell#26>", line 1, in <module>
    iter(42)
TypeError: 'int' object is not iterable

>>> iter(3.1)                                  # Float
Traceback (most recent call last):
  File "<pyshell#27>", line 1, in <module>
    iter(3.1)
TypeError: 'float' object is not iterable

>>> iter(len)                                  # Built-in function
Traceback (most recent call last):
  File "<pyshell#28>", line 1, in <module>
    iter(len)
TypeError: 'builtin_function_or_method' object is not iterable

All the data types you have encountered so far that are collection or container types are iterable. These include the string, list, tuple, dict, set, and frozenset types.

But these are by no means the only types that you can iterate over. Many objects that are built into Python or defined in modules are designed to be iterable. For example, open files in Python are iterable. As you will see soon in the tutorial on file I/O, iterating over an open file object reads data from the file.

In fact, almost any object in Python can be made iterable. Even user-defined objects can be designed in such a way that they can be iterated over. (You will find out how that is done in the upcoming article on object-oriented programming.)

Iterators

Okay, now you know what it means for an object to be iterable, and you know how to use iter() to obtain an iterator from it. Once you’ve got an iterator, what can you do with it?

An iterator is essentially a value producer that yields successive values from its associated iterable object. The built-in function next() is used to obtain the next value from in iterator.

Here is an example using the same list as above:

>>>

>>> a = ['foo', 'bar', 'baz']

>>> itr = iter(a)
>>> itr
<list_iterator object at 0x031EFD10>

>>> next(itr)
'foo'
>>> next(itr)
'bar'
>>> next(itr)
'baz'

In this example, a is an iterable list and itr is the associated iterator, obtained with iter(). Each next(itr) call obtains the next value from itr.

Notice how an iterator retains its state internally. It knows which values have been obtained already, so when you call next(), it knows what value to return next.

What happens when the iterator runs out of values? Let’s make one more next() call on the iterator above:

>>>

>>> next(itr)
Traceback (most recent call last):
  File "<pyshell#10>", line 1, in <module>
    next(itr)
StopIteration

If all the values from an iterator have been returned already, a subsequent next() call raises a StopIteration exception. Any further attempts to obtain values from the iterator will fail.

You can only obtain values from an iterator in one direction. You can’t go backward. There is no prev() function. But you can define two independent iterators on the same iterable object:

>>>

>>> a
['foo', 'bar', 'baz']

>>> itr1 = iter(a)
>>> itr2 = iter(a)

>>> next(itr1)
'foo'
>>> next(itr1)
'bar'
>>> next(itr1)
'baz'

>>> next(itr2)
'foo'

Even when iterator itr1 is already at the end of the list, itr2 is still at the beginning. Each iterator maintains its own internal state, independent of the other.

If you want to grab all the values from an iterator at once, you can use the built-in list() function. Among other possible uses, list() takes an iterator as its argument, and returns a list consisting of all the values that the iterator yielded:

>>>

>>> a = ['foo', 'bar', 'baz']
>>> itr = iter(a)
>>> list(itr)
['foo', 'bar', 'baz']

Similarly, the built-in tuple() and set() functions return a tuple and a set, respectively, from all the values an iterator yields:

>>>

>>> a = ['foo', 'bar', 'baz']

>>> itr = iter(a)
>>> tuple(itr)
('foo', 'bar', 'baz')

>>> itr = iter(a)
>>> set(itr)
{'baz', 'foo', 'bar'}

It isn’t necessarily advised to make a habit of this. Part of the elegance of iterators is that they are “lazy.” That means that when you create an iterator, it doesn’t generate all the items it can yield just then. It waits until you ask for them with next(). Items are not created until they are requested.

When you use list(), tuple(), or the like, you are forcing the iterator to generate all its values at once, so they can all be returned. If the total number of objects the iterator returns is very large, that may take a long time.

In fact, it is possible to create an iterator in Python that returns an endless series of objects. (You will learn how to do this in upcoming tutorials on generator functions and itertools.) If you try to grab all the values at once from an endless iterator, the program will hang.

The Guts of the Python `for` Loop

You now have been introduced to all the concepts you need to fully understand how Python’s for loop works. Before proceeding, let’s review the relevant terms:

Term	Meaning
Iteration	The process of looping through the objects or items in a collection
Iterable	An object (or the adjective used to describe an object) that can be iterated over
Iterator	The object that produces successive items or values from its associated iterable
`iter()`	The built-in function used to obtain an iterator from an iterable

Now, consider again the simple for loop presented at the start of this tutorial:

>>>

>>> a = ['foo', 'bar', 'baz']
>>> for i in a:
...     print(i)
...
foo
bar
baz

This loop can be described entirely in terms of the concepts you have just learned about. To carry out the iteration this for loop describes, Python does the following:

Calls iter() to obtain an iterator for a
Calls next() repeatedly to obtain each item from the iterator in turn
Terminates the loop when next() raises the StopIteration exception

The loop body is executed once for each item next() returns, with loop variable i set to the given item for each iteration.

This sequence of events is summarized in the following diagram:

Schematic Diagram of a Python for Loop

Perhaps this seems like a lot of unnecessary monkey business, but the benefit is substantial. Python treats looping over all iterables in exactly this way, and in Python, iterables and iterators abound:

Many built-in and library objects are iterable.
There is a Standard Library module called itertools containing many functions that return iterables.
User-defined objects created with Python’s object-oriented capability can be made to be iterable.
Python features a construct called a generator that allows you to create your own iterator in a simple, straightforward way.

You will discover more about all the above throughout this series. They can all be the target of a for loop, and the syntax is the same across the board. It’s elegant in its simplicity and eminently versatile.

Iterating Through a Dictionary

You saw earlier that an iterator can be obtained from a dictionary with iter(), so you know dictionaries must be iterable. What happens when you loop through a dictionary? Let’s see:

>>>

>>> d = {'foo': 1, 'bar': 2, 'baz': 3}
>>> for k in d:
...     print(k)
...
foo
bar
baz

As you can see, when a for loop iterates through a dictionary, the loop variable is assigned to the dictionary’s keys.

To access the dictionary values within the loop, you can make a dictionary reference using the key as usual:

>>>

>>> for k in d:
...     print(d[k])
...
1
2
3

You can also iterate through a dictionary’s values directly by using .values():

>>>

>>> for v in d.values():
...     print(v)
...
1
2
3

In fact, you can iterate through both the keys and values of a dictionary simultaneously. That is because the loop variable of a for loop isn’t limited to just a single variable. It can also be a tuple, in which case the assignments are made from the items in the iterable using packing and unpacking, just as with an assignment statement:

>>>

>>> i, j = (1, 2)
>>> print(i, j)
1 2

>>> for i, j in [(1, 2), (3, 4), (5, 6)]:
...     print(i, j)
...
1 2
3 4
5 6

As noted in the tutorial on Python dictionaries, the dictionary method .items() effectively returns a list of key/value pairs as tuples:

>>>

>>> d = {'foo': 1, 'bar': 2, 'baz': 3}

>>> d.items()
dict_items([('foo', 1), ('bar', 2), ('baz', 3)])

Thus, the Pythonic way to iterate through a dictionary accessing both the keys and values looks like this:

>>>

>>> d = {'foo': 1, 'bar': 2, 'baz': 3}
>>> for k, v in d.items():
...     print('k =', k, ', v =', v)
...
k = foo , v = 1
k = bar , v = 2
k = baz , v = 3

The `range()` Function

In the first section of this tutorial, you saw a type of for loop called a numeric range loop, in which starting and ending numeric values are specified. Although this form of for loop isn’t directly built into Python, it is easily arrived at.

For example, if you wanted to iterate through the values from 0 to 4, you could simply do this:

>>>

>>> for n in (0, 1, 2, 3, 4):
...     print(n)
...
0
1
2
3
4

This solution isn’t too bad when there are just a few numbers. But if the number range were much larger, it would become tedious pretty quickly.

Happily, Python provides a better option—the built-in range() function, which returns an iterable that yields a sequence of integers.

range(<end>) returns an iterable that yields integers starting with 0, up to but not including <end>:

>>>

>>> x = range(5)
>>> x
range(0, 5)
>>> type(x)
<class 'range'>

Note that range() returns an object of class range, not a list or tuple of the values. Because a range object is an iterable, you can obtain the values by iterating over them with a for loop:

>>>

>>> for n in x:
...     print(n)
...
0
1
2
3
4

You could also snag all the values at once with list() or tuple(). In a REPL session, that can be a convenient way to quickly display what the values are:

>>>

>>> list(x)
[0, 1, 2, 3, 4]

>>> tuple(x)
(0, 1, 2, 3, 4)

However, when range() is used in code that is part of a larger application, it is typically considered poor practice to use list() or tuple() in this way. Like iterators, range objects are lazy—the values in the specified range are not generated until they are requested. Using list() or tuple() on a range object forces all the values to be returned at once. This is rarely necessary, and if the list is long, it can waste time and memory.

range(<begin>, <end>, <stride>) returns an iterable that yields integers starting with <begin>, up to but not including <end>. If specified, <stride> indicates an amount to skip between values (analogous to the stride value used for string and list slicing):

>>>

>>> list(range(5, 20, 3))
[5, 8, 11, 14, 17]

If <stride> is omitted, it defaults to 1:

>>>

>>> list(range(5, 10, 1))
[5, 6, 7, 8, 9]
>>> list(range(5, 10))
[5, 6, 7, 8, 9]

All the parameters specified to range() must be integers, but any of them can be negative. Naturally, if <begin> is greater than <end>, <stride> must be negative (if you want any results):

>>>

>>> list(range(-5, 5))
[-5, -4, -3, -2, -1, 0, 1, 2, 3, 4]

>>> list(range(5, -5))
[]
>>> list(range(5, -5, -1))
[5, 4, 3, 2, 1, 0, -1, -2, -3, -4]

Technical Note: Strictly speaking, range() isn’t exactly a built-in function. It is implemented as a callable class that creates an immutable sequence type. But for practical purposes, it behaves like a built-in function.

For more information on range(), see the Real Python article Python’s range() Function (Guide).

Altering `for` Loop Behavior

You saw in the previous tutorial in this introductory series how execution of a while loop can be interrupted with break and continue statements and modified with an else clause. These capabilities are available with the for loop as well.

The `break` and `continue` Statements

break and continue work the same way with for loops as with while loops. break terminates the loop completely and proceeds to the first statement following the loop:

>>>

>>> for i in ['foo', 'bar', 'baz', 'qux']:
...     if 'b' in i:
...         break
...     print(i)
...
foo

continue terminates the current iteration and proceeds to the next iteration:

>>>

>>> for i in ['foo', 'bar', 'baz', 'qux']:
...     if 'b' in i:
...         continue
...     print(i)
...
foo
qux

The `else` Clause

A for loop can have an else clause as well. The interpretation is analogous to that of a while loop. The else clause will be executed if the loop terminates through exhaustion of the iterable:

>>>

>>> for i in ['foo', 'bar', 'baz', 'qux']:
...     print(i)
... else:
...     print('Done.')  # Will execute
...
foo
bar
baz
qux
Done.

The else clause won’t be executed if the list is broken out of with a break statement:

>>>

>>> for i in ['foo', 'bar', 'baz', 'qux']:
...     if i == 'bar':
...         break
...     print(i)
... else:
...     print('Done.')  # Will not execute
...
foo

Conclusion

This tutorial presented the for loop, the workhorse of definite iteration in Python.

You also learned about the inner workings of iterables and iterators, two important object types that underlie definite iteration, but also figure prominently in a wide variety of other Python code.

In the next two tutorials in this introductory series, you will shift gears a little and explore how Python programs can interact with the user via input from the keyboard and output to the console.

« Python "while" Loops (Indefinite Iteration)

Python "for" Loops (Definite Iteration)

Basic Input and Output in Python »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Jupyter Notebook: An Introduction

2019-01-28T06:00:00+00:00

The Jupyter Notebook is an open source web application that you can use to create and share documents that contain live code, equations, visualizations, and text. Jupyter Notebook is maintained by the people at Project Jupyter.

Jupyter Notebooks are a spin-off project from the IPython project, which used to have an IPython Notebook project itself. The name, Jupyter, comes from the core supported programming languages that it supports: Julia, Python, and R. Jupyter ships with the IPython kernel, which allows you to write your programs in Python, but there are currently over 100 other kernels that you can also use.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Getting Up and Running With Jupyter Notebook

The Jupyter Notebook is not included with Python, so if you want to try it out, you will need to install Jupyter.

There are many distributions of the Python language. This article will focus on just two of them for the purposes of installing Jupyter Notebook. The most popular is CPython, which is the reference version of Python that you can get from their website. It is also assumed that you are using Python 3.

Installation

If so, then you can use a handy tool that comes with Python called pip to install Jupyter Notebook like this:

$ pip install jupyter

The next most popular distribution of Python is Anaconda. Anaconda has its own installer tool called conda that you could use for installing a third-party package. However, Anaconda comes with many scientific libraries preinstalled, including the Jupyter Notebook, so you don’t actually need to do anything other than install Anaconda itself.

Starting the Jupyter Notebook Server

Now that you have Jupyter installed, let’s learn how to use it. To get started, all you need to do is open up your terminal application and go to a folder of your choice. I recommend using something like your Documents folder to start out with and create a subfolder there called Notebooks or something else that is easy to remember.

Then just go to that location in your terminal and run the following command:

$ jupyter notebook

This will start up Jupyter and your default browser should start (or open a new tab) to the following URL: http://localhost:8888/tree

Your browser should now look something like this:

Note that right now you are not actually running a Notebook, but instead you are just running the Notebook server. Let’s actually create a Notebook now!

Creating a Notebook

Now that you know how to start a Notebook server, you should probably learn how to create an actually Notebook document.

All you need to do is click on the New button (upper right), and it will open up a list of choices. On my machine, I happen to have Python 2 and Python 3 installed, so I can create a Notebook that uses either of these. For simplicity’s sake, let’s choose Python 3.

Your web page should now look like this:

Naming

You will notice that at the top of the page is the word Untitled. This is the title for the page and the name of your Notebook. Since that isn’t a very descriptive name, let’s change it!

Just move your mouse over the word Untitled and click on the text. You should now see an in-browser dialog titled Rename Notebook. Let’s rename this one to Hello Jupyter:

Running Cells

A Notebook’s cell defaults to using code whenever you first create one, and that cell uses the kernel that you chose when you started your Notebook.

In this case, you started yours with Python 3 as your kernel, so that means you can write Python code in your code cells. Since your initial Notebook has only one empty cell in it, the Notebook can’t really do anything.

Thus, to verify that everything is working as it should, you can add some Python code to the cell and try running its contents.

Let’s try adding the following code to that cell:

print('Hello Jupyter!')

Running a cell means that you will execute the cell’s contents. To execute a cell, you can just select the cell and click the Run button that is in the row of buttons along the top. It’s towards the middle. If you prefer using your keyboard, you can just press Shift+Enter.

When I ran the code above, the output looked like this:

If you have multiple cells in your Notebook, and you run the cells in order, you can share your variables and imports across cells. This makes it easy to separate out your code into logical chunks without needing to reimport libraries or recreate variables or functions in every cell.

When you run a cell, you will notice that there are some square braces next to the word In to the left of the cell. The square braces will auto fill with a number that indicates the order that you ran the cells. For example, if you open a fresh Notebook and run the first cell at the top of the Notebook, the square braces will fill with the number 1.

The Menus

The Jupyter Notebook has several menus that you can use to interact with your Notebook. The menu runs along the top of the Notebook just like menus do in other applications. Here is a list of the current menus:

File
Edit
View
Insert
Cell
Kernel
Widgets
Help

Let’s go over the menus one by one. This article won’t go into detail for every single option in every menu, but it will focus on the items that are unique to the Notebook application.

The first menu is the File menu. In it, you can create a new Notebook or open a preexisting one. This is also where you would go to rename a Notebook. I think the most interesting menu item is the Save and Checkpoint option. This allows you to create checkpoints that you can roll back to if you need to.

Next is the Edit menu. Here you can cut, copy, and paste cells. This is also where you would go if you wanted to delete, split, or merge a cell. You can reorder cells here too.

Note that some of the items in this menu are greyed out. The reason for this is that they do not apply to the currently selected cell. For example, a code cell cannot have an image inserted into it, but a Markdown cell can. If you see a greyed out menu item, try changing the cell’s type and see if the item becomes available to use.

The View menu is useful for toggling the visibility of the header and toolbar. You can also toggle Line Numbers within cells on or off. This is also where you would go if you want to mess about with the cell’s toolbar.

The Insert menu is just for inserting cells above or below the currently selected cell.

The Cell menu allows you to run one cell, a group of cells, or all the cells. You can also go here to change a cell’s type, although I personally find the toolbar to be more intuitive for that.

The other handy feature in this menu is the ability to clear a cell’s output. If you are planning to share your Notebook with others, you will probably want to clear the output first so that the next person can run the cells themselves.

The Kernel cell is for working with the kernel that is running in the background. Here you can restart the kernel, reconnect to it, shut it down, or even change which kernel your Notebook is using.

You probably won’t be working with the Kernel all that often, but there are times when you are debugging a Notebook that you will find you need to restart the Kernel. When that happens, this is where you would go.

The Widgets menu is for saving and clearing widget state. Widgets are basically JavaScript widgets that you can add to your cells to make dynamic content using Python (or another Kernel).

Finally you have the Help menu, which is where you go to learn about the Notebook’s keyboard shortcuts, a user interface tour, and lots of reference material.

Starting Terminals and Other Things

Jupyter Notebook also allows you to start more than just Notebooks. You can also create a text file, a folder, or a Terminal in your browser. Go back to the home page that opened when you first started the Jupyter server at http://localhost:8888/tree. Go to the New button and choose one of the other options.

The Terminal is probably the most interesting of the bunch, as it is running your operating systems terminal in the browser. This allows you to run bash, Powershell, and so on in your browser and run any shell command that you might need to there.

Viewing What’s Running

Also on the home page of your Jupyter server (http://localhost:8888/tree) are two other tabs: Running and Clusters.

The Running tab will tell you which Notebooks and Terminals you are currently running. This is useful for when you want to shut down your server but you need to make sure that you have saved all your data. Fortunately, Notebooks auto-save pretty frequently, so you rarely lose data. But it’s good to be able to see what’s running when you need to.

The other nice thing about this tab is that you can go through your running applications and shut them down there.

Adding Rich Content

Jupyter Notebook supports adding rich content to its cells. In this section, you will get an overview of just some of the things you can do with your cells using Markup and Code.

Cell Types

There are technically four cell types: Code, Markdown, Raw NBConvert, and Heading.

The Heading cell type is no longer supported and will display a dialog that says as much. Instead, you are supposed to use Markdown for your Headings.

The Raw NBConvert cell type is only intended for special use cases when using the nbconvert command line tool. Basically it allows you to control the formatting in a very specific way when converting from a Notebook to another format.

The primary cell types that you will use are the Code and Markdown cell types. You have already learned how code cells work, so let’s learn how to style your text with Markdown.

Styling Your Text

Jupyter Notebook supports Markdown, which is a markup language that is a superset of HTML. This turotial will cover some of the basics of what you can do with Markdown.

Set a new cell to Markdown and then add the following text to the cell:

When you run the cell, the output should look like this:

If you would prefer to bold your text, use a double underscore or double asterisk.

Headers

Creating headers in Markdown is also quite simple. You just have to use the humble pound sign. The more pound signs you use, the smaller the header. Jupyter Notebook even kind of previews it for you:

Then when you run the cell, you will end up with a nicely formatted header:

Creating Lists

You can create a list (bullet points) by using dashes, plus signs, or asterisks. Here is an example:

Code and Syntax Highlighting

If you want to insert a code example that you don’t want your end user to actually run, you can use Markdown to insert it. For inline code highlighting, just surround the code with backticks. If you want to insert a block of code, you can use triple backticks and also specify the programming language:

Exporting Notebooks

When you are working with Jupyter Notebooks, you will find that you need to share your results with non-technical people. When that happens, you can use the nbconvert tool which comes with Jupyter Notebook to convert or export your Notebook into one of the following formats:

HTML
LaTeX
PDF
RevealJS
Markdown
ReStructured Text
Executable script

The nbconvert tool uses Jinja templates under the covers to convert your Notebook files (.ipynb) into these other formats.

Jinja is a template engine that was made for Python. Also note that nbconvert also depends on Pandoc and TeX to be able to export to all the formats above. If you don’t have one or more of these, some of the export types may not work. For more information, you should check out the documentation.

How to Use `nbconvert`

The nbconvert command does not take very many parameters, which makes learning how to use it easier. Open up a terminal and navigate to the folder that contains the Notebook you wish to convert. The basic conversion command looks like this:

$ jupyter nbconvert <input 
notebook> --to <output format>

Example Usage

Let’s pretend that you have a Notebook named py_examples.ipynb and you want to convert it to PDF. Here is the command you would use to do that:

$ jupyter nbconvert py_examples.ipynb --to pdf

When you run this command, you should see some output that tells you about the conversion process. nbconvert will display warnings and errors if there are any. Assuming everything goes according to plan, you will now have a py_examples.pdf file in your folder.

The conversion process for the other file types is quite similar. You just have to tell nbconvert what type to convert to (PDF, Markdown, HTML, and so on).

You can also export your currently running Notebook by going to the File menu and choosing the Download as option.

This option allows you to download in all the formats that nbconvert supports. The benefit of using the menu is that you don’t need to learn the nbconvert at all if you don’t want to. However I recommend doing so as you can use nbconvert to export multiple Notebooks at once, which is something that the menu does not support.

Notebook Extensions

While Jupyter Notebooks have lots of functionality built in, you can add new functionality through extensions. Jupyter actually supports four types of extensions:

Kernel
IPython kernel
Notebook
Notebook server

This tutorial will focus on Notebook extensions.

What Are Extensions?

A Notebook extension (nbextension) is a JavaScript module that you load in most of the views in the Notebook’s frontend. If you are handy with JavaScript, you can even write your own extension. An extension can access the page’s DOM and the Jupyter JavaScript API.

Where Do I Get Extensions?

You can use Google or search for Jupyter Notebook extensions. There are actually quite a few out there. One of the most popular extension sets is called jupyter_contrib_nbextensions, which you can get from GitHub. This is actually a collection of extensions that is provided by the Jupyter community and installed with pip.

How Do I Install Them?

Most Jupyter Notebook extensions can be installed using Python’s pip tool. If you find an extension that can’t be installed with pip, then you will likely have to use the following command:

$ jupyter nbextension install EXTENSION_NAME

This only installs the extension but does not make it active. You will need to enable an extension after installing it by running the following:

$ jupyter nbextension enable EXTENSION_NAME

You may need to restart your Jupyter Notebook kernel to see the extension.

There is a nice meta extension called Jupyter NbExtensions Configurator that is worth getting for managing other extensions. It allows you to enable and disable your extensions from within the Jupyter Notebook’s user interface and also shows all the currently installed extensions.

Conclusion

The Jupyter Notebook is quite useful not only for learning and teaching a programming language such as Python but also for sharing your data.

You can turn your Notebook into a slideshow or share it online with GitHub. If you want to share a Notebook without requiring your users to install anything, you can use binder for that.

Google and Microsoft both have their own version of the Notebook that you can use to create and share your Notebooks at Google Colaboratory and Microsoft Azure Notebooks respectively. You can browse really interesting Notebooks there as well.

Project Jupyter recently launched their latest product, JupyterLab. JupyterLab incorporates Jupyter Notebook into an Integrated Development type Editor that you run in your browser. You can kind of think of JupyterLab as an advanced version of Jupyter Notebook. JupyterLab allows you to run terminals, text editors and code consoles in your browser in addition to Notebooks.

As always, it is best to try out a new piece of software yourself to see if it suits you and is worth using. I encourage you to give Jupyter Notebook or JupyterLab a spin and see what you think!

Python’s Requests Library (Guide)

2019-01-23T06:00:00+00:00

The requests library is the de facto standard for making HTTP requests in Python. It abstracts the complexities of making requests behind a beautiful, simple API so that you can focus on interacting with services and consuming data in your application.

Throughout this article, you’ll see some of the most useful features that requests has to offer as well as how to customize and optimize those features for different situations you may come across. You’ll also learn how to use requests in an efficient way as well as how to prevent requests to external services from slowing down your application.

In this tutorial, you’ll learn how to:

Make requests using the most common HTTP methods
Customize your requests’ headers and data, using the query string and message body
Inspect data from your requests and responses
Make authenticated requests
Configure your requests to help prevent your application from backing up or slowing down

Though I’ve tried to include as much information as you need to understand the features and examples included in this article, I do assume a very basic general knowledge of HTTP. That said, you still may be able to follow along fine anyway.

Now that that is out of the way, let’s dive in and see how you can use requests in your application!

Getting Started With `requests`

Let’s begin by installing the requests library. To do so, run the following command:

$ pip install requests

If you prefer to use Pipenv for managing Python packages, you can run the following:

$ pipenv install requests

Once requests is installed, you can use it in your application. Importing requests looks like this:

import requests

Now that you’re all set up, it’s time to begin your journey through requests. Your first goal will be learning how to make a GET request.

The GET Request

HTTP methods such as GET and POST, determine which action you’re trying to perform when making an HTTP request. Besides GET and POST, there are several other common methods that you’ll use later in this tutorial.

One of the most common HTTP methods is GET. The GET method indicates that you’re trying to get or retrieve data from a specified resource. To make a GET request, invoke requests.get().

To test this out, you can make a GET request to GitHub’s Root REST API by calling get() with the following URL:

>>>

>>> requests.get('https://api.github.com')
<Response [200]>

Congratulations! You’ve made your first request. Let’s dive a little deeper into the response of that request.

The Response

A Response is a powerful object for inspecting the results of the request. Let’s make that same request again, but this time store the return value in a variable so that you can get a closer look at its attributes and behaviors:

>>>

>>> response = requests.get('https://api.github.com')

In this example, you’ve captured the return value of get(), which is an instance of Response, and stored it in a variable called response. You can now use response to see a lot of information about the results of your GET request.

Status Codes

The first bit of information that you can gather from Response is the status code. A status code informs you of the status of the request.

For example, a 200 OK status means that your request was successful, whereas a 404 NOT FOUND status means that the resource you were looking for was not found. There are many other possible status codes as well to give you specific insights into what happened with your request.

By accessing .status_code, you can see the status code that the server returned:

>>>

>>> response.status_code
200

.status_code returned a 200, which means your request was successful and the server responded with the data you were requesting.

Sometimes, you might want to use this information to make decisions in your code:

if response.status_code == 200:
    print('Success!')
elif response.status_code == 404:
    print('Not Found.')

With this logic, if the server returns a 200 status code, your program will print Success!. If the result is a 404, your program will print Not Found.

requests goes one step further in simplifying this process for you. If you use a Response instance in a conditional expression, it will evaluate to True if the status code was between 200 and 400, and False otherwise.

Therefore, you can simplify the last example by rewriting the if statement:

if response:
    print('Success!')
else:
    print('An error has occurred.')

Technical Detail: This Truth Value Test is made possible because __bool__() is an overloaded method on Response.

This means that the default behavior of Response has been redefined to take the status code into account when determining the truth value of the object.

Keep in mind that this method is not verifying that the status code is equal to 200. The reason for this is that other status codes within the 200 to 400 range, such as 204 NO CONTENT and 304 NOT MODIFIED, are also considered successful in the sense that they provide some workable response.

For example, the 204 tells you that the response was successful, but there’s no content to return in the message body.

So, make sure you use this convenient shorthand only if you want to know if the request was generally successful and then, if necessary, handle the response appropriately based on the status code.

Let’s say you don’t want to check the response’s status code in an if statement. Instead, you want to raise an exception if the request was unsuccessful. You can do this using .raise_for_status():

import requests
from requests.exceptions import HTTPError

for url in ['https://api.github.com', 'https://api.github.com/invalid']:
    try:
        response = requests.get(url)

        # If the response was successful, no Exception will be raised
        response.raise_for_status()
    except HTTPError as http_err:
        print(f'HTTP error occurred: {http_err}')  # Python 3.6
    except Exception as err:
        print(f'Other error occurred: {err}')  # Python 3.6
    else:
        print('Success!')

If you invoke .raise_for_status(), an HTTPError will be raised for certain status codes. If the status code indicates a successful request, the program will proceed without that exception being raised.

Further Reading: If you’re not familiar with Python 3.6’s f-strings, I encourage you to take advantage of them as they are a great way to simplify your formatted strings.

Now, you know a lot about how to deal with the status code of the response you got back from the server. However, when you make a GET request, you rarely only care about the status code of the response. Usually, you want to see more. Next, you’ll see how to view the actual data that the server sent back in the body of the response.

Content

The response of a GET request often has some valuable information, known as a payload, in the message body. Using the attributes and methods of Response, you can view the payload in a variety of different formats.

To see the response’s content in bytes, you use .content:

>>>

>>> response = requests.get('https://api.github.com')
>>> response.content
b'{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'

While .content gives you access to the raw bytes of the response payload, you will often want to convert them into a string using a character encoding such as UTF-8. response will do that for you when you access .text:

>>>

>>> response.text
'{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'

Because the decoding of bytes to a str requires an encoding scheme, requests will try to guess the encoding based on the response’s headers if you do not specify one. You can provide an explicit encoding by setting .encoding before accessing .text:

>>>

>>> response.encoding = 'utf-8' # Optional: requests infers this internally
>>> response.text
'{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'

If you take a look at the response, you’ll see that it is actually serialized JSON content. To get a dictionary, you could take the str you retrieved from .text and deserialize it using json.loads(). However, a simpler way to accomplish this task is to use .json():

>>>

>>> response.json()
{'current_user_url': 'https://api.github.com/user', 'current_user_authorizations_html_url': 'https://github.com/settings/connections/applications{/client_id}', 'authorizations_url': 'https://api.github.com/authorizations', 'code_search_url': 'https://api.github.com/search/code?q={query}{&page,per_page,sort,order}', 'commit_search_url': 'https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}', 'emails_url': 'https://api.github.com/user/emails', 'emojis_url': 'https://api.github.com/emojis', 'events_url': 'https://api.github.com/events', 'feeds_url': 'https://api.github.com/feeds', 'followers_url': 'https://api.github.com/user/followers', 'following_url': 'https://api.github.com/user/following{/target}', 'gists_url': 'https://api.github.com/gists{/gist_id}', 'hub_url': 'https://api.github.com/hub', 'issue_search_url': 'https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}', 'issues_url': 'https://api.github.com/issues', 'keys_url': 'https://api.github.com/user/keys', 'notifications_url': 'https://api.github.com/notifications', 'organization_repositories_url': 'https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}', 'organization_url': 'https://api.github.com/orgs/{org}', 'public_gists_url': 'https://api.github.com/gists/public', 'rate_limit_url': 'https://api.github.com/rate_limit', 'repository_url': 'https://api.github.com/repos/{owner}/{repo}', 'repository_search_url': 'https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}', 'current_user_repositories_url': 'https://api.github.com/user/repos{?type,page,per_page,sort}', 'starred_url': 'https://api.github.com/user/starred{/owner}{/repo}', 'starred_gists_url': 'https://api.github.com/gists/starred', 'team_url': 'https://api.github.com/teams', 'user_url': 'https://api.github.com/users/{user}', 'user_organizations_url': 'https://api.github.com/user/orgs', 'user_repositories_url': 'https://api.github.com/users/{user}/repos{?type,page,per_page,sort}', 'user_search_url': 'https://api.github.com/search/users?q={query}{&page,per_page,sort,order}'}

The type of the return value of .json() is a dictionary, so you can access values in the object by key.

You can do a lot with status codes and message bodies. But, if you need more information, like metadata about the response itself, you’ll need to look at the response’s headers.

Headers

The response headers can give you useful information, such as the content type of the response payload and a time limit on how long to cache the response. To view these headers, access .headers:

>>>

>>> response.headers
{'Server': 'GitHub.com', 'Date': 'Mon, 10 Dec 2018 17:49:54 GMT', 'Content-Type': 'application/json; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Status': '200 OK', 'X-RateLimit-Limit': '60', 'X-RateLimit-Remaining': '59', 'X-RateLimit-Reset': '1544467794', 'Cache-Control': 'public, max-age=60, s-maxage=60', 'Vary': 'Accept', 'ETag': 'W/"7dc470913f1fe9bb6c7355b50a0737bc"', 'X-GitHub-Media-Type': 'github.v3; format=json', 'Access-Control-Expose-Headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload', 'X-Frame-Options': 'deny', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Referrer-Policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'Content-Security-Policy': "default-src 'none'", 'Content-Encoding': 'gzip', 'X-GitHub-Request-Id': 'E439:4581:CF2351:1CA3E06:5C0EA741'}

.headers returns a dictionary-like object, allowing you to access header values by key. For example, to see the content type of the response payload, you can access Content-Type:

>>>

>>> response.headers['Content-Type']
'application/json; charset=utf-8'

There is something special about this dictionary-like headers object, though. The HTTP spec defines headers to be case-insensitive, which means we are able to access these headers without worrying about their capitalization:

>>>

>>> response.headers['content-type']
'application/json; charset=utf-8'

Whether you use the key 'content-type' or 'Content-Type', you’ll get the same value.

Now, you’ve learned the basics about Response. You’ve seen its most useful attributes and methods in action. Let’s take a step back and see how your responses change when you customize your GET requests.

Query String Parameters

One common way to customize a GET request is to pass values through query string parameters in the URL. To do this using get(), you pass data to params. For example, you can use GitHub’s Search API to look for the requests library:

import requests

# Search GitHub's repositories for requests
response = requests.get(
    'https://api.github.com/search/repositories',
    params={'q': 'requests+language:python'},
)

# Inspect some attributes of the `requests` repository
json_response = response.json()
repository = json_response['items'][0]
print(f'Repository name: {repository["name"]}')  # Python 3.6+
print(f'Repository description: {repository["description"]}')  # Python 3.6+

By passing the dictionary {'q': 'requests+language:python'} to the params parameter of .get(), you are able to modify the results that come back from the Search API.

You can pass params to get() in the form of a dictionary, as you have just done, or as a list of tuples:

>>>

>>> requests.get(
...     'https://api.github.com/search/repositories',
...     params=[('q', 'requests+language:python')],
... )
<Response [200]>

You can even pass the values as bytes:

>>>

>>> requests.get(
...     'https://api.github.com/search/repositories',
...     params=b'q=requests+language:python',
... )
<Response [200]>

Query strings are useful for parameterizing GET requests. You can also customize your requests by adding or modifying the headers you send.

Request Headers

To customize headers, you pass a dictionary of HTTP headers to get() using the headers parameter. For example, you can change your previous search request to highlight matching search terms in the results by specifying the text-match media type in the Accept header:

import requests

response = requests.get(
    'https://api.github.com/search/repositories',
    params={'q': 'requests+language:python'},
    headers={'Accept': 'application/vnd.github.v3.text-match+json'},
)

# View the new `text-matches` array which provides information
# about your search term within the results
json_response = response.json()
repository = json_response['items'][0]
print(f'Text matches: {repository["text_matches"]}')

The Accept header tells the server what content types your application can handle. In this case, since you’re expecting the matching search terms to be highlighted, you’re using the header value application/vnd.github.v3.text-match+json, which is a proprietary GitHub Accept header where the content is a special JSON format.

Before you learn more ways to customize requests, let’s broaden the horizon by exploring other HTTP methods.

Other HTTP Methods

Aside from GET, other popular HTTP methods include POST, PUT, DELETE, HEAD, PATCH, and OPTIONS. requests provides a method, with a similar signature to get(), for each of these HTTP methods:

>>>

>>> requests.post('https://httpbin.org/post', data={'key':'value'})
>>> requests.put('https://httpbin.org/put', data={'key':'value'})
>>> requests.delete('https://httpbin.org/delete')
>>> requests.head('https://httpbin.org/get')
>>> requests.patch('https://httpbin.org/patch', data={'key':'value'})
>>> requests.options('https://httpbin.org/get')

Each function call makes a request to the httpbin service using the corresponding HTTP method. For each method, you can inspect their responses in the same way you did before:

>>>

>>> response = requests.head('https://httpbin.org/get')
>>> response.headers['Content-Type']
'application/json'

>>> response = requests.delete('https://httpbin.org/delete')
>>> json_response = response.json()
>>> json_response['args']
{}

Headers, response bodies, status codes, and more are returned in the Response for each method. Next you’ll take a closer look at the POST, PUT, and PATCH methods and learn how they differ from the other request types.

The Message Body

According to the HTTP specification, POST, PUT, and the less common PATCH requests pass their data through the message body rather than through parameters in the query string. Using requests, you’ll pass the payload to the corresponding function’s data parameter.

data takes a dictionary, a list of tuples, bytes, or a file-like object. You’ll want to adapt the data you send in the body of your request to the specific needs of the service you’re interacting with.

For example, if your request’s content type is application/x-www-form-urlencoded, you can send the form data as a dictionary:

>>>

>>> requests.post('https://httpbin.org/post', data={'key':'value'})
<Response [200]>

You can also send that same data as a list of tuples:

>>>

>>> requests.post('https://httpbin.org/post', data=[('key', 'value')])
<Response [200]>

If, however, you need to send JSON data, you can use the json parameter. When you pass JSON data via json, requests will serialize your data and add the correct Content-Type header for you.

httpbin.org is a great resource created by the author of requests, Kenneth Reitz. It’s a service that accepts test requests and responds with data about the requests. For instance, you can use it to inspect a basic POST request:

>>>

>>> response = requests.post('https://httpbin.org/post', json={'key':'value'})
>>> json_response = response.json()
>>> json_response['data']
'{"key": "value"}'
>>> json_response['headers']['Content-Type']
'application/json'

You can see from the response that the server received your request data and headers as you sent them. requests also provides this information to you in the form of a PreparedRequest.

Inspecting Your Request

When you make a request, the requests library prepares the request before actually sending it to the destination server. Request preparation includes things like validating headers and serializing JSON content.

You can view the PreparedRequest by accessing .request:

>>>

>>> response = requests.post('https://httpbin.org/post', json={'key':'value'})
>>> response.request.headers['Content-Type']
'application/json'
>>> response.request.url
'https://httpbin.org/post'
>>> response.request.body
b'{"key": "value"}'

Inspecting the PreparedRequest gives you access to all kinds of information about the request being made such as payload, URL, headers, authentication, and more.

So far, you’ve made a lot of different kinds of requests, but they’ve all had one thing in common: they’re unauthenticated requests to public APIs. Many services you may come across will want you to authenticate in some way.

Authentication

Authentication helps a service understand who you are. Typically, you provide your credentials to a server by passing data through the Authorization header or a custom header defined by the service. All the request functions you’ve seen to this point provide a parameter called auth, which allows you to pass your credentials.

One example of an API that requires authentication is GitHub’s Authenticated User API. This endpoint provides information about the authenticated user’s profile. To make a request to the Authenticated User API, you can pass your GitHub username and password in a tuple to get():

>>>

>>> from getpass import getpass
>>> requests.get('https://api.github.com/user', auth=('username', getpass()))
<Response [200]>

The request succeeded if the credentials you passed in the tuple to auth are valid. If you try to make this request with no credentials, you’ll see that the status code is 401 Unauthorized:

>>>

>>> requests.get('https://api.github.com/user')
<Response [401]>

When you pass your username and password in a tuple to the auth parameter, requests is applying the credentials using HTTP’s Basic access authentication scheme under the hood.

Therefore, you could make the same request by passing explicit Basic authentication credentials using HTTPBasicAuth:

>>>

>>> from requests.auth import HTTPBasicAuth
>>> from getpass import getpass
>>> requests.get(
...     'https://api.github.com/user',
...     auth=HTTPBasicAuth('username', getpass())
... )
<Response [200]>

Though you don’t need to be explicit for Basic authentication, you may want to authenticate using another method. requests provides other methods of authentication out of the box such as HTTPDigestAuth and HTTPProxyAuth.

You can even supply your own authentication mechanism. To do so, you must first create a subclass of AuthBase. Then, you implement __call__():

import requests
from requests.auth import AuthBase

class TokenAuth(AuthBase):
    """Implements a custom authentication scheme."""

    def __init__(self, token):
        self.token = token

    def __call__(self, r):
        """Attach an API token to a custom auth header."""
        r.headers['X-TokenAuth'] = f'{self.token}'  # Python 3.6+
        return r


requests.get('https://httpbin.org/get', auth=TokenAuth('12345abcde-token'))

Here, your custom TokenAuth mechanism receives a token, then includes that token in the X-TokenAuth header of your request.

Bad authentication mechanisms can lead to security vulnerabilities, so unless a service requires a custom authentication mechanism for some reason, you’ll always want to use a tried-and-true auth scheme like Basic or OAuth.

While you’re thinking about security, let’s consider dealing with SSL Certificates using requests.

SSL Certificate Verification

Any time the data you are trying to send or receive is sensitive, security is important. The way that you communicate with secure sites over HTTP is by establishing an encrypted connection using SSL, which means that verifying the target server’s SSL Certificate is critical.

The good news is that requests does this for you by default. However, there are some cases where you might want to change this behavior.

If you want to disable SSL Certificate verification, you pass False to the verify parameter of the request function:

>>>

>>> requests.get('https://api.github.com', verify=False)
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
<Response [200]>

requests even warns you when you’re making an insecure request to help you keep your data safe!

Note: requests uses a package called certifi to provide Certificate Authorities. This lets requests know which authorities it can trust. Therefore, you should update certifi frequently to keep your connections as secure as possible.

Performance

When using requests, especially in a production application environment, it’s important to consider performance implications. Features like timeout control, sessions, and retry limits can help you keep your application running smoothly.

Timeouts

When you make an inline request to an external service, your system will need to wait upon the response before moving on. If your application waits too long for that response, requests to your service could back up, your user experience could suffer, or your background jobs could hang.

By default, requests will wait indefinitely on the response, so you should almost always specify a timeout duration to prevent these things from happening. To set the request’s timeout, use the timeout parameter. timeout can be an integer or float representing the number of seconds to wait on a response before timing out:

>>>

>>> requests.get('https://api.github.com', timeout=1)
<Response [200]>
>>> requests.get('https://api.github.com', timeout=3.05)
<Response [200]>

In the first request, the request will timeout after 1 second. In the second request, the request will timeout after 3.05 seconds.

You can also pass a tuple to timeout with the first element being a connect timeout (the time it allows for the client to establish a connection to the server), and the second being a read timeout (the time it will wait on a response once your client has established a connection):

>>>

>>> requests.get('https://api.github.com', timeout=(2, 5))
<Response [200]>

If the request establishes a connection within 2 seconds and receives data within 5 seconds of the connection being established, then the response will be returned as it was before. If the request times out, then the function will raise a Timeout exception:

import requests
from requests.exceptions import Timeout

try:
    response = requests.get('https://api.github.com', timeout=1)
except Timeout:
    print('The request timed out')
else:
    print('The request did not time out')

Your program can catch the Timeout exception and respond accordingly.

The Session Object

Until now, you’ve been dealing with high level requests APIs such as get() and post(). These functions are abstractions of what’s going on when you make your requests. They hide implementation details such as how connections are managed so that you don’t have to worry about them.

Underneath those abstractions is a class called Session. If you need to fine-tune your control over how requests are being made or improve the performance of your requests, you may need to use a Session instance directly.

Sessions are used to persist parameters across requests. For example, if you want to use the same authentication across multiple requests, you could use a session:

import requests
from getpass import getpass

# By using a context manager, you can ensure the resources used by
# the session will be released after use
with requests.Session() as session:
    session.auth = ('username', getpass())

    # Instead of requests.get(), you'll use session.get()
    response = session.get('https://api.github.com/user')

# You can inspect the response just like you did before
print(response.headers)
print(response.json())

Each time you make a request with session, once it has been initialized with authentication credentials, the credentials will be persisted.

The primary performance optimization of sessions comes in the form of persistent connections. When your app makes a connection to a server using a Session, it keeps that connection around in a connection pool. When your app wants to connect to the same server again, it will reuse a connection from the pool rather than establishing a new one.

Max Retries

When a request fails, you may want your application to retry the same request. However, requests will not do this for you by default. To apply this functionality, you need to implement a custom Transport Adapter.

Transport Adapters let you define a set of configurations per service you’re interacting with. For example, let’s say you want all requests to https://api.github.com to retry three times before finally raising a ConnectionError. You would build a Transport Adapter, set its max_retries parameter, and mount it to an existing Session:

import requests
from requests.adapters import HTTPAdapter
from requests.exceptions import ConnectionError

github_adapter = HTTPAdapter(max_retries=3)

session = requests.Session()

# Use `github_adapter` for all requests to endpoints that start with this URL
session.mount('https://api.github.com', github_adapter)

try:
    session.get('https://api.github.com')
except ConnectionError as ce:
    print(ce)

When you mount the HTTPAdapter, github_adapter, to session, session will adhere to its configuration for each request to https://api.github.com.

Timeouts, Transport Adapters, and sessions are for keeping your code efficient and your application resilient.

Conclusion

You’ve come a long way in learning about Python’s powerful requests library.

You’re now able to:

Make requests using a variety of different HTTP methods such as GET, POST, and PUT
Customize your requests by modifying headers, authentication, query strings, and message bodies
Inspect the data you send to the server and the data the server sends back to you
Work with SSL Certificate verification
Use requests effectively using max_retries, timeout, Sessions, and Transport Adapters

Because you learned how to use requests, you’re equipped to explore the wide world of web services and build awesome applications using the fascinating data they provide.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Working With Files in Python

2019-01-21T06:00:00+00:00

Python has several built-in modules and functions for handling files. These functions are spread out over several modules such as os, os.path, shutil, and pathlib, to name a few. This article gathers in one place many of the functions you need to know in order to perform the most common operations on files in Python.

In this tutorial, you’ll learn how to:

Retrieve file properties
Create directories
Match patterns in filenames
Traverse directory trees
Make temporary files and directories
Delete files and directories
Copy, move, or rename files and directories
Create and extract ZIP and TAR archives
Open multiple files using the fileinput module

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Reading and Writing Data to Files in Python

Reading and writing data to files using Python is pretty straightforward. To do this, you must first open files in the appropriate mode. Here’s an example of how to open a text file and read its contents:

with open('data.txt', 'r') as f:
    data = f.read()

open() takes a filename and a mode as its arguments. r opens the file in read only mode. To write data to a file, pass in w as an argument instead:

with open('data.txt', 'w') as f:
    data = 'some data to be written to the file'
    f.write(data)

In the examples above, open() opens files for reading or writing and returns a file handle (f in this case) that provides methods that can be used to read or write data to the file. Read Working With File I/O in Python for more information on how to read and write to files.

Getting a Directory Listing

Suppose your current working directory has a subdirectory called my_directory that has the following contents:

my_directory/
|
├── sub_dir/
|   ├── bar.py
|   └── foo.py
|
├── sub_dir_b/
|   └── file4.txt
|
├── sub_dir_c/
|   ├── config.py
|   └── file5.txt
|
├── file1.py
├── file2.csv
└── file3.txt

The built-in os module has a number of useful functions that can be used to list directory contents and filter the results. To get a list of all the files and folders in a particular directory in the filesystem, use os.listdir() in legacy versions of Python or os.scandir() in Python 3.x. os.scandir() is the preferred method to use if you also want to get file and directory properties such as file size and modification date.

Directory Listing in Legacy Python Versions

In versions of Python prior to Python 3, os.listdir() is the method to use to get a directory listing:

>>>

>>> import os
>>> entries = os.listdir('my_directory/')

os.listdir() returns a Python list containing the names of the files and subdirectories in the directory given by the path argument:

>>>

>>> os.listdir('my_directory/')
['sub_dir_c', 'file1.py', 'sub_dir_b', 'file3.txt', 'file2.csv', 'sub_dir']

A directory listing like that isn’t easy to read. Printing out the output of a call to os.listdir() using a loop helps clean things up:

>>>

>>> entries = os.listdir('my_directory/')
>>> for entry in entries:
...     print(entry)
...
...
sub_dir_c
file1.py
sub_dir_b
file3.txt
file2.csv
sub_dir

Directory Listing in Modern Python Versions

In modern versions of Python, an alternative to os.listdir() is to use os.scandir() and pathlib.Path().

os.scandir() was introduced in Python 3.5 and is documented in PEP 471. os.scandir() returns an iterator as opposed to a list when called:

>>>

>>> import os
>>> entries = os.scandir('my_directory/')
>>> entries
<posix.ScandirIterator object at 0x7f5b047f3690>

The ScandirIterator points to all the entries in the current directory. You can loop over the contents of the iterator and print out the filenames:

import os

with os.scandir('my_directory/') as entries:
    for entry in entries:
        print(entry.name)

Here, os.scandir() is used in conjunction with the with statement because it supports the context manager protocol. Using a context manager closes the iterator and frees up acquired resources automatically after the iterator has been exhausted. The result is a print out of the filenames in my_directory/ just like you saw in the os.listdir() example:

sub_dir_c
file1.py
sub_dir_b
file3.txt
file2.csv
sub_dir

Another way to get a directory listing is to use the pathlib module:

from pathlib import Path

entries = Path('my_directory/')
for entry in entries.iterdir():
    print(entry.name)

The objects returned by Path are either PosixPath or WindowsPath objects depending on the OS.

pathlib.Path() objects have an .iterdir() method for creating an iterator of all files and folders in a directory. Each entry yielded by .iterdir() contains information about the file or directory such as its name and file attributes. pathlib was first introduced in Python 3.4 and is a great addition to Python that provides an object oriented interface to the filesystem.

In the example above, you call pathlib.Path() and pass a path argument to it. Next is the call to .iterdir() to get a list of all files and directories in my_directory.

pathlib offers a set of classes featuring most of the common operations on paths in an easy, object-oriented way. Using pathlib is more if not equally efficient as using the functions in os. Another benefit of using pathlib over os is that it reduces the number of imports you need to make to manipulate filesystem paths. For more information, read Python 3’s pathlib Module: Taming the File System.

Running the code above produces the following:

sub_dir_c
file1.py
sub_dir_b
file3.txt
file2.csv
sub_dir

Using pathlib.Path() or os.scandir() instead of os.listdir() is the preferred way of getting a directory listing, especially when you’re working with code that needs the file type and file attribute information. pathlib.Path() offers much of the file and path handling functionality found in os and shutil, and it’s methods are more efficient than some found in these modules. We will discuss how to get file properties shortly.

Here are the directory-listing functions again:

Function	Description
`os.listdir()`	Returns a list of all files and folders in a directory
`os.scandir()`	Returns an iterator of all the objects in a directory including file attribute information
`pathlib.Path.iterdir()`	Returns an iterator of all the objects in a directory including file attribute information

These functions return a list of everything in the directory, including subdirectories. This might not always be the behavior you want. The next section will show you how to filter the results from a directory listing.

Listing All Files in a Directory

This section will show you how to print out the names of files in a directory using os.listdir(), os.scandir(), and pathlib.Path(). To filter out directories and only list files from a directory listing produced by os.listdir(), use os.path:

import os

# List all files in a directory using os.listdir
basepath = 'my_directory/'
for entry in os.listdir(basepath):
    if os.path.isfile(os.path.join(basepath, entry)):
        print(entry)

Here, the call to os.listdir() returns a list of everything in the specified path, and then that list is filtered by os.path.isfile() to only print out files and not directories. This produces the following output:

file1.py
file3.txt
file2.csv

An easier way to list files in a directory is to use os.scandir() or pathlib.Path():

import os

# List all files in a directory using scandir()
basepath = 'my_directory/'
with os.scandir(basepath) as entries:
    for entry in entries:
        if entry.is_file():
            print(entry.name)

Using os.scandir() has the advantage of looking cleaner and being easier to understand than using os.listdir(), even though it is one line of code longer. Calling entry.is_file() on each item in the ScandirIterator returns True if the object is a file. Printing out the names of all files in the directory gives you the following output:

file1.py
file3.txt
file2.csv

Here’s how to list files in a directory using pathlib.Path():

from pathlib import Path

basepath = Path('my_directory/')
files_in_basepath = basepath.iterdir()
for item in files_in_basepath:
    if item.is_file():
        print(item.name)

Here, you call .is_file() on each entry yielded by .iterdir(). The output produced is the same:

file1.py
file3.txt
file2.csv

The code above can be made more concise if you combine the for loop and the if statement into a single generator expression. Dan Bader has an excellent article on generator expressions and list comprehensions.

The modified version looks like this:

from pathlib import Path

# List all files in directory using pathlib
basepath = Path('my_directory/')
files_in_basepath = (entry for entry in basepath.iterdir() if entry.is_file())
for item in files_in_basepath:
    print(item.name)

This produces exactly the same output as the example before it. This section showed that filtering files or directories using os.scandir() and pathlib.Path() feels more intuitive and looks cleaner than using os.listdir() in conjunction with os.path.

Listing Subdirectories

To list subdirectories instead of files, use one of the methods below. Here’s how to use os.listdir() and os.path():

import os

# List all subdirectories using os.listdir
basepath = 'my_directory/'
for entry in os.listdir(basepath):
    if os.path.isdir(os.path.join(basepath, entry)):
        print(entry)

Manipulating filesystem paths this way can quickly become cumbersome when you have multiple calls to os.path.join(). Running this on my computer produces the following output:

sub_dir_c
sub_dir_b
sub_dir

Here’s how to use os.scandir():

import os

# List all subdirectories using scandir()
basepath = 'my_directory/'
with os.scandir(basepath) as entries:
    for entry in entries:
        if entry.is_dir():
            print(entry.name)

As in the file listing example, here you call .is_dir() on each entry returned by os.scandir(). If the entry is a directory, .is_dir() returns True, and the directory’s name is printed out. The output is the same as above:

sub_dir_c
sub_dir_b
sub_dir

Here’s how to use pathlib.Path():

from pathlib import Path

# List all subdirectory using pathlib
basepath = Path('my_directory/')
for entry in basepath.iterdir():
    if entry.is_dir():
        print(entry.name)

Calling .is_dir() on each entry of the basepath iterator checks if an entry is a file or a directory. If the entry is a directory, its name is printed out to the screen, and the output produced is the same as the one from the previous example:

sub_dir_c
sub_dir_b
sub_dir

Getting File Attributes

Python makes retrieving file attributes such as file size and modified times easy. This is done through os.stat(), os.scandir(), or pathlib.Path().

os.scandir() and pathlib.Path() retrieve a directory listing with file attributes combined. This can be potentially more efficient than using os.listdir() to list files and then getting file attribute information for each file.

The examples below show how to get the time the files in my_directory/ were last modified. The output is in seconds:

>>>

>>> import os
>>> with os.scandir('my_directory/') as dir_contents:
...     for entry in dir_contents:
...         info = entry.stat()
...         print(info.st_mtime)
...
1539032199.0052035
1539032469.6324475
1538998552.2402923
1540233322.4009316
1537192240.0497339
1540266380.3434134

os.scandir() returns a ScandirIterator object. Each entry in a ScandirIterator object has a .stat() method that retrieves information about the file or directory it points to. .stat() provides information such as file size and the time of last modification. In the example above, the code prints out the st_mtime attribute, which is the time the content of the file was last modified.

The pathlib module has corresponding methods for retrieving file information that give the same results:

>>>

>>> from pathlib import Path
>>> current_dir = Path('my_directory')
>>> for path in current_dir.iterdir():
...     info = path.stat()
...     print(info.st_mtime)
...
1539032199.0052035
1539032469.6324475
1538998552.2402923
1540233322.4009316
1537192240.0497339
1540266380.3434134

In the example above, the code loops through the object returned by .iterdir() and retrieves file attributes through a .stat() call for each file in the directory list. The st_mtime attribute returns a float value that represents seconds since the epoch. To convert the values returned by st_mtime for display purposes, you could write a helper function to convert the seconds into a datetime object:

from datetime import datetime
from os import scandir

def convert_date(timestamp):
    d = datetime.utcfromtimestamp(timestamp)
    formated_date = d.strftime('%d %b %Y')
    return formated_date

def get_files():
    dir_entries = scandir('my_directory/')
    for entry in dir_entries:
        if entry.is_file():
            info = entry.stat()
            print(f'{entry.name}\t Last Modified: {convert_date(info.st_mtime)}')

This will first get a list of files in my_directory and their attributes and then call convert_date() to convert each file’s last modified time into a human readable form. convert_date() makes use of .strftime() to convert the time in seconds into a string.

The arguments passed to .strftime() are the following:

%d: the day of the month
%b: the month, in abbreviated form
%Y: the year

Together, these directives produce output that looks like this:

>>>

>>> get_files()
file1.py        Last modified:  04 Oct 2018
file3.txt       Last modified:  17 Sep 2018
file2.txt       Last modified:  17 Sep 2018

The syntax for converting dates and times into strings can be quite confusing. To read more about it, check out the official documentation on it. Another handy reference that is easy to remember is http://strftime.org/ .

Making Directories

Sooner or later, the programs you write will have to create directories in order to store data in them. os and pathlib include functions for creating directories. We’ll consider these:

Function	Description
`os.mkdir()`	Creates a single subdirectory
`pathlib.Path.mkdir()`	Creates single or multiple directories
`os.makedirs()`	Creates multiple directories, including intermediate directories

Creating a Single Directory

To create a single directory, pass a path to the directory as a parameter to os.mkdir():

import os

os.mkdir('example_directory/')

If a directory already exists, os.mkdir() raises FileExistsError. Alternatively, you can create a directory using pathlib:

from pathlib import Path

p = Path('example_directory/')
p.mkdir()

If the path already exists, mkdir() raises a FileExistsError:

>>>

>>> p.mkdir()
Traceback (most recent call last):
  File '<stdin>', line 1, in <module>
  File '/usr/lib/python3.5/pathlib.py', line 1214, in mkdir
    self._accessor.mkdir(self, mode)
  File '/usr/lib/python3.5/pathlib.py', line 371, in wrapped
    return strfunc(str(pathobj), *args)
FileExistsError: [Errno 17] File exists: '.'
[Errno 17] File exists: '.'

To avoid errors like this, catch the error when it happens and let your user know:

from pathlin Path

p = Path('example_directory')
try:
    p.mkdir()
except FileExistsError as exc:
    print(exc)

Alternatively, you can ignore the FileExistsError by passing the exist_ok=True argument to .mkdir():

from pathlib import Path

p = Path('example_directory')
p.mkdir(exist_ok=True)

This will not raise an error if the directory already exists.

Creating Multiple Directories

os.makedirs() is similar to os.mkdir(). The difference between the two is that not only can os.makedirs() create individual directories, it can also be used to create directory trees. In other words, it can create any necessary intermediate folders in order to ensure a full path exists.

os.makedirs() is similar to running mkdir -p in Bash. For example, to create a group of directories like 2018/10/05, all you have to do is the following:

import os


os.makedirs('2018/10/05')

This will create a nested directory structure that contains the folders 2018, 10, and 05:

.
|
└── 2018/
    └── 10/
        └── 05/

.makedirs() creates directories with default permissions. If you need to create directories with different permissions call .makedirs() and pass in the mode you would like the directories to be created in:

import os

os.makedirs('2018/10/05', mode=0o770)

This creates the 2018/10/05 directory structure and gives the owner and group users read, write, and execute permissions. The default mode is 0o777, and the file permission bits of existing parent directories are not changed. For more details on file permissions, and how the mode is applied, see the docs.

Run tree to confirm that the right permissions were applied:

$ tree -p -i .
.
[drwxrwx---]  2018
[drwxrwx---]  10
[drwxrwx---]  05

This prints out a directory tree of the current directory. tree is normally used to list contents of directories in a tree-like format. Passing the -p and -i arguments to it prints out the directory names and their file permission information in a vertical list. -p prints out the file permissions, and -i makes tree produce a vertical list without indentation lines.

As you can see, all of the directories have 770 permissions. An alternative way to create directories is to use .mkdir() from pathlib.Path:

import pathlib

p = pathlib.Path('2018/10/05')
p.mkdir(parents=True)

Passing parents=True to Path.mkdir() makes it create the directory 05 and any parent directories necessary to make the path valid.

By default, os.makedirs() and Path.mkdir() raise an OSError if the target directory already exists. This behavior can be overridden (as of Python 3.2) by passing exist_ok=True as a keyword argument when calling each function.

Running the code above produces a directory structure like the one below in one go:

.
|
└── 2018/
    └── 10/
        └── 05/

I prefer using pathlib when creating directories because I can use the same function to create single or nested directories.

Filename Pattern Matching

After getting a list of files in a directory using one of the methods above, you will most probably want to search for files that match a particular pattern.

These are the methods and functions available to you:

endswith() and startswith() string methods
fnmatch.fnmatch()
glob.glob()
pathlib.Path.glob()

Each of these is discussed below. The examples in this section will be performed on a directory called some_directory that has the following structure:

.
|
├── sub_dir/
|   ├── file1.py
|   └── file2.py
|
├── admin.py
├── data_01_backup.txt
├── data_01.txt
├── data_02_backup.txt
├── data_02.txt
├── data_03_backup.txt
├── data_03.txt
└── tests.py

If you’re following along using a Bash shell, you can create the above directory structure using the following commands:

$ mkdir some_directory
$ cd some_directory/
$ mkdir sub_dir
$ touch sub_dir/file1.py sub_dir/file2.py
$ touch data_{01..03}.txt data_{01..03}_backup.txt admin.py tests.py

This will create the some_directory/ directory, change into it, and then create sub_dir. The next line creates file1.py and file2.py in sub_dir, and the last line creates all the other files using expansion. To learn more about shell expansion, visit this site.

Using String Methods

Python has several built-in methods for modifying and manipulating strings. Two of these methods, .startswith() and .endswith(), are useful when you’re searching for patterns in filenames. To do this, first get a directory listing and then iterate over it:

>>>

>>> import os

>>> # Get .txt files
>>> for f_name in os.listdir('some_directory'):
...     if f_name.endswith('.txt'):
...         print(f_name)

The code above finds all the files in some_directory/, iterates over them and uses .endswith() to print out the filenames that have the .txt file extension. Running this on my computer produces the following output:

data_01.txt
data_03.txt
data_03_backup.txt
data_02_backup.txt
data_02.txt
data_01_backup.txt

Simple Filename Pattern Matching Using `fnmatch`

String methods are limited in their matching abilities. fnmatch has more advanced functions and methods for pattern matching. We will consider fnmatch.fnmatch(), a function that supports the use of wildcards such as * and ? to match filenames. For example, in order to find all .txt files in a directory using fnmatch, you would do the following:

>>>

>>> import os
>>> import fnmatch

>>> for file_name in os.listdir('some_directory/'):
...     if fnmatch.fnmatch(file_name, '*.txt'):
...         print(file_name)

This iterates over the list of files in some_directory and uses .fnmatch() to perform a wildcard search for files that have the .txt extension.

More Advanced Pattern Matching

Let’s suppose you want to find .txt files that meet certain criteria. For example, you could be only interested in finding .txt files that contain the word data, a number between a set of underscores, and the word backup in their filename. Something similar to data_01_backup, data_02_backup, or data_03_backup.

Using fnmatch.fnmatch(), you could do it this way:

>>>

>>> for filename in os.listdir('.'):
...     if fnmatch.fnmatch(filename, 'data_*_backup.txt'):
...         print(filename)

Here, you print only the names of files that match the data_*_backup.txt pattern. The asterisk in the pattern will match any character, so running this will find all text files whose filenames start with the word data and end in backup.txt, as you can see from the output below:

data_03_backup.txt
data_02_backup.txt
data_01_backup.txt

Filename Pattern Matching Using `glob`

Another useful module for pattern matching is glob.

.glob() in the glob module works just like fnmatch.fnmatch(), but unlike fnmatch.fnmatch(), it treats files beginning with a period (.) as special.

UNIX and related systems translate name patterns with wildcards like ? and * into a list of files. This is called globbing.

For example, typing mv *.py python_files/ in a UNIX shell moves (mv) all files with the .py extension from the current directory to the directory python_files. The * character is a wildcard that means “any number of characters,” and *.py is the glob pattern. This shell capability is not available in the Windows Operating System. The glob module adds this capability in Python, which enables Windows programs to use this feature.

Here’s an example of how to use glob to search for all Python (.py) source files in the current directory:

>>>

>>> import glob
>>> glob.glob('*.py')
['admin.py', 'tests.py']

glob.glob('*.py') searches for all files that have the .py extension in the current directory and returns them as a list. glob also supports shell-style wildcards to match patterns:

>>>

>>> import glob
>>> for name in glob.glob('*[0-9]*.txt'):
...     print(name)

This finds all text (.txt) files that contain digits in the filename:

data_01.txt
data_03.txt
data_03_backup.txt
data_02_backup.txt
data_02.txt
data_01_backup.txt

glob makes it easy to search for files recursively in subdirectories too:

>>>

>>> import glob
>>> for file in glob.iglob('**/*.py', recursive=True):
...     print(file)

This example makes use of glob.iglob() to search for .py files in the current directory and subdirectories. Passing recursive=True as an argument to .iglob() makes it search for .py files in the current directory and any subdirectories. The difference between glob.iglob() and glob.glob() is that .iglob() returns an iterator instead of a list.

Running the program above produces the following:

admin.py
tests.py
sub_dir/file1.py
sub_dir/file2.py

pathlib contains similar methods for making flexible file listings. The example below shows how you can use .Path.glob() to list file types that start with the letter p:

>>>

>>> from pathlib import Path
>>> p = Path('.')
>>> for name in p.glob('*.p*'):
...     print(name)

admin.py
scraper.py
docs.pdf

Calling p.glob('*.p*') returns a generator object that points to all files in the current directory that start with the letter p in their file extension.

Path.glob() is similar to os.glob() discussed above. As you can see, pathlib combines many of the best features of the os, os.path, and glob modules into one single module, which makes it a joy to use.

To recap, here is a table of the functions we have covered in this section:

Function	Description
`startswith()`	Tests if a string starts with a specified pattern and returns `True` or `False`
`endswith()`	Tests if a string ends with a specified pattern and returns `True` or `False`
`fnmatch.fnmatch(filename, pattern)`	Tests whether the filename matches the pattern and returns `True` or `False`
`glob.glob()`	Returns a list of filenames that match a pattern
`pathlib.Path.glob()`	Finds patterns in path names and returns a generator object

Traversing Directories and Processing Files

A common programming task is walking a directory tree and processing files in the tree. Let’s explore how the built-in Python function os.walk() can be used to do this. os.walk() is used to generate filename in a directory tree by walking the tree either top-down or bottom-up. For the purposes of this section, we’ll be manipulating the following directory tree:

.
|
├── folder_1/
|   ├── file1.py
|   ├── file2.py
|   └── file3.py
|
├── folder_2/
|   ├── file4.py
|   ├── file5.py
|   └── file6.py
|
├── test1.txt
└── test2.txt

The following is an example that shows you how to list all files and directories in a directory tree using os.walk().

os.walk() defaults to traversing directories in a top-down manner:

# Walking a directory tree and printing the names of the directories and files
for dirpath, dirnames, files in os.walk('.'):
    print(f'Found directory: {dirpath}')
    for file_name in files:
        print(file_name)

os.walk() returns three values on each iteration of the loop:

The name of the current folder
A list of folders in the current folder
A list of files in the current folder

On each iteration, it prints out the names of the subdirectories and files it finds:

Found directory: .
test1.txt
test2.txt
Found directory: ./folder_1
file1.py
file3.py
file2.py
Found directory: ./folder_2
file4.py
file5.py
file6.py

To traverse the directory tree in a bottom-up manner, pass in a topdown=False keyword argument to os.walk():

for dirpath, dirnames, files in os.walk('.', topdown=False):
    print(f'Found directory: {dirpath}')
    for file_name in files:
        print(file_name)

Passing the topdown=False argument will make os.walk() print out the files it finds in the subdirectories first:

Found directory: ./folder_1
file1.py
file3.py
file2.py
Found directory: ./folder_2
file4.py
file5.py
file6.py
Found directory: .
test1.txt
test2.txt

As you can see, the program started by listing the contents of the subdirectories before listing the contents of the root directory. This is very useful in situations where you want to recursively delete files and directories. You will learn how to do this in the sections below. By default, os.walk does not walk down into symbolic links that resolve to directories. This behavior can be overridden by calling it with a followlinks=True argument.

Making Temporary Files and Directories

Python provides a handy module for creating temporary files and directories called tempfile.

tempfile can be used to open and store data temporarily in a file or directory while your program is running. tempfile handles the deletion of the temporary files when your program is done with them.

Here’s how to create a temporary file:

from tempfile import TemporaryFile

# Create a temporary file and write some data to it
fp = TemporaryFile('w+t')
fp.write('Hello universe!')

# Go back to the beginning and read data from file
fp.seek(0)
data = fp.read()

# Close the file, after which it will be removed
fp.close()

The first step is to import TemporaryFile from the tempfile module. Next, create a file like object using the TemporaryFile() method by calling it and passing the mode you want to open the file in. This will create and open a file that can be used as a temporary storage area.

In the example above, the mode is 'w+t', which makes tempfile create a temporary text file in write mode. There is no need to give the temporary file a filename since it will be destroyed after the script is done running.

After writing to the file, you can read from it and close it when you’re done processing it. Once the file is closed, it will be deleted from the filesystem. If you need to name the temporary files produced using tempfile, use tempfile.NamedTemporaryFile().

The temporary files and directories created using tempfile are stored in a special system directory for storing temporary files. Python searches a standard list of directories to find one that the user can create files in.

On Windows, the directories are C:\TEMP, C:\TMP, \TEMP, and \TMP, in that order. On all other platforms, the directories are /tmp, /var/tmp, and /usr/tmp, in that order. As a last resort, tempfile will save temporary files and directories in the current directory.

.TemporaryFile() is also a context manager so it can be used in conjunction with the with statement. Using a context manager takes care of closing and deleting the file automatically after it has been read:

with TemporaryFile('w+t') as fp:
    fp.write('Hello universe!')
    fp.seek(0)
    fp.read()
# File is now closed and removed

This creates a temporary file and reads data from it. As soon as the file’s contents are read, the temporary file is closed and deleted from the file system.

tempfile can also be used to create temporary directories. Let’s look at how you can do this using tempfile.TemporaryDirectory():

>>>

>>> import tempfile
>>> with tempfile.TemporaryDirectory() as tmpdir:
...     print('Created temporary directory ', tmpdir)
...     os.path.exists(tmpdir)
...
Created temporary directory  /tmp/tmpoxbkrm6c
True

>>> # Directory contents have been removed
...
>>> tmpdir
'/tmp/tmpoxbkrm6c'
>>> os.path.exists(tmpdir)
False

Calling tempfile.TemporaryDirectory() creates a temporary directory in the file system and returns an object representing this directory. In the example above, the directory is created using a context manager, and the name of the directory is stored in tmpdir. The third line prints out the name of the temporary directory, and os.path.exists(tmpdir) confirms if the directory was actually created in the file system.

After the context manager goes out of context, the temporary directory is deleted and a call to os.path.exists(tmpdir) returns False, which means that the directory was succesfully deleted.

Deleting Files and Directories

You can delete single files, directories, and entire directory trees using the methods found in the os, shutil, and pathlib modules. The following sections describe how to delete files and directories that you no longer need.

Deleting Files in Python

To delete a single file, use pathlib.Path.unlink(), os.remove(). or os.unlink().

os.remove() and os.unlink() are semantically identical. To delete a file using os.remove(), do the following:

import os

data_file = 'C:\\Users\\vuyisile\\Desktop\\Test\\data.txt'
os.remove(data_file)

Deleting a file using os.unlink() is similar to how you do it using os.remove():

import os

data_file = 'C:\\Users\\vuyisile\\Desktop\\Test\\data.txt'
os.unlink(data_file)

Calling .unlink() or .remove() on a file deletes the file from the filesystem. These two functions will throw an OSError if the path passed to them points to a directory instead of a file. To avoid this, you can either check that what you’re trying to delete is actually a file and only delete it if it is, or you can use exception handling to handle the OSError:

import os

data_file = 'home/data.txt'

# If the file exists, delete it
if os.path.isfile(data_file):
    os.remove(data_file)
else:
    print(f'Error: {data_file} not a valid filename')

os.path.isfile() checks whether data_file is actually a file. If it is, it is deleted by the call to os.remove(). If data_file points to a folder, an error message is printed to the console.

The following example shows how to use exception handling to handle errors when deleting files:

import os

data_file = 'home/data.txt'

# Use exception handling
try:
    os.remove(data_file)
except OSError as e:
    print(f'Error: {data_file} : {e.strerror}')

The code above attempts to delete the file first before checking its type. If data_file isn’t actually a file, the OSError that is thrown is handled in the except clause, and an error message is printed to the console. The error message that gets printed out is formatted using Python f-strings.

Finally, you can also use pathlib.Path.unlink() to delete files:

from pathlib import Path

data_file = Path('home/data.txt')

try:
    data_file.unlink()
except IsADirectoryError as e:
    print(f'Error: {data_file} : {e.strerror}')

This creates a Path object called data_file that points to a file. Calling .remove() on data_file will delete home/data.txt. If data_file points to a directory, an IsADirectoryError is raised. It is worth noting that the Python program above has the same permissions as the user running it. If the user does not have permission to delete the file, a PermissionError is raised.

Deleting Directories

The standard library offers the following functions for deleting directories:

os.rmdir()
pathlib.Path.rmdir()
shutil.rmtree()

To delete a single directory or folder, use os.rmdir() or pathlib.rmdir(). These two functions only work if the directory you’re trying to delete is empty. If the directory isn’t empty, an OSError is raised. Here is how to delete a folder:

import os

trash_dir = 'my_documents/bad_dir'

try:
    os.rmdir(trash_dir)
except OSError as e:
    print(f'Error: {trash_dir} : {e.strerror}')

Here, the trash_dir directory is deleted by passing its path to os.rmdir(). If the directory isn’t empty, an error message is printed to the screen:

>>>

Traceback (most recent call last):
  File '<stdin>', line 1, in <module>
OSError: [Errno 39] Directory not empty: 'my_documents/bad_dir'

Alternatively, you can use pathlib to delete directories:

from pathlib import Path

trash_dir = Path('my_documents/bad_dir')

try:
    trash_dir.rmdir()
except OSError as e:
    print(f'Error: {trash_dir} : {e.strerror}')

Here, you create a Path object that points to the directory to be deleted. Calling .rmdir() on the Path object will delete it if it is empty.

Deleting Entire Directory Trees

To delete non-empty directories and entire directory trees, Python offers shutil.rmtree():

import shutil

trash_dir = 'my_documents/bad_dir'

try:
    shutil.rmtree(trash_dir)
except OSError as e:
    print(f'Error: {trash_dir} : {e.strerror}')

Everything in trash_dir is deleted when shutil.rmtree() is called on it. There may be cases where you want to delete empty folders recursively. You can do this using one of the methods discussed above in conjunction with os.walk():

import os

for dirpath, dirnames, files in os.walk('.', topdown=False):
    try:
        os.rmdir(dirpath)
    except OSError as ex:
        pass

This walks down the directory tree and tries to delete each directory it finds. If the directory isn’t empty, an OSError is raised and that directory is skipped. The table below lists the functions covered in this section:

Function	Description
`os.remove()`	Deletes a file and does not delete directories
`os.unlink()`	Is identical to `os.remove()` and deletes a single file
`pathlib.Path.unlink()`	Deletes a file and cannot delete directories
`os.rmdir()`	Deletes an empty directory
`pathlib.Path.rmdir()`	Deletes an empty directory
`shutil.rmtree()`	Deletes entire directory tree and can be used to delete non-empty directories

Copying, Moving, and Renaming Files and Directories

Python ships with the shutil module. shutil is short for shell utilities. It provides a number of high-level operations on files to support copying, archiving, and removal of files and directories. In this section, you’ll learn how to move and copy files and directories.

Copying Files in Python

shutil offers a couple of functions for copying files. The most commonly used functions are shutil.copy() and shutil.copy2(). To copy a file from one location to another using shutil.copy(), do the following:

import shutil

src = 'path/to/file.txt'
dst = 'path/to/dest_dir'
shutil.copy(src, dst)

shutil.copy() is comparable to the cp command in UNIX based systems. shutil.copy(src, dst) will copy the file src to the location specified in dst. If dst is a file, the contents of that file are replaced with the contents of src. If dst is a directory, then src will be copied into that directory. shutil.copy() only copies the file’s contents and the file’s permissions. Other metadata like the file’s creation and modification times are not preserved.

To preserve all file metadata when copying, use shutil.copy2():

import shutil

src = 'path/to/file.txt'
dst = 'path/to/dest_dir'
shutil.copy2(src, dst)

Using .copy2() preserves details about the file such as last access time, permission bits, last modification time, and flags.

Copying Directories

While shutil.copy() only copies a single file, shutil.copytree() will copy an entire directory and everything contained in it. shutil.copytree(src, dest) takes two arguments: a source directory and the destination directory where files and folders will be copied to.

Here’s an example of how to copy the contents of one folder to a different location:

>>>

>>> import shutil
>>> shutil.copytree('data_1', 'data1_backup')
'data1_backup'

In this example, .copytree() copies the contents of data_1 to a new location data1_backup and returns the destination directory. The destination directory must not already exist. It will be created as well as missing parent directories. shutil.copytree() is a good way to back up your files.

Moving Files and Directories

To move a file or directory to another location, use shutil.move(src, dst).

src is the file or directory to be moved and dst is the destination:

>>>

>>> import shutil
>>> shutil.move('dir_1/', 'backup/')
'backup'

shutil.move('dir_1/', 'backup/') moves dir_1/ into backup/ if backup/ exists. If backup/ does not exist, dir_1/ will be renamed to backup.

Renaming Files and Directories

Python includes os.rename(src, dst) for renaming files and directories:

>>>

>>> os.rename('first.zip', 'first_01.zip')

The line above will rename first.zip to first_01.zip. If the destination path points to a directory, it will raise an OSError.

Another way to rename files or directories is to use rename() from the pathlib module:

>>>

>>> from pathlib import Path
>>> data_file = Path('data_01.txt')
>>> data_file.rename('data.txt')

To rename files using pathlib, you first create a pathlib.Path() object that contains a path to the file you want to replace. The next step is to call rename() on the path object and pass a new filename for the file or directory you’re renaming.

Archiving

Archives are a convenient way to package several files into one. The two most common archive types are ZIP and TAR. The Python programs you write can create, read, and extract data from archives. You will learn how to read and write to both archive formats in this section.

Reading ZIP Files

The zipfile module is a low level module that is part of the Python Standard Library. zipfile has functions that make it easy to open and extract ZIP files. To read the contents of a ZIP file, the first thing to do is to create a ZipFile object. ZipFile objects are similar to file objects created using open(). ZipFile is also a context manager and therefore supports the with statement:

import zipfile

with zipfile.ZipFile('data.zip', 'r') as zipobj:

Here, you create a ZipFile object, passing in the name of the ZIP file to open in read mode. After opening a ZIP file, information about the archive can be accessed through functions provided by the zipfile module. The data.zip archive in the example above was created from a directory named data that contains a total of 5 files and 1 subdirectory:

.
|
├── sub_dir/
|   ├── bar.py
|   └── foo.py
|
├── file1.py
├── file2.py
└── file3.py

To get a list of files in the archive, call namelist() on the ZipFile object:

import zipfile

with zipfile.ZipFile('data.zip', 'r') as zipobj:
    zipobj.namelist()

This produces a list:

['file1.py', 'file2.py', 'file3.py', 'sub_dir/', 'sub_dir/bar.py', 'sub_dir/foo.py']

.namelist() returns a list of names of the files and directories in the archive. To retrieve information about the files in the archive, use .getinfo():

import zipfile

with zipfile.ZipFile('data.zip', 'r') as zipobj:
    bar_info = zipobj.getinfo('sub_dir/bar.py')
    bar_info.file_size

Here’s the output:

.getinfo() returns a ZipInfo object that stores information about a single member of the archive. To get information about a file in the archive, you pass its path as an argument to .getinfo(). Using getinfo(), you’re able to retrieve information about archive members such as the date the files were last modified, their compressed sizes, and their full filenames. Accessing .file_size retrieves the file’s original size in bytes.

The following example shows how to retrieve more details about archived files in a Python REPL. Assume that the zipfile module has been imported and bar_info is the same object you created in previous examples:

>>>

>>> bar_info.date_time
(2018, 10, 7, 23, 30, 10)
>>> bar_info.compress_size
2856
>>> bar_info.filename
'sub_dir/bar.py'

bar_info contains details about bar.py such as its size when compressed and its full path.

The first line shows how to retrieve a file’s last modified date. The next line shows how to get the size of the file after compression. The last line shows the full path of bar.py in the archive.

ZipFile supports the context manager protocol, which is why you’re able to use it with the with statement. Doing this automatically closes the ZipFile object after you’re done with it. Trying to open or extract files from a closed ZipFile object will result in an error.

Extracting ZIP Archives

The zipfile module allows you to extract one or more files from ZIP archives through .extract() and .extractall().

These methods extract files to the current directory by default. They both take an optional path parameter that allows you to specify a different directory to extract files to. If the directory does not exist, it is automatically created. To extract files from the archive, do the following:

>>>

>>> import zipfile
>>> import os

>>> os.listdir('.')
['data.zip']

>>> data_zip = zipfile.ZipFile('data.zip', 'r')

>>> # Extract a single file to current directory
>>> data_zip.extract('file1.py')
'/home/terra/test/dir1/zip_extract/file1.py'

>>> os.listdir('.')
['file1.py', 'data.zip']

>>> # Extract all files into a different directory
>>> data_zip.extractall(path='extract_dir/')

>>> os.listdir('.')
['file1.py', 'extract_dir', 'data.zip']

>>> os.listdir('extract_dir')
['file1.py', 'file3.py', 'file2.py', 'sub_dir']

>>> data_zip.close()

The third line of code is a call to os.listdir(), which shows that the current directory has only one file, data.zip.

Next, you open data.zip in read mode and call .extract() to extract file1.py from it. .extract() returns the full file path of the extracted file. Since there’s no path specified, .extract() extracts file1.py to the current directory.

The next line prints a directory listing showing that the current directory now includes the extracted file in addition to the original archive. The line after that shows how to extract the entire archive into the zip_extract directory. .extractall() creates the extract_dir and extracts the contents of data.zip into it. The last line closes the ZIP archive.

Extracting Data From Password Protected Archives

zipfile supports extracting password protected ZIPs. To extract password protected ZIP files, pass in the password to the .extract() or .extractall() method as an argument:

>>>

>>> import zipfile

>>> with zipfile.ZipFile('secret.zip', 'r') as pwd_zip:
...     # Extract from a password protected archive
...     pwd_zip.extractall(path='extract_dir', pwd='Quish3@o')

This opens the secret.zip archive in read mode. A password is supplied to .extractall(), and the archive contents are extracted to extract_dir. The archive is closed automatically after the extraction is complete thanks to the with statement.

Creating New ZIP Archives

To create a new ZIP archive, you open a ZipFile object in write mode (w) and add the files you want to archive:

>>>

>>> import zipfile

>>> file_list = ['file1.py', 'sub_dir/', 'sub_dir/bar.py', 'sub_dir/foo.py']
>>> with zipfile.ZipFile('new.zip', 'w') as new_zip:
...     for name in file_list:
...         new_zip.write(name)

In the example, new_zip is opened in write mode and each file in file_list is added to the archive. When the with statement suite is finished, new_zip is closed. Opening a ZIP file in write mode erases the contents of the archive and creates a new archive.

To add files to an existing archive, open a ZipFile object in append mode and then add the files:

>>>

>>> # Open a ZipFile object in append mode
>>> with zipfile.ZipFile('new.zip', 'a') as new_zip:
...     new_zip.write('data.txt')
...     new_zip.write('latin.txt')

Here, you open the new.zip archive you created in the previous example in append mode. Opening the ZipFile object in append mode allows you to add new files to the ZIP file without deleting its current contents. After adding files to the ZIP file, the with statement goes out of context and closes the ZIP file.

Opening TAR Archives

TAR files are uncompressed file archives like ZIP. They can be compressed using gzip, bzip2, and lzma compression methods. The TarFile class allows reading and writing of TAR archives.

Do this to read from an archive:

import tarfile

with tarfile.open('example.tar', 'r') as tar_file:
    print(tar_file.getnames())

tarfile objects open like most file-like objects. They have an open() function that takes a mode that determines how the file is to be opened.

Use the 'r', 'w' or 'a' modes to open an uncompressed TAR file for reading, writing, and appending, respectively. To open compressed TAR files, pass in a mode argument to tarfile.open() that is in the form filemode[:compression]. The table below lists the possible modes TAR files can be opened in:

Mode	Action
`r`	Opens archive for reading with transparent compression
`r:gz`	Opens archive for reading with gzip compression
`r:bz2`	Opens archive for reading with bzip2 compression
`r:xz`	Opens archive for reading with lzma compression
`w`	Opens archive for uncompressed writing
`w:gz`	Opens archive for gzip compressed writing
`w:xz`	Opens archive for lzma compressed writing
`a`	Opens archive for appending with no compression

.open() defaults to 'r' mode. To read an uncompressed TAR file and retrieve the names of the files in it, use .getnames():

>>>

>>> import tarfile

>>> tar = tarfile.open('example.tar', mode='r')
>>> tar.getnames()
['CONTRIBUTING.rst', 'README.md', 'app.py']

This returns a list with the names of the archive contents.

Note: For the purposes of showing you how to use different tarfile object methods, the TAR file in the examples is opened and closed manually in an interactive REPL session.

Interacting with the TAR file this way allows you to see the output of running each command. Normally, you would want to use a context manager to open file-like objects.

The metadata of each entry in the archive can be accessed using special attributes:

>>>

>>> for entry in tar.getmembers():
...     print(entry.name)
...     print(' Modified:', time.ctime(entry.mtime))
...     print(' Size    :', entry.size, 'bytes')
...     print()
CONTRIBUTING.rst
 Modified: Sat Nov  1 09:09:51 2018
 Size    : 402 bytes

README.md
 Modified: Sat Nov  3 07:29:40 2018
 Size    : 5426 bytes

app.py
 Modified: Sat Nov  3 07:29:13 2018
 Size    : 6218 bytes

In this example, you loop through the list of files returned by .getmembers() and print out each file’s attributes. The objects returned by .getmembers() have attributes that can be accessed programmatically such as the name, size, and last modified time of each of the files in the archive. After reading or writing to the archive, it must be closed to free up system resources.

Extracting Files From a TAR Archive

In this section, you’ll learn how to extract files from TAR archives using the following methods:

.extract()
.extractfile()
.extractall()

To extract a single file from a TAR archive, use extract(), passing in the filename:

>>>

>>> tar.extract('README.md')
>>> os.listdir('.')
['README.md', 'example.tar']

The README.md file is extracted from the archive to the file system. Calling os.listdir() confirms that README.md file was successfully extracted into the current directory. To unpack or extract everything from the archive, use .extractall():

>>>

>>> tar.extractall(path="extracted/")

.extractall() has an optional path argument to specify where extracted files should go. Here, the archive is unpacked into the extracted directory. The following commands show that the archive was successfully extracted:

$ ls
example.tar  extracted  README.md

$ tree
.
├── example.tar
├── extracted
|   ├── app.py
|   ├── CONTRIBUTING.rst
|   └── README.md
└── README.md

1 directory, 5 files

$ ls extracted/
app.py  CONTRIBUTING.rst  README.md

To extract a file object for reading or writing, use .extractfile(), which takes a filename or TarInfo object to extract as an argument. .extractfile() returns a file-like object that can be read and used:

>>>

>>> f = tar.extractfile('app.py')
>>> f.read()
>>> tar.close()

Opened archives should always be closed after they have been read or written to. To close an archive, call .close() on the archive file handle or use the with statement when creating tarfile objects to automatically close the archive when you’re done. This frees up system resources and writes any changes you made to the archive to the filesystem.

Creating New TAR Archives

Here’s how you do it:

>>>

>>> import tarfile

>>> file_list = ['app.py', 'config.py', 'CONTRIBUTORS.md', 'tests.py']
>>> with tarfile.open('packages.tar', mode='w') as tar:
...     for file in file_list:
...         tar.add(file)

>>> # Read the contents of the newly created archive
>>> with tarfile.open('package.tar', mode='r') as t:
...     for member in t.getmembers():
...         print(member.name)
app.py
config.py
CONTRIBUTORS.md
tests.py

First, you make a list of files to be added to the archive so that you don’t have to add each file manually.

The next line uses the with context manager to open a new archive called packages.tar in write mode. Opening an archive in write mode('w') enables you to write new files to the archive. Any existing files in the archive are deleted and a new archive is created.

After the archive is created and populated, the with context manager automatically closes it and saves it to the filesystem. The last three lines open the archive you just created and print out the names of the files contained in it.

To add new files to an existing archive, open the archive in append mode ('a'):

>>>

>>> with tarfile.open('package.tar', mode='a') as tar:
...     tar.add('foo.bar')

>>> with tarfile.open('package.tar', mode='r') as tar:
...     for member in tar.getmembers():
...         print(member.name)
app.py
config.py
CONTRIBUTORS.md
tests.py
foo.bar

Opening an archive in append mode allows you to add new files to it without deleting the ones already in it.

Working With Compressed Archives

tarfile can also read and write TAR archives compressed using gzip, bzip2, and lzma compression. To read or write to a compressed archive, use tarfile.open(), passing in the appropriate mode for the compression type.

For example, to read or write data to a TAR archive compressed using gzip, use the 'r:gz' or 'w:gz' modes respectively:

>>>

>>> files = ['app.py', 'config.py', 'tests.py']
>>> with tarfile.open('packages.tar.gz', mode='w:gz') as tar:
...     tar.add('app.py')
...     tar.add('config.py')
...     tar.add('tests.py')

>>> with tarfile.open('packages.tar.gz', mode='r:gz') as t:
...     for member in t.getmembers():
...         print(member.name)
app.py
config.py
tests.py

The 'w:gz' mode opens the archive for gzip compressed writing and 'r:gz' opens the archive for gzip compressed reading. Opening compressed archives in append mode is not possible. To add files to a compressed archive, you have to create a new archive.

An Easier Way of Creating Archives

The Python Standard Library also supports creating TAR and ZIP archives using the high-level methods in the shutil module. The archiving utilities in shutil allow you to create, read, and extract ZIP and TAR archives. These utilities rely on the lower level tarfile and zipfile modules.

Working With Archives Using shutil.make_archive()

shutil.make_archive() takes at least two arguments: the name of the archive and an archive format.

By default, it compresses all the files in the current directory into the archive format specified in the format argument. You can pass in an optional root_dir argument to compress files in a different directory. .make_archive() supports the zip, tar, bztar, and gztar archive formats.

This is how to create a TAR archive using shutil:

import shutil

# shutil.make_archive(base_name, format, root_dir)
shutil.make_archive('data/backup', 'tar', 'data/')

This copies everything in data/ and creates an archive called backup.tar in the filesystem and returns its name. To extract the archive, call .unpack_archive():

shutil.unpack_archive('backup.tar', 'extract_dir/')

Calling .unpack_archive() and passing in an archive name and destination directory extracts the contents of backup.tar into extract_dir/. ZIP archives can be created and extracted in the same way.

Reading Multiple Files

Python supports reading data from multiple input streams or from a list of files through the fileinput module. This module allows you to loop over the contents of one or more text files quickly and easily. Here’s the typical way fileinput is used:

import fileinput
for line in fileinput.input()
    process(line)

fileinput gets its input from command line arguments passed to sys.argv by default.

Using fileinput to Loop Over Multiple Files

Let’s use fileinput to build a crude version of the common UNIX utility cat. The cat utility reads files sequentially, writing them to standard output. When given more than one file in its command line arguments, cat will concatenate the text files and display the result in the terminal:

# File: fileinput-example.py
import fileinput
import sys

files = fileinput.input()
for line in files:
    if fileinput.isfirstline():
        print(f'\n--- Reading {fileinput.filename()} ---')
    print(' -> ' + line, end='')
print()

Running this on two text files in my current directory produces the following output:

$ python3 fileinput-example.py bacon.txt cupcake.txt
--- Reading bacon.txt ---
 -> Spicy jalapeno bacon ipsum dolor amet in in aute est qui enim aliquip,
 -> irure cillum drumstick elit.
 -> Doner jowl shank ea exercitation landjaeger incididunt ut porchetta.
 -> Tenderloin bacon aliquip cupidatat chicken chuck quis anim et swine.
 -> Tri-tip doner kevin cillum ham veniam cow hamburger.
 -> Turkey pork loin cupidatat filet mignon capicola brisket cupim ad in.
 -> Ball tip dolor do magna laboris nisi pancetta nostrud doner.

--- Reading cupcake.txt ---
 -> Cupcake ipsum dolor sit amet candy I love cheesecake fruitcake.
 -> Topping muffin cotton candy.
 -> Gummies macaroon jujubes jelly beans marzipan.

fileinput allows you to retrieve more information about each line such as whether or not it is the first line (.isfirstline()), the line number (.lineno()), and the filename (.filename()). You can read more about it here.

Conclusion

You now know how to use Python to perform the most common operations on files and groups of files. You’ve learned about the different built-in modules used to read, find, and manipulate them.

You’re now equipped to use Python to:

Get directory contents and file properties
Create directories and directory trees
Find patterns in filenames
Create temporary files and directories
Move, rename, copy, and delete files or directories
Read and extract data from different types of archives
Read multiple files simultaneously using fileinput

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Async IO in Python: A Complete Walkthrough

2019-01-16T14:00:00+00:00

Async IO is a concurrent programming design that has received dedicated support in Python, evolving rapidly from Python 3.4 through 3.7, and probably beyond.

You may be thinking with dread, “Concurrency, parallelism, threading, multiprocessing. That’s a lot to grasp already. Where does async IO fit in?”

This tutorial is built to help you answer that question, giving you a firmer grasp of Python’s approach to async IO.

Here’s what you’ll cover:

Asynchronous IO (async IO): a language-agnostic paradigm (model) that has implementations across a host of programming languages
async/await: two new Python keywords that are used to define coroutines
asyncio: the Python package that provides a foundation and API for running and managing coroutines

Coroutines (specialized generator functions) are the heart of async IO in Python, and we’ll dive into them later on.

Note: In this article, I use the term async IO to denote the language-agnostic design of asynchronous IO, while asyncio refers to the Python package.

Before you get started, you’ll need to make sure you’re set up to use asyncio and other libraries found in this tutorial.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Setting Up Your Environment

You’ll need Python 3.7 or above to follow this article in its entirety, as well as the aiohttp and aiofiles packages:

$ python3.7 -m venv ./py37async
$ source ./py37async/bin/activate  # Windows: .\py37async\Scripts\activate.bat
$ pip install --upgrade pip aiohttp aiofiles  # Optional: aiodns

For help with installing Python 3.7 and setting up a virtual environment, check out Python 3 Installation & Setup Guide or Virtual Environments Primer.

With that, let’s jump in.

The 10,000-Foot View of Async IO

Async IO is a bit lesser known than its tried-and-true cousins, multiprocessing and threading. This section will give you a fuller picture of what async IO is and how it fits into its surrounding landscape.

Where Does Async IO Fit In?

Concurrency and parallelism are expansive subjects that are not easy to wade into. While this article focuses on async IO and its implementation in Python, it’s worth taking a minute to compare async IO to its counterparts in order to have context about how async IO fits into the larger, sometimes dizzying puzzle.

Parallelism consists of performing multiple operations at the same time. Multiprocessing is a means to effect parallelism, and it entails spreading tasks over a computer’s central processing units (CPUs, or cores). Multiprocessing is well-suited for CPU-bound tasks: tightly bound for loops and mathematical computations usually fall into this category.

Concurrency is a slightly broader term than parallelism. It suggests that multiple tasks have the ability to run in an overlapping manner. (There’s a saying that concurrency does not imply parallelism.)

Threading is a concurrent execution model whereby multiple threads take turns executing tasks. One process can contain multiple threads. Python has a complicated relationship with threading thanks to its GIL, but that’s beyond the scope of this article.

What’s important to know about threading is that it’s better for IO-bound tasks. While a CPU-bound task is characterized by the computer’s cores continually working hard from start to finish, an IO-bound job is dominated by a lot of waiting on input/output to complete.

To recap the above, concurrency encompasses both multiprocessing (ideal for CPU-bound tasks) and threading (suited for IO-bound tasks). Multiprocessing is a form of parallelism, with parallelism being a specific type (subset) of concurrency. The Python standard library has offered longstanding support for both of these through its multiprocessing, threading, and concurrent.futures packages.

Now it’s time to bring a new member to the mix. Over the last few years, a separate design has been more comprehensively built into CPython: asynchronous IO, enabled through the standard library’s asyncio package and the new async and await language keywords. To be clear, async IO is not a newly invented concept, and it has existed or is being built into other languages and runtime environments, such as Go, C#, or Scala.

The asyncio package is billed by the Python documentation as a library to write concurrent code. However, async IO is not threading, nor is it multiprocessing. It is not built on top of either of these.

In fact, async IO is a single-threaded, single-process design: it uses cooperative multitasking, a term that you’ll flesh out by the end of this tutorial. It has been said in other words that async IO gives a feeling of concurrency despite using a single thread in a single process. Coroutines (a central feature of async IO) can be scheduled concurrently, but they are not inherently concurrent.

To reiterate, async IO is a style of concurrent programming, but it is not parallelism. It’s more closely aligned with threading than with multiprocessing but is very much distinct from both of these and is a standalone member in concurrency’s bag of tricks.

That leaves one more term. What does it mean for something to be asynchronous? This isn’t a rigorous definition, but for our purposes here, I can think of two properties:

Asynchronous routines are able to “pause” while waiting on their ultimate result and let other routines run in the meantime.
Asynchronous code, through the mechanism above, facilitates concurrent execution. To put it differently, asynchronous code gives the look and feel of concurrency.

Here’s a diagram to put it all together. The white terms represent concepts, and the green terms represent ways in which they are implemented or effected:

I’ll stop there on the comparisons between concurrent programming models. This tutorial is focused on the subcomponent that is async IO, how to use it, and the APIs that have sprung up around it. For a thorough exploration of threading versus multiprocessing versus async IO, pause here and check out Jim Anderson’s overview of concurrency in Python. Jim is way funnier than me and has sat in more meetings than me, to boot.

Async IO Explained

Async IO may at first seem counterintuitive and paradoxical. How does something that facilitates concurrent code use a single thread and a single CPU core? I’ve never been very good at conjuring up examples, so I’d like to paraphrase one from Miguel Grinberg’s 2017 PyCon talk, which explains everything quite beautifully:

Chess master Judit Polgár hosts a chess exhibition in which she plays multiple amateur players. She has two ways of conducting the exhibition: synchronously and asynchronously.

Assumptions:

24 opponents

Judit makes each chess move in 5 seconds

Opponents each take 55 seconds to make a move

Games average 30 pair-moves (60 moves total)

Synchronous version: Judit plays one game at a time, never two at the same time, until the game is complete. Each game takes (55 + 5) * 30 == 1800 seconds, or 30 minutes. The entire exhibition takes 24 * 30 == 720 minutes, or 12 hours.

Asynchronous version: Judit moves from table to table, making one move at each table. She leaves the table and lets the opponent make their next move during the wait time. One move on all 24 games takes Judit 24 * 5 == 120 seconds, or 2 minutes. The entire exhibition is now cut down to 120 * 30 == 3600 seconds, or just 1 hour. (Source)

There is only one Judit Polgár, who has only two hands and makes only one move at a time by herself. But playing asynchronously cuts the exhibition time down from 12 hours to one. So, cooperative multitasking is a fancy way of saying that a program’s event loop (more on that later) communicates with multiple tasks to let each take turns running at the optimal time.

Async IO takes long waiting periods in which functions would otherwise be blocking and allows other functions to run during that downtime. (A function that blocks effectively forbids others from running from the time that it starts until the time that it returns.)

Async IO Is Not Easy

I’ve heard it said, “Use async IO when you can; use threading when you must.” The truth is that building durable multithreaded code can be hard and error-prone. Async IO avoids some of the potential speedbumps that you might otherwise encounter with a threaded design.

But that’s not to say that async IO in Python is easy. Be warned: when you venture a bit below the surface level, async programming can be difficult too! Python’s async model is built around concepts such as callbacks, events, transports, protocols, and futures—just the terminology can be intimidating. The fact that its API has been changing continually makes it no easier.

Luckily, asyncio has matured to a point where most of its features are no longer provisional, while its documentation has received a huge overhaul and some quality resources on the subject are starting to emerge as well.

The `asyncio` Package and `async`/`await`

Now that you have some background on async IO as a design, let’s explore Python’s implementation. Python’s asyncio package (introduced in Python 3.4) and its two keywords, async and await, serve different purposes but come together to help you declare, build, execute, and manage asynchronous code.

The `async`/`await` Syntax and Native Coroutines

A Word of Caution: Be careful what you read out there on the Internet. Python’s async IO API has evolved rapidly from Python 3.4 to Python 3.7. Some old patterns are no longer used, and some things that were at first disallowed are now allowed through new introductions. For all I know, this tutorial will join the club of the outdated soon too.

At the heart of async IO are coroutines. A coroutine is a specialized version of a Python generator function. Let’s start with a baseline definition and then build off of it as you progress here: a coroutine is a function that can suspend its execution before reaching return, and it can indirectly pass control to another coroutine for some time.

Later, you’ll dive a lot deeper into how exactly the traditional generator is repurposed into a coroutine. For now, the easiest way to pick up how coroutines work is to start making some.

Let’s take the immersive approach and write some async IO code. This short program is the Hello World of async IO but goes a long way towards illustrating its core functionality:

#!/usr/bin/env python3
# countasync.py

import asyncio

async def count():
    print("One")
    await asyncio.sleep(1)
    print("Two")

async def main():
    await asyncio.gather(count(), count(), count())

if __name__ == "__main__":
    import time
    s = time.perf_counter()
    asyncio.run(main())
    elapsed = time.perf_counter() - s
    print(f"{__file__} executed in {elapsed:0.2f} seconds.")

When you execute this file, take note of what looks different than if you were to define the functions with just def and time.sleep():

$ python3 countasync.py
One
One
One
Two
Two
Two
countasync.py executed in 1.01 seconds.

The order of this output is the heart of async IO. Talking to each of the calls to count() is a single event loop, or coordinator. When each task reaches await asyncio.sleep(1), the function yells up to the event loop and gives control back to it, saying, “I’m going to be sleeping for 1 second. Go ahead and let something else meaningful be done in the meantime.”

Contrast this to the synchronous version:

#!/usr/bin/env python3
# countsync.py

import time

def count():
    print("One")
    time.sleep(1)
    print("Two")

def main():
    for _ in range(3):
        count()

if __name__ == "__main__":
    s = time.perf_counter()
    main()
    elapsed = time.perf_counter() - s
    print(f"{__file__} executed in {elapsed:0.2f} seconds.")

When executed, there is a slight but critical change in order and execution time:

$ python3 countsync.py
One
Two
One
Two
One
Two
countsync.py executed in 3.01 seconds.

While using time.sleep() and asyncio.sleep() may seem banal, they are used as stand-ins for any time-intensive processes that involve wait time. (The most mundane thing you can wait on is a sleep() call that does basically nothing.) That is, time.sleep() can represent any time-consuming blocking function call, while asyncio.sleep() is used to stand in for a non-blocking call (but one that also takes some time to complete).

As you’ll see in the next section, the benefit of awaiting something, including asyncio.sleep(), is that the surrounding function can temporarily cede control to another function that’s more readily able to do something immediately. In contrast, time.sleep() or any other blocking call is incompatible with asynchronous Python code, because it will stop everything in its tracks for the duration of the sleep time.

The Rules of Async IO

At this point, a more formal definition of async, await, and the coroutine functions that they create are in order. This section is a little dense, but getting a hold of async/await is instrumental, so come back to this if you need to:

The syntax async def introduces either a native coroutine or an asynchronous generator. The expressions async with and async for are also valid, and you’ll see them later on.
The keyword await passes function control back to the event loop. (It suspends the execution of the surrounding coroutine.) If Python encounters an await f() expression in the scope of g(), this is how await tells the event loop, “Suspend execution of g() until whatever I’m waiting on—the result of f()—is returned. In the meantime, go let something else run.”

In code, that second bullet point looks roughly like this:

async def g():
    # Pause here and come back to g() when f() is ready
    r = await f()
    return r

There’s also a strict set of rules around when and how you can and cannot use async/await. These can be handy whether you are still picking up the syntax or already have exposure to using async/await:

A function that you introduce with async def is a coroutine. It may use await, return, or yield, but all of these are optional. Declaring async def noop(): pass is valid:
- Using await and/or return creates a coroutine function. To call a coroutine function, you must await it to get its results.
- It is less common (and only recently legal in Python) to use yield in an async def block. This creates an asynchronous generator, which you iterate over with async for. Forget about async generators for the time being and focus on getting down the syntax for coroutine functions, which use await and/or return.
- Anything defined with async def may not use yield from, which will raise a SyntaxError.
Just like it’s a SyntaxError to use yield outside of a def function, it is a SyntaxError to use await outside of an async def coroutine. You can only use await in the body of coroutines.

Here are some terse examples meant to summarize the above few rules:

async def f(x):
    y = await z(x)  # OK - `await` and `return` allowed in coroutines
    return y

async def g(x):
    yield x  # OK - this is an async generator

async def m(x):
    yield from gen(x)  # No - SyntaxError

def m(x):
    y = await z(x)  # Still no - SyntaxError (no `async def` here)
    return y

Finally, when you use await f(), it’s required that f() be an object that is awaitable. Well, that’s not very helpful, is it? For now, just know that an awaitable object is either (1) another coroutine or (2) an object defining an .__await__() dunder method that returns an iterator. If you’re writing a program, for the large majority of purposes, you should only need to worry about case #1.

That brings us to one more technical distinction that you may see pop up: an older way of marking a function as a coroutine is to decorate a normal def function with @asyncio.coroutine. The result is a generator-based coroutine. This construction has been outdated since the async/await syntax was put in place in Python 3.5.

These two coroutines are essentially equivalent (both are awaitable), but the first is generator-based, while the second is a native coroutine:

import asyncio

@asyncio.coroutine
def py34_coro():
    """Generator-based coroutine, older syntax"""
    yield from stuff()

async def py35_coro():
    """Native coroutine, modern syntax"""
    await stuff()

If you’re writing any code yourself, prefer native coroutines for the sake of being explicit rather than implicit. Generator-based coroutines will be removed in Python 3.10.

Towards the latter half of this tutorial, we’ll touch on generator-based coroutines for explanation’s sake only. The reason that async/await were introduced is to make coroutines a standalone feature of Python that can be easily differentiated from a normal generator function, thus reducing ambiguity.

Don’t get bogged down in generator-based coroutines, which have been deliberately outdated by async/await. They have their own small set of rules (for instance, await cannot be used in a generator-based coroutine) that are largely irrelevant if you stick to the async/await syntax.

Without further ado, let’s take on a few more involved examples.

Here’s one example of how async IO cuts down on wait time: given a coroutine makerandom() that keeps producing random integers in the range [0, 10], until one of them exceeds a threshold, you want to let multiple calls of this coroutine not need to wait for each other to complete in succession. You can largely follow the patterns from the two scripts above, with slight changes:

#!/usr/bin/env python3
# rand.py

import asyncio
import random

# ANSI colors
c = (
    "\033[0m",   # End of color
    "\033[36m",  # Cyan
    "\033[91m",  # Red
    "\033[35m",  # Magenta
)

async def randint(a: int, b: int) -> int:
    return random.randint(a, b)

async def makerandom(idx: int, threshold: int = 6) -> int:
    print(c[idx + 1] + f"Initiated makerandom({idx}).")
    i = await randint(0, 10)
    while i <= threshold:
        print(c[idx + 1] + f"makerandom({idx}) == {i} too low; retrying.")
        await asyncio.sleep(idx + 1)
        i = await randint(0, 10)
    print(c[idx + 1] + f"---> Finished: makerandom({idx}) == {i}" + c[0])
    return i

async def main():
    res = await asyncio.gather(*(makerandom(i, 10 - i - 1) for i in range(3)))
    return res

if __name__ == "__main__":
    random.seed(444)
    r1, r2, r3 = asyncio.run(main())
    print()
    print(f"r1: {r1}, r2: {r2}, r3: {r3}")

The colorized output says a lot more than I can and gives you a sense for how this script is carried out:

rand.py execution

This program uses one main coroutine, makerandom(), and runs it concurrently across 3 different inputs. Most programs will contain small, modular coroutines and one wrapper function that serves to chain each of the smaller coroutines together. main() is then used to gather tasks (futures) by mapping the central coroutine across some iterable or pool.

In this miniature example, the pool is range(3). In a fuller example presented later, it is a set of URLs that need to be requested, parsed, and processed concurrently, and main() encapsulates that entire routine for each URL.

While “making random integers” (which is CPU-bound more than anything) is maybe not the greatest choice as a candidate for asyncio, it’s the presence of asyncio.sleep() in the example that is designed to mimic an IO-bound process where there is uncertain wait time involved. For example, the asyncio.sleep() call might represent sending and receiving not-so-random integers between two clients in a message application.

Async IO Design Patterns

Async IO comes with its own set of possible script designs, which you’ll get introduced to in this section.

Chaining Coroutines

A key feature of coroutines is that they can be chained together. (Remember, a coroutine object is awaitable, so another coroutine can await it.) This allows you to break programs into smaller, manageable, recyclable coroutine:

#!/usr/bin/env python3
# chained.py

import asyncio
import random
import time

async def randint(a: int, b: int) -> int:
    return random.randint(a, b)

async def part1(n: int) -> str:
    i = await randint(0, 10)
    print(f"part1({n}) sleeping for {i} seconds.")
    await asyncio.sleep(i)
    result = f"result{n}-1"
    print(f"Returning part1({n}) == {result}.")
    return result

async def part2(n: int, arg: str) -> str:
    i = await randint(0, 10)
    print(f"part2{n, arg} sleeping for {i} seconds.")
    await asyncio.sleep(i)
    result = f"result{n}-2 derived from {arg}"
    print(f"Returning part2{n, arg} == {result}.")
    return result

async def chain(n: int) -> None:
    start = time.perf_counter()
    p1 = await part1(n)
    p2 = await part2(n, p1)
    end = time.perf_counter() - start
    print(f"-->Chained result{n} => {p2} (took {end:0.2f} seconds).")

async def main(*args):
    await asyncio.gather(*(chain(n) for n in args))

if __name__ == "__main__":
    import sys
    random.seed(444)
    args = [1, 2, 3] if len(sys.argv) == 1 else map(int, sys.argv[1:])
    start = time.perf_counter()
    asyncio.run(main(*args))
    end = time.perf_counter() - start
    print(f"Program finished in {end:0.2f} seconds.")

Pay careful attention to the output, where part1() sleeps for a variable amount of time, and part2() begins working with the results as they become available:

$ python3 chained.py 9 6 3
part1(9) sleeping for 4 seconds.
part1(6) sleeping for 4 seconds.
part1(3) sleeping for 0 seconds.
Returning part1(3) == result3-1.
part2(3, 'result3-1') sleeping for 4 seconds.
Returning part1(9) == result9-1.
part2(9, 'result9-1') sleeping for 7 seconds.
Returning part1(6) == result6-1.
part2(6, 'result6-1') sleeping for 4 seconds.
Returning part2(3, 'result3-1') == result3-2 derived from result3-1.
-->Chained result3 => result3-2 derived from result3-1 (took 4.00 seconds).
Returning part2(6, 'result6-1') == result6-2 derived from result6-1.
-->Chained result6 => result6-2 derived from result6-1 (took 8.01 seconds).
Returning part2(9, 'result9-1') == result9-2 derived from result9-1.
-->Chained result9 => result9-2 derived from result9-1 (took 11.01 seconds).
Program finished in 11.01 seconds.

In this setup, the runtime of main() will be equal to the maximum runtime of the tasks that it gathers together and schedules.

Using a Queue

The asyncio package provides queue classes that are designed to be similar to classes of the queue module. In our examples so far, we haven’t really had a need for a queue structure. In chained.py, each task (future) is composed of a set of coroutines that explicitly await each other and pass through a single input per chain.

There is an alternative structure that can also work with async IO: a number of producers, which are not associated with each other, add items to a queue. Each producer may add multiple items to the queue at staggered, random, unannounced times. A group of consumers pull items from the queue as they show up, greedily and without waiting for any other signal.

In this design, there is no chaining of any individual consumer to a producer. The consumers don’t know the number of producers, or even the cumulative number of items that will be added to the queue, in advance.

It takes an individual producer or consumer a variable amount of time to put and extract items from the queue, respectively. The queue serves as a throughput that can communicate with the producers and consumers without them talking to each other directly.

Note: While queues are often used in threaded programs because of the thread-safety of queue.Queue(), you shouldn’t need to concern yourself with thread safety when it comes to async IO. (The exception is when you’re combining the two, but that isn’t done in this tutorial.)

One use-case for queues (as is the case here) is for the queue to act as a transmitter for producers and consumers that aren’t otherwise directly chained or associated with each other.

The synchronous version of this program would look pretty dismal: a group of blocking producers serially add items to the queue, one producer at a time. Only after all producers are done can the queue be processed, by one consumer at a time processing item-by-item. There is a ton of latency in this design. Items may sit idly in the queue rather than be picked up and processed immediately.

An asynchronous version, asyncq.py, is below. The challenging part of this workflow is that there needs to be a signal to the consumers that production is done. Otherwise, await q.get() will hang indefinitely, because the queue will have been fully processed, but consumers won’t have any idea that production is complete.

(Big thanks for some help from a StackOverflow user for helping to straighten out main(): the key is to await q.join(), which blocks until all items in the queue have been received and processed, and then to cancel the consumer tasks, which would otherwise hang up and wait endlessly for additional queue items to appear.)

Here is the full script:

#!/usr/bin/env python3
# asyncq.py

import asyncio
import itertools as it
import os
import random
import time

async def makeitem(size: int = 5) -> str:
    return os.urandom(size).hex()

async def randint(a: int, b: int) -> int:
    return random.randint(a, b)

async def randsleep(a: int = 1, b: int = 5, caller=None) -> None:
    i = await randint(a, b)
    if caller:
        print(f"{caller} sleeping for {i} seconds.")
    await asyncio.sleep(i)

async def produce(name: int, q: asyncio.Queue) -> None:
    n = await randint(1, 5)
    for _ in it.repeat(None, n):  # Synchronous loop for each single producer
        await randsleep(caller=f"Producer {name}")
        i = await makeitem()
        t = time.perf_counter()
        await q.put((i, t))
        print(f"Producer {name} added <{i}> to queue.")

async def consume(name: int, q: asyncio.Queue) -> None:
    while True:
        await randsleep(caller=f"Consumer {name}")
        i, t = await q.get()
        now = time.perf_counter()
        print(f"Consumer {name} got element <{i}>"
              f" in {now-t:0.5f} seconds.")
        q.task_done()

async def main(nprod: int, ncon: int):
    q = asyncio.Queue()
    producers = [asyncio.create_task(produce(n, q)) for n in range(nprod)]
    consumers = [asyncio.create_task(consume(n, q)) for n in range(ncon)]
    await asyncio.gather(*producers)
    await q.join()  # Implicitly awaits consumers, too
    for c in consumers:
        c.cancel()

if __name__ == "__main__":
    import argparse
    random.seed(444)
    parser = argparse.ArgumentParser()
    parser.add_argument("-p", "--nprod", type=int, default=5)
    parser.add_argument("-c", "--ncon", type=int, default=10)
    ns = parser.parse_args()
    start = time.perf_counter()
    asyncio.run(main(**ns.__dict__))
    elapsed = time.perf_counter() - start
    print(f"Program completed in {elapsed:0.5f} seconds.")

The first few coroutines are helper functions that return a random string, a fractional-second performance counter, and a random integer. A producer puts anywhere from 1 to 5 items into the queue. Each item is a tuple of (i, t) where i is a random string and t is the time at which the producer attempts to put the tuple into the queue.

When a consumer pulls an item out, it simply calculates the elapsed time that the item sat in the queue using the timestamp that the item was put in with.

Keep in mind that asyncio.sleep() is used to mimic some other, more complex coroutine that would eat up time and block all other execution if it were a regular blocking function.

Here is a test run with two producers and five consumers:

$ python3 asyncq.py -p 2 -c 5
Producer 0 sleeping for 3 seconds.
Producer 1 sleeping for 3 seconds.
Consumer 0 sleeping for 4 seconds.
Consumer 1 sleeping for 3 seconds.
Consumer 2 sleeping for 3 seconds.
Consumer 3 sleeping for 5 seconds.
Consumer 4 sleeping for 4 seconds.
Producer 0 added <377b1e8f82> to queue.
Producer 0 sleeping for 5 seconds.
Producer 1 added <413b8802f8> to queue.
Consumer 1 got element <377b1e8f82> in 0.00013 seconds.
Consumer 1 sleeping for 3 seconds.
Consumer 2 got element <413b8802f8> in 0.00009 seconds.
Consumer 2 sleeping for 4 seconds.
Producer 0 added <06c055b3ab> to queue.
Producer 0 sleeping for 1 seconds.
Consumer 0 got element <06c055b3ab> in 0.00021 seconds.
Consumer 0 sleeping for 4 seconds.
Producer 0 added <17a8613276> to queue.
Consumer 4 got element <17a8613276> in 0.00022 seconds.
Consumer 4 sleeping for 5 seconds.
Program completed in 9.00954 seconds.

In this case, the items process in fractions of a second. A delay can be due to two reasons:

Standard, largely unavoidable overhead
Situations where all consumers are sleeping when an item appears in the queue

With regards to the second reason, luckily, it is perfectly normal to scale to hundreds or thousands of consumers. You should have no problem with python3 asyncq.py -p 5 -c 100. The point here is that, theoretically, you could have different users on different systems controlling the management of producers and consumers, with the queue serving as the central throughput.

So far, you’ve been thrown right into the fire and seen three related examples of asyncio calling coroutines defined with async and await. If you’re not completely following or just want to get deeper into the mechanics of how modern coroutines came to be in Python, you’ll start from square one with the next section.

Async IO’s Roots in Generators

Earlier, you saw an example of the old-style generator-based coroutines, which have been outdated by more explicit native coroutines. The example is worth re-showing with a small tweak:

import asyncio

@asyncio.coroutine
def py34_coro():
    """Generator-based coroutine"""
    # No need to build these yourself, but be aware of what they are
    s = yield from stuff()
    return s

async def py35_coro():
    """Native coroutine, modern syntax"""
    s = await stuff()
    return s

async def stuff():
    return 0x10, 0x20, 0x30

As an experiment, what happens if you call py34_coro() or py35_coro() on its own, without await, or without any calls to asyncio.run() or other asyncio “porcelain” functions? Calling a coroutine in isolation returns a coroutine object:

>>>

>>> py35_coro()
<coroutine object py35_coro at 0x10126dcc8>

This isn’t very interesting on its surface. The result of calling a coroutine on its own is an awaitable coroutine object.

Time for a quiz: what other feature of Python looks like this? (What feature of Python doesn’t actually “do much” when it’s called on its own?)

Hopefully you’re thinking of generators as an answer to this question, because coroutines are enhanced generators under the hood. The behavior is similar in this regard:

>>>

>>> def gen():
...     yield 0x10, 0x20, 0x30
...
>>> g = gen()
>>> g  # Nothing much happens - need to iterate with `.__next__()`
<generator object gen at 0x1012705e8>
>>> next(g)
(16, 32, 48)

Generator functions are, as it so happens, the foundation of async IO (regardless of whether you declare coroutines with async def rather than the older @asyncio.coroutine wrapper). Technically, await is more closely analogous to yield from than it is to yield. (But remember that yield from x() is just syntactic sugar to replace for i in x(): yield i.)

One critical feature of generators as it pertains to async IO is that they can effectively be stopped and restarted at will. For example, you can break out of iterating over a generator object and then resume iteration on the remaining values later. When a generator function reaches yield, it yields that value, but then it sits idle until it is told to yield its subsequent value.

This can be fleshed out through an example:

>>>

>>> from itertools import cycle
>>> def endless():
...     """Yields 9, 8, 7, 6, 9, 8, 7, 6, ... forever"""
...     yield from cycle((9, 8, 7, 6))

>>> e = endless()
>>> total = 0
>>> for i in e:
...     if total < 30:
...         print(i, end=" ")
...         total += i
...     else:
...         print()
...         # Pause execution. We can resume later.
...         break
9 8 7 6 9 8 7 6 9 8 7 6 9 8

>>> # Resume
>>> next(e), next(e), next(e)
(6, 9, 8)

The await keyword behaves similarly, marking a break point at which the coroutine suspends itself and lets other coroutines work. “Suspended,” in this case, means a coroutine that has temporarily ceded control but not totally exited or finished. Keep in mind that yield, and by extension yield from and await, mark a break point in a generator’s execution.

This is the fundamental difference between functions and generators. A function is all-or-nothing. Once it starts, it won’t stop until it hits a return, then pushes that value to the caller (the function that calls it). A generator, on the other hand, pauses each time it hits a yield and goes no further. Not only can it push this value to calling stack, but it can keep a hold of its local variables when you resume it by calling next() on it.

There’s a second and lesser-known feature of generators that also matters. You can send a value into a generator as well through its .send() method. This allows generators (and coroutines) to call (await) each other without blocking. I won’t get any further into the nuts and bolts of this feature, because it matters mainly for the implementation of coroutines behind the scenes, but you shouldn’t ever really need to use it directly yourself.

If you’re interested in exploring more, you can start at PEP 342, where coroutines were formally introduced. Brett Cannon’s How the Heck Does Async-Await Work in Python is also a good read, as is the PYMOTW writeup on asyncio. Lastly, there’s David Beazley’s Curious Course on Coroutines and Concurrency, which dives deep into the mechanism by which coroutines run.

Let’s try to condense all of the above articles into a few sentences: there is a particularly unconventional mechanism by which these coroutines actually get run. Their result is an attribute of the exception object that gets thrown when their .send() method is called. There’s some more wonky detail to all of this, but it probably won’t help you use this part of the language in practice, so let’s move on for now.

To tie things together, here are some key points on the topic of coroutines as generators:

Coroutines are repurposed generators that take advantage of the peculiarities of generator methods.
Old generator-based coroutines use yield from to wait for a coroutine result. Modern Python syntax in native coroutines simply replaces yield from with await as the means of waiting on a coroutine result. The await is analogous to yield from, and it often helps to think of it as such.
The use of await is a signal that marks a break point. It lets a coroutine temporarily suspend execution and permits the program to come back to it later.

Other Features: `async for` and Async Generators + Comprehensions

Along with plain async/await, Python also enables async for to iterate over an asynchronous iterator. The purpose of an asynchronous iterator is for it to be able to call asynchronous code at each stage when it is iterated over.

A natural extension of this concept is an asynchronous generator. Recall that you can use await, return, or yield in a native coroutine. Using yield within a coroutine became possible in Python 3.6 (via PEP 525), which introduced asynchronous generators with the purpose of allowing await and yield to be used in the same coroutine function body:

>>>

>>> async def mygen(u: int = 10):
...     """Yield powers of 2."""
...     i = 0
...     while i < u:
...         yield 2 ** i
...         i += 1
...         await asyncio.sleep(0.1)

Last but not least, Python enables asynchronous comprehension with async for. Like its synchronous cousin, this is largely syntactic sugar:

>>>

>>> async def main():
...     # This does *not* introduce concurrent execution
...     # It is meant to show syntax only
...     g = [i async for i in mygen()]
...     f = [j async for j in mygen() if not (j // 3 % 5)]
...     return g, f
...
>>> g, f = asyncio.run(main())
>>> g
[1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
>>> f
[1, 2, 16, 32, 256, 512]

This is a crucial distinction: neither asynchronous generators nor comprehensions make the iteration concurrent. All that they do is provide the look-and-feel of their synchronous counterparts, but with the ability for the loop in question to give up control to the event loop for some other coroutine to run.

In other words, asynchronous iterators and asynchronous generators are not designed to concurrently map some function over a sequence or iterator. They’re merely designed to let the enclosing coroutine allow other tasks to take their turn. The async for and async with statements are only needed to the extent that using plain for or with would “break” the nature of await in the coroutine. This distinction between asynchronicity and concurrency is a key one to grasp.

The Event Loop and `asyncio.run()`

You can think of an event loop as something like a while True loop that monitors coroutines, taking feedback on what’s idle, and looking around for things that can be executed in the meantime. It is able to wake up an idle coroutine when whatever that coroutine is waiting on becomes available.

Thus far, the entire management of the event loop has been implicitly handled by one function call:

asyncio.run(main())  # Python 3.7+

asyncio.run(), introduced in Python 3.7, is responsible for getting the event loop, running tasks until they are marked as complete, and then closing the event loop.

There’s a more long-winded way of managing the asyncio event loop, with get_event_loop(). The typical pattern looks like this:

loop = asyncio.get_event_loop()
try:
    loop.run_until_complete(main())
finally:
    loop.close()

You’ll probably see loop.get_event_loop() floating around in older examples, but unless you have a specific need to fine-tune control over the event loop management, asyncio.run() should be sufficient for most programs.

If you do need to interact with the event loop within a Python program, loop is a good-old-fashioned Python object that supports introspection with loop.is_running() and loop.is_closed(). You can manipulate it if you need to get more fine-tuned control, such as in scheduling a callback by passing the loop as an argument.

What is more crucial is understanding a bit beneath the surface about the mechanics of the event loop. Here are a few points worth stressing about the event loop.

#1: Coroutines don’t do much on their own until they are tied to the event loop.

You saw this point before in the explanation on generators, but it’s worth restating. If you have a main coroutine that awaits others, simply calling it in isolation has little effect:

>>>

>>> import asyncio

>>> async def main():
...     print("Hello ...")
...     await asyncio.sleep(1)
...     print("World!")

>>> routine = main()
>>> routine
<coroutine object main at 0x1027a6150>

Remember to use asyncio.run() to actually force execution by scheduling the main() coroutine (future object) for execution on the event loop:

>>>

>>> asyncio.run(routine)
Hello ...
World!

(Other coroutines can be executed with await. It is typical to wrap just main() in asyncio.run(), and chained coroutines with await will be called from there.)

#2: By default, an async IO event loop runs in a single thread and on a single CPU core. Usually, running one single-threaded event loop in one CPU core is more than sufficient. It is also possible to run event loops across multiple cores. Check out this talk by John Reese for more, and be warned that your laptop may spontaneously combust.

#3. Event loops are pluggable. That is, you could, if you really wanted, write your own event loop implementation and have it run tasks just the same. This is wonderfully demonstrated in the uvloop package, which is an implementation of the event loop in Cython.

That is what is meant by the term “pluggable event loop”: you can use any working implementation of an event loop, unrelated to the structure of the coroutines themselves. The asyncio package itself ships with two different event loop implementations, with the default being based on the selectors module. (The second implementation is built for Windows only.)

A Full Program: Asynchronous Requests

You’ve made it this far, and now it’s time for the fun and painless part. In this section, you’ll build a web-scraping URL collector, areq.py, using aiohttp, a blazingly fast async HTTP client/server framework. (We just need the client part.) Such a tool could be used to map connections between a cluster of sites, with the links forming a directed graph.

Note: You may be wondering why Python’s requests package isn’t compatible with async IO. requests is built on top of urllib3, which in turn uses Python’s http and socket modules.

By default, socket operations are blocking. This means that Python won’t like await requests.get(url) because .get() is not awaitable. In contrast, almost everything in aiohttp is an awaitable coroutine, such as session.request() and response.text(). It’s a great package otherwise, but you’re doing yourself a disservice by using requests in asynchronous code.

The high-level program structure will look like this:

Read a sequence of URLs from a local file, urls.txt.
Send GET requests for the URLs and decode the resulting content. If this fails, stop there for a URL.
Search for the URLs within href tags in the HTML of the responses.
Write the results to foundurls.txt.
Do all of the above as asynchronously and concurrently as possible. (Use aiohttp for the requests, and aiofiles for the file-appends. These are two primary examples of IO that are well-suited for the async IO model.)

Here are the contents of urls.txt. It’s not huge, and contains mostly highly trafficked sites:

$ cat urls.txt
https://regex101.com/
https://docs.python.org/3/this-url-will-404.html
https://www.nytimes.com/guides/
https://www.mediamatters.org/
https://1.1.1.1/
https://www.politico.com/tipsheets/morning-money
https://www.bloomberg.com/markets/economics
https://www.ietf.org/rfc/rfc2616.txt

The second URL in the list should return a 404 response, which you’ll need to handle gracefully. If you’re running an expanded version of this program, you’ll probably need to deal with much hairier problems than this, such a server disconnections and endless redirects.

The requests themselves should be made using a single session, to take advantage of reusage of the session’s internal connection pool.

Let’s take a look at the full program. We’ll walk through things step-by-step after:

#!/usr/bin/env python3
# areq.py

"""Asynchronously get links embedded in multiple pages' HMTL."""

import asyncio
import logging
import re
import sys
from typing import IO
import urllib.error
import urllib.parse

import aiofiles
import aiohttp
from aiohttp import ClientSession

logging.basicConfig(
    format="%(asctime)s %(levelname)s:%(name)s: %(message)s",
    level=logging.DEBUG,
    datefmt="%H:%M:%S",
    stream=sys.stderr,
)
logger = logging.getLogger("areq")
logging.getLogger("chardet.charsetprober").disabled = True

HREF_RE = re.compile(r'href="(.*?)"')

async def fetch_html(url: str, session: ClientSession, **kwargs) -> str:
    """GET request wrapper to fetch page HTML.

    kwargs are passed to `session.request()`.
    """

    resp = await session.request(method="GET", url=url, **kwargs)
    resp.raise_for_status()
    logger.info("Got response [%s] for URL: %s", resp.status, url)
    html = await resp.text()
    return html

async def parse(url: str, session: ClientSession, **kwargs) -> set:
    """Find HREFs in the HTML of `url`."""
    found = set()
    try:
        html = await fetch_html(url=url, session=session, **kwargs)
    except (
        aiohttp.ClientError,
        aiohttp.http_exceptions.HttpProcessingError,
    ) as e:
        logger.error(
            "aiohttp exception for %s [%s]: %s",
            url,
            getattr(e, "status", None),
            getattr(e, "message", None),
        )
        return found
    except Exception as e:
        logger.exception(
            "Non-aiohttp exception occured:  %s", getattr(e, "__dict__", {})
        )
        return found
    else:
        for link in HREF_RE.findall(html):
            try:
                abslink = urllib.parse.urljoin(url, link)
            except (urllib.error.URLError, ValueError):
                logger.exception("Error parsing URL: %s", link)
                pass
            else:
                found.add(abslink)
        logger.info("Found %d links for %s", len(found), url)
        return found

async def write_one(file: IO, url: str, **kwargs) -> None:
    """Write the found HREFs from `url` to `file`."""
    res = await parse(url=url, **kwargs)
    if not res:
        return None
    async with aiofiles.open(file, "a") as f:
        for p in res:
            await f.write(f"{url}\t{p}\n")
        logger.info("Wrote results for source URL: %s", url)

async def bulk_crawl_and_write(file: IO, urls: set, **kwargs) -> None:
    """Crawl & write concurrently to `file` for multiple `urls`."""
    async with ClientSession() as session:
        tasks = []
        for url in urls:
            tasks.append(
                write_one(file=file, url=url, session=session, **kwargs)
            )
        await asyncio.gather(*tasks)

if __name__ == "__main__":
    import pathlib
    import sys

    assert sys.version_info >= (3, 7), "Script requires Python 3.7+."
    here = pathlib.Path(__file__).parent

    with open(here.joinpath("urls.txt")) as infile:
        urls = set(map(str.strip, infile))

    outpath = here.joinpath("foundurls.txt")
    with open(outpath, "w") as outfile:
        outfile.write("source_url\tparsed_url\n")

    asyncio.run(bulk_crawl_and_write(file=outpath, urls=urls))

This script is longer than our initial toy programs, so let’s break it down.

The constant HREF_RE is a regular expression to extract what we’re ultimately searching for, href tags within HTML:

>>>

>>> HREF_RE.search('Go to <a href="https://realpython.com/">Real Python</a>')
<re.Match object; span=(15, 45), match='href="https://realpython.com/"'>

The coroutine fetch_html() is a wrapper around a GET request to make the request and decode the resulting page HTML. It makes the request, awaits the response, and raises right away in the case of a non-200 status:

resp = await session.request(method="GET", url=url, **kwargs)
resp.raise_for_status()

If the status is okay, fetch_html() returns the page HTML (a str). Notably, there is no exception handling done in this function. The logic is to propagate that exception to the caller and let it be handled there:

html = await resp.text()

We await session.request() and resp.text() because they’re awaitable coroutines. The request/response cycle would otherwise be the long-tailed, time-hogging portion of the application, but with async IO, fetch_html() lets the event loop work on other readily available jobs such as parsing and writing URLs that have already been fetched.

Next in the chain of coroutines comes parse(), which waits on fetch_html() for a given URL, and then extracts all of the href tags from that page’s HTML, making sure that each is valid and formatting it as an absolute path.

Admittedly, the second portion of parse() is blocking, but it consists of a quick regex match and ensuring that the links discovered are made into absolute paths.

In this specific case, this synchronous code should be quick and inconspicuous. But just remember that any line within a given coroutine will block other coroutines unless that line uses yield, await, or return. If the parsing was a more intensive process, you might want to consider running this portion in its own process with loop.run_in_executor().

Next, the coroutine write() takes a file object and a single URL, and waits on parse() to return a set of the parsed URLs, writing each to the file asynchronously along with its source URL through use of aiofiles, a package for async file IO.

Lastly, bulk_crawl_and_write() serves as the main entry point into the script’s chain of coroutines. It uses a single session, and a task is created for each URL that is ultimately read from urls.txt.

Here are a few additional points that deserve mention:

The default ClientSession has an adapter with a maximum of 100 open connections. To change that, pass an instance of asyncio.connector.TCPConnector to ClientSession. You can also specify limits on a per-host basis.
You can specify max timeouts for both the session as a whole and for individual requests.
This script also uses async with, which works with an asynchronous context manager. I haven’t devoted a whole section to this concept because the transition from synchronous to asynchronous context managers is fairly straightforward. The latter has to define .__aenter__() and .__aexit__() rather than .__exit__() and .__enter__(). As you might expect, async with can only be used inside a coroutine function declared with async def.

If you’d like to explore a bit more, the companion files for this tutorial up at GitHub have comments and docstrings attached as well.

Here’s the execution in all of its glory, as areq.py gets, parses, and saves results for 9 URLs in under a second:

$ python3 areq.py
21:33:22 DEBUG:asyncio: Using selector: KqueueSelector
21:33:22 INFO:areq: Got response [200] for URL: https://www.mediamatters.org/
21:33:22 INFO:areq: Found 115 links for https://www.mediamatters.org/
21:33:22 INFO:areq: Got response [200] for URL: https://www.nytimes.com/guides/
21:33:22 INFO:areq: Got response [200] for URL: https://www.politico.com/tipsheets/morning-money
21:33:22 INFO:areq: Got response [200] for URL: https://www.ietf.org/rfc/rfc2616.txt
21:33:22 ERROR:areq: aiohttp exception for https://docs.python.org/3/this-url-will-404.html [404]: Not Found
21:33:22 INFO:areq: Found 120 links for https://www.nytimes.com/guides/
21:33:22 INFO:areq: Found 143 links for https://www.politico.com/tipsheets/morning-money
21:33:22 INFO:areq: Wrote results for source URL: https://www.mediamatters.org/
21:33:22 INFO:areq: Found 0 links for https://www.ietf.org/rfc/rfc2616.txt
21:33:22 INFO:areq: Got response [200] for URL: https://1.1.1.1/
21:33:22 INFO:areq: Wrote results for source URL: https://www.nytimes.com/guides/
21:33:22 INFO:areq: Wrote results for source URL: https://www.politico.com/tipsheets/morning-money
21:33:22 INFO:areq: Got response [200] for URL: https://www.bloomberg.com/markets/economics
21:33:22 INFO:areq: Found 3 links for https://www.bloomberg.com/markets/economics
21:33:22 INFO:areq: Wrote results for source URL: https://www.bloomberg.com/markets/economics
21:33:23 INFO:areq: Found 36 links for https://1.1.1.1/
21:33:23 INFO:areq: Got response [200] for URL: https://regex101.com/
21:33:23 INFO:areq: Found 23 links for https://regex101.com/
21:33:23 INFO:areq: Wrote results for source URL: https://regex101.com/
21:33:23 INFO:areq: Wrote results for source URL: https://1.1.1.1/

That’s not too shabby! As a sanity check, you can check the line-count on the output. In my case, it’s 626, though keep in mind this may fluctuate:

$ wc -l foundurls.txt
     626 foundurls.txt

$ head -n 3 foundurls.txt
source_url  parsed_url
https://www.bloomberg.com/markets/economics https://www.bloomberg.com/feedback
https://www.bloomberg.com/markets/economics https://www.bloomberg.com/notices/tos

Next Steps: If you’d like to up the ante, make this webcrawler recursive. You can use aio-redis to keep track of which URLs have been crawled within the tree to avoid requesting them twice, and connect links with Python’s networkx library.

Remember to be nice. Sending 1000 concurrent requests to a small, unsuspecting website is bad, bad, bad. There are ways to limit how many concurrent requests you’re making in one batch, such as in using the sempahore objects of asyncio or using a pattern like this one. If you don’t heed this warning, you may get a massive batch of TimeoutError exceptions and only end up hurting your own program.

Async IO in Context

Now that you’ve seen a healthy dose of code, let’s step back for a minute and consider when async IO is an ideal option and how you can make the comparison to arrive at that conclusion or otherwise choose a different model of concurrency.

When and Why Is Async IO the Right Choice?

This tutorial is no place for an extended treatise on async IO versus threading versus multiprocessing. However, it’s useful to have an idea of when async IO is probably the best candidate of the three.

The battle over async IO versus multiprocessing is not really a battle at all. In fact, they can be used in concert. If you have multiple, fairly uniform CPU-bound tasks (a great example is a grid search in libraries such as scikit-learn or keras), multiprocessing should be an obvious choice.

Simply putting async before every function is a bad idea if all of the functions use blocking calls. (This can actually slow down your code.) But as mentioned previously, there are places where async IO and multiprocessing can live in harmony.

The contest between async IO and threading is a little bit more direct. I mentioned in the introduction that “threading is hard.” The full story is that, even in cases where threading seems easy to implement, it can still lead to infamous impossible-to-trace bugs due to race conditions and memory usage, among other things.

Threading also tends to scale less elegantly than async IO, because threads are a system resource with a finite availability. Creating thousands of threads will fail on many machines, and I don’t recommend trying it in the first place. Creating thousands of async IO tasks is completely feasible.

Async IO shines when you have multiple IO-bound tasks where the tasks would otherwise be dominated by blocking IO-bound wait time, such as:

Network IO, whether your program is the server or the client side
Serverless designs, such as a peer-to-peer, multi-user network like a group chatroom
Read/write operations where you want to mimic a “fire-and-forget” style but worry less about holding a lock on whatever you’re reading and writing to

The biggest reason not to use it is that await only supports a specific set of objects that define a specific set of methods. If you want to do async read operations with a certain DBMS, you’ll need to find not just a Python wrapper for that DBMS, but one that supports the async/await syntax. Coroutines that contain synchronous calls block other coroutines and tasks from running.

For a shortlist of libraries that work with async/await, see the list at the end of this tutorial.

Async IO It Is, but Which One?

This tutorial focuses on async IO, the async/await syntax, and using asyncio for event-loop management and specifying tasks. asyncio certainly isn’t the only async IO library out there. This observation from Nathaniel J. Smith says a lot:

[In] a few years, asyncio might find itself relegated to becoming one of those stdlib libraries that savvy developers avoid, like urllib2.

…

What I’m arguing, in effect, is that asyncio is a victim of its own success: when it was designed, it used the best approach possible; but since then, work inspired by asyncio – like the addition of async/await – has shifted the landscape so that we can do even better, and now asyncio is hamstrung by its earlier commitments. (Source)

To that end, a few big-name alternatives that do what asyncio does, albeit with different APIs and different approaches, are curio and trio. Personally, I think that if you’re building a moderately sized, straightforward program, just using asyncio is plenty sufficient and understandable, and lets you avoid adding yet another large dependency outside of Python’s standard library.

But by all means, check out curio and trio, and you might find that they get the same thing done in a way that’s more intuitive for you as the user. Many of the package-agnostic concepts presented here should permeate to alternative async IO packages as well.

Odds and Ends

In these next few sections, you’ll cover some miscellaneous parts of asyncio and async/await that haven’t fit neatly into the tutorial thus far, but are still important for building and understanding a full program.

Other Top-Level `asyncio` Functions

In addition to asyncio.run(), you’ve seen a few other package-level functions such as asyncio.create_task() and asyncio.gather().

You can use create_task() to schedule the execution of a coroutine object, followed by asyncio.run():

>>>

>>> import asyncio

>>> async def coro(seq) -> list:
...     """'IO' wait time is proportional to the max element."""
...     await asyncio.sleep(max(seq))
...     return list(reversed(seq))
...
>>> async def main():
...     # This is a bit redundant in the case of one task
...     # We could use `await coro([3, 2, 1])` on its own
...     t = asyncio.create_task(coro([3, 2, 1]))  # Python 3.7+
...     await t
...     print(f't: type {type(t)}')
...     print(f't done: {t.done()}')
...
>>> t = asyncio.run(main())
t: type <class '_asyncio.Task'>
t done: True

There’s a subtlety to this pattern: if you don’t await t within main(), it may finish before main() itself signals that it is complete. Because asyncio.run(main()) calls loop.run_until_complete(main()), the event loop is only concerned (without await t present) that main() is done, not that the tasks that get created within main() are done. Without await t, the loop’s other tasks will be cancelled, possibly before they are completed. If you need to get a list of currently pending tasks, you can use asyncio.Task.all_tasks().

Note: asyncio.create_task() was introduced in Python 3.7. In Python 3.6 or lower, use asyncio.ensure_future() in place of create_task().

Separately, there’s asyncio.gather(). While it doesn’t do anything tremendously special, gather() is meant to neatly put a collection of coroutines (futures) into a single future. As a result, it returns a single future object, and, if you await asyncio.gather() and specify multiple tasks or coroutines, you’re waiting for all of them to be completed. (This somewhat parallels queue.join() from our earlier example.) The result of gather() will be a list of the results across the inputs:

>>>

>>> import time
>>> async def main():
...     t = asyncio.create_task(coro([3, 2, 1]))
...     t2 = asyncio.create_task(coro([10, 5, 0]))  # Python 3.7+
...     print('Start:', time.strftime('%X'))
...     a = await asyncio.gather(t, t2)
...     print('End:', time.strftime('%X'))  # Should be 10 seconds
...     print(f'Both tasks done: {all((t.done(), t2.done()))}')
...     return a
...
>>> a = asyncio.run(main())
Start: 16:20:11
End: 16:20:21
Both tasks done: True
>>> a
[[1, 2, 3], [0, 5, 10]]

You probably noticed that gather() waits on the entire result set of the Futures or coroutines that you pass it. Alternatively, you can loop over asyncio.as_completed() to get tasks as they are completed, in the order of completion. The function returns an iterator that yields tasks as they finish. Below, the result of coro([3, 2, 1]) will be available before coro([10, 5, 0]) is complete, which is not the case with gather():

>>>

>>> async def main():
...     t = asyncio.create_task(coro([3, 2, 1]))
...     t2 = asyncio.create_task(coro([10, 5, 0]))
...     print('Start:', time.strftime('%X'))
...     for res in asyncio.as_completed((t, t2)):
...         compl = await res
...         print(f'res: {compl} completed at {time.strftime("%X")}')
...     print('End:', time.strftime('%X'))
...     print(f'Both tasks done: {all((t.done(), t2.done()))}')
...
>>> a = asyncio.run(main())
Start: 09:49:07
res: [1, 2, 3] completed at 09:49:10
res: [0, 5, 10] completed at 09:49:17
End: 09:49:17
Both tasks done: True

Lastly, you may also see asyncio.ensure_future(). You should rarely need it, because it’s a lower-level plumbing API and largely replaced by create_task(), which was introduced later.

The Precedence of `await`

While they behave somewhat similarly, the await keyword has significantly higher precedence than yield. This means that, because it is more tightly bound, there are a number of instances where you’d need parentheses in a yield from statement that are not required in an analogous await statement. For more information, see examples of await expressions from PEP 492.

Conclusion

You’re now equipped to use async/await and the libraries built off of it. Here’s a recap of what you’ve covered:

Asynchronous IO as a language-agnostic model and a way to effect concurrency by letting coroutines indirectly communicate with each other
The specifics of Python’s new async and await keywords, used to mark and define coroutines
asyncio, the Python package that provides the API to run and manage coroutines

Resources

Python Version Specifics

Async IO in Python has evolved swiftly, and it can be hard to keep track of what came when. Here’s a list of Python minor-version changes and introductions related to asyncio:

3.3: The yield from expression allows for generator delegation.
3.4: asyncio was introduced in the Python standard library with provisional API status.
3.5: async and await became a part of the Python grammar, used to signify and wait on coroutines. They were not yet reserved keywords. (You could still define functions or variables named async and await.)
3.6: Asynchronous generators and asynchronous comprehensions were introduced. The API of asyncio was declared stable rather than provisional.
3.7: async and await became reserved keywords. (They cannot be used as identifiers.) They are intended to replace the asyncio.coroutine() decorator. asyncio.run() was introduced to the asyncio package, among a bunch of other features.

If you want to be safe (and be able to use asyncio.run()), go with Python 3.7 or above to get the full set of features.

Articles

Here’s a curated list of additional resources:

Real Python: Speed up your Python Program with Concurrency
Real Python: What is the Python Global Interpreter Lock?
CPython: The asyncio package source
Python docs: Data model > Coroutines
TalkPython: Async Techniques and Examples in Python
Brett Cannon: How the Heck Does Async-Await Work in Python 3.5?
PYMOTW: asyncio
A. Jesse Jiryu Davis and Guido van Rossum: A Web Crawler With asyncio Coroutines
Andy Pearce: The State of Python Coroutines: yield from
Nathaniel J. Smith: Some Thoughts on Asynchronous API Design in a Post-async/await World
Armin Ronacher: I don’t understand Python’s Asyncio
Andy Balaam: series on asyncio (4 posts)
Stack Overflow: Python asyncio.semaphore in async-await function
Yeray Diaz:
- AsyncIO for the Working Python Developer
- Asyncio Coroutine Patterns: Beyond await

A few Python What’s New sections explain the motivation behind language changes in more detail:

What’s New in Python 3.3 (yield from and PEP 380)
What’s New in Python 3.6 (PEP 525 & 530)

From David Beazley:

YouTube talks:

PEP	Date Created
PEP 342 – Coroutines via Enhanced Generators	2005-05
PEP 380 – Syntax for Delegating to a Subgenerator	2009-02
PEP 3153 – Asynchronous IO support	2011-05
PEP 3156 – Asynchronous IO Support Rebooted: the “asyncio” Module	2012-12
PEP 492 – Coroutines with async and await syntax	2015-04
PEP 525 – Asynchronous Generators	2016-07
PEP 530 – Asynchronous Comprehensions	2016-09

Libraries That Work With `async`/`await`

From aio-libs:

aiohttp: Asynchronous HTTP client/server framework
aioredis: Async IO Redis support
aiopg: Async IO PostgreSQL support
aiomcache: Async IO memcached client
aiokafka: Async IO Kafka client
aiozmq: Async IO ZeroMQ support
aiojobs: Jobs scheduler for managing background tasks
async_lru: Simple LRU cache for async IO

From magicstack:

uvloop: Ultra fast async IO event loop
asyncpg: (Also very fast) async IO PostgreSQL support

From other hosts:

trio: Friendlier asyncio intended to showcase a radically simpler design
aiofiles: Async file IO
asks: Async requests-like http library
asyncio-redis: Async IO Redis support
aioprocessing: Integrates multiprocessing module with asyncio
umongo: Async IO MongoDB client
unsync: Unsynchronize asyncio
aiostream: Like itertools, but async

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Speed Up Your Python Program With Concurrency

2019-01-14T14:00:00+00:00

If you’ve heard lots of talk about asyncio being added to Python but are curious how it compares to other concurrency methods or are wondering what concurrency is and how it might speed up your program, you’ve come to the right place.

In this article, you’ll learn the following:

What concurrency is
What parallelism is
How some of Python’s concurrency methods compare, including threading, asyncio, and multiprocessing
When to use concurrency in your program and which module to use

This article assumes that you have a basic understanding of Python and that you’re using at least version 3.6 to run the examples. You can download the examples from the Real Python GitHub repo.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Take the Quiz: Test your knowledge with our interactive “Python Concurrency” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Take the Quiz »

What Is Concurrency?

The dictionary definition of concurrency is simultaneous occurrence. In Python, the things that are occurring simultaneously are called by different names (thread, task, process) but at a high level, they all refer to a sequence of instructions that run in order.

I like to think of them as different trains of thought. Each one can be stopped at certain points, and the CPU or brain that is processing them can switch to a different one. The state of each one is saved so it can be restarted right where it was interrupted.

You might wonder why Python uses different words for the same concept. It turns out that threads, tasks, and processes are only the same if you view them from a high level. Once you start digging into the details, they all represent slightly different things. You’ll see more of how they are different as you progress through the examples.

Now let’s talk about the simultaneous part of that definition. You have to be a little careful because, when you get down to the details, only multiprocessing actually runs these trains of thought at literally the same time. Threading and asyncio both run on a single processor and therefore only run one at a time. They just cleverly find ways to take turns to speed up the overall process. Even though they don’t run different trains of thought simultaneously, we still call this concurrency.

The way the threads or tasks take turns is the big difference between threading and asyncio. In threading, the operating system actually knows about each thread and can interrupt it at any time to start running a different thread. This is called pre-emptive multitasking since the operating system can pre-empt your thread to make the switch.

Pre-emptive multitasking is handy in that the code in the thread doesn’t need to do anything to make the switch. It can also be difficult because of that “at any time” phrase. This switch can happen in the middle of a single Python statement, even a trivial one like x = x + 1.

Asyncio, on the other hand, uses cooperative multitasking. The tasks must cooperate by announcing when they are ready to be switched out. That means that the code in the task has to change slightly to make this happen.

The benefit of doing this extra work up front is that you always know where your task will be swapped out. It will not be swapped out in the middle of a Python statement unless that statement is marked. You’ll see later how this can simplify parts of your design.

What Is Parallelism?

So far, you’ve looked at concurrency that happens on a single processor. What about all of those CPU cores your cool, new laptop has? How can you make use of them? multiprocessing is the answer.

With multiprocessing, Python creates new processes. A process here can be thought of as almost a completely different program, though technically they’re usually defined as a collection of resources where the resources include memory, file handles and things like that. One way to think about it is that each process runs in its own Python interpreter.

Because they are different processes, each of your trains of thought in a multiprocessing program can run on a different core. Running on a different core means that they actually can run at the same time, which is fabulous. There are some complications that arise from doing this, but Python does a pretty good job of smoothing them over most of the time.

Now that you have an idea of what concurrency and parallelism are, let’s review their differences, and then we can look at why they can be useful:

Concurrency Type	Switching Decision	Number of Processors
Pre-emptive multitasking (`threading`)	The operating system decides when to switch tasks external to Python.	1
Cooperative multitasking (`asyncio`)	The tasks decide when to give up control.	1
Multiprocessing (`multiprocessing`)	The processes all run at the same time on different processors.	Many

Each of these types of concurrency can be useful. Let’s take a look at what types of programs they can help you speed up.

When Is Concurrency Useful?

Concurrency can make a big difference for two types of problems. These are generally called CPU-bound and I/O-bound.

I/O-bound problems cause your program to slow down because it frequently must wait for input/output (I/O) from some external resource. They arise frequently when your program is working with things that are much slower than your CPU.

Examples of things that are slower than your CPU are legion, but your program thankfully does not interact with most of them. The slow things your program will interact with most frequently are the file system and network connections.

Let’s see what that looks like:

In the diagram above, the blue boxes show time when your program is doing work, and the red boxes are time spent waiting for an I/O operation to complete. This diagram is not to scale because requests on the internet can take several orders of magnitude longer than CPU instructions, so your program can end up spending most of its time waiting. This is what your browser is doing most of the time.

On the flip side, there are classes of programs that do significant computation without talking to the network or accessing a file. These are the CPU-bound programs, because the resource limiting the speed of your program is the CPU, not the network or the file system.

Here’s a corresponding diagram for a CPU-bound program:

As you work through the examples in the following section, you’ll see that different forms of concurrency work better or worse with CPU-bound and I/O-bound programs. Adding concurrency to your program adds extra code and complications, so you’ll need to decide if the potential speed up is worth the extra effort. By the end of this article, you should have enough info to start making that decision.

Here’s a quick summary to clarify this concept:

I/O-Bound Process	CPU-Bound Process
Your program spends most of its time talking to a slow device, like a network connection, a hard drive, or a printer.	You program spends most of its time doing CPU operations.
Speeding it up involves overlapping the times spent waiting for these devices.	Speeding it up involves finding ways to do more computations in the same amount of time.

You’ll look at I/O-bound programs first. Then, you’ll get to see some code dealing with CPU-bound programs.

How to Speed Up an I/O-Bound Program

Let’s start by focusing on I/O-bound programs and a common problem: downloading content over the network. For our example, you will be downloading web pages from a few sites, but it really could be any network traffic. It’s just easier to visualize and set up with web pages.

Synchronous Version

We’ll start with a non-concurrent version of this task. Note that this program requires the requests module. You should run pip install requests before running it, probably using a virtualenv. This version does not use concurrency at all:

import requests
import time


def download_site(url, session):
    with session.get(url) as response:
        print(f"Read {len(response.content)} from {url}")


def download_all_sites(sites):
    with requests.Session() as session:
        for url in sites:
            download_site(url, session)


if __name__ == "__main__":
    sites = [
        "http://www.jython.org",
        "http://olympus.realpython.org/dice",
    ] * 80
    start_time = time.time()
    download_all_sites(sites)
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} in {duration} seconds")

As you can see, this is a fairly short program. download_site() just downloads the contents from a URL and prints the size. One small thing to point out is that we’re using a Session object from requests.

It is possible to simply use get() from requests directly, but creating a Session object allows requests to do some fancy networking tricks and really speed things up.

download_all_sites() creates the Session and then walks through the list of sites, downloading each one in turn. Finally, it prints out how long this process took so you can have the satisfaction of seeing how much concurrency has helped us in the following examples.

The processing diagram for this program will look much like the I/O-bound diagram in the last section.

Note: Network traffic is dependent on many factors that can vary from second to second. I’ve seen the times of these tests double from one run to another due to network issues.

Why the Synchronous Version Rocks

The great thing about this version of code is that, well, it’s easy. It was comparatively easy to write and debug. It’s also more straight-forward to think about. There’s only one train of thought running through it, so you can predict what the next step is and how it will behave.

The Problems With the Synchronous Version

The big problem here is that it’s relatively slow compared to the other solutions we’ll provide. Here’s an example of what the final output gave on my machine:

$ ./io_non_concurrent.py
   [most output skipped]
Downloaded 160 in 14.289619207382202 seconds

Note: Your results may vary significantly. When running this script, I saw the times vary from 14.2 to 21.9 seconds. For this article, I took the fastest of three runs as the time. The differences between the methods will still be clear.

Being slower isn’t always a big issue, however. If the program you’re running takes only 2 seconds with a synchronous version and is only run rarely, it’s probably not worth adding concurrency. You can stop here.

What if your program is run frequently? What if it takes hours to run? Let’s move on to concurrency by rewriting this program using threading.

`threading` Version

As you probably guessed, writing a threaded program takes more effort. You might be surprised at how little extra effort it takes for simple cases, however. Here’s what the same program looks like with threading:

import concurrent.futures
import requests
import threading
import time


thread_local = threading.local()


def get_session():
    if not getattr(thread_local, "session", None):
        thread_local.session = requests.Session()
    return thread_local.session


def download_site(url):
    session = get_session()
    with session.get(url) as response:
        print(f"Read {len(response.content)} from {url}")


def download_all_sites(sites):
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        executor.map(download_site, sites)


if __name__ == "__main__":
    sites = [
        "http://www.jython.org",
        "http://olympus.realpython.org/dice",
    ] * 80
    start_time = time.time()
    download_all_sites(sites)
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} in {duration} seconds")

When you add threading, the overall structure is the same and you only needed to make a few changes. download_all_sites() changed from calling the function once per site to a more complex structure.

In this version, you’re creating a ThreadPoolExecutor, which seems like a complicated thing. Let’s break that down: ThreadPoolExecutor = Thread + Pool + Executor.

You already know about the Thread part. That’s just a train of thought we mentioned earlier. The Pool portion is where it starts to get interesting. This object is going to create a pool of threads, each of which can run concurrently. Finally, the Executor is the part that’s going to control how and when each of the threads in the pool will run. It will execute the request in the pool.

Helpfully, the standard library implements ThreadPoolExecutor as a context manager so you can use the with syntax to manage creating and freeing the pool of Threads.

Once you have a ThreadPoolExecutor, you can use its handy .map() method. This method runs the passed-in function on each of the sites in the list. The great part is that it automatically runs them concurrently using the pool of threads it is managing.

Those of you coming from other languages, or even Python 2, are probably wondering where the usual objects and functions are that manage the details you’re used to when dealing with threading, things like Thread.start(), Thread.join(), and Queue.

These are all still there, and you can use them to achieve fine-grained control of how your threads are run. But, starting with Python 3.2, the standard library added a higher-level abstraction called Executors that manage many of the details for you if you don’t need that fine-grained control.

The other interesting change in our example is that each thread needs to create its own requests.Session() object. When you’re looking at the documentation for requests, it’s not necessarily easy to tell, but reading this issue, it seems fairly clear that you need a separate Session for each thread.

This is one of the interesting and difficult issues with threading. Because the operating system is in control of when your task gets interrupted and another task starts, any data that is shared between the threads needs to be protected, or thread-safe. Unfortunately requests.Session() is not thread-safe.

There are several strategies for making data accesses thread-safe depending on what the data is and how you’re using it. One of them is to use thread-safe data structures like Queue from Python’s queue module.

These objects use low-level primitives like threading.Lock to ensure that only one thread can access a block of code or a bit of memory at the same time. You are using this strategy indirectly by way of the ThreadPoolExecutor object.

Another strategy to use here is something called thread local storage. Threading.local() creates an object that look like a global but is specific to each individual thread. In your example, this is done with threadLocal and get_session():

threadLocal = threading.local()


def get_session():
    if getattr(threadLocal, "session", None) is None:
        threadLocal.session = requests.Session()
    return threadLocal.session

ThreadLocal is in the threading module to specifically solve this problem. It looks a little odd, but you only want to create one of these objects, not one for each thread. The object itself takes care of separating accesses from different threads to different data.

When get_session() is called, the session it looks up is specific to the particular thread on which it’s running. So each thread will create a single session the first time it calls get_session() and then will simply use that session on each subsequent call throughout its lifetime.

Finally, a quick note about picking the number of threads. You can see that the example code uses 5 threads. Feel free to play around with this number and see how the overall time changes. You might expect that having one thread per download would be the fastest but, at least on my system it was not. I found the fastest results somewhere between 5 and 10 threads. If you go any higher than that, then the extra overhead of creating and destroying the threads erases any time savings.

The difficult answer here is that the correct number of threads is not a constant from one task to another. Some experimentation is required.

Why the threading Version Rocks

It’s fast! Here’s the fastest run of my tests. Remember that the non-concurrent version took more than 14 seconds:

$ ./io_threading.py
   [most output skipped]
Downloaded 160 in 3.7238826751708984 seconds

Here’s what its execution timing diagram looks like:

It uses multiple threads to have multiple open requests out to web sites at the same time, allowing your program to overlap the waiting times and get the final result faster! Yippee! That was the goal.

The Problems with the threading Version

Well, as you can see from the example, it takes a little more code to make this happen, and you really have to give some thought to what data is shared between threads.

Threads can interact in ways that are subtle and hard to detect. These interactions can cause race conditions that frequently result in random, intermittent bugs that can be quite difficult to find. Those of you who are unfamiliar with the concept of race conditions might want to expand and read the section below.

Race conditions are an entire class of subtle bugs that can and frequently do happen in multi-threaded code. Race conditions happen because the programmer has not sufficiently protected data accesses to prevent threads from interfering with each other. You need to take extra steps when writing threaded code to ensure things are thread-safe.

What’s going on here is that the operating system is controlling when your thread runs and when it gets swapped out to let another thread run. This thread swapping can occur at any point, even while doing sub-steps of a Python statement. As a quick example, look at this function:

import concurrent.futures


counter = 0


def increment_counter(fake_value):
    global counter
    for _ in range(100):
        counter += 1


if __name__ == "__main__":
    fake_data = [x for x in range(5000)]
    counter = 0
    with concurrent.futures.ThreadPoolExecutor(max_workers=5000) as executor:
        executor.map(increment_counter, fake_data)

This code is quite similar to the structure you used in the threading example above. The difference is that each of the threads is accessing the same global variable counter and incrementing it. Counter is not protected in any way, so it is not thread-safe.

In order to increment counter, each of the threads needs to read the current value, add one to it, and the save that value back to the variable. That happens in this line: counter += 1.

Because the operating system knows nothing about your code and can swap threads at any point in the execution, it’s possible for this swap to happen after a thread has read the value but before it has had the chance to write it back. If the new code that is running modifies counter as well, then the first thread has a stale copy of the data and trouble will ensue.

As you can imagine, hitting this exact situation is fairly rare. You can run this program thousands of times and never see the problem. That’s what makes this type of problem quite difficult to debug as it can be quite hard to reproduce and can cause random-looking errors to show up.

As a further example, I want to remind you that requests.Session() is not thread-safe. This means that there are places where the type of interaction described above could happen if multiple threads use the same Session. I bring this up not to cast aspersions on requests but rather to point out that these are difficult problems to resolve.

`asyncio` Version

Before you jump into examining the asyncio example code, let’s talk more about how asyncio works.

asyncio Basics

This will be a simplified version of asycio. There are many details that are glossed over here, but it still conveys the idea of how it works.

The general concept of asyncio is that a single Python object, called the event loop, controls how and when each task gets run. The event loop is aware of each task and knows what state it’s in. In reality, there are many states that tasks could be in, but for now let’s imagine a simplified event loop that just has two states.

The ready state will indicate that a task has work to do and is ready to be run, and the waiting state means that the task is waiting for some external thing to finish, such as a network operation.

Your simplified event loop maintains two lists of tasks, one for each of these states. It selects one of the ready tasks and starts it back to running. That task is in complete control until it cooperatively hands the control back to the event loop.

When the running task gives control back to the event loop, the event loop places that task into either the ready or waiting list and then goes through each of the tasks in the waiting list to see if it has become ready by an I/O operation completing. It knows that the tasks in the ready list are still ready because it knows they haven’t run yet.

Once all of the tasks have been sorted into the right list again, the event loop picks the next task to run, and the process repeats. Your simplified event loop picks the task that has been waiting the longest and runs that. This process repeats until the event loop is finished.

An important point of asyncio is that the tasks never give up control without intentionally doing so. They never get interrupted in the middle of an operation. This allows us to share resources a bit more easily in asyncio than in threading. You don’t have to worry about making your code thread-safe.

That’s a high-level view of what’s happening with asyncio. If you want more detail, this StackOverflow answer provides some good details if you want to dig deeper.

async and await

Now let’s talk about two new keywords that were added to Python: async and await. In light of the discussion above, you can view await as the magic that allows the task to hand control back to the event loop. When your code awaits a function call, it’s a signal that the call is likely to be something that takes a while and that the task should give up control.

It’s easiest to think of async as a flag to Python telling it that the function about to be defined uses await. There are some cases where this is not strictly true, like asynchronous generators, but it holds for many cases and gives you a simple model while you’re getting started.

One exception to this that you’ll see in the next code is the async with statement, which creates a context manager from an object you would normally await. While the semantics are a little different, the idea is the same: to flag this context manager as something that can get swapped out.

As I’m sure you can imagine, there’s some complexity in managing the interaction between the event loop and the tasks. For developers starting out with asyncio, these details aren’t important, but you do need to remember that any function that calls await needs to be marked with async. You’ll get a syntax error otherwise.

Back to Code

Now that you’ve got a basic understanding of what asyncio is, let’s walk through the asyncio version of the example code and figure out how it works. Note that this version adds aiohttp. You should run pip install aiohttp before running it:

import asyncio
import time
import aiohttp


async def download_site(session, url):
    async with session.get(url) as response:
        print("Read {0} from {1}".format(response.content_length, url))


async def download_all_sites(sites):
    async with aiohttp.ClientSession() as session:
        tasks = []
        for url in sites:
            task = asyncio.ensure_future(download_site(session, url))
            tasks.append(task)
        await asyncio.gather(*tasks, return_exceptions=True)


if __name__ == "__main__":
    sites = [
        "http://www.jython.org",
        "http://olympus.realpython.org/dice",
    ] * 80
    start_time = time.time()
    asyncio.get_event_loop().run_until_complete(download_all_sites(sites))
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} sites in {duration} seconds")

This version is a bit more complex than the previous two. It has a similar structure, but there’s a bit more work setting up the tasks than there was creating the ThreadPoolExecutor. Let’s start at the top of the example.

download_site()

download_site() at the top is almost identical to the threading version with the exception of the async keyword on the function definition line and the async with keywords when you actually call session.get(). You’ll see later why Session can be passed in here rather than using thread-local storage.

download_all_sites()

download_all_sites() is where you will see the biggest change from the threading example.

You can share the session across all tasks, so the session is created here as a context manager. The tasks can share the session because they are all running on the same thread. There is no way one task could interrupt another while the session is in a bad state.

Inside that context manager, it creates a list of tasks using asyncio.ensure_future(), which also takes care of starting them. Once all the tasks are created, this function uses asyncio.gather() to keep the session context alive until all of the tasks have completed.

The threading code does something similar to this, but the details are conveniently handled in the ThreadPoolExecutor. There currently is not an AsyncioPoolExecutor class.

There is one small but important change buried in the details here, however. Remember how we talked about the number of threads to create? It wasn’t obvious in the threading example what the optimal number of threads was.

One of the cool advantages of asyncio is that it scales far better than threading. Each task takes far fewer resources and less time to create than a thread, so creating and running more of them works well. This example just creates a separate task for each site to download, which works out quite well.

__main__

Finally, the nature of asyncio means that you have to start up the event loop and tell it which tasks to run. The __main__ section at the bottom of the file contains the code to get_event_loop() and then run_until_complete(). If nothing else, they’ve done an excellent job in naming those functions.

If you’ve updated to Python 3.7, the Python core developers simplified this syntax for you. Instead of the asyncio.get_event_loop().run_until_complete() tongue-twister, you can just use asyncio.run().

Why the asyncio Version Rocks

It’s really fast! In the tests on my machine, this was the fastest version of the code by a good margin:

$ ./io_asyncio.py
   [most output skipped]
Downloaded 160 in 2.5727896690368652 seconds

The execution timing diagram looks quite similar to what’s happening in the threading example. It’s just that the I/O requests are all done by the same thread:

The lack of a nice wrapper like the ThreadPoolExecutor makes this code a bit more complex than the threading example. This is a case where you have to do a little extra work to get much better performance.

Also there’s a common argument that having to add async and await in the proper locations is an extra complication. To a small extent, that is true. The flip side of this argument is that it forces you to think about when a given task will get swapped out, which can help you create a better, faster, design.

The scaling issue also looms large here. Running the threading example above with a thread for each site is noticeably slower than running it with a handful of threads. Running the asyncio example with hundreds of tasks didn’t slow it down at all.

The Problems With the asyncio Version

There are a couple of issues with asyncio at this point. You need special async versions of libraries to gain the full advantage of asycio. Had you just used requests for downloading the sites, it would have been much slower because requests is not designed to notify the event loop that it’s blocked. This issue is getting smaller and smaller as time goes on and more libraries embrace asyncio.

Another, more subtle, issue is that all of the advantages of cooperative multitasking get thrown away if one of the tasks doesn’t cooperate. A minor mistake in code can cause a task to run off and hold the processor for a long time, starving other tasks that need running. There is no way for the event loop to break in if a task does not hand control back to it.

With that in mind, let’s step up to a radically different approach to concurrency, multiprocessing.

`multiprocessing` Version

Unlike the previous approaches, the multiprocessing version of the code takes full advantage of the multiple CPUs that your cool, new computer has. Or, in my case, that my clunky, old laptop has. Let’s start with the code:

import requests
import multiprocessing
import time

session = None


def set_global_session():
    global session
    if not session:
        session = requests.Session()


def download_site(url):
    with session.get(url) as response:
        name = multiprocessing.current_process().name
        print(f"{name}:Read {len(response.content)} from {url}")


def download_all_sites(sites):
    with multiprocessing.Pool(initializer=set_global_session) as pool:
        pool.map(download_site, sites)


if __name__ == "__main__":
    sites = [
        "http://www.jython.org",
        "http://olympus.realpython.org/dice",
    ] * 80
    start_time = time.time()
    download_all_sites(sites)
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} in {duration} seconds")

This is much shorter than the asyncio example and actually looks quite similar to the threading example, but before we dive into the code, let’s take a quick tour of what multiprocessing does for you.

multiprocessing in a Nutshell

Up until this point, all of the examples of concurrency in this article run only on a single CPU or core in your computer. The reasons for this have to do with the current design of CPython and something called the Global Interpreter Lock, or GIL.

This article won’t dive into the hows and whys of the GIL. It’s enough for now to know that the synchronous, threading, and asyncio versions of this example all run on a single CPU.

multiprocessing in the standard library was designed to break down that barrier and run your code across multiple CPUs. At a high level, it does this by creating a new instance of the Python interpreter to run on each CPU and then farming out part of your program to run on it.

As you can imagine, bringing up a separate Python interpreter is not as fast as starting a new thread in the current Python interpreter. It’s a heavyweight operation and comes with some restrictions and difficulties, but for the correct problem, it can make a huge difference.

multiprocessing Code

The code has a few small changes from our synchronous version. The first one is in download_all_sites(). Instead of simply calling download_site() repeatedly, it creates a multiprocessing.Pool object and has it map download_site to the iterable sites. This should look familiar from the threading example.

What happens here is that the Pool creates a number of separate Python interpreter processes and has each one run the specified function on some of the items in the iterable, which in our case is the list of sites. The communication between the main process and the other processes is handled by the multiprocessing module for you.

The line that creates Pool is worth your attention. First off, it does not specify how many processes to create in the Pool, although that is an optional parameter. By default, multiprocessing.Pool() will determine the number of CPUs in your computer and match that. This is frequently the best answer, and it is in our case.

For this problem, increasing the number of processes did not make things faster. It actually slowed things down because the cost for setting up and tearing down all those processes was larger than the benefit of doing the I/O requests in parallel.

Next we have the initializer=set_global_session part of that call. Remember that each process in our Pool has its own memory space. That means that they cannot share things like a Session object. You don’t want to create a new Session each time the function is called, you want to create one for each process.

The initializer function parameter is built for just this case. There is not a way to pass a return value back from the initializer to the function called by the process download_site(), but you can initialize a global session variable to hold the single session for each process. Because each process has its own memory space, the global for each one will be different.

That’s really all there is to it. The rest of the code is quite similar to what you’ve seen before.

Why the multiprocessing Version Rocks

The multiprocessing version of this example is great because it’s relatively easy to set up and requires little extra code. It also takes full advantage of the CPU power in your computer. The execution timing diagram for this code looks like this:

The Problems With the multiprocessing Version

This version of the example does require some extra setup, and the global session object is strange. You have to spend some time thinking about which variables will be accessed in each process.

Finally, it is clearly slower than the asyncio and threading versions in this example:

$ ./io_mp.py
    [most output skipped]
Downloaded 160 in 5.718175172805786 seconds

That’s not surprising, as I/O-bound problems are not really why multiprocessing exists. You’ll see more as you step into the next section and look at CPU-bound examples.

How to Speed Up a CPU-Bound Program

Let’s shift gears here a little bit. The examples so far have all dealt with an I/O-bound problem. Now, you’ll look into a CPU-bound problem. As you saw, an I/O-bound problem spends most of its time waiting for external operations, like a network call, to complete. A CPU-bound problem, on the other hand, does few I/O operations, and its overall execution time is a factor of how fast it can process the required data.

For the purposes of our example, we’ll use a somewhat silly function to create something that takes a long time to run on the CPU. This function computes the sum of the squares of each number from 0 to the passed-in value:

def cpu_bound(number):
    return sum(i * i for i in range(number))

You’ll be passing in large numbers, so this will take a while. Remember, this is just a placeholder for your code that actually does something useful and requires significant processing time, like computing the roots of equations or sorting a large data structure.

CPU-Bound Synchronous Version

Now let’s look at the non-concurrent version of the example:

import time


def cpu_bound(number):
    return sum(i * i for i in range(number))


def find_sums(numbers):
    for number in numbers:
        cpu_bound(number)


if __name__ == "__main__":
    numbers = [5_000_000 + x for x in range(20)]

    start_time = time.time()
    find_sums(numbers)
    duration = time.time() - start_time
    print(f"Duration {duration} seconds")

This code calls cpu_bound() 20 times with a different large number each time. It does all of this on a single thread in a single process on a single CPU. The execution timing diagram looks like this:

Unlike the I/O-bound examples, the CPU-bound examples are usually fairly consistent in their run times. This one takes about 7.8 seconds on my machine:

$ ./cpu_non_concurrent.py
Duration 7.834432125091553 seconds

Clearly we can do better than this. This is all running on a single CPU with no concurrency. Let’s see what we can do to make it better.

`threading` and `asyncio` Versions

How much do you think rewriting this code using threading or asyncio will speed this up?

If you answered “Not at all,” give yourself a cookie. If you answered, “It will slow it down,” give yourself two cookies.

Here’s why: In your I/O-bound example above, much of the overall time was spent waiting for slow operations to finish. threading and asyncio sped this up by allowing you to overlap the times you were waiting instead of doing them sequentially.

On a CPU-bound problem, however, there is no waiting. The CPU is cranking away as fast as it can to finish the problem. In Python, both threads and tasks run on the same CPU in the same process. That means that the one CPU is doing all of the work of the non-concurrent code plus the extra work of setting up threads or tasks. It takes more than 10 seconds:

$ ./cpu_threading.py
Duration 10.407078266143799 seconds

I’ve written up a threading version of this code and placed it with the other example code in the GitHub repo so you can go test this yourself. Let’s not look at that just yet, however.

CPU-Bound `multiprocessing` Version

Now you’ve finally reached where multiprocessing really shines. Unlike the other concurrency libraries, multiprocessing is explicitly designed to share heavy CPU workloads across multiple CPUs. Here’s what its execution timing diagram looks like:

Here’s what the code looks like:

import multiprocessing
import time


def cpu_bound(number):
    return sum(i * i for i in range(number))


def find_sums(numbers):
    with multiprocessing.Pool() as pool:
        pool.map(cpu_bound, numbers)


if __name__ == "__main__":
    numbers = [5_000_000 + x for x in range(20)]

    start_time = time.time()
    find_sums(numbers)
    duration = time.time() - start_time
    print(f"Duration {duration} seconds")

Little of this code had to change from the non-concurrent version. You had to import multiprocessing and then just change from looping through the numbers to creating a multiprocessing.Pool object and using its .map() method to send individual numbers to worker-processes as they become free.

This was just what you did for the I/O-bound multiprocessing code, but here you don’t need to worry about the Session object.

As mentioned above, the processes optional parameter to the multiprocessing.Pool() constructor deserves some attention. You can specify how many Process objects you want created and managed in the Pool. By default, it will determine how many CPUs are in your machine and create a process for each one. While this works great for our simple example, you might want to have a little more control in a production environment.

Also, as we mentioned in the first section about threading, the multiprocessing.Pool code is built upon building blocks like Queue and Semaphore that will be familiar to those of you who have done multithreaded and multiprocessing code in other languages.

Why the multiprocessing Version Rocks

The multiprocessing version of this example is great because it’s relatively easy to set up and requires little extra code. It also takes full advantage of the CPU power in your computer.

Hey, that’s exactly what I said the last time we looked at multiprocessing. The big difference is that this time it is clearly the best option. It takes 2.5 seconds on my machine:

$ ./cpu_mp.py
Duration 2.5175397396087646 seconds

That’s much better than we saw with the other options.

The Problems With the multiprocessing Version

There are some drawbacks to using multiprocessing. They don’t really show up in this simple example, but splitting your problem up so each processor can work independently can sometimes be difficult.

Also, many solutions require more communication between the processes. This can add some complexity to your solution that a non-concurrent program would not need to deal with.

When to Use Concurrency

You’ve covered a lot of ground here, so let’s review some of the key ideas and then discuss some decision points that will help you determine which, if any, concurrency module you want to use in your project.

The first step of this process is deciding if you should use a concurrency module. While the examples here make each of the libraries look pretty simple, concurrency always comes with extra complexity and can often result in bugs that are difficult to find.

Hold out on adding concurrency until you have a known performance issue and then determine which type of concurrency you need. As Donald Knuth has said, “Premature optimization is the root of all evil (or at least most of it) in programming.”

Once you’ve decided that you should optimize your program, figuring out if your program is CPU-bound or I/O-bound is a great next step. Remember that I/O-bound programs are those that spend most of their time waiting for something to happen while CPU-bound programs spend their time processing data or crunching numbers as fast as they can.

As you saw, CPU-bound problems only really gain from using multiprocessing. threading and asyncio did not help this type of problem at all.

For I/O-bound problems, there’s a general rule of thumb in the Python community: “Use asyncio when you can, threading when you must.” asyncio can provide the best speed up for this type of program, but sometimes you will require critical libraries that have not been ported to take advantage of asyncio. Remember that any task that doesn’t give up control to the event loop will block all of the other tasks.

Conclusion

You’ve now seen the basic types of concurrency available in Python:

threading
asyncio
multiprocessing

You’ve got the understanding to decide which concurrency method you should use for a given problem, or if you should use any at all! In addition, you’ve achieved a better understanding of some of the problems that can arise when you’re using concurrency.

I hope you’ve learned a lot from this article and that you find a great use for concurrency in your own projects! Be sure to take our “Python Concurrency” quiz linked below to check your learning:

Take the Quiz: Test your knowledge with our interactive “Python Concurrency” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Take the Quiz »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Django Migrations: A Primer

2019-01-09T12:10:21+00:00

Since version 1.7, Django has come with built-in support for database migrations. In Django, database migrations usually go hand in hand with models: whenever you code up a new model, you also generate a migration to create the necessary table in the database. However, migrations can do much more.

You are going to learn how Django Migrations work and how you can get the most out of them over the course of four articles and one video:

Part 1: Django Migrations: A Primer (current article)
Part 2: Digging Deeper into Migrations
Part 3: Data Migrations
Video: Django 1.7 Migrations - primer

In this article, you’ll get comfortable with Django migrations and learn the following:

How to create database tables without writing any SQL
How to automatically modify your database after you changed your models
How to revert changes made to your database

Free Bonus: Click here to get access to a free Django Learning Resources Guide (PDF) that shows you tips and tricks as well as common pitfalls to avoid when building Python + Django web applications.

The Problems That Migrations Solve

If you are new to Django or web development in general, you might not be familiar with the concept of database migrations, and it might not seem obvious why they’re a good idea.

First, let’s quickly define a couple of terms to make sure everybody is on the same page. Django is designed to work with a relational database, stored in a relational database management system like PostgreSQL, MySQL, or SQLite.

In a relational database, data is organized in tables. A database table has a certain number of columns, but it can have any number of rows. Each column has a specific datatype, like a string of a certain maximum length or a positive integer. The description of all tables with their columns and their respective datatypes is called a database schema.

All database systems supported by Django use the language SQL to create, read, update and delete data in a relational database. SQL is also used to create, change, and delete the database tables themselves.

Working directly with SQL can be quite cumbersome, so to make your life easier, Django comes with an object-relational mapper, or ORM for short. The ORM maps the relational database to the world of object oriented programming. Instead of defining database tables in SQL, you write Django models in Python. Your models define database fields, which correspond to the columns in their database tables.

Here’s an example of how a Django model class is mapped to a database table:

But just defining a model class in a Python file does not make a database table magically appear out of nowhere. Creating the database tables to store your Django models is the job of a database migration. Additionally, whenever you make a change to your models, like adding a field, the database has to be changed too. Migrations handle that as well.

Here are a few ways Django migrations make your life easier.

Making Database Changes Without SQL

Without migrations, you would have to connect to your database and type in a bunch of SQL commands or use a graphical tool like PHPMyAdmin to modify the database schema every time you wanted to change your model definition.

In Django, migrations are primarily written in Python, so you don’t have to know any SQL unless you have really advanced use cases.

Avoiding Repetition

Creating a model and then writing SQL to create the database tables for it would be repetitive.

Migrations are generated from your models, making sure you don’t repeat yourself.

Ensuring Model Definitions and the Database Schema in Sync

Usually, you have multiple instances of your database, for example one database for each developer in your team, a database for testing and a database with live data.

Without migrations, you will have to perform any schema changes on each one of your database, and you will have to keep track which changes have already been made to which database.

With Django Migrations, you can easily keep multiple databases in sync with your models.

Tracking Database Schema Change in Version Control

A version control system, like Git is excellent for code, but not so much for database schemas.

As migrations are plain Python in Django, you can put them in a version control system just like any other piece of code.

By now, you’re hopefully convinced that migrations are a useful and powerful tool. Let’s start learning how to unleash that power.

Setting Up a Django Project

Throughout this tutorial, you are going to work on a simple Bitcoin tracker app as an example project.

The first step is to install Django. Here is how you do that on Linux or macOS X using a virtual environment:

$ python3 -m venv env
$ source env/bin/activate
(env) $ $ pip install "Django==2.1.*"
...
Successfully installed Django-2.1.3

Now you have created a new virtual environment and activated it, as well as installed Django in that virtual environment.

Note that on Windows, you would run env/bin/activate.bat instead of source env/bin/activate to activate your virtual environment.

For easier readability, console examples won’t include the (env) part of the prompt from now on.

With Django installed, you can create the project using the following commands:

$ django-admin.py startproject bitcoin_tracker
$ cd bitcoin_tracker
$ python manage.py startapp historical_data

This gives you a simple project and an app called historical_data. You should now have this directory structure:

bitcoin_tracker/
|
├── bitcoin_tracker/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
|
├── historical_data/
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── migrations/
│   │   └── __init__.py
|   |
│   ├── models.py
│   ├── tests.py
│   └── views.py
|
└── manage.py

Within the bitcoin_tracker directory, there are two sub-directories: bitcoin_tracker for project-wide files and historical_data containing files for the app you created.

Now, to create a model, add this class in historical_data/models.py:

class PriceHistory(models.Model):
    date = models.DateTimeField(auto_now_add=True)
    price = models.DecimalField(max_digits=7, decimal_places=2)
    volume = models.PositiveIntegerField()

This is the basic model to keep track of Bitcoin prices.

Also, don’t forget to add the newly created app to settings.INSTALLED_APPS. Open bitcoin_tracker/settings.py and append historical_data to the list INSTALLED_APPS, like this:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'historical_data',
]

The other settings are fine for this project. This tutorial assumes that your project is configured to use an SQLite database, which is the default.

Creating Migrations

With the model created, the first thing you need to do is create a migration for it. You can do this with the following command:

$ python manage.py makemigrations historical_data
Migrations for 'historical_data':
  historical_data/migrations/0001_initial.py
    - Create model PriceHistory

Note: Specifying the name of the application, historical_data, is optional. Leaving it off creates migrations for all apps.

This creates the migrations file that instructs Django on how to create the database tables for the models defined in your application. Let’s have another look at the directory tree:

bitcoin_tracker/
|
├── bitcoin_tracker/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
|
├── historical_data/
│   ├── migrations/
│   │   ├── 0001_initial.py
│   │   └── __init__.py
|   |
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
|
├── db.sqlite3
└── manage.py

As you can see, the migrations directory now contains a new file: 0001_initial.py.

Note: You might notice that running the makemigrations command also created the file db.sqlite3, which contains your SQLite database.

When you try to access a non-existing SQLite3 database file, it will automatically be created.

This behavior is unique to SQLite3. If you use any other database backend like PostgreSQL or MySQL, you have to create the database yourself before running makemigrations.

You can take a peek at the database with the dbshell management command. In SQLite, the command to list all tables is simply .tables:

$ python manage.py dbshell
SQLite version 3.19.3 2017-06-27 16:48:08
Enter ".help" for usage hints.
sqlite> .tables
sqlite>

The database is still empty. That will change when you apply the migration. Type .quit to exit the SQLite shell.

Applying Migrations

You have now created the migration, but to actually make any changes in the database, you have to apply it with the management command migrate:

$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, historical_data, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying historical_data.0001_initial... OK
  Applying sessions.0001_initial... OK

There is a lot going on here! According to the output, your migration has been successfully applied. But where do all the other migrations come from?

Remember the setting INSTALLED_APPS? Some of the other apps listed there also come with migrations, and the migrate management command applies the migrations for all installed apps by default.

Have another look at the database:

$ python manage.py dbshell
SQLite version 3.19.3 2017-06-27 16:48:08
Enter ".help" for usage hints.
sqlite> .tables
auth_group                    django_admin_log
auth_group_permissions        django_content_type
auth_permission               django_migrations
auth_user                     django_session
auth_user_groups              historical_data_pricehistory
auth_user_user_permissions
sqlite>

Now there are multiple tables. Their names give you an idea of their purpose. The migration that you generated in the previous step has created the historical_data_pricehistory table. Let’s inspect it using the .schema command:

sqlite> .schema --indent historical_data_pricehistory
CREATE TABLE IF NOT EXISTS "historical_data_pricehistory"(
  "id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
  "date" datetime NOT NULL,
  "price" decimal NOT NULL,
  "volume" integer unsigned NOT NULL
);

The .schema command prints out the CREATE statement that you would execute to create the table. The parameter --indent formats it nicely. Even if you are not familiar with SQL syntax, you can see that the schema of the historical_data_pricehistory table reflects the fields of the PriceHistory model.

There is a column for each field and an additional column id for the primary key, which Django creates automatically unless you explicitly specify a primary key in your model.

Here’s what happens if you run the migrate command again:

$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, historical_data, sessions
Running migrations:
  No migrations to apply.

Nothing! Django remembers which migrations have already been applied and does not try to rerun them.

It is worth noting that you can also limit the migrate management command to a single app:

$ python manage.py migrate historical_data
Operations to perform:
 Apply all migrations: historical_data
Running migrations:
 No migrations to apply.

As you can see, Django now only applies migrations for the historical_data app.

When you are running the migrations for the first time, it is a good idea to apply all migrations to ensure your database contains the necessary tables for the features you might take for granted, like user authentication and sessions.

Changing Models

Your models are not set in stone. Your models will change as your Django project gains more features. You might add or remove fields or change their types and options.

When you change the definition of a model, the database tables used to store these models have to be changed too. If your model definitions don’t match your current database schema, you will most likely run into a django.db.utils.OperationalError.

So how do you change the database tables? By creating and applying a migration.

While testing your Bitcoin tracker, you realize that you made a mistake. People are selling fractions of a Bitcoin, so the field volume should be of the type DecimalField instead of PositiveIntegerField.

Let’s change the model to look like this:

class PriceHistory(models.Model):
    date = models.DateTimeField(auto_now_add=True)
    price = models.DecimalField(max_digits=7, decimal_places=2)
    volume = models.DecimalField(max_digits=7, decimal_places=3)

Without migrations, you would have to figure out the SQL syntax to turn a PositiveIntegerField into a DecimalField. Luckily, Django will handle that for you. Just tell it to make migrations:

$ python manage.py makemigrations
Migrations for 'historical_data':
  historical_data/migrations/0002_auto_20181112_1950.py
    - Alter field volume on pricehistory

Note: The name of the migration file (0002_auto_20181112_1950.py) is based on the current time and will be different if you follow along on your system.

Now you apply this migration to your database:

$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, historical_data, sessions
Running migrations:
  Applying historical_data.0002_auto_20181112_1950... OK

The migration has been applied successfully, so you can use dbshell to verify that the changes had an effect:

$ python manage.py dbshell
SQLite version 3.19.3 2017-06-27 16:48:08
Enter ".help" for usage hints.
sqlite> .schema --indent historical_data_pricehistory
CREATE TABLE IF NOT EXISTS "historical_data_pricehistory" (
  "id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
  "date" datetime NOT NULL,
  "price" decimal NOT NULL,
  "volume" decimal NOT NULL
);

If you compare the new schema with the schema you saw earlier, you will notice that the type of the volume column has changed from integer to decimal to reflect the change of the volume field in the model from PositiveIntegerField to DecimalField.

Listing Out Migrations

If you want to know what migrations exist in a Django project, you don’t have to dig trough the migrations directories of your installed apps. You can use the showmigrations command:

$ ./manage.py showmigrations
admin
 [X] 0001_initial
 [X] 0002_logentry_remove_auto_add
 [X] 0003_logentry_add_action_flag_choices
auth
 [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [X] 0007_alter_validators_add_error_messages
 [X] 0008_alter_user_username_max_length
 [X] 0009_alter_user_last_name_max_length
contenttypes
 [X] 0001_initial
 [X] 0002_remove_content_type_name
historical_data
 [X] 0001_initial
 [X] 0002_auto_20181112_1950
sessions
 [X] 0001_initial

This lists all apps in the project and the migrations associated with each app. Also, it will put a big X next to the migrations that have already been applied.

For our little example, the showmigrations command is not particularly exciting, but it comes in handy when you start working on an existing code base or work in a team where you are not the only person adding migrations.

Unapplying Migrations

Now you know how to make changes to your database schema by creating and applying migrations. At some point, you might want to undo changes and switch back to an earlier database schema because you:

Want to test a migration a colleague wrote
Realize that a change you made was a bad idea
Work on multiple features with different database changes in parallel
Want to restore a backup that was created when the database still had an older schema

Luckily, migrations don’t have to be a one-way street. In many cases, the effects of a migration can be undone by unapplying a migration. To unapply a migration, you have to call migrate with the name of the app and the name of the migration before the migration you want to unapply.

If you want to revert the migration 0002_auto_20181112_1950 in your historical_data app, you have to pass 0001_initial as an argument to the migrate command:

$ python manage.py migrate historical_data 0001_initial
Operations to perform:
  Target specific migration: 0001_initial, from historical_data
Running migrations:
  Rendering model states... DONE
  Unapplying historical_data.0002_auto_20181112_1950... OK

The migration has been unapplied, meaning that the changes to the database have been reversed.

Unapplying a migration does not remove its migration file. The next time you run the migrate command, the migration will be applied again.

Caution: Don’t confuse unapplying migrations with the undo operation you are used to from your favorite text editor.

Not all database operations can be completely reverted. If you remove a field from a model, create a migration, and apply it, Django will remove the respective column from the database.

Unapplying that migration will re-create the column, but it won’t bring back the data that was stored in that column!

When you’re dealing with migration names, Django saves you a few keystrokes by not forcing you to spell out the whole name of the migration. It needs just enough of the name to identify it uniquely.

In the previous example, it would have been enough to run python manage.py migrate historical_data 0001.

Naming Migrations

In the above example, Django came up with a name for the migration based on the timestamp—something like *0002_auto_20181112_1950. If you’re not happy with that, then you can use the --name parameter to provide a custom name (without the .py extension).

To try that out, you first have to remove the old migration. You have already unapplied it, so you can safely delete the file:

$ rm historical_data/migrations/0002_auto_20181112_1950.py

Now you can recreate it with a more descriptive name:

$ ./manage.py makemigrations historical_data --name switch_to_decimals

This will create the same migration as before except with the new name of 0002_switch_to_decimals.

Conclusion

You covered quite a bit of ground in this tutorial and learned the fundamentals of Django migrations.

To recap, the basic steps to use Django migrations look like this:

Create or update a model
Run ./manage.py makemigrations <app_name>
Run ./manage.py migrate to migrate everything or ./manage.py migrate <app_name> to migrate an individual app
Repeat as necessary

That’s it! This workflow will work the majority of the time, but if things don’t work out as expected, you also know how to list and unapply migrations.

If you previously created and modified your database tables with hand-written SQL, you have now become much more efficient by delegating this work to Django migrations.

In the next tutorial in this series, you will dig deeper into the topic and learn how Django Migrations work under the hood.

Cheers!

Video

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Ultimate Guide to Python Type Checking

2019-01-07T14:00:00+00:00

In this guide, you will get a look into Python type checking. Traditionally, types have been handled by the Python interpreter in a flexible but implicit way. Recent versions of Python allow you to specify explicit type hints that can be used by different tools to help you develop your code more efficiently.

In this tutorial, you’ll learn about the following:

Type annotations and type hints
Adding static types to code, both your code and the code of others
Running a static type checker
Enforcing types at runtime

This is a comprehensive guide that will cover a lot of ground. If you want to just get a quick glimpse of how type hints work in Python, and see whether type checking is something you would include in your code, you don’t need to read all of it. The two sections Hello Types and Pros and Cons will give you a taste of how type checking works and recommendations about when it’ll be useful.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Type Systems

All programming languages include some kind of type system that formalizes which categories of objects it can work with and how those categories are treated. For instance, a type system can define a numerical type, with 42 as one example of an object of numerical type.

Dynamic Typing

Python is a dynamically typed language. This means that the Python interpreter does type checking only as code runs, and that the type of a variable is allowed to change over its lifetime. The following dummy examples demonstrate that Python has dynamic typing:

>>>

>>> if False:
...     1 + "two"  # This line never runs, so no TypeError is raised
... else:
...     1 + 2
...
3

>>> 1 + "two"  # Now this is type checked, and a TypeError is raised
TypeError: unsupported operand type(s) for +: 'int' and 'str'

In the first example, the branch 1 + "two" never runs so it’s never type checked. The second example shows that when 1 + "two" is evaluated it raises a TypeError since you can’t add an integer and a string in Python.

Next, let’s see if variables can change type:

>>>

>>> thing = "Hello"
>>> type(thing)
<class 'str'>

>>> thing = 28.1
>>> type(thing)
<class 'float'>

type() returns the type of an object. These examples confirm that the type of thing is allowed to change, and Python correctly infers the type as it changes.

Static Typing

The opposite of dynamic typing is static typing. Static type checks are performed without running the program. In most statically typed languages, for instance C and Java, this is done as your program is compiled.

With static typing, variables generally are not allowed to change types, although mechanisms for casting a variable to a different type may exist.

Let’s look at a quick example from a statically typed language. Consider the following Java snippet:

String thing;
thing = "Hello";

The first line declares that the variable name thing is bound to the String type at compile time. The name can never be rebound to another type. In the second line, thing is assigned a value. It can never be assigned a value that is not a String object. For instance, if you were to later say thing = 28.1f the compiler would raise an error because of incompatible types.

Python will always remain a dynamically typed language. However, PEP 484 introduced type hints, which make it possible to also do static type checking of Python code.

Unlike how types work in most other statically typed languages, type hints by themselves don’t cause Python to enforce types. As the name says, type hints just suggest types. There are other tools, which you’ll see later, that perform static type checking using type hints.

Duck Typing

Another term that is often used when talking about Python is duck typing. This moniker comes from the phrase “if it walks like a duck and it quacks like a duck, then it must be a duck” (or any of its variations).

Duck typing is a concept related to dynamic typing, where the type or the class of an object is less important than the methods it defines. Using duck typing you do not check types at all. Instead you check for the presence of a given method or attribute.

As an example, you can call len() on any Python object that defines a .__len__() method:

>>>

>>> class TheHobbit:
...     def __len__(self):
...         return 95022
...
>>> the_hobbit = TheHobbit()
>>> len(the_hobbit)
95022

Note that the call to len() gives the return value of the .__len__() method. In fact, the implementation of len() is essentially equivalent to the following:

def len(obj):
    return obj.__len__()

In order to call len(obj), the only real constraint on obj is that it must define a .__len__() method. Otherwise, the object can be of types as different as str, list, dict, or TheHobbit.

Duck typing is somewhat supported when doing static type checking of Python code, using structural subtyping. You’ll learn more about duck typing later.

Hello Types

In this section you’ll see how to add type hints to a function. The following function turns a text string into a headline by adding proper capitalization and a decorative line:

def headline(text, align=True):
    if align:
        return f"{text.title()}\n{'-' * len(text)}"
    else:
        return f" {text.title()} ".center(50, "o")

By default the function returns the headline left aligned with an underline. By setting the align flag to False you can alternatively have the headline be centered with a surrounding line of o:

>>>

>>> print(headline("python type checking"))
Python Type Checking
--------------------

>>> print(headline("python type checking", align=False))
oooooooooooooo Python Type Checking oooooooooooooo

It’s time for our first type hints! To add information about types to the function, you simply annotate its arguments and return value as follows:

def headline(text: str, align: bool = True) -> str:
    ...

The text: str syntax says that the text argument should be of type str. Similarly, the optional align argument should have type bool with the default value True. Finally, the -> str notation specifies that headline() will return a string.

In terms of style, PEP 8 recommends the following:

Use normal rules for colons, that is, no space before and one space after a colon: text: str.
Use spaces around the = sign when combining an argument annotation with a default value: align: bool = True.
Use spaces around the -> arrow: def headline(...) -> str.

Adding type hints like this has no runtime effect: they are only hints and are not enforced on their own. For instance, if we use a wrong type for the (admittedly badly named) align argument, the code still runs without any problems or warnings:

>>>

>>> print(headline("python type checking", align="left"))
Python Type Checking
--------------------

Note: The reason this seemingly works is that the string "left" compares as truthy. Using align="center" would not have the desired effect as "center" is also truthy.

To catch this kind of error you can use a static type checker. That is, a tool that checks the types of your code without actually running it in the traditional sense.

You might already have such a type checker built into your editor. For instance PyCharm immediately gives you a warning:

The most common tool for doing type checking is Mypy though. You’ll get a short introduction to Mypy in a moment, while you can learn much more about how it works later.

If you don’t already have Mypy on your system, you can install it using pip:

$ pip install mypy

Put the following code in a file called headlines.py:

 1 # headlines.py
 2 
 3 def headline(text: str, align: bool = True) -> str:
 4     if align:
 5         return f"{text.title()}\n{'-' * len(text)}"
 6     else:
 7         return f" {text.title()} ".center(50, "o")
 8 
 9 print(headline("python type checking"))
10 print(headline("use mypy", align="center"))

This is essentially the same code you saw earlier: the definition of headline() and two examples that are using it.

Now run Mypy on this code:

$ mypy headlines.py
headlines.py:10: error: Argument "align" to "headline" has incompatible
                        type "str"; expected "bool"

Based on the type hints, Mypy is able to tell us that we are using the wrong type on line 10.

To fix the issue in the code you should change the value of the align argument you are passing in. You might also rename the align flag to something less confusing:

 1 # headlines.py
 2 
 3 def headline(text: str, centered: bool = False):
 4     if not centered:
 5         return f"{text.title()}\n{'-' * len(text)}"
 6     else:
 7         return f" {text.title()} ".center(50, "o")
 8 
 9 print(headline("python type checking"))
10 print(headline("use mypy", centered=True))

Here you’ve changed align to centered, and correctly used a boolean value for centered when calling headline(). The code now passes Mypy:

$ mypy headlines.py
$

No output from Mypy means that no type errors were detected. Furthermore, when you run the code you see the expected output:

$ python headlines.py
Python Type Checking
--------------------
oooooooooooooooooooo Use Mypy oooooooooooooooooooo

The first headline is aligned to the left, while the second one is centered.

Pros and Cons

The previous section gave you a little taste of what type checking in Python looks like. You also saw an example of one of the advantages of adding types to your code: type hints help catch certain errors. Other advantages include:

Type hints help document your code. Traditionally, you would use docstrings if you wanted to document the expected types of a function’s arguments. This works, but as there is no standard for docstrings (despite PEP 257 they can’t be easily used for automatic checks.
Type hints improve IDEs and linters. They make it much easier to statically reason about your code. This in turn allows IDEs to offer better code completion and similar features. With the type annotation, PyCharm knows that text is a string, and can give specific suggestions based on this:
Type hints help you build and maintain a cleaner architecture. The act of writing type hints forces you to think about the types in your program. While the dynamic nature of Python is one of its great assets, being conscious about relying on duck typing, overloaded methods, or multiple return types is a good thing.

Of course, static type checking is not all peaches and cream. There are also some downsides you should consider:

Type hints take developer time and effort to add. Even though it probably pays off in spending less time debugging, you will spend more time entering code.
Type hints work best in modern Pythons. Annotations were introduced in Python 3.0, and it’s possible to use type comments in Python 2.7. Still, improvements like variable annotations and postponed evaluation of type hints mean that you’ll have a better experience doing type checks using Python 3.6 or even Python 3.7.
Type hints introduce a slight penalty in start-up time. If you need to use the typing module the import time may be significant, especially in short scripts.

You’ll later learn about the typing module, and how it’s necessary in most cases when you add type hints. Importing modules necessarily take some time, but how much?

To get some idea about this, create two files: empty_file.py should be an empty file, while import_typing.py should contain the following line:

import typing

On Linux it’s quite easy to check how much time the typing import takes using the perf utility:

$ perf stat -r 1000 python3.6 import_typing.py

 Performance counter stats for 'python3.6 import_typing.py' (1000 runs):

 [ ... extra information hidden for brevity ... ]

       0.045161650 seconds time elapsed    ( +-  0.77% )

So running the import typing.py script takes about 45 milliseconds. Of course this is not all time spent on importing typing. Some of this is overhead in starting the Python interpreter, so let’s compare to running Python on an empty file:

$ perf stat -r 1000 python3.6 empty_file.py

 Performance counter stats for 'python3.6 empty_file.py' (1000 runs):

 [ ... extra information hidden for brevity ... ]

       0.028077845 seconds time elapsed    ( +-  0.49% )

Based on this test, the import of the typing module takes around 17 milliseconds on Python 3.6.

One of the advertised improvements in Python 3.7 is faster startup. Let’s see if the results are different:

$ perf stat -r 1000 python3.7 import_typing.py
 [...]
       0.025979806 seconds time elapsed    ( +-  0.31% )

$ perf stat -r 1000 python3.7 empty_file.py
 [...]
       0.020002505 seconds time elapsed    ( +-  0.30% )

Indeed, the general startup time is reduced by about 8 milliseconds, and the time to import typing is down from 17 to around 6 milliseconds—almost 3 times faster.

Using timeit

There are similar tools on other platforms. Python itself comes with the timeit module in the standard library. Typically, we would directly use timeit for the timings above. However, timeit struggles to time imports reliably because Python is clever about importing modules only once. Consider the following example:

$ python3.6 -m timeit "import typing"
10000000 loops, best of 3: 0.134 usec per loop

While you get a result, you should be suspicious about the result: 0.1 microsecond is more than 100000 times faster than what perf measured! What timeit has actually done is to run the import typing statement 30 million times, with Python actually only importing typing once.

To get reasonable results you can tell timeit to only run once:

$ python3.6 -m timeit -n 1 -r 1 "import typing"
1 loops, best of 1: 9.77 msec per loop
$ python3.7 -m timeit -n 1 -r 1 "import typing"
1 loop, best of 1: 1.97 msec per loop

These results are on the same scale as the results from perf above. However, since these are based on only one execution of the code, they are not as reliable as those based on multiple runs.

The conclusion in both these cases is that importing typing takes a few milliseconds. For the majority of programs and scripts you write this will probably not be an issue.

The New importtime Option

In Python 3.7 there is also a new command line option that can be used to figure out how much time imports take. Using -X importtime you’ll get a report about all imports that are made:

$ python3.7 -X importtime import_typing.py
import time: self [us] | cumulative | imported package
[ ... some information hidden for brevity ... ]
import time:       358 |        358 | zipimport
import time:      2107 |      14610 | site
import time:       272 |        272 |   collections.abc
import time:       664 |       3058 |   re
import time:      3044 |       6373 | typing

This shows a similar result. Importing typing takes about 6 milliseconds. If you’ll read the report closely you can notice that around half of this time is spent on importing the collections.abc and re modules which typing depends on.

So, should you use static type checking in your own code? Well, it’s not an all-or-nothing question. Luckily, Python supports the concept of gradual typing. This means that you can gradually introduce types into your code. Code without type hints will be ignored by the static type checker. Therefore, you can start adding types to critical components, and continue as long as it adds value to you.

Looking at the lists above of pros and cons you’ll notice that adding types will have no effect on your running program or the users of your program. Type checking is meant to make your life as a developer better and more convenient.

A few rules of thumb on whether to add types to your project are:

If you are just beginning to learn Python, you can safely wait with type hints until you have more experience.
Type hints add little value in short throw-away scripts.
In libraries that will be used by others, especially ones published on PyPI, type hints add a lot of value. Other code using your libraries need these type hints to be properly type checked itself. For examples of projects using type hints see cursive_re, black, our own Real Python Reader, and Mypy itself.
In bigger projects, type hints help you understand how types flow through your code, and are highly recommended. Even more so in projects where you cooperate with others.

In his excellent article The State of Type Hints in Python Bernát Gábor recommends that “type hints should be used whenever unit tests are worth writing.” Indeed, type hints play a similar role as tests in your code: they help you as a developer write better code.

Hopefully you now have an idea about how type checking works in Python and whether it’s something you would like to employ in your own projects.

In the rest of this guide, we’ll go into more detail about the Python type system, including how you run static type checkers (with particular focus on Mypy), how you type check code that uses libraries without type hints, and how you use annotations at runtime.

Annotations

Annotations were introduced in Python 3.0, originally without any specific purpose. They were simply a way to associate arbitrary expressions to function arguments and return values.

Years later, PEP 484 defined how to add type hints to your Python code, based off work that Jukka Lehtosalo had done on his Ph.D. project—Mypy. The main way to add type hints is using annotations. As type checking is becoming more and more common, this also means that annotations should mainly be reserved for type hints.

The next sections explain how annotations work in the context of type hints.

Function Annotations

For functions, you can annotate arguments and the return value. This is done as follows:

def func(arg: arg_type, optarg: arg_type = default) -> return_type:
    ...

For arguments the syntax is argument: annotation, while the return type is annotated using -> annotation. Note that the annotation must be a valid Python expression.

The following simple example adds annotations to a function that calculates the circumference of a circle:

import math

def circumference(radius: float) -> float:
    return 2 * math.pi * radius

When running the code, you can also inspect the annotations. They are stored in a special .__annotations__ attribute on the function:

>>>

>>> circumference(1.23)
7.728317927830891

>>> circumference.__annotations__
{'radius': <class 'float'>, 'return': <class 'float'>}

Sometimes you might be confused by how Mypy is interpreting your type hints. For those cases there are special Mypy expressions: reveal_type() and reveal_locals(). You can add these to your code before running Mypy, and Mypy will dutifully report which types it has inferred. As an example, save the following code to reveal.py:

 1 # reveal.py
 2 
 3 import math
 4 reveal_type(math.pi)
 5 
 6 radius = 1
 7 circumference = 2 * math.pi * radius
 8 reveal_locals()

Next, run this code through Mypy:

$ mypy reveal.py
reveal.py:4: error: Revealed type is 'builtins.float'

reveal.py:8: error: Revealed local types are:
reveal.py:8: error: circumference: builtins.float
reveal.py:8: error: radius: builtins.int

Even without any annotations Mypy has correctly inferred the types of the built-in math.pi, as well as our local variables radius and circumference.

Note: The reveal expressions are only meant as a tool helping you add types and debug your type hints. If you try to run the reveal.py file as a Python script it will crash with a NameError since reveal_type() is not a function known to the Python interpreter.

If Mypy says that “Name ‘reveal_locals‘ is not defined” you might need to update your Mypy installation. The reveal_locals() expression is available in Mypy version 0.610 and later.

Variable Annotations

In the definition of circumference() in the previous section, you only annotated the arguments and the return value. You did not add any annotations inside the function body. More often than not, this is enough.

However, sometimes the type checker needs help in figuring out the types of variables as well. Variable annotations were defined in PEP 526 and introduced in Python 3.6. The syntax is the same as for function argument annotations:

pi: float = 3.142

def circumference(radius: float) -> float:
    return 2 * pi * radius

The variable pi has been annotated with the float type hint.

Note: Static type checkers are more than able to figure out that 3.142 is a float, so in this example the annotation of pi is not necessary. As you learn more about the Python type system, you’ll see more relevant examples of variable annotations.

Annotations of variables are stored in the module level __annotations__ dictionary:

>>>

>>> circumference(1)
6.284

>>> __annotations__
{'pi': <class 'float'>}

You’re allowed to annotate a variable without giving it a value. This adds the annotation to the __annotations__ dictionary, while the variable remains undefined:

>>>

>>> nothing: str
>>> nothing
NameError: name 'nothing' is not defined

>>> __annotations__
{'nothing': <class 'str'>}

Since no value was assigned to nothing, the name nothing is not yet defined.

Type Comments

As mentioned, annotations were introduced in Python 3, and they’ve not been backported to Python 2. This means that if you’re writing code that needs to support legacy Python, you can’t use annotations.

Instead, you can use type comments. These are specially formatted comments that can be used to add type hints compatible with older code. To add type comments to a function you do something like this:

import math

def circumference(radius):
    # type: (float) -> float
    return 2 * math.pi * radius

The type comments are just comments, so they can be used in any version of Python.

Type comments are handled directly by the type checker, so these types are not available in the __annotations__ dictionary:

>>>

>>> circumference.__annotations__
{}

A type comment must start with the type: literal, and be on the same or the following line as the function definition. If you want to annotate a function with several arguments, you write each type separated by comma:

def headline(text, width=80, fill_char="-"):
    # type: (str, int, str) -> str
    return f" {text.title()} ".center(width, fill_char)

print(headline("type comments work", width=40))

You are also allowed to write each argument on a separate line with its own annotation:

 1 # headlines.py
 2 
 3 def headline(
 4     text,           # type: str
 5     width=80,       # type: int
 6     fill_char="-",  # type: str
 7 ):                  # type: (...) -> str
 8     return f" {text.title()} ".center(width, fill_char)
 9 
10 print(headline("type comments work", width=40))

Run the example through Python and Mypy:

$  python headlines.py
---------- Type Comments Work ----------

$ mypy headline.py
$

If you have errors, for instance if you happened to call headline() with width="full" on line 10, Mypy will tell you:

$ mypy headline.py
headline.py:10: error: Argument "width" to "headline" has incompatible
                       type "str"; expected "int"

You can also add type comments to variables. This is done similarly to how you add type comments to arguments:

pi = 3.142  # type: float

In this example, pi will be type checked as a float variable.

So, Type Annotations or Type Comments?

Should you use annotations or type comments when adding type hints to your own code? In short: Use annotations if you can, use type comments if you must.

Annotations provide a cleaner syntax keeping type information closer to your code. They are also the officially recommended way of writing type hints, and will be further developed and properly maintained in the future.

Type comments are more verbose and might conflict with other kinds of comments in your code like linter directives. However, they can be used in code bases that don’t support annotations.

There is also hidden option number three: stub files. You will learn about these later, when we discuss adding types to third party libraries.

Stub files will work in any version of Python, at the expense of having to maintain a second set of files. In general, you only want to use stub files if you can’t change the original source code.

Playing With Python Types, Part 1

Up until now you’ve only used basic types like str, float, and bool in your type hints. The Python type system is quite powerful, and supports many kinds of more complex types. This is necessary as it needs to be able to reasonably model Python’s dynamic duck typing nature.

In this section you will learn more about this type system, while implementing a simple card game. You will see how to specify:

The type of sequences and mappings like tuples, lists and dictionaries
Type aliases that make code easier to read
That functions and methods do not return anything
Objects that may be of any type

After a short detour into some type theory you will then see even more ways to specify types in Python. You can find the code examples from this section here.

Example: A Deck of Cards

The following example shows an implementation of a regular (French) deck of cards:

 1 # game.py
 2 
 3 import random
 4 
 5 SUITS = "♠ ♡ ♢ ♣".split()
 6 RANKS = "2 3 4 5 6 7 8 9 10 J Q K A".split()
 7 
 8 def create_deck(shuffle=False):
 9     """Create a new deck of 52 cards"""
10     deck = [(s, r) for r in RANKS for s in SUITS]
11     if shuffle:
12         random.shuffle(deck)
13     return deck
14 
15 def deal_hands(deck):
16     """Deal the cards in the deck into four hands"""
17     return (deck[0::4], deck[1::4], deck[2::4], deck[3::4])
18 
19 def play():
20     """Play a 4-player card game"""
21     deck = create_deck(shuffle=True)
22     names = "P1 P2 P3 P4".split()
23     hands = {n: h for n, h in zip(names, deal_hands(deck))}
24 
25     for name, cards in hands.items():
26         card_str = " ".join(f"{s}{r}" for (s, r) in cards)
27         print(f"{name}: {card_str}")
28 
29 if __name__ == "__main__":
30     play()

Each card is represented as a tuple of strings denoting the suit and rank. The deck is represented as a list of cards. create_deck() creates a regular deck of 52 playing cards, and optionally shuffles the cards. deal_hands() deals the deck of cards to four players.

Finally, play() plays the game. As of now, it only prepares for a card game by constructing a shuffled deck and dealing cards to each player. The following is a typical output:

$ python game.py
P4: ♣9 ♢9 ♡2 ♢7 ♡7 ♣A ♠6 ♡K ♡5 ♢6 ♢3 ♣3 ♣Q
P1: ♡A ♠2 ♠10 ♢J ♣10 ♣4 ♠5 ♡Q ♢5 ♣6 ♠A ♣5 ♢4
P2: ♢2 ♠7 ♡8 ♢K ♠3 ♡3 ♣K ♠J ♢A ♣7 ♡6 ♡10 ♠K
P3: ♣2 ♣8 ♠8 ♣J ♢Q ♡9 ♡J ♠4 ♢8 ♢10 ♠9 ♡4 ♠Q

You will see how to extend this example into a more interesting game as we move along.

Sequences and Mappings

Let’s add type hints to our card game. In other words, let’s annotate the functions create_deck(), deal_hands(), and play(). The first challenge is that you need to annotate composite types like the list used to represent the deck of cards and the tuples used to represent the cards themselves.

With simple types like str, float, and bool, adding type hints is as easy as using the type itself:

>>>

>>> name: str = "Guido"
>>> pi: float = 3.142
>>> centered: bool = False

With composite types, you are allowed to do the same:

>>>

>>> names: list = ["Guido", "Jukka", "Ivan"]
>>> version: tuple = (3, 7, 1)
>>> options: dict = {"centered": False, "capitalize": True}

However, this does not really tell the full story. What will be the types of names[2], version[0], and options["centered"]? In this concrete case you can see that they are str, int, and bool, respectively. However, the type hints themselves give no information about this.

Instead, you should use the special types defined in the typing module. These types add syntax for specifying the types of elements of composite types. You can write the following:

>>>

>>> from typing import Dict, List, Tuple

>>> names: List[str] = ["Guido", "Jukka", "Ivan"]
>>> version: Tuple[int, int, int] = (3, 7, 1)
>>> options: Dict[str, bool] = {"centered": False, "capitalize": True}

Note that each of these types start with a capital letter and that they all use square brackets to define item types:

names is a list of strings
version is a 3-tuple consisting of three integers
options is a dictionary mapping strings to boolean values

The typing module contains many more composite types, including Counter, Deque, FrozenSet, NamedTuple, and Set. In addition, the module includes other kinds of types that you’ll see in later sections.

Let’s return to the card game. A card is represented by a tuple of two strings. You can write this as Tuple[str, str], so the type of the deck of cards becomes List[Tuple[str, str]]. Therefore you can annotate create_deck() as follows:

 8 def create_deck(shuffle: bool = False) -> List[Tuple[str, str]]:
 9     """Create a new deck of 52 cards"""
10     deck = [(s, r) for r in RANKS for s in SUITS]
11     if shuffle:
12         random.shuffle(deck)
13     return deck

In addition to the return value, you’ve also added the bool type to the optional shuffle argument.

Note: Tuples and lists are annotated differently.

A tuple is an immutable sequence, and typically consists of a fixed number of possibly differently typed elements. For example, we represent a card as a tuple of suit and rank. In general, you write Tuple[t_1, t_2, ..., t_n] for an n-tuple.

A list is a mutable sequence and usually consists of an unknown number of elements of the same type, for instance a list of cards. No matter how many elements are in the list there is only one type in the annotation: List[t].

In many cases your functions will expect some kind of sequence, and not really care whether it is a list or a tuple. In these cases you should use typing.Sequence when annotating the function argument:

from typing import List, Sequence

def square(elems: Sequence[float]) -> List[float]:
    return [x**2 for x in elems]

Using Sequence is an example of using duck typing. A Sequence is anything that supports len() and .__getitem__(), independent of its actual type.

Type Aliases

The type hints might become quite oblique when working with nested types like the deck of cards. You may need to stare at List[Tuple[str, str]] a bit before figuring out that it matches our representation of a deck of cards.

Now consider how you would annotate deal_hands():

15 def deal_hands(
16     deck: List[Tuple[str, str]]
17 ) -> Tuple[
18     List[Tuple[str, str]],
19     List[Tuple[str, str]],
20     List[Tuple[str, str]],
21     List[Tuple[str, str]],
22 ]:
23     """Deal the cards in the deck into four hands"""
24     return (deck[0::4], deck[1::4], deck[2::4], deck[3::4])

That’s just terrible!

Recall that type annotations are regular Python expressions. That means that you can define your own type aliases by assigning them to new variables. You can for instance create Card and Deck type aliases:

from typing import List, Tuple

Card = Tuple[str, str]
Deck = List[Card]

Card can now be used in type hints or in the definition of new type aliases, like Deck in the example above.

Using these aliases, the annotations of deal_hands() become much more readable:

15 def deal_hands(deck: Deck) -> Tuple[Deck, Deck, Deck, Deck]:
16     """Deal the cards in the deck into four hands"""
17     return (deck[0::4], deck[1::4], deck[2::4], deck[3::4])

Type aliases are great for making your code and its intent clearer. At the same time, these aliases can be inspected to see what they represent:

>>>

>>> from typing import List, Tuple
>>> Card = Tuple[str, str]
>>> Deck = List[Card]

>>> Deck
typing.List[typing.Tuple[str, str]]

Note that when printing Deck, it shows that it’s an alias for a list of 2-tuples of strings.

Functions Without Return Values

You may know that functions without an explicit return still return None:

>>>

>>> def play(player_name):
...     print(f"{player_name} plays")
...

>>> ret_val = play("Jacob")
Jacob plays

>>> print(ret_val)
None

While such functions technically return something, that return value is not useful. You should add type hints saying as much by using None also as the return type:

 1 # play.py
 2 
 3 def play(player_name: str) -> None:
 4     print(f"{player_name} plays")
 5 
 6 ret_val = play("Filip")

The annotations help catch the kinds of subtle bugs where you are trying to use a meaningless return value. Mypy will give you a helpful warning:

$ mypy play.py
play.py:6: error: "play" does not return a value

Note that being explicit about a function not returning anything is different from not adding a type hint about the return value:

# play.py

def play(player_name: str):
    print(f"{player_name} plays")

ret_val = play("Henrik")

In this latter case Mypy has no information about the return value so it will not generate any warning:

$ mypy play.py
$

As a more exotic case, note that you can also annotate functions that are never expected to return normally. This is done using NoReturn:

from typing import NoReturn

def black_hole() -> NoReturn:
    raise Exception("There is no going back ...")

Since black_hole() always raises an exception, it will never return properly.

Example: Play Some Cards

Let’s return to our card game example. In this second version of the game, we deal a hand of cards to each player as before. Then a start player is chosen and the players take turns playing their cards. There are not really any rules in the game though, so the players will just play random cards:

 1 # game.py
 2 
 3 import random
 4 from typing import List, Tuple
 5 
 6 SUITS = "♠ ♡ ♢ ♣".split()
 7 RANKS = "2 3 4 5 6 7 8 9 10 J Q K A".split()
 8 
 9 Card = Tuple[str, str]
10 Deck = List[Card]
11 
12 def create_deck(shuffle: bool = False) -> Deck:
13     """Create a new deck of 52 cards"""
14     deck = [(s, r) for r in RANKS for s in SUITS]
15     if shuffle:
16         random.shuffle(deck)
17     return deck
18 
19 def deal_hands(deck: Deck) -> Tuple[Deck, Deck, Deck, Deck]:
20     """Deal the cards in the deck into four hands"""
21     return (deck[0::4], deck[1::4], deck[2::4], deck[3::4])
22 
23 def choose(items):
24     """Choose and return a random item"""
25     return random.choice(items)
26 
27 def player_order(names, start=None):
28     """Rotate player order so that start goes first"""
29     if start is None:
30         start = choose(names)
31     start_idx = names.index(start)
32     return names[start_idx:] + names[:start_idx]
33 
34 def play() -> None:
35     """Play a 4-player card game"""
36     deck = create_deck(shuffle=True)
37     names = "P1 P2 P3 P4".split()
38     hands = {n: h for n, h in zip(names, deal_hands(deck))}
39     start_player = choose(names)
40     turn_order = player_order(names, start=start_player)
41 
42     # Randomly play cards from each player's hand until empty
43     while hands[start_player]:
44         for name in turn_order:
45             card = choose(hands[name])
46             hands[name].remove(card)
47             print(f"{name}: {card[0] + card[1]:<3}  ", end="")
48         print()
49 
50 if __name__ == "__main__":
51     play()

Note that in addition to changing play(), we have added two new functions that need type hints: choose() and player_order(). Before discussing how we’ll add type hints to them, here is an example output from running the game:

$ python game.py
P3: ♢10  P4: ♣4   P1: ♡8   P2: ♡Q
P3: ♣8   P4: ♠6   P1: ♠5   P2: ♡K
P3: ♢9   P4: ♡J   P1: ♣A   P2: ♡A
P3: ♠Q   P4: ♠3   P1: ♠7   P2: ♠A
P3: ♡4   P4: ♡6   P1: ♣2   P2: ♠K
P3: ♣K   P4: ♣7   P1: ♡7   P2: ♠2
P3: ♣10  P4: ♠4   P1: ♢5   P2: ♡3
P3: ♣Q   P4: ♢K   P1: ♣J   P2: ♡9
P3: ♢2   P4: ♢4   P1: ♠9   P2: ♠10
P3: ♢A   P4: ♡5   P1: ♠J   P2: ♢Q
P3: ♠8   P4: ♢7   P1: ♢3   P2: ♢J
P3: ♣3   P4: ♡10  P1: ♣9   P2: ♡2
P3: ♢6   P4: ♣6   P1: ♣5   P2: ♢8

In this example, player P3 was randomly chosen as the starting player. In turn, each player plays a card: first P3, then P4, then P1, and finally P2. The players keep playing cards as long as they have any left in their hand.

The `Any` Type

choose() works for both lists of names and lists of cards (and any other sequence for that matter). One way to add type hints for this would be the following:

import random
from typing import Any, Sequence

def choose(items: Sequence[Any]) -> Any:
    return random.choice(items)

This means more or less what it says: items is a sequence that can contain items of any type and choose() will return one such item of any type. Unfortunately, this is not that useful. Consider the following example:

 1 # choose.py
 2 
 3 import random
 4 from typing import Any, Sequence
 5 
 6 def choose(items: Sequence[Any]) -> Any:
 7     return random.choice(items)
 8 
 9 names = ["Guido", "Jukka", "Ivan"]
10 reveal_type(names)
11 
12 name = choose(names)
13 reveal_type(name)

While Mypy will correctly infer that names is a list of strings, that information is lost after the call to choose() because of the use of the Any type:

$ mypy choose.py
choose.py:10: error: Revealed type is 'builtins.list[builtins.str*]'
choose.py:13: error: Revealed type is 'Any'

You’ll see a better way shortly. First though, let’s have a more theoretical look at the Python type system, and the special role Any plays.

Type Theory

This tutorial is mainly a practical guide and we will only scratch the surface of the theory underpinning Python type hints. For more details PEP 483 is a good starting point. If you want to get back to the practical examples, feel free to skip to the next section.

Subtypes

One important concept is that of subtypes. Formally, we say that a type T is a subtype of U if the following two conditions hold:

Every value from T is also in the set of values of U type.
Every function from U type is also in the set of functions of T type.

These two conditions guarantees that even if type T is different from U, variables of type T can always pretend to be U.

For a concrete example, consider T = bool and U = int. The bool type takes only two values. Usually these are denoted True and False, but these names are just aliases for the integer values 1 and 0, respectively:

>>>

>>> int(False)
0

>>> int(True)
1

>>> True + True
2

>>> issubclass(bool, int)
True

Since 0 and 1 are both integers, the first condition holds. Above you can see that booleans can be added together, but they can also do anything else integers can. This is the second condition above. In other words, bool is a subtype of int.

The importance of subtypes is that a subtype can always pretend to be its supertype. For instance, the following code type checks as correct:

def double(number: int) -> int:
    return number * 2

print(double(True))  # Passing in bool instead of int

Subtypes are somewhat related to subclasses. In fact all subclasses corresponds to subtypes, and bool is a subtype of int because bool is a subclass of int. However, there are also subtypes that do not correspond to subclasses. For instance int is a subtype of float, but int is not a subclass of float.

Covariant, Contravariant, and Invariant

What happens when you use subtypes inside composite types? For instance, is Tuple[bool] a subtype of Tuple[int]? The answer depends on the composite type, and whether that type is covariant, contravariant, or invariant. This gets technical fast, so let’s just give a few examples:

Tuple is covariant. This means that it preserves the type hierarchy of its item types: Tuple[bool] is a subtype of Tuple[int] because bool is a subtype of int.
List is invariant. Invariant types give no guarantee about subtypes. While all values of List[bool] are values of List[int], you can append an int to List[int] and not to List[bool]. In other words, the second condition for subtypes does not hold, and List[bool] is not a subtype of List[int].
Callable is contravariant in its arguments. This means that it reverses the type hierarchy. You will see how Callable works later, but for now think of Callable[[T], ...] as a function with its only argument being of type T. An example of a Callable[[int], ...] is the double() function defined above. Being contravariant means that if a function operating on a bool is expected, then a function operating on an int would be acceptable.

In general, you don’t need to keep these expression straight. However, you should be aware that subtypes and composite types may not be simple and intuitive.

Gradual Typing and Consistent Types

Earlier we mentioned that Python supports gradual typing, where you can gradually add type hints to your Python code. Gradual typing is essentially made possible by the Any type.

Somehow Any sits both at the top and at the bottom of the type hierarchy of subtypes. Any type behaves as if it is a subtype of Any, and Any behaves as if it is a subtype of any other type. Looking at the definition of subtypes above this is not really possible. Instead we talk about consistent types.

The type T is consistent with the type U if T is a subtype of U or either T or U is Any.

The type checker only complains about inconsistent types. The takeaway is therefore that you will never see type errors arising from the Any type.

This means that you can use Any to explicitly fall back to dynamic typing, describe types that are too complex to describe in the Python type system, or describe items in composite types. For instance, a dictionary with string keys that can take any type as its values can be annotated Dict[str, Any].

Do remember, though, if you use Any the static type checker will effectively not do any type any checking.

Playing With Python Types, Part 2

Let’s return to our practical examples. Recall that you were trying to annotate the general choose() function:

import random
from typing import Any, Sequence

def choose(items: Sequence[Any]) -> Any:
    return random.choice(items)

The problem with using Any is that you are needlessly losing type information. You know that if you pass a list of strings to choose(), it will return a string. Below you’ll see how to express this using type variables, as well as how to work with:

Duck types and protocols
Arguments with None as default value
Class methods
The type of your own classes
Variable number of arguments

Type Variables

A type variable is a special variable that can take on any type, depending on the situation.

Let’s create a type variable that will effectively encapsulate the behavior of choose():

 1 # choose.py
 2 
 3 import random
 4 from typing import Sequence, TypeVar
 5 
 6 Choosable = TypeVar("Chooseable")
 7 
 8 def choose(items: Sequence[Choosable]) -> Choosable:
 9     return random.choice(items)
10 
11 names = ["Guido", "Jukka", "Ivan"]
12 reveal_type(names)
13 
14 name = choose(names)
15 reveal_type(name)

A type variable must be defined using TypeVar from the typing module. When used, a type variable ranges over all possible types and takes the most specific type possible. In the example, name is now a str:

$ mypy choose.py
choose.py:12: error: Revealed type is 'builtins.list[builtins.str*]'
choose.py:15: error: Revealed type is 'builtins.str*'

Consider a few other examples:

 1 # choose_examples.py
 2 
 3 from choose import choose
 4 
 5 reveal_type(choose(["Guido", "Jukka", "Ivan"]))
 6 reveal_type(choose([1, 2, 3]))
 7 reveal_type(choose([True, 42, 3.14]))
 8 reveal_type(choose(["Python", 3, 7])

The first two examples should have type str and int, but what about the last two? The individual list items have different types, and in that case the Choosable type variable does its best to accommodate:

$ mypy choose_examples.py
choose_examples.py:5: error: Revealed type is 'builtins.str*'
choose_examples.py:6: error: Revealed type is 'builtins.int*'
choose_examples.py:7: error: Revealed type is 'builtins.float*'
choose_examples.py:8: error: Revealed type is 'builtins.object*'

As you’ve already seen bool is a subtype of int, which again is a subtype of float. So in the third example the return value of choose() is guaranteed to be something that can be thought of as a float. In the last example, there is no subtype relationship between str and int, so the best that can be said about the return value is that it is an object.

Note that none of these examples raised a type error. Is there a way to tell the type checker that choose() should accept both strings and numbers, but not both at the same time?

You can constrain type variables by listing the acceptable types:

 1 # choose.py
 2 
 3 import random
 4 from typing import Sequence, TypeVar
 5 
 6 Choosable = TypeVar("Choosable", str, float)
 7 
 8 def choose(items: Sequence[Choosable]) -> Choosable:
 9     return random.choice(items)
10 
11 reveal_type(choose(["Guido", "Jukka", "Ivan"]))
12 reveal_type(choose([1, 2, 3]))
13 reveal_type(choose([True, 42, 3.14]))
14 reveal_type(choose(["Python", 3, 7]))

Now Choosable can only be either str or float, and Mypy will note that the last example is an error:

$ mypy choose.py
choose.py:11: error: Revealed type is 'builtins.str*'
choose.py:12: error: Revealed type is 'builtins.float*'
choose.py:13: error: Revealed type is 'builtins.float*'
choose.py:14: error: Revealed type is 'builtins.object*'
choose.py:14: error: Value of type variable "Choosable" of "choose"
                     cannot be "object"

Also note that in the second example the type is considered float even though the input list only contains int objects. This is because Choosable was restricted to strings and floats and int is a subtype of float.

In our card game we want to restrict choose() to be used for str and Card:

Choosable = TypeVar("Choosable", str, Card)

def choose(items: Sequence[Choosable]) -> Choosable:
    ...

We briefly mentioned that Sequence represents both lists and tuples. As we noted, a Sequence can be thought of as a duck type, since it can be any object with .__len__() and .__getitem__() implemented.

Duck Types and Protocols

Recall the following example from the introduction:

def len(obj):
    return obj.__len__()

len() can return the length of any object that has implemented the .__len__() method. How can we add type hints to len(), and in particular the obj argument?

The answer hides behind the academic sounding term structural subtyping. One way to categorize type systems is by whether they are nominal or structural:

In a nominal system, comparisons between types are based on names and declarations. The Python type system is mostly nominal, where an int can be used in place of a float because of their subtype relationship.
In a structural system, comparisons between types are based on structure. You could define a structural type Sized that includes all instances that define .__len__(), irrespective of their nominal type.

There is ongoing work to bring a full-fledged structural type system to Python through PEP 544 which aims at adding a concept called protocols. Most of PEP 544 is already implemented in Mypy though.

A protocol specifies one or more methods that must be implemented. For example, all classes defining .__len__() fulfill the typing.Sized protocol. We can therefore annotate len() as follows:

from typing import Sized

def len(obj: Sized) -> int:
    return obj.__len__()

Other examples of protocols defined in the typing module include Container, Iterable, Awaitable, and ContextManager.

You can also define your own protocols. This is done by inheriting from Protocol and defining the function signatures (with empty function bodies) that the protocol expects. The following example shows how len() and Sized could have been implemented:

from typing_extensions import Protocol

class Sized(Protocol):
    def __len__(self) -> int: ...

def len(obj: Sized) -> int:
    return obj.__len__()

At the time of writing the support for self-defined protocols is still experimental and only available through the typing_extensions module. This module must be explicitly installed from PyPI by doing pip install typing-extensions.

The `Optional` Type

A common pattern in Python is to use None as a default value for an argument. This is usually done either to avoid problems with mutable default values or to have a sentinel value flagging special behavior.

In the card example, the player_order() function uses None as a sentinel value for start saying that if no start player is given it should be chosen randomly:

27 def player_order(names, start=None):
28     """Rotate player order so that start goes first"""
29     if start is None:
30         start = choose(names)
31     start_idx = names.index(start)
32     return names[start_idx:] + names[:start_idx]

The challenge this creates for type hinting is that in general start should be a string. However, it may also take the special non-string value None.

In order to annotate such arguments you can use the Optional type:

from typing import Sequence, Optional

def player_order(
    names: Sequence[str], start: Optional[str] = None
) -> Sequence[str]:
    ...

The Optional type simply says that a variable either has the type specified or is None. An equivalent way of specifying the same would be using the Union type: Union[None, str]

Note that when using either Optional or Union you must take care that the variable has the correct type when you operate on it. This is done in the example by testing whether start is None. Not doing so would cause both static type errors as well as possible runtime errors:

 1 # player_order.py
 2 
 3 from typing import Sequence, Optional
 4 
 5 def player_order(
 6     names: Sequence[str], start: Optional[str] = None
 7 ) -> Sequence[str]:
 8     start_idx = names.index(start)
 9     return names[start_idx:] + names[:start_idx]

Mypy tells you that you have not taken care of the case where start is None:

$ mypy player_order.py
player_order.py:8: error: Argument 1 to "index" of "list" has incompatible
                          type "Optional[str]"; expected "str"

Note: The use of None for optional arguments is so common that Mypy handles it automatically. Mypy assumes that a default argument of None indicates an optional argument even if the type hint does not explicitly say so. You could have used the following:

def player_order(names: Sequence[str], start: str = None) -> Sequence[str]:
    ...

If you don’t want Mypy to make this assumption you can turn it off with the --no-implicit-optional command line option.

Example: The Object(ive) of the Game

Let’s rewrite the card game to be more object-oriented. This will allow us to discuss how to properly annotate classes and methods.

A more or less direct translation of our card game into code that uses classes for Card, Deck, Player, and Game looks something like the following:

 1 # game.py
 2 
 3 import random
 4 import sys
 5 
 6 class Card:
 7     SUITS = "♠ ♡ ♢ ♣".split()
 8     RANKS = "2 3 4 5 6 7 8 9 10 J Q K A".split()
 9 
10     def __init__(self, suit, rank):
11         self.suit = suit
12         self.rank = rank
13 
14     def __repr__(self):
15         return f"{self.suit}{self.rank}"
16 
17 class Deck:
18     def __init__(self, cards):
19         self.cards = cards
20 
21     @classmethod
22     def create(cls, shuffle=False):
23         """Create a new deck of 52 cards"""
24         cards = [Card(s, r) for r in Card.RANKS for s in Card.SUITS]
25         if shuffle:
26             random.shuffle(cards)
27         return cls(cards)
28 
29     def deal(self, num_hands):
30         """Deal the cards in the deck into a number of hands"""
31         cls = self.__class__
32         return tuple(cls(self.cards[i::num_hands]) for i in range(num_hands))
33 
34 class Player:
35     def __init__(self, name, hand):
36         self.name = name
37         self.hand = hand
38 
39     def play_card(self):
40         """Play a card from the player's hand"""
41         card = random.choice(self.hand.cards)
42         self.hand.cards.remove(card)
43         print(f"{self.name}: {card!r:<3}  ", end="")
44         return card
45 
46 class Game:
47     def __init__(self, *names):
48         """Set up the deck and deal cards to 4 players"""
49         deck = Deck.create(shuffle=True)
50         self.names = (list(names) + "P1 P2 P3 P4".split())[:4]
51         self.hands = {
52             n: Player(n, h) for n, h in zip(self.names, deck.deal(4))
53         }
54 
55     def play(self):
56         """Play a card game"""
57         start_player = random.choice(self.names)
58         turn_order = self.player_order(start=start_player)
59 
60         # Play cards from each player's hand until empty
61         while self.hands[start_player].hand.cards:
62             for name in turn_order:
63                 self.hands[name].play_card()
64             print()
65 
66     def player_order(self, start=None):
67         """Rotate player order so that start goes first"""
68         if start is None:
69             start = random.choice(self.names)
70         start_idx = self.names.index(start)
71         return self.names[start_idx:] + self.names[:start_idx]
72 
73 if __name__ == "__main__":
74     # Read player names from command line
75     player_names = sys.argv[1:]
76     game = Game(*player_names)
77     game.play()

Now let’s add types to this code.

Type Hints for Methods

First of all type hints for methods work much the same as type hints for functions. The only difference is that the self argument need not be annotated, as it always will be a class instance. The types of the Card class are easy to add:

 6 class Card:
 7     SUITS = "♠ ♡ ♢ ♣".split()
 8     RANKS = "2 3 4 5 6 7 8 9 10 J Q K A".split()
 9 
10     def __init__(self, suit: str, rank: str) -> None:
11         self.suit = suit
12         self.rank = rank
13 
14     def __repr__(self) -> str:
15         return f"{self.suit}{self.rank}"

Note that the .__init__() method always should have None as its return type.

Classes as Types

There is a correspondence between classes and types. For example, all instances of the Card class together form the Card type. To use classes as types you simply use the name of the class.

For example, a Deck essentially consists of a list of Card objects. You can annotate this as follows:

17 class Deck:
18     def __init__(self, cards: List[Card]) -> None:
19         self.cards = cards

Mypy is able to connect your use of Card in the annotation with the definition of the Card class.

This doesn’t work as cleanly though when you need to refer to the class currently being defined. For example, the Deck.create() class method returns an object with type Deck. However, you can’t simply add -> Deck as the Deck class is not yet fully defined.

Instead, you are allowed to use string literals in annotations. These strings will only be evaluated by the type checker later, and can therefore contain self and forward references. The .create() method should use such string literals for its types:

20 class Deck:
21     @classmethod
22     def create(cls, shuffle: bool = False) -> "Deck":
23         """Create a new deck of 52 cards"""
24         cards = [Card(s, r) for r in Card.RANKS for s in Card.SUITS]
25         if shuffle:
26             random.shuffle(cards)
27         return cls(cards)

Note that the Player class also will reference the Deck class. This is however no problem, since Deck is defined before Player:

34 class Player:
35     def __init__(self, name: str, hand: Deck) -> None:
36         self.name = name
37         self.hand = hand

Usually annotations are not used at runtime. This has given wings to the idea of postponing the evaluation of annotations. Instead of evaluating annotations as Python expressions and storing their value, the proposal is to store the string representation of the annotation and only evaluate it when needed.

Such functionality is planned to become standard in the still mythical Python 4.0. However, in Python 3.7 and later, forward references are available through a __future__ import:

from __future__ import annotations

class Deck:
    @classmethod
    def create(cls, shuffle: bool = False) -> Deck:
        ...

With the __future__ import you can use Deck instead of "Deck" even before Deck is defined.

Returning `self` or `cls`

As noted, you should typically not annotate the self or cls arguments. Partly, this is not necessary as self points to an instance of the class, so it will have the type of the class. In the Card example, self has the implicit type Card. Also, adding this type explicitly would be cumbersome since the class is not defined yet. You would have to use the string literal syntax, self: "Card".

There is one case where you might want to annotate self or cls, though. Consider what happens if you have a superclass that other classes inherit from, and which has methods that return self or cls:

 1 # dogs.py
 2 
 3 from datetime import date
 4 
 5 class Animal:
 6     def __init__(self, name: str, birthday: date) -> None:
 7         self.name = name
 8         self.birthday = birthday
 9 
10     @classmethod
11     def newborn(cls, name: str) -> "Animal":
12         return cls(name, date.today())
13 
14     def twin(self, name: str) -> "Animal":
15         cls = self.__class__
16         return cls(name, self.birthday)
17 
18 class Dog(Animal):
19     def bark(self) -> None:
20         print(f"{self.name} says woof!")
21 
22 fido = Dog.newborn("Fido")
23 pluto = fido.twin("Pluto")
24 fido.bark()
25 pluto.bark()

While the code runs without problems, Mypy will flag a problem:

$ mypy dogs.py
dogs.py:24: error: "Animal" has no attribute "bark"
dogs.py:25: error: "Animal" has no attribute "bark"

The issue is that even though the inherited Dog.newborn() and Dog.twin() methods will return a Dog the annotation says that they return an Animal.

In cases like this you want to be more careful to make sure the annotation is correct. The return type should match the type of self or the instance type of cls. This can be done using type variables that keep track of what is actually passed to self and cls:

# dogs.py

from datetime import date
from typing import Type, TypeVar

TAnimal = TypeVar("TAnimal", bound="Animal")

class Animal:
    def __init__(self, name: str, birthday: date) -> None:
        self.name = name
        self.birthday = birthday

    @classmethod
    def newborn(cls: Type[TAnimal], name: str) -> TAnimal:
        return cls(name, date.today())

    def twin(self: TAnimal, name: str) -> TAnimal:
        cls = self.__class__
        return cls(name, self.birthday)

class Dog(Animal):
    def bark(self) -> None:
        print(f"{self.name} says woof!")

fido = Dog.newborn("Fido")
pluto = fido.twin("Pluto")
fido.bark()
pluto.bark()

There are a few things to note in this example:

The type variable TAnimal is used to denote that return values might be instances of subclasses of Animal.
We specify that Animal is an upper bound for TAnimal. Specifying bound means that TAnimal will only be Animal or one of its subclasses. This is needed to properly restrict the types that are allowed.
The typing.Type[] construct is the typing equivalent of type(). You need it to note that the class method expects a class and returns an instance of that class.

Annotating `*args` and `**kwargs`

In the object oriented version of the game, we added the option to name the players on the command line. This is done by listing player names after the name of the program:

$ python game.py GeirArne Dan Joanna
Dan: ♢A   Joanna: ♡9   P1: ♣A   GeirArne: ♣2
Dan: ♡A   Joanna: ♡6   P1: ♠4   GeirArne: ♢8
Dan: ♢K   Joanna: ♢Q   P1: ♣K   GeirArne: ♠5
Dan: ♡2   Joanna: ♡J   P1: ♠7   GeirArne: ♡K
Dan: ♢10  Joanna: ♣3   P1: ♢4   GeirArne: ♠8
Dan: ♣6   Joanna: ♡Q   P1: ♣Q   GeirArne: ♢J
Dan: ♢2   Joanna: ♡4   P1: ♣8   GeirArne: ♡7
Dan: ♡10  Joanna: ♢3   P1: ♡3   GeirArne: ♠2
Dan: ♠K   Joanna: ♣5   P1: ♣7   GeirArne: ♠J
Dan: ♠6   Joanna: ♢9   P1: ♣J   GeirArne: ♣10
Dan: ♠3   Joanna: ♡5   P1: ♣9   GeirArne: ♠Q
Dan: ♠A   Joanna: ♠9   P1: ♠10  GeirArne: ♡8
Dan: ♢6   Joanna: ♢5   P1: ♢7   GeirArne: ♣4

This is implemented by unpacking and passing in sys.argv to Game() when it’s instantiated. The .__init__() method uses *names to pack the given names into a tuple.

Regarding type annotations: even though names will be a tuple of strings, you should only annotate the type of each name. In other words, you should use str and not Tuple[str]:

46 class Game:
47     def __init__(self, *names: str) -> None:
48         """Set up the deck and deal cards to 4 players"""
49         deck = Deck.create(shuffle=True)
50         self.names = (list(names) + "P1 P2 P3 P4".split())[:4]
51         self.hands = {
52             n: Player(n, h) for n, h in zip(self.names, deck.deal(4))
53         }

Similarly, if you have a function or method accepting **kwargs you should only annotate the type of each possible keyword argument.

Callables

Functions are first-class objects in Python. This means that you can use functions as arguments to other functions. That also means that you need to be able to add type hints representing functions.

Functions, as well as lambdas, methods and classes, are represented by typing.Callable. The types of the arguments and the return value are usually also represented. For instance, Callable[[A1, A2, A3], Rt] represents a function with three arguments with types A1, A2, and A3, respectively. The return type of the function is Rt.

In the following example, the function do_twice() calls a given function twice and prints the return values:

 1 # do_twice.py
 2 
 3 from typing import Callable
 4 
 5 def do_twice(func: Callable[[str], str], argument: str) -> None:
 6     print(func(argument))
 7     print(func(argument))
 8 
 9 def create_greeting(name: str) -> str:
10     return f"Hello {name}"
11 
12 do_twice(create_greeting, "Jekyll")

Note the annotation of the func argument to do_twice() on line 5. It says that func should be a callable with one string argument, that also returns a string. One example of such a callable is create_greeting() defined on line 9.

Most callable types can be annotated in a similar manner. However, if you need more flexibility, check out callback protocols and extended callable types.

Example: Hearts

Let’s end with a full example of the game of Hearts. You might already know this game from other computer simulations. Here is a quick recap of the rules:

Four players play with a hand of 13 cards each.
The player holding the ♣2 starts the first round, and must play ♣2.
Players take turns playing cards, following the leading suit if possible.
The player playing the highest card in the leading suit wins the trick, and becomes start player in the next turn.
A player can not lead with a ♡ until a ♡ has already been played in an earlier trick.
After all cards are played, players get points if they take certain cards:
- 13 points for the ♠Q
- 1 point for each ♡
A game lasts several rounds, until one player has 100 points or more. The player with the least points wins.

More details can be found found online.

There are not many new typing concepts in this example that you have not already seen. We’ll therefore not go through this code in detail, but leave it as an example of annotated code.

You can download this code and other examples from GitHub:

# hearts.py

from collections import Counter
import random
import sys
from typing import Any, Dict, List, Optional, Sequence, Tuple, Union
from typing import overload

class Card:
    SUITS = "♠ ♡ ♢ ♣".split()
    RANKS = "2 3 4 5 6 7 8 9 10 J Q K A".split()

    def __init__(self, suit: str, rank: str) -> None:
        self.suit = suit
        self.rank = rank

    @property
    def value(self) -> int:
        """The value of a card is rank as a number"""
        return self.RANKS.index(self.rank)

    @property
    def points(self) -> int:
        """Points this card is worth"""
        if self.suit == "♠" and self.rank == "Q":
            return 13
        if self.suit == "♡":
            return 1
        return 0

    def __eq__(self, other: Any) -> Any:
        return self.suit == other.suit and self.rank == other.rank

    def __lt__(self, other: Any) -> Any:
        return self.value < other.value

    def __repr__(self) -> str:
        return f"{self.suit}{self.rank}"

class Deck(Sequence[Card]):
    def __init__(self, cards: List[Card]) -> None:
        self.cards = cards

    @classmethod
    def create(cls, shuffle: bool = False) -> "Deck":
        """Create a new deck of 52 cards"""
        cards = [Card(s, r) for r in Card.RANKS for s in Card.SUITS]
        if shuffle:
            random.shuffle(cards)
        return cls(cards)

    def play(self, card: Card) -> None:
        """Play one card by removing it from the deck"""
        self.cards.remove(card)

    def deal(self, num_hands: int) -> Tuple["Deck", ...]:
        """Deal the cards in the deck into a number of hands"""
        return tuple(self[i::num_hands] for i in range(num_hands))

    def add_cards(self, cards: List[Card]) -> None:
        """Add a list of cards to the deck"""
        self.cards += cards

    def __len__(self) -> int:
        return len(self.cards)

    @overload
    def __getitem__(self, key: int) -> Card: ...

    @overload
    def __getitem__(self, key: slice) -> "Deck": ...

    def __getitem__(self, key: Union[int, slice]) -> Union[Card, "Deck"]:
        if isinstance(key, int):
            return self.cards[key]
        elif isinstance(key, slice):
            cls = self.__class__
            return cls(self.cards[key])
        else:
            raise TypeError("Indices must be integers or slices")

    def __repr__(self) -> str:
        return " ".join(repr(c) for c in self.cards)

class Player:
    def __init__(self, name: str, hand: Optional[Deck] = None) -> None:
        self.name = name
        self.hand = Deck([]) if hand is None else hand

    def playable_cards(self, played: List[Card], hearts_broken: bool) -> Deck:
        """List which cards in hand are playable this round"""
        if Card("♣", "2") in self.hand:
            return Deck([Card("♣", "2")])

        lead = played[0].suit if played else None
        playable = Deck([c for c in self.hand if c.suit == lead]) or self.hand
        if lead is None and not hearts_broken:
            playable = Deck([c for c in playable if c.suit != "♡"])
        return playable or Deck(self.hand.cards)

    def non_winning_cards(self, played: List[Card], playable: Deck) -> Deck:
        """List playable cards that are guaranteed to not win the trick"""
        if not played:
            return Deck([])

        lead = played[0].suit
        best_card = max(c for c in played if c.suit == lead)
        return Deck([c for c in playable if c < best_card or c.suit != lead])

    def play_card(self, played: List[Card], hearts_broken: bool) -> Card:
        """Play a card from a cpu player's hand"""
        playable = self.playable_cards(played, hearts_broken)
        non_winning = self.non_winning_cards(played, playable)

        # Strategy
        if non_winning:
            # Highest card not winning the trick, prefer points
            card = max(non_winning, key=lambda c: (c.points, c.value))
        elif len(played) < 3:
            # Lowest card maybe winning, avoid points
            card = min(playable, key=lambda c: (c.points, c.value))
        else:
            # Highest card guaranteed winning, avoid points
            card = max(playable, key=lambda c: (-c.points, c.value))
        self.hand.cards.remove(card)
        print(f"{self.name} -> {card}")
        return card

    def has_card(self, card: Card) -> bool:
        return card in self.hand

    def __repr__(self) -> str:
        return f"{self.__class__.__name__}({self.name!r}, {self.hand})"

class HumanPlayer(Player):
    def play_card(self, played: List[Card], hearts_broken: bool) -> Card:
        """Play a card from a human player's hand"""
        playable = sorted(self.playable_cards(played, hearts_broken))
        p_str = "  ".join(f"{n}: {c}" for n, c in enumerate(playable))
        np_str = " ".join(repr(c) for c in self.hand if c not in playable)
        print(f"  {p_str}  (Rest: {np_str})")
        while True:
            try:
                card_num = int(input(f"  {self.name}, choose card: "))
                card = playable[card_num]
            except (ValueError, IndexError):
                pass
            else:
                break
        self.hand.play(card)
        print(f"{self.name} => {card}")
        return card

class HeartsGame:
    def __init__(self, *names: str) -> None:
        self.names = (list(names) + "P1 P2 P3 P4".split())[:4]
        self.players = [Player(n) for n in self.names[1:]]
        self.players.append(HumanPlayer(self.names[0]))

    def play(self) -> None:
        """Play a game of Hearts until one player go bust"""
        score = Counter({n: 0 for n in self.names})
        while all(s < 100 for s in score.values()):
            print("\nStarting new round:")
            round_score = self.play_round()
            score.update(Counter(round_score))
            print("Scores:")
            for name, total_score in score.most_common(4):
                print(f"{name:<15} {round_score[name]:>3} {total_score:>3}")

        winners = [n for n in self.names if score[n] == min(score.values())]
        print(f"\n{' and '.join(winners)} won the game")

    def play_round(self) -> Dict[str, int]:
        """Play a round of the Hearts card game"""
        deck = Deck.create(shuffle=True)
        for player, hand in zip(self.players, deck.deal(4)):
            player.hand.add_cards(hand.cards)
        start_player = next(
            p for p in self.players if p.has_card(Card("♣", "2"))
        )
        tricks = {p.name: Deck([]) for p in self.players}
        hearts = False

        # Play cards from each player's hand until empty
        while start_player.hand:
            played: List[Card] = []
            turn_order = self.player_order(start=start_player)
            for player in turn_order:
                card = player.play_card(played, hearts_broken=hearts)
                played.append(card)
            start_player = self.trick_winner(played, turn_order)
            tricks[start_player.name].add_cards(played)
            print(f"{start_player.name} wins the trick\n")
            hearts = hearts or any(c.suit == "♡" for c in played)
        return self.count_points(tricks)

    def player_order(self, start: Optional[Player] = None) -> List[Player]:
        """Rotate player order so that start goes first"""
        if start is None:
            start = random.choice(self.players)
        start_idx = self.players.index(start)
        return self.players[start_idx:] + self.players[:start_idx]

    @staticmethod
    def trick_winner(trick: List[Card], players: List[Player]) -> Player:
        lead = trick[0].suit
        valid = [
            (c.value, p) for c, p in zip(trick, players) if c.suit == lead
        ]
        return max(valid)[1]

    @staticmethod
    def count_points(tricks: Dict[str, Deck]) -> Dict[str, int]:
        return {n: sum(c.points for c in cards) for n, cards in tricks.items()}

if __name__ == "__main__":
    # Read player names from the command line
    player_names = sys.argv[1:]
    game = HeartsGame(*player_names)
    game.play()

Here are a few points to note in the code:

For type relationships that are hard to express using Union or type variables, you can use the @overload decorator. See Deck.__getitem__() for an example and the documentation for more information.
Subclasses correspond to subtypes, so that a HumanPlayer can be used wherever a Player is expected.
When a subclass reimplements a method from a superclass, the type annotations must match. See HumanPlayer.play_card() for an example.

When starting the game, you control the first player. Enter numbers to choose which cards to play. The following is an example of game play, with the highlighted lines showing where the player made a choice:

$ python hearts.py GeirArne Aldren Joanna Brad

Starting new round:
Brad -> ♣2
  0: ♣5  1: ♣Q  2: ♣K  (Rest: ♢6 ♡10 ♡6 ♠J ♡3 ♡9 ♢10 ♠7 ♠K ♠4)
  GeirArne, choose card: 2
GeirArne => ♣K
Aldren -> ♣10
Joanna -> ♣9
GeirArne wins the trick

  0: ♠4  1: ♣5  2: ♢6  3: ♠7  4: ♢10  5: ♠J  6: ♣Q  7: ♠K  (Rest: ♡10 ♡6 ♡3 ♡9)
  GeirArne, choose card: 0
GeirArne => ♠4
Aldren -> ♠5
Joanna -> ♠3
Brad -> ♠2
Aldren wins the trick

...

Joanna -> ♡J
Brad -> ♡2
  0: ♡6  1: ♡9  (Rest: )
  GeirArne, choose card: 1
GeirArne => ♡9
Aldren -> ♡A
Aldren wins the trick

Aldren -> ♣A
Joanna -> ♡Q
Brad -> ♣J
  0: ♡6  (Rest: )
  GeirArne, choose card: 0
GeirArne => ♡6
Aldren wins the trick

Scores:
Brad             14  14
Aldren           10  10
GeirArne          1   1
Joanna            1   1

Static Type Checking

So far you have seen how to add type hints to your code. In this section you’ll learn more about how to actually perform static type checking of Python code.

The Mypy Project

Mypy was started by Jukka Lehtosalo during his Ph.D. studies at Cambridge around 2012. Mypy was originally envisioned as a Python variant with seamless dynamic and static typing. See Jukka’s slides from PyCon Finland 2012 for examples of the original vision of Mypy.

Most of those original ideas still play a big part in the Mypy project. In fact, the slogan “Seamless dynamic and static typing” is still prominently visible on Mypy’s home page and describes the motivation for using type hints in Python well.

The biggest change since 2012 is that Mypy is no longer a variant of Python. In its first versions Mypy was a stand-alone language that was compatible with Python except for its type declarations. Following a suggestion by Guido van Rossum, Mypy was rewritten to use annotations instead. Today Mypy is a static type checker for regular Python code.

Running Mypy

Before running Mypy for the first time, you must install the program. This is most easily done using pip:

$ pip install mypy

With Mypy installed, you can run it as a regular command line program:

$ mypy my_program.py

Running Mypy on your my_program.py Python file will check it for type errors without actually executing the code.

There are many available options when type checking your code. As Mypy is still under very active development, command line options are liable to change between versions. You should refer to Mypy’s help to see which settings are default on your version:

$ mypy --help
usage: mypy [-h] [-v] [-V] [more options; see below]
            [-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...]

Mypy is a program that will type check your Python code.

[... The rest of the help hidden for brevity ...]

Additionally, the Mypy command line documentation online has a lot of information.

Let’s look at some of the most common options. First of all, if you are using third-party packages without type hints, you may want to silence Mypy’s warnings about these. This can be done with the --ignore-missing-imports option.

The following example uses Numpy to calculate and print the cosine of several numbers:

 1 # cosine.py
 2 
 3 import numpy as np
 4 
 5 def print_cosine(x: np.ndarray) -> None:
 6     with np.printoptions(precision=3, suppress=True):
 7         print(np.cos(x))
 8 
 9 x = np.linspace(0, 2 * np.pi, 9)
10 print_cosine(x)

Note that np.printoptions() is only available in version 1.15 and later of Numpy. Running this example prints some numbers to the console:

$ python cosine.py
[ 1.     0.707  0.    -0.707 -1.    -0.707 -0.     0.707  1.   ]

The actual output of this example is not important. However, you should note that the argument x is annotated with np.ndarray on line 5, as we want to print the cosine of a full array of numbers.

You can run Mypy on this file as usual:

$ mypy cosine.py 
cosine.py:3: error: No library stub file for module 'numpy'
cosine.py:3: note: (Stub files are from https://github.com/python/typeshed)

These warnings may not immediately make much sense to you, but you’ll learn about stubs and typeshed soon. You can essentially read the warnings as Mypy saying that the Numpy package does not contain type hints.

In most cases, missing type hints in third-party packages is not something you want to be bothered with so you can silence these messages:

$ mypy --ignore-missing-imports cosine.py 
$

If you use the --ignore-missing-import command line option, Mypy will not try to follow or warn about any missing imports. This might be a bit heavy-handed though, as it also ignores actual mistakes, like misspelling the name of a package.

Two less intrusive ways of handling third-party packages are using type comments or configuration files.

In a simple example as the one above, you can silence the numpy warning by adding a type comment to the line containing the import:

 3 import numpy as np  # type: ignore

The literal # type: ignore tells Mypy to ignore the import of Numpy.

If you have several files, it might be easier to keep track of which imports to ignore in a configuration file. Mypy reads a file called mypy.ini in the current directory if it is present. This configuration file must contain a section called [mypy] and may contain module specific sections of the form [mypy-module].

The following configuration file will ignore that Numpy is missing type hints:

# mypy.ini

[mypy]

[mypy-numpy]
ignore_missing_imports = True

There are many options that can be specified in the configuration file. It is also possible to specify a global configuration file. See the documentation for more information.

Adding Stubs

Type hints are available for all the packages in the Python standard library. However, if you are using third-party packages you’ve already seen that the situation can be different.

The following example uses the Parse package to do simple text parsing. To follow along you should first install Parse:

$ pip install parse

Parse can be used to recognize simple patterns. Here is a small program that tries its best to figure out your name:

 1 # parse_name.py
 2 
 3 import parse
 4 
 5 def parse_name(text: str) -> str:
 6     patterns = (
 7         "my name is {name}",
 8         "i'm {name}",
 9         "i am {name}",
10         "call me {name}",
11         "{name}",
12     )
13     for pattern in patterns:
14         result = parse.parse(pattern, text)
15         if result:
16             return result["name"]
17     return ""
18 
19 answer = input("What is your name? ")
20 name = parse_name(answer)
21 print(f"Hi {name}, nice to meet you!")

The main flow is defined in the last three lines: ask for your name, parse the answer, and print a greeting. The parse package is called on line 14 in order to try to find a name based on one of the patterns listed on lines 7-11.

The program can be used as follows:

$ python parse_name.py
What is your name? I am Geir Arne
Hi Geir Arne, nice to meet you!

Note that even though I answer I am Geir Arne, the program figures out that I am is not part of my name.

Let’s add a small bug to the program, and see if Mypy is able to help us detect it. Change line 16 from return result["name"] to return result. This will return a parse.Result object instead of the string containing the name.

Next run Mypy on the program:

$ mypy parse_name.py 
parse_name.py:3: error: Cannot find module named 'parse'
parse_name.py:3: note: (Perhaps setting MYPYPATH or using the
                       "--ignore-missing-imports" flag would help)

Mypy prints a similar error to the one you saw in the previous section: It doesn’t know about the parse package. You could try to ignore the import:

$ mypy parse_name.py --ignore-missing-imports
$

Unfortunately, ignoring the import means that Mypy has no way of discovering the bug in our program. A better solution would be to add type hints to the Parse package itself. As Parse is open source you can actually add types to the source code and send a pull request.

Alternatively, you can add the types in a stub file. A stub file is a text file that contains the signatures of methods and functions, but not their implementations. Their main function is to add type hints to code that you for some reason can’t change. To show how this works, we will add some stubs for the Parse package.

First of all, you should put all your stub files inside one common directory, and set the MYPYPATH environment variable to point to this directory. On Mac and Linux you can set MYPYPATH as follows:

$ export MYPYPATH=/home/gahjelle/python/stubs

You can set the variable permanently by adding the line to your .bashrc file. On Windows you can click the start menu and search for environment variables to set MYPYPATH.

Next, create a file inside your stubs directory that you call parse.pyi. It must be named for the package that you are adding type hints for, with a .pyi suffix. Leave this file empty for now. Then run Mypy again:

$ mypy parse_name.py
parse_name.py:14: error: Module has no attribute "parse"

If you have set everything up correctly, you should see this new error message. Mypy uses the new parse.pyi file to figure out which functions are available in the parse package. Since the stub file is empty, Mypy assumes that parse.parse() does not exist, and then gives the error you see above.

The following example does not add types for the whole parse package. Instead it shows the type hints you need to add in order for Mypy to type check your use of parse.parse():

# parse.pyi

from typing import Any, Mapping, Optional, Sequence, Tuple, Union

class Result:
    def __init__(
        self,
        fixed: Sequence[str],
        named: Mapping[str, str],
        spans: Mapping[int, Tuple[int, int]],
    ) -> None: ...
    def __getitem__(self, item: Union[int, str]) -> str: ...
    def __repr__(self) -> str: ...

def parse(
    format: str,
    string: str,
    evaluate_result: bool = ...,
    case_sensitive: bool = ...,
) -> Optional[Result]: ...

The ellipsis ... are part of the file, and should be written exactly as above. The stub file should only contain type hints for variables, attributes, functions, and methods, so the implementations should be left out and replaced by ... markers.

Finally Mypy is able to spot the bug we introduced:

$ mypy parse_name.py
parse_name.py:16: error: Incompatible return value type (got
                         "Result", expected "str")

This points straight to line 16 and the fact that we return a Result object and not the name string. Change return result back to return result["name"], and run Mypy again to see that it’s happy.

Typeshed

You’ve seen how to use stubs to add type hints without changing the source code itself. In the previous section we added some type hints to the third-party Parse package. Now, it wouldn’t be very effective if everybody needs to create their own stubs files for all third-party packages they are using.

Typeshed is a Github repository that contains type hints for the Python standard library, as well as many third-party packages. Typeshed comes included with Mypy so if you are using a package that already has type hints defined in Typeshed, the type checking will just work.

You can also contribute type hints to Typeshed. Make sure to get the permission of the owner of the package first though, especially because they might be working on adding type hints into the source code itself—which is the preferred approach.

Other Static Type Checkers

In this tutorial, we have mainly focused on type checking using Mypy. However, there are other static type checkers in the Python ecosystem.

The PyCharm editor comes with its own type checker included. If you are using PyCharm to write your Python code, it will be automatically type checked.

Facebook has developed Pyre. One of its stated goals is to be fast and performant. While there are some differences, Pyre functions mostly similar to Mypy. See the documentation if you’re interested in trying out Pyre.

Furthermore, Google has created Pytype. This type checker also works mostly the same as Mypy. In addition to checking annotated code, Pytype has some support for running type checks on unannotated code and even adding annotations to code automatically. See the quickstart document for more information.

Using Types at Runtime

As a final note, it’s possible to use type hints also at runtime during execution of your Python program. Runtime type checking will probably never be natively supported in Python.

However, the type hints are available at runtime in the __annotations__ dictionary, and you can use those to do type checks if you desire. Before you run off and write your own package for enforcing types, you should know that there are already several packages doing this for you. Have a look at Enforce, Pydantic, or Pytypes for some examples.

Another use of type hints is for translating your Python code to C and compiling it for optimization. The popular Cython project uses a hybrid C/Python language to write statically typed Python code. However, since version 0.27 Cython has also supported type annotations. Recently, the Mypyc project has become available. While not yet ready for general use, it can compile some type annotated Python code to C extensions.

Conclusion

Type hinting in Python is a very useful feature that you can happily live without. Type hints don’t make you capable of writing any code you can’t write without using type hints. Instead, using type hints makes it easier for you to reason about code, find subtle bugs, and maintain a clean architecture.

In this tutorial you have learned how type hinting works in Python, and how gradual typing make type checks in Python more flexible than in many other languages. You’ve seen some of the pros and cons of using type hints, and how they can be added to code using annotations or type comments. Finally you saw many of the different types that Python supports, as well as how to perform static type checking.

There are many resources to learn more about static type checking in Python. PEP 483 and PEP 484 give a lot of background about how type checking is implemented in Python. The Mypy documentation has a great reference section detailing all the different types available.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Modeling Polymorphism in Django With Python

2019-01-02T14:00:00+00:00

Modeling polymorphism in relational databases is a challenging task. In this article, we present several modeling techniques to represent polymorphic objects in a relational database using the Django object-relational mapping (ORM).

This intermediate-level tutorial is designed for readers who are already familiar with the fundamental design of Django.

Free Bonus: Click here to get the most popular Django tutorials and resources on Real Python and improve your Django + Python web development skills.

What Is Polymorphism?

Polymorphism is the ability of an object to take on many forms. Common examples of polymorphic objects include event streams, different types of users, and products in an e-commerce website. A polymorphic model is used when a single entity requires different functionality or information.

In the examples above, all events are logged for future use, but they can contain different data. All users need be able to log in, but they might have different profile structures. In every e-commerce website, a user wants to put different products in their shopping cart.

Why Is Modeling Polymorphism Challenging?

There are many ways to model polymorphism. Some approaches use standard features of the Django ORM, and some use special features of the Django ORM. The main challenges you’ll encounter when modeling polymorphic objects are the following:

How to represent a single polymorphic object: Polymorphic objects have different attributes. The Django ORM maps attributes to columns in the database. In that case, how should the Django ORM map attributes to the columns in the table? Should different objects reside in the same table? Should you have multiple tables?
How to reference instances of a polymorphic model: To utilize database and Django ORM features, you need to reference objects using foreign keys. How you decide to represent a single polymorphic object is crucial to your ability to reference it.

To truly understand the challenges of modeling polymorphism, you are going to take a small bookstore from its first online website to a big online shop selling all sorts of products. Along the way, you’ll experience and analyze different approaches for modeling polymorphism using the Django ORM.

Note: To follow this tutorial, it is recommended that you use a PostgreSQL backend, Django 2.x, and Python 3.

It’s possible to follow along with other database backends as well. In places where features unique to PostgreSQL are used, an alternative will be presented for other databases.

Naive Implementation

You have a bookstore in a nice part of town right next to a coffee shop, and you want to start selling books online.

You sell only one type of product: books. In your online store, you want to show details about the books, like name and price. You want your users to browse around the website and collect many books, so you also need a cart. You eventually need to ship the books to the user, so you need to know the weight of each book to calculate the delivery fee.

Let’s create a simple model for your new book store:

from django.contrib.auth import get_user_model
from django.db import models


class Book(models.Model):
    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )
    weight = models.PositiveIntegerField(
        help_text='in grams',
    )

    def __str__(self) -> str:
        return self.name


class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    books = models.ManyToManyField(Book)

To create a new book, you provide a name, price, and weight:

>>>

>>> from naive.models import Book
>>> book = Book.objects.create(name='Python Tricks', price=1000, weight=200)
>>> book
<Product: Python Tricks>

To create a cart, you first need to associate it with a user:

>>>

>>> from django.contrib.auth import get_user_model
>>> haki = get_user_model().create_user('haki')

>>> from naive.models import Cart
>>> cart = Cart.objects.create(user=haki)

Then the user can start adding items to it:

>>>

>>> cart.products.add(book)
>>> cart.products.all()
<QuerySet [<Book: Python Tricks>]>

Pro

Easy to understand and maintain: It’s sufficient for a single type of product.

Con

Restricted to homogeneous products: It only supports products with the same set of attributes. Polymorphism is not captured or permitted at all.

Sparse Model

With the success of your online bookstore, users started to ask if you also sell e-books. E-books are a great product for your online store, and you want to start selling them right away.

A physical book is different from an e-book:

An e-book has no weight. It’s a virtual product.
An e-book does not require shipment. Users download it from the website.

To make your existing model support the additional information for selling e-books, you add some fields to the existing Book model:

from django.contrib.auth import get_user_model
from django.db import models


class Book(models.Model):
    TYPE_PHYSICAL = 'physical'
    TYPE_VIRTUAL = 'virtual'
    TYPE_CHOICES = (
        (TYPE_PHYSICAL, 'Physical'),
        (TYPE_VIRTUAL, 'Virtual'),
    )
    type = models.CharField(
        max_length=20,
        choices=TYPE_CHOICES,
    )

    # Common attributes
    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )

    # Specific attributes
    weight = models.PositiveIntegerField(
        help_text='in grams',
    )
    download_link = models.URLField(
        null=True, blank=True,
    )

    def __str__(self) -> str:
        return f'[{self.get_type_display()}] {self.name}'


class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    books = models.ManyToManyField(
        Book,
    )

First, you added a type field to indicate what type of book it is. Then, you added a URL field to store the download link of the e-book.

To add a physical book to your bookstore, do the following:

>>>

>>> from sparse.models import Book
>>> physical_book = Book.objects.create(
...     type=Book.TYPE_PHYSICAL,
...     name='Python Tricks',
...     price=1000,
...     weight=200,
...     download_link=None,
... )
>>> physical_book
<Book: [Physical] Python Tricks>

To add a new e-book, you do the following:

>>>

>>> virtual_book = Book.objects.create(
...     type=Book.TYPE_VIRTUAL,
...     name='The Old Man and the Sea',
...     price=1500,
...     weight=0,
...     download_link='https://books.com/12345',
... )
>>> virtual_book
<Book: [Virtual] The Old Man and the Sea>

Your users can now add both books and e-books to the cart:

>>>

>>> from sparse.models import Cart
>>> cart = Cart.objects.create(user=user)
>>> cart.books.add(physical_book, virtual_book)
>>> cart.books.all()
<QuerySet [<Book: [Physical] Python Tricks>, <Book: [Virtual] The Old Man and the Sea>]>

The virtual books are a big hit, and you decide to hire employees. The new employees are apparently not so tech savvy, and you start seeing weird things in the database:

>>>

>>> Book.objects.create(
...     type=Book.TYPE_PHYSICAL,
...     name='Python Tricks',
...     price=1000,
...     weight=0,
...     download_link='http://books.com/54321',
... )

That book apparently weighs 0 pounds and has a download link.

This e-book apparently weighs 100g and has no download link:

>>>

>>> Book.objects.create(
...     type=Book.TYPE_VIRTUAL,
...     name='Python Tricks',
...     price=1000,
...     weight=100,
...     download_link=None,
... )

This doesn’t make any sense. You have a data integrity problem.

To overcome integrity problems, you add validations to the model:

from django.core.exceptions import ValidationError


class Book(models.Model):

    # ...

    def clean(self) -> None:
        if self.type == Book.TYPE_VIRTUAL:
            if self.weight != 0:
                raise ValidationError(
                    'A virtual product weight cannot exceed zero.'
                )

            if self.download_link is None:
                raise ValidationError(
                    'A virtual product must have a download link.'
                )

        elif self.type == Book.TYPE_PHYSICAL:
            if self.weight == 0:
                raise ValidationError(
                    'A physical product weight must exceed zero.'
                )

            if self.download_link is not None:
                raise ValidationError(
                    'A physical product cannot have a download link.'
                )

        else:
            assert False, f'Unknown product type "{self.type}"'

You used Django’s built-in validation mechanism to enforce data integrity rules. clean() is only called automatically by Django forms. For objects that are not created by a Django form, you need to make sure to explicitly validate the object.

To keep the integrity of the Book model intact, you need to make a little change to the way you create books:

>>>

>>> book = Book(
...    type=Book.TYPE_PHYSICAL,
...    name='Python Tricks',
...    price=1000,
...    weight=0,
...    download_link='http://books.com/54321',
... )
>>> book.full_clean()
ValidationError: {'__all__': ['A physical product weight must exceed zero.']}

>>> book = Book(
...    type=Book.TYPE_VIRTUAL,
...    name='Python Tricks',
...    price=1000,
...    weight=100,
...    download_link=None,
... )
>>> book.full_clean()
ValidationError: {'__all__': ['A virtual product weight cannot exceed zero.']}

When creating objects using the default manager (Book.objects.create(...)), Django will create an object and immediately persist it to the database.

In your case, you want to validate the object before saving if to the database. You first create the object (Book(...)), validate it (book.full_clean()), and only then save it (book.save()).

Denormalization:

A sparse model is a product of denormalization. In a denormalization process, you inline attributes from multiple normalized models into a single table for better performance. A denormalized table will usually have a lot of nullable columns.

Denormalizing is often used in decision support systems such as data warehouses where read performance is most important. Unlike OLTP systems, data warehouses are usually not required to enforce data integrity rules, which makes denormalization ideal.

Pro

Easy to understand and maintain: The sparse model is usually the first step we take when certain types of objects need more information. It’s very intuitive and easy to understand.

Cons

Unable to utilize NOT NULL database constraints: Null values are used for attributes that are not defined for all types of objects.
Complex validation logic: Complex validation logic is required to enforce data integrity rules. The complex logic also requires more tests.
Many Null fields create clutter: Representing multiple types of products in a single model makes it harder to understand and maintain.
New types require schema changes: New types of products require additional fields and validations.

Use Case

The sparse model is ideal when you’re representing heterogeneous objects that share most attributes, and when new items are not added very often.

Semi-Structured Model

Your bookstore is now a huge success, and you are selling more and more books. You have books from different genres and publishers, e-books with different formats, books with odd shapes and sizes, and so on.

In the sparse model approach, you added fields for every new type of product. The model now has a lot of nullable fields, and new developers and employees are having trouble keeping up.

To address the clutter, you decide to keep only the common fields (name and price) on the model. You store the rest of the fields in a single JSONField:

from django.contrib.auth import get_user_model
from django.contrib.postgres.fields import JSONField
from django.db import models

class Book(models.Model):
    TYPE_PHYSICAL = 'physical'
    TYPE_VIRTUAL = 'virtual'
    TYPE_CHOICES = (
        (TYPE_PHYSICAL, 'Physical'),
        (TYPE_VIRTUAL, 'Virtual'),
    )
    type = models.CharField(
        max_length=20,
        choices=TYPE_CHOICES,
    )

    # Common attributes
    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )

    extra = JSONField()

    def __str__(self) -> str:
        return f'[{self.get_type_display()}] {self.name}'


class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    books = models.ManyToManyField(
        Book,
        related_name='+',
    )

JSONField:

In this example, you use PostgreSQL as a database backend. Django provides a built-in JSON field for PostgreSQL in django.contrib.postgres.fields.

For other databases, such as SQLite and MySQL, there are packages that provide similar functionality.

Your Book model is now clutter-free. Common attributes are modeled as fields. Attributes that are not common to all types of products are stored in the extra JSON field:

>>>

>>> from semi_structured.models import Book
>>> physical_book = Book(
...     type=Book.TYPE_PHYSICAL,
...     name='Python Tricks',
...     price=1000,
...     extra={'weight': 200},
... )
>>> physical_book.full_clean()
>>> physical_book.save()
<Book: [Physical] Python Tricks>

>>> virtual_book = Book(
...     type=Book.TYPE_VIRTUAL,
...     name='The Old Man and the Sea',
...     price=1500,
...     extra={'download_link': 'http://books.com/12345'},
... )
>>> virtual_book.full_clean()
>>> virtual_book.save()
<Book: [Virtual] The Old Man and the Sea>

>>> from semi_structured.models import Cart
>>> cart = Cart.objects.create(user=user)
>>> cart.books.add(physical_book, virtual_book)
>>> cart.books.all()
<QuerySet [<Book: [Physical] Python Tricks>, <Book: [Virtual] The Old Man and the Sea>]>

Clearing up the clutter is important, but it comes with a cost. The validation logic is a lot more complicated:

from django.core.exceptions import ValidationError
from django.core.validators import URLValidator

class Book(models.Model):

    # ...

    def clean(self) -> None:

        if self.type == Book.TYPE_VIRTUAL:

            try:
                weight = int(self.extra['weight'])
            except ValueError:
                raise ValidationError(
                    'Weight must be a number'
                )
            except KeyError:
                pass
            else:
                if weight != 0:
                    raise ValidationError(
                        'A virtual product weight cannot exceed zero.'
                    )

            try:
                download_link = self.extra['download_link']
            except KeyError:
                pass
            else:
                # Will raise a validation error
                URLValidator()(download_link)

        elif self.type == Book.TYPE_PHYSICAL:

            try:
                weight = int(self.extra['weight'])
            except ValueError:
                raise ValidationError(
                    'Weight must be a number'
                 )
            except KeyError:
                pass
            else:
                if weight == 0:
                    raise ValidationError(
                        'A physical product weight must exceed zero.'
                     )

            try:
                download_link = self.extra['download_link']
            except KeyError:
                pass
            else:
                if download_link is not None:
                    raise ValidationError(
                        'A physical product cannot have a download link.'
                    )

        else:
            raise ValidationError(f'Unknown product type "{self.type}"')

The benefit of using a proper field is that it validates the type. Both Django and the Django ORM can perform checks to make sure the right type is used for the field. When using a JSONField, you need to validate both the type and the value:

>>>

>>> book = Book.objects.create(
...     type=Book.TYPE_VIRTUAL,
...     name='Python Tricks',
...     price=1000,
...     extra={'weight': 100},
... )
>>> book.full_clean()
ValidationError: {'__all__': ['A virtual product weight cannot exceed zero.']}

Another issue with using JSON is that not all databases have proper support for querying and indexing values in JSON fields.

In PostgreSQL for example, you can query all the books that weigh more than 100:

>>>

>>> Book.objects.filter(extra__weight__gt=100)
<QuerySet [<Book: [Physical] Python Tricks>]>

However, not all database vendors support that.

Another restriction imposed when using JSON is that you are unable to use database constraints such as not null, unique, and foreign keys. You will have to implement these constraints in the application.

This semi-structured approach resembles NoSQL architecture and has many of its advantages and disadvantages. The JSON field is a way to get around the strict schema of a relational database. This hybrid approach provides us with the flexibility to squash many object types into a single table while still maintaining some of the benefits of a relational, strictly and strongly typed database. For many common NoSQL use cases, this approach might actually be more suitable.

Pros

Reduce clutter: Common fields are stored on the model. Other fields are stored in a single JSON field.
Easier to add new types: New types of products don’t require schema changes.

Cons

Complicated and ad hoc validation logic: Validating a JSON field requires validating types as well as values. This challenge can be addressed by using other solutions to validate JSON data such as JSON schema.
Unable to utilize database constraints: Database constraints such as null null, unique and foreign key constraints, which enforce type and data integrity at the database level, cannot be used.
Restricted by database support for JSON: Not all database vendors support querying and indexing JSON fields.
Schema is not enforced by the database system: Schema changes might require backward compatibility or ad hoc migrations. Data can “rot.”
No deep integration with the database metadata system: Metadata about the fields is not stored in the database. Schema is only enforced at the application level.

Use Case

A semi-structured model is ideal when you need to represent heterogeneous objects that don’t share many common attributes, and when new items are added often.

A classic use case for the semi-structured approach is storing events (like logs, analytics, and event stores). Most events have a timestamp, type and metadata like device, user agent, user, and so on. The data for each type is stored in a JSON field. For analytics and log events, it’s important to be able to add new types of events with minimal effort, so this approach is ideal.

Abstract Base Model

So far, you’ve worked around the problem of actually treating your products as heterogeneous. You worked under the assumption that the differences between the products is minimal, so it made sense to maintain them in the same model. This assumption can take you only so far.

Your little store is growing fast, and you want to start selling entirely different types of products, such as e-readers, pens, and notebooks.

A book and an e-book are both products. A product is defined using common attributes such as name and price. In an object-oriented environment, you could look at a Product as a base class or an interface. Every new type of product you add must implement the Product class and extend it with its own attributes.

Django offers the ability to create abstract base classes. Let’s define a Product abstract base class and add two models for Book and EBook:

from django.contrib.auth import get_user_model
from django.db import models


class Product(models.Model):
    class Meta:
        abstract = True

    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )

    def __str__(self) -> str:
        return self.name


class Book(Product):
    weight = models.PositiveIntegerField(
        help_text='in grams',
    )


class EBook(Product):
    download_link = models.URLField()

Notice that both Book and EBook inherit from Product. The fields defined in the base class Product are inherited, so the derived models Book and Ebook don’t need to repeat them.

To add new products, you use the derived classes:

>>>

>>> from abstract_base_model.models import Book
>>> book = Book.objects.create(name='Python Tricks', price=1000, weight=200)
>>> book
<Book: Python Tricks>

>>> ebook = EBook.objects.create(
...     name='The Old Man and the Sea',
...     price=1500,
...     download_link='http://books.com/12345',
... )
>>> ebook
<Book: The Old Man and the Sea>

You might have noticed that the Cart model is missing. You can try to create a Cart model with a ManyToMany field to Product:

class Cart(models.Model):
    user = models.OneToOneField(
       get_user_model(),
       primary_key=True,
       on_delete=models.CASCADE,
    )
    items = models.ManyToManyField(Product)

If you try to reference a ManyToMany field to an abstract model, you will get the following error:

abstract_base_model.Cart.items: (fields.E300) Field defines a relation with model 'Product', which is either not installed, or is abstract.

A foreign key constraint can only point to a concrete table. The abstract base model Product only exists in the code, so there is no products table in the database. The Django ORM will only create tables for the derived models Book and EBook.

Given that you can’t reference the abstract base class Product, you need to reference books and e-books directly:

class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    books = models.ManyToManyField(Book)
    ebooks = models.ManyToManyField(EBook)

You can now add both books and e-books to the cart:

>>>

>>> user = get_user_model().objects.first()
>>> cart = Cart.objects.create(user=user)
>>> cart.books.add(book)
>>> cart.ebooks.add(ebook)

This model is a bit more complicated now. Let’s query the total price of the items in the cart:

>>>

>>> from django.db.models import Sum
>>> from django.db.models.functions import Coalesce
>>> (
...     Cart.objects
...     .filter(pk=cart.pk)
...     .aggregate(total_price=Sum(
...         Coalesce('books__price', 'ebooks__price')
...     ))
... )
{'total_price': 1000}

Because you have more than one type of book, you use Coalesce to fetch either the price of the book or the price of the e-book for each row.

Pro

Easier to implement specific logic: A separate model for each product makes it easier to implement, test, and maintain specific logic.

Cons

Require multiple foreign keys: To reference all types of products, each type needs a foreign key.
Harder to implement and maintain: Operations on all types of products require checking all foreign keys. This adds complexity to the code and makes maintenance and testing harder.
Very hard to scale: New types of products require additional models. Managing many models can be tedious and very hard to scale.

Use Case

An abstract base model is a good choice when there are very few types of objects that required very distinct logic.

An intuitive example is modeling a payment process for your online shop. You want to accept payments with credit cards, PayPal, and store credit. Each payment method goes through a very different process that requires very distinct logic. Adding a new type of payment is not very common, and you don’t plan on adding new payment methods in the near future.

You create a payment process base class with derived classes for credit card payment process, PayPal payment process, and store credit payment process. For each of the derived classes, you implement the payment process in a very different way that cannot be easily shared. In this case, it might make sense to handle each payment process specifically.

Concrete Base Model

Django offers another way to implement inheritance in models. Instead of using an abstract base class that only exists in the code, you can make the base class concrete. “Concrete” means that the base class exists in the database as a table, unlike in the abstract base class solution, where the base class only exists in the code.

Using the abstract base model, you were unable to reference multiple type of products. You were forced to create a many-to-many relation for each type of product. This made it harder to perform tasks on the common fields such as getting the total price of all the items in the cart.

Using a concrete base class, Django will create a table in the database for the Product model. The Product model will have all the common fields you defined in the base model. Derived models such as Book and EBook will reference the Product table using a one-to-one field. To reference a product, you create a foreign key to the base model:

from django.contrib.auth import get_user_model
from django.db import models


class Product(models.Model):
    name = models.CharField(
        max_length=100,
    )
    price = models.PositiveIntegerField(
        help_text='in cents',
    )

    def __str__(self) -> str:
        return self.name


class Book(Product):
    weight = models.PositiveIntegerField()


class EBook(Product):
    download_link = models.URLField()

The only difference between this example and the previous one is that the Product model is not defined with abstract=True.

To create new products, you use derived Book and EBook models directly:

>>>

>>> from concrete_base_model.models import Book, EBook
>>> book = Book.objects.create(
...     name='Python Tricks',
...     price=1000,
...     weight=200,
... )
>>> book
<Book: Python Tricks>

>>> ebook = EBook.objects.create(
...     name='The Old Man and the Sea',
...     price=1500,
...     download_link='http://books.com/12345',
... )
>>> ebook
<Book: The Old Man and the Sea>

In the case of concrete base class, it’s interesting to see what’s happening in the underlying database. Let’s look at the tables created by Django in the database:

> \d concrete_base_model_product

Column |          Type          |                         Default
--------+-----------------------+---------------------------------------------------------
id     | integer                | nextval('concrete_base_model_product_id_seq'::regclass)
name   | character varying(100) |
price  | integer                |

Indexes:
   "concrete_base_model_product_pkey" PRIMARY KEY, btree (id)

Referenced by:
   TABLE "concrete_base_model_cart_items" CONSTRAINT "..." FOREIGN KEY (product_id) 
   REFERENCES concrete_base_model_product(id) DEFERRABLE INITIALLY DEFERRED

   TABLE "concrete_base_model_book" CONSTRAINT "..." FOREIGN KEY (product_ptr_id) 
   REFERENCES concrete_base_model_product(id) DEFERRABLE INITIALLY DEFERRED

   TABLE "concrete_base_model_ebook" CONSTRAINT "..." FOREIGN KEY (product_ptr_id) 
   REFERENCES concrete_base_model_product(id) DEFERRABLE INITIALLY DEFERRED

The product table has two familiar fields: name and price. These are the common fields you defined in the Product model. Django also created an ID primary key for you.

In the constraints section, you see multiple tables that are referencing the product table. Two tables that stand out are concrete_base_model_book and concrete_base_model_ebook:

> \d concrete_base_model_book

    Column     |  Type
---------------+---------
product_ptr_id | integer
weight         | integer

Indexes:
   "concrete_base_model_book_pkey" PRIMARY KEY, btree (product_ptr_id)

Foreign-key constraints:
   "..." FOREIGN KEY (product_ptr_id) REFERENCES concrete_base_model_product(id) 
   DEFERRABLE INITIALLY DEFERRED

The Book model has only two fields:

weight is the field you added in the derived Book model.
product_ptr_id is both the primary of the table and a foreign key to the base product model.

Behind the scenes, Django created a base table for product. Then, for each derived model, Django created another table that includes the additional fields, and a field that acts both as a primary key and a foreign key to the product table.

Let’s take a look at a query generated by Django to fetch a single book. Here are the results of print(Book.objects.filter(pk=1).query):

SELECT
    "concrete_base_model_product"."id",
    "concrete_base_model_product"."name",
    "concrete_base_model_product"."price",
    "concrete_base_model_book"."product_ptr_id",
    "concrete_base_model_book"."weight"
FROM
    "concrete_base_model_book"
    INNER JOIN "concrete_base_model_product" ON
        "concrete_base_model_book"."product_ptr_id" = "concrete_base_model_product"."id"
WHERE
    "concrete_base_model_book"."product_ptr_id" = 1

To fetch a single book, Django joined concrete_base_model_product and concrete_base_model_book on the product_ptr_id field. The name and price are in the product table and the weight is in the book table.

Since all the products are managed in the Product table, you can now reference it in a foreign key from the Cart model:

class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )
    items = models.ManyToManyField(Product)

Adding items to the cart is the same as before:

>>>

>>> from concrete_base_model.models import Cart
>>> cart = Cart.objects.create(user=user)
>>> cart.items.add(book, ebook)
>>> cart.items.all()
<QuerySet [<Book: Python Tricks>, <Book: The Old Man and the Sea>]>

Working with common fields is also simple:

>>>

>>> from django.db.models import Sum
>>> cart.items.aggregate(total_price=Sum('price'))
{'total_price': 2500}

Migrating base classes in Django:

When a derived model is created, Django adds a bases attribute to the migration:

  migrations.CreateModel(
      name='Book',
      fields=[...],
      bases=('concrete_base_model.product',),
  ),

If in the future you remove or change the base class, Django might not be able to perform the migration automatically. You might get this error:

TypeError: metaclass conflict: the metaclass of a derived class must 
be a (non-strict) subclass of the metaclasses of all its bases

This is a known issue in Django (#23818, #23521, #26488). To work around it, you must edit the original migration manually and adjust the bases attribute.

Pros

Primary key is consistent across all types: The product is issued by a single sequence in the base table. This restriction can be easily resolved by using a UUID instead of a sequence.
Common attributes can be queried from a single table: Common queries such as total price, list of product names, and prices can be fetched directly from the base table.

Cons

New product types require schema changes: A new type requires a new model.
Can produce inefficient queries: The data for a single item is in two database tables. Fetching a product requires a join with the base table.
Cannot access extended data from base class instance: A type field is required to downcast an item. This adds complexity to the code. django-polymorphic is a popular module that might eliminate some of these challenges.

Use Case

The concrete base model approach is useful when common fields in the base class are sufficient to satisfy most common queries.

For example, if you often need to query for the cart total price, show a list of items in the cart, or run ad hoc analytic queries on the cart model, you can benefit from having all the common attributes in a single database table.

Generic Foreign Key

Inheritance can sometimes be a nasty business. It forces you to create (possibly premature) abstractions, and it doesn’t always fit nicely into the ORM.

The main problem you have is referencing different products from the cart model. You first tried to squash all the product types into one model (sparse model, semi-structured model), and you got clutter. Then you tried splitting products into separate models and providing a unified interface using a concrete base model. You got a complicated schema and a lot of joins.

Django offers a special way of referencing any model in the project called GenericForeignKey. Generic foreign keys are part of the Content Types framework built into Django. The content type framework is used by Django itself to keep track of models. This is necessary for some core capabilities such as migrations and permissions.

To better understand what content types are and how they facilitate generic foreign keys, let’s look at the content type related to the Book model:

>>>

>>> from django.contrib.contenttypes.models import ContentType
>>> ct = ContentType.objects.get_for_model(Book)
>>> vars(ct)
{'_state': <django.db.models.base.ModelState at 0x7f1c9ea64400>,
'id': 22,
'app_label': 'concrete_base_model',
'model': 'book'}

Each model has a unique identifier. If you want to reference a book with PK 54, you can say, “Get object with PK 54 in the model represented by content type 22.”

GenericForeignKey is implemented exactly like that. To create a generic foreign key, you define two fields:

A reference to a content type (the model)
The primary key of the referenced object (the model instance’s pk attribute)

To implement a many-to-many relation using GenericForeignKey, you need to manually create a model to connect carts with items.

The Cart model remains roughly similar to what you have seen so far:

from django.db import models
from django.contrib.auth import get_user_model


class Cart(models.Model):
    user = models.OneToOneField(
        get_user_model(),
        primary_key=True,
        on_delete=models.CASCADE,
    )

Unlike previous Cart models, this Cart no longer includes a ManyToMany field. You are going need to do that yourself.

To represent a single item in the cart, you need to reference both the cart and any product:

from django.contrib.contenttypes.fields import GenericForeignKey
from django.contrib.contenttypes.models import ContentType


class CartItem(models.Model):
    cart = models.ForeignKey(
        Cart,
        on_delete=models.CASCADE,
        related_name='items',
    )
    product_object_id = models.IntegerField()
    product_content_type = models.ForeignKey(
        ContentType,
        on_delete=models.PROTECT,
    )
    product = GenericForeignKey(
        'product_content_type',
        'product_object_id',
    )

To add a new item in the Cart, you provide the content type and the primary key:

>>>

>>> book = Book.objects.first()

>>> CartItem.objects.create(
...     product_content_type=ContentType.objects.get_for_model(book),
...     product_object_id=book.pk,
... )
>>> ebook = EBook.objects.first()

>>> CartItem.objects.create(
...    product_content_type=ContentType.objects.get_for_model(ebook),
...    product_object_id=ebook.pk,
... )

Adding an item to a cart is a common task. You can add a method on the cart to add any product to the cart:

class Cart(models.Model):

    # ...

    def add_item(self, product) -> 'CartItem':
        product_content_type = ContentType.objects.get_for_model(product)

        return CartItem.objects.create(
            cart=self,
            product_content_type=product_content_type,
            product_object_id=product.pk,
        )

Adding a new item to a cart is now much shorter:

>>>

>>> cart.add_item(book)
>>> cart.add_item(ebook)

Getting information about the items in the cart is also possible:

>>>

>>> cart.items.all()
<QuerySet [<CartItem: CartItem object (1)>, <CartItem: CartItem object (2)>]

>>> item = cart.items.first()
>>> item.product
<Book: Python Tricks>

>>> item.product.price
1000

So far so good. Where’s the catch?

Let’s try to calculate the total price of the products in the cart:

>>>

>>> from django.db.models import Sum
>>> cart.items.aggregate(total=Sum('product__price'))

FieldError: Field 'product' does not generate an automatic reverse 
relation and therefore cannot be used for reverse querying. 
If it is a GenericForeignKey, consider adding a GenericRelation.

Django tells us it isn’t possible to traverse the generic relation from the generic model to the referenced model. The reason for that is that Django has no idea which table to join to. Remember, the Item model can point to any ContentType.

The error message does mention a GenericRelation. Using a GenericRelation, you can define a reverse relation from the referenced model to the Item model. For example, you can define a reverse relation from the Book model to items of books:

from django.contrib.contenttypes.fields import GenericRelation

class Book(model.Model):
    # ...
    cart_items = GenericRelation(
        'CartItem',
        'product_object_id',
        'product_content_type_id',
        related_query_name='books',
    )

Using the reverse relation, you can answer questions like how many carts include a specific book:

>>>

>>> book.cart_items.count()
4

>>> CartItem.objects.filter(books__id=book.id).count()
4

The two statement are identical.

You still need to know the price of the entire cart. You already saw that fetching the price from each product table is impossible using the ORM. To do that, you have to iterate the items, fetch each item separately, and aggregate:

>>>

>>> sum(item.product.price for item in cart.items.all())
2500

This is one of the major disadvantages of generic foreign keys. The flexibility comes with a great performance cost. It’s very hard to optimize for performance using just the Django ORM.

Structural Subtyping

In the abstract and concrete base class approaches, you used nominal subtyping, which is based on a class hierarchy. Mypy is able to detect this form of relation between two classes and infer types from it.

In the generic relation approach, you used structural subtyping. Structural subtyping exists when a class implements all the methods and attributes of another class. This form of subtyping is very useful when you wish to avoid direct dependency between modules.

Mypy provides a way to utilize structural subtyping using Protocols.

You already identified a product entity with common methods and attributes. You can define a Protocol:

from typing_extensions import Protocol

class Product(Protocol):
    pk: int
    name: str
    price: int

    def __str__(self) -> str:
        ...

Note: The use of class attributes and ellipses (...) in the method definition are new features in Python 3.7. In earlier versions of Python, it isn’t possible to define a Protocol using this syntax. Instead of an ellipsis, methods should have pass in the body. Class attributes such as pk and name can be defined using the @attribute decorator, but it will not work with Django models.

You can now use the Product protocol to add type information. For example, in add_item(), you accept an instance of a product and add it to the cart:

def add_item(
    self,
    product: Product,
) -> 'CartItem':
    product_content_type = ContentType.objects.get_for_model(product)

    return CartItem.objects.create(
        cart=self,
        product_content_type=product_content_type,
        product_object_id=product.pk,
    )

Running mypy on this function will not yield any warnings. Let’s say you change product.pk to product.id, which is not defined in the Product protocol:

def add_item(
    self,
    product: Product,
) -> 'CartItem':
    product_content_type = ContentType.objects.get_for_model(product)

    return CartItem.objects.create(
        cart=self,
        product_content_type=product_content_type,
        product_object_id=product.id,
    )

You will get the following warning from Mypy:

$ mypy
models.py:62: error: "Product" has no attribute "id"

Note: Protocol is not yet a part of Mypy. It’s part of the complementary package called mypy_extentions. The package is developed by the Mypy team and includes features that they thought weren’t ready for the main Mypy package yet.

Pros

Migrations are not needed to add product types: The generic foreign key can reference any model. Adding a new type of product does not require migrations.
Any model can be used as an item: Using generic foreign key, any model can be referenced by the Item model.
Built-in admin support: Django has built-in support for generic foreign keys in the admin. It can inline, for example, information about the referenced models in the detail page.
Self-contained module: There is no direct dependency between the products module and the cart module. This makes this approach ideal for existing projects and pluggable modules.

Cons

Can produce inefficient queries: The ORM cannot determine in advance what models are referenced by the generic foreign key. This makes it very difficult for it to optimize queries that fetch multiple types of products.
Harder to understand and maintain: Generic foreign key eliminates some Django ORM features that require access to specific product models. Accessing information from the product models requires writing more code.
Typing requires Protocol: Mypy is unable to provide type checking for generic models. A Protocol is required.

Use Case

Generic foreign keys are a great choice for pluggable modules or existing projects. The use of GenericForeignKey and structural subtyping abstract any direct dependency between the modules.

In the bookstore example, the book and e-book models can exist in a separate app and new products can be added without changing the cart module. For existing projects, a Cart module can be added with minimal changes to existing code.

The patterns presented in this article play nicely together. Using a mixture of patterns, you can eliminate some of the disadvantages and optimize the schema for your use case.

For example, in the generic foreign key approach, you were unable to get the price of the entire cart quickly. You had to fetch each item separately and aggregate. You can address this specific concern by inlining the price of the product on the Item model (the sparse model approach). This will allow you to query only the Item model to get the total price very quickly.

Conclusion

In this article, you started with a small town bookstore and grew it to a big e-commerce website. You tackled different types of problems and adjusted your model to accommodate the changes. You learned that problems such as complex code and difficulty adding new programmers to the team are often symptoms of a larger problem. You learned how to identify these problems and solve them.

You now know how to plan and implement a polymorphic model using the Django ORM. You’re familiar with multiple approaches, and you understand their pros and cons. You’re able to analyze your use case and decide on the best course of action.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Corey Schafer

2018-12-31T14:00:00+00:00

For this week’s community interview, I am joined by Corey Schafer, of YouTube fame.

Corey is a full-time content creator publishing regular Python tutorials on YouTube. In this interview, we talk to Corey about his YouTube channel and his advice for budding YouTubers and content creators, getting his first developer job, and his passion for woodworking.

Ricky: Welcome to Real Python! We might as well start at the beginning. How’d you get into programming, and when did you start using Python?

Corey: Thanks for having me. I actually got started programming a little later in life than most people you’ve probably interviewed. I used to be a little self-conscious about getting started later, but now I try to be upfront about it so that others aren’t as intimidated if they are just getting started at an “older” age.

I suppose I technically got started in college when I began my degree in Computer Science, but even at that time, I wasn’t taking it very seriously. I would do enough to pass tests, but I wasn’t absorbing any of the information. I definitely wasn’t doing any side projects or using coding in any real-world applications.

It wasn’t until my mid-20s that I randomly applied to an internship for NASA at Kennedy Space Center and started to take programming seriously. To my surprise, I was chosen for the internship. Imposter syndrome set in full-force on day 1. I was definitely out of my league.

But the longer I worked there, the more I realized that these people weren’t superhuman. These were people just like me, with the exception that they’ve spent much more time putting in the work to master their skill.

I thought to myself, “If they can do it, there’s no reason I can’t.” So I left Cape Canaveral with a newfound motivation to really dive into programming and learn as much as I could. I really didn’t feel like I could call myself an actual programmer until my late 20s. I am in my early 30s now, so I feel like I haven’t even scratched the surface in terms of the skills I would like the master.

As far as learning Python, I didn’t start using it until about 4 years ago. I was a full-time front-end JavaScript developer at the West Virginia University GIS Technical Center doing some mapping work. We were using Python for some of the backend scripts, and I was assigned to maintain/update some of those. I found that I enjoyed that much more than the front-end JavaScript work and began using Python on a daily basis.

Ricky: You mentioned that you came to programming a little later in life (though still young). With that context in mind, what was your experience getting your first junior developer job, and do you have any advice for anyone looking for their first developer job later in life?

Corey: I got my first development job in my mid-20s, after my internship in Cape Canaveral. That job was at a small research company in West Virginia (WV). The competition isn’t as fierce in WV as it is in places like San Francisco or Silicon Valley, so I was able to land a position without much prior experience.

I was extremely intimidated when I first started there. I never wanted anyone to see any code that I had written out of fear that it would expose me as not knowing nearly as much as anyone else. I later came to find out that this is a common fear within the community.

Actually working as a developer full-time taught me much more than anything I had learned in school or self-study. Knowing the fundamentals is definitely helpful, but there’s no replacement for actually writing real-world applications and having your code critiqued by people who have been doing this for years.

It’s definitely uncomfortable for your mistakes to be on display for your coworkers to see, but once you get over that discomfort, you come out on the other side less likely to make the same mistakes in the future. Many of my viewers ask me, “How do you just know how to solve these problems?” Well, the truth is, many problems in programming are very similar.

Once you solve problems incorrectly over the years and are shown better and better ways, you eventually learn to recognize certain patterns and use the most efficient methods from the start. This isn’t a skill that most people have naturally. It is developed over years of trial and error.

Ricky: Most of our readers might recognize you from your very popular YouTube channel. I certainly heard of you through it. Not only is it on our Ultimate List of Python YouTube Channels, but I regularly watch your videos and even built one of my most recent Flask apps as a direct result of your Flask course. How have you found YouTube as a platform from a teaching point of view? Any surprising lessons learned?

Corey: Thanks. I was honored to have my channel featured in your article. YouTube is a terrific platform for learning, and I’m excited to see where online learning evolves in the future. Websites like YouTube have definitely lowered the barrier-to-entry for anyone who wants to create content.

Many people think you need a nice recording studio or some financial backing before you get started, but that is no longer the case. As long as you have a computer and a cell phone then you basically have all of the tools you need to get started. I have upgraded my equipment over time, but I first started on YouTube by doing screen recordings on a cheap laptop using the built-in laptop microphone.

For anyone who is thinking about starting a YouTube channel or creating content in general, I do have some lessons I’ve learned over time. I believe the most important lesson I’ve learned is that you should make content for yourself. I try not to create tutorials for topics for the sole purpose of them being popular or what will get the most views… I instead try to create the lessons that I wish I had when learning that topic.

Take notes while you learn certain subjects and keep track of what you found difficult to digest and why. If you get stuck on something, then most likely others are getting stuck on it as well. And once you find solutions to these problems, then you can look back and see if there is a way anyone could have explained it to you that would have helped you understand more easily. If there is, then be sure to pass that on to others.

I really believe that advice carries over into other fields. If you’re an educator, then make content that you personally would have found helpful. If you are a musician, then make music that you personally enjoy. If you’re a comedian, then tell jokes that you think are funny. If you do that, then most likely there are many people out there like yourself who will share your same mindset and love your content.

Ricky: What’s next for the YouTube channel going forward? Any plans to branch out into paid courses or other forms of teaching?

Corey: At the moment, I am sticking to YouTube courses. I feel very fortunate to have a platform that allows me to put my content out there for free and be funded by ad revenue. This allows people who can’t afford courses or who can’t pay for content at the moment to get access to all of my content.

Ideally, my content will always remain free as long as I have enough coming in to live on. It also wouldn’t be possible without generous supporters contributing through sites like Patreon and YouTube Channel Memberships. They support me financially on a monthly basis so that the content can remain free for those who can’t afford it. I would love to continue with this model into the future for as long as I can.

As far as branching out into other forms of teaching, I have thought about creating an online learning platform of some kind. I have used a lot of tools in my personal life that have helped me learn topics quickly. Tools such as spaced-repetition-learning applications and daily coding challenges.

I would love to tackle a project that brought these concepts to a learning platform in a way that would help students absorb material more quickly and fully. Any type of project like that will be something I wouldn’t work on for some time though. For now, I am focused on creating video content.

Ricky: You, sir, are a talented woodworker! Your stuff looks impressive. Is this a new hobby or a long-time passion of yours? How did you get started?

Corey: Oh, thank you for saying that. It is a hobby that I wish I had more time to explore. It is something I have been toying with for many years now. Personally, it is a fantastic way to clear my mind and drain any stress that’s built up.

Woodworking and programming can be similar in many ways. Sometimes I will start a woodworking project and have an idea of what I want the finished product to look like, but not a detailed understanding of where to begin. So you start by jotting down some outlines, then knock out the smaller components, and then after many hours of work you can combine all of that together into the result you were hoping for. There’s a lot of pride that comes with finishing those types of projects.

Ricky: Now for my last question. What else do you get up to in your spare time? What other hobbies and interests do you have, aside from Python? Any you’d like to share and/or plug?

Corey: Since I spend my professional career at a computer, I like to spend my spare time outside as much as possible. This includes hiking, kayaking, camping, swimming, and trips to the dog park. Now that I am working from home and have a more flexible schedule, my girlfriend and I would love to start traveling more. I believe I might mark some international PyCons on my calendar and use those as an excuse to visit some places we have wanted to see for a very long time, with the added bonus of meeting more folks from the Python community.

Thank you, Corey, for joining me this week. You can find Corey’s YouTube channel here. You can get in touch with him via Twitter or his website. And if you’ve benefited from Corey’s videos, consider supporting his efforts with via Patreon.

If there’s someone you’d like me to interview from the Python community, then reach out to me in the comments below, or send me a message on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Make a Location-Based Web App With Django and GeoDjango

2018-12-24T14:00:00+00:00

Throughout this tutorial, you’ll learn how to use Django and GeoDjango to build a location-based web application from scratch. You’ll be building a simple nearby shops application that lists the shops closest to a user’s location.

By the end of this tutorial, you’ll be able to:

Use Django to build a simple web application from scratch
Use the GeoDjango sub-framework to implement geolocation features in your Django application
Use a spatial database (PostgreSQL and PostGIS) to get benefits from the spatial features and easily implement location-aware web apps

Free Bonus: Click here to get free access to additional Django tutorials and resources you can use to deepen your Python web development skills.

Example App Source Code: The full source code for the app you’ll be building in this tutorial is available on the realpython/materials repository on GitHub.

The Tools You Will Be Using

You’ll be using the following tools to develop your nearby shops web application:

The Python programming language
The Django web framework
The PostgreSQL database for persisting data
The PostGIS extension for supporting spatial features in the PostgreSQL database
pip for installing dependencies
The venv module for managing a virtual environment
Docker for installing PostgreSQL and PostGIS

Before diving into practical steps, let’s first start by introducing the frameworks you’ll be using.

Django is the most popular Python framework for building web apps. It makes it easy for developers to quickly build prototypes and meet their project deadlines by providing a plethora of built-in APIs and sub-frameworks such as GeoDjango.

GeoDjango is a built-in application that is included as a contrib module in Django. It’s actually a complete framework itself that can also be used separately from Django. It provides a toolbox of utilities for building GIS web applications.

GIS stands for Geographic Information System. It’s an information system (an organized system for the collection, organization, storage, and communication of information) designed for processing and manipulating data that have geographic or spatial characteristics.

GeoDjango also provides Python bindings to popular spatial libraries such as GEOS, GDAL, and GeoIP, which can be used separately without Django in any Python application or interactively in the shell.

GeoDjango aims to provide a world-class geographic web framework. It has been refactored over the years with the goal of making it easier to work with geospatial data, in other words data that identifies the geographic location of natural or artificial features on Earth and is stored as coordinates and topologies.

GeoDjango integrates very well with the Django ORM and provides a set of geometry fields defined by the Open Geospatial Consortium (OGS) that can be used to map to different types of geometries in geospatial databases:

GeometryField is the base class for all geometric fields in GeoDjango.
PointField is used for storing GEOS Point objects.
PolygonField is used for storing GEOS Polygon objects and so on.

GeoDjango is a very powerful framework for storing and working with geographic data using the Django ORM. It provides an easy-to-use API to find the distances between two points on a map, areas of polygons, the points within a polygon, and so on.

To be able to work with GeoDjango, you’ll need to have two things: a spatial database and geospatial libraries. A spatial database is a database that is optimized for storing and querying data that represents objects defined in a geometric space.

To fully use all features of GeoDjango, you’ll need to install the following open-source geospatial libraries:

GEOS stands for Geometry Engine Open Source. It’s a C++ port of the JTS (Java Topology Suite) that implements the OCG Simple Feature for SQL specification.
GDAL stands for Geospatial Data Abstraction Library. It’s an open-source library for working with raster and vector geospatial data formats.
PROJ.4 is for Cartographic Projections library. It’s an open-source GIS library for easily working with spatial reference systems and projections.
GeoIP is a library that helps users find geographical information based on an IP address.

This tutorial makes use of an Ubuntu 18.04 system for installing prerequisites and a Unix bash for running the commands, but this shouldn’t be a problem for you if you’re using any other system, particularly Unix-based systems like macOS.

For most installation instructions, you’ll be using the aptitude package manager, so you should simply replace that with the equivalent package manager for your system.

Prerequisites

In this section, you’ll be installing the prerequisites needed before you can bootstrap your project, such as Python 3 and GeoDjango dependencies (GEOS, GDAL, and PROJ.4). You’ll also use Docker to set up a PostgreSQL and PostGIS database for your project.

Installing Python 3

There is a big chance that you already have Python 3 installed on your system. If you don’t, you can simply head to the official website and download the binaries for your operating system.

Depending on your system, you may also be able to install Python 3 or upgrade it to the latest version if it’s already installed by using the official package manager.

If you have a problem installing Python 3 or want more information, you can check the Python 3 Installation & Setup Guide, which provides different ways to install Python 3 on your system.

Finally, you can check if you have Python 3 installed by running the following command:

$ python3 --version
Python 3.6.5

Installing GeoDjango Dependencies (GEOS, GDAL, and PROJ.4)

GeoDjango requires a spatial database and a set of open-source geospatial libraries:

GEOS is an open-source geometry engine and a C++ port of the JTS (Java Topology Suite). It’s required by GeoDjango for performing geometric operations.
PROJ.4 is an open-source GIS library for easily working with spatial reference systems and projections. You need it because you’ll be using PostGIS as the spatial database.
GDAL is an open-source geospatial data abstraction library for working with raster and vector data formats. It’s needed for many utilities used by GeoDjango.

You can refer to the docs for more information about spatial databases and the required libraries.

GDAL 2.2 is included in Ubuntu 18.04 so you can simply run the following command to install it:

$ sudo aptitude install gdal-bin libgdal-dev
$ sudo aptitude install python3-gdal

Note: python3-gdal is the Python 3 binding for GDAL.

Next, you can install the other libraries using the following:

$ sudo aptitude install binutils libproj-dev

Note: Since you’re using a binary package for GEOS, you also need to install binutils.

Refer to the docs for detailed instructions about how to install these dependencies on macOS and Windows.

For more information about PROJ.4, you can refer to its official docs.

Setting up a Spatial Database With PostgreSQL and PostGIS

You’ll use PostgreSQL, the most commonly used database with Django. It’s not a spatial database, but thanks to PostGIS you can power up your database with powerful geospatial features.

PostGIS is a spatial database extension that needs to be installed on a PostgreSQL database, which gives it the capability to store and work with spatial data and perform spatial operations. It adds support for geographic objects allowing location queries to be run in SQL.

You can either install PostgreSQL on your system, create a database, and then add the PostGIS extension, or better yet use Docker to quickly create a database using the kartoza postgis image, which provides a container with PostgreSQL and PostGIS already installed:

$ docker run --name=postgis -d -e POSTGRES_USER=user001 -e POSTGRES_PASS=123456789 -e POSTGRES_DBNAME=gis -p 5432:5432 kartoza/postgis:9.6-2.4

After running the command, you’ll have a PostgreSQL server listening on the 5432 port with a database called gis. The database uses the user001 username and the 123456789 password.

Note: You need to have Docker installed on your system. For instructions, you can simply refer to the official docs.

Setting up Your Project

Now that you have a spatial database set up and ready, you can go ahead and setup your Django project. In this section, you’ll use venv to create an isolated virtual environment for your project and install all the required packages such as Django.

Creating a Virtual Environment

A virtual environment allows you to create an isolated environment for the dependencies of your current project. This will allow you to avoid conflicts between the same packages that have different versions.

In Python 3, you can create virtual environments using virtualenv or the venv module.

For more information about Python virtual environments, check out Python Virtual Environments: A Primer.

Now, head over to your terminal and run the following command to create a virtual environment based on Python 3:

$ python3 -m venv env
(env) $

Next, you need to activate the following command:

$ source env/bin/activate

That’s it. You now have your virtual environment activated, and you can install the packages for your project.

Installing Django

The first step after creating and activating a virtual environment is to install Django. The Django package is available from the Python Package Index (PyPI) so you can simply use pip to install it by running the following command in your terminal:

$ pip install django

Creating a Django Project

The project you’ll be creating is a web application that lists shops sorted by distance so your users will be able to discover the shops that are close to their location.

The web application makes use of GeoDjango for easily implementing location requirements like calculating the distances of shops from the user’s location and ordering the shops by distance.

Using GeoDjango, you can get and display the nearest shops that are stored in a PostgreSQL database configured with the PostGIS extension to enable spatial operations.

Now, you’re ready to create a Django project using the django-admin.py script. Simply run the following command:

$ django-admin.py startproject nearbyshops

This will create a project named nearbyshops.

Configuring the PostgreSQL Database

Now that you’ve created a project, let’s continue by configuring the connection to the PostgreSQL and PostGIS spatial database. Open the settings.py file and add django.contrib.gis.db.backends.postgis as the engine with the credentials for the PostGIS database you configured earlier:

DATABASES = {
    'default': {
        'ENGINE': 'django.contrib.gis.db.backends.postgis',
        'NAME': 'gis',
        'USER': 'user001',
        'PASSWORD': '123456789',
        'HOST': 'localhost',
        'PORT': '5432'
    }
}

Note: You need to change the database credentials accordingly if you didn’t specify the same credentials when running the Docker container.

If you try to run your Django server at this point, you’ll get the ImportError: No module named 'psycopg2' error related to psycopg2, which is the most popular PostgreSQL adapter for Python. To solve the error, you simply need to install the psycopg2-binary in your virtual environment using the following:

$ pip install psycopg2-binary

Adding GeoDjango

GeoDjango is a framework that makes it as easy as possible to build GIS and location aware web applications. You can add it by simply including the gis contrib module in the list of installed apps.

Open the settings.py file and locate the INSTALLED_APPS array. Then add the 'django.contrib.gis' module:

INSTALLED_APPS = [
    # [...]
    'django.contrib.gis'
]

Creating a Django Application

A Django project is made up of applications. By default, it contains several core or built-in apps like django.contrib.admin, but you will usually add at least one app that contains your custom project’s code.

Note: For simple projects, you may only need one app, but once your project becomes bigger and has different requirements, you can organize your code in multiple separate apps.

Now that you have created a Django project, configured the connection with the spatial database, and added GeoDjango to the project, you need to create a Django application that you may call shops.

The shops application will contain the code for creating and displaying the shops closest to a user’s location. In the next steps, you are going to perform the following tasks:

Create the app
Add a Shop model
Add a data migration for loading initial demo data (shops)
Add a view function
Add a template

First run the following command to create the app:

$ python manage.py startapp shops

Next, you need to add it to the list of installed apps in the settings.py file, which will make Django recognize it as a part of your project:

INSTALLED_APPS = [
    # [...]
    'shops'
]

Creating a Django Model

After creating the shops application, which will contain the actual code of your project, you need to add models in your app. Django uses an ORM (Object Relational Mapper), which is an abstraction layer between Django and the database that transforms Python objects (or models) into database tables.

In this case, you need one model that represents a shop in the database. You’ll create a Shop model that has the following fields:

name: the name of the shop
location: the location of the shop in latitude and longitude coordinates
address: the address of the shop
city: the city the shop is in

Open the shops/models.py file and add the following code:

from django.contrib.gis.db import models

class Shop(models.Model):
    name = models.CharField(max_length=100)
    location = models.PointField()
    address = models.CharField(max_length=100)
    city = models.CharField(max_length=50)

For the location, you are using the PointField, a GeoDjango-specific geometric field for storing a GEOS Point object that represents a pair of longitude and latitude coordinates.

The other fields are normal Django fields of type CharField that can be used to store strings of small and large size.

Note: Please note that the models module is imported from django.contrib.gis.db and not the usual django.db module.

Creating the Database Tables

With Django, you don’t need to use SQL to create the database tables thanks to its ORM. Let’s create the database tables by using the makemigrations and migrate commands. Head back to your terminal and run the following:

$ python manage.py makemigrations
$ python manage.py migrate

For more information about these commands, check out Django Migrations – A Primer.

Adding a Super User

You need to create a super user so you can access the admin interface. This can be done using the following command:

$ python manage.py createsuperuser

The prompt will ask you for the username, email, and password you want to use for accessing the user account. Enter them and hit Enter.

Registering the Model in the Admin Interface

Django’s admin application provides a complete CRUD interface for managing data.

GeoDjango extends the admin application to add support for working with geometry fields.

Before you can access your models from Django admin, you need to register them.

Open the shops/admin.py file and add the following code:

from django.contrib.gis.admin import OSMGeoAdmin
from .models import Shop

@admin.register(Shop)
class ShopAdmin(OSMGeoAdmin):
    list_display = ('name', 'location')

You are using the @admin.register decorator to register the Shop model in the admin application. The decorated class is a representation of the Shop model in the admin interface and allows you to customize different aspects such as the Shop fields that you want to display. (In your case, it’s the name and location.) For more information about decorators, you can read Primer on Python Decorators.

Since the Shop model includes a GeoDjango field, you need to use the special OSMGeoAdmin class that’s available from the django.contrib.gis.admin package .

You can either use GeoModelAdmin or OSMGeoAdmin, which is a subclass of GeoModelAdmin that uses an Open Street Map layer in the admin to display geometric fields. This provides more information like street and thoroughfare details than would be available with the GeoModelAdmin class, which uses Vector Map Level 0.

You can now run the Django server:

$ python manage.py runserver

You application will be running from localhost:8000, and you can access the admin interface from localhost:8000/admin.

This is a screenshot from the Add shop interface:

You can see that the Location geometric field is displayed as an interactive map. You can zoom the map in and out, and you can choose different selectors at the top right corner of the map to select a location, which is marked by the green circle.

Adding Initial Data

You need some initial demo data for your application, but instead of manually adding data, you can use a data migration.

Data migrations can be used for multiple scenarios, including adding initial data in your database. For more information, check out Data Migrations.

Before creating a migration, let’s first get some real-world data from OpenStreetMap using overpass turbo, a web-based data filtering tool for OpenStreetMap. You can run Overpass API queries and analyze the resulting data interactively on the map.

You can also use the integrated Wizard, which makes it easy to create queries.

In your case, you want to get all the shops in a city. Simply click on the Wizard button. A small window will pop up. In the text field, write a query like “shop in Miami” and click on build and run query.

Next, click on the export button and click on download/copy as raw OSM data to download a JSON file that contains the raw OSM data. Save the file as data.json in your project’s root folder:

This is a screenshot of example data from the file:

You need to get the objects in the elements array. Specifically the lat, lon, and tags (name) fields for each shop.

You can find more details on how you can write Overpass queries from this wiki.

Now, let’s create an empty migration for importing the content of the data.json file in the database, using the following command:

$ python manage.py makemigrations shops --empty

Open the migration file. It has the following code:

from django.db import migrations

class Migration(migrations.Migration):
    dependencies = [
        ('shops', '0001_initial'),
    ]
    operations = [
    ]

You next need to create load_data() to be executed by RunPython(). First, in the import area, add the following imports:

from django.db import migrations
import json
from django.contrib.gis.geos import fromstr
from pathlib import Path

You are importing the Path class from the pathlib package for accessing low-level system functions, the json package for working with JSON, the Django built-in migrations API, and fromstr(), part of the geos package.

Next, add load_data():

DATA_FILENAME = 'data.json'
def load_data(apps, schema_editor):
    Shop = apps.get_model('shops', 'Shop')
    jsonfile = Path(__file__).parents[2] / DATA_FILENAME

    with open(str(jsonfile)) as datafile:
        objects = json.load(datafile)
        for obj in objects['elements']:
            try:
                objType = obj['type']
                if objType == 'node':
                    tags = obj['tags']
                    name = tags.get('name','no-name')
                    longitude = obj.get('lon', 0)
                    latitude = obj.get('lat', 0)
                    location = fromstr(f'POINT({longitude} {latitude})', srid=4326)
                    Shop(name=name, location = location).save()
            except KeyError:
                pass

Let’s explain the code you’ve just added. You first construct the absolute path using the Path class of the pathlib library and open the data.json file. Next, you parse the JSON file into a Python object.

You loop through the elements object containing the locations and tags of shops. Inside the loop, you extract the name and the longitude and latitude coordinates. Then you use formstr() to return a valid GEOSGeometry object corresponding to the spatial data in the string that can be assigned to the location field of the Shop model. Finally, you create and save the instance of the Shop model corresponding to the extracted data.

You are using the with statement, so you don’t have to explicitly close the file, and an f-string for formatting the argument of fromstr().

fromstr() takes a srid as the second parameter. srid stands for Spatial Reference System Identifier. It’s a unique value to identify spatial reference systems (projection systems used for interpreting the data in the spatial database).

The 4326 srid is the most popular system used with PostGIS. It’s also known as WGS84, where units are specified in degrees of longitude and latitude. You can refer to spatialreference.org for a Django-powered database of spatial reference systems.

Next, add the migration class to execute the above function when you run the migrate command:

class Migration(migrations.Migration):

    dependencies = [
        ('shops', '0005_auto_20181018_2050'),
    ]

    operations = [
        migrations.RunPython(load_data)
    ]

That’s it. You can now return to your terminal and run the following:

$ python manage.py migrate

The data from the data.json file will be loaded on your database. Run your Django server and head to your admin interface. You should see your data in the table. In my case, this is a screenshot of a portion of the table:

Displaying Nearby Shops

At this point of the tutorial, you have created:

The shops application, which encapsulates the code for creating and getting nearby shops in your project
The Shop model and the corresponding tables in the database
The initial admin user for accessing the admin interface
The initial demo data for loading real-world shops in the database that you can play with without manually entering a lot of fake data

You also registered the Shop model in the admin application so you can create, update, delete, and list shops from the admin interface.

Next, you’ll add a view function using the generic ListView class that you can use to display a list of nearby shops. You also create an HTML template that will be used by the view function to render the shops and add the URL that will be used to display the shops.

Let’s start by adding a template and a view function that will be used to display the nearby shops from a user’s location.

Open the shops/views.py file and start by importing the necessary APIs:

from django.views import generic
from django.contrib.gis.geos import fromstr
from django.contrib.gis.db.models.functions import Distance
from .models import Shop

Next add a user_location variable where you can hard code a user location:

longitude = -80.191788
latitude = 25.761681

user_location = Point(longitude, latitude, srid=4326)

In this part, you’ll simply hard code the user’s location (the coordinates of Miami in the USA), but this ideally should be specified by the user or retrieved automatically from the user’s browser with their permission using JavaScript and the HTML5 GeoLocation API. You can scroll down to the middle of that page to see a live example of implementing the Geolocation API.

Finally, add the following view class:

class Home(generic.ListView):
    model = Shop
    context_object_name = 'shops'
    queryset = Shop.objects.annotate(distance=Distance('location',
    user_location)
    ).order_by('distance')[0:6]
    template_name = 'shops/index.html'

You are using the generic class-based ListView to create a view.

Class-based views are an alternative way to implement views as Python classes instead of functions. They are used to handle common use cases in web development without re-inventing the wheel. In this example, you’ve just sub-classed the ListView generic view and overridden the model, context_object_name, queryset, and template_name attributes to create a list view that handles HTTP requests without any extra code.

Now let’s focus on the queryset attribute. To get the nearby shops, you simply use .annotate() to annotate each object on the returned queryset with a distance annotation that’s calculated using Distance(), available from GeoDjango, between the location of each shop and the user’s location. You also order the returned queryset by the distance annotation and take only the nearest six shops.

You can learn more about class-based views from the official docs.

Next let’s add the shops/index.html template with the following content:

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <title>Nearby Shops</title>
    </head>
<body>
    <h1>Nearby Shops</h1>
    {% if shops %}
    <ul>
    {% for shop in shops %}
        <li>
        {{ shop.name }}: {{shop.distance}}
        </li>
    {% endfor %}
    </ul>
    {% endif %}
</body>
</html>

The nearest shops are available from the shops context object that you specified as the context_object_name in the class-based view. You loop through the shops object and you display the name and distance from the user’s location for each shop.

Finally, let’s add a URL to our urls.py file:

from django.urls import path
from shops import views

urlpatterns = [
    # [...]
    path('', views.ShopList.as_view())
]

You use .as_view() to return a callable view that takes a request and returns a response, which can be passed as the second parameter for path() that maps paths to views.

Now you can run your Django server. The home page will display a simple un-styled list with the nearest shops from the hard-coded user’s location. This is an example screenshot:

In the screenshot, each item in the list displays the name of the shop (before the colon) and the distance in meters from the user’s location (after the colon). The letter m refers to meters.

Note: The view function you used to display the results in the screenshot is just for testing the queryset annotated by Distance().

In the next part of the series, you’ll use a RESTful endpoint to return the nearest shops. Instead of a Django template, you’ll use Vue.js with some CSS styling for a better look, so stay tuned!

Wrap-Up

Congratulations on creating your location based web application using GeoDjango, which aims to become a world-class geographic framework for implementing GIS apps. You now have the basic skills that you can use to either add simple geolocation stuff to your applications or create GIS apps. You can read the GeoDjango docs for a complete resource of the available APIs and what you can do with them.

You also learned to use Docker to quickly pull and launch a PostgreSQL and PostGIS server. Docker can be used for more than that. It’s a containerization tool for spinning up isolated, reproducible application environments. You can read Django Development with Docker Compose and Machine if you want to learn how to containerize your Django project.

Nowadays, location aware apps (apps that know your location and help you discover nearby objects and services by offering you results based on your location) are all the rage. Using the knowledge you gained in this tutorial, you’ll be able to incorporate this modern feature in your apps developed with Django.

The only requirement, besides the dependencies of GeoDjango, is to use a spatial database (a database that’s capable of storing and manipulating spatial data). For PostgreSQL, one of the most popular database management systems used with Django, you can simply install the PostGIS extension with your database to turn it into a spatial database. Other popular databases like Oracle and MySQL have built-in support for spatial data.

Conclusion

Throughout this tutorial, you used PostgreSQL, PostGIS, Django, and GeoDjango to build a simple nearby shops web application.

Example App Source Code: The full source code for the app you’ll be building in this tutorial is available on the realpython/materials repository on GitHub.

Free Bonus: Click here to get free access to additional Django tutorials and resources you can use to deepen your Python web development skills.

In the next tutorial, you’ll continue building the application with Django REST framework to expose a RESTful API, and you’ll use the JavaScript Vue.js library with the Axios HTTP client to consume the API and render the data.

You’ll also use the HTML5 Geolocation API to automatically get the user’s location from the user’s browser instead of hard-coding it like you did in this tutorial, so stay tuned!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Write Beautiful Python Code With PEP 8

2018-12-19T14:00:00+00:00

PEP 8, sometimes spelled PEP8 or PEP-8, is a document that provides guidelines and best practices on how to write Python code. It was written in 2001 by Guido van Rossum, Barry Warsaw, and Nick Coghlan. The primary focus of PEP 8 is to improve the readability and consistency of Python code.

PEP stands for Python Enhancement Proposal, and there are several of them. A PEP is a document that describes new features proposed for Python and documents aspects of Python, like design and style, for the community.

This tutorial outlines the key guidelines laid out in PEP 8. It’s aimed at beginner to intermediate programmers, and as such I have not covered some of the most advanced topics. You can learn about these by reading the full PEP 8 documentation.

By the end of this tutorial, you’ll be able to:

Write Python code that conforms to PEP 8
Understand the reasoning behind the guidelines laid out in PEP 8
Set up your development environment so that you can start writing PEP 8 compliant Python code

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Why We Need PEP 8

“Readability counts.”

— The Zen of Python

PEP 8 exists to improve the readability of Python code. But why is readability so important? Why is writing readable code one of the guiding principles of the Python language?

As Guido van Rossum said, “Code is read much more often than it is written.” You may spend a few minutes, or a whole day, writing a piece of code to process user authentication. Once you’ve written it, you’re never going to write it again. But you’ll definitely have to read it again. That piece of code might remain part of a project you’re working on. Every time you go back to that file, you’ll have to remember what that code does and why you wrote it, so readability matters.

If you’re new to Python, it can be difficult to remember what a piece of code does a few days, or weeks, after you wrote it. If you follow PEP 8, you can be sure that you’ve named your variables well. You’ll know that you’ve added enough whitespace so it’s easier to follow logical steps in your code. You’ll also have commented your code well. All this will mean your code is more readable and easier to come back to. As a beginner, following the rules of PEP 8 can make learning Python a much more pleasant task.

Following PEP 8 is particularly important if you’re looking for a development job. Writing clear, readable code shows professionalism. It’ll tell an employer that you understand how to structure your code well.

If you have more experience writing Python code, then you may need to collaborate with others. Writing readable code here is crucial. Other people, who may have never met you or seen your coding style before, will have to read and understand your code. Having guidelines that you follow and recognize will make it easier for others to read your code.

Naming Conventions

“Explicit is better than implicit.”

— The Zen of Python

When you write Python code, you have to name a lot of things: variables, functions, classes, packages, and so on. Choosing sensible names will save you time and energy later. You’ll be able to figure out, from the name, what a certain variable, function, or class represents. You’ll also avoid using inappropriate names that might result in errors that are difficult to debug.

Note: Never use l, O, or I single letter names as these can be mistaken for 1 and 0, depending on typeface:

O = 2  # This may look like you're trying to reassign 2 to zero

Naming Styles

The table below outlines some of the common naming styles in Python code and when you should use them:

Type	Naming Convention	Examples
Function	Use a lowercase word or words. Separate words by underscores to improve readability.	`function`, `my_function`
Variable	Use a lowercase single letter, word, or words. Separate words with underscores to improve readability.	`x`, `var`, `my_variable`
Class	Start each word with a capital letter. Do not separate words with underscores. This style is called camel case.	`Model`, `MyClass`
Method	Use a lowercase word or words. Separate words with underscores to improve readability.	`class_method`, `method`
Constant	Use an uppercase single letter, word, or words. Separate words with underscores to improve readability.	`CONSTANT`, `MY_CONSTANT`, `MY_LONG_CONSTANT`
Module	Use a short, lowercase word or words. Separate words with underscores to improve readability.	`module.py`, `my_module.py`
Package	Use a short, lowercase word or words. Do not separate words with underscores.	`package`, `mypackage`

These are some of the common naming conventions and examples of how to use them. But in order to write readable code, you still have to be careful with your choice of letters and words. In addition to choosing the correct naming styles in your code, you also have to choose the names carefully. Below are a few pointers on how to do this as effectively as possible.

How to Choose Names

Choosing names for your variables, functions, classes, and so forth can be challenging. You should put a fair amount of thought into your naming choices when writing code as it will make your code more readable. The best way to name your objects in Python is to use descriptive names to make it clear what the object represents.

When naming variables, you may be tempted to choose simple, single-letter lowercase names, like x. But, unless you’re using x as the argument of a mathematical function, it’s not clear what x represents. Imagine you are storing a person’s name as a string, and you want to use string slicing to format their name differently. You could end up with something like this:

>>>

>>> # Not recommended
>>> x = 'John Smith'
>>> y, z = x.split()
>>> print(z, y, sep=', ')
'Smith, John'

This will work, but you’ll have to keep track of what x, y, and z represent. It may also be confusing for collaborators. A much clearer choice of names would be something like this:

>>>

>>> # Recommended
>>> name = 'John Smith'
>>> first_name, last_name = name.split()
>>> print(last_name, first_name, sep=', ')
'Smith, John'

Similarly, to reduce the amount of typing you do, it can be tempting to use abbreviations when choosing names. In the example below, I have defined a function db() that takes a single argument x and doubles it:

# Not recommended
def db(x):
    return x * 2

At first glance, this could seem like a sensible choice. db() could easily be an abbreviation for double. But imagine coming back to this code in a few days. You may have forgotten what you were trying to achieve with this function, and that would make guessing how you abbreviated it difficult.

The following example is much clearer. If you come back to this code a couple of days after writing it, you’ll still be able to read and understand the purpose of this function:

# Recommended
def multiply_by_two(x):
    return x * 2

The same philosophy applies to all other data types and objects in Python. Always try to use the most concise but descriptive names possible.

Code Layout

“Beautiful is better than ugly.”

— The Zen of Python

How you lay out your code has a huge role in how readable it is. In this section, you’ll learn how to add vertical whitespace to improve the readability of your code. You’ll also learn how to handle the 79 character line limit recommended in PEP 8.

Blank Lines

Vertical whitespace, or blank lines, can greatly improve the readability of your code. Code that’s bunched up together can be overwhelming and hard to read. Similarly, too many blank lines in your code makes it look very sparse, and the reader might need to scroll more than necessary. Below are three key guidelines on how to use vertical whitespace.

Surround top-level functions and classes with two blank lines. Top-level functions and classes should be fairly self-contained and handle separate functionality. It makes sense to put extra vertical space around them, so that it’s clear they are separate:

class MyFirstClass:
    pass


class MySecondClass:
    pass


def top_level_function():
    return None

Surround method definitions inside classes with a single blank line. Inside a class, functions are all related to one another. It’s good practice to leave only a single line between them:

class MyClass:
    def first_method(self):
        return None

    def second_method(self):
        return None

Use blank lines sparingly inside functions to show clear steps. Sometimes, a complicated function has to complete several steps before the return statement. To help the reader understand the logic inside the function, it can be helpful to leave a blank line between each step.

In the example below, there is a function to calculate the variance of a list. This is two-step problem, so I have indicated each step by leaving a blank line between them. There is also a blank line before the return statement. This helps the reader clearly see what’s returned:

def calculate_variance(number_list):
    sum_list = 0
    for number in number_list:
        sum_list = sum_list + number
    mean = sum_list / len(number_list)

    sum_squares = 0
    for number in number_list:
        sum_squares = sum_squares + number**2
    mean_squares = sum_squares / len(number_list)

    return mean_squares - mean**2

If you use vertical whitespace carefully, it can greatly improved the readability of your code. It helps the reader visually understand how your code splits up into sections, and how those sections relate to one another.

Maximum Line Length and Line Breaking

PEP 8 suggests lines should be limited to 79 characters. This is because it allows you to have multiple files open next to one another, while also avoiding line wrapping.

Of course, keeping statements to 79 characters or less is not always possible. PEP 8 outlines ways to allow statements to run over several lines.

Python will assume line continuation if code is contained within parentheses, brackets, or braces:

def function(arg_one, arg_two,
             arg_three, arg_four):
    return arg_one

If it is impossible to use implied continuation, then you can use backslashes to break lines instead:

from mypkg import example1, \
    example2, example3

However, if you can use implied continuation, then you should do so.

If line breaking needs to occur around binary operators, like + and *, it should occur before the operator. This rule stems from mathematics. Mathematicians agree that breaking before binary operators improves readability. Compare the following two examples.

Below is an example of breaking before a binary operator:

# Recommended
total = (first_variable
         + second_variable
         - third_variable)

You can immediately see which variable is being added or subtracted, as the operator is right next to the variable being operated on.

Now, let’s see an example of breaking after a binary operator:

# Not Recommended
total = (first_variable +
         second_variable -
         third_variable)

Here, it’s harder to see which variable is being added and which is subtracted.

Breaking before binary operators produces more readable code, so PEP 8 encourages it. Code that consistently breaks after a binary operator is still PEP 8 compliant. However, you’re encouraged to break before a binary operator.

Indentation

“There should be one—and preferably only one—obvious way to do it.”

— The Zen of Python

Indentation, or leading whitespace, is extremely important in Python. The indentation level of lines of code in Python determines how statements are grouped together.

Consider the following example:

x = 3
if x > 5:
    print('x is larger than 5')

The indented print statement lets Python know that it should only be executed if the if statement returns True. The same indentation applies to tell Python what code to execute when a function is called or what code belongs to a given class.

The key indentation rules laid out by PEP 8 are the following:

Use 4 consecutive spaces to indicate indentation.
Prefer spaces over tabs.

Tabs vs. Spaces

As mentioned above, you should use spaces instead of tabs when indenting code. You can adjust the settings in your text editor to output 4 spaces instead of a tab character, when you press the Tab key.

If you’re using Python 2 and have used a mixture of tabs and spaces to indent your code, you won’t see errors when trying to run it. To help you to check consistency, you can add a -t flag when running Python 2 code from the command line. The interpreter will issue warnings when you are inconsistent with your use of tabs and spaces:

$ python2 -t code.py
code.py: inconsistent use of tabs and spaces in indentation

If, instead, you use the -tt flag, the interpreter will issue errors instead of warnings, and your code will not run. The benefit of using this method is that the interpreter tells you where the inconsistencies are:

$ python2 -tt code.py
  File "code.py", line 3
    print(i, j)
             ^
TabError: inconsistent use of tabs and spaces in indentation

Python 3 does not allow mixing of tabs and spaces. Therefore, if you are using Python 3, then these errors are issued automatically:

$ python3 code.py
  File "code.py", line 3
    print(i, j)
              ^
TabError: inconsistent use of tabs and spaces in indentation

You can write Python code with either tabs or spaces indicating indentation. But, if you’re using Python 3, you must be consistent with your choice. Otherwise, your code will not run. PEP 8 recommends that you always use 4 consecutive spaces to indicate indentation.

Indentation Following Line Breaks

When you’re using line continuations to keep lines to under 79 characters, it is useful to use indentation to improve readability. It allows the reader to distinguish between two lines of code and a single line of code that spans two lines. There are two styles of indentation you can use.

The first of these is to align the indented block with the opening delimiter:

def function(arg_one, arg_two,
             arg_three, arg_four):
    return arg_one

Sometimes you can find that only 4 spaces are needed to align with the opening delimiter. This will often occur in if statements that span multiple lines as the if, space, and opening bracket make up 4 characters. In this case, it can be difficult to determine where the nested code block inside the if statement begins:

x = 5
if (x > 3 and
    x < 10):
    print(x)

In this case, PEP 8 provides two alternatives to help improve readability:

Add a comment after the final condition. Due to syntax highlighting in most editors, this will separate the conditions from the nested code:
```
x = 5
if (x > 3 and
    x < 10):
    # Both conditions satisfied
    print(x)
```

Add extra indentation on the line continuation:

x = 5
if (x > 3 and
        x < 10):
    print(x)

An alternative style of indentation following a line break is a hanging indent. This is a typographical term meaning that every line but the first in a paragraph or statement is indented. You can use a hanging indent to visually represent a continuation of a line of code. Here’s an example:

var = function(
    arg_one, arg_two,
    arg_three, arg_four)

Note: When you’re using a hanging indent, there must not be any arguments on the first line. The following example is not PEP 8 compliant:

# Not Recommended
var = function(arg_one, arg_two,
    arg_three, arg_four)

When using a hanging indent, add extra indentation to distinguish the continued line from code contained inside the function. The following example is difficult to read because the code inside the function is at the same indentation level as the continued lines:

# Not Recommended
def function(
    arg_one, arg_two,
    arg_three, arg_four):
    return arg_one

Instead, it’s better to use a double indent on the line continuation. This helps you to distinguish between function arguments and the function body, improving readability:

def function(
        arg_one, arg_two,
        arg_three, arg_four):
    return arg_one

When you write PEP 8 compliant code, the 79 character line limit forces you to add line breaks in your code. To improve readability, you should indent a continued line to show that it is a continued line. There are two ways of doing this. The first is to align the indented block with the opening delimiter. The second is to use a hanging indent. You are free to chose which method of indentation you use following a line break.

Where to Put the Closing Brace

Line continuations allow you to break lines inside parentheses, brackets, or braces. It’s easy to forget about the closing brace, but it’s important to put it somewhere sensible. Otherwise, it can confuse the reader. PEP 8 provides two options for the position of the closing brace in implied line continuations:

Line up the closing brace with the first non-whitespace character of the previous line:

list_of_numbers = [
    1, 2, 3,
    4, 5, 6,
    7, 8, 9
    ]

Line up the closing brace with the first character of the line that starts the construct:
```
list_of_numbers = [
    1, 2, 3,
    4, 5, 6,
    7, 8, 9
]
```

You are free to chose which option you use. But, as always, consistency is key, so try to stick to one of the above methods.

Comments

“If the implementation is hard to explain, it’s a bad idea.”

— The Zen of Python

You should use comments to document code as it’s written. It is important to document your code so that you, and any collaborators, can understand it. When you or someone else reads a comment, they should be able to easily understand the code the comment applies to and how it fits in with the rest of your code.

Here are some key points to remember when adding comments to your code:

Limit the line length of comments and docstrings to 72 characters.
Use complete sentences, starting with a capital letter.
Make sure to update comments if you change your code.

Block Comments

Use block comments to document a small section of code. They are useful when you have to write several lines of code to perform a single action, such as importing data from a file or updating a database entry. They are important as they help others understand the purpose and functionality of a given code block.

PEP 8 provides the following rules for writing block comments:

Indent block comments to the same level as the code they describe.
Start each line with a # followed by a single space.
Separate paragraphs by a line containing a single #.

Here is a block comment explaining the function of a for loop. Note that the sentence wraps to a new line to preserve the 79 character line limit:

for i in range(0, 10):
    # Loop over i ten times and print out the value of i, followed by a
    # new line character
    print(i, '\n')

Sometimes, if the code is very technical, then it is necessary to use more than one paragraph in a block comment:

def quadratic(a, b, c, x):
    # Calculate the solution to a quadratic equation using the quadratic
    # formula.
    #
    # There are always two solutions to a quadratic equation, x_1 and x_2.
    x_1 = (- b+(b**2-4*a*c)**(1/2)) / (2*a)
    x_2 = (- b-(b**2-4*a*c)**(1/2)) / (2*a)
    return x_1, x_2

If you’re ever in doubt as to what comment type is suitable, then block comments are often the way to go. Use them as much as possible throughout your code, but make sure to update them if you make changes to your code!

Inline Comments

Inline comments explain a single statement in a piece of code. They are useful to remind you, or explain to others, why a certain line of code is necessary. Here’s what PEP 8 has to say about them:

Use inline comments sparingly.
Write inline comments on the same line as the statement they refer to.
Separate inline comments by two or more spaces from the statement.
Start inline comments with a # and a single space, like block comments.
Don’t use them to explain the obvious.

Below is an example of an inline comment:

x = 5  # This is an inline comment

Sometimes, inline comments can seem necessary, but you can use better naming conventions instead. Here’s an example:

x = 'John Smith'  # Student Name

Here, the inline comment does give extra information. However using x as a variable name for a person’s name is bad practice. There’s no need for the inline comment if you rename your variable:

student_name = 'John Smith'

Finally, inline comments such as these are bad practice as they state the obvious and clutter code:

empty_list = []  # Initialize empty list

x = 5
x = x * 5  # Multiply x by 5

Inline comments are more specific than block comments, and it’s easy to add them when they’re not necessary, which leads to clutter. You could get away with only using block comments so, unless you are sure you need an inline comment, your code is more likely to be PEP 8 compliant if you stick to block comments.

Documentation Strings

Documentation strings, or docstrings, are strings enclosed in double (""") or single (''') quotation marks that appear on the first line of any function, class, method, or module. You can use them to explain and document a specific block of code. There is an entire PEP, PEP 257, that covers docstrings, but you’ll get a summary in this section.

The most important rules applying to docstrings are the following:

Surround docstrings with three double quotes on either side, as in """This is a docstring""".
Write them for all public modules, functions, classes, and methods.

Put the """ that ends a multiline docstring on a line by itself:

def quadratic(a, b, c, x):
    """Solve quadratic equation via the quadratic formula.

    A quadratic equation has the following form:
    ax**2 + bx + c = 0

    There always two solutions to a quadratic equation: x_1 & x_2.
    """
    x_1 = (- b+(b**2-4*a*c)**(1/2)) / (2*a)
    x_2 = (- b-(b**2-4*a*c)**(1/2)) / (2*a)

    return x_1, x_2

For one-line docstrings, keep the """ on the same line:

def quadratic(a, b, c, x):
    """Use the quadratic formula"""
    x_1 = (- b+(b**2-4*a*c)**(1/2)) / (2*a)
    x_2 = (- b-(b**2-4*a*c)**(1/2)) / (2*a)

    return x_1, x_2

For a more detailed article on documenting Python code, see Documenting Python Code: A Complete Guide by James Mertz.

Whitespace in Expressions and Statements

“Sparse is better than dense.”

— The Zen of Python

Whitespace can be very helpful in expressions and statements when used properly. If there is not enough whitespace, then code can be difficult to read, as it’s all bunched together. If there’s too much whitespace, then it can be difficult to visually combine related terms in a statement.

Whitespace Around Binary Operators

Surround the following binary operators with a single space on either side:

Assignment operators (=, +=, -=, and so forth)
Comparisons (==, !=, >, <. >=, <=) and (is, is not, in, not in)
Booleans (and, not, or)

Note: When = is used to assign a default value to a function argument, do not surround it with spaces.

# Recommended
def function(default_parameter=5):
    # ...


# Not recommended
def function(default_parameter = 5):
    # ...

When there’s more than one operator in a statement, adding a single space before and after each operator can look confusing. Instead, it is better to only add whitespace around the operators with the lowest priority, especially when performing mathematical manipulation. Here are a couple examples:

# Recommended
y = x**2 + 5
z = (x+y) * (x-y)

# Not Recommended
y = x ** 2 + 5
z = (x + y) * (x - y)

You can also apply this to if statements where there are multiple conditions:

# Not recommended
if x > 5 and x % 2 == 0:
    print('x is larger than 5 and divisible by 2!')

In the above example, the and operator has lowest priority. It may therefore be clearer to express the if statement as below:

# Recommended
if x>5 and x%2==0:
    print('x is larger than 5 and divisible by 2!')

You are free to choose which is clearer, with the caveat that you must use the same amount of whitespace either side of the operator.

The following is not acceptable:

# Definitely do not do this!
if x >5 and x% 2== 0:
    print('x is larger than 5 and divisible by 2!')

In slices, colons act as a binary operators. Therefore, the rules outlined in the previous section apply, and there should be the same amount of whitespace either side. The following examples of list slices are valid:

list[3:4]

# Treat the colon as the operator with lowest priority
list[x+1 : x+2]

# In an extended slice, both colons must be
# surrounded by the same amount of whitespace
list[3:4:5]
list[x+1 : x+2 : x+3]

# The space is omitted if a slice parameter is omitted
list[x+1 : x+2 :]

In summary, you should surround most operators with whitespace. However, there are some caveats to this rule, such as in function arguments or when you’re combining multiple operators in one statement.

When to Avoid Adding Whitespace

In some cases, adding whitespace can make code harder to read. Too much whitespace can make code overly sparse and difficult to follow. PEP 8 outlines very clear examples where whitespace is inappropriate.

The most important place to avoid adding whitespace is at the end of a line. This is known as trailing whitespace. It is invisible and can produce errors that are difficult to trace.

The following list outlines some cases where you should avoid adding whitespace:

Immediately inside parentheses, brackets, or braces:

# Recommended
my_list = [1, 2, 3]

# Not recommended
my_list = [ 1, 2, 3, ]

Before a comma, semicolon, or colon:

x = 5
y = 6

# Recommended
print(x, y)

# Not recommended
print(x , y)

Before the open parenthesis that starts the argument list of a function call:

def double(x):
    return x * 2

# Recommended
double(3)

# Not recommended
double (3)

Before the open bracket that starts an index or slice:

# Recommended
list[3]

# Not recommended
list [3]

Between a trailing comma and a closing parenthesis:

# Recommended
tuple = (1,)

# Not recommended
tuple = (1, )

To align assignment operators:

# Recommended
var1 = 5
var2 = 6
some_long_var = 7

# Not recommended
var1          = 5
var2          = 6
some_long_var = 7

Make sure that there is no trailing whitespace anywhere in your code. There are other cases where PEP 8 discourages adding extra whitespace, such as immediately inside brackets, as well as before commas and colons. You should also never add extra whitespace in order to align operators.

Programming Recommendations

“Simple is better than complex.”

— The Zen of Python

You will often find that there are several ways to perform a similar action in Python (and any other programming language for that matter). In this section, you’ll see some of the suggestions PEP 8 provides to remove that ambiguity and preserve consistency.

Don’t compare boolean values to True or False using the equivalence operator. You’ll often need to check if a boolean value is True or False. When doing so, it is intuitive to do this with a statement like the one below:

# Not recommended
my_bool = 6 > 5
if my_bool == True:
    return '6 is bigger than 5'

The use of the equivalence operator, ==, is unnecessary here. bool can only take values True or False. It is enough to write the following:

# Recommended
if my_bool:
    return '6 is bigger than 5'

This way of performing an if statement with a boolean requires less code and is simpler, so PEP 8 encourages it.

Use the fact that empty sequences are falsy in if statements. If you want to check whether a list is empty, you might be tempted to check the length of the list. If the list is empty, it’s length is 0 which is equivalent to False when used in an if statement. Here’s an example:

# Not recommended
my_list = []
if not len(my_list):
    print('List is empty!')

However, in Python any empty list, string, or tuple is falsy. We can therefore come up with a simpler alternative to the above:

# Recommended
my_list = []
if not my_list:
    print('List is empty!')

While both examples will print out List is empty!, the second option is simpler, so PEP 8 encourages it.

Use is not rather than not ... is in if statements. If you are trying to check whether a variable has a defined value, there are two options. The first is to evaluate an if statement with x is not None, as in the example below:

# Recommended
if x is not None:
    return 'x exists!'

A second option would be to evaluate x is None and then have an if statement based on not the outcome:

# Not recommended
if not x is None:
    return 'x exists!'

While both options will be evaluated correctly, the first is simpler, so PEP 8 encourages it.

Don’t use if x: when you mean if x is not None:. Sometimes, you may have a function with arguments that are None by default. A common mistake when checking if such an argument, arg, has been given a different value is to use the following:

# Not Recommended
if arg:
    # Do something with arg...

This code checks that arg is truthy. Instead, you want to check that arg is not None, so it would be better to use the following:

# Recommended
if arg is not None:
    # Do something with arg...

The mistake being made here is assuming that not None and truthy are equivalent. You could have set arg = []. As we saw above, empty lists are evaluated as falsy in Python. So, even though the argument arg has been assigned, the condition is not met, and so the code in the body of the if statement will not be executed.

Use .startswith() and .endswith() instead of slicing. If you were trying to check if a string word was prefixed, or suffixed, with the word cat, it might seem sensible to use list slicing. However, list slicing is prone to error, and you have to hardcode the number of characters in the prefix or suffix. It is also not clear to someone less familiar with Python list slicing what you are trying to achieve:

# Not recommended
if word[:3] == 'cat':
    print('The word starts with "cat"')

However, this is not as readable as using .startswith():

# Recommended
if word.startswith('cat'):
    print('The word starts with "cat"')

Similarly, the same principle applies when you’re checking for suffixes. The example below outlines how you might check whether a string ends in jpg:

# Not recommended
if file_name[-3:] == 'jpg':
    print('The file is a JPEG')

While the outcome is correct, the notation is a bit clunky and hard to read. Instead, you could use .endswith() as in the example below:

# Recommended
if file_name.endswith('jpg'):
    print('The file is a JPEG')

As with most of these programming recommendations, the goal is readability and simplicity. In Python, there are many different ways to perform the same action, so guidelines on which methods to chose are helpful.

When to Ignore PEP 8

The short answer to this question is never. If you follow PEP 8 to the letter, you can guarantee that you’ll have clean, professional, and readable code. This will benefit you as well as collaborators and potential employers.

However, some guidelines in PEP 8 are inconvenient in the following instances:

If complying with PEP 8 would break compatibility with existing software
If code surrounding what you’re working on is inconsistent with PEP 8
If code needs to remain compatible with older versions of Python

Tips and Tricks to Help Ensure Your Code Follows PEP 8

There is a lot to remember to make sure your code is PEP 8 compliant. It can be a tall order to remember all these rules when you’re developing code. It’s particularly time consuming to update past projects to be PEP 8 compliant. Luckily, there are tools that can help speed up this process. There are two classes of tools that you can use to enforce PEP 8 compliance: linters and autoformatters.

Linters

Linters are programs that analyze code and flag errors. They provide suggestions on how to fix the error. Linters are particularly useful when installed as extensions to your text editor, as they flag errors and stylistic problems while you write. In this section, you’ll see an outline of how the linters work, with links to the text editor extensions at the end.

The best linters for Python code are the following:

pycodestyle is a tool to check your Python code against some of the style conventions in PEP 8.

Install pycodestyle using pip:

$ pip install pycodestyle

You can run pycodestyle from the terminal using the following command:

$ pycodestyle code.py
code.py:1:17: E231 missing whitespace after ','
code.py:2:21: E231 missing whitespace after ','
code.py:6:19: E711 comparison to None should be 'if cond is None:'

flake8 is a tool that combines a debugger, pyflakes, with pycodestyle.

Install flake8 using pip:

$ pip install flake8

Run flake8 from the terminal using the following command:

$ flake8 code.py
code.py:1:17: E231 missing whitespace after ','
code.py:2:21: E231 missing whitespace after ','
code.py:3:17: E999 SyntaxError: invalid syntax
code.py:6:19: E711 comparison to None should be 'if cond is None:'

An example of the output is also shown.

Note: The extra line of output indicates a syntax error.

These are also available as extensions for Atom, Sublime Text, Visual Studio Code, and VIM. You can also find guides on setting up Sublime Text and VIM for Python development, as well as an overview of some popular text editors at Real Python.

Autoformatters

Autoformatters are programs that refactor your code to conform with PEP 8 automatically. Once such program is black, which autoformats code following most of the rules in PEP 8. One big difference is that it limits line length to 88 characters, rather than 79. However, you can overwrite this by adding a command line flag, as you’ll see in an example below.

Install black using pip. It requires Python 3.6+ to run:

$ pip install black

It can be run via the command line, as with the linters. Let’s say you start with the following code that isn’t PEP 8 compliant in a file called code.py:

for i in range(0,3):
    for j in range(0,3):
        if (i==2):
            print(i,j)

You can then run the following command via the command line:

$ black code.py
reformatted code.py
All done! ✨ 🍰 ✨

code.py will be automatically reformatted to look like this:

for i in range(0, 3):
    for j in range(0, 3):
        if i == 2:
            print(i, j)

If you want to alter the line length limit, then you can use the --line-length flag:

$ black --line-length=79 code.py
reformatted code.py
All done! ✨ 🍰 ✨

Two other autoformatters, autopep8 and yapf, perform actions that are similar to what black does.

Another Real Python tutorial, Python Code Quality: Tools & Best Practices by Alexander van Tol, gives a thorough explanation of how to use these tools.

Conclusion

You now know how to write high-quality, readable Python code by using the guidelines laid out in PEP 8. While the guidelines can seem pedantic, following them can really improve your code, especially when it comes to sharing your code with potential employers or collaborators.

In this tutorial, you learned:

What PEP 8 is and why it exists
Why you should aim to write PEP 8 compliant code
How to write code that is PEP 8 compliant

On top of all this, you also saw how to use linters and autoformatters to check your code against PEP 8 guidelines.

If you want to learn more about PEP 8, then you can read the full documentation, or visit pep8.org, which contains the same information but has been nicely formatted. In these documents, you will find the rest of the PEP 8 guidelines not covered in this tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

A Pythonista's Holiday Wish List

2018-12-17T14:00:00+00:00

It’s that time of year again when everyone is looking to get last minute gifts. Whether you’re a friend of a Python developer or one yourself, I’ve got the perfect wish list if you’re looking to get a little something special. I’ve combed through the interwebs and polled my fellow Real Python authors to find the best presents for Pythonistas.

I’ve broken up these suggestions into five categories:

Good causes
Learning resources
Python swag
Hardware
Gadgets

Donating to a Good Cause

The greatest gift that anyone can receive is the feeling of giving to someone else. Whether you want to make a donation yourself or make one in someone’s name, here are some great Python related organizations that could use a donation:

Python Software Foundation (PSF): the official organization that maintains Python and hosts the Python Conference each year
Django Software Foundation (DSF): the official organization that maintains Django, our favorite web app framework
PyLadies: an organization dedicated to helping more women become participants and leaders in the Python open-source community

Donating your money isn’t the only thing that you can do to help the Python Community. You can also donate your time as a volunteer.

Python Learning Resources

I think that one of the greatest gifts that can be given is knowledge. Want to get a good read that’s Python related? Check out The Best Python Books for a great collection of Python related books. Maybe you want something a little more interactive? Check out the courses at Real Python!

One of the best ways to learn about Python is to learn from the masters themselves. A ticket to PyCon 2019 would be the greatest learning gift of all: three days of classes, demonstrations, coding challenges, and rubbing shoulders with the people who dream in Python code! This would be the ultimate gift that would keep on giving. You’ll also get swag bags that will last you all year round.

Python Stickers, T-Shirts, and Coffee Mugs… Oh My!

We’re Pythonistas, so why not show off the programming language we love most! I love showing off my love for Python on my laptop.

Nerdlettering has got your back for all your sticker needs, and if stickers aren’t your thing, then perhaps a sweatshirt or t-shirt might be better. Also don’t forget about that old coffee mug of yours. It could probably use an upgrade as well.

Want something that sports the Real Python logo? They’ve got you covered as well.

Hardware

With the way that tech is advancing every day, every programmer needs an upgrade of their hardware. This may be because you’re running out of hard drive space, have worn out keyboards, or just want something fun. (Or maybe you want something more advanced. Everyone loves a good challenge right?) This is my recommended hardware to check out this season.

External HDD

Whether you have a massive code base or are running several hundred Docker images, you can quickly run out of space to do all your work. Good thing the cost of hard drive capacity is almost constantly dropping! Plus, with USB 3.0, it’s almost as fast or even faster than an internal hard drive. You can get 2 TB of portable space for less than $60! Need something bigger? For a little more than double the price, you can get 8 TB! I recommend sticking to the name brands like Seagate and Western Digital if you can.

A New Mechanical Keyboard

As developers, we’re using the keyboard 99% of the time we’re on the computer. That means that it’s important to have the right keyboard that feels right for you. Don’t know what to get? The folks over at the MechanicalKeyboards subreddit have created a great guide to all things mechanical keyboards.

Recommending a keyboard is like recommending underwear: you never know how it’s going to feel for anyone else, so I highly recommend that you look at the guide above. But if you want my opinion, the keyboard that I use and think feels the best is the Microsoft Sculpt Ergo Keyboard.

Headphones

Let’s face it: there’s nothing like getting into the zone with your favorite music set and crunching away at your codebase. So why not make sure that you get a good set of headphones? I love my Beyerdynamic DT 770 Pro headphones and can’t imagine coding without them. When you’re buying headphones, I recommend using this useful guide created by the helpful folks over at the headphones subreddit.

Gadgets

It’s always fun to have new toys to play with, and us Python developers have a lot to choose from. Here, we’ve collected our favorite gadgets that use Python.

Raspberry Pi

Let’s start off with a little pie—Raspberry Pi that is. There are so many things that you can do with a Raspberry Pi, including the following:

With its starting price of only $35, it’s amazing that you’re getting a multi-core, Bluetooth and WiFi enabled device! I highly recommend getting a kit to get started.

The Open Source Rover

Speaking of Raspberry Pi, how about creating your own Mars Science Lab! NASA’s Jet Propulsion Lab (JPL), the creators of the Mars Rovers, has created an Open Source replica of the Mars Science Laboratory rover that uses the Raspberry Pi.

This comes with a hefty price tag (~$2,500) and isn’t for the faint of heart as it requires purchasing each part separately and assembling the entire rover. Don’t worry about programming the rover though. That was already done by JPL in Python! If the price and difficulty don’t scare you off, then this is a great gift for anyone wanting to learn about robotics.

Vector and Cozmo Robots From Anki

If making your own robot seems a little daunting, then check out Vector and Cozmo. Vector is “the good robot” who can help out with multiple things such as a kitchen timer, taking a selfie, or telling you what the weather is going to be like. If that’s not enough, there is the Python Software Development Kit (SDK) that enables you to make Vector fully customizable!

If you’re new to programming or want to teach someone about programming robots, then Cozmo’s got you covered. With both a full Python SDK and a more simple option available, this robot is a good choice for the beginner and the seasoned coder.

DJI Drones

A little tired of being stuck on the ground? How about flying? DJI has multiple versions of their drones that you can take flight with either manually or automated using their software. If they don’t have the feature you’re looking for, how about checking out the SDK and doing it yourself! DJI has a lot of drone options available, but I’ve got my eye on the Tello, since it’s on the cheaper end and has an easier-to-use Python Library.

Happy Pythonic Holidays

Whatever you decide to get you or your Python developer friends, be sure to enjoy this holiday season and keep on Pythoning! I’m about to push checkout on my Amazon cart now and tell my wife that #PythonMadeMeDoIt.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Thonny: The Beginner-Friendly Python Editor

2018-12-12T14:00:00+00:00

Are you a Python beginner looking for a tool that can support your learning? This article is for you! Every programmer needs a place to write their code. This article will discuss an awesome tool called Thonny that will enable you to start working with Python in a beginner-friendly environment.

In this article, you’ll learn:

How to install Thonny on your computer
How to navigate Thonny’s user interface to use its built-in features
How to use Thonny to write and run your code
How to use Thonny to debug your code

By the end of this article, you’ll be comfortable with the development workflow in Thonny and ready to use it for your Python learning.

So what is Thonny? Great question!

Thonny is a free Python Integrated Development Environment (IDE) that was especially designed with the beginner Pythonista in mind. Specifically, it has a built-in debugger that can help when you run into nasty bugs, and it offers the ability to do step through expression evaluation, among other really awesome features.

Free Sample Chapter: Download a free sample chapter from the Real Python course and gain practical Python programming skills.

Installing Thonny

This article assumes that you have Python 3 installed on your computer. If not, please review Python 3 Installation & Setup.

Web Download

The web download can be accessed via a web browser by visiting the Thonny website. Once on the page, you will see a light gray box in the top right corner like this:

Once you’ve found the gray box, click the appropriate link for your operating system. This tutorial assumes you’ve downloaded version 3.0.1.

Command Line Download

You can also install Thonny via your system’s command line. On Windows, you can do this by starting a program called Command Prompt, while on macOS and Linux you start a program called Terminal. Once you’ve done that, enter the following command:

$ pip install thonny

The User Interface

Let’s make sure you understand what Thonny has to offer. Think of Thonny as the workroom in which you will create amazing Python projects. Your workroom contains a toolbox containing many tools that will enable you to be a rock star Pythonista. In this section, you’ll learn about each of the features of the UI that’ll help you use each of the tools in your Thonny toolbox.

The Code Editor and Shell

Now that you have Thonny installed, open the application. You should see a window with several icons across the top, and two white areas:

Notice the two main sections of the window. The top section is your code editor, where you will write all of your code. The bottom half is your Shell, where you will see outputs from your code.

The Icons

Across the top you’ll see several icons. Let’s explore what each of them does. You’ll see an image of the icons below, with a letter above each one. We will use these letters to talk about each of the icons:

Working our way from left to right, below is a description of each of the icons in the image.

A: The paper icon allows you to create a new file. Typically in Python you want to separate your programs into separate files. You’ll use this button later in the tutorial to create your first program in Thonny!

B: The open folder icon allows you to open a file that already exists on your computer. This might be useful if you come back to a program that you worked on previously.

C: The floppy disk icon allows you to save your code. Press this early and often. You’ll use this later to save your first Thonny Python program.

D: The play icon allows you to run your code. Remember that the code you write is meant to be executed. Running your code means you’re telling Python, “Do what I told you to do!” (In other words, “Read through my code and execute what I wrote.”)

E: The bug icon allows you to debug your code. It’s inevitable that you will encounter bugs when you’re writing code. A bug is another word for a problem. Bugs can come in many forms, sometimes appearing when you use inappropriate syntax and sometimes when your logic is incorrect.

Thonny’s bug button is typically used to spot and investigate bugs. You’ll work with this later in the tutorial. By the way, if you’re wondering why they’re called bugs, there’s also a fun story of how it came about!

F-H: The arrow icons allow you to run your programs step by step. This can be very useful when you’re debugging or, in other words, trying to find those nasty bugs in your code. These icons are used after you press the bug icon. You’ll notice as you hit each arrow, a yellow highlighted bar will indicate which line or section Python is currently evaluating:

The F arrow tells Python to take a big step, meaning jumping to the next line or block of code.
The G arrow tells Python to take a small step, meaning diving deep into each component of an expression.
The H arrow tells Python to exit out of the debugger.

I: The resume icon allows you to return to play mode from debug mode. This is useful in the instance when you no longer want to go step by step through the code, and instead want your program to finish running.

J: The stop icon allows you to stop running your code. This can be particularly useful if, let’s say, your code runs a program that opens a new window, and you want to stop that program. You’ll use the stop icon later in the tutorial.

Let’s Try It!

Get ready to write your first official Python program in Thonny:

Enter the following code into the code editor:
```
print("Hello World")
```
Click the play button to run your program.
See the output in the Shell window.
Click the play button again to see that it says hello one more time.

Congratulations! You’ve now completed your first program in Thonny! You should see Hello world! printed inside the Shell, also known as the console. This is because your program told Python to print this phrase, and the console is where you see the output of this execution.

Other UI Features

To see more of the other features that Thonny has to offer, navigate to the menu bar and select the View dropdown. You should see that Shell has a check mark next to it, which is why you see the Shell section in Thonny’s application window:

Let’s explore some of the other offerings, specifically those that will be useful to a beginning Pythonista:

Help: You’ll select the Help view if you want more information about working with Thonny. Currently this section offers more reading on the following topics: Running Programs Step-wise, how to install 3rd Party Packages, or using Scientific Python Packages.
Variables: This feature can be very valuable. A variable in Python is a value that you define in code. Variables can be numbers, strings, or other complex data structures. This section allows you to see the values assigned to all of the variables in your program.
Assistant: The Assistant is there to give you helpful hints when you hit Exceptions or other types of errors.

The other features will become useful as you advance your skills. Check them out once you get more comfortable with Thonny!

The Code Editor

Now that you have an understanding of the UI, let’s use Thonny to write another little program. In this section, you’ll go through the features of Thonny that will help guide you through your development workflow.

Write Some Code

In the code editor (top portion of the UI), add the following function:

def factorial(num):
    if num == 1:
        return 1
    else:
        return num * factorial(num - 1)

print(factorial(3))

Save Your Code

Before we move on, let’s save your program. Last time, you were prompted to do this after pressing the play button. You can also do this by clicking the blue floppy disk icon or by going to the menu bar and selecting File > Save. Let’s call the program factorial.py.

Run Your Code

In order to run your code, find and press the play icon. The output should look like this:

Debug Your Code

To truly understand what this function is doing, try the step feature. Take a few large and small steps through the function to see what is happening. Remember you can do this by pressing the arrow icons:

As you can see, the steps will show how the computer is evaluating each part of the code. Each pop up window is like a piece of scratch paper that the computer is using to compute each portion of the code. Without this awesome feature, this may have been hard to conceptualize—but now you’ve got it!

Stop Running Your Code

So far, there hasn’t been a need to hit the stop icon for this program, particularly because it exits as soon as it has executed print(). Try increasing the number being passed to the factorial function to 100:

def factorial(num):
    if num == 1:
        return 1
    else:
        return num * factorial(num - 1)

print(factorial(100))

Then step through the function. After a while, you will notice that you will be clicking for a long time to reach the end. This is a good time to use the stop button. The stop button can be really useful to stop a program that is either intentionally or unintentionally running continuously.

Find Syntax Errors in Your Code

Now that you have a simple program that works, let’s break it! By intentionally creating an error in your factorial program, you’ll be able to see how Thonny handles these types of issues.

We will be creating what is called a syntax error. A syntax error is an error that indicates that your code is syntactically incorrect. In other words, your code does not follow the proper way to write Python. When Python notices the error, it will display a syntax error to complain about your invalid code.

Above the print statement, let’s add another print statement that says print("The factorial of 100 is:"). Now let’s go ahead and create syntax errors. In the first print statement, remove the second quotation mark, and in the other remove the second parenthesis.

As you do this, you should see that Thonny will highlight your SyntaxErrors. Missing quotations are highlighted in green, and missing parenthesis are in gray:

For beginners, this is a great resource that will allow you to help spot any typos while you’re writing. Some of the most common and frustrating errors when you start programming are missing quotes and mismatched parentheses.

If you have your Assistant View turned on, you will also notice that it will give you a helpful message to guide you in the right direction when you are debugging:

As you get more comfortable with Thonny, the Assistant can be a useful tool to help you get unstuck!

The Package Manager

As you continue to learn Python, it can be quite useful to download a Python package to use inside of your code. This allows you to use code that someone else has written inside of your program.

Consider an example where you want to do some calculations in your code. Instead of writing your own calculator, you might want to use a third-party package called simplecalculator. In order to do this, you’ll use Thonny’s package manager.

The package manager will allow you to install packages that you will need to use with your program. Specifically, it allows you to add more tools to your toolbox. Thonny has the built-in benefit of handling any conflicts with other Python interpreters.

To access the package manager, go to the menu bar and select Tools > Manage Packages… This should pop open a new window with a search field. Type simplecalculator into that field and click the Search button.

The output should look similar to this:

Go ahead and click Install to install this package. You will see a small window pop up showing the system’s logs while it installs the package. Once it completes, you are ready to use simplecalculator in your code!

In the next section, you will use the simplecalculator package along with some of the other skills you’ve learned in this tutorial to create a simple calculator program.

Check Your Understanding

You’ve learned so much about Thonny so far! Here’s what you’ve learned:

Where to write your code
How to save your code
How to run your code
How to stop your code from running
Where to see your code execute
How to spot SyntaxErrors
How to install third party packages

Let’s check your understanding of these concepts.

Now that you have simplecalculator installed, let’s create a simple program that will use this package. You’ll also use this as an opportunity to check that you understand some of the UI and development features that you’ve learned thus far in the tutorial.

Part 1: Create a File, Add Some Code, and Understand the Code

In Part 1, you will create a file, and add some code to it! Do your best to try to dig into what the code is actually doing. If you get stuck, check out the Take a Deeper Look window. Let’s get started:

Start a new file.
Add the following code into your Thonny code editor:

 1 from calculator.simple import SimpleCalculator
 2 
 3 my_calculator = SimpleCalculator()  
 4 my_calculator.run('2 * 2')
 5 print(my_calculator.lcd)

This code will print out the result of 2 * 2 to the Thonny Shell in the main UI. To understand what each part of the code is doing, check out the Take a Deeper Look section below.

Line 1: This code imports the library calculator inside of the package called simplecalculator. From this library, we import the class called SimpleCalculator from a file called simple.py. You can see the code here.
Lines 2: This is a blank line behind code blocks, which is generally a preferred style. Read more about Python Code Quality in this article.
Line 3: Here we create an instance of the class SimpleCalculator and assign it to a variable called my_calculator. This can be used to run different calculators. If you’re new to classes, you can learn more about object-oriented programming here.
Line 4: Here we have the calculator run the operation 2 * 2 by calling run() and passing in the expression as a string.
Line 5: Here we print the result of the calculation. You’ll notice in order to get the most recent calculation result, we must access the attribute called lcd.

Great! Now that you know exactly what your calculator code is doing, let’s move on to running this code!

Part 2: Save the File, View the Variables, and Run Your Code

Now it’s time to save and run your code. In this section, you’ll make use of two of the icons we reviewed earlier:

Save your new file as calculations.py.
Open the Variables window and make note of the two variables listed. You should see SimpleCalculator and my_calculator. This section also gives you insight into the value that each variable is pointing to.
Run your code! You should see 4.0 in the output:

Great job! Next you’ll explore how Thonny’s debugger can help you to better understand this code.

Other Great Beginner Features

As you get more comfortable with Thonny, the features in this section will come in quite handy.

Debugging

Using your calculations.py script, you’re going to use the debugger to investigate what is happening. Update your code in calculations.py to the following:

from calculator.simple import SimpleCalculator

def create_add_string(x, y):
    '''Returns a string containing an addition expression.'''
    return 'x + y'

my_calculator = SimpleCalculator()  
my_calculator.run(create_add_string(2, 2))
print(my_calculator.lcd)

Hit the save icon to save this version.

You’ll notice the code has a new function called create_add_string(). If you’re unfamiliar with Python functions, learn more in this awesome Real Python course!

As you inspect the function, you may notice why this script will not work as expected. If not, that’s okay! Thonny is going to help you see exactly what is going on, and squash that bug! Go ahead and run your program and see what happens. The Shell output should be the following:

>>> %Run calculations.py
0

Oh no! Now you can see there is a bug in your program. The answer should be 4! Next, you’ll use Thonny’s debugger to find the bug.

Let’s Try It!

Now that we have a bug in our program, this is a great chance to use Thonny’s debugging features:

Click the bug icon at the top of the window. This enters debugger mode.
You should see the import statements highlighted. Click the small step arrow icon, the yellow arrow in the middle. Keep pressing this to see how the debugger works. You should notice that it highlights each step that Python takes to evaluate your program. Once it hits create_add_string(), you should see a new window pop up.
Examine the pop up window carefully. You should see that it shows the values for x and y. Keep pressing the small step icon until you see the value that Python will return to your program. It will be enclosed in a light-blue box: Oh no! There’s the bug! It looks like Python will return a string containing the letters x and y (meaning 'x + y' and not a string containing the values of those variables, like '2 + 2', which is what the calculator is expecting.) Each time you see a light-blue box, you can think of this as Python replacing subexpressions with their values, step by step. The pop up window can be thought of as a piece of scratch paper that Python uses to figure out those values. Continue to step through the program to see how this bug results in a calculation of 0.
The bug here has to do with string formatting. If you are unfamiliar with string formatting, check out this article on Python String Formatting Best Practices. Inside create_add_string(), the f-string formatting method should be used. Update this function to the following:
```
def create_add_string(x, y):
    '''Returns a string containing an addition expression.'''
    return f'{x} + {y}'
```
Run your program again. You should see the following output:
```
>>> %Run calculations.py
4.0
```

Success! You have just demonstrated how the step-by-step debugger can help you find a problem in your code! Next you’ll learn about some other fun Thonny features.

Variable Scope Highlighting

Thonny offers variable highlighting to remind you that the same name doesn’t always mean the same variable. In order for this feature to work, on the menu bar, go to Thonny > Preferences and ensure that Highlight matching names is checked.

Notice in the code snippet below, that create_add_string() now has a new variable called my_calculator, though this is not the same as the my_calculator on lines 10 and 11. You should be able to tell because Thonny highlights the variables that reference the same thing. This my_calculator inside the function only exists within the scope of that function, which is why it is not highlighted when the cursor is on the other my_calculator variable on line 10:

This feature can really help you avoid typos and understand the scope of your variables.

Code Completion

Thonny also offers code completion for APIs. Notice in the snapshot below how pressing the Tab key shows the methods available from the random library:

This can be very useful when you’re working with libraries and don’t want to look at the documentation to find a method or attribute name.

Working on a Pre-Existing Project

Now that you’ve learned the basic features of Thonny, let’s explore how you can use it to work on a pre-existing project.

Find a File on Your Computer

Opening a file on your computer is as easy as going to the menu bar, selecting File > Open, and using your browser to navigate to the file. You can also use the open folder icon at the top of the screen to do this as well.

If you have a requirements.txt file and pip locally installed, you can pip install these from the Thonny system Shell. If you don’t have pip installed, remember you can use the Package Manager to install it:

$ pip install -r requirements.txt

Work on a Project From Github

Now that you are a Thonny expert, you can use it to work on the exercises from Real Python Course 1: Introduction to Python:

Navigate to the Real Python GitHub repo called book1-exercises.
Click the green button labeled Clone or download and select Download Zip.
Click the opening folder icon to navigate and find the downloaded files. You should find a folder called book1-exercises1.
Open one of the files and start working!

This is useful because there are tons of cool projects available on GitHub!

Conclusion

Awesome job getting through this tutorial on Thonny!

You can now start using Thonny to write, debug, and run Python code! If you like Thonny, you might also like some of the other IDEs we’ve listed in Python IDEs and Code Editors (Guide).

Thonny is actively maintained, and new features are being added all the time. There are several awesome new features that are currently in beta that can be found on the Thonny Blog. Thonny’s main development takes place at the Institute of Computer Science of the University of Tartu, Estonia, as well as by contributors around the world.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Brian Peterson

2018-12-10T14:00:00+00:00

To date, I’ve interviewed people you’ve likely heard of before from the Python community. But this column isn’t just about interviewing the rock stars and core devs. It’s also a means to shine light on the huge contributions to the community that can often go unthanked and overlooked. As such, I present to you Brian Peterson.

Brian is a project manager by day, and by night he’s one of the moderators of the Pythonista Café, a peer-to-peer learning community for Pythonistas. In our interview, we talk about how Python helps him in his role as a project manager, and how moderating a large forum for Python enthusiasts has impacted his coding chops. Let’s dig in!

Ricky: Welcome to Real Python! Let’s get started with the question I ask everyone. How did you get into programming, and when did you start using Python?

Brian: Well… thinking back, my first real exposure to programming was testing near object detection system prototypes using an old HP-87. The program was used to move and change target and rotating antenna positions along test tracks inside an automotive laboratory anechoic chamber, all while collecting and processing data from a spectrum analyzer. Seeing the code translate into motion did it for me—something real, and tangible. I was hooked on using programming as an R&D tool to make things come alive.

Over the years, I started spending more time with Linux automation, control systems, data collection and analysis, which naturally led to spending more time writing C code, and then it was like, “Hey, Python is already on the system. Why not take it for a spin?”

I think it was Python 2.3 at that time. I liked Python because it felt natural, and I didn’t have to throw away all my C knowledge. It had some functional programming sprinkled in, and an impressive set of engineering and scientific libraries. To top it off, I could use the same language seamlessly across scripting, interactive data analysis, and writing code. And Python syntax, well, it can be so darned readable that personally, I think it’s hard not to like it. I was hooked.

Ricky: Working in project management and using Python might not seem a natural fit for some people. How do you use Python to help you in the project management office, and what tools and libraries are the most helpful for you on a day-to-day basis?

Brian: Yeah, at first glance Python may not seem like a natural fit. This is in large part due to the way project management is commonly taught—like cooking from a recipe book rather than learning and understanding the culinary arts.

And cookie-cutter project management tools are like the microwave ovens. Yeah, sure, use ‘em for the prepacked, boxed, canned stuff, but for every meal? Seriously? Python also starts making sense when you’re considering different flavors of project management:

Traditional project managers use template-driven approaches.
Agile practitioners use story-driven approaches.
Adaptive practitioners blend existing, new, emerging approaches and creativity to develop the best fit for each unique project or group of projects.

So while off-the-shelf project management tools and services are an excellent fit for traditional projects, and a good fit for some Agile projects, adaptive project management calls for a different breed. In other words, rather than force all projects to fit a tool, why not use Python to construct a toolbox filled with best-of-breed tools to solve different types of problems for different kinds of projects?

It’s also worth considering that modern day Project Management Offices (PMO) don’t just manage projects: there’s ideation, portfolio management, road mapping, strategic planning, KPI monitoring, communications, governance, dashboards, stakeholder portals, working closely with DevOps, business analysts, architects, developers, UX designers, CX folks, subject matter experts, other technologists, and the list goes on and on.

Now imagine a vendor who knows nothing about your particular projects or PMO services with a big box of Legos gluing pieces into place and selling it to you. Yeah, it’s probably a nice contraption, or it wouldn’t be on the market for long.

But compare it to owning your own big box of Legos (Python and goodies), where you can add Legos to pre-built contraptions, use Legos to bridge multiple contraptions together, use them with existing Legos already lying around the organization, or build your own and mix and match any or all of the above.

And to answer the second part of the question, I currently use Python and RESTful APIs to interact with a variety of flexible SaaS tools, for example, ProductPlan, Smartsheet (with the dashboard extension), Airtable, and a commercial ticketing system, and enterprise visualization software. Python makes it easy to wire these tools together in creative ways and fill gaps in functionality to solve many different kinds of problems.

Ricky: How have those tools changed the way you work? And how do you see them changing going forward? Will Python continue to play a role in the future of project management offices?

Brian: I think the most significant change is a sense of freedom, the ability to creatively solve problems, focus on what’s essential, and leverage modern solutions rather than being chained to tools, falling into tool ruts, or suffering from tool rot. As an added bonus, Python makes project management fun and refreshing.

I’m not sure that Python will become mainstream in project management, but I do think it can give project managers who learn it a unique advantage in the automation of tedious tasks, solving complex problems, adapting to the evolution of organizations, and providing valuable outside-the-box PMO services.

Ricky: Do you have any Python or tech related side projects you’re working on right now?

Brian: Yes, Road Maps to the Future is a project in partnership with a network of volunteers and in cooperation with agencies to help pilot it. With over 80,000 local governments in the United States alone, I believe there needs to be a better way to map out where we are, where we are going in the future, and the best ways to get there.

In Python, we have an amazing assortment of tools for doing fantastic things with data, and we’re getting better at collecting discrete data.

But there are opportunities in the exploration and non-linear collection of ideas, the identification of unseen problems, gaps in services, areas for improvement in the overall customer experience, the development of creative and viable alternative solutions to these problems, and then translating them into successful projects with demonstrable benefits.

The road maps to the future are not just compilations of random ideas with new ways to collect information, but cohesive collections of core elements, processes, and partnerships necessary to propel the ideas into action through the organization of projects.

To make it sustainable, it has to be discoverable, simple, quick and fun, so a lot of extra focus is being put into the UX side of the project. The current design uses Python, Vue.js, Azure Cosmos DB, Service Bus and Cognitive Services. It’s still in early development. Project information, details, and code will be released into the public domain next year.

Ricky: One of the reasons I was keen to interview you is because of your contribution to the Python community. You are one of two moderators in the Pythonista Café, a peer-to-peer learning community for Python enthusiasts. How has your journey been from being a member to a moderator, and how has it influenced your Python chops?

Brian: It’s been great! Dan Bader has done such a fantastic job, along with you, Jennifer (co-moderator), the Real Python team, and especially all the amazing contributors from a wide range of backgrounds, geographical locations, and levels of experience who have made the Café into the lively forum it is today.

It really is truly community-driven from the ground up. It has a fun, productive, open-source vibe without fear of public shaming. It’s okay to make mistakes and learn. You can post stuff without having to worry about how the whole world will interpret it or misinterpret it.

I initially joined the Café for several reasons. It’s like a non-cliquey 24/7/365 Python meetup that you can drop in and out of at any time. As forum members have mentioned, it has a much better signal-to-noise ratio than the public forums, not to mention freedom from ads and social media algorithms that manipulate behaviors.

It’s very welcoming and ultra-Python friendly. The passion for programming is seriously contagious, which leads to your question on how it has influenced my Python chops. The answer is that it has made Python even more fun, and I’m now even more committed and focused to continuous learning in Python and its ecosystem. It has also increased my excitement about the possibilities of what can be done with Python in the future.

Ricky: Now for my last question. What other hobbies and interests do you have, aside from Python? Any you’d like to share and/or plug?

Brian: Outside of work and Python, I like to mix it up a lot. I currently work in the Panhandle and spend my off-time on the First Coast, so lately one of my favorite things to do is wandering around the south kayaking the many rivers and swamps; under certain conditions, it’s absolutely surreal. You can read and watch all you want about swamps, but there’s no substitute for the real thing.

Each year I also pick out one or two new subjects to research and study for fun. I am an amateur radio operator, I love music, and I’m an avid reader of books, particularly history, survival stories, and science fiction.

I’m also blessed to have an amazing wife and son. Outside of my projects, my wife keeps things lively with home projects, going to festivals and other events, helping animal rescues, along with exploring new areas and hiking with the dogs. Boredom is an alien concept—it definitely keeps life fun.

Thank you, Brian, for joining me this week. If you’re looking for a friendly Python place to hang your metaphorical hat, you can find the Pythonista Café here. Just be sure to say hi to Brian when you get there.

Do you know an unsung hero in the Python community? If they’d like me to interview them in the future, I can be reached in the comments below, or you can send me a message on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Sending Emails With Python

2018-12-05T14:00:00+00:00

You probably found this tutorial because you want to send emails using Python. Perhaps you want to receive email reminders from your code, send a confirmation email to users when they create an account, or send emails to members of your organization to remind them to pay their dues. Sending emails manually is a time-consuming and error-prone task, but it’s easy to automate with Python.

In this tutorial you’ll learn how to:

Set up a secure connection using SMTP_SSL() and .starttls()
Use Python’s built-in smtplib library to send basic emails
Send emails with HTML content and attachments using the email package
Send multiple personalized emails using a CSV file with contact data
Use the Yagmail package to send email through your Gmail account using only a few lines of code

You’ll find a few transactional email services at the end of this tutorial, which will come in useful when you want to send a large number of emails.

Getting Started

Python comes with the built-in smtplib module for sending emails using the Simple Mail Transfer Protocol (SMTP). smtplib uses the RFC 821 protocol for SMTP. The examples in this tutorial will use the Gmail SMTP server to send emails, but the same principles apply to other email services. Although the majority of email providers use the same connection ports as the ones in this tutorial, you can run a quick Google search to confirm yours.

To get started with this tutorial, set up a Gmail account for development, or set up an SMTP debugging server that discards emails you send and prints them to the command prompt instead. Both options are laid out for you below. A local SMTP debugging server can be useful for fixing any issues with email functionality and ensuring your email functions are bug-free before sending out any emails.

Option 1: Setting up a Gmail Account for Development

If you decide to use a Gmail account to send your emails, I highly recommend setting up a throwaway account for the development of your code. This is because you’ll have to adjust your Gmail account’s security settings to allow access from your Python code, and because there’s a chance you might accidentally expose your login details. Also, I found that the inbox of my testing account rapidly filled up with test emails, which is reason enough to set up a new Gmail account for development.

A nice feature of Gmail is that you can use the + sign to add any modifiers to your email address, right before the @ sign. For example, mail sent to my+person1@gmail.com and my+person2@gmail.com will both arrive at my@gmail.com. When testing email functionality, you can use this to emulate multiple addresses that all point to the same inbox.

To set up a Gmail address for testing your code, do the following:

Create a new Google account.
Turn Allow less secure apps to ON. Be aware that this makes it easier for others to gain access to your account.

If you don’t want to lower the security settings of your Gmail account, check out Google’s documentation on how to gain access credentials for your Python script, using the OAuth2 authorization framework.

Option 2: Setting up a Local SMTP Server

You can test email functionality by running a local SMTP debugging server, using the smptd module that comes pre-installed with Python. Rather than sending emails to the specified address, it discards them and prints their content to the console. Running a local debugging server means it’s not necessary to deal with encryption of messages or use credentials to log in to an email server.

You can start a local SMTP debugging server by typing the following in Command Prompt:

$ python -m smtpd -c DebuggingServer -n localhost:1025

On Linux, use the same command preceded by sudo.

Any emails sent through this server will be discarded and shown in the terminal window as a bytes object for each line:

---------- MESSAGE FOLLOWS ----------
b'X-Peer: ::1'
b''
b'From: my@address.com'
b'To: your@address.com'
b'Subject: a local test mail'
b''
b'Hello there, here is a test email'
------------ END MESSAGE ------------

For the rest of the tutorial, I’ll assume you’re using a Gmail account, but if you’re using a local debugging server, just make sure to use localhost as your SMTP server and use port 1025 rather than port 465 or 587. Besides this, you won’t need to use login() or encrypt the communication using SSL/TLS.

Sending a Plain-Text Email

Before we dive into sending emails with HTML content and attachments, you’ll learn to send plain-text emails using Python. These are emails that you could write up in a simple text editor. There’s no fancy stuff like text formatting or hyperlinks. You’ll learn that a bit later.

Starting a Secure SMTP Connection

When you send emails through Python, you should make sure that your SMTP connection is encrypted, so that your message and login credentials are not easily accessed by others. SSL (Secure Sockets Layer) and TLS (Transport Layer Security) are two protocols that can be used to encrypt an SMTP connection. It’s not necessary to use either of these when using a local debugging server.

There are two ways to start a secure connection with your email server:

Start an SMTP connection that is secured from the beginning using SMTP_SSL().
Start an unsecured SMTP connection that can then be encrypted using .starttls().

In both instances, Gmail will encrypt emails using TLS, as this is the more secure successor of SSL. As per Python’s Security considerations, it is highly recommended that you use create_default_context() from the ssl module. This will load the system’s trusted CA certificates, enable host name checking and certificate validation, and try to choose reasonably secure protocol and cipher settings.

If you want to check the encryption for an email in your Gmail inbox, go to More → Show original to see the encryption type listed under the Received header.

smtplib is Python’s built-in module for sending emails to any Internet machine with an SMTP or ESMTP listener daemon.

I’ll show you how to use SMTP_SSL() first, as it instantiates a connection that is secure from the outset and is slightly more concise than the .starttls() alternative. Keep in mind that Gmail requires that you connect to port 465 if using SMTP_SSL(), and to port 587 when using .starttls().

Option 1: Using `SMTP_SSL()`

The code example below creates a secure connection with Gmail’s SMTP server, using the SMTP_SSL() of smtplib to initiate a TLS-encrypted connection. The default context of ssl validates the host name and its certificates and optimizes the security of the connection. Make sure to fill in your own email address instead of my@gmail.com:

import smtplib, ssl

port = 465  # For SSL
password = input("Type your password and press enter: ")

# Create a secure SSL context
context = ssl.create_default_context()

with smtplib.SMTP_SSL("smtp.gmail.com", port, context=context) as server:
    server.login("my@gmail.com", password)
    # TODO: Send email here

Using with smtplib.SMTP_SSL() as server: makes sure that the connection is automatically closed at the end of the indented code block. If port is zero, or not specified, .SMTP_SSL() will use the standard port for SMTP over SSL (port 465).

It’s not safe practice to store your email password in your code, especially if you intend to share it with others. Instead, use input() to let the user type in their password when running the script, as in the example above. If you don’t want your password to show on your screen when you type it, you can import the getpass module and use .getpass() instead for blind input of your password.

Option 2: Using `.starttls()`

Instead of using .SMTP_SSL() to create a connection that is secure from the outset, we can create an unsecured SMTP connection and encrypt it using .starttls().

To do this, create an instance of smtplib.SMTP, which encapsulates an SMTP connection and allows you access to its methods. I recommend defining your SMTP server and port at the beginning of your script to configure them easily.

The code snippet below uses the construction server = SMTP(), rather than the format with SMTP() as server: which we used in the previous example. To make sure that your code doesn’t crash when something goes wrong, put your main code in a try block, and let an except block print any error messages to stdout:

import smtplib, ssl

smtp_server = "smtp.gmail.com"
port = 587  # For starttls
sender_email = "my@gmail.com"
password = input("Type your password and press enter: ")

# Create a secure SSL context
context = ssl.create_default_context()

# Try to log in to server and send email
try:
    server = smtplib.SMTP(smtp_server,port)
    server.ehlo() # Can be omitted
    server.starttls(context=context) # Secure the connection
    server.ehlo() # Can be omitted
    server.login(sender_email, password)
    # TODO: Send email here
except Exception as e:
    # Print any error messages to stdout
    print(e)
finally:
    server.quit()

To identify yourself to the server, .helo() (SMTP) or .ehlo() (ESMTP) should be called after creating an .SMTP() object, and again after .starttls(). This function is implicitly called by .starttls() and .sendmail() if needed, so unless you want to check the SMTP service extensions of the server, it is not necessary to use .helo() or .ehlo() explicitly.

Sending Your Plain-text Email

After you initiated a secure SMTP connection using either of the above methods, you can send your email using .sendmail(), which pretty much does what it says on the tin:

server.sendmail(sender_email, receiver_email, message)

I recommend defining the email addresses and message content at the top of your script, after the imports, so you can change them easily:

sender_email = "my@gmail.com"
receiver_email = "your@gmail.com"
message = """\
Subject: Hi there

This message is sent from Python."""

# Send email here

The message string starts with "Subject: Hi there" followed by two newlines (\n). This ensures Hi there shows up as the subject of the email, and the text following the newlines will be treated as the message body.

The code example below sends a plain-text email using SMTP_SSL():

import smtplib, ssl

port = 465  # For SSL
smtp_server = "smtp.gmail.com"
sender_email = "my@gmail.com"  # Enter your address
receiver_email = "your@gmail.com"  # Enter receiver address
password = input("Type your password and press enter: ")
message = """\
Subject: Hi there

This message is sent from Python."""

context = ssl.create_default_context()
with smtplib.SMTP_SSL(smtp_server, port, context=context) as server:
    server.login(sender_email, password)
    server.sendmail(sender_email, receiver_email, message)

For comparison, here is a code example that sends a plain-text email over an SMTP connection secured with .starttls(). The server.ehlo() lines may be omitted, as they are called implicitly by .starttls() and .sendmail(), if required:

import smtplib, ssl

port = 587  # For starttls
smtp_server = "smtp.gmail.com"
sender_email = "my@gmail.com"
receiver_email = "your@gmail.com"
password = input("Type your password and press enter:")
message = """\
Subject: Hi there

This message is sent from Python."""

context = ssl.create_default_context()
with smtplib.SMTP(smtp_server, port) as server:
    server.ehlo()  # Can be omitted
    server.starttls(context=context)
    server.ehlo()  # Can be omitted
    server.login(sender_email, password)
    server.sendmail(sender_email, receiver_email, message)

Sending Fancy Emails

Python’s built-in email package allows you to structure more fancy emails, which can then be transferred with smptlib as you have done already. Below, you’ll learn how use the email package to send emails with HTML content and attachments.

Including HTML Content

If you want to format the text in your email (bold, italics, and so on), or if you want to add any images, hyperlinks, or responsive content, then HTML comes in very handy. Today’s most common type of email is the MIME (Multipurpose Internet Mail Extensions) Multipart email, combining HTML and plain-text. MIME messages are handled by Python’s email.mime module. For a detailed description, check the documentation.

As not all email clients display HTML content by default, and some people choose only to receive plain-text emails for security reasons, it is important to include a plain-text alternative for HTML messages. As the email client will render the last multipart attachment first, make sure to add the HTML message after the plain-text version.

In the example below, our MIMEText() objects will contain the HTML and plain-text versions of our message, and the MIMEMultipart("alternative") instance combines these into a single message with two alternative rendering options:

import smtplib, ssl
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

sender_email = "my@gmail.com"
receiver_email = "your@gmail.com"
password = input("Type your password and press enter:")

message = MIMEMultipart("alternative")
message["Subject"] = "multipart test"
message["From"] = sender_email
message["To"] = receiver_email

# Create the plain-text and HTML version of your message
text = """\
Hi,
How are you?
Real Python has many great tutorials:
www.realpython.com"""
html = """\
<html>
  <body>
    <p>Hi,<br>
       How are you?<br>
       <a href="http://www.realpython.com">Real Python</a> 
       has many great tutorials.
    </p>
  </body>
</html>
"""

# Turn these into plain/html MIMEText objects
part1 = MIMEText(text, "plain")
part2 = MIMEText(html, "html")

# Add HTML/plain-text parts to MIMEMultipart message
# The email client will try to render the last part first
message.attach(part1)
message.attach(part2)

# Create secure connection with server and send email
context = ssl.create_default_context()
with smtplib.SMTP_SSL("smtp.gmail.com", 465, context=context) as server:
    server.login(sender_email, password)
    server.sendmail(
        sender_email, receiver_email, message.as_string()
    )

In this example, you first define the plain-text and HTML message as string literals, and then store them as plain/html MIMEText objects. These can then be added in this order to the MIMEMultipart("alternative") message and sent through your secure connection with the email server. Remember to add the HTML message after the plain-text alternative, as email clients will try to render the last subpart first.

Adding Attachments Using the `email` Package

In order to send binary files to an email server that is designed to work with textual data, they need to be encoded before transport. This is most commonly done using base64, which encodes binary data into printable ASCII characters.

The code example below shows how to send an email with a PDF file as an attachment:

import email, smtplib, ssl

from email import encoders
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

subject = "An email with attachment from Python"
body = "This is an email with attachment sent from Python"
sender_email = "my@gmail.com"
receiver_email = "your@gmail.com"
password = input("Type your password and press enter:")

# Create a multipart message and set headers
message = MIMEMultipart()
message["From"] = sender_email
message["To"] = receiver_email
message["Subject"] = subject
message["Bcc"] = receiver_email  # Recommended for mass emails

# Add body to email
message.attach(MIMEText(body, "plain"))

filename = "document.pdf"  # In same directory as script

# Open PDF file in binary mode
with open(filename, "rb") as attachment:
    # Add file as application/octet-stream
    # Email client can usually download this automatically as attachment
    part = MIMEBase("application", "octet-stream")
    part.set_payload(attachment.read())

# Encode file in ASCII characters to send by email    
encoders.encode_base64(part)

# Add header as key/value pair to attachment part
part.add_header(
    "Content-Disposition",
    f"attachment; filename= {filename}",
)

# Add attachment to message and convert message to string
message.attach(part)
text = message.as_string()

# Log in to server using secure context and send email
context = ssl.create_default_context()
with smtplib.SMTP_SSL("smtp.gmail.com", 465, context=context) as server:
    server.login(sender_email, password)
    server.sendmail(sender_email, receiver_email, text)

The MIMEultipart() message accepts parameters in the form of RFC5233-style key/value pairs, which are stored in a dictionary and passed to the .add_header method of the Message base class.

Check out the documentation for Python’s email.mime module to learn more about using MIME classes.

Sending Multiple Personalized Emails

Imagine you want to send emails to members of your organization, to remind them to pay their contribution fees. Or maybe you want to send students in your class personalized emails with the grades for their recent assignment. These tasks are a breeze in Python.

Make a CSV File With Relevant Personal Info

An easy starting point for sending multiple personalized emails is to create a CSV (comma-separated values) file that contains all the required personal information. (Make sure not to share other people’s private information without their consent.) A CSV file can be thought of as a simple table, where the first line often contains the column headers.

Below are the contents of the file contacts_file.csv, which I saved in the same folder as my Python code. It contains the names, addresses, and grades for a set of fictional people. I used my+modifier@gmail.com constructions to make sure all emails end up in my own inbox, which in this example is my@gmail.com:

name,email,grade
Ron Obvious,my+ovious@gmail.com,B+
Killer Rabbit of Caerbannog,my+rabbit@gmail.com,A
Brian Cohen,my+brian@gmail.com,C

When creating a CSV file, make sure to separate your values by a comma, without any surrounding whitespaces.

Loop Over Rows to Send Multiple Emails

The code example below shows you how to open a CSV file and loop over its lines of content (skipping the header row). To make sure that the code works correctly before you send emails to all your contacts, I’ve printed Sending email to ... for each contact, which we can later replace with functionality that actually sends out emails:

import csv

with open("contacts_file.csv") as file:
    reader = csv.reader(file)
    next(reader)  # Skip header row
    for name, email, grade in reader:
        print(f"Sending email to {name}")
        # Send email here

In the example above, using with open(filename) as file:makes sure that your file closes at the end of the code block. csv.reader() makes it easy to read a CSV file line by line and extract its values. The next(reader) line skips the header row, so that the following line for name, email, grade in reader: splits subsequent rows at each comma, and stores the resulting values in the strings name, email and grade for the current contact.

If the values in your CSV file contain whitespaces on either or both sides, you can remove them using the .strip() method.

Personalized Content

You can put personalized content in a message by using str.format() to fill in curly-bracket placeholders. For example, "hi {name}, you {result} your assignment".format(name="John", result="passed") will give you "hi John, you passed your assignment".

As of Python 3.6, string formatting can be done more elegantly using f-strings, but these require the placeholders to be defined before the f-string itself. In order to define the email message at the beginning of the script, and fill in placeholders for each contact when looping over the CSV file, the older .format() method is used.

With this in mind, you can set up a general message body, with placeholders that can be tailored to individuals.

Code Example

The following code example lets you send personalized emails to multiple contacts. It loops over a CSV file with name,email,grade for each contact, as in the example above.

The general message is defined in the beginning of the script, and for each contact in the CSV file its {name} and {grade} placeholders are filled in, and a personalized email is sent out through a secure connection with the Gmail server, as you saw before:

import csv, smtplib, ssl

message = """Subject: Your grade

Hi {name}, your grade is {grade}"""
from_address = "my@gmail.com"
password = input("Type your password and press enter: ")

context = ssl.create_default_context()
with smtplib.SMTP_SSL("smtp.gmail.com", 465, context=context) as server:
    server.login(from_address, password)
    with open("contacts_file.csv") as file:
        reader = csv.reader(file)
        next(reader)  # Skip header row
        for name, email, grade in reader:
            server.sendmail(
                from_address,
                email,
                message.format(name=name,grade=grade),
            )

Yagmail

There are multiple libraries designed to make sending emails easier, such as Envelopes, Flanker and Yagmail. Yagmail is designed to work specifically with Gmail, and it greatly simplifies the process of sending emails through a friendly API, as you can see in the code example below:

import yagmail

receiver = "your@gmail.com"
body = "Hello there from Yagmail"
filename = "document.pdf"

yag = yagmail.SMTP("my@gmail.com")
yag.send(
    to=receiver,
    subject="Yagmail test with attachment",
    contents=body, 
    attachments=filename,
)

This code example sends an email with a PDF attachment in a fraction of the lines needed for our example using email and smtplib.

When setting up Yagmail, you can add your Gmail validations to the keyring of your OS, as described in the documentation. If you don’t do this, Yagmail will prompt you to enter your password when required and store it in the keyring automatically.

Transactional Email Services

If you plan to send a large volume of emails, want to see email statistics, and want to ensure reliable delivery, it may be worth looking into transactional email services. Although all of the following services have paid plans for sending large volumes of emails, they also come with a free plan so you can try them out. Some of these free plans are valid indefinitely and may be sufficient for your email needs.

Below is an overview of the free plans for some of the major transactional email services. Clicking on the provider name will take you to the pricing section of their website.

Provider	Free plan
Sendgrid	40,000 emails for your first 30 days, then 100/day
Sendinblue	300 emails/day
Mailgun	First 10,000 emails free
Mailjet	200 emails/day
Amazon SES	62,000 emails/month

You can run a Google search to see which provider best fits your needs, or just try out a few of the free plans to see which API you like working with most.

Sendgrid Code Example

Here’s a code example for sending emails with Sendgrid to give you a flavor of how to use a transactional email service with Python:

import os
import sendgrid
from sendgrid.helpers.mail import Content, Email, Mail

sg = sendgrid.SendGridAPIClient(
    apikey=os.environ.get("SENDGRID_API_KEY")
)
from_email = Email("my@gmail.com")
to_email = Email("your@gmail.com")
subject = "A test email from Sendgrid"
content = Content(
    "text/plain", "Here's a test email sent through Python"
)
mail = Mail(from_email, subject, to_email, content)
response = sg.client.mail.send.post(request_body=mail.get())

# The statements below can be included for debugging purposes
print(response.status_code)
print(response.body)
print(response.headers)

To run this code, you must first:

Sign up for a (free) Sendgrid account
Request an API key for user validation
Add your API key by typing setx SENDGRID_API_KEY "YOUR_API_KEY" in Command Prompt (to store this API key permanently) or set SENDGRID_API_KEY YOUR_API_KEY to store it only for the current client session

More information on how to set up Sendgrid for Mac and Windows can be found in the repository’s README on Github.

Conclusion

You can now start a secure SMTP connection and send multiple personalized emails to the people in your contacts list!

You’ve learned how to send an HTML email with a plain-text alternative and attach files to your emails. The Yagmail package simplifies all these tasks when you’re using a Gmail account. If you plan to send large volumes of email, it is worth looking into transactional email services.

Enjoy sending emails with Python, and remember: no spam please!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Building Serverless Python Apps Using AWS Chalice

2018-12-03T14:00:00+00:00

Shipping a web application usually involves having your code up and running on single or multiple servers. In this model, you end up setting up processes for monitoring, provisioning, and scaling your servers up or down. Although this seems to work well, having all the logistics around a web application handled in an automated manner reduces a lot of manual overhead. Enter Serverless.

With Serverless Architecture, you don’t manage servers. Instead, you only need to ship the code or the executable package to the platform that executes it. It’s not really serverless. The servers do exist, but the developer doesn’t need to worry about them.

AWS introduced Lambda Services, a platform that enables developers to simply have their code executed in a particular runtime environment. To make the platform easy to use, many communities have come up with some really good frameworks around it in order to make the serverless apps a working solution.

By the end of this tutorial, you’ll be able to:

Discuss the benefits of a serverless architecture
Explore Chalice, a Python serverless framework
Build a full blown serverless app for a real world use case
Deploy to Amazon Web Services (AWS) Lambda
Compare Pure and Lambda functions

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Getting Started With AWS Chalice

Chalice, a Python Serverless Microframework developed by AWS, enables you to quickly spin up and deploy a working serverless app that scales up and down on its own as required using AWS Lambda.

Why Chalice?

For Python developers accustomed to the Flask web framework, Chalice should be a breeze in terms of building and shipping your first app. Highly inspired by Flask, Chalice keeps it pretty minimalist in terms of defining what the service should be like and finally making an executable package of the same.

Enough theory! Let’s start with a basic hello-world app and kick-start our serverless journey.

Project Setup

Before diving into Chalice, you’ll set up a working environment on your local machine, which will set you up for the rest of the tutorial.

First, create and activate a virtual environment and install Chalice:

$ python3.6 -m venv env
$ source env/bin/activate
(env)$ pip install chalice

Follow our comprehensive guide on the Pipenv packaging tool.

Note: Chalice comes with a user-friendly CLI that makes it easy to play around with your serverless app.

Now that you have Chalice installed on your virtual environment, let’s use the Chalice CLI to generate some boilerplate code:

(env)$ chalice new-project

Enter the name of the project when prompted and hit return. A new directory is created with that name:

<project-name>/
|
├── .chalice/
│   └── config.json
|
├── .gitignore
├── app.py
└── requirements.txt

See how minimalist the Chalice codebase is. A .chalice directory, app.py, and requirements.txt is all that it requires to have a serverless app up and running. Let’s quickly run the app on our local machine.

Chalice CLI consists of really great utility functions allowing you to perform a number of operations from running locally to deploying in a Lambda environment.

Build and Run Locally

You can simulate the app by running it locally using the local utility of Chalice:

(env)$ chalice local
Serving on 127.0.0.1:8000

By default, Chalice runs on port 8000. We can now check the index route by making a curl request to http://localhost:8000/:

$ curl -X GET http://localhost:8000/
{"hello": "world"}

Now if we look at app.py, we can appreciate the simplicity with which Chalice allows you to build a serverless service. All the complex stuff is handled by the decorators:

from chalice import Chalice
app = Chalice(app_name='serverless-sms-service')

@app.route('/')
def index():
    return {'hello': 'world'}

Note: We haven’t named our app hello-world, as we will build our SMS service on the same app.

Now, let’s move on to deploying our app on the AWS Lambda.

Deploy on AWS Lambda

Chalice makes deploying your serverless app completely effortless. Using the deploy utility, you can simply instruct Chalice to deploy and create a Lambda function that can be accessible via a REST API.

Before we begin deployment, we need to make sure we have our AWS credentials in place, usually located at ~/.aws/config. The contents of the file look as follows:

[default]
aws_access_key_id=<your-access-key-id>
aws_secret_access_key=<your-secret-access-key>
region=<your-region>

With AWS credentials in place, let’s begin our deployment process with just a single command:

(env)$ chalice deploy
Creating deployment package.
Updating policy for IAM role: hello-world-dev
Creating lambda function: hello-world-dev
Creating Rest API
Resources deployed:
  - Lambda ARN: arn:aws:lambda:ap-south-1:679337104153:function:hello-world-dev
  - Rest API URL: https://fqcdyzvytc.execute-api.ap-south-1.amazonaws.com/api/

Note: The generated ARN and API URL in the above snippet will vary from user to user.

Wow! Yes, it is really this easy to get your serverless app up and running. To verify simply make a curl request on the generated Rest API URL:

$ curl -X GET https://fqcdyzvytc.execute-api.ap-south-1.amazonaws.com/api/
{"hello": "world"}

Typically, this is all that you need to get your serverless app up and running. You can also go to your AWS console and see the Lambda function created under the Lambda service section. Each Lambda service has a unique REST API endpoint that can be consumed in any web application.

Next, you will begin building your Serverless SMS Sender service using Twilio as an SMS service provider.

Building a Serverless SMS Sender Service

With a basic hello-world app deployed, let’s move on to building a more real-world application that can be used along with everyday web apps. In this section, you’ll build a completely serverless SMS-sending app that can be plugged into any system and work as expected as long as the input parameters are correct.

In order to send SMS, we will be using Twilio, a developer-friendly SMS service. Before we begin using Twilio, we need to take care of a few prerequisites:

Create an account and acquire ACCOUNT_SID and AUTH_TOKEN.
Get a mobile phone number, which is available for free at Twilio for minor testing stuff.
Install the twilio package in our virtual environment using pip install twilio.

With all the above prerequisites checked, you can start building your SMS service client using Twilio’s Python library. Let’s begin by cloning the repository and creating a new feature branch:

$ git clone <project-url>
$ cd <project-dir>
$ git checkout tags/1.0 -b twilio-support

Now make the following changes to app.py to evolve it from a simple hello-world app to enable support for Twilio service too.

First, let’s include all the import statements:

from os import environ as env

# 3rd party imports
from chalice import Chalice, Response
from twilio.rest import Client
from twilio.base.exceptions import TwilioRestException

# Twilio Config
ACCOUNT_SID = env.get('ACCOUNT_SID')
AUTH_TOKEN = env.get('AUTH_TOKEN')
FROM_NUMBER = env.get('FROM_NUMBER')
TO_NUMBER = env.get('TO_NUMBER')

Next, you’ll encapsulate the Twilio API and use it to send SMS:

app = Chalice(app_name='sms-shooter')

# Create a Twilio client using account_sid and auth token
tw_client = Client(ACCOUNT_SID, AUTH_TOKEN)

@app.route('/service/sms/send', methods=['POST'])
def send_sms():
    request_body = app.current_request.json_body
    if request_body:
        try:
            msg = tw_client.messages.create(
                from_=FROM_NUMBER,
                body=request_body['msg'],
                to=TO_NUMBER)

            if msg.sid:
                return Response(status_code=201,
                                headers={'Content-Type': 'application/json'},
                                body={'status': 'success',
                                      'data': msg.sid,
                                      'message': 'SMS successfully sent'})
            else:
                return Response(status_code=200,
                                headers={'Content-Type': 'application/json'},
                                body={'status': 'failure',
                                      'message': 'Please try again!!!'})
        except TwilioRestException as exc:
            return Response(status_code=400,
                            headers={'Content-Type': 'application/json'},
                            body={'status': 'failure',
                                  'message': exc.msg})

In the above snippet, you simply create a Twilio client object using ACCOUNT_SID and AUTH_TOKEN and use it to send messages under the send_sms view. send_sms is a bare bones function that uses the Twilio client’s API to send the SMS to the specified destination. Before proceeding further, let’s give it a try and run it on our local machine.

Build and Run Locally

Now you can run your app on your machine using the local utility and verify that everything is working fine:

(env)$ chalice local

Now make a curl POST request to http://localhost:8000/service/sms/send with a specific payload and test the app locally:

$ curl -H "Content-Type: application/json" -X POST -d '{"msg": "hey mate!!!"}' http://localhost:8000/service/sms/send

The above request responds as follows:

{
  "status": "success",
  "data": "SM60f11033de4f4e39b1c193025bcd5cd8",
  "message": "SMS successfully sent"
}

The response indicates that the message was successfully sent. Now, let’s move on to deploying the app on AWS Lambda.

Deploy on AWS Lambda

As suggested in the previous deployment section, you just need to issue the following command:

(env)$ chalice deploy
Creating deployment package.
Updating policy for IAM role: sms-shooter-dev
Creating lambda function: sms-shooter-dev
Creating Rest API
Resources deployed:
  - Lambda ARN: arn:aws:lambda:ap-south-1:679337104153:function:sms-shooter-dev
  - Rest API URL: https://qtvndnjdyc.execute-api.ap-south-1.amazonaws.com/api/

Note: The above command succeeds, and you have your API URL in the output as expected. Now on testing the URL, the API throws an error message. What went wrong?

As per AWS Lambda logs, twilio package is not found or installed, so you need to tell the Lambda service to install the dependencies. To do so, you need to add twilio as a dependency to requirements.txt:

twilio==6.18.1

Other packages such as Chalice and its dependencies should not be included in requirements.txt, as they are not a part of Python’s WSGI runtime. Instead, we should maintain a requirements-dev.txt, which is applicable to only the development environment and contains all Chalice-related dependencies. To learn more, check out this GitHub issue.

Once all the package dependencies are sorted, you need to make sure all the environment variables are also shipped along and set correctly during the Lambda runtime. To do so, you have to add all the environment variables in .chalice/config.json in the following manner:

{
  "version": "2.0",
  "app_name": "sms-shooter",
  "stages": {
    "dev": {
      "api_gateway_stage": "api",
      "environment_variables": {
          "ACCOUNT_SID": "<your-account-sid>",
          "AUTH_TOKEN": "<your-auth-token>",
          "FROM_NUMBER": "<source-number>",
          "TO_NUMBER": "<destination-number>"
      }
    }
  }
}

Now we’re good to deploy:

Creating deployment package.
Updating policy for IAM role: sms-shooter-dev
Updating lambda function: sms-shooter-dev
Updating rest API
Resources deployed:
  - Lambda ARN: arn:aws:lambda:ap-south-1:679337104153:function:sms-shooter-dev
  - Rest API URL: https://fqcdyzvytc.execute-api.ap-south-1.amazonaws.com/api/

Do a sanity check by making a curl request to the generated API endpoint:

$ curl -H "Content-Type: application/json" -X POST -d '{"msg": "hey mate!!!"}' https://fqcdyzvytc.execute-api.ap-south-1.amazonaws.com/api/service/sms/send

The above request responds as expected:

{
  "status": "success",
  "data": "SM60f11033de4f4e39b1c193025bcd5cd8",
  "message": "SMS successfully sent"
}

Now, you have a completely serverless SMS sending service up and running. With the front end of this service being a REST API, it can be used in other applications as a plug-and-play feature that is scalable, secure, and reliable.

Refactoring

Finally, we will refactor our SMS app to not contain all the business logic in app.py completely. Instead, we will follow the Chalice prescribed best practices and abstract the business logic under the chalicelib/ directory.

Let’s begin by creating a new branch:

$ git checkout tags/2.0 -b sms-app-refactor

First, create a new directory in the root directory of the project named chalicelib/ and create a new file named sms.py:

(env)$ mkdir chalicelib
(env)$ touch chalicelib/sms.py

Update the above created chalicelib/sms.py with the SMS sending logic by abstracting things from app.py:

from os import environ as env
from twilio.rest import Client

# Twilio Config
ACCOUNT_SID = env.get('ACCOUNT_SID')
AUTH_TOKEN = env.get('AUTH_TOKEN')
FROM_NUMBER = env.get('FROM_NUMBER')
TO_NUMBER = env.get('TO_NUMBER')

# Create a twilio client using account_sid and auth token
tw_client = Client(ACCOUNT_SID, AUTH_TOKEN)


def send(payload_params=None):
    """ send sms to the specified number """
    msg = tw_client.messages.create(
        from_=FROM_NUMBER,
        body=payload_params['msg'],
        to=TO_NUMBER)

    if msg.sid:
        return msg

The above snippet only accepts the input params and responds as required. Now to make this work, we need to make changes to app.py as well:

# Core imports
from chalice import Chalice, Response
from twilio.base.exceptions import TwilioRestException

# App level imports
from chalicelib import sms

app = Chalice(app_name='sms-shooter')


@app.route('/')
def index():
    return {'hello': 'world'}


@app.route('/service/sms/send', methods=['POST'])
def send_sms():
    request_body = app.current_request.json_body
    if request_body:
        try:
            resp = sms.send(request_body)
            if resp:
                return Response(status_code=201,
                                headers={'Content-Type': 'application/json'},
                                body={'status': 'success',
                                      'data': resp.sid,
                                      'message': 'SMS successfully sent'})
            else:
                return Response(status_code=200,
                                headers={'Content-Type': 'application/json'},
                                body={'status': 'failure',
                                      'message': 'Please try again!!!'})
        except TwilioRestException as exc:
            return Response(status_code=400,
                            headers={'Content-Type': 'application/json'},
                            body={'status': 'failure',
                                  'message': exc.msg})

In the above snippet, all the SMS sending logic is invoked from the chalicelib.sms module, making the view layer a lot cleaner in terms of readability. This abstraction lets you add much more complex business logic and customize the functionality as required.

Sanity Check

After refactoring our code, let’s ensure it is running as expected.

Build and Run Locally

Run the app once again using the local utility:

(env)$ chalice local

Make a curl request and verify. Once that’s done, move on to deployment.

Deploy on AWS Lambda

Once you are sure everything is working as expected, you can now finally deploy your app:

(env)$ chalice deploy

As usual, the command executes successfully and you can verify the endpoint.

Conclusion

You now know how to do the following:

Build a serverless application using AWS Chalice in accordance with best practices
Deploy your working app on the Lambda runtime environment

Lambda services under the hood are analogous to pure functions, which have a certain behavior on a set of input/output. Developing precise Lambda services allows for better testing, readability, and atomicity. Since Chalice is a minimalist framework, you can just focus on the business logic, and the rest is taken care of, from deployment to IAM policy generation. This is all with just a single command deployment!

Moreover, Lambda services are mostly focused on heavy CPU bound processing and scale in a self-governed manner, as per the number of requests in a unit of time. Using serverless architecture allows your codebase to be more like SOA (Service Oriented Architecture). Using AWS’s other products in their ecosystem that plug in well with Lambda functions is even more powerful.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Emily Morehouse

2018-11-28T14:00:00+00:00

I’m very pleased to be joined this week by Emily Morehouse.

Emily is one of the newest additions to the CPython core developer team, and the founder and director of engineering of Cuttlesoft. Emily and I talk about the recent CPython core developer sprint and the fact that she completed three majors in college at the same time! We’ll also get into her passion for compilers and abstract syntax trees.

Ricky: Let’s start with the obvious: how did you get into programming, and when did you start using Python?

Emily: My path to programming started with falling in love with Enigma machines. Really, I stumbled into programming during college. I had recently switched one of my majors from Biochemistry to Criminology, and the department had just launched a Computer Criminology program.

I was encouraged to try out the Intro To Programming course to see how I liked it. One of our final projects was building an Enigma machine simulator (in C++, mind you), and I was hooked. I decided to add a third major to take on a full Computer Science degree (more on that later!).

Since the CS program was highly theoretical and focused on languages like C and C++, I started to find ways outside of coursework to learn different things. I picked up Python working on web scrapers on the weekends and was eventually hired as a researcher where we used Python to scrape and analyze public data from various sites.

For me, programming spans this wide range of challenging logic and technical problems to more abstract concepts of how humans think and interact with machines and how technology can enhance our daily lives. It fills a gap between academics and art that I didn’t know I needed to, or could, fill.

Ricky: As you’ve already alluded to, you attended Florida State University, where you completed your CS degree. And a degree in Criminology. And another one in Theater… Did you ever sleep? One degree is hard, but three at once? I’m really curious to know your secrets and any time management hacks for studying and learning to code when you have so much else going on.

Emily: I definitely did not sleep much. On top of all of my schoolwork, I worked a nearly full-time job and even worked as an overnight manager at our local coffee shop while still participating in theater rehearsals and performances.

I was able to get a research position in the CS department to eliminate some of that pressure. I was lucky to have started college with a lot of credits and tested out of a few courses, so I was technically already a year ahead, which gave me more freedom to try out courses like Programming.

It’s all very much how I was raised. From a very young age, I knew that my day started around 7 a.m. I went straight from school to rehearsals and dance classes, then had to do homework until I fell asleep. I had to learn how to retain information and figure things out quickly—and I had to stay organized, so I made a lot of lists.

I’ve asked my parents how I came to be this way, and they just shrug! I’ve always felt very in control of how I spend my time to ensure it’s what I want to be doing, and I think that’s important when staying so busy. You have to want to be doing everything, or else things will fall by the wayside.

I definitely suggest finding a manner of keeping to-do lists and prioritizing your time. I use an app called Bear (like a simplified Evernote, but with programmer-friendly themes and markdown support) along with a lot of task prioritization.

I also figured out that I learn things quickly by writing them down multiple times. I used this method to memorize lines for shows. I’d white-out my lines then go back and physically write them down from memory on a separate piece of paper, rinse, and repeat. I got to the point where if I wrote something down 1 to 2 times, it’d stick.

Ricky: You are the co-founder and director of engineering at Cuttlesoft. It looks as if you started the company before finishing college. What was your motivation for starting your own business instead of applying for junior software developer jobs straight out of college?

Emily: Cuttlesoft was a matter of circumstance. I never imagined I’d run my own company, especially not alongside my now-husband Frank. I was in a weird timing limbo where I’d finished my undergraduate degrees earlier than expected which meant that I missed all of the grad school deadlines.

FSU agreed to let me jump in and start my Masters there, and my intention was to stay for a year then transfer elsewhere where I could continue working with parsers, compilers, and formal verification. I was also getting recruited by huge tech companies, and I was a bit enamored with the idea of living in San Francisco or Boston. (I’d only ever lived in Florida at the time.)

But then Frank and I had found our way into this budding entrepreneurship ecosystem in Tallahassee. We met a few people who became great mentors, and before we could even get our Is dotted, we had our first couple of clients. I thought, “Why do I want to leave all of these people who are invested in my future and success to go be one of the thousands somewhere else?”

I figured that I should take the chance on starting something of my own and continuing on this rapid growth path. I knew I’d learn a lot more in a shorter amount of time than I would almost anywhere else. So I dropped out of graduate school after my first semester and put all of my time into Cuttlesoft.

Looking back, I can’t imagine a different path for me. Soon after I turned down those job offers, Susan Fowler’s story came to light. I couldn’t help but think, “That could have been me.” I truly believe that a company’s culture is top-down, and I’m grateful to get to contribute to a company where I can make a huge impact in a positive manner.

Ricky: This year, you got to fulfill a dream and speak at PyCon, with your talk titled The AST and Me. I’ll admit, some of it went over my head, but I’m still learning. I got the impression that language internals fascinate you. What advice would you give to someone who is at the start of their coding journey and wants to know more about how the sausage is made? What resources would you recommend?

Emily: Yes! I was the weird kid in university who loved the classes that most everyone else hated (Programming Languages, Compilers, Theory of Computation…). I would spend hours drawing out non-deterministic finite automata and state machines for my course notes.

I’m a huge fan of Seven Languages in Seven Weeks: A Pragmatic Guide to Learning Programming Languages by Bruce A. Tate to get a feel for different programming language paradigms. The Dragon Book (Compilers: Principles, Techniques, and Tools) is a classic and is the backbone of so much we still use today. (Python’s compiler is based on this.) Philip Guo’s video series on CPython internals is also awesome and helped me in my journey diving into how Python works under the hood.

Ricky: Huge congratulations are in order, as you have just been promoted to a CPython core developer! You must be so thrilled. How was the initiation at the recent CPython sprints? Anything exciting to share or any stories to tell? Don’t worry, we can keep a secret…

Emily: Thank you! The CPython Sprint was a lot of fun. It’s rare that we get so many core developers in the same room working together. We’re all incredibly grateful for the PSF and this year’s sprint sponsor, Microsoft, for supporting CPython.

I was able to attend sprints and Language Summits at PyCons past and had the chance to get to know a lot of the group previously, so this sprint felt surprisingly normal, but it was super cool to see and work with everyone in person.

I spent most of the sprint implementing PEP 572, the (in)famous assignment expressions PEP, with Guido’s guidance. No matter which side of the fence you fell on with assignment expressions (or I as lovingly now call it, the walrus operator), it’s been incredibly cool to add new syntax to the language and deep dive into the internals to get variable scoping to work as intended. It will be in the alpha versions of 3.8 early next year, so keep an eye out!

One of my favorite parts of the sprint was getting to know more about the history of CPython. Since the beginning of my path in core development, I’ve found that it’s really interesting to hear the stories of how others became core developers, so I pose that question to everyone I can.

Understanding everyone’s journey and motivations for devoting so much of their time and energy to a project (especially those who have been involved since the very early days) is an important step to understanding how to continue growing the group and increasing diversity.

Ricky: Now for my last question: what other hobbies and interests do you have, aside from Python? Any you’d like to share and/or plug?

Emily: I try to take advantage of everything Colorado has to offer in my spare time—coming from Florida, I’m still totally enamored with the Rocky Mountains and love hiking. Denver is also a great foodie city.

When I make the time for it, I also really enjoy yoga, reading, listening to podcasts, playing video games (though I’m still slowly working through the most recent God of War), and trying to keep my houseplants alive. I also enjoy spending time with my husband and our dog—they’re my world.

Thank you, Emily, for joining me this week. You can follow Emily’s work on Twitter or Github. Find out more about her company, Cuttlesoft, here.

If there’s someone you’d like me to interview in the future, reach out to me in the comments below, or send me a message on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

The Ultimate List of Data Science Podcasts

Active Data Science Podcasts

Data Skeptic

Linear Digressions

Talking Machines

O’Reilly Data Show

Not So Standard Deviations

Data Stories

SuperDataScience

Data Science at Home

This Week in Machine Learning & Artificial Intelligence (TWiML&AI)

DataFramed

Learning Machines 101

Artificial Intelligence in Industry

Archived Data Science Podcasts

Partially Derivative

Machine Learning Guide / Machine Learning Applied

Becoming a Data Scientist

Conclusion

Python Development in Visual Studio Code

Installing and Configuring Visual Studio Code for Python Development

Extensions for Python Development

Visual Studio Code Configuration Files

Start a New Python Program

Entering Python Code

Running Python Code

Python Linting Support

Editing an Existing Python Project

Testing Support

Debugging Support

Git Integration

Conclusion

Python "for" Loops (Definite Iteration)

A Survey of Definite Iteration in Programming

Numeric Range Loop

Three-Expression Loop

Collection-Based or Iterator-Based Loop

The Python for Loop

Iterables

Iterators

The Guts of the Python for Loop

Iterating Through a Dictionary

The range() Function

Altering for Loop Behavior

The break and continue Statements

The else Clause

Conclusion

Jupyter Notebook: An Introduction

Getting Up and Running With Jupyter Notebook

Installation

Starting the Jupyter Notebook Server

Creating a Notebook

Naming

Running Cells

The Menus

Starting Terminals and Other Things

Viewing What’s Running

Adding Rich Content

Cell Types

Styling Your Text

Headers

Creating Lists

Code and Syntax Highlighting

Exporting Notebooks

How to Use nbconvert

Example Usage

Use the Menu

Notebook Extensions

What Are Extensions?

Where Do I Get Extensions?

How Do I Install Them?

Conclusion

Further Reading

Python’s Requests Library (Guide)

Getting Started With requests

The GET Request

The Response

Status Codes

Content

The Python `for` Loop

The Guts of the Python `for` Loop

The `range()` Function

Altering `for` Loop Behavior

The `break` and `continue` Statements

The `else` Clause

How to Use `nbconvert`

Getting Started With `requests`

Simple Filename Pattern Matching Using `fnmatch`

Filename Pattern Matching Using `glob`

The `asyncio` Package and `async`/`await`

The `async`/`await` Syntax and Native Coroutines

Other Features: `async for` and Async Generators + Comprehensions

The Event Loop and `asyncio.run()`