Real Python

Working With JSON Data in Python

2019-03-28T14:00:00+00:00

JSON is a lightweight data-interchange format. It allows us to represent the objects in our Python programs as human-readable text that can be sent over the internet. Lots of APIs and databases use JSON for communication.

You’ll learn how to work with Python’s built-in json module to serialize the data in your programs into JSON format. Then, you’ll deserialize some JSON from an online API and convert it into Python objects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Stand Out in a Python Coding Interview

2019-03-27T14:00:00+00:00

You’ve made it past the phone call with the recruiter, and now it’s time to show that you know how to solve problems with actual code. Whether it’s a HackerRank exercise, a take-home assignment, or an onsite whiteboard interview, this is your moment to prove your coding interview skills.

But interviews aren’t just about solving problems: they’re also about showing that you can write clean production code. This means that you have a deep knowledge of Python’s built-in functionality and libraries. This knowledge shows companies that you can move quickly and won’t duplicate functionality that comes with the language just because you don’t know it exists.

At Real Python, we’ve put our heads together and discussed what tools we’re always impressed to see in coding interviews. This article will walk you through the best of that functionality, starting with Python built-ins, then Python’s native support for data structures, and finally Python’s powerful (and often underappreciated) standard library.

In this article, you’ll learn how to:

Use enumerate() to iterate over both indices and values
Debug problematic code with breakpoint()
Format strings effectively with f-strings
Sort lists with custom arguments
Use generators instead of list comprehensions to conserve memory
Define default values when looking up dictionary keys
Count hashable objects with the collections.Counter class
Use the standard library to get lists of permutations and combinations

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Select the Right Built-In Function for the Job

Python has a large standard library but only a small library of built-in functions, which are always available and don’t need to be imported. It’s worth going through each one, but until you get the chance to do so, here are a few built-in functions worth understanding how to use, and in the case of some of them, what alternatives to use instead.

Iterate With `enumerate()` Instead of `range()`

This scenario might come up more than any other in coding interviews: you have a list of elements, and you need to iterate over the list with access to both the indices and the values.

There’s a classic coding interview question named FizzBuzz that can be solved by iterating over both indices and values. In FizzBuzz, you are given a list of integers. Your task is to do the following:

Replace all integers that are evenly divisible by 3 with "fizz"
Replace all integers divisible by 5 with "buzz"
Replace all integers divisible by both 3 and 5 with "fizzbuzz"

Often, developers will solve this problem with range():

>>>

>>> numbers = [45, 22, 14, 65, 97, 72]
>>> for i in range(len(numbers)):
...     if numbers[i] % 3 == 0 and numbers[i] % 5 == 0:
...         numbers[i] = 'fizzbuzz'
...     elif numbers[i] % 3 == 0:
...         numbers[i] = 'fizz'
...     elif numbers[i] % 5 == 0:
...         numbers[i] = 'buzz'
...
>>> numbers
['fizzbuzz', 22, 14, 'buzz', 97, 'fizz']

Range allows you to access the elements of numbers by index and is a useful tool for some situations. But in this case, where you want to get each element’s index and value at the same time, a more elegant solution uses enumerate():

>>>

>>> numbers = [45, 22, 14, 65, 97, 72]
>>> for i, num in enumerate(numbers):
...     if num % 3 == 0 and num % 5 == 0:
...         numbers[i] = 'fizzbuzz'
...     elif num % 3 == 0:
...         numbers[i] = 'fizz'
...     elif num % 5 == 0:
...         numbers[i] = 'buzz'
...
>>> numbers
['fizzbuzz', 22, 14, 'buzz', 97, 'fizz']

For each element, enumerate() returns a counter and the element value. The counter defaults to 0, which conveniently is also the element’s index. Don’t want to start your count at 0? Just use the optional start parameter to set an offset:

>>>

>>> numbers = [45, 22, 14, 65, 97, 72]
>>> for i, num in enumerate(numbers, start=52):
...     print(i, num)
...
52 45
53 22
54 14
55 65
56 97
57 72

By using the start parameter, we access all of the same elements, starting with the first index, but now our count starts from the specified integer value.

Use List Comprehensions Instead of `map()` and `filter()`

“I think dropping filter() and map() is pretty uncontroversial[.]”

— Guido van Rossum, Python’s creator

He may have been wrong about it being uncontroversial, but Guido had good reasons for wanting to remove map() and filter() from Python. One reason is that Python supports list comprehensions, which are often easier to read and support the same functionality as map() and filter().

Let’s first take a look at how we’d structure a call to map() and the equivalent list comprehension:

>>>

>>> numbers = [4, 2, 1, 6, 9, 7]
>>> def square(x):
...     return x*x
...
>>> list(map(square, numbers))
[16, 4, 1, 36, 81, 49]

>>> [square(x) for x in numbers]
[16, 4, 1, 36, 81, 49]

Both approaches, using map() and the list comprehension, return the same values, but the list comprehension is easier to read and understand.

Now we can do the same thing for the filter() and its equivalent list comprehension:

>>>

>>> def is_odd(x):
...    return bool(x % 2)
...
>>> list(filter(is_odd, numbers))
[1, 9, 7]

>>> [x for x in numbers if is_odd(x)]
[1, 9, 7]

Like we saw with map(), the filter() and list comprehension approaches return the same value, but the list comprehension is easier to follow.

Developers who come from other languages may disagree that list comprehensions are easier to read than map() and filter(), but in my experience beginners are able to pick up list comprehensions much more intuitively.

Either way, you’ll rarely go wrong using list comprehensions in a coding interview, as it will communicate that you know what’s most common in Python.

Debug With `breakpoint()` Instead of `print()`

You may have debugged a small problem by adding print() to your code and seeing what was printed out. This approach works well at first but quickly becomes cumbersome. Plus, in a coding interview setting, you hardly want print() calls peppered throughout your code.

Instead, you should be using a debugger. For non-trivial bugs, it’s almost always faster than using print(), and given that debugging is a big part of writing software, it shows that you know how to use tools that will let you develop quickly on the job.

If you’re using Python 3.7, you don’t need to import anything and can just call breakpoint() at the location in your code where you’d like to drop into the debugger:

# Some complicated code with bugs

breakpoint()

Calling breakpoint() will put you into pdb, which is the default Python debugger. On Python 3.6 and older, you can do the same by importing pdb explicitly:

import pdb; pdb.set_trace()

Like breakpoint(), pdb.set_trace() will put you into the pdb debugger. It’s just not quite as clean and is a tad more to remember.

There are other debuggers available that you may want to try, but pdb is part of the standard library, so it’s always available. Whatever debugger you prefer, it’s worth trying them out to get used to the workflow before you’re in a coding interview setting.

Format strings With f-Strings

Python has a lot of different ways to handle string formatting, and it can be tricky to know what to use. In fact, we tackle formatting in depth in two separate articles: one about string formatting in general and one specifically focused on f-strings. In a coding interview, where you’re (hopefully) using Python 3.6+, the suggested formatting approach is Python’s f-strings.

f-strings support use of the string formatting mini-language, as well as powerful string interpolation. These features allow you to add variables or even valid Python expressions and have them evaluated at runtime before being added to the string:

>>>

>>> def get_name_and_decades(name, age):
...     return f"My name is {name} and I'm {age / 10:.5f} decades old."
...
>>> get_name_and_decades("Maria", 31)
My name is Maria and I'm 3.10000 decades old.

The f-string allows you to put Maria into the string and add her age with the desired formatting in one succinct operation.

The one risk to be aware of is that if you’re outputting user-generated values, then that can introduce security risks, in which case Template Strings may be a safer option.

Sort Complex Lists With `sorted()`

Plenty of coding interview questions require some kind of sorting, and there are multiple valid ways you can sort items. Unless the interviewer wants you to implement your own sorting algorithm, it’s usually best to use sorted().

You’ve probably seen the most simple uses of sorting, such as sorting lists of numbers or strings in ascending or descending order:

>>>

>>> sorted([6,5,3,7,2,4,1])
[1, 2, 3, 4, 5, 6, 7]

>>> sorted(['cat', 'dog', 'cheetah', 'rhino', 'bear'], reverse=True)
['rhino', 'dog', 'cheetah', 'cat', 'bear]

By default, sorted() has sorted the input in ascending order, and the reverse keyword argument causes it to sort in descending order.

It’s worth knowing about the optional keyword argument key that lets you specify a function that will be called on every element prior to sorting. Adding a function allows custom sorting rules, which are especially helpful if you want to sort more complex data types:

>>>

>>> animals = [
...     {'type': 'penguin', 'name': 'Stephanie', 'age': 8},
...     {'type': 'elephant', 'name': 'Devon', 'age': 3},
...     {'type': 'puma', 'name': 'Moe', 'age': 5},
... ]
>>> sorted(animals, key=lambda animal: animal['age'])
[
    {'type': 'elephant', 'name': 'Devon', 'age': 3},
    {'type': 'puma', 'name': 'Moe', 'age': 5},
    {'type': 'penguin', 'name': 'Stephanie, 'age': 8},
]

By passing in a lambda function that returns each element’s age, you can easily sort a list of dictionaries by a single value of each of those dictionaries. In this case, the dictionary is now sorted in ascending order by age.

Leverage Data Structures Effectively

Algorithms get a lot of attention in coding interviews, but data structures are arguably even more important. In a coding interviewing context, picking the right data structure can have a major impact on performance.

Beyond theoretical data structures, Python has powerful and convenient functionality built into its standard data structure implementations. These data structures are incredibly useful in coding interviews because they give you lots of functionality by default and let you focus your time on other parts of the problem.

Store Unique Values With Sets

You’ll often need to remove duplicate elements from an existing dataset. New developers will sometimes do so with lists when they should be using sets, which enforce the uniqueness of all elements.

Pretend you have a function named get_random_word(). It will always return a random selection from a small set of words:

>>>

>>> import random
>>> all_words = "all the words in the world".split()
>>> def get_random_word():
...    return random.choice(all_words)

You’re supposed to call get_random_word() repeatedly to get 1000 random words and then return a data structure containing every unique word. Here are two common, suboptimal approaches and one good approach.

Bad Approach

get_unique_words() stores values in a list then converts the list into a set:

>>>

>>> def get_unique_words():
...     words = []
...     for _ in range(1000):
...         words.append(get_random_word())
...     return set(words)
>>> get_unique_words()
{'world', 'all', 'the', 'words'}

This approach isn’t terrible, but it unnecessarily creates a list and then converts it to a set. Interviewers almost always notice (and ask about) this type of design choice.

Worse Approach

To avoid converting from a list to a set, you now store values in a list without using any other data structures. You then test for uniqueness by comparing new values with all elements currently in the list:

>>>

>>> def get_unique_words():
...     words = []
...     for _ in range(1000):
...         word = get_random_word()
...         if word not in words:
...             words.append(word)
...     return words
>>> get_unique_words()
['world', 'all', 'the', 'words']

This is worse than the first approach, because you have to compare every new word against every word already in the list. That means that as the number of words grows, the number of lookups grows quadratically. In other words, the time complexity grows on the order of O(N²).

Good Approach

Now, you skip using lists altogether and instead use a set from the start:

>>>

>>> def get_unique_words():
...     words = set()
...     for _ in range(1000):
...         words.add(get_random_word())
...     return words
>>> get_unique_words()
{'world', 'all', 'the', 'words'}

This may not look much different than the other approaches except for making use of a set from the beginning. If you consider what’s happening within .add(), it even sounds like the second approach: get the word, check if it’s already in the set, and if not, add it to the data structure.

So why is using a set different from the second approach?

It’s different because sets store elements in a manner that allows near-constant-time checks whether a value is in the set or not, unlike lists, which require linear-time lookups. The difference in lookup time means that the time complexity for adding to a set grows at a rate of O(N), which is much better than the O(N²) from the second approach in most cases.

Save Memory With Generators

List comprehensions are convenient tools but can sometimes lead to unnecessary memory usage.

Imagine you’ve been asked to find the sum of the first 1000 perfect squares, starting with 1. You know about list comprehensions, so you quickly code up a working solution:

>>>

>>> sum([i * i for i in range(1, 1001)])
333833500

Your solution makes a list of every perfect square between 1 and 1,000,000 and sums the values. Your code returns the right answer, but then your interviewer starts increasing the number of perfect squares you need to sum.

At first, your function keeps popping out the right answer, but soon it starts slowing down until eventually the process seems to hang for an eternity. This is the last thing you want happening in a coding interview.

What’s going on here?

It’s making a list of every perfect square you’ve requested and summing them all. A list with 1000 perfect squares may not be large in computer-terms, but 100 million or 1 billion is quite a bit of information and can easily overwhelm your computer’s available memory resources. That’s what’s happening here.

Thankfully, there’s a quick way to solve the memory problem. You just replace the brackets with parentheses:

>>>

>>> sum((i * i for i in range(1, 1001)))
333833500

Swapping out the brackets changes your list comprehension into a generator expression. Generator expressions are perfect for when you know you want to retrieve data from a sequence, but you don’t need to access all of it at the same time.

Instead of creating a list, the generator expression returns a generator object. That object knows where it is in the current state (for example, i = 49) and only calculates the next value when it’s asked for.

So when sum iterates over the generator object by calling .__next__() repeatedly, the generator checks what i equals, calculates i * i, increments i internally, and returns the proper value to sum. The design allows generators to be used on massive sequences of data, because only one element exists in memory at a time.

Define Default Values in Dictionaries With `.get()` and `.setdefault()`

One of the most common programming tasks involves adding, modifying, or retrieving an item that may or may not be in a dictionary. Python dictionaries have elegant functionality to make these tasks clean and easy, but developers often check explicitly for values when it isn’t necessary.

Imagine you have a dictionary named cowboy, and you want to get that cowboy’s name. One approach is to explicitly check for the key with a conditional:

>>>

>>> cowboy = {'age': 32, 'horse': 'mustang', 'hat_size': 'large'}
>>> if 'name' in cowboy:
...     name = cowboy['name']
... else:
...     name = 'The Man with No Name'
...
>>> name
'The Man with No Name'

This approach first checks if the name key exists in the dictionary, and if so, it returns the corresponding value. Otherwise, it returns a default value.

While explicitly checking for keys does work, it can easily be replaced with one line if you use .get():

>>>

>>> name = cowboy.get('name', 'The Man with No Name')

get() performs the same operations that were performed in the first approach, but now they’re handled automatically. If the key exists, then the proper value will be returned. Otherwise, the default value will get returned.

But what if you want to update the dictionary with a default value while still accessing the name key? .get() doesn’t really help you here, so you’re left with explicitly checking for the value again:

>>>

>>> if 'name' not in cowboy:
...     cowboy['name'] = 'The Man with No Name'
...
>>> name = cowboy['name']

Checking for the value and setting a default is a valid approach and is easy to read, but again Python offers a more elegant method with .setdefault():

>>>

>>> name = cowboy.setdefault('name', 'The Man with No Name')

.setdefault() accomplishes exactly the same thing as the snippet above. It checks if name exists in cowboy, and if so it returns that value. Otherwise, it sets cowboy['name'] to The Man with No Name and returns the new value.

Take Advantage of Python’s Standard Library

By default, Python comes with a lot of functionality that’s just an import statement away. It’s powerful on its own, but knowing how to leverage the standard library can supercharge your coding interview skills.

It’s hard to pick the most useful pieces from all of the available modules, so this section will focus on just a small subset of its utility functions. Hopefully, these will prove useful to you in coding interviews and also whet your appetite to learn more about the advanced functionality of these and other modules.

Handle Missing Dictionary Keys With `collections.defaultdict()`

.get() and .setdefault() work well when you’re setting a default for a single key, but it’s common to want a default value for all possible unset keys, especially when programming in a coding interview context.

Pretend you have a group of students, and you need to keep track of their grades on homework assignments. The input value is a list of tuples with the format (student_name, grade), but you want to easily look up all the grades for a single student without iterating over the list.

One way to store the grade data uses a dictionary that maps student names to lists of grades:

>>>

>>> student_grades = {}
>>> grades = [
...     ('elliot', 91),
...     ('neelam', 98),
...     ('bianca', 81),
...     ('elliot', 88),
... ]
>>> for name, grade in grades:
...     if name not in student_grades:
...         student_grades[name] = []
...     student_grades[name].append(grade)
...
>>> student_grades
{'elliot': [91, 88], 'neelam': [98], 'bianca': [81]}

In this approach, you iterate over the students and check if their names are already properties in the dictionary. If not, you add them to the dictionary with an empty list as the default value. You then append their actual grades to that student’s list of grades.

But there’s an even cleaner approach that uses a defaultdict, which extends standard dict functionality to allow you to set a default value that will be operated upon if the key doesn’t exist:

>>>

>>> from collections import defaultdict
>>> student_grades = defaultdict(list)
>>> for name, grade in grades:
...     student_grades[name].append(grade)

In this case, you’re creating a defaultdict that uses the list() constructor with no arguments as a default factory method. list() with no arguments returns an empty list, so defaultdict calls list() if the name doesn’t exist and then allows the grade to be appended. If you want to get fancy, you could also use a lambda function as your factory value to return an arbitrary constant.

Leveraging a defaultdict can lead to cleaner application code because you don’t have to worry about default values at the key level. Instead, you can handle them once at the defaultdict level and afterwards act as if the key is always present.

Count Hashable Objects With `collections.Counter`

You have a long string of words with no punctuation or capital letters and you want to count how many times each word appears.

You could use a dictionary or defaultdict and increment the counts, but collections.Counter provides a cleaner and more convenient way to do exactly that. Counter is a subclass of dict that uses 0 as the default value for any missing element and makes it easier to count occurrences of objects:

>>>

>>> from collections import Counter
>>> words = "if there was there was but if \
... there was not there was not".split()
>>> counts = Counter(words)
>>> counts
Counter({'if': 2, 'there': 4, 'was': 4, 'not': 2, 'but': 1})

When you pass in the list of words to Counter, it stores each word along with a count of how many times that word occurred in the list.

Are you curious what the two most common words were? Just use .most_common():

>>>

>>> counts.most_common(2)
[('there', 4), ('was', 4)]

.most_common() is a convenience method and simply returns the n most frequent inputs by count.

Access Common String Groups With `string` Constants

It’s trivia time! Is 'A' > 'a' true or false?

It’s false, because the ASCII code for A is 65, but a is 97, and 65 is not greater than 97.

Why does the answer matter? Because if you want to check if a character is part of the English alphabet, one popular way is to see if it’s between A and z (65 and 122 on the ASCII chart).

Checking the ASCII code works but is clumsy and easy to mess up in coding interviews, especially if you can’t remember whether lowercase or uppercase ASCII characters come first. It’s much easier to use the constants defined as part of the string module.

You can see one in use in is_upper(), which returns whether all characters in a string are uppercase letters:

>>>

>>> import string
>>> def is_upper(word):
...     for letter in word:
...         if letter not in string.ascii_uppercase:
...             return False
...     return True
...
>>> is_upper('Thanks Geir')
False
>>> is_upper('LOL')
True

is_upper() iterates over the letters in word, and checks if the letters are part of string.ascii_uppercase. If you print out string.ascii_uppercase you’ll see that it’s just a lowly string. The value is set to the literal 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'.

All string constants are just strings of frequently referenced string values. They include the following:

string.ascii_letters
string.ascii_uppercase
string.ascii_lowercase
string.digits
string.hexdigits
string.octdigits
string.punctuation
string.printable
string.whitespace

These are easier to use and, even more importantly, easier to read.

Generate Permutations and Combinations With `itertools`

Interviewers love to give real life scenarios to make coding interviews seem less intimidating, so here’s a contrived example: you go to an amusement park and decide to figure out every possible pair of friends that could sit together on a roller coaster.

Unless generating these pairs is the primary purpose of the interview question, it’s likely that generating all the possible pairs is just a tedious step on the way towards a working algorithm. You could calculate them yourself with nested for-loops, or you could use the powerful itertools library.

itertools has multiple tools for generating iterable sequences of input data, but right now we’ll just focus on two common functions: itertools.permutations() and itertools.combinations().

itertools.permutations() builds a list of all permutations, meaning it’s a list of every possible grouping of input values with a length matching the count parameter. The r keyword argument lets us specify how many values go in each grouping:

>>>

>>> import itertools
>>> friends = ['Monique', 'Ashish', 'Devon', 'Bernie']
>>> list(itertools.permutations(friends, r=2))
[('Monique', 'Ashish'), ('Monique', 'Devon'), ('Monique', 'Bernie'),
('Ashish', 'Monique'), ('Ashish', 'Devon'), ('Ashish', 'Bernie'),
('Devon', 'Monique'), ('Devon', 'Ashish'), ('Devon', 'Bernie'),
('Bernie', 'Monique'), ('Bernie', 'Ashish'), ('Bernie', 'Devon')]

With permutations, the order of the elements matters, so ('sam', 'devon') represents a different pairing than ('devon', 'sam'), meaning that they would both be included in the list.

itertools.combinations() builds combinations. These are also the possible groupings of the input values, but now the order of the values doesn’t matter. Because ('sam', 'devon') and ('devon', 'sam') represent the same pair, only one of them would be included in the output list:

>>>

>>> list(itertools.combinations(friends, r=2))
[('Monique', 'Ashish'), ('Monique', 'Devon'), ('Monique', 'Bernie'),
('Ashish', 'Devon'), ('Ashish', 'Bernie'), ('Devon', 'Bernie')]

Since the order of the values matters with combinations, there are fewer combinations than permutations for the same input list. Again, because we set r to 2, each grouping has two names in it.

.combinations() and .permutations() are just small examples of a powerful library, but even these two functions can be quite useful when you’re trying to solve an algorithm problem quickly.

Conclusion: Coding Interview Superpowers

You can now feel comfortable using some of Python’s less common, but more powerful, standard features in your next coding interview. There’s a lot to learn about the language as a whole but this article should have given you a starting point to go deeper while letting you use Python more effectively when you interview.

In this article, you’ve learned different types of standard tools to supercharge your coding interview skills:

Powerful built-in functions
Data structures built to handle common scenarios with barely any code
Standard library packages that have feature-rich solutions for specific problems, letting you write better code faster

Interviewing may not be the best approximation of real software development, but it’s worth knowing how to succeed in any programming environment, even interviews. Thankfully, learning how to use Python during coding interviews can help you understand the language more deeply, which will pay dividends during day-to-day development.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Django Migrations 101

2019-03-26T14:00:00+00:00

Since version 1.7, Django has come with built-in support for database migrations. In Django, database migrations usually go hand in hand with models: whenever you code up a new model, you also generate a migration to create the necessary table in the database. However, migrations can do much more.

With this course you’ll get comfortable with Django migrations and learn how to create database tables without writing any SQL, how to automatically modify your database after you changed your models, and how to revert changes made to your database.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Documenting Python Projects With Sphinx and Read The Docs

2019-03-26T13:30:00+00:00

In this video series we’ll cover creating Python documentation from scratch using Sphinx, as well as getting your code repository hooked up to Read The Docs, to automatically build and publish your code documentation.

Note: This course uses Python 2.7 in its coding examples.

Resources

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

An Intro to Threading in Python

2019-03-25T14:00:00+00:00

Python threading allows you to have different parts of your program run concurrently and can simplify your design. If you’ve got some experience in Python and want to speed up your program using threads, then this tutorial is for you!

In this article, you’ll learn:

What threads are
How to create threads and wait for them to finish
How to use a ThreadPoolExecutor
How to avoid race conditions
How to use the common tools that Python threading provides

This article assumes you’ve got the Python basics down pat and that you’re using at least version 3.6 to run the examples. If you need a refresher, you can start with the Python Learning Paths and get up to speed.

If you’re not sure if you want to use Python threading, asyncio, or multiprocessing, then you can check out Speed Up Your Python Program With Concurrency.

All of the sources used in this tutorial are available to you in the Real Python GitHub repo.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Take the Quiz: Test your knowledge with our interactive “Python Threading” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Take the Quiz »

What Is a Thread?

A thread is a separate flow of execution. This means that your program will have two things happening at once. But for most Python 3 implementations the different threads do not actually execute at the same time: they merely appear to.

It’s tempting to think of threading as having two (or more) different processors running on your program, each one doing an independent task at the same time. That’s almost right. The threads may be running on different processors, but they will only be running one at a time.

Getting multiple tasks running simultaneously requires a non-standard implementation of Python, writing some of your code in a different language, or using multiprocessing which comes with some extra overhead.

Because of the way CPython implementation of Python works, threading may not speed up all tasks. This is due to interactions with the GIL that essentially limit one Python thread to run at a time.

Tasks that spend much of their time waiting for external events are generally good candidates for threading. Problems that require heavy CPU computation and spend little time waiting for external events might not run faster at all.

This is true for code written in Python and running on the standard CPython implementation. If your threads are written in C they have the ability to release the GIL and run concurrently. If you are running on a different Python implementation, check with the documentation too see how it handles threads.

If you are running a standard Python implementation, writing in only Python, and have a CPU-bound problem, you should check out the multiprocessing module instead.

Architecting your program to use threading can also provide gains in design clarity. Most of the examples you’ll learn about in this tutorial are not necessarily going to run faster because they use threads. Using threading in them helps to make the design cleaner and easier to reason about.

So, let’s stop talking about threading and start using it!

Starting a Thread

Now that you’ve got an idea of what a thread is, let’s learn how to make one. The Python standard library provides threading, which contains most of the primitives you’ll see in this article. Thread, in this module, nicely encapsulates threads, providing a clean interface to work with them.

To start a separate thread, you create a Thread instance and then tell it to .start():

 1 import logging
 2 import threading
 3 import time
 4 
 5 def thread_function(name):
 6     logging.info("Thread %s: starting", name)
 7     time.sleep(2)
 8     logging.info("Thread %s: finishing", name)
 9 
10 if __name__ == "__main__":
11     format = "%(asctime)s: %(message)s"
12     logging.basicConfig(format=format, level=logging.INFO,
13                         datefmt="%H:%M:%S")
14 
15     logging.info("Main    : before creating thread")
16     x = threading.Thread(target=thread_function, args=(1,))
17     logging.info("Main    : before running thread")
18     x.start()
19     logging.info("Main    : wait for the thread to finish")
20     # x.join()
21     logging.info("Main    : all done")

If you look around the logging statements, you can see that the main section is creating and starting the thread:

x = threading.Thread(target=thread_function, args=(1,))
x.start()

When you create a Thread, you pass it a function and a list containing the arguments to that function. In this case, you’re telling the Thread to run thread_function() and to pass it 1 as an argument.

For this article, you’ll use sequential integers as names for your threads. There is threading.get_ident(), which returns a unique name for each thread, but these are usually neither short nor easily readable.

thread_function() itself doesn’t do much. It simply logs some messages with a time.sleep() in between them.

When you run this program as it is (with line twenty commented out), the output will look like this:

$ ./single_thread.py
Main    : before creating thread
Main    : before running thread
Thread 1: starting
Main    : wait for the thread to finish
Main    : all done
Thread 1: finishing

You’ll notice that the Thread finished after the Main section of your code did. You’ll come back to why that is and talk about the mysterious line twenty in the next section.

Daemon Threads

In computer science, a daemon is a process that runs in the background.

Python threading has a more specific meaning for daemon. A daemon thread will shut down immediately when the program exits. One way to think about these definitions is to consider the daemon thread a thread that runs in the background without worrying about shutting it down.

If a program is running Threads that are not daemons, then the program will wait for those threads to complete before it terminates. Threads that are daemons, however, are just killed wherever they are when the program is exiting.

Let’s look a little more closely at the output of your program above. The last two lines are the interesting bit. When you run the program, you’ll notice that there is a pause (of about 2 seconds) after __main__ has printed its all done message and before the thread is finished.

This pause is Python waiting for the non-daemonic thread to complete. When your Python program ends, part of the shutdown process is to clean up the threading routine.

If you look at the source for Python threading, you’ll see that threading._shutdown() walks through all of the running threads and calls .join() on every one that does not have the daemon flag set.

So your program waits to exit because the thread itself is waiting in a sleep. As soon as it has completed and printed the message, .join() will return and the program can exit.

Frequently, this behavior is what you want, but there are other options available to us. Let’s first repeat the program with a daemon thread. You do that by changing how you construct the Thread, adding the daemon=True flag:

x = threading.Thread(target=thread_function, args=(1,), daemon=True)

When you run the program now, you should see this output:

$ ./daemon_thread.py
Main    : before creating thread
Main    : before running thread
Thread 1: starting
Main    : wait for the thread to finish
Main    : all done

The difference here is that the final line of the output is missing. thread_function() did not get a chance to complete. It was a daemon thread, so when __main__ reached the end of its code and the program wanted to finish, the daemon was killed.

`join()` a Thread

Daemon threads are handy, but what about when you want to wait for a thread to stop? What about when you want to do that and not exit your program? Now let’s go back to your original program and look at that commented out line twenty:

# x.join()

To tell one thread to wait for another thread to finish, you call .join(). If you uncomment that line, the main thread will pause and wait for the thread x to complete running.

Did you test this on the code with the daemon thread or the regular thread? It turns out that it doesn’t matter. If you .join() a thread, that statement will wait until either kind of thread is finished.

Working With Many Threads

The example code so far has only been working with two threads: the main thread and one you started with the threading.Thread object.

Frequently, you’ll want to start a number of threads and have them do interesting work. Let’s start by looking at the harder way of doing that, and then you’ll move on to an easier method.

The harder way of starting multiple threads is the one you already know:

import logging
import threading
import time

def thread_function(name):
    logging.info("Thread %s: starting", name)
    time.sleep(2)
    logging.info("Thread %s: finishing", name)

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")

    threads = list()
    for index in range(3):
        logging.info("Main    : create and start thread %d.", index)
        x = threading.Thread(target=thread_function, args=(index,))
        threads.append(x)
        x.start()

    for index, thread in enumerate(threads):
        logging.info("Main    : before joining thread %d.", index)
        thread.join()
        logging.info("Main    : thread %d done", index)

This code uses the same mechanism you saw above to start a thread, create a Thread object, and then call .start(). The program keeps a list of Thread objects so that it can then wait for them later using .join().

Running this code multiple times will likely produce some interesting results. Here’s an example output from my machine:

$ ./multiple_threads.py
Main    : create and start thread 0.
Thread 0: starting
Main    : create and start thread 1.
Thread 1: starting
Main    : create and start thread 2.
Thread 2: starting
Main    : before joining thread 0.
Thread 2: finishing
Thread 1: finishing
Thread 0: finishing
Main    : thread 0 done
Main    : before joining thread 1.
Main    : thread 1 done
Main    : before joining thread 2.
Main    : thread 2 done

If you walk through the output carefully, you’ll see all three threads getting started in the order you might expect, but in this case they finish in the opposite order! Multiple runs will produce different orderings. Look for the Thread x: finishing message to tell you when each thread is done.

The order in which threads are run is determined by the operating system and can be quite hard to predict. It may (and likely will) vary from run to run, so you need to be aware of that when you design algorithms that use threading.

Fortunately, Python gives you several primitives that you’ll look at later to help coordinate threads and get them running together. Before that, let’s look at how to make managing a group of threads a bit easier.

Using a `ThreadPoolExecutor`

There’s an easier way to start up a group of threads than the one you saw above. It’s called a ThreadPoolExecutor, and it’s part of the standard library in concurrent.futures (as of Python 3.2).

The easiest way to create it is as a context manager, using the with statement to manage the creation and destruction of the pool.

Here’s the __main__ from the last example rewritten to use a ThreadPoolExecutor:

import concurrent.futures

# [rest of code]

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")

    with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
        executor.map(thread_function, range(3))

The code creates a ThreadPoolExecutor as a context manager, telling it how many worker threads it wants in the pool. It then uses .map() to step through an iterable of things, in your case range(3), passing each one to a thread in the pool.

The end of the with block causes the ThreadPoolExecutor to do a .join() on each of the threads in the pool. It is strongly recommended that you use ThreadPoolExecutor as a context manager when you can so that you never forget to .join() the threads.

Note: Using a ThreadPoolExecutor can cause some confusing errors.

For example, if you call a function that takes no parameters, but you pass it parameters in .map(), the thread will throw an exception.

Unfortunately, ThreadPoolExecutor will hide that exception, and (in the case above) the program terminates with no output. This can be quite confusing to debug at first.

Running your corrected example code will produce output that looks like this:

$ ./executor.py
Thread 0: starting
Thread 1: starting
Thread 2: starting
Thread 1: finishing
Thread 0: finishing
Thread 2: finishing

Again, notice how Thread 1 finished before Thread 0. The scheduling of threads is done by the operating system and does not follow a plan that’s easy to figure out.

Race Conditions

Before you move on to some of the other features tucked away in Python threading, let’s talk a bit about one of the more difficult issues you’ll run into when writing threaded programs: race conditions.

Once you’ve seen what a race condition is and looked at one happening, you’ll move on to some of the primitives provided by the standard library to prevent race conditions from happening.

Race conditions can occur when two or more threads access a shared piece of data or resource. In this example, you’re going to create a large race condition that happens every time, but be aware that most race conditions are not this obvious. Frequently, they only occur rarely, and they can produce confusing results. As you can imagine, this makes them quite difficult to debug.

Fortunately, this race condition will happen every time, and you’ll walk through it in detail to explain what is happening.

For this example, you’re going to write a class that updates a database. Okay, you’re not really going to have a database: you’re just going to fake it, because that’s not the point of this article.

Your FakeDatabase will have .__init__() and .update() methods:

class FakeDatabase:
    def __init__(self):
        self.value = 0

    def update(self, name):
        logging.info("Thread %s: starting update", name)
        local_copy = self.value
        local_copy += 1
        time.sleep(0.1)
        self.value = local_copy
        logging.info("Thread %s: finishing update", name)

FakeDatabase is keeping track of a single number: .value. This is going to be the shared data on which you’ll see the race condition.

.__init__() simply initializes .value to zero. So far, so good.

.update() looks a little strange. It’s simulating reading a value from a database, doing some computation on it, and then writing a new value back to the database.

In this case, reading from the database just means copying .value to a local variable. The computation is just to add one to the value and then .sleep() for a little bit. Finally, it writes the value back by copying the local value back to .value.

Here’s how you’ll use this FakeDatabase:

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")

    database = FakeDatabase()
    logging.info("Testing update. Starting value is %d.", database.value)
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        for index in range(2):
            executor.submit(database.update, index)
    logging.info("Testing update. Ending value is %d.", database.value)

The program creates a ThreadPoolExecutor with two threads and then calls .submit() on each of them, telling them to run database.update().

.submit() has a signature that allows both positional and named arguments to be passed to the function running in the thread:

.submit(function, *args, **kwargs)

In the usage above, index is passed as the first and only positional argument to database.update(). You’ll see later in this article where you can pass multiple arguments in a similar manner.

Since each thread runs .update(), and .update() adds one to .value, you might expect database.value to be 2 when it’s printed out at the end. But you wouldn’t be looking at this example if that was the case. If you run the above code, the output looks like this:

$ ./racecond.py
Testing unlocked update. Starting value is 0.
Thread 0: starting update
Thread 1: starting update
Thread 0: finishing update
Thread 1: finishing update
Testing unlocked update. Ending value is 1.

You might have expected that to happen, but let’s look at the details of what’s really going on here, as that will make the solution to this problem easier to understand.

One Thread

Before you dive into this issue with two threads, let’s step back and talk a bit about some details of how threads work.

You won’t be diving into all of the details here, as that’s not important at this level. We’ll also be simplifying a few things in a way that won’t be technically accurate but will give you the right idea of what is happening.

When you tell your ThreadPoolExecutor to run each thread, you tell it which function to run and what parameters to pass to it: executor.submit(database.update, index).

The result of this is that each of the threads in the pool will call database.update(index). Note that database is a reference to the one FakeDatabase object created in __main__. Calling .update() on that object calls an instance method on that object.

Each thread is going to have a reference to the same FakeDatabase object, database. Each thread will also have a unique value, index, to make the logging statements a bit easier to read:

When the thread starts running .update(), it has its own version of all of the data local to the function. In the case of .update(), this is local_copy. This is definitely a good thing. Otherwise, two threads running the same function would always confuse each other. It means that all variables that are scoped (or local) to a function are thread-safe.

Now you can start walking through what happens if you run the program above with a single thread and a single call to .update().

The image below steps through the execution of .update() if only a single thread is run. The statement is shown on the left followed by a diagram showing the values in the thread’s local_value and the shared database.value:

The diagram is laid out so that time increases as you move from top to bottom. It begins when Thread 1 is created and ends when it is terminated.

When Thread 1 starts, FakeDatabase.value is zero. The first line of code in the method, local_copy = self.value, copies the value zero to the local variable. Next it increments the value of local_copy with the local_copy += 1 statement. You can see .value in Thread 1 getting set to one.

Next time.sleep() is called, which makes the current thread pause and allows other threads to run. Since there is only one thread in this example, this has no effect.

When Thread 1 wakes up and continues, it copies the new value from local_copy to FakeDatabase.value, and then the thread is complete. You can see that database.value is set to one.

So far, so good. You ran .update() once and FakeDatabase.value was incremented to one.

Two Threads

Getting back to the race condition, the two threads will be running concurrently but not at the same time. They will each have their own version of local_copy and will each point to the same database. It is this shared database object that is going to cause the problems.

The program starts with Thread 1 running .update():

When Thread 1 calls time.sleep(), it allows the other thread to start running. This is where things get interesting.

Thread 2 starts up and does the same operations. It’s also copying database.value into its private local_copy, and this shared database.value has not yet been updated:

When Thread 2 finally goes to sleep, the shared database.value is still unmodified at zero, and both private versions of local_copy have the value one.

Thread 1 now wakes up and saves its version of local_copy and then terminates, giving Thread 2 a final chance to run. Thread 2 has no idea that Thread 1 ran and updated database.value while it was sleeping. It stores its version of local_copy into database.value, also setting it to one:

The two threads have interleaving access to a single shared object, overwriting each other’s results. Similar race conditions can arise when one thread frees memory or closes a file handle before the other thread is finished accessing it.

Why This Isn’t a Silly Example

The example above is contrived to make sure that the race condition happens every time you run your program. Because the operating system can swap out a thread at any time, it is possible to interrupt a statement like x = x + 1 after it has read the value of x but before it has written back the incremented value.

The details of how this happens are quite interesting, but not needed for the rest of this article, so feel free to skip over this hidden section.

The code above isn’t quite as out there as you might originally have thought. It was designed to force a race condition every time you run it, but that makes it much easier to solve than most race conditions.

There are two things to keep in mind when thinking about race conditions:

Even an operation like x += 1 takes the processor many steps. Each of these steps is a separate instruction to the processor.
The operating system can swap which thread is running at any time. A thread can be swapped out after any of these small instructions. This means that a thread can be put to sleep to let another thread run in the middle of a Python statement.

Let’s look at this in detail. The REPL below shows a function that takes a parameter and increments it:

>>>

>>> def inc(x):
...    x += 1
...
>>> import dis
>>> dis.dis(inc)
  2           0 LOAD_FAST                0 (x)
              2 LOAD_CONST               1 (1)
              4 INPLACE_ADD
              6 STORE_FAST               0 (x)
              8 LOAD_CONST               0 (None)
             10 RETURN_VALUE

The REPL example uses dis from the Python standard library to show the smaller steps that the processor does to implement your function. It does a LOAD_FAST of the data value x, it does a LOAD_CONST 1, and then it uses the INPLACE_ADD to add those values together.

We’re stopping here for a specific reason. This is the point in .update() above where time.sleep() forced the threads to switch. It is entirely possible that, every once in while, the operating system would switch threads at that exact point even without sleep(), but the call to sleep() makes it happen every time.

As you learned above, the operating system can swap threads at any time. You’ve walked down this listing to the statement marked 4. If the operating system swaps out this thread and runs a different thread that also modifies x, then when this thread resumes, it will overwrite x with an incorrect value.

Technically, this example won’t have a race condition because x is local to inc(). It does illustrate how a thread can be interrupted during a single Python operation, however. The same LOAD, MODIFY, STORE set of operations also happens on global and shared values. You can explore with the dis module and prove that yourself.

It’s rare to get a race condition like this to occur, but remember that an infrequent event taken over millions of iterations becomes likely to happen. The rarity of these race conditions makes them much, much harder to debug than regular bugs.

Now back to your regularly scheduled tutorial!

Now that you’ve seen a race condition in action, let’s find out how to solve them!

Basic Synchronization Using `Lock`

There are a number of ways to avoid or solve race conditions. You won’t look at all of them here, but there are a couple that are used frequently. Let’s start with Lock.

To solve your race condition above, you need to find a way to allow only one thread at a time into the read-modify-write section of your code. The most common way to do this is called Lock in Python. In some other languages this same idea is called a mutex. Mutex comes from MUTual EXclusion, which is exactly what a Lock does.

A Lock is an object that acts like a hall pass. Only one thread at a time can have the Lock. Any other thread that wants the Lock must wait until the owner of the Lock gives it up.

The basic functions to do this are .acquire() and .release(). A thread will call my_lock.acquire() to get the lock. If the lock is already held, the calling thread will wait until it is released. There’s an important point here. If one thread gets the lock but never gives it back, your program will be stuck. You’ll read more about this later.

Fortunately, Python’s Lock will also operate as a context manager, so you can use it in a with statement, and it gets released automatically when the with block exits for any reason.

Let’s look at the FakeDatabase with a Lock added to it. The calling function stays the same:

class FakeDatabase:
    def __init__(self):
        self.value = 0
        self._lock = threading.Lock()

    def locked_update(self, name):
        logging.info("Thread %s: starting update", name)
        logging.debug("Thread %s about to lock", name)
        with self._lock:
            logging.debug("Thread %s has lock", name)
            local_copy = self.value
            local_copy += 1
            time.sleep(0.1)
            self.value = local_copy
            logging.debug("Thread %s about to release lock", name)
        logging.debug("Thread %s after release", name)
        logging.info("Thread %s: finishing update", name)

Other than adding a bunch of debug logging so you can see the locking more clearly, the big change here is to add a member called ._lock, which is a threading.Lock() object. This ._lock is initialized in the unlocked state and locked and released by the with statement.

It’s worth noting here that the thread running this function will hold on to that Lock until it is completely finished updating the database. In this case, that means it will hold the Lock while it copies, updates, sleeps, and then writes the value back to the database.

If you run this version with logging set to warning level, you’ll see this:

$ ./fixrace.py
Testing locked update. Starting value is 0.
Thread 0: starting update
Thread 1: starting update
Thread 0: finishing update
Thread 1: finishing update
Testing locked update. Ending value is 2.

Look at that. Your program finally works!

You can turn on full logging by setting the level to DEBUG by adding this statement after you configure the logging output in __main__:

logging.getLogger().setLevel(logging.DEBUG)

Running this program with DEBUG logging turned on looks like this:

$ ./fixrace.py
Testing locked update. Starting value is 0.
Thread 0: starting update
Thread 0 about to lock
Thread 0 has lock
Thread 1: starting update
Thread 1 about to lock
Thread 0 about to release lock
Thread 0 after release
Thread 0: finishing update
Thread 1 has lock
Thread 1 about to release lock
Thread 1 after release
Thread 1: finishing update
Testing locked update. Ending value is 2.

In this output you can see Thread 0 acquires the lock and is still holding it when it goes to sleep. Thread 1 then starts and attempts to acquire the same lock. Because Thread 0 is still holding it, Thread 1 has to wait. This is the mutual exclusion that a Lock provides.

Many of the examples in the rest of this article will have WARNING and DEBUG level logging. We’ll generally only show the WARNING level output, as the DEBUG logs can be quite lengthy. Try out the programs with the logging turned up and see what they do.

Deadlock

Before you move on, you should look at a common problem when using Locks. As you saw, if the Lock has already been acquired, a second call to .acquire() will wait until the thread that is holding the Lock calls .release(). What do you think happens when you run this code:

import threading

l = threading.Lock()
print("before first acquire")
l.acquire()
print("before second acquire")
l.acquire()
print("acquired lock twice")

When the program calls l.acquire() the second time, it hangs waiting for the Lock to be released. In this example, you can fix the deadlock by removing the second call, but deadlocks usually happen from one of two subtle things:

An implementation bug where a Lock is not released properly
A design issue where a utility function needs to be called by functions that might or might not already have the Lock

The first situation happens sometimes, but using a Lock as a context manager greatly reduces how often. It is recommended to write code whenever possible to make use of context managers, as they help to avoid situations where an exception skips you over the .release() call.

The design issue can be a bit trickier in some languages. Thankfully, Python threading has a second object, called RLock, that is designed for just this situation. It allows a thread to .acquire() an RLock multiple times before it calls .release(). That thread is still required to call .release() the same number of times it called .acquire(), but it should be doing that anyway.

Lock and RLock are two of the basic tools used in threaded programming to prevent race conditions. There are a few other that work in different ways. Before you look at them, let’s shift to a slightly different problem domain.

Producer-Consumer Threading

The Producer-Consumer Problem is a standard computer science problem used to look at threading or process synchronization issues. You’re going to look at a variant of it to get some ideas of what primitives the Python threading module provides.

For this example, you’re going to imagine a program that needs to read messages from a network and write them to disk. The program does not request a message when it wants. It must be listening and accept messages as they come in. The messages will not come in at a regular pace, but will be coming in bursts. This part of the program is called the producer.

On the other side, once you have a message, you need to write it to a database. The database access is slow, but fast enough to keep up to the average pace of messages. It is not fast enough to keep up when a burst of messages comes in. This part is the consumer.

In between the producer and the consumer, you will create a Pipeline that will be the part that changes as you learn about different synchronization objects.

That’s the basic layout. Let’s look at a solution using Lock. It doesn’t work perfectly, but it uses tools you already know, so it’s a good place to start.

Producer-Consumer Using `Lock`

Since this is an article about Python threading, and since you just read about the Lock primitive, let’s try to solve this problem with two threads using a Lock or two.

The general design is that there is a producer thread that reads from the fake network and puts the message into a Pipeline:

SENTINEL = object()

def producer(pipeline):
    """Pretend we're getting a message from the network."""
    for index in range(10):
        message = random.randint(1, 101)
        logging.info("Producer got message: %s", message)
        pipeline.set_message(message, "Producer")

    # Send a sentinel message to tell consumer we're done
    pipeline.set_message(SENTINEL, "Producer")

To generate a fake message, the producer gets a random number between one and one hundred. It calls .set_message() on the pipeline to send it to the consumer.

The producer also uses a SENTINEL value to signal the consumer to stop after it has sent ten values. This is a little awkward, but don’t worry, you’ll see ways to get rid of this SENTINEL value after you work through this example.

On the other side of the pipeline is the consumer:

def consumer(pipeline):
    """Pretend we're saving a number in the database."""
    message = 0
    while message is not SENTINEL:
        message = pipeline.get_message("Consumer")
        if message is not SENTINEL:
            logging.info("Consumer storing message: %s", message)

The consumer reads a message from the pipeline and writes it to a fake database, which in this case is just printing it to the display. If it gets the SENTINEL value, it returns from the function, which will terminate the thread.

Before you look at the really interesting part, the Pipeline, here’s the __main__ section, which spawns these threads:

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")
    # logging.getLogger().setLevel(logging.DEBUG)

    pipeline = Pipeline()
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        executor.submit(producer, pipeline)
        executor.submit(consumer, pipeline)

This should look fairly familiar as it’s close to the __main__ code in the previous examples.

Remember that you can turn on DEBUG logging to see all of the logging messages by uncommenting this line:

# logging.getLogger().setLevel(logging.DEBUG)

It can be worthwhile to walk through the DEBUG logging messages to see exactly where each thread acquires and releases the locks.

Now let’s take a look at the Pipeline that passes messages from the producer to the consumer:

class Pipeline:
    """
    Class to allow a single element pipeline between producer and consumer.
    """
    def __init__(self):
        self.message = 0
        self.producer_lock = threading.Lock()
        self.consumer_lock = threading.Lock()
        self.consumer_lock.acquire()

    def get_message(self, name):
        logging.debug("%s:about to acquire getlock", name)
        self.consumer_lock.acquire()
        logging.debug("%s:have getlock", name)
        message = self.message
        logging.debug("%s:about to release setlock", name)
        self.producer_lock.release()
        logging.debug("%s:setlock released", name)
        return message

    def set_message(self, message, name):
        logging.debug("%s:about to acquire setlock", name)
        self.producer_lock.acquire()
        logging.debug("%s:have setlock", name)
        self.message = message
        logging.debug("%s:about to release getlock", name)
        self.consumer_lock.release()
        logging.debug("%s:getlock released", name)

Woah! That’s a lot of code. A pretty high percentage of that is just logging statements to make it easier to see what’s happening when you run it. Here’s the same code with all of the logging statements removed:

class Pipeline:
    """
    Class to allow a single element pipeline between producer and consumer.
    """
    def __init__(self):
        self.message = 0
        self.producer_lock = threading.Lock()
        self.consumer_lock = threading.Lock()
        self.consumer_lock.acquire()

    def get_message(self, name):
        self.coNsumer_lock.acquire()
        message = self.message
        self.producer_lock.release()
        return message

    def set_message(self, message, name):
        self.producer_lock.acquire()
        self.message = message
        self.consumer_lock.release()

That seems a bit more manageable. The Pipeline in this version of your code has three members:

.message stores the message to pass.
.producer_lock is a threading.Lock object that restricts access to the message by the producer thread.
.consumer_lock is also a threading.Lock that restricts access to the message by the consumer thread.

__init__() initializes these three members and then calls .acquire() on the .consumer_lock. This is the state you want to start in. The producer is allowed to add a new message, but the consumer needs to wait until a message is present.

.get_message() and .set_messages() are nearly opposites. .get_message() calls .acquire() on the consumer_lock. This is the call that will make the consumer wait until a message is ready.

Once the consumer has acquired the .consumer_lock, it copies out the value in .message and then calls .release() on the .producer_lock. Releasing this lock is what allows the producer to insert the next message into the pipeline.

Before you go on to .set_message(), there’s something subtle going on in .get_message() that’s pretty easy to miss. It might seem tempting to get rid of message and just have the function end with return self.message. See if you can figure out why you don’t want to do that before moving on.

Here’s the answer. As soon as the consumer calls .producer_lock.release(), it can be swapped out, and the producer can start running. That could happen before .release() returns! This means that there is a slight possibility that when the function returns self.message, that could actually be the next message generated, so you would lose the first message. This is another example of a race condition.

Moving on to .set_message(), you can see the opposite side of the transaction. The producer will call this with a message. It will acquire the .producer_lock, set the .message, and the call .release() on then consumer_lock, which will allow the consumer to read that value.

Let’s run the code that has logging set to WARNING and see what it looks like:

$ ./prodcom_lock.py
Producer got data 43
Producer got data 45
Consumer storing data: 43
Producer got data 86
Consumer storing data: 45
Producer got data 40
Consumer storing data: 86
Producer got data 62
Consumer storing data: 40
Producer got data 15
Consumer storing data: 62
Producer got data 16
Consumer storing data: 15
Producer got data 61
Consumer storing data: 16
Producer got data 73
Consumer storing data: 61
Producer got data 22
Consumer storing data: 73
Consumer storing data: 22

At first, you might find it odd that the producer gets two messages before the consumer even runs. If you look back at the producer and .set_message(), you will notice that the only place it will wait for a Lock is when it attempts to put the message into the pipeline. This is done after the producer gets the message and logs that it has it.

When the producer attempts to send this second message, it will call .set_message() the second time and it will block.

The operating system can swap threads at any time, but it generally lets each thread have a reasonable amount of time to run before swapping it out. That’s why the producer usually runs until it blocks in the second call to .set_message().

Once a thread is blocked, however, the operating system will always swap it out and find a different thread to run. In this case, the only other thread with anything to do is the consumer.

The consumer calls .get_message(), which reads the message and calls .release() on the .producer_lock, thus allowing the producer to run again the next time threads are swapped.

Notice that the first message was 43, and that is exactly what the consumer read, even though the producer had already generated the 45 message.

While it works for this limited test, it is not a great solution to the producer-consumer problem in general because it only allows a single value in the pipeline at a time. When the producer gets a burst of messages, it will have nowhere to put them.

Let’s move on to a better way to solve this problem, using a Queue.

Producer-Consumer Using `Queue`

If you want to be able to handle more than one value in the pipeline at a time, you’ll need a data structure for the pipeline that allows the number to grow and shrink as data backs up from the producer.

Python’s standard library has a queue module which, in turn, has a Queue class. Let’s change the Pipeline to use a Queue instead of just a variable protected by a Lock. You’ll also use a different way to stop the worker threads by using a different primitive from Python threading, an Event.

Let’s start with the Event. The threading.Event object allows one thread to signal an event while many other threads can be waiting for that event to happen. The key usage in this code is that the threads that are waiting for the event do not necessarily need to stop what they are doing, they can just check the status of the Event every once in a while.

The triggering of the event can be many things. In this example, the main thread will simply sleep for a while and then .set() it:

 1 if __name__ == "__main__":
 2     format = "%(asctime)s: %(message)s"
 3     logging.basicConfig(format=format, level=logging.INFO,
 4                         datefmt="%H:%M:%S")
 5     # logging.getLogger().setLevel(logging.DEBUG)
 6 
 7     pipeline = Pipeline()
 8     event = threading.Event()
 9     with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
10         executor.submit(producer, pipeline, event)
11         executor.submit(consumer, pipeline, event)
12 
13         time.sleep(0.1)
14         logging.info("Main: about to set event")
15         event.set()

The only changes here are the creation of the event object on line 6, passing the event as a parameter on lines 8 and 9, and the final section on lines 11 to 13, which sleep for a second, log a message, and then call .set() on the event.

The producer also did not have to change too much:

 1 def producer(pipeline, event):
 2     """Pretend we're getting a number from the network."""
 3     while not event.is_set():
 4         message = random.randint(1, 101)
 5         logging.info("Producer got message: %s", message)
 6         pipeline.set_message(message, "Producer")
 7 
 8     logging.info("Producer received EXIT event. Exiting")

It now will loop until it sees that the event was set on line 3. It also no longer puts the SENTINEL value into the pipeline.

consumer had to change a little more:

 1 def consumer(pipeline, event):
 2     """Pretend we're saving a number in the database."""
 3     while not event.is_set() or not pipeline.empty():
 4         message = pipeline.get_message("Consumer")
 5         logging.info(
 6             "Consumer storing message: %s  (queue size=%s)",
 7             message,
 8             pipeline.qsize(),
 9         )
10 
11     logging.info("Consumer received EXIT event. Exiting")

While you got to take out the code related to the SENTINEL value, you did have to do a slightly more complicated while condition. Not only does it loop until the event is set, but it also needs to keep looping until the pipeline has been emptied.

Making sure the queue is empty before the consumer finishes prevents another fun issue. If the consumer does exit while the pipeline has messages in it, there are two bad things that can happen. The first is that you lose those final messages, but the more serious one is that the producer can get caught on the producer_lock and never return.

This happens if the event gets triggered after the producer has checked the .is_set() condition but before it calls pipeline.set_message().

If that happens, it’s possible for the producer to wake up and exit with the .producer_lock still being held. The producer will then try to .acquire() the .producer_lock, but the consumer has exited and will never .release() it.

The rest of the consumer should look familiar.

The Pipeline has changed dramatically, however:

 1 class Pipeline(queue.Queue):
 2     def __init__(self):
 3         super().__init__(maxsize=10)
 4 
 5     def get_message(self, name):
 6         logging.debug("%s:about to get from queue", name)
 7         value = self.get()
 8         logging.debug("%s:got %d from queue", name, value)
 9         return value
10 
11     def set_message(self, value, name):
12         logging.debug("%s:about to add %d to queue", name, value)
13         self.put(value)
14         logging.debug("%s:added %d to queue", name, value)

You can see that Pipeline is a subclass of queue.Queue. Queue has an optional parameter when initializing to specify a maximum size of the queue.

If you give a positive number for maxsize, it will limit the queue to that number of elements, causing .put() to block until there are fewer than maxsize elements. If you don’t specify maxsize, then the queue will grow to the limits of your computer’s memory.

.get_message() and .set_message() got much smaller. They basically wrap .get() and .put() on the Queue. You might be wondering where all of the locking code that prevents the threads from causing race conditions went.

The core devs who wrote the standard library knew that a Queue is frequently used in multi-threading environments and incorporated all of that locking code inside the Queue itself. Queue is thread-safe.

Running this program looks like the following:

$ ./prodcom_queue.py
Producer got message: 32
Producer got message: 51
Producer got message: 25
Producer got message: 94
Producer got message: 29
Consumer storing message: 32 (queue size=3)
Producer got message: 96
Consumer storing message: 51 (queue size=3)
Producer got message: 6
Consumer storing message: 25 (queue size=3)
Producer got message: 31

[many lines deleted]

Producer got message: 80
Consumer storing message: 94 (queue size=6)
Producer got message: 33
Consumer storing message: 20 (queue size=6)
Producer got message: 48
Consumer storing message: 31 (queue size=6)
Producer got message: 52
Consumer storing message: 98 (queue size=6)
Main: about to set event
Producer got message: 13
Consumer storing message: 59 (queue size=6)
Producer received EXIT event. Exiting
Consumer storing message: 75 (queue size=6)
Consumer storing message: 97 (queue size=5)
Consumer storing message: 80 (queue size=4)
Consumer storing message: 33 (queue size=3)
Consumer storing message: 48 (queue size=2)
Consumer storing message: 52 (queue size=1)
Consumer storing message: 13 (queue size=0)
Consumer received EXIT event. Exiting

If you read through the output in my example, you can see some interesting things happening. Right at the top, you can see the producer got to create five messages and place four of them on the queue. It got swapped out by the operating system before it could place the fifth one.

The consumer then ran and pulled off the first message. It printed out that message as well as how deep the queue was at that point:

Consumer storing message: 32 (queue size=3)

This is how you know that the fifth message hasn’t made it into the pipeline yet. The queue is down to size three after a single message was removed. You also know that the queue can hold ten messages, so the producer thread didn’t get blocked by the queue. It was swapped out by the OS.

Note: Your output will be different. Your output will change from run to run. That’s the fun part of working with threads!

As the program starts to wrap up, can you see the main thread generating the event which causes the producer to exit immediately. The consumer still has a bunch of work do to, so it keeps running until it has cleaned out the pipeline.

Try playing with different queue sizes and calls to time.sleep() in the producer or the consumer to simulate longer network or disk access times respectively. Even slight changes to these elements of the program will make large differences in your results.

This is a much better solution to the producer-consumer problem, but you can simplify it even more. The Pipeline really isn’t needed for this problem. Once you take away the logging, it just becomes a queue.Queue.

Here’s what the final code looks like using queue.Queue directly:

import concurrent.futures
import logging
import queue
import random
import threading
import time

def producer(queue, event):
    """Pretend we're getting a number from the network."""
    while not event.is_set():
        message = random.randint(1, 101)
        logging.info("Producer got message: %s", message)
        queue.put(message)

    logging.info("Producer received event. Exiting")

def consumer(queue, event):
    """Pretend we're saving a number in the database."""
    while not event.is_set() or not pipeline.empty():
        message = queue.get()
        logging.info(
            "Consumer storing message: %s (size=%d)", message, queue.qsize()
        )

    logging.info("Consumer received event. Exiting")

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")

    pipeline = queue.Queue(maxsize=10)
    event = threading.Event()
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        executor.submit(producer, pipeline, event)
        executor.submit(consumer, pipeline, event)

        time.sleep(0.1)
        logging.info("Main: about to set event")
        event.set()

That’s easier to read and shows how using Python’s built-in primitives can simplify a complex problem.

Lock and Queue are handy classes to solve concurrency issues, but there are others provided by the standard library. Before you wrap up this tutorial, let’s do a quick survey of some of them.

Threading Objects

There are a few more primitives offered by the Python threading module. While you didn’t need these for the examples above, they can come in handy in different use cases, so it’s good to be familiar with them.

Semaphore

The first Python threading object to look at is threading.Semaphore. A Semaphore is a counter with a few special properties. The first one is that the counting is atomic. This means that there is a guarantee that the operating system will not swap out the thread in the middle of incrementing or decrementing the counter.

The internal counter is incremented when you call .release() and decremented when you call .acquire().

The next special property is that if a thread calls .acquire() when the counter is zero, that thread will block until a different thread calls .release() and increments the counter to one.

Semaphores are frequently used to protect a resource that has a limited capacity. An example would be if you have a pool of connections and want to limit the size of that pool to a specific number.

Timer

A threading.Timer is a way to schedule a function to be called after a certain amount of time has passed. You create a Timer by passing in a number of seconds to wait and a function to call:

t = threading.Timer(30.0, my_function)

You start the Timer by calling .start(). The function will be called on a new thread at some point after the specified time, but be aware that there is no promise that it will be called exactly at the time you want.

If you want to stop a Timer that you’ve already started, you can cancel it by calling .cancel(). Calling .cancel() after the Timer has triggered does nothing and does not produce an exception.

A Timer can be used to prompt a user for action after a specific amount of time. If the user does the action before the Timer expires, .cancel() can be called.

Barrier

A threading.Barrier can be used to keep a fixed number of threads in sync. When creating a Barrier, the caller must specify how many threads will be synchronizing on it. Each thread calls .wait() on the Barrier. They all will remain blocked until the specified number of threads are waiting, and then the are all released at the same time.

Remember that threads are scheduled by the operating system so, even though all of the threads are released simultaneously, they will be scheduled to run one at a time.

One use for a Barrier is to allow a pool of threads to initialize themselves. Having the threads wait on a Barrier after they are initialized will ensure that none of the threads start running before all of the threads are finished with their initialization.

Conclusion: Threading in Python

You’ve now seen much of what Python threading has to offer and some examples of how to build threaded programs and the problems they solve. You’ve also seen a few instances of the problems that arise when writing and debugging threaded programs.

If you’d like to explore other options for concurrency in Python, check out Speed Up Your Python Program With Concurrency.

If you’re interested in doing a deep dive on the asyncio module, go read Async IO in Python: A Complete Walkthrough.

Whatever you do, you now have the information and confidence you need to write programs using Python threading!

Special thanks to reader JL Diaz for helping to clean up the introduction.

Take the Quiz: Test your knowledge with our interactive “Python Threading” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Take the Quiz »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Intro to Object-Oriented Programming (OOP) in Python

2019-03-21T14:00:00+00:00

Object-oriented programming is one of the biggest and most important subjects in all of programming. This series will provide you with a basic conceptual understanding of Object-Oriented Programming so you can take your Python programming skills to the next level.

You’ll learn how to define custom types using classes, and how to instantiate those classes into python objects that can be used throughout your program.

Finally, you’ll learn how classes can subclass one another with brief introduction to inheritance, which allows you to write maintainable and less redundant Python code.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

13 Project Ideas for Intermediate Python Developers

2019-03-20T14:00:00+00:00

Learning the basics of Python is a wonderful experience. But the euphoria of just learning can be replaced by the hunger for hands-on projects. It’s normal to want to build projects, hence the need for project ideas.

The problem though is that some projects are either too simple for an intermediate Python developer or too hard. This article will suggest projects you can work on as an intermediate Python developer. These project ideas will provide the appropriate level of challenge for you.

In this article, you’ll learn:

The importance of building projects
The major platforms you can build projects for
Thirteen project ideas you can work on
Some tips for working on projects

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

The Importance of Building Projects

Working on projects is vital to pushing your career as a Python developer forward. They make you apply the skills and knowledge you’re acquiring.

Projects can help you:

Build confidence: You will believe more in your ability to create software regardless the level of complexity.
Explore other technologies: You will learn about other technologies needed in building a complete product such as databases, servers, and other languages.
Understand programming concepts better: You will learn to write code better and understand concepts such as design patterns and object-oriented programming.
Experience a complete software development life cycle: You will learn how to plan before writing code, manage the coding process and update software.

There is a lot to gain from building projects as a Python developer.

Choosing a Project Platform

You need to build your software to run on a platform so that people who lack certain technical knowledge can use your software. The web, desktop, and command-line are the three major platforms you’ll want to build your projects for.

Web

Web applications are applications that run on the web, they can be accessed on any device without being downloaded, provided there is access to the internet. If you want your projects to be accessible by everyone with internet access, it needs to be a web application.

A web application has a back end and front end. The back end is the part where the business logic is: your back-end code will manipulate and store data. The front end is the interface of the application: your front-end code will determine the look of a web application.

As an intermediate Python developer, your major focus will be the back-end code. However, the front-end code is important too, so you will need some knowledge of HTML, CSS, and maybe JavaScript to create a simple-looking interface. Just the basics will be enough.

Another option is to use Python for both the front end and back end. Thanks to the anvil library, which eliminates the need for HTML, CSS, and JavaScript, you can focus on Python code alone.

You can build web applications with Python through web frameworks such as django and flask. The list of frameworks for building web applications using Python is long. There are plenty to choose from, but django and flask remain the most popular web frameworks.

Desktop GUI

Every time you perform a task on your PC, be it a desktop or laptop, it is through an application. As an intermediate Python developer, you can make your own desktop applications.

You do not have to learn any front-end technology to create your own Graphical User Interface (GUI) applications, as you saw with web applications. You can build all the parts using Python.

There are frameworks for building your desktop applications. PySimpleGUI is one of them, and it’s pretty user-friendly for an intermediate Python developer.

An advanced GUI framework like PyQt5 is quite powerful, but it may have a steep learning curve.

The software you create for the Desktop GUI is able to work on any of the Windows, Linux, or Mac operating systems. All you have to do after creating the project is compile it to an executable for your operating system of choice.

Command-Line

Command-line applications are those applications that work in a console window. This is the command prompt on Windows and the Terminal on Linux and Mac.

You’d click to use a web or GUI application, but you’d type in commands for command-line applications. Users of command-line applications need to have some technical knowledge since they’ll need to use commands.

Command-line applications may not be as beautiful or easy to use as web or GUI applications, but that doesn’t make them less powerful than web or GUI applications.

You can improve the look of your command-line applications by applying colors to the text. There are libraries you can use for coloring, such as colorama and colored. You can spice things up and use some color.

You can use frameworks such as docopt, argparse, and click to build your applications.

Web Project Ideas

In this section, you’ll see project ideas for the web. These project ideas can be classified as utility and education tools.

Here are the project ideas:

Content Aggregator
Regex Query Tool
URL Shortener
Post-It Note
Quiz Application

Content Aggregator

Content is king. It exists everywhere on the web, from blogs to social media platforms. To keep up, you need to search for new information on the internet constantly. One way to stay updated is to check all the sites manually to see what the new posts are. But this is time consuming and quite tiring.

This is where the content aggregator comes in: A content aggregator fetches information from various places online and gathers all of that information in one place. Therefore, you don’t have to visit multiple sites to get the latest info: one website is enough.

With the content aggregator, all of the latest information can be gotten from one site that aggregates all the content. People can see the posts that interest them and can decide to find out more about them without traipsing all over the internet.

Examples of Content Aggregators

Here are some implementations of the Content Aggregator idea:

AllTop
Hvper

Technical Details

The main objective of this project idea is to aggregate content. First, you need to know what sites you’ll want the Content Aggregator to get content from. Then, you can use libraries such as requests for sending HTTP requests and BeautifulSoup to parse and scrape the necessary content from the sites.

Your application can implement its content aggregation as a background process. Libraries such as celery or apscheduler can help with that. You can try out apscheduler. It’s great for small background processes.

After scraping content from various sites, you’ll need to save it somewhere. So, you’ll use a database to save the scraped content.

Extra Challenge

For a tougher challenge, you can add more websites. This will help you learn how to study and extract information from websites.

You can also have users subscribe to certain sites that you aggregate. Then, at the end of the day, the content aggregator will send the articles for that day to the email address of the user.

Regex Query Tool

You and I deal with text daily. This article, which is also text, has a structure. This makes it easier for you to understand. Sometimes, you need to find certain information in text, and using the regular search tool in text editors can be ineffective.

This is where the Regex Query Tool comes in. A regex is a set of strings, so the regex query tool will check for the validity of the queries. When the regex matches patterns in the text, it tells the user and highlights the matched patterns. So, your Regex Query Tool will check the validity of the regex strings passed in by the user.

With the Regex Query Tool, users can quickly check the validity of their regex strings on the web. This makes it easier for them, instead of having to check the strings with a text editor.

Examples of Regex Query Tools

Here are some implementations of the Regex Query Tool idea:

Technical Details

The main objective of this type of project is to tell the user the validity of the inputted query strings. You can make it give a positive or negative response such as Query String Is Valid and Query String Is Invalid, implementing the positive response in green and the negative in red.

You don’t have to implement the query tool from scratch. You can use Python’s standard re library, which you can use to run the query strings on the inputted text. The re library will return None when the query string matches nothing, and it’ll return the matched strings when positive.

Some users may not understand regex fully, so you can make a page to explain how regex works. You can make documentation that is interesting enough to keep the users excited about learning and understanding regex.

Extra Challenge

Making a project that just returns the validity of the regex is fine. But you can also add a replacement feature. This means the application will check for the validity of the regex and also allow users to replace the matched strings with something else. So the tool is no longer a find tool but also a replace tool.

URL Shortener

URLs can be extremely long and not user-friendly. When people share links or even try to remember a URL, it’s difficult because most URLs are filled with more difficult characters and don’t form meaningful words.

This is where the URL Shortener comes in. A URL Shortener reduces the characters or letters in a URL, making them easier to read and remember. A URL like xyz.com/wwryb78&svnhkn%sghq?sfiyh can be shortened to xyz.com/piojwr.

With the URL Shortener, URLs become a joy to work with.

Examples of URL Shorteners

Here are some implementations of the URL Shortener idea:

Technical Details

The main objective of this project idea is to shorten URLs. The main task the application will accomplish is to shorten URLs and then redirect users to the original URL when the shortened URL is visited.

In the application, the users will input the original URL, and they will get the new, shortened URL as the result. To do this, you can use a combination of the random and string modules to generate the characters for the shortened URL.

Since users will visit the shortened URL days, months, or even years after, you’ll need to save the original and shortened URLs in a database. When a request comes in, the application checks if the URL exists and redirects to the original, or else it redirects to a 404 page.

Extra Challenge

Generating a shortened URL with random characters makes for a better URL than the long, random ones. But, you can make the result better for the users. You can add a feature to customize URLs, so the users can customize the generated URLs themselves.

Without a doubt, a custom xyz.com/mysite URL is better than a randomly generated xyz.com/piojwr URL.

Post-It Note

It’s human to have many thoughts and ideas in a day, but it’s also human to forget. One way to work around forgetting things is to jot them down before they disappear into thin air. While some of forgotten thoughts and ideas may be trivial, some can be quite powerful.

This is where a Post-It note comes in: A Post-It note is a small paper with low-tack adhesive at the back, making it attachable to surfaces such as documents, walls. Post-It notes make it easier to jot things down. The Post-It note project idea is something similar. It allows users to jot things down, making them accessible anywhere, since it’s a web application.

With the Post-It note, people can now jot things down anywhere, without the fear of forgetting things or misplacing the notes—which is a possibility with physical notes.

Examples of Post-It Notes

Here are some implementations of the Post-It Note idea:

Technical Details

The main objective of this project is to allow users to jot down thoughts. This means that each user will have their own notes, so the application will need to have an account creation feature. This ensures that the notes of each user remain private to them.

django comes with a user authentication system, so it may be a good choice. You can use other frameworks like bottle or flask, but you’ll have to implement the user authentication system on your own.

Since users may need to separate their notes under different sections, implementing a feature to allow users to categorize their notes will make the application more useful.

As an example, you may need to have notes on algorithms and data structures, so you’ll need to be able to separate the notes in those categories.

You’ll need to store the information and notes of each user, so a database becomes an essential part of this project. The MySQLdb module can be used if you want to use a MySQL database or the psycopg2 module for a PostgreSQL database. There are other modules you can use, but it all depends on the database you choose to use.

Extra Challenge

Since it’s human for users to forget their ideas, it’s also human for them to forget that they even made a note somewhere. You can add a feature to remind users of their notes. This feature will allow users to set a time for the reminder, so the application will send the reminder to the users when it’s time, by email.

Quiz Application

Knowledge is power. There are so many things in the world to learn, and quizzes help in testing the understanding of those concepts. You, as an intermediate Python developer, do not have to understand everything about the language. Taking tests is one way to find out things you don’t fully understand.

This is where the Quiz Application comes in. The Quiz Application will present questions to the users and expect the right answers to those questions. Think of the Quiz Application as a kind of questionnaire.

With the Quiz Application, special users you can call administrators will be allowed to create tests, so regular users can answer the questions and test their understanding of the topics in the quiz.

Examples of Quiz Applications

Here are some implementations of the Quiz Application idea:

Technical Details

The main objective of this project is to set quizzes and have people answer them. Therefore, users should be able to set questions, and other users should be able to answer those questions. The application will then display the final score and the right answers.

If you want users to be able to have a record of their scores, you may have to implement an account creation feature.

Users creating the tests should be able to create tests with the questions and answers by simply uploading a text file. The text file will have a format that you can decide, so the application can convert from a file to a quiz.

You’ll need to implement a database for this project. The database will store the questions, possible answers, correct answers, and the scores for each user.

Extra Challenge

For more of a challenge, you can allow users to add timers to the quizzes. This way, the creators of a quiz can determine how many seconds or minutes a user should spend on each question in the quiz.

It would be great to also have a quiz-sharing feature, so users can share interesting quizzes with their friends on other platforms.

GUI Project Ideas

In this section, you’ll see project ideas for Graphical User Interfaces. These project ideas can be classified as entertainment, finance, and utility tools.

Here’s are the project ideas:

MP3 Player
Alarm Tool
File Manager
Expense Tracker

MP3 Player

Audio is as important as text today if not more important. Since audio files are digital files, you’ll need a tool that can play them. Without a player, you’ll never be able to listen to the contents of an audio file.

This is where the MP3 Player comes in. The MP3 Player is a device for playing MP3s and other digital audio files. This MP3 Player GUI project idea attempts to emulate the physical MP3 Player. You can build software that allows you play an MP3 files on your desktop or laptop computer.

When you are done building the MP3 Player project, users can play their MP3 files and other digital audio files without having to purchase a physical MP3 Player. They’ll be able to play the MP3 files using their computers.

Examples of MP3 Players

Here are some implementations of the MP3 Player idea:

Technical Details

The main objective of this project is to allow users to play MP3 and digital audio files. To be engaging for users, the application has to have a simple but beautiful user interface.

You can have an interface for listing the available MP3 files. You can also give users the option to list other digital audio files that are not MP3.

The users will also expect the MP3 Player to have an interface that shows information on the file that is playing. Some of the information you can include are the name of the file, its length, the amount played, and the amount not played, in minutes and seconds.

Python has libraries that can play audio files, such as pygame, which allows you to work with multimedia files in few lines of code. You can also check out pymedia and simpleaudio.

These libraries can handle a lot of digital audio files. They can handle other file types, not just the MP3 files.

You can also implement a feature that allows users to create a playlist. To do this, you’ll need a database to store information on the created playlists. Python’s sqlite3 module allows you to use the SQLite database.

The SQLite database is a better option in this case, because it is file based and easier to set up than other SQL databases. While SQLite is file based, it is better for saving data than a regular file.

Extra Challenge

For a more exciting challenge, you can add a feature to allow the MP3 player to repeat currently playing files or even shuffle the list of files to be played.

It’s also possible to implement a feature that allows users to increase and decrease the playing speed of the audio file. Users will find this interesting, as they’ll be able to play files at a slower or faster pace than usual.

Alarm Tool

As they say, “Time and tide wait for no man.” But with a lot of things going on in our lives, it’s difficult to not lose track of time. To be able to keep track of time, a reminder is needed.

This is where the Alarm Tool comes in. An alarm is a device that gives an audio or visual signal about a certain condition. This Alarm Tool project idea is an attempt to build an alarm as software. The Alarm Tool gives an audio signal when a certain condition is met. The set time is the certain condition in this case.

With the Alarm Tool, users can set alarms to remind them of things at certain times of the day. The Alarm Tool project will work from the user’s laptop or desktop device, so they do not have to purchase a physical timer.

Examples of Alarm Tools

Here are some implementations of the Alarm Tool idea:

Technical Details

The main objective of this project is to activate audio signals at certain times of the day. So, timing and the audio signal to be played are the most important parts of the Alarm Tool.

The Alarm Tool should allow users to create, edit, and delete alarms. It should also have an interface that lists all the alarms, provided they have not being deleted by the user. So, it should list the active and inactive alarms.

Since it is an alarm, the application has to play tones at the set time. There are libraries for playing audio, like the pygame library.

In your code logic, the application has to keep checking for set alarm times. When the time is reached, it triggers a function to play the alarm tone.

Since the application will check for set alarm times, it means the application has to save the alarms in a database. The database should store things like the alarm date, time, and tone location.

Extra Challenge

As an extra feature, you can allow users to set recurring alarms. They’ll be able to set alarms that will ring at a certain time on certain days of the week, every week. As an example, an alarm can be set at 2:00 PM every Monday.

You can also add a snooze feature, so your users can snooze alarms instead of only dismissing them.

File Manager

The number of files on the personal computer of an average PC user is pretty high. If all of those files were placed in a single directory, it would be difficult to navigate and find files or directories. So, there is a need to arrange the files and manage them properly.

This is where a file manager comes in. A file manager allows users to manage files and directories through a user interface. While files can be managed through the command-line, not all users know how to do that.

With a file manager, users can arrange, access, and administer their files and directories properly without knowing how to use the command line. Some of the tasks a file manager allows users to perform includes copying, moving, and renaming files or directories.

Examples of File Manager Tools

Here are some implementations of the File Manager idea:

Technical Details

The main objective of the file manager project is to give users an interface to manage their files. Users want a file manager that has a file management tool that looks good and is easy to use.

You can use the PySimpleGUI library to create unique user interfaces with a powerful widget, without having to deal with a lot of complexity.

Your users should be able to perform simple tasks like creating new directories or empty text files. They should also be able to copy and move files or directories.

The sys, os, and shutil libraries will be quite useful for this project, as they can be used to execute actions on the files in the background, while the user clicks away.

The grid and list views are popular views today, so you can implement both in the application. This gives the user the option to choose which view option is suitable for them.

Extra Challenge

To make the file manager a bit more advanced, you can implement a search feature. So users can search for files and directories without having to find them manually.

You can also implement a sort feature. This will allow users to sort files according to different orders, such as time, alphabetical order, or size.

Expense Tracker

We have daily expenses, from groceries to clothing to bills. There are so many expenses that it’s normal to lose track of them and keep spending till we’re almost out of cash. A tracker can help people watch their expenses.

This is where the expense tracker comes in. An expense tracker is a software tool that allows users to keep track of their expenses. It can also analyze the expenses, depending on how advanced it is, but let’s keep it simple for now.

With the expense tracker, users can set a budget and track their spending so as to make better financial decisions.

Examples of Expense Trackers

Here are some implementations of the Expense Tracker idea:

Technical Details

The main objective of this project is to keep track of the user’s expenses. Some statistical analysis has to be done to be able to give users correct information on their expenses and help them spend better.

While tracking the expenses is the key thing, a good interface is also important. With PySimpleGUI, you can create a unique interface to improve the experience of the users.

PyData libraries such as pandas and matplotlib can be helpful for building the expense tracker.

The pandas library can be used for the data analysis, and the matplotlib library can be used for plotting graphs. Graphs will give the users a visual representation of their expenses, and a visual representation is usually easier to understand.

The application will receive data from the users. The data here is the inputted expenses. So, you’ll have to store the expenses in a database. The SQLite database is a good database choice for this project since it can be set up quickly. You can use sqlite3 module for the SQLite database.

Extra Challenge

For your users to benefit from this project, they’ll have to input their expenses regularly, which might slip their mind. It could be useful for you to implement a reminder feature. So the application will send a notification at certain times of the day or the week, reminding them to make use of the expense tracker.

Command-Line Project Ideas

In this section, you’ll see project ideas for the command-line. The project ideas discussed can be classified as utility tools.

Here’s are the project ideas:

Contact Book
File Connectivity Checker
Bulk File Rename Tool
Directory Tree Generator

Contact Book

We come across lots of people daily. We make acquaintances and friends. We get their contacts to keep in touch later on. Sadly, keeping the received contact details can be hard. One way to do this is to write the contact details down. But this is not secure as the physical book can easily be lost.

This is where the Contact Book project comes in. A contact book is a tool for saving a contact’s details, such as name, address, phone number, and email address. With this contact book project, you can build a software tool that people can use to save and find contact details.

With the contact book project idea, users can save their contacts with less risk of losing the saved contact details. It’ll always be accessible from their computer, through the command-line.

Examples of Contact Book Tools

There are Contact Book applications, but it’s rare to find command-line Contact Book products, as most are web, mobile, or GUI applications.

Here are some implementations of the Contact Book idea:

Technical Details

The main objective of this project is to save contact details. It’s important that you set up the commands users can use to enter the contact details. You can use the argparse or click command-line frameworks. They abstract a lot of complex stuff, so you only have to focus on the logic to be run when executing commands.

Some features you should implement include the commands to delete contacts, update contact information, and list saved contacts. You can also allow users to list contacts using different parameters, such as alphabetical order or contact creation date.

Since it’s a command-line project, the SQLite database will be fine for saving contacts. SQLite is user-friendly to set up. You may save the contact details in a file, but a file will not offer the benefits you can gain from using SQLite, such as performance and security.

To use the SQLite database in this project, the Python sqlite3 module will be very useful.

Extra Challenge

Remember how the database is stored on the user’s computer? What if something happens, like the user losing their files? It means they’ll also lose the contact details.

You can challenge yourself further and backup the database to an online storage platform. To do this, you can upload the database files to the cloud at certain intervals.

You can also add a command that allows users to backup the database themselves. This way, the user can still have access to the contacts if the database file is lost.

You should note that you may need some form of identification, so the contact book can tell which database file belongs to which user. Implementing a user authentication feature is one way to go about it.

Site Connectivity Checker

When you visit a URL, you expect to get the requested pages on your browser. But this is not always the case. Sometimes, sites can be down, so you won’t get the desired results. Instead, you’ll be presented with error messages. You can keep trying a site that is down, till it comes up and you get the information you need.

This is where the Site Connectivity Checker project comes in. The Site Connectivity Checker visits a URL and returns the status of the URL: it is either live or not. The Site Connectivity Checker will visit the URL at intervals, returning the results of each visit.

Instead of manually visiting a URL, a Site Connectivity Checker can do all of that manual work for you. This way, you’ll only get the results of the check without having to spend time on the browser, waiting for the site to go live.

Examples of Site Connectivity Checkers

Here are some implementations of the Site Connectivity Checker idea:

Technical Details

The main objective of this project is to check the status of sites. So, you need to write code for checking the status of a website.

You can choose to use either TCP or ICMP for your connections. The socket module is one to check out. You can also read Socket Programming in Python (Guide).

Through your chosen framework, be it the docopt, click, or argparse framework, you can add commands to allow users to add and remove sites from the list of sites to be checked.

The users should also be able to start the tool, stop it, and determine the intervals.

Since you’ll have to save the list of files to be checked, you can either save it in a file (just a list of sites) or use a SQLite database through the sqlite3 module.

Extra Challenge

The application can check for the connectivity status of sites and display the results to the command-line. But this will require the user to keep checking the command-line.

You can increase the challenge and implement a notification feature. The notification feature can be a sound played in the background to alert the user when a site’s status changes. You’ll need a database to store the previous status of a site. That’s the only way the tool can tell when the status changes.

Bulk File Rename Tool

Sometimes, you need to name all the files in a directory according to certain conventions. For example, you can name all the files in a directory with File0001.jpg, where the numbers increase based on the number of files in the directory. Doing this manually can be stressful and repetitive.

The Bulk File Rename Tool allows users to rename a large number of files, without having to manually rename files.

This saves users a lot of time. It spares them the trouble of having to do boring repetitive work and make mistakes. With the Bulk File Rename Tool, users can rename files in a couple of seconds without any mistakes.

Examples of Bulk File Rename Tools

Here are some implementations of the Bulk File Rename idea:

Ren
Rename

Technical Details

The main objective of this project idea is to rename files. So, the application needs to find a way to manipulate the target files. The os, sys, and shutil libraries will be useful for a large part of this project.

Your users will be able to rename all the files in the directory, using naming conventions. Therefore, they should be able to pass in the naming convention of choice. The regex module will help match the required naming patterns, if you understand how regex works.

A user may want to pass in a naming convention such as myfiles as part of the commands and expect that the tool renames all the files like myfilesXYZ, where XYZ is a number. They should also be able to choose the directory where the files to be renamed are.

Extra Challenge

The major challenge in this project is to rename all the files in a directory. But users may only need to name a certain number of files. To test your skills, you can implement a feature to allow users to choose the number of files to be renamed, instead of all the files.

Note that renaming only a certain number of files will require the tool to sort the files based on alphabetical order, time of file creation, or file size, depending on the user’s requirements.

Directory Tree Generator

Directories are like family trees: each directory has a particular relationship with other directories. No directories ever stays on its own, except an empty root directory.

When you’re working with files and directories, it is difficult to see the relationship between directories, as you can only see what exists in the current directory. You’re either using a file manager or working from the command-line.

With a Directory Tree Generator, you can see the relationship between files and directories like a tree or a map.

This makes it easier to understand the positioning of files and directories. A directory tree map is important when you’re explaining certain concepts, and a Directory Tree Generator makes it easier to get a visual representation of the file and directory relationships.

Examples of Directory Tree Generators

Here are some implementations of the Directory Tree Generator idea:

Technical Details

The main objective of the Directory Tree Generator is to visualize the relationships between files and directories. The os library can be very useful in listing the files and directories in a chosen directory.

Using a framework such as docopt or argparse helps abstract a lot of stuff, allowing you to focus on writing code for the application’s logic.

In the application’s logic, you can decide how you want to represent files or directories. Using different colors is a brilliant way to go about it. You can use the colored library to print the files and directories in different colors.

You can also decide how deep you’d like the Directory Tree Generator to go. For example, if a directory has children directories twelve levels deep, you may decide to go only as deep as the fifth level.

If you wish, you can also let the user decide how deep they want the Directory Tree Generator to go.

Extra Challenge

Since the results of the generated directory tree will be on the command-line, you can go one step further. You can have the generator create images of the directory tree, so it’ll basically turn the text into an image.

You’ll find the pillow library useful for doing this.

Tips for Working on Projects

Working on projects can be difficult. That’s one reason why motivation and interest in a project will make it a less daunting task.

If you’re interested in a project, you’ll be able to put in the time to research as well as find libraries and tools that will help you with the project.

Here are some tips:

Find a source of motivation
Break the project into subtasks
Do research on the subtasks
Build each subtasks, one step at a time
Reach out for help if you’re stuck
Put the subtasks together

Conclusion

In this article, you’ve seen a couple of Python project ideas you may find interesting.

The project ideas cover a range of platforms. You saw project ideas for the Web, GUI, and Command-line platforms.

You can choose to build a project for different platforms. Using the URL Shortener as an example, you may choose to build one for the Web, GUI, or the Command-line.

Since you’re an intermediate Python developer, these projects can be quite challenging but interesting.

The best way to make a project happen is to just get started. In no time, you’ll be finished and discover how much you’ve benefited from working on a project!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Decorators 101

2019-03-19T14:00:00+00:00

In this course on Python decorators, you’ll learn what they are and how to create and use them. Decorators provide a simple syntax for calling higher-order functions in Python. By definition, a decorator is a function that takes another function and extends the behavior of the latter function without explicitly modifying it.

Decorators Slack Q&A With the Real Python Team

We’re hosting a Python decorators Q&A session with Chris Bailey, Geir Arne Hjelle, and Dan Bader on the members-only Real Python Community Slack on Thursday, March 28 1pm-2pm EST (click here to see when that is in your local timezone).

Join us to talk about all things decorators and how they can make your code more Pythonic. See you on Slack :)

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Build a Python GUI Application With wxPython

2019-03-18T14:00:00+00:00

There are many graphical user interface (GUI) toolkits that you can use with the Python programming language. The big three are Tkinter, wxPython, and PyQt. Each of these toolkits will work with Windows, macOS, and Linux, with PyQt having the additional capability of working on mobile.

A graphical user interface is an application that has buttons, windows, and lots of other widgets that the user can use to interact with your application. A good example would be a web browser. It has buttons, tabs, and a main window where all the content loads.

In this article, you’ll learn how to build a graphical user interface with Python using the wxPython GUI toolkit.

Here are the topics covered:

Getting Started with wxPython
Definition of a GUI
Creating a Skeleton Application
Creating a Working Application

Let’s start learning!

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

Getting Started With wxPython

The wxPython GUI toolkit is a Python wrapper around a C++ library called wxWidgets. The initial release of wxPython was in 1998, so wxPython has been around quite a long time. wxPython’s primary difference from other toolkits, such as PyQt or Tkinter, is that wxPython uses the actual widgets on the native platform whenever possible. This makes wxPython applications look native to the operating system that it is running on.

PyQt and Tkinter both draw their widgets themselves, which is why they don’t always match the native widgets, although PyQt is very close.

This is not to say that wxPython does not support custom widgets. In fact, the wxPython toolkit has many custom widgets included with it, along with dozens upon dozens of core widgets. The wxPython downloads page has a section called Extra Files that is worth checking out.

Here, there is a download of the wxPython Demo package. This is a nice little application that demonstrates the vast majority of the widgets that are included with wxPython. The demo allows a developer to view the code in one tab and run it in a second tab. You can even edit and re-run the code in the demo to see how your changes affect the application.

Installing wxPython

You will be using the latest wxPython for this article, which is wxPython 4, also known as the Phoenix release. The wxPython 3 and wxPython 2 versions are built only for Python 2. When Robin Dunn, the primary maintainer of wxPython, created the wxPython 4 release, he deprecated a lot of aliases and cleaned up a lot of code to make wxPython more Pythonic and easier to maintain.

You will want to consult the following links if you are migrating from an older version of wxPython to wxPython 4 (Phoenix):

Classic vs Phoenix
wxPython Project Phoenix Migration Guide

The wxPython 4 package is compatible with both Python 2.7 and Python 3.

You can now use pip to install wxPython 4, which was not possible in the legacy versions of wxPython. You can do the following to install it on your machine:

$ pip install wxpython

Note: On Mac OS X you will need a compiler installed such as XCode for the install to complete successfully. Linux may also require you to install some dependencies before the pip installer will work correctly.

For example, I needed to install freeglut3-dev, libgstreamer-plugins-base0.10-dev, and libwebkitgtk-3.0-dev on Xubuntu to get it to install.

Fortunately, the error messages that pip displays are helpful in figuring out what is missing, and you can use the prerequisites section on the wxPython Github page to help you find the information you need if you want to install wxPython on Linux.

There are some Python wheels available for the most popular Linux versions that you can find in the Extras Linux section with both GTK2 and GTK3 versions. To install one of these wheels, you would use the following command:

$ pip install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04/ wxPython

Be sure you have modified the command above to match your version of Linux.

Definition of a GUI

As was mentioned in the introduction, a graphical user interface (GUI) is an interface that is drawn on the screen for the user to interact with.

User interfaces have some common components:

Main window
Menu
Toolbar
Buttons
Text Entry
Labels

All of these items are known generically as widgets. There are many other common widgets and many custom widgets that wxPython supports. A developer will take the widgets and arrange them logically on a window for the user to interact with.

Event Loops

A graphical user interface works by waiting for the user to do something. The something is called an event. Events happen when the user types something while your application is in focus or when the user uses their mouse to press a button or other widget.

Underneath the covers, the GUI toolkit is running an infinite loop that is called an event loop. The event loop just waits for events to occur and then acts on those events according to what the developer has coded the application to do. When the application doesn’t catch an event, it effectively ignores that it even happened.

When you are programming a graphical user interface, you will want to keep in mind that you will need to hook up each of the widgets to event handlers so that your application will do something.

There is a special consideration that you need to keep in mind when working with event loops: they can be blocked. When you block an event loop, the GUI will become unresponsive and appear to freeze to the user.

Any process that you launch in a GUI that will take longer than a quarter second should probably be launched as a separate thread or process. This will prevent your GUI from freezing and give the user a better user experience.

The wxPython framework has special thread-safe methods that you can use to communicate back to your application to let it know that the thread is finished or to give it an update.

Let’s create a skeleton application to demonstrate how events work.

Creating a Skeleton Application

An application skeleton in a GUI context is a user interface with widgets that don’t have any event handlers. These are useful for prototyping. You basically just create the GUI and present it to your stakeholders for sign-off before spending a lot of time on the backend logic.

Let’s start by creating a Hello World application with wxPython:

import wx

app = wx.App()
frame = wx.Frame(parent=None, title='Hello World')
frame.Show()
app.MainLoop()

Note: Mac users may get the following message: This program needs access to the screen. Please run with a Framework build of python, and only when you are logged in on the main display of your Mac. If you see this message and you are not running in a virtualenv, then you need to run your application with pythonw instead of python. If you are running wxPython from within a virtualenv, then see the wxPython wiki for the solution.

In this example, you have two parts: wx.App and the wx.Frame. The wx.App is wxPython’s application object and is required for running your GUI. The wx.App starts something called a .MainLoop(). This is the event loop that you learned about in the previous section.

The other piece of the puzzle is wx.Frame, which will create a window for the user to interact with. In this case, you told wxPython that the frame has no parent and that its title is Hello World. Here is what it looks like when you run the code:

Note: The application will look different when you run it on Mac or Windows.

By default, a wx.Frame will include minimize, maximize, and exit buttons along the top. You won’t normally create an application in this manner though. Most wxPython code will require you to subclass the wx.Frame and other widgets so that you can get the full power of the toolkit.

Let’s take a moment and rewrite your code as a class:

import wx

class MyFrame(wx.Frame):    
    def __init__(self):
        super().__init__(parent=None, title='Hello World')
        self.Show()

if __name__ == '__main__':
    app = wx.App()
    frame = MyFrame()
    app.MainLoop()

You can use this code as a template for your application. However, this application doesn’t do very much, so let’s take a moment to learn a little about some of the other widgets you could add.

Widgets

The wxPython toolkit has more than one hundred widgets to choose from. This allows you to create rich applications, but it can also be daunting trying to figure out which widget to use. This is why the wxPython Demo is helpful, as it has a search filter that you can use to help you find the widgets that might apply to your project.

Most GUI applications allow the user to enter some text and press a button. Let’s go ahead and add those widgets:

import wx

class MyFrame(wx.Frame):    
    def __init__(self):
        super().__init__(parent=None, title='Hello World')
        panel = wx.Panel(self)

        self.text_ctrl = wx.TextCtrl(panel, pos=(5, 5))
        my_btn = wx.Button(panel, label='Press Me', pos=(5, 55))

        self.Show()

if __name__ == '__main__':
    app = wx.App()
    frame = MyFrame()
    app.MainLoop()

When you run this code, your application should look like this:

The first widget you need to add is something called wx.Panel. This widget is not required, but recommended. On Windows, you are actually required to use a Panel so that the background color of the frame is the right shade of gray. Tab traversal is disabled without a Panel on Windows.

When you add the panel widget to a frame and the panel is the sole child of the frame, it will automatically expand to fill the frame with itself.

The next step is to add a wx.TextCtrl to the panel. The first argument for almost all widgets is which parent the widget should go onto. In this case, you want the text control and the button to be on top of the panel, so it is the parent you specify.

You also need to tell wxPython where to place the widget, which you can do by passing in a position via the pos parameter. In wxPython, the origin location is (0,0) which is the upper left corner of the parent. So for the text control, you tell wxPython that you want to position its top left corner 5 pixels from the left (x) and 5 pixels from the top (y).

Then you add your button to the panel and give it a label. To prevent the widgets from overlapping, you need to set the y-coordinate to 55 for the button’s position.

Absolute Positioning

When you provide exact coordinates for your widget’s position, the technique that you used is called absolute positioning. Most GUI toolkits provide this capability, but it’s not actually recommended.

As your application becomes more complex, it becomes difficult to keep track of all the widget locations and if you have to move the widgets around. Resetting all those positions becomes a nightmare.

Fortunately all modern GUI toolkits provide a solution for this, which is what you will learn about next.

Sizers (Dynamic Sizing)

The wxPython toolkit includes sizers, which are used for creating dynamic layouts. They manage the placement of your widgets for you and will adjust them when you resize the application window. Other GUI toolkits will refer to sizers as layouts, which is what PyQt does.

Here are the primary types of sizers that you will see used most often:

wx.BoxSizer
wx.GridSizer
wx.FlexGridSizer

Let’s add a wx.BoxSizer to your example and see if we can make it work a bit more nicely:

import wx

class MyFrame(wx.Frame):    
    def __init__(self):
        super().__init__(parent=None, title='Hello World')
        panel = wx.Panel(self)        
        my_sizer = wx.BoxSizer(wx.VERTICAL)        
        self.text_ctrl = wx.TextCtrl(panel)
        my_sizer.Add(self.text_ctrl, 0, wx.ALL | wx.EXPAND, 5)        
        my_btn = wx.Button(panel, label='Press Me')
        my_sizer.Add(my_btn, 0, wx.ALL | wx.CENTER, 5)        
        panel.SetSizer(my_sizer)        
        self.Show()

if __name__ == '__main__':
    app = wx.App()
    frame = MyFrame()
    app.MainLoop()

Here you create an instance of a wx.BoxSizer and pass it wx.VERTICAL, which is the orientation that widgets are added to the sizer.

In this case, the widgets will be added vertically, which means they will be added one at a time from top to bottom. You may also set a BoxSizer’s orientation to wx.HORIZONTAL. When you do that, the widgets would be added from left to right.

To add a widget to a sizer, you will use .Add(). It accepts up to five arguments:

window (the widget)
proportion
flag
border
userData

The window argument is the widget to be added while proportion sets how much space relative to other widgets in the sizer this particular widget should take. By default, it is zero, which tells wxPython to leave the widget at its default proportion.

The third argument is flag. You can actually pass in multiple flags if you wish as long as you separate them with a pipe character: |. The wxPython toolkit uses | to add flags using a series of bitwise ORs.

In this example, you add the text control with the wx.ALL and wx.EXPAND flags. The wx.ALL flag tells wxPython that you want to add a border on all sides of the widget while wx.EXPAND makes the widgets expand as much as they can within the sizer.

Finally, you have the border parameter, which tells wxPython how many pixels of border you want around the widget. The userData parameter is only used when you want to do something complex with your sizing of the widget and is actually quite rare to see in practice.

Adding the button to the sizer follows the exact same steps. However, to make things a bit more interesting, I went ahead and switched out the wx.EXPAND flag for wx.CENTER so that the button would be centered on-screen.

When you run this version of the code, your application should look like the following:

If you’d like to learn more about sizers, the wxPython documentation has a nice page on the topic.

Adding an Event

While your application looks more interesting visually, it still doesn’t really do anything. For example, if you press the button, nothing really happens.

Let’s give the button a job:

import wx

class MyFrame(wx.Frame):    
    def __init__(self):
        super().__init__(parent=None, title='Hello World')
        panel = wx.Panel(self)        
        my_sizer = wx.BoxSizer(wx.VERTICAL)        
        self.text_ctrl = wx.TextCtrl(panel)
        my_sizer.Add(self.text_ctrl, 0, wx.ALL | wx.EXPAND, 5)        
        my_btn = wx.Button(panel, label='Press Me')
        my_btn.Bind(wx.EVT_BUTTON, self.on_press)
        my_sizer.Add(my_btn, 0, wx.ALL | wx.CENTER, 5)        
        panel.SetSizer(my_sizer)        
        self.Show()

    def on_press(self, event):
        value = self.text_ctrl.GetValue()
        if not value:
            print("You didn't enter anything!")
        else:
            print(f'You typed: "{value}"')

if __name__ == '__main__':
    app = wx.App()
    frame = MyFrame()
    app.MainLoop()

The widgets in wxPython allow you to attach event bindings to them so that they can respond to certain types of events.

Note: The code block above uses f-strings. You can read all about them in Python 3’s f-Strings: An Improved String Formatting Syntax (Guide).

You want the button to do something when the user presses it. You can accomplish this by calling the button’s .Bind() method. .Bind() takes the event you want to bind to, the handler to call when the event happens, an optional source, and a couple of optional ids.

In this example, you bind your button object to the wx.EVT_BUTTON event and tell it to call on_press() when that event gets fired.

An event gets “fired” when the user does the event you have bound to. In this case, the event that you set up is the button press event, wx.EVT_BUTTON.

.on_press() accepts a second argument that you can call event. This is by convention. You could call it something else if you wanted to. However, the event parameter here refers to the fact that when this method is called, its second argument should be an event object of some sort.

Within .on_press(), you can get the text control’s contents by calling its GetValue() method. You then print a string to stdout depending on what the contents of the text control is.

Now that you have the basics out of the way, let’s learn how to create an application that does something useful!

Creating a Working Application

The first step when creating something new is to figure out what you want to create. In this case, I have taken the liberty of making that decision for you. You will learn how to create a MP3 tag editor! The next step when creating something new is to find out what packages can help you accomplish your task.

If you do a Google search for Python mp3 tagging, you will find you have several options:

mp3-tagger
eyeD3
mutagen

I tried out a couple of these and decided that eyeD3 had a nice API that you could use without getting bogged down with the MP3’s ID3 specification. You can install eyeD3 using pip, like this:

$ pip install eyed3

When installing this package on macOS, you may need to install libmagic using brew. Windows and Linux users shouldn’t have any issues installing eyeD3.

Designing the User Interface

When it comes to designing an interface, it’s always nice to just kind of sketch out how you think the user interface should look.

You will need to be able to do the following:

Open up one or more MP3 files
Display the current MP3 tags
Edit an MP3 tag

Most user interfaces use a menu or a button for opening files or folders. You can go with a File menu for this. Since you will probably want to see tags for multiple MP3 files, you will need to find a widget that can do this in a nice manner.

Something that is tabular with columns and rows would be ideal because then you can have labeled columns for the MP3 tags. The wxPython toolkit has a few widgets that would work for this, with the top two being the following:

wx.grid.Grid
wx.ListCtrl

You should use wx.ListCtrl in this case as the Grid widget is overkill, and frankly it is also quite a bit more complex. Finally, you need a button to use to edit a selected MP3’s tag.

Now that you know what you want, you can draw it up:

The illustration above gives us an idea of how the application should look. Now that you know what you want to do, it’s time to code!

Creating the User Interface

There are many different approaches when it comes to writing a new application. For example, do you need to follow the Model-View-Controller design pattern? How do you split up the classes? One class per file? There are many such questions, and as you get more experienced with GUI design, you’ll know how you want to answer them.

In your case, you really only need two classes:

A wx.Panel class
A wx.Frame class

You could argue for creating a controller type module as well, but for something like this, you really do not need it. A case could also be made for putting each class into its own module, but to keep it compact, you will create a single Python file for all of your code.

Let’s start with imports and the panel class:

import eyed3
import glob
import wx

class Mp3Panel(wx.Panel):    
    def __init__(self, parent):
        super().__init__(parent)
        main_sizer = wx.BoxSizer(wx.VERTICAL)
        self.row_obj_dict = {}

        self.list_ctrl = wx.ListCtrl(
            self, size=(-1, 100), 
            style=wx.LC_REPORT | wx.BORDER_SUNKEN
        )
        self.list_ctrl.InsertColumn(0, 'Artist', width=140)
        self.list_ctrl.InsertColumn(1, 'Album', width=140)
        self.list_ctrl.InsertColumn(2, 'Title', width=200)
        main_sizer.Add(self.list_ctrl, 0, wx.ALL | wx.EXPAND, 5)        
        edit_button = wx.Button(self, label='Edit')
        edit_button.Bind(wx.EVT_BUTTON, self.on_edit)
        main_sizer.Add(edit_button, 0, wx.ALL | wx.CENTER, 5)        
        self.SetSizer(main_sizer)

    def on_edit(self, event):
        print('in on_edit')

    def update_mp3_listing(self, folder_path):
        print(folder_path)

Here, you import the eyed3 package, Python’s glob package, and the wx package for your user interface. Next, you subclass wx.Panel and create your user interface. You need a dictionary for storing data about your MP3s, which you can name row_obj_dict.

Then you create a wx.ListCtrl and set it to report mode (wx.LC_REPORT) with a sunken border (wx.BORDER_SUNKEN). The list control can take on a few other forms depending on the style flag that you pass in, but the report flag is the most popular.

To make the ListCtrl have the correct headers, you will need to call .InsertColumn() for each column header. You then supply the index of the column, its label, and how wide in pixels the column should be.

The last step is to add your Edit button, an event handler, and a method. You can create the binding to the event and leave the method that it calls empty for now.

Now you should write the code for the frame:

class Mp3Frame(wx.Frame):    
    def __init__(self):
        super().__init__(parent=None,
                         title='Mp3 Tag Editor')
        self.panel = Mp3Panel(self)
        self.Show()

if __name__ == '__main__':
    app = wx.App(False)
    frame = Mp3Frame()
    app.MainLoop()

This class is much simpler than the first one in that all you need to do is set the title of the frame and instantiate the panel class, Mp3Panel. When you are all done, your user interface should look like this:

The user interface looks almost right, but you don’t have a File menu. This makes it impossible to add MP3s to the application and edit their tags!

Let’s fix that now.

Make a Functioning Application

The first step in making your application work is to update the application so that it has a File menu because then you can add MP3 files to your creation. Menus are almost always added to the wx.Frame class, so that is the class you need to modify.

Note: Some applications have moved away from having menus in their applications. One of the first to do so was Microsoft Office when they added the Ribbon Bar. The wxPython toolkit has a custom widget that you can use to create ribbons in wx.lib.agw.ribbon.

The other type of application that has dropped menus of late are web browsers, such as Google Chrome and Mozilla Firefox. They just use toolbars nowadays.

Let’s learn how to add a menu bar to our application:

class Mp3Frame(wx.Frame):

    def __init__(self):
        wx.Frame.__init__(self, parent=None, 
                          title='Mp3 Tag Editor')
        self.panel = Mp3Panel(self)
        self.create_menu()
        self.Show()

    def create_menu(self):
        menu_bar = wx.MenuBar()
        file_menu = wx.Menu()
        open_folder_menu_item = file_menu.Append(
            wx.ID_ANY, 'Open Folder', 
            'Open a folder with MP3s'
        )
        menu_bar.Append(file_menu, '&File')
        self.Bind(
            event=wx.EVT_MENU, 
            handler=self.on_open_folder,
            source=open_folder_menu_item,
        )
        self.SetMenuBar(menu_bar)

    def on_open_folder(self, event):
        title = "Choose a directory:"
        dlg = wx.DirDialog(self, title, 
                           style=wx.DD_DEFAULT_STYLE)
        if dlg.ShowModal() == wx.ID_OK:
            self.panel.update_mp3_listing(dlg.GetPath())
        dlg.Destroy()

Here, you add a call to .create_menu() within the class’s constructor. Then in .create_menu() itself, you will create a wx.MenuBar instance and a wx.Menu instance.

To add a menu item to a menu, you call the menu instance’s .Append() and pass it the following:

A unique identifier
The label for the new menu item
A help string

Next, you need to add the menu to the menubar, so you will need to call the menubar’s .Append(). It takes the menu instance and the label for menu. This label is a bit odd in that you called it &File instead of File. The ampersand tells wxPython to create a keyboard shortcut of Alt+F to open the File menu using just your keyboard.

Note: If you would like to add keyboard shortcuts to your application, then you will want to use an instance of wx.AcceleratorTable to create them. You can read more about Accerator Tables in the wxPython documentation.

To create an event binding, you will need to call self.Bind(), which binds the frame to wx.EVT_MENU. When you use self.Bind() for a menu event, you need to not only tell wxPython which handler to use, but also which source to bind the handler to.

Finally, you must call the frame’s .SetMenuBar() and pass it the menubar instance for it to be shown to the user.

Now that you have the menu added to your frame, let’s go over the menu item’s event handler, which is reproduced again below:

def on_open_folder(self, event):
    title = "Choose a directory:"
    dlg = wx.DirDialog(self, title, style=wx.DD_DEFAULT_STYLE)
    if dlg.ShowModal() == wx.ID_OK:
        self.panel.update_mp3_listing(dlg.GetPath())
    dlg.Destroy()

Since you want the user to choose a folder that contains MP3s, you will want to use wxPython’s wx.DirDialog. The wx.DirDialog allows the user to only open directories.

You can set the dialog’s title and various style flags. To show the dialog, you will need to call .ShowModal(). This will cause the dialog to show modally, which means that the user won’t be able to interact with your main application while the dialog is shown.

If the user presses the dialog’s OK button, you can get the user’s path choice via the dialog’s .GetPath(). You will want to pass that path to your panel class, which you can do here by calling the panel’s .update_mp3_listing().

Finally you need to close the dialog. To close a dialog, the recommended method is to call its .Destroy().

Dialogs do have a .Close() method, but that basically just hides the dialog, and it will not destroy itself when you close your application, which can lead to weird issues such as your application now shutting down properly. It’s simpler to call .Destroy() on the dialog to prevent this issue.

Now let’s update your Mp3Panel class. You can start by updating .update_mp3_listing():

def update_mp3_listing(self, folder_path):
    self.current_folder_path = folder_path
    self.list_ctrl.ClearAll()

    self.list_ctrl.InsertColumn(0, 'Artist', width=140)
    self.list_ctrl.InsertColumn(1, 'Album', width=140)
    self.list_ctrl.InsertColumn(2, 'Title', width=200)
    self.list_ctrl.InsertColumn(3, 'Year', width=200)

    mp3s = glob.glob(folder_path + '/*.mp3')
    mp3_objects = []
    index = 0
    for mp3 in mp3s:
        mp3_object = eyed3.load(mp3)
        self.list_ctrl.InsertItem(index, 
            mp3_object.tag.artist)
        self.list_ctrl.SetItem(index, 1, 
            mp3_object.tag.album)
        self.list_ctrl.SetItem(index, 2, 
            mp3_object.tag.title)
        mp3_objects.append(mp3_object)
        self.row_obj_dict[index] = mp3_object
        index += 1

Here you set the current directory to the specified folder and then you clear the list control. This keeps the list control fresh and only showing the MP3s that you are currently working on. That also means that you need to re-insert all the columns again.

Next, you’ll want to take the folder that was passed in and use Python’s glob module to search for MP3 files.

Then you can loop over the MP3s and turn them into eyed3 objects. You can do this by calling the .load() of eyed3. Assuming that the MP3s have the appropriate tags already, you can then add the artist, album, and title of the MP3 to the list control.

Interestingly, the method of adding a new row to a list control object is by calling .InsertItem() for the first column and SetItem() for all the subsequent columns.

The last step is to save off your MP3 object to your Python dictionary, row_obj_dict.

Now you need to update the .on_edit() event handler so that you can edit an MP3’s tags:

def on_edit(self, event):
    selection = self.list_ctrl.GetFocusedItem()
    if selection >= 0:
        mp3 = self.row_obj_dict[selection]
        dlg = EditDialog(mp3)
        dlg.ShowModal()
        self.update_mp3_listing(self.current_folder_path)
        dlg.Destroy()

The first thing you need to do is get the user’s selection by calling the list control’s .GetFocusedItem().

If the user has not selected anything in the list control, it will return -1. Assuming that the user did select something, you will want to extract the MP3 object from your dictionary and open a MP3 tag editor dialog. This will be a custom dialog that you will use to edit the artist, album, and title tags of the MP3 file.

As usual, show the dialog modally. When the dialog closes, the last two lines in .on_edit() will execute. These two lines will update the list control so it displays the current MP3 tag information that the user just edited and destroy the dialog.

Creating an Editing Dialog

The final piece of the puzzle is creating an MP3 tag editing dialog. For brevity, we will skip sketching out this interface as it is a series of rows that contains labels and text controls. The text controls should have the existing tag information pre-populated within them. You can create a label for the text controls by creating instances of wx.StaticText.

When you need to create a custom dialog, the wx.Dialog class is your friend. You can use that to design the editor:

class EditDialog(wx.Dialog):    
    def __init__(self, mp3):
        title = f'Editing "{mp3.tag.title}"'
        super().__init__(parent=None, title=title)        
        self.mp3 = mp3        
        self.main_sizer = wx.BoxSizer(wx.VERTICAL)        
        self.artist = wx.TextCtrl(
            self, value=self.mp3.tag.artist)
        self.add_widgets('Artist', self.artist)        
        self.album = wx.TextCtrl(
            self, value=self.mp3.tag.album)
        self.add_widgets('Album', self.album)        
        self.title = wx.TextCtrl(
            self, value=self.mp3.tag.title)
        self.add_widgets('Title', self.title)        
        btn_sizer = wx.BoxSizer()
        save_btn = wx.Button(self, label='Save')
        save_btn.Bind(wx.EVT_BUTTON, self.on_save)        
        btn_sizer.Add(save_btn, 0, wx.ALL, 5)
        btn_sizer.Add(wx.Button(
            self, id=wx.ID_CANCEL), 0, wx.ALL, 5)
        self.main_sizer.Add(btn_sizer, 0, wx.CENTER)        
        self.SetSizer(self.main_sizer)

Here you want to start off by sub-classing wx.Dialog and giving it a custom title based on the title of the MP3 that you are editing.

Next you can create the sizer you want to use and the widgets. To make things easier, you can create a helper method called .add_widgets() for adding the wx.StaticText widgets as rows with the text control instances. The only other widget here is the Save button.

Let’s write the add_widgets method next:

    def add_widgets(self, label_text, text_ctrl):
        row_sizer = wx.BoxSizer(wx.HORIZONTAL)
        label = wx.StaticText(self, label=label_text,
                              size=(50, -1))
        row_sizer.Add(label, 0, wx.ALL, 5)
        row_sizer.Add(text_ctrl, 1, wx.ALL | wx.EXPAND, 5)
        self.main_sizer.Add(row_sizer, 0, wx.EXPAND)

add_widgets() takes the label’s text and the text control instance. It then creates a horizontally oriented BoxSizer.

Next you will create an instance of wx.StaticText using the passed-in text for its label parameter. You will also set its size to be 50 pixels wide and the default height is set with a -1. Since you want the label before the text control, you will add the StaticText widget to your BoxSizer first and then add the text control .

Finally, you want to add the horizontal sizer to the top level vertical sizer. By nesting the sizers in each other, you can design complex applications.

Now you will need to create the on_save() event handler so that you can save your changes:

    def on_save(self, event):
        self.mp3.tag.artist = self.artist.GetValue()
        self.mp3.tag.album = self.album.GetValue()
        self.mp3.tag.title = self.title.GetValue()
        self.mp3.tag.save()
        self.Close()

Here you set the tags to the contents of the text controls and then call the eyed3 object’s .save(). Finally, you call the .Close() of the dialog. The reason you call .Close() here instead of .Destroy() is that you already call .Destroy() in the .on_edit() of your panel subclass.

Now your application is complete!

Conclusion

You learned a lot about wxPython in this article. You became familiar with the basics of creating GUI applications using wxPython.

You now know more about the following:

How to work with some of wxPython’s widgets
How events work in wxPython
How absolute positioning compares with sizers
How to create a skeleton application

Finally you learned how to create a working application, an MP3 tag editor. You can use what you learned in this article to continue to enhance this application or perhaps create an amazing application on your own.

The wxPython GUI toolkit is robust and full of interesting widgets that you can use to build cross-platform applications. You are limited by only your imagination.

Understanding the Python Mock Object Library

2019-03-13T14:00:00+00:00

When you’re writing robust code, tests are essential for verifying that your application logic is correct, reliable, and efficient. However, the value of your tests depends on how well they demonstrate these criteria. Obstacles such as complex logic and unpredictable dependencies make writing valuable tests difficult. The Python mock object library, unittest.mock, can help you overcome these obstacles.

By the end of this article, you’ll be able to:

Create Python mock objects using Mock
Assert you’re using objects as you intended
Inspect usage data stored on your Python mocks
Configure certain aspects of your Python mock objects
Substitute your mocks for real objects using patch()
Avoid common problems inherent in Python mocking

You’ll begin by seeing what mocking is and how it will improve your tests.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

What Is Mocking?

A mock object substitutes and imitates a real object within a testing environment. It is a versatile and powerful tool for improving the quality of your tests.

One reason to use Python mock objects is to control your code’s behavior during testing.

For example, if your code makes HTTP requests to external services, then your tests execute predictably only so far as the services are behaving as you expected. Sometimes, a temporary change in the behavior of these external services can cause intermittent failures within your test suite.

Because of this, it would be better for you to test your code in a controlled environment. Replacing the actual request with a mock object would allow you to simulate external service outages and successful responses in a predictable way.

Sometimes, it is difficult to test certain areas of your codebase. Such areas include except blocks and if statements that are hard to satisfy. Using Python mock objects can help you control the execution path of your code to reach these areas and improve your code coverage.

Another reason to use mock objects is to better understand how you’re using their real counterparts in your code. A Python mock object contains data about its usage that you can inspect such as:

If you called a method
How you called the method
How often you called the method

Understanding what a mock object does is the first step to learning how to use one.

Now, you’ll see how to use Python mock objects.

The Python Mock Library

The Python mock object library is unittest.mock. It provides an easy way to introduce mocks into your tests.

Note: The standard library includes unittest.mock in Python 3.3 and later. If you’re using an older version of Python, you’ll need to install the official backport of the library. To do so, install mock from PyPI:

$ pip install mock

unittest.mock provides a class called Mock which you will use to imitate real objects in your codebase. Mock offers incredible flexibility and insightful data. This, along with its subclasses, will meet most Python mocking needs that you will face in your tests.

The library also provides a function, called patch(), which replaces the real objects in your code with Mock instances. You can use patch() as either a decorator or a context manager, giving you control over the scope in which the object will be mocked. Once the designated scope exits, patch() will clean up your code by replacing the mocked objects with their original counterparts.

Finally, unittest.mock provides solutions for some of the issues inherent in mocking objects.

Now, you have a better understanding of what mocking is and the library you’ll be using to do it. Let’s dive in and explore what features and functionalities unittest.mock offers.

The `Mock` Object

unittest.mock offers a base class for mocking objects called Mock. The use cases for Mock are practically limitless because Mock is so flexible.

Begin by instantiating a new Mock instance:

>>>

>>> from unittest.mock import Mock
>>> mock = Mock()
>>> mock
<Mock id='4561344720'>

Now, you are able to substitute an object in your code with your new Mock. You can do this by passing it as an argument to a function or by redefining another object:

# Pass mock as an argument to do_something()
do_something(mock)

# Patch the json library
json = mock

When you substitute an object in your code, the Mock must look like the real object it is replacing. Otherwise, your code will not be able to use the Mock in place of the original object.

For example, if you are mocking the json library and your program calls dumps(), then your Python mock object must also contain dumps().

Next, you’ll see how Mock deals with this challenge.

Lazy Attributes and Methods

A Mock must simulate any object that it replaces. To achieve such flexibility, it creates its attributes when you access them:

>>>

>>> mock.some_attribute
<Mock name='mock.some_attribute' id='4394778696'>
>>> mock.do_something()
<Mock name='mock.do_something()' id='4394778920'>

Since Mock can create arbitrary attributes on the fly, it is suitable to replace any object.

Using an example from earlier, if you’re mocking the json library and you call dumps(), the Python mock object will create the method so that its interface can match the library’s interface:

>>>

>>> json = Mock()
>>> json.dumps()
<Mock name='mock.dumps()' id='4392249776'>

Notice two key characteristics of this mocked version of dumps():

Unlike the real dumps(), this mocked method requires no arguments. In fact, it will accept any arguments that you pass to it.
The return value of dumps() is also a Mock. The capability of Mock to recursively define other mocks allows for you to use mocks in complex situations:

>>>

>>> json = Mock()
>>> json.loads('{"k": "v"}').get('k')
<Mock name='mock.loads().get()' id='4379599424'>

Because the return value of each mocked method is also a Mock, you can use your mocks in a multitude of ways.

Mocks are flexible, but they’re also informative. Next, you’ll learn how you can use mocks to understand your code better.

Assertions and Inspection

Mock instances store data on how you used them. For instance, you can see if you called a method, how you called the method, and so on. There are two main ways to use this information.

First, you can assert that your program used an object as you expected:

>>>

>>> from unittest.mock import Mock

>>> # Create a mock object
... json = Mock()

>>> json.loads('{"key": "value"}')
<Mock name='mock.loads()' id='4550144184'>

>>> # You know that you called loads() so you can
>>> # make assertions to test that expectation
... json.loads.assert_called()
>>> json.loads.assert_called_once()
>>> json.loads.assert_called_with('{"key": "value"}')
>>> json.loads.assert_called_once_with('{"key": "value"}')

>>> json.loads('{"key": "value"}')
<Mock name='mock.loads()' id='4550144184'>

>>> # If an assertion fails, the mock will raise an AssertionError
... json.loads.assert_called_once()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 795, in assert_called_once
    raise AssertionError(msg)
AssertionError: Expected 'loads' to have been called once. Called 2 times.

>>> json.loads.assert_called_once_with('{"key": "value"}')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 824, in assert_called_once_with
    raise AssertionError(msg)
AssertionError: Expected 'loads' to be called once. Called 2 times.

>>> json.loads.assert_not_called()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 777, in assert_not_called
    raise AssertionError(msg)
AssertionError: Expected 'loads' to not have been called. Called 2 times.

.assert_called() ensures you called the mocked method while .assert_called_once() checks that you called the method exactly one time.

Both assertion functions have variants that let you inspect the arguments passed to the mocked method:

.assert_called_with(*args, **kwargs)
.assert_called_once_with(*args, **kwargs)

To pass these assertions, you must call the mocked method with the same arguments that you pass to the actual method:

>>>

>>> json = Mock()
>>> json.loads(s='{"key": "value"}')
>>> json.loads.assert_called_with('{"key": "value"}')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 814, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: loads('{"key": "value"}')
Actual call: loads(s='{"key": "value"}')
>>> json.loads.assert_called_with(s='{"key": "value"}')

json.loads.assert_called_with('{"key": "value"}') raised an AssertionError because it expected you to call loads() with a positional argument, but you actually called it with a keyword argument. json.loads.assert_called_with(s='{"key": "value"}') gets this assertion correct.

Second, you can view special attributes to understand how your application used an object:

>>>

>>> from unittest.mock import Mock

>>> # Create a mock object
... json = Mock()
>>> json.loads('{"key": "value"}')
<Mock name='mock.loads()' id='4391026640'>

>>> # Number of times you called loads():
... json.loads.call_count
1
>>> # The last loads() call:
... json.loads.call_args
call('{"key": "value"}')
>>> # List of loads() calls:
... json.loads.call_args_list
[call('{"key": "value"}')]
>>> # List of calls to json's methods (recursively):
... json.method_calls
[call.loads('{"key": "value"}')]

You can write tests using these attributes to make sure that your objects behave as you intended.

Now, you can create mocks and inspect their usage data. Next, you’ll see how to customize mocked methods so that they become more useful in your testing environment.

Managing a Mock’s Return Value

One reason to use mocks is to control your code’s behavior during tests. One way to do this is to specify a function’s return value. Let’s use an example to see how this works.

First, create a file called my_calendar.py. Add is_weekday(), a function that uses Python’s datetime library to determine whether or not today is a week day. Finally, write a test that asserts that the function works as expected:

from datetime import datetime

def is_weekday():
    today = datetime.today()
    # Python's datetime library treats Monday as 0 and Sunday as 6
    return (0 <= today.weekday() < 5)

# Test if today is a weekday
assert is_weekday()

Since you’re testing if today is a weekday, the result depends on the day you run your test:

$ python my_calendar.py

If this command produces no output, the assertion was successful. Unfortunately, if you run the command on a weekend, you’ll get an AssertionError:

$ python my_calendar.py
Traceback (most recent call last):
  File "test.py", line 9, in <module>
    assert is_weekday()
AssertionError

When writing tests, it is important to ensure that the results are predictable. You can use Mock to eliminate uncertainty from your code during testing. In this case, you can mock datetime and set the .return_value for .today() to a day that you choose:

import datetime
from unittest.mock import Mock

# Save a couple of test days
tuesday = datetime.datetime(year=2019, month=1, day=1)
saturday = datetime.datetime(year=2019, month=1, day=5)

# Mock datetime to control today's date
datetime = Mock()

def is_weekday():
    today = datetime.datetime.today()
    # Python's datetime library treats Monday as 0 and Sunday as 6
    return (0 <= today.weekday() < 5)

# Mock .today() to return Tuesday
datetime.datetime.today.return_value = tuesday
# Test Tuesday is a weekday
assert is_weekday()
# Mock .today() to return Saturday
datetime.datetime.today.return_value = saturday
# Test Saturday is not a weekday
assert not is_weekday()

In the example, .today() is a mocked method. You’ve removed the inconsistency by assigning a specific day to the mock’s .return_value. That way, when you call .today(), it returns the datetime that you specified.

In the first test, you ensure tuesday is a weekday. In the second test, you verify that saturday is not a weekday. Now, it doesn’t matter what day you run your tests on because you’ve mocked datetime and have control over the object’s behavior.

Further Reading: Though mocking datetime like this is a good practice example for using Mock, a fantastic library already exists for mocking datetime called freezegun.

When building your tests, you will likely come across cases where mocking a function’s return value will not be enough. This is because functions are often more complicated than a simple one-way flow of logic.

Sometimes, you’ll want to make functions return different values when you call them more than once or even raise exceptions. You can do this using .side_effect.

Managing a Mock’s Side Effects

You can control your code’s behavior by specifying a mocked function’s side effects. A .side_effect defines what happens when you call the mocked function.

To test how this works, add a new function to my_calendar.py:

import requests

def get_holidays():
    r = requests.get('http://localhost/api/holidays')
    if r.status_code == 200:
        return r.json()
    return None

get_holidays() makes a request to the localhost server for a set of holidays. If the server responds successfully, get_holidays() will return a dictionary. Otherwise, the method will return None.

You can test how get_holidays() will respond to a connection timeout by setting requests.get.side_effect.

For this example, you’ll only see the relevant code from my_calendar.py. You’ll build a test case using Python’s unittest library:

import unittest
from requests.exceptions import Timeout
from unittest.mock import Mock

# Mock requests to control its behavior
requests = Mock()

def get_holidays():
    r = requests.get('http://localhost/api/holidays')
    if r.status_code == 200:
        return r.json()
    return None

class TestCalendar(unittest.TestCase):
    def test_get_holidays_timeout(self):
        # Test a connection timeout
        requests.get.side_effect = Timeout
        with self.assertRaises(Timeout):
            get_holidays()

if __name__ == '__main__':
    unittest.main()

You use .assertRaises() to verify that get_holidays() raises an exception given the new side effect of get().

Run this test to see the result of your test:

$ python my_calendar.py
.
-------------------------------------------------------
Ran 1 test in 0.000s

OK

If you want to be a little more dynamic, you can set .side_effect to a function that Mock will invoke when you call your mocked method. The mock shares the arguments and return value of the .side_effect function:

import requests
import unittest
from unittest.mock import Mock

# Mock requests to control its behavior
requests = Mock()

def get_holidays():
    r = requests.get('http://localhost/api/holidays')
    if r.status_code == 200:
        return r.json()
    return None

class TestCalendar(unittest.TestCase):
    def log_request(self, url):
        # Log a fake request for test output purposes
        print(f'Making a request to {url}.')
        print('Request received!')

        # Create a new Mock to imitate a Response
        response_mock = Mock()
        response_mock.status_code = 200
        response_mock.json.return_value = {
            '12/25': 'Christmas',
            '7/4': 'Independence Day',
        }
        return response_mock

    def test_get_holidays_logging(self):
        # Test a successful, logged request
        requests.get.side_effect = self.log_request
        assert get_holidays()['12/25'] == 'Christmas'

if __name__ == '__main__':
    unittest.main()

First, you created .log_request(), which takes a URL, logs some output using print(), then returns a Mock response. Next, you set the .side_effect of get() to .log_request(), which you’ll use when you call get_holidays(). When you run your test, you’ll see that get() forwards its arguments to .log_request() then accepts the return value and returns it as well:

$ python my_calendar.py
Making a request to http://localhost/api/holidays.
Request received!
.
-------------------------------------------------------
Ran 1 test in 0.000s

OK

Great! The print() statements logged the correct values. Also, get_holidays() returned the holidays dictionary.

.side_effect can also be an iterable. The iterable must consist of return values, exceptions, or a mixture of both. The iterable will produce its next value every time you call your mocked method. For example, you can test that a retry after a Timeout returns a successful response:

import unittest
from requests.exceptions import Timeout
from unittest.mock import Mock

# Mock requests to control its behavior
requests = Mock()

def get_holidays():
    r = requests.get('http://localhost/api/holidays')
    if r.status_code == 200:
        return r.json()
    return None

class TestCalendar(unittest.TestCase):
    def test_get_holidays_retry(self):
        # Create a new Mock to imitate a Response
        response_mock = Mock()
        response_mock.status_code = 200
        response_mock.json.return_value = {
            '12/25': 'Christmas',
            '7/4': 'Independence Day',
        }
        # Set the side effect of .get()
        requests.get.side_effect = [Timeout, response_mock]
        # Test that the first request raises a Timeout
        with self.assertRaises(Timeout):
            get_holidays()
        # Now retry, expecting a successful response
        assert get_holidays()['12/25'] == 'Christmas'
        # Finally, assert .get() was called twice
        assert requests.get.call_count == 2

if __name__ == '__main__':
    unittest.main()

The first time you call get_holidays(), get() raises a Timeout. The second time, the method returns a valid holidays dictionary. These side effects match the order they appear in the list passed to .side_effect.

You can set .return_value and .side_effect on a Mock directly. However, because a Python mock object needs to be flexible in creating its attributes, there is a better way to configure these and other settings.

Configuring Your Mock

You can configure a Mock to set up some of the object’s behaviors. Some configurable members include .side_effect, .return_value, and .name. You configure a Mock when you create one or when you use .configure_mock().

You can configure a Mock by specifying certain attributes when you initialize an object:

>>>

>>> mock = Mock(side_effect=Exception)
>>> mock()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 939, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 995, in _mock_call
    raise effect
Exception

>>> mock = Mock(name='Real Python Mock')
>>> mock
<Mock name='Real Python Mock' id='4434041432'>

>>> mock = Mock(return_value=True)
>>> mock()
True

While .side_effect and .return_value can be set on the Mock instance, itself, other attributes like .name can only be set through .__init__() or .configure_mock(). If you try to set the .name of the Mock on the instance, you will get a different result:

>>>

>>> mock = Mock(name='Real Python Mock')
>>> mock.name
<Mock name='Real Python Mock.name' id='4434041544'>

>>> mock = Mock()
>>> mock.name = 'Real Python Mock'
>>> mock.name
'Real Python Mock'

.name is a common attribute for objects to use. So, Mock doesn’t let you set that value on the instance in the same way you can with .return_value or .side_effect. If you access mock.name you will create a .name attribute instead of configuring your mock.

You can configure an existing Mock using .configure_mock():

>>>

>>> mock = Mock()
>>> mock.configure_mock(return_value=True)
>>> mock()
True

By unpacking a dictionary into either .configure_mock() or Mock.__init__(), you can even configure your Python mock object’s attributes. Using Mock configurations, you could simplify a previous example:

# Verbose, old Mock
response_mock = Mock()
response_mock.json.return_value = {
    '12/25': 'Christmas',
    '7/4': 'Independence Day',
}

# Shiny, new .configure_mock()
holidays = {'12/25': 'Christmas', '7/4': 'Independence Day'}
response_mock = Mock(**{'json.return_value': holidays})

Now, you can create and configure Python mock objects. You can also use mocks to control the behavior of your application. So far, you’ve used mocks as arguments to functions or patching objects in the same module as your tests.

Next, you’ll learn how to substitute your mocks for real objects in other modules.

`patch()`

unittest.mock provides a powerful mechanism for mocking objects, called patch(), which looks up an object in a given module and replaces that object with a Mock.

Usually, you use patch() as a decorator or a context manager to provide a scope in which you will mock the target object.

`patch()` as a Decorator

If you want to mock an object for the duration of your entire test function, you can use patch() as a function decorator.

To see how this works, reorganize your my_calendar.py file by putting the logic and tests into separate files:

import requests
from datetime import datetime

def is_weekday():
    today = datetime.today()
    # Python's datetime library treats Monday as 0 and Sunday as 6
    return (0 <= today.weekday() < 5)

def get_holidays():
    r = requests.get('http://localhost/api/holidays')
    if r.status_code == 200:
        return r.json()
    return None

These functions are now in their own file, separate from their tests. Next, you’ll re-create your tests in a file called tests.py.

Up to this point, you’ve monkey patched objects in the file in which they exist. Monkey patching is the replacement of one object with another at runtime. Now, you’ll use patch() to replace your objects in my_calendar.py:

import unittest
from my_calendar import get_holidays
from requests.exceptions import Timeout
from unittest.mock import patch

class TestCalendar(unittest.TestCase):
    @patch('my_calendar.requests')
    def test_get_holidays_timeout(self, mock_requests):
            mock_requests.get.side_effect = Timeout
            with self.assertRaises(Timeout):
                get_holidays()
                mock_requests.get.assert_called_once()

if __name__ == '__main__':
    unittest.main()

Originally, you created a Mock and patched requests in the local scope. Now, you need to access the requests library in my_calendar.py from tests.py.

For this case, you used patch() as a decorator and passed the target object’s path. The target path was 'my_calendar.requests' which consists of the module name and the object.

You also defined a new parameter for the test function. patch() uses this parameter to pass the mocked object into your test. From there, you can modify the mock or make assertions as necessary.

You can execute this test module to ensure it’s working as expected:

$ python tests.py
.
-------------------------------------------------------
Ran 1 test in 0.001s

OK

Technical Detail: patch() returns an instance of MagicMock, which is a Mock subclass. MagicMock is useful because it implements most magic methods for you, such as .__len__(), .__str__(), and .__iter__(), with reasonable defaults.

Using patch() as a decorator worked well in this example. In some cases, it is more readable, more effective, or easier to use patch() as a context manager.

`patch()` as a Context Manager

Sometimes, you’ll want to use patch() as a context manager rather than a decorator. Some reasons why you might prefer a context manager include the following:

You only want to mock an object for a part of the test scope.
You are already using too many decorators or parameters, which hurts your test’s readability.

To use patch() as a context manager, you use Python’s with statement:

import unittest
from my_calendar import get_holidays
from requests.exceptions import Timeout
from unittest.mock import patch

class TestCalendar(unittest.TestCase):
    def test_get_holidays_timeout(self):
        with patch('my_calendar.requests') as mock_requests:
            mock_requests.get.side_effect = Timeout
            with self.assertRaises(Timeout):
                get_holidays()
                mock_requests.get.assert_called_once()

if __name__ == '__main__':
    unittest.main()

When the test exits the with statement, patch() replaces the mocked object with the original.

Until now, you’ve mocked complete objects, but sometimes you’ll only want to mock a part of an object.

Patching an Object’s Attributes

Let’s say you only want to mock one method of an object instead of the entire object. You can do so by using patch.object().

For example, .test_get_holidays_timeout() really only needs to mock requests.get() and set its .side_effect to Timeout:

import unittest
from my_calendar import requests, get_holidays
from unittest.mock import patch

class TestCalendar(unittest.TestCase):
    @patch.object(requests, 'get', side_effect=requests.exceptions.Timeout)
    def test_get_holidays_timeout(self, mock_requests):
            with self.assertRaises(requests.exceptions.Timeout):
                get_holidays()

if __name__ == '__main__':
    unittest.main()

In this example, you’ve mocked only get() rather than all of requests. Every other attribute remains the same.

object() takes the same configuration parameters that patch() does. But instead of passing the target’s path, you provide the target object, itself, as the first parameter. The second parameter is the attribute of the target object that you are trying to mock. You can also use object() as a context manager like patch().

Further Reading: Besides objects and attributes, you can also patch() dictionaries with patch.dict().

Learning how to use patch() is critical to mocking objects in other modules. However, sometimes it’s not obvious what the target object’s path is.

Where to Patch

Knowing where to tell patch() to look for the object you want mocked is important because if you choose the wrong target location, the result of patch() could be something you didn’t expect.

Let’s say you are mocking is_weekday() in my_calendar.py using patch():

>>>

>>> import my_calendar
>>> from unittest.mock import patch

>>> with patch('my_calendar.is_weekday'):
...     my_calendar.is_weekday()
...
<MagicMock name='is_weekday()' id='4336501256'>

First, you import my_calendar.py. Then you patch is_weekday(), replacing it with a Mock. Great! This is working as expected.

Now, let’s change this example slightly and import the function directly:

>>>

>>> from my_calendar import is_weekday
>>> from unittest.mock import patch

>>> with patch('my_calendar.is_weekday'):
...     is_weekday()
...
False

Note: Depending on what day you are reading this tutorial, your console output may read True or False. The important thing is that the output is not a Mock like before.

Notice that even though the target location you passed to patch() did not change, the result of calling is_weekday() is different. The difference is due to the change in how you imported the function.

from my_calendar import is_weekday binds the real function to the local scope. So, even though you patch() the function later, you ignore the mock because you already have a local reference to the un-mocked function.

A good rule of thumb is to patch() the object where it is looked up.

In the first example, mocking 'my_calendar.is_weekday()' works because you look up the function in the my_calendar module. In the second example, you have a local reference to is_weekday(). Since you use the function found in the local scope, you should mock the local function:

>>>

>>> from unittest.mock import patch
>>> from my_calendar import is_weekday

>>> with patch('__main__.is_weekday'):
...     is_weekday()
...
<MagicMock name='is_weekday()' id='4502362992'>

Now, you have a firm grasp on the power of patch(). You’ve seen how to patch() objects and attributes as well as where to patch them.

Next, you’ll see some common problems inherent in object mocking and the solutions that unittest.mock provides.

Common Mocking Problems

Mocking objects can introduce several problems into your tests. Some problems are inherent in mocking while others are specific to unittest.mock. Keep in mind that there are other issues with mocking that are not mentioned in this tutorial.

The ones covered here are similar to each other in that the problem they cause is fundamentally the same. In each case, the test assertions are irrelevant. Though the intention of each mock is valid, the mocks themselves are not.

Changes to Object Interfaces and Misspellings

Classes and function definitions change all the time. When the interface of an object changes, any tests relying on a Mock of that object may become irrelevant.

For example, you rename a method but forget that a test mocks that method and invokes .assert_not_called(). After the change, .assert_not_called() is still True. The assertion is not useful, though, because the method no longer exists.

Irrelevant tests may not sound critical, but if they are your only tests and you assume that they work properly, the situation could be disastrous for your application.

A problem specific to Mock is that a misspelling can break a test. Recall that a Mock creates its interface when you access its members. So, you will inadvertently create a new attribute if you misspell its name.

If you call .asert_called() instead of .assert_called(), your test will not raise an AssertionError. This is because you’ve created a new method on the Python mock object named .asert_called() instead of evaluating an actual assertion.

Technical Detail: Interestingly, assret is a special misspelling of assert. If you try to access an attribute that starts with assret (or assert), Mock will automatically raise an AttributeError.

These problems occur when you mock objects within your own codebase. A different problem arises when you mock objects interacting with external codebases.

Changes to External Dependencies

Imagine again that your code makes a request to an external API. In this case, the external dependency is the API which is susceptible to change without your consent.

On one hand, unit tests test isolated components of code. So, mocking the code that makes the request helps you to test your isolated components under controlled conditions. However, it also presents a potential problem.

If an external dependency changes its interface, your Python mock objects will become invalid. If this happens (and the interface change is a breaking one), your tests will pass because your mock objects have masked the change, but your production code will fail.

Unfortunately, this is not a problem that unittest.mock provides a solution for. You must exercise judgment when mocking external dependencies.

All three of these issues can cause test irrelevancy and potentially costly issues because they threaten the integrity of your mocks. unittest.mock gives you some tools for dealing with these problems.

Avoiding Common Problems Using Specifications

As mentioned before, if you change a class or function definition or you misspell a Python mock object’s attribute, you can cause problems with your tests.

These problems occur because Mock creates attributes and methods when you access them. The answer to these issues is to prevent Mock from creating attributes that don’t conform to the object you’re trying to mock.

When configuring a Mock, you can pass an object specification to the spec parameter. The spec parameter accepts a list of names or another object and defines the mock’s interface. If you attempt to access an attribute that does not belong to the specification, Mock will raise an AttributeError:

>>>

>>> from unittest.mock import Mock
>>> calendar = Mock(spec=['is_weekday', 'get_holidays'])

>>> calendar.is_weekday()
<Mock name='mock.is_weekday()' id='4569015856'>
>>> calendar.create_event()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 582, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'create_event'

Here, you’ve specified that calendar has methods called .is_weekday() and .get_holidays(). When you access .is_weekday(), it returns a Mock. When you access .create_event(), a method that does not match the specification, Mock raises an AttributeError.

Specifications work the same way if you configure the Mock with an object:

>>>

>>> import my_calendar
>>> from unittest.mock import Mock

>>> calendar = Mock(spec=my_calendar)
>>> calendar.is_weekday()
<Mock name='mock.is_weekday()' id='4569435216'>
>>> calendar.create_event()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 582, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'create_event'

.is_weekday() is available to calendar because you configured calendar to match the my_calendar module’s interface.

Furthermore, unittest.mock provides convenient methods of automatically specifying a Mock instance’s interface.

One way to implement automatic specifications is create_autospec:

>>>

>>> import my_calendar
>>> from unittest.mock import create_autospec

>>> calendar = create_autospec(my_calendar)
>>> calendar.is_weekday()
<MagicMock name='mock.is_weekday()' id='4579049424'>
>>> calendar.create_event()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 582, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'create_event'

Like before, calendar is a Mock instance whose interface matches my_calendar. If you’re using patch(), you can send an argument to the autospec parameter to achieve the same result:

>>>

>>> import my_calendar
>>> from unittest.mock import patch

>>> with patch('__main__.my_calendar', autospec=True) as calendar:
...     calendar.is_weekday()
...     calendar.create_event()
...
<MagicMock name='my_calendar.is_weekday()' id='4579094312'>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 582, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'create_event'

Conclusion

You’ve learned so much about mocking objects using unittest.mock!

Now, you’re able to:

Use Mock to imitate objects in your tests
Check usage data to understand how you use your objects
Customize your mock objects’ return values and side effects
patch() objects throughout your codebase
See and avoid problems with using Python mock objects

You have built a foundation of understanding that will help you build better tests. You can use mocks to gain insights into your code that you would not have been able to get otherwise.

I leave you with one final disclaimer. Beware of overusing mock objects!

It’s easy to take advantage of the power of Python mock objects and mock so much that you actually decrease the value of your tests.

If you’re interested in learning more about unittest.mock, I encourage you to read its excellent documentation.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Managing Multiple Python Versions With pyenv

2019-03-11T14:00:00+00:00

Have you ever wanted to contribute to a project that supports multiple versions of Python but aren’t sure how you would easily test all the versions? Are you ever curious about the latest and greatest versions of Python? Maybe you’d like to try out these new features, but you don’t want to worry about messing up your development environment. Luckily, managing multiple version of Python doesn’t have to be confusing if you use pyenv.

This article will provide you with a great overview of how to maximize your time spent working on projects and minimize the time spent in headaches trying to use the right version of Python.

In this article, you’ll learn how to:

Install multiple versions of Python
Install the latest development version of Python
Switch between the installed versions
Use virtual environments with pyenv
Activate different Python versions and virtual environments automatically

Why Use `pyenv`?

pyenv is a wonderful tool for managing multiple Python versions. Even if you already have Python installed on your system, it is worth having pyenv installed so that you can easily try out new language features or help contribute to a project that is on a different version of Python.

Why Not Use System Python?

“System Python” is the Python that comes installed on your operating system. If you’re on Mac or Linux, then by default, when you type python in your terminal, you get a nice Python REPL.

So, why not use it? One way to look at it is that this Python really belongs to the operating system. After all, it came installed with the operating system. That’s even reflected when you run which:

$ which python
/usr/bin/python

Here, python is available to all users as evidenced by its location /usr/bin/python. Chances are, this isn’t the version of Python you want either:

$ python -V
Pyhton 2.7.12

To install a package into your system Python, you have to run sudo pip install. That’s because you’re installing the Python package globally, which is a real problem if another user comes along and wants to install a slightly older version of the package.

Problems with multiple versions of the same package tend to creep up on you and bite you when you least expect it. One common way this problem presents itself is a popular and stable package suddenly misbehaving on your system. After hours of troubleshooting and Googling, you may find that you’ve installed the wrong version of a dependency, and it’s ruining your day.

Even if your Python version is installed in /usr/local/bin/python3, you’re still not safe. You will run into the same permissions and flexibility problems described above.

In addition, you don’t really have much control over what version of Python comes installed on your OS. If you want to use the latest features in Python, and you’re on Ubuntu for example, you might just be out of luck. The default versions might be too old, which means you’ll just have to wait for a new OS to come out.

Finally, some operating systems actually use the packaged Python for operation. Take yum for example, which makes heavy use of Python to do its job. If you install a new version of Python and aren’t careful to install it into your user space, you could seriously damage your ability to use your OS.

What About a Package Manager?

The next logical place to look is package managers. Programs such as apt, yum, brew, or port are typical next options. After all, this is how you install most packages to your system. Unfortunately, you’ll find some of the same problems using a package manager.

By default, package managers tend to install their packages into the global system space instead of the user space. Again, these system level packages pollute your development environment and make it hard to share a workspace with others.

Once again, you still don’t have control over what version of Python you can install. It’s true some repositories give you a greater selection, but by default, you’re looking at whatever version of Python your particular vendor is up to on any given day.

Even if you do install Python from a package manager, consider what would happen if you’re writing a package and want to support and test on Python 3.4 - 3.7.

What would happen on your system when you type python3? How would you switch quickly between the different versions? You can certainly do it, but it is tedious and prone to error. Nevermind the fact that if you want PyPy, Jython, or Miniconda, then you’re probably just out of luck with your package manager.

With these constraints in mind, let’s recap the criteria that would let you install and manage Python versions easily and flexibly:

Install Python in your user space
Install multiple versions of Python
Specify the exact Python version you want
Switch between the installed versions

pyenv lets you do all of these things and more.

Installing `pyenv`

Before you install pyenv itself, you’re going to need some OS-specific dependencies. These dependencies are mostly development utilities written in C and are required because pyenv installs Python by building from source. For a more detailed breakdown and explanation of the build dependencies, you can check out the official docs. In this tutorial, you’ll see the most common ways to install these dependencies.

Note: pyenv did not originally support Windows. However, there appears to be some basic support with the pyenv-win project that recently became active. If you use Windows, feel free to check it out.

Build Dependencies

pyenv builds Python from source, which means you’ll need build dependencies to actually use pyenv. The build dependencies vary by platform. If you are on Ubuntu/Debian and want to install the build dependencies, you could use the following:

$ sudo apt-get install -y make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev \
libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl

This uses Apt to install all the build dependencies. Let this run, and you’ll be ready to go for Debian systems.

If you use Fedora/CentOS/RHEL, you could use yum to install your build dependencies:

$ sudo yum install zlib-devel bzip2 bzip2-devel readline-devel sqlite \
sqlite-devel openssl-devel xz xz-devel libffi-devel

This command will install all the build dependencies for Python using yum.

macOS users can use the following command:

$ brew install readline xz

This command relies on Homebrew and installs the few dependencies for macOS users.

If you’re instead using openSUSE then you would run the following:

$ zypper in zlib-devel bzip2 libbz2-devel libffi-devel \
libopenssl-devel readline-devel sqlite3 sqlite3-devel xz xz-devel

Once again, this command installs all the Python build dependencies for your system.

Finally, for Alpine users, you can use this:

$ apk add libffi-dev ncurses-dev openssl-dev readline-dev \
tk-dev xz-dev zlib-dev

This command uses apk as the package manager and will install all build dependencies for Python on Alpine.

Using the pyenv-installer

After you’ve installed the build dependencies, you’re ready to install pyenv itself. I recommend using the pyenv-installer project:

$ curl https://pyenv.run | bash

This will install pyenv along with a few plugins that are useful:

pyenv: The actual pyenv application
pyenv-virtualenv: Plugin for pyenv and virtual environments
pyenv-update: Plugin for updating pyenv
pyenv-doctor: Plugin to verify that pyenv and build dependencies are installed
pyenv-which-ext: Plugin to automatically lookup system commands

Note: The above command is the same as downloading the pyenv-installer script and running it locally. So if you’d like to see exactly what you’re running, you can view the file yourself. Alternatively, if you really don’t want to run a script, you can checkout the manual installation instructions.

At the end of the run, you should see something like this:

WARNING: seems you still have not added 'pyenv' to the load path.

# Load pyenv automatically by adding
# the following to ~/.bashrc:

export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

The output will be based on your shell. But you should follow the instructions to add pyenv to your path and to initialize pyenv/pyenv-virtualenv auto completion. Once you’ve done this, you need to reload your shell:

$ exec "$SHELL" # Or just restart your terminal

That’s it. You now have pyenv and four useful plugins installed.

Using `pyenv` to Install Python

Now that you have pyenv installed, installing Python is the next step. You have many versions of Python to choose from. If you wanted to see all the available CPython 3.6 through 3.8, you can do this:

$ pyenv install --list | grep " 3\.[678]"
  3.6.0
  3.6-dev
  3.6.1
  3.6.2
  3.6.3
  3.6.4
  3.6.5
  3.6.6
  3.6.7
  3.6.8
  3.7.0
  3.7-dev
  3.7.1
  3.7.2
  3.8-dev

The above shows all the Python versions that pyenv knows about that match the regular expression. In this case, that is all available CPython versions 3.6 through 3.8. Likewise, if you wanted to see all the Jython versions, you could do this:

$ pyenv install --list | grep "jython"
  jython-dev
  jython-2.5.0
  jython-2.5-dev
  jython-2.5.1
  jython-2.5.2
  jython-2.5.3
  jython-2.5.4-rc1
  jython-2.7.0
  jython-2.7.1

Again, you can see all the Jython versions that pyenv has to offer. If you want to see all the versions, you can do the following:

$ pyenv install --list
...
# There are a lot

Once you find the version you want, you can install it with a single command:

$ pyenv install -v 3.7.2
/tmp/python-build.20190208022403.30568 ~
Downloading Python-3.7.2.tar.xz...
-> https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tar.xz
Installing Python-3.7.2...
/tmp/python-build.20190208022403.30568/Python-3.7.2 /tmp/python-build.20190208022403.30568 ~
[...]
Installing collected packages: setuptools, pip
Successfully installed pip-18.1 setuptools-40.6.2
Installed Python-3.7.2 to /home/realpython/.pyenv/versions/3.7.2

Having Problems? The pyenv documentation has great installation notes as well as a useful FAQ along with common build problems.

This will take a while because pyenv is building Python from source, but once it’s done, you’ll have Python 3.7.2 available on your local machine. If you don’t want to see all the output, just remove the -v flag. Even development versions of CPython can be installed:

$ pyenv install 3.8-dev

Pro Tip: If you’ve been using pyenv for a while and don’t see the version you’re looking for, you may need to run pyenv update to update the tool and make sure you have access to the latest versions.

For the rest of the tutorial, the examples assume you’ve installed 3.6.8 and 2.7.15, but you’re free to substitute these values for the Python versions you actually installed. Also note that the system Python version in the examples is 2.7.12.

Installation Location

As mentioned before, pyenv works by building Python from source. Each version that you have installed is located nicely in your pyenv root directory:

$ ls ~/.pyenv/versions/
2.7.15  3.6.8  3.8-dev

All of your versions will be located here. This is handy because removing these versions is trivial:

$ rm -rf ~/.pyenv/versions/2.7.15

Of course pyenv also provides a command to uninstall a particular Python version:

$ pyenv uninstall 2.7.15

Using Your New Python

Now that you’ve installed a couple of different Python versions, let’s see some basics on how to use them. First, check what versions of Python you have available:

$ pyenv versions
* system (set by /home/realpython/.pyenv/version)
  2.7.15
  3.6.8
  3.8-dev

The * indicates that the system Python version is active currently. You’ll also notice that this is set by a file in your root pyenv directory. This means that, by default, you are still using your system Python:

$ python -V
Python 2.7.12

If you try to confirm this using which, you’ll see this:

$ which python
/home/realpython/.pyenv/shims/python

This might be surprising, but this is how pyenv works. pyenv inserts itself into your PATH and from your OS’s perspective is the executable that is getting called. If you want to see the actual path, you can run the following:

$ pyenv which python
/usr/bin/python

If, for example, you wanted to use version 2.7.15, then you can use the global command:

$ pyenv global 2.7.15
$ python -V
Python 2.7.15

$ pyenv versions
  system
* 2.7.15 (set by /home/realpython/.pyenv/version)
  3.6.8
  3.8-dev

Pro Tip: A great way to get peace of mind that the version of Python you just installed is working properly is to run the built-in test suite:

$ pyenv global 3.8-dev
$ python -m test

This will kick off lots of internal Python tests that will verify your installation. You can just kick back and watch the tests pass.

If you ever want to go back to the system version of Python as the default, you can run this:

$ pyenv global system
$ python -V
Python 2.7.12

You can now switch between different versions of Python with ease. This is just the beginning. If you have many versions that you want to switch between, typing these commands consistently is tedious. This section goes over the basics, but a better workflow is described in working with multiple environments.

Exploring `pyenv` Commands

pyenv offers many commands. You can see a complete list of all available commands with this:

$ pyenv commands
activate
commands
completions
deactivate
...
virtualenvs
whence
which

This outputs all command names. Each command has a --help flag that will give you more detailed information. For example, if you wanted to see more information on the shims command you could run the following:

$ pyenv shims --help
Usage: pyenv shims [--short]

List existing pyenv shims

The help message describes what the command is used for and any options you can use in conjunction with the command. In the following sections, you’ll find a quick, high-level overview of the most used commands.

`install`

You’ve already seen the install command above. This command can be used to install a specific version of Python. For example, if you wanted to install 3.6.8 you would use this:

$ pyenv install 3.6.8

The output shows us pyenv downloading and installing Python. Some of the common flags you may want to use are the following:

Flag	Description
`-l/--list`	Lists all available Python versions for installation
`-g/--debug`	Builds a debug version of Python
`-v/--verbose`	Verbose mode: print compilation status to stdout

`versions`

The versions command displays all currently installed Python versions:

$ pyenv versions
* system (set by /home/realpython/.pyenv/version)
  2.7.15
  3.6.8
  3.8-dev

This output shows not only that 2.7.15, 3.6.8, 3.8-dev, and your system Python are installed, but also shows you that the system Python is active. If you only care about the current active version, you can use the following command:

$ pyenv version
system (set by /home/realpython/.pyenv/version)

This command is similar to versions but only shows you the current active Python version.

`which`

The which command is helpful for determining the full path to a system executable. Because pyenv works by using shims, this command allows you to see the full path to the executable pyenv is running. For example, if you wanted to see where pip is installed, you could run this:

$ pyenv which pip
/home/realpython/.pyenv/versions/3.6.8/bin/pip

The output displays the full system path for pip. This can be helpful when you’ve installed command-line applications.

`global`

The global command sets the global Python version. This can be overridden with other commands, but is useful for ensuring you use a particular Python version by default. If you wanted to use 3.6.8 by default, then you could run this:

$ pyenv global 3.6.8

This command sets the ~/.pyenv/version to 3.6.8. For more information, see the section on specifying your Python version.

`local`

The local command is often used to set an application-specific Python version. You could use it to set the version to 2.7.15:

$ pyenv local 2.7.15

This command creates a .python-version file in your current directory. If you have pyenv active in your environment, this file will automatically activate this version for you.

`shell`

The shell command is used to set a shell-specific Python version. For example, if you wanted to test out the 3.8-dev version of Python, you can do this:

$ pyenv shell 3.8-dev

This command activates the version specified by setting the PYENV_VERSION environment variable. This command overwrites any applications or global settings you may have. If you want to deactivate the version, you can use the --unset flag.

Specifying Your Python Version

One of the more confusing parts of pyenv is how exactly the python command gets resolved and what commands can be used to modify it. As mentioned in the commands, there are 3 ways to modify which version of python you’re using. So how do all these commands interact with one another? The resolution order looks a little something like this:

This pyramid is meant to be read from top to bottom. The first of these options that pyenv can find is the option it will use. Let’s see a quick example:

$ pyenv versions
* system (set by /home/realpython/.pyenv/version)
  2.7.15
  3.6.8
  3.8-dev

Here, your system Python is being used as denoted by the *. To exercise the next most global setting, you use global:

$ pyenv global 3.6.8
$ pyenv versions
  system
  2.7.15
* 3.6.8 (set by /home/realpython/.pyenv/version)
  3.8-dev

You can see that now pyenv wants to use 3.6.8 as our Python version. It even indicates the location of the file it found. That file does indeed exist, and you can list its contents:

$ cat ~/.pyenv/version
3.6.8

Now, let’s create a .python-version file with local:

$ pyenv local 2.7.15
$ pyenv versions
  system
* 2.7.15 (set by /home/realpython/.python-version)
  3.6.8
  3.8-dev
$ ls -a
.  ..  .python-version
$ cat .python-version
2.7.15

Here again, pyenv indicates how it would resolve our python command. This time it comes from ~/.python-version. Note that the searching for .python-version is recursive:

$ mkdir subdirectory
$ cd subdirectory
$ ls -la # Notice no .python-version file
. ..
$ pyenv versions
  system
* 2.7.15 (set by /home/realpython/.python-version)
  3.6.8
  3.8-dev

Even though there isn’t a .python-version in subdirectory, the version is still set to 2.7.15 because .python-version exists in a parent directory.

Finally, you can set the Python version with shell:

$ pyenv shell 3.8-dev
$ pyenv versions
  system
  2.7.15
  3.6.8
* 3.8-dev (set by PYENV_VERSION environment variable)

All this did is set the $PYENV_VERSION environment variable:

$ echo $PYENV_VERSION
3.8-dev

If you’re feeling overwhelmed by the options, the section on working with multiple environments goes over an opinionated process for managing these files, mostly using local.

Virtual Environments and `pyenv`

Virtual environments are a big part of managing Python installations and applications. If you haven’t heard of virtual environments before, you can check out Python Virtual Environments: A Primer.

Virtual environments and pyenv are a match made in heaven. pyenv has a wonderful plugin called pyenv-virtualenv that makes working with multiple Python version and multiple virtual environments a breeze. If you’re wondering what the difference is between pyenv, pyenv-virtualenv, and tools like virtualenv or venv, then don’t worry. You’re not alone.

Here’s what you need to know:

pyenv manages multiple versions of Python itself.
virtualenv/venv manages virtual environments for a specific Python version.
pyenv-virtualenv manages virtual environments for across varying versions of Python.

If you’re a die-hard virtualenv or venv user, don’t worry: pyenv plays nicely with either. In fact, you can keep the same workflow you’ve had if you’d prefer, though I think pyenv-virtualenv makes for a nicer experience when you’re switching between multiple environments that require different Python versions.

The good news is that since you used the pyenv-installer script to install pyenv, you already have pyenv-virtualenv installed and ready to go.

Creating Virtual Environments

Creating a virtual environment is a single command:

$ pyenv virtualenv <python_version> <environment_name>

Technically, the <python_version> is optional, but you should consider always specifying it so that you’re certain of what Python version you’re using.

The <environment_name> is just a name for you to help keep your environments separate. A good practice is to name your environments the same name as your project. For example, if you were working on myproject and wanted to develop against Python 3.6.8, you would run this:

$ pyenv virtualenv 3.6.8 myproject

The output includes messages that show a couple of extra Python packages getting installed, namely wheel, pip, and setuptools. This is strictly for convenience and just sets up a more full featured environment for each of your virtual environments.

Activating Your Versions

Now that you’ve created your virtual environment, using it is the next step. Normally, you should activate your environments by running the following:

$ pyenv local myproject

You’ve seen the pyenv local command before, but this time, instead of specifying a Python version, you specify an environment. This creates a .python-version file in your current working directory and because you ran eval "$(pyenv virtualenv-init -)" in your environment, the environment will automatically be activated.

You can verify this by running the following:

$ pyenv which python
/home/realpython/.pyenv/versions/myproject/bin/python

You can see a new version has been created called myproject and the python executable is pointing to that version. If you look at any executable this environment provides, you will see the same thing. Take, for example, pip:

$ pyenv which pip
/home/realpython/.pyenv/versions/myproject/bin/pip

If you did not configure eval "$(pyenv virtualenv-init -)" to run in your shell, you can manually activate/deactivate your Python versions with this:

$ pyenv activate <environment_name>
$ pyenv deactivate

The above is what pyenv-virtualenv is doing when it enters or exits a directory with a .python-version file in it.

Working With Multiple Environments

Putting everything you’ve learned together, you can work effectively with multiple environments. Let’s assume you have the following versions installed:

$ pyenv versions
* system (set by /home/realpython/.pyenv/version)
  2.7.15
  3.6.8
  3.8-dev

Now you want to work on two different, aptly named, projects:

project1 supports Python 2.7 and 3.6.
project2 supports Python 3.6 and experiments with 3.8-dev.

You can see that, by default, you are using the system Python, which is indicated by the * in the pyenv versions output. First, create a virtual environment for the first project:

$ cd project1/
$ pyenv which python
/usr/bin/python
$ pyenv virtualenv 3.6.8 project1
...
$ pyenv local project1
$ python -V
/home/realpython/.pyenv/versions/project1/bin/python

Finally, notice that when you cd out of the directory, you default back to the system Python:

$ cd $HOME
$ pyenv which python
/usr/bin/python

You can follow the above steps and create a virtual environment for project2:

$ cd project2/
$ pyenv which python
/usr/bin/python
$ pyenv virtualenv 3.8-dev project2
...
$ pyenv local 3.8-dev
$ pyenv which python
/home/realpython/.pyenv/versions/3.8-dev/bin/python

These are one time steps for your projects. Now, as you cd between the projects, your environments will automatically activate:

$ cd project2/
$ python -V
Python 3.8.0a0
$ cd ../project1
$ python -V
Python 3.6.8

No more remembering to activate environments: you can switch between all your projects, and pyenv will take care of automatically activating the correct Python versions and the correct virtual environments.

Activating Multiple Versions Simultaneously

As described in the example above, project2 uses experimental features in 3.8. Suppose you wanted to ensure that your code still works on Python 3.6. If you try running python3.6, you’ll get this:

$ cd project2/
$ python3.6 -V
pyenv: python3.6: command not found

The `python3.6' command exists in these Python versions:
  3.6.8
  3.6.8/envs/project1
  project1

pyenv informs you that, while Python 3.6 is not available in the current active environment, it is available in other environments. pyenv gives you a way to activate multiple environments at once using a familiar command:

$ pyenv local project2 3.6.8

This indicates to pyenv that you would like to use the virtual environment project2 as the first option. So if a command, for example python, can be resolved in both environments, it will pick project2 before 3.6.8. Let’s see what happens if you run this:

$ python3.6 -V
Python 3.6.8

Here, pyenv attempts to find the python3.6 command, and because it finds it in an environment that is active, it allows the command to execute. This is extremely useful for tools like tox that require multiple versions of Python to be available on your PATH in order to execute.

Pro Tip: If you’re using tox and pyenv, you should checkout the tox-pyenv package.

Suppose that in the above example, you’ve found a compatibility problem with your library and would like to do some local testing. The testing requires that you install all the dependencies. You should follow the steps to create a new environment:

$ pyenv virtualenv 3.6.8 project2-tmp
$ pyenv local project2-tmp

Once you’re satisfied with your local testing, you can easily switch back to your default environment:

$ pyenv local project2 3.6.8

Conclusion

You can now more easily contribute to a project that wants to support multiple environments. You can also more easily test out the latest and greatest Python versions without having to worry about messing up your development machine, all with a wonderful tool: pyenv.

You’ve seen how pyenv can help you:

Install multiple versions of Python
Switch between the installed versions
Use virtual environments with pyenv
Activate different Python versions and virtual environments automatically

If you still have questions, feel free to reach out either in the comments section or on Twitter. Additionally, the pyenv documentation is a great resource.

Bonus: Displaying Your Environment Name in Your Command Prompt

If you’re like me and constantly switching between various virtual environments and Python versions, it’s easy to get confused about which version is currently active. I use oh-my-zsh and the agnoster theme, which by default makes my prompt look like this:

At a glance, I don’t know which Python version is active. To figure it out, I would have to run python -V or pyenv version. To help reduce my time spent on figuring out my active Python environment, I add the pyenv virtual environment I’m using to my prompt:

My Python version in this case is project1-venv and is displayed immediately at the beginning of the prompt. This allows me to quickly see what version of Python I’m using right away. If you’d like to use this too, you can use my agnoster-pyenv theme.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Writing Comments in Python

2019-03-08T00:02:05+00:00

Learn how to write Python comments that are clean, concise, and useful. Quickly get up to speed on what the best practices are, which types of comments it’s best to avoid, and how you can practice writing cleaner comments.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Making HTTP Requests With Python

2019-03-06T15:17:19+00:00

The requests library is the de facto standard for making HTTP requests in Python. It abstracts the complexities of making requests behind a beautiful, simple API so that you can focus on interacting with services and consuming data in your application.

Throughout this course, you’ll see some of the most useful features that requests has to offer as well as how to customize and optimize those features for different situations you may come across. You’ll also learn how to use requests in an efficient way as well as how to prevent requests to external services from slowing down your application.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Using PyInstaller to Easily Distribute Python Applications

2019-03-06T14:00:00+00:00

Are you jealous of Go developers building an executable and easily shipping it to users? Wouldn’t it be great if your users could run your application without installing anything? That is the dream, and PyInstaller is one way to get there in the Python ecosystem.

There are countless tutorials on how to set up virtual environments, manage dependencies, and publish to PyPI, which is useful when you’re creating Python libraries. There is much less information for developers building Python applications. This tutorial is for developers who want to distribute applications to users who may or may not be Python developers.

In this tutorial, you’ll learn the following:

How PyInstaller can simplify application distribution
How to use PyInstaller on your own projects
How to debug PyInstaller errors
What PyInstaller can’t do

PyInstaller gives you the ability to create a folder or executable that users can immediately run without any extra installation. To fully appreciate PyInstaller’s power, it’s useful to revisit some of the distribution problems PyInstaller helps you avoid.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Distribution Problems

Setting up a Python project can be frustrating, especially for non-developers. Often, the setup starts with opening a Terminal, which is a non-starter for a huge group of potential users. This roadblock stops users even before the installation guide delves into the complicated details of virtual environments, Python versions, and the myriad of potential dependencies.

Think about what you typically go through when setting up a new machine for Python development. It probably goes something like this:

Download and install a specific version of Python
Set up pip
Set up a virtual environment
Get a copy of your code
Install dependencies

Stop for a moment and consider if any of the above steps make any sense if you’re not a developer, let alone a Python developer. Probably not.

These problems explode if your user is lucky enough to get to the dependencies portion of the installation. This has gotten much better in the last few years with the prevalence of wheels, but some dependencies still require C/C++ or even FORTRAN compilers!

This barrier to entry is way too high if your goal is to make an application available to as many users as possible. As Raymond Hettinger often says in his excellent talks, “There has to be a better way.”

PyInstaller

PyInstaller abstracts these details from the user by finding all your dependencies and bundling them together. Your users won’t even know they’re running a Python project because the Python Interpreter itself is bundled into your application. Goodbye complicated installation instructions!

PyInstaller performs this amazing feat by introspecting your Python code, detecting your dependencies, and then packaging them into a suitable format depending on your Operating System.

There are lots of interesting details about PyInstaller, but for now you’ll learn the basics of how it works and how to use it. You can always refer to the excellent PyInstaller docs if you want more details.

In addition, PyInstaller can create executables for Windows, Linux, or macOS. This means Windows users will get a .exe, Linux users get a regular executable, and macOS users get a .app bundle. There are some caveats to this. See the limitations section for more information.

Preparing Your Project

PyInstaller requires your application to conform to some minimal structure, namely that you have a CLI script to start your application. Often, this means creating a small script outside of your Python package that simply imports your package and runs main().

The entry-point script is a Python script. You can technically do anything you want in the entry-point script, but you should avoid using explicit relative imports. You can still use relative imports throughout the rest your application if that’s your preferred style.

Note: An entry-point is the code that starts your project or application.

You can give this a try with your own project or follow along with the Real Python feed reader project. For more detailed information on the reader project, check out the the tutorial on Publishing a Package on PyPI.

The first step to building an executable version of this project is to add the entry-point script. Luckily, the feed reader project is well structured, so all you need is a short script outside the package to run it. For example, you can create a file called cli.py alongside the reader package with the following code:

from reader.__main__ import main

if __name__ == '__main__':
    main()

This cli.py script calls main() to start up the feed reader.

Creating this entry-point script is straightforward when you’re working on your own project because you’re familiar with the code. However, it’s not as easy to find the entry-point of another person’s code. In this case, you can start by looking at the setup.py file in the third-party project.

Look for a reference to the entry_points argument in the project’s setup.py. For example, here’s the reader project’s setup.py:

setup(
    name="realpython-reader",
    version="1.0.0",
    description="Read the latest Real Python tutorials",
    long_description=README,
    long_description_content_type="text/markdown",
    url="https://github.com/realpython/reader",
    author="Real Python",
    author_email="office@realpython.com",
    license="MIT",
    classifiers=[
        "License :: OSI Approved :: MIT License",
        "Programming Language :: Python",
        "Programming Language :: Python :: 2",
        "Programming Language :: Python :: 3",
    ],
    packages=["reader"],
    include_package_data=True,
    install_requires=[
        "feedparser", "html2text", "importlib_resources", "typing"
    ],
    entry_points={"console_scripts": ["realpython=reader.__main__:main"]},
)

As you can see, the entry-point cli.py script calls the same function mentioned in the entry_points argument.

After this change, the reader project directory should look like this, assuming you checked it out into a folder called reader:

reader/
|
├── reader/
|   ├── __init__.py
|   ├── __main__.py
|   ├── config.cfg
|   ├── feed.py
|   └── viewer.py
|
├── cli.py
├── LICENSE
├── MANIFEST.in
├── README.md
├── setup.py
└── tests

Notice there is no change to the reader code itself, just a new file called cli.py. This entry-point script is usually all that’s necessary to use your project with PyInstaller.

However, you’ll also want to look out for uses of __import__() or imports inside of functions. These are referred to as hidden imports in PyInstaller terminology.

You can manually specify the hidden imports to force PyInstaller to include those dependencies if changing the imports in your application is too difficult. You’ll see how to do this later in this tutorial.

Once you can launch your application with a Python script outside of your package, you’re ready to give PyInstaller a try at creating an executable.

Using PyInstaller

The first step is to install PyInstaller from PyPI. You can do this using pip like other Python packages:

$ pip install pyinstaller

pip will install PyInstaller’s dependencies along with a new command: pyinstaller. PyInstaller can be imported in your Python code and used as a library, but you’ll likely only use it as a CLI tool.

You’ll use the library interface if you create your own hook files.

You’ll increase the likelihood of PyInstaller’s defaults creating an executable if you only have pure Python dependencies. However, don’t stress too much if you have more complicated dependencies with C/C++ extensions.

PyInstaller supports lots of popular packages like NumPy, PyQt, and Matplotlib without any additional work from you. You can see more about the list of packages that PyInstaller officially supports by referring to the PyInstaller documentation.

Don’t worry if some of your dependencies aren’t listed in the official docs. Many Python packages work fine. In fact, PyInstaller is popular enough that many projects have explanations on how to get things working with PyInstaller.

In short, the chances of your project working out of the box are high.

To try creating an executable with all the defaults, simply give PyInstaller the name of your main entry-point script.

First, cd in the folder with your entry-point and pass it as an argument to the pyinstaller command that was added to your PATH when PyInstaller was installed.

For example, type the following after you cd into the top-level reader directory if you’re following along with the feed reader project:

$ pyinstaller cli.py

Don’t be alarmed if you see a lot of output while building your executable. PyInstaller is verbose by default, and the verbosity can be cranked way up for debugging, which you’ll see later.

Digging Into PyInstaller Artifacts

PyInstaller is complicated under the hood and will create a lot of output. So, it’s important to know what to focus on first. Namely, the executable you can distribute to your users and potential debugging information. By default, the pyinstaller command will create a few things of interest:

A *.spec file
A build/ folder
A dist/ folder

Spec File

The spec file will be named after your CLI script by default. Sticking with our previous example, you’ll see a file called cli.spec. Here’s what the default spec file looks like after running PyInstaller on the cli.py file:

# -*- mode: python -*-

block_cipher = None

a = Analysis(['cli.py'],
             pathex=['/Users/realpython/pyinstaller/reader'],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='cli',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               name='cli')

This file will be automatically created by the pyinstaller command. Your version will have different paths, but the majority should be the same.

Don’t worry, you don’t need to understand the above code to effectively use PyInstaller!

This file can be modified and re-used to create executables later. You can make future builds a bit faster by providing this spec file instead of the entry-point script to the pyinstaller command.

There are a few specific use-cases for PyInstaller spec files. However, for simple projects, you won’t need to worry about those details unless you want to heavily customize how your project is built.

Build Folder

The build/ folder is where PyInstaller puts most of the metadata and internal bookkeeping for building your executable. The default contents will look something like this:

build/
|
└── cli/
    ├── Analysis-00.toc
    ├── base_library.zip
    ├── COLLECT-00.toc
    ├── EXE-00.toc
    ├── PKG-00.pkg
    ├── PKG-00.toc
    ├── PYZ-00.pyz
    ├── PYZ-00.toc
    ├── warn-cli.txt
    └── xref-cli.html

The build folder can be useful for debugging, but unless you have problems, this folder can largely be ignored. You’ll learn more about debugging later in this tutorial.

Dist Folder

After building, you’ll end up with a dist/ folder similar to the following:

dist/
|
└── cli/
    └── cli

The dist/ folder contains the final artifact you’ll want to ship to your users. Inside the dist/ folder, there is a folder named after your entry-point. So in this example, you’ll have a dist/cli folder that contains all the dependencies and executable for our application. The executable to run is dist/cli/cli or dist/cli/cli.exe if you’re on Windows.

You’ll also find lots of files with the extension .so, .pyd, and .dll depending on your Operating System. These are the shared libraries that represent the dependencies of your project that PyInstaller created and collected.

Note: You can add *.spec, build/, and dist/ to your .gitignore file to keep git status clean if you’re using git for version control. The default GitHub gitignore file for Python projects already does this for you.

You’ll want to distribute the entire dist/cli folder, but you can rename cli to anything that suits you.

At this point you can try running the dist/cli/cli executable if you’re following along with the feed reader example.

You’ll notice that running the executable results in errors mentioning the version.txt file. This is because the feed reader and its dependencies require some extra data files that PyInstaller doesn’t know about. To fix that, you’ll have to tell PyInstaller that version.txt is required, which you’ll learn about when testing your new executable.

Customizing Your Builds

PyInstaller comes with lots of options that can be provided as spec files or normal CLI options. Below, you’ll find some of the most common and useful options.

--name

Change the name of your executable.

This is a way to avoid your executable, spec file, and build artifact folders being named after your entry-point script. --name is useful if you have a habit of naming your entry-point script something like cli.py, as I do.

You can build an executable called realpython from the cli.py script with a command like this:

$ pyinstaller cli.py --name realpython

--onefile

Package your entire application into a single executable file.

The default options create a folder of dependencies and and executable, whereas --onefile keeps distribution easier by creating only an executable.

This option takes no arguments. To bundle your project into a single file, you can build with a command like this:

$ pyinstaller cli.py --onefile

With the above command, your dist/ folder will only contain a single executable instead of a folder with all the dependencies in separate files.

--hidden-import

List multiple top-level imports that PyInstaller was unable to detect automatically.

This is one way to work around your code using import inside functions and __import__(). You can also use --hidden-import multiple times in the same command.

This option requires the name of the package that you want to include in your executable. For example, if your project imported the requests library inside of a function, then PyInstaller would not automatically include requests in your executable. You could use the following command to force requests to be included:

$ pyinstaller cli.py --hiddenimport=requests

You can specify this multiple times in your build command, once for each hidden import.

--add-data and --add-binary

Instruct PyInstaller to insert additional data or binary files into your build.

This is useful when you want to bundle in configuration files, examples, or other non-code data. You’ll see an example of this later if you’re following along with the feed reader project.

--exclude-module

Exclude some modules from being included with your executable

This is useful to exclude developer-only requirements like testing frameworks. This is a great way to keep the artifact you give users as small as possible. For example, if you use pytest, you may want to exclude this from your executable:

$ pyinstaller cli.py --exclude-module=pytest

-w

Avoid automatically opening a console window for stdout logging.

This is only useful if you’re building a GUI-enabled application. This helps your hide the details of your implementation by allowing users to never see a terminal.

Similar to the --onefile option, -w takes no arguments:

$ pyinstaller cli.py -w

.spec file

As mentioned earlier, you can reuse the automatically generated .spec file to further customize your executable. The .spec file is a regular Python script that implicitly uses the PyInstaller library API.

Since it’s a regular Python script, you can do almost anything inside of it. You can refer to the official PyInstaller Spec file documentation for more information on that API.

Testing Your New Executable

The best way to test your new executable is on a new machine. The new machine should have the same OS as your build machine. Ideally, this machine should be as similar as possible to what your users use. That may not always be possible, so the next best thing is testing on your own machine.

The key is to run the resulting executable without your development environment activated. This means run without virtualenv, conda, or any other environment that can access your Python installation. Remember, one of the main goals of a PyInstaller-created executable is for users to not need anything installed on their machine.

Picking up with the feed reader example, you’ll notice that running the default cli executable in the dist/cli folder fails. Luckily the error points you to the problem:

FileNotFoundError: 'version.txt' resource not found in 'importlib_resources'
[15110] Failed to execute script cli

The importlib_resources package requires a version.txt file. You can add this file to the build using the --add-data option. Here’s an example of how to include the required version.txt file:

$ pyinstaller cli.py \
    --add-data venv/reader/lib/python3.6/site-packages/importlib_resources/version.txt:importlib_resources

This command tells PyInstaller to include the version.txt file in the importlib_resources folder in a new folder in your build called importlib_resources.

Note: The pyinstaller commands use the \ character to make the command easier to read. You can omit the \ when running commands on your own or copy and paste the commands as-is below provided you’re using the same paths.

You’ll want to adjust the path in the above command to match where you installed the feed reader dependencies.

Now running the new executable will result in a new error about a config.cfg file.

This file is required by the feed reader project, so you’ll need to make sure to include it in your build:

$ pyinstaller cli.py \
    --add-data venv/reader/lib/python3.6/site-packages/importlib_resources/version.txt:importlib_resources \
    --add-data reader/config.cfg:reader

Again, you’ll need to adjust the path to the file based on where you have the feed reader project.

At this point, you should have a working executable that can be given directly to users!

Debugging PyInstaller Executables

As you saw above, you might encounter problems when running your executable. Depending on the complexity of your project, the fixes could be as simple as including data files like the feed reader example. However, sometimes you need more debugging techniques.

Below are a few common strategies that are in no particular order. Often times one of these strategies or a combination will lead to a break-through in tough debugging sessions.

Use the Terminal

First, try running the executable from a terminal so you can see all the output.

Remember to remove the -w build flag to see all the stdout in a console window. Often, you’ll see ImportError exceptions if a dependency is missing.

Debug Files

Inspect the build/cli/warn-cli.txt file for any problems. PyInstaller creates lots of output to help you understand exactly what it’s creating. Digging around in the build/ folder is a great place to start.

Single Directory Builds

Use the --onedir distribution mode of creating distribution folder instead of a single executable. Again, this is the default mode. Building with --onedir gives you the opportunity to inspect all the dependencies included instead of everything being hidden in a single executable.

--onedir is useful for debugging, but --onefile is typically easier for users to comprehend. After debugging you may want to switch to --onefile mode to simplify distribution.

Additional CLI Options

PyInstaller also has options to control the amount of information printed during the build process. Rebuild the executable with the --log-level=DEBUG option to PyInstaller and review the output.

PyInstaller will create a lot of output when increasing the verbosity with --log-level=DEBUG. It’s useful to save this output to a file you can refer to later instead of scrolling in your Terminal. To do this, you can use your shell’s redirection functionality. Here’s an example:

$ pyinstaller --log-level=DEBUG cli.py 2> build.txt

By using the above command, you’ll have a file called build.txt containing lots of additional DEBUG messages.

Note: The standard redirection with > is not sufficient. PyInstaller prints to the stderr stream, not stdout. This means you need to redirect the stderr stream to a file, which can be done using a 2 as in the previous command.

Here’s a sample of what your build.txt file might look like:

67 INFO: PyInstaller: 3.4
67 INFO: Python: 3.6.6
73 INFO: Platform: Darwin-18.2.0-x86_64-i386-64bit
74 INFO: wrote /Users/realpython/pyinstaller/reader/cli.spec
74 DEBUG: Testing for UPX ...
77 INFO: UPX is not available.
78 DEBUG: script: /Users/realptyhon/pyinstaller/reader/cli.py
78 INFO: Extending PYTHONPATH with paths
['/Users/realpython/pyinstaller/reader',
 '/Users/realpython/pyinstaller/reader']

This file will have a lot of detailed information about what was included in your build, why something was not included, and how the executable was packaged.

You can also rebuild your executable using the --debug option in addition to using the --log-level option for even more information.

Note: The -y and --clean options are useful when rebuilding, especially when initially configuring your builds or building with Continuous Integration. These options remove old builds and omit the need for user input during the build process.

Additional PyInstaller Docs

The PyInstaller GitHub Wiki has lots of useful links and debugging tips. Most notably are the sections on making sure everything is packaged correctly and what to do if things go wrong.

Assisting in Dependency Detection

The most common problem you’ll see is ImportError exceptions if PyInstaller couldn’t properly detect all your dependencies. As mentioned before, this can happen if you’re using __import__(), imports inside functions, or other types of hidden imports.

Many of these types of problems can be resolved by using the --hidden-import PyInstaller CLI option. This tells PyInstaller to include a module or package even if it doesn’t automatically detect it. This is the easiest way to work around lots of dynamic import magic in your application.

Another way to work around problems is hook files. These files contain additional information to help PyInstaller package up a dependency. You can write your own hooks and tell PyInstaller to use them with the --additional-hooks-dir CLI option.

Hook files are how PyInstaller itself works internally so you can find lots of example hook files in the PyInstaller source code.

Limitations

PyInstaller is incredibly powerful, but it does have some limitations. Some of the limitations were discussed previously: hidden imports and relative imports in entry-point scripts.

PyInstaller supports making executables for Windows, Linux, and macOS, but it cannot cross compile. Therefore, you cannot make an executable targeting one Operating System from another Operating System. So, to distribute executables for multiple types of OS, you’ll need a build machine for each supported OS.

Related to the cross compile limitation, it’s useful to know that PyInstaller does not technically bundle absolutely everything your application needs to run. Your executable is still dependent on the users’ glibc. Typically, you can work around the glibc limitation by building on the oldest version of each OS you intend to target.

For example, if you want to target a wide array of Linux machines, then you can build on an older version of CentOS. This will give you compatibility with most versions newer than the one you build on. This is the same strategy described in PEP 0513 and is what the PyPA recommends for building compatible wheels.

In fact, you might want to investigate using the PyPA’s manylinux docker image for your Linux build environment. You could start with the base image then install PyInstaller along with all your dependencies and have a build image that supports most variants of Linux.

Conclusion

PyInstaller can help make complicated installation documents unnecessary. Instead, your users can simply run your executable to get started as quickly as possible. The PyInstaller workflow can be summed up by doing the following:

Create an entry-point script that calls your main function.
Install PyInstaller.
Run PyInstaller on your entry-point.
Test your new executable.
Ship your resulting dist/ folder to users.

Your users don’t have to know what version of Python you used or that your application uses Python at all!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Introduction to Python Exceptions

2019-03-05T23:39:23+00:00

A Python program terminates as soon as it encounters an error. In Python, an error can be a syntax error or an exception. In this course, you will see what an exception is and how it differs from a syntax error. After that, you will learn about raising exceptions and making assertions. Then, you’ll finish with a demonstration of the try and except block.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Writing Beautiful Pythonic Code With PEP 8

2019-03-05T18:51:13+00:00

Learn how to write high-quality, readable code by using the Python style guidelines laid out in PEP 8. Following these guidelines helps you make a great impression when sharing your work with potential employers and collaborators.

This course outlines the key guidelines laid out in PEP 8. It’s aimed at beginner to intermediate programmers.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python 3's f-Strings: An Improved String Formatting Syntax

2019-03-05T18:49:19+00:00

As of Python 3.6, f-strings are a great new way to format strings. Not only are they more readable, more concise, and less prone to error than other ways of formatting, but they are also faster! By the end of this course, you’ll know how and why to start using f-strings today.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Refactoring Python Applications for Simplicity

2019-03-04T14:00:00+00:00

Do you want simpler Python code? You always start a project with the best intentions, a clean codebase, and a nice structure. But over time, there are changes to your apps, and things can get a little messy.

If you can write and maintain clean, simple Python code, then it’ll save you lots of time in the long term. You can spend less time testing, finding bugs, and making changes when your code is well laid out and simple to follow.

In this tutorial you’ll learn:

How to measure the complexity of Python code and your applications
How to change your code without breaking it
What the common issues in Python code that cause extra complexity are and how you can fix them

Throughout this tutorial, I’m going to use the theme of subterranean railway networks to explain complexity because navigating a subway system in a large city can be complicated! Some are well designed, and others seem overly complex.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Code Complexity in Python

The complexity of an application and its codebase is relative to the task it’s performing. If you’re writing code for NASA’s jet propulsion laboratory (literally rocket science), then it’s going to be complicated.

The question isn’t so much, “Is my code complicated?” as, “Is my code more complicated than it needs to be?”

The Tokyo railway network is one of the most extensive and complicated in the world. This is partly because Tokyo is a metropolis of over 30 million people, but it’s also because there are 3 networks overlapping each other.

The author of this article getting lost on the Tokyo Metro

There are the Toei and Tokyo Metro rapid-transport networks as well as the Japan Rail East trains going through Central Tokyo. To even the most experienced traveler, navigating central Tokyo can be mind-bogglingly complicated.

Here is a map of the Tokyo railway network to give you some perspective:

Image: Tokyo Metro Co.

If your code is starting to look a bit like this map, then this is the tutorial for you.

First, we’ll go through 4 metrics of complexity that can give you a scale to measure your relative progress in the mission to make your code simpler:

After you’ve explored the metrics, you’ll learn about a tool called wily to automate calculating those metrics.

Metrics for Measuring Complexity

Much time and research have been put into analyzing the complexity of computer software. Overly complex and unmaintainable applications can have a very real cost.

The complexity of software correlates to the quality. Code that is easy to read and understand is more likely to be updated by developers in the future.

Here are some metrics for programming languages. They apply to many languages, not just Python.

Lines of Code

LOC, or Lines of Code, is the crudest measure of complexity. It is debatable whether there is any direct correlation between the lines of code and the complexity of an application, but the indirect correlation is clear. After all, a program with 5 lines is likely simpler than one with 5 million.

When looking at Python metrics, we try to ignore blank lines and lines containing comments.

Lines of code can be calculated using the wc command on Linux and Mac OS, where file.py is the name of the file you want to measure:

$ wc -l file.py

If you want to add the combined lines in a folder by recursively searching for all .py files, you can combine wc with the find command:

$ find . -name \*.py | xargs wc -l

For Windows, PowerShell offers a word count command in Measure-Object and a recursive file search in Get-ChildItem:

$ Get-ChildItem -Path *.py -Recurse | Measure-Object –Line

In the response, you will see the total number of lines.

Why are lines of code used to quantify the amount of code in your application? The assumption is that a line of code roughly equates to a statement. Lines is a better measure than characters, which would include whitespace.

In Python, we are encouraged to put a single statement on each line. This example is 9 lines of code:

 1 x = 5
 2 value = input("Enter a number: ")
 3 y = int(value)
 4 if x < y:
 5     print(f"{x} is less than {y}")
 6 elif x == y:
 7     print(f"{x} is equal to {y}")
 8 else:
 9     print(f"{x} is more than {y}")

If you used only lines of code as your measure of complexity, it could encourage the wrong behaviors.

Python code should be easy to read and understand. Taking that last example, you could reduce the number of lines of code to 3:

 1 x = 5; y = int(input("Enter a number:"))
 2 equality = "is equal to" if x == y else "is less than" if x < y else "is more than"
 3 print(f"{x} {equality} {y}")

But the result is hard to read, and PEP 8 has guidelines around maximum line length and line breaking. You can check out How to Write Beautiful Python Code With PEP 8 for more on PEP 8.

This code block uses 2 Python language features to make the code shorter:

Compound statements: using ;
Chained conditional or ternary statements: name = value if condition else value if condition2 else value2

We have reduced the number of lines of code but violated one of the fundamental laws of Python:

“Readability counts”

— Tim Peters, Zen of Python

This shortened code is potentially harder to maintain because code maintainers are humans, and this short code is harder to read. We will explore some more advanced and useful metrics for complexity.

Cyclomatic Complexity

Cyclomatic complexity is the measure of how many independent code paths there are through your application. A path is a sequence of statements that the interpreter can follow to get to the end of the application.

One way to think of cyclomatic complexity and code paths is imagine your code is like a railway network.

For a journey, you may need to change trains to reach your destination. The Lisbon Metropolitan railway system in Portugal is simple and easy to navigate. The cyclomatic complexity for any trip is equal to the number of lines you need to travel on:

Image: Metro Lisboa

If you needed to get from Alvalade to Anjos, then you would travel 5 stops on the linha verde (green line):

Image: Metro Lisboa

This trip has a cyclomatic complexity of 1 because you only take 1 train. It’s an easy trip. That train is equivalent in this analogy to a code branch.

If you needed to travel from the Aeroporto (airport) to sample the food in the district of Belém, then it’s a more complicated journey. You would have to change trains at Alameda and Cais do Sodré:

Image: Metro Lisboa

This trip has a cyclomatic complexity of 3, because you take 3 trains. You might be better off taking a taxi!

Seeing as how you’re not navigating Lisbon, but rather writing code, the changes of train line become a branch in execution, like an if statement. Let’s explore this example:

x = 1

There is only 1 way this code can be executed, so it has a cyclomatic complexity of 1.

If we add a decision, or branch to the code as an if statement, it increases the complexity:

x = 1
if x < 2:
    x += 1

Even though there is only 1 way this code can be executed, as x is a constant, this has a cyclomatic complexity of 2. All of the cyclomatic complexity analyzers will treat an if statement as a branch.

This is also an example of overly complex code. The if statement is useless as x has a fixed value. You could simply refactor this example to the following:

x = 2

That was a toy example, so let’s explore something a little more real.

main() has a cyclomatic complexity of 5. I’ll comment each branch in the code so you can see where they are:

# cyclomatic_example.py
import sys

def main():
    if len(sys.argv) > 1:  # 1
        filepath = sys.argv[1]
    else:
        print("Provide a file path")
        exit(1)
    if filepath:  # 2
        with open(filepath) as fp:  # 3
            for line in fp.readlines():  # 4
                if line != "\n":  # 5
                    print(line, end="")

if __name__ == "__main__":  # Ignored.
    main()

There are certainly ways that code can be refactored into a far simpler alternative. We’ll get to that later.

Note: The Cyclomatic Complexity measure was developed by Thomas J. McCabe, Sr in 1976. You may see it referred to as the McCabe metric or McCabe number.

In the following examples, we will use the radon library from PyPi to calculate metrics. You can install it now:

$ pip install radon

To calculate cyclomatic complexity using radon, you can save the example into a file called cyclomatic_example.py and use radon from the command line.

The radon command takes 2 main arguments:

The type of analysis (cc for cyclomatic complexity)
A path to the file or folder to analyze

Execute the radon command with the cc analysis against the cyclomatic_example.py file. Adding -s will give the cyclomatic complexity in the output:

$ radon cc cyclomatic_example.py -s
cyclomatic_example.py
    F 4:0 main - B (6)

The output is a little cryptic. Here is what each part means:

F means function, M means method, and C means class.
main is the name of the function.
4 is the line the function starts on.
B is the rating from A to F. A is the best grade, meaning the least complexity.
The number in parentheses, 6, is the cyclomatic complexity of the code.

Halstead Metrics

The Halstead complexity metrics relate to the size of a program’s codebase. They were developed by Maurice H. Halstead in 1977. There are 4 measures in the Halstead equations:

Operands are values and names of variables.
Operators are all of the built-in keywords, like if, else, for or while.
Length (N) is the number of operators plus the number of operands in your program.
Vocabulary (h) is the number of unique operators plus the number of unique operands in your a program.

There are then 3 additional metrics with those measures:

Volume (V) represents a product of the length and the vocabulary.
Difficulty (D) represents a product of half the unique operands and the reuse of operands.
Effort (E) is the overall metric that is a product of volume and difficulty.

All of this is very abstract, so let’s put it in relative terms:

The effort of your application is highest if you use a lot of operators and unique operands.
The effort of your application is lower if you use a few operators and fewer variables.

For the cyclomatic_complexity.py example, operators and operands both occur on the first line:

import sys  # import (operator), sys (operand)

import is an operator, and sys is the name of the module, so it’s an operand.

In a slightly more complex example, there are a number of operators and operands:

if len(sys.argv) > 1:
    ...

There are 5 operators in this example:

if
(
)
>
:

Furthermore, there are 2 operands:

sys.argv
1

Be aware that radon only counts a subset of operators. For example, parentheses are excluded in any calculations.

To calculate the Halstead measures in radon, you can run the following command:

$ radon hal cyclomatic_example.py
cyclomatic_example.py:
    h1: 3
    h2: 6
    N1: 3
    N2: 6
    vocabulary: 9
    length: 9
    calculated_length: 20.264662506490406
    volume: 28.529325012980813
    difficulty: 1.5
    effort: 42.793987519471216
    time: 2.377443751081734
    bugs: 0.009509775004326938

Why does radon give a metric for time and bugs?

Halstead theorized that you could estimate the time taken in seconds to code by dividing the effort (E) by 18.

Halstead also stated that the expected number of bugs could be estimated dividing the volume (V) by 3000. Keep in mind this was written in 1977, before Python was even invented! So don’t panic and start looking for bugs just yet.

Maintainability Index

The maintainability index brings the McCabe Cyclomatic Complexity and the Halstead Volume measures in a scale roughly between zero and one-hundred.

If you’re interested, the original equation is as follows:

In the equation, V is the Halstead volume metric, C is the cyclomatic complexity, and L is the number of lines of code.

If you’re as baffled as I was when I first saw this equation, here’s it means: it calculates a scale that includes the number of variables, operations, decision paths, and lines of code.

It is used across many tools and languages, so it’s one of the more standard metrics. However, there are numerous revisions of the equation, so the exact number shouldn’t be taken as fact. radon, wily, and Visual Studio cap the number between 0 and 100.

On the maintainability index scale, all you need to be paying attention to is when your code is getting significantly lower (toward 0). The scale considers anything lower than 25 as hard to maintain, and anything over 75 as easy to maintain. The Maintainability Index is also referred to as MI.

The maintainability index can be used as a measure to get the current maintainability of your application and see if you’re making progress as you refactor it.

To calculate the maintainability index from radon, run the following command:

$ radon mi cyclomatic_example.py -s
cyclomatic_example.py - A (87.42)

In this result, A is the grade that radon has applied to the number 87.42 on a scale. On this scale, A is most maintainable and F the least.

Using `wily` to Capture and Track Your Projects’ Complexity

wily is an open-source software project for collecting code-complexity metrics, including the ones we’ve covered so far like Halstead, Cyclomatic, and LOC. wily integrates with Git and can automate the collection of metrics across Git branches and revisions.

The purpose of wily is to give you the ability to see trends and changes in the complexity of your code over time. If you were trying to fine-tune a car or improve your fitness, you’d start off with measuring a baseline and tracking improvements over time.

Installing `wily`

wily is available on PyPi and can be installed using pip:

$ pip install wily

Once wily is installed, you have some commands available in your command-line:

wily build: iterate through the Git history and analyze the metrics for each file
wily report: see the historical trend in metrics for a given file or folder
wily graph: graph a set of metrics in an HTML file

Building a Cache

Before you can use wily, you need to analyze your project. This is done using the wily build command.

For this section of the tutorial, we will analyze the very popular requests package, used for talking to HTTP APIs. Because this project is open-source and available on GitHub, we can easily access and download a copy of the source code:

$ git clone https://github.com/requests/requests
$ cd requests
$ ls
AUTHORS.rst        CONTRIBUTING.md    LICENSE            Makefile
Pipfile.lock       _appveyor          docs               pytest.ini
setup.cfg          tests              CODE_OF_CONDUCT.md HISTORY.md
MANIFEST.in        Pipfile            README.md          appveyor.yml
ext                requests           setup.py           tox.ini

Note: Windows users should use the PowerShell command prompt for the following examples instead of traditional MS-DOS Command-Line. To start the PowerShell CLI press Win+R and type powershell then Enter.

You will see a number of folders here, for tests, documentation, and configuration. We’re only interested in the source code for the requests Python package, which is in a folder called requests.

Call the wily build command from the cloned source code and provide the name of the source code folder as the first argument:

$ wily build requests

This will take a few minutes to analyze, depending on how much CPU power your computer has:

Collecting Data on Your Project

Once you have analyzed the requests source code, you can query any file or folder to see key metrics. Earlier in the tutorial, we discussed the following:

Lines of Code
Maintainability Index
Cyclomatic Complexity

Those are the 3 default metrics in wily. To see those metrics for a specific file (such as requests/api.py), run the following command:

$ wily report requests/api.py

wily will print a tabular report on the default metrics for each Git commit in reverse date order. You will see the most recent commit at the top and the oldest at the bottom:

Revision	Author	Date	MI	Lines of Code	Cyclomatic Complexity
f37daf2	Nate Prewitt	2019-01-13	100 (0.0)	158 (0)	9 (0)
6dd410f	Ofek Lev	2019-01-13	100 (0.0)	158 (0)	9 (0)
5c1f72e	Nate Prewitt	2018-12-14	100 (0.0)	158 (0)	9 (0)
c4d7680	Matthieu Moy	2018-12-14	100 (0.0)	158 (0)	9 (0)
c452e3b	Nate Prewitt	2018-12-11	100 (0.0)	158 (0)	9 (0)
5a1e738	Nate Prewitt	2018-12-10	100 (0.0)	158 (0)	9 (0)

This tells us that the requests/api.py file has:

158 lines of code
A perfect maintainability index of 100
A cyclomatic complexity of 9

To see other metrics, you first need to know the names of them. You can see this by running the following command:

$ wily list-metrics

You will see a list of operators, modules that analyze the code, and the metrics they provide.

To query alternative metrics on the report command, add their names after the filename. You can add as many metrics as you wish. Here’s an example with the Maintainability Rank and the Source Lines of Code:

$ wily report requests/api.py maintainability.rank raw.sloc

You will see the table now has 2 different columns with the alternative metrics.

Graphing Metrics

Now that you know the names of the metrics and how to query them on the command line, you can also visualize them in graphs. wily supports HTML and interactive charts with a similar interface as the report command:

$ wily graph requests/sessions.py maintainability.mi

Your default browser will open with an interactive chart like this:

You can hover over specific data points, and it will show the Git commit message as well as the data.

If you want to save the HTML file in a folder or repository, you can add the -o flag with the path to a file:

$ wily graph requests/sessions.py maintainability.mi -o my_report.html

There will now be a file called my_report.html that you can share with others. This command is ideal for team dashboards.

`wily` as a `pre-commit` Hook

wily can be configured so that before you commit changes to your project, it can alert you to improvements or degradations in complexity.

wily has a wily diff command, that compares the last indexed data with the current working copy of a file.

To run a wily diff command, provide the names of the files you have changed. For example, if I made some changes to requests/api.py you will see the impact on the metrics by running wily diff with the file path:

$ wily diff requests/api.py

In the response, you will see all of the changed metrics, as well as the functions or classes that have changed for cyclomatic complexity:

The diff command can be paired with a tool called pre-commit. pre-commit inserts a hook into your Git configuration that calls a script every time you run the git commit command.

To install pre-commit, you can install from PyPI:

$ pip install pre-commit

Add the following to a .pre-commit-config.yaml in your projects root directory:

repos:
-   repo: local
    hooks:
    -   id: wily
        name: wily
        entry: wily diff
        verbose: true
        language: python
        additional_dependencies: [wily]

Once setting this, you run the pre-commit install command to finalize things:

$ pre-commit install

Whenever you run the git commit command, it will call wily diff along with the list of files you’ve added to your staged changes.

wily is a useful utility to baseline the complexity of your code and measure the improvements you make when you start to refactor.

Refactoring in Python

Refactoring is the technique of changing an application (either the code or the architecture) so that it behaves the same way on the outside, but internally has improved. These improvements can be stability, performance, or reduction in complexity.

One of the world’s oldest underground railways, the London Underground, started in 1863 with the opening of the Metropolitan line. It had gas-lit wooden carriages hauled by steam locomotives. On the opening of the railway, it was fit for purpose. 1900 brought the invention of the electric railways.

By 1908, the London Underground had expanded to 8 railways. During the Second World War, the London Underground stations were closed to trains and used as air-raid shelters. The modern London Underground carries millions of passengers a day with over 270 stations:

Joint London Underground Railways Map, c. 1908 (Image: Wikipedia)

It’s almost impossible to write perfect code the first time, and requirements change frequently. If you would have asked the original designers of the railway to design a network fit for 10 million passengers a day in 2020, they would not design the network that exists today.

Instead, the railway has undergone a series of continuous changes to optimize its operation, design, and layout to match the changes in the city. It has been refactored.

In this section, you’ll explore how to safely refactor by leveraging tests and tools. You’ll also see how to use the refactoring functionality in Visual Studio Code and PyCharm:

Avoiding Risks With Refactoring: Leveraging Tools and Having Tests

If the point of refactoring is to improve the internals of an application without impacting the externals, how do you ensure the externals haven’t changed?

Before you charge into a major refactoring project, you need to make sure you have a solid test suite for your application. Ideally, that test suite should be mostly automated, so that as you make changes, you see the impact on the user and address it quickly.

If you want to learn more about testing in Python, Getting Started With Testing in Python is a great place to start.

There is no perfect number of tests to have on your application. But, the more robust and thorough the test suite, the more aggressively you can refactor your code.

The two most common tasks you will perform when doing refactoring are:

Renaming modules, functions, classes, and methods
Finding usages of functions, classes, and methods to see where they are called

You can simply do this by hand using search and replace, but it is both time consuming and risky. Instead, there are some great tools to perform these tasks.

Using `rope` for Refactoring

rope is a free Python utility for refactoring Python code. It comes with an extensive set of APIs for refactoring and renaming components in your Python codebase.

rope can be used in two ways:

By using an editor plugin, for Visual Studio Code, Emacs, or Vim
Directly by writing scripts to refactor your application

To use rope as a library, first install rope by executing pip:

$ pip install rope

It is useful to work with rope on the REPL so that you can explore the project and see changes in real time. To start, import the Project type and instantiate it with the path to the project:

>>>

>>> from rope.base.project import Project

>>> proj = Project('requests')

The proj variable can now perform a series of commands, like get_files and get_file, to get a specific file. Get the file api.py and assign it to a variable called api:

>>>

>>> [f.name for f in proj.get_files()]
['structures.py', 'status_codes.py', ...,'api.py', 'cookies.py']

>>> api = proj.get_file('api.py')

If you wanted to rename this file, you could simply rename it on the filesystem. However, any other Python files in your project that imported the old name would now be broken. Let’s rename the api.py to new_api.py:

>>>

>>> from rope.refactor.rename import Rename

>>> change = Rename(proj, api).get_changes('new_api')

>>> proj.do(change)

Running git status, you will see that rope made some changes to the repository:

$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   requests/__init__.py
    deleted:    requests/api.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)

   requests/.ropeproject/
   requests/new_api.py

no changes added to commit (use "git add" and/or "git commit -a")

The three changes made by rope are the following:

Deleted requests/api.py and created requests/new_api.py
Modified requests/__init__.py to import from new_api instead of api
Created a project folder named .ropeproject

To reset the change, run git reset.

There are hundreds of other refactorings that can be done with rope.

Using Visual Studio Code for Refactoring

Visual Studio Code opens up a small subset of the refactoring commands available in rope through its own UI.

You can:

Extract variables from a statement
Extract methods from a block of code
Sort imports into a logical order

Here is an example of using the Extract methods command from the command palette:

Using PyCharm for Refactoring

If you use or are considering using PyCharm as a Python editor, it’s worth taking note of the powerful refactoring capabilities it has.

You can access all the refactoring shortcuts with the Ctrl+T command on Windows and macOS. The shortcut to access refactoring in Linux is Ctrl+Shift+Alt+T.

Finding Callers and Usages of Functions and Classes

Before you remove a method or class or change the way it behaves, you’ll need to know what code depends on it. PyCharm can search for all usages of a method, function, or class within your project.

To access this feature, select a method, class, or variable by right-clicking and select Find Usages:

All of the code that uses your search criteria is shown in a panel at the bottom. You can double-click on any item to navigate directly to the line in question.

Using the PyCharm Refactoring Tools

Some of the other refactoring commands include the ability to:

Extract methods, variables, and constants from existing code
Extract abstract classes from existing class signatures, including the ability to specify abstract methods
Rename practically anything, from a variable to a method, file, class, or module

Here is an example of renaming the same api.py module you renamed earlier using the rope module to new_api.py:

The rename command is contextualized to the UI, which makes refactoring quick and simple. It has updated the imports automatically in __init__.py with the new module name.

Another useful refactor is the Change Signature command. This can be used to add, remove, or rename arguments to a function or method. It will search for usages and update them for you:

You can set default values and also decide how the refactoring should handle the new arguments.

Summary

Refactoring is an important skill for any developer. As you’ve learned in this chapter, you aren’t alone. The tools and IDEs already come with powerful refactoring features to be able to make changes quickly.

Complexity Anti-Patterns

Now that you know how complexity can be measured, how to measure it, and how to refactor your code, it’s time to learn 5 common anti-patterns that make code more complex than it need be:

If you can master these patterns and know how to refactor them, you’ll soon be on track (pun intended) to a more maintainable Python application.

1. Functions That Should Be Objects

Python supports procedural programming using functions and also inheritable classes. Both are very powerful and should be applied to different problems.

Take this example of a module for working with images. The logic in the functions has been removed for brevity:

# imagelib.py

def load_image(path):
    with open(path, "rb") as file:
        fb = file.load()
    image = img_lib.parse(fb)
    return image

def crop_image(image, width, height):
    ...
    return image

def get_image_thumbnail(image, resolution=100):
    ...
    return image

There are a few issues with this design:

It’s not clear if crop_image() and get_image_thumbnail() modify the original image variable or create new images. If you wanted to load an image then create both a cropped and thumbnail image, would you have to copy the instance first? You could read the source code in the functions, but you can’t rely on every developer doing this.
You have to pass the image variable as an argument in every call to the image functions.

This is how the calling code might look:

from imagelib import load_image, crop_image, get_image_thumbnail

image = load_image('~/face.jpg')
image = crop_image(image, 400, 500)
thumb = get_image_thumbnail(image)

Here are some symptoms of code using functions that could be refactored into classes:

Similar arguments across functions
Higher number of Halstead h2 unique operands
Mix of mutable and immutable functions
Functions spread across multiple Python files

Here is a refactored version of those 3 functions, where the following happens:

.__init__() replaces load_image().
crop() becomes a class method.
get_image_thumbnail() becomes a property.

The thumbnail resolution has become a class property, so it can be changed globally or on that particular instance:

# imagelib.py

class Image(object):
    thumbnail_resolution = 100
    def __init__(self, path):
        ...

    def crop(self, width, height):
        ...

    @property
    def thumbnail(self):
        ...
        return thumb

If there were many more image-related functions in this code, the refactoring to a class could make a drastic change. The next consideration would be the complexity of the consuming code.

This is how the refactored example would look:

from imagelib import Image

image = Image('~/face.jpg')
image.crop(400, 500)
thumb = image.thumbnail

In the resulting code, we have solved the original problems:

It is clear that thumbnail returns a thumbnail since it is a property, and that it doesn’t modify the instance.
The code no longer requires creating new variables for the crop operation.

2. Objects That Should Be Functions

Sometimes, the reverse is true. There is object-oriented code which would be better suited to a simple function or two.

Here are some tell-tale signs of incorrect use of classes:

Classes with 1 method (other than .__init__())
Classes that contain only static methods

Take this example of an authentication class:

# authenticate.py

class Authenticator(object):
    def __init__(self, username, password):
        self.username = username
        self.password = password

    def authenticate(self):
        ...
        return result

It would make more sense to just have a simple function named authenticate() that takes username and password as arguments:

# authenticate.py

def authenticate(username, password):
    ...
    return result

You don’t have to sit down and look for classes that match these criteria by hand: pylint comes with a rule that classes should have a minimum of 2 public methods. For more on PyLint and other code quality tools, you can check out Python Code Quality.

To install pylint, run the following command in your console:

$ pip install pylint

pylint takes a number of optional arguments and then the path to one or more files and folders. If you run pylint with its default settings, it’s going to give a lot of output as pylint has a huge number of rules. Instead, you can run specific rules. The too-few-public-methods rule id is R0903. You can look this up on the documentation website:

$ pylint --disable=all --enable=R0903 requests
************* Module requests.auth
requests/auth.py:72:0: R0903: Too few public methods (1/2) (too-few-public-methods)
requests/auth.py:100:0: R0903: Too few public methods (1/2) (too-few-public-methods)
************* Module requests.models
requests/models.py:60:0: R0903: Too few public methods (1/2) (too-few-public-methods)

-----------------------------------
Your code has been rated at 9.99/10

This output tells us that auth.py contains 2 classes that have only 1 public method. Those classes are on lines 72 and 100. There is also a class on line 60 of models.py with only 1 public method.

3. Converting “Triangular” Code to Flat Code

If you were to zoom out on your source code and tilt your head 90 degrees to the right, does the whitespace look flat like Holland or mountainous like the Himalayas? Mountainous code is a sign that your code contains a lot of nesting.

Here’s one of the principles in the Zen of Python:

“Flat is better than nested”

— Tim Peters, Zen of Python

Why would flat code be better than nested code? Because nested code makes it harder to read and understand what is happening. The reader has to understand and memorize the conditions as they go through the branches.

These are the symptoms of highly nested code:

A high cyclomatic complexity because of the number of code branches
A low Maintainability Index because of the high cyclomatic complexity relative to the number of lines of code

Take this example that looks at the argument data for strings that match the word error. It first checks if the data argument is a list. Then, it iterates over each and checks if the item is a string. If it is a string and the value is "error", then it returns True. Otherwise, it returns False:

def contains_errors(data):
    if isinstance(data, list):
        for item in data:
            if isinstance(item, str):
                if item == "error":
                    return True
    return False

This function would have a low maintainability index because it is small, but it has a high cyclomatic complexity.

Instead, we can refactor this function by “returning early” to remove a level of nesting and returning False if the value of data is not list. Then using .count() on the list object to count for instances of "error". The return value is then an evaluation that the .count() is greater than zero:

def contains_errors(data):
    if not isinstance(data, list):
        return False
    return data.count("error") > 0

Another technique for reducing nesting is to leverage list comprehensions. This common pattern of creating a new list, going through each item in a list to see if it matches a criterion, then adding all matches to the new list:

results = []
for item in iterable:
    if item == match:
        results.append(item)

This code can be replaced with a faster and more efficient list comprehension.

Refactor the last example into a list comprehension and an if statement:

results = [item for item in iterable if item == match]

This new example is smaller, has less complexity, and is more performant.

If your data is not a single dimension list, then you can leverage the itertools package in the standard library, which contains functions for creating iterators from data structures. You can use it for chaining iterables together, mapping structures, cycling or repeating over existing iterables.

Itertools also contains functions for filtering data, like filterfalse(). For more on Itertools, check out Itertools in Python 3, By Example.

4. Handling Complex Dictionaries With Query Tools

One of Python’s most powerful and widely used core types is the dictionary. It’s fast, efficient, scalable, and highly flexible.

If you’re new to dictionaries, or think you could leverage them more, you can read Dictionaries in Python for more information.

It does have one major side-effect: when dictionaries are highly nested, the code that queries them becomes nested too.

Take this example piece of data, a sample of the Tokyo Metro lines you saw earlier:

data = {
 "network": {
  "lines": [
    {
     "name.en": "Ginza",
     "name.jp": "銀座線",
     "color": "orange",
     "number": 3,
     "sign": "G"
    },
    {
     "name.en": "Marunouchi",
     "name.jp": "丸ノ内線",
     "color": "red",
     "number": 4,
     "sign": "M"
    }
  ]
 }
}

If you wanted to get the line that matched a certain number, this could be achieved in a small function:

def find_line_by_number(data, number):
    matches = [line for line in data if line['number'] == number]
    if len(matches) > 0:
        return matches[0]
    else:
        raise ValueError(f"Line {number} does not exist.")

Even though the function itself is small, calling the function is unnecessarily complicated because the data is so nested:

>>>

>>> find_line_by_number(data["network"]["lines"], 3)

There are third party tools for querying dictionaries in Python. Some of the most popular are JMESPath, glom, asq, and flupy.

JMESPath can help with our train network. JMESPath is a querying language designed for JSON, with a plugin available for Python that works with Python dictionaries. To install JMESPath, do the following:

$ pip install jmespath

Then open up a Python REPL to explore the JMESPath API, copying in the data dictionary. To get started, import jmespath and call search() with a query string as the first argument and the data as the second. The query string "network.lines" means return data['network']['lines']:

>>>

>>> import jmespath

>>> jmespath.search("network.lines", data)
[{'name.en': 'Ginza', 'name.jp': '銀座線',
  'color': 'orange', 'number': 3, 'sign': 'G'}, 
 {'name.en': 'Marunouchi', 'name.jp': '丸ノ内線',
  'color': 'red', 'number': 4, 'sign': 'M'}]

When working with lists, you can use square brackets and provide a query inside. The “everything” query is simply *. You can then add the name of the attribute inside each matching item to return. If you wanted to get the line number for every line, you could do this:

>>>

>>> jmespath.search("network.lines[*].number", data)
[3, 4]

You can provide more complex queries, like a == or <. The syntax is a little unusual for Python developers, so keep the documentation handy for reference.

If we wanted to find the line with the number 3, this can be done in a single query:

>>>

>>> jmespath.search("network.lines[?number==`3`]", data)
[{'name.en': 'Ginza', 'name.jp': '銀座線', 'color': 'orange', 'number': 3, 'sign': 'G'}]

If we wanted to get the color of that line, you could add the attribute in the end of the query:

>>>

>>> jmespath.search("network.lines[?number==`3`].color", data)
['orange']

JMESPath can be used to reduce and simplify code that queries and searches through complex dictionaries.

5. Using `attrs` and `dataclasses` to Reduce Code

Another goal when refactoring is to simply reduce the amount of code in the codebase while achieving the same behaviors. The techniques shown so far can go a long way to refactoring code into smaller and simpler modules.

Some other techniques require a knowledge of the standard library and some third party libraries.

What Is Boilerplate?

Boilerplate code is code that has to be used in many places with little or no alterations.

Taking our train network as an example, if we were to convert that into types using Python classes and Python 3 type hints, it might look something like this:

from typing import List

class Line(object):
    def __init__(self, name_en: str, name_jp: str, color: str, number: int, sign: str):
        self.name_en = name_en
        self.name_jp = name_jp
        self.color = color
        self.number = number
        self.sign = sign

    def __repr__(self):
        return f"<Line {self.name_en} color='{self.color}' number={self.number} sign='{self.sign}'>"

    def __str__(self):
        return f"The {self.name_en} line"

class Network(object):
    def __init__(self, lines: List[Line]):
        self._lines = lines

    @property
    def lines(self) -> List[Line]:
        return self._lines

Now, you might also want to add other magic methods, like .__eq__(). This code is boilerplate. There’s no business logic or any other functionality here: we’re just copying data from one place to another.

A Case for `dataclasses`

Introduced into the standard library in Python 3.7, with a backport package for Python 3.6 on PyPI, the dataclasses module can help remove a lot of boilerplate for these types of classes where you’re just storing data.

To convert the Line class above to a dataclass, convert all of the fields to class attributes and ensure they have type annotations:

from dataclasses import dataclass

@dataclass
class Line(object):
    name_en: str
    name_jp: str
    color: str
    number: int
    sign: str

You can then create an instance of the Line type with the same arguments as before, with the same fields, and even .__str__(), .__repr__(), and .__eq__() are implemented:

>>>

>>> line = Line('Marunouchi', "丸ノ内線", "red", 4, "M")

>>> line.color
red

>>> line2 = Line('Marunouchi', "丸ノ内線", "red", 4, "M")

>>> line == line2
True

Dataclasses are a great way to reduce code with a single import that’s already available in the standard library. For a full walkthrough, you can checkout The Ultimate Guide to Data Classes in Python 3.7.

Some `attrs` Use Cases

attrs is a third party package that’s been around a lot longer than dataclasses. attrs has a lot more functionality, and it’s available on Python 2.7 and 3.4+.

If you are using Python 3.5 or below, attrs is a great alternative to dataclasses. Also, it provides many more features.

The equivalent dataclasses example in attrs would look similar. Instead of using type annotations, the class attributes are assigned with a value from attrib(). This can take additional arguments, such as default values and callbacks for validating input:

from attr import attrs, attrib

@attrs
class Line(object):
    name_en = attrib()
    name_jp = attrib()
    color = attrib()
    number = attrib()
    sign = attrib()

attrs can be a useful package for removing boilerplate code and input validation on data classes.

Conclusion

Now that you’ve learned how to identify and tackle complicated code, think back to the steps you can now take to make your application easier to change and manage:

Start off by creating a baseline of your project using a tool like wily.
Look at some of the metrics and start with the module that has the lowest maintainability index.
Refactor that module using the safety provided in tests and the knowledge of tools like PyCharm and rope.

Once you follow these steps and the best practices in this article, you can do other exciting things to your application, like adding new features and improving performance.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Traditional Face Detection With Python

2019-02-27T14:00:00+00:00

Computer vision is an exciting and growing field. There are tons of interesting problems to solve! One of them is face detection: the ability of a computer to recognize that a photograph contains a human face, and tell you where it is located. In this article, you’ll learn about face detection with Python.

To detect any object in an image, it is necessary to understand how images are represented inside a computer, and how that objects differs visually from any other object.

Once that is done, the process of scanning an image and looking for those visual cues needs to be automated and optimized. All these steps come together to form a fast and reliable computer vision algorithm.

In this tutorial you’ll learn:

What face detection is
How computers understand features in images
How to quickly analyze many different features to reach a decision
How to use a minimal Python solution for detecting human faces in images

Free Bonus: Click here to get the Python Face Detection & OpenCV Examples Mini-Guide that shows you practical code examples of real-world Python computer vision techniques.

What Is Face Detection?

Face detection is a type of computer vision technology that is able to identify people’s faces within digital images. This is very easy for humans, but computers need precise instructions. The images might contain many objects that aren’t human faces, like buildings, cars, animals, and so on.

It is distinct from other computer vision technologies that involve human faces, like facial recognition, analysis, and tracking.

Facial recognition involves identifying the face in the image as belonging to person X and not person Y. It is often used for biometric purposes, like unlocking your smartphone.

Facial analysis tries to understand something about people from their facial features, like determining their age, gender, or the emotion they are displaying.

Facial tracking is mostly present in video analysis and tries to follow a face and its features (eyes, nose, and lips) from frame to frame. The most popular applications are various filters available in mobile apps like Snapchat.

All of these problems have different technological solutions. This tutorial will focus on a traditional solution for the first challenge: face detection.

How Do Computers “See” Images?

The smallest element of an image is called a pixel, or a picture element. It is basically a dot in the picture. An image contains multiple pixels arranged in rows and columns.

You will often see the number of rows and columns expressed as the image resolution. For example, an Ultra HD TV has the resolution of 3840x2160, meaning it is 3840 pixels wide and 2160 pixels high.

But a computer does not understand pixels as dots of color. It only understands numbers. To convert colors to numbers, the computer uses various color models.

In color images, pixels are often represented in the RGB color model. RGB stands for Red Green Blue. Each pixel is a mix of those three colors. RGB is great at modeling all the colors humans perceive by combining various amounts of red, green, and blue.

Since a computer only understand numbers, every pixel is represented by three numbers, corresponding to the amounts of red, green, and blue present in that pixel. You can learn more about color spaces in Image Segmentation Using Color Spaces in OpenCV + Python.

In grayscale (black and white) images, each pixel is a single number, representing the amount of light, or intensity, it carries. In many applications, the range of intensities is from 0 (black) to 255 (white). Everything between 0 and 255 is various shades of gray.

If each grayscale pixel is a number, an image is nothing more than a matrix (or table) of numbers:

Example 3x3 image with pixel values and colors

In color images, there are three such matrices representing the red, green, and blue channels.

What Are Features?

A feature is a piece of information in an image that is relevant to solving a certain problem. It could be something as simple as a single pixel value, or more complex like edges, corners, and shapes. You can combine multiple simple features into a complex feature.

Applying certain operations to an image produces information that could be considered features as well. Computer vision and image processing have a large collection of useful features and feature extracting operations.

Basically, any inherent or derived property of an image could be used as a feature to solve tasks.

Preparation

To run the code examples, you need to set up an environment with all the necessary libraries installed. The simplest way is to use conda.

You will need three libraries:

scikit-image
scikit-learn
opencv

To create an environment in conda, run these commands in your shell:

$ conda create --name face-detection python=3.7
$ source activate face-detection
(face-detection)$ conda install scikit-learn
(face-detection)$ conda install -c conda-forge scikit-image
(face-detection)$ conda install -c menpo opencv3

If you are having problems installing OpenCV correctly and running the examples, try consulting their Installation Guide or the article on OpenCV Tutorials, Resources, and Guides.

Now you have all the packages necessary to practice what you learn in this tutorial.

Viola-Jones Object Detection Framework

This algorithm is named after two computer vision researchers who proposed the method in 2001: Paul Viola and Michael Jones.

They developed a general object detection framework that was able to provide competitive object detection rates in real time. It can be used to solve a variety of detection problems, but the main motivation comes from face detection.

The Viola-Jones algorithm has 4 main steps, and you’ll learn more about each of them in the sections that follow:

Selecting Haar-like features
Creating an integral image
Running AdaBoost training
Creating classifier cascades

Given an image, the algorithm looks at many smaller subregions and tries to find a face by looking for specific features in each subregion. It needs to check many different positions and scales because an image can contain many faces of various sizes. Viola and Jones used Haar-like features to detect faces.

Haar-Like Features

All human faces share some similarities. If you look at a photograph showing a person’s face, you will see, for example, that the eye region is darker than the bridge of the nose. The cheeks are also brighter than the eye region. We can use these properties to help us understand if an image contains a human face.

A simple way to find out which region is lighter or darker is to sum up the pixel values of both regions and comparing them. The sum of pixel values in the darker region will be smaller than the sum of pixels in the lighter region. This can be accomplished using Haar-like features.

A Haar-like feature is represented by taking a rectangular part of an image and dividing that rectangle into multiple parts. They are often visualized as black and white adjacent rectangles:

Basic Haar-like rectangle features

In this image, you can see 4 basic types of Haar-like features:

Horizontal feature with two rectangles
Vertical feature with two rectangles
Vertical feature with three rectangles
Diagonal feature with four rectangles

The first two examples are useful for detecting edges. The third one detects lines, and the fourth one is good for finding diagonal features. But how do they work?

The value of the feature is calculated as a single number: the sum of pixel values in the black area minus the sum of pixel values in the white area. For uniform areas like a wall, this number would be close to zero and won’t give you any meaningful information.

To be useful, a Haar-like feature needs to give you a large number, meaning that the areas in the black and white rectangles are very different. There are known features that perform very well to detect human faces:

Haar-like feature applied on the eye region. (Image: Wikipedia)

In this example, the eye region is darker than the region below. You can use this property to find which areas of an image give a strong response (large number) for a specific feature:

Haar-like feature applied on the bridge of the nose. (Image: Wikipedia)

This example gives a strong response when applied to the bridge of the nose. You can combine many of these features to understand if an image region contains a human face.

As mentioned, the Viola-Jones algorithm calculates a lot of these features in many subregions of an image. This quickly becomes computationally expensive: it takes a lot of time using the limited resources of a computer.

To tackle this problem, Viola and Jones used integral images.

Integral Images

An integral image (also known as a summed-area table) is the name of both a data structure and an algorithm used to obtain this data structure. It is used as a quick and efficient way to calculate the sum of pixel values in an image or rectangular part of an image.

In an integral image, the value of each point is the sum of all pixels above and to the left, including the target pixel:

Calculating an integral image from pixel values

The integral image can be calculated in a single pass over the original image. This reduces summing the pixel intensities within a rectangle into only three operations with four numbers, regardless of rectangle size:

Calculate the sum of pixels in the orange rectangle.

The sum of pixels in the rectangle ABCD can be derived from the values of points A, B, C, and D, using the formula D - B - C + A. It is easier to understand this formula visually:

Calculating the sum of pixels step by step

You’ll notice that subtracting both B and C means that the area defined with A has been subtracted twice, so we need to add it back again.

Now you have a simple way to calculate the difference between the sums of pixel values of two rectangles. This is perfect for Haar-like features!

But how do you decide which of these features and in what sizes to use for finding faces in images? This is solved by a machine learning algorithm called boosting. Specifically, you will learn about AdaBoost, short for Adaptive Boosting.

AdaBoost

Boosting is based on the following question: “Can a set of weak learners create a single strong learner?” A weak learner (or weak classifier) is defined as a classifier that is only slightly better than random guessing.

In face detection, this means that a weak learner can classify a subregion of an image as a face or not-face only slightly better than random guessing. A strong learner is substantially better at picking faces from non-faces.

The power of boosting comes from combining many (thousands) of weak classifiers into a single strong classifier. In the Viola-Jones algorithm, each Haar-like feature represents a weak learner. To decide the type and size of a feature that goes into the final classifier, AdaBoost checks the performance of all classifiers that you supply to it.

To calculate the performance of a classifier, you evaluate it on all subregions of all the images used for training. Some subregions will produce a strong response in the classifier. Those will be classified as positives, meaning the classifier thinks it contains a human face.

Subregions that don’t produce a strong response don’t contain a human face, in the classifiers opinion. They will be classified as negatives.

The classifiers that performed well are given higher importance or weight. The final result is a strong classifier, also called a boosted classifier, that contains the best performing weak classifiers.

The algorithm is called adaptive because, as training progresses, it gives more emphasis on those images that were incorrectly classified. The weak classifiers that perform better on these hard examples are weighted more strongly than others.

Let’s look at an example:

The blue and orange circles are samples that belong to different categories.

Imagine that you are supposed to classify blue and orange circles in the following image using a set of weak classifiers:

The first weak classifier classifies some of the blue circles correctly.

The first classifier you use captures some of the blue circles but misses the others. In the next iteration, you give more importance to the missed examples:

The missed blue samples are given more importance, indicated by size.

The second classifier that manages to correctly classify those examples will get a higher weight. Remember, if a weak classifier performs better, it will get a higher weight and thus higher chances to be included in the final, strong classifiers:

The second classifier captures the bigger blue circles.

Now you have managed to capture all of the blue circles, but incorrectly captured some of the orange circles. These incorrectly classified orange circles are given more importance in the next iteration:

The misclassified orange circles are given more importance, and others are reduced.

The final classifier manages to capture those orange circles correctly:

The third classifier captures the remaining orange circles.

To create a strong classifier, you combine all three classifiers to correctly classify all examples:

The final, strong classifier combines all three weak classifiers.

Using a variation of this process, Viola and Jones have evaluated hundreds of thousands of classifiers that specialize in finding faces in images. But it would be computationally expensive to run all these classifiers on every region in every image, so they created something called a classifier cascade.

Cascading Classifiers

The definition of a cascade is a series of waterfalls coming one after another. A similar concept is used in computer science to solve a complex problem with simple units. The problem here is reducing the number of computations for each image.

To solve it, Viola and Jones turned their strong classifier (consisting of thousands of weak classifiers) into a cascade where each weak classifier represents one stage. The job of the cascade is to quickly discard non-faces and avoid wasting precious time and computations.

When an image subregion enters the cascade, it is evaluated by the first stage. If that stage evaluates the subregion as positive, meaning that it thinks it’s a face, the output of the stage is maybe.

If a subregion gets a maybe, it is sent to the next stage of the cascade. If that one gives a positive evaluation, then that’s another maybe, and the image is sent to the third stage:

A weak classifier in a cascade

This process is repeated until the image passes through all stages of the cascade. If all classifiers approve the image, it is finally classified as a human face and is presented to the user as a detection.

If, however, the first stage gives a negative evaluation, then the image is immediately discarded as not containing a human face. If it passes the first stage but fails the second stage, it is discarded as well. Basically, the image can get discarded at any stage of the classifier:

A cascade of _n_ classifiers for face detection

This is designed so that non-faces get discarded very quickly, which saves a lot of time and computational resources. Since every classifier represents a feature of a human face, a positive detection basically says, “Yes, this subregion contains all the features of a human face.” But as soon as one feature is missing, it rejects the whole subregion.

To accomplish this effectively, it is important to put your best performing classifiers early in the cascade. In the Viola-Jones algorithm, the eyes and nose bridge classifiers are examples of best performing weak classifiers.

Now that you understand how the algorithm works, it is time to use it to detect faces with Python.

Using a Viola-Jones Classifier

Training a Viola-Jones classifier from scratch can take a long time. Fortunately, a pre-trained Viola-Jones classifier comes out-of-the-box with OpenCV! You will use that one to see the algorithm in action.

First, find and download an image that you would like to scan for the presence of human faces. Here’s an example:

Example stock photo for face detection (Image source)

Import OpenCV and load the image into memory:

import cv2 as cv

# Read image from your local file system
original_image = cv.imread('path/to/your-image.jpg')

# Convert color image to grayscale for Viola-Jones
grayscale_image = cv.cvtColor(original_image, cv.COLOR_BGR2GRAY)

Next, you need to load the Viola-Jones classifier. If you installed OpenCV from source, it will be in the folder where you installed the OpenCV library.

Depending on the version, the exact path might vary, but the folder name will be haarcascades, and it will contain multiple files. The one you need is called haarcascade_frontalface_alt.xml.

If for some reason, your installation of OpenCV did not get the pre-trained classifier, you can obtain it from the OpenCV GitHub repo:

# Load the classifier and create a cascade object for face detection
face_cascade = cv.CascadeClassifier('path/to/haarcascade_frontalface_alt.xml')

The face_cascade object has a method detectMultiScale(), which receives an image as an argument and runs the classifier cascade over the image. The term MultiScale indicates that the algorithm looks at subregions of the image in multiple scales, to detect faces of varying sizes:

detected_faces = face_cascade.detectMultiScale(grayscale_image)

The variable detected_faces now contains all the detections for the target image. To visualize the detections, you need to iterate over all detections and draw rectangles over the detected faces.

OpenCV’s rectangle() draws rectangles over images, and it needs to know the pixel coordinates of the top-left and bottom-right corner. The coordinates indicate the row and column of pixels in the image.

Luckily, detections are saved as pixel coordinates. Each detection is defined by its top-left corner coordinates and width and height of the rectangle that encompasses the detected face.

Adding the width to the row and height to the column will give you the bottom-right corner of the image:

for (column, row, width, height) in detected_faces:
    cv.rectangle(
        original_image,
        (column, row),
        (column + width, row + height),
        (0, 255, 0),
        2
    )

rectangle() accepts the following arguments:

The original image
The coordinates of the top-left point of the detection
The coordinates of the bottom-right point of the detection
The color of the rectangle (a tuple that defines the amount of red, green, and blue (0-255))
The thickness of the rectangle lines

Finally, you need to display the image:

cv.imshow('Image', original_image)
cv.waitKey(0)
cv.destroyAllWindows()

imshow() displays the image. waitKey() waits for a keystroke. Otherwise, imshow() would display the image and immediately close the window. Passing 0 as the argument tells it to wait indefinitely. Finally, destroyAllWindows() closes the window when you press a key.

Here’s the result:

Original image with detections

That’s it! You now have a ready-to-use face detector in Python.

If you really want to train the classifier yourself, scikit-image offers a tutorial with the accompanying code on their website.

Conclusion

Good work! You are now able to find faces in images.

In this tutorial, you have learned how to represent regions in an image with Haar-like features. These features can be calculated very quickly using integral images.

You have learned how AdaBoost finds the best performing Haar-like features from thousands of available features and turns them into a series of weak classifiers.

Finally, you have learned how to create a cascade of weak classifiers that can quickly and reliably distinguish faces from non-faces.

These steps illustrate many important elements of computer vision:

Finding useful features
Combining them to solve complex problems
Balancing between performance and managing computational resources

These ideas apply to object detection in general and will help you solve many real-world challenges. Good luck!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Ali Spittel

2019-02-25T14:00:00+00:00

Happy 2019! For our first community interview of the year, I’m joined by Ali Spittel.

Ali is a Pythonista with a passion for CSS art and teaching Python. Join us as we talk about her non-traditional path to learning to code, teaching at a Bootcamp, and her recent move to one of the most positive developer communities around.

Ricky: Welcome, Ali! Thank you for joining me for this interview. Let’s start with the usual first questions. How’d you get into programming? When did you start using Python?

Ali: Thanks for having me! I got into programming during my sophomore year of college. I randomly went to a computer science class because I had an extra credit hour for the semester.

It was a Python class, and I fell in love with it. I decided part way through the semester to try to double major in computer science. They even offered me a teaching assistant job for the next semester!

Then, the next semester, I took Data Structures and Algorithms in C++. I hated it, and I had to put so much work into the class. I decided coding wasn’t for me, and so I quit programming. I only got back into it because I was offered a paid programming internship and couldn’t find one in politics, the field I was interested in at the time!

Ricky: Until recently, you were the Lead Instructor for the Web Development Immersive program at General Assembly in Washington, D.C.

I’m curious to know what it’s like to teach at a coding Bootcamp and how it has changed as Bootcamps become more common and accepted as a non-traditional path to becoming a developer?

Ali: Teaching is one of the most rewarding things I’ve ever done. Seeing material click for students and seeing their growth is awesome. I also love building a curriculum and shaping how people see code at the beginning of their journeys.

I think Bootcamps are so important for increasing tech diversity, and there are so many great programs within Bootcamps that serve people from economically disadvantaged or underrepresented demographic groups in tech.

For my last 9 months working at GA, I worked for their enterprise program, where I was teaching onsite for companies who were training employees.

I love that those companies were really investing in their employees and that those students could really focus on their education instead of having to worry about the financial aspects of doing a coding Bootcamp.

I’m actually still teaching their Python part-time program in D.C. even though I don’t teach full-time anymore!

Ricky: Teaching seems to be something you enjoy doing, between working as a Teaching Assistant in college and working at the General Assembly. Last year, you expanded on that and started a blog that has become really popular. What made you decide to start the blog, and has success you’ve had early on with it surprised you?

Ali: I started writing pretty randomly. I had an idea for a CSS art blog post, and then decided I could go further with it and write a series about learning new things. I started out on Medium, where I got next to no traction, and then found out about DEV.to a little later on, where more people were reading my writing.

I fell out of writing for a few months because I was getting more and more speaking roles, and I was more focused on that.

Then this summer I moved to Connecticut for work, and I didn’t know a lot of people. I had a concept for a blog that was a hybrid of a programming blog and a lifestyle blog. Those posts started to really gain some traction, which was awesome.

Ricky: You’re also a member of the D.C. meetup circuit, and most notably, the D.C. chapter of Women Who Code, which now has almost 9000 members! How is the meetup scene in D.C., particularly for women and minorities? What advice would you give to someone who is looking to go to their first meetup?

Ali: The meetup scene in DC is great! There’s an event pretty much every night (usually multiple), and I always learn so much from them and meet great people.

My advice for someone going to a meetup for the first time would be to bring a friend if you’re nervous the first time! I still pretty much always make sure I know somebody before heading to a meetup. It just makes it easier when I have at least one person to talk to.

Ricky: When we first met on Twitter, you were posting daily code challenges from code wars, and then solving them in Python. It was great to see so much participation from others, as they compared answers in various languages. Are you still doing those? What did you get out of doing something like that with the community?

Ali: I loved doing the code challenges on Twitter! I did them for six months, I think, and would love to get back into them.

Unfortunately, my life is a little busy right now between a new puppy and a new job! I may be able to start putting them up again soon though. I have a lot of fun with them.

I did them for a few reasons: first to prove that I can actually code since I have had people doubt that, and second because they’re fun to do. Hopefully they’re educational to some people!

Ricky: You just alluded to your new job. Why don’t you tell us more about your exciting news and the reason for the change?

Ali: I just started a new job last week! I’m now working as a frontend developer and doing developer advocacy work for DEV.to.

I’m going to be working on their open source codebase, which will be awesome. On top of that, I’ll be doing more speaking and writing for them, as well as developer community outreach!

I’m so excited to work for them because they’re such an awesome community. Their tagline is Where programmers share ideas and help each other grow.

It is an online community for sharing and discovering great ideas, having debates, and making friends. It’s definitely been that way for me as a community member. It was a huge part of how I gained a reader-base for my blog—their users are so supportive. I’m so excited to be a bigger part of the community!

Ricky: Now for my last question. What else do you get up to in your spare time? What other hobbies and interests do you have, aside from Python and coding?

Ali: In my spare time, I rock climb, check out the awesome food and bar scene in D.C., and hang out with my puppy!

Thank you Ali for taking the time to talk to me, and good luck in your new role! To keep up with Ali, you can check out her blog, follow her on Twitter, or check out her awesome content on DEV.to.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

Working With JSON Data in Python

How to Stand Out in a Python Coding Interview

Select the Right Built-In Function for the Job

Iterate With enumerate() Instead of range()

Use List Comprehensions Instead of map() and filter()

Debug With breakpoint() Instead of print()

Format strings With f-Strings

Sort Complex Lists With sorted()

Leverage Data Structures Effectively

Store Unique Values With Sets

Save Memory With Generators

Define Default Values in Dictionaries With .get() and .setdefault()

Take Advantage of Python’s Standard Library

Handle Missing Dictionary Keys With collections.defaultdict()

Count Hashable Objects With collections.Counter

Access Common String Groups With string Constants

Generate Permutations and Combinations With itertools

Conclusion: Coding Interview Superpowers

Django Migrations 101

Documenting Python Projects With Sphinx and Read The Docs

Resources

An Intro to Threading in Python

What Is a Thread?

Starting a Thread

Daemon Threads

join() a Thread

Working With Many Threads

Using a ThreadPoolExecutor

Race Conditions

One Thread

Two Threads

Why This Isn’t a Silly Example

Basic Synchronization Using Lock

Deadlock

Producer-Consumer Threading

Producer-Consumer Using Lock

Producer-Consumer Using Queue

Threading Objects

Semaphore

Timer

Barrier

Conclusion: Threading in Python

Intro to Object-Oriented Programming (OOP) in Python

13 Project Ideas for Intermediate Python Developers

The Importance of Building Projects

Choosing a Project Platform

Web

Desktop GUI

Command-Line

Web Project Ideas

Content Aggregator

Regex Query Tool

URL Shortener

Post-It Note

Quiz Application

GUI Project Ideas

MP3 Player

Alarm Tool

File Manager

Expense Tracker

Command-Line Project Ideas

Contact Book

Site Connectivity Checker

Bulk File Rename Tool

Directory Tree Generator

Tips for Working on Projects

Conclusion

Python Decorators 101

How to Build a Python GUI Application With wxPython

Getting Started With wxPython

Installing wxPython

Definition of a GUI

Event Loops

Creating a Skeleton Application

Widgets

Absolute Positioning

Sizers (Dynamic Sizing)

Adding an Event

Creating a Working Application

Iterate With `enumerate()` Instead of `range()`

Use List Comprehensions Instead of `map()` and `filter()`

Debug With `breakpoint()` Instead of `print()`

Sort Complex Lists With `sorted()`

Define Default Values in Dictionaries With `.get()` and `.setdefault()`

Handle Missing Dictionary Keys With `collections.defaultdict()`

Count Hashable Objects With `collections.Counter`

Access Common String Groups With `string` Constants

Generate Permutations and Combinations With `itertools`

`join()` a Thread

Using a `ThreadPoolExecutor`

Basic Synchronization Using `Lock`

Producer-Consumer Using `Lock`

Producer-Consumer Using `Queue`

The `Mock` Object

`patch()`

`patch()` as a Decorator

`patch()` as a Context Manager

Why Use `pyenv`?

Installing `pyenv`

Using `pyenv` to Install Python

Exploring `pyenv` Commands

`install`

`versions`

`which`

`global`

`local`

`shell`

Virtual Environments and `pyenv`