Real Python

Linked Lists in Python: An Introduction

2020-04-01T14:00:00+00:00

Linked lists are like a lesser-known cousin of lists. They’re not as popular or as cool, and you might not even remember them from your algorithms class. But in the right context, they can really shine.

In this article, you’ll learn:

What linked lists are and when you should use them
How to use collections.deque for all of your linked list needs
How to implement your own linked lists
What the other types of linked lists are and what they can be used for

If you’re looking to brush up on your coding skills for a job interview, or if you want to learn more about Python data structures besides the usual dictionaries and lists, then you’ve come to the right place!

You can follow along with the examples in this tutorial by downloading the source code available at the link below:

Get the Source Code: Click here to get the source code you'll use to learn about linked lists in this tutorial.

Understanding Linked Lists

Linked lists are an ordered collection of objects. So what makes them different from normal lists? Linked lists differ from lists in the way that they store elements in memory. While lists use a contiguous memory block to store references to their data, linked lists store references as part of their own elements.

Main Concepts

Before going more in depth on what linked lists are and how you can use them, you should first learn how they are structured. Each element of a linked list is called a node, and every node has two different fields:

Data contains the value to be stored in the node.
Next contains a reference to the next node on the list.

Here’s what a typical node looks like:

Node

A linked list is a collection of nodes. The first node is called the head, and it’s used as the starting point for any iteration through the list. The last node must have its next reference pointing to None to determine the end of the list. Here’s how it looks:

Linked List

Now that you know how a linked list is structured, you’re ready to look at some practical use cases for it.

Practical Applications

Linked lists serve a variety of purposes in the real world. They can be used to implement (spoiler alert!) queues or stacks as well as graphs. They’re also useful for much more complex tasks, such as lifecycle management for an operating system application.

Queues or Stacks

Queues and stacks differ only in the way elements are retrieved. For a queue, you use a First-In/First-Out (FIFO) approach. That means that the first element inserted in the list is the first one to be retrieved:

Queue

In the diagram above, you can see the front and rear elements of the queue. When you append new elements to the queue, they’ll go to the rear end. When you retrieve elements, they’ll be taken from the front of the queue.

For a stack, you use a Last-In/Fist-Out (LIFO) approach, meaning that the last element inserted in the list is the first to be retrieved:

Stack

In the above diagram you can see that the first element inserted on the stack (index 0) is at the bottom, and the last element inserted is at the top. Since stacks use the LIFO approach, the last element inserted (at the top) will be the first to be retrieved.

Because of the way you insert and retrieve elements from the edges of queues and stacks, linked lists are one of the most convenient ways to implement these data structures. You’ll see examples of these implementations later in the article.

Graphs

Graphs can be used to show relationships between objects or to represent different types of networks. For example, a visual representation of a graph—say a directed acyclic graph (DAG)—might look like this:

Directed Acyclic Graph

There are different ways to implement graphs like the above, but one of the most common is to use an adjacency list. An adjacency list is, in essence, a list of linked lists where each vertex of the graph is stored alongside a collection of connected vertices:

Vertex	Linked List of Vertices
1	2 → 3 → None
2	4 → None
3	None
4	5 → 6 → None
5	6 → None
6	None

In the table above, each vertex of your graph is listed in the left column. The right column contains a series of linked lists storing the other vertices connected with the corresponding vertex in the left column. This adjacency list could also be represented in code using a dict:

>>>

>>> graph = {
...     1: [2, 3, None],
...     2: [4, None],
...     3: [None],
...     4: [5, 6, None],
...     5: [6, None],
...     6: [None]
... }

The keys of this dictionary are the source vertices, and the value for each key is a list. This list is usually implemented as a linked list.

Note: In the above example you could avoid storing the None values, but we’ve retained them here for clarity and consistency with later examples.

In terms of both speed and memory, implementing graphs using adjacency lists is very efficient in comparison with, for example, an adjacency matrix. That’s why linked lists are so useful for graph implementation.

Performance Comparison: Lists vs Linked Lists

In most programming languages, there are clear differences in the way linked lists and arrays are stored in memory. In Python, however, lists are dynamic arrays. That means that the memory usage of both lists and linked lists is very similar.

Further reading: Python’s implementation of dynamic arrays is quite interesting and definitely worth a read. Make sure to have a look and use that knowledge to stand out at your next company party!

Since the difference in memory usage between lists and linked lists is so insignificant, it’s better if you focus on their performance differences when it comes to time complexity.

Insertion and Deletion of Elements

List insertion times vary depending on the need to resize the existing array. If you read the article above on how lists are implemented, then you’ll notice that Python overallocates space for arrays based on their growth. This means that inserting a new element into a list can take anywhere from Θ(n) to Θ(1) depending on whether the existing array needs to be resized.

The same applies for deleting elements from the array. If the size of the array is less than half the allocated size, then Python will shrink the array. So, all in all, inserting or deleting elements from an array has an average time complexity of Θ(n).

Linked lists, however, are much better when it comes to insertion and deletion of elements at the beginning or the end of the list, where their time complexity is constant: Θ(1). This performance advantage is the reason linked lists are so useful for queues and stacks, where elements are continuously inserted and removed from the edges.

Retrieval of Elements

When it comes to element lookup, lists perform much better than linked lists. When you know which element you want to access, lists can perform this operation in Θ(1) time. Trying to do the same with a linked list would take Θ(n) because you need to traverse the whole list to find the element.

When searching for a specific element, however, both lists and linked lists perform very similarly, with a time complexity of Θ(n). In both cases, you need to iterate through the entire list to find the element you’re looking for.

Introducing `collections.deque`

In Python, there’s a specific object in the collections module that you can use for linked lists called deque(pronounced “deck”), which stands for double-ended queue.

collections.deque uses an implementation of a linked list in which you can access, insert, or remove elements from the beginning or end of a list with constant O(1) performance.

How to Use `collections.deque`

There are quite a few methods that come, by default, with a deque object. However, in this article you’ll only touch on a few of them, mostly for adding or removing elements.

First, you need to create a linked list. You can use the following piece of code to do that with deque:

>>>

>>> from collections import deque
>>> deque()
deque([])

The code above will create an empty linked list. If you want to populate it at creation, then you can give it an iterable as input:

>>>

>>> deque(['a','b','c'])
deque(['a', 'b', 'c'])

>>> deque('abc')
deque(['a', 'b', 'c'])

>>> deque([{'data': 'a'}, {'data': 'b'}])
deque([{'data': 'a'}, {'data': 'b'}])

When initializing a deque object, you can pass any iterable as an input, such as a string (also an iterable) or a list of objects.

Now that you know how to create a deque object, you can interact with it by adding or removing elements. You can create an abcde linked list and add a new element f like this:

>>>

>>> llist = deque("abcde")
>>> llist
deque(['a', 'b', 'c', 'd', 'e'])

>>> llist.append("f")
>>> llist
deque(['a', 'b', 'c', 'd', 'e', 'f'])

>>> llist.pop()
'f'

>>> llist
deque(['a', 'b', 'c', 'd', 'e'])

Both append() and pop() add or remove elements from the right side of the linked list. However, you can also use deque to quickly add or remove elements from the left side, or head, of the list:

>>>

>>> llist.appendleft("z")
>>> llist
deque(['z', 'a', 'b', 'c', 'd', 'e'])

>>> llist.popleft()
'z'

>>> llist
deque(['a', 'b', 'c', 'd', 'e'])

Adding or removing elements from both ends of the list is pretty straightforward using the deque object. Now you’re ready to learn how to use collections.deque to implement a queue or a stack.

How to Implement Queues and Stacks

As you learned above, the main difference between a queue and a stack is the way you retrieve elements from each. Next, you’ll find out how to use collections.deque to implement both data structures.

Queues

With queues, you want to add values to a list (enqueue), and when the timing is right, you want to remove the element that has been on the list the longest (dequeue). For example, imagine a queue at a trendy and fully booked restaurant. If you were trying to implement a fair system for seating guests, then you’d start by creating a queue and adding people as they arrive:

>>>

>>> from collections import deque
>>> queue = deque()
>>> queue
deque([])

>>> queue.append("Mary")
>>> queue.append("John")
>>> queue.append("Susan")
>>> queue
deque(['Mary', 'John', 'Susan'])

Now you have Mary, John, and Susan in the queue. Remember that since queues are FIFO, the first person who got into the queue should be the first to get out.

Now imagine some time goes by and a few tables become available. At this stage, you want to remove people from the queue in the correct order. This is how you would do that:

>>>

>>> queue.popleft()
'Mary'

>>> queue
deque(['John', 'Susan'])

>>> queue.popleft()
'John'

>>> queue
deque(['Susan'])

Every time you call popleft(), you remove the head element from the linked list, mimicking a real-life queue.

Stacks

What if you wanted to create a stack instead? Well, the idea is more or less the same as with the queue. The only difference is that the stack uses the LIFO approach, meaning that the last element to be inserted in the stack should be the first to be removed.

Imagine you’re creating a web browser’s history functionality in which store every page a user visits so they can go back in time easily. Assume these are the actions a random user takes on their browser:

Visits Real Python’s website
Navigates to Pandas: How to Read and Write Files
Clicks on a link for Reading and Writing CSV Files in Python

If you’d like to map this behavior into a stack, then you could do something like this:

>>>

>>> from collections import deque
>>> history = deque()

>>> history.appendleft("https://realpython.com/")
>>> history.appendleft("https://realpython.com/pandas-read-write-files/")
>>> history.appendleft("https://realpython.com/python-csv/")
>>> history
deque(['https://realpython.com/python-csv/',
       'https://realpython.com/pandas-read-write-files/',
       'https://realpython.com/'])

In this example, you created an empty history object, and every time the user visited a new site, you added it to your history variable using appendleft(). Doing so ensured that each new element was added to the head of the linked list.

Now suppose that after the user read both articles, they wanted to go back to the Real Python home page to pick a new article to read. Knowing that you have a stack and want to remove elements using LIFO, you could do the following:

>>>

>>> history.popleft()
'https://realpython.com/python-csv/'

>>> history.popleft()
'https://realpython.com/pandas-read-write-files/'

>>> history
deque(['https://realpython.com/'])

There you go! Using popleft(), you removed elements from the head of the linked list until you reached the Real Python home page.

From the examples above, you can see how useful it can be to have collections.deque in your toolbox, so make sure to use it the next time you have a queue- or stack-based challenge to solve.

Implementing Your Own Linked List

Now that you know how to use collections.deque for handling linked lists, you might be wondering why you would ever implement your own linked list in Python. There are a few reasons to do it:

Practicing your Python algorithm skills
Learning about data structure theory
Preparing for job interviews

Feel free to skip this next section if you’re not interested in any of the above, or if you already aced implementing your own linked list in Python. Otherwise, it’s time to implement some linked lists!

How to Create a Linked List

First things first, create a class to represent your linked list:

class LinkedList:
    def __init__(self):
        self.head = None

The only information you need to store for a linked list is where the list starts (the head of the list). Next, create another class to represent each node of the linked list:

class Node:
    def __init__(self, data):
        self.data = data
        self.next = None

In the above class definition, you can see the two main elements of every single node: data and next. You can also add a __repr__ to both classes to have a more helpful representation of the objects:

class Node:
    def __init__(self, data):
        self.data = data
        self.next = None

    def __repr__(self):
        return self.data

class LinkedList:
    def __init__(self):
        self.head = None

    def __repr__(self):
        node = self.head
        nodes = []
        while node is not None:
            nodes.append(node.data)
            node = node.next
        nodes.append("None")
        return " -> ".join(nodes)

Have a look at an example of using the above classes to quickly create a linked list with three nodes:

>>>

>>> llist = LinkedList()
>>> llist
None

>>> first_node = Node("a")
>>> llist.head = first_node
>>> llist
a -> None

>>> second_node = Node("b")
>>> third_node = Node("c")
>>> first_node.next = second_node
>>> second_node.next = third_node
>>> llist
a -> b -> c -> None

By defining a node’s data and next values, you can create a linked list quite quickly. These LinkedList and Node classes are the starting points for our implementation. From now on, it’s all about increasing their functionality.

Here’s a slight change to the linked list’s __init__() that allows you to quickly create linked lists with some data:

def __init__(self, nodes=None):
    self.head = None
    if nodes is not None:
        node = Node(data=nodes.pop(0))
        self.head = node
        for elem in nodes:
            node.next = Node(data=elem)
            node = node.next

With the above modification, creating linked lists to use in the examples below will be much faster.

How to Traverse a Linked List

One of the most common things you will do with a linked list is to traverse it. Traversing means going through every single node, starting with the head of the linked list and ending on the node that has a next value of None.

Traversing is just a fancier way to say iterating. So, with that in mind, create an __iter__ to add the same behavior to linked lists that you would expect from a normal list:

def __iter__(self):
    node = self.head
    while node is not None:
        yield node
        node = node.next

The method above goes through the list and yields every single node. The most important thing to remember about this __iter__ is that you need to always validate that the current node is not None. When that condition is True, it means you’ve reached the end of your linked list.

After yielding the current node, you want to move to the next node on the list. That’s why you add node = node.next. Here’s an example of traversing a random list and printing each node:

>>>

>>> llist = LinkedList(["a", "b", "c", "d", "e"])
>>> llist
a -> b -> c -> d -> e -> None

>>> for node in llist:
...     print(node)
a
b
c
d
e

In other articles, you might see the traversing defined into a specific method called traverse(). However, using Python’s built-in methods to achieve said behavior makes this linked list implementation a bit more Pythonic.

How to Insert a New Node

There are different ways to insert new nodes into a linked list, each with its own implementation and level of complexity. That’s why you’ll see them split into specific methods for inserting at the beginning, end, or between nodes of a list.

Inserting at the Beginning

Inserting a new node at the beginning of a list is probably the most straightforward insertion since you don’t have to traverse the whole list to do it. It’s all about creating a new node and then pointing the head of the list to it.

Have a look at the following implementation of add_first() for the class LinkedList:

def add_first(self, node):
    node.next = self.head
    self.head = node

In the above example, you’re setting self.head as the next reference of the new node so that the new node points to the old self.head. After that, you need to state that the new head of the list is the inserted node.

Here’s how it behaves with a sample list:

>>>

>>> llist = LinkedList()
>>> llist
None

>>> llist.add_first(Node("b"))
>>> llist
b -> None

>>> llist.add_first(Node("a"))
>>> llist
a -> b -> None

As you can see, add_first() always adds the node to the head of the list, even if the list was empty before.

Inserting at the End

Inserting a new node at the end of the list forces you to traverse the whole linked list first and to add the new node when you reach the end. You can’t just append to the end as you would with a normal list because in a linked list you don’t know which node is last.

Here’s an example implementation of a function for inserting a node to the end of a linked list:

def add_last(self, node):
    if not self.head:
        self.head = node
        return
    for current_node in self:
        pass
    current_node.next = node

First, you want to traverse the whole list until you reach the end (that is, until the for loop raises a StopIteration exception). Next, you want to set the current_node as the last node on the list. Finally, you want to add the new node as the next value of that current_node.

Here’s an example of add_last() in action:

>>>

>>> llist = LinkedList(["a", "b", "c", "d"])
>>> llist
a -> b -> c -> d -> None

>>> llist.add_last(Node("e"))
>>> llist
a -> b -> c -> d -> e -> None

>>> llist.add_last(Node("f"))
>>> llist
a -> b -> c -> d -> e -> f -> None

In the code above, you start by creating a list with four values (a, b, c, and d). Then, when you add new nodes using add_last(), you can see that the nodes are always appended to the end of the list.

Inserting Between Two Nodes

Inserting between two nodes adds yet another layer of complexity to the linked list’s already complex insertions because there are two different approaches that you can use:

Inserting after an existing node
Inserting before an existing node

It might seem weird to split these into two methods, but linked lists behave differently than normal lists, and you need a different implementation for each case.

Here’s a method that adds a node after an existing node with a specific data value:

def add_after(self, target_node_data, new_node):
    if not self.head:
        raise Exception("List is empty")

    for node in self:
        if node.data == target_node_data:
            new_node.next = node.next
            node.next = new_node
            return

    raise Exception("Node with data '%s' not found" % target_node_data)

In the above code, you’re traversing the linked list looking for the node with data indicating where you want to insert a new node. When you find the node you’re looking for, you’ll insert the new node immediately after it and rewire the next reference to maintain the consistency of the list.

The only exceptions are if the list is empty, making it impossible to insert a new node after an existing node, or if the list does not contain the value you’re searching for. Here are a few examples of how add_after() behaves:

>>>

>>> llist = LinkedList()
>>> llist.add_after("a", Node("b"))
Exception: List is empty

>>> llist = LinkedList(["a", "b", "c", "d"])
>>> llist
a -> b -> c -> d -> None

>>> llist.add_after("c", Node("cc"))
>>> llist
a -> b -> c -> cc -> d -> None

>>> llist.add_after("f", Node("g"))
Exception: Node with data 'f' not found

Trying to use add_after() on an empty list results in an exception. The same happens when you try to add after a nonexistent node. Everything else works as expected.

Now, if you want to implement add_before(), then it will look something like this:

 1 def add_before(self, target_node_data, new_node):
 2     if not self.head:
 3         raise Exception("List is empty")
 4 
 5     if self.head.data == target_node_data:
 6         return self.add_first(new_node)
 7 
 8     prev_node = self.head
 9     for node in self:
10         if node.data == target_node_data:
11             prev_node.next = new_node
12             new_node.next = node
13             return
14         prev_node = node
15 
16     raise Exception("Node with data '%s' not found" % target_node_data)

There are a few things to keep in mind while implementing the above. First, as with add_after(), you want to make sure to raise an exception if the linked list is empty (line 2) or the node you’re looking for is not present (line 16).

Second, if you’re trying to add a new node before the head of the list (line 5), then you can reuse add_first() because the node you’re inserting will be the new head of the list.

Finally, for any other case (line 9), you should keep track of the last-checked node using the prev_node variable. Then, when you find the target node, you can use that prev_node variable to rewire the next values.

Once again, an example is worth a thousand words:

>>>

>>> llist = LinkedList()
>>> llist.add_before("a", Node("a"))
Exception: List is empty

>>> llist = LinkedList(["b", "c"])
>>> llist
b -> c -> None

>>> llist.add_before("b", Node("a"))
>>> llist
a -> b -> c -> None

>>> llist.add_before("b", Node("aa"))
>>> llist.add_before("c", Node("bb"))
>>> llist
a -> aa -> b -> bb -> c -> None

>>> llist.add_before("n", Node("m"))
Exception: Node with data 'n' not found

With add_before(), you now have all the methods you need to insert nodes anywhere you’d like in your list.

How to Remove a Node

To remove a node from a linked list, you first need to traverse the list until you find the node you want to remove. Once you find the target, you want to link its previous and next nodes. This re-linking is what removes the target node from the list.

That means you need to keep track of the previous node as you traverse the list. Have a look at an example implementation:

 1 def remove_node(self, target_node_data):
 2     if not self.head:
 3         raise Exception("List is empty")
 4 
 5     if self.head.data == target_node_data:
 6         self.head = self.head.next
 7         return
 8 
 9     previous_node = self.head
10     for node in self:
11         if node.data == target_node_data:
12             previous_node.next = node.next
13             return
14         previous_node = node
15 
16     raise Exception("Node with data '%s' not found" % target_node_data)

In the above code, you first check that your list is not empty (line 2). If it is, then you raise an exception. After that, you check if the node to be removed is the current head of the list (line 5). If it is, then you want the next node in the list to become the new head.

If none of the above happens, then you start traversing the list looking for the node to be removed (line 10). If you find it, then you need to update its previous node to point to its next node, automatically removing the found node from the list. Finally, if you traverse the whole list without finding the node to be removed (line 16), then you raise an exception.

Notice how in the above code you use previous_node to keep track of the, well, previous node. Doing so ensures that the whole process will be much more straightforward when you find the right node to be deleted.

Here’s an example using a list:

>>>

>>> llist = LinkedList()
>>> llist.remove_node("a")
Exception: List is empty

>>> llist = LinkedList(["a", "b", "c", "d", "e"])
>>> llist
a -> b -> c -> d -> e -> None

>>> llist.remove_node("a")
>>> llist
b -> c -> d -> e -> None

>>> llist.remove_node("e")
>>> llist
b -> c -> d -> None

>>> llist.remove_node("c")
>>> llist
b -> d -> None

>>> llist.remove_node("a")
Exception: Node with data 'a' not found

That’s it! You now know how to implement a linked list and all of the main methods for traversing, inserting, and removing nodes. If you feel comfortable with what you’ve learned and you’re craving more, then feel free to pick one of the challenges below:

Create a method to retrieve an element from a specific position: get(i) or even llist[i].
Create a method to reverse the linked list: llist.reverse().
Create a Queue() object inheriting this article’s linked list with enqueue() and dequeue() methods.

Apart from being great practice, doing some extra challenges on your own is an effective way to assimilate all the knowledge you’ve gained. If you want to get a head start by reusing all the source code from this article, then you can download everything you need at the link below:

Get the Source Code: Click here to get the source code you'll use to learn about linked lists in this tutorial.

Using Advanced Linked Lists

Until now, you’ve been learning about a specific type of linked list called singly linked lists. But there are more types of linked lists that can be used for slightly different purposes.

How to Use Doubly Linked Lists

Doubly linked lists are different from singly linked lists in that they have two references:

The previous field references the previous node.
The next field references the next node.

The end result looks like this:

Node (Doubly Linked List)

If you wanted to implement the above, then you could make some changes to your existing Node class in order to include a previous field:

class Node:
    def __init__(self, data):
        self.data = data
        self.next = None
        self.previous = None

This kind of implementation would allow you to traverse a list in both directions instead of only traversing using next. You could use next to go forward and previous to go backward.

In terms of structure, this is how a doubly linked list would look:

Doubly Linked List

You learned earlier that collections.deque uses a linked list as part of its data structure. This is the kind of linked list it uses. With doubly linked lists, deque is capable of inserting or deleting elements from both ends of a queue with constant O(1) performance.

How to Use Circular Linked Lists

Circular linked lists are a type of linked list in which the last node points back to the head of the list instead of pointing to None. This is what makes them circular. Circular linked lists have quite a few interesting use cases:

Going around each player’s turn in a multiplayer game
Managing the application life cycle of a given operating system
Implementing a Fibonacci heap

This is what a circular linked list looks like:

Circular Linked List

One of the advantages of circular linked lists is that you can traverse the whole list starting at any node. Since the last node points to the head of the list, you need to make sure that you stop traversing when you reach the starting point. Otherwise, you’ll end up in an infinite loop.

In terms of implementation, circular linked lists are very similar to singly linked list. The only difference is that you can define the starting point when you traverse the list:

class CircularLinkedList:
    def __init__(self):
        self.head = None

    def traverse(self, starting_point=None):
        if starting_point is None:
            starting_point = self.head
        node = starting_point
        while node is not None and (node.next != starting_point):
            yield node
            node = node.next
        yield node

    def print_list(self, starting_point=None):
        nodes = []
        for node in self.traverse(starting_point):
            nodes.append(str(node))
        print(" -> ".join(nodes))

Traversing the list now receives an additional argument, starting_point, that is used to define the start and (because the list is circular) the end of the iteration process. Apart from that, much of the code is the same as what we had in our LinkedList class.

To wrap up with a final example, have a look at how this new type of list behaves when you give it some data:

>>>

>>> circular_llist = CircularLinkedList()
>>> circular_llist.print_list()
None

>>> a = Node("a")
>>> b = Node("b")
>>> c = Node("c")
>>> d = Node("d")
>>> a.next = b
>>> b.next = c
>>> c.next = d
>>> d.next = a
>>> circular_llist.head = a
>>> circular_llist.print_list()
a -> b -> c -> d

>>> circular_llist.print_list(b)
b -> c -> d -> a

>>> circular_llist.print_list(d)
d -> a -> b -> c

There you have it! You’ll notice that you no longer have the None while traversing the list. That’s because there is no specific end to a circular list. You can also see that choosing different starting nodes will render slightly different representations of the same list.

Conclusion

In this article, you learned quite a few things! The most important are:

What linked lists are and when you should use them
How to use collections.deque to implement queues and stacks
How to implement your own linked list and node classes, plus relevant methods
What the other types of linked lists are and what they can be used for

If you want to learn more about linked lists, then check out Vaidehi Joshi’s Medium post for a nice visual explanation. If you’re interested in a more in-depth guide, then the Wikipedia article is quite thorough. Finally, if you’re curious about the reasoning behind the current implementation of collections.deque, then check out Raymond Hettinger’s thread.

You can download the source code used throughout this tutorial by clicking on the following link:

Get the Source Code: Click here to get the source code you'll use to learn about linked lists in this tutorial.

Feel free to leave any questions or comments below. Happy Pythoning!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Comparing Python Objects the Right Way: "is" vs "=="

2020-03-31T14:00:00+00:00

There’s a subtle difference between the Python identity operator (is) and the equality operator (==). Your code can run fine when you use the Python is operator to compare numbers, until it suddenly doesn’t. You might have heard somewhere that the Python is operator is faster than the == operator, or you may feel that it looks more Pythonic. However, it’s crucial to keep in mind that these operators don’t behave quite the same.

The == operator compares the value or equality of two objects, whereas the Python is operator checks whether two variables point to the same object in memory. In the vast majority of cases, this means you should use the equality operators == and !=, except when you’re comparing to None.

In this course, you’ll learn:

What the difference is between object equality and identity
When to use equality and identity operators to compare objects
What these Python operators do under the hood
Why using is and is not to compare values leads to unexpected behavior
How to write a custom __eq__() class method to define equality operator behavior

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Use any() in Python

2020-03-30T14:00:00+00:00

As a Python programmer, you’ll frequently deal with Booleans and conditional statements—sometimes very complex ones. In those situations, you may need to rely on tools that can simplify logic and consolidate information. Fortunately, any() in Python is such a tool. It looks through the elements in an iterable and returns a single value indicating whether any element is true in a Boolean context, or truthy.

In this tutorial, you’ll learn:

How to use any()
How to decide between any() and or

Let’s dive right in!

Python Pit Stop: This tutorial is a quick and practical way to find the info you need, so you’ll be back to your project in no time!

Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

How to Use `any()` in Python

Imagine that you’re writing a program for your employer’s recruiting department. You might want to schedule interviews with candidates who meet any of the following criteria:

Know Python already
Have five or more years of developer experience
Have a degree

One tool you could use to write this conditional expression is or:

# recruit_developer.py
def schedule_interview(applicant):
    print(f"Scheduled interview with {applicant['name']}")

applicants = [
    {
        "name": "Devon Smith",
        "programming_languages": ["c++", "ada"],
        "years_of_experience": 1,
        "has_degree": False,
        "email_address": "devon@email.com",
    },
    {
        "name": "Susan Jones",
        "programming_languages": ["python", "javascript"],
        "years_of_experience": 2,
        "has_degree": False,
        "email_address": "susan@email.com",
    },
    {
        "name": "Sam Hughes",
        "programming_languages": ["java"],
        "years_of_experience": 4,
        "has_degree": True,
        "email_address": "sam@email.com",
    },
]
for applicant in applicants:
    knows_python = "python" in applicant["programming_languages"]
    experienced_dev = applicant["years_of_experience"] >= 5

    meets_criteria = (
        knows_python
        or experienced_dev
        or applicant["has_degree"]
    )
    if meets_criteria:
        schedule_interview(applicant)

In the above example, you check each applicant’s credentials and schedule an interview if the applicant meets any of your three criteria.

Technical Detail: Python’s any() and or aren’t limited to evaluating Boolean expressions. Instead, Python performs a truth value test on each argument, evaluating whether the expression is truthy or falsy. For example, nonzero integer values are considered truthy and zero is considered falsy:

>>>

>>> 1 or 0
1

In this example, or evaluated the nonzero value 1 as truthy even though it’s not of type Boolean. or returned 1 and didn’t need to evaluate the truthiness of 0. Later in this tutorial, you’ll learn more about the return value and argument evaluation of or.

If you execute this code, then you’ll see that Susan and Sam will get interviews:

$ python recruit_developer.py
Scheduled interview with Susan Jones
Scheduled interview with Sam Hughes

The reason the program chose to schedule interviews with Susan and Sam is that Susan already knows Python and Sam has a degree. Notice each candidate only needed to meet one criterion.

Another way to evaluate the applicants’ credentials is to use any(). When you use any() in Python, you must pass the applicants’ credentials as an iterable argument:

for applicant in applicants:
    knows_python = "python" in applicant["programming_languages"]
    experienced_dev = applicant["years_of_experience"] >= 5

    credentials = (
        knows_python,
        experienced_dev,
        applicant["has_degree"],
    )
    if any(credentials):
        schedule_interview(applicant)

When you use any() in Python, keep in mind that you can pass any iterable as an argument:

>>>

>>> any([0, 0, 1, 0])
True

>>> any(set((True, False, True)))
True

>>> any(map(str.isdigit, "hello world"))
False

In each example, any() loops through a different Python iterable, testing the truth of each element until it finds a truthy value or checks every element.

Note: The last example uses Python’s built-in map(), which returns an iterator in which every element is the result of passing the next character in the string to str.isdigit(). This is a useful way to use any() for more complex checks.

You may be wondering if any() is merely a dressed-up version of or. In the next section, you’ll learn the differences between these tools.

How to Distinguish Between `or` and `any()`

There are two main differences between or and any() in Python:

Syntax
Return value

First, you’ll learn about how syntax affects the usability and reliability of each tool. Second, you’ll learn the types of values that each tool returns. Knowing these differences will help you decide which tool is best for a given situation.

Syntax

or is an operator, so it takes two arguments, one on either side:

>>>

>>> True or False
True

any(), on the other hand, is a function that takes one argument, an iterable of objects that it loops through to evaluate truthiness:

>>>

>>> any((False, True))
True

This difference in syntax is significant because it affects each tool’s usability and readability. For example, if you have an iterable, then you can pass the iterable directly to any(). To get similar behavior from or, you’d need to use a loop or a function like reduce():

>>>

>>> import functools
>>> functools.reduce(lambda x, y: x or y, (True, False, False))
True

In the above example, you used reduce() to pass an iterable as an argument to or. This could be done much more efficiently with any, which directly accepts iterables as arguments.

To illustrate another way that the syntax of each tool affects its usability, imagine that you want to avoid testing a condition if any preceding condition is True:

def knows_python(applicant):
    print(f"Determining if {applicant['name']} knows Python...")
    return "python" in applicant["programming_languages"]

def is_local(applicant):
    print(f"Determine if {applicant['name']} lives near the office...")

should_interview = knows_python(applicant) or is_local(applicant)

If is_local() takes a relatively long time to execute, then you don’t want to call it when knows_python() has already returned True. This is called lazy evaluation, or short-circuit evaluation. By default, or evaluates conditions lazily, whereas any does not.

In the above example, the program wouldn’t even need to determine if Susan is local because it already confirmed that she knows Python. That’s good enough to schedule an interview. In this situation, calling functions lazily with or would be the most efficient approach.

Why not use any() instead? You learned above that any() takes an iterable as an argument, and Python evaluates the conditions according to the iterable type. So, if you use a list, Python will execute both knows_python() and is_local() during the creation of that list before calling any():

should_interview = any([knows_python(applicant), is_local(applicant)])

Here, Python will call is_local() for every applicant, even for those who know Python. Because is_local() will take a long time to execute and is sometimes unnecessary, this is an inefficient implementation of the logic.

There are ways to make Python call functions lazily when you’re using iterables, such as building an iterator with map() or using a generator expression:

any((meets_criteria(applicant) for applicant in applicants))

This example uses a generator expression to generate Boolean values indicating whether an applicant meets the criteria for an interview. Once an applicant meets the criteria, any() will return True without checking the remaining applicants. But keep in mind that these types of workarounds also present their own issues and may not be appropriate in every situation.

The most important thing to remember is that the syntactic difference between any() and or can affect their usability.

Syntax isn’t the only difference that affects the usability of these tools. Next, let’s take a look at the different return values for any() and or and how they might influence your decision on which tool to use.

Return Value

Python’s any() and or return different types of values. any() returns a Boolean, which indicates whether it found a truthy value in the iterable:

>>>

>>> any((1, 0))
True

In this example, any() found a truthy value (the integer 1), so it returned the Boolean value True.

or, on the other hand, returns the first truthy value it finds, which will not necessarily be a Boolean. If there are no truthy values, then or returns the last value:

>>>

>>> 1 or 0
1

>>> None or 0
0

In the first example, or evaluated 1, which is truthy, and returned it without evaluating 0. In the second example, None is falsy, so or evaluated 0 next, which is also falsy. But since there are no more expressions to check, or returns the last value, 0.

When you’re deciding which tool to use, it’s helpful to consider if you want to know the actual value of the object or just whether a truthy value exists somewhere in the collection of objects.

Conclusion

Congratulations! You’ve learned the ins and outs of using any() in Python and the differences between any() and or. With a deeper understanding of these two tools, you’re well prepared to decide between them in your own code.

You now know:

How to use any() in Python
Why you’d use any() instead of or

If you would like to continue learning about conditional expressions and how to use tools like or and any() in Python, then you can check out the following resources:

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #2: Learn Python Skills While Creating Games

2020-03-27T12:00:00+00:00

In this episode, Christopher interviews Jon Fincher from the Real Python Team. Jon talks about his recent articles on PyGame and Arcade. They discuss if game programming is a good way to develop your Python programming skills, and if a game would make a good portfolio piece. He compares the two popular Python game libraries of Arcade and PyGame, and discusses about how to find assets for your own creations.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

SimPy: Simulating Real-World Processes With Python

2020-03-25T14:00:00+00:00

The real world is full of systems, like airports and highways, that frequently experience congestion and delay. When these systems are not optimized, their inefficiency can lead to countless unhappy customers and hours of wasted time. In this tutorial, you’ll learn how to use Python’s simpy framework to create virtual simulations that will help you solve problems like these.

In this tutorial, you’ll learn how to:

Use a simulation to model a real-world process
Create a step-by-step algorithm to approximate a complex system
Design and run a real-world simulation in Python with simpy

In this tutorial, you’ll create a simulation for a local movie theater. Your goal is to provide the manager with a script to help find the optimal number of employees to have on staff. You can download the source code for this script by clicking on the link below:

Download Code: Click here to download the code you'll use to learn about SimPy in this tutorial.

What Simulation Is

A simulation is a representation of a real-world system. One can use mathematical or computational models of this system to study how it works, or what happens when parts of it are changed. Simulations are used in airports, restaurants, mechanics, government agencies, and many other systems where poor resource allocation can lead to congestion, customer dissatisfaction, and critical transportation delays.

A system can be any environment where things happen. Examples of real-world systems include car washes, banks, manufacturing plants, airports, post offices, call centers, and more. These systems have agents that undergo processes within them. For instance:

A car wash will have cars go through the washing process.
An airport will have passengers go through the security check process.
A call center will have customers go through the process of speaking with a telemarketer.

This relationship is summed up in the table below:

System	Agent	Process
Car wash	Car	Wash
Airport	Passenger	Security check
Call center	Customer	Speak with a telemarketer

Understanding the processes that agents go through within a system is an important component of logistical planning, especially for large-scale organizations. For example, an airport can see passenger wait times at a security checkpoint skyrocket if there aren’t enough workers that day. Similarly, time-sensitive mail can be delayed by days (or even weeks) if it isn’t routed properly.

These instances of congestion can have real-life consequences on time and money, so it’s important to be able to model these processes beforehand. This gives you an idea of where the system might run into problems and how resources should be allocated ahead of time to solve those problems in the most efficient way possible.

How Simulation Works

In Python, you can use the simpy framework for event simulation. First, take a quick look at how a simulated process would run in Python. Below is a code snippet from a simulation of a security checkpoint system. The following three lines of code set up the environment, pass all necessary functions, and run the simulation:

# Set up the environment
env = simpy.Environment()

# Assume you've defined checkpoint_run() beforehand
env.process(checkpoint_run(env, num_booths, check_time, passenger_arrival))

# Let's go!
env.run(until=10)

The first line of code above establishes the environment. You’ll do this by assigning simpy.Environment() to the desired variable. Here, it’s simply named env. This tells simpy to create an environment object named env that will manage the simulation time and move the simulation through each subsequent time step.

Once you have your environment established, you’ll pass in all of the variables that will act as your parameters. These are the things you can vary to see how the system will react to changes. For this security checkpoint system, you’re using the following parameters:

env: the environment object to schedule and process events
num_booths: the number of ID check booths
check_time: the length of time it takes to check a passenger’s ID
passenger_arrival: the rate at which passengers arrive at the queue

Then, it’s time to run the simulation! You can do this by calling env.run() and specifying how long you want the simulation to run for. The simulation runs in minutes, so this sample code will run the simulation for 10 real-time minutes.

Note: Don’t worry! You won’t have to wait 10 actual minutes for the simulation to finish. Because simulation gives you a virtual look at a real-time process, those 10 minutes will pass in mere seconds on the computer.

To recap, here are the three steps to running a simulation in Python:

Establish the environment.
Pass in the parameters.
Run the simulation.

But there’s a lot more going on underneath the hood! You’ll need to understand how to choose those parameters, and you’ll have to define all the functions that will be called when the simulation is run.

Let’s get started!

How to Get Started With `simpy`

There are a few to-dos you should check off your list before creating simulations in Python. The first thing you need to do is make sure you have a solid understanding of Python basics. In particular, you’ll need to have a good grasp of classes and generators.

Note: If you need to freshen up on these topics, then check out Intro to Object-Oriented Programming (OOP) in Python and Introduction to Python Generators. These are crucial pieces of the simulation process, so you’ll need to understand them before moving forward.

The next thing you’ll want to do is install the required package. The main framework you’ll be using is simpy. This is the core package that will create, manage, and run your simulation. You can install it with pip:

$ python3 -m pip install simpy

You’ll also need a few built-in Python modules. You’ll use the statistics module to calculate average wait times and the random module to generate random numbers. These come as part of the Python standard library, so you don’t need to install anything new.

Finally, you’ll need to choose how you want to run your simulation. In general, you can choose one of two options:

Run it interactively: Use a Jupyter Notebook, where each code block will contain its own class or function definition. The output will be displayed at the bottom of the notebook.
Run it in the shell: Save your simulation as a .py file and tell Python to run it in your terminal. The output will be printed directly to the console.

Choose whichever method you’re most comfortable with! The outcome should be the same.

Throughout this tutorial, you’ll see references to a standalone file named simulate.py. As you move through this tutorial, the code blocks will reference simulate.py to help you keep track of how all the pieces fit together. For your reference, you can access the full code for simulate.py at the link below:

Download Code: Click here to download the code you'll use to learn about SimPy in this tutorial.

Feel free to save the file simulate.py and follow along in your favorite editor!

How to Simulate With the `simpy` Package

The first step to running a simulation in simpy is to choose a process to model. Simulation is all about creating a virtual environment to reflect a real-world system. In that same spirit, you’ll “simulate” a situation for your simulation!

Imagine you’ve been hired to help the manager for a small, local movie theater. The theater has been receiving poor reviews due to their long wait times. The manager, who is just as concerned about cost as he is about customer satisfaction, can only afford to keep so many employees on staff.

The manager is particularly worried about what chaos can unfold once those blockbusters start coming out: lines wrapping around the theater, employees stretched to their limit, angry moviegoers missing the opening scenes… This is definitely a situation to avoid!

After checking the reviews, the manager was able to determine that a given moviegoer to their theater is willing to spend at most 10 minutes from the time they arrive until the time they put their butt in a seat. In other words, the average wait time for a night at the theater needs to be 10 minutes or less. The manager has asked for your help to figure out a solution to getting customer wait times under this 10 minute requirement.

Brainstorming a Simulation Algorithm

Before you write out a single line of code, it’s important that you first figure out how your process would run in real life. This is to ensure that, when you pass it along to the machine, the process is an accurate reflection of what customers will really experience. Here’s how you might think through the steps a moviegoer might take to write out your algorithm:

Arrive at the theater, get in line, and wait to purchase a ticket.
Buy a ticket from the box office.
Wait in line to have the ticket checked.
Get the ticket checked by an usher.
Choose whether or not to get in line for the concession stand:
- If they get in line, then they purchase food.
- If they don’t get in line, then they skip to the last step.
Go find their seat.

This is a step-by-step iteration for a moviegoer who purchases their ticket at the theater box office. You can already see which parts of this process can be controlled. You can affect how long a customer is waiting by having more cashiers available at the box office.

There are also parts of the process that can’t be controlled, like the very first step. You can’t control how many customers will arrive, or how quickly they’ll do so. You can make a guess, but you can’t simply choose a number, because that would be a poor reflection of reality. For this parameter, the best thing you can do is use available data to determine an appropriate arrival time.

Note: Using historical data ensures that the solution you find will accurately reflect what you can expect to see in real life.

With these things in mind, it’s time to build your simulation!

Setting Up the Environment

Before you start building your simulation, you need to make sure that your development environment is properly configured. The very first thing you’ll want to do is import the necessary packages. You can do this by declaring import statements at the top of your file:

import simpy
import random
import statistics

These are the main libraries you’ll use to build a script for the theater manager. Remember, the goal is to find the optimal number of employees that gives an average wait time of less than 10 minutes. To do this, you’ll need to collect the length of time that it takes for each moviegoer to make it to their seats. The next step is to declare a list to hold these times:

wait_times = []

This list will contain the total amount of time each moviegoer spends moving through the theater, from arrival to sitting in their seat. You declare this list at the very top of the file so that you can use it inside any function you define later on.

Creating the Environment: Class Definition

The first part of the simulation you’ll want to build is the blueprint for the system. This is going to be the overall environment inside which things happen, and people or objects move from one place to another. Remember, an environment can be one of many different systems, like a bank, a car wash, or a security checkpoint. In this case, the environment is a movie theater, so that will be the name of your class:

class Theater(object):
    def __init__(self):
        # More to come!

Now it’s time to think through the parts of a movie theater. Of course, there’s the theater itself, which is what you’ve called your environment. Later, you’ll explicitly declare the theater as an actual environment using one of the simpy functions. For now, call it env for short and add it to the class definition:

class Theater(object):
    def __init__(self, env):
        self.env = env

Alright, what else might there be in a theater? You can figure this out by thinking through the simulation algorithm you planned out earlier. When a moviegoer arrives, they’ll need to get in line at the box office, where a cashier will be waiting to help them out. Now you’ve discovered two things about the theater environment:

There are cashiers.
Moviegoers can purchase tickets from them.

Cashiers are a resource that the theater makes available to its customers, and they help moviegoers through the process of purchasing a ticket. Right now, you don’t know how many cashiers are available in the simulated theater. In fact, that’s the very problem you’re trying to solve. How do wait times change, depending on the number of cashiers working on a given night?

You can go ahead and call this unknown variable num_cashiers. The exact value this variable will take can be sorted out later. For now, just know that it’s an indispensable part of the theater environment. Add it to the class definition:

class Theater(object):
    def __init__(self, env, num_cashiers):
        self.env = env
        self.cashier = simpy.Resource(env, num_cashiers)

Here, you add the new parameter num_cashiers to your __init__() definition. Then, you create a resource self.cashier and use simpy.Resource() to declare how many can be in this environment at any given time.

Note: In simpy, resources are the parts of the environment (env) that are limited in number. Using one of them takes time, and only so many (num_cashiers) are available to be used at once.

There’s one more step that you’ll need to take. A cashier isn’t going to purchase a ticket for themselves, right? They’re going to help the moviegoer! Again, you know that this process of purchasing a ticket is going to take a certain amount of time. But just how much time?

Say you’ve asked the manager for historical data on the theater, like employee performance reviews or ticket purchase receipts. Based on this data, you’ve learned that it takes, on average, between 1 and 2 minutes to issue a ticket at the box office. How do you get simpy to mimic this behavior? It only takes one line of code:

yield self.env.timeout(random.randint(1, 3))

env.timeout() tells simpy to trigger an event after a certain amount of time has passed. In this case, the event is that a ticket was purchased.

The time this takes could be one minute, two minutes, or three minutes. You want each moviegoer to spend a different amount of time at the cashier. To do this, you use random.randint() to choose a random number between the given low and high values. Then, for each moviegoer, the simulation will wait for the chosen amount of time.

Let’s wrap this up in a tidy function and add it to the class definition:

class Theater(object):
    def __init__(self, env, num_cashiers):
        self.env = env
        self.cashier = simpy.Resource(env, num_cashiers)

    def purchase_ticket(self, moviegoer):
        yield self.env.timeout(random.randint(1, 3))

The one initiating the event in purchase_ticket() is the moviegoer, so they must be passed as a required argument.

Note: You’ll see how the moviegoer actually purchases the ticket in the next section!

That’s it! You’ve selected a time-bound resource, defined its related process, and codified this in your class definition. For this tutorial, there are two more resources you’ll need to declare:

Ushers to check tickets
Servers to sell food

After checking the data the manager sent over, you determine that servers take anywhere between 1 and 5 minutes to complete an order. In addition, ushers are remarkably fast at checking tickets, with an average speed of 3 seconds!

You’ll need to add these resources to your class and define the corresponding functions check_ticket() and sell_food(). Can you figure out what the code should look like? When you’ve got an idea, you can expand the code block below to check your understanding:

class Theater(object):
    def __init__(self, env, num_cashiers, num_servers, num_ushers):
        self.env = env
        self.cashier = simpy.Resource(env, num_cashiers)
        self.server = simpy.Resource(env, num_servers)
        self.usher = simpy.Resource(env, num_ushers)

    def purchase_ticket(self, moviegoer):
        yield self.env.timeout(random.randint(1, 3))

    def check_ticket(self, moviegoer):
        yield self.env.timeout(3 / 60)

    def sell_food(self, moviegoer):
        yield self.env.timeout(random.randint(1, 5))

Take a close look at the new resources and functions. Notice how they follow the same format as described above. sell_food() uses random.randint() to generate a random number between 1 and 5 minutes, representing the time it would take a moviegoer to place an order and receive their food.

The time delay for check_ticket() is a bit different because the ushers only take 3 seconds. Since simpy works in minutes, this value needs to be passed as a fraction of a minute, or 3 / 60.

Moving Through the Environment: Function Definition

Alright, you’ve set up the environment by defining a class. You have resources and processes. Now you need a moviegoer to use them. When a moviegoer arrives at the theater, they’ll request a resource, wait for its process to complete, and then leave. You’ll create a function, called go_to_movies(), to keep track of this:

def go_to_movies(env, moviegoer, theater):
    # Moviegoer arrives at the theater
    arrival_time = env.now

There are three arguments passed to this function:

env: The moviegoer will be controlled by the environment, so you’ll pass this as the first argument.
moviegoer: This variable tracks each person as they move through the system.
theater: This parameter gives you access to the processes you defined in the overall class definition.

You also declare a variable arrival_time to hold the time at which each moviegoer arrives at the theater. You can get this time using the simpy call to env.now.

You’ll want each of the processes from your theater to have corresponding requests in go_to_movies(). For example, the first process in the class is purchase_ticket(), which uses a cashier resource. The moviegoer will need to make a request to the cashier resource to help them through the purchase_ticket() process. Here’s a table to summarize this:

Process in `theater`	Request in `go_to_movies()`
`purchase_ticket()`	Request a `cashier`
`check_ticket()`	Request an `usher`
`sell_food()`	Request a `server`

The cashier is a shared resource, which means that many moviegoers will use the same cashier. However, a cashier can only help one moviegoer at a time, so you’ll need to include some waiting behavior in your code. Here’s how that works:

def go_to_movies(env, moviegoer, theater):
    # Moviegoer arrives at the theater
    arrival_time = env.now

    with theater.cashier.request() as request:
        yield request
        yield env.process(theater.purchase_ticket(moviegoer))

Here’s how this code works:

theater.cashier.request(): moviegoer generates a request to use a cashier.
yield request: moviegoer waits for a cashier to become available if all are currently in use. To learn more about the yield keyword, check out How to Use Generators and yield in Python.
yield env.process(): moviegoer uses an available cashier to complete the given process. In this case, that’s to purchase a ticket with a call to theater.purchase_ticket().

After a resource is used, it must be freed up for the next agent to use. You could do this explicitly with release(), but in the code above, you use a with statement instead. This shortcut tells the simulation to automatically release the resource once the process is complete. In other words, once the ticket is bought, the moviegoer will leave, and the cashier will automatically be ready to take the next customer.

When a cashier is freed up, the moviegoer will spend some time buying their ticket. env.process() tells the simulation to go to the Theater instance and run the purchase_ticket() process on this moviegoer. The moviegoer will repeat this request, use, release cycle to have their ticket checked:

def go_to_movies(env, moviegoer, theater):
    # Moviegoer arrives at the theater
    arrival_time = env.now

    with theater.cashier.request() as request:
        yield request
        yield env.process(theater.purchase_ticket(moviegoer))

    with theater.usher.request() as request:
        yield request
        yield env.process(theater.check_ticket(moviegoer))

Here, the structure for the code is the same.

Then, there’s the optional step of buying food from the concession stand. You can’t know whether a moviegoer will want to purchase snacks and drinks. One way to deal with this uncertainty is to introduce a bit of randomness to the function.

Each moviegoer either will or will not want to buy food, which you can store as the Boolean values True or False. Then, use the random module to have the simulation randomly decide whether or not this particular moviegoer is going to proceed to the concession stand:

def go_to_movies(env, moviegoer, theater):
    # Moviegoer arrives at the theater
    arrival_time = env.now

    with theater.cashier.request() as request:
        yield request
        yield env.process(theater.purchase_ticket(moviegoer))

    with theater.usher.request() as request:
        yield request
        yield env.process(theater.check_ticket(moviegoer))

    if random.choice([True, False]):
        with theater.server.request() as request:
            yield request
            yield env.process(theater.sell_food(moviegoer))

This conditional statement will return one of two outcomes:

True: The moviegoer will request a server and order food.
False: The moviegoer will instead go to find their seats without purchasing any snacks.

Now, remember the goal of this simulation is to determine the number of cashiers, ushers, and servers that should be on staff to keep wait times under 10 minutes. To do this, you’ll need to know how long it took any given moviegoer to make it to their seats. You use env.now at the beginning of the function to track the arrival_time, and again at the end when each moviegoer is finished with all processes and heading into the theater:

def go_to_movies(env, moviegoer, theater):
    # Moviegoer arrives at the theater
    arrival_time = env.now

    with theater.cashier.request() as request:
        yield request
        yield env.process(theater.purchase_ticket(moviegoer))

    with theater.usher.request() as request:
        yield request
        yield env.process(theater.check_ticket(moviegoer))

    if random.choice([True, False]):
        with theater.server.request() as request:
            yield request
            yield env.process(theater.sell_food(moviegoer))

    # Moviegoer heads into the theater
    wait_times.append(env.now - arrival_time)

You use env.now to get the time at which the moviegoer has finished all processes and made it to their seats. You subtract the moviegoer’s arrival_time from this departure time and append the resulting time difference to your waiting list, wait_times.

Note: You could store the departure time in a separate variable like departure_time, but this would make your code very repetitive, which violates the D.R.Y. principle.

This moviegoer is ready to watch some previews!

Making Things Happen: Function Definition

Now you’ll need to define a function to run the simulation. run_theater() will be responsible for creating an instance of a theater and generating moviegoers until the simulation stops. The first thing this function should do is create an instance of a theater:

def run_theater(env, num_cashiers, num_servers, num_ushers):
    theater = Theater(env, num_cashiers, num_servers, num_ushers)

Since this is the main process, you’ll need to pass all of the unknowns you’ve declared so far:

num_cashiers
num_servers
num_ushers

These are all variables that the simulation needs to create and control the environment, so it’s absolutely vital to pass them all. Then, you define a variable theater and tell the simulation to set up the theater with a certain number of cashiers, servers, and ushers.

You also might want to start your simulation with a few moviegoers waiting at the theater. There will probably be a few people ready to go as soon as the doors open! The manager says to expect around 3 moviegoers in line ready to buy tickets as soon as the box office opens. You can tell the simulation to go ahead and move through this initial group like so:

def run_theater(env, num_cashiers, num_servers, num_ushers):
    theater = Theater(env, num_cashiers, num_servers, num_ushers)

    for moviegoer in range(3):
        env.process(go_to_movies(env, moviegoer, theater))

You use range() to populate the theater with 3 moviegoers. Then, you use env.process() to tell the simulation to prepare to move them through the theater. The rest of the moviegoers will make it to the theater in their own time. So, the function should keep sending new customers into the theater as long as the simulation is running.

You don’t know how long it will take new moviegoers to make it to the theater, so you decide to look at past data. Using timestamped receipts from the box office, you learn that moviegoers tend to arrive at the theater, on average, every 12 seconds. Now all you have to do is tell the function to wait this long before generating a new person:

def run_theater(env, num_cashiers, num_servers, num_ushers):
    theater = Theater(env, num_cashiers, num_servers, num_ushers)

    for moviegoer in range(3):
        env.process(go_to_movies(env, moviegoer, theater))

    while True:
        yield env.timeout(0.20)  # Wait a bit before generating a new person

        # Almost done!...

Note that you use the decimal number 0.20 to represent 12 seconds. To get this number, you simply divide 12 seconds by 60 seconds, which is the number of seconds in a minute.

After waiting, the function should increment moviegoer by 1 and generate the next person. The generator function is the same one you used to initialize the first 3 moviegoers:

def run_theater(env, num_cashiers, num_servers, num_ushers):
    theater = Theater(env, num_cashiers, num_servers, num_ushers)

    for moviegoer in range(3):
        env.process(go_to_movies(env, moviegoer, theater))

    while True:
        yield env.timeout(0.20)  # Wait a bit before generating a new person

        moviegoer += 1
        env.process(go_to_movies(env, moviegoer, theater))

That’s it! When you call this function, the simulation will generate 3 moviegoers to start and begin moving them through the theater with go_to_movies(). After that, new moviegoers will arrive at the theater with an interval of 12 seconds and move through the theater in their own time.

Calculating the Wait Time: Function Definition

At this point, you should have a list wait_times that contains the total amount of time it took each moviegoer to make it to their seat. Now you’ll want to define a function to help calculate the average time a moviegoer spends from the time they arrive to the time they finish checking their ticket. get_average_wait_time() does just this:

def get_average_wait_time(wait_times):
    average_wait = statistics.mean(wait_times)

This function takes your wait_times list as an argument and uses statistics.mean() to calculate the average wait time.

Since you’re creating a script that will be used by the movie theater manager, you’ll want to make sure that the output can be read easily by the user. You can add a function called calculate_wait_time() to do this:

def calculate_wait_time(arrival_times, departure_times):
    average_wait = statistics.mean(wait_times)
    # Pretty print the results
    minutes, frac_minutes = divmod(average_wait, 1)
    seconds = frac_minutes * 60
    return round(minutes), round(seconds)

The last part of the function uses divmod() to return the results in minutes and seconds, so the manager can easily understand the program’s output.

Choosing Parameters: User Input Function Definition

As you’ve built these functions, you’ve run into a few variables that have not been clearly defined:

num_cashiers
num_servers
num_ushers

These variables are the parameters that you can change to see how the simulation changes. If a blockbuster movie has customers lining up around the block, how many cashiers should be working? What if people are flying through the box office but getting stuck at concessions? What value of num_servers will help ease the flow?

Note: That’s the beauty of simulation. It allows you to try these things out so that you can determine the best possible decision in real life.

Whoever is using your simulation needs to be able to change the values of these parameters to try out different scenarios. To this end, you’ll create a helper function to get these values from the user:

def get_user_input():
    num_cashiers = input("Input # of cashiers working: ")
    num_servers = input("Input # of servers working: ")
    num_ushers = input("Input # of ushers working: ")
    params = [num_cashiers, num_servers, num_ushers]
    if all(str(i).isdigit() for i in params):  # Check input is valid
        params = [int(x) for x in params]
    else:
        print(
            "Could not parse input. The simulation will use default values:",
            "\n1 cashier, 1 server, 1 usher.",
        )
        params = [1, 1, 1]
    return params

This function simply calls Python’s input() function to retrieve data from the user. Because user input runs the risk of being messy, you can include an if/else clause to catch anything invalid. If the user inputs bad data, then the simulation will run with default values.

Finalizing the Setup: Main Function Definition

The last function you’ll want to create is main(). This will ensure your script runs in the proper order when you execute it on the command line. You can read more about main() in Defining Main Functions in Python. Here’s what your main() should look like:

def main():
  # Setup
  random.seed(42)
  num_cashiers, num_servers, num_ushers = get_user_input()

  # Run the simulation
  env = simpy.Environment()
  env.process(run_theater(env, num_cashiers, num_servers, num_ushers))
  env.run(until=90)

  # View the results
  mins, secs = get_average_wait_time(wait_times)
  print(
      "Running simulation...",
      f"\nThe average wait time is {mins} minutes and {secs} seconds.",
  )

Here’s how main() works:

Set up your environment by declaring a random seed. This ensures your output will look like what you see in this tutorial.
Query the user of your program for some input.
Create the environment and save it as the variable env, which will move the simulation through each time step.
Tell simpy to run the process run_theater(), which creates the theater environment and generates moviegoers to move through it.
Determine how long you want the simulation to run. As a default value, the simulation is set to run for 90 minutes.
Store the output of get_average_wait_time() in two variables, mins and secs.
Use print() to show the results to the user.

With this, the setup is complete!

How to Run the Simulation

With just a few more lines of code, you’ll be able to watch your simulation come to life. But first, here’s an overview of the functions and classes you’ve defined so far:

Theater: This class definition serves as a blueprint for the environment you want to simulate. It determines some information about that environment, like what kinds of resources are available, and what processes are associated with them.
go_to_movies(): This function makes explicit requests to use a resource, goes through the associated process, and then releases it to the next moviegoer.
run_theater(): This function controls the simulation. It uses the Theater class blueprint to create an instance of a theater, and then calls on go_to_movies() to generate and move people through the theater.
get_average_wait_time(): This function finds the average time it takes a moviegoer to make it through the theater.
calculate_wait_time(): This function ensures the final output is easy for the user to read.
get_user_input(): This function allows the user to define some parameters, like how many cashiers are available.
main(): This function ensures that your script runs properly in the command line.

Now, you only need two more lines of code to invoke your main function:

if __name__ == '__main__':
    main()

With that, your script is ready to run! Open up your terminal, navigate to where you’ve stored simulate.py, and run the following command:

$ python simulate.py
Input # of cashiers working:

You’ll be prompted to select the parameters you want for your simulation. Here’s what the output looks like with default parameters:

$ python simulate.py
Input # of cashiers working: 1
Input # of servers working: 1
Input # of ushers working: 1
Running simulation...
The average wait time is 42 minutes and 53 seconds.

Whoa! That’s a long time to be waiting around!

When to Change Things Up

Remember, your goal is to approach the manager with a solution for how many employees he’ll need on staff to keep wait times under 10 minutes. To this end, you’ll want to play around with your parameters to see which numbers offer an optimal solution.

First, try something completely insane and max out the resources! Say there were 100 cashiers, 100 servers, and 100 ushers working in this theater. This is impossible, of course, but using insanely high numbers will quickly tell you what the system’s limit is. Try it now:

$ python simulate.py
Input # of cashiers working: 100
Input # of servers working: 100
Input # of ushers working: 100
Running simulation...
The average wait time is 3 minutes and 29 seconds.

Even if you maxed out the resources, you would only get wait times down to 3 and a half minutes. Now try and change the numbers to see if you can get wait times down to 10 minutes, like the manager requested. What solution did you come up with? You can expand the code block below to see one possible solution:

$ python simulate.py
Input # of cashiers working: 9
Input # of servers working: 6
Input # of ushers working: 1
Running simulation...
The average wait time is 9 minutes and 60 seconds.

At this point, you would present your results to the manager and make a suggestion to help improve the theater. For instance, to cut down on costs, he might want to install 10 ticket kiosks at the front of the theater instead of keeping 10 cashiers on hand each night.

Conclusion

In this tutorial, you’ve learned how to build and run a simulation in Python using the simpy framework. You’ve come to understand how systems have agents undergo processes, and how you can create virtual representations of those systems to fortify them against congestion and delay. While the type of simulation can vary, the overall execution is the same! You’ll be able to apply what you’ve learned here to a variety of different scenarios.

Now you can:

Brainstorm a simulation algorithm step by step
Create a virtual environment in Python with simpy
Define functions that represent agents and processes
Change parameters of your simulation to find the optimal solution

There’s so much you can do with simpy, so don’t let your exploration stop here. Take what you’ve learned and apply it to new scenarios. Your solutions could help save people valuable time and money, so dive in and see what other processes you can optimize! You can download the source code for the script you built in this tutorial by clicking on the link below:

Download Code: Click here to download the code you'll use to learn about SimPy in this tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Using NumPy's np.arange() Effectively

2020-03-24T14:00:00+00:00

NumPy is the fundamental Python library for numerical computing. Its most important type is an array type called ndarray. NumPy offers a lot of array creation routines for different circumstances. arange() is one such function based on numerical ranges. It’s often referred to as np.arange() because np is a widely used abbreviation for NumPy.

Creating NumPy arrays is important when you’re working with other Python libraries that rely on them, like SciPy, Pandas, Matplotlib, scikit-learn, and more. NumPy is suitable for creating and working with arrays because it offers useful routines, enables performance boosts, and allows you to write concise code.

By the end of this course, you’ll know:

What np.arange() is
How to use np.arange()
How np.arange() compares to the Python built-in class range
Which routines are similar to np.arange()

Let’s see np.arange() in action!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Python math Module: Everything You Need to Know

2020-03-23T14:00:00+00:00

In this article, you’ll learn all about Python’s math module. Mathematical calculations are an essential part of most Python development. Whether you’re working on a scientific project, a financial application, or any other type of programming endeavor, you just can’t escape the need for math.

For straightforward mathematical calculations in Python, you can use the built-in mathematical operators, such as addition (+), subtraction (-), division (/), and multiplication (*). But more advanced operations, such as exponential, logarithmic, trigonometric, or power functions, are not built in. Does that mean you need to implement all of these functions from scratch?

Fortunately, no. Python provides a module specifically designed for higher-level mathematical operations: the math module.

By the end of this article, you’ll learn:

What the Python math module is
How to use math module functions to solve real-life problems
What the constants of the math module are, including pi, tau, and Euler’s number
What the differences between built-in functions and math functions are
What the differences between math, cmath, and NumPy are

A background in mathematics will be helpful here, but don’t worry if math isn’t your strong suit. This article will explain the basics of everything you need to know.

So let’s get started!

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

Getting to Know the Python `math` Module

The Python math module is an important feature designed to deal with mathematical operations. It comes packaged with the standard Python release and has been there from the beginning. Most of the math module’s functions are thin wrappers around the C platform’s mathematical functions. Since its underlying functions are written in CPython, the math module is efficient and conforms to the C standard.

The Python math module offers you the ability to perform common and useful mathematical calculations within your application. Here are a few practical uses for the math module:

Calculating combinations and permutations using factorials
Calculating the height of a pole using trigonometric functions
Calculating radioactive decay using the exponential function
Calculating the curve of a suspension bridge using hyperbolic functions
Solving quadratic equations
Simulating periodic functions, such as sound and light waves, using trigonometric functions

Since the math module comes packaged with the Python release, you don’t have to install it separately. Using it is just a matter of importing the module:

>>>

>>> import math

You can import the Python math module using the above command. After importing, you can use it straightaway.

Constants of the `math` Module

The Python math module offers a variety of predefined constants. Having access to these constants provides several advantages. For one, you don’t have to manually hardcode them into your application, which saves you a lot of time. Plus, they provide consistency throughout your code. The module includes several famous mathematical constants and important values:

Pi
Tau
Euler’s number
Infinity
Not a number (NaN)

In this section, you’ll learn about the constants and how to use them in your Python code.

Pi

Pi (π) is the ratio of a circle’s circumference (c) to its diameter (d):

π = c/d

This ratio is always the same for any circle.

Pi is an irrational number, which means it can’t be expressed as a simple fraction. Therefore, pi has an infinite number of decimal places, but it can be approximated as 22/7, or 3.141.

Interesting Fact: Pi is the most recognized and well-known mathematical constant in the world. It has its own celebration date, called Pi Day, which falls on March 14th (3/14).

You can access pi as follows:

>>>

>>> math.pi
3.141592653589793

As you can see, the pi value is given to fifteen decimal places in Python. The number of digits provided depends on the underlying C compiler. Python prints the first fifteen digits by default, and math.pi always returns a float value.

So what are some of the ways that pi can be useful to you? You can calculate the circumference of a circle using 2πr, where r is the radius of the circle:

>>>

>>> r = 3
>>> circumference = 2 * math.pi * r
>>> f"Circumference of a Circle = 2 * {math.pi:.4} * {r} = {circumference:.4}"
'Circumference of a Circle = 2 * 3.142 * 3 = 18.85'

You can use math.pi to calculate the circumference of a circle. You can also calculate the area of a circle using the formula πr² as follows:

>>>

>>> r = 5
>>> area = math.pi * r * r
>>> f"Area of a Circle = {math.pi:.4} * {r} * {r} = {area:.4}"
'Area of a Circle = 3.142 * 5 * 5 = 78.54'

You can use math.pi to calculate the area and the circumference of a circle. When you are doing mathematical calculations with Python and you come across a formula that uses π, it’s a best practice to use the pi value given by the math module instead of hardcoding the value.

Tau

Tau (τ) is the ratio of a circle’s circumference to its radius. This constant is equal to 2π, or roughly 6.28. Like pi, tau is an irrational number because it’s just pi times two.

Many mathematical expressions use 2π, and using tau instead can help simplify your equations. For example, instead of calculating the circumference of a circle with 2πr, we can substitute tau and use the simpler equation τr.

The use of tau as the circle constant, however, is still under debate. You have the freedom to use either 2π or τ as necessary.

You can use tau as below:

>>>

>>> math.tau
6.283185307179586

Like math.pi, math.tau returns fifteen digits and is a float value. You can use tau to calculate the circumference of a circle with τr, where r is the radius, as follows:

>>>

>>> r = 3
>>> circumference = math.tau * r
>>> f"Circumference of a Circle = {math.tau:.4} * {r} = {circumference:.4}"
'Circumference of a Circle = 6.283 * 3 = 18.85'

You can use math.tau in place of 2 * math.pi to tidy up equations that include the expression 2π.

Euler’s Number

Euler’s number (e) is a constant that is the base of the natural logarithm, a mathematical function that is commonly used to calculate rates of growth or decay. As with pi and tau, Euler’s number is an irrational number with infinite decimal places. The value of e is often approximated as 2.718.

Euler’s number is an important constant because it has many practical uses, such as calculating population growth over time or determining rates of radioactive decay. You can access Euler’s number from the math module as follows:

>>>

>>> math.e
2.718281828459045

As with math.pi and math.tau, the value of math.e is given to fifteen decimal places and is returned as a float value.

Infinity

Infinity can’t be defined by a number. Rather, it’s a mathematical concept representing something that is never-ending or boundless. Infinity can go in either direction, positive or negative.

You can use infinity in algorithms when you want to compare a given value to an absolute maximum or minimum value. The values of positive and negative infinity in Python are as follows:

>>>

>>> f"Positive Infinity = {math.inf}"
'Positive Infinity = inf'
>>> f"Negative Infinity = {-math.inf}"
'Negative Infinity = -inf'

Infinity is not a numerical value. Instead, it’s defined as math.inf. Python introduced this constant in version 3.5 as an equivalent to float("inf"):

>>>

>>> float("inf") == math.inf
True

Both float("inf") and math.inf represent the concept of infinity, making math.inf greater than any numerical value:

>>>

>>> x = 1e308
>>> math.inf > x
True

In the above code, math.inf is greater than the value of x, 10³⁰⁸ (the maximum size of a floating-point number), which is a double precision number.

Similarly, -math.inf is smaller than any value:

>>>

>>> y = -1e308
>>> y > -math.inf
True

Negative infinity is smaller than the value of y, which is -10³⁰⁸. No number can be greater than infinity or smaller than negative infinity. That’s why mathematical operations with math.inf don’t change the value of infinity:

>>>

>>> math.inf + 1e308
inf
>>> math.inf / 1e308
inf

As you can see, neither addition nor division changes the value of math.inf.

Not a Number (NaN)

Not a number, or NaN, isn’t really a mathematical concept. It originated in the computer science field as a reference to values that are not numeric. A NaN value can be due to invalid inputs, or it can indicate that a variable that should be numerical has been corrupted by text characters or symbols.

It’s always a best practice to check if a value is NaN. If it is, then it could lead to invalid values in your program. Python introduced the NaN constant in version 3.5.

You can observe the value of math.nan below:

>>>

>>> math.nan
nan

NaN is not a numerical value. You can see that the value of math.nan is nan, the same value as float("nan").

Arithmetic Functions

Number theory is a branch of pure mathematics, which is the study of natural numbers. Number theory usually deals with positive whole numbers or integers.

The Python math module provides functions that are useful in number theory as well as in representation theory, a related field. These functions allow you to calculate a range of important values, including the following:

The factorials of a number
The greatest common divisor of two numbers
The sum of iterables

Find Factorials With Python `factorial()`

You may have seen mathematical expressions like 7! or 4! before. The exclamation marks don’t mean that the numbers are excited. Rather, “!” is the factorial symbol. Factorials are used in finding permutations or combinations. You can determine the factorial of a number by multiplying all whole numbers from the chosen number down to 1.

The following table shows the factorial values for 4, 6, and 7:

Symbol	In Words	Expression	Result
4!	Four factorial	4 x 3 x 2 x 1	24
6!	Six factorial	6 x 5 x 4 x 3 x 2 x 1	720
7!	Seven factorial	7 x 6 x 5 x 4 x 3 x 2 x 1	5040

You can see from the table that 4!, or four factorial, gives the value 24 by multiplying the range of whole numbers from 4 to 1. Similarly, 6! and 7! give the values 720 and 5040, respectively.

You can implement a factorial function in Python using one of several tools:

for loops
Recursive functions
math.factorial()

First you are going to look at a factorial implementation using a for loop. This is a relatively straightforward approach:

def fact_loop(num):
    if num < 0:
        return 0
    if num == 0:
        return 1

    factorial = 1
    for i in range(1, num + 1):
        factorial = factorial * i
    return factorial

You can also use a recursive function to find the factorial. This is more complicated but also more elegant than using a for loop. You can implement the recursive function as follows:

def fact_recursion(num):
    if num < 0:
        return 0
    if num == 0:
        return 1

    return num * fact_recursion(num - 1)

Note: There is a limit to the recursion depth in Python, but that subject is outside the scope of this article.

The following example illustrates how you can use the for loop and recursive functions:

>>>

>>> fact_loop(7)
5040

>>> fact_recursion(7)
5040

Even though their implementations are different, their return values are the same.

However, implementing functions of your own just to get the factorial of a number is time consuming and inefficient. A better method is to use math.factorial(). Here’s how you can find the factorial of a number using math.factorial():

>>>

>>> math.factorial(7)
5040

This approach returns the desired output with a minimal amount of code.

factorial() accepts only positive integer values. If you try to input a negative value, then you will get a ValueError:

>>>

>>> math.factorial(-5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: factorial() not defined for negative values

Inputting a negative value will result in a ValueError reading factorial() not defined for negative values.

factorial() doesn’t accept decimal numbers, either. It will give you a ValueError:

>>>

>>> math.factorial(4.3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: factorial() only accepts integral values

Inputting a decimal value results in a ValueError reading factorial() only accepts integral values.

You can compare the execution times for each of the factorial methods using timeit():

>>>

>>> import timeit
>>> timeit.timeit("fact_loop(10)", globals=globals())
1.063997201999996

>>> timeit.timeit("fact_recursion(10)", globals=globals())
1.815312818999928

>>> timeit.timeit("math.factorial(10)", setup="import math")
0.10671788000001925

The sample above illustrates the results of timeit() for each of the three factorial methods.

timeit() executes one million loops each time it is run. The following table compares the execution times of the three factorial methods:

Type	Execution Time
With loops	1.0640 s
With recursion	1.8153 s
With `factorial()`	0.1067 s

As you can see from the execution times, factorial() is faster than the other methods. That’s because of its underlying C implementation. The recursion-based method is the slowest out of the three. Although you might get different timings depending on your CPU, the order of the functions should be the same.

Not only is factorial() faster than the other methods, but it’s also more stable. When you implement your own function, you have to explicitly code for disaster cases such as handling negative or decimal numbers. One mistake in the implementation could lead to bugs. But when using factorial(), you don’t have to worry about disaster cases because the function handles them all. Therefore, it’s a best practice to use factorial() whenever possible.

Find the Ceiling Value With `ceil()`

math.ceil() will return the smallest integer value that is greater than or equal to the given number. If the number is a positive or negative decimal, then the function will return the next integer value greater than the given value.

For example, an input of 5.43 will return the value 6, and an input of -12.43 will return the value -12. math.ceil() can take positive or negative real numbers as input values and will always return an integer value.

When you input an integer value to ceil(), it will return the same number:

>>>

>>> math.ceil(6)
6
>>> math.ceil(-11)
-11

math.ceil() always returns the same value when an integer is given as input. To see the true nature of ceil(), you have to input decimal values:

>>>

>>> math.ceil(4.23)
5
>>> math.ceil(-11.453)
-11

When the value is positive (4.23), the function returns the next integer greater than the value (5). When the value is negative (-11.453), the function likewise returns the next integer greater than the value (-11).

The function will return a TypeError if you input a value that is not a number:

>>>

>>> math.ceil("x")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: must be real number, not str

You must input a number to the function. If you try to input any other value, then you will get a TypeError.

Find the Floor Value With `floor()`

floor() will return the closest integer value that is less than or equal to the given number. This function behaves opposite to ceil(). For example, an input of 8.72 will return 8, and an input of -12.34 will return -13. floor() can take either positive or negative numbers as input and will return an integer value.

If you input an integer value, then the function will return the same value:

>>>

>>> math.floor(4)
4
>>> math.floor(-17)
-17

As with ceil(), when the input for floor() is an integer, the result will be the same as the input number. The output only differs from the input when you input decimal values:

>>>

>>> math.floor(5.532)
5
>>> math.floor(-6.432)
-7

When you input a positive decimal value (5.532), it will return the closest integer that is less than the input number (5). If you input a negative number (-6.432), then it will return the next lowest integer value (-7).

If you try to input a value that is not a number, then the function will return a TypeError:

>>>

>>> math.floor("x")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: must be real number, not str

You can’t give non-number values as input to ceil(). Doing so will result in a TypeError.

Truncate Numbers With `trunc()`

When you get a number with a decimal point, you might want to keep only the integer part and eliminate the decimal part. The math module has a function called trunc() which lets you do just that.

Dropping the decimal value is a type of rounding. With trunc(), negative numbers are always rounded upward toward zero and positive numbers are always rounded downward toward zero.

Here is how the trunc() function rounds off positive or negative numbers:

>>>

>>> math.trunc(12.32)
12
>>> math.trunc(-43.24)
-43

As you can see, 12.32 is rounded downwards towards 0, which gives the result 12. In the same way, -43.24 is rounded upwards towards 0, which gives the value -43. trunc() always rounds towards zero regardless of whether the number is positive or negative.

When dealing with positive numbers, trunc() behaves the same as floor():

>>>

>>> math.trunc(12.32) == math.floor(12.32)
True

trunc() behaves the same as floor() for positive numbers. As you can see, the return value of both functions is the same.

When dealing with negative numbers, trunc() behaves the same as ceil():

>>>

>>> math.trunc(-43.24) == math.ceil(-43.24)
True

When the number is negative, floor() behaves the same as ceil(). The return values of both functions are the same.

Find the Closeness of Numbers With Python `isclose()`

In certain situations—particularly in the data science field—you may need to determine whether two numbers are close to each other. But to do so, you first need to answer an important question: How close is close? In other words, what is the definition of close?

Well, Merriam-Webster will tell you that close means “near in time, space, effect, or degree.” Not very helpful, is it?

For example, take the following set of numbers: 2.32, 2.33, and 2.331. When you measure closeness by two decimal points, 2.32 and 2.33 are close. But in reality, 2.33 and 2.331 are closer. Closeness, therefore, is a relative concept. You can’t determine closeness without some kind of threshold.

Fortunately, the math module provides a function called isclose() that lets you set your own threshold, or tolerance, for closeness. It returns True if two numbers are within your established tolerance for closeness and otherwise returns False.

Let’s check out how to compare two numbers using the default tolerances:

Relative tolerance, or rel_tol, is the maximum difference for being considered “close” relative to the magnitude of the input values. This is the percentage of tolerance. The default value is 1e-09 or 0.000000001.
Absolute tolerance, or abs_tol, is the maximum difference for being considered “close” regardless of the magnitude of the input values. The default value is 0.0.

isclose() will return True when the following condition is satisfied:

abs(a-b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol).

isclose uses the above expression to determine the closeness of two numbers. You can substitute your own values and observe whether any two numbers are close.

In the following case, 6 and 7 aren’t close:

>>>

>>> math.isclose(6, 7)
False

The numbers 6 and 7 aren’t considered close because the relative tolerance is set for nine decimal places. But if you input 6.999999999 and 7 under the same tolerance, then they are considered close:

>>>

>>> math.isclose(6.999999999, 7)
True

You can see that the value 6.999999999 is within nine decimal places of 7. Therefore, based on the default relative tolerance, 6.999999999 and 7 are considered close.

You can adjust the relative tolerance however you want depending on your need. If you set rel_tol to 0.2, then 6 and 7 are considered close:

>>>

>>> math.isclose(6, 7, rel_tol=0.2)
True

You can observe that 6 and 7 are close now. This is because they are within 20% of each other.

As with rel_tol, you can adjust the abs_tol value according to your needs. To be considered close, the difference between the input values must be less than or equal to the absolute tolerance value. You can set the abs_tol as follows:

>>>

>>> math.isclose(6, 7, abs_tol=1.0)
True
>>> math.isclose(6, 7, abs_tol=0.2)
False

When you set the absolute tolerance to 1, the numbers 6 and 7 are close because the difference between them is equal to the absolute tolerance. However, in the second case, the difference between 6 and 7 is not less than or equal to the established absolute tolerance of 0.2.

You can use the abs_tol for very small values:

>>>

>>> math.isclose(1, 1.0000001, abs_tol=1e-08)
False
>>> math.isclose(1, 1.00000001, abs_tol=1e-08)
True

As you can see, you can determine the closeness of very small numbers with isclose. A few special cases regarding closeness can be illustrated using nan and inf values:

>>>

>>> math.isclose(math.nan, 1e308)
False
>>> math.isclose(math.nan, math.nan)
False

>>> math.isclose(math.inf, 1e308)
False
>>> math.isclose(math.inf, math.inf)
True

You can see from the above examples that nan is not close to any value, not even to itself. On the other hand, inf is not close to any numerical values, not even to very large ones, but it is close to itself.

Power Functions

The power function takes any number x as input, raises x to some power n, and returns xⁿ as output. Python’s math module provides several power-related functions. In this section, you’ll learn about power functions, exponential functions, and square root functions.

Calculate the Power of a Number With `pow()`

Power functions have the following formula where the variable x is the base, the variable n is the power, and a can be any constant:

Power Function

In the formula above, the value of the base x is raised to the power of n.

You can use math.pow() to get the power of a number. There is a built-in function, pow(), that is different from math.pow(). You will learn the difference later in this section.

math.pow() takes two parameters as follows:

>>>

>>> math.pow(2, 5)
32.0
>>> math.pow(5, 2.4)
47.59134846789696

The first argument is the base value and the second argument is the power value. You can give an integer or a decimal value as input and the function always returns a float value. There are some special cases defined in math.pow().

When the base 1 is raised to the power of any number n, it gives the result 1.0:

>>>

>>> math.pow(1.0, 3)
1.0

When you raise base value 1 to any power value, you will always get 1.0 as the result. Likewise, any base number raised to the power of 0 gives the result 1.0:

>>>

>>> math.pow(4, 0.0)
1.0
>>> math.pow(-4, 0.0)
1.0

As you can see, any number raised to the power of 0 will give 1.0 as the result. You can see that result even if the base is nan:

>>>

>>> math.pow(math.nan, 0.0)
1.0

Zero raised to the power of any positive number will give 0.0 as the result:

>>>

>>> math.pow(0.0, 2)
0.0
>>> math.pow(0.0, 2.3)
0.0

But if you try to raise 0.0 to a negative power, then the result will be a ValueError:

>>>

>>> math.pow(0.0, -2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: math domain error

The ValueError only occurs when the base is 0. If the base is any other number except 0, then the function will return a valid power value.

Apart from math.pow(), there are two built-in ways of finding the power of a number in Python:

x ** y
pow()

The first option is straightforward. You may have used it a time or two already. The return type of the value is determined by the inputs:

>>>

>>> 3 ** 2
9
>>> 2 ** 3.3
9.849155306759329

When you use integers, you get an integer value. When you use decimal values, the return type changes to a decimal value.

The second option is a versatile built-in function. You don’t have to use any imports to use it. The built-in pow() method has three parameters:

The base number
The power number
The modulus number

The first two parameters are mandatory, whereas the third parameter is optional. You can input integers or decimal numbers and the function will return the appropriate result based on the input:

>>>

>>> pow(3, 2)
9
>>> pow(2, 3.3)
9.849155306759329

The built-in pow() has two required arguments that work the same as the base and power in the x ** y syntax. The pow() method also has a third parameter that is optional: modulus. This parameter is often used in cryptography. Built-in pow() with the optional modulus parameter is equivalent to the equation (x ** y) % z. The Python syntax looks like this:

>>>

>>> pow(32, 6, 5)
4
>>> (32 ** 6) % 5 == pow(32, 6, 5)
True

pow() raises the base (32) to the power (6), and then the result value is modulo divided by the modulus number (5). In this case, the result is 4. You can substitute your own values and see that both pow() and the given equation provide the same results.

Even though all three methods of calculating power do the same thing, there are some implementation differences between them. The execution times for each method are as follows:

>>>

>>> timeit.timeit("10 ** 308")
1.0078728999942541

>>> timeit.timeit("pow(10, 308)")
1.047615700008464

>>> timeit.timeit("math.pow(10, 308)", setup="import math")
0.1837239999877056

The following table compares the execution times of the three methods as measured by timeit():

Type	Execution Time
`x ** y`	1.0079 s
`pow(x, y)`	1.0476 s
`math.pow(x, y)`	0.1837 s

You can observe from the table that math.pow() is faster than the other methods and built-in pow() is the slowest.

The reason behind the efficiency of math.pow() is the way that it’s implemented. It relies on the underlying C language. On the other hand, pow() and x ** y use the input object’s own implementation of the ** operator. However, math.pow() can’t handle complex numbers (which will be explained in a later section), whereas pow() and ** can.

Find the Natural Exponent With `exp()`

You learned about power functions in the previous section. With exponential functions, things are a bit different. Instead of the base being the variable, power becomes the variable. It looks something like this:

General Exponential Function

Here a can be any constant, and x, which is the power value, becomes the variable.

So what’s so special about exponential functions? The value of the function grows rapidly as the x value increases. If the base is greater than 1, then the function continuously increases in value as x increases. A special property of exponential functions is that the slope of the function also continuously increases as x increases.

You learned about the Euler’s number in a previous section. It is the base of the natural logarithm. It also plays a role with the exponential function. When Euler’s number is incorporated into the exponential function, it becomes the natural exponential function:

Natural Exponential Function

This function is used in many real-life situations. You may have heard of the term exponential growth, which is often used in relation to human population growth or rates of radioactive decay. Both of these can be calculated using the natural exponential function.

The Python math module provides a function, exp(), that lets you calculate the natural exponent of a number. You can find the value as follows:

>>>

>>> math.exp(21)
1318815734.4832146
>>> math.exp(-1.2)
0.30119421191220214

The input number can be positive or negative, and the function always returns a float value. If the number is not a numerical value, then the method will return a TypeError:

>>>

>>> math.exp("x")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: must be real number, not str

As you can see, if the input is a string value, then the function returns a TypeError reading must be real number, not str.

You can also calculate the exponent using the math.e ** x expression or by using pow(math.e, x). The execution times of these three methods are as follows:

>>>

>>> timeit.timeit("math.e ** 308", setup="import math")
0.17853009998701513

>>> timeit.timeit("pow(math.e, 308)", setup="import math")
0.21040189999621361

>>> timeit.timeit("math.exp(308)", setup="import math")
0.125878200007719

The following table compares the execution times of the above methods as measured by timeit():

Type	Execution Time
`e ** x`	0.1785 s
`pow(e, x)`	0.2104 s
`math.exp(x)`	0.1259 s

You can see that math.exp() is faster than the other methods and pow(e, x) is the slowest. This is the expected behavior because of the underlying C implementation of the math module.

It’s also worth noting that e ** x and pow(e, x) return the same values, but exp() returns a slightly different value. This is due to implementation differences. Python documentation notes that exp() is more accurate than the other two methods.

Practical Example With `exp()`

Radioactive decay happens when an unstable atom loses energy by emitting ionizing radiation. The rate of radioactive decay is measured using half-life, which is the time it takes for half the amount of the parent nucleus to decay. You can calculate the decay process using the following formula:

Radioactive Decay Equation

You can use the above formula to calculate the remaining quantity of a radioactive element after a certain number of years. The variables of the given formula are as follows:

N(0) is the initial quantity of the substance.
N(t) is the quantity that still remains and has not yet decayed after a time (t).
T is the half-life of the decaying quantity.
e is Euler’s number.

Scientific research has identified the half-lives of all radioactive elements. You can substitute values to the equation to calculate the remaining quantity of any radioactive substance. Let’s try that now.

The radioisotope strontium-90 has a half-life of 38.1 years. A sample contains 100 mg of Sr-90. You can calculate the remaining milligrams of Sr-90 after 100 years:

>>>

>>> half_life = 38.1
>>> initial = 100
>>> time = 100
>>> remaining = initial * math.exp(-0.693 * time / half_life)
>>> f"Remaining quantity of Sr-90: {remaining}"
'Remaining quantity of Sr-90: 16.22044604811303'

As you can see, the half-life is set to 38.1 and the duration is set to 100 years. You can use math.exp to simplify the equation. By substituting the values to the equation you can find that, after 100 years, 16.22mg of Sr-90 remains.

Logarithmic Functions

Logarithmic functions can be considered the inverse of exponential functions. They are denoted in the following form:

General Logarithmic Function

Here a is the base of the logarithm, which can be any number. You learned about exponential functions in a previous section. Exponential functions can be expressed in the form of logarithmic functions and vice versa.

Python Natural Log With `log()`

The natural logarithm of a number is its logarithm to the base of the mathematical constant e, or Euler’s number:

Natural Logarithmic Function

As with the exponential function, natural log uses the constant e. It’s generally depicted as f(x) = ln(x), where e is implicit.

You can use the natural log in the same way that you use the exponential function. It’s used to calculate values such as the rate of population growth or the rate of radioactive decay in elements.

log() has two arguments. The first one is mandatory and the second one is optional. With one argument you can get the natural log (to the base e) of the input number:

>>>

>>> math.log(4)
1.3862943611198906
>>> math.log(3.4)
1.2237754316221157

However, the function returns a ValueError if you input a non-positive number:

>>>

>>> math.log(-3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: math domain error

As you can see, you can’t input a negative value to log(). This is because log values are undefined for negative numbers and zero.

With two arguments, you can calculate the log of the first argument to the base of the second argument:

>>>

>>> math.log(math.pi, 2)
1.651496129472319
>>> math.log(math.pi, 5)
0.711260668712669

You can see how the value changes when the log base is changed.

Understand `log2()` and `log10()`

The Python math module also provides two separate functions that let you calculate the log values to the base of 2 and 10:

log2() is used to calculate the log value to the base 2.
log10() is used to calculate the log value to the base 10.

With log2() you can get the log value to the base 2:

>>>

>>> math.log2(math.pi)
1.6514961294723187
>>> math.log(math.pi, 2)
1.651496129472319

Both functions have the same objective, but the Python documentation notes that log2() is more accurate than using log(x, 2).

You can calculate the log value of a number to base 10 with log10():

>>>

>>> math.log10(math.pi)
0.4971498726941338
>>> math.log(math.pi, 10)
0.4971498726941338

The Python documentation also mentions that log10() is more accurate than log(x, 10) even though both functions have the same objective.

Practical Example With Natural Log

In a previous section, you saw how to use math.exp() to calculate the remaining amount of a radioactive element after a certain period of time. With math.log(), you can find the half-life of an unknown radioactive element by measuring the mass at an interval. The following equation can be used to calculate the half-life of a radioactive element:

Radioactive Decay Equation

By rearranging the radioactive decay formula, you can make the half-life (T) the subject of the formula. The variables of the given formula are as follows:

T is the half-life of the decaying quantity.
N(0) is the initial quantity of the substance.
N(t) is the quantity that remains and has not yet decayed after a period of time (t).
ln is the natural log.

You can substitute the known values to the equation to calculate the half-life of a radioactive substance.

For example, imagine you are studying an unidentified radioactive element sample. When it was discovered 100 years ago, the sample size was 100mg. After 100 years of decay, only 16.22mg is remaining. Using the formula above, you can calculate the half-life of this unknown element:

>>>

>>> initial = 100
>>> remaining = 16.22
>>> time = 100
>>> half_life = (-0.693 * time) / math.log(remaining / initial)
>>> f"Half-life of the unknown element: {half_life}"
'Half-life of the unknown element: 38.09942398335152'

You can see that the unknown element has a half-life of roughly 38.1 years. Based on this information, you can identify the unknown element as strontium-90.

Other Important `math` Module Functions

The Python math module has many useful functions for mathematical calculations, and this article only covered a few of them in depth. In this section, you will briefly learn about some of the other important functions available in the math module.

Calculate the Greatest Common Divisor

The greatest common divisor (GCD) of two positive numbers is the largest positive integer that divides both numbers without a remainder.

For example, the GCD of 15 and 25 is 5. You can divide both 15 and 25 by 5 without any remainder. There is no greater number that does the same. If you take 15 and 30, then the GCD is 15 because both 15 and 30 can be divided by 15 without a remainder.

You don’t have to implement your own functions to calculate GCD. The Python math module provides a function called math.gcd() that allows you to calculate the GCD of two numbers. You can give positive or negative numbers as input, and it returns the appropriate GCD value. You can’t input a decimal number, however.

Calculate the Sum of Iterables

If you ever want to find the sum of the values of an iterable without using a loop, then math.fsum() is probably the easiest way to do so. You can use iterables such as arrays, tuples, or lists as input and the function returns the sum of the values. A built-in function called sum() lets you calculate the sum of iterables as well, but fsum() is more accurate than sum(). You can read more about that in the documentation.

Calculate the Square Root

The square root of a number is a value that, when multiplied by itself, gives the number. You can use math.sqrt() to find the square root of any positive real number (integer or decimal). The return value is always a float value. The function will throw a ValueError if you try to enter a negative number.

Convert Angle Values

In real-life scenarios as well as in mathematics, you often come across instances where you have to measure angles to perform calculations. Angles can be measured either by degrees or by radians. Sometimes you have to convert degrees to radians and vice versa. The math module provides functions that let you do so.

If you want to convert degrees to radians, then you can use math.radians(). It returns the radian value of the degree input. Likewise, if you want to convert radians to degrees, then you can use math.degrees().

Calculate Trigonometric Values

Trigonometry is the study of triangles. It deals with the relationship between angles and the sides of a triangle. Trigonometry is mostly interested in right-angled triangles (in which one internal angle is 90 degrees), but it can also be applied to other types of triangles. The Python math module provides very useful functions that let you perform trigonometric calculations.

You can calculate the sine value of an angle with math.sin(), the cosine value with math.cos(), and the tangent value with math.tan(). The math module also provides functions to calculate arc sine with math.asin(), arc cosine with math.acos(), and arc tangent with math.atan(). Finally, you can calculate the hypotenuse of a triangle using math.hypot().

New Additions to the `math` Module in Python 3.8

With the release of Python version 3.8, a few new additions and changes have been made to the math module. The new additions and changes are as follows:

comb(n, k) returns the number of ways to choose k items from n items without repetition and without particular order.
perm(n, k) returns the number of ways to choose k items from n items without repetition and with order.
isqrt() returns the integer square root of a non-negative integer.
prod() calculates the product of all of the elements in the input iterable. As with fsum(), this method can take iterables such as arrays, lists, or tuples.
dist() returns the Euclidean distance between two points p and q, each given as a sequence (or iterable) of coordinates. The two points must have the same dimension.
hypot() now handles more than two dimensions. Previously, it supported a maximum of two dimensions.

`cmath` vs `math`

A complex number is a combination of a real number and an imaginary number. It has the formula of a + bi, where a is the real number and bi is the imaginary number. Real and imaginary numbers can be explained as follows:

A real number is literally any number you can think of.
An imaginary number is a number that gives a negative result when squared.

A real number can be any number. For example, 12, 4.3, -19.0 are all real numbers. Imaginary numbers are shown as i. The following image shows an example of a complex number:

Complex Number

In the example above, 7 is the real number and 3i is the imaginary number. Complex numbers are mostly used in geometry, calculus, scientific calculations, and especially in electronics.

The functions of the Python math module aren’t equipped to handle complex numbers. However, Python provides a different module that can specifically deal with complex numbers, the cmath module. The Python math module is complemented by the cmath module, which implements many of the same functions but for complex numbers.

You can import the cmath module as follows:

>>>

>>> import cmath

Since the cmath module is also packaged with Python, you can import it the same way you imported the math module. Before you work with the cmath module, you have to know how to define a complex number. You can define a complex number as follows:

>>>

>>> c = 2 + 3j
>>> c
(2+3j)

>>> type(c)
<class 'complex'>

As you can see, you can determine that a number is indeed complex by using type().

Note: In mathematics, the imaginary unit is usually denoted i. In some fields, it’s more customary to use j for the same thing. In Python, you use j to denote imaginary numbers.

Python also provides a special built-in function called complex() that lets you create complex numbers. You can use complex() as follows:

>>>

>>> c = complex(2, 3)
>>> c
(2+3j)

>>> type(c)
<class 'complex'>

You can use either method to create complex numbers. You can also use the cmath module to calculate mathematical functions for complex numbers as follows:

>>>

>>> cmath.sqrt(c)
(1.8581072140693775+0.6727275964137814j)

>>> cmath.log(c)
(1.3622897515267103+0.6947382761967031j)

>>> cmath.exp(c)
(-16.091399670844+12.02063434789931j)

This example shows you how to calculate the square root, logarithmic value, and exponential value of a complex number. You can read the documentation if you want to learn more about the cmath module.

NumPy vs `math`

Several notable Python libraries can be used for mathematical calculations. One of the most prominent libraries is Numerical Python, or NumPy. It is mainly used in scientific computing and in data science fields. Unlike the math module, which is part of the standard Python release, you have to install NumPy in order to work with it.

The heart of NumPy is the high-performance N-dimensional (multidimensional) array data structure. This array allows you to perform mathematical operations on an entire array without looping over the elements. All of the functions in the library are optimized to work with the N-dimensional array objects.

Both the math module and the NumPy library can be used for mathematical calculations. NumPy has several similarities with the math module. NumPy has a subset of functions, similar to math module functions, that deal with mathematical calculations. Both NumPy and math provide functions that deal with trigonometric, exponential, logarithmic, hyperbolic and arithmetic calculations.

There are also several fundamental differences between math and NumPy. The Python math module is geared more towards working with scalar values, whereas NumPy is better suited for working with arrays, vectors, and even matrices.

When working with scalar values, math module functions can be faster than their NumPy counterparts. This is because the NumPy functions convert the values to arrays under the hood in order to perform calculations on them. NumPy is much faster when working with N-dimensional arrays because of the optimizations for them. Except for fsum() and prod(), the math module functions can’t handle arrays.

Conclusion

In this article, you learned about the Python math module. The module provides useful functions for performing mathematical calculations that have many practical applications.

In this article you’ve learned:

What the Python math module is
How to use math functions with practical examples
What the constants of the math module, including pi, tau, and Euler’s number are
What the differences between built-in functions and math functions are
What the differences between math, cmath, and NumPy are

Understanding how to use the math functions is the first step. Now it’s time to start applying what you learned to real-life situations. If you have any questions or comments, then please leave them in the comments section below.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast Is Here!

2020-03-20T20:18:40+00:00

The Real Python Podcast is finally live! Tune in for interesting guests, interviews with expert Pythonistas, and lots of behind-the-scenes with the Real Python Team.

Today we’re officially launching the Real Python Podcast, a new (and freely accessible) podcast for Pythonistas like you.

This has been in the making for a while, and both Christopher and I are super proud to finally release the first episode to you this week.

We’ll have a roster of interesting guests for you, interviews with expert Pythonistas, and lots of behind-the-scenes with the Real Python Team.

Here’s more info about the show and how you can listen to it:

What Is the Real Python Podcast?

It’s a free, weekly show with interviews, coding tips, and conversation with guests from the Python community, hosted by Real Python’s Christopher Bailey.

We’ll cover a wide range of topics including Python programming best practices, career tips, and related software development topics. And take a look behind the scenes at Real Python.

So join us to hear what’s new in the world of Python programming and become a more effective Pythonista.

How Can You Listen to the Show?

The show is available now on all common podcast directories, like Apple Podcasts, Google Podcasts, Spotify, Stitcher, and so on. Simply launch your favourite podcast listening app and search for “Real Python”.

Or alternatively, you can listen online at realpython.com/podcast

When Does the Show Launch?

As of Friday, March 20, 2020 the Real Python Podcast is available on all common podcast directories. So warm up your podcast apps 🙂

We're Celebrating the Podcast Launch in the RP Slack Community

We Want to Hear From You!

This is a podcast by the community, for the community. And we want you to be involved, we want to hear your feedback, and we want to make the podcast better and better over time.

Do you have an idea for an episode? Do you want to share your story with us or ask Christopher a question? Tweet at @realpython, or leave us a voicemail for a chance to be featured on the show.

How Can You Support the Show?

Subscribe using your favorite podcast app, leave a review for the podcast on Apple Podcasts or other directories, and tell the world about the show on Facebook, LinkedIn, and Twitter.

Here’s the link to the podcast and the show notes again:

Listen to the Real Python Podcast »

Happy Pythoning!

— Dan Bader

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #1: Python Decorators and Writing for Real Python

2020-03-20T14:00:00+00:00

In this first episode, Christopher interviews Geir Arne Hjelle from the Real Python Team. You’ll learn about Geir Arne’s background as a Pythonista and PyCon speaker, the tutorials he’s written for the site, how Python decorators can help you write better code, and what Real Python’s tutorial publishing process looks like behind the scenes.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Scope & the LEGB Rule: Resolving Names in Your Code

2020-03-18T14:00:00+00:00

The concept of scope rules how variables and names are looked up in your code. It determines the visibility of a variable within the code. The scope of a name or variable depends on the place in your code where you create that variable. The Python scope concept is generally presented using a rule known as the LEGB rule.

The letters in the acronym LEGB stand for Local, Enclosing, Global, and Built-in scopes. This summarizes not only the Python scope levels but also the sequence of steps that Python follows when resolving names in a program.

In this tutorial, you’ll learn:

What scopes are and how they work in Python
Why it’s important to know about Python scope
What the LEGB rule is and how Python uses it to resolve names
How to modify the standard behavior of Python scope using global and nonlocal
What scope-related tools Python offers and how you can use them

With this knowledge at hand, you can take advantage of Python scopes to write more reliable and maintainable programs. Using Python scope will help you avoid or minimize bugs related to name collision as well as bad use of global names across your programs.

You’ll get the most out of this tutorial if you’re familiar with intermediate Python concepts like classes, functions, inner functions, variables, exceptions, comprehensions, built-in functions, and standard data structures.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Understanding Scope

In programming, the scope of a name defines the area of a program in which you can unambiguously access that name, such as variables, functions, objects, and so on. A name will only be visible to and accessible by the code in its scope. Several programming languages take advantage of scope for avoiding name collisions and unpredictable behaviors. Most commonly, you’ll distinguish two general scopes:

Global scope: The names that you define in this scope are available to all your code.
Local scope: The names that you define in this scope are only available or visible to the code within the scope.

Scope came about because early programming languages (like BASIC) only had global names. With this kind of name, any part of the program could modify any variable at any time, so maintaining and debugging large programs could become a real nightmare. To work with global names, you’d need to keep all the code in mind at the same time to know what the value of a given name is at any time. This was an important side-effect of not having scopes.

Some languages like Python use scope to avoid this kind of problem. When you use a language that implements scope, there’s no way for you to access all the variables in a program at all locations in that program. In this case, your ability to access a given name will depend on where you’ve defined that name.

Note: You’ll be using the term name to refer to the identifiers of variables, constants, functions, classes, or any other object that can be assigned a name.

The names in your programs will have the scope of the block of code in which you define them. When you can access the value of a given name from someplace in your code, you’ll say that the name is in scope. If you can’t access the name, then you’ll say that the name is out of scope.

Names and Scopes in Python

Since Python is a dynamically-typed language, variables in Python come into existence when you first assign them a value. On the other hand, functions and classes are available after you define them using def or class, respectively. Finally, modules exist after you import them. As a summary, you can create Python names through one of the following operations:

Operation	Statement
Assignments	`x = value`
Import operations	`import module` or `from module import name`
Function definitions	`def my_func(): ...`
Argument definitions in the context of functions	`def my_func(arg1, arg2,... argN): ...`
Class definitions	`class MyClass: ...`

All these operations create or, in the case of assignments, update new Python names because all of them assign a name to a variable, constant, function, class, instance, module, or other Python object.

Note: There’s an important difference between assignment operations and reference or access operations. When you reference a name, you’re just retrieving its content or value. When you assign a name, you’re either creating that name or modifying it.

Python uses the location of the name assignment or definition to associate it with a particular scope. In other words, where you assign or define a name in your code determines the scope or visibility of that name.

For example, if you assign a value to a name inside a function, then that name will have a local Python scope. In contrast, if you assign a value to a name outside of all functions—say, at the top level of a module—then that name will have a global Python scope.

Python Scope vs Namespace

In Python, the concept of scope is closely related to the concept of the namespace. As you’ve learned so far, a Python scope determines where in your program a name is visible. Python scopes are implemented as dictionaries that map names to objects. These dictionaries are commonly called namespaces. These are the concrete mechanisms that Python uses to store names. They’re stored in a special attribute called .__dict__.

Names at the top level of a module are stored in the module’s namespace. In other words, they’re stored in the module’s .__dict__ attribute. Take a look at the following code:

>>>

>>> import sys
>>> sys.__dict__.keys()
dict_keys(['__name__', '__doc__', '__package__',..., 'argv', 'ps1', 'ps2'])

After you import sys, you can use .keys() to inspect the keys of sys.__dict__. This returns a list with all the names defined at the top level of the module. In this case, you can say that .__dict__ holds the namespace of sys and is a concrete representation of the module scope.

Note: The output of some of the examples in this tutorial has been abbreviated (...) to save space. The output may vary based on your platform, Python version, or even on how long you’ve been using your current Python interactive session.

As a further example, suppose that you need to use the name ps1, which is defined in sys. If you know how .__dict__ and namespaces work in Python, then you can reference ps1 in at least two different ways:

Using the dot notation on the module’s name in the form module.name
Using a subscription operation on .__dict__ in the form module.__dict__['name']

Take a look at the following code:

>>>

>>> sys.ps1
'>>> '
>>> sys.__dict__['ps1']
'>>> '

Once you’ve imported sys you can access ps1 using the dot notation on sys. You can also access ps1 using a dictionary key lookup with the key 'ps1'. Both actions return the same result, '>>> '.

Note: ps1 is a string specifying the primary prompt of the Python interpreter. ps1 is only defined if the interpreter is in interactive mode and its initial value is '>>> '.

Whenever you use a name, such as a variable or a function name, Python searches through different scope levels (or namespaces) to determine whether the name exists or not. If the name exists, then you’ll always get the first occurrence of it. Otherwise, you’ll get an error. You’ll cover this search mechanism in the next section.

Using the LEGB Rule for Python Scope

Python resolves names using the so-called LEGB rule, which is named after the Python scope for names. The letters in LEGB stand for Local, Enclosing, Global, and Built-in. Here’s a quick overview of what these terms mean:

Local (or function) scope is the code block or body of any Python function or lambda expression. This Python scope contains the names that you define inside the function. These names will only be visible from the code of the function. It’s created at function call, not at function definition, so you’ll have as many different local scopes as function calls. This is true even if you call the same function multiple times, or recursively. Each call will result in a new local scope being created.
Enclosing (or nonlocal) scope is a special scope that only exists for nested functions. If the local scope is an inner or nested function, then the enclosing scope is the scope of the outer or enclosing function. This scope contains the names that you define in the enclosing function. The names in the enclosing scope are visible from the code of the inner and enclosing functions.
Global (or module) scope is the top-most scope in a Python program, script, or module. This Python scope contains all of the names that you define at the top level of a program or a module. Names in this Python scope are visible from everywhere in your code.
Built-in scope is a special Python scope that’s created or loaded whenever you run a script or open an interactive session. This scope contains names such as keywords, functions, exceptions, and other attributes that are built into Python. Names in this Python scope are also available from everywhere in your code. It’s automatically loaded by Python when you run a program or script.

The LEGB rule is a kind of name lookup procedure, which determines the order in which Python looks up names. For example, if you reference a given name, then Python will look that name up sequentially in the local, enclosing, global, and built-in scope. If the name exists, then you’ll get the first occurrence of it. Otherwise, you’ll get an error.

Note: Notice that the local and enclosing Python scopes are searched only if you use a name inside a function (local scope) or a nested or inner function (local and enclosing scope).

In summary, when you use nested functions, names are resolved by first checking the local scope or the innermost function’s local scope. Then, Python looks at all enclosing scopes of outer functions from the innermost scope to the outermost scope. If no match is found, then Python looks at the global and built-in scopes. If it can’t find the name, then you’ll get an error.

At any given time during execution, you’ll have at most four active Python scopes—local, enclosing, global, and built-in—depending on where you are in the code. On the other hand, you’ll always have at least two active scopes, which are the global and built-in scopes. These two scopes will always be available for you.

Functions: The Local Scope

The local scope or function scope is a Python scope created at function calls. Every time you call a function, you’re also creating a new local scope. On the other hand, you can think of each def statement and lambda expression as a blueprint for new local scopes. These local scopes will come into existence whenever you call the function at hand.

By default, parameters and names that you assign inside a function exist only within the function or local scope associated with the function call. When the function returns, the local scope is destroyed and the names are forgotten. Here’s how this works:

>>>

>>> def square(base):
...     result = base ** 2
...     print(f'The square of {base} is: {result}')
...
>>> square(10)
The square of 10 is: 100
>>> result  # Isn't accessible from outside square()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    result
NameError: name 'result' is not defined
>>> base  # Isn't accessible from outside square()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    base
NameError: name 'base' is not defined
>>> square(20)
The square of 20 is: 400

square() is a function that computes the square of a given number, base. When you call the function, Python creates a local scope containing the names base (an argument) and result (a local variable). After the first call to square(), base holds a value of 10 and result holds a value of 100. The second time, the local names will not remember the values that were stored in them the first time the function was called. Notice that base now holds the value 20 and result holds 400.

Note: If you try to access result or base after the function call, then you get a NameError, because these only exist in the local scope created by the call to square(). Whenever you try to access a name that isn’t defined in any Python scope, you’ll get a NameError. The error message will include the name that couldn’t be found.

Since you can’t access local names from statements that are outside the function, different functions can define objects with the same name. Check out this example:

>>>

>>> def cube(base):
...     result = base ** 3
...     print(f'The cube of {base} is: {result}')
...
>>> cube(30)
The cube of 30 is: 27000

Notice that you define cube() using the same variable and parameter that you used in square(). However, since cube() can’t see the names inside the local scope of square() and vice versa, both functions work as expected without any name collision.

You can avoid name collisions in your programs by properly using the local Python scope. This also makes functions more self-contained and creates maintainable program units. Additionally, since you can’t change local names from remote places in your code, your programs will be easier to debug, read, and modify.

You can inspect the names and parameters of a function using .__code__, which is an attribute that holds information on the function’s internal code. Take a look at the code below:

>>>

>>> square.__code__.co_varnames
('base', 'result')
>>> square.__code__.co_argcount
1
>>> square.__code__.co_consts
(None, 2, 'The square of ', ' is: ')
>>> square.__code__.co_name
'square'

In this code example, you inspect .__code__ on square(). This is a special attribute that holds information about the code of a Python function. In this case, you see that .co_varnames holds a tuple containing the names that you define inside square().

Nested Functions: The Enclosing Scope

Enclosing or nonlocal scope is observed when you nest functions inside other functions. The enclosing scope was added in Python 2.2. It takes the form of the local scope of any enclosing function’s local scopes. Names that you define in the enclosing Python scope are commonly known as nonlocal names. Consider the following code:

>>>

>>> def outer_func():
...     # This block is the Local scope of outer_func()
...     var = 100  # A nonlocal var
...     # It's also the enclosing scope of inner_func()
...     def inner_func():
...         # This block is the Local scope of inner_func()
...         print(f"Printing var from inner_func(): {var}")
...
...     inner_func()
...     print(f"Printing var from outer_func(): {var}")
...
>>> outer_func()
Printing var from inner_func(): 100
Printing var from outer_func(): 100
>>> inner_func()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'inner_func' is not defined

When you call outer_func(), you’re also creating a local scope. The local scope of outer_func() is, at the same time, the enclosing scope of inner_func(). From inside inner_func(), this scope is neither the global scope nor the local scope. It’s a special scope that lies in between those two scopes and is known as the enclosing scope.

Note: In a sense, inner_func() is a temporary function that comes to life only during the execution of its enclosing function, outer_func(). Note that inner_func() is only visible to the code in outer_func().

All the names that you create in the enclosing scope are visible from inside inner_func(), except for those created after you call inner_func(). Here’s a new version of outer_fun() that shows this point:

>>>

>>> def outer_func():
...     var = 100
...     def inner_func():
...         print(f"Printing var from inner_func(): {var}")
...         print(f"Printing another_var from inner_func(): {another_var}")
...
...     inner_func()
...     another_var = 200  # This is defined after calling inner_func()
...     print(f"Printing var from outer_func(): {var}")
...
>>> outer_func()
Printing var from inner_func(): 100
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    outer_func()
  File "<stdin>", line 7, in outer_func
    inner_func()
  File "<stdin>", line 5, in inner_func
    print(f"Printing another_var from inner_func(): {another_var}")
NameError: free variable 'another_var' referenced before assignment in enclosing
 scope

When you call outer_func() the code runs down to the point in which you call inner_func(). The last statement of inner_func() tries to access another_var. At this point, another_var isn’t defined yet, so Python raises a NameError because it can’t find the name that you’re trying to use.

Last but not least, you can’t modify names in the enclosing scope from inside a nested function unless you declare them as nonlocal in the nested function. You’ll cover how to use nonlocal later in this tutorial.

Modules: The Global Scope

From the moment you start a Python program, you’re in the global Python scope. Internally, Python turns your program’s main script into a module called __main__ to hold the main program’s execution. The namespace of this module is the main global scope of your program.

Note: In Python, the notions of global scope and global names are tightly associated with module files. For example, if you define a name at the top level of any Python module, then that name is considered global to the module. That’s the reason why this kind of scope is also called module scope.

If you’re working in a Python interactive session, then you’ll notice that '__main__' is also the name of its main module. To check that out, open an interactive session and type in the following:

>>>

>>> __name__
'__main__'

Whenever you run a Python program or an interactive session like in the above code, the interpreter executes the code in the module or script that serves as an entry point to your program. This module or script is loaded with the special name, __main__. From this point on, you can say that your main global scope is the scope of __main__.

To inspect the names within your main global scope, you can use dir(). If you call dir() without arguments, then you’ll get the list of names that live in your current global scope. Take a look at this code:

>>>

>>> dir()
['__annotations__', '__builtins__',..., '__package__', '__spec__']
>>> var = 100  # Assign var at the top level of __main__
>>> dir()
['__annotations__', '__builtins__',..., '__package__', '__spec__', 'var']

When you call dir() with no arguments, you get the list of names available in your main global Python scope. Note that if you assign a new name (like var here) at the top level of the module (which is __main__ here), then that name will be added to the list returned by dir().

Note: You’ll cover dir() in more detail later on in this tutorial.

There’s only one global Python scope per program execution. This scope remains in existence until the program terminates and all its names are forgotten. Otherwise, the next time you were to run the program, the names would remember their values from the previous run.

You can access or reference the value of any global name from any place in your code. This includes functions and classes. Here’s an example that clarifies these points:

>>>

>>> var = 100
>>> def func():
...     return var  # You can access var from inside func()
...
>>> func()
100
>>> var  # Remains unchanged
100

Inside func(), you can freely access or reference the value of var. This has no effect on your global name var, but it shows you that var can be freely accessed from within func(). On the other hand, you can’t assign global names inside functions unless you explicitly declare them as global names using a global statement, which you’ll see later on.

Whenever you assign a value to a name in Python, one of two things can happen:

You create a new name
You update an existing name

The concrete behavior will depend on the Python scope in which you’re assigning the name. If you try to assign a value to a global name inside a function, then you’ll be creating that name in the function’s local scope, shadowing or overriding the global name. This means that you won’t be able to change most variables that have been defined outside the function from within the function.

If you follow this logic, then you’ll realize that the following code won’t work as you might expect:

>>>

>>> var = 100  # A global variable
>>> def increment():
...     var = var + 1  # Try to update a global variable
...
>>> increment()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    increment()
  File "<stdin>", line 2, in increment
    var = var + 1
UnboundLocalError: local variable 'var' referenced before assignment

Within increment(), you try to increment the global variable, var. Since var isn’t declared global inside increment(), Python creates a new local variable with the same name, var, inside the function. In the process, Python realizes that you’re trying to use the local var before its first assignment (var + 1), so it raises an UnboundLocalError.

Here’s another example:

>>>

>>> var = 100  # A global variable
>>> def func():
...     print(var)  # Reference the global variable, var
...     var = 200   # Define a new local variable using the same name, var
...
>>> func()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    func()
  File "<stdin>", line 2, in func
    print(var)
UnboundLocalError: local variable 'var' referenced before assignment

You likely expect to be able to print the global var and be able to update var later, but again you get an UnboundLocalError. What happens here is that when you run the body of func(), Python decides that var is a local variable because it’s assigned within the function scope. This isn’t a bug, but a design choice. Python assumes that names assigned in the body of a function are local to that function.

Note: Global names can be updated or modified from any place in your global Python scope. Beyond that, the global statement can be used to modify global names from almost any place in your code, as you’ll see in The global Statement.

Modifying global names is generally considered bad programming practice because it can lead to code that is:

Difficult to debug: Almost any statement in the program can change the value of a global name.
Hard to understand: You need to be aware of all the statements that access and modify global names.
Impossible to reuse: The code is dependent on global names that are specific to a concrete program.

Good programming practice recommends using local names rather than global names. Here are some tips:

Write self-contained functions that rely on local names rather than global ones.
Try to use unique objects names, no matter what scope you’re in.
Avoid global name modifications throughout your programs.
Avoid cross-module name modifications.
Use global names as constants that don’t change during your program’s execution.

Up to this point, you’ve covered three Python scopes. Check out the following example for a summary on where they’re located in your code and how Python looks up names through them:

>>>

>>> # This area is the global or module scope
>>> number = 100
>>> def outer_func():
...     # This block is the local scope of outer_func()
...     # It's also the enclosing scope of inner_func()
...     def inner_func():
...         # This block is the local scope of inner_func()
...         print(number)
...
...     inner_func()
...
>>> outer_func()
100

When you call outer_func(), you get 100 printed on your screen. But how does Python look up the name number in this case? Following the LEGB rule, you’ll look up number in the following places:

Inside inner_func(): This is the local scope, but number doesn’t exist there.
Inside outer_func(): This is the enclosing scope, but number isn’t defined there either.
In the module scope: This is the global scope, and you find number there, so you can print number to the screen.

If number isn’t defined inside the global scope, then Python continues the search by looking at the built-in scope. This is the last component of the LEGB rule, as you’ll see in the next section.

`builtins`: The Built-In Scope

The built-in scope is a special Python scope that’s implemented as a standard library module named builtins in Python 3.x. All of Python’s built-in objects live in this module. They’re automatically loaded to the built-in scope when you run the Python interpreter. Python searches builtins last in its LEGB lookup, so you get all the names it defines for free. This means that you can use them without importing any module.

Notice that the names in builtins are always loaded into your global Python scope with the special name __builtins__, as you can see in the following code:

>>>

>>> dir()
['__annotations__', '__builtins__',..., '__package__', '__spec__']
>>> dir(__builtins__)
['ArithmeticError', 'AssertionError',..., 'tuple', 'type', 'vars', 'zip']

In the output of the first call to dir(), you can see that __builtins__ is always present in the global Python scope. If you inspect __builtins__ itself using dir(), then you’ll get the whole list of Python built-in names.

The built-in scope brings more than 150 names to your current global Python scope. For example, in Python 3.8 you can get to know the exact number of names as follows:

>>>

>>> len(dir(__builtins__))
152

With the call to len(), you get the number of items in the list returned by dir(). This returns 152 names that include exceptions, functions, types, special attributes, and other Python built-in objects.

Even though you can access all of these Python built-in objects for free (without importing anything), you can also explicitly import builtins and access the names using the dot notation. Here’s how this works:

>>>

>>> import builtins  # Import builtins as a regular module
>>> dir(builtins)
['ArithmeticError', 'AssertionError',..., 'tuple', 'type', 'vars', 'zip']
>>> builtins.sum([1, 2, 3, 4, 5])
15
>>> builtins.max([1, 5, 8, 7, 3])
8
>>> builtins.sorted([1, 5, 8, 7, 3])
[1, 3, 5, 7, 8]
>>> builtins.pow(10, 2)
100

You can import builtins as you would any other Python module. From this point on, you can access all the names in builtins by using the dotted attribute lookup or fully-qualified names. This can be quite useful if you want to make sure that you won’t have a name collision if any of your global names override any built-in name.

You can override or redefine any built-in name in your global scope. If you do so, then keep in mind that this will affect all your code. Take a look at the following example:

>>>

>>> abs(-15)  # Standard use of a built-in function
15
>>> abs = 20  # Redefine a built-in name in the global scope
>>> abs(-15)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not callable

If you override or re-assign abs, then the original built-in abs() is affected all over your code. Now, suppose that you need to call the original abs() and you forget that you re-assigned the name. In this case, when you call abs() again, you’d get a TypeError because abs now holds a reference to an integer, which is not callable.

Note: Accidentally or inadvertently overriding or redefining built-in names in your global scope can be a source of dangerous and hard-to-find bugs. It’s better to try and avoid this kind of practice.

If you’re experimenting with some code and you accidentally re-assign a built-in name at the interactive prompt, then you can either restart your session or run del name to remove the redefinition from your global Python scope. This way, you’re restoring the original name in the built-in scope. If you revisit the example of abs(), then you can do something like this:

>>>

>>> del abs  # Remove the redefined abs from your global scope
>>> abs(-15)  # Restore the original abs()
15

When you delete the custom abs name, you’re removing the name from your global scope. This allows you to access the original abs() in the built-in scope again.

To work around this kind of situation, you can explicitly import builtins and then use fully-qualified names, like in the following code fragment:

>>>

>>> import builtins
>>> builtins.abs(-15)
15

Once you explicitly import builtins, you have the module name available in your global Python scope. From this point on, you can use fully-qualified names to unambiguously get the names you need from builtins, just like you did with builtins.abs() in the above example.

As a quick summary, some of the implications of Python scope are shown in the following table:

Action	Global Code	Local Code	Nested Function Code
Access or reference names that live in the global scope	Yes	Yes	Yes
Modify or update names that live in the global scope	Yes	No (unless declared `global`)	No (unless declared `global`)
Access or reference names that live in a local scope	No	Yes (its own local scope), No (other local scope)	Yes (its own local scope), No (other local scope)
Override names in the built-in scope	Yes	Yes (during function execution)	Yes (during function execution)
Access or reference names that live in their enclosing scope	N/A	N/A	Yes
Modify or update names that live in their enclosing scope	N/A	N/A	No (unless declared `nonlocal`)

Additionally, code in different scopes can use the same name for different objects. This way, you can use a local variable named spam and also a global variable with the same name, spam. However, this is considered bad programming practice.

Modifying the Behavior of a Python Scope

So far, you’ve learned how a Python scope works and how they restrict the visibility of variables, functions, classes, and other Python objects to certain portions of your code. You now know that you can access or reference global names from any place in your code, but they can be modified or updated from within the global Python scope.

You also know that you can access local names only from inside the local Python scope they were created in or from inside a nested function, but you can’t access them from the global Python scope or from other local scopes. Additionally, you’ve learned that nonlocal names can be accessed from inside nested functions, but they can’t be modified or updated from there.

Even though Python scopes follow these general rules by default, there are ways to modify this standard behavior. Python provides two keywords that allow you to modify the content of global and nonlocal names. These two keywords are:

In the next two sections, you’ll cover how to use these Python keywords to modify the standard behavior of Python scopes.

The `global` Statement

You already know that when you try to assign a value to a global name inside a function, you create a new local name in the function scope. To modify this behavior, you can use a global statement. With this statement, you can define a list of names that are going to be treated as global names.

The statement consists of the global keyword followed by one or more names separated by commas. You can also use multiple global statements with a name (or a list of names). All the names that you list in a global statement will be mapped to the global or module scope in which you define them.

Here’s an example where you try to update a global variable from within a function:

>>>

>>> counter = 0  # A global name
>>> def update_counter():
...     counter = counter + 1  # Fail trying to update counter
...
>>> update_counter()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in update_counter
UnboundLocalError: local variable 'counter' referenced before assignment

When you try to assign counter inside update_counter(), Python assumes that counter is local to update_counter() and raises an UnboundLocalError because you’re trying to access a name that isn’t defined yet.

If you want this code to work the way you expect here, then you can use a global statement as follows:

>>>

>>> counter = 0  # A global name
>>> def update_counter():
...     global counter  # Declare counter as global
...     counter = counter + 1  # Successfully update the counter
...
>>> update_counter()
>>> counter
1
>>> update_counter()
>>> counter
2
>>> update_counter()
>>> counter
3

In this new version of update_counter(), you add the statement global counter to the body of the function right before you try to change counter. With this tiny change, you’re mapping the name counter in the function scope to the same name in the global or module scope. From this point on, you can freely modify counter inside update_counter(). All the changes will reflect in the global variable.

With the statement global counter, you’re telling Python to look in the global scope for the name counter. This way, the expression counter = counter + 1 doesn’t create a new name in the function scope, but updates it in the global scope.

Note: The use of global is considered bad practice in general. If you find yourself using global to fix problems like the one above, then stop and think if there is a better way to write your code.

For example, you can try to write a self-contained function that relies on local names rather than on global names as follows:

>>>

>>> global_counter = 0  # A global name
>>> def update_counter(counter):
...     return counter + 1  # Rely on a local name
...
>>> global_counter = update_counter(global_counter)
>>> global_counter
1
>>> global_counter = update_counter(global_counter)
>>> global_counter
2
>>> global_counter = update_counter(global_counter)
>>> global_counter
3

This implementation of update_counter() defines counter as a parameter and returns its value augmented by 1 unit every time the function is called. This way, the result of update_counter() depends on the counter you use as an input and not on the changes that other functions (or pieces of code) can perform on the global variable, global_counter.

You can also use a global statement to create lazy global names by declaring them inside a function. Take a look at the following code:

>>>

>>> def create_lazy_name():
...     global lazy  # Create a global name, lazy
...     lazy = 100
...     return lazy
...
>>> create_lazy_name()
100
>>> lazy  # The name is now available in the global scope
100
>>> dir()
['__annotations__', '__builtins__',..., 'create_lazy_name', 'lazy']

When you call create_lazy_name(), you’re also creating a global variable called lazy. Notice that after calling the function, the name lazy is available in the global Python scope. If you inspect the global namespace using dir(), then you’ll see that lazy appears last in the list.

Note: Even though you can use a global statement to create lazy global names, this can be a dangerous practice that can lead to buggy code. So, it’s best to avoid things like this in your code.

For example, suppose you’re trying to get access to one of those lazy names and, for some reason, your code hasn’t called the function that creates that name yet. In this case, you’ll get a NameError and your program will crash.

Finally, it’s worth noting that you can use global from inside any function or nested function and the names listed will always be mapped to names in the global Python scope.

Also notice that, even though using a global statement at the top level of a module is legal, it doesn’t make much sense because any name assigned in the global scope is already a global name by definition. Take a look at the following code:

>>>

>>> name = 100
>>> dir()
['__annotations__', '__builtins__',..., '__spec__', 'name']
>>> global name
>>> dir()
['__annotations__', '__builtins__',..., '__spec__', 'name']

The use of a global statement like global name doesn’t change anything in your current global scope, as you can see in the output of dir(). The variable name is a global variable whether you use global or not.

The `nonlocal` Statement

Similarly to global names, nonlocal names can be accessed from inner functions, but not assigned or updated. If you want to modify them, then you need to use a nonlocal statement. With a nonlocal statement, you can define a list of names that are going to be treated as nonlocal.

The nonlocal statement consists of the nonlocal keyword followed by one or more names separated by commas. These names will refer to the same names in the enclosing Python scope. The following example shows how you can use nonlocal to modify a variable defined in the enclosing or nonlocal scope:

>>>

>>> def func():
...     var = 100  # A nonlocal variable
...     def nested():
...         nonlocal var  # Declare var as nonlocal
...         var += 100
...
...     nested()
...     print(var)
...
>>> func()
200

With the statement nonlocal var, you tell Python that you’ll be modifying var inside nested(). Then, you increment var using an augmented assignment operation. This change is reflected in the nonlocal name var, which now has a value of 200.

Unlike global, you can’t use nonlocal outside of a nested or enclosed function. To be more precise, you can’t use a nonlocal statement in either the global scope or in a local scope. Here’s an example:

>>>

>>> nonlocal my_var  # Try to use nonlocal in the global scope
  File "<stdin>", line 1
SyntaxError: nonlocal declaration not allowed at module level
>>> def func():
...     nonlocal var  # Try to use nonlocal in a local scope
...     print(var)
...
  File "<stdin>", line 2
SyntaxError: no binding for nonlocal 'var' found

Here, you first try to use a nonlocal statement in the global Python scope. Since nonlocal only works inside an inner or nested function, you get a SyntaxError telling you that you can’t use nonlocal in a module scope. Notice that nonlocal doesn’t work inside a local scope either.

Note: For more detailed information on the nonlocal statement, check out PEP 3104 — Access to Names in Outer Scopes.

In contrast to global, you can’t use nonlocal to create lazy nonlocal names. Names must already exist in the enclosing Python scope if you want to use them as nonlocal names. This means that you can’t create nonlocal names by declaring them in a nonlocal statement in a nested function. Take a look at the following code example:

>>>

>>> def func():
...     def nested():
...         nonlocal lazy_var  # Try to create a nonlocal lazy name
...
  File "<stdin>", line 3
SyntaxError: no binding for nonlocal 'lazy_var' found

In this example, when you try to define a nonlocal name using nonlocal lazy_var, Python immediately raises a SyntaxError because lazy_var doesn’t exist in the enclosing scope of nested().

Using Enclosing Scopes as Closures

Closures are a special use case of the enclosing Python scope. When you handle a nested function as data, the statements that make up that function are packaged together with the environment in which they execute. The resulting object is known as a closure. In other words, a closure is an inner or nested function that carries information about its enclosing scope, even though this scope has completed its execution.

Note: You can also call this kind of function a factory, a factory function, or—to be more precise—a closure factory to specify that the function builds and returns closures (an inner function), rather than classes or instances.

Closures provide a way to retain state information between function calls. This can be useful when you want to write code based on the concept of lazy or delayed evaluation. Take a look at the following code for an example of how closures work and how you can take advantage of them in Python:

>>>

>>> def power_factory(exp):
...     def power(base):
...         return base ** exp
...     return power
...
>>> square = power_factory(2)
>>> square(10)
100
>>> cube = power_factory(3)
>>> cube(10)
1000
>>> cube(5)
125
>>> square(15)
225

Your closure factory function power_factory() takes an argument called exp. You can use this function to build closures that run different power operations. This works because each call to power_factory() gets its own set of state information. In other words, it gets its value for exp.

Note: Variables like exp are called free variables. They are variables that are used in a code block but not defined there. Free variables are the mechanism that closures use to retain state information between calls.

In the above example, the inner function power() is first assigned to square. In this case, the function remembers that exp equals 2. In the second example, you call power_factory() using 3 as an argument. This way, cube holds a function object, which remembers that exp is 3. Notice that you can freely reuse square and cube because they don’t forget their respective state information.

For a final example on how to use closures, suppose that you need to calculate the mean of some sample data. You collect the data through a stream of successive measurements of the parameter you’re analyzing. In this case, you can use a closure factory to generate a closure that remembers the previous measurements in the sample. Take a look at the following code:

>>>

>>> def mean():
...     sample = []
...     def _mean(number):
...         sample.append(number)
...         return sum(sample) / len(sample)
...     return _mean
...
>>> current_mean = mean()
>>> current_mean(10)
10.0
>>> current_mean(15)
12.5
>>> current_mean(12)
12.333333333333334
>>> current_mean(11)
12.0
>>> current_mean(13)
12.2

The closure that you create in the above code remembers the state information of sample between calls of current_mean. This way, you can solve the problem in an elegant and Pythonic way.

Notice that if your data stream gets too large, then this function can become a problem in terms of memory usage. That’s because with each call to current_mean, sample will hold a bigger and bigger list of values. Take a look at the following code for an alternative implementation using nonlocal:

>>>

>>> def mean():
...     total = 0
...     length = 0
...     def _mean(number):
...         nonlocal total, length
...         total += number
...         length += 1
...         return total / length
...     return _mean
...
>>> current_mean = mean()
>>> current_mean(10)
10.0
>>> current_mean(15)
12.5
>>> current_mean(12)
12.333333333333334
>>> current_mean(11)
12.0
>>> current_mean(13)
12.2

Even though this solution is more verbose, you don’t have an endlessly growing list anymore. You now have a single value for total and length. This implementation is a lot more efficient in terms of memory consumption than the previous solution.

Finally, you can find some examples of using closures in the Python standard library. For example, functools provides a function named partial() that makes use of the closure technique to create new function objects that can be called using predefined arguments. Here’s an example:

>>>

>>> from functools import partial
>>> def power(exp, base):
...     return base ** exp
...
>>> square = partial(power, 2)
>>> square(10)
100

You use partial to build a function object that remembers the state information, where exp=2. Then, you call this object to perform the power operation and get the final result.

Bringing Names to Scope With `import`

When you write a Python program, you typically organize the code into several modules. For your program to work, you’ll need to bring the names in those separate modules to your __main__ module. To do that, you need to import the modules or the names explicitly. This is the only way you can use those names in your main global Python scope.

Take a look at the following code for an example of what happens when you import some standard modules and names:

>>>

>>> dir()
['__annotations__', '__builtins__',..., '__spec__']
>>> import sys
>>> dir()
['__annotations__', '__builtins__',..., '__spec__', 'sys']
>>> import os
>>> dir()
['__annotations__', '__builtins__',..., '__spec__', 'os', 'sys']
>>> from functools import partial
>>> dir()
['__annotations__', '__builtins__',..., '__spec__', 'os', 'partial', 'sys']

You first import sys and os from the Python standard library. By calling dir() with no arguments, you can see that these modules are now available for you as names in your current global scope. This way, you can use dot notation to get access to the names that are defined in sys and os.

Note: If you want to dive deeper into how imports work in Python, then check out Absolute vs Relative Imports in Python.

In the latest import operation, you use the form from <module> import <name>. This way, you can use the imported name directly in your code. In other words, you don’t need to explicitly use the dot notation.

Discovering Unusual Python Scopes

You’ll find some Python structures where name resolution seems not to fit into the LEGB rule for Python scopes. These structures include:

In the next few sections, you’ll cover how Python scope works on these three structures. With this knowledge, you’ll be able to avoid subtle errors related to the use of names in these kinds of Python structures.

Comprehension Variables Scope

The first structure you’ll cover is the comprehension. A comprehension is a compact way to process all or part of the elements in a collection or sequence. You can use comprehensions to create lists, dictionaries, and sets.

Comprehensions consist of a pair of brackets ([]) or curly braces ({}) containing an expression, followed by one or more for clauses and then zero or one if clause per for clause.

The for clause in a comprehension works similarly to a traditional for loop. The loop variable in a comprehension is local to the structure. Check out the following code:

>>>

>>> [item for item in range(5)]
[0, 1, 2, 3, 4]
>>> item  # Try to access the comprehension variable
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    item
NameError: name 'item' is not defined

Once you run the list comprehension, the variable item is forgotten and you can’t access its value anymore. It’s unlikely that you need to use this variable outside of the comprehension, but regardless, Python makes sure that its value is no longer available once the comprehension finishes.

Note that this only applies to comprehensions. When it comes to regular for loops, the loop variable holds the last value processed by the loop:

>>>

>>> for item in range(5):
...     print(item)
...
0
1
2
3
4
>>> item  # Access the loop variable
4

You can freely access the loop variable item once the loop has finished. Here, the loop variable holds the last value processed by the loop, which is 4 in this example.

Exception Variables Scope

Another atypical case of Python scope that you’ll encounter is the case of the exception variable. The exception variable is a variable that holds a reference to the exception raised by a try statement. In Python 3.x, such variables are local to the except block and are forgotten when the block ends. Check out the following code:

>>>

>>> lst = [1, 2, 3]
>>> try:
...     lst[4]
... except IndexError as err:
...     # The variable err is local to this block
...     # Here you can do anything with err
...     print(err)
...
list index out of range
>>> err # Is out of scope
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    err
NameError: name 'err' is not defined

err holds a reference to the exception raised by the try clause. You can use err only inside the code block of the except clause. This way, you can say that the Python scope for the exception variable is local to the except code block. Also note that if you try to access err from outside the except block, then you’ll get a NameError. That’s because once the except block finishes, the name doesn’t exist anymore.

To work around this behavior, you can define an auxiliary variable out of the try statement and then assign the exception to that variable inside the except block. Check out the following example:

>>>

>>> lst = [1, 2, 3]
>>> ex = None
>>> try:
...     lst[4]
... except IndexError as err:
...     ex = err
...     print(err)
...
list index out of range
>>> err  # Is out of scope
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'err' is not defined
>>> ex  # Holds a reference to the exception
list index out of range

You use ex as an auxiliary variable to hold a reference to the exception raised by the try clause. This can be useful when you need to do something with the exception object once the code block has finished. Note that if no exception is raised, then ex remains None.

Class and Instance Attributes Scope

When you define a class, you’re creating a new local Python scope. The names assigned at the top level of the class live in this local scope. The names that you assigned inside a class statement don’t clash with names elsewhere. You can say that these names follow the LEGB rule, where the class block represents the L level.

Unlike functions, the class local scope isn’t created at call time, but at execution time. Each class object has its own .__dict__ attribute that holds the class scope or namespace where all the class attributes live. Check out this code:

>>>

>>> class A:
...     attr = 100
...
>>> A.__dict__.keys()
dict_keys(['__module__', 'attr', '__dict__', '__weakref__', '__doc__'])

When you inspect the keys of .__dict__ you’ll see that attr is in the list along with other special names. This dictionary represents the class local scope. The names in this scope are visible to all instances of the class and to the class itself.

To get access to a class attribute from outside the class, you need to use the dot notation as follows:

>>>

>>> class A:
...     attr = 100
...     print(attr)  # Access class attributes directly
...
100
>>> A.attr  # Access a class attribute from outside the class
100
>>> attr  # Isn't defined outside A
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    attr
NameError: name 'attr' is not defined

Inside the local scope of A, you can access the class attributes directly, just like you did in the statement print(attr). To access any class attribute once the code block of the class is executed, you’ll need to use the dot notation or attribute reference, as you did with A.attr. Otherwise, you’ll get a NameError, because the attribute attr is local to the class block.

On the other hand, if you try to access an attribute that isn’t defined inside a class, then you’ll get an AttributeError. Check out the following example:

>>>

>>> A.undefined  # Try to access an undefined class attribute
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    A.undefined
AttributeError: type object 'A' has no attribute 'undefined'

In this example, you try to access the attribute undefined. Since this attribute doesn’t exist in A, you get an AttributeError telling you that A doesn’t have an attribute named undefined.

You can also access any class attribute using an instance of the class as follows:

>>>

>>> obj = A()
>>> obj.attr
100

Once you have the instance you can access the class attributes using the dot notation, as you did here with obj.attr. Class attributes are specific to the class object, but you can access them from any instances of the class. It’s worth noting that class attributes are common to all instances of a class. If you modify a class attribute, then the changes will be visible in all instances of the class.

Note: Think of the dot notation as if you were telling Python, “Look for the attribute called attr in obj. If you find it, then give it back to me.”

Whenever you call a class, you’re creating a new instance of that class. Instances have their own .__dict__ attribute that holds the names in the instance local scope or namespace. These names are commonly called instance attributes and are local and specific to each instance. This means that if you modify an instance attribute, then the changes will be visible only to that specific instance.

To create, update, or access any instance attribute from inside the class, you need to use self along with the dot notation. Here, self is a special attribute that represents the current instance. On the other hand, to update or access any instance attribute from outside the class, you need to create an instance and then use the dot notation. Here’s how this works:

>>>

>>> class A:
...     def __init__(self, var):
...         self.var = var  # Create a new instance attribute
...         self.var *= 2  # Update the instance attribute
...
>>> obj = A(100)
>>> obj.__dict__
{'var': 200}
>>> obj.var
200

The class A takes an argument called var, which is automatically doubled inside .__init__() using the assignment operation self.var *= 2. Note that when you inspect .__dict__ on obj, you get a dictionary containing all instance attributes. In this case, the dictionary contains only the name var, whose value is now 200.

Note: For a more on how classes work in Python, check out Introduction to Object-Oriented Programming in Python.

Even though you can create instance attributes within any method in a class, it’s good practice to create and initialize them inside .__init__(). Take a look at this new version of A:

>>>

>>> class A:
...     def __init__(self, var):
...         self.var = var
...
...     def duplicate_var(self):
...         return self.var * 2
...
>>> obj = A(100)
>>> obj.var
100
>>> obj.duplicate_var()
200
>>> A.var
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    A.var
AttributeError: type object 'A' has no attribute 'var'

Here, you modify A to add a new method called duplicate_var(). Then, you create an instance of A by passing in 100 to the class initializer. After that, you can now call duplicate_var() on obj to duplicate the value stored in self.var. Finally, if you try to access var using the class object instead of an instance, then you’ll get an AttributeError because instance attributes can’t be accessed using class objects.

In general, when you’re writing object-oriented code in Python and you try to access an attribute, your program takes the following steps:

Check the instance local scope or namespace first.
If the attribute is not found there, then check the class local scope or namespace.
If the name doesn’t exist in the class namespace either, then you’ll get an AttributeError.

This is the underlying mechanism by which Python resolves names in classes and instances.

Although classes define a class local scope or namespace, they don’t create an enclosing scope for methods. Therefore, when you’re implementing a class, references to attributes and methods must be done using the dot notation:

>>>

>>> class A:
...     var = 100
...     def print_var(self):
...         print(var)  # Try to access a class attribute directly
...
>>> A().print_var()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    A().print_var()
  File "<stdin>", line 4, in print_var
    print(var)
NameError: name 'var' is not defined

Since classes don’t create an enclosing scope for methods, you can’t access var directly from within print_var() like you try to do here. To get access to class attributes from inside any method, you need to use the dot notation. To fix the problem in this example, change the statement print(var) inside print_var() to print(A.var) and see what happens.

You can override a class attribute with an instance attribute, which will modify the general behavior of your class. However, you can access both attributes unambiguously using the dot notation like in the following example:

>>>

>>> class A:
...     var = 100
...     def __init__(self):
...         self.var = 200
...
...     def access_attr(self):
...         # Use dot notation to access class and instance attributes
...         print(f'The instance attribute is: {self.var}')
...         print(f'The class attribute is: {A.var}')
...
>>> obj = A()
>>> obj.access_attr()
The instance attribute is: 200
The class attribute is: 100
>>> A.var  # Access class attributes
100
>>> A().var # Access instance attributes
200
>>> A.__dict__.keys()
dict_keys(['__module__', 'var', '__init__',..., '__getattribute__'])
>>> A().__dict__.keys()
dict_keys(['var'])

The above class has an instance attribute and a class attribute with the same name var. You can use the following code to access each of them:

Instance: Use self.var to access this attribute.
Class: Use A.var to access this attribute.

Since both cases use the dot notation, there are no name collision problems.

Note: In general, good OOP practices recommend not to shadow class attributes with instance attributes that have different responsibilities or perform different actions. Doing so can lead to subtle and hard-to-find bugs.

Finally, notice that the class .__dict__ and the instance .__dict__ are totally different and independent dictionaries. That’s why class attributes are available immediately after you run or import the module in which the class was defined. In contrast, instance attributes come to life only after an object or instance is created.

There are many built-in functions that are closely related to the concept of Python scope and namespaces. In previous sections, you’ve used dir() to get information on the names that exist in a given scope. Besides dir(), there are some other built-in functions that can help you out when you’re trying to get information about a Python scope or namespace. In this section, you’ll cover how to work with:

Since all these are built-in functions, they’re available for free in the built-in scope. This means that you can use them at any time without importing anything. Most of these functions are intended to be used in an interactive session to get information on different Python objects. However, you can find some interesting use cases for them in your code as well.

`globals()`

In Python, globals() is a built-in function that returns a reference to the current global scope or namespace dictionary. This dictionary always stores the names of the current module. This means that if you call globals() in a given module, then you’ll get a dictionary containing all the names that you’ve defined in that module, right before the call to globals(). Here’s an example:

>>>

>>> globals()
{'__name__': '__main__',..., '__builtins__': <module 'builtins' (built-in)>}
>>> my_var = 100
>>> globals()
{'__name__': '__main__',..., 'my_var': 100}

The first call to globals() returns a dictionary containing the names in your __main__ module or program. Note that when you assign a new name at the top level of the module, like in my_var = 100, the name is added to the dictionary returned by globals().

An interesting example of how you can use globals() in your code would be to dynamically dispatch functions that live in the global scope. Suppose you want to dynamically dispatch platform-dependent functions. To do this, you can use globals() as follows:

 1 # Filename: dispatch.py
 2 
 3 from sys import platform
 4 
 5 def linux_print():
 6     print('Printing from Linux...')
 7 
 8 def win32_print():
 9     print('Printing from Windows...')
10 
11 def darwin_print():
12     print('Printing from macOS...')
13 
14 printer = globals()[platform + '_print']
15 
16 printer()

If you run this script in your command line, then you’ll get an output that will depend on your current platform.

Another example of how to use globals() would be to inspect the list of special names in the global scope. Take a look at the following list comprehension:

>>>

>>> [name for name in globals() if name.startswith('__')]
['__name__', '__doc__', '__package__',..., '__annotations__', '__builtins__']

This list comprehension will return a list with all the special names that are defined in your current global Python scope. Note that you can use the globals() dictionary just like you would use any regular dictionary. For example, you can iterate through it through it using these traditional methods:

.keys()
.values()
.items()

You can also perform regular subscription operations over globals() by using square brackets like in globals()['name']. For example, you can modify the content of globals() even though this isn’t recommended. Take a look at this example:

>>>

>>> globals()['__doc__'] = """Docstring for __main__."""
>>> __doc__
'Docstring for __main__.'

Here, you change the key __doc__ to include a docstring for __main__ so that from now on, the main module’s docstring will have the value 'Docstring for __main__.'.

`locals()`

Another function related to Python scope and namespaces is locals(). This function updates and returns a dictionary that holds a copy of the current state of the local Python scope or namespace. When you call locals() in a function block, you get all the names assigned in the local or function scope up to the point where you call locals(). Here’s an example:

>>>

>>> def func(arg):
...     var = 100
...     print(locals())
...     another = 200
...
>>> func(300)
{'var': 100, 'arg': 300}

Whenever you call locals() inside func(), the resulting dictionary contains the name var mapped to the value 100 and arg mapped to 300. Since locals() only grabs the names assigned before you call it, another is not in the dictionary.

If you call locals() in the global Python scope, then you’ll get the same dictionary that you would get if you were to call globals():

>>>

>>> locals()
{'__name__': '__main__',..., '__builtins__': <module 'builtins' (built-in)>}
>>> locals() is globals()
True

When you call locals() in the global Python scope, you get a dictionary that’s identical to the dictionary returned by the call to globals().

Note that you shouldn’t modify the content of locals() because changes may have no effect on the values of local and free names. Check out the following example:

>>>

>>> def func():
...     var = 100
...     locals()['var'] = 200
...     print(var)
...
>>> func()
100

When you try to modify the content of var using locals(), the change doesn’t reflect in the value of var. So, you can say that locals() is only useful for read operations since updates to the locals dictionary are ignored by Python.

`vars()`

vars() is a Python built-in function that returns the .__dict__ attribute of a module, class, instance, or any other object which has a dictionary attribute. Remember that .__dict__ is a special dictionary that Python uses to implement namespaces. Take a look at the following examples:

>>>

>>> import sys
>>> vars(sys) # With a module object
{'__name__': 'sys',..., 'ps1': '>>> ', 'ps2': '... '}
>>> vars(sys) is sys.__dict__
True
>>> class MyClass:
...     def __init__(self, var):
...         self.var = var
...
>>> obj = MyClass(100)
>>> vars(obj)  # With a user-defined object
{'var': 100}
>>> vars(MyClass)  # With a class
mappingproxy({'__module__': '__main__',..., '__doc__': None})

When you call vars() using sys as an argument, you get the .__dict__ of sys. You can also call vars() using different types of Python objects, as long as they have this dictionary attribute.

Without any argument, vars() acts like locals() and returns a dictionary with all the names in the local Python scope:

>>>

>>> vars()
{'__name__': '__main__',..., '__builtins__': <module 'builtins' (built-in)>}
>>> vars() is locals()
True

Here, you call vars() at the top level of an interactive session. With no argument, this call returns a dictionary containing all the names in the global Python scope. Note that, at this level, vars() and locals() return the same dictionary.

If you call vars() with an object that doesn’t have a .__dict__, then you’ll get a TypeError, like in the following example:

>>>

>>> vars(10)  # Call vars() with objects that don't have a .__dict__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: vars() argument must have __dict__ attribute

If you call vars() with an integer object, then you’ll get a TypeError because this type of Python object doesn’t have a .__dict__.

`dir()`

You can use dir() without arguments to get the list of names in the current Python scope. If you call dir() with an argument, then the function attempts to return a list of valid attributes for that object:

>>>

>>> dir()  # With no arguments
['__annotations__', '__builtins__',..., '__package__', '__spec__']
>>> dir(zip)  # With a function object
['__class__', '__delattr__',..., '__str__', '__subclasshook__']
>>> import sys
>>> dir(sys)  # With a module object
['__displayhook__', '__doc__',..., 'version_info', 'warnoptions']
>>> var = 100
>>> dir(var)  # With an integer variable
['__abs__', '__add__',..., 'imag', 'numerator', 'real', 'to_bytes']

If you call dir() with no arguments, then you get a list containing the names that live in the global scope. You can also use dir() to inspect the list of names or attributes of different objects. This includes functions, modules, variables, and so on.

Even though the official documentation says that dir() is intended for interactive use, you can use the function to provide a comprehensive list of attributes of a given object. Note that you can also call dir() from inside a function. In this case, you’ll get the list of names defined in the function scope:

>>>

>>> def func():
...     var = 100
...     print(dir())
...     another = 200  # Is defined after calling dir()
...
>>> func()
['var']

In this example, you use dir() inside func(). When you call the function, you get a list containing the names that you define in the local scope. It’s worth noting that in this case, dir() only shows the names you declared before the function call.

Conclusion

The scope of a variable or name defines its visibility throughout your code. In Python, scope is implemented as either a Local, Enclosing, Global, or Built-in scope. When you use a variable or name, Python searches these scopes sequentially to resolve it. If the name isn’t found, then you’ll get an error. This is the general mechanism that Python uses for name resolution and is known as the LEGB rule.

You’re now able to:

Take advantage of Python scope to avoid or minimize bugs related to name collision
Make good use of global and local names across your programs to improve code maintainability
Use a coherent strategy to access, modify, or update names across all your Python code

Additionally, you’ve covered some scope-related tools and techniques that Python offers and how you can use them to gather information about the names that live in a given scope or to modify the standard behavior of Python scope. Of course, there’s more to this topic that’s outside the scope of this tutorial, so get out there and continue to tackle name resolution in Python!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Make a 2D Side-Scroller Game With PyGame

2020-03-17T14:00:00+00:00

In this course, you’ll learn about creating games using Python and the library PyGame.

By the end of this course, you’ll be able to:

Draw items on your screen
Play sound effects and music
Handle user input
Implement event loops
Describe how game programming differs from standard procedural Python programming

This primer assumes you have a basic understanding of writing Python programs, including user-defined functions, imports, loops, and conditionals. You should also be familiar with how to open files on your platform. A basic understanding of object-oriented Python is helpful as well. pygame works with most versions of Python, but Python 3.6 is recommended and used throughout this article.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Do a Binary Search in Python

2020-03-16T14:00:00+00:00

Binary search is a classic algorithm in computer science. It often comes up in programming contests and technical interviews. Implementing binary search turns out to be a challenging task, even when you understand the concept. Unless you’re curious or have a specific assignment, you should always leverage existing libraries to do a binary search in Python or any other language.

In this tutorial, you’ll learn how to:

Use the bisect module to do a binary search in Python
Implement a binary search in Python both recursively and iteratively
Recognize and fix defects in a binary search Python implementation
Analyze the time-space complexity of the binary search algorithm
Search even faster than binary search

This tutorial assumes you’re a student or an intermediate programmer with an interest in algorithms and data structures. At the very least, you should be familiar with Python’s built-in data types, such as lists and tuples. In addition, some familiarity with recursion, classes, data classes, and lambdas will help you better understand the concepts you’ll see in this tutorial.

Below you’ll find a link to the sample code you’ll see throughout this tutorial, which requires Python 3.7 or later to run:

Get Sample Code: Click here to get the sample code you'll use to learn about binary search in Python in this tutorial.

Benchmarking

In the next section of this tutorial, you’ll be using a subset of the Internet Movie Database (IMDb) to benchmark the performance of a few search algorithms. This dataset is free of charge for personal and non-commercial use. It’s distributed as a bunch of compressed tab-separated values (TSV) files, which get daily updates.

To make your life easier, you can use a Python script included in the sample code. It’ll automatically fetch the relevant file from IMDb, decompress it, and extract the interesting pieces:

$ python download_imdb.py
Fetching data from IMDb...
Created "names.txt" and "sorted_names.txt"

Be warned that this will download and extract approximately 600 MB of data, as well as produce two additional files, which are about half of that in size. The download, as well as the processing of this data, might take a minute or two to complete.

Download IMDb

To manually obtain the data, navigate your web browser to https://datasets.imdbws.com/ and grab the file called name.basics.tsv.gz, which contains the records of actors, directors, writers, and so on. When you decompress the file, you’ll see the following content:

nconst     primaryName      birthYear  deathYear  (...)
nm0000001  Fred Astaire     1899       1987       (...)
nm0000002  Lauren Bacall    1924       2014       (...)
nm0000003  Brigitte Bardot  1934       \N         (...)
nm0000004  John Belushi     1949       1982       (...)

It has a header with the column names in the first line, followed by data records in each of the subsequent lines. Each record contains a unique identifier, a full name, birth year, and a few other attributes. These are all delimited with a tab character.

There are millions of records, so don’t try to open the file with a regular text editor to avoid crashing your computer. Even specialized software such as spreadsheets can have problems opening it. Instead, you might take advantage of the high-performance data grid viewer included in JupyterLab, for example.

Read Tab-Separated Values

There are a few ways to parse a TSV file. For example, you can read it with Pandas, use a dedicated application, or leverage a few command-line tools. However, it’s recommended that you use the hassle-free Python script included in the sample code.

Note: As a rule of thumb, you should avoid parsing files manually because you might overlook edge cases. For example, in one of the fields, the delimiting tab character could be used literally inside quotation marks, which would break the number of columns. Whenever possible, try to find a relevant module in the standard library or a trustworthy third-party one.

Ultimately, you want to end up with two text files at your disposal:

names.txt
sorted_names.txt

One will contain a list of names obtained by cutting out the second column from the original TSV file:

Fred Astaire
Lauren Bacall
Brigitte Bardot
John Belushi
Ingmar Bergman
...

The second one will be the sorted version of this.

Once both files are ready, you can load them into Python using this function:

def load_names(path):
    with open(path) as text_file:
        return text_file.read().splitlines()

names = load_names('names.txt')
sorted_names = load_names('sorted_names.txt')

This code returns a list of names pulled from the given file. Note that calling .splitlines() on the resulting string removes the trailing newline character from each line. As an alternative, you could call text_file.readlines(), but that would keep the unwanted newlines.

Measure the Execution Time

To evaluate the performance of a particular algorithm, you can measure its execution time against the IMDb dataset. This is usually done with the help of the built-in time or timeit modules, which are useful for timing a block of code.

You could also define a custom decorator to time a function if you wanted to. The sample code provided uses time.perf_counter_ns(), introduced in Python 3.7, because it offers high precision in nanoseconds.

Understanding Search Algorithms

Searching is ubiquitous and lies at the heart of computer science. You probably did several web searches today alone, but have you ever wondered what searching really means?

Search algorithms take many different forms. For example, you can:

Do a full-text search
Match strings with fuzzy searching
Find the shortest path in a graph
Query a database
Look for a minimum or maximum value

In this tutorial, you’ll learn about searching for an element in a sorted list of items, like a phone book. When you search for such an element, you might be asking one of the following questions:

Question	Answer
Is it there?	Yes
Where is it?	On the 42nd page
Which one is it?	A person named John Doe

The answer to the first question tells you whether an element is present in the collection. It always holds either true or false. The second answer is the location of an element within the collection, which may be unavailable if that element was missing. Finally, the third answer is the element itself, or a lack of it.

Note: Sometimes there might be more than one correct answer due to duplicate or similar items. For example, if you have a few contacts with the same name, then they will all fit your search criteria. At other times, there might only be an approximate answer or no answer at all.

In the most common case, you’ll be searching by value, which compares elements in the collection against the exact one you provide as a reference. In other words, your search criteria are the entire element, such as a number, a string, or an object like a person. Even the tiniest difference between the two compared elements won’t result in a match.

On the other hand, you can be more granular with your search criteria by choosing some property of an element, such as a person’s last name. This is called searching by key because you pick one or more attributes to compare. Before you dive into binary search in Python, let’s take a quick look at other search algorithms to get a bigger picture and understand how they work.

Random Search

How might you look for something in your backpack? You might just dig your hand into it, pick an item at random, and see if it’s the one you wanted. If you’re out of luck, then you put the item back, rinse, and repeat. This example is a good way to understand random search, which is one of the least efficient search algorithms. The inefficiency of this approach stems from the fact that you’re running the risk of picking the same wrong thing multiple times.

Note: Funnily enough, this strategy could be the most efficient one, in theory, if you were very lucky or had a small number of elements in the collection.

The fundamental principle of this algorithm can be expressed with the following snippet of Python code:

import random

def find(elements, value):
    while True:
        random_element = random.choice(elements)
        if random_element == value:
            return random_element

The function loops until some element chosen at random matches the value given as input. However, this isn’t very useful because the function returns either None implicitly or the same value it already received in a parameter. You can find the full implementation in the sample code available for download at the link below:

Get Sample Code: Click here to get the sample code you'll use to learn about binary search in Python in this tutorial.

For microscopic datasets, the random search algorithm appears to be doing its job reasonably fast:

>>>

>>> from search.random import *  # Sample code to download
>>> fruits = ['orange', 'plum', 'banana', 'apple']
>>> contains(fruits, 'banana')
True
>>> find_index(fruits, 'banana')
2
>>> find(fruits, key=len, value=4)
'plum'

However, imagine having to search like that through millions of elements! Here’s a quick rundown of a performance test that was done against the IMDb dataset:

Search Term	Element Index	Best Time	Average Time	Worst Time
Fred Astaire	`0`	`0.74s`	`21.69s`	`43.16s`
Alicia Monica	`4,500,000`	`1.02s`	`26.17s`	`66.34s`
Baoyin Liu	`9,500,000`	`0.11s`	`17.41s`	`51.03s`
missing	`N/A`	`5m 16s`	`5m 40s`	`5m 54s`

Unique elements at different memory locations were specifically chosen to avoid bias. Each term was searched for ten times to account for the randomness of the algorithm and other factors such as garbage collection or system processes running in the background.

Note: If you’d like to conduct this experiment yourself, then refer back to the instructions in the introduction to this tutorial. To measure the performance of your code, you can use the built-in time and timeit modules, or you can time functions with a custom decorator.

The algorithm has a non-deterministic performance. While the average time to find an element doesn’t depend on its whereabouts, the best and worst times are two to three orders of magnitude apart. It also suffers from inconsistent behavior. Consider having a collection of elements containing some duplicates. Because the algorithm picks elements at random, it’ll inevitably return different copies upon subsequent runs.

How can you improve on this? One way to address both issues at once is by using a linear search.

Linear Search

When you’re deciding what to have for lunch, you may be looking around the menu chaotically until something catches your eye. Alternatively, you can take a more systematic approach by scanning the menu from top to bottom and scrutinizing every item in a sequence. That’s linear search in a nutshell. To implement it in Python, you could enumerate() elements to keep track of the current element’s index:

def find_index(elements, value):
    for index, element in enumerate(elements):
        if element == value:
            return index

The function loops over a collection of elements in a predefined and consistent order. It stops when the element is found, or when there are no more elements to check. This strategy guarantees that no element is visited more than once because you’re traversing them in order by index.

Let’s see how well linear search copes with the IMDb dataset you used before:

Search Term	Element Index	Best Time	Average Time	Worst Time
Fred Astaire	`0`	`491ns`	`1.17µs`	`6.1µs`
Alicia Monica	`4,500,000`	`0.37s`	`0.38s`	`0.39s`
Baoyin Liu	`9,500,000`	`0.77s`	`0.79s`	`0.82s`
missing	`N/A`	`0.79s`	`0.81s`	`0.83s`

There’s hardly any variance in the lookup time of an individual element. The average time is virtually the same as the best and the worst one. Since the elements are always browsed in the same order, the number of comparisons required to find the same element doesn’t change.

However, the lookup time grows with the increasing index of an element in the collection. The further the element is from the beginning of the list, the more comparisons have to run. In the worst case, when an element is missing, the whole collection has to be checked to give a definite answer.

When you project experimental data onto a plot and connect the dots, then you’ll immediately see the relationship between element location and the time it takes to find it:

All samples lie on a straight line and can be described by a linear function, which is where the name of the algorithm comes from. You can assume that, on average, the time required to find any element using a linear search will be proportional to the number of all elements in the collection. They don’t scale well as the amount of data to search increases.

For example, biometric scanners available at some airports wouldn’t recognize passengers in a matter of seconds, had they been implemented using linear search. On the other hand, the linear search algorithm may be a good choice for smaller datasets, because it doesn’t require preprocessing the data. In such a case, the benefits of preprocessing wouldn’t pay back its cost.

Python already ships with linear search, so there’s no point in writing it yourself. The list data structure, for example, exposes a method that will return the index of an element or raise an exception otherwise:

>>>

>>> fruits = ['orange', 'plum', 'banana', 'apple']
>>> fruits.index('banana')
2
>>> fruits.index('blueberry')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: 'blueberry' is not in list

This can also tell you if the element is present in the collection, but a more Pythonic way would involve using the versatile in operator:

>>>

>>> 'banana' in fruits
True
>>> 'blueberry' in fruits
False

It’s worth noting that despite using linear search under the hood, these built-in functions and operators will blow your implementation out of the water. That’s because they were written in pure C, which compiles to native machine code. The standard Python interpreter is no match for it, no matter how hard you try.

A quick test with the timeit module reveals that the Python implementation might run almost ten times slower than the equivalent native one:

>>>

>>> import timeit
>>> from search.linear import contains
>>> fruits = ['orange', 'plum', 'banana', 'apple']
>>> timeit.timeit(lambda: contains(fruits, 'blueberry'))
1.8904765040024358
>>> timeit.timeit(lambda: 'blueberry' in fruits)
0.22473459799948614

However, for sufficiently large datasets, even the native code will hit its limits, and the only solution will be to rethink the algorithm.

Note: The in operator doesn’t always do a linear search. When you use it on a set, for example, it does a hash-based search instead. The operator can work with any iterable, including tuple, list, set, dict, and str. You can even support your custom classes with it by implementing the magic method .__contains__() to define the underlying logic.

In real-life scenarios, the linear search algorithm should usually be avoided. For example, there was a time I wasn’t able to register my cat at the vet clinic because their system kept crashing. The doctor told me he must eventually upgrade his computer because adding more records into the database makes it run slower and slower.

I remember thinking to myself at that point that the person who wrote that software clearly didn’t know about the binary search algorithm!

Binary Search

The word binary is generally associated with the number 2. In this context, it refers to dividing a collection of elements into two halves and throwing away one of them at each step of the algorithm. This can dramatically reduce the number of comparisons required to find an element. But there’s a catch—elements in the collection must be sorted first.

The idea behind it resembles the steps for finding a page in a book. At first, you typically open the book to a completely random page or at least one that’s close to where you think your desired page might be.

Occasionally, you’ll be fortunate enough to find that page on the first try. However, if the page number is too low, then you know the page must be to the right. If you overshoot on the next try, and the current page number is higher than the page you’re looking for, then you know for sure that it must be somewhere in between.

You repeat the process, but rather than choosing a page at random, you check the page located right in the middle of that new range. This minimizes the number of tries. A similar approach can be used in the number guessing game. If you haven’t heard of that game, then you can look it up on the Internet to get a plethora of examples implemented in Python.

Note: Sometimes, if the values are uniformly distributed, you can calculate the middle index with linear interpolation rather than taking the average. This variation of the algorithm will require even fewer steps.

The page numbers that restrict the range of pages to search through are known as the lower bound and the upper bound. In binary search, you commonly start with the first page as the lower bound and the last page as the upper bound. You must update both bounds as you go. For example, if the page you turn to is lower than the one you’re looking for, then that’s your new lower bound.

Let’s say you were looking for a strawberry in a collection of fruits sorted in ascending order by size:

On the first attempt, the element in the middle happens to be a lemon. Since it’s bigger than a strawberry, you can discard all elements to the right, including the lemon. You’ll move the upper bound to a new position and update the middle index:

Now, you’re left with only half of the fruits you began with. The current middle element is indeed the strawberry you were looking for, which concludes the search. If it wasn’t, then you’d just update the bounds accordingly and continue until they pass each other. For example, looking for a missing plum, which would go between the strawberry and a kiwi, will end with the following result:

Notice there weren’t that many comparisons that had to be made in order to find the desired element. That’s the magic of binary search. Even if you’re dealing with a million elements, you’d only require at most a handful of checks. This number won’t exceed the logarithm base two of the total number of elements due to halving. In other words, the number of remaining elements is reduced by half at each step.

This is possible because the elements are already sorted by size. However, if you wanted to find fruits by another key, such as a color, then you’d have to sort the entire collection once again. To avoid the costly overhead of sorting, you might try to compute different views of the same collection in advance. This is somewhat similar to creating a database index.

Consider what happens if you add, delete or update an element in a collection. For a binary search to continue working, you’d need to maintain the proper sort order. This can be done with the bisect module, which you’ll read about in the upcoming section.

You’ll see how to implement the binary search algorithm in Python later on in this tutorial. For now, let’s confront it with the IMDb dataset. Notice that there are different people to search for than before. That’s because the dataset must be sorted for binary search, which reorders the elements. The new elements are located roughly at the same indices as before, to keep the measurements comparable:

Search Term	Element Index	Average Time	Comparisons
(…) Berendse	`0`	`6.52µs`	23
Jonathan Samuangte	`4,499,997`	`6.99µs`	24
Yorgos Rahmatoulin	`9,500,001`	`6.5µs`	23
missing	`N/A`	`7.2µs`	23

The answers are nearly instantaneous. In the average case, it takes only a few microseconds for the binary search to find one element among all nine million! Other than that, the number of comparisons for the chosen elements remains almost constant, which coincides with the following formula:

Finding most elements will require the highest number of comparisons, which can be derived from a logarithm of the size of the collection. Conversely, there’s just one element in the middle that can be found on the first try with one comparison.

Binary search is a great example of a divide-and-conquer technique, which partitions one problem into a bunch of smaller problems of the same kind. The individual solutions are then combined to form the final answer. Another well-known example of this technique is the quicksort algorithm.

Note: Don’t confuse divide-and-conquer with dynamic programming, which is a somewhat similar technique.

Unlike other search algorithms, binary search can be used beyond just searching. For example, it allows for set membership testing, finding the largest or smallest value, finding the nearest neighbor of the target value, performing range queries, and more.

If speed is a top priority, then binary search is not always the best choice. There are even faster algorithms that can take advantage of hash-based data structures. However, those algorithms require a lot of additional memory, whereas binary search offers a good space-time tradeoff.

Hash-Based Search

To search faster, you need to narrow down the problem space. Binary search achieves that goal by halving the number of candidates at each step. That means that even if you have one million elements, it takes at most twenty comparisons to determine if the element is present, provided that all elements are sorted.

The fastest way to search is to know where to find what you’re looking for. If you knew the exact memory location of an element, then you’d access it directly without the need for searching in the first place. Mapping an element or (more commonly) one of its keys to the element location in memory is referred to as hashing.

You can think of hashing not as searching for the specific element, but instead computing the index based on the element itself. That’s the job of a hash function, which needs to hold certain mathematical properties. A good hash function should:

Take arbitrary input and turn it into a fixed-size output.
Have a uniform value distribution to mitigate hash collisions.
Produce deterministic results.
Be a one-way function.
Amplify input changes to achieve the avalanche effect.

At the same time, it shouldn’t be too computationally expensive, or else its cost would outweigh the gains. A hash function is also used for data integrity verification as well as in cryptography.

A data structure that uses this concept to map keys into values is called a map, a hash table, a dictionary, or an associative array.

Note: Python has two built-in data structures, namely set and dict, which rely on the hash function to find elements. While a set hashes its elements, a dict uses the hash function against element keys. To find out exactly how a dict is implemented in Python, check out Raymond Hettinger’s conference talk on Modern Python Dictionaries.

Another way to visualize hashing is to imagine so-called buckets of similar elements grouped under their respective keys. For example, you may be harvesting fruits into different buckets based on color:

The coconut and a kiwi fruit go to the bucket labeled brown, while an apple ends up in a bucket with the red label, and so on. This allows you to glance through a fraction of the elements quickly. Ideally, you want to have only one fruit in each bucket. Otherwise, you get what’s known as a collision, which leads to extra work.

Note: The buckets, as well as their contents, are typically in no particular order.

Let’s put the names from the IMDb dataset into a dictionary, so that each name becomes a key, and the corresponding value becomes its index:

>>>

>>> from benchmark import load_names  # Sample code to download
>>> names = load_names('names.txt')
>>> index_by_name = {
...     name: index for index, name in enumerate(names)
... }

After loading textual names into a flat list, you can enumerate() it inside a dictionary comprehension to create the mapping. Now, checking the element’s presence as well as getting its index is straightforward:

>>>

>>> 'Guido van Rossum' in index_by_name
False
>>> 'Arnold Schwarzenegger' in index_by_name
True
>>> index_by_name['Arnold Schwarzenegger']
215

Thanks to the hash function used behind the scenes, you don’t have to implement any search at all!

Here’s how the hash-based search algorithm performs against the IMDb dataset:

Search Term	Element Index	Best Time	Average Time	Worst Time
Fred Astaire	`0`	`0.18µs`	`0.4µs`	`1.9µs`
Alicia Monica	`4,500,000`	`0.17µs`	`0.4µs`	`2.4µs`
Baoyin Liu	`9,500,000`	`0.17µs`	`0.4µs`	`2.6µs`
missing	`N/A`	`0.19µs`	`0.4µs`	`1.7µs`

Not only is the average time an order of magnitude faster than the already fast binary search Python implementation, but the speed is also sustained across all elements regardless of where they are.

The price for that gain is approximately 0.5 GB of more memory consumed by the Python process, slower load time, and the need to keep that additional data consistent with dictionary contents. In turn, the lookup is very quick, while updates and insertions are slightly slower when compared to a list.

Another constraint that dictionaries impose on their keys is that they must be hashable, and their hash values can’t change over time. You can check if a particular data type is hashable in Python by calling hash() on it:

>>>

>>> key = ['round', 'juicy']
>>> hash(key)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'

Mutable collections—such as a list, set, and dict—aren’t hashable. In practice, dictionary keys should be immutable because their hash value often depends on some attributes of the key. If a mutable collection was hashable and could be used as a key, then its hash value would be different every time the contents changed. Consider what would happen if a particular fruit changed color due to ripening. You’d be looking for it in the wrong bucket!

The hash function has many other uses. For example, it’s used in cryptography to avoid storing passwords in plain text form, as well as for data integrity verification.

Using the `bisect` Module

Binary search in Python can be performed using the built-in bisect module, which also helps with preserving a list in sorted order. It’s based on the bisection method for finding roots of functions. This module comes with six functions divided into two categories:

Find Index	Insert Element
`bisect()`	`insort()`
`bisect_left()`	`insort_left()`
`bisect_right()`	`insort_right()`

These functions allow you to either find an index of an element or add a new element in the right position. Those in the first row are just aliases for bisect_right() and insort_right(), respectively. In reality, you’re dealing with only four functions.

Note: It’s your responsibility to sort the list before passing it to one of the functions. If the elements aren’t sorted, then you’ll most likely get incorrect results.

Without further ado, let’s see the bisect module in action.

Finding an Element

To find the index of an existing element in a sorted list, you want to bisect_left():

>>>

>>> import bisect
>>> sorted_fruits = ['apple', 'banana', 'orange', 'plum']
>>> bisect.bisect_left(sorted_fruits, 'banana')
1

The output tells you that a banana is the second fruit on the list because it was found at index 1. However, if an element was missing, then you’d still get its expected position:

>>>

>>> bisect.bisect_left(sorted_fruits, 'apricot')
1
>>> bisect.bisect_left(sorted_fruits, 'watermelon')
4

Even though these fruits aren’t on the list yet, you can get an idea of where to put them. For example, an apricot should come between the apple and the banana, whereas a watermelon should become the last element. You’ll know if an element was found by evaluating two conditions:

Is the index within the size of the list?
Is the value of the element the desired one?

This can be translated to a universal function for finding elements by value:

def find_index(elements, value):
    index = bisect.bisect_left(elements, value)
    if index < len(elements) and elements[index] == value:
        return index

When there’s a match, the function will return the corresponding element index. Otherwise, it’ll return None implicitly.

To search by key, you have to maintain a separate list of keys. Since this incurs an additional cost, it’s worthwhile to calculate the keys upfront and reuse them as much as possible. You can define a helper class to be able to search by different keys without introducing much code duplication:

class SearchBy:
    def __init__(self, key, elements):
        self.elements_by_key = sorted([(key(x), x) for x in elements])
        self.keys = [x[0] for x in self.elements_by_key]

The key is a function passed as the first parameter to __init__(). Once you have it, you make a sorted list of key-value pairs to be able to retrieve an element from its key at a later time. Representing pairs with tuples guarantees that the first element of each pair will be sorted. In the next step, you extract the keys to make a flat list that’s suitable for your binary search Python implementation.

Then there’s the actual method for finding elements by key:

class SearchBy:
    def __init__(self, key, elements):
        ...

    def find(self, value):
        index = bisect.bisect_left(self.keys, value)
        if index < len(self.keys) and self.keys[index] == value:
            return self.elements_by_key[index][1]

This code bisects the list of sorted keys to get the index of an element by key. If such a key exists, then its index can be used to get the corresponding pair from the previously computed list of key-value pairs. The second element of that pair is the desired value.

Note: This is just an illustrative example. You’ll be better off using the recommended recipe, which is mentioned in the official documentation.

If you had multiple bananas, then bisect_left() would return the leftmost instance:

>>>

>>> sorted_fruits = [
...     'apple',
...     'banana', 'banana', 'banana',
...     'orange',
...     'plum'
... ]
>>> bisect.bisect_left(sorted_fruits, 'banana')
1

Predictably, to get the rightmost banana, you’d need to call bisect_right() or its bisect() alias. However, those two functions return one index further from the actual rightmost banana, which is useful for finding the insertion point of a new element:

>>>

>>> bisect.bisect_right(sorted_fruits, 'banana')
4
>>> bisect.bisect(sorted_fruits, 'banana')
4
>>> sorted_fruits[4]
'orange'

When you combine the code, you can see how many bananas you have:

>>>

>>> l = bisect.bisect_left(sorted_fruits, 'banana')
>>> r = bisect.bisect_right(sorted_fruits, 'banana')
>>> r - l
3

If an element were missing, then both bisect_left() and bisect_right() would return the same index yielding zero bananas.

Inserting a New Element

Another practical application of the bisect module is maintaining the order of elements in an already sorted list. After all, you wouldn’t want to sort the whole list every time you had to insert something into it. In most cases, all three functions can be used interchangeably:

>>>

>>> import bisect
>>> sorted_fruits = ['apple', 'banana', 'orange']
>>> bisect.insort(sorted_fruits, 'apricot')
>>> bisect.insort_left(sorted_fruits, 'watermelon')
>>> bisect.insort_right(sorted_fruits, 'plum')
>>> sorted_fruits
['apple', 'apricot', 'banana', 'orange', 'plum', 'watermelon']

You won’t see any difference until there are duplicates in your list. But even then, it won’t become apparent as long as those duplicates are simple values. Adding another banana to the left will have the same effect as adding it to the right.

To notice the difference, you need a data type whose objects can have unique identities despite having equal values. Let’s define a Person type using the @dataclass decorator, which was introduced in Python 3.7:

from dataclasses import dataclass, field

@dataclass
class Person:
    name: str
    number: int = field(compare=False)

    def __repr__(self):
        return f'{self.name}({self.number})'

A person has a name and an arbitrary number assigned to it. By excluding the number field from the equality test, you make two people equal even if they have different values of that attribute:

>>>

>>> p1 = Person('John', 1)
>>> p2 = Person('John', 2)
>>> p1 == p2
True

On the other hand, those two variables refer to completely separate entities, which allows you to make a distinction between them:

>>>

>>> p1 is p2
False
>>> p1
John(1)
>>> p2
John(2)

The variables p1 and p2 are indeed different objects.

Note that instances of a data class aren’t comparable by default, which prevents you from using the bisection algorithm on them:

>>>

>>> alice, bob = Person('Alice', 1), Person('Bob', 1)
>>> alice < bob
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'Person' and 'Person'

Python doesn’t know to order alice and bob, because they’re objects of a custom class. Traditionally, you’d implement the magic method .__lt__() in your class, which stands for less than, to tell the interpreter how to compare such elements. However, the @dataclass decorator accepts a few optional Boolean flags. One of them is order, which results in an automatic generation of the magic methods for comparison when set to True:

@dataclass(order=True)
class Person:
    ...

In turn, this allows you to compare two people and decide which one comes first:

>>>

>>> alice < bob
True
>>> bob < alice
False

Finally, you can take advantage of the name and number properties to observe where various functions insert new people to the list:

>>>

>>> sorted_people = [Person('John', 1)]
>>> bisect.insort_left(sorted_people, Person('John', 2))
>>> bisect.insort_right(sorted_people, Person('John', 3))
>>> sorted_people
[John(2), John(1), John(3)]

The numbers in parentheses after the names indicate the insertion order. In the beginning, there was just one John, who got the number 1. Then, you added its duplicate to the left, and later one more to the right.

Implementing Binary Search in Python

Keep in mind that you probably shouldn’t implement the algorithm unless you have a strong reason to. You’ll save time and won’t need to reinvent the wheel. The chances are that the library code is mature, already tested by real users in a production environment, and has extensive functionality delivered by multiple contributors.

That said, there are times when it makes sense to roll up your sleeves and do it yourself. Your company might have a policy banning certain open source libraries due to licensing or security matters. Maybe you can’t afford another dependency due to memory or network bandwidth constraints. Lastly, writing code yourself might be a great learning tool!

You can implement most algorithms in two ways:

Iteratively
Recursively

However, there are exceptions to that rule. One notable example is the Ackermann function, which can only be expressed in terms of recursion.

Before you go any further, make sure that you have a good grasp of the binary search algorithm. You can refer to an earlier part of this tutorial for a quick refresher.

Iteratively

The iterative version of the algorithm involves a loop, which will repeat some steps until the stopping condition is met. Let’s begin by implementing a function that will search elements by value and return their index:

def find_index(elements, value):
    ...

You’re going to reuse this function later.

Assuming that all elements are sorted, you can set the lower and the upper boundaries at the opposite ends of the sequence:

def find_index(elements, value):
    left, right = 0, len(elements) - 1

Now, you want to identify the middle element to see if it has the desired value. Calculating the middle index can be done by taking the average of both boundaries:

def find_index(elements, value):
    left, right = 0, len(elements) - 1
    middle = (left + right) // 2

Notice how an integer division helps to handle both an odd and even number of elements in the bounded range by flooring the result. Depending on how you’re going to update the boundaries and define the stopping condition, you could also use a ceiling function.

Next, you either finish or split the sequence in two and continue searching in one of the resultant halves:

def find_index(elements, value):
    left, right = 0, len(elements) - 1
    middle = (left + right) // 2

    if elements[middle] == value:
        return middle

    if elements[middle] < value:
        left = middle + 1
    elif elements[middle] > value:
        right = middle - 1

If the element in the middle was a match, then you return its index. Otherwise, if it was too small, then you need to move the lower boundary up. If it was too big, then you need to move the upper boundary down.

To keep going, you have to enclose most of the steps in a loop, which will stop when the lower boundary overtakes the upper one:

def find_index(elements, value):
    left, right = 0, len(elements) - 1

    while left <= right:
        middle = (left + right) // 2

        if elements[middle] == value:
            return middle

        if elements[middle] < value:
            left = middle + 1
        elif elements[middle] > value:
            right = middle - 1

In other words, you want to iterate as long as the lower boundary is below or equal to the upper one. Otherwise, there was no match, and the function returns None implicitly.

Searching by key boils down to looking at an object’s attributes instead of its literal value. A key could be the number of characters in a fruit’s name, for example. You can adapt find_index() to accept and use a key parameter:

def find_index(elements, value, key):
    left, right = 0, len(elements) - 1

    while left <= right:
        middle = (left + right) // 2
        middle_element = key(elements[middle])

        if middle_element == value:
            return middle

        if middle_element < value:
            left = middle + 1
        elif middle_element > value:
            right = middle - 1

However, you must also remember to sort the list using the same key that you’re going to search with:

>>>

>>> fruits = ['orange', 'plum', 'watermelon', 'apple']
>>> fruits.sort(key=len)
>>> fruits
['plum', 'apple', 'orange', 'watermelon']
>>> fruits[find_index(fruits, key=len, value=10)]
'watermelon'
>>> print(find_index(fruits, key=len, value=3))
None

In the example above, watermelon was chosen because its name is precisely ten characters long, while no fruits on the list have names made up of three letters.

That’s great, but at the same time, you’ve just lost the ability to search by value. To remedy this, you could assign the key a default value of None and then check if it was given or not. However, in a more streamlined solution, you’d always want to call the key. By default, it would be an identity function returning the element itself:

def identity(element):
    return element

def find_index(elements, value, key=identity):
    ...

Alternatively, you might define the identity function inline with an anonymous lambda expression:

def find_index(elements, value, key=lambda x: x):
    ...

find_index() answers only one question. There are still two others, which are “Is it there?” and “What is it?” To answer these two, you can build on top of it:

def find_index(elements, value, key):
    ...

def contains(elements, value, key=identity):
    return find_index(elements, value, key) is not None

def find(elements, value, key=identity):
    index = find_index(elements, value, key)
    return None if index is None else elements[index]

With these three functions, you can tell almost everything about an element. However, you still haven’t addressed duplicates in your implementation. What if you had a collection of people, and some of them shared a common name or surname? For example, there might be a Smith family or a few guys going by the name of John among the people:

people = [
    Person('Bob', 'Williams'),
    Person('John', 'Doe'),
    Person('Paul', 'Brown'),
    Person('Alice', 'Smith'),
    Person('John', 'Smith'),
]

To model the Person type, you can modify a data class defined earlier:

from dataclasses import dataclass

@dataclass(order=True)
class Person:
    name: str
    surname: str

Notice the use of the order attribute to enable automatic generation of magic methods for comparing instances of the class by all fields. Alternatively, you might prefer to take advantage of the namedtuple, which has a shorter syntax:

from collections import namedtuple
Person = namedtuple('Person', 'name surname')

Both definitions are fine and interchangeable. Each person has a name and a surname attribute. To sort and search by one of them, you can conveniently define the key function with an attrgetter() available in the built-in operator module:

>>>

>>> from operator import attrgetter
>>> by_surname = attrgetter('surname')
>>> people.sort(key=by_surname)
>>> people
[Person(name='Paul', surname='Brown'),
 Person(name='John', surname='Doe'),
 Person(name='Alice', surname='Smith'),
 Person(name='John', surname='Smith'),
 Person(name='Bob', surname='Williams')]

Notice how people are now sorted by surname in ascending order. There’s John Smith and Alice Smith, but binary searching for the Smith surname currently gives you only one arbitrary result:

>>>

>>> find(people, key=by_surname, value='Smith')
Person(name='Alice', surname='Smith')

To mimic the features of the bisect module shown before, you can write your own version of bisect_left() and bisect_right(). Before finding the leftmost instance of a duplicate element, you want to determine if there’s such an element at all:

def find_leftmost_index(elements, value, key=identity):
    index = find_index(elements, value, key)
    if index is not None:
        ...
    return index

If some index has been found, then you can look to the left and keep moving until you come across an element with a different key or there are no more elements:

def find_leftmost_index(elements, value, key=identity):
    index = find_index(elements, value, key)
    if index is not None:
        while index >= 0 and key(elements[index]) == value:
            index -= 1
        index += 1
    return index

Once you go past the leftmost element, you need to move the index back by one position to the right.

Finding the rightmost instance is quite similar, but you need to flip the conditions:

def find_rightmost_index(elements, value, key=identity):
    index = find_index(elements, value, key)
    if index is not None:
        while index < len(elements) and key(elements[index]) == value:
            index += 1
        index -= 1
    return index

Instead of going left, now you’re going to the right until the end of the list. Using both functions allows you to find all occurrences of duplicate items:

def find_all_indices(elements, value, key=identity):
    left = find_leftmost_index(elements, value, key)
    right = find_rightmost_index(elements, value, key)
    if left and right:
        return set(range(left, right + 1))
    return set()

This function always returns a set. If the element isn’t found, then the set will be empty. If the element is unique, then the set will be made up of only a single index. Otherwise, there will be multiple indices in the set.

To wrap up, you can define even more abstract functions to complete your binary search Python library:

def find_leftmost(elements, value, key=identity):
    index = find_leftmost_index(elements, value, key)
    return None if index is None else elements[index]

def find_rightmost(elements, value, key=identity):
    index = find_rightmost_index(elements, value, key)
    return None if index is None else elements[index]

def find_all(elements, value, key=identity):
    return {elements[i] for i in find_all_indices(elements, value, key)}

Not only does this allow you to pinpoint the exact location of elements on the list, but also to retrieve those elements. You’re able to ask very specific questions:

Is it there?	Where is it?	What is it?
`contains()`	`find_index()`	`find()`
	`find_leftmost_index()`	`find_leftmost()`
	`find_rightmost_index()`	`find_rightmost()`
	`find_all_indices()`	`find_all()`

The complete code of this binary search Python library can be found at the link below:

Get Sample Code: Click here to get the sample code you'll use to learn about binary search in Python in this tutorial.

Recursively

For the sake of simplicity, you’re only going to consider the recursive version of contains(), which tells you if an element was found.

Note: My favorite definition of recursion was given in an episode of the Fun Fun Function series about functional programming in JavaScript:

“Recursion is when a function calls itself until it doesn’t.”

— Mattias Petter Johansson

The most straightforward approach would be to take the iterative version of binary search and use the slicing operator to chop the list:

def contains(elements, value):
    left, right = 0, len(elements) - 1

    if left <= right:
        middle = (left + right) // 2

        if elements[middle] == value:
            return True

        if elements[middle] < value:
            return contains(elements[middle + 1:], value)
        elif elements[middle] > value:
            return contains(elements[:middle], value)

    return False

Instead of looping, you check the condition once and sometimes call the same function on a smaller list. What could go wrong with that? Well, it turns out that slicing generates copies of element references, which can have noticeable memory and computational overhead.

To avoid copying, you might reuse the same list but pass different boundaries into the function whenever necessary:

def contains(elements, value, left, right):
    if left <= right:
        middle = (left + right) // 2

        if elements[middle] == value:
            return True

        if elements[middle] < value:
            return contains(elements, value, middle + 1, right)
        elif elements[middle] > value:
            return contains(elements, value, left, middle - 1)

    return False

The downside is that every time you want to call that function, you have to pass initial boundaries, making sure they’re correct:

>>>

>>> sorted_fruits = ['apple', 'banana', 'orange', 'plum']
>>> contains(sorted_fruits, 'apple', 0, len(sorted_fruits) - 1)
True

If you were to make a mistake, then it would potentially not find that element. You can improve this by using default function arguments or by introducing a helper function that delegates to the recursive one:

def contains(elements, value):
    return recursive(elements, value, 0, len(elements) - 1)

def recursive(elements, value, left, right):
    ...

Going further, you might prefer to nest one function in another to hide the technical details and to take advantage of variable reuse from outer scope:

def contains(elements, value):
    def recursive(left, right):
        if left <= right:
            middle = (left + right) // 2
            if elements[middle] == value:
                return True
            if elements[middle] < value:
                return recursive(middle + 1, right)
            elif elements[middle] > value:
                return recursive(left, middle - 1)
        return False
    return recursive(0, len(elements) - 1)

The recursive() inner function can access both elements and value parameters even though they’re defined in the enclosing scope. The life cycle and visibility of variables in Python is dictated by the so-called LEGB rule, which tells the interpreter to look for symbols in the following order:

Local scope
Enclosing scope
Global scope
Built-in symbols

This allows variables that are defined in outer scope to be accessed from within nested blocks of code.

The choice between an iterative and a recursive implementation is often the net result of performance considerations, convenience, as well as personal taste. However, there are also certain risks involved with recursion, which is one of the subjects of the next section.

Covering Tricky Details

Here’s what the author of The Art of Computer Programming has to say about implementing the binary search algorithm:

“Although the basic idea of binary search is comparatively straightforward, the details can be surprisingly tricky, and many good programmers have done it wrong the first few times they tried.”

— Donald Knuth

If that doesn’t deter you enough from the idea of writing the algorithm yourself, then maybe this will. The standard library in Java had a subtle bug in their implementation of binary search, which remained undiscovered for a decade! But the bug itself traces its roots much earlier than that.

Note: I once fell victim to the binary search algorithm during a technical screening. There were a couple of coding puzzles to solve, including a binary search one. Guess which one I failed to complete? Yeah.

The following list isn’t exhaustive, but at the same time, it doesn’t talk about common mistakes like forgetting to sort the list.

Integer Overflow

This is the Java bug that was just mentioned. If you recall, the binary search Python algorithm inspects the middle element of a bounded range in a sorted collection. But how is that middle element chosen exactly? Usually, you take the average of the lower and upper boundary to find the middle index:

middle = (left + right) // 2

This method of calculating the average works just fine in the overwhelming majority of cases. However, once the collection of elements becomes sufficiently large, the sum of both boundaries won’t fit the integer data type. It’ll be larger than the maximum value allowed for integers.

Some programming languages might raise an error in such situations, which would immediately stop program execution. Unfortunately, that’s not always the case. For example, Java silently ignores this problem, letting the value flip around and become some seemingly random number. You’ll only know about the problem as long as the resulting number happens to be negative, which throws an IndexOutOfBoundsException.

Here’s an example that demonstrates this behavior in jshell, which is kind of like an interactive interpreter for Java:

jshell> var a = Integer.MAX_VALUE
a ==> 2147483647

jshell> a + 1
$2 ==> -2147483648

A safer way to find the middle index could be calculating the offset first and then adding it to the lower boundary:

middle = left + (right - left) // 2

Even if both values are maxed out, the sum in the formula above will never be. There are a few more ways, but the good news is that you don’t need to worry about any of these, because Python is free from the integer overflow error. There’s no upper limit on how big integers can be other than your memory:

>>>

>>> 2147483647**7
210624582650556372047028295576838759252690170086892944262392971263

However, there’s a catch. When you call functions from a library, that code might be subject to the C language constraints and still cause an overflow. There are plenty of libraries based on the C language in Python. You could even build your own C extension module or load a dynamically-linked library into Python using ctypes.

Stack Overflow

The stack overflow problem may, theoretically, concern the recursive implementation of binary search. Most programming languages impose a limit on the number of nested function calls. Each call is associated with a return address stored on a stack. In Python, the default limit is a few thousand levels of such calls:

>>>

>>> import sys
>>> sys.getrecursionlimit()
3000

This won’t be enough for a lot of recursive functions. However, it’s very unlikely that a binary search in Python would ever need more due to its logarithmic nature. You’d need a collection of two to the power of three thousand elements. That’s a number with over nine hundred digits!

Nevertheless, it’s still possible for the infinite recursion error to arise if the stopping condition is stated incorrectly due to a bug. In such a case, the infinite recursion will eventually cause a stack overflow.

Note: The stack overflow error is very common among languages with manual memory management. People would often google those errors to see if someone else already had similar issues, which gave the name to a popular Q&A site for programmers.

You can temporarily lift or decrease the recursion limit to simulate a stack overflow error. Note that the effective limit will be smaller because of the functions that the Python runtime environment has to call:

>>>

>>> def countup(limit, n=1):
...     print(n)
...     if n < limit:
...         countup(limit, n + 1)
...
>>> import sys
>>> sys.setrecursionlimit(7)  # Actual limit is 3
>>> countup(10)
1
2
3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in countup
  File "<stdin>", line 4, in countup
  File "<stdin>", line 2, in countup
RecursionError: maximum recursion depth exceeded while calling a Python object

The recursive function was called three times before saturating the stack. The remaining four calls must have been made by the interactive interpreter. If you run that same code in PyCharm or an alternative Python shell, then you might get a different result.

Duplicate Elements

You’re aware of the possibility of having duplicate elements in the list and you know how to deal with them. This is just to emphasize that a conventional binary search in Python might not produce deterministic results. Depending on how the list was sorted or how many elements it has, you’ll get a different answer:

>>>

>>> from search.binary import *
>>> sorted_fruits = ['apple', 'banana', 'banana', 'orange']
>>> find_index(sorted_fruits, 'banana')
1
>>> sorted_fruits.append('plum')
>>> find_index(sorted_fruits, 'banana')
2

There are two bananas on the list. At first, the call to find_index() returns the left one. However, adding a completely unrelated element at the end of the list makes the same call give you a different banana.

The same principle, known as algorithm stability, applies to sorting algorithms. Some are stable, meaning they don’t change the relative positions of equivalent elements. Others don’t make such guarantees. If you ever need to sort elements by multiple criteria, then you should always start from the least significant key to retain stability.

Floating-Point Rounding

So far you’ve only searched for fruits or people, but what about numbers? They should be no different, right? Let’s make a list of floating-point numbers at 0.1 increments using a list comprehension:

>>>

>>> sorted_numbers = [0.1*i for i in range(1, 4)]

The list should contain numbers the one-tenth, two-tenths, and three-tenths. Surprisingly, only two of those three numbers can be found:

>>>

>>> from search.binary import contains
>>> contains(sorted_numbers, 0.1)
True
>>> contains(sorted_numbers, 0.2)
True
>>> contains(sorted_numbers, 0.3)
False

This isn’t a problem strictly related to binary search in Python, as the built-in linear search is consistent with it:

>>>

>>> 0.1 in sorted_numbers
True
>>> 0.2 in sorted_numbers
True
>>> 0.3 in sorted_numbers
False

It’s not even a problem related to Python but rather to how floating-point numbers are represented in computer memory. This is defined by the IEEE 754 standard for floating-point arithmetic. Without going into much detail, some decimal numbers don’t have a finite representation in binary form. Because of limited memory, those numbers get rounded, causing a floating-point rounding error.

Note: If you require maximum precision, then steer away from floating-point numbers. They’re great for engineering purposes. However, for monetary operations, you don’t want rounding errors to accumulate. It’s recommended to scale down all prices and amounts to the smallest unit, such as cents or pennies, and treat them as integers.

Alternatively, many programming languages have support for fixed-point numbers, such as the decimal type in Python. This puts you in control of when and how rounding is taking place.

If you do need to work with floating-point numbers, then you should replace exact matching with an approximate comparison. Let’s consider two variables with slightly different values:

>>>

>>> a = 0.3
>>> b = 0.1 * 3
>>> b
0.30000000000000004
>>> a == b
False

Regular comparison gives a negative result, although both values are nearly identical. Fortunately, Python comes with a function that will test if two values are close to each other within some small neighborhood:

>>>

>>> import math
>>> math.isclose(a, b)
True

That neighborhood, which is the maximum distance between the values, can be adjusted if needed:

>>>

>>> math.isclose(a, b, rel_tol=1e-16)
False

You can use that function to do a binary search in Python in the following way:

import math

def find_index(elements, value):
    left, right = 0, len(elements) - 1

    while left <= right:
        middle = (left + right) // 2

        if math.isclose(elements[middle], value):
            return middle

        if elements[middle] < value:
            left = middle + 1
        elif elements[middle] > value:
            right = middle - 1

On the other hand, this implementation of binary search in Python is specific to floating-point numbers only. You couldn’t use it to search for anything else without getting an error.

Analyzing the Time-Space Complexity of Binary Search

The following section will contain no code and some math concepts.

In computing, you can optimize the performance of pretty much any algorithm at the expense of increased memory use. For instance, you saw that a hash-based search of the IMDb dataset required an extra 0.5 GB of memory to achieve unparalleled speed.

Conversely, to save bandwidth, you’d compress a video stream before sending it over the network, increasing the amount of work to be done. This phenomenon is known as the space-time tradeoff and is useful in evaluating an algorithm’s complexity.

Time-Space Complexity

The computational complexity is a relative measure of how many resources an algorithm needs to do its job. The resources include computation time as well as the amount of memory it uses. Comparing the complexity of various algorithms allows you to make an informed decision about which is better in a given situation.

Note: Algorithms that don’t need to allocate more memory than their input data already consumes are called in-place, or in-situ, algorithms. This results in mutating the original data, which sometimes may have unwanted side-effects.

You looked at a few search algorithms and their average performance against a large dataset. It’s clear from those measurements that a binary search is faster than a linear search. You can even tell by what factor.

However, if you took the same measurements in a different environment, you’d probably get slightly or perhaps entirely different results. There are invisible factors at play that can be influencing your test. Besides, such measurements aren’t always feasible. So, how can you compare time complexities quickly and objectively?

The first step is to break down the algorithm into smaller pieces and find the one that is doing the most work. It’s likely going to be some elementary operation that gets called a lot and consistently takes about the same time to run. For search algorithms, such an operation might be the comparison of two elements.

Having established that, you can now analyze the algorithm. To find the time complexity, you want to describe the relationship between the number of elementary operations executed versus the size of the input. Formally, such a relationship is a mathematical function. However, you’re not interested in looking for its exact algebraic formula but rather estimating its overall shape.

There are a few well-known classes of functions that most algorithms fit in. Once you classify an algorithm according to one of them, you can put it on a scale:

Common Classes of Time Complexity

These classes tell you how the number of elementary operations increases with the growing size of the input. They are, from left to right:

Constant
Logarithmic
Linear
Quasilinear
Quadratic
Exponential
Factorial

This can give you an idea about the performance of the algorithm you’re considering. A constant complexity, regardless of the input size, is the most desired one. A logarithmic complexity is still pretty good, indicating a divide-and-conquer technique at use. The further to the right on this scale, the worse the complexity of the algorithm, because it has more work to do.

When you’re talking about the time complexity, what you typically mean is the asymptotic complexity, which describes the behavior under very large data sets. This simplifies the function formula by eliminating all terms and coefficients but the one that grows at the fastest rate (for example, n squared).

However, a single function doesn’t provide enough information to compare two algorithms accurately. The time complexity may vary depending on the volume of data. For example, the binary search algorithm is like a turbocharged engine, which builds pressure before it’s ready to deliver power. On the other hand, the linear search algorithm is fast from the start but quickly reaches its peak power and ultimately loses the race:

In terms of speed, the binary search algorithm starts to overtake the linear search when there’s a certain number of elements in the collection. For smaller collections, a linear search might be a better choice.

Note: Note that the same algorithm may have different optimistic, pessimistic, and average time complexity. For example, in the best-case scenario, a linear search algorithm will find the element at the first index, after running just one comparison.

On the other end of the spectrum, it’ll have to compare a reference value to all elements in the collection. In practice, you want to know the pessimistic complexity of an algorithm.

There are a few mathematical notations of the asymptotic complexity, which are used to compare algorithms. By far the most popular one is the Big O notation.

The Big O Notation

The Big O notation represents the worst-case scenario of asymptotic complexity. Although this might sound rather intimidating, you don’t need to know the formal definition. Intuitively, it’s a very rough measure of the rate of growth at the tail of the function that describes the complexity. You pronounce it as “big oh” of something:

That “something” is usually a function of data size or just the digit “one” that stands for a constant. For example, the linear search algorithm has a time complexity of O(n), while a hash-based search has O(1) complexity.

Note: When you say that some algorithm has complexity O(f(n)), where n is the size of the input data, then it means that the function f(n) is an upper bound of the graph of that complexity. In other words, the actual complexity of that algorithm won’t grow faster than f(n) multiplied by some constant, when n approaches infinity.

In real-life, the Big O notation is used less formally as both an upper and a lower bound. This is useful for the classification and comparison of algorithms without having to worry about the exact function formulas.

The Complexity of Binary Search

You’ll estimate the asymptotic time complexity of binary search by determining the number of comparisons in the worst-case scenario—when an element is missing—as a function of input size. You can approach this problem in three different ways:

Tabular
Graphical
Analytical

The tabular method is about collecting empirical data, putting it in a table, and trying to guess the formula by eyeballing sampled values:

Number of Elements	Number of Comparisons
0	0
1	1
2	2
3	2
4	3
5	3
6	3
7	3
8	4

The number of comparisons grows as you increase the number of elements in the collection, but the rate of growth is slower than if it was a linear function. That’s an indication of a good algorithm that can scale with data.

If that doesn’t help you, you can try the graphical method, which visualizes the sampled data by drawing a graph:

The data points seem to overlay with a curve, but you don’t have enough information to provide a conclusive answer. It could be a polynomial, whose graph turns up and down for larger inputs.

Taking the analytical approach, you can choose some relationship and look for patterns. For example, you might study how the number of elements shrinks in each step of the algorithm:

Comparison	Number of Elements
-	n
1st	n/2
2nd	n/4
3rd	n/8
⋮	⋮
k-th	n/2 ^k

In the beginning, you start with the whole collection of n elements. After the first comparison, you’re left with only half of them. Next, you have a quarter, and so on. The pattern that arises from this observation is that after k-th comparison, there are n/2 ^k elements. Variable k is the expected number of elementary operations.

After all k comparisons, there will be no more elements left. However, when you take one step back, that is k - 1, there will be exactly one element left. This gives you a convenient equation:

Multiply both sides of the equation by the denominator, then take the logarithm base two of the result, and move the remaining constant to the right. You’ve just found the formula for the binary search complexity, which is on the order of O(log(n)).

Conclusion

Now you know the binary search algorithm inside and out. You can flawlessly implement it yourself, or take advantage of the standard library in Python. Having tapped into the concept of time-space complexity, you’re able to choose the best search algorithm for the given situation.

Now you can:

Use the bisect module to do a binary search in Python
Implement binary search in Python recursively and iteratively
Recognize and fix defects in a binary search Python implementation
Analyze the time-space complexity of the binary search algorithm
Search even faster than binary search

With all this knowledge, you’ll rock your programming interview! Whether the binary search algorithm is an optimal solution to a particular problem, you have the tools to figure it out on your own. You don’t need a computer science degree to do so.

You can grab all of the code you’ve seen in this tutorial at the link below:

Get Sample Code: Click here to get the sample code you'll use to learn about binary search in Python in this tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Using the Python defaultdict Type for Handling Missing Keys

2020-03-11T14:00:00+00:00

A common problem that you can face when working with Python dictionaries is to try to access or modify keys that don’t exist in the dictionary. This will raise a KeyError and break up your code execution. To handle these kinds of situations, the standard library provides the Python defaultdict type, a dictionary-like class that’s available for you in collections.

The Python defaultdict type behaves almost exactly like a regular Python dictionary, but if you try to access or modify a missing key, then defaultdict will automatically create the key and generate a default value for it. This makes defaultdict a valuable option for handling missing keys in dictionaries.

In this tutorial, you’ll learn:

How to use the Python defaultdict type for handling missing keys in a dictionary
When and why to use a Python defaultdict rather than a regular dict
How to use a defaultdict for grouping, counting, and accumulating operations

With this knowledge under your belt, you’ll be in a better condition to effectively use the Python defaultdict type in your day-to-day programming challenges.

To get the most out of this tutorial, you should have some previous understanding of what Python dictionaries are and how to work with them. If you need to freshen up, then check out the following resources:

Dictionaries in Python (Tutorial)
Dictionaries in Python (Course)
How to Iterate Through a Dictionary in Python

Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Handling Missing Keys in Dictionaries

A common issue that you can face when working with Python dictionaries is how to handle missing keys. If your code is heavily based on dictionaries, or if you’re creating dictionaries on the fly all the time, then you’ll soon notice that dealing with frequent KeyError exceptions can be quite annoying and can add extra complexity to your code. With Python dictionaries, you have at least four available ways to handle missing keys:

Use .setdefault()
Use .get()
Use the key in dict idiom
Use a try and except block

The Python docs explain .setdefault() and .get() as follows:

setdefault(key[, default])

If key is in the dictionary, return its value. If not, insert key with a value of default and return default. default defaults to None.

get(key[, default])

Return the value for key if key is in the dictionary, else default. If default is not given, it defaults to None, so that this method never raises a KeyError.

(Source)

Here’s an example of how you can use .setdefault() to handle missing keys in a dictionary:

>>>

>>> a_dict = {}
>>> a_dict['missing_key']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    a_dict['missing_key']
KeyError: 'missing_key'
>>> a_dict.setdefault('missing_key', 'default value')
'default value'
>>> a_dict['missing_key']
'default value'
>>> a_dict.setdefault('missing_key', 'another default value')
'default value'
>>> a_dict
{'missing_key': 'default value'}

In the above code, you use .setdefault() to generate a default value for missing_key. Notice that your dictionary, a_dict, now has a new key called missing_key whose value is 'default value'. This key didn’t exist before you called .setdefault(). Finally, if you call .setdefault() on an existing key, then the call won’t have any effect on the dictionary. Your key will hold the original value instead of the new default value.

Note: In the above code example, you get an exception, and Python shows you a traceback message, which tells you that you’re trying to access a missing key in a_dict. If you want to dive deeper into how to decipher and understand a Python traceback, then check out Understanding the Python Traceback.

On the other hand, if you use .get(), then you can code something like this:

>>>

>>> a_dict = {}
>>> a_dict.get('missing_key', 'default value')
'default value'
>>> a_dict
{}

Here, you use .get() to generate a default value for missing_key, but this time, your dictionary stays empty. This is because .get() returns the default value, but this value isn’t added to the underlying dictionary. For example, if you have a dictionary called D, then you can assume that .get() works something like this:

D.get(key, default) -> D[key] if key in D, else default

With this pseudo-code, you can understand how .get() works internally. If the key exists, then .get() returns the value mapped to that key. Otherwise, the default value is returned. Your code never creates or assigns a value to key. In this example, default defaults to None.

You can also use conditional statements to handle missing keys in dictionaries. Take a look at the following example, which uses the key in dict idiom:

>>>

>>> a_dict = {}
>>> if 'key' in a_dict:
...     # Do something with 'key'...
...     a_dict['key']
... else:
...     a_dict['key'] = 'default value'
...
>>> a_dict
{'key': 'default value'}

In this code, you use an if statement along with the in operator to check if key is present in a_dict. If so, then you can perform any action with key or with its value. Otherwise, you create the new key, key, and assign it a 'default value'. Note that the above code works similar to .setdefault() but takes four lines of code, while .setdefault() would only take one line (in addition to being more readable).

You can also walk around the KeyError by using a try and except block to handle the exception. Consider the following piece of code:

>>>

>>> a_dict = {}
>>> try:
...     # Do something with 'key'...
...     a_dict['key']
... except KeyError:
...     a_dict['key'] = 'default value'
...
>>> a_dict
{'key': 'default value'}

The try and except block in the above example catches the KeyError whenever you try to get access to a missing key. In the except clause, you create the key and assign it a 'default value'.

Note: If missing keys are uncommon in your code, then you might prefer to use a try and except block (EAFP coding style) to catch the KeyError exception. This is because the code doesn’t check the existence of every key and only handles a few exceptions, if any.

On the other hand, if missing keys are quite common in your code, then the conditional statement (LBYL coding style) can be a better choice because checking for keys can be less costly than handling frequent exceptions.

So far, you’ve learned how to handle missing keys using the tools that dict and Python offer you. However, the examples you saw here are quite verbose and hard to read. They might not be as straightforward as you might want. That’s why the Python standard library provides a more elegant, Pythonic, and efficient solution. That solution is collections.defaultdict, and that’s what you’ll be covering from now on.

Understanding the Python `defaultdict` Type

The Python standard library provides collections, which is a module that implements specialized container types. One of those is the Python defaultdict type, which is an alternative to dict that’s specifically designed to help you out with missing keys. defaultdict is a Python type that inherits from dict:

>>>

>>> from collections import defaultdict
>>> issubclass(defaultdict, dict)
True

The above code shows that the Python defaultdict type is a subclass of dict. This means that defaultdict inherits most of the behavior of dict. So, you can say that defaultdict is much like an ordinary dictionary.

The main difference between defaultdict and dict is that when you try to access or modify a key that’s not present in the dictionary, a default value is automatically given to that key. In order to provide this functionality, the Python defaultdict type does two things:

It overrides .__missing__().
It adds .default_factory, a writable instance variable that needs to be provided at the time of instantiation.

The instance variable .default_factory will hold the first argument passed into defaultdict.__init__(). This argument can take a valid Python callable or None. If a callable is provided, then it’ll automatically be called by defaultdict whenever you try to access or modify the value associated with a missing key.

Note: All the remaining arguments to the class initializer are treated as if they were passed to the initializer of regular dict, including the keyword arguments.

Take a look at how you can create and properly initialize a defaultdict:

>>>

>>> # Correct instantiation
>>> def_dict = defaultdict(list)  # Pass list to .default_factory
>>> def_dict['one'] = 1  # Add a key-value pair
>>> def_dict['missing']  # Access a missing key returns an empty list
[]
>>> def_dict['another_missing'].append(4)  # Modify a missing key
>>> def_dict
defaultdict(<class 'list'>, {'one': 1, 'missing': [], 'another_missing': [4]})

Here, you pass list to .default_factory when you create the dictionary. Then, you use def_dict just like a regular dictionary. Note that when you try to access or modify the value mapped to a non-existent key, the dictionary assigns it the default value that results from calling list().

Keep in mind that you must pass a valid Python callable object to .default_factory, so remember not to call it using the parentheses at initialization time. This can be a common issue when you start using the Python defaultdict type. Take a look at the following code:

>>>

>>> # Wrong instantiation
>>> def_dict = defaultdict(list())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    def_dict = defaultdict(list())
TypeError: first argument must be callable or None

Here, you try to create a defaultdict by passing list() to .default_factory. The call to list() raises a TypeError, which tells you that the first argument must be callable or None.

With this introduction to the Python defaultdict type, you can get start coding with practical examples. The next few sections will walk you through some common use cases where you can rely on a defaultdict to provide an elegant, efficient, and Pythonic solution.

Using the Python `defaultdict` Type

Sometimes, you’ll use a mutable built-in collection (a list, dict, or set) as values in your Python dictionaries. In these cases, you’ll need to initialize the keys before first use, or you’ll get a KeyError. You can either do this process manually or automate it using a Python defaultdict. In this section, you’ll learn how to use the Python defaultdict type for solving some common programming problems:

Grouping the items in a collection
Counting the items in a collection
Accumulating the values in a collection

You’ll be covering some examples that use list, set, int, and float to perform grouping, counting, and accumulating operations in a user-friendly and efficient way.

Grouping Items

A typical use of the Python defaultdict type is to set .default_factory to list and then build a dictionary that maps keys to lists of values. With this defaultdict, if you try to get access to any missing key, then the dictionary runs the following steps:

Call list() to create a new empty list
Insert the empty list into the dictionary using the missing key as key
Return a reference to that list

This allows you to write code like this:

>>>

>>> from collections import defaultdict
>>> dd = defaultdict(list)
>>> dd['key'].append(1)
>>> dd
defaultdict(<class 'list'>, {'key': [1]})
>>> dd['key'].append(2)
>>> dd
defaultdict(<class 'list'>, {'key': [1, 2]})
>>> dd['key'].append(3)
>>> dd
defaultdict(<class 'list'>, {'key': [1, 2, 3]})

Here, you create a Python defaultdict called dd and pass list to .default_factory. Notice that even when key isn’t defined, you can append values to it without getting a KeyError. That’s because dd automatically calls .default_factory to generate a default value for the missing key.

You can use defaultdict along with list to group the items in a sequence or a collection. Suppose that you’ve retrieved the following data from your company’s database:

Department	Employee Name
Sales	John Doe
Sales	Martin Smith
Accounting	Jane Doe
Marketing	Elizabeth Smith
Marketing	Adam Doe
…	…

With this data, you create an initial list of tuple objects like the following:

dep = [('Sales', 'John Doe'),
       ('Sales', 'Martin Smith'),
       ('Accounting', 'Jane Doe'),
       ('Marketing', 'Elizabeth Smith'),
       ('Marketing', 'Adam Doe')]

Now, you need to create a dictionary that groups the employees by department. To do this, you can use a defaultdict as follows:

from collections import defaultdict

dep_dd = defaultdict(list)
for department, employee in dep:
    dep_dd[department].append(employee)

Here, you create a defaultdict called dep_dd and use a for loop to iterate through your dep list. The statement dep_dd[department].append(employee) creates the keys for the departments, initializes them to an empty list, and then appends the employees to each department. Once you run this code, your dep_dd will look something like this:

>>>

defaultdict(<class 'list'>, {'Sales': ['John Doe', 'Martin Smith'],
                             'Accounting' : ['Jane Doe'],
                             'Marketing': ['Elizabeth Smith', 'Adam Doe']})

In this example, you group the employees by their department using a defaultdict with .default_factory set to list. To do this with a regular dictionary, you can use dict.setdefault() as follows:

dep_d = dict()
for department, employee in dep:
    dep_d.setdefault(department, []).append(employee)

This code is straightforward, and you’ll find similar code quite often in your work as a Python coder. However, the defaultdict version is arguably more readable, and for large datasets, it can also be a lot faster and more efficient. So, if speed is a concern for you, then you should consider using a defaultdict instead of a standard dict.

Grouping Unique Items

Continue working with the data of departments and employees from the previous section. After some processing, you realize that a few employees have been duplicated in the database by mistake. You need to clean up the data and remove the duplicated employees from your dep_dd dictionary. To do this, you can use a set as the .default_factory and rewrite your code as follows:

dep = [('Sales', 'John Doe'),
       ('Sales', 'Martin Smith'),
       ('Accounting', 'Jane Doe'),
       ('Marketing', 'Elizabeth Smith'),
       ('Marketing', 'Elizabeth Smith'),
       ('Marketing', 'Adam Doe'),
       ('Marketing', 'Adam Doe'),
       ('Marketing', 'Adam Doe')]

dep_dd = defaultdict(set)
for department, employee in dep:
    dep_dd[department].add(employee)

In this example, you set .default_factory to set. Sets are collections of unique objects, which means that you can’t create a set with repeated items. This is a really interesting feature of sets, which guarantees that you won’t have repeated items in your final dictionary.

Counting Items

If you set .default_factory to int, then your defaultdict will be useful for counting the items in a sequence or collection. When you call int() with no arguments, the function returns 0, which is the typical value you’d use to initialize a counter.

To continue with the example of the company database, suppose you want to build a dictionary that counts the number of employees per department. In this case, you can code something like this:

>>>

>>> from collections import defaultdict
>>> dep = [('Sales', 'John Doe'),
...        ('Sales', 'Martin Smith'),
...        ('Accounting', 'Jane Doe'),
...        ('Marketing', 'Elizabeth Smith'),
...        ('Marketing', 'Adam Doe')]
>>> dd = defaultdict(int)
>>> for department, _ in dep:
...     dd[department] += 1
>>> dd
defaultdict(<class 'int'>, {'Sales': 2, 'Accounting': 1, 'Marketing': 2})

Here, you set .default_factory to int. When you call int() with no argument, the returned value is 0. You can use this default value to start counting the employees that work in each department. For this code to work correctly, you need a clean dataset. There must be no repeated data. Otherwise, you’ll need to filter out the repeated employees.

Another example of counting items is the mississippi example, where you count the number of times each letter in a word is repeated. Take a look at the following code:

>>>

>>> from collections import defaultdict
>>> s = 'mississippi'
>>> dd = defaultdict(int)
>>> for letter in s:
...     dd[letter] += 1
...
>>> dd
defaultdict(<class 'int'>, {'m': 1, 'i': 4, 's': 4, 'p': 2})

In the above code, you create a defaultdict with .default_factory set to int. This sets the default value for any given key to 0. Then, you use a for loop to traverse the string s and use an augmented assignment operation to add 1 to the counter in every iteration. The keys of dd will be the letters in mississippi.

Note: Python’s augmented assignment operators are a handy shortcut to common operations.

Take a look at the following examples:

var += 1 is equivalent to var = var + 1
var -= 1 is equivalent to var = var - 1
var *= 1 is equivalent to var = var * 1

This is just a sample of how the augmented assignment operators work. You can take a look at the official documentation to learn more about this feature.

As counting is a relatively common task in programming, the Python dictionary-like class collections.Counter is specially designed for counting items in a sequence. With Counter, you can write the mississippi example as follows:

>>>

>>> from collections import Counter
>>> counter = Counter('mississippi')
>>> counter
Counter({'i': 4, 's': 4, 'p': 2, 'm': 1})

In this case, Counter does all the work for you! You only need to pass in a sequence, and the dictionary will count its items, storing them as keys and the counts as values. Note that this example works because Python strings are also a sequence type.

Accumulating Values

Sometimes you’ll need to calculate the total sum of the values in a sequence or collection. Let’s say you have the following Excel sheet with data about the sales of your Python website:

Products	July	August	September
Books	1250.00	1300.00	1420.00
Tutorials	560.00	630.00	750.00
Courses	2500.00	2430.00	2750.00

Next, you process the data using Python and get the following list of tuple objects:

incomes = [('Books', 1250.00),
           ('Books', 1300.00),
           ('Books', 1420.00),
           ('Tutorials', 560.00),
           ('Tutorials', 630.00),
           ('Tutorials', 750.00),
           ('Courses', 2500.00),
           ('Courses', 2430.00),
           ('Courses', 2750.00),]

With this data, you want to calculate the total income per product. To do that, you can use a Python defaultdict with float as .default_factory and then code something like this:

 1 from collections import defaultdict
 2 
 3 dd = defaultdict(float)
 4 for product, income in incomes:
 5     dd[product] += income
 6 
 7 for product, income in dd.items():
 8     print(f'Total income for {product}: ${income:,.2f}')

Here’s what this code does:

In line 1, you import the Python defaultdict type.
In line 3, you create a defaultdict object with .default_factory set to float.
In line 4, you define a for loop to iterate through the items of incomes.
In line 5, you use an augmented assignment operation (+=) to accumulate the incomes per product in the dictionary.

The second loop iterates through the items of dd and prints the incomes to your screen.

Note: If you want to dive deeper into dictionary iteration, check out How to Iterate Through a Dictionary in Python.

If you put all this code into a file called incomes.py and run it from your command line, then you’ll get the following output:

$ python3 incomes.py
Total income for Books: $3,970.00
Total income for Tutorials: $1,940.00
Total income for Courses: $7,680.00

You now have a summary of incomes per product, so you can make decisions on which strategy to follow for increasing the total income of your site.

Diving Deeper Into `defaultdict`

So far, you’ve learned how to use the Python defaultdict type by coding some practical examples. At this point, you can dive deeper into type implementation and other working details. That’s what you’ll be covering in the next few sections.

`defaultdict` vs `dict`

For you to better understand the Python defaultdict type, a good exercise would be to compare it with its superclass, dict. If you want to know the methods and attributes that are specific to the Python defaultdict type, then you can run the following line of code:

>>>

>>> set(dir(defaultdict)) - set(dir(dict))
{'__copy__', 'default_factory', '__missing__'}

In the above code, you use dir() to get the list of valid attributes for dict and defaultdict. Then, you use a set difference to get the set of methods and attributes that you can only find in defaultdict. As you can see, the differences between these two classes are. You have two methods and one instance attribute. The following table shows what the methods and the attribute are for:

Method or Attribute	Description
`.__copy__()`	Provides support for `copy.copy()`
`.default_factory`	Holds the callable invoked by `.__missing__()` to automatically provide default values for missing keys
`.__missing__(key)`	Gets called when `.__getitem__()` can’t find `key`

In the above table, you can see the methods and the attribute that make a defaultdict different from a regular dict. The rest of the methods are the same in both classes.

Note: If you initialize a defaultdict using a valid callable, then you won’t get a KeyError when you try to get access to a missing key. Any key that doesn’t exist gets the value returned by .default_factory.

Additionally, you might notice that a defaultdict is equal to a dict with the same items:

>>>

>>> std_dict = dict(numbers=[1, 2, 3], letters=['a', 'b', 'c'])
>>> std_dict
{'numbers': [1, 2, 3], 'letters': ['a', 'b', 'c']}
>>> def_dict = defaultdict(list, numbers=[1, 2, 3], letters=['a', 'b', 'c'])
>>> def_dict
defaultdict(<class 'list'>, {'numbers': [1, 2, 3], 'letters': ['a', 'b', 'c']})
>>> std_dict == def_dict
True

Here, you create a regular dictionary std_dict with some arbitrary items. Then, you create a defaultdict with the same items. If you test both dictionaries for content equality, then you’ll see that they’re equal.

`defaultdict.default_factory`

The first argument to the Python defaultdict type must be a callable that takes no arguments and returns a value. This argument is assigned to the instance attribute, .default_factory. For this, you can use any callable, including functions, methods, classes, type objects, or any other valid callable. The default value of .default_factory is None.

If you instantiate defaultdict without passing a value to .default_factory, then the dictionary will behave like a regular dict and the usual KeyError will be raised for missing key lookup or modification attempts:

>>>

>>> from collections import defaultdict
>>> dd = defaultdict()
>>> dd['missing_key']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    dd['missing_key']
KeyError: 'missing_key'

Here, you instantiate the Python defaultdict type with no arguments. In this case, the instance behaves like a standard dictionary. So, if you try to access or modify a missing key, then you’ll get the usual KeyError. From this point on, you can use dd as a normal Python dictionary and, unless you assign a new callable to .default_factory, you won’t be able to use the ability of defaultdict to handle missing keys automatically.

If you pass None to the first argument of defaultdict, then the instance will behave the same way you saw in the above example. That’s because .default_factory defaults to None, so both initializations are equivalent. On the other hand, if you pass a valid callable object to .default_factory, then you can use it to handle missing keys in a user-friendly way. Here’s an example where you pass list to .default_factory:

>>>

>>> dd = defaultdict(list, letters=['a', 'b', 'c'])
>>> dd.default_factory
<class 'list'>
>>> dd
defaultdict(<class 'list'>, {'letters': ['a', 'b', 'c']})
>>> dd['numbers']
[]
>>> dd
defaultdict(<class 'list'>, {'letters': ['a', 'b', 'c'], 'numbers': []})
>>> dd['numbers'].append(1)
>>> dd
defaultdict(<class 'list'>, {'letters': ['a', 'b', 'c'], 'numbers': [1]})
>>> dd['numbers'] += [2, 3]
>>> dd
defaultdict(<class 'list'>, {'letters': ['a', 'b', 'c'], 'numbers': [1, 2, 3]})

In this example, you create a Python defaultdict called dd, then you use list for its first argument. The second argument is called letters and holds a list of letters. You see that .default_factory now holds a list object that will be called when you need to supply a default value for any missing key.

Notice that when you try to access numbers, dd tests if numbers is in the dictionary. If it’s not, then it calls .default_factory(). Since .default_factory holds a list object, the returned value is an empty list ([]).

Now that dd['numbers'] is initialized with an empty list, you can use .append() to add elements to the list. You can also use an augmented assignment operator (+=) to concatenate the lists [1] and [2, 3]. This way, you can handle missing keys in a more Pythonic and more efficient way.

On the other hand, if you pass a non-callable object to the initializer of the Python defaultdict type, then you’ll get a TypeError like in the following code:

>>>

>>> defaultdict(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    defaultdict(0)
TypeError: first argument must be callable or None

Here, you pass 0 to .default_factory. Since 0 is not a callable object, you get a TypeError telling you that the first argument must be callable or None. Otherwise, defaultdict doesn’t work.

Keep in mind that .default_factory is only called from .__getitem__() and not from other methods. This means that if dd is a defaultdict and key is a missing key, then dd[key] will call .default_factory to provide a default value, but dd.get(key) still returns None instead of the value that .default_factory would provide. That’s because .get() doesn’t call .__getitem__() to retrieve the key.

Take a look at the following code:

>>>

>>> dd = defaultdict(list)
>>> # Calls dd.__getitem__('missing')
>>> dd['missing']
[]
>>> # Don't call dd.__getitem__('another_missing')
>>> print(dd.get('another_missing'))
None
>>> dd
defaultdict(<class 'list'>, {'missing': []})

In this code fragment, you can see that dd.get() returns None rather than the default value that .default_factory would provide. That’s because .default_factory is only called from .__missing__(), which is not called by .get().

Notice that you can also add arbitrary values to a Python defaultdict. This means that you’re not limited to values with the same type as the values generated by .default_factory. Here’s an example:

>>>

>>> dd = defaultdict(list)
>>> dd
defaultdict(<class 'list'>, {})
>>> dd['string'] = 'some string'
>>> dd
defaultdict(<class 'list'>, {'string': 'some string'})
>>> dd['list']
[]
>>> dd
defaultdict(<class 'list'>, {'string': 'some string', 'list': []})

Here, you create a defaultdict and pass in a list object to .default_factory. This sets your default values to be empty lists. However, you can freely add a new key that holds values of a different type. That’s the case with the key string, which holds a str object instead of a list object.

Finally, you can always change or update the callable you initially assign to .default_factory in the same way you would do with any instance attribute:

>>>

>>> dd.default_factory = str
>>> dd['missing_key']
''

In the above code, you change .default_factory from list to str. Now, whenever you try to get access to a missing key, your default value will be an empty string ('').

Depending on your use cases for the Python defaultdict type, you might need to freeze the dictionary once you finish creating it and make it read-only. To do this, you can set .default_factory to None after you finish populating the dictionary. This way, your dictionary will behave like a standard dict, which means you won’t have more automatically generated default values.

`defaultdict` vs `dict.setdefault()`

As you saw before, dict provides .setdefault(), which will allow you to assign values to missing keys on the fly. In contrast, with a defaultdict you can specify the default value up front when you initialize the container. You can use .setdefault() to assign default values as follows:

>>>

>>> d = dict()
>>> d.setdefault('missing_key', [])
[]
>>> d
{'missing_key': []}

In this code, you create a regular dictionary and then use .setdefault() to assign a value ([]) to the key missing_key, which wasn’t defined yet.

Note: You can assign any type of Python object using .setdefault(). This is an important difference compared to defaultdict if you consider that defaultdict only accepts a callable or None.

On the other hand, if you use a defaultdict to accomplish the same task, then the default value is generated on demand whenever you try to access or modify a missing key. Notice that, with defaultdict, the default value is generated by the callable you pass upfront to the initializer of the class. Here’s how it works:

>>>

>>> from collections import defaultdict
>>> dd = defaultdict(list)
>>> dd['missing_key']
[]
>>> dd
defaultdict(<class 'list'>, {'missing_key': []})

Here, you first import the Python defaultdict type from collections. Then, you create a defaultdict and pass list to .default_factory. When you try to get access to a missing key, defaultdict internally calls .default_factory(), which holds a reference to list, and assigns the resulting value (an empty list) to missing_key.

The code in the above two examples does the same work, but the defaultdict version is arguably more readable, user-friendly, Pythonic, and straightforward.

Note: A call to a built-in type like list, set, dict, str, int, or float will return an empty object or zero for numeric types.

Take a look at the following code examples:

>>>

>>> list()
[]
>>> set()
set([])
>>> dict()
{}
>>> str()
''
>>> float()
0.0
>>> int()
0

In this code, you call some built-in types with no arguments and get an empty object or zero for the numeric types.

Finally, using a defaultdict to handle missing keys can be faster than using dict.setdefault(). Take a look a the following example:

# Filename: exec_time.py

from collections import defaultdict
from timeit import timeit

animals = [('cat', 1), ('rabbit', 2), ('cat', 3), ('dog', 4), ('dog', 1)]
std_dict = dict()
def_dict = defaultdict(list)

def group_with_dict():
    for animal, count in animals:
        std_dict.setdefault(animal, []).append(count)
    return std_dict

def group_with_defaultdict():
    for animal, count in animals:
        def_dict[animal].append(count)
    return def_dict

print(f'dict.setdefault() takes {timeit(group_with_dict)} seconds.')
print(f'defaultdict takes {timeit(group_with_defaultdict)} seconds.')

If you run the script from your system’s command line, then you’ll get something like this:

$ python3 exec_time.py
dict.setdefault() takes 1.0281260240008123 seconds.
defaultdict takes 0.6704721650003194 seconds.

Here, you use timeit.timeit() to measure the execution time of group_with_dict() and group_with_defaultdict(). These functions perform equivalent actions, but the first uses dict.setdefault(), and the second uses a defaultdict. The time measure will depend on your current hardware, but you can see here that defaultdict is faster than dict.setdefault(). This difference can become more important as the dataset gets larger.

Additionally, you need to consider that creating a regular dict can be faster than creating a defaultdict. Take a look at this code:

>>>

>>> from timeit import timeit
>>> from collections import defaultdict
>>> print(f'dict() takes {timeit(dict)} seconds.')
dict() takes 0.08921320698573254 seconds.
>>> print(f'defaultdict() takes {timeit(defaultdict)} seconds.')
defaultdict() takes 0.14101867799763568 seconds.

This time, you use timeit.timeit() to measure the execution time of dict and defaultdict instantiation. Notice that creating a dict takes almost half the time of creating a defaultdict. This might not be a problem if you consider that, in real-world code, you normally instantiate defaultdict only once.

Also notice that, by default, timeit.timeit() will run your code a million times. That’s the reason for defining std_dict and def_dict out of the scope of group_with_dict() and group_with_defaultdict() in exec_time.py. Otherwise, the time measure will be affected by the instantiation time of dict and defaultdict.

At this point, you may have an idea of when to use a defaultdict rather than a regular dict. Here are three things to take into account:

If your code is heavily base on dictionaries and you’re dealing with missing keys all the time, then you should consider using a defaultdict rather than a regular dict.
If your dictionary items need to be initialized with a constant default value, then you should consider using a defaultdict instead of a dict.
If your code relies on dictionaries for aggregating, accumulating, counting, or grouping values, and performance is a concern, then you should consider using a defaultdict.

You can consider the above guidelines when deciding whether to use a dict or a defaultdict.

`defaultdict.missing()`

Behind the scenes, the Python defaultdict type works by calling .default_factory to supply default values to missing keys. The mechanism that makes this possible is .__missing__(), a special method supported by all the standard mapping types, including dict and defaultdict.

Note: Note that .__missing__() is automatically called by .__getitem__() to handle missing keys and that .__getitem__() is automatically called by Python at the same time for subscription operations like d[key].

So, how does .__missing__() work? If you set .default_factory to None, then .__missing__() raises a KeyError with the key as an argument. Otherwise, .default_factory is called without arguments to provide a default value for the given key. This value is inserted into the dictionary and finally returned. If calling .default_factory raises an exception, then the exception is propagated unchanged.

The following code shows a viable Python implementation for .__missing__():

 1 def __missing__(self, key):
 2     if self.default_factory is None:
 3         raise KeyError(key)
 4     if key not in self:
 5         self[key] = self.default_factory()
 6     return self[key]

Here’s what this code does:

In line 1, you define the method and its signature.
In lines 2 and 3, you test to see if .default_factory is None. If so, then you raise a KeyError with the key as an argument.
In lines 4 and 5, you check if the key is not in the dictionary. If it’s not, then you call .default_factory and assign its return value to the key.
In line 6, you return the key as expected.

Keep in mind that the presence of .__missing__() in a mapping has no effect on the behavior of other methods that look up keys, such as .get() or .__contains__(), which implements the in operator. That’s because .__missing__() is only called by .__getitem__() when the requested key is not found in the dictionary. Whatever .__missing__() returns or raises is then returned or raised by .__getitem__().

Now that you’ve covered an alternative Python implementation for .__missing__(), it would be a good exercise to try to emulate defaultdict with some Python code. That’s what you’ll be doing in the next section.

Emulating the Python `defaultdict` Type

In this section, you’ll be coding a Python class that will behave much like a defaultdict. To do that, you’ll subclass collections.UserDict and then add .__missing__(). Also, you need to add an instance attribute called .default_factory, which will hold the callable for generating default values on demand. Here’s a piece of code that emulates most of the behavior of the Python defaultdict type:

 1 import collections
 2 
 3 class my_defaultdict(collections.UserDict):
 4     def __init__(self, default_factory=None, *args, **kwargs):
 5         super().__init__(*args, **kwargs)
 6         if not callable(default_factory) and default_factory is not None:
 7             raise TypeError('first argument must be callable or None')
 8         self.default_factory = default_factory
 9 
10     def __missing__(self, key):
11         if self.default_factory is None:
12             raise KeyError(key)
13         if key not in self:
14             self[key] = self.default_factory()
15         return self[key]

Here’s how this code works:

In line 1, you import collections to get access to UserDict.
In line 3, you create a class that subclasses UserDict.
In line 4, you define the class initializer .__init__(). This method takes an argument called default_factory to hold the callable that you’ll use to generate the default values. Notice that default_factory defaults to None, just like in a defaultdict. You also need the *args and **kwargs for emulating the normal behavior of a regular dict.
In line 5, you call the superclass .__init__(). This means that you’re calling UserDict.__init__() and passing *args and **kwargs to it.
In line 6, you first check if default_factory is a valid callable object. In this case, you use callable(object), which is a built-in function that returns True if object appears to be a callable and otherwise returns False. This check ensures that you can call .default_factory() if you need to generate a default value for any missing key. Then, you check if .default_factory is not None.
In line 7, you raise a TypeError just like a regular dict would do if default_factory is None.
In line 8, you initialize .default_factory.
In line 10, you define .__missing__(), which is implemented as you saw before. Recall that .__missing__() is automatically called by .__getitem__() when a given key is not in a dictionary.

If you feel in the mood to read some C code, then you can take a look at the full code for the Python defaultdict Type in the CPython source code.

Now that you’ve finished coding this class, you can test it by putting the code into a Python script called my_dd.py and importing it from an interactive session. Here’s an example:

>>>

>>> from my_dd import my_defaultdict
>>> dd_one = my_defaultdict(list)
>>> dd_one
{}
>>> dd_one['missing']
[]
>>> dd_one
{'missing': []}
>>> dd_one.default_factory = int
>>> dd_one['another_missing']
0
>>> dd_one
{'missing': [], 'another_missing': 0}
>>> dd_two = my_defaultdict(None)
>>> dd_two['missing']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    dd_two['missing']
  File "/home/user/my_dd.py", line 10,
 in __missing__
    raise KeyError(key)
KeyError: 'missing'

Here, you first import my_defaultdict from my_dd. Then, you create an instance of my_defaultdict and pass list to .default_factory. If you try to get access to a key with a subscription operation, like dd_one['missing'], then .__getitem__() is automatically called by Python. If the key is not in the dictionary, then .__missing__() is called, which generates a default value by calling .default_factory().

You can also change the callable assigned to .default_factory using a normal assignment operation like in dd_one.default_factory = int. Finally, if you pass None to .default_factory, then you’ll get a KeyError when trying to retrieve a missing key.

Note: The behavior of a defaultdict is essentially the same as this Python equivalent. However, you’ll soon note that your Python implementation doesn’t print as a real defaultdict but as a standard dict. You can modify this detail by overriding .__str__() and .__repr__().

You may be wondering why you subclass collections.UserDict instead of a regular dict for this example. The main reason for this is that subclassing built-in types can be error-prone because the C code of the built-ins doesn’t seem to consistently call special methods overridden by the user.

Here’s an example that shows some issues that you can face when subclassing dict:

>>>

>>> class MyDict(dict):
...     def __setitem__(self, key, value):
...         super().__setitem__(key, None)
...
>>> my_dict = MyDict(first=1)
>>> my_dict
{'first': 1}
>>> my_dict['second'] = 2
>>> my_dict
{'first': 1, 'second': None}
>>> my_dict.setdefault('third', 3)
3
>>> my_dict
{'first': 1, 'second': None, 'third': 3}

In this example, you create MyDict, which is a class that subclasses dict. Your implementation of .__setitem__() always sets values to None. If you create an instance of MyDict and pass a keyword argument to its initializer, then you’ll notice the class is not calling your .__setitem__() to handle the assignment. You know that because the key first wasn’t assigned None.

By contrast, if you run a subscription operation like my_dict['second'] = 2, then you’ll notice that second is set to None rather than to 2. So, this time you can say that subscription operations call your custom .__setitem__(). Finally, notice that .setdefault() doesn’t call .__setitem__() either, because your third key ends up with a value of 3.

UserDict doesn’t inherit from dict but simulates the behavior of a standard dictionary. The class has an internal dict instance called .data, which is used to store the content of the dictionary. UserDict is a more reliable class when it comes to creating custom mappings. If you use UserDict, then you’ll be avoiding the issues you saw before. To prove this, go back to the code for my_defaultdict and add the following method:

 1 class my_defaultdict(collections.UserDict):
 2     # Snip
 3     def __setitem__(self, key, value):
 4         print('__setitem__() gets called')
 5         super().__setitem__(key, None)

Here, you add a custom .__setitem__() that calls the superclass .__setitem__(), which always sets the value to None. Update this code in your script my_dd.py and import it from an interactive session as follows:

>>>

>>> from my_dd import my_defaultdict
>>> my_dict = my_defaultdict(list, first=1)
__setitem__() gets called
>>> my_dict
{'first': None}
>>> my_dict['second'] = 2
__setitem__() gets called
>>> my_dict
{'first': None, 'second': None}

In this case, when you instantiate my_defaultdict and pass first to the class initializer, your custom __setitem__() gets called. Also, when you assign a value to the key second, __setitem__() gets called as well. You now have a my_defaultdict that consistently calls your custom special methods. Notice that all the values in the dictionary are equal to None now.

Passing Arguments to `.default_factory`

As you saw earlier, .default_factory must be set to a callable object that takes no argument and returns a value. This value will be used to supply a default value for any missing key in the dictionary. Even when .default_factory shouldn’t take arguments, Python offers some tricks that you can use if you need to supply arguments to it. In this section, you’ll cover two Python tools that can serve this purpose:

With these two tools, you can add extra flexibility to the Python defaultdict type. For example, you can initialize a defaultdict with a callable that takes an argument and, after some processing, you can update the callable with a new argument to change the default value for the keys you’ll create from this point on.

Using `lambda`

A flexible way to pass arguments to .default_factory is to use lambda. Suppose you want to create a function to generate default values in a defaultdict. The function does some processing and returns a value, but you need to pass an argument for the function to work correctly. Here’s an example:

>>>

>>> def factory(arg):
...     # Do some processing here...
...     result = arg.upper()
...     return result
...
>>> def_dict = defaultdict(lambda: factory('default value'))
>>> def_dict['missing']
'DEFAULT VALUE'

In the above code, you create a function called factory(). The function takes an argument, does some processing, and returns the final result. Then, you create a defaultdict and use lambda to pass the string 'default value' to factory(). When you try to get access to a missing key, the following steps are run:

The dictionary def_dict calls its .default_factory, which holds a reference to a lambda function.
The lambda function gets called and returns the value that results from calling factory() with 'default value' as an argument.

If you’re working with def_dict and suddenly need to change the argument to factory(), then you can do something like this:

>>>

>>> def_dict.default_factory = lambda: factory('another default value')
>>> def_dict['another_missing']
'ANOTHER DEFAULT VALUE'

This time, factory() takes a new string argument ('another default value'). From now on, if you try to access or modify a missing key, then you’ll get a new default value, which is the string 'ANOTHER DEFAULT VALUE'.

Finally, you can possibly face a situation where you need a default value that’s different from 0 or []. In this case, you can also use lambda to generate a different default value. For example, suppose you have a list of integer numbers, and you need to calculate the cumulative product of each number. Then, you can use a defaultdict along with lambda as follows:

>>>

>>> from collections import defaultdict
>>> lst = [1, 1, 2, 1, 2, 2, 3, 4, 3, 3, 4, 4]
>>> def_dict = defaultdict(lambda: 1)
>>> for number in lst:
...     def_dict[number] *= number
...
>>> def_dict
defaultdict(<function <lambda> at 0x...70>, {1: 1, 2: 8, 3: 27, 4: 64})

Here, you use lambda to supply a default value of 1. With this initial value, you can calculate the cumulative product of each number in lst. Notice that you can’t get the same result using int because the default value returned by int is always 0, which is not a good initial value for the multiplication operations you need to perform here.

Using `functools.partial()`

functools.partial(func, *args, **keywords) is a function that returns a partial object. When you call this object with the positional arguments (args) and keyword arguments (keywords), it behaves similar to when you call func(*args, **keywords). You can take advantage of this behavior of partial() and use it to pass arguments to .default_factory in a Python defaultdict. Here’s an example:

>>>

>>> def factory(arg):
...     # Do some processing here...
...     result = arg.upper()
...     return result
...
>>> from functools import partial
>>> def_dict = defaultdict(partial(factory, 'default value'))
>>> def_dict['missing']
'DEFAULT VALUE'
>>> def_dict.default_factory = partial(factory, 'another default value')
>>> def_dict['another_missing']
'ANOTHER DEFAULT VALUE'

Here, you create a Python defaultdict and use partial() to supply an argument to .default_factory. Notice that you can also update .default_factory to use another argument for the callable factory(). This kind of behavior can add a lot of flexibility to your defaultdict objects.

Conclusion

The Python defaultdict type is a dictionary-like data structure provided by the Python standard library in a module called collections. The class inherits from dict, and its main added functionality is to supply default values for missing keys. In this tutorial, you’ve learned how to use the Python defaultdict type for handling the missing keys in a dictionary.

You’re now able to:

Create and use a Python defaultdict to handle missing keys
Solve real-world problems related to grouping, counting, and accumulating operations
Know the implementation differences between defaultdict and dict
Decide when and why to use a Python defaultdict rather than a standard dict

The Python defaultdict type is a convenient and efficient data structure that’s designed to help you out when you’re dealing with missing keys in a dictionary. Give it a try and make your code faster, more readable, and more Pythonic!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Defining Main Functions in Python

2020-03-10T14:00:00+00:00

Many programming languages have a special function that is automatically executed when an operating system starts to run a program. This function is usually called main() and must have a specific return type and arguments according to the language standard. On the other hand, the Python interpreter executes scripts starting at the top of the file, and there is no specific function that Python automatically executes.

Nevertheless, having a defined starting point for the execution of a program is useful for understanding how a program works. Python programmers have come up with several conventions to define this starting point.

By the end of this course, you’ll understand:

What the special __name__ variable is and how Python defines it
Why you would want to use a main() in Python
What conventions there are for defining main() in Python
What the best practices are for what code to put into your main()

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Defining Your Own Python Function

2020-03-09T14:00:00+00:00

Throughout the previous tutorials in this series, you’ve seen many examples demonstrating the use of built-in Python functions. In this tutorial, you’ll learn how to define your own Python function. You’ll learn when to divide your program into separate user-defined functions and what tools you’ll need to do this.

Here’s what you’ll learn in this tutorial:

How functions work in Python and why they’re beneficial
How to define and call your own Python function
Mechanisms for passing arguments to your function
How to return data from your function back to the calling environment

Free PDF Download: Python 3 Cheat Sheet

Functions in Python

You may be familiar with the mathematical concept of a function. A function is a relationship or mapping between one or more inputs and a set of outputs. In mathematics, a function is typically represented like this:

Here, f is a function that operates on the inputs x and y. The output of the function is z. However, programming functions are much more generalized and versatile than this mathematical definition. In fact, appropriate function definition and use is so critical to proper software development that virtually all modern programming languages support both built-in and user-defined functions.

In programming, a function is a self-contained block of code that encapsulates a specific task or related group of tasks. In previous tutorials in this series, you’ve been introduced to some of the built-in functions provided by Python. id(), for example, takes one argument and returns that object’s unique integer identifier:

>>>

>>> s = 'foobar'
>>> id(s)
56313440

len() returns the length of the argument passed to it:

>>>

>>> a = ['foo', 'bar', 'baz', 'qux']
>>> len(a)
4

any() takes an iterable as its argument and returns True if any of the items in the iterable are truthy and False otherwise:

>>>

>>> any([False, False, False])
False
>>> any([False, True, False])
True

>>> any(['bar' == 'baz', len('foo') == 4, 'qux' in {'foo', 'bar', 'baz'}])
False
>>> any(['bar' == 'baz', len('foo') == 3, 'qux' in {'foo', 'bar', 'baz'}])
True

Each of these built-in functions performs a specific task. The code that accomplishes the task is defined somewhere, but you don’t need to know where or even how the code works. All you need to know about is the function’s interface:

What arguments (if any) it takes
What values (if any) it returns

Then you call the function and pass the appropriate arguments. Program execution goes off to the designated body of code and does its useful thing. When the function is finished, execution returns to your code where it left off. The function may or may not return data for your code to use, as the examples above do.

When you define your own Python function, it works just the same. From somewhere in your code, you’ll call your Python function and program execution will transfer to the body of code that makes up the function.

Note: In this case, you will know where the code is and exactly how it works because you wrote it!

When the function is finished, execution returns to the location where the function was called. Depending on how you designed the function’s interface, data may be passed in when the function is called, and return values may be passed back when it finishes.

The Importance of Python Functions

Virtually all programming languages used today support a form of user-defined functions, although they aren’t always called functions. In other languages, you may see them referred to as one of the following:

Subroutines
Procedures
Methods
Subprograms

So, why bother defining functions? There are several very good reasons. Let’s go over a few now.

Abstraction and Reusability

Suppose you write some code that does something useful. As you continue development, you find that the task performed by that code is one you need often, in many different locations within your application. What should you do? Well, you could just replicate the code over and over again, using your editor’s copy-and-paste capability.

Later on, you’ll probably decide that the code in question needs to be modified. You’ll either find something wrong with it that needs to be fixed, or you’ll want to enhance it in some way. If copies of the code are scattered all over your application, then you’ll need to make the necessary changes in every location.

Note: At first blush, that may seem like a reasonable solution, but in the long term, it’s likely to be a maintenance nightmare! While your code editor may help by providing a search-and-replace function, this method is error-prone, and you could easily introduce bugs into your code that will be difficult to find.

A better solution is to define a Python function that performs the task. Anywhere in your application that you need to accomplish the task, you simply call the function. Down the line, if you decide to change how it works, then you only need to change the code in one location, which is the place where the function is defined. The changes will automatically be picked up anywhere the function is called.

The abstraction of functionality into a function definition is an example of the Don’t Repeat Yourself (DRY) Principle of software development. This is arguably the strongest motivation for using functions.

Modularity

Functions allow complex processes to be broken up into smaller steps. Imagine, for example, that you have a program that reads in a file, processes the file contents, and then writes an output file. Your code could look like this:

# Main program

# Code to read file in
<statement>
<statement>
<statement>
<statement>

# Code to process file
<statement>
<statement>
<statement>
<statement>

# Code to write file out
<statement>
<statement>
<statement>
<statement>

In this example, the main program is a bunch of code strung together in a long sequence, with whitespace and comments to help organize it. However, if the code were to get much lengthier and more complex, then you’d have an increasingly difficult time wrapping your head around it.

Alternatively, you could structure the code more like the following:

def read_file():
    # Code to read file in
    <statement>
    <statement>
    <statement>
    <statement>

def process_file():
    # Code to process file
    <statement>
    <statement>
    <statement>
    <statement>

def write_file():
    # Code to write file out
    <statement>
    <statement>
    <statement>
    <statement>


# Main program
read_file()
process_file()
write_file()

This example is modularized. Instead of all the code being strung together, it’s broken out into separate functions, each of which focuses on a specific task. Those tasks are read, process, and write. The main program now simply needs to call each of these in turn.

Note: The def keyword introduces a new Python function definition. You’ll learn all about this very soon.

In life, you do this sort of thing all the time, even if you don’t explicitly think of it that way. If you wanted to move some shelves full of stuff from one side of your garage to the other, then you hopefully wouldn’t just stand there and aimlessly think, “Oh, geez. I need to move all that stuff over there! How do I do that???” You’d divide the job into manageable steps:

Take all the stuff off the shelves.
Take the shelves apart.
Carry the shelf parts across the garage to the new location.
Re-assemble the shelves.
Carry the stuff across the garage.
Put the stuff back on the shelves.

Breaking a large task into smaller, bite-sized sub-tasks helps make the large task easier to think about and manage. As programs become more complicated, it becomes increasingly beneficial to modularize them in this way.

Namespace Separation

A namespace is a region of a program in which identifiers have meaning. As you’ll see below, when a Python function is called, a new namespace is created for that function, one that is distinct from all other namespaces that already exist.

The practical upshot of this is that variables can be defined and used within a Python function even if they have the same name as variables defined in other functions or in the main program. In these cases, there will be no confusion or interference because they’re kept in separate namespaces.

This means that when you write code within a function, you can use variable names and identifiers without worrying about whether they’re already used elsewhere outside the function. This helps minimize errors in code considerably.

Note: You’ll learn much more about namespaces in the next tutorial in this series.

Hopefully, you’re sufficiently convinced of the virtues of functions and eager to create some! Let’s see how.

Function Calls and Definition

The usual syntax for defining a Python function is as follows:

def <function_name>([<parameters>]):
    <statement(s)>

The components of the definition are explained in the table below:

Component	Meaning
`def`	The keyword that informs Python that a function is being defined
`<function_name>`	A valid Python identifier that names the function
`<parameters>`	An optional, comma-separated list of parameters that may be passed to the function
`:`	Punctuation that denotes the end of the Python function header (the name and parameter list)
`<statement(s)>`	A block of valid Python statements

The final item, <statement(s)>, is called the body of the function. The body is a block of statements that will be executed when the function is called. The body of a Python function is defined by indentation in accordance with the off-side rule. This is the same as code blocks associated with a control structure, like an if or while statement.

The syntax for calling a Python function is as follows:

<function_name>([<arguments>])

<arguments> are the values passed into the function. They correspond to the <parameters> in the Python function definition. You can define a function that doesn’t take any arguments, but the parentheses are still required. Both a function definition and a function call must always include parentheses, even if they’re empty.

As usual, you’ll start with a small example and add complexity from there. Keeping the time-honored mathematical tradition in mind, you’ll call your first Python function f(). Here’s a script file, foo.py, that defines and calls f():

 1 def f():
 2     s = '-- Inside f()'
 3     print(s)
 4 
 5 print('Before calling f()')
 6 f()
 7 print('After calling f()')

Here’s how this code works:

Line 1 uses the def keyword to indicate that a function is being defined. Execution of the def statement merely creates the definition of f(). All the following lines that are indented (lines 2 to 3) become part of the body of f() and are stored as its definition, but they aren’t executed yet.
Line 4 is a bit of whitespace between the function definition and the first line of the main program. While it isn’t syntactically necessary, it is nice to have. To learn more about whitespace around top-level Python function definitions, check out Writing Beautiful Pythonic Code With PEP 8.
Line 5 is the first statement that isn’t indented because it isn’t a part of the definition of f(). It’s the start of the main program. When the main program executes, this statement is executed first.
Line 6 is a call to f(). Note that empty parentheses are always required in both a function definition and a function call, even when there are no parameters or arguments. Execution proceeds to f() and the statements in the body of f() are executed.
Line 7 is the next line to execute once the body of f() has finished. Execution returns to this print() statement.

The sequence of execution (or control flow) for foo.py is shown in the following diagram:

When foo.py is run from a Windows command prompt, the result is as follows:

C:\Users\john\Documents\Python\doc>python foo.py
Before calling f()
-- Inside f()
After calling f()

Occasionally, you may want to define an empty function that does nothing. This is referred to as a stub, which is usually a temporary placeholder for a Python function that will be fully implemented at a later time. Just as a block in a control structure can’t be empty, neither can the body of a function. To define a stub function, use the pass statement:

>>>

>>> def f():
...     pass
...
>>> f()

As you can see above, a call to a stub function is syntactically valid but doesn’t do anything.

Argument Passing

So far in this tutorial, the functions you’ve defined haven’t taken any arguments. That can sometimes be useful, and you’ll occasionally write such functions. More often, though, you’ll want to pass data into a function so that its behavior can vary from one invocation to the next. Let’s see how to do that.

Positional Arguments

The most straightforward way to pass arguments to a Python function is with positional arguments (also called required arguments). In the function definition, you specify a comma-separated list of parameters inside the parentheses:

>>>

>>> def f(qty, item, price):
...     print(f'{qty} {item} cost ${price:.2f}')
...

When the function is called, you specify a corresponding list of arguments:

>>>

>>> f(6, 'bananas', 1.74)
6 bananas cost $1.74

The parameters (qty, item, and price) behave like variables that are defined locally to the function. When the function is called, the arguments that are passed (6, 'bananas', and 1.74) are bound to the parameters in order, as though by variable assignment:

Parameter		Argument
`qty`	←	`6`
`item`	←	`bananas`
`price`	←	`1.74`

In some programming texts, the parameters given in the function definition are referred to as formal parameters, and the arguments in the function call are referred to as actual parameters:

Although positional arguments are the most straightforward way to pass data to a function, they also afford the least flexibility. For starters, the order of the arguments in the call must match the order of the parameters in the definition. There’s nothing to stop you from specifying positional arguments out of order, of course:

>>>

>>> f('bananas', 1.74, 6)
bananas 1.74 cost $6.00

The function may even still run, as it did in the example above, but it’s very unlikely to produce the correct results. It’s the responsibility of the programmer who defines the function to document what the appropriate arguments should be, and it’s the responsibility of the user of the function to be aware of that information and abide by it.

With positional arguments, the arguments in the call and the parameters in the definition must agree not only in order but in number as well. That’s the reason positional arguments are also referred to as required arguments. You can’t leave any out when calling the function:

>>>

>>> # Too few arguments
>>> f(6, 'bananas')
Traceback (most recent call last):
  File "<pyshell#6>", line 1, in <module>
    f(6, 'bananas')
TypeError: f() missing 1 required positional argument: 'price'

Nor can you specify extra ones:

>>>

>>> # Too many arguments
>>> f(6, 'bananas', 1.74, 'kumquats')
Traceback (most recent call last):
  File "<pyshell#5>", line 1, in <module>
    f(6, 'bananas', 1.74, 'kumquats')
TypeError: f() takes 3 positional arguments but 4 were given

Positional arguments are conceptually straightforward to use, but they’re not very forgiving. You must specify the same number of arguments in the function call as there are parameters in the definition, and in exactly the same order. In the sections that follow, you’ll see some argument-passing techniques that relax these restrictions.

Keyword Arguments

When you’re calling a function, you can specify arguments in the form <keyword>=<value>. In that case, each <keyword> must match a parameter in the Python function definition. For example, the previously defined function f() may be called with keyword arguments as follows:

>>>

>>> f(qty=6, item='bananas', price=1.74)
6 bananas cost $1.74

Referencing a keyword that doesn’t match any of the declared parameters generates an exception:

>>>

>>> f(qty=6, item='bananas', cost=1.74)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() got an unexpected keyword argument 'cost'

Using keyword arguments lifts the restriction on argument order. Each keyword argument explicitly designates a specific parameter by name, so you can specify them in any order and Python will still know which argument goes with which parameter:

>>>

>>> f(item='bananas', price=1.74, qty=6)
6 bananas cost $1.74

Like with positional arguments, though, the number of arguments and parameters must still match:

>>>

>>> # Still too few arguments
>>> f(qty=6, item='bananas')
Traceback (most recent call last):
  File "<pyshell#16>", line 1, in <module>
    f(qty=6, item='bananas')
TypeError: f() missing 1 required positional argument: 'price'

So, keyword arguments allow flexibility in the order that function arguments are specified, but the number of arguments is still rigid.

You can call a function using both positional and keyword arguments:

>>>

>>> f(6, price=1.74, item='bananas')
6 bananas cost $1.74

>>> f(6, 'bananas', price=1.74)
6 bananas cost $1.74

When positional and keyword arguments are both present, all the positional arguments must come first:

>>>

>>> f(6, item='bananas', 1.74)
SyntaxError: positional argument follows keyword argument

Once you’ve specified a keyword argument, there can’t be any positional arguments to the right of it.

Default Parameters

If a parameter specified in a Python function definition has the form <name>=<value>, then <value> becomes a default value for that parameter. Parameters defined this way are referred to as default or optional parameters. An example of a function definition with default parameters is shown below:

>>>

>>> def f(qty=6, item='bananas', price=1.74):
...     print(f'{qty} {item} cost ${price:.2f}')
...

When this version of f() is called, any argument that’s left out assumes its default value:

>>>

>>> f(4, 'apples', 2.24)
4 apples cost $2.24
>>> f(4, 'apples')
4 apples cost $1.74

>>> f(4)
4 bananas cost $1.74
>>> f()
6 bananas cost $1.74

>>> f(item='kumquats', qty=9)
9 kumquats cost $1.74
>>> f(price=2.29)
6 bananas cost $2.29

In summary:

Positional arguments must agree in order and number with the parameters declared in the function definition.
Keyword arguments must agree with declared parameters in number, but they may be specified in arbitrary order.
Default parameters allow some arguments to be omitted when the function is called.

Mutable Default Parameter Values

Things can get weird if you specify a default parameter value that is a mutable object. Consider this Python function definition:

>>>

>>> def f(my_list=[]):
...     my_list.append('###')
...     return my_list
...

f() takes a single list parameter, appends the string '###' to the end of the list, and returns the result:

>>>

>>> f(['foo', 'bar', 'baz'])
['foo', 'bar', 'baz', '###']

>>> f([1, 2, 3, 4, 5])
[1, 2, 3, 4, 5, '###']

The default value for parameter my_list is the empty list, so if f() is called without any arguments, then the return value is a list with the single element '###':

>>>

>>> f()
['###']

Everything makes sense so far. Now, what would you expect to happen if f() is called without any parameters a second and a third time? Let’s see:

>>>

>>> f()
['###', '###']
>>> f()
['###', '###', '###']

Oops! You might have expected each subsequent call to also return the singleton list ['###'], just like the first. Instead, the return value keeps growing. What happened?

In Python, default parameter values are defined only once when the function is defined (that is, when the def statement is executed). The default value isn’t re-defined each time the function is called. Thus, each time you call f() without a parameter, you’re performing .append() on the same list.

You can demonstrate this with id():

>>>

>>> def f(my_list=[]):
...     print(id(my_list))
...     my_list.append('###')
...     return my_list
...
>>> f()
140095566958408
['###']        
>>> f()
140095566958408
['###', '###']
>>> f()
140095566958408
['###', '###', '###']

The object identifier displayed confirms that, when my_list is allowed to default, the value is the same object with each call. Since lists are mutable, each subsequent .append() call causes the list to get longer. This is a common and pretty well-documented pitfall when you’re using a mutable object as a parameter’s default value. It potentially leads to confusing code behavior, and is probably best avoided.

As a workaround, consider using a default argument value that signals no argument has been specified. Most any value would work, but None is a common choice. When the sentinel value indicates no argument is given, create a new empty list inside the function:

>>>

>>> def f(my_list=None):
...     if my_list is None:
...         my_list = []
...         my_list.append('###')
...     return my_list
...

>>> f()
['###']
>>> f()
['###']
>>> f()
['###']

>>> f(['foo', 'bar', 'baz'])
['foo', 'bar', 'baz', '###']

>>> f([1, 2, 3, 4, 5])
[1, 2, 3, 4, 5, '###']

Note how this ensures that my_list now truly defaults to an empty list whenever f() is called without an argument.

Pass-By-Value vs Pass-By-Reference in Pascal

In programming language design, there are two common paradigms for passing an argument to a function:

Pass-by-value: A copy of the argument is passed to the function.
Pass-by-reference: A reference to the argument is passed to the function.

Other mechanisms exist, but they are essentially variations on these two. In this section, you’re going to take a short detour from Python and briefly look at Pascal, a programming language that makes a particularly clear distinction between these two.

Note: Don’t worry if you aren’t familiar with Pascal! The concepts are similar to those of Python, and the examples shown are accompanied by enough detailed explanation that you should get a general idea. Once you’ve seen how argument passing works in Pascal, we’ll circle back around to Python, and you’ll see how it compares.

Here’s what you need to know about Pascal syntax:

Procedures: A procedure in Pascal is similar to a Python function.
Colon-equals: This operator (:=) is used for assignment in Pascal. It’s analogous to the equals sign (=) in Python.
writeln(): This function displays data to the console, similar to Python’s print().

With that bit of groundwork in place, here’s the first Pascal example:

 1 // Pascal Example #1
 2 
 3 procedure f(fx : integer);
 4 begin
 5     writeln('Start  f():  fx = ', fx);
 6     fx := 10;
 7     writeln('End    f():  fx = ', fx);
 8 end;
 9 
10 // Main program
11 var
12     x : integer;
13 
14 begin
15     x := 5;
16     writeln('Before f():  x = ', x);
17     f(x);
18     writeln('After  f():  x = ', x);
19 end.

Here’s what’s going on:

Line 12: The main program defines an integer variable x.
Line 15: It initially assigns x the value 5.
Line 17: It then calls procedure f(), passing x as an argument.
Line 5: Inside f(), the writeln() statement shows that the corresponding parameter fx is initially 5, the value passed in.
Line 6: fx is then assigned the value 10.
Line 7: This value is verified by this writeln() statement executed just before f() exits.
Line 18: Back in the calling environment of the main program, this writeln() statement shows that after f() returns, x is still 5, as it was prior to the procedure call.

Running this code generates the following output:

Before f():  x = 5
Start  f():  fx = 5
End    f():  fx = 10
After  f():  x = 5

In this example, x is passed by value, so f() receives only a copy. When the corresponding parameter fx is modified, x is unaffected.

Note: If you want to see this in action, then you can run the code for yourself using an online Pascal compiler.

Just follow these steps:

Copy the code from the code box above.
Visit the Online Pascal Compiler.
In the code box on the left, replace any existing contents with the code you copied in step 1.
Click Execute.

You should see the same output as above.

Now, compare this with the next example:

 1 // Pascal Example #2
 2 
 3 procedure f(var fx : integer);
 4 begin
 5     writeln('Start  f():  fx = ', fx);
 6     fx := 10;
 7     writeln('End    f():  fx = ', fx);
 8 end;
 9 
10 // Main program
11 var
12     x : integer;
13 
14 begin
15     x := 5;
16     writeln('Before f():  x = ', x);
17     f(x);
18     writeln('After  f():  x = ', x);
19 end.

This code is identical to the first example, with one change. It’s the presence of the word var in front of fx in the definition of procedure f() on line 3. That indicates that the argument to f() is passed by reference. Changes made to the corresponding parameter fx will also modify the argument in the calling environment.

The output from this code is the same as before, except for the last line:

Before f():  x = 5
Start  f():  fx = 5
End    f():  fx = 10
After  f():  x = 10

Again, fx is assigned the value 10 inside f() as before. But this time, when f() returns, x in the main program has also been modified.

In many programming languages, that’s essentially the distinction between pass-by-value and pass-by-reference:

If a variable is passed by value, then the function has a copy to work on, but it can’t modify the original value in the calling environment.
If a variable is passed by reference, then any changes the function makes to the corresponding parameter will affect the value in the calling environment.

The reason why comes from what a reference means in these languages. Variable values are stored in memory. In Pascal and similar languages, a reference is essentially the address of that memory location, as demonstrated below:

In the diagram on the left, x has memory allocated in the main program’s namespace. When f() is called, x is passed by value, so memory for the corresponding parameter fx is allocated in the namespace of f(), and the value of x is copied there. When f() modifies fx, it’s this local copy that is changed. The value of x in the calling environment remains unaffected.

In the diagram on the right, x is passed by reference. The corresponding parameter fx points to the actual address in the main program’s namespace where the value of x is stored. When f() modifies fx, it’s modifying the value in that location, just the same as if the main program were modifying x itself.

Pass-By-Value vs Pass-By-Reference in Python

Are parameters in Python pass-by-value or pass-by-reference? The answer is they’re neither, exactly. That’s because a reference doesn’t mean quite the same thing in Python as it does in Pascal.

Recall that in Python, every piece of data is an object. A reference points to an object, not a specific memory location. That means assignment isn’t interpreted the same way in Python as it is in Pascal. Consider the following pair of statements in Pascal:

x := 5
x := 10

These are interpreted this way:

The variable x references a specific memory location.
The first statement puts the value 5 in that location.
The next statement overwrites the 5 and puts 10 there instead.

By contrast, in Python, the analogous assignment statements are as follows:

x = 5
x = 10

These assignment statements have the following meaning:

The first statement causes x to point to an object whose value is 5.
The next statement reassigns x as a new reference to a different object whose value is 10. Stated another way, the second assignment rebinds x to a different object with value 10.

In Python, when you pass an argument to a function, a similar rebinding occurs. Consider this example:

>>>

 1 >>> def f(fx):
 2 ...     fx = 10
 3 ...
 4 >>> x = 5
 5 >>> f(x)
 6 >>> x
 7 5

In the main program, the statement x = 5 on line 5 creates a reference named x bound to an object whose value is 5. f() is then called on line 7, with x as an argument. When f() first starts, a new reference called fx is created, which initially points to the same 5 object as x does:

However, when the statement fx = 10 on line 2 is executed, f() rebinds fx to a new object whose value is 10. The two references, x and fx, are uncoupled from one another. Nothing else that f() does will affect x, and when f() terminates, x will still point to the object 5, as it did prior to the function call:

You can confirm all this using id(). Here’s a slightly augmented version of the above example that displays the numeric identifiers of the objects involved:

>>>

 1 >>> def f(fx):
 2 ...     print('fx =', fx, '/ id(fx) = ', id(fx))
 3 ...     fx = 10
 4 ...     print('fx =', fx, '/ id(fx) = ', id(fx))
 5 ...
 6 
 7 >>> x = 5
 8 >>> print('x =', x, '/ id(x) = ', id(x))
 9 x = 5 / id(x) =  1357924048
10 
11 >>> f(x)
12 fx = 5 / id(fx) =  1357924048
13 fx = 10 / id(fx) =  1357924128
14 
15 >>> print('x =', x, '/ id(x) = ', id(x))
16 x = 5 / id(x) =  1357924048

When f() first starts, fx and x both point to the same object, whose id() is 1357924048. After f() executes the statement fx = 10 on line 3, fx points to a different object whose id() is 1357924128. The connection to the original object in the calling environment is lost.

Argument passing in Python is somewhat of a hybrid between pass-by-value and pass-by-reference. What gets passed to the function is a reference to an object, but the reference is passed by value.

Note: Python’s argument-passing mechanism has been called pass-by-assignment. This is because parameter names are bound to objects on function entry in Python, and assignment is also the process of binding a name to an object. You may also see the terms pass-by-object, pass-by-object-reference, or pass-by-sharing.

The key takeaway here is that a Python function can’t change the value of an argument by reassigning the corresponding parameter to something else. The following example demonstrates this:

>>>

>>> def f(x):
...     x = 'foo'
...
>>> for i in (
...         40,
...         dict(foo=1, bar=2),
...         {1, 2, 3},
...         'bar',
...         ['foo', 'bar', 'baz']):
...     f(i)
...     print(i)
...
40
{'foo': 1, 'bar': 2}
{1, 2, 3}
bar
['foo', 'bar', 'baz']

Here, objects of type int, dict, set, str, and list are passed to f() as arguments. f() tries to assign each to the string object 'foo', but as you can see, once back in the calling environment, they are all unchanged. As soon as f() executes the assignment x = 'foo', the reference is rebound, and the connection to the original object is lost.

Does that mean a Python function can never modify its arguments at all? Actually, no, that isn’t the case! Watch what happens here:

>>>

>>> def f(x):
...     x[0] = '---'
...

>>> my_list = ['foo', 'bar', 'baz', 'qux']

>>> f(my_list)
>>> my_list
['---', 'bar', 'baz', 'qux']

In this case, the argument to f() is a list. When f() is called, a reference to my_list is passed. You’ve already seen that f() can’t reassign my_list wholesale. If x were assigned to something else, then it would be bound to a different object, and the connection to my_list would be lost.

However, f() can use the reference to make modifications inside my_list. Here, f() has modified the first element. You can see that once the function returns, my_list has, in fact, been changed in the calling environment. The same concept applies to a dictionary:

>>>

>>> def f(x):
...     x['bar'] = 22
...

>>> my_dict = {'foo': 1, 'bar': 2, 'baz': 3}

>>> f(my_dict)
>>> my_dict
{'foo': 1, 'bar': 22, 'baz': 3}

Here, f() uses x as a reference to make a change inside my_dict. That change is reflected in the calling environment after f() returns.

Argument Passing Summary

Argument passing in Python can be summarized as follows. Passing an immutable object, like an int, str, tuple, or frozenset, to a Python function acts like pass-by-value. The function can’t modify the object in the calling environment.

Passing a mutable object such as a list, dict, or set acts somewhat—but not exactly—like pass-by-reference. The function can’t reassign the object wholesale, but it can change items in place within the object, and these changes will be reflected in the calling environment.

Side Effects

So, in Python, it’s possible for you to modify an argument from within a function so that the change is reflected in the calling environment. But should you do this? This is an example of what’s referred to in programming lingo as a side effect.

More generally, a Python function is said to cause a side effect if it modifies its calling environment in any way. Changing the value of a function argument is just one of the possibilities.

Note: You’re probably familiar with side effects from the field of human health, where the term typically refers to an unintended consequence of medication. Often, the consequence is undesirable, like vomiting or sedation. On the other hand, side effects can be used intentionally. For example, some medications cause appetite stimulation, which can be used to an advantage, even if that isn’t the medication’s primary intent.

The concept is similar in programming. If a side effect is a well-documented part of the function specification, and the user of the function is expressly aware of when and how the calling environment might be modified, then it can be okay. But a programmer may not always properly document side effects, or they may not even be aware that side effects are occurring.

When they’re hidden or unexpected, side effects can lead to program errors that are very difficult to track down. Generally, it’s best to avoid them.

The `return` Statement

What’s a Python function to do then? After all, in many cases, if a function doesn’t cause some change in the calling environment, then there isn’t much point in calling it at all. How should a function affect its caller?

Well, one possibility is to use function return values. A return statement in a Python function serves two purposes:

It immediately terminates the function and passes execution control back to the caller.
It provides a mechanism by which the function can pass data back to the caller.

Exiting a Function

Within a function, a return statement causes immediate exit from the Python function and transfer of execution back to the caller:

>>>

>>> def f():
...     print('foo')
...     print('bar')
...     return
...

>>> f()
foo
bar

In this example, the return statement is actually superfluous. A function will return to the caller when it falls off the end—that is, after the last statement of the function body is executed. So, this function would behave identically without the return statement.

However, return statements don’t need to be at the end of a function. They can appear anywhere in a function body, and even multiple times. Consider this example:

>>>

 1 >>> def f(x):
 2 ...     if x < 0:
 3 ...         return
 4 ...     if x > 100:
 5 ...         return
 6 ...     print(x)
 7 ...
 8 
 9 >>> f(-3)
10 >>> f(105)
11 >>> f(64)
12 64

The first two calls to f() don’t cause any output, because a return statement is executed and the function exits prematurely, before the print() statement on line 6 is reached.

This sort of paradigm can be useful for error checking in a function. You can check several error conditions at the start of the function, with return statements that bail out if there’s a problem:

def f():
    if error_cond1:
        return
    if error_cond2:
        return
    if error_cond3:
        return

    <normal processing>

If none of the error conditions are encountered, then the function can proceed with its normal processing.

Returning Data to the Caller

In addition to exiting a function, the return statement is also used to pass data back to the caller. If a return statement inside a Python function is followed by an expression, then in the calling environment, the function call evaluates to the value of that expression:

>>>

 1 >>> def f():
 2 ...     return 'foo'
 3 ...
 4 
 5 >>> s = f()
 6 >>> s
 7 'foo'

Here, the value of the expression f() on line 5 is 'foo', which is subsequently assigned to variable s.

A function can return any type of object. In Python, that means pretty much anything whatsoever. In the calling environment, the function call can be used syntactically in any way that makes sense for the type of object the function returns.

For example, in this code, f() returns a dictionary. In the calling environment then, the expression f() represents a dictionary, and f()['baz'] is a valid key reference into that dictionary:

>>>

>>> def f():
...     return dict(foo=1, bar=2, baz=3)
...

>>> f()
{'foo': 1, 'bar': 2, 'baz': 3}
>>> f()['baz']
3

In the next example, f() returns a string that you can slice like any other string:

>>>

>>> def f():
...     return 'foobar'
...

>>> f()[2:4]
'ob'

Here, f() returns a list that can be indexed or sliced:

>>>

>>> def f():
...     return ['foo', 'bar', 'baz', 'qux']
...  

>>> f()
['foo', 'bar', 'baz', 'qux']
>>> f()[2]
'baz'
>>> f()[::-1]
['qux', 'baz', 'bar', 'foo']

If multiple comma-separated expressions are specified in a return statement, then they’re packed and returned as a tuple:

>>>

>>> def f():
...     return 'foo', 'bar', 'baz', 'qux'
...

>>> type(f())
<class 'tuple'>
>>> t = f()
>>> t
('foo', 'bar', 'baz', 'qux')

>>> a, b, c, d = f()
>>> print(f'a = {a}, b = {b}, c = {c}, d = {d}')
a = foo, b = bar, c = baz, d = qux

When no return value is given, a Python function returns the special Python value None:

>>>

>>> def f():
...     return
...

>>> print(f())
None

The same thing happens if the function body doesn’t contain a return statement at all and the function falls off the end:

>>>

>>> def g():
...     pass
...

>>> print(g())
None

Recall that None is falsy when evaluated in a boolean context.

Since functions that exit through a bare return statement or fall off the end return None, a call to such a function can be used in a boolean context:

>>>

>>> def f():
...     return
...
>>> def g():
...     pass
...

>>> if f() or g():
...     print('yes')
... else:
...     print('no')
...
no

Here, calls to both f() and g() are falsy, so f() or g() is as well, and the else clause executes.

Revisiting Side Effects

Suppose you want to write a function that takes an integer argument and doubles it. That is, you want to pass an integer variable to the function, and when the function returns, the value of the variable in the calling environment should be twice what it was. In Pascal, you could accomplish this using pass-by-reference:

 1 procedure double(var x : integer);
 2 begin
 3     x := x * 2;
 4 end;
 5 
 6 var
 7     x : integer;
 8 
 9 begin
10     x := 5;
11     writeln('Before procedure call: ', x);
12     double(x);
13     writeln('After procedure call:  ', x);
14 end.

Executing this code produces the following output, which verifies that double() does indeed modify x in the calling environment:

Before procedure call: 5
After procedure call:  10

In Python, this won’t work. As you now know, Python integers are immutable, so a Python function can’t change an integer argument by side effect:

>>>

>>> def double(x):
...     x *= 2
...  

>>> x = 5
>>> double(x)
>>> x
5

However, you can use a return value to obtain a similar effect. Simply write double() so that it takes an integer argument, doubles it, and returns the doubled value. Then, the caller is responsible for the assignment that modifies the original value:

>>>

>>> def double(x):
...     return x * 2
...  

>>> x = 5
>>> x = double(x)
>>> x
10

This is arguably preferable to modifying by side effect. It’s very clear that x is being modified in the calling environment because the caller is doing so itself. Anyway, it’s the only option, because modification by side effect doesn’t work in this case.

Still, even in cases where it’s possible to modify an argument by side effect, using a return value may still be clearer. Suppose you want to double every item in a list. Because lists are mutable, you could define a Python function that modifies the list in place:

>>>

>>> def double_list(x):
...     i = 0
...     while i < len(x):
...             x[i] *= 2
...             i += 1
...  

>>> a = [1, 2, 3, 4, 5]
>>> double_list(a)
>>> a
[2, 4, 6, 8, 10]

Unlike double() in the previous example, double_list() actually works as intended. If the documentation for the function clearly states that the list argument’s contents are changed, then this may be a reasonable implementation.

However, you can also write double_list() to pass the desired list back by return value and allow the caller to make the assignment, similar to how double() was re-written in the previous example:

>>>

>>> def double_list(x):
...     r = []
...     for i in x:
...             r.append(i * 2)
...     return r
...

>>> a = [1, 2, 3, 4, 5]
>>> a = double_list(a)
>>> a
[2, 4, 6, 8, 10]

Either approach works equally well. As is often the case, this is a matter of style, and personal preferences vary. Side effects aren’t necessarily consummate evil, and they have their place, but because virtually anything can be returned from a function, the same thing can usually be accomplished through return values as well.

Variable-Length Argument Lists

In some cases, when you’re defining a function, you may not know beforehand how many arguments you’ll want it to take. Suppose, for example, that you want to write a Python function that computes the average of several values. You could start with something like this:

>>>

>>> def avg(a, b, c):
...     return (a + b + c) / 3
...

All is well if you want to average three values:

>>>

>>> avg(1, 2, 3)
2.0

However, as you’ve already seen, when positional arguments are used, the number of arguments passed must agree with the number of parameters declared. Clearly then, all isn’t well with this implementation of avg() for any number of values other than three:

>>>

>>> avg(1, 2, 3, 4)
Traceback (most recent call last):
  File "<pyshell#34>", line 1, in <module>
    avg(1, 2, 3, 4)
TypeError: avg() takes 3 positional arguments but 4 were given

You could try to define avg() with optional parameters:

>>>

>>> def avg(a, b=0, c=0, d=0, e=0):
...     .
...     .
...     .
...

This allows for a variable number of arguments to be specified. The following calls are at least syntactically correct:

avg(1)
avg(1, 2)
avg(1, 2, 3)
avg(1, 2, 3, 4)
avg(1, 2, 3, 4, 5)

But this approach still suffers from a couple of problems. For starters, it still only allows up to five arguments, not an arbitrary number. Worse yet, there’s no way to distinguish between the arguments that were specified and those that were allowed to default. The function has no way to know how many arguments were actually passed, so it doesn’t know what to divide by:

>>>

>>> def avg(a, b=0, c=0, d=0, e=0):
...     return (a + b + c + d + e) / # Divided by what???
...

Evidently, this won’t do either.

You could write avg() to take a single list argument:

>>>

>>> def avg(a):
...     total = 0
...     for v in a:
...             total += v
...     return total / len(a)
...  

>>> avg([1, 2, 3])
2.0  
>>> avg([1, 2, 3, 4, 5])
3.0

At least this works. It allows an arbitrary number of values and produces a correct result. As an added bonus, it works when the argument is a tuple as well:

>>>

>>> t = (1, 2, 3, 4, 5)
>>> avg(t)
3.0

The drawback is that the added step of having to group the values into a list or tuple is probably not something the user of the function would expect, and it isn’t very elegant. Whenever you find Python code that looks inelegant, there’s probably a better option.

In this case, indeed there is! Python provides a way to pass a function a variable number of arguments with argument tuple packing and unpacking using the asterisk (*) operator.

Argument Tuple Packing

When a parameter name in a Python function definition is preceded by an asterisk (*), it indicates argument tuple packing. Any corresponding arguments in the function call are packed into a tuple that the function can refer to by the given parameter name. Here’s an example:

>>>

>>> def f(*args):
...     print(args)
...     print(type(args), len(args))
...     for x in args:
...             print(x)
...

>>> f(1, 2, 3)
(1, 2, 3)        
<class 'tuple'> 3
1
2
3

>>> f('foo', 'bar', 'baz', 'qux', 'quux')
('foo', 'bar', 'baz', 'qux', 'quux')
<class 'tuple'> 5
foo
bar
baz
qux
quux

In the definition of f(), the parameter specification *args indicates tuple packing. In each call to f(), the arguments are packed into a tuple that the function can refer to by the name args. Any name can be used, but args is so commonly chosen that it’s practically a standard.

Using tuple packing, you can clean up avg() like this:

>>>

>>> def avg(*args):
...     total = 0
...     for i in args:
...         total += i
...     return total / len(args)
...

>>> avg(1, 2, 3)
2.0
>>> avg(1, 2, 3, 4, 5)
3.0

Better still, you can tidy it up even further by replacing the for loop with the built-in Python function sum(), which sums the numeric values in any iterable:

>>>

>>> def avg(*args):
...     return sum(args) / len(args)
...

>>> avg(1, 2, 3)
2.0
>>> avg(1, 2, 3, 4, 5)
3.0

Now, avg() is concisely written and works as intended.

Still, depending on how this code will be used, there may still be work to do. As written, avg() will produce a TypeError exception if any arguments are non-numeric:

>>>

>>> avg(1, 'foo', 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in avg
TypeError: unsupported operand type(s) for +: 'int' and 'str'

To be as robust as possible, you should add code to check that the arguments are of the proper type. Later in this tutorial series, you’ll learn how to catch exceptions like TypeError and handle them appropriately. You can also check out Python Exceptions: An Introduction.

Argument Tuple Unpacking

An analogous operation is available on the other side of the equation in a Python function call. When an argument in a function call is preceded by an asterisk (*), it indicates that the argument is a tuple that should be unpacked and passed to the function as separate values:

>>>

>>> def f(x, y, z):
...     print(f'x = {x}')
...     print(f'y = {y}')
...     print(f'z = {z}')
...

>>> f(1, 2, 3)
x = 1
y = 2
z = 3

>>> t = ('foo', 'bar', 'baz')
>>> f(*t)
x = foo
y = bar
z = baz

In this example, *t in the function call indicates that t is a tuple that should be unpacked. The unpacked values 'foo', 'bar', and 'baz' are assigned to the parameters x, y, and z, respectively.

Although this type of unpacking is called tuple unpacking, it doesn’t only work with tuples. The asterisk (*) operator can be applied to any iterable in a Python function call. For example, a list or set can be unpacked as well:

>>>

>>> a = ['foo', 'bar', 'baz']
>>> type(a)
<class 'list'>
>>> f(*a)
x = foo
y = bar
z = baz

>>> s = {1, 2, 3}
>>> type(s)
<class 'set'>
>>> f(*s)
x = 1
y = 2
z = 3

You can even use tuple packing and unpacking at the same time:

>>>

>>> def f(*args):
...     print(type(args), args)
...

>>> a = ['foo', 'bar', 'baz', 'qux']
>>> f(*a)
<class 'tuple'> ('foo', 'bar', 'baz', 'qux')

Here, f(*a) indicates that list a should be unpacked and the items passed to f() as individual values. The parameter specification *args causes the values to be packed back up into the tuple args.

Argument Dictionary Packing

Python has a similar operator, the double asterisk (**), which can be used with Python function parameters and arguments to specify dictionary packing and unpacking. Preceding a parameter in a Python function definition by a double asterisk (**) indicates that the corresponding arguments, which are expected to be key=value pairs, should be packed into a dictionary:

>>>

>>> def f(**kwargs):
...     print(kwargs)
...     print(type(kwargs))
...     for key, val in kwargs.items():
...             print(key, '->', val)
...

>>> f(foo=1, bar=2, baz=3)
{'foo': 1, 'bar': 2, 'baz': 3}
<class 'dict'>
foo -> 1
bar -> 2
baz -> 3

In this case, the arguments foo=1, bar=2, and baz=3 are packed into a dictionary that the function can reference by the name kwargs. Again, any name can be used, but the peculiar kwargs (which is short for keyword args) is nearly standard. You don’t have to adhere to it, but if you do, then anyone familiar with Python coding conventions will know straightaway what you mean.

Argument Dictionary Unpacking

Argument dictionary unpacking is analogous to argument tuple unpacking. When the double asterisk (**) precedes an argument in a Python function call, it specifies that the argument is a dictionary that should be unpacked, with the resulting items passed to the function as keyword arguments:

>>>

>>> def f(a, b, c):
...     print(F'a = {a}')
...     print(F'b = {b}')
...     print(F'c = {c}')
...

>>> d = {'a': 'foo', 'b': 25, 'c': 'qux'}
>>> f(**d)
a = foo
b = 25
c = qux

The items in the dictionary d are unpacked and passed to f() as keyword arguments. So, f(**d) is equivalent to f(a='foo', b=25, c='qux'):

>>>

>>> f(a='foo', b=25, c='qux')
a = foo
b = 25
c = qux

In fact, check this out:

>>>

>>> f(**dict(a='foo', b=25, c='qux'))
a = foo
b = 25
c = qux

Here, dict(a='foo', b=25, c='qux') creates a dictionary from the specified key/value pairs. Then, the double asterisk operator (**) unpacks it and passes the keywords to f().

Putting It All Together

Think of *args as a variable-length positional argument list, and **kwargs as a variable-length keyword argument list.

Note: For another look at *args and **kwargs, see Python args and kwargs: Demystified.

All three—standard positional parameters, *args, and **kwargs—can be used in one Python function definition. If so, then they should be specified in that order:

>>>

>>> def f(a, b, *args, **kwargs):
...     print(F'a = {a}')
...     print(F'b = {b}')
...     print(F'args = {args}')
...     print(F'kwargs = {kwargs}')
...

>>> f(1, 2, 'foo', 'bar', 'baz', 'qux', x=100, y=200, z=300)
a = 1
b = 2
args = ('foo', 'bar', 'baz', 'qux')
kwargs = {'x': 100, 'y': 200, 'z': 300}

This provides just about as much flexibility as you could ever need in a function interface!

Multiple Unpackings in a Python Function Call

Python version 3.5 introduced support for additional unpacking generalizations, as outlined in PEP 448. One thing these enhancements allow is multiple unpackings in a single Python function call:

>>>

>>> def f(*args):
...     for i in args:
...             print(i)
...

>>> a = [1, 2, 3]
>>> t = (4, 5, 6)
>>> s = {7, 8, 9}

>>> f(*a, *t, *s)
1
2
3
4
5
6
8
9
7

You can specify multiple dictionary unpackings in a Python function call as well:

>>>

>>> def f(**kwargs):
...     for k, v in kwargs.items():
...             print(k, '->', v)
...

>>> d1 = {'a': 1, 'b': 2}
>>> d2 = {'x': 3, 'y': 4}

>>> f(**d1, **d2)
a -> 1
b -> 2
x -> 3
y -> 4

Note: This enhancement is available only in Python version 3.5 or later. If you try this in an earlier version, then you’ll get a SyntaxError exception.

By the way, the unpacking operators * and ** don’t apply only to variables, as in the examples above. You can also use them with literals that are iterable:

>>>

>>> def f(*args):
...     for i in args:
...             print(i)
...

>>> f(*[1, 2, 3], *[4, 5, 6])
1
2
3
4
5
6

>>> def f(**kwargs):
...     for k, v in kwargs.items():
...             print(k, '->', v)
...

>>> f(**{'a': 1, 'b': 2}, **{'x': 3, 'y': 4})
a -> 1
b -> 2
x -> 3
y -> 4

Here, the literal lists [1, 2, 3] and [4, 5, 6] are specified for tuple unpacking, and the literal dictionaries {'a': 1, 'b': 2} and {'x': 3, 'y': 4} are specified for dictionary unpacking.

Keyword-Only Arguments

A Python function in version 3.x can be defined so that it takes keyword-only arguments. These are function arguments that must be specified by keyword. Let’s explore a situation where this might be beneficial.

Suppose you want to write a Python function that takes a variable number of string arguments, concatenates them together separated by a dot ("."), and prints them to the console. Something like this will do to start:

>>>

>>> def concat(*args):
...     print(f'-> {".".join(args)}')
...

>>> concat('a', 'b', 'c')
-> a.b.c
>>> concat('foo', 'bar', 'baz', 'qux')
-> foo.bar.baz.qux

As it stands, the output prefix is hard-coded to the string '-> '. What if you want to modify the function to accept this as an argument as well, so the user can specify something else? This is one possibility:

>>>

>>> def concat(prefix, *args):
...     print(f'{prefix}{".".join(args)}')
...

>>> concat('//', 'a', 'b', 'c')
//a.b.c
>>> concat('... ', 'foo', 'bar', 'baz', 'qux')
... foo.bar.baz.qux

This works as advertised, but there are a couple of undesirable things about this solution:

The prefix string is lumped together with the strings to be concatenated. Just from looking at the function call, it isn’t clear that the first argument is treated differently from the rest. To know that, you’d have to go back and look at the function definition.
prefix isn’t optional. It always has to be included, and there’s no way to assume a default value.

You might think you could overcome the second issue by specifying a parameter with a default value, like this, perhaps:

>>>

>>> def concat(prefix='-> ', *args):
...     print(f'{prefix}{".".join(args)}')
...

Unfortunately, this doesn’t work quite right. prefix is a positional parameter, so the interpreter assumes that the first argument specified in the function call is the intended output prefix. This means there isn’t any way to omit it and obtain the default value:

>>>

>>> concat('a', 'b', 'c')
ab.c

What if you try to specify prefix as a keyword argument? Well, you can’t specify it first:

>>>

>>> concat(prefix='//', 'a', 'b', 'c')
  File "<stdin>", line 1
SyntaxError: positional argument follows keyword argument

As you’ve seen previously, when both types of arguments are given, all positional arguments must come before any keyword arguments.

However, you can’t specify it last either:

>>>

>>> concat('a', 'b', 'c', prefix='... ')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: concat() got multiple values for argument 'prefix'

Again, prefix is a positional parameter, so it’s assigned the first argument specified in the call (which is 'a' in this case). Then, when it’s specified again as a keyword argument at the end, Python thinks it’s been assigned twice.

Keyword-only parameters help solve this dilemma. In the function definition, specify *args to indicate a variable number of positional arguments, and then specify prefix after that:

>>>

>>> def concat(*args, prefix='-> '):
...     print(f'{prefix}{".".join(args)}')
...

In that case, prefix becomes a keyword-only parameter. Its value will never be filled by a positional argument. It can only be specified by a named keyword argument:

>>>

>>> concat('a', 'b', 'c', prefix='... ')
... a.b.c

Note that this is only possible in Python 3. In versions 2.x of Python, specifying additional parameters after the *args variable arguments parameter raises an error.

Keyword-only arguments allow a Python function to take a variable number of arguments, followed by one or more additional options as keyword arguments. If you wanted to modify concat() so that the separator character can optionally be specified as well, then you could add an additional keyword-only argument:

>>>

>>> def concat(*args, prefix='-> ', sep='.'):
...     print(f'{prefix}{sep.join(args)}')
...

>>> concat('a', 'b', 'c')
-> a.b.c
>>> concat('a', 'b', 'c', prefix='//')
//a.b.c
>>> concat('a', 'b', 'c', prefix='//', sep='-')
//a-b-c

If a keyword-only parameter is given a default value in the function definition (as it is in the example above), and the keyword is omitted when the function is called, then the default value is supplied:

>>>

>>> concat('a', 'b', 'c')
-> a.b.c

If, on the other hand, the parameter isn’t given a default value, then it becomes required, and failure to specify it results in an error:

>>>

>>> def concat(*args, prefix):
...     print(f'{prefix}{".".join(args)}')
...

>>> concat('a', 'b', 'c', prefix='... ')
... a.b.c

>>> concat('a', 'b', 'c')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: concat() missing 1 required keyword-only argument: 'prefix'

What if you want to define a Python function that takes a keyword-only argument but doesn’t take a variable number of positional arguments? For example, the following function performs the specified operation on two numerical arguments:

>>>

>>> def oper(x, y, op='+'):
...     if op == '+':
...             return x + y
...     elif op == '-':
...             return x - y
...     elif op == '/':
...             return x / y
...     else:
...             return None
...

>>> oper(3, 4)
7
>>> oper(3, 4, '+')
7
>>> oper(3, 4, '/')
0.75

If you wanted to make op a keyword-only parameter, then you could add an extraneous dummy variable argument parameter and just ignore it:

>>>

>>> def oper(x, y, *ignore, op='+'):
...     if op == '+':
...             return x + y
...     elif op == '-':
...             return x - y
...     elif op == '/':
...             return x / y
...     else:
...             return None
...

>>> oper(3, 4, op='+')
7
>>> oper(3, 4, op='/')
0.75

The problem with this solution is that *ignore absorbs any extraneous positional arguments that might happen to be included:

>>>

>>> oper(3, 4, "I don't belong here")
7
>>> oper(3, 4, "I don't belong here", op='/')
0.75

In this example, the extra argument shouldn’t be there (as the argument itself announces). Instead of quietly succeeding, it should really result in an error. The fact that it doesn’t is untidy at best. At worst, it may cause a result that appears misleading:

>>>

>>> oper(3, 4, '/')
7

To remedy this, version 3 allows a variable argument parameter in a Python function definition to be just a bare asterisk (*), with the name omitted:

>>>

>>> def oper(x, y, *, op='+'):
...     if op == '+':
...             return x + y
...     elif op == '-':
...             return x - y
...     elif op == '/':
...             return x / y
...     else:
...             return None
...

>>> oper(3, 4, op='+')
7
>>> oper(3, 4, op='/')
0.75

>>> oper(3, 4, "I don't belong here")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: oper() takes 2 positional arguments but 3 were given

>>> oper(3, 4, '+')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: oper() takes 2 positional arguments but 3 were given

The bare variable argument parameter * indicates that there aren’t any more positional parameters. This behavior generates appropriate error messages if extra ones are specified. It allows keyword-only parameters to follow.

Positional-Only Arguments

As of Python 3.8, function parameters can also be declared positional-only, meaning the corresponding arguments must be supplied positionally and can’t be specified by keyword.

To designate some parameters as positional-only, you specify a bare slash (/) in the parameter list of a function definition. Any parameters to the left of the slash (/) must be specified positionally. For example, in the following function definition, x and y are positional-only parameters, but z may be specified by keyword:

>>>

>>> # This is Python 3.8
>>> def f(x, y, /, z):
...     print(f'x: {x}')
...     print(f'y: {y}')
...     print(f'z: {z}')
...

This means that the following calls are valid:

>>>

>>> f(1, 2, 3)
x: 1
y: 2
z: 3

>>> f(1, 2, z=3)
x: 1
y: 2
z: 3

The following call to f(), however, is not valid:

>>>

>>> f(x=1, y=2, z=3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() got some positional-only arguments passed as keyword arguments:
'x, y'

The positional-only and keyword-only designators may both be used in the same function definition:

>>>

>>> # This is Python 3.8
>>> def f(x, y, /, z, w, *, a, b):
...     print(x, y, z, w, a, b)
...

>>> f(1, 2, z=3, w=4, a=5, b=6)
1 2 3 4 5 6

>>> f(1, 2, 3, w=4, a=5, b=6)
1 2 3 4 5 6

In this example:

x and y are positional-only.
a and b are keyword-only.
z and w may be specified positionally or by keyword.

For more information on positional-only parameters, see the Python 3.8 release highlights.

Docstrings

When the first statement in the body of a Python function is a string literal, it’s known as the function’s docstring. A docstring is used to supply documentation for a function. It can contain the function’s purpose, what arguments it takes, information about return values, or any other information you think would be useful.

The following is an example of a function definition with a docstring:

>>>

>>> def avg(*args):
...     """Returns the average of a list of numeric values."""
...     return sum(args) / len(args)
...

Technically, docstrings can use any of Python’s quoting mechanisms, but the recommended convention is to triple-quote using double-quote characters ("""), as shown above. If the docstring fits on one line, then the closing quotes should be on the same line as the opening quotes.

Multi-line docstrings are used for lengthier documentation. A multi-line docstring should consist of a summary line, followed by a blank line, followed by a more detailed description. The closing quotes should be on a line by themselves:

>>>

>>> def foo(bar=0, baz=1):
...     """Perform a foo transformation.
...
...     Keyword arguments:
...     bar -- magnitude along the bar axis (default=0)
...     baz -- magnitude along the baz axis (default=1)
...     """
...     <function_body>
...

Docstring formatting and semantic conventions are detailed in PEP 257.

When a docstring is defined, the Python interpreter assigns it to a special attribute of the function called __doc__. This attribute is one of a set of specialized identifiers in Python that are sometimes called magic attributes or magic methods because they provide special language functionality.

Note: These attributes are also referred to by the colorful nickname dunder attributes and dunder methods. The word dunder combines the d from double and under from the underscore character (_). You’ll encounter many more dunder attributes and methods in future tutorials in this series.

You can access a function’s docstring with the expression <function_name>.__doc__. The docstrings for the above examples can be displayed as follows:

>>>

>>> print(avg.__doc__)
Returns the average of a list of numeric values.

>>> print(foo.__doc__)
Perform a foo transformation.

    Keyword arguments:
    bar -- magnitude along the bar axis (default=0)
    baz -- magnitude along the baz axis (default=1)

In the interactive Python interpreter, you can type help(<function_name>) to display the docstring for <function_name>:

>>>

>>> help(avg)
Help on function avg in module __main__:

avg(*args)
    Returns the average of a list of numeric values.

>>> help(foo)
Help on function foo in module __main__:

foo(bar=0, baz=1)
    Perform a foo transformation.

    Keyword arguments:
    bar -- magnitude along the bar axis (default=0)
    baz -- magnitude along the baz axis (default=1)

It’s considered good coding practice to specify a docstring for each Python function you define. For more on docstrings, check out Documenting Python Code: A Complete Guide.

Python Function Annotations

As of version 3.0, Python provides an additional feature for documenting a function called a function annotation. Annotations provide a way to attach metadata to a function’s parameters and return value.

To add an annotation to a Python function parameter, insert a colon (:) followed by any expression after the parameter name in the function definition. To add an annotation to the return value, add the characters -> and any expression between the closing parenthesis of the parameter list and the colon that terminates the function header. Here’s an example:

>>>

>>> def f(a: '<a>', b: '<b>') -> '<ret_value>':
...     pass
...

The annotation for parameter a is the string '<a>', for b the string '<b>', and for the function return value the string '<ret_value>'.

The Python interpreter creates a dictionary from the annotations and assigns them to another special dunder attribute of the function called __annotations__. The annotations for the Python function f() shown above can be displayed as follows:

>>>

>>> f.__annotations__
{'a': '<a>', 'b': '<b>', 'return': '<ret_value>'}

The keys for the parameters are the parameter names. The key for the return value is the string 'return':

>>>

>>> f.__annotations__['a']
'<a>'
>>> f.__annotations__['b']
'<b>'
>>> f.__annotations__['return']
'<ret_value>'

Note that annotations aren’t restricted to string values. They can be any expression or object. For example, you might annotate with type objects:

>>>

>>> def f(a: int, b: str) -> float:
...     print(a, b)
...     return(3.5)
...

>>> f(1, 'foo')
1 foo
3.5

>>> f.__annotations__
{'a': <class 'int'>, 'b': <class 'str'>, 'return': <class 'float'>}

An annotation can even be a composite object like a list or a dictionary, so it’s possible to attach multiple items of metadata to the parameters and return value:

>>>

>>> def area(
...     r: {
...            'desc': 'radius of circle',
...            'type': float
...        }) -> \
...        {
...            'desc': 'area of circle',
...            'type': float
...        }:
...     return 3.14159 * (r ** 2)
...

>>> area(2.5)
19.6349375

>>> area.__annotations__
{'r': {'desc': 'radius of circle', 'type': <class 'float'>},
'return': {'desc': 'area of circle', 'type': <class 'float'>}}

>>> area.__annotations__['r']['desc']
'radius of circle'
>>> area.__annotations__['return']['type']
<class 'float'>

In the example above, an annotation is attached to the parameter r and to the return value. Each annotation is a dictionary containing a string description and a type object.

If you want to assign a default value to a parameter that has an annotation, then the default value goes after the annotation:

>>>

>>> def f(a: int = 12, b: str = 'baz') -> float:
...     print(a, b)
...     return(3.5)
...

>>> f.__annotations__
{'a': <class 'int'>, 'b': <class 'str'>, 'return': <class 'float'>}

>>> f()
12 baz
3.5

What do annotations do? Frankly, they don’t do much of anything. They’re just kind of there. Let’s look at one of the examples from above again, but with a few minor modifications:

>>>

>>> def f(a: int, b: str) -> float:
...     print(a, b)
...     return 1, 2, 3
...

>>> f('foo', 2.5)
foo 2.5
(1, 2, 3)

What’s going on here? The annotations for f() indicate that the first argument is int, the second argument str, and the return value float. But the subsequent call to f() breaks all the rules! The arguments are str and float, respectively, and the return value is a tuple. Yet the interpreter lets it all slide with no complaint at all.

Annotations don’t impose any semantic restrictions on the code whatsoever. They’re simply bits of metadata attached to the Python function parameters and return value. Python dutifully stashes them in a dictionary, assigns the dictionary to the function’s __annotations__ dunder attribute, and that’s it. Annotations are completely optional and don’t have any impact on Python function execution at all.

To quote Amahl in Amahl and the Night Visitors, “What’s the use of having it then?”

For starters, annotations make good documentation. You can specify the same information in the docstring, of course, but placing it directly in the function definition adds clarity. The types of the arguments and the return value are obvious on sight for a function header like this:

def f(a: int, b: str) -> float:

Granted, the interpreter doesn’t enforce adherence to the types specified, but at least they’re clear to someone reading the function definition.

Deep Dive: Enforcing Type-Checking

If you were inclined to, you could add code to enforce the types specified in the function annotations. Here’s a function that checks the actual type of each argument against what’s specified in the annotation for the corresponding parameter. It displays True if they match ans False if they don’t:
>>>
>>> def f(a: int, b: str, c: float):
...     import inspect
...     args = inspect.getfullargspec(f).args
...     annotations = inspect.getfullargspec(f).annotations
...     for x in args:
...         print(x, '->',
...               'arg is', type(locals()[x]), ',',
...               'annotation is', annotations[x],
...               '/', (type(locals()[x])) is annotations[x])
...

>>> f(1, 'foo', 3.3)
a -> arg is <class 'int'> , annotation is <class 'int'> / True
b -> arg is <class 'str'> , annotation is <class 'str'> / True
c -> arg is <class 'float'> , annotation is <class 'float'> / True

>>> f('foo', 4.3, 9)
a -> arg is <class 'str'> , annotation is <class 'int'> / False
b -> arg is <class 'float'> , annotation is <class 'str'> / False
c -> arg is <class 'int'> , annotation is <class 'float'> / False

>>> f(1, 'foo', 'bar')
a -> arg is <class 'int'> , annotation is <class 'int'> / True
b -> arg is <class 'str'> , annotation is <class 'str'> / True
c -> arg is <class 'str'> , annotation is <class 'float'> / False
(The inspect module contains functions that obtain useful information about live objects—in this case, function f().)

A function defined like the one above could, if desired, take some sort of corrective action when it detects that the passed arguments don’t conform to the types specified in the annotations.

In fact, a scheme for using annotations to perform static type checking in Python is described in PEP 484. A free static type checker for Python called mypy is available, which is built on the PEP 484 specification.

There’s another benefit to using annotations as well. The standardized format in which annotation information is stored in the __annotations__ attribute lends itself to the parsing of function signatures by automated tools.

When it comes down to it, annotations aren’t anything especially magical. You could even define your own without the special syntax that Python provides. Here’s a Python function definition with type object annotations attached to the parameters and return value:

>>>

>>> def f(a: int, b: str) -> float:
...     return
...

>>> f.__annotations__
{'a': <class 'int'>, 'b': <class 'str'>, 'return': <class 'float'>}

The following is essentially the same function, with the __annotations__ dictionary constructed manually:

>>>

>>> def f(a, b):
...     return
...

>>> f.__annotations__ = {'a': int, 'b': str, 'return': float}

>>> f.__annotations__
{'a': <class 'int'>, 'b': <class 'str'>, 'return': <class 'float'>}

The effect is identical in both cases, but the first is more visually appealing and readable at first glance.

In fact, the __annotations__ attribute isn’t significantly different from most other attributes of a function. For example, it can be modified dynamically. You could choose to use the return value attribute to count how many times a function is executed:

>>>

>>> def f() -> 0:
...     f.__annotations__['return'] += 1
...     print(f"f() has been executed {f.__annotations__['return']} time(s)")
...

>>> f()
f() has been executed 1 time(s)
>>> f()
f() has been executed 2 time(s)
>>> f()
f() has been executed 3 time(s)

Python function annotations are nothing more than dictionaries of metadata. It just happens that you can create them with convenient syntax that’s supported by the interpreter. They’re whatever you choose to make of them.

Conclusion

As applications grow larger, it becomes increasingly important to modularize code by breaking it up into smaller functions of manageable size. You now hopefully have all the tools you need to do this.

You’ve learned:

How to create a user-defined function in Python
Several different ways you can pass arguments to a function
How you can return data from a function to its caller
How to add documentation to functions with docstrings and annotations

Next up in this series are two tutorials that cover searching and pattern matching. You will get an in-depth look at a Python module called re, which contains functionality for searching and matching using a versatile pattern syntax called a regular expression.

« Python String Formatting Techniques

Functions in Python

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #0: About the Show

2020-03-06T00:00:00+00:00

A weekly Python podcast hosted by Christopher Bailey with interviews, coding tips, and conversation with guests from the Python community. The show covers a wide range of topics including Python programming best practices, career tips, and related software development topics. Join us to hear what's new in the world of Python programming and become a better coder.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Alexa Python Development: Build and Deploy an Alexa Skill

2020-03-04T14:00:00+00:00

Smart home speakers were a novel idea just a couple of years ago. Today, they’ve become a central part of many people’s homes and offices and their adoption is only expected to grow. Among the most popular of these devices are those controlled by Amazon Alexa. In this tutorial, you’ll become an Alexa Python developer by deploying your own Alexa skill, an application that users will interact with using voice commands to Amazon Alexa devices.

In this tutorial, you’ll learn:

What the main components of an Alexa skill are
How to set up an Alexa skill and create Intents
What the ask_sdk_core Alexa Python package is
How to use ask_sdk_core to create the business logic of your Alexa Python skill
How to build, deploy, and test your Alexa Python skill using the online developer console

Free Bonus: Click here to download a Python speech recognition sample project with full source code that you can use as a basis for your own speech recognition apps.

Getting Started With Alexa Python Development

To follow this tutorial, you’ll need to make a free Alexa developer account. On that page, you’ll take the following steps:

Click the Get Started button.
Click the Sign-Up button on the subsequent page.
Click Create your Amazon Account.
Fill out the form with the required details.
Click Submit to complete the sign-up process.

You’ll also need to be familiar with concepts such as lists and dictionaries in Python, as well as JavaScript Object Notation (JSON). If you’re new to JSON, then check out Working With JSON Data in Python.

Let’s get started!

Understanding Alexa Skills

An Alexa Python developer must be familiar with a number of different Alexa skill components, but the two most important components are the interface and the service:

The skill interface processes the user’s speech inputs and maps it to an intent.
The skill service contains all the business logic that determines the response for a given user input and returns it as a JSON object.

The skill interface will be the frontend of your Alexa skill. This is where you’ll define the intents and the invocation phrases that will perform a certain function. Essentially, this is the part of the skill that’s responsible for interacting with the users.

The skill service will be the backend of your Alexa skill. When a specific intent is triggered by the user, it will send that information as a request to the skill service. This will contain the business logic to be returned along with valuable information to the frontend, which will finally be relayed back to the user.

Setting Up Your Environment

It’s time to start building your first Alexa Python skill! Sign in to the Alexa developer console and click on the Create Skill button to get started. On the next page, enter the Skill name, which will be Joke Bot:

This will be the invocation phrase of your skill. It’s the phrase a user will speak to start using your Alexa skill. You can change this to something else later on if you’d like. Also, note that Alexa skills can interact in many languages, which you can see from the Default Language dropdown menu. For now, just set it to English (US).

Next, you’ll need to choose a model to add to your skill. These models are like templates that have been pre-designed by the Amazon team to help you get started with Alexa Python development, based on some common use cases. For this tutorial, you should select the Custom model.

Finally, you need to select a method to host the backend of your Alexa skill. This service will contain the business logic of your application.

Note: If you select the Provision your own option, then you’ll have to host your own backend for your Alexa Python projects. This can be an API built and hosted on a platform of your choice. The other option is to create a separate AWS Lambda function and link it to your Alexa skill. You can learn more about AWS Lambda pricing on their pricing page.

For now, select Alexa-Hosted (Python) as the backend for your Alexa skill. This will automatically provide you with a hosted backend within the AWS free tier so you don’t have to pay anything upfront or set up a complicated backend right now.

Finally, click the Create Skill button to proceed. You might be asked to fill out a CAPTCHA here, so complete that as well. After a minute or so, you should be redirected to the Build section of the developer console.

Understanding the Alexa Skill Model

Once you’ve logged into the Alexa developer console and selected or created a skill, you’ll be greeted with the Build section. This section provides you with a lot of options and controls to set up the interaction model of the skill. The components of this interaction model allow you to define how the users will interact with your skill. These properties can be accessed through the left-side panel, which looks something like this:

As an Alexa Python developer, there are a few components of an Alexa skill interaction model that you’ll need to know about. The first is the invocation. This is what users will say to begin interacting with your Alexa skill. For example, the user will say, “Joke Bot,” to invoke the Alexa skill you’ll build in this tutorial. You can change this from the Invocation section at any time.

Another component is the intent, which represents the core functionality of your application. Your app will have a set of intents that will represent what kinds of actions your skill can perform. To provide contextual information for a given intent, you’ll use a slot, which is a variable in an utterance phrase.

Consider the following example. A sample utterance to invoke the weather intent could be, “Tell me about the weather.” To make the skill more useful, you can set the intent to be, “Tell me about the weather in Chicago,” where the word “Chicago” will be passed as a slot variable, which improves the user experience.

Lastly, there are slot types, which define how data in a slot is handled and recognized. For example, the AMAZON.DATE slot type easily converts words that indicate a date—like “today, “tomorrow”, and others—into a standard date format (such as “2019-07-05”). You can check out the official slot type reference page to learn more.

Note: To learn more about the Alexa skill interaction model, check out the official documentation.

At this point, the Intents panel should be open. If it’s not, then you can open it by selecting Intents from the sidebar on the left. You’ll notice that there are five intents already set up by default:

The Intents panel includes a HelloWorldIntent and five Built-in Intents. The built-in intents are there to remind you to account for some common cases that are important to making a user-friendly bot. Here’s a brief overview:

AMAZON.CancelIntent lets the user cancel a transaction or task. Examples include, “Never mind,” “Forget it,” “Exit,” and “Cancel,” though there are others.
AMAZON.HelpIntent provides help about how to use the skill. This could be used to return a sentence that serves as a manual for the user on how to interact with your skill.
AMAZON.StopIntent allows the user to exit the skill.
AMAZON.NavigateHomeIntent navigates the user to the device home screen (if a screen is being used) and ends the skill session.

By default, there are no sample utterances assigned to trigger these intents, so you’ll have to add those as well. Consider it part of your training as an Alexa Python developer. You can learn more about these built-in intents in the official documentation.

Viewing a Sample Intent

Later in this tutorial, you’ll learn how to make a new intent, but for now, it’s a good idea to take a look at some existing intents that are part of every new skill you create. To start, click the HelloWorldIntent to see its properties:

You can see the sample utterances that a user can speak to invoke this intent. When this intent is invoked, this information is sent to the backend service of your Alexa skill, which will then execute the required business logic and return a response.

Below this, you have the option to set up the Dialog Delegation Strategy, which allows you to delegate a specific dialog that you define to a particular intent. While you won’t cover this in this tutorial, you can read more about it in the official documentation.

Next, you have the option to define slots for some particular data that your intent is supposed to collect. For example, if you were to create an intent that tells the weather for a given day, then you’d have a Date slot here that would collect the date information and send it to your backend service.

Note: In addition, the Intent Confirmation option can be useful in a case when you’re collecting a number of different data points from your user in a single intent and you want to prompt the user before sending it on for further processing.

Whenever you make changes to an intent, you need to click the Save Model button to save it. Then, you can click the Build Model button to go ahead and test your Alexa Python skill.

It’s helpful to know that the interaction model of a skill can be completely represented in a JSON format. To see the current structure of your Alexa skill, click the JSON Editor option from the left side panel of the console:

If you make a change directly using the JSON editor, then the changes are also reflected in the developer console UI. To test this behavior, add a new intent and click Save Model.

Once you’ve made all the necessary changes to the interaction model of your skill, you can open the Test section of the developer console to test out your skill. Testing is an important part of becoming an Alexa Python developer, so be sure not to skip this step! Click the Test button from the top navigation bar on the developer console. By default, testing will be disabled. From the drop-down menu, select Development to start testing:

Here, you have a number of ways that you can test out your Alexa Python skill. Let’s do a quick test so that you can get an idea of how your Alexa skill will respond to an utterance.

Select the Alexa Simulator option from the left side panel, then enter the phrase, “Hey Alexa, open Joke Bot.” You can do this either by typing it in the input box or by using the Mic option. After a couple of seconds, a response will be returned back to you:

In addition to the voice response, you can also see the JSON Input that was sent to the backend service of your Alexa skill, as well as the JSON Output that was received back to the console:

Here’s what’s happened so far:

The JSON input object was constructed from input data that the user entered through voice or text.
The Alexa simulator packaged up the input along with other relevant metadata and sent it to the backend service. You can see this in the JSON Input box.
The backend service received the input JSON object and parsed it to check the type of the request. Then, it passed the JSON to the relevant intent handler function.
The intent handler function processed the input and gathered the required response, which is sent back as a JSON response to the Alexa simulator. You can see this in the JSON Output box.
The Alexa simulator parsed this JSON and read the speech response back to you.

Note: You can read about the JSON request-response mechanism for Alexa skills in the official docs.

Now that you have an overview of the different components of an Alexa skill and how information flows from one part to the other, it’s time to start building your Joke Bot! In the next section, you’ll put your Alexa Python developer skills to the test by creating a new intent.

Creating New Intents

Let’s start by creating the JokeIntent, which will return a random joke from a list to the user. Open the Build section of your Alexa developer console. Then, click the Add button next to the Intents option from the left side panel:

With the Create custom intent option selected, set the name to JokeIntent and then click the Create custom intent button:

Next, you need to add sample utterances that the user will speak to invoke this intent. These can be phrases like “Tell me a joke” or “I want to hear a joke.” Type in a phrase and click the plus sign (+) to add it as a sample utterance. Here’s what that should look like:

You can add more sample utterances, but for now, these will do just fine. Lastly, click the Save Model button in the top-left corner of the window to save these changes.

Remember, you’ll need to build your model before you can test it out. Click the Build Model button to re-build the interaction model of your Alexa Python skill. You’ll see a progress notification on the bottom-right of your browser window. Once the build process is successful, you should see another pop-up notification indicating the status of the build process.

You can check to see if the JokeIntent is successfully triggered or not. Click the Evaluate Model button in the top-right corner of the developer console. A small window will pop in from the side allowing you to check what intent will be triggered by a given input utterance. Type in any of the sample utterances to make sure that the JokeIntent is being invoked successfully.

To get rid of the evaluate pop-up window, click the Evaluate Model button again.

Note: A key thing to remember here is that the model is very flexible in terms of the keywords that are part of the sample utterance phrases. For example, take the phrase, “Is this some kind of a joke?” Even this phrase will trigger the JokeIntent. As an Alexa Python developer, it’s important to select utterances that have a low probability of executing other intents in your skill.

Now that you’ve successfully created an intent, it’s time to write the Python code that will handle this intent and return back a joke as a response.

Building the Skill Backend

Now that you have an intent created that can be triggered by the user, you need to add functionality in the skill backend to handle this intent and return useful information. Open the Code section of the Alexa developer console to get started.

Note: Since you selected the Alexa-Hosted Python option during the setup process, you’re provided with a complete online code editor where you can write, test, build, and deploy the backend of your Alexa skill, all within the developer console.

When you open the Code section of the developer console, you can see an online code editor with some files already set up for you to get started. In particular, you’ll see the following three files in the lambda sub-directory:

lambda_function.py: This is the main entry point of the backend service. All the request data from the Alexa intent is received here and is supposed to be returned from this file only.
requirements.txt: This file contains the list of Python packages that are being used in this project. This is especially useful if you’re choosing to set up your own backend service instead of using what’s provided by Amazon. To learn more about requirements files, check out Using Requirements Files.
utils.py: This file contains some utility functions required for the lambda function to interact with the Amazon S3 service. It contains some sample code on how to fetch data from an Amazon S3 bucket, which you might find useful later on. Right now, this file is not being used in lambda_function.py.

For now, you’ll only be making changes in lambda_function.py, so let’s take a closer look at the structure of the file:

 7 import logging
 8 import ask_sdk_core.utils as ask_utils
 9 
10 from ask_sdk_core.skill_builder import SkillBuilder
11 from ask_sdk_core.dispatch_components import AbstractRequestHandler
12 from ask_sdk_core.dispatch_components import AbstractExceptionHandler
13 from ask_sdk_core.handler_input import HandlerInput
14 
15 from ask_sdk_model import Response
16 
17 logger = logging.getLogger(__name__)
18 logger.setLevel(logging.INFO)
19 
20 
21 class LaunchRequestHandler(AbstractRequestHandler):
22     """Handler for Skill Launch."""
23     def can_handle(self, handler_input):
24         # type: (HandlerInput) -> bool
25 
26         return ask_utils.is_request_type("LaunchRequest")(handler_input)
27 
28     def handle(self, handler_input):
29         # type: (HandlerInput) -> Response
30         speak_output = "Welcome, you can say Hello or Help. " \
31                        "Which would you like to try?"
32 
33         return (
34             handler_input.response_builder
35                 .speak(speak_output)
36                 .ask(speak_output)
37                 .response
38         )
39 ...

First, you import the necessary utilities that were provided in the ask_sdk_core Alexa Python package. Then, there are three main tasks you need to perform in lambda_function.py to handle a request from an intent received from the front-end of the Alexa skill:

Create an intent handler class, which inherits from the AbstractRequestHandler class, with functions can_handle() and handle(). There are already a couple of handler classes defined in lambda_function.py, such as LaunchRequestHandler, HelpIntentHandler, and so on. These handle the fundamental intents of an Alexa skill. An important point to note here is that you need to create a new intent handler class for each of the intents you define.
Create a SkillBuilder object, which acts as the entry point for your Alexa Python skill. This routes all the incoming request and response payloads to the intent handlers that you define.
Pass the intent handler class as an argument to .add_request_handler() so that they’re called in order whenever a new request is received. The SkillBuilder is a singleton, so only one instance of it is needed to handle the routing of all incoming requests.

This is a good time for you to go through lambda_function.py. You’ll notice that the same pattern is followed over and over again to handle different intents that can be triggered by your Alexa Python skill.

Now that you have a broad overview of all the different things you need to do to handle an intent in your backend, it’s time to write the code that will handle the JokeIntent that you built in the previous section.

Creating the JokeIntent Handler

Since the important utilities from the ask_sdk_core Alexa Python package have already been imported, you don’t need to import them again. If you want to learn more about these in-depth, then you can check out the official documentation.

Next, you’ll create a new intent handler that will handle the request received from the JokeIntent. In the code snippet below, the intent handler will simply return with a sample phrase. This indicates that the response to the JokeIntent was received from the backend. Add the following code to lambda_function.py above the class definition of LaunchRequestHandler():

20 class JokeIntentHandler(AbstractRequestHandler):
21     def can_handle(self, handler_input):
22         return ask_utils.is_intent_name("JokeIntent")(handler_input)
23 
24     def handle(self, handler_input):
25         speak_output = "Here's a sample joke for you."
26 
27         return (
28             handler_input.response_builder
29                 .speak(speak_output)
30                 .ask(speak_output)
31                 .response
32         )

Let’s take a look at what each section does. In line 20 you create a new intent handler class for the JokeIntent, which is a child class of the AbstractRequestHandler class. When you create an intent in the frontend, you need to create an intent handler class in the backend that can handle requests from Alexa. The code you write for this needs to do two things:

JokeIntentHandler.can_handle() recognizes each incoming request that Alexa sends.
JokeIntentHandler.handle() returns an appropriate response.

In line 21 you define .can_handle(). It takes in handler_input as a parameter, which is an object of type dict() that contains all the input request information. Then, it uses ask_utils.is_intent_name() or ask_utils.is_request_type() to check whether the JSON input it received can be handled by this intent handler function or not.

You use .is_intent_name() and pass in the name of the intent. This returns a predicate, which is a function object that returns True if the given handler_input originates from the indicated intent. If this is true, then the SkillBuilder object will call JokeIntentHandler.handle().

Note: If the JokeIntent is triggered from the Alexa skill frontend, then it will send a JSON object containing a key type in the body of request that indicates that the intent named JokeIntent was received as input.

This statement subsequently calls .handle(), which you define in line 24. This method receives the input request along with any other important information that might be needed. It contains the business logic that’s required to successfully handle a particular intent. In the case of the JokeIntent, this method is required to send a response containing a joke back to the Alexa frontend.

The speak_ouput variable contains the sentence which will be spoken back to the user by the Alexa skill frontend. speak(speak_output) indicates what the Alexa frontend will play to the user as speech. ask("Question to ask...") can be used to ask a follow-up question. In this method, an object of class response_builder returns the response back to the Alexa skill.

Note: A default response message (Sorry, I had trouble doing what you asked. Please try again.) will be sent back if .handle() does not exist.

Notice that the value of speak_output is set to a fixed response right now. You’ll change this later on to return a random joke from a list of jokes.

Here’s what your code looks like in an editor:

Once you’ve created an intent handler class, you need to pass it as an argument to SkillBuilder.add_request_handler. Scroll to the bottom of lambda_function.py and add the following line:

sb.add_request_handler(JokeIntentHandler())

An important thing to note here is that the placement of this line is important, as the code is processed from top to bottom. So, make sure that the call for your custom intent handler is above the call for the InstantReflectHandler() class. This is how it should look:

171 sb = SkillBuilder()
172 
173 sb.add_request_handler(LaunchRequestHandler())
174 sb.add_request_handler(JokeIntentHandler())
175 sb.add_request_handler(HelloWorldIntentHandler())
176 sb.add_request_handler(HelpIntentHandler())
177 sb.add_request_handler(CancelOrStopIntentHandler())
178 sb.add_request_handler(SessionEndedRequestHandler())
179 
180 # Make sure IntentReflectorHandler is last so it
181 # Doesn't override your custom intent handlers
182 sb.add_request_handler(IntentReflectorHandler())
183 
184 sb.add_exception_handler(CatchAllExceptionHandler())
185 
186 ...

Alright, it’s time to test your code! Click the Deploy button to save the changes and deploy the backend service. You’ll be checking whether it’s going to work as expected from the Alexa skill frontend.

Once the Deploy process is successful, head back to the Test section of the developer console and invoke the JokeIntent. Remember, enter the utterance phrase to invoke your Alexa Python skill, then input a phrase to execute an intent:

If you get a response similar to the one in the image above, then it means you’ve successfully created an intent handler for the JokeIntent in your skill’s backend service. Congratulations! Now, all that’s left to do is to return a random joke from a list back to the skill frontend.

Adding Jokes

Open the Code section of the developer console. Then, add the jokes variable in lambda_function.py:

15 from ask_sdk_model import Response
16 
17 logger = logging.getLogger(__name__)
18 logger.setLevel(logging.INFO)
19 
20 jokes = [
21         "Did you hear about the semi-colon that broke the law? He was given two consecutive sentences.",
22         "I ate a clock yesterday, it was very time-consuming.",
23         "I've just written a song about tortillas; actually, it's more of a rap.",
24         "I woke up this morning and forgot which side the sun rises from, then it dawned on me.",
25         "I recently decided to sell my vacuum cleaner as all it was doing was gathering dust.",
26         "If you shouldn't eat at night, why do they put a light in the fridge?",
27         ]
28 
29 class JokeIntentHandler(AbstractRequestHandler):
30 ...

Here, jokes is a variable of type list containing some one-liner jokes. Make sure to add this outside of a function or class definition so that it has global scope.

Note: Since this list will only be referenced by the JokeIntentHandler() class, it doesn’t really matter if you declare this in the body of a function or not. However, doing it this way does help the function body to be free of clutter.

Next, you’ll add the functionality that .handle() needs to randomly pick one joke from the list of jokes and return it to the user. Modify the body of JokeIntentHandler.handle() with the following code:

29 class JokeIntentHandler(AbstractRequestHandler):
30     def can_handle(self, handler_input):
31         return ask_utils.is_intent_name("JokeIntent")(handler_input)
32 
33     def handle(self, handler_input):
34         speak_output = random.choice(jokes)
35 
36         return (
37             handler_input.response_builder
38                 .speak(speak_output)
39                 .ask(speak_output)
40                 .response
41         )

In the body of .handle(), you select a random joke from the list jokes using random.choice() and return it back as a response to the Alexa skill frontend.

Finally, import the random package by adding an import statement to the top of lambda_function.py:

15 from ask_sdk_model import Response
16 
17 import random
18 
19 logger = logging.getLogger(__name__)
20 logger.setLevel(logging.INFO)
21 
22 ...

This is how the editor should look at this point:

There’s one final change to make before testing. You need to allow Alexa to give an acknowledgment that the skill has been triggered. To do this, look inside LaunchRequestHandler.handle() for the speak_output variable and set its value to the text in the highlighted line below:

45 class LaunchRequestHandler(AbstractRequestHandler):
46     """Handler for Skill Launch."""
47     def can_handle(self, handler_input):
48         # type: (HandlerInput) -> bool
49 
50         return ask_utils.is_request_type("LaunchRequest")(handler_input)
51 
52     def handle(self, handler_input):
53         # type: (HandlerInput) -> Response
54         speak_output = "Hey there! I am a Joke Bot. You can ask me to tell you a random Joke that might just make your day better!"
55 
56         return (
57             handler_input.response_builder
58                 .speak(speak_output)
59                 .ask(speak_output)
60                 .response
61         )
62 ...

Your Joke Bot is ready for final testing! Click the Deploy button to save the changes and head back to the Test section of the developer console. This time, you should see a new greeting message when your skill is first invoked. Then, when you ask the bot to tell you a joke, it should give you a different joke every time:

That’s it! You’ve successfully created your first skill as an Alexa Python developer!

Conclusion

Congratulations on taking your first steps into Alexa Python development! You’ve now successfully built your very own Alexa Python skill. You now know how to create a new skill, create intents, write Python code to handle those intents, and return valuable information back to the user.

Level-up your skills by trying some of the following:

Increase the list of jokes in the backend.
Create a new Intent named Trivia which will respond with a fun trivia fact.
Publish your skill to the Amazon Marketplace.

The possibilities are endless, so go ahead and dive right in! To learn more about Alexa Python development, check out the official docs. You can also check out How to Make a Twitter Bot in Python With Tweepy and How to Make a Discord Bot in Python to learn more about how you can make bots for different platforms using Python.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Implement a Python Stack

2020-03-03T14:00:00+00:00

Have you heard of stacks and wondered what they are? Do you have a general idea but are wondering how to implement a Python stack? You’ve come to the right place!

In this course, you’ll learn:

How to recognize when a stack is a good choice for a data structure
How to decide which implementation is best for your program
What extra considerations to make about stacks in a threading or multiprocessing environment

This course is for Pythonistas who are comfortable running scripts, know what a list is and how to use it, and are wondering how to implement Python stacks.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Work With a PDF in Python

2020-02-25T14:00:00+00:00

The Portable Document Format or PDF is a file format that can be used to present and exchange documents reliably across operating systems. While the PDF was originally invented by Adobe, it is now an open standard that is maintained by the International Organization for Standardization (ISO). You can work with a preexisting PDF in Python by using the PyPDF2 package.

PyPDF2 is a pure-Python package that you can use for many different types of PDF operations.

By the end of this course, you’ll know how to:

Extract document information from a PDF in Python
Rotate pages
Merge PDFs
Split PDFs
Add watermarks
Encrypt a PDF

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Finding the Perfect Python Code Editor

2020-02-18T14:00:00+00:00

Find your perfect Python development setup with this review of Python IDEs and code editors. Writing Python using IDLE or the Python REPL is great for simple things, but not ideal for larger programming projects. With this course you’ll get an overview of the most common Python coding environments to help you make an informed decision.

By the end of this course, you’ll know how to:

Choose the Python editing environment that’s right for you
Perform common tasks like creating, running, and debugging code
Dig deeper into optimizing your favorite editing setup

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Playing and Recording Sound in Python

2020-02-11T14:00:00+00:00

In this course, you’ll learn how to play and record sound in Python using some of the most popular audio libraries. You’ll learn about the most straightforward methods for playing and recording sound first, and then you’ll learn about some libraries that offer some more functionality in exchange for a few extra lines of code.

By the end of this course, you’ll know how to:

Play MP3 and WAV files, as well as a range of other audio formats
Play NumPy and Python arrays containing sound
Record sound using Python
Save your recordings or audio files in a range of different file formats

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Sets in Python

2020-02-04T14:00:00+00:00

In this course, you’ll learn about sets. They’re a useful data structure that allows you to do some complex operations more easily. They come up everywhere in the real world and are important to understand.

By the end of this course, you’ll know:

What a set is
How to define a set in Python
How to operate on a set
How to modify a set
When to use sets
Why sets are a good choice for checking membership

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Modules and Packages: An Introduction

2020-01-28T14:00:00+00:00

In this course, you’ll learn about Python modules and Python packages, two mechanisms that facilitate modular programming.

Modular programming is the process of breaking a large, unwieldy programming task into separate, smaller, more manageable subtasks or modules. Individual modules can then be put together like building blocks to create a larger application.

Learn how to write and import modules so you can optimize the structure of your own programs and make them easier to maintain and grow.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

Linked Lists in Python: An Introduction

Understanding Linked Lists

Main Concepts

Practical Applications

Queues or Stacks

Graphs

Performance Comparison: Lists vs Linked Lists

Insertion and Deletion of Elements

Retrieval of Elements

Introducing collections.deque

How to Use collections.deque

How to Implement Queues and Stacks

Queues

Stacks

Implementing Your Own Linked List

How to Create a Linked List

How to Traverse a Linked List

How to Insert a New Node

Inserting at the Beginning

Inserting at the End

Inserting Between Two Nodes

How to Remove a Node

Using Advanced Linked Lists

How to Use Doubly Linked Lists

How to Use Circular Linked Lists

Conclusion

Comparing Python Objects the Right Way: "is" vs "=="

How to Use any() in Python

How to Use any() in Python

How to Distinguish Between or and any()

Syntax

Return Value

Conclusion

The Real Python Podcast – Episode #2: Learn Python Skills While Creating Games

SimPy: Simulating Real-World Processes With Python

What Simulation Is

How Simulation Works

How to Get Started With simpy

How to Simulate With the simpy Package

Brainstorming a Simulation Algorithm

Setting Up the Environment

Creating the Environment: Class Definition

Moving Through the Environment: Function Definition

Making Things Happen: Function Definition

Calculating the Wait Time: Function Definition

Choosing Parameters: User Input Function Definition

Finalizing the Setup: Main Function Definition

How to Run the Simulation

When to Change Things Up

Conclusion

Using NumPy's np.arange() Effectively

The Python math Module: Everything You Need to Know

Getting to Know the Python math Module

Constants of the math Module

Pi

Tau

Euler’s Number

Infinity

Not a Number (NaN)

Arithmetic Functions

Find Factorials With Python factorial()

Find the Ceiling Value With ceil()

Find the Floor Value With floor()

Truncate Numbers With trunc()

Find the Closeness of Numbers With Python isclose()

Power Functions

Calculate the Power of a Number With pow()

Find the Natural Exponent With exp()

Practical Example With exp()

Logarithmic Functions

Python Natural Log With log()

Understand log2() and log10()

Practical Example With Natural Log

Other Important math Module Functions

Calculate the Greatest Common Divisor

Calculate the Sum of Iterables

Calculate the Square Root

Convert Angle Values

Calculate Trigonometric Values

Introducing `collections.deque`

How to Use `collections.deque`

How to Use `any()` in Python

How to Distinguish Between `or` and `any()`

How to Get Started With `simpy`

How to Simulate With the `simpy` Package

Getting to Know the Python `math` Module

Constants of the `math` Module

Find Factorials With Python `factorial()`

Find the Ceiling Value With `ceil()`

Find the Floor Value With `floor()`

Truncate Numbers With `trunc()`

Find the Closeness of Numbers With Python `isclose()`

Calculate the Power of a Number With `pow()`

Find the Natural Exponent With `exp()`

Practical Example With `exp()`

Python Natural Log With `log()`

Understand `log2()` and `log10()`

Other Important `math` Module Functions

New Additions to the `math` Module in Python 3.8

`cmath` vs `math`

NumPy vs `math`

`builtins`: The Built-In Scope

The `global` Statement

The `nonlocal` Statement

Bringing Names to Scope With `import`

`globals()`

`locals()`

`vars()`

`dir()`

Using the `bisect` Module

Understanding the Python `defaultdict` Type

Using the Python `defaultdict` Type