Real Python

Pointers in Python: What's the Point?

2019-05-29T14:00:00+00:00

If you’ve ever worked with lower level languages like C or C++, then you’ve probably heard of pointers. Pointers allow you to create great efficiency in parts of your code. They also cause confusion for beginners and can lead to various memory management bugs, even for experts. So where are they in Python, and how can you simulate pointers in Python?

Pointers are widely used in C and C++. Essentially, they are variables that hold the memory address of another variable. For a refresher on pointers, you might consider checking out this overview on C Pointers.

In this article, you’ll gain a better understanding of Python’s object model and learn why pointers in Python don’t really exist. For the cases where you need to mimic pointer behavior, you’ll learn ways to simulate pointers in Python without the memory-management nightmare.

In this article, you’ll:

Learn why pointers in Python don’t exist
Explore the difference between C variables and Python names
Simulate pointers in Python
Experiment with real pointers using ctypes

Note: In this article, “Python” will refer to the reference implementation of Python in C, otherwise known as CPython. As the article discusses some internals of the language, these notes are true for CPython 3.7 but may not be true in future or past iterations of the language.

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

Why Doesn’t Python Have Pointers?

The truth is that I don’t know. Could pointers in Python exist natively? Probably, but pointers seem to go against the Zen of Python. Pointers encourage implicit changes rather than explicit. Often, they are complex instead of simple, especially for beginners. Even worse, they beg for ways to shoot yourself in the foot, or do something really dangerous like read from a section of memory you were not supposed to.

Python tends to try to abstract away implementation details like memory addresses from its users. Python often focuses on usability instead of speed. As a result, pointers in Python doesn’t really make sense. Not to fear though, Python does, by default, give you some of the benefits of using pointers.

Understanding pointers in Python requires a short detour into Python’s implementation details. Specifically, you’ll need to understand:

Immutable vs mutable objects
Python variables/names

Hold onto your memory addresses, and let’s get started.

Objects in Python

In Python, everything is an object. For proof, you can open up a REPL and explore using isinstance():

>>>

>>> isinstance(1, object)
True
>>> isinstance(list(), object)
True
>>> isinstance(True, object)
True
>>> def foo():
...     pass
...
>>> isinstance(foo, object)
True

This code shows you that everything in Python is indeed an object. Each object contains at least three pieces of data:

Reference count
Type
Value

The reference count is for memory management. For an in-depth look at the internals of memory management in Python, you can read Memory Management in Python.

The type is used at the CPython layer to ensure type safety during runtime. Finally, there’s the value, which is the actual value associated with the object.

Not all objects are the same though. There is one other important distinction you’ll need to understand: immutable vs mutable objects. Understanding the difference between the types of objects really helps clarify the first layer of the onion that is pointers in Python.

Immutable vs Mutable Objects

In Python, there are two types of objects:

Immutable objects can’t be changed.
Mutable objects can be changed.

Understanding this difference is the first key to navigating the landscape of pointers in Python. Here’s a breakdown of common types and whether or not they are mutable or immutable:

Type	Immutable?
`int`	Yes
`float`	Yes
`bool`	Yes
`complex`	Yes
`tuple`	Yes
`frozenset`	Yes
`str`	Yes
`list`	No
`set`	No
`dict`	No

As you can see, lots of commonly used primitive types are immutable. You can prove this yourself by writing some Python. You’ll need a couple of tools from the Python standard library:

id() returns the object’s memory address.
is returns True if and only if two objects have the same memory address.

Once again, you can use these in a REPL environment:

>>>

>>> x = 5
>>> id(x)
94529957049376

In the above code, you have assigned the value 5 to x. If you tried to modify this value with addition, then you’d get a new object:

>>>

>>> x += 1
>>> x
6
>>> id(x)
94529957049408

Even though the above code appears to modify the value of x, you’re getting a new object as a response.

The str type is also immutable:

>>>

>>> s = "real_python"
>>> id(s)
140637819584048
>>> s += "_rocks"
>>> s
'real_python_rocks'
>>> id(s)
140637819609424

Again, s ends up with a different memory addresses after the += operation.

Bonus: The += operator translates to various method calls.

For some objects like list, += will translate into __iadd__() (in-place add). This will modify self and return the same ID. However, str and int don’t have these methods and result in __add__() calls instead of __iadd__().

For more detailed information, check out the Python data model docs.

Trying to directly mutate the string s results in an error:

>>>

>>> s[0] = "R"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'str' object does not support item assignment

The above code fails, and Python indicates that str doesn’t support this mutation, which is in line with the definition that the str type is immutable.

Contrast that with a mutable object, like list:

>>>

>>> my_list = [1, 2, 3]
>>> id(my_list)
140637819575368
>>> my_list.append(4)
>>> my_list
[1, 2, 3, 4]
>>> id(my_list)
140637819575368

This code shows a major difference in the two types of objects. my_list has an id originally. Even after 4 is appended to the list, my_list has the same id. This is because the list type is mutable.

Another way to demonstrate that the list is mutable is with assignment:

>>>

>>> my_list[0] = 0
>>> my_list
[0, 2, 3, 4]
>>> id(my_list)
140637819575368

In this code, you mutate my_list and set its first element to 0. However, it maintains the same id even after this assignment. With mutable and immutable objects out of the way, the next step on your journey to Python enlightenment is understanding Python’s variable ecosystem.

Understanding Variables

Python variables are fundamentally different than variables in C or C++. In fact, Python doesn’t even have variables. Python has names, not variables.

This might seem pedantic, and for the most part, it is. Most of the time, it’s perfectly acceptable to think about Python names as variables, but understanding the difference is important. This is especially true when you’re navigating the tricky subject of pointers in Python.

To help drive home the difference, you can take a look at how variables work in C, what they represent, and then contrast that with how names work in Python.

Variables in C

Let’s say you had the following code that defines the variable x:

int x = 2337;

This one line of code has several, distinct steps when executed:

Allocate enough memory for an integer
Assign the value 2337 to that memory location
Indicate that x points to that value

Shown in a simplified view of memory, it might look like this:

Here, you can see that the variable x has a fake memory location of 0x7f1 and the value 2337. If, later in the program, you want to change the value of x, you can do the following:

x = 2338;

The above code assigns a new value (2338) to the variable x, thereby overwriting the previous value. This means that the variable x is mutable. The updated memory layout shows the new value:

Notice that the location of x didn’t change, just the value itself. This is a significant point. It means that x is the memory location, not just a name for it.

Another way to think of this concept is in terms of ownership. In one sense, x owns the memory location. x is, at first, an empty box that can fit exactly one integer in which integer values can be stored.

When you assign a value to x, you’re placing a value in the box that x owns. If you wanted to introduce a new variable (y), you could add this line of code:

int y = x;

This code creates a new box called y and copies the value from x into the box. Now the memory layout will look like this:

Notice the new location 0x7f5 of y. Even though the value of x was copied to y, the variable y owns some new address in memory. Therefore, you could overwrite the value of y without affecting x:

y = 2339;

Now the memory layout will look like this:

Again, you have modified the value at y, but not its location. In addition, you have not affected the original x variable at all. This is in stark contrast with how Python names work.

Names in Python

Python does not have variables. It has names. Yes, this is a pedantic point, and you can certainly use the term variables as much as you like. It is important to know that there is a difference between variables and names.

Let’s take the equivalent code from the above C example and write it in Python:

>>>

>>> x = 2337

Much like in C, the above code is broken down into several distinct steps during execution:

Create a PyObject
Set the typecode to integer for the PyObject
Set the value to 2337 for the PyObject
Create a name called x
Point x to the new PyObject
Increase the refcount of the PyObject by 1

Note: The PyObject is not the same as Python’s object. It’s specific to CPython and represents the base structure for all Python objects.

PyObject is defined as a C struct, so if you’re wondering why you can’t call typecode or refcount directly, its because you don’t have access to the structures directly. Method calls like sys.getrefcount() can help get some internals.

In memory, it might looks something like this:

You can see that the memory layout is vastly different than the C layout from before. Instead of x owning the block of memory where the value 2337 resides, the newly created Python object owns the memory where 2337 lives. The Python name x doesn’t directly own any memory address in the way the C variable x owned a static slot in memory.

If you were to try to assign a new value to x, you could try the following:

>>>

>>> x = 2338

What’s happening here is different than the C equivalent, but not too different from the original bind in Python.

This code:

Creates a new PyObject
Sets the typecode to integer for the PyObject
Sets the value to 2338 for the PyObject
Points x to the new PyObject
Increases the refcount of the new PyObject by 1
Decreases the refcount of the old PyObject by 1

Now in memory, it would look something like this:

This diagram helps illustrate that x points to a reference to an object and doesn’t own the memory space as before. It also shows that the x = 2338 command is not an assignment, but rather binding the name x to a reference.

In addition, the previous object (which held the 2337 value) is now sitting in memory with a ref count of 0 and will get cleaned up by the garbage collector.

You could introduce a new name, y, to the mix as in the C example:

>>>

>>> y = x

In memory, you would have a new name, but not necessarily a new object:

Now you can see that a new Python object has not been created, just a new name that points to the same object. Also, the object’s refcount has increased by one. You could check for object identity equality to confirm that they are the same:

>>>

>>> y is x
True

The above code indicates that x and y are the same object. Make no mistake though: y is still immutable.

For example, you could perform addition on y:

>>>

>>> y += 1
>>> y is x
False

After the addition call, you are returned with a new Python object. Now, the memory looks like this:

A new object has been created, and y now points to the new object. Interestingly, this is the same end-state if you had bound y to 2339 directly:

>>>

>>> y = 2339

The above statement results in the same end-memory state as the addition. To recap, in Python, you don’t assign variables. Instead, you bind names to references.

A Note on Intern Objects in Python

Now that you understand how Python objects get created and names get bound to those objects, its time to throw a wrench in the machinery. That wrench goes by the name of interned objects.

Suppose you have the following Python code:

>>>

>>> x = 1000
>>> y = 1000
>>> x is y
True

As above, x and y are both names that point to the same Python object. But the Python object that holds the value 1000 is not always guaranteed to have the same memory address. For example, if you were to add two numbers together to get 1000, you would end up with a different memory address:

>>>

>>> x = 1000
>>> y = 499 + 501
>>> x is y
False

This time, the line x is y returns False. If this is confusing, then don’t worry. Here are the steps that occur when this code is executed:

Create Python object(1000)
Assign the name x to that object
Create Python object (499)
Create Python object (501)
Add these two objects together
Create a new Python object (1000)
Assign the name y to that object

Technical Note: The above steps occur only when this code is executed inside a REPL. If you were to take the example above, paste it into a file, and run the file, then you would find that the x is y line would return True.

This occurs because compilers are smart. The CPython compiler attempts to make optimizations called peephole optimizations, which help save execution steps whenever possible. For detailed information, you can check out CPython’s peephole optimizer source code.

Isn’t this wasteful? Well, yes it is, but that’s the price you pay for all of the great benefits of Python. You never have to worry about cleaning up these intermediate objects or even need to know that they exist! The joy is that these operations are relatively fast, and you never had to know any of those details until now.

The core Python developers, in their wisdom, also noticed this waste and decided to make a few optimizations. These optimizations result in behavior that can be surprising to newcomers:

>>>

>>> x = 20
>>> y = 19 + 1
>>> x is y
True

In this example, you see nearly the same code as before, except this time the result is True. This is the result of interned objects. Python pre-creates a certain subset of objects in memory and keeps them in the global namespace for everyday use.

Which objects depend on the implementation of Python. CPython 3.7 interns the following:

Integer numbers between -5 and 256
Strings that contain ASCII letters, digits, or underscores only

The reasoning behind this is that these variables are extremely likely to be used in many programs. By interning these objects, Python prevents memory allocation calls for consistently used objects.

Strings that are less than 20 characters and contain ASCII letters, digits, or underscores will be interned. The reasoning behind this is that these are assumed to be some kind of identity:

>>>

>>> s1 = "realpython"
>>> id(s1)
140696485006960
>>> s2 = "realpython"
>>> id(s2)
140696485006960
>>> s1 is s2
True

Here you can see that s1 and s2 both point to the same address in memory. If you were to introduce a non-ASCII letter, digit, or underscore, then you would get a different result:

>>>

>>> s1 = "Real Python!"
>>> s2 = "Real Python!"
>>> s1 is s2
False

Because this example has an exclamation mark (!) in it, these strings are not interned and are different objects in memory.

Bonus: If you really want these objects to reference the same internal object, then you may want to check out sys.intern(). One of the use cases for this function is outlined in the documentation:

Interning strings is useful to gain a little performance on dictionary lookup—if the keys in a dictionary are interned, and the lookup key is interned, the key comparisons (after hashing) can be done by a pointer compare instead of a string compare. (Source)

Interned objects are often a source of confusion. Just remember, if you’re ever in doubt, that you can always use id() and is to determine object equality.

Simulating Pointers in Python

Just because pointers in Python don’t exist natively doesn’t mean you can’t get the benefits of using pointers. In fact, there are multiple ways to simulate pointers in Python. You’ll learn two in this section:

Using mutable types as pointers
Using custom Python objects

Okay, let’s get to the point.

Using Mutable Types as Pointers

You’ve already learned about mutable types. Because these objects are mutable, you can treat them as if they were pointers to simulate pointer behavior. Suppose you wanted to replicate the following c code:

void add_one(int *x) {
    *x += 1;
}

This code takes a pointer to an integer (*x) and then increments the value by one. Here is a main function to exercise the code:

#include <stdio.h>

int main(void) {
    int y = 2337;
    printf("y = %d\n", y);
    add_one(&y);
    printf("y = %d\n", y);
    return 0;
}

In the above code, you assign 2337 to y, print out the current value, increment the value by one, and then print out the modified value. The output of executing this code would be the following:

y = 2337
y = 2338

One way to replicate this type of behavior in Python is by using a mutable type. Consider using a list and modifying the first element:

>>>

>>> def add_one(x):
...     x[0] += 1
...
>>> y = [2337]
>>> add_one(y)
>>> y[0]
2338

Here, add_one(x) accesses the first element and increments its value by one. Using a list means that the end result appears to have modified the value. So pointers in Python do exist? Well, no. This is only possible because list is a mutable type. If you tried to use a tuple, you would get an error:

>>>

>>> z = (2337,)
>>> add_one(z)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in add_one
TypeError: 'tuple' object does not support item assignment

The above code demonstrates that tuple is immutable. Therefore, it does not support item assignment. list is not the only mutable type. Another common approach to mimicking pointers in Python is to use a dict.

Let’s say you had an application where you wanted to keep track of every time an interesting event happened. One way to achieve this would be to create a dict and use one of the items as a counter:

>>>

>>> counters = {"func_calls": 0}
>>> def bar():
...     counters["func_calls"] += 1
...
>>> def foo():
...     counters["func_calls"] += 1
...     bar()
...
>>> foo()
>>> counters["func_calls"]
2

In this example, the counters dictionary is used to keep track of the number of function calls. After you call foo(), the counter has increased to 2 as expected. All because dict is mutable.

Keep in mind, this is only simulates pointer behavior and does not directly map to true pointers in C or C++. That is to say, these operations are more expensive than they would be in C or C++.

Using Python Objects

The dict option is a great way to emulate pointers in Python, but sometimes it gets tedious to remember the key name you used. This is especially true if you’re using the dictionary in various parts of your application. This is where a custom Python class can really help.

To build on the last example, assume that you want to track metrics in your application. Creating a class is a great way to abstract the pesky details:

class Metrics(object):
    def __init__(self):
        self._metrics = {
            "func_calls": 0,
            "cat_pictures_served": 0,
        }

This code defines a Metrics class. This class still uses a dict for holding the actual data, which is in the _metrics member variable. This will give you the mutability you need. Now you just need to be able to access these values. One nice way to do this is with properties:

class Metrics(object):
    # ...

    @property
    def func_calls(self):
        return self._metrics["func_calls"]

    @property
    def cat_pictures_served(self):
        return self._metrics["cat_pictures_served"]

This code makes use of @property. If you’re not familiar with decorators, you can check out this Primer on Python Decorators. The @property decorator here allows you to access func_calls and cat_pictures_served as if they were attributes:

>>>

>>> metrics = Metrics()
>>> metrics.func_calls
0
>>> metrics.cat_pictures_served
0

The fact that you can access these names as attributes means that you abstracted the fact that these values are in a dict. You also make it more explicit what the names of the attributes are. Of course, you need to be able to increment these values:

class Metrics(object):
    # ...

    def inc_func_calls(self):
        self._metrics["func_calls"] += 1

    def inc_cat_pics(self):
        self._metrics["cat_pictures_served"] += 1

You have introduced two new methods:

inc_func_calls()
inc_cat_pics()

These methods modify the values in the metrics dict. You now have a class that you modify as if you’re modifying a pointer:

>>>

>>> metrics = Metrics()
>>> metrics.inc_func_calls()
>>> metrics.inc_func_calls()
>>> metrics.func_calls
2

Here, you can access func_calls and call inc_func_calls() in various places in your applications and simulate pointers in Python. This is useful when you have something like metrics that need to be used and updated frequently in various parts of your applications.

Note: In this class in particular, making inc_func_calls() and inc_cat_pics() explicit instead of using @property.setter prevents users from setting these values to an arbitrary int or an invalid value like a dict.

Here’s the full source for the Metrics class:

class Metrics(object):
    def __init__(self):
        self._metrics = {
            "func_calls": 0,
            "cat_pictures_served": 0,
        }

    @property
    def func_calls(self):
        return self._metrics["func_calls"]

    @property
    def cat_pictures_served(self):
        return self._metrics["cat_pictures_served"]

    def inc_func_calls(self):
        self._metrics["func_calls"] += 1

    def inc_cat_pics(self):
        self._metrics["cat_pictures_served"] += 1

Real Pointers With `ctypes`

Okay, so maybe there are pointers in Python, specifically CPython. Using the builtin ctypes module, you can create real C-style pointers in Python. If you are unfamiliar with ctypes, then you can take a look at Extending Python With C Libraries and the “ctypes” Module.

The real reason you would use this is if you needed to make a function call to a C library that requires a pointer. Let’s go back to the add_one() C-function from before:

void add_one(int *x) {
    *x += 1;
}

Here again, this code is incrementing the value of x by one. To use this, first compile it into a shared object. Assuming the above file is stored in add.c, you could accomplish this with gcc:

$ gcc -c -Wall -Werror -fpic add.c
$ gcc -shared -o libadd1.so add.o

The first command compiles the C source file into an object called add.o. The second command takes that unlinked object file and produces a shared object called libadd1.so.

libadd1.so should be in your current directory. You can load it into Python using ctypes:

>>>

>>> import ctypes
>>> add_lib = ctypes.CDLL("./libadd1.so")
>>> add_lib.add_one
<_FuncPtr object at 0x7f9f3b8852a0>

The ctypes.CDLL code returns an object that represents the libadd1 shared object. Because you defined add_one() in this shared object, you can access it as if it were any other Python object. Before you call the function though, you should specify the function signature. This helps Python ensure that you pass the right type to the function.

In this case, the function signature is a pointer to an integer. ctypes will allow you to specify this using the following code:

>>>

>>> add_one = add_lib.add_one
>>> add_one.argtypes = [ctypes.POINTER(ctypes.c_int)]

In this code, you’re setting the function signature to match what C is expecting. Now, if you were to try to call this code with the wrong type, then you would get a nice warning instead of undefined behavior:

>>>

>>> add_one(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ctypes.ArgumentError: argument 1: <class 'TypeError'>: \
expected LP_c_int instance instead of int

Python throws an error, explaining that add_one() wants a pointer instead of just an integer. Luckily, ctypes has a way to pass pointers to these functions. First, declare a C-style integer:

>>>

>>> x = ctypes.c_int()
>>> x
c_int(0)

The above code creates a C-style integer x with a value of 0. ctypes provides the handy byref() to allow passing a variable by reference.

Note: The term by reference is opposed to passing a variable by value.

When passing by reference, you’re passing the reference to the original variable, and thus modifications will be reflected in the original variable. Passing by value results in a copy of the original variable, and modifications are not reflected in the original.

You can use this to call add_one():

>>>

>>> add_one(ctypes.byref(x))
998793640
>>> x
c_int(1)

Nice! Your integer was incremented by one. Congratulations, you have successfully used real pointers in Python.

Conclusion

You now have a better understanding of the intersection between Python objects and pointers. Even though some of the distinctions between names and variables seem pedantic, fundamentally understanding these key terms expands your understanding of how Python handles variables.

You’ve also learned some excellent ways to simulate pointers in Python:

Utilizing mutable objects as low-overhead pointers
Creating custom Python objects for ease of use
Unlocking real pointers with the ctypes module

These methods allow you to simulate pointers in Python without sacrificing the memory safety that Python provides.

Thanks for reading. If you still have questions, feel free to reach out either in the comments section or on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Continuous Integration With Python

2019-05-28T14:00:00+00:00

In this course, you’ll learn the core concepts behind Continuous Integration (CI) and why they are essential for modern software engineering teams.

Find out how to how set up Continuous Integration for your Python project to automatically create environments, install dependencies, and run tests.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Iterate Through a Dictionary in Python

2019-05-27T14:00:00+00:00

Dictionaries are one of the most important and useful data structures in Python. They can help you solve a wide variety of programming problems. This tutorial will take you on a deep dive into how to iterate through a dictionary in Python.

By the end of this tutorial, you’ll know:

What dictionaries are, as well as some of their main features and implementation details
How to iterate through a dictionary in Python by using the basic tools the language offers
What kind of real-world tasks you can perform by iterating through a dictionary in Python
How to use some more advanced techniques and strategies to iterate through a dictionary in Python

For more information on dictionaries, you can check out the following resources:

Dictionaries in Python
Itertools in Python 3, By Example
The documentation for map() and filter()

Ready? Let’s go!

A Few Words on Dictionaries

Dictionaries are a cornerstone of Python. The language itself is built around dictionaries. Modules, classes, objects, globals(), locals(): all of these are dictionaries. Dictionaries have been central to Python from its very beginning.

Python’s official documentation defines a dictionary as follows:

An associative array, where arbitrary keys are mapped to values. The keys can be any object with __hash__() and __eq__() methods. (Source)

There are a couple points to keep in mind:

Dictionaries map keys to values and store them in an array or collection.
The keys must be of a hashable type, which means that they must have a hash value that never changes during the key’s lifetime.

Dictionaries are frequently used for solving all kinds of programming problems, so they are a fundamental piece of your tool kit as a Python developer.

Unlike sequences, which are iterables that support element access using integer indices, dictionaries are indexed by keys.

The keys in a dictionary are much like a set, which is a collection of hashable and unique objects. Because the objects need to be hashable, mutable objects can’t be used as dictionary keys.

On the other hand, values can be of any Python type, whether they are hashable or not. There are literally no restrictions for values.

In Python 3.6 and beyond, the keys and values of a dictionary are iterated over in the same order in which they were created. However, this behavior may vary across different Python versions, and it depends on the dictionary’s history of insertions and deletions.

In Python 2.7, dictionaries are unordered structures. The order of the dictionaries’ items is scrambled. This means that the order of the items is deterministic and repeatable. Let’s see an example:

>>>

>>> # Python 2.7
>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> a_dict
{'color': 'blue', 'pet': 'dog', 'fruit': 'apple'}
>>> a_dict
{'color': 'blue', 'pet': 'dog', 'fruit': 'apple'}

If you leave the interpreter and open a new interactive session later, you’ll get the same item order:

>>>

>>> # Python 2.7. New interactive session
>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> a_dict
{'color': 'blue', 'pet': 'dog', 'fruit': 'apple'}
>>> a_dict
{'color': 'blue', 'pet': 'dog', 'fruit': 'apple'}

A closer look at these two outputs shows you that the resulting order is exactly the same in both cases. That’s why you can say that the ordering is deterministic.

In Python 3.5, dictionaries are still unordered, but this time, randomized data structures. This means that every time you re-run the dictionary, you’ll get a different items order. Let’s take a look:

>>>

>>> # Python 3.5
>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> a_dict
{'color': 'blue', 'pet': 'dog', 'fruit': 'apple'}
>>> a_dict
{'color': 'blue', 'pet': 'dog', 'fruit': 'apple'}

If you enter a new interactive session, then you’ll get the following:

>>>

>>> # Python 3.5. New interactive session
>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> a_dict
{'fruit': 'apple', 'pet': 'dog', 'color': 'blue'}
>>> a_dict
{'fruit': 'apple', 'pet': 'dog', 'color': 'blue'}

This time, you can see that the order of the items is different in both outputs. That’s why you can say they are randomized data structures.

In Python 3.6 and beyond, dictionaries are ordered data structures, which means that they keep their elements in the same order in which they were introduced, as you can see here:

>>>

>>> # Python 3.6 and beyond
>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> a_dict
{'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> a_dict
{'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}

This is a relatively new feature of Python’s dictionaries, and it’s a very useful one. But if you’re writing code that is supposed to be run in different Python versions, then you must not rely on this feature, because it can generate buggy behaviors.

Another important feature of dictionaries is that they are mutable data structures, which means that you can add, delete, and update their items. It’s worth noting that this also means that they can’t be used as keys to other dictionaries, as they are not hashable objects.

Note: Everything you’ve learned in this section is related to the core Python implementation, CPython.

Other Python implementations, like PyPy, IronPython or Jython, could exhibit different dictionary behaviors and features that are beyond the scope of this article.

How to Iterate Through a Dictionary in Python: The Basics

Dictionaries are an useful and widely used data structure in Python. As a Python coder, you’ll often be in situations where you’ll need to iterate through a dictionary in Python, while you perform some actions on its key-value pairs.

When it comes to iterating through a dictionary in Python, the language provides you with some great tools that we’ll cover in this article.

Iterating Through Keys Directly

Python’s dictionaries are mapping objects. This means that they inherit some special methods, which Python uses internally to perform some operations. These methods are named using the naming convention of adding a double underscore at the beginning of and at the end of the method’s name.

To visualize the methods and attributes of any Python object, you can use dir(), which is a built-in function that serves that purpose. If you run dir() with an empty dictionary as an argument, then you’ll be able to see all the methods and attributes that dictionaries implement:

>>>

>>> dir({})
['__class__', '__contains__', '__delattr__', ... , '__iter__', ...]

If you take a closer look at the previous output, you’ll see '__iter__'. This is a method that is called when an iterator is required for a container, and it should return a new iterator object that can iterate through all the objects in the container.

Note: The output of the previous code has been abbreviated (...) in order to save space.

For mappings (like dictionaries), .__iter__() should iterate over the keys. This means that if you put a dictionary directly into a for loop, Python will automatically call .__iter__() on that dictionary, and you’ll get an iterator over its keys:

>>>

>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> for key in a_dict:
...     print(key)
...
color
fruit
pet

Python is smart enough to know that a_dict is a dictionary and that it implements .__iter__(). In this example, Python called .__iter__() automatically, and this allowed you to iterate over the keys of a_dict.

This is the simplest way to iterate through a dictionary in Python. Just put it directly into a for loop, and you’re done!

If you use this approach along with a small trick, then you can process the keys and values of any dictionary. The trick consists of using the indexing operator [] with the dictionary and its keys to get access to the values:

>>>

>>> for key in a_dict:
...     print(key, '->', a_dict[key])
...
color -> blue
fruit -> apple
pet -> dog

The preceding code allowed you to get access to the keys (key) and the values (a_dict[key]) of a_dict at the same time. This way, you can do any operation with both the keys and the values.

Iterating Through `.items()`

When you’re working with dictionaries, it’s likely that you’ll want to work with both the keys and the values. One of the most useful ways to iterate through a dictionary in Python is by using .items(), which is a method that returns a new view of the dictionary’s items:

>>>

>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> d_items = a_dict.items()
>>> d_items  # Here d_items is a view of items
dict_items([('color', 'blue'), ('fruit', 'apple'), ('pet', 'dog')])

Dictionary views like d_items provide a dynamic view on the dictionary’s entries, which means that when the dictionary changes, the views reflect these changes.

Views can be iterated over to yield their respective data, so you can iterate through a dictionary in Python by using the view object returned by .items():

>>>

>>> for item in a_dict.items():
...     print(item)
...
('color', 'blue')
('fruit', 'apple')
('pet', 'dog')

The view object returned by .items() yields the key-value pairs one at a time and allows you to iterate through a dictionary in Python, but in such a way that you get access to the keys and values at the same time.

If you take a closer look at the individual items yielded by .items(), you’ll notice that they’re really tuple objects. Let’s take a look:

>>>

>>> for item in a_dict.items():
...     print(item)
...     print(type(item))
...
('color', 'blue')
<class 'tuple'>
('fruit', 'apple')
<class 'tuple'>
('pet', 'dog')
<class 'tuple'>

Once you know this, you can use tuple unpacking to iterate through the keys and values of the dictionary you are working with. To achieve this, you just need to unpack the elements of every item into two different variables representing the key and the value:

>>>

>>> for key, value in a_dict.items():
...     print(key, '->', value)
...
color -> blue
fruit -> apple
pet -> dog

Here, the variables key and value in the header of your for loop do the unpacking. Every time the loop runs, key will store the key, and value will store the value of the item that is been processed. This way, you’ll have more control over the items of the dictionary, and you’ll be able to process the keys and values separately and in a way that is more readable and Pythonic.

Note: Notice that .values() and .keys() return view objects just like .items(), as you’ll see in the next two sections.

Iterating Through `.keys()`

If you just need to work with the keys of a dictionary, then you can use .keys(), which is a method that returns a new view object containing the dictionary’s keys:

>>>

>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> keys = a_dict.keys()
>>> keys
dict_keys(['color', 'fruit', 'pet'])

The object returned by .keys() here provided a dynamic view on the keys of a_dict. This view can be used to iterate through the keys of a_dict.

To iterate through a dictionary in Python by using .keys(), you just need to call .keys() in the header of a for loop:

>>>

>>> for key in a_dict.keys():
...     print(key)
...
color
fruit
pet

When you call .keys() on a_dict, you get a view of keys. Python knows that view objects are iterables, so it starts looping, and you can process the keys of a_dict.

On the other hand, using the same trick you’ve seen before (indexing operator []), you can get access to the values of the dictionary:

>>>

>>> for key in a_dict.keys():
...     print(key, '->', a_dict[key])
...
color -> blue
fruit -> apple
pet -> dog

This way you’ve gotten access to the keys (key) and values (a_dict[key]) of a_dict at the same time, and you’ll be able to perform any action on them.

Iterating Through `.values()`

It’s also common to only use the values to iterate through a dictionary in Python. One way to do that is to use .values(), which returns a view with the values of the dictionary:

>>>

>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> values = a_dict.values()
>>> values
dict_values(['blue', 'apple', 'dog'])

In the previous code, values holds a reference to a view object containing the values of a_dict.

As any view object, the object returned by .values() can also be iterated over. In this case, .values() yields the values of a_dict:

>>>

>>> for value in a_dict.values():
...     print(value)
...
blue
apple
dog

Using .values(), you’ll be getting access to only the values of a_dict, without dealing with the keys.

It’s worth noting that they also support membership tests (in), which is an important feature if you’re trying to know if a specific element is in a dictionary or not:

>>>

>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> 'pet' in a_dict.keys()
True
>>> 'apple' in a_dict.values()
True
>>> 'onion' in a_dict.values()
False

The membership test using in returns True if the key (or value or item) is present in the dictionary you’re testing, and returns False otherwise. The membership test allows you to not iterate through a dictionary in Python if you just want to know if certain key (or value or item) is present in a dictionary or not.

Modifying Values and Keys

It can be pretty common to need to modify the values and keys when you’re iterating through a dictionary in Python. There are some points you’ll need to take into account to accomplish this task.

The values, for example, can be modified whenever you need, but you’ll need to use the original dictionary and the key that maps the value you want to modify:

>>>

>>> prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> for k, v in prices.items():
...     prices[k] = round(v * 0.9, 2)  # Apply a 10% discount
...
>>> prices
{'apple': 0.36, 'orange': 0.32, 'banana': 0.23}

In the previous code example, to modify the values of prices and apply a 10% discount, you used the expression prices[k] = round(v * 0.9, 2).

So why do you have to use the original dictionary if you have access to its key (k) and its values (v)? Should you be able to modify them directly?

The real problem is that k and v changes aren’t reflected in the original dictionary. That is, if you modify any of them (k or v) directly inside the loop, then what really happens is that you’ll lose the reference to the relevant dictionary component without changing anything in the dictionary.

On the other hand, the keys can be added or removed from a dictionary by converting the view returned by .keys() into a list object:

>>>

>>> prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> for key in list(prices.keys()):  # Use a list instead of a view
...     if key == 'orange':
...         del prices[key]  # Delete a key from prices
...
>>> prices
{'apple': 0.4, 'banana': 0.25}

This approach may have some performance implications, mainly related to memory consumption. For example, instead of a view object that yields elements on demand, you’ll have an entire new list in your system’s memory. However, this could be a safe way to modify the keys while you iterate through a dictionary in Python.

Finally, if you try to remove a key from prices by using .keys() directly, then Python will raise a RuntimeError telling you that the dictionary’s size has changed during iteration:

>>>

>>> # Python 3. dict.keys() returns a view object, not a list
>>> prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> for key in prices.keys():
...     if key == 'orange':
...         del prices[key]
...
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    for key in prices.keys():
RuntimeError: dictionary changed size during iteration

This is because .keys() returns a dictionary-view object, which yields keys on demand one at a time, and if you delete an item (del prices[key]), then Python raises a RuntimeError, because you’ve modified the dictionary during iteration.

Note: In Python 2, .items(), .keys(), and .values() return list objects. But .iteritems(), iterkeys(), and .itervalues() return iterators. So, if you’re using Python 2, then you can modify the dictionary’s keys by using .keys() directly.

On the other hand, if you’re using iterkeys() in your Python 2 code and you try to modify the keys of a dictionary, then you’ll get a RuntimeError.

Real-World Examples

So far, you’ve seen the more basic ways of iterating through a dictionary in Python. Now it’s time to see how you can perform some actions with the items of a dictionary during iteration. Let’s look at some real-world examples.

Note: Later on in this article, you’ll see another way of solving these very same problems by using other Python tools.

Turning Keys Into Values and Vice Versa

Suppose you have a dictionary and for some reason need to turn keys into values and vice versa. In this situation, you can use a for loop to iterate through the dictionary and build the new dictionary by using the keys as values and vice versa:

>>>

>>> a_dict = {'one': 1, 'two': 2, 'thee': 3, 'four': 4}
>>> new_dict = {}
>>> for key, value in a_dict.items():
...     new_dict[value] = key
...
>>> new_dict
{1: 'one', 2: 'two', 3: 'thee', 4: 'four'}

The expression new_dict[value] = key did all the work for you by turning the keys into values and using the values as keys. For this code to work, the data stored in the original values must be of a hashable data type.

Filtering Items

Sometimes you’ll be in situations where you have a dictionary and you want to create a new one to store only the data that satisfies a given condition. You can do this with an if statement inside a for loop as follows:

>>>

>>> a_dict = {'one': 1, 'two': 2, 'thee': 3, 'four': 4}
>>> new_dict = {}  # Create a new empty dictionary
>>> for key, value in a_dict.items():
...     # If value satisfies the condition, then store it in new_dict
...     if value <= 2:
...         new_dict[key] = value
...
>>> new_dict
{'one': 1, 'two': 2}

In this example, you’ve filtered out the items with a value greater than 2. Now new_dict only contains the items that satisfy the condition value <= 2. This is one possible solution for this kind of problem. Later on, you’ll see a more Pythonic and readable way to get the same result.

Doing Some Calculations

It’s also common to need to do some calculations while you iterate through a dictionary in Python. Suppose you’ve stored the data for your company’s sales in a dictionary, and now you want to know the total income of the year.

To solve this problem you could define a variable with an initial value of zero. Then, you can accumulate every value of your dictionary in that variable:

>>>

>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> total_income = 0.00
>>> for value in incomes.values():
...     total_income += value  # Accumulate the values in total_income
...
>>> total_income
14100.0

Here, you’ve iterated through incomes and sequentially accumulated its values in total_income as you wanted to do. The expression total_income += value does the magic, and at the end of the loop, you’ll get the total income of the year. Note that total_income += value is equivalent to total_income = total_income + value.

Using Comprehensions

A dictionary comprehension is a compact way to process all or part of the elements in a collection and return a dictionary as a results. In contrast to list comprehensions, they need two expressions separated with a colon followed by for and if (optional) clauses. When a dictionary comprehension is run, the resulting key-value pairs are inserted into a new dictionary in the same order in which they were produced.

Suppose, for example, that you have two lists of data, and you need to create a new dictionary from them. In this case, you can use Python’s zip(*iterables) to loop over the elements of both lists in pairs:

>>>

>>> objects = ['blue', 'apple', 'dog']
>>> categories = ['color', 'fruit', 'pet']
>>> a_dict = {key: value for key, value in zip(categories, objects)}
>>> a_dict
{'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}

Here, zip() receives two iterables (categories and objects) as arguments and makes an iterator that aggregates elements from each iterable. The tuple objects generated by zip() are then unpacked into key and value, which are finally used to create the new dictionary.

Dictionary comprehensions open up a wide spectrum of new possibilities and provide you with a great tool to iterate through a dictionary in Python.

Turning Keys Into Values and Vice Versa: Revisited

If you take another look at the problem of turning keys into values and vice versa, you’ll see that you could write a more Pythonic and efficient solution by using a dictionary comprehension:

>>>

>>> a_dict = {'one': 1, 'two': 2, 'thee': 3, 'four': 4}
>>> new_dict = {value: key for key, value in a_dict.items()}
>>> new_dict
{1: 'one', 2: 'two', 3: 'thee', 4: 'four'}

With this dictionary comprehension, you’ve created a totally new dictionary where the keys have taken the place of the values and vice versa. This new approach gave you the ability to write more readable, succinct, efficient, and Pythonic code.

The condition for this code to work is the same one you saw before: the values must be hashable objects. Otherwise, you won’t be able to use them as keys for a_dict.

Filtering Items: Revisited

To filter the items in a dictionary with a comprehension, you just need to add an if clause that defines the condition you want to meet. In the previous example where you filtered a dictionary, that condition was if v <= 2. With this if clause added to the end of the dictionary comprehension, you’ll filter out the items whose values are greater than 2. Let’s take a look:

>>>

>>> a_dict = {'one': 1, 'two': 2, 'thee': 3, 'four': 4}
>>> new_dict = {k: v for k, v in a_dict.items() if v <= 2}
>>> new_dict
{'one': 1, 'two': 2}

Now new_dict contains only the items that satisfy your condition. Compared to the previous solutions, this one is more Pythonic and efficient.

Doing Some Calculations: Revisited

Remember the example with the company’s sales? If you use a list comprehension to iterate through the dictionary’s values, then you’ll get code that is more compact, fast, and Pythonic:

>>>

>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> total_income = sum([value for value in incomes.values()])
>>> total_income
14100.0

The list comprehension created a list object containing the values of incomes, and then you summed up all of them by using sum() and stored the result in total_income.

If you’re working with a really large dictionary, and memory usage is a problem for you, then you can use a generator expression instead of a list comprehension. A generator expression is an expression that returns an iterator. It looks like a list comprehension, but instead of brackets you need to use parentheses to define it:

>>>

>>> total_income = sum(value for value in incomes.values())
>>> total_income
14100.0

If you change the square brackets for a pair of parentheses (the parentheses of sum() here), you’ll be turning the list comprehension into a generator expression, and your code will be memory efficient, because generator expressions yield elements on demand. Instead of creating and storing the whole list in memory, you’ll only have to store one element at a time.

Note: If you are totally new to generator expressions, you can take a look at Introduction to Python Generators to get a better understanding of the topic.

Finally, there is a simpler way to solve this problem by just using incomes.values() directly as an argument to sum():

>>>

>>> total_income = sum(incomes.values())
>>> total_income
14100.0

sum() receives an iterable as an argument and returns the total sum of its elements. Here, incomes.values() plays the role of the iterable passed to sum(). The result is the total income you were looking for.

Removing Specific Items

Now, suppose you have a dictionary and need to create a new one with selected keys removed. Remember how key-view objects are like sets? Well, these similarities go beyond just being collections of hashable and unique objects. Key-view objects also support common set operations. Let’s see how you can take advantage of this to remove specific items in a dictionary:

>>>

>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> non_citric = {k: incomes[k] for k in incomes.keys() - {'orange'}}
>>> non_citric
{'apple': 5600.0, 'banana': 5000.0}

This code works because key-view objects support set operations like unions, intersections, and differences. When you wrote incomes.keys() - {'orange'} inside the dictionary comprehension, you were really doing a set difference operation. If you need to perform any set operations with the keys of a dictionary, then you can just use the key-view object directly without first converting it into a set. This is a little-known feature of key-view objects that can be useful in some situations.

Sorting a Dictionary

It’s often necessary to sort the elements of a collection. Since Python 3.6, dictionaries are ordered data structures, so if you use Python 3.6 (and beyond), you’ll be able to sort the items of any dictionary by using sorted() and with the help of a dictionary comprehension:

>>>

>>> # Python 3.6, and beyond
>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> sorted_income = {k: incomes[k] for k in sorted(incomes)}
>>> sorted_income
{'apple': 5600.0, 'banana': 5000.0, 'orange': 3500.0}

This code allows you to create a new dictionary with its keys in sorted order. This is possible because sorted(incomes) returns a list of sorted keys that you can use to generate the new dictionary sorted_dict.

Iterating in Sorted Order

Sometimes you may need to iterate through a dictionary in Python but want to do it in sorted order. This can be achieved by using sorted(). When you call sorted(iterable), you get a list with the elements of iterable in sorted order.

Let’s see how you can use sorted() to iterate through a dictionary in Python when you need to do it in sorted order.

Sorted by Keys

If you need to iterate through a dictionary in Python and want it to be sorted by keys, then you can use your dictionary as an argument to sorted(). This will return a list containing the keys in sorted order, and you’ll be able to iterate through them:

>>>

>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> for key in sorted(incomes):
...     print(key, '->', incomes[key])
...
apple -> 5600.0
banana -> 5000.0
orange -> 3500.0

In this example, you sorted the dictionary (alphabetically) by keys using sorted(incomes) in the header of the for loop. Notice that you can also use sorted(incomes.keys()) to get the same result. In both cases, you’ll get a list containing the keys of your dictionary in sorted order.

Note: The sorting order will depend on the data type you are using for keys or values and the internal rules that Python uses to sort those data types.

Sorted by Values

You could also need to iterate through a dictionary in Python with its items sorted by values. You can use sorted() too, but with a second argument called key.

The key keyword argument specifies a function of one argument that is used to extract a comparison key from each element you’re processing.

To sort the items of a dictionary by values, you can write a function that returns the value of each item and use this function as the key argument to sorted():

>>>

>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> def by_value(item):
...     return item[1]
...
>>> for k, v in sorted(incomes.items(), key=by_value):
...     print(k, '->', v)
...
('orange', '->', 3500.0)
('banana', '->', 5000.0)
('apple', '->', 5600.0)

In this example, you defined by_value() and used it to sort the items of incomes by value. Then you iterated through the dictionary in sorted order by using sorted(). The key function (by_value()) tells sorted() to sort incomes.items() by the second element of each item, that is, by the value (item[1]).

You may also just want to iterate through the values of a dictionary in sorted order, without worrying about the keys. In that case, you can use .values() as follows:

>>>

>>> for value in sorted(incomes.values()):
...     print(value)
...
3500.0
5000.0
5600.0

sorted(incomes.values()) returned the values of the dictionary in sorted order as you desired. The keys won’t be accessible if you use incomes.values(), but sometimes you don’t really need the keys, just the values, and this is a fast way to get access to them.

Reversed

If you need to sort your dictionaries in reverse order, you can add reverse=True as an argument to sorted(). The keyword argument reverse should take a boolean value. If it’s set to True, then the elements are sorted in reverse order:

>>>

>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> for key in sorted(incomes, reverse=True):
...     print(key, '->', incomes[key])
...
orange -> 3500.0
banana -> 5000.0
apple -> 5600.0

Here, you iterated over the keys of incomes in reverse order by using sorted(incomes, reverse=True) in the header of the for loop.

Finally, it’s important to note that sorted() doesn’t really modify the order of the underlying dictionary. What really happen is that sorted() creates an independent list with its element in sorted order, so incomes remains the same:

>>>

>>> incomes
{'apple': 5600.0, 'orange': 3500.0, 'banana': 5000.0}

This code shows you that incomes didn’t change. sorted() didn’t modify incomes. It just created a new sorted list from the keys of incomes.

Iterating Destructively With `.popitem`()

Sometimes you need to iterate through a dictionary in Python and delete its items sequentially. To accomplish this task, you can use .popitem(), which will remove and return an arbitrary key-value pair from a dictionary. On the other hand, when you call .popitem() on an empty dictionary, it raises a KeyError.

If you really need to destructively iterate through a dictionary in Python, then .popitem() can be useful. Here’s an example:

 1 # File: dict_popitem.py
 2 
 3 a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
 4 
 5 while True:
 6     try:
 7         print(f'Dictionary length: {len(a_dict)}')
 8         item = a_dict.popitem()
 9         # Do something with item here...
10         print(f'{item} removed')
11     except KeyError:
12         print('The dictionary has no item now...')
13         break

Here, you used a while loop instead of a for loop. The reason for this is that it’s never safe to iterate through a dictionary in Python if you pretend to modify it this way, that is, if you’re deleting or adding items to it.

Inside the while loop, you defined a try...except block to catch the KeyError raised by .popitems() when a_dict turns empty. In the try...except block, you process the dictionary, removing an item in each iteration. The variable item keeps a reference to the successive items and allows you to do some actions with them.

Note: In the previous code example, you used Python’s f-strings for string formatting. If you want to dive deeper into f-strings, then you can take a look at Python 3’s f-Strings: An Improved String Formatting Syntax (Guide).

If you run this script from your command-line, then you’ll get the following results:

$ python3 dict_popitem.py
Dictionary length: 3
('pet', 'dog') removed
Dictionary length: 2
('fruit', 'apple') removed
Dictionary length: 1
('color', 'blue') removed
Dictionary length: 0
The dictionary has no item now...

Here .popitem() sequentially removed the items of a_dict. The loop broke when the dictionary became empty, and .popitem() raised a KeyError exception.

Using Some of Python’s Built-In Functions

Python provides some built-in functions that could be useful when you’re working with collections, like dictionaries. These functions are a sort of iteration tool that provides you with another way of iterating through a dictionary in Python. Let’s see some of them.

`map()`

Python’s map() is defined as map(function, iterable, ...) and returns an iterator that applies function to every item of iterable, yielding the results on demand. So, map() could be viewed as an iteration tool that you can use to iterate through a dictionary in Python.

Suppose you have a dictionary containing the prices of a bunch of products, and you need to apply a discount to them. In this case, you can define a function that manages the discount and then uses it as the first argument to map(). The second argument can be prices.items():

>>>

>>> prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> def discount(current_price):
...     return (current_price[0], round(current_price[1] * 0.95, 2))
...
>>> new_prices = dict(map(discount, prices.items()))
>>> new_prices
{'apple': 0.38, 'orange': 0.33, 'banana': 0.24}

Here, map() iterated through the items of the dictionary (prices.items()) to apply a 5% discount to each fruit by using discount(). In this case, you need to use dict() to generate the new_prices dictionary from the iterator returned by map().

Note that discount() returns a tuple of the form (key, value), where current_price[0] represents the key and round(current_price[1] * 0.95, 2) represents the new value.

`filter()`

filter() is another built-in function that you can use to iterate through a dictionary in Python and filter out some of its items. This function is defined as filter(function, iterable) and returns an iterator from those elements of iterable for which function returns True.

Suppose you want to know the products with a price lower than 0.40. You need to define a function to determine if the price satisfies that condition and pass it as first argument to filter(). The second argument can be prices.keys():

>>>

>>> prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> def has_low_price(price):
...     return prices[price] < 0.4
...
>>> low_price = list(filter(has_low_price, prices.keys()))
>>> low_price
['orange', 'banana']

Here, you iterated through the keys of prices with filter(). Then filter() applies has_low_price() to every key of prices. Finally, you need to use list() to generate the list of products with a low price, because filter() returns an iterator, and you really need a list object.

Using `collections.ChainMap`

collections is a useful module from the Python Standard Library that provides specialized container data types. One of these data types is ChainMap, which is a dictionary-like class for creating a single view of multiple mappings (like dictionaries). With ChainMap, you can group multiple dictionaries together to create a single, updateable view.

Now, suppose you have two (or more) dictionaries, and you need to iterate through them together as one. To achieve this, you can create a ChainMap object and initialize it with your dictionaries:

>>>

>>> from collections import ChainMap
>>> fruit_prices = {'apple': 0.40, 'orange': 0.35}
>>> vegetable_prices = {'pepper': 0.20, 'onion': 0.55}
>>> chained_dict = ChainMap(fruit_prices, vegetable_prices)
>>> chained_dict  # A ChainMap object
ChainMap({'apple': 0.4, 'orange': 0.35}, {'pepper': 0.2, 'onion': 0.55})
>>> for key in chained_dict:
...     print(key, '->', chained_dict[key])
...
pepper -> 0.2
orange -> 0.35
onion -> 0.55
apple -> 0.4

After importing ChainMap from collections, you need to create a ChainMap object with the dictionaries you want to chain, and then you can freely iterate through the resulting object as you would do with a regular dictionary.

ChainMap objects also implement .keys(), values(), and .items() as a standard dictionary does, so you can use these methods to iterate through the dictionary-like object generated by ChainMap, just like you would do with a regular dictionary:

>>>

>>> for key, value in chained_dict.items():
...     print(key, '->', value)
...
apple -> 0.4
pepper -> 0.2
orange -> 0.35
onion -> 0.55

In this case, you’ve called .items() on a ChainMap object. The ChainMap object behaved as if it were a regular dictionary, and .items() returned a dictionary view object that can be iterated over as usual.

Using `itertools`

Python’s itertools is a module that provides some useful tools to perform iteration tasks. Let’s see how you can use some of them to iterate through a dictionary in Python.

Cyclic Iteration With `cycle()`

Suppose you want to iterate through a dictionary in Python, but you need to iterate through it repeatedly in a single loop. To get this task done, you can use itertools.cycle(iterable), which makes an iterator returning elements from iterable and saving a copy of each. When iterable is exhausted, cycle() returns elements from the saved copy. This is performed in cyclic fashion, so it’s up to you to stop the cycle.

In the following example, you’ll be iterating through the items of a dictionary three consecutive times:

>>>

>>> from itertools import cycle
>>> prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> times = 3  # Define how many times you need to iterate through prices
>>> total_items = times * len(prices)
>>> for item in cycle(prices.items()):
...     if not total_items:
...         break
...     total_items -= 1
...     print(item)
...
('apple', 0.4)
('orange', 0.35)
('banana', 0.25)
('apple', 0.4)
('orange', 0.35)
('banana', 0.25)
('apple', 0.4)
('orange', 0.35)
('banana', 0.25)

The preceding code allowed you to iterate through prices a given number of times (3 in this case). This cycle could be as long as you need, but you are responsible for stopping it. The if condition breaks the cycle when total_items counts down to zero.

Chained Iteration With `chain()`

itertools also provides chain(*iterables), which gets some iterables as arguments and makes an iterator that yields elements from the first iterable until it’s exhausted, then iterates over the next iterable and so on, until all of them are exhausted.

This allows you to iterate through multiple dictionaries in a chain, like to what you did with collections.ChainMap:

>>>

>>> from itertools import chain
>>> fruit_prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> vegetable_prices = {'pepper': 0.20, 'onion': 0.55, 'tomato': 0.42}
>>> for item in chain(fruit_prices.items(), vegetable_prices.items()):
...     print(item)
...
('apple', 0.4)
('orange', 0.35)
('banana', 0.25)
('pepper', 0.2)
('onion', 0.55)
('tomato', 0.42)

In the above code, chain() returned an iterable that combined the items from fruit_prices and vegetable_prices.

It’s also possible to use .keys() or .values(), depending on your needs, with the condition of being homogeneous: if you use .keys() for an argument to chain(), then you need to use .keys() for the rest of them.

Using the Dictionary Unpacking Operator (`**`)

Python 3.5 brings a new and interesting feature. PEP 448 - Additional Unpacking Generalizations can make your life easier when it comes to iterating through multiple dictionaries in Python. Let’s see how this works with a short example.

Suppose you have two (or more) dictionaries, and you need to iterate through them together, without using collections.ChainMap or itertools.chain(), as you’ve seen in the previous sections. In this case, you can use the dictionary unpacking operator (**) to merge the two dictionaries into a new one and then iterate through it:

>>>

>>> fruit_prices = {'apple': 0.40, 'orange': 0.35}
>>> vegetable_prices = {'pepper': 0.20, 'onion': 0.55}
>>> # How to use the unpacking operator **
>>> {**vegetable_prices, **fruit_prices}
{'pepper': 0.2, 'onion': 0.55, 'apple': 0.4, 'orange': 0.35}
>>> # You can use this feature to iterate through multiple dictionaries
>>> for k, v in {**vegetable_prices, **fruit_prices}.items():
...     print(k, '->', v)
...
pepper -> 0.2
onion -> 0.55
apple -> 0.4
orange -> 0.35

The dictionary unpacking operator (**) is really an awesome feature in Python. It allows you to merge multiple dictionaries into a new one, as you did in the example with vegetable_prices and fruit_prices. Once you’ve merged the dictionaries with the unpacking operator, you can iterate through the new dictionary as usual.

It’s important to note that if the dictionaries you’re trying to merge have repeated or common keys, then the values of the right-most dictionary will prevail:

>>>

>>> vegetable_prices = {'pepper': 0.20, 'onion': 0.55}
>>> fruit_prices = {'apple': 0.40, 'orange': 0.35, 'pepper': .25}
>>> {**vegetable_prices, **fruit_prices}
{'pepper': 0.25, 'onion': 0.55, 'apple': 0.4, 'orange': 0.35}

The pepper key is present in both dictionaries. After you merge them, the fruit_prices value for pepper (0.25) prevailed, because fruit_prices is the right-most dictionary.

Conclusion

You now know the basics of how to iterate through a dictionary in Python, as well as some more advanced techniques and strategies!

You’ve learned:

What dictionaries are, as well as some of their main features and implementation details
What the basic ways to iterate through a dictionary in Python are
What kind of tasks you can accomplish by iterating through a dictionary in Python
How to use some more elaborated techniques and strategies to iterate through a dictionary in Python

You have the tools and knowledge you’ll need to get the most out of dictionaries in Python. This will help you be more efficient and effective in your use of dictionary iteration in the future.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Logging: A Stroll Through the Source Code

2019-05-22T14:00:00+00:00

The Python logging package is a a lightweight but extensible package for keeping better track of what your own code does. Using it gives you much more flexibility than just littering your code with superfluous print() calls.

However, Python’s logging package can be complicated in certain spots. Handlers, loggers, levels, namespaces, filters: it’s not easy to keep track of all of these pieces and how they interact.

One way to tie up the loose ends in your understanding of logging is to peek under the hood to its CPython source code. The Python code behind logging is concise and modular, and reading through it can help you get that aha moment.

This article is meant to complement the logging HOWTO document as well as Logging in Python, which is a walkthrough on how to use the package.

By the end of this article, you’ll be familiar with the following:

logging levels and how they work
Thread-safety versus process-safety in logging
The design of logging from an OOP perspective
Logging in libraries vs applications
Best practices and design patterns for using logging

For the most part, we’ll go line-by-line down the core module in Python’s logging package in order to build a picture of how it’s laid out.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

How to Follow Along

Because the logging source code is central to this article, you can assume that any code block or link is based on a specific commit in the Python 3.7 CPython repository, namely commit d730719. You can find the logging package itself in the Lib/ directory within the CPython source.

Within the logging package, most of the heavy lifting occurs within logging/__init__.py, which is the file you’ll spend the most time on here:

cpython/
│
├── Lib/
│   ├── logging/
│   │   ├── __init__.py
│   │   ├── config.py
│   │   └── handlers.py
│   ├── ...
├── Modules/
├── Include/
...
... [truncated]

With that, let’s jump in.

Preliminaries

Before we get to the heavyweight classes, the top hundred lines or so of __init__.py introduce a few subtle but important concepts.

Preliminary #1: A Level Is Just an `int`!

Objects like logging.INFO or logging.DEBUG can seem a bit opaque. What are these variables internally, and how are they defined?

In fact, the uppercase constants from Python’s logging are just integers, forming an enum-like collection of numerical levels:

CRITICAL = 50
FATAL = CRITICAL
ERROR = 40
WARNING = 30
WARN = WARNING
INFO = 20
DEBUG = 10
NOTSET = 0

Why not just use the strings "INFO" or "DEBUG"? Levels are int constants to allow for the simple, unambiguous comparison of one level with another. They are given names as well to lend them semantic meaning. Saying that a message has a severity of 50 may not be immediately clear, but saying that it has a level of CRITICAL lets you know that you’ve got a flashing red light somewhere in your program.

Now, technically, you can pass just the str form of a level in some places, such as logger.setLevel("DEBUG"). Internally, this will call _checkLevel(), which ultimately does a dict lookup for the corresponding int:

_nameToLevel = {
    'CRITICAL': CRITICAL,
    'FATAL': FATAL,
    'ERROR': ERROR,
    'WARN': WARNING,
    'WARNING': WARNING,
    'INFO': INFO,
    'DEBUG': DEBUG,
    'NOTSET': NOTSET,
}

def _checkLevel(level):
    if isinstance(level, int):
        rv = level
    elif str(level) == level:
        if level not in _nameToLevel:
            raise ValueError("Unknown level: %r" % level)
        rv = _nameToLevel[level]
    else:
        raise TypeError("Level not an integer or a valid string: %r" % level)
    return rv

Which should you prefer? I’m not too opinionated on this, but it’s notable that the logging docs consistently use the form logging.DEBUG rather than "DEBUG" or 10. Also, passing the str form isn’t an option in Python 2, and some logging methods such as logger.isEnabledFor() will accept only an int, not its str cousin.

Preliminary #2: Logging Is Thread-Safe, but Not Process-Safe

A few lines down, you’ll find the following short code block, which is sneakily critical to the whole package:

import threading

_lock = threading.RLock()

def _acquireLock():
     if _lock:
        _lock.acquire()

def _releaseLock():
    if _lock:
        _lock.release()

The _lock object is a reentrant lock that sits in the global namespace of the logging/__init__.py module. It makes pretty much every object and operation in the entire logging package thread-safe, enabling threads to do read and write operations without the threat of a race condition. You can see in the module source code that _acquireLock() and _releaseLock() are ubiquitous to the module and its classes.

There’s something not accounted for here, though: what about process safety? The short answer is that the logging module is not process safe. This isn’t inherently a fault of logging—generally, two processes can’t write to same file without a lot of proactive effort on behalf of the programmer first.

This means that you’ll want to be careful before using classes such as a logging.FileHandler with multiprocessing involved. If two processes want to read from and write to the same underlying file concurrently, then you can run into a nasty bug halfway through a long-running routine.

If you want to get around this limitation, there’s a thorough recipe in the official Logging Cookbook. Because this entails a decent amount of setup, one alternative is to have each process log to a separate file based on its process ID, which you can grab with os.getpid().

Package Architecture: Logging’s MRO

Now that we’ve covered some preliminary setup code, let’s take a high-level look at how logging is laid out. The logging package uses a healthy dose of OOP and inheritance. Here’s a partial look at the method resolution order (MRO) for some of the most important classes in the package:

object
│
├── LogRecord
├── Filterer
│   ├── Logger
│   │   └── RootLogger
│   └── Handler
│       ├── StreamHandler
│       └── NullHandler
├── Filter
└── Manager

The tree diagram above doesn’t cover all of the classes in the module, just those that are most worth highlighting.

Note: You can use the dunder attribute logging.StreamHandler.__mro__ to see the chain of inheritance. A definitive guide to the MRO can be found in the Python 2 docs, though it is applicable to Python 3 as well.

This litany of classes is typically one source of confusion because there’s a lot going on, and it’s all jargon-heavy. Filter versus Filterer? Logger versus Handler? It can be challenging to keep track of everything, much less visualize how it fits together. A picture is worth a thousand words, so here’s a diagram of a scenario where one logger with two handlers attached to it writes a log message with level logging.INFO:

Flow of logging objects (Image: Real Python)

In Python code, everything above would look like this:

import logging
import sys

logger = logging.getLogger("pylog")
logger.setLevel(logging.DEBUG)
h1 = logging.FileHandler(filename="/tmp/records.log")
h1.setLevel(logging.INFO)
h2 = logging.StreamHandler(sys.stderr)
h2.setLevel(logging.ERROR)
logger.addHandler(h1)
logger.addHandler(h2)
logger.info("testing %d.. %d.. %d..", 1, 2, 3)

There’s a more detailed map of this flow in the Logging HOWTO. What’s shown above is a simplified scenario.

Your code defines just one Logger instance, logger, along with two Handler instances, h1 and h2.

When you call logger.info("testing %d.. %d.. %d..", 1, 2, 3), the logger object serves as a filter because it also has a level associated with it. Only if the message level is severe enough will the logger do anything with the message. Because the logger has level DEBUG, and the message carries a higher INFO level, it gets the go-ahead to move on.

Internally, logger calls logger.makeRecord() to put the message string "testing %d.. %d.. %d.." and its arguments (1, 2, 3) into a bona fide class instance of a LogRecord, which is just a container for the message and its metadata.

The logger object looks around for its handlers (instances of Handler), which may be tied directly to logger itself or to its parents (a concept that we’ll touch on later). In this example, it finds two handlers:

One with level INFO that dumps log data to a file at /tmp/records.log
One that writes to sys.stderr but only if the incoming message is at level ERROR or higher

At this point, there’s another round of tests that kicks in. Because the LogRecord and its message only carry level INFO, the record gets written to Handler 1 (green arrow), but not to Handler 2’s stderr stream (red arrow). For Handlers, writing the LogRecord to their stream is called emitting it, which is captured in their .emit().

Next, let’s further dissect everything from above.

The `LogRecord` Class

What is a LogRecord? When you log a message, an instance of the LogRecord class is the object you send to be logged. It’s created for you by a Logger instance and encapsulates all the pertinent info about that event. Internally, it’s little more than a wrapper around a dict that contains attributes for the record. A Logger instance sends a LogRecord instance to zero or more Handler instances.

The LogRecord contains some metadata, such as the following:

A name
The creation time as a Unix timestamp
The message itself
Information on what function made the logging call

Here’s a peek into the metadata that it carries with it, which you can introspect by stepping through a logging.error() call with the pdb module:

>>>

>>> import logging
>>> import pdb

>>> def f(x):
...     logging.error("bad vibes")
...     return x / 0
... 
>>> pdb.run("f(1)")

After stepping through some higher-level functions, you end up at line 1517:

(Pdb) l
1514                     exc_info = (type(exc_info), exc_info, exc_info.__traceback__)
1515                 elif not isinstance(exc_info, tuple):
1516                     exc_info = sys.exc_info()
1517             record = self.makeRecord(self.name, level, fn, lno, msg, args,
1518                                      exc_info, func, extra, sinfo)
1519 ->          self.handle(record)
1520
1521         def handle(self, record):
1522             """
1523             Call the handlers for the specified record.
1524
(Pdb) from pprint import pprint
(Pdb) pprint(vars(record))
{'args': (),
 'created': 1550671851.660067,
 'exc_info': None,
 'exc_text': None,
 'filename': '<stdin>',
 'funcName': 'f',
 'levelname': 'ERROR',
 'levelno': 40,
 'lineno': 2,
 'module': '<stdin>',
 'msecs': 660.067081451416,
 'msg': 'bad vibes',
 'name': 'root',
 'pathname': '<stdin>',
 'process': 2360,
 'processName': 'MainProcess',
 'relativeCreated': 295145.5490589142,
 'stack_info': None,
 'thread': 4372293056,
 'threadName': 'MainThread'}

A LogRecord, internally, contains a trove of metadata that’s used in one way or another.

You’ll rarely need to deal with a LogRecord directly, since the Logger and Handler do this for you. It’s still worthwhile to know what information is wrapped up in a LogRecord, because this is where all that useful info, like the timestamp, come from when you see record log messages.

Note: Below the LogRecord class, you’ll also find the setLogRecordFactory(), getLogRecordFactory(), and makeLogRecord() factory functions. You won’t need these unless you want to use a custom class instead of LogRecord to encapsulate log messages and their metadata.

The `Logger` and `Handler` Classes

The Logger and Handler classes are both central to how logging works, and they interact with each other frequently. A Logger, a Handler, and a LogRecord each have a .level associated with them.

The Logger takes the LogRecord and passes it off to the Handler, but only if the effective level of the LogRecord is equal to or higher than that of the Logger. The same goes for the LogRecord versus Handler test. This is called level-based filtering, which Logger and Handler implement in slightly different ways.

In other words, there is an (at least) two-step test applied before the message that you log gets to go anywhere. In order to be fully passed from a logger to handler and then logged to the end stream (which could be sys.stdout, a file, or an email via SMTP), a LogRecord must have a level at least as high as both the logger and handler.

PEP 282 describes how this works:

Each Logger object keeps track of a log level (or threshold) that it is interested in, and discards log requests below that level. (Source)

So where does this level-based filtering actually occur for both Logger and Handler?

For the Logger class, it’s a reasonable first assumption that the logger would compare its .level attribute to the level of the LogRecord, and be done there. However, it’s slightly more involved than that.

Level-based filtering for loggers occurs in .isEnabledFor(), which in turn calls .getEffectiveLevel(). Always use logger.getEffectiveLevel() rather than just consulting logger.level. The reason has to do with the organization of Logger objects in a hierarchical namespace. (You’ll see more on this later.)

By default, a Logger instance has a level of 0 (NOTSET). However, loggers also have parent loggers, one of which is the root logger, which functions as the parent of all other loggers. A Logger will walk upwards in its hierarchy and get its effective level vis-à-vis its parent (which ultimately may be root if no other parents are found).

Here’s where this happens in the Logger class:

class Logger(Filterer):
    # ...
    def getEffectiveLevel(self):
        logger = self
        while logger:
            if logger.level:
                return logger.level
            logger = logger.parent
        return NOTSET

    def isEnabledFor(self, level):
        try:
            return self._cache[level]
        except KeyError:
            _acquireLock()
            if self.manager.disable >= level:
                is_enabled = self._cache[level] = False
            else:
                is_enabled = self._cache[level] = level >= self.getEffectiveLevel()
            _releaseLock()
        return is_enabled

Correspondingly, here’s an example that calls the source code you see above:

>>>

>>> import logging
>>> logger = logging.getLogger("app")
>>> logger.level  # No!
0
>>> logger.getEffectiveLevel()
30
>>> logger.parent
<RootLogger root (WARNING)>
>>> logger.parent.level
30

Here’s the takeaway: don’t rely on .level. If you haven’t explicitly set a level on your logger object, and you’re depending on .level for some reason, then your logging setup will likely behave differently than you expected it to.

What about Handler? For handlers, the level-to-level comparison is simpler, though it actually happens in .callHandlers() from the Logger class:

class Logger(Filterer):
    # ...
    def callHandlers(self, record):
        c = self
        found = 0
        while c:
            for hdlr in c.handlers:
                found = found + 1
                if record.levelno >= hdlr.level:
                    hdlr.handle(record)

For a given LogRecord instance (named record in the source code above), a logger checks with each of its registered handlers and does a quick check on the .level attribute of that Handler instance. If the .levelno of the LogRecord is greater than or equal to that of the handler, only then does the record get passed on. A docstring in logging refers to this as “conditionally emit[ting] the specified logging record.”

The most important attribute for a Handler subclass instance is its .stream attribute. This is the final destination where logs get written to and can be pretty much any file-like object. Here’s an example with io.StringIO, which is an in-memory stream (buffer) for text I/O.

First, set up a Logger instance with a level of DEBUG. You’ll see that, by default, it has no direct handlers:

>>>

>>> import io
>>> import logging
>>> logger = logging.getLogger("abc")
>>> logger.setLevel(logging.DEBUG)
>>> print(logger.handlers)
[]

Next, you can subclass logging.StreamHandler to make the .flush() call a no-op. We would want to flush sys.stderr or sys.stdout, but not the in-memory buffer in this case:

class IOHandler(logging.StreamHandler):
    def flush(self):
        pass  # No-op

Now, declare the buffer object itself and tie it in as the .stream for your custom handler with a level of INFO, and then tie that handler into the logger:

>>>

>>> stream = io.StringIO()
>>> h = IOHandler(stream)
>>> h.setLevel(logging.INFO)
>>> logger.addHandler(h)

>>> logger.debug("extraneous info")
>>> logger.warning("you've been warned")
>>> logger.critical("SOS")

>>> try:
...     print(stream.getvalue())
... finally:
...     stream.close()
... 
you've been warned
SOS

This last chunk is another illustration of level-based filtering.

Three messages with levels DEBUG, WARNING, and CRITICAL are passed through the chain. At first, it may look as if they don’t go anywhere, but two of them do. All three of them make it out of the gates from logger (which has level DEBUG).

However, only two of them get emitted by the handler because it has a higher level of INFO, which exceeds DEBUG. Finally, you get the entire contents of the buffer as a str and close the buffer to explicitly free up system resources.

The `Filter` and `Filterer` Classes

Above, we asked the question, “Where does level-based filtering happen?” In answering this question, it’s easy to get distracted by the Filter and Filterer classes. Paradoxically, level-based filtering for Logger and Handler instances occurs without the help of either of the Filter or Filterer classes.

Filter and Filterer are designed to let you add additional function-based filters on top of the level-based filtering that is done by default. I like to think of it as à la carte filtering.

Filterer is the base class for Logger and Handler because both of these classes are eligible for receiving additional custom filters that you specify. You add instances of Filter to them with logger.addFilter() or handler.addFilter(), which is what self.filters refers to in the following method:

class Filterer(object):
    # ...
    def filter(self, record):
        rv = True
        for f in self.filters:
            if hasattr(f, 'filter'):
                result = f.filter(record)
            else:
                result = f(record)
            if not result:
                rv = False
                break
        return rv

Given a record (which is a LogRecord instance), .filter() returns True or False depending on whether that record gets the okay from this class’s filters.

Here is .handle() in turn, for the Logger and Handler classes:

class Logger(Filterer):
    # ...
    def handle(self, record):
        if (not self.disabled) and self.filter(record):
            self.callHandlers(record)

# ...

class Handler(Filterer):
    # ...
    def handle(self, record):
        rv = self.filter(record)
        if rv:
            self.acquire()
            try:
                self.emit(record)
            finally:
                self.release()
        return rv

Neither Logger nor Handler come with any additional filters by default, but here’s a quick example of how you could add one:

>>>

>>> import logging

>>> logger = logging.getLogger("rp")
>>> logger.setLevel(logging.INFO)
>>> logger.addHandler(logging.StreamHandler())
>>> logger.filters  # Initially empty
[]
>>> class ShortMsgFilter(logging.Filter):
...     """Only allow records that contain long messages (> 25 chars)."""
...     def filter(self, record):
...         msg = record.msg
...         if isinstance(msg, str):
...             return len(msg) > 25
...         return False
... 
>>> logger.addFilter(ShortMsgFilter())
>>> logger.filters
[<__main__.ShortMsgFilter object at 0x10c28b208>]
>>> logger.info("Reeeeaaaaallllllly long message")  # Length: 31
Reeeeaaaaallllllly long message
>>> logger.info("Done")  # Length: <25, no output

Above, you define a class ShortMsgFilter and override its .filter(). In .addHandler(), you could also just pass a callable, such as a function or lambda or a class that defines .__call__().

The `Manager` Class

There’s one more behind-the-scenes actor of logging that is worth touching on: the Manager class. What matters most is not the Manager class but a single instance of it that acts as a container for the growing hierarchy of loggers that are defined across packages. You’ll see in the next section how just a single instance of this class is central to gluing the module together and allowing its parts to talk to each other.

The All-Important Root Logger

When it comes to Logger instances, one stands out. It’s called the root logger:

class RootLogger(Logger):
    def __init__(self, level):
        Logger.__init__(self, "root", level)

# ...

root = RootLogger(WARNING)
Logger.root = root
Logger.manager = Manager(Logger.root)

The last three lines of this code block are one of the ingenious tricks employed by the logging package. Here are a few points:

The root logger is just a no-frills Python object with the identifier root. It has a level of logging.WARNING and a .name of "root". As far as the class RootLogger is concerned, this unique name is all that’s special about it.
The root object in turn becomes a class attribute for the Logger class. This means that all instances of Logger, and the Logger class itself, all have a .root attribute that is the root logger. This is another example of a singleton-like pattern being enforced in the logging package.
A Manager instance is set as the .manager class attribute for Logger. This eventually comes into play in logging.getLogger("name"). The .manager does all the facilitation of searching for existing loggers with the name "name" and creating them if they don’t exist.

The Logger Hierarchy

Everything is a child of root in the logger namespace, and I mean everything. That includes loggers that you specify yourself as well as those from third-party libraries that you import.

Remember earlier how the .getEffectiveLevel() for our logger instances was 30 (WARNING) even though we had not explicitly set it? That’s because the root logger sits at the top of the hierarchy, and its level is a fallback if any nested loggers have a null level of NOTSET:

>>>

>>> root = logging.getLogger()  # Or getLogger("")
>>> root
<RootLogger root (WARNING)>
>>> root.parent is None
True
>>> root.root is root  # Self-referential
True
>>> root is logging.root
True
>>> root.getEffectiveLevel()
30

The same logic applies to the search for a logger’s handlers. The search is effectively a reverse-order search up the tree of a logger’s parents.

A Multi-Handler Design

The logger hierarchy may seem neat in theory, but how beneficial is it in practice?

Let’s take a break from exploring the logging code and foray into writing our own mini-application—one that takes advantage of the logger hierarchy in a way that reduces boilerplate code and keeps things scalable if the project’s codebase grows.

Here’s the project structure:

project/
│
└── project/
    ├── __init__.py
    ├── utils.py
    └── base.py

Don’t worry about the application’s main functions in utils.py and base.py. What we’re paying more attention to here is the interaction in logging objects between the modules in project/.

In this case, say that you want to design a multipronged logging setup:

Each module gets a logger with multiple handlers.
Some of the handlers are shared between different logger instances in different modules. These handlers only care about level-based filtering, not the module where the log record emanated from. There is a handler for DEBUG messages, one for INFO, one for WARNING, and so on.
Each logger is also tied to one more additional handler that only receives LogRecord instances from that lone logger. You can call this a module-based file handler.

Visually, what we’re shooting for would look something like this:

A multipronged logging design (Image: Real Python)

The two turquoise objects are instances of Logger, established with logging.getLogger(__name__) for each module in a package. Everything else is a Handler instance.

The thinking behind this design is that it’s neatly compartmentalized. You can conveniently look at messages coming from a single logger, or look at messages of a certain level and above coming from any logger or module.

The properties of the logger hierarchy make it suitable for setting up this multipronged logger-handler layout. What does that mean? Here’s a concise explanation from the Django documentation:

Why is the hierarchy important? Well, because loggers can be set to propagate their logging calls to their parents. In this way, you can define a single set of handlers at the root of a logger tree, and capture all logging calls in the subtree of loggers. A logging handler defined in the project namespace will catch all logging messages issued on the project.interesting and project.interesting.stuff loggers. (Source)

The term propagate refers to how a logger keeps walking up its chain of parents looking for handlers. The .propagate attribute is True for a Logger instance by default:

>>>

>>> logger = logging.getLogger(__name__)
>>> logger.propagate
True

In .callHandlers(), if propagate is True, each successive parent gets reassigned to the local variable c until the hierarchy is exhausted:

class Logger(Filterer):
    # ...
    def callHandlers(self, record):
        c = self
        found = 0
        while c:
            for hdlr in c.handlers:
                found = found + 1
                if record.levelno >= hdlr.level:
                    hdlr.handle(record)
            if not c.propagate:
                c = None
            else:
                c = c.parent

Here’s what this means: because the __name__ dunder variable within a package’s __init__.py module is just the name of the package, a logger there becomes a parent to any loggers present in other modules in the same package.

Here are the resulting .name attributes from assigning to logger with logging.getLogger(__name__):

Module	`.name` Attribute
`project/__init__.py`	`'project'`
`project/utils.py`	`'project.utils'`
`project/base.py`	`'project.base'`

Because the 'project.utils' and 'project.base' loggers are children of 'project', they will latch onto not only their own direct handlers but whatever handlers are attached to 'project'.

Let’s build out the modules. First comes __init__.py:

# __init__.py
import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

levels = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
for level in levels:
    handler = logging.FileHandler(f"/tmp/level-{level.lower()}.log")
    handler.setLevel(getattr(logging, level))
    logger.addHandler(handler)

def add_module_handler(logger, level=logging.DEBUG):
    handler = logging.FileHandler(
        f"/tmp/module-{logger.name.replace('.', '-')}.log"
    )
    handler.setLevel(level)
    logger.addHandler(handler)

This module is imported when the project package is imported. You add a handler for each level in DEBUG through CRITICAL, then attach it to a single logger at the top of the hierarchy.

You also define a utility function that adds one more FileHandler to a logger, where the filename of the handler corresponds to the module name where the logger is defined. (This assumes the logger is defined with __name__.)

You can then add some minimal boilerplate logger setup in base.py and utils.py. Notice that you only need to add one additional handler with add_module_handler() from __init__.py. You don’t need to worry about the level-oriented handlers because they are already added to their parent logger named 'project':

# base.py
import logging

from project import add_module_handler

logger = logging.getLogger(__name__)
add_module_handler(logger)

def func1():
    logger.debug("debug called from base.func1()")
    logger.critical("critical called from base.func1()")

Here’s utils.py:

# utils.py
import logging

from project import add_module_handler

logger = logging.getLogger(__name__)
add_module_handler(logger)

def func2():
    logger.debug("debug called from utils.func2()")
    logger.critical("critical called from utils.func2()")

Let’s see how all of this works together from a fresh Python session:

>>>

>>> from pprint import pprint
>>> import project
>>> from project import base, utils

>>> project.logger
<Logger project (DEBUG)>
>>> base.logger, utils.logger
(<Logger project.base (DEBUG)>, <Logger project.utils (DEBUG)>)
>>> base.logger.handlers
[<FileHandler /tmp/module-project-base.log (DEBUG)>]
>>> pprint(base.logger.parent.handlers)
[<FileHandler /tmp/level-debug.log (DEBUG)>,
 <FileHandler /tmp/level-info.log (INFO)>,
 <FileHandler /tmp/level-warning.log (WARNING)>,
 <FileHandler /tmp/level-error.log (ERROR)>,
 <FileHandler /tmp/level-critical.log (CRITICAL)>]
>>> base.func1()
>>> utils.func2()

You’ll see in the resulting log files that our filtration system works as intended. Module-oriented handlers direct one logger to a specific file, while level-oriented handlers direct multiple loggers to a different file:

$ cat /tmp/level-debug.log 
debug called from base.func1()
critical called from base.func1()
debug called from utils.func2()
critical called from utils.func2()

$ cat /tmp/level-critical.log 
critical called from base.func1()
critical called from utils.func2()

$ cat /tmp/module-project-base.log
debug called from base.func1()
critical called from base.func1()

$ cat /tmp/module-project-utils.log 
debug called from utils.func2()
critical called from utils.func2()

A drawback worth mentioning is that this design introduces a lot of redundancy. One LogRecord instance may go to no less than six files. That’s also a non-negligible amount of file I/O that may add up in a performance-critical application.

Now that you’ve seen a practical example, let’s switch gears and delve into a possible source of confusion in logging.

The “Why Didn’t My Log Message Go Anywhere?” Dilemma

There are two common situations with logging when it’s easy to get tripped up:

You logged a message that seemingly went nowhere, and you’re not sure why.
Instead of being suppressed, a log message appeared in a place that you didn’t expect it to.

Each of these has a reason or two commonly associated with it.

You logged a message that seemingly went nowhere, and you’re not sure why.

Don’t forget that the effective level of a logger for which you don’t otherwise set a custom level is WARNING, because a logger will walk up its hierarchy until it finds the root logger with its own WARNING level:

>>>

>>> import logging
>>> logger = logging.getLogger("xyz")
>>> logger.debug("mind numbing info here")
>>> logger.critical("storm is coming")
storm is coming

Because of this default, the .debug() call goes nowhere.

Instead of being suppressed, a log message appeared in a place that you didn’t expect it to.

When you defined your logger above, you didn’t add any handlers to it. So, why is it writing to the console?

The reason for this is that logging sneakily uses a handler called lastResort that writes to sys.stderr if no other handlers are found:

class _StderrHandler(StreamHandler):
    # ...
    @property
    def stream(self):
        return sys.stderr

_defaultLastResort = _StderrHandler(WARNING)
lastResort = _defaultLastResort

This kicks in when a logger goes to find its handlers:

class Logger(Filterer):
    # ...
    def callHandlers(self, record):
        c = self
        found = 0
        while c:
            for hdlr in c.handlers:
                found = found + 1
                if record.levelno >= hdlr.level:
                    hdlr.handle(record)
            if not c.propagate:
                c = None
            else:
                c = c.parent
        if (found == 0):
            if lastResort:
                if record.levelno >= lastResort.level:
                     lastResort.handle(record)

If the logger gives up on its search for handlers (both its own direct handlers and attributes of parent loggers), then it picks up the lastResort handler and uses that.

There’s one more subtle detail worth knowing about. This section has largely talked about the instance methods (methods that a class defines) rather than the module-level functions of the logging package that carry the same name.

If you use the functions, such as logging.info() rather than logger.info(), then something slightly different happens internally. The function calls logging.basicConfig(), which adds a StreamHandler that writes to sys.stderr. In the end, the behavior is virtually the same:

>>>

>>> import logging
>>> root = logging.getLogger("")
>>> root.handlers
[]
>>> root.hasHandlers()
False
>>> logging.basicConfig()
>>> root.handlers
[<StreamHandler <stderr> (NOTSET)>]
>>> root.hasHandlers()
True

Taking Advantage of Lazy Formatting

It’s time to switch gears and take a closer look at how messages themselves are joined with their data. While it’s been supplanted by str.format() and f-strings, you’ve probably used Python’s percent-style formatting to do something like this:

>>>

>>> print("To iterate is %s, to recurse %s" % ("human", "divine"))
To iterate is human, to recurse divine

As a result, you may be tempted to do the same thing in a logging call:

>>>

>>> # Bad!  Check out a more efficient alternative below.
>>> logging.warning("To iterate is %s, to recurse %s" % ("human", "divine"))
WARNING:root:To iterate is human, to recurse divine

This uses the entire format string and its arguments as the msg argument to logging.warning().

Here is the recommended alternative, straight from the logging docs:

>>>

>>> # Better: formatting doesn't occur until it really needs to.
>>> logging.warning("To iterate is %s, to recurse %s", "human", "divine")
WARNING:root:To iterate is human, to recurse divine

It looks a little weird, right? This seems to defy the conventions of how percent-style string formatting works, but it’s a more efficient function call because the format string gets formatted lazily rather than greedily. Here’s what that means.

The method signature for Logger.warning() looks like this:

def warning(self, msg, *args, **kwargs)

The same applies to the other methods, such as .debug(). When you call warning("To iterate is %s, to recurse %s", "human", "divine"), both "human" and "divine" get caught as *args and, within the scope of the method’s body, args is equal to ("human", "divine").

Contrast this to the first call above:

logging.warning("To iterate is %s, to recurse %s" % ("human", "divine"))

In this form, everything in the parentheses gets immediately merged together into "To iterate is human, to recurse divine" and passed as msg, while args is an empty tuple.

Why does this matter? Repeated logging calls can degrade runtime performance slightly, but the logging package does its very best to control that and keep it in check. By not merging the format string with its arguments right away, logging is delaying the string formatting until the LogRecord is requested by a Handler.

This happens in LogRecord.getMessage(), so only after logging deems that the LogRecord will actually be passed to a handler does it become its fully merged self.

All that is to say that the logging package makes some very fine-tuned performance optimizations in the right places. This may seem like minutia, but if you’re making the same logging.debug() call a million times inside a loop, and the args are function calls, then the lazy nature of how logging does string formatting can make a difference.

Before doing any merging of msg and args, a Logger instance will check its .isEnabledFor() to see if that merging should be done in the first place.

Functions vs Methods

Towards the bottom of logging/__init__.py sit the module-level functions that are advertised up front in the public API of logging. You already saw the Logger methods such as .debug(), .info(), and .warning(). The top-level functions are wrappers around the corresponding methods of the same name, but they have two important features:

They always call their corresponding method from the root logger, root.
Before calling the root logger methods, they call logging.basicConfig() with no arguments if root doesn’t have any handlers. As you saw earlier, it is this call that sets a sys.stdout handler for the root logger.

For illustration, here’s logging.error():

def error(msg, *args, **kwargs):
    if len(root.handlers) == 0:
        basicConfig()
    root.error(msg, *args, **kwargs)

You’ll find the same pattern for logging.debug(), logging.info(), and the others as well. Tracing the chain of commands is interesting. Eventually, you’ll end up at the same place, which is where the internal Logger._log() is called.

The calls to debug(), info(), warning(), and the other level-based functions all route to here. _log() primarily has two purposes:

Call self.makeRecord(): Make a LogRecord instance from the msg and other arguments you pass to it.
Call self.handle(): This determines what actually gets done with the record. Where does it get sent? Does it make it there or get filtered out?

Here’s that entire process in one diagram:

Internals of a logging call (Image: Real Python)

You can also trace the call stack with pdb.

>>>

>>> import logging
>>> import pdb
>>> pdb.run('logging.warning("%s-%s", "uh", "oh")')
> <string>(1)<module>()
(Pdb) s
--Call--
> lib/python3.7/logging/__init__.py(1971)warning()
-> def warning(msg, *args, **kwargs):
(Pdb) s
> lib/python3.7/logging/__init__.py(1977)warning()
-> if len(root.handlers) == 0:
(Pdb) unt
> lib/python3.7/logging/__init__.py(1978)warning()
-> basicConfig()
(Pdb) unt
> lib/python3.7/logging/__init__.py(1979)warning()
-> root.warning(msg, *args, **kwargs)
(Pdb) s
--Call--
> lib/python3.7/logging/__init__.py(1385)warning()
-> def warning(self, msg, *args, **kwargs):
(Pdb) l
1380             logger.info("Houston, we have a %s", "interesting problem", exc_info=1)
1381             """
1382             if self.isEnabledFor(INFO):
1383                 self._log(INFO, msg, args, **kwargs)
1384     
1385 ->        def warning(self, msg, *args, **kwargs):
1386             """
1387             Log 'msg % args' with severity 'WARNING'.
1388     
1389             To pass exception information, use the keyword argument exc_info with
1390             a true value, e.g.
(Pdb) s
> lib/python3.7/logging/__init__.py(1394)warning()
-> if self.isEnabledFor(WARNING):
(Pdb) unt
> lib/python3.7/logging/__init__.py(1395)warning()
-> self._log(WARNING, msg, args, **kwargs)
(Pdb) s
--Call--
> lib/python3.7/logging/__init__.py(1496)_log()
-> def _log(self, level, msg, args, exc_info=None, extra=None, stack_info=False):
(Pdb) s
> lib/python3.7/logging/__init__.py(1501)_log()
-> sinfo = None
(Pdb) unt 1517
> lib/python3.7/logging/__init__.py(1517)_log()
-> record = self.makeRecord(self.name, level, fn, lno, msg, args,
(Pdb) s
> lib/python3.7/logging/__init__.py(1518)_log()
-> exc_info, func, extra, sinfo)
(Pdb) s
--Call--
> lib/python3.7/logging/__init__.py(1481)makeRecord()
-> def makeRecord(self, name, level, fn, lno, msg, args, exc_info,
(Pdb) p name
'root'
(Pdb) p level
30
(Pdb) p msg
'%s-%s'
(Pdb) p args
('uh', 'oh')
(Pdb) up
> lib/python3.7/logging/__init__.py(1518)_log()
-> exc_info, func, extra, sinfo)
(Pdb) unt
> lib/python3.7/logging/__init__.py(1519)_log()
-> self.handle(record)
(Pdb) n
WARNING:root:uh-oh

What Does `getLogger()` Really Do?

Also hiding in this section of the source code is the top-level getLogger(), which wraps Logger.manager.getLogger():

def getLogger(name=None):
    if name:
        return Logger.manager.getLogger(name)
    else:
        return root

This is the entry point for enforcing the singleton logger design:

If you specify a name, then the underlying .getLogger() does a dict lookup on the string name. What this comes down to is a lookup in the loggerDict of logging.Manager. This is a dictionary of all registered loggers, including the intermediate PlaceHolder instances that are generated when you reference a logger far down in the hierarchy before referencing its parents.
Otherwise, root is returned. There is only one root—the instance of RootLogger discussed above.

This feature is what lies behind a trick that can let you peek into all of the registered loggers:

>>>

>>> import logging
>>> logging.Logger.manager.loggerDict
{}

>>> from pprint import pprint
>>> import asyncio
>>> pprint(logging.Logger.manager.loggerDict)
{'asyncio': <Logger asyncio (WARNING)>,
 'concurrent': <logging.PlaceHolder object at 0x10d153710>,
 'concurrent.futures': <Logger concurrent.futures (WARNING)>}

Whoa, hold on a minute. What’s happening here? It looks like something changed internally to the logging package as a result of an import of another library, and that’s exactly what happened.

Firstly, recall that Logger.manager is a class attribute, where an instance of Manager is tacked onto the Logger class. The manager is designed to track and manage all of the singleton instances of Logger. These are housed in .loggerDict.

Now, when you initially import logging, this dictionary is empty. But after you import asyncio, the same dictionary gets populated with three loggers. This is an example of one module setting the attributes of another module in-place. Sure enough, inside of asyncio/log.py, you’ll find the following:

import logging

logger = logging.getLogger(__package__)  # "asyncio"

The key-value pair is set in Logger.getLogger() so that the manager can oversee the entire namespace of loggers. This means that the object asyncio.log.logger gets registered in the logger dictionary that belongs to the logging package. Something similar happens in the concurrent.futures package as well, which is imported by asyncio.

You can see the power of the singleton design in an equivalence test:

>>>

>>> obj1 = logging.getLogger("asyncio")
>>> obj2 = logging.Logger.manager.loggerDict["asyncio"]
>>> obj1 is obj2
True

This comparison illustrates (glossing over a few details) what getLogger() ultimately does.

Library vs Application Logging: What Is `NullHandler`?

That brings us to the final hundred or so lines in the logging/__init__.py source, where NullHandler is defined. Here’s the definition in all its glory:

class NullHandler(Handler):
    def handle(self, record):
        pass

    def emit(self, record):
        pass

    def createLock(self):
        self.lock = None

The NullHandler is all about the distinctions between logging in a library versus an application. Let’s see what that means.

A library is an extensible, generalizable Python package that is intended for other users to install and set up. It is built by a developer with the express purpose of being distributed to users. Examples include popular open-source projects like NumPy, dateutil, and cryptography.

An application (or app, or program) is designed for a more specific purpose and a much smaller set of users (possibly just one user). It’s a program or set of programs highly tailored by the user to do a limited set of things. An example of an application is a Django app that sits behind a web page. Applications commonly use (import) libraries and the tools they contain.

When it comes to logging, there are different best practices in a library versus an app.

That’s where NullHandler fits in. It’s basically a do-nothing stub class.

If you’re writing a Python library, you really need to do this one minimalist piece of setup in your package’s __init__.py:

# Place this in your library's uppermost `__init__.py`
# Nothing else!

import logging

logging.getLogger(__name__).addHandler(NullHandler())

This serves two critical purposes.

Firstly, a library logger that is declared with logger = logging.getLogger(__name__) (without any further configuration) will log to sys.stderr by default, even if that’s not what the end user wants. This could be described as an opt-out approach, where the end user of the library has to go in and disable logging to their console if they don’t want it.

Common wisdom says to use an opt-in approach instead: don’t emit any log messages by default, and let the end users of the library determine if they want to further configure the library’s loggers and add handlers to them. Here’s that philosophy worded more bluntly by the author of the logging package, Vinay Sajip:

A third party library which uses logging should not spew logging output by default which may not be wanted by a developer/user of an application which uses it. (Source)

This leaves it up to the library user, not library developer, to incrementally call methods such as logger.addHandler() or logger.setLevel().

The second reason that NullHandler exists is more archaic. In Python 2.7 and earlier, trying to log a LogRecord from a logger that has no handler set would raise a warning. Adding the no-op class NullHandler will avert this.

Here’s what specifically happens in the line logging.getLogger(__name__).addHandler(NullHandler()) from above:

Python gets (creates) the Logger instance with the same name as your package. If you’re designing the calculus package, within __init__.py, then __name__ will be equal to 'calculus'.
A NullHandler instance gets attached to this logger. That means that Python will not default to using the lastResort handler.

Keep in mind that any logger created in any of the other .py modules of the package will be children of this logger in the logger hierarchy and that, because this handler also belongs to them, they won’t need to use the lastResort handler and won’t default to logging to standard error (stderr).

As a quick example, let’s say your library has the following structure:

calculus/
│
├── __init__.py
└── integration.py

In integration.py, as the library developer you are free to do the following:

# calculus/integration.py
import logging

logger = logging.getLogger(__name__)

def func(x):
    logger.warning("Look!")
    # Do stuff
    return None

Now, a user comes along and installs your library from PyPI via pip install calculus. They use from calculus.integration import func in some application code. This user is free to manipulate and configure the logger object from the library like any other Python object, to their heart’s content.

What Logging Does With Exceptions

One thing that you may be wary of is the danger of exceptions that stem from your calls to logging. If you have a logging.error() call that is designed to give you some more verbose debugging information, but that call itself for some reason raises an exception, that would be the height of irony, right?

Cleverly, if the logging package encounters an exception that has to do with logging itself, then it will print the traceback, but not raise the exception itself.

Here’s an example that deals with a common typo: passing two arguments to a format string that is only expecting one argument. The important distinction is that what you see below is not an exception being raised, but rather a prettified printed traceback of the internal exception, which itself was suppressed:

>>>

>>> logging.critical("This %s has too many arguments", "msg", "other")
--- Logging error ---
Traceback (most recent call last):
  File "lib/python3.7/logging/__init__.py", line 1034, in emit
    msg = self.format(record)
  File "lib/python3.7/logging/__init__.py", line 880, in format
    return fmt.format(record)
  File "lib/python3.7/logging/__init__.py", line 619, in format
    record.message = record.getMessage()
  File "lib/python3.7/logging/__init__.py", line 380, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "<stdin>", line 1, in <module>
Message: 'This %s has too many arguments'
Arguments: ('msg', 'other')

This lets your program gracefully carry on with its actual program flow. The rationale is that you wouldn’t want an uncaught exception to come from a logging call itself and stop a program dead in its tracks.

Tracebacks can be messy, but this one is informative and relatively straightforward. What enables the suppression of exceptions related to logging is Handler.handleError(). When the handler calls .emit(), which is the method where it attempts to log the record, it falls back to .handleError() if something goes awry. Here’s the implementation of .emit() for the StreamHandler class:

def emit(self, record):
    try:
        msg = self.format(record)
        stream = self.stream
        stream.write(msg + self.terminator)
        self.flush()
    except Exception:
        self.handleError(record)

Any exception related to the formatting and writing gets caught rather than being raised, and handleError gracefully writes the traceback to sys.stderr.

Logging Python Tracebacks

Speaking of exceptions and their tracebacks, what about cases where your program encounters them but should log the exception and keep chugging along in its execution?

Let’s walk through a couple of ways to do this.

Here’s a contrived example of a lottery simulator using code that isn’t Pythonic on purpose. You’re developing an online lottery game where users can wager on their lucky number:

import random

class Lottery(object):
    def __init__(self, n):
        self.n = n

    def make_tickets(self):
        for i in range(self.n):
            yield i

    def draw(self):
        pool = self.make_tickets()
        random.shuffle(pool)
        return next(pool)

Behind the frontend application sits the critical code below. You want to make sure that you keep track of any errors caused by the site that may make a user lose their money. The first (suboptimal) way is to use logging.error() and log the str form of the exception instance itself:

try:
    lucky_number = int(input("Enter your ticket number: "))
    drawn = Lottery(n=20).draw()
    if lucky_number == drawn:
        print("Winner chicken dinner!")
except Exception as e:
    # NOTE: See below for a better way to do this.
    logging.error("Could not draw ticket: %s", e)

This will only get you the actual exception message, rather than the traceback. You check the logs on your website’s server and find this cryptic message:

ERROR:root:Could not draw ticket: object of type 'generator' has no len()

Hmm. As the application developer, you’ve got a serious problem, and a user got ripped off as a result. But maybe this exception message itself isn’t very informative. Wouldn’t it be nice to see the lineage of the traceback that led to this exception?

The proper solution is to use logging.exception(), which logs a message with level ERROR and also displays the exception traceback. Replace the two final lines above with these:

except Exception:
    logging.exception("Could not draw ticket")

Now you get a better indication of what’s going on:

>>>

ERROR:root:Could not draw ticket
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "<stdin>", line 9, in draw
  File "lib/python3.7/random.py", line 275, in shuffle
    for i in reversed(range(1, len(x))):
TypeError: object of type 'generator' has no len()

Using exception() saves you from having to reference the exception yourself because logging pulls it in with sys.exc_info().

This makes things clearer that the problem stems from random.shuffle(), which needs to know the length of the object it is shuffling. Because our Lottery class passes a generator to shuffle(), it gets held up and raises before the pool can be shuffled, much less generate a winning ticket.

In large, full-blown applications, you’ll find logging.exception() to be even more useful when deep, multi-library tracebacks are involved, and you can’t step into them with a live debugger like pdb.

The code for logging.Logger.exception(), and hence logging.exception(), is just a single line:

def exception(self, msg, *args, exc_info=True, **kwargs):
    self.error(msg, *args, exc_info=exc_info, **kwargs)

That is, logging.exception() just calls logging.error() with exc_info=True, which is otherwise False by default. If you want to log an exception traceback but at a level different than logging.ERROR, just call that function or method with exc_info=True.

Keep in mind that exception() should only be called in the context of an exception handler, inside of an except block:

for i in data:
    try:
        result = my_longwinded_nested_function(i)
    except ValueError:
        # We are in the context of exception handler now.
        # If it's unclear exactly *why* we couldn't process
        # `i`, then log the traceback and move on rather than
        # ditching completely.
        logger.exception("Could not process %s", i)
        continue

Use this pattern sparingly rather than as a means to suppress any exception. It can be most helpful when you’re debugging a long function call stack where you’re otherwise seeing an ambiguous, unclear, and hard-to-track error.

Conclusion

Pat yourself on the back, because you’ve just walked through almost 2,000 lines of dense source code. You’re now better equipped to deal with the logging package!

Keep in mind that this tutorial has been far from exhaustive in covering all of the classes found in the logging package. There’s even more machinery that glues everything together. If you’d like to learn more, then you can look into the Formatter classes and the separate modules logging/config.py and logging/handlers.py.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Interactive Data Visualization in Python With Bokeh

2019-05-21T14:00:00+00:00

Bokeh prides itself on being a library for interactive data visualization. The graphics are rendered using HTML and JavaScript, and your visualizations are easy to share as an HTML page. You will create a number of visualizations based on a real-world dataset.

The goal of this course is to get you up and running with Bokeh.

You will learn how to:

Transform your data into visualizations
Customize and organize your visualizations
Add interactivity to your visualizations

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Unicode & Character Encodings in Python: A Painless Guide

2019-05-20T14:00:00+00:00

Handling character encodings in Python or any other language can at times seem painful. Places such as Stack Overflow have thousands of questions stemming from confusion over exceptions like UnicodeDecodeError and UnicodeEncodeError. This tutorial is designed to clear the Exception fog and illustrate that working with text and binary data in Python 3 can be a smooth experience. Python’s Unicode support is strong and robust, but it takes some time to master.

This tutorial is different because it’s not language-agnostic but instead deliberately Python-centric. You’ll still get a language-agnostic primer, but you’ll then dive into illustrations in Python, with text-heavy paragraphs kept to a minimum. You’ll see how to use concepts of character encodings in live Python code.

By the end of this tutorial, you’ll:

Get conceptual overviews on character encodings and numbering systems
Understand how encoding comes into play with Python’s str and bytes
Know about support in Python for numbering systems through its various forms of int literals
Be familiar with Python’s built-in functions related to character encodings and numbering systems

Character encoding and numbering systems are so closely connected that they need to be covered in the same tutorial or else the treatment of either would be totally inadequate.

Note: This article is Python 3-centric. Specifically, all code examples in this tutorial were generated from a CPython 3.7.2 shell, although all minor versions of Python 3 should behave (mostly) the same in their treatment of text.

If you’re still using Python 2 and are intimidated by the differences in how Python 2 and Python 3 treat text and binary data, then hopefully this tutorial will help you make the switch.

What’s a Character Encoding?

There are tens if not hundreds of character encodings. The best way to start understanding what they are is to cover one of the simplest character encodings, ASCII.

Whether you’re self-taught or have a formal computer science background, chances are you’ve seen an ASCII table once or twice. ASCII is a good place to start learning about character encoding because it is a small and contained encoding. (Too small, as it turns out.)

It encompasses the following:

Lowercase English letters: a through z
Uppercase English letters: A through Z
Some punctuation and symbols: "$" and "!", to name a couple
Whitespace characters: an actual space (" "), as well as a newline, carriage return, horizontal tab, vertical tab, and a few others
Some non-printable characters: characters such as backspace, "\b", that can’t be printed literally in the way that the letter A can

So what is a more formal definition of a character encoding?

At a very high level, it’s a way of translating characters (such as letters, punctuation, symbols, whitespace, and control characters) to integers and ultimately to bits. Each character can be encoded to a unique sequence of bits. Don’t worry if you’re shaky on the concept of bits, because we’ll get to them shortly.

The various categories outlined represent groups of characters. Each single character has a corresponding code point, which you can think of as just an integer. Characters are segmented into different ranges within the ASCII table:

Code Point Range	Class
0 through 31	Control/non-printable characters
32 through 64	Punctuation, symbols, numbers, and space
65 through 90	Uppercase English alphabet letters
91 through 96	Additional graphemes, such as `[` and `\`
97 through 122	Lowercase English alphabet letters
123 through 126	Additional graphemes, such as `{` and `\|`
127	Control/non-printable character (`DEL`)

The entire ASCII table contains 128 characters. This table captures the complete character set that ASCII permits. If you don’t see a character here, then you simply can’t express it as printed text under the ASCII encoding scheme.

Code Point	Character (Name)	Code Point	Character (Name)
0	NUL (Null)	64	`@`
1	SOH (Start of Heading)	65	`A`
2	STX (Start of Text)	66	`B`
3	ETX (End of Text)	67	`C`
4	EOT (End of Transmission)	68	`D`
5	ENQ (Enquiry)	69	`E`
6	ACK (Acknowledgment)	70	`F`
7	BEL (Bell)	71	`G`
8	BS (Backspace)	72	`H`
9	HT (Horizontal Tab)	73	`I`
10	LF (Line Feed)	74	`J`
11	VT (Vertical Tab)	75	`K`
12	FF (Form Feed)	76	`L`
13	CR (Carriage Return)	77	`M`
14	SO (Shift Out)	78	`N`
15	SI (Shift In)	79	`O`
16	DLE (Data Link Escape)	80	`P`
17	DC1 (Device Control 1)	81	`Q`
18	DC2 (Device Control 2)	82	`R`
19	DC3 (Device Control 3)	83	`S`
20	DC4 (Device Control 4)	84	`T`
21	NAK (Negative Acknowledgment)	85	`U`
22	SYN (Synchronous Idle)	86	`V`
23	ETB (End of Transmission Block)	87	`W`
24	CAN (Cancel)	88	`X`
25	EM (End of Medium)	89	`Y`
26	SUB (Substitute)	90	`Z`
27	ESC (Escape)	91	`[`
28	FS (File Separator)	92	`\`
29	GS (Group Separator)	93	`]`
30	RS (Record Separator)	94	`^`
31	US (Unit Separator)	95	`_`
32	SP (Space)	96	`
33	`!`	97	`a`
34	`"`	98	`b`
35	`#`	99	`c`
36	`$`	100	`d`
37	`%`	101	`e`
38	`&`	102	`f`
39	`'`	103	`g`
40	`(`	104	`h`
41	`)`	105	`i`
42	`*`	106	`j`
43	`+`	107	`k`
44	`,`	108	`l`
45	`-`	109	`m`
46	`.`	110	`n`
47	`/`	111	`o`
48	`0`	112	`p`
49	`1`	113	`q`
50	`2`	114	`r`
51	`3`	115	`s`
52	`4`	116	`t`
53	`5`	117	`u`
54	`6`	118	`v`
55	`7`	119	`w`
56	`8`	120	`x`
57	`9`	121	`y`
58	`:`	122	`z`
59	`;`	123	`{`
60	`<`	124	`\|`
61	`=`	125	`}`
62	`>`	126	`~`
63	`?`	127	DEL (delete)

The `string` Module

Python’s string module is a convenient one-stop-shop for string constants that fall in ASCII’s character set.

Here’s the core of the module in all its glory:

# From lib/python3.7/string.py

whitespace = ' \t\n\r\v\f'
ascii_lowercase = 'abcdefghijklmnopqrstuvwxyz'
ascii_uppercase = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
ascii_letters = ascii_lowercase + ascii_uppercase
digits = '0123456789'
hexdigits = digits + 'abcdef' + 'ABCDEF'
octdigits = '01234567'
punctuation = r"""!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~"""
printable = digits + ascii_letters + punctuation + whitespace

Most of these constants should be self-documenting in their identifier name. We’ll cover what hexdigits and octdigits are shortly.

You can use these constants for everyday string manipulation:

>>>

>>> import string

>>> s = "What's wrong with ASCII?!?!?"
>>> s.rstrip(string.punctuation)
'What's wrong with ASCII'

Note: string.printable includes all of string.whitespace. This disagrees slightly with another method for testing whether a character is considered printable, namely str.isprintable(), which will tell you that none of {'\v', '\n', '\r', '\f', '\t'} are considered printable.

The subtle difference is because of definition: str.isprintable() considers something printable if “all of its characters are considered printable in repr().”

A Bit of a Refresher

Now is a good time for a short refresher on the bit, the most fundamental unit of information that a computer knows.

A bit is a signal that has only two possible states. There are different ways of symbolically representing a bit that all mean the same thing:

0 or 1
“yes” or “no”
True or False
“on” or “off”

Our ASCII table from the previous section uses what you and I would just call numbers (0 through 127), but what are more precisely called numbers in base 10 (decimal).

You can also express each of these base-10 numbers with a sequence of bits (base 2). Here are the binary versions of 0 through 10 in decimal:

Decimal	Binary (Compact)	Binary (Padded Form)
0	0	00000000
1	1	00000001
2	10	00000010
3	11	00000011
4	100	00000100
5	101	00000101
6	110	00000110
7	111	00000111
8	1000	00001000
9	1001	00001001
10	1010	00001010

Notice that as the decimal number n increases, you need more significant bits to represent the character set up to and including that number.

Here’s a handy way to represent ASCII strings as sequences of bits in Python. Each character from the ASCII string gets pseudo-encoded into 8 bits, with spaces in between the 8-bit sequences that each represent a single character:

>>>

>>> def make_bitseq(s: str) -> str:
...     if not s.isascii():
...         raise ValueError("ASCII only allowed")
...     return " ".join(f"{ord(i):08b}" for i in s)

>>> make_bitseq("bits")
'01100010 01101001 01110100 01110011'

>>> make_bitseq("CAPS")
'01000011 01000001 01010000 01010011'

>>> make_bitseq("$25.43")
'00100100 00110010 00110101 00101110 00110100 00110011'

>>> make_bitseq("~5")
'01111110 00110101'

Note: .isascii() was introduced in Python 3.7.

The f-string f"{ord(i):08b}" uses Python’s Format Specification Mini-Language, which is a way of specifying formatting for replacement fields in format strings:

The left side of the colon, ord(i), is the actual object whose value will be formatted and inserted into the output. Using ord() gives you the base-10 code point for a single str character.
The right hand side of the colon is the format specifier. 08 means width 8, 0 padded, and the b functions as a sign to output the resulting number in base 2 (binary).

This trick is mainly just for fun, and it will fail very badly for any character that you don’t see present in the ASCII table. We’ll discuss how other encodings fix this problem later on.

We Need More Bits!

There’s a critically important formula that’s related to the definition of a bit. Given a number of bits, n, the number of distinct possible values that can be represented in n bits is 2ⁿ:

def n_possible_values(nbits: int) -> int:
    return 2 ** nbits

Here’s what that means:

1 bit will let you express 2¹ == 2 possible values.
8 bits will let you express 2⁸ == 256 possible values.
64 bits will let you express 2⁶⁴ == 18,446,744,073,709,551,616 possible values.

There’s a corollary to this formula: given a range of distinct possible values, how can we find the number of bits, n, that is required for the range to be fully represented? What you’re trying to solve for is n in the equation 2ⁿ = x (where you already know x).

Here’s what that works out to:

>>>

>>> from math import ceil, log

>>> def n_bits_required(nvalues: int) -> int:
...     return ceil(log(nvalues) / log(2))

>>> n_bits_required(256)
8

The reason that you need to use a ceiling in n_bits_required() is to account for values that are not clean powers of 2. Say you need to store a character set of 110 characters total. Naively, this should take log(110) / log(2) == 6.781 bits, but there’s no such thing as 0.781 bits. 110 values will require 7 bits, not 6, with the final slots being unneeded:

>>>

>>> n_bits_required(110)
7

All of this serves to to prove one concept: ASCII is, strictly speaking, a 7-bit code. The ASCII table that you saw above contains 128 code points and characters, 0 through 127 inclusive. This requires 7 bits:

>>>

>>> n_bits_required(128)  # 0 through 127
7
>>> n_possible_values(7)
128

The issue with this is that modern computers don’t store much of anything in 7-bit slots. They traffic in units of 8 bits, conventionally known as a byte.

Note: Throughout this tutorial, I assume that a byte refers to 8 bits, as it has since the 1960s, rather than some other unit of storage. You are free to call this an octet if you prefer.

This means that the storage space used by ASCII is half-empty. If it’s not clear why this is, think back to the decimal-to-binary table from above. You can express the numbers 0 and 1 with just 1 bit, or you can use 8 bits to express them as 00000000 and 00000001, respectively.

You can express the numbers 0 through 3 with just 2 bits, or 00 through 11, or you can use 8 bits to express them as 00000000, 00000001, 00000010, and 00000011, respectively. The highest ASCII code point, 127, requires only 7 significant bits.

Knowing this, you can see that make_bitseq() converts ASCII strings into a str representation of bytes, where every character consumes one byte:

>>>

>>> make_bitseq("bits")
'01100010 01101001 01110100 01110011'

ASCII’s underutilization of the 8-bit bytes offered by modern computers led to a family of conflicting, informalized encodings that each specified additional characters to be used with the remaining 128 available code points allowed in an 8-bit character encoding scheme.

Not only did these different encodings clash with each other, but each one of them was by itself still a grossly incomplete representation of the world’s characters, regardless of the fact that they made use of one additional bit.

Over the years, one character encoding mega-scheme came to rule them all. However, before we get there, let’s talk for a minute about numbering systems, which are a fundamental underpinning of character encoding schemes.

Covering All the Bases: Other Number Systems

In the discussion of ASCII above, you saw that each character maps to an integer in the range 0 through 127.

This range of numbers is expressed in decimal (base 10). It’s the way that you, me, and the rest of us humans are used to counting, for no reason more complicated than that we have 10 fingers.

But there are other numbering systems as well that are especially prevalent throughout the CPython source code. While the “underlying number” is the same, all numbering systems are just different ways of expressing the same number.

If I asked you what number the string "11" represents, you’d be right to give me a strange look before answering that it represents eleven.

However, this string representation can express different underlying numbers in different numbering systems. In addition to decimal, the alternatives include the following common numbering systems:

Binary: base 2
Octal: base 8
Hexadecimal (hex): base 16

But what does it mean for us to say that, in a certain numbering system, numbers are represented in base N?

Here is the best way that I know of to articulate what this means: it’s the number of fingers that you’d count on in that system.

If you want a much fuller but still gentle introduction to numbering systems, Charles Petzold’s Code is an incredibly cool book that explores the foundations of computer code in detail.

One way to demonstrate how different numbering systems interpret the same thing is with Python’s int() constructor. If you pass a str to int(), Python will assume by default that the string expresses a number in base 10 unless you tell it otherwise:

>>>

>>> int('11')
11
>>> int('11', base=10)  # 10 is already default
11
>>> int('11', base=2)  # Binary
3
>>> int('11', base=8)  # Octal
9
>>> int('11', base=16)  # Hex
17

There’s a more common way of telling Python that your integer is typed in a base other than 10. Python accepts literal forms of each of the 3 alternative numbering systems above:

Type of Literal	Prefix	Example
n/a	n/a	`11`
Binary literal	`0b` or `0B`	`0b11`
Octal literal	`0o` or `0O`	`0o11`
Hex literal	`0x` or `0X`	`0x11`

All of these are sub-forms of integer literals. You can see that these produce the same results, respectively, as the calls to int() with non-default base values. They’re all just int to Python:

>>>

>>> 11
11
>>> 0b11  # Binary literal
3
>>> 0o11  # Octal literal
9
>>> 0x11  # Hex literal
17

Here’s how you could type the binary, octal, and hexadecimal equivalents of the decimal numbers 0 through 20. Any of these are perfectly valid in a Python interpreter shell or source code, and all work out to be of type int:

Decimal	Binary	Octal	Hex
`0`	`0b0`	`0o0`	`0x0`
`1`	`0b1`	`0o1`	`0x1`
`2`	`0b10`	`0o2`	`0x2`
`3`	`0b11`	`0o3`	`0x3`
`4`	`0b100`	`0o4`	`0x4`
`5`	`0b101`	`0o5`	`0x5`
`6`	`0b110`	`0o6`	`0x6`
`7`	`0b111`	`0o7`	`0x7`
`8`	`0b1000`	`0o10`	`0x8`
`9`	`0b1001`	`0o11`	`0x9`
`10`	`0b1010`	`0o12`	`0xa`
`11`	`0b1011`	`0o13`	`0xb`
`12`	`0b1100`	`0o14`	`0xc`
`13`	`0b1101`	`0o15`	`0xd`
`14`	`0b1110`	`0o16`	`0xe`
`15`	`0b1111`	`0o17`	`0xf`
`16`	`0b10000`	`0o20`	`0x10`
`17`	`0b10001`	`0o21`	`0x11`
`18`	`0b10010`	`0o22`	`0x12`
`19`	`0b10011`	`0o23`	`0x13`
`20`	`0b10100`	`0o24`	`0x14`

It’s amazing just how prevalent these expressions are in the Python Standard Library. If you want to see for yourself, navigate to wherever your lib/python3.7/ directory sits, and check out the use of hex literals like this:

$ grep -nri --include "*\.py" -e "\b0x" lib/python3.7

This should work on any Unix system that has grep. You could use "\b0o" to search for octal literals or “\b0b” to search for binary literals.

What’s the argument for using these alternate int literal syntaxes? In short, it’s because 2, 8, and 16 are all powers of 2, while 10 is not. These three alternate number systems occasionally offer a way for expressing values in a computer-friendly manner. For example, the number 65536 or 2¹⁶, is just 10000 in hexadecimal, or 0x10000 as a Python hexadecimal literal.

Enter Unicode

As you saw, the problem with ASCII is that it’s not nearly a big enough set of characters to accommodate the world’s set of languages, dialects, symbols, and glyphs. (It’s not even big enough for English alone.)

Unicode fundamentally serves the same purpose as ASCII, but it just encompasses a way, way, way bigger set of code points. There are a handful of encodings that emerged chronologically between ASCII and Unicode, but they are not really worth mentioning just yet because Unicode and one of its encoding schemes, UTF-8, has become so predominantly used.

Think of Unicode as a massive version of the ASCII table—one that has 1,114,112 possible code points. That’s 0 through 1,114,111, or 0 through 17 * (2¹⁶) - 1, or 0x10ffff hexadecimal. In fact, ASCII is a perfect subset of Unicode. The first 128 characters in the Unicode table correspond precisely to the ASCII characters that you’d reasonably expect them to.

In the interest of being technically exacting, Unicode itself is not an encoding. Rather, Unicode is implemented by different character encodings, which you’ll see soon. Unicode is better thought of as a map (something like a dict) or a 2-column database table. It maps characters (like "a", "¢", or even "ቈ") to distinct, positive integers. A character encoding needs to offer a bit more.

Unicode contains virtually every character that you can imagine, including additional non-printable ones too. One of my favorites is the pesky right-to-left mark, which has code point 8207 and is used in text with both left-to-right and right-to-left language scripts, such as an article containing both English and Arabic paragraphs.

Note: The world of character encodings is one of many fine-grained technical details over which some people love to nitpick about. One such detail is that only 1,111,998 of the Unicode code points are actually usable, due to a couple of archaic reasons.

Unicode vs UTF-8

It didn’t take long for people to realize that all of the world’s characters could not be packed into one byte each. It’s evident from this that modern, more comprehensive encodings would need to use multiple bytes to encode some characters.

You also saw above that Unicode is not technically a full-blown character encoding. Why is that?

There is one thing that Unicode doesn’t tell you: it doesn’t tell you how to get actual bits from text—just code points. It doesn’t tell you enough about how to convert text to binary data and vice versa.

Unicode is an abstract encoding standard, not an encoding. That’s where UTF-8 and other encoding schemes come into play. The Unicode standard (a map of characters to code points) defines several different encodings from its single character set.

UTF-8 as well as its lesser-used cousins, UTF-16 and UTF-32, are encoding formats for representing Unicode characters as binary data of one or more bytes per character. We’ll discuss UTF-16 and UTF-32 in a moment, but UTF-8 has taken the largest share of the pie by far.

That brings us to a definition that is long overdue. What does it mean, formally, to encode and decode?

Encoding and Decoding in Python 3

Python 3’s str type is meant to represent human-readable text and can contain any Unicode character.

The bytes type, conversely, represents binary data, or sequences of raw bytes, that do not intrinsically have an encoding attached to it.

Encoding and decoding is the process of going from one to the other:

Encoding vs decoding (Image: Real Python)

In .encode() and .decode(), the encoding parameter is "utf-8" by default, though it’s generally safer and more unambiguous to specify it:

>>>

>>> "résumé".encode("utf-8")
b'r\xc3\xa9sum\xc3\xa9'
>>> "El Niño".encode("utf-8")
b'El Ni\xc3\xb1o'

>>> b"r\xc3\xa9sum\xc3\xa9".decode("utf-8")
'résumé'
>>> b"El Ni\xc3\xb1o".decode("utf-8")
'El Niño'

The results of str.encode() is a bytes object. Both bytes literals (such as b"r\xc3\xa9sum\xc3\xa9") and the representations of bytes permit only ASCII characters.

This is why, when calling "El Niño".encode("utf-8"), the ASCII-compatible "El" is allowed to be represented as it is, but the n with tilde is escaped to "\xc3\xb1". That messy-looking sequence represents two bytes, 0xc3 and 0xb1 in hex:

>>>

>>> " ".join(f"{i:08b}" for i in (0xc3, 0xb1))
'11000011 10110001'

That is, the character ñ requires two bytes for its binary representation under UTF-8.

Note: If you type help(str.encode), you’ll probably see a default of encoding='utf-8'. Be careful about excluding this and just using "résumé".encode(), because the default may be different in Windows prior to Python 3.6.

Python 3: All-In on Unicode

Python 3 is all-in on Unicode and UTF-8 specifically. Here’s what that means:

Python 3 source code is assumed to be UTF-8 by default. This means that you don’t need # -*- coding: UTF-8 -*- at the top of .py files in Python 3.
All text (str) is Unicode by default. Encoded Unicode text is represented as binary data (bytes). The str type can contain any literal Unicode character, such as "Δv / Δt", all of which will be stored as Unicode.
Anything from the Unicode character set is kosher in identifiers, meaning résumé = "~/Documents/resume.pdf" is valid if this strikes your fancy.
Python’s re module defaults to the re.UNICODE flag rather than re.ASCII. This means, for instance, that r"\w" matches Unicode word characters, not just ASCII letters.
The default encoding in str.encode() and bytes.decode() is UTF-8.

There is one other property that is more nuanced, which is that the default encoding to the built-in open() is platform-dependent and depends on the value of locale.getpreferredencoding():

>>>

>>> # Mac OS X High Sierra
>>> import locale
>>> locale.getpreferredencoding()
'UTF-8'

>>> # Windows Server 2012; other Windows builds may use UTF-16
>>> import locale
>>> locale.getpreferredencoding()
'cp1252'

Again, the lesson here is to be careful about making assumptions when it comes to the universality of UTF-8, even if it is the predominant encoding. It never hurts to be explicit in your code.

One Byte, Two Bytes, Three Bytes, Four

A crucial feature is that UTF-8 is a variable-length encoding. It’s tempting to gloss over what this means, but it’s worth delving into.

Think back to the section on ASCII. Everything in extended-ASCII-land demands at most one byte of space. You can quickly prove this with the following generator expression:

>>>

>>> all(len(chr(i).encode("ascii")) == 1 for i in range(128))
True

UTF-8 is quite different. A given Unicode character can occupy anywhere from one to four bytes. Here’s an example of a single Unicode character taking up four bytes:

>>>

>>> ibrow = "🤨"
>>> len(ibrow)
1
>>> ibrow.encode("utf-8")
b'\xf0\x9f\xa4\xa8'
>>> len(ibrow.encode("utf-8"))
4

>>> # Calling list() on a bytes object gives you
>>> # the decimal value for each byte
>>> list(b'\xf0\x9f\xa4\xa8')
[240, 159, 164, 168]

This is a subtle but important feature of len():

The length of a single Unicode character as a Python str will always be 1, no matter how many bytes it occupies.
The length of the same character encoded to bytes will be anywhere between 1 and 4.

The table below summarizes what general types of characters fit into each byte-length bucket:

Decimal Range	Hex Range	What’s Included	Examples
0 to 127	`"\u0000"` to `"\u007F"`	U.S. ASCII	`"A"`, `"\n"`, `"7"`, `"&"`
128 to 2047	`"\u0080"` to `"\u07FF"`	Most Latinic alphabets*	`"ę"`, `"±"`, `"ƌ"`, `"ñ"`
2048 to 65535	`"\u0800"` to `"\uFFFF"`	Additional parts of the multilingual plane (BMP)**	`"ത"`, `"ᄇ"`, `"ᮈ"`, `"‰"`
65536 to 1114111	`"\U00010000"` to `"\U0010FFFF"`	Other***	`"𝕂"`, `"𐀀"`, `"😓"`, `"🂲"`,

_{*Such as English, Arabic, Greek, and Irish}
_{**A huge array of languages and symbols—mostly Chinese, Japanese, and Korean by volume (also ASCII and Latin alphabets)}
_{***Additional Chinese, Japanese, Korean, and Vietnamese characters, plus more symbols and emojis}

Note: In the interest of not losing sight of the big picture, there is an additional set of technical features of UTF-8 that aren’t covered here because they are rarely visible to a Python user.

For instance, UTF-8 actually uses prefix codes that indicate the number of bytes in a sequence. This enables a decoder to tell what bytes belong together in a variable-length encoding, and lets the first byte serve as an indicator of the number of bytes in the coming sequence.

Wikipedia’s UTF-8 article does not shy away from technical detail, and there is always the official Unicode Standard for your reading enjoyment as well.

What About UTF-16 and UTF-32?

Let’s get back to two other encoding variants, UTF-16 and UTF-32.

The difference between these and UTF-8 is substantial in practice. Here’s an example of how major the difference is with a round-trip conversion:

>>>

>>> letters = "αβγδ"
>>> rawdata = letters.encode("utf-8")
>>> rawdata.decode("utf-8")
'αβγδ'
>>> rawdata.decode("utf-16")  # 😧
'뇎닎돎듎'

In this case, encoding four Greek letters with UTF-8 and then decoding back to text in UTF-16 would produce a text str that is in a completely different language (Korean).

Glaringly wrong results like this are possible when the same encoding isn’t used bidirectionally. Two variations of decoding the same bytes object may produce results that aren’t even in the same language.

This table summarizes the range or number of bytes under UTF-8, UTF-16, and UTF-32:

Encoding	Bytes Per Character (Inclusive)	Variable Length
UTF-8	1 to 4	Yes
UTF-16	2 to 4	Yes
UTF-32	4	No

One other curious aspect of the UTF family is that UTF-8 will not always take up less space than UTF-16. That may seem mathematically counterintuitive, but it’s quite possible:

>>>

>>> text = "記者 鄭啟源 羅智堅"
>>> len(text.encode("utf-8"))
26
>>> len(text.encode("utf-16"))
22

The reason for this is that the code points in the range U+0800 through U+FFFF (2048 through 65535 in decimal) take up three bytes in UTF-8 versus only two in UTF-16.

I’m not by any means recommending that you jump aboard the UTF-16 train, regardless of whether or not you operate in a language whose characters are commonly in this range. Among other reasons, one of the strong arguments for using UTF-8 is that, in the world of encoding, it’s a great idea to blend in with the crowd.

Not to mention, it’s 2019: computer memory is cheap, so saving 4 bytes by going out of your way to use UTF-16 is arguably not worth it.

Python’s Built-In Functions

You’ve made it through the hard part. Time to use what you’ve seen thus far in Python.

Python has a group of built-in functions that relate in some way to numbering systems and character encoding:

These can be logically grouped together based on their purpose:

ascii(), bin(), hex(), and oct() are for obtaining a different representation of an input. Each one produces a str. The first, ascii(), produces an ASCII only representation of an object, with non-ASCII characters escaped. The remaining three give binary, hexadecimal, and octal representations of an integer, respectively. These are only representations, not a fundamental change in the input.
bytes(), str(), and int() are class constructors for their respective types, bytes, str, and int. They each offer ways of coercing the input into the desired type. For instance, as you saw earlier, while int(11.0) is probably more common, you might also see int('11', base=16).
ord() and chr() are inverses of each other in that ord() converts a str character to its base-10 code point, while chr() does the opposite.

Here’s a more detailed look at each of these nine functions:

Function	Signature	Accepts	Return Type	Purpose
`ascii()`	`ascii(obj)`	Varies	`str`	ASCII only representation of an object, with non-ASCII characters escaped
`bin()`	`bin(number)`	`number: int`	`str`	Binary representation of an integer, with the prefix `"0b"`
`bytes()`	`bytes(iterable_of_ints)` `bytes(s, enc[, errors])` `bytes(bytes_or_buffer)` `bytes([i])`	Varies	`bytes`	Coerce (convert) the input to `bytes`, raw binary data
`chr()`	`chr(i)`	`i: int` `i>=0` `i<=1114111`	`str`	Convert an integer code point to a single Unicode character
`hex()`	`hex(number)`	`number: int`	`str`	Hexadecimal representation of an integer, with the prefix `"0x"`
`int()`	`int([x])` `int(x, base=10)`	Varies	`int`	Coerce (convert) the input to `int`
`oct()`	`oct(number)`	`number: int`	`str`	Octal representation of an integer, with the prefix `"0o"`
`ord()`	`ord(c)`	`c: str` `len(c) == 1`	`int`	Convert a single Unicode character to its integer code point
`str()`	`str(object=’‘)` `str(b[, enc[, errors]])`	Varies	`str`	Coerce (convert) the input to `str`, text

You can expand the section below to see some examples of each function.

ascii() gives you an ASCII-only representation of an object, with non-ASCII characters escaped:

>>>

>>> ascii("abcdefg")
"'abcdefg'"

>>> ascii("jalepeño")
"'jalepe\\xf1o'"

>>> ascii((1, 2, 3))
'(1, 2, 3)'

>>> ascii(0xc0ffee)  # Hex literal (int)
'12648430'

bin() gives you a binary representation of an integer, with the prefix "0b":

>>>

>>> bin(0)
'0b0'

>>> bin(400)
'0b110010000'

>>> bin(0xc0ffee)  # Hex literal (int)
'0b110000001111111111101110'

>>> [bin(i) for i in [1, 2, 4, 8, 16]]  # `int` + list comprehension
['0b1', '0b10', '0b100', '0b1000', '0b10000']

bytes() coerces the input to bytes, representing raw binary data:

>>>

>>> # Iterable of ints
>>> bytes((104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100))
b'hello world'

>>> bytes(range(97, 123))  # Iterable of ints
b'abcdefghijklmnopqrstuvwxyz'

>>> bytes("real 🐍", "utf-8")  # String + encoding
b'real \xf0\x9f\x90\x8d'

>>> bytes(10)
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

>>> bytes.fromhex('c0 ff ee')
b'\xc0\xff\xee'

>>> bytes.fromhex("72 65 61 6c 70 79 74 68 6f 6e")
b'realpython'

chr() converts an integer code point to a single Unicode character:

>>>

>>> chr(97)
'a'

>>> chr(7048)
'ᮈ'

>>> chr(1114111)
'\U0010ffff'

>>> chr(0x10FFFF)  # Hex literal (int)
'\U0010ffff'

>>> chr(0b01100100)  # Binary literal (int)
'd'

hex() gives the hexadecimal representation of an integer, with the prefix "0x":

>>>

>>> hex(100)
'0x64'

>>> [hex(i) for i in [1, 2, 4, 8, 16]]
['0x1', '0x2', '0x4', '0x8', '0x10']

>>> [hex(i) for i in range(16)]
['0x0', '0x1', '0x2', '0x3', '0x4', '0x5', '0x6', '0x7',
 '0x8', '0x9', '0xa', '0xb', '0xc', '0xd', '0xe', '0xf']

int() coerces the input to int, optionally interpreting the input in a given base:

>>>

>>> int(11.0)
11

>>> int('11')
11

>>> int('11', base=2)
3

>>> int('11', base=8)
9

>>> int('11', base=16)
17

>>> int(0xc0ffee - 1.0)
12648429

>>> int.from_bytes(b"\x0f", "little")
15

>>> int.from_bytes(b'\xc0\xff\xee', "big")
12648430

ord() converts a single Unicode character to its integer code point:

>>>

>>> ord("a")
97

>>> ord("ę")
281

>>> ord("ᮈ")
7048

>>> [ord(i) for i in "hello world"]
[104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100]

str() coerces the input to str, representing text:

>>>

>>> str("str of string")
'str of string'

>>> str(5)
'5'

>>> str([1, 2, 3, 4])  # Like [1, 2, 3, 4].__str__(), but use str()
'[1, 2, 3, 4]'

>>> str(b"\xc2\xbc cup of flour", "utf-8")
'¼ cup of flour'

>>> str(0xc0ffee)
'12648430'

Python String Literals: Ways to Skin a Cat

Rather than using the str() constructor, it’s commonplace to type a str literally:

>>>

>>> meal = "shrimp and grits"

That may seem easy enough. But the interesting side of things is that, because Python 3 is Unicode-centric through and through, you can “type” Unicode characters that you probably won’t even find on your keyboard. You can copy and paste this right into a Python 3 interpreter shell:

>>>

>>> alphabet = 'αβγδεζηθικλμνξοπρςστυφχψ'
>>> print(alphabet)
αβγδεζηθικλμνξοπρςστυφχψ

Besides placing the actual, unescaped Unicode characters in the console, there are other ways to type Unicode strings as well.

One of the densest sections of Python’s documentation is the portion on lexical analysis, specifically the section on string and bytes literals. Personally, I had to read this section about one, two, or maybe nine times for it to really sink in.

Part of what it says is that there are up to six ways that Python will allow you to type the same Unicode character.

The first and most common way is to type the character itself literally, as you’ve already seen. The tough part with this method is finding the actual keystrokes. That’s where the other methods for getting and representing characters come into play. Here’s the full list:

Escape Sequence	Meaning	How To Express `"a"`
`"\ooo"`	Character with octal value `ooo`	`"\141"`
`"\xhh"`	Character with hex value `hh`	`"\x61"`
`"\N{name}"`	Character named `name` in the Unicode database	`"\N{LATIN SMALL LETTER A}"`
`"\uxxxx"`	Character with 16-bit (2-byte) hex value `xxxx`	`"\u0061"`
`"\Uxxxxxxxx"`	Character with 32-bit (4-byte) hex value `xxxxxxxx`	`"\U00000061"`

Here’s some proof and validation of the above:

>>>

>>> (
...     "a" ==
...     "\x61" == 
...     "\N{LATIN SMALL LETTER A}" ==
...     "\u0061" ==
...     "\U00000061"
... )
True

Now, there are two main caveats:

Not all of these forms work for all characters. The hex representation of the integer 300 is 0x012c, which simply isn’t going to fit into the 2-hex-digit escape code "\xhh". The highest code point that you can squeeze into this escape sequence is "\xff" ("ÿ"). Similarly for "\ooo", it will only work up to "\777" ("ǿ").
For \xhh, \uxxxx, and \Uxxxxxxxx, exactly as many digits are required as are shown in these examples. This can throw you for a loop because of the way that Unicode tables conventionally display the codes for characters, with a leading U+ and variable number of hex characters. They key is that Unicode tables most often do not zero-pad these codes.

For instance, if you consult unicode-table.com for information on the Gothic letter faihu (or fehu), "𐍆", you’ll see that it is listed as having the code U+10346.

How do you put this into "\uxxxx" or "\Uxxxxxxxx"? Well, you can’t fit it in "\uxxxx" because it’s a 4-byte character, and to use "\Uxxxxxxxx" to represent this character, you’ll need to left-pad the sequence:

>>>

>>> "\U00010346"
'𐍆'

This also means that the "\Uxxxxxxxx" form is the only escape sequence that is capable of holding any Unicode character.

Note: Here’s a short function to convert strings that look like "U+10346" into something Python can work with. It uses str.zfill():

>>>

>>> def make_uchr(code: str):
...     return chr(int(code.lstrip("U+").zfill(8), 16))
>>> make_uchr("U+10346")
'𐍆'
>>> make_uchr("U+0026")
'&'

Other Encodings Available in Python

So far, you’ve seen four character encodings:

ASCII
UTF-8
UTF-16
UTF-32

There are a ton of other ones out there.

One example is Latin-1 (also called ISO-8859-1), which is technically the default for the Hypertext Transfer Protocol (HTTP), per RFC 2616. Windows has its own Latin-1 variant called cp1252.

Note: ISO-8859-1 is still very much present out in the wild. The requests library follows RFC 2616 “to the letter” in using it as the default encoding for the content of an HTTP/HTTPS response. If the word “text” is found in the Content-Type header, and no other encoding is specified, then requests will use ISO-8859-1.

The complete list of accepted encodings is buried way down in the documentation for the codecs module, which is part of Python’s Standard Library.

There’s one more useful recognized encoding to be aware of, which is "unicode-escape". If you have a decoded str and want to quickly get a representation of its escaped Unicode literal, then you can specify this encoding in .encode():

>>>

>>> alef = chr(1575)  # Or "\u0627"
>>> alef_hamza = chr(1571)  # Or "\u0623"
>>> alef, alef_hamza
('ا', 'أ')
>>> alef.encode("unicode-escape")
b'\\u0627'
>>> alef_hamza.encode("unicode-escape")
b'\\u0623'

You Know What They Say About Assumptions…

Just because Python makes the assumption of UTF-8 encoding for files and code that you generate doesn’t mean that you, the programmer, should operate with the same assumption for external data.

Let’s say that again because it’s a rule to live by: when you receive binary data (bytes) from a third party source, whether it be from a file or over a network, the best practice is to check that the data specifies an encoding. If it doesn’t, then it’s on you to ask.

All I/O happens in bytes, not text, and bytes are just ones and zeros to a computer until you tell it otherwise by informing it of an encoding.

Here’s an example of where things can go wrong. You’re subscribed to an API that sends you a recipe of the day, which you receive in bytes and have always decoded using .decode("utf-8") with no problem. On this particular day, part of the recipe looks like this:

>>>

>>> data = b"\xbc cup of flour"

It looks as if the recipe calls for some flour, but we don’t know how much:

>>>

>>> data.decode("utf-8")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbc in position 0: invalid start byte

Uh oh. There’s that pesky UnicodeDecodeError that can bite you when you make assumptions about encoding. You check with the API host. Lo and behold, the data is actually sent over encoded in Latin-1:

>>>

>>> data.decode("latin-1")
'¼ cup of flour'

There we go. In Latin-1, every character fits into a single byte, whereas the “¼” character takes up two bytes in UTF-8 ("\xc2\xbc").

The lesson here is that it can be dangerous to assume the encoding of any data that is handed off to you. It’s usually UTF-8 these days, but it’s the small percentage of cases where it’s not that will blow things up.

If you really do need to abandon ship and guess an encoding, then have a look at the chardet library, which uses methodology from Mozilla to make an educated guess about ambiguously encoded text. That said, a tool like chardet should be your last resort, not your first.

Odds and Ends: `unicodedata`

We would be remiss not to mention unicodedata from the Python Standard Library, which lets you interact with and do lookups on the Unicode Character Database (UCD):

>>>

>>> import unicodedata

>>> unicodedata.name("€")
'EURO SIGN'
>>> unicodedata.lookup("EURO SIGN")
'€'

Wrapping Up

In this article, you’ve decoded the wide and imposing subject of character encoding in Python.

You’ve covered a lot of ground here:

Fundamental concepts of character encodings and numbering systems
Integer, binary, octal, hex, str, and bytes literals in Python
Python’s built-in functions related to character encoding and numbering systems
Python 3’s treatment of text versus binary data

Now, go forth and encode!

Resources

For even more detail about the topics covered here, check out these resources:

Joel Spolsky: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
David Zentgraf: What every programmer absolutely, positively needs to know about encodings and character sets to work with text
Mozilla: A composite approach to language/encoding detection
Wikipedia: UTF-8
John Skeet: Unicode and .NET
Charles Petzold: Code: The Hidden Language of Computer Hardware and Software
Network Working Group, RFC 3629: UTF-8, a transformation format of ISO 10646
Unicode Technical Standard #18: Unicode Regular Expressions

The Python docs have two pages on the subject:

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Installing Python on Windows, macOS, and Linux

2019-05-16T14:00:00+00:00

To get started working with Python, you’ll need to have access to the Python interpreter. There are several common ways to accomplish this.

In this course, you will learn how to install the latest version of Python 3 on your computer.

Many operating systems, such as macOS and Linux, come with Python pre-installed. The version of Python that comes with your operating system is called your system Python.

The system Python is almost always out-of-date, and may not even be a full Python installation. It’s essential that you have the most recent version of Python so that you can follow along successfully with the examples in this book.

There are two major versions of Python available: Python 2, also known as legacy Python, and Python 3. Python 2 was released in the year 2000 and will reach its end-of-life on January 1, 2020. This course focuses on Python 3.

It is split into three sections: Windows, macOS, and Linux. Just find the section for your operating system and follow the steps to get your computer set-up.

If you have a different operating system, check out the Python 3 Installation & Setup Guide maintained on realpython.com to see if your OS is covered.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Three Ways of Storing and Accessing Lots of Images in Python

2019-05-15T14:00:00+00:00

Why would you want to know more about different ways of storing and accessing images in Python? If you’re segmenting a handful of images by color or detecting faces one by one using OpenCV, then you don’t need to worry about it. Even if you’re using the Python Imaging Library (PIL) to draw on a few hundred photos, you still don’t need to. Storing images on disk, as .png or .jpg files, is both suitable and appropriate.

Increasingly, however, the number of images required for a given task is getting larger and larger. Algorithms like convolutional neural networks, also known as convnets or CNNs, can handle enormous datasets of images and even learn from them. If you’re interested, you can read more about how convnets can be used for ranking selfies or for sentiment analysis.

ImageNet is a well-known public image database put together for training models on tasks like object classification, detection, and segmentation, and it consists of over 14 million images.

Think about how long it would take to load all of them into memory for training, in batches, perhaps hundreds or thousands of times. Keep reading, and you’ll be convinced that it would take quite awhile—at least long enough to leave your computer and do many other things while you wish you worked at Google or NVIDIA.

In this tutorial, you’ll learn about:

Storing images on disk as .png files
Storing images in lightning memory-mapped databases (LMDB)
Storing images in hierarchical data format (HDF5)

You’ll also explore the following:

Why alternate storage methods are worth considering
What the performance differences are when you’re reading and writing single images
What the performance differences are when you’re reading and writing many images
How the three methods compare in terms of disk usage

If none of the storage methods ring a bell, don’t worry: for this article, all you need is a reasonably solid foundation in Python and a basic understanding of images (that they are really composed of multi-dimensional arrays of numbers) and relative memory, such as the difference between 10MB and 10GB.

Let’s get started!

Free Bonus: Click here to get the Python Face Detection & OpenCV Examples Mini-Guide that shows you practical code examples of real-world Python computer vision techniques.

Setup

You will need an image dataset to experiment with, as well as a few Python packages.

A Dataset to Play With

We will be using the Canadian Institute for Advanced Research image dataset, better known as CIFAR-10, which consists of 60,000 32x32 pixel color images belonging to different object classes, such as dogs, cats, and airplanes. Relatively, CIFAR is not a very large dataset, but if we were to use the full TinyImages dataset, then you would need about 400GB of free disk space, which would probably be a limiting factor.

Credits for the dataset as described in chapter 3 of this tech report go to Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.

If you’d like to follow along with the code examples in this article, you can download CIFAR-10 here, selecting the Python version. You’ll be sacrificing 163MB of disk space:

Image: A. Krizhevsky

When you download and unzip the folder, you’ll discover that the files are not human-readable image files. They have actually been serialized and saved in batches using cPickle.

While we won’t consider pickle or cPickle in this article, other than to extract the CIFAR dataset, it’s worth mentioning that the Python pickle module has the key advantage of being able to serialize any Python object without any extra code or transformation on your part. It also has a potentially serious disadvantage of posing a security risk and not coping well when dealing with very large quantities of data.

The following code unpickles each of the five batch files and loads all of the images into a NumPy array:

import numpy as np
import pickle
from pathlib import Path

# Path to the unzipped CIFAR data
data_dir = Path("data/cifar-10-batches-py/")

# Unpickle function provided by the CIFAR hosts
def unpickle(file):
    with open(file, "rb") as fo:
        dict = pickle.load(fo, encoding="bytes")
    return dict

images, labels = [], []
for batch in data_dir.glob("data_batch_*"):
    batch_data = unpickle(batch)
    for i, flat_im in enumerate(batch_data[b"data"]):
        im_channels = []
        # Each image is flattened, with channels in order of R, G, B
        for j in range(3):
            im_channels.append(
                flat_im[j * 1024 : (j + 1) * 1024].reshape((32, 32))
            )
        # Reconstruct the original image
        images.append(np.dstack((im_channels)))
        # Save the label
        labels.append(batch_data[b"labels"][i])

print("Loaded CIFAR-10 training set:")
print(f" - np.shape(images)     {np.shape(images)}")
print(f" - np.shape(labels)     {np.shape(labels)}")

All the images are now in RAM in the images variable, with their corresponding meta data in labels, and are ready for you to manipulate. Next, you can install the Python packages you’ll use for the three methods.

Note: That last code block used f-strings. You can read more about them in Python 3’s f-Strings: An Improved String Formatting Syntax (Guide).

Setup for Storing Images on Disk

You’ll need to set up your environment for the default method of saving and accessing these images from disk. This article will assume you have Python 3.x installed on your system, and will use Pillow for the image manipulation:

$ pip install Pillow

Alternatively, if you prefer, you can install it using Anaconda:

$ conda install -c conda-forge pillow

Note: PIL is the original version of the Python Imaging Library, which is no longer maintained and is not compatible with Python 3.x. If you have previously installed PIL, make sure to uninstall it before installing Pillow, as they can’t exist together.

Now you’re ready for storing and reading images from disk.

Getting Started With LMDB

LMDB, sometimes referred to as the “Lightning Database,” stands for Lightning Memory-Mapped Database because it’s fast and uses memory-mapped files. It’s a key-value store, not a relational database.

In terms of implementation, LMDB is a B+ tree, which basically means that it is a tree-like graph structure stored in memory where each key-value element is a node, and nodes can have many children. Nodes on the same level are linked to one another for fast traversal.

Critically, key components of the B+ tree are set to correspond to the page size of the host operating system, maximizing efficiency when accessing any key-value pair in the database. Since LMDB high-performance heavily relies on this particular point, LMDB efficiency has been shown to be dependent on the underlying file system and its implementation.

Another key reason for the efficiency of LMDB is that it is memory-mapped. This means that it returns direct pointers to the memory addresses of both keys and values, without needing to copy anything in memory as most other databases do.

Those who want to dive into a bit more of the internal implementation details of B+ trees can check out this article on B+ trees and then play with this visualization of node insertion.

If B+ trees don’t interest you, don’t worry. You don’t need to know much about their internal implementation in order to use LMDB. We will be using the Python binding for the LMDB C library, which can be installed via pip:

$ pip install lmdb

You also have the option of installing via Anaconda:

$ conda install -c conda-forge python-lmdb

Check that you can import lmdb from a Python shell, and you’re good to go.

Getting Started With HDF5

HDF5 stands for Hierarchical Data Format, a file format referred to as HDF4 or HDF5. We don’t need to worry about HDF4, as HDF5 is the current maintained version.

Interestingly, HDF has its origins in the National Center for Supercomputing Applications, as a portable, compact scientific data format. If you’re wondering if it’s widely used, check out NASA’s blurb on HDF5 from their Earth Data project.

HDF files consist of two types of objects:

Datasets
Groups

Datasets are multidimensional arrays, and groups consist of datasets or other groups. Multidimensional arrays of any size and type can be stored as a dataset, but the dimensions and type have to be uniform within a dataset. Each dataset must contain a homogeneous N-dimensional array. That said, because groups and datasets may be nested, you can still get the heterogeneity you may need:

$ pip install h5py

As with the other libraries, you can alternately install via Anaconda:

$ conda install -c conda-forge h5py

If you can import h5py from a Python shell, everything is set up properly.

Storing a Single Image

Now that you have a general overview of the methods, let’s dive straight in and look at a quantitative comparison of the basic tasks we care about: how long it takes to read and write files, and how much disk memory will be used. This will also serve as a basic introduction to how the methods work, with code examples of how to use them.

When I refer to “files,” I generally mean a lot of them. However, it is important to make a distinction since some methods may be optimized for different operations and quantities of files.

For the purposes of experimentation, we can compare the performance between various quantities of files, by factors of 10 from a single image to 100,000 images. Since our five batches of CIFAR-10 add up to 50,000 images, we can use each image twice to get to 100,000 images.

To prepare for the experiments, you will want to create a folder for each method, which will contain all the database files or images, and save the paths to those directories in variables:

from pathlib import Path

disk_dir = Path("data/disk/")
lmdb_dir = Path("data/lmdb/")
hdf5_dir = Path("data/hdf5/")

Path does not automatically create the folders for you unless you specifically ask it to:

disk_dir.mkdir(parents=True, exist_ok=True)
lmdb_dir.mkdir(parents=True, exist_ok=True)
hdf5_dir.mkdir(parents=True, exist_ok=True)

Now you can move on to running the actual experiments, with code examples of how to perform basic tasks with the three different methods. We can use the timeit module, which is included in the Python standard library, to help time the experiments.

Although the main purpose of this article is not to learn the APIs of the different Python packages, it is helpful to have an understanding of how they can be implemented. We will go through the general principles alongside all the code used to conduct the storing experiments.

Storing to Disk

Our input for this experiment is a single image image, currently in memory as a NumPy array. You want to save it first to disk as a .png image, and name it using a unique image ID image_id. This can be done using the Pillow package you installed earlier:

from PIL import Image
import csv

def store_single_disk(image, image_id, label):
    """ Stores a single image as a .png file on disk.
        Parameters:
        ---------------
        image       image array, (32, 32, 3) to be stored
        image_id    integer unique ID for image
        label       image label
    """
    Image.fromarray(image).save(disk_dir / f"{image_id}.png")

    with open(disk_dir / f"{image_id}.csv", "wt") as csvfile:
        writer = csv.writer(
            csvfile, delimiter=" ", quotechar="|", quoting=csv.QUOTE_MINIMAL
        )
        writer.writerow([label])

This saves the image. In all realistic applications, you also care about the meta data attached to the image, which in our example dataset is the image label. When you’re storing images to disk, there are several options for saving the meta data.

One solution is to encode the labels into the image name. This has the advantage of not requiring any extra files.

However, it also has the big disadvantage of forcing you to deal with all the files whenever you do anything with labels. Storing the labels in a separate file allows you to play around with the labels alone, without having to load the images. Above, I have stored the labels in a separate .csv files for this experiment.

Now let’s move on to doing the exact same task with LMDB.

Storing to LMDB

Firstly, LMDB is a key-value storage system where each entry is saved as a byte array, so in our case, keys will be a unique identifier for each image, and the value will be the image itself. Both the keys and values are expected to be strings, so the common usage is to serialize the value as a string, and then unserialize it when reading it back out.

You can use pickle for the serializing. Any Python object can be serialized, so you might as well include the image meta data in the database as well. This saves you the trouble of attaching meta data back to the image data when we load the dataset from disk.

You can create a basic Python class for the image and its meta data:

class CIFAR_Image:
    def __init__(self, image, label):
        # Dimensions of image for reconstruction - not really necessary 
        # for this dataset, but some datasets may include images of 
        # varying sizes
        self.channels = image.shape[2]
        self.size = image.shape[:2]

        self.image = image.tobytes()
        self.label = label

    def get_image(self):
        """ Returns the image as a numpy array. """
        image = np.frombuffer(self.image, dtype=np.uint8)
        return image.reshape(*self.size, self.channels)

Secondly, because LMDB is memory-mapped, new databases need to know how much memory they are expected to use up. This is relatively straightforward in our case, but it can be a massive pain in other cases, which you will see in more depth in a later section. LMDB calls this variable the map_size.

Finally, read and write operations with LMDB are performed in transactions. You can think of them as similar to those of a traditional database, consisting of a group of operations on the database. This may look already significantly more complicated than the disk version, but hang on and keep reading!

With those three points in mind, let’s look at the code to save a single image to a LMDB:

import lmdb
import pickle

def store_single_lmdb(image, image_id, label):
    """ Stores a single image to a LMDB.
        Parameters:
        ---------------
        image       image array, (32, 32, 3) to be stored
        image_id    integer unique ID for image
        label       image label
    """
    map_size = image.nbytes * 10

    # Create a new LMDB environment
    env = lmdb.open(str(lmdb_dir / f"single_lmdb"), map_size=map_size)

    # Start a new write transaction
    with env.begin(write=True) as txn:
        # All key-value pairs need to be strings
        value = CIFAR_Image(image, label)
        key = f"{image_id:08}"
        txn.put(key.encode("ascii"), pickle.dumps(value))
    env.close()

Note: It’s a good idea to calculate the exact number of bytes each key-value pair will take up.

With a dataset of images of varying size, this will be an approximation, but you can use sys.getsizeof() to get a reasonable approximation. Keep in mind that sys.getsizeof(CIFAR_Image) will only return the size of a class definition, which is 1056, not the size of an instantiated object.

The function will also not be able to fully calculate nested items, lists, or objects containing references to other objects.

Alternately, you could use pympler to save you some calculations by determining the exact size of an object.

You are now ready to save an image to LMDB. Lastly, let’s look at the final method, HDF5.

Storing With HDF5

Remember that an HDF5 file can contain more than one dataset. In this rather trivial case, you can create two datasets, one for the image, and one for its meta data:

import h5py

def store_single_hdf5(image, image_id, label):
    """ Stores a single image to an HDF5 file.
        Parameters:
        ---------------
        image       image array, (32, 32, 3) to be stored
        image_id    integer unique ID for image
        label       image label
    """
    # Create a new HDF5 file
    file = h5py.File(hdf5_dir / f"{image_id}.h5", "w")

    # Create a dataset in the file
    dataset = file.create_dataset(
        "image", np.shape(image), h5py.h5t.STD_U8BE, data=image
    )
    meta_set = file.create_dataset(
        "meta", np.shape(label), h5py.h5t.STD_U8BE, data=label
    )
    file.close()

h5py.h5t.STD_U8BE specifies the type of data that will be stored in the dataset, which in this case is unsigned 8-bit integers. You can see a full list of HDF’s predefined datatypes here.

Note: The choice of datatype will strongly affect the runtime and storage requirements of HDF5, so it is best to choose your minimum requirements.

Now that we have reviewed the three methods of saving a single image, let’s move on to the next step.

Experiments for Storing a Single Image

Now you can put all three functions for saving a single image into a dictionary, which can be called later during the timing experiments:

_store_single_funcs = dict(
    disk=store_single_disk, lmdb=store_single_lmdb, hdf5=store_single_hdf5
)

Finally, everything is ready for conducting the timed experiment. Let’s try saving the first image from CIFAR and its corresponding label, and storing it in the three different ways:

from timeit import timeit

store_single_timings = dict()

for method in ("disk", "lmdb", "hdf5"):
    t = timeit(
        "_store_single_funcs[method](image, 0, label)",
        setup="image=images[0]; label=labels[0]",
        number=1,
        globals=globals(),
    )
    store_single_timings[method] = t
    print(f"Method: {method}, Time usage: {t}")

Note: While you’re playing around with LMDB, you may see a MapFullError: mdb_txn_commit: MDB_MAP_FULL: Environment mapsize limit reached error. It’s important to note that LMDB does not overwrite preexisting values, even if they have the same key.

This contributes to the fast write time, but it also means that if you store an image more than once in the same LMDB file, then you will use up the map size. If you run a store function, be sure to delete any preexisting LMDB files first.

Remember that we’re interested in runtime, displayed here in seconds, and also the memory usage:

Method	Save Single Image + Meta	Memory
Disk	1.915 ms	8 K
LMDB	1.203 ms	32 K
HDF5	8.243 ms	8 K

There are two takeaways here:

All of the methods are trivially quick.
In terms of disk usage, LMDB uses more.

Clearly, despite LMDB having a slight performance lead, we haven’t convinced anyone why to not just store images on disk. After all, it’s a human readable format, and you can open and view them from any file system browser! Well, it’s time to look at a lot more images…

Storing Many Images

You have seen the code for using the various storage methods to save a single image, so now we need to adjust the code to save many images and then run the timed experiment.

Adjusting the Code for Many Images

Saving multiple images as .png files is as straightforward as calling store_single_method() multiple times. But this isn’t true for LMDB or HDF5, since you don’t want a different database file for each image. Rather, you want to put all of the images into one or more files.

You will need to slightly alter the code and create three new functions that accept multiple images, store_many_disk(), store_many_lmdb(), and store_many_hdf5:

 store_many_disk(images, labels):
    """ Stores an array of images to disk
        Parameters:
        ---------------
        images       images array, (N, 32, 32, 3) to be stored
        labels       labels array, (N, 1) to be stored
    """
    num_images = len(images)

    # Save all the images one by one
    for i, image in enumerate(images):
        Image.fromarray(image).save(disk_dir / f"{i}.png")

    # Save all the labels to the csv file
    with open(disk_dir / f"{num_images}.csv", "w") as csvfile:
        writer = csv.writer(
            csvfile, delimiter=" ", quotechar="|", quoting=csv.QUOTE_MINIMAL
        )
        for label in labels:
            # This typically would be more than just one value per row
            writer.writerow([label])

def store_many_lmdb(images, labels):
    """ Stores an array of images to LMDB.
        Parameters:
        ---------------
        images       images array, (N, 32, 32, 3) to be stored
        labels       labels array, (N, 1) to be stored
    """
    num_images = len(images)

    map_size = num_images * images[0].nbytes * 10

    # Create a new LMDB DB for all the images
    env = lmdb.open(str(lmdb_dir / f"{num_images}_lmdb"), map_size=map_size)

    # Same as before — but let's write all the images in a single transaction
    with env.begin(write=True) as txn:
        for i in range(num_images):
            # All key-value pairs need to be Strings
            value = CIFAR_Image(images[i], labels[i])
            key = f"{i:08}"
            txn.put(key.encode("ascii"), pickle.dumps(value))
    env.close()

def store_many_hdf5(images, labels):
    """ Stores an array of images to HDF5.
        Parameters:
        ---------------
        images       images array, (N, 32, 32, 3) to be stored
        labels       labels array, (N, 1) to be stored
    """
    num_images = len(images)

    # Create a new HDF5 file
    file = h5py.File(hdf5_dir / f"{num_images}_many.h5", "w")

    # Create a dataset in the file
    dataset = file.create_dataset(
        "images", np.shape(images), h5py.h5t.STD_U8BE, data=images
    )
    meta_set = file.create_dataset(
        "meta", np.shape(labels), h5py.h5t.STD_U8BE, data=labels
    )
    file.close()

So you could store more than one file to disk, the image files method was altered to loop over each image in the list. For LMDB, a loop is also needed since we are creating a CIFAR_Image object for each image and its meta data.

The smallest adjustment is with the HDF5 method. In fact, there’s hardly an adjustment at all! HFD5 files have no limitation on file size aside from external restrictions or dataset size, so all the images were stuffed into a single dataset, just like before.

Next, you will need to prepare the dataset for the experiments by increasing its size.

Preparing the Dataset

Before running the experiments again, let’s first double our dataset size so that we can test with up to 100,000 images:

cutoffs = [10, 100, 1000, 10000, 100000]

# Let's double our images so that we have 100,000
images = np.concatenate((images, images), axis=0)
labels = np.concatenate((labels, labels), axis=0)

# Make sure you actually have 100,000 images and labels
print(np.shape(images))
print(np.shape(labels))

Now that there are enough images, it’s time for the experiment.

Experiment for Storing Many Images

As you did with reading many images, you can create a dictionary handling all the functions with store_many_ and run the experiments:

_store_many_funcs = dict(
    disk=store_many_disk, lmdb=store_many_lmdb, hdf5=store_many_hdf5
)

from timeit import timeit

store_many_timings = {"disk": [], "lmdb": [], "hdf5": []}

for cutoff in cutoffs:
    for method in ("disk", "lmdb", "hdf5"):
        t = timeit(
            "_store_many_funcs[method](images_, labels_)",
            setup="images_=images[:cutoff]; labels_=labels[:cutoff]",
            number=1,
            globals=globals(),
        )
        store_many_timings[method].append(t)

        # Print out the method, cutoff, and elapsed time
        print(f"Method: {method}, Time usage: {t}")

If you’re following along and running the code yourself, you’ll need to sit back a moment in suspense and wait for 111,110 images to be stored three times each to your disk, in three different formats. You’ll also need to say goodbye to approximately 2 GB of disk space.

Now for the moment of truth! How long did all of that storing take? A picture is worth a thousand words:

The first graph shows the normal, unadjusted storage time, highlighting the drastic difference between storing to .png files and LMDB or HDF5.

The second graph shows the log of the timings, highlighting that HDF5 starts out slower than LMDB but, with larger quantities of images, comes out slightly ahead.

While exact results may vary depending on your machine, this is why LMDB and HDF5 are worth thinking about. Here’s the code that generated the above graph:

import matplotlib.pyplot as plt

def plot_with_legend(
    x_range, y_data, legend_labels, x_label, y_label, title, log=False
):
    """ Displays a single plot with multiple datasets and matching legends.
        Parameters:
        --------------
        x_range         list of lists containing x data
        y_data          list of lists containing y values
        legend_labels   list of string legend labels
        x_label         x axis label
        y_label         y axis label
    """
    plt.style.use("seaborn-whitegrid")
    plt.figure(figsize=(10, 7))

    if len(y_data) != len(legend_labels):
        raise TypeError(
            "Error: number of data sets does not match number of labels."
        )

    all_plots = []
    for data, label in zip(y_data, legend_labels):
        if log:
            temp, = plt.loglog(x_range, data, label=label)
        else:
            temp, = plt.plot(x_range, data, label=label)
        all_plots.append(temp)

    plt.title(title)
    plt.xlabel(x_label)
    plt.ylabel(y_label)
    plt.legend(handles=all_plots)
    plt.show()

# Getting the store timings data to display
disk_x = store_many_timings["disk"]
lmdb_x = store_many_timings["lmdb"]
hdf5_x = store_many_timings["hdf5"]

plot_with_legend(
    cutoffs,
    [disk_x, lmdb_x, hdf5_x],
    ["PNG files", "LMDB", "HDF5"],
    "Number of images",
    "Seconds to store",
    "Storage time",
    log=False,
)

plot_with_legend(
    cutoffs,
    [disk_x, lmdb_x, hdf5_x],
    ["PNG files", "LMDB", "HDF5"],
    "Number of images",
    "Seconds to store",
    "Log storage time",
    log=True,
)

Now let’s go on to reading the images back out.

Reading a Single Image

First, let’s consider the case for reading a single image back into an array for each of the three methods.

Reading From Disk

Of the three methods, LMDB requires the most legwork when reading image files back out of memory, because of the serialization step. Let’s walk through these functions that read a single image out for each of the three storage formats.

First, read a single image and its meta from a .png and .csv file:

def read_single_disk(image_id):
    """ Stores a single image to disk.
        Parameters:
        ---------------
        image_id    integer unique ID for image

        Returns:
        ----------
        image       image array, (32, 32, 3) to be stored
        label       associated meta data, int label
    """
    image = np.array(Image.open(disk_dir / f"{image_id}.png"))

    with open(disk_dir / f"{image_id}.csv", "r") as csvfile:
        reader = csv.reader(
            csvfile, delimiter=" ", quotechar="|", quoting=csv.QUOTE_MINIMAL
        )
        label = int(next(reader)[0])

    return image, label

Reading From LMDB

Next, read the same image and meta from an LMDB by opening the environment and starting a read transaction:

 1 def read_single_lmdb(image_id):
 2     """ Stores a single image to LMDB.
 3         Parameters:
 4         ---------------
 5         image_id    integer unique ID for image
 6 
 7         Returns:
 8         ----------
 9         image       image array, (32, 32, 3) to be stored
10         label       associated meta data, int label
11     """
12     # Open the LMDB environment
13     env = lmdb.open(str(lmdb_dir / f"single_lmdb"), readonly=True)
14 
15     # Start a new read transaction
16     with env.begin() as txn:
17         # Encode the key the same way as we stored it
18         data = txn.get(f"{image_id:08}".encode("ascii"))
19         # Remember it's a CIFAR_Image object that is loaded
20         cifar_image = pickle.loads(data)
21         # Retrieve the relevant bits
22         image = cifar_image.get_image()
23         label = cifar_image.label
24     env.close()
25 
26     return image, label

Here are a couple points to not about the code snippet above:

Line 13: The readonly=True flag specifies that no writes will be allowed on the LMDB file until the transaction is finished. In database lingo, it’s equivalent to taking a read lock.
Line 20: To retrieve the CIFAR_Image object, you need to reverse the steps we took to pickle it when we were writing it. This is where the get_image() of the object is helpful.

This wraps up reading the image back out from LMDB. Finally, you will want to do the same with HDF5.

Reading From HDF5

Reading from HDF5 looks very similar to the writing process. Here is the code to open and read the HDF5 file and parse the same image and meta:

def read_single_hdf5(image_id):
    """ Stores a single image to HDF5.
        Parameters:
        ---------------
        image_id    integer unique ID for image

        Returns:
        ----------
        image       image array, (32, 32, 3) to be stored
        label       associated meta data, int label
    """
    # Open the HDF5 file
    file = h5py.File(hdf5_dir / f"{image_id}.h5", "r+")

    image = np.array(file["/image"]).astype("uint8")
    label = int(np.array(file["/meta"]).astype("uint8"))

    return image, label

Note that you access the various datasets in the file by indexing the file object using the dataset name preceded by a forward slash /. As before, you can create a dictionary containing all the read functions:

_read_single_funcs = dict(
    disk=read_single_disk, lmdb=read_single_lmdb, hdf5=read_single_hdf5
)

With this dictionary prepared, you are ready for running the experiment.

Experiment for Reading a Single Image

You might expect that the experiment for reading a single image in will have somewhat trivial results, but here’s the experiment code:

from timeit import timeit

read_single_timings = dict()

for method in ("disk", "lmdb", "hdf5"):
    t = timeit(
        "_read_single_funcs[method](0)",
        setup="image=images[0]; label=labels[0]",
        number=1,
        globals=globals(),
    )
    read_single_timings[method] = t
    print(f"Method: {method}, Time usage: {t}")

Here are the results of the experiment for reading a single image:

Method	Read Single Image + Meta
Disk	1.61970 ms
LMDB	4.52063 ms
HDF5	1.98036 ms

It’s slightly faster to read the .png and .csv files directly from disk, but all three methods perform trivially quickly. The experiments we’ll do next are much more interesting.

Reading Many Images

Now you can adjust the code to read many images at once. This is likely the action you’ll be performing most often, so the runtime performance is essential.

Adjusting the Code for Many Images

Extending the functions above, you can create functions with read_many_, which can be used for the next experiments. Like before, it is interesting to compare performance when reading different quantities of images, which are repeated in the code below for reference:

def read_many_disk(num_images):
    """ Reads image from disk.
        Parameters:
        ---------------
        num_images   number of images to read

        Returns:
        ----------
        images      images array, (N, 32, 32, 3) to be stored
        labels      associated meta data, int label (N, 1)
    """
    images, labels = [], []

    # Loop over all IDs and read each image in one by one
    for image_id in range(num_images):
        images.append(np.array(Image.open(disk_dir / f"{image_id}.png")))

    with open(disk_dir / f"{num_images}.csv", "r") as csvfile:
        reader = csv.reader(
            csvfile, delimiter=" ", quotechar="|", quoting=csv.QUOTE_MINIMAL
        )
        for row in reader:
            labels.append(int(row[0]))
    return images, labels

def read_many_lmdb(num_images):
    """ Reads image from LMDB.
        Parameters:
        ---------------
        num_images   number of images to read

        Returns:
        ----------
        images      images array, (N, 32, 32, 3) to be stored
        labels      associated meta data, int label (N, 1)
    """
    images, labels = [], []
    env = lmdb.open(str(lmdb_dir / f"{num_images}_lmdb"), readonly=True)

    # Start a new read transaction
    with env.begin() as txn:
        # Read all images in one single transaction, with one lock
        # We could split this up into multiple transactions if needed
        for image_id in range(num_images):
            data = txn.get(f"{image_id:08}".encode("ascii"))
            # Remember that it's a CIFAR_Image object 
            # that is stored as the value
            cifar_image = pickle.loads(data)
            # Retrieve the relevant bits
            images.append(cifar_image.get_image())
            labels.append(cifar_image.label)
    env.close()
    return images, labels

def read_many_hdf5(num_images):
    """ Reads image from HDF5.
        Parameters:
        ---------------
        num_images   number of images to read

        Returns:
        ----------
        images      images array, (N, 32, 32, 3) to be stored
        labels      associated meta data, int label (N, 1)
    """
    images, labels = [], []

    # Open the HDF5 file
    file = h5py.File(hdf5_dir / f"{num_images}_many.h5", "r+")

    images = np.array(file["/images"]).astype("uint8")
    labels = np.array(file["/meta"]).astype("uint8")

    return images, labels

_read_many_funcs = dict(
    disk=read_many_disk, lmdb=read_many_lmdb, hdf5=read_many_hdf5
)

With the reading functions stored in a dictionary as with the writing functions, you’re all set for the experiment.

Experiment for Reading Many Images

You can now run the experiment for reading many images out:

from timeit import timeit

read_many_timings = {"disk": [], "lmdb": [], "hdf5": []}

for cutoff in cutoffs:
    for method in ("disk", "lmdb", "hdf5"):
        t = timeit(
            "_read_many_funcs[method](num_images)",
            setup="num_images=cutoff",
            number=1,
            globals=globals(),
        )
        read_many_timings[method].append(t)

        # Print out the method, cutoff, and elapsed time
        print(f"Method: {method}, No. images: {cutoff}, Time usage: {t}")

As we did previously, you can graph the read experiment results:

The top graph shows the normal, unadjusted read times, showing the drastic difference between reading from .png files and LMDB or HDF5.

In contrast, the graph on the bottom shows the log of the timings, highlighting the relative differences with fewer images. Namely, we can see how HDF5 starts out behind but, with more images, becomes consistently faster than LMDB by a small margin.

Using the same plotting function as for the write timings, we have the following:

disk_x_r = read_many_timings["disk"]
lmdb_x_r = read_many_timings["lmdb"]
hdf5_x_r = read_many_timings["hdf5"]

plot_with_legend(
    cutoffs,
    [disk_x_r, lmdb_x_r, hdf5_x_r],
    ["PNG files", "LMDB", "HDF5"],
    "Number of images",
    "Seconds to read",
    "Read time",
    log=False,
)

plot_with_legend(
    cutoffs,
    [disk_x_r, lmdb_x_r, hdf5_x_r],
    ["PNG files", "LMDB", "HDF5"],
    "Number of images",
    "Seconds to read",
    "Log read time",
    log=True,
)

In practice, the write time is often less critical than the read time. Imagine that you are training a deep neural network on images, and only half of your entire image dataset fits into RAM at once. Each epoch of training a network requires the entire dataset, and the model needs a few hundred epochs to converge. You will essentially be reading half of the dataset into memory every epoch.

There are several tricks people do, such as training pseudo-epochs to make this slightly better, but you get the idea.

Now, look again at the read graph above. The difference between a 40-second and 4-second read time suddenly is the difference between waiting six hours for your model to train, or forty minutes!

If we view the read and write times on the same chart, we have the following:

You can plot all the read and write timings on a single graph using the same plotting function:

plot_with_legend(
    cutoffs,
    [disk_x_r, lmdb_x_r, hdf5_x_r, disk_x, lmdb_x, hdf5_x],
    [
        "Read PNG",
        "Read LMDB",
        "Read HDF5",
        "Write PNG",
        "Write LMDB",
        "Write HDF5",
    ],
    "Number of images",
    "Seconds",
    "Log Store and Read Times",
    log=False,
)

When you’re storing images as .png files, there is a big difference between write and read times. However, with LMDB and HDF5, the difference is much less marked. Overall, even if read time is more critical than write time, there is a strong argument for storing images using LMDB or HDF5.

Now that you’ve seen the performance benefits of LMDB and HDF5, let’s look at another crucial metric: disk usage.

Considering Disk Usage

Speed is not the only performance metric you may be interested in. We’re already dealing with very large datasets, so disk space is also a very valid and relevant concern.

Suppose you have an image dataset of 3TB. Presumably, you have them already on disk somewhere, unlike our CIFAR example, so by using an alternate storage method, you are essentially making a copy of them, which also has to be stored. Doing so will give you huge performance benefits when you use the images, but you’ll need to make sure you have enough disk space.

How much disk space do the various storage methods use? Here’s the disk space used for each method for each quantity of images:

I used the Linux du -h -c folder_name/* command to compute the disk usage on my system. There is some approximation inherent with this method due to rounding, but here’s the general comparison:

# Memory used in KB
disk_mem = [24, 204, 2004, 20032, 200296]
lmdb_mem = [60, 420, 4000, 39000, 393000]
hdf5_mem = [36, 304, 2900, 29000, 293000]

X = [disk_mem, lmdb_mem, hdf5_mem]

ind = np.arange(3)
width = 0.35

plt.subplots(figsize=(8, 10))
plots = [plt.bar(ind, [row[0] for row in X], width)]
for i in range(1, len(cutoffs)):
    plots.append(
        plt.bar(
            ind, [row[i] for row in X], width, bottom=[row[i - 1] for row in X]
        )
    )

plt.ylabel("Memory in KB")
plt.title("Disk memory used by method")
plt.xticks(ind, ("PNG", "LMDB", "HDF5"))
plt.yticks(np.arange(0, 400000, 100000))

plt.legend(
    [plot[0] for plot in plots], ("10", "100", "1,000", "10,000", "100,000")
)
plt.show()

Both HDF5 and LMDB take up more disk space than if you store using normal .png images. It’s important to note that both LMDB and HDF5 disk usage and performance depend highly on various factors, including operating system and, more critically, the size of the data you store.

LMDB gains its efficiency from caching and taking advantage of OS page sizes. You don’t need to understand its inner workings, but note that with larger images, you will end up with significantly more disk usage with LMDB, because images won’t fit on LMDB’s leaf pages, the regular storage location in the tree, and instead you will have many overflow pages. The LMDB bar in the chart above will shoot off the chart.

Our 32x32x3 pixel images are relatively small compared to the average images you may use, and they allow for optimal LMDB performance.

While we won’t explore it here experimentally, in my own experience with images of 256x256x3 or 512x512x3 pixels, HDF5 is usually slightly more efficient in terms of disk usage than LMDB. This is a good transition into the final section, a qualitative discussion of the differences between the methods.

Discussion

There are other distinguishing features of LMDB and HDF5 that are worth knowing about, and it’s also important to briefly discuss some of the criticisms of both methods. Several links are included along with the discussion if you want to learn more.

Parallel Access

A key comparison that we didn’t test in the experiments above is concurrent reads and writes. Often, with such large datasets, you may want to speed up your operation through parallelization.

In the majority of cases, you won’t be interested in reading parts of the same image at the same time, but you will want to read multiple images at once. With this definition of concurrency, storing to disk as .png files actually allows for complete concurrency. Nothing prevents you from reading several images at once from different threads, or writing multiple files at once, as long as the image names are different.

How about LMDB? There can be multiple readers on an LMDB environment at a time, but only one writer, and writers do not block readers. You can read more about that at the LMDB technology website.

Multiple applications can access the same LMDB database at the same time, and multiple threads from the same process can also concurrently access the LMDB for reads. This allows for even quicker read times: if you divided all of CIFAR into ten sets, then you could set up ten processes to each read in one set, and it would divide the loading time by ten.

HDF5 also offers parallel I/O, allowing concurrent reads and writes. However, in implementation, a write lock is held, and access is sequential, unless you have a parallel file system.

There are two main options if you are working on such a system, which are discussed more in depth in this article by the HDF Group on parallel IO. It can get quite complicated, and the simplest option is to intelligently split your dataset into multiple HDF5 files, such that each process can deal with one .h5 file independently of the others.

Documentation

If you Google lmdb, at least in the United Kingdom, the third search result is IMDb, the Internet Movie Database. That’s not what you were looking for!

Actually, there is one main source of documentation for the Python binding of LMDB, which is hosted on Read the Docs LMDB. While the Python package hasn’t even reached version > 0.94, it is quite widely used and is considered stable.

As for the LMDB technology itself, there is more detailed documentation at the LMDB technology website, which can feel a bit like learning calculus in second grade, unless you start from their Getting Started page.

For HDF5, there is very clear documentation at the h5py docs site, as well as a helpful blog post by Christopher Lovell, which is an excellent overview of how to use the h5py package. The O’Reilly book, Python and HDF5 also is a good way to get started.

While not as documented as perhaps a beginner would appreciate, both LMDB and HDF5 have large user communities, so a deeper Google search usually yields helpful results.

A More Critical Look at Implementation

There is no utopia in storage systems, and both LMDB and HDF5 have their share of pitfalls.

A key point to understand about LMDB is that new data is written without overwriting or moving existing data. This is a design decision that allows for the extremely quick reads you witnessed in our experiments, and also guarantees data integrity and reliability without the additional need of keeping transaction logs.

Remember, however, that you needed to define the map_size parameter for memory allocation before writing to a new database? This is where LMDB can be a hassle. Suppose you have created an LMDB database, and everything is wonderful. You’ve waited patiently for your enormous dataset to be packed into a LMDB.

Then, later down the line, you remember that you need to add new data. Even with the buffer you specified on your map_size, you may easily expect to see the lmdb.MapFullError error. Unless you want to re-write your entire database, with the updated map_size, you’ll have to store that new data in a separate LMDB file. Even though one transaction can span multiple LMDB files, having multiple files can still be a pain.

Additionally, some systems have restrictions on how much memory may be claimed at once. In my own experience, working with high-performance computing (HPC) systems, this has proved extremely frustrating, and has often made me prefer HDF5 over LMDB.

With both LMDB and HDF5, only the requested item is read into memory at once. With LMDB, key-unit pairs are read into memory one by one, while with HDF5, the dataset object can be accessed like a Python array, with indexing dataset[i], ranges, dataset[i:j] and other splicing dataset[i:j:interval].

Because of the way the systems are optimized, and depending on your operating system, the order in which you access items can impact performance.

In my experience, it’s generally true that for LMDB, you may get better performance when accessing items sequentially by key (key-value pairs being kept in memory ordered alphanumerically by key), and that for HDF5, accessing large ranges will perform better than reading every element of the dataset one by one using the following:

# Slightly slower
for i in range(len(dataset)):
    # Read the ith value in the dataset, one at a time
    do_something_with(dataset[i])

# This is better
data = dataset[:]
for d in data:
    do_something_with(d)

If you are considering a choice of file storage format to write your software around, it would be remiss not to mention Moving away from HDF5 by Cyrille Rossant on the pitfalls of HDF5, and Konrad Hinsen’s response On HDF5 and the future of data management, which shows how some of the pitfalls can be avoided in his own use cases with many smaller datasets rather than a few enormous ones. Note that a relatively smaller dataset is still several GB in size.

Integration With Other Libraries

If you’re dealing with really large datasets, it’s highly likely that you’ll be doing something significant with them. It’s worthwhile to consider deep learning libraries and what kind of integration there is with LMDB and HDF5.

First of all, all libraries support reading images from disk as .png files, as long as you convert them into NumPy arrays of the expected format. This holds true for all the methods, and we have already seen above that it is relatively straightforward to read in images as arrays.

Here are several of the most popular deep learning libraries and their LMDB and HDF5 integration:

Caffe has a stable, well-supported LMDB integration, and it handles the reading step transparently. The LMDB layer can also easily be replaced with a HDF5 database.
Keras uses the HDF5 format to save and restore models. This implies that TensorFlow can as well.
TensorFlow has a built-in class LMDBDataset that provides an interface for reading in input data from an LMDB file and can produce iterators and tensors in batches. TensorFlow does not have a built-in class for HDF5, but one can be written that inherits from the Dataset class. I personally use a custom class altogether that is designed for optimal read access based on the way I structure my HDF5 files.
Theano does not natively support any particular file format or database, but as previously stated, can use anything as long as it is read in as an N-dimensional array.

While far from comprehensive, this hopefully gives you a feel for the LMDB/HDF5 integration by some key deep learning libraries.

A Few Personal Insights on Storing Images in Python

In my own daily work analyzing terabytes of medical images, I use both LMDB and HDF5, and have learned that, with any storage method, forethought is critical.

Often, models need to be trained using k-fold cross validation, which involves splitting the entire dataset into k-sets (k typically being 10), and k models being trained, each with a different k-set used as test set. This ensures that the model is not overfitting the dataset, or, in other words, unable to make good predictions on unseen data.

A standard way to craft a k-set is to put an equal representation of each type of data represented in the dataset in each k-set. Thus, saving each k-set into a separate HDF5 dataset maximizes efficiency. Sometimes, a single k-set cannot be loaded into memory at once, so even the ordering of data within a dataset requires some forethought.

With LMDB, I similarly am careful to plan ahead before creating the database(s). There are a few good questions worth asking before you save images:

How can I save the images such that most of the reads will be sequential?
What are good keys?
How can I calculate a good map_size, anticipating potential future changes in the dataset?
How large can a single transaction be, and how should transactions be subdivided?

Regardless of the storage method, when you’re dealing with large image datasets, a little planning goes a long way.

Conclusion

You’ve made it to the end! You’ve now had a bird’s eye view of a large topic.

In this article, you’ve been introduced to three ways of storing and accessing lots of images in Python, and perhaps had a chance to play with some of them. All the code for this article is in a Jupyter notebook here or Python script here. Run at your own risk, as a few GB of your disk space will be overtaken by little square images of cars, boats, and so on.

You’ve seen evidence of how various storage methods can drastically affect read and write time, as well as a few pros and cons of the three methods considered in this article. While storing images as .png files may be the most intuitive, there are large performance benefits to considering methods such as HDF5 or LMDB.

Feel free to discuss in the comment section the excellent storage methods not covered in this article, such as LevelDB, Feather, TileDB, Badger, BoltDB, or anything else. There is no perfect storage method, and the best method depends on your specific dataset and use cases.

Writing Cleaner Python Code With PyLint

2019-05-14T14:00:00+00:00

PyLint is a well-known static analysis tool for Python 2 and 3. It has a number of useful features, like checking your code for compliance with the PEP 8 Python style guide. It makes sure that your code follows the code style guide and it can also automatically identify common bugs and errors in your Python code.

In this video series you’ll see how to install and set up the PyLint code linter tool. You’ll learn why you should use code linters like PyLint, Flake8, PyFlakes, or other static analysis tools—and how they can help you write cleaner and more Pythonic code.

You can get this setup up and running in a few minutes and it’ll quickly help you write better and cleaner Python code.

Additional Resources

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Playing and Recording Sound in Python

2019-05-13T14:00:00+00:00

If you want to use Python to play or record sound, then you’ve come to the right place! In this tutorial, you’ll learn how to play and record sound in Python using some of the most popular audio libraries. You will learn about the most straight-forward methods for playing and recording sound first, and then you’ll learn about some libraries that offer some more functionality in exchange for a few extra lines of code.

By the end of this tutorial, you’ll know how to:

Play MP3 and WAV files, as well as a range of other audio formats
Play NumPy and Python arrays containing sound
Record sound using Python
Save your recordings or audio files in a range of different file formats

For a comprehensive list of audio-related Python libraries, have a look at the wiki page on audio in Python.

Playing Audio Files

Below, you’ll see how to play audio files with a selection of Python libraries. A few of these libraries let you play a range of audio formats, including MP3 and NumPy arrays. All of the libraries below let you play WAV files, some with a few more lines of code than others:

playsound is the most straightforward package to use if you simply want to play a WAV or MP3 file. It offers no functionality other than simple playback.
simpleaudio lets you play WAV files and NumPy arrays, and gives you options to check whether a file is still playing.
winsound allows you to play WAV files or beep your speakers, but it works only on Windows.
python-sounddevice and pyaudio provide bindings for the PortAudio library for cross-platform playback of WAV files.
pydub requires pyaudio for audio playback, but with ffmpeg installed, it lets you play a large range of audio formats with only a few lines of code.

Let’s have a look at these libraries for audio playback one by one.

`playsound`

playsound is a “pure Python, cross platform, single function module with no dependencies for playing sounds.” With this module, you can play a sound file with a single line of code:

from playsound import playsound

playsound('myfile.wav')

The documentation of playsound states that it has been tested on WAV and MP3 files, but it may work for other file formats as well.

This library was last updated in June 2017. It seems to work well at the time of writing this article, but it’s not clear whether it will still support newer Python releases.

`simpleaudio`

simpleaudio is a cross-platform library for playback of (mono and stereo) WAV files with no dependencies. The following code can be used to play a WAV file, and wait for the file to finish playing before terminating the script:

import simpleaudio as sa

filename = 'myfile.wav'
wave_obj = sa.WaveObject.from_wave_file(filename)
play_obj = wave_obj.play()
play_obj.wait_done()  # Wait until sound has finished playing

WAV files contain a sequence of bits representing the raw audio data, as well as headers with metadata in RIFF (Resource Interchange File Format) format.

For CD recordings, the industry standard is to store each audio sample (an individual audio datapoint relating to air pressure) as a 16-bit value, at 44100 samples per second.

To reduce file size, it may be sufficient to store some recordings (for example of human speech) at a lower sampling rate, such as 8000 samples per second, although this does mean that higher sound frequencies may not be as accurately represented.

A few of the libraries discussed in this tutorial play and record bytes objects, whereas others use NumPy arrays to store raw audio data.

Both correspond to a sequence of data points that can be played back at a specified sample rate in order to play a sound. For a bytes object, each sample is stored as a set of two 8-bit values, whereas in a NumPy array, each element can contain a 16-bit value corresponding to a single sample.

An important difference between these two data types is that bytes objects are immutable, whereas NumPy arrays are mutable, making the latter more suitable for generating sounds and for more complex signal processing. For more information on how to work with NumPy, have a look at our NumPy tutorials.

simpleaudio allows you to play NumPy and Python arrays and bytes objects using simpleaudio.play_buffer(). Make sure you have NumPy installed for the following example to work, as well as simpleaudio. (With pip installed, you can do this by running pip install numpy from your console.)

For more information on how to use pip for installing packages, have a look at Pipenv: A Guide to the New Python Packaging Tool.

Below you’ll see how to generate a NumPy array corresponding to a 440 Hz tone and play it back using simpleaudio.play_buffer():

import numpy as np
import simpleaudio as sa

frequency = 440  # Our played note will be 440 Hz
fs = 44100  # 44100 samples per second
seconds = 3  # Note duration of 3 seconds

# Generate array with seconds*sample_rate steps, ranging between 0 and seconds
t = np.linspace(0, seconds, seconds * fs, False)

# Generate a 440 Hz sine wave
note = np.sin(frequency * t * 2 * np.pi)

# Ensure that highest value is in 16-bit range
audio = note * (2**15 - 1) / np.max(np.abs(note))
# Convert to 16-bit data
audio = audio.astype(np.int16)

# Start playback
play_obj = sa.play_buffer(audio, 1, 2, fs)

# Wait for playback to finish before exiting
play_obj.wait_done()

Next, let’s see how you can use winsound to play WAV files on a Windows machine.

`winsound`

If you use Windows, you can use the built-in winsound module to access its basic sound-playing machinery. Playing a WAV file can be done in a few lines of code:

import winsound

filename = 'myfile.wav'
winsound.PlaySound(filename, winsound.SND_FILENAME)

winsound does not support playback of any files other than WAV files. It does allow you to beep your speakers using winsound.Beep(frequency, duration). For example, you can beep a 1000 Hz tone for 100 milliseconds with the following code:

import winsound

winsound.Beep(1000, 100)  # Beep at 1000 Hz for 100 ms

Next, you’ll learn how to use the python-sounddevice module for cross-platform audio playback.

`python-sounddevice`

As stated in its documentation, python-sounddevice “provides bindings for the PortAudio library and a few convenience functions to play and record NumPy arrays containing audio signals”. In order to play WAV files, numpy and soundfile need to be installed, to open WAV files as NumPy arrays.

With python-sounddevice, numpy, and soundfile installed, you can now read a WAV file as a NumPy array and play it back:

import sounddevice as sd
import soundfile as sf

filename = 'myfile.wav'
# Extract data and sampling rate from file
data, fs = sf.read(filename, dtype='float32')  
sd.play(data, fs)
status = sd.wait()  # Wait until file is done playing

The line containing sf.read() extracts the raw audio data, as well as the sampling rate of the file as stored in its RIFF header, and sounddevice.wait() ensures that the script is only terminated after the sound finishes playing.

Next, we’ll learn how to use pydub to play sound. With the right dependencies installed, it allows you to play a wide range of audio files, and it offers you more options for working with audio than python-soundevice does.

`pydub`

Although pydub can open and save WAV files without any dependencies, you need to have an audio playback package installed to play audio. simpleaudio is strongly recommended, but pyaudio, ffplay, and avplay are alternative options.

The following code can be used to play a WAV file with pydub:

from pydub import AudioSegment
from pydub.playback import play

sound = AudioSegment.from_wav('myfile.wav')
play(sound)

In order to play back other audio types, such as MP3 files, ffmpeg or libav should be installed. Have a look at the documentation of pydub for instructions. As an alternative to the steps described in the documentation, ffmpeg-python provides bindings for ffmpeg, and can be installed using pip:

$ pip install ffmpeg-python

With ffmpeg installed, playing back an MP3 file requires only a small change in our earlier code:

from pydub import AudioSegment
from pydub.playback import play

sound = AudioSegment.from_mp3('myfile.mp3')
play(sound)

Using the AudioSegment.from_file(filename, filetype) construction, you can play any type of audio file that ffmpeg supports. For example, you may play a WMA file using the ollowing:

sound = AudioSegment.from_file('myfile.wma', 'wma')

In addition to playing back sound files, pydub lets you save audio in different file formats (more on this later), slice audio, calculate the length of audio files, fade in or out, and apply cross-fades.

AudioSegment.reverse() creates a copy of the AudioSegment that plays backwards, which the documentation describes as “useful for Pink Floyd, screwing around, and some audio processing algorithms.”

`pyaudio`

pyaudio provides bindings for PortAudio, the cross-platform audio I/O library. This means that you can use pyaudio to play and record audio on a variety of platforms, including Windows, Linux, and Mac. With pyaudio, playing audio is done by writing to a .Stream:

import pyaudio
import wave

filename = 'myfile.wav'

# Set chunk size of 1024 samples per data frame
chunk = 1024  

# Open the sound file 
wf = wave.open(filename, 'rb')

# Create an interface to PortAudio
p = pyaudio.PyAudio()

# Open a .Stream object to write the WAV file to
# 'output = True' indicates that the sound will be played rather than recorded
stream = p.open(format = p.get_format_from_width(wf.getsampwidth()),
                channels = wf.getnchannels(),
                rate = wf.getframerate(),
                output = True)

# Read data in chunks
data = wf.readframes(chunk)

# Play the sound by writing the audio data to the stream
while data != '':
    stream.write(data)
    data = wf.readframes(chunk)

# Close and terminate the stream
stream.close()
p.terminate()

As you may have noticed, playing sounds with pyaudio is a bit more complex than playing sounds with the libraries you’ve seen earlier. This means that it may not be your first choice if you just want to play a sound effect in your Python application.

However, because pyaudio gives you more low-level control, it is possible to get and set parameters for your input and output devices, and to check your CPU load and input or output latency.

It also allows you to play and record audio in callback mode, where a specified callback function is called when new data is required for playback, or available for recording. These options make pyaudio a suitable library to use if your audio needs go beyond simple playback.

Now that you’ve seen how you can use a number of different libraries to play audio, it’s time to see how you can use Python to record audio yourself.

Recording Audio

The python-sounddevice and pyaudio libraries provide ways to record audio with Python. python-sounddevice records to NumPy arrays and pyaudio records to bytes objects. Both of these can be stored as WAV files using the scipy and wave libraries, respectively.

`python-sounddevice`

python-sounddevice allows you to record audio from your microphone and store it as a NumPy array. This is a handy datatype for sound processing that can be converted to WAV format for storage using the scipy.io.wavfile module. Make sure to install the scipy module for the following example (pip install scipy). This automatically installs NumPy as one of its dependencies:

import sounddevice as sd
from scipy.io.wavfile import write

fs = 44100  # Sample rate
seconds = 3  # Duration of recording

myrecording = sd.rec(int(seconds * fs), samplerate=fs, channels=2)
sd.wait()  # Wait until recording is finished
write('output.wav', fs, myrecording)  # Save as WAV file

`pyaudio`

Earlier in this article, you learned how to play sounds by reading a pyaudio.Stream(). Recording audio can be done by writing to this stream instead:

import pyaudio
import wave

chunk = 1024  # Record in chunks of 1024 samples
sample_format = pyaudio.paInt16  # 16 bits per sample
channels = 2
fs = 44100  # Record at 44100 samples per second
seconds = 3
filename = "output.wav"

p = pyaudio.PyAudio()  # Create an interface to PortAudio

print('Recording')

stream = p.open(format=sample_format,
                channels=channels,
                rate=fs,
                frames_per_buffer=chunk,
                input=True)

frames = []  # Initialize array to store frames

# Store data in chunks for 3 seconds
for i in range(0, int(fs / chunk * seconds)):
    data = stream.read(chunk)
    frames.append(data)

# Stop and close the stream 
stream.stop_stream()
stream.close()
# Terminate the PortAudio interface
p.terminate()

print('Finished recording')

# Save the recorded data as a WAV file
wf = wave.open(filename, 'wb')
wf.setnchannels(channels)
wf.setsampwidth(p.get_sample_size(sample_format))
wf.setframerate(fs)
wf.writeframes(b''.join(frames))
wf.close()

Now that you’ve seen how to record audio with python-sounddevice and pyaudio, you’ll learn how to convert your recording (or any other audio files) to a range of different audio formats.

Saving and Converting Audio

You saw earlier that you can use the scipy.io.wavfile module to store NumPy arrays as WAV files. The wavio module similarly lets you convert between WAV files and NumPy arrays. If you want to store your audio in a different file format, pydub and soundfile come in handy, as they allow you to read and write a range of popular file formats (such as MP3, FLAC, WMA and FLV).

`wavio`

This module depends on numpy and lets you read WAV files as NumPy arrays, and save NumPy arrays as WAV files.

To save a NumPy array as a WAV file, you can use wavio.write():

import wavio

wavio.write("myfile.wav", my_np_array, fs, sampwidth=2)

In this example, my_np_array is a NumPy array containing audio, fs is the sample rate of the recording (usually 44100 or 44800 Hz), and sampwidth is the sampling width of the audio (the number of bytes per sample, typically 1 or 2 bytes).

`soundfile`

The soundfile library can read and write all file formats supported by libsndfile. Although it can’t play back audio, it allows you to convert audio from and to FLAC, AIFF, and a few audio formats that are less common . To convert a WAV file to FLAC, you can use the following code:

import soundfile as sf

# Extract audio data and sampling rate from file 
data, fs = sf.read('myfile.wav') 
# Save as FLAC file at correct sampling rate
sf.write('myfile.flac', data, fs)

Similar code will work for converting between other file formats supported by libsndfile.

`pydub`

pydub lets you save audio in any format that ffmpeg supports, which includes nearly all audio types you might encounter in your daily life. For example, you can convert your WAV file to MP3 with the following code:

from pydub import AudioSegment
sound = AudioSegment.from_wav('myfile.wav')

sound.export('myfile.mp3', format='mp3')

Using AudioSegment.from_file() is a more general way of loading audio files. For example, if you want to convert your file back from MP3 to WAV, you can do the following:

from pydub import AudioSegment
sound = AudioSegment.from_file('myfile.mp3', format='mp3')

sound.export('myfile.wav', format='wav')

This code should work for any audio file format that ffmpeg supports.

Comparison of Audio Libraries

Below is handy table that compares the functionality of the libraries discussed in this tutorial:

Library	Platform	Playback	Record	Convert	Dependencies
`playsound`	Cross-platform	WAV, MP3	-	-	None
`simpleaudio`	Cross-platform	WAV, array, `bytes`	-	-	None
`winsound`	Windows	WAV	-	-	None
`sounddevice`	Cross-platform	NumPy array	NumPy array	-	`numpy`, `soundfile`
`pydub`	Cross-platform	Any type supported by `ffmpeg`	-	Any type supported by `ffmpeg`	`simpleaudio`, `ffmpeg`
`pyaudio`	Cross-platform	`bytes`	`bytes`	-	`wave`
`wavio`	Cross-platform	-	-	WAV, NumPy array	`numpy`, `wave`
`soundfile`	Cross-platform	-	-	Any type supported by `libsndfile`	`CFFI`, `libsndfile`, `numpy`

The libraries that are included in this tutorial are chosen for their ease of use and popularity. For a more comprehensive list of audio libraries for Python, have a look at the wiki page on audio in Python.

Conclusion: Playing and Recording Sound in Python

In this tutorial, you learned how to use some of the most popular audio libraries to play and record audio in Python. You also saw how to save your audio in a range of different formats.

You are now able to:

Play a large range of audio formats, including WAV, MP3 and NumPy arrays
Record audio from your microphone to a NumPy or Python array
Store your recorded audio a range of different formats, including WAV and MP3
Convert your sound files to a range of different audio formats

You now have the information you need to help you decide which libraries to use to start working with audio in Python. Go forth, and develop some awesome audio applications!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Bob and Julian of PyBites

2019-05-08T14:00:00+00:00

This week, I’m joined by Bob Belderbos and Julian Sequeira of PyBites fame. Bob is a Software Developer at Oracle, Spain. Julian is a Data Center Technician at Amazon Web Services, in Australia.

Join us as we discuss how PyBites got started and what they have in store for its future. We’ll also look into Bob’s secret love of drawing and Julian’s not so secret love of a good pint.

Ricky: Welcome, Bob and Julian! Or is it Julian and Bob? In either case, thank you for joining me for this interview. Let’s start with the usual first questions. How’d you get into programming? When did you start using Python?

Julian: We prefer to go by “Boolean.” Not really, but let’s make it stick.

I was first bitten by the programming bug (pun intended!) back in high school but really dove into coding in late 2015.

I needed a way to track the overtime hours I was clocking in my Field Engineer role at Oracle to ensure I was being paid correctly. Enter Python (at Bob’s recommendation).

I made myself a CLI overtime tracker that comprised of a rudimentary menu system with simple options to calculate my post-tax overtime pay. It was basic, worked a treat, and got me hooked on Python.

Tip: It’s the real-world use cases that make the learning stick!

The speed and simplicity of Python compared to my high school days using C++ had me hooked. Then we started PyBites, and the Python got real. (Real Python, get it?)

Bob: For me, it all started at Sun Microsystems back in 2009. I moved into the Systems support group and found a niche to develop a web app to parse diagnostic outputs, of which we were getting thousands every month. This became a staple tool for the support organization, saving them countless hours of tedious work every day.

This diagnostic tool was a wild mix of shell scripting (bash/sed/awk), Perl, PHP, and some jQuery. The business decided to move to full automation of service requests, for which I wrote a new framework in Perl, which soon became a maintenance nightmare, so I started looking for other solutions.

This was back in 2012. I stumbled on Python and started learning it. It was love at first sight: with its clean and elegant design (type import this in your REPL), lack of braces, and other C-like syntax, it just reads like English! It’s simple to learn the basics of Python, yet under the covers, it’s very versatile.

There is a great article by Eric Raymond in Linux Journal titled Why Python?, which describes his revelation when he changed from Perl to Python.

I experienced something similar. After a fast rewrite of the automation framework in Python, it became relatively “easy” to make changes, and we expanded it with many exciting new features, all without major headaches. Although I transitioned into another role, the solution is still actively maintained. Since then, I have been fortunate to use Python for almost all my work.

Ricky: You are both most famous (for better or worse) for being the co-founders of PyBites. For those who don’t know about PyBites, what is it, and how did it get started?

Boolean: PyBites just started out as a simple blog to share what we were learning in the Python space. (We use a static site generator called Pelican.) We were keen to push ourselves past just reading books and watching videos. Writing blog posts made us really dive deep to ensure we were communicating the right concepts.

Shortly after its inception, we started to test out some ideas. One was to challenge ourselves, as inspired by Noah Kagan’s coffee challenge. So one day Bob came up with a Python exercise for us to solve by the end of the week.

We found a JavaScript course page with mm:ss timings per video, but not the total duration. The task was to scrape the site and add up all the video timestamps, calculating the total duration of the course.

At the end of the week (sprint), we compared our solutions and retrospected on our learnings. We had this “Fight Club moment” (maybe not as violent…) of “we should do this again sometime,” and that’s how our blog/community code challenges line was born.

We looked into GitHub, set up a challenges repo for us to publish our challenges and solutions to, and then created a separate community branch to have others pull request their work to.

The traction we gained from this sparked the idea that became our coding platform: CodeChalleng.es. It comprises of a growing collection of almost 200 Python exercises (called Bites of Py), which you can code in the comfort of your browser.

Another huge thing for us was completing the #100DaysOfCode challenge, which we actively shared on our blog and social media. This got us interviewed on the Talk Python podcast and, in turn, led to us producing the #100DaysOfCode in Python course together with Michael Kennedy.

An important lesson for us here was that you just need to get started and drop perfectionism. A lot of awesome stuff that happened was not planned but came to us because we put something out there that generated interest, and we acted upon the valuable feedback we received in the process!

Ricky: Julian, you’re not a programmer by day, and you work as a Data Center Technician at Amazon Web Services. How has it been learning to code alongside someone as accomplished as Bob? I would assume it would help to have someone more experienced as your business partner? Is there anything that you were surprised by or found difficult as you learned to code?

Julian: The Pythonic gap between Bob and me has been the cornerstone of PyBites. It’s actually what we figured would give our blog a unique spin!

To that end, it’s been a bit of a mixed bag. When you code alongside someone as capable as Bob, the first feeling is that of awe. You see the elegance of their code and their ability to solve problems you’ve spent hours on, in mere moments. It’s truly inspirational!

But not always. It becomes easy to fall into the Imposter Syndrome trap:

“Is my code ever going to be as good?”
“Why do I even bother?”
“He’s just always going to be better!”

The reality is that there’s always going to be someone better when it comes to code, and life in general! I quickly learned that rather than focusing on the skill gap, I should be focusing on my own code, being proud of my achievements, and embracing what I bring to the table.

Funnily enough, and this applies to both of us, the difficulties we hit with learning and managing code were negligible. That is, we accept it and love it. (I say this now but never in the moment!)

The surprisingly difficult part has been the actual business side of PyBites! I’ll also add that my days of not being a professional coder may be coming to an end! (Shhh!)

Ricky: Bob, in your own words, when you’re “not hard at work for the Man” in your day job, you love to tinker and build projects in your spare time. What kind of projects do you have on the go currently?

Bob: Currently, almost all of my time goes into producing course content and improving our platform by adding more exercises and features. With increased use, there are more questions and feedback/requests, but I enjoy every minute of it because it provides a great way to teach Python and mentor other developers, something I really enjoy.

I am also taking the Coursera Applied Data Science specialization because I love data and want to integrate this more in my day-to-day work.

When this and our upcoming course (suspense…) is done, my default workflow will still be similar:

Take a new concept (Python module, data, web technology, automation opportunity, and so on)
Study it
Build something cool and blog/share the process

It’s what I am passionate about and adds to PyBites’ growing body of content.

This is actually something I recommend for any programmer/dev. CVs are legacy. Start building your blog/GitHub/brand so you have a portfolio you can show. It also enables you to re-use what you build for future projects, and it’s a great way to network/collaborate with others.

Ricky: It’s obvious you both share the entrepreneurial gene. (You only have to look at Bob’s bookshelf to see the evidence.) What’s been the hardest or most challenging part for you with starting your own business/side-hustle (PyBites)? Do you find your respective talents offset each other?

Boolean: Hands down the most challenging part has been managing our priorities.

The key priorities in both of our lives happen to be very similar:

Family/kids
Full-time day jobs
Learning
PyBites

As PyBites grows, so too does the required time investment. We can’t take that time away from our day jobs or our families, so it’s definitely been the greatest hurdle trying to find an acceptable balance.

The outlook we’ve adopted is that “if we have time to watch Netflix, then we have time to work on PyBites.”

To that end, more often than not, we consciously make the choice to not watch TV, play games, or go out, and make that time our PyBites time. It doesn’t sound like much, but after a long day at work and getting the kids to bed, getting back on the tools to work on PyBites can be extremely challenging!

That said, we openly tell anyone who’ll listen that we wouldn’t be where we are without our mutual support of one another.

If you’re taking on anything worthwhile, then get yourself an accountability partner. We hold each other accountable for the items we take on and build each other up when things don’t exactly go our way.

Our different talents most definitely offset each other. Julian tends to be more of the writer, storyteller, marketer, business manager, and “mouth,” whereas Bob pushes deep into code, builds and maintains the tools, comes up with incredible ideas, and supports the community technically. Combine these two talent sets, and you have a well-oiled PyBites machine.

The best part (and sticking to our core belief that you learn by doing) is that we learn from one another. We push each other to continuously improve, to live outside the box, and always try something new.

Ricky: Now for my last questions. What else do you get up to in your spare time? What other hobbies and interests do you have, aside from Python and coding?

Bob: Haha Python and coding is a big chunk, but outside of that I love every minute I get to spend with my wife and two kids.

I am also keen on sticking to a daily fitness routine, especially as our work requires so much sitting and screen time! I love reading books and consuming podcasts. When I free up more time, language learning and drawing are two other things I really enjoy.

Julian: Okay, cutting out coding and Python, phew!

Like Bob, the priority is spending time with my wife and two kids as well. Anyone with young kids will know that you really don’t have much time after they’re done and dusted, so hobby time is limited these days.

When I do grab a few rare, non-PyBites minutes, I love to play my electric guitar, play video games, read, and tinker with my Raspberry Pi and home automation.

Additionally, being the social butterfly I am, I love to hang out with mates and enjoy a few beers. Get me to a pub, shout me a nice Pilsner, and I’ll be your best mate!

(Wait a sec. Did Bob just say he likes to draw?)

Ricky Are there any projects you wish to share with our readers? Where can we find out more about you guys and what you’re up to?

Boolean: Other than the blog, our current pride and joy is our online Python Coding Exercises platform.

We also created the #100DaysOfCode in Python course with Michael Kennedy and have become trainers on the Talk Python to Me training platform. Again, something we’re super proud of. (More to come in this space!)

You can find and follow Julian in the dark places below:

Twitter: @_juliansequeira
LinkedIn: https://www.linkedin.com/in/juliansequeira/
Udemy: I also created an introductory Flask course on Udemy!
Personal blog: https://www.techmoneykids.com/

Bob:

Twitter: @bbelderbos
LinkedIn: https://www.linkedin.com/in/bbelderbos/
Personal blog: https://bobbelderbos.com

Thank you Julian and Bob for an entertaining interview. If you haven’t checked out PyBites lately, then I encourage you to do so. If you have someone you’d like me to interview as part of this series, then leave a comment below. Happy coding!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Context Managers and the "with" Statement

2019-05-07T14:00:00+00:00

Python’s with statement is powered by a language feature called context managers. It helps you simplify some common resource management patterns by abstracting their functionality and allowing them to be factored out and reused.

In this course you’ll learn how context managers and the with statement work in Python, including the difference between class-based and function-based context managers, as well as some other tricks that will make you a more efficient Python developer.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Use sorted() and sort() in Python

2019-05-06T14:00:00+00:00

All programmers will have to write code to sort items or data at some point. Sorting can be critical to the user experience in your application, whether it’s ordering a user’s most recent activity by timestamp, or putting a list of email recipients in alphabetical order by last name. Python sorting functionality offers robust features to do basic sorting or customize ordering at a granular level.

In this guide, you’ll learn how to sort various types of data in different data structures, customize the order, and work with two different methods of sorting in Python.

By the end of this tutorial, you’ll know how to:

Implement basic Python sorting and ordering on data structures
Differentiate between sorted() and .sort()
Customize a complex sort order in your code based on unique requirements

For this tutorial, you’ll need a basic understanding of lists and tuples as well as sets. Those data structures will be used in this tutorial, and some basic operations will be performed on them. Also, this tutorial uses Python 3, so example output in this tutorial might vary slightly if you’re using Python 2.

Ordering Values With `sorted()`

To get started with Python sorting, you’re first going to see how to sort both numeric data and string data.

Sorting Numbers

You can use Python to sort a list by using sorted(). In this example, a list of integers is defined, and then sorted() is called with the numbers variable as the argument:

>>>

>>> numbers = [6, 9, 3, 1]
>>> sorted(numbers)
[1, 3, 6, 9]
>>> numbers
[6, 9, 3, 1]

The output from this code is a new, sorted list. When the original variable is printed, the initial values are unchanged.

This example shows four important characteristics of sorted():

The function sorted() did not have to be defined. It’s a built-in function that is available in a standard installation of Python.
sorted(), with no additional arguments or parameters, is ordering the values in numbers in an ascending order, meaning smallest to largest.
The original numbers variable is unchanged because sorted() provides sorted output and does not change the original value in place.
When sorted() is called, it provides an ordered list as a return value.

This last point means that sorted() can be used on a list, and the output can immediately be assigned to a variable:

>>>

>>> numbers = [6, 9, 3, 1]
>>> numbers_sorted = sorted(numbers)
>>> numbers_sorted
[1, 3, 6, 9]
>>> numbers
[6, 9, 3, 1]

In this example, there is now a new variable numbers_sorted that stored the output of sorted().

You can confirm all of these observations by calling help() on sorted(). The optional arguments key and reverse will be covered later in the tutorial:

>>>

>>> # Python 3
>>> help(sorted)
Help on built-in function sorted in module builtins:

sorted(iterable, /, *, key=None, reverse=False)
    Return a new list containing all items from the iterable in ascending order.

    A custom key function can be supplied to customize the sort order, and the
    reverse flag can be set to request the result in descending order.

Technical Detail: If you’re transitioning from Python 2 and are familiar with its function of the same name, you should be aware of a couple important changes in Python 3:

Python 3’s sorted() does not have a cmp parameter. Instead, only key is used to introduce custom sorting logic.
key and reverse must be passed as keyword arguments, unlike in Python 2, where they could be passed as positional arguments.

If you need to convert a Python 2 cmp function to a key function, then check out functools.cmp_to_key(). This tutorial will not cover any examples using Python 2.

sorted() can be used on tuples and sets very similarly:

>>>

>>> numbers_tuple = (6, 9, 3, 1)
>>> numbers_set = {5, 5, 10, 1, 0}
>>> numbers_tuple_sorted = sorted(numbers_tuple)
>>> numbers_set_sorted = sorted(numbers_set)
>>> numbers_tuple_sorted
[1, 3, 6, 9]
>>> numbers_set_sorted
[0, 1, 5, 10]

Notice how even though the input was a set and a tuple, the output is a list because sorted() returns a new list by definition. The returned object can be cast to a new type if it needs to match the input type. Be careful if attempting to cast the resulting list back to a set, as a set by definition is unordered:

>>>

>>> numbers_tuple = (6, 9, 3, 1)
>>> numbers_set = {5, 5, 10, 1, 0}
>>> numbers_tuple_sorted = sorted(numbers_tuple)
>>> numbers_set_sorted = sorted(numbers_set)
>>> numbers_tuple_sorted
[1, 3, 6, 9]
>>> numbers_set_sorted
[0, 1, 5, 10]
>>> tuple(numbers_tuple_sorted)
(1, 3, 6, 9)
>>> set(numbers_set_sorted)
{0, 1, 10, 5}

The numbers_set_sorted value when cast to a set is not ordered, as expected. The other variable, numbers_tuple_sorted, retained the sorted order.

Sorting Strings

str types sort similarly to other iterables, like list and tuple. The example below shows how sorted() iterates through each character in the value passed to it and orders them in the output:

>>>

>>> string_number_value = '34521'
>>> string_value = 'I like to sort'
>>> sorted_string_number = sorted(string_number_value)
>>> sorted_string = sorted(string_value)
>>> sorted_string_number
['1', '2', '3', '4', '5']
>>> sorted_string
[' ', ' ', ' ', 'I', 'e', 'i', 'k', 'l', 'o', 'o', 'r', 's', 't', 't']

sorted() will treat a str like a list and iterate through each element. In a str, each element means each character in the str. sorted() will not treat a sentence differently, and it will sort each character, including spaces.

.split() can change this behavior and clean up the output, and .join() can put it all back together. We will cover the specific order of the output and why it is that way shortly:

>>>

>>> string_value = 'I like to sort'
>>> sorted_string = sorted(string_value.split())
>>> sorted_string
['I', 'like', 'sort', 'to']
>>> ' '.join(sorted_string)
'I like sort to'

The original sentence in this example is converted into a list of words instead of leaving it as a str. That list is then sorted and combined to form a str again instead of a list.

Limitations and Gotchas With Python Sorting

It is worth noting some limitations and odd behavior that can arise when you’re using Python to sort values besides integers.

Lists With Non-Comparable Data Types Can’t Be `sorted()`

There are data types that can’t be compared to each other using just sorted() because they are too different. Python will return an error if you attempt to use sorted() on a list containing non-comparable data. In this example, a None and an int in the same list can’t be sorted because of their incompatibility:

>>>

>>> mixed_types = [None, 0]
>>> sorted(mixed_types)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'int' and 'NoneType'

This error shows why Python can’t sort the values given to it. It’s trying to put the values in order by using the less than operator (<) to determine which value is lower in sorting order. You can replicate this error by manually comparing the two values:

>>>

>>> None < 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'NoneType' and 'int'

The same TypeError is thrown when you try to compare two non-comparable values without using sorted().

If the values within the list can be compared and will not throw a TypeError, then the list can be sorted. This prevents sorting iterables with intrinsically unorderable values and producing output that may not make sense.

For example, should the number 1 come before the word apple? However, if a iterable contains a combination of integers and strings that are all numbers, they can be cast to comparable data types by using a list comprehension:

>>>

>>> mixed_numbers = [5, "1", 100, "34"]
>>> sorted(mixed_numbers)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'str' and 'int'
>>> # List comprehension to convert all values to integers
>>> [int(x) for x in mixed_numbers]
[5, 1, 100, 34]
>>> sorted([int(x) for x in mixed_numbers])
[1, 5, 34, 100]

Each element in mixed_numbers has int() called on it to convert any str values to int values. sorted() is then called and can successfully compare each element and provide a sorted output.

Python can also implicitly convert a value to another type. In the example below, the evaluation of 1 <= 0 is a false statement, so the output of the evaluation will be False. The number 1 can be converted to True as a bool type, while 0 converts to False.

Even though the elements in the list look different, they can all be converted to Booleans (True or False) and compared to each other using sorted():

>>>

>>> similar_values = [False, 0, 1, 'A' == 'B', 1 <= 0]
>>> sorted(similar_values)
[False, 0, False, False, 1]

'A' == 'B' and 1 <= 0 are converted to False and returned in the ordered output.

This example illustrates an important aspect of sorting: sort stability. In Python, when you sort equal values, they will retain their original order in the output. Even though the 1 moved, all the other values are equal so they retain their original order relative to each other. In the example below, all the values are considered equal and will retain their original positions:

>>>

>>> false_values = [False, 0, 0, 1 == 2, 0, False, False]
>>> sorted(false_values)
[False, 0, 0, False, 0, False, False]

If you inspect the original order and the sorted output, you will see that 1 == 2 is converted to False, and all sorted output is in the original order.

When You’re Sorting Strings, Case Matters

sorted() can be used on a list of strings to sort the values in ascending order, which appears to be alphabetically by default:

>>>

>>> names = ['Harry', 'Suzy', 'Al', 'Mark']
>>> sorted(names)
['Al', 'Harry', 'Mark', 'Suzy']

However, Python is using the Unicode Code Point of the first letter in each string to determine ascending sort order. This means that sorted() will not treat the names Al and al the same. This example uses ord() to return the Unicode Code Point of the first letter in each string:

>>>

>>> names_with_case = ['harry', 'Suzy', 'al', 'Mark']
>>> sorted(names_with_case)
['Mark', 'Suzy', 'al', 'harry']
>>> # List comprehension for Unicode Code Point of first letter in each word
>>> [(ord(name[0]), name[0]) for name in sorted(names_with_case)]
[(77, 'M'), (83, 'S'), (97, 'a'), (104, 'h')]

name[0] is returning the first character in each element of sorted(names_with_case), and ord() is providing the Unicode Code Point. Even though a comes before M in the alphabet, the code point for M comes before a, so the sorted output has M first.

If the first letter is the same, then sorted() will use the second character to determine order, and the third character if that is the same, and so on, all the way to the end of the string:

>>>

>>> very_similar_strs = ['hhhhhd', 'hhhhha', 'hhhhhc','hhhhhb']
>>> sorted(very_similar_strs)
['hhhhha', 'hhhhhb', 'hhhhhc', 'hhhhhd']

Each value of very_similar_strs is identical except for the last character. sorted() will compare the strings, and because the first five characters are the same, the output will be based on the sixth character.

Strings that contain identical values will end up sorted shortest to longest due to the shorter strings not having elements to compare to with the longer strings:

>>>

>>> different_lengths = ['hhhh', 'hh', 'hhhhh','h']
>>> sorted(different_lengths)
['h', 'hh', 'hhhh', 'hhhhh']

The shortest string, h, is ordered first with the longest, hhhhh, ordered last.

Using `sorted()` With a `reverse` Argument

As shown in the help() documentation for sorted(), there is an optional keyword argument called reverse, which will change the sorting behavior based on the Boolean assigned to it. If reverse is assigned True, then the sorting will be in descending order:

>>>

>>> names = ['Harry', 'Suzy', 'Al', 'Mark']
>>> sorted(names)
['Al', 'Harry', 'Mark', 'Suzy']
>>> sorted(names, reverse=True)
['Suzy', 'Mark', 'Harry', 'Al']

The sorting logic remains the same, meaning that the names are still being sorted by their first letter. But the output has been reversed with the reverse keyword set to True.

When False is assigned, the ordering will remain ascending. Any of the previous examples can be used to see the behavior of reverse using both True or False:

>>>

>>> names_with_case = ['harry', 'Suzy', 'al', 'Mark']
>>> sorted(names_with_case, reverse=True)
['harry', 'al', 'Suzy', 'Mark']
>>> similar_values = [False, 1, 'A' == 'B', 1 <= 0]
>>> sorted(similar_values, reverse=True)
[1, False, False, False]
>>> numbers = [6, 9, 3, 1]
>>> sorted(numbers, reverse=False)
[1, 3, 6, 9]

`sorted()` With a `key` Argument

One of the most powerful components of sorted() is the keyword argument called key. This argument expects a function to be passed to it, and that function will be used on each value in the list being sorted to determine the resulting order.

To demonstrate a basic example, let’s assume the requirement for ordering a specific list is the length of the strings in the list, shortest to longest. The function to return the length of a string, len(), will be used with the key argument:

>>>

>>> word = 'paper'
>>> len(word)
5
>>> words = ['banana', 'pie', 'Washington', 'book']
>>> sorted(words, key=len)
['pie', 'book', 'banana', 'Washington']

The resulting order is a list with a string order of shortest to longest. The length of each element in the list is determined by len() and then returned in ascending order.

Let’s return to the earlier example of sorting by first letter when the case is different. key can be used to solve that problem by converting the entire string to lowercase:

>>>

>>> names_with_case = ['harry', 'Suzy', 'al', 'Mark']
>>> sorted(names_with_case)
['Mark', 'Suzy', 'al', 'harry']
>>> sorted(names_with_case, key=str.lower)
['al', 'harry', 'Mark', 'Suzy']

The output values have not been converted to lowercase because key does not manipulate the data in the original list. During sorting, the function passed to key is being called on each element to determine sort order, but the original values will be in the output.

There are two main limitations when you’re using functions with the key argument.

First, the number of required arguments in the function passed to key must be one.

The example below shows the definition of an addition function that takes two arguments. When that function is used in key on a list of numbers, it fails because it is missing a second argument. Each time add() is called during the sort, it is only receiving one element from the list at a time:

>>>

>>> def add(x, y):
...     return x + y
... 
>>> values_to_add = [1, 2, 3]
>>> sorted(values_to_add, key=add)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: add() missing 1 required positional argument: 'y'

The second limitation is that the function used with key must be able to handle all the values in the iterable. For example, you have a list of numbers represented as strings to be used in sorted(), and key is going to attempt to convert them to numbers using int. If a value in the iterable can’t be cast to an integer, then the function will fail:

>>>

>>> values_to_cast = ['1', '2', '3', 'four']
>>> sorted(values_to_cast, key=int)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: 'four'

Each numeric value as a str can be converted to int, but four can’t. This causes a ValueError to be raised and explain that four can’t be converted to int because it is invalid.

The key functionality is extremely powerful because almost any function, built-in or user-defined, can be used to manipulate the output order.

If the ordering requirement is to order an iterable by the last letter in each string (and if the letter is the same, then to use the next letter), then a function can be defined and then used in the sorting. The example below defines a function that reverses the string passed to it, and then that function is used as the argument for key:

>>>

>>> def reverse_word(word):
...     return word[::-1]
...
>>> words = ['banana', 'pie', 'Washington', 'book']
>>> sorted(words, key=reverse_word)
['banana', 'pie', 'book', 'Washington']

The word[::-1] slice syntax is used to reverse a string. Each element will have reverse_word() applied to it, and the sorting order will be based on the characters in the backwards word.

Instead of writing a standalone function, you can use a lambda function defined in the key argument.

A lambda is an anonymous function that:

Must be defined inline
Doesn’t have a name
Can’t contain statements
Will execute just like a function

In the example below, the key is defined as a lambda with no name, the argument taken by the lambda is x, and x[::-1] is the operation that will be performed on the argument:

>>>

>>> words = ['banana', 'pie', 'Washington', 'book']
>>> sorted(words, key=lambda x: x[::-1])
['banana', 'pie', 'book', 'Washington']

x[::-1] is called on each element and reverses the word. That reversed output is then used for sorting, but the original words are still returned.

If the requirement changes, and the order should be reversed as well, then the reverse keyword can be used alongside the key argument:

>>>

>>> words = ['banana', 'pie', 'Washington', 'book']
>>> sorted(words, key=lambda x: x[::-1], reverse=True)
['Washington', 'book', 'pie', 'banana']

lambda functions are also useful when you need to sort class objects based on a property. If you have a group of students and need to sort them by their final grade, highest to lowest, then a lambda can be used to get the grade property from the class:

>>>

>>> from collections import namedtuple

>>> StudentFinal = namedtuple('StudentFinal', 'name grade')
>>> bill = StudentFinal('Bill', 90)
>>> patty = StudentFinal('Patty', 94)
>>> bart = StudentFinal('Bart', 89)
>>> students = [bill, patty, bart]
>>> sorted(students, key=lambda x: getattr(x, 'grade'), reverse=True)
[StudentFinal(name='Patty', grade=94), StudentFinal(name='Bill', grade=90), StudentFinal(name='Bart', grade=89)]

This example uses namedtuple to produce classes with name and grade attributes. The lambda calls getattr() on each element and returns the value for grade.

reverse is set to True to make the ascending output flipped to be descending so that the highest grades are ordered first.

The possibilities are endless for how ordering can be done when you leverage both the key and reverse keyword arguments on sorted(). Code can be kept clean and short when you use a basic lambda for a small function, or you can write a whole new function, import it, and use it in the key argument.

Ordering Values With `.sort()`

The very similarly named .sort() differs quite a bit from the sorted() built-in. They accomplish more or less the same thing, but the help() documentation for list.sort() highlights two of the most critical differences between .sort() and sorted():

>>>

>>> # Python2
Help on method_descriptor:

sort(...)
    L.sort(cmp=None, key=None, reverse=False) -- stable sort *IN PLACE*;
    cmp(x, y) -> -1, 0, 1

>>> # Python3
>>> help(list.sort)
Help on method_descriptor:

sort(...)
    L.sort(key=None, reverse=False) -> None -- stable sort *IN PLACE*

First, sort is a method of the list class and can only be used with lists. It is not a built-in with an iterable passed to it.

Second, .sort() returns None and modifies the values in place. Let’s take a look at the impacts of both of these differences in code:

>>>

>>> values_to_sort = [5, 2, 6, 1]
>>> # Try to call .sort() like sorted()
>>> sort(values_to_sort)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'sort' is not defined

>>> # Try to use .sort() on a tuple
>>> tuple_val = (5, 1, 3, 5)
>>> tuple_val.sort()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'tuple' object has no attribute 'sort'

>>> # Sort the list and assign to new variable
>>> sorted_values = values_to_sort.sort()
>>> print(sorted_values)
None

>>> # Print original variable
>>> print(values_to_sort)
[1, 2, 5, 6]

There are some pretty dramatic differences in how .sort() operates compared to sorted() in this code example:

There is no ordered output of .sort(), so the assignment to a new variable only passes a None type.
The values_to_sort list has been changed in place, and the original order is not maintained in any way.

These differences in behavior make .sort() and sorted() absolutely not interchangeable in code, and they can produce wildly unexpected outcomes if one is used in the wrong way.

.sort() has the same key and reverse optional keyword arguments that produce the same robust functionality as sorted(). Here, you can sort a list of phrases by the second letter of the third word and return the list in reverse:

>>>

>>> phrases = ['when in rome', 
...     'what goes around comes around', 
...     'all is fair in love and war'
...     ]
>>> phrases.sort(key=lambda x: x.split()[2][1], reverse=True)
>>> phrases
['what goes around comes around', 'when in rome', 'all is fair in love and war']

In this sample, a lambda is used to do the following:

Split each phrase into a list of words
Find the third element, or word in this case
Find the second letter in that word

When to Use `sorted()` and When to Use `.sort()`

You’ve seen the differences between sorted() and .sort(), but when do you use which?

Let’s say there is a 5k race coming up: The First Annual Python 5k. The data from the race needs to be captured and sorted. The data that needs to be captured is the runner’s bib number and the number of seconds it took to finish the race:

>>>

>>> from collections import namedtuple

>>> Runner = namedtuple('Runner', 'bibnumber duration')

As the runners cross the finish line, each Runner will be added to a list called runners. In 5k races, not all runners cross the starting line at the same time, so the first person to cross the finish line might not actually be the fastest person:

>>>

>>> runners = []
>>> runners.append(Runner('2528567', 1500))
>>> runners.append(Runner('7575234', 1420))
>>> runners.append(Runner('2666234', 1600))
>>> runners.append(Runner('2425234', 1490))
>>> runners.append(Runner('1235234', 1620))
>>> # Thousands and Thousands of entries later...
>>> runners.append(Runner('2526674', 1906))

Each time a runner crosses the finish line, their bib number and their total duration in seconds is added to runners.

Now, the dutiful programmer in charge of handling the outcome data sees this list, knows that the top 5 fastest participants are the winners that get prizes, and the remaining runners are going to be sorted by fastest time.

There are no requirements for multiple types of sorting by various attributes. The list is a reasonable size. There is no mention of storing the list somewhere. Just sort by duration and grab the five participants with the lowest duration:

>>>

>>> runners.sort(key=lambda x: getattr(x, 'duration'))
>>> top_five_runners = runners[:5]

The programmer chooses to use a lambda in the key argument to get the duration attribute from each runner and sort runners in place using .sort(). After runners is sorted, the first 5 elements are stored in top_five_runners.

Mission accomplished! The race director comes over and informs the programmer that since the current release of Python is 3.7, they have decided that every thirty-seventh person that crossed the finish line is going to get a free gym bag.

At this point, the programmer starts to sweat because the list of runners has been irreversibly changed. There is no way to recover the original list of runners in the order they finished and find every thirty-seventh person.

If you’re working with important data, and there is even a remote possibility that the original data will need to be recovered, then .sort() is not the best option. If the data is a copy, of if it is unimportant working data, of if no one will mind losing it because it can be retrieved, then .sort() can be a fine option.

Alternatively, the runners could have been sorted using sorted() and using the same lambda:

>>>

>>> runners_by_duration = sorted(runners, key=lambda x: getattr(x, 'duration'))
>>> top_five_runners = runners_by_duration[:5]

In this scenario with sorted(), the original list of runners is still intact and has not been overwritten. The impromptu requirement of finding every thirty-seventh person to cross the finish line can be accomplished by interacting with the original values:

>>>

>>> every_thirtyseventh_runners = runners[::37]

every_thirtyseventh_runners is created by using a stride in the list slice syntax on runners, which still contains the original order in which the runners crossed the finish line.

How to Sort in Python: Conclusion

.sort() and sorted() can provide exactly the sort order you need if you use them properly with both the reverse and key optional keyword arguments.

Both have very different characteristics when it comes to output and in-place modifications, so make sure you think through any application functionality or program that will be using .sort() as it can irrevocably overwrite data.

For the avid Pythonistas looking for a challenge with sorting, try using more complex data types in sorting: nested iterables. Also, feel free to dive into the open source Python code implementations for the built-ins and read about the sort algorithm used in Python called Timsort.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Defining Main Functions in Python

2019-05-01T14:00:00+00:00

Many programming languages have a special function that is automatically executed when an operating system starts to run a program. This function is usually called main() and must have a specific return type and arguments according to the language standard. On the other hand, the Python interpreter executes scripts starting at the top of the file, and there is no specific function that Python automatically executes.

Nevertheless, having a defined starting point for the execution of a program is useful for understanding how a program works. Python programmers have come up with several conventions to define this starting point.

By the end of this article, you’ll understand:

What the special __name__ variable is and how Python defines it
Why you would want to use a main() in Python
What conventions there are for defining main() in Python
What the best-practices are for what code to put into your main()

A Basic Python main()

In some Python scripts, you may see a function definition and a conditional statement that looks like the example below:

def main():
    print("Hello World!")

if __name__ == "__main__":
    main()

In this code, there is a function called main() that prints the phrase Hello World! when the Python interpreter executes it. There is also a conditional (or if) statement that checks the value of __name__ and compares it to the string "__main__". When the if statement evaluates to True, the Python interpreter executes main(). You can read more about conditional statements in Conditional Statements in Python.

This code pattern is quite common in Python files that you want to be executed as a script and imported in another module. To help understand how this code will execute, you should first understand how the Python interpreter sets __name__ depending on how the code is being executed.

Execution Modes in Python

There are two primary ways that you can instruct the Python interpreter to execute or use code:

You can execute the Python file as a script using the command line.
You can import the code from one Python file into another file or into the interactive interpreter.

You can read a lot more about these approaches in How to Run Your Python Scripts. No matter which way of running your code you’re using, Python defines a special variable called __name__ that contains a string whose value depends on how the code is being used.

We’ll use this example file, saved as execution_methods.py, to explore how the behavior of the code changes depending on the context:

print("This is my file to test Python's execution methods.")
print("The variable __name__ tells me which context this file is running in.")
print("The value of __name__ is:", repr(__name__))

In this file, there are three calls to print() defined. The first two print some introductory phrases. The third print() will first print the phrase The value of __name__ is, and then it will print the representation of the __name__ variable using Python’s built-in repr().

In Python, repr() displays the printable representation of an object. This example uses repr() to emphasize that the value of __name__ is a string. You can read more about repr() in the Python documentation.

You’ll see the words file, module, and script used throughout this article. Practically, there isn’t much difference between them. However, there are slight differences in meaning that emphasize the purpose of a piece of code:

File: Typically, a Python file is any file that contains code. Most Python files have the extension .py.
Script: A Python script is a file that you intend to execute from the command line to accomplish a task.
Module: A Python module is a file that you intend to import from within another module or a script, or from the interactive interpreter. You can read more about modules in the Python documentation.

This distinction is also discussed in How to Run Your Python Scripts.

Executing From the Command Line

In this approach, you want to execute your Python script from the command line.

When you execute a script, you will not be able to interactively define the code that the Python interpreter is executing. The details of how you can execute Python from your command line are not that important for the purpose of this article, but you can expand the box below to read more about the differences between the command line on Windows, Linux, and macOS.

The way that you tell the computer to execute code from the command line is slightly different depending on your operating system.

On Linux and macOS, the command line typically looks like the example below:

eleanor@realpython:~/Documents$

The part before the dollar sign ($) may look different, depending on your username and your computer’s name. The commands that you type will go after the $. On Linux or macOS, the name of the Python 3 executable is python3, so you should run Python scripts by typing python3 script_name.py after the $.

On Windows, the command prompt typically looks like the example below:

C:\Users\Eleanor\Documents>

The part before the > may look different, depending on your username. The commands that you type will go after the >. On Windows, the name of the Python 3 executable is typically python, so you should run Python scripts by typing python script_name.py after the >.

Regardless of your operating system, the output from the Python scripts that you use in this article will be the same, so only the Linux and macOS style of input is shown in this article, and the input line will start at the $.

Now you should execute the execution_methods.py script from the command line, as shown below:

$ python3 execution_methods.py
This is my file to test Python's execution methods.
The variable __name__ tells me which context this file is running in.
The value of __name__ is: '__main__'

In this example, you can see that __name__ has the value '__main__', where the quote symbols (') tell you that the value has the string type.

Remember that, in Python, there is no difference between strings defined with single quotes (') and double quotes ("). You can read more about defining strings in Basic Data Types in Python.

You will find identical output if you include a shebang line in your script and execute it directly (./execution_methods.py), or use the %run magic in IPython or Jupyter Notebooks.

You may also see Python scripts executed from within packages by adding the -m argument to the command. Most often, you will see this recommended when you’re using pip: python3 -m pip install package_name.

Adding the -m argument runs the code in the __main__.py module of a package. You can find more information about the __main__.py file in How to Publish an Open-Source Python Package to PyPI.

In all three of these cases, __name__ has the same value: the string '__main__'.

Technical detail: The Python documentation defines specifically when __name__ will have the value '__main__':

A module’s __name__ is set equal to '__main__' when read from standard input, a script, or from an interactive prompt. (Source)

__name__ is stored in the global namespace of the module along with the __doc__, __package__, and other attributes. You can read more about these attributes in the Python Data Model documentation and, specifically for modules and packages, in the Python Import documentation.

Importing Into a Module or the Interactive Interpreter

Now let’s take a look at the second way that the Python interpreter will execute your code: imports. When you are developing a module or script, you will most likely want to take advantage of modules that someone else has already built, which you can do with the import keyword.

During the import process, Python executes the statements defined in the specified module (but only the first time you import a module). To demonstrate the results of importing your execution_methods.py file, start the interactive Python interpreter and then import your execution_methods.py file:

>>>

>>> import execution_methods
This is my file to test Python's execution methods.
The variable __name__ tells me which context this file is running in.
The value of __name__ is: 'execution_methods'

In this code output, you can see that the Python interpreter executes the three calls to print(). The first two lines of output are exactly the same as when you executed the file as a script on the command line because there are no variables in either of the first two lines. However, there is a difference in the output from the third print().

When the Python interpreter imports code, the value of __name__ is set to be the same as the name of the module that is being imported. You can see this in the third line of output above. __name__ has the value 'execution_methods', which is the name of the .py file that Python is importing from.

Note that if you import the module again without quitting Python, there will be no output.

Note: For more information on how importing works in Python, check out the official documentation as well as Absolute vs Relative Imports in Python.

Best Practices for Python Main Functions

Now that you can see the differences in how Python handles its different execution modes, it’s useful for you to know some best practices to use. These will apply whenever you want to write code that you can run as a script and import in another module or an interactive session.

You will learn about four best practices to make sure that your code can serve a dual purpose:

Put most code into a function or class.
Use __name__ to control execution of your code.
Create a function called main() to contain the code you want to run.
Call other functions from main().

Put Most Code Into a Function or Class

Remember that the Python interpreter executes all the code in a module when it imports the module. Sometimes the code you write will have side effects that you want the user to control, such as:

Running a computation that takes a long time
Writing to a file on the disk
Printing information that would clutter the user’s terminal

In these cases, you want the user to control triggering the execution of this code, rather than letting the Python interpreter execute the code when it imports your module.

Therefore, the best practice is to include most code inside a function or a class. This is because when the Python interpreter encounters the def or class keywords, it only stores those definitions for later use and doesn’t actually execute them until you tell it to.

Save the code below to a file called best_practices.py to demonstrate this idea:

 1 from time import sleep
 2 
 3 print("This is my file to demonstrate best practices.")
 4 
 5 def process_data(data):
 6     print("Beginning data processing...")
 7     modified_data = data + " that has been modified"
 8     sleep(3)
 9     print("Data processing finished.")
10     return modified_data

In this code, you first import sleep() from the time module.

sleep() pauses the interpreter for however many seconds you give as an argument and will produce a function that takes a long time to run for this example. Next, you use print() to print a sentence describing the purpose of this code.

Then, you define a function called process_data() that does five things:

Prints some output to tell the user that the data processing is starting
Modifies the input data
Pauses the execution for three seconds using sleep()
Prints some output to tell the user that the processing is finished
Returns the modified data

Execute the Best Practices File on the Command Line

Now, what will happen when you execute this file as a script on the command line?

The Python interpreter will execute the from time import sleep and print() lines that are outside the function definition, then it will create the definition of the function called process_data(). Then, the script will exit without doing anything further, because the script does not have any code that executes process_data().

The code block below shows the result of running this file as a script:

$ python3 best_practices.py
This is my file to demonstrate best practices.

The output that we can see here is the result of the first print(). Notice that importing from time and defining process_data() produce no output. Specifically, the outputs of the calls to print() that are inside the definition of process_data() are not printed!

Import the Best Practices File in Another Module or the Interactive Interpreter

When you import this file in an interactive session (or another module), the Python interpreter will perform exactly the same steps as when it executes file as a script.

Once the Python interpreter imports the file, you can use any variables, classes, or functions defined in the module you’ve imported. To demonstrate this, we will use the interactive Python interpreter. Start the interactive interpreter and then type import best_practices:

>>>

>>> import best_practices
This is my file to demonstrate best practices.

The only output from importing the best_practices.py file is from the first print() call defined outside process_data(). Importing from time and defining process_data() produce no output, just like when you executed the code from the command line.

Use `name` to Control the Execution of Your Code

What if you want process_data() to execute when you run the script from the command line but not when the Python interpreter imports the file?

You can use __name__ to determine the execution context and conditionally run process_data() only when __name__ is equal to "__main__". Add the code below to the bottom of your best_practices.py file:

11 if __name__ == "__main__":
12     data = "My data read from the Web"
13     print(data)
14     modified_data = process_data(data)
15     print(modified_data)

In this code, you’ve added a conditional statement that checks the value of __name__. This conditional will evaluate to True when __name__ is equal to the string "__main__". Remember that the special value of "__main__" for the __name__ variable means the Python interpreter is executing your script and not importing it.

Inside the conditional block, you have added four lines of code (lines 12, 13, 14, and 15):

Lines 12 and 13: You are creating a variable data that stores the data you’ve acquired from the Web and printing it.
Line 14: You are processing the data.
Line 15: You are printing the modified data.

Now, run your best_practices.py script from the command line to see how the output will change:

$ python3 best_practices.py
This is my file to demonstrate best practices.
My data read from the Web
Beginning data processing...
Data processing finished.
My data read from the Web that has been modified

First, the output shows the result of the print() call outside of process_data().

After that, the value of data is printed. This happened because the variable __name__ has the value "__main__" when the Python interpreter executes the file as a script, so the conditional statement evaluated to True.

Next, your script called process_data() and passed data in for modification. When process_data() executes, it prints some status messages to the output. Finally, the value of modified_data is printed.

Now you should check what happens when you import the best_practices.py file from the interactive interpreter (or another module). The example below demonstrates this situation:

>>>

>>> import best_practices
This is my file to demonstrate best practices.

Notice that you get the same behavior as before you added the conditional statement to the end of the file! This is because the __name__ variable had the value "best_practices", so Python did not execute the code inside the block, including process_data(), because the conditional statement evaluated to False.

Create a Function Called main() to Contain the Code You Want to Run

Now you are able to write Python code that can be run from the command line as a script and imported without unwanted side effects. Next, you are going to learn about how to write your code to make it easy for other Python programmers to follow what you mean.

Many languages, such as C, C++, Java, and several others, define a special function that must be called main() that the operating system automatically calls when it executes the compiled program. This function is often called the entry point because it is where execution enters the program.

By contrast, Python does not have a special function that serves as the entry point to a script. You can actually give the entry point function in a Python script any name you want!

Although Python does not assign any significance to a function named main(), the best practice is to name the entry point function main() anyways. That way, any other programmers who read your script immediately know that this function is the starting point of the code that accomplishes the primary task of the script.

In addition, main() should contain any code that you want to run when the Python interpreter executes the file. This is better than putting the code directly into the conditional block because a user can reuse main()if they import your module.

Change the best_practices.py file so that it looks like the code below:

 1 from time import sleep
 2 
 3 print("This is my file to demonstrate best practices.")
 4 
 5 def process_data(data):
 6     print("Beginning data processing...")
 7     modified_data = data + " that has been modified"
 8     sleep(3)
 9     print("Data processing finished.")
10     return modified_data
11 
12 def main():
13     data = "My data read from the Web"
14     print(data)
15     modified_data = process_data(data)
16     print(modified_data)
17 
18 if __name__ == "__main__":
19     main()

In this example, you added the definition of main() that includes the code that was previously inside the conditional block. Then, you changed the conditional block so that it executes main(). If you run this code as a script or import it, you will get the same output as in the previous section.

Call Other Functions From main()

Another common practice in Python is to have main() execute other functions, rather than including the task-accomplishing code in main(). This is especially useful when you can compose your overall task from several smaller sub-tasks that can execute independently.

For example, you may have a script that does the following:

Reads a data file from a source that could be a database, a file on the disk, or a web API
Processes the data
Writes the processed data to another location

If you implement each of these sub-tasks in separate functions, then it is easy for a you (or another user) to re-use a few of the steps and ignore the ones you don’t want. Then you can create a default workflow in main(), and you can have the best of both worlds.

Whether to apply this practice to your code is a judgment call on your part. Splitting the work into several functions makes reuse easier but increases the difficulty for someone else trying to interpret your code because they have to follow several jumps in the flow of the program.

Modify your best_practices.py file so that it looks like the code below:

 1 from time import sleep
 2 
 3 print("This is my file to demonstrate best practices.")
 4 
 5 def process_data(data):
 6     print("Beginning data processing...")
 7     modified_data = data + " that has been modified"
 8     sleep(3)
 9     print("Data processing finished.")
10     return modified_data
11 
12 def read_data_from_web():
13     print("Reading data from the Web")
14     data = "Data from the web"
15     return data
16 
17 def write_data_to_database(data):
18     print("Writing data to a database")
19     print(data)
20 
21 def main():
22     data = read_data_from_web()
23     modified_data = process_data(data)
24     write_data_to_database(modified_data)
25 
26 if __name__ == "__main__":
27     main()

In this example code, the first 10 lines of the file have the same content that they had before. The second function definition on line 12 creates and returns some sample data, and the third function definition on line 17 simulates writing the modified data to a database.

On line 21, main() is defined. In this example, you have modified main() so that it calls the data reading, data processing, and data writing functions in turn.

First, the data is created from read_data_from_web(). This data is passed to process_data(), which returns the modified_data. Finally, modified_data is passed into write_data_to_database().

The last two lines of the script are the conditional block that checks __name__ and runs main() if the if statement is True.

Now, you can run the whole processing pipeline from the command line, as shown below:

$ python3 best_practices.py
This is my file to demonstrate best practices.
Reading data from the Web
Beginning data processing...
Data processing finished.
Writing processed data to a database
Data from the web that has been modified

In the output from this execution, you can see that the Python interpreter executed main(), which executed read_data_from_web(), process_data(), and write_data_to_database(). However, you can also import the best_practices.py file and re-use process_data() for a different input data source, as shown below:

>>>

>>> import best_practices as bp
This is my file to demonstrate best practices.
>>> data = "Data from a file"
>>> modified_data = bp.process_data(data)
Beginning data processing...
Data processing finished.
>>> bp.write_data_to_database(modified_data)
Writing processed data to a database
Data from a file that has been modified

In this example, you imported best_practices and shortened the name to bp for this code.

The import process caused the Python interpreter to execute all of the lines of code in the best_practices.py file, so the output shows the line explaining the purpose of the file.

Then, you stored data from a file in data instead of reading the data from the Web. Then, you reused process_data()and write_data_to_database() from the best_practices.py file. In this case, you took advantage of reusing your code instead of defining all of the logic in main().

Summary of Python Main Function Best Practices

Here are four key best practices about main() in Python that you just saw:

Put code that takes a long time to run or has other effects on the computer in a function or class, so you can control exactly when that code is executed.
Use the different values of __name__ to determine the context and change the behavior of your code with a conditional statement.
You should name your entry point function main() in order to communicate the intention of the function, even though Python does not assign any special significance to a function named main().
If you want to reuse functionality from your code, define the logic in functions outside main() and call those functions within main().

Conclusion

Congratulations! You now know how to create Python main() functions.

You learned the following:

Knowing the value of the __name__ variable is important to write code that serves the dual purpose of executable script and importable module.
__name__ takes on different values depending on how you executed your Python file. __name__ will be equal to:
- "__main__" when the file is executed from the command line or with python -m (to execute a package’s __main__.py file)
- The name of the module, if the module is being imported
Python programmers have developed a set of good practices to use when you want to develop reusable code.

Now you’re ready to go write some awesome Python main() function code!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Make a Location-Based Web App With Django and GeoDjango

2019-04-30T14:00:00+00:00

Throughout this course, you’ll learn how to use Django and GeoDjango to build a location-based web application from scratch. You’ll be building a simple nearby shops application that lists the shops closest to a user’s location.

By the end of this course, you’ll be able to:

Use Django to build a simple web application from scratch
Use the GeoDjango sub-framework to implement geolocation features in your Django application
Use a spatial database (PostgreSQL and PostGIS) to get benefits from the spatial features and easily implement location-aware web apps

Downloadable Sample Project: This course includes a downloadable sample project that demonstrates the final state of the Django app you’ll build throughout the course. You can get the source code for it here on the materials repo.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Get the Most Out of PyCon

2019-04-29T14:00:00+00:00

Congratulations! You’re going to PyCon!

Whether this is your first time or not, going to a conference full of people who love the same thing as you is always a fun experience. There’s so much more to PyCon than just a bunch of people talking about the Python language, and that can be intimidating for first-time attendees. This guide will help you navigate all there is to see and do at PyCon.

PyCon (US/North America) is the biggest conference centered around the Python language. Originally formed in 2003, this conference has grown exponentially and has even spawned several other PyCons and workshops around the world.

This year will mark my fourth year in a row attending PyCon, so I thought that I would share some of the notes that I’ve taken over the years to help you get the most out of your PyCon experience.

Everyone who attends PyCon will have a different experience, and that’s what makes the conference truly unique. This guide is meant to help you, but you don’t need to follow it strictly.

This guide will have links that are specific to PyCon 2019, but it should be useful for future PyCons as well.

By the end of this article, you’ll know:

What PyCon involves
What to do before you go
What to do while you’re at PyCon
What to do after PyCon
How to have a great PyCon

What PyCon Involves

Before we go into how to get the most out of PyCon, let’s first establish what PyCon involves.

PyCon is broken up into three stages, as described by the official PyCon site:

Tutorials: We start with two days of 3-hour classes during which you get to learn in depth with instructors. These are great to go to since the class sizes are small, and you can ask questions of the instructors. I highly recommend going to at least one of these if you can, but they do have an additional cost of $150 per class.
Conference: Next, we have three days of talks. Each presentation lasts for 30 to 45 minutes, and there are about 5 talks going on at a time. But that’s not all: there are also open spaces, sponsors, lightning talks, dinners, and so much more.
Sprints: During this stage, you can take what you’ve learned and apply it! This is a 3-day exercise in which people group up to work on various open source projects all related to Python. If you’ve got the time, I highly suggest that you go to this as it’s a great way to practice what you’ve learned, become associated with an open source project, and network with very smart and talented people.

Since most PyCon attendees go to the Conference part, that will be the focus of this article. However, don’t let that deter you from attending the Tutorials or Sprints if you can!

I’ve personally found that I’ve learned more technical skills by attending the Tutorials rather than listening to the talks. The Sprints are great for networking and applying the skills that you’ve already got as well as learning new ones from the people you’ll be working with.

What to Do Before You Go

In general, the more prepared you are for something, the better your experience will be. The same applies for PyCon.

It’s really helpful to plan and prepare ahead of time, which you’re already doing just by reading this article!

Look through the talk schedule and see which talks sound most interesting to you. This doesn’t mean you need to plan out all of the talks you are going to see for every slot possible, but it helps to get an idea of which topics are going to be presented so you can decide what you’re most interested in.

Getting the Guidebook app will help you plan you schedule. This app lets you view the schedule for the talks and add reminders for the ones you want to attend. If you’re having a hard time picking which talks to go to, you can come prepared with a question or problem you need solved. Doing this can help you focus on the topics that are important to you.

If you can, come the day before to check in and attend the opening reception. The line up to check in on the first day is always long, so you’ll save time if you check in the day before. There’s also usually an opening reception that evening, so you can meet other attendees and speakers, as well as get a chance to check out the various sponsors and their booths.

If you’re brand new to PyCon, there’s also a Newcomer Orientation that can help you get caught up on what the conference involves and how you can participate.

Recap:

Here’s what to do before you go:

Look at the talk schedule.
Get the Guidebook App.
Come up with a specific question you want answered.
Check in the day before the conference.

What to Do at PyCon

It’s okay if you’re nervous or excited on your first day at the conference.

There will be a lot of people from all different walks of life, and that’s what makes it so great. You may see some of your Python heroes, such as Guido van Rossum and and have a chance to go up to them and say hello.

Guido van Rossum With the Python Staff of Enlightenment

The Python community is very welcoming! But there are also some designated quiet rooms, where speakers and others will go to work in peace. You should refrain from talking to anyone in those rooms to allow them that safe space.

Let’s break down the conference into some key elements and see how you can get the most out of them:

Talks
Open spaces
Sponsors
Volunteering opportunities
Lightning talks
After hours activities
Time for yourself

Talks

Go to as many of the talks as you want, but you don’t need to go to all of them. You’ll only find yourself getting stressed as you run from room to room. Instead, make sure that you hit up the talks that you selected before coming. Although it’s pretty rare, the schedule can change, so check the schedule board daily to catch any changes.

'Data Visualization in Mixed Reality with Python' Talk Given by Anna Nicanorova at PyCon 2018

If there’s a conflict between two talks that you to really want to go to, remember that each talk is recorded and uploaded onto YouTube. Sometimes, they’re even available that same day! Pick the talk that is going to be the most relevant to your situation and seems the most interesting to you. Then take note of the talk that you missed to watch it later that evening or the next day when you have some free time.

Even though all talks are available online, actually going to the talks is still valuable. You’ll retain the information better if you attend, and you’ll have the opportunity to ask questions directly of the presenter.

Keep in mind that you don’t need to see all of the “celebrity” speakers. These speakers get a lot of attention in the Python community and may seem like more worthwhile speakers because of that. But the process of getting a talk approved at PyCon is rigorous and ensures that each speaker and topic are worth listening to.

In fact, sometimes it’s better to go see some of the less famous speakers because you can get better seats and have a greater chance to ask questions.

When you go to a talk, remember to silence your phone and computer. Noise can be very distracting to the audience and the speaker. It might be helpful to put away all your devices and simply listen or take notes using a pad and paper.

Try to think of questions that you may have about what’s being discussed. Typically, there’s time allotted at the end for questions from the audience. If that doesn’t happen, the presenters are usually very accommodating about answering questions in the hall afterwards.

Recap:

Here’s what you need to know about talks at PyCon:

You don’t need to go to all of the talks and will be able to watch them on YouTube.
All talks and speakers are amazing.
Keep the noise to a minimum during the talks.
Think of questions to ask.
Talk to the speakers afterwards.

Open Spaces

Open Spaces are rooms that can be booked by conference attendees. There are 1-hour slots throughout the day available to anyone that wants to use them. These rooms have been used as places to teach people, hold meetups, and even do yoga classes. They’re open for whatever activities you need them for, as long as you follow the Code of Conduct of course.

Open Spaces Board at PyCon 2018

You might want to go to these open spaces instead of a talk or a sponsor booth. Be sure to check out the open space board each day since it constantly changes. It might be helpful to take a picture of the board to reference later.

Feel free to create your own open space. Remember that problem or specific question that you were looking for help with? Sign up for a room and ask for advice on that topic! You never know who might show up to help out.

I did that one year. While I was working on one of my open source projects, I ran into a testing problem that I just couldn’t solve. I grabbed an open space for an hour and asked for help. I learned all about the mock module for testing. That saved me literally hours of work!

If you’re an expert or even have some limited knowledge about a topic that you want to share, feel free to grab an open space for that as well! These spaces are intended to be whatever you want and need them to be, so don’t be shy about using them. In fact, be on the lookout for a Real Python open space and feel free to join us as we talk about what we envision for the future.

Recap:

Here’s what you need to know about open spaces at PyCon:

Open spaces can be whatever you need them to be.
Use an open space slot to ask for help.
Use an open space slot to teach others.

Volunteering Opportunities

Have you ever wished you could contribute or give back to the Python Community? Well, you can at PyCon.

So much work goes into making a conference happen, and none of it would be possible without volunteers. The 2019 PyCon conference is looking for over 300-man hours of volunteering alone!

That sounds like a lot, but you can still make a difference. An hour or two of your time can be a big help but still not take away from your learning time. You also never know who you might rub shoulders with as a volunteer. You can learn more about helping out in the Call for On-Site Volunteers. There’s a little something for everyone.

Recap:

Here’s what you need to know about volunteering at PyCon:

Just Do It.
Really, Just Do It!

After Hours Activities

Even though the conference ends in the early evening, there’s still more to do after the conference wraps up for the day.

The first thing you should check out, even if only for a little while, is the lightning talks. These are 5 minute “soap boxes” for anyone to share on a topic. These talks cover a wide range of topics: anything from a new open source project to social commentary to philanthropic topics. Did you know that Docker was first publicly announced during a PyCon lightning talk? The lightning talks have become a popular staple of regular attendees.

Also be on the lookout for sponsors doing sponsored dinners with members of their company. This is a great way to build up your network and get a free dinner as well. PyCon does host dinner parties in the evenings, but these cost money and typically sell out really fast.

Even if you don’t get in on the official PyCon dinners or the sponsored ones, there’s always plenty to check out near the conference. Each conference location is selected in part because of the interesting things nearby. When you’re getting ready for the conference, look for fun places to go eat.

Recap:

Here’s what you need to know about after-hours events at PyCon:

Check out the lightning talks.
If there are spots available, sign up for one of the official PyCon dinners.
Check early and often for any sponsored dinners.
Have fun exploring the local cuisine and culture.

New Friends

One of the greatest pieces of advice that I received when I first started going to PyCon was to make a new friend each day.

Dan Bader and Anthony Shaw

Some of the best times to get to know someone at PyCon are the lunch and snack breaks. Instead of trying to pick an empty table to sit at, find one that already has a person or two and ask if you can join them. Strike up a conversation about what their favorite talk is thus far, or how they use Python in their day to day activities. Pretty soon you’ll make a new friend. You can take a few notes, whether mentally or literally, about your conversation so you can remember that person later on.

You might want to make some business cards with the contact info you want to share with people you meet in order to stay in contact with them. PyCon will actually give you a few to give out, but you can go through them really quickly! Be sure to update any profiles that you share with people such as LinkedIn or GitHub.

Recap:

Here’s what you need to know about making new friends at PyCon:

Take the challenge to meet at least one new person each day.
Make a note of who that person is so that you don’t forget.
Bring business cards to share with people you meet.

What to Do After PyCon

Once the conference is over, there’s still a lot you can do. First off, if you have the time, there are the Sprints, which are a great opportunity for you to hone your skills and even gain new ones as a Python developer. They last for four days after the Conference, but you don’t need to stay for the entire time. Stay for as long as you like, whether that’s for a few hours or a few days.

After you get home, make sure to check out the YouTube videos of talks you missed or made a note to watch again. There’s also all of the tutorials, keynote speaker talks, and even the lightning talks to check out. There’s plenty there for you to get your PyCon fix throughout the year.

The greatest thing about PyCon is the feeling of belonging to a community. That’s only made possible by the great people who give back to the Python community, and you can become one of them!

There are lots of ways that you can give back to this great community:

Contribute to an open source project that uses Python.
Join a local Python meetup group. Don’t have one? Create one!
Share with others what you’ve learned.
Submit a talk or poster for next year’s PyCon.

Finally, you can start preparing for the next PyCon. When you purchase tickets early, you get a discount on the pricing, but those tickets go pretty fast too. You can also start taking note of any problems or questions that you can’t find the answer to in preparation for selecting talks to check out at the next PyCon.

Recap:

Here’s what you need to know about what to do after the conference:

If you have the time, stay for the Sprints.
Check out the YouTube videos of the talks you missed or loved.
Look for a way to contribute back to the Python community with what you learned.
Start looking at next year’s PyCon.

Welcome to the Greatest Community!

Congratulation! You’re about to attend one of the greatest technical conferences there is out there.

In this article, you learned:

What PyCon is all about
What you can do before coming to PyCon
What you can do at PyCon
What you can do after PyCon

With the tips in this article, you’ll be able to have a great PyCon. We look forward to seeing you there!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Conditional Statements in Python (if/elif/else)

2019-04-25T14:00:00+00:00

In this step-by-step course you’ll learn how to work with conditional (“if”) statements in Python. Master if-statements step-by-step and see how to write complex decision making code in your programs.

Take the Quiz: Test your knowledge with our interactive “Python Conditional Statements” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Take the Quiz »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Sending Emails With Python

2019-04-23T14:00:00+00:00

In this course, you’ll learn how to send emails using Python. Find out how to send plain-text and HTML messages, add files as attachments, and send personalized emails to multiple people. Later on you’ll build a CSV-powered email sending script from scratch.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Immutability in Python

2019-04-18T14:00:00+00:00

In Python, immutable vs mutable data types and objects types can cause some confusion—and weird bugs. With this video course you’ll see what the difference between mutable and immutable data types is in Python, and how you can use it to your advantage in your own programs.

You’ll also see how to deal with a language quirk in Python that allows objects referenced by immutable types to me modified. Python’s definition of “immutable” can be a bit misleading.

Basically, the promise of “immutability” on tuples is only partly true. The tuple itself cannot be modified, but objects referenced by the tuple can be. This is sometimes called “non-transitive immutability.”

If the tuple has an immutable field like a string for example, then it cannot be modified. A mutable field like a list however, can be edited, even if it’s embedded in the “immutable” tuple.

When the Python documentation refers to an object as being “immutable” they mean the behavior above observed. Other immutable types in Python behave the same way, e.g. namedtuples or frozensets.

If you’ve ever encountered the following exception and now you want to know why, this is the video course for you:

TypeError: 'tuple' object does not support item assignment

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Hands-on Python 3 Concurrency With the asyncio Module

2019-04-16T14:00:00+00:00

Learn how to speed up your Python 3 programs using concurrency and the new asyncio module in the standard library.

First, you’ll explore the key terms of parallel programming. Next, you’ll see step-by-step how to leverage concurrency and parallelism in your own programs, all the way to building a complete HTTP downloader example app using asyncio and aiohttp.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

Pointers in Python: What's the Point?

Why Doesn’t Python Have Pointers?

Objects in Python

Immutable vs Mutable Objects

Understanding Variables

Variables in C

Names in Python

A Note on Intern Objects in Python

Simulating Pointers in Python

Using Mutable Types as Pointers

Using Python Objects

Real Pointers With ctypes

Conclusion

Continuous Integration With Python

How to Iterate Through a Dictionary in Python

A Few Words on Dictionaries

How to Iterate Through a Dictionary in Python: The Basics

Iterating Through Keys Directly

Iterating Through .items()

Iterating Through .keys()

Iterating Through .values()

Modifying Values and Keys

Real-World Examples

Turning Keys Into Values and Vice Versa

Filtering Items

Doing Some Calculations

Using Comprehensions

Turning Keys Into Values and Vice Versa: Revisited

Filtering Items: Revisited

Doing Some Calculations: Revisited

Removing Specific Items

Sorting a Dictionary

Iterating in Sorted Order

Sorted by Keys

Sorted by Values

Reversed

Iterating Destructively With .popitem()

Using Some of Python’s Built-In Functions

map()

filter()

Using collections.ChainMap

Using itertools

Cyclic Iteration With cycle()

Chained Iteration With chain()

Using the Dictionary Unpacking Operator (**)

Conclusion

Python Logging: A Stroll Through the Source Code

How to Follow Along

Preliminaries

Preliminary #1: A Level Is Just an int!

Preliminary #2: Logging Is Thread-Safe, but Not Process-Safe

Package Architecture: Logging’s MRO

The LogRecord Class

The Logger and Handler Classes

The Filter and Filterer Classes

The Manager Class

The All-Important Root Logger

The Logger Hierarchy

A Multi-Handler Design

The “Why Didn’t My Log Message Go Anywhere?” Dilemma

Taking Advantage of Lazy Formatting

Functions vs Methods

What Does getLogger() Really Do?

Library vs Application Logging: What Is NullHandler?

What Logging Does With Exceptions

Logging Python Tracebacks

Conclusion

Interactive Data Visualization in Python With Bokeh

Unicode & Character Encodings in Python: A Painless Guide

What’s a Character Encoding?

The string Module

A Bit of a Refresher

We Need More Bits!

Covering All the Bases: Other Number Systems

Enter Unicode

Unicode vs UTF-8

Encoding and Decoding in Python 3

Python 3: All-In on Unicode

One Byte, Two Bytes, Three Bytes, Four

Real Pointers With `ctypes`

Iterating Through `.items()`

Iterating Through `.keys()`

Iterating Through `.values()`

Iterating Destructively With `.popitem`()

`map()`

`filter()`

Using `collections.ChainMap`

Using `itertools`

Cyclic Iteration With `cycle()`

Chained Iteration With `chain()`

Using the Dictionary Unpacking Operator (`**`)

Preliminary #1: A Level Is Just an `int`!

The `LogRecord` Class

The `Logger` and `Handler` Classes

The `Filter` and `Filterer` Classes

The `Manager` Class

What Does `getLogger()` Really Do?

Library vs Application Logging: What Is `NullHandler`?

The `string` Module

Odds and Ends: `unicodedata`

`playsound`

`simpleaudio`

`winsound`

`python-sounddevice`

`pydub`

`pyaudio`

`python-sounddevice`

`pyaudio`

`wavio`

`soundfile`

`pydub`

Ordering Values With `sorted()`

Lists With Non-Comparable Data Types Can’t Be `sorted()`

Using `sorted()` With a `reverse` Argument