Real Python

How to Publish Your Own Python Package to PyPI

2019-06-18T14:00:00+00:00

Learn how to create a Python package for your project and how to publish it to PyPI, the Python Package Repository with this step-by-step course. Quickly get up to speed on everything from naming your package to configuring it using setup.py.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Marlene Mhangami

2019-06-17T14:00:00+00:00

We are joined today by Marlene Mhangami. Marlene is a passionate Pythonista who is not only using tech to facilitate social change and empower Zimbabwean women but is also the chair of the very first PyCon Africa. Join me as we talk about her non-traditional start in tech, as well as her passion for using technology to create social change for good.

Ricky: Welcome, Marlene! I’m so glad you could join me for this interview. Let’s start with the usual first question. How’d you get into programming, and when did you start using Python?

Marlene: Well, my background is in molecular biology, so in college, although I had some interaction with code, it wasn’t really something I used regularly. I’ve always been interested in technology and the impact it has on the world, but growing up I didn’t really have anyone to introduce me to the concept of coding. We had one computer that we shared in my house, and my brothers were super into gaming. They were on it all the time.

In high school, I took a computer class, but my teachers, who I’m sure were really nice people, didn’t go much further than the basics of using Microsoft Office and having us memorize names of different parts of hardware. So, in general, I focused more on life and physical sciences, and I’m still really interested in those fields today.

I studied for a bit in the United States for college, and I remember coming home one summer and being really aware of how different Zimbabwe was from the U.S. From really small cultural differences like how people address conflict (which is something I’m still trying to figure out with my US friends) to much more impactful things like access to knowledge and education.

I decided I wanted to start being more involved with my local community, and for some reason, I decided that I wanted to leverage technology to help me do that. I also had a really vivid dream that made me re-evaluate what I was doing with my life.

After googling around and actually organizing a meetup, I got introduced to one of my co-founders, Ronald Maravanykia, who was at the time running a Django girls workshop in Harare. He introduced me to Python as a great educational tool for teaching programming to people who don’t have a computer science background. That was almost three years ago, and I’m super grateful Python ended up being the language I chose to focus on. It’s really great!

Ricky: As a Zimbabwean, you are passionate about empowering Zimbabwean women in tech and helping today’s young girls pursue a career in tech. So much so that you are the programs director at the non-profit ZimboPy?

Marlene: Yes, as I mentioned earlier, using technology to create social change for good is something that’s consistently at the forefront of my mind. My co-founders and I decided to create ZimboPy as a way to give girls in Zim access to knowledge and experiences they wouldn’t necessarily get anywhere else in the country.

I think for me, the end goal is to create a model where we introduce girls to programming, and if they enjoy it, offering them a pathway to get jobs in the field. The tech world is not perfect, but unlike most fields, there are ways that you can by-pass traditional gatekeepers and be successful regardless of whether or not you have the privilege of receiving formal higher education.

We are still a small non-profit, and I’m learning new things every day, but we’ve had some great success stories that give me so much hope! Also, our mentorship weeks, where we introduce girls to women who are doing incredible things in the field and help them build something new, are my favorite thing to do all year! We’ve had incredible women like Ashley McNamara and Emily Freeman speak. It’s always so much fun!

Ricky: You are one of the 13 members of the Python Software Foundation’s (PSF) Board of Directors. How did you come to be involved with the PSF, and what do you think is the biggest challenge as we look towards 2020 and beyond?

Marlene: I’m actually up for re-election this year, so hopefully, I’ll still be on the board for 2020. I really had such a positive experience with it so far!

I was actually introduced to the PSF by Lorena Mesa, who was volunteering to speak at one of our ZimboPy mentorship weeks in early 2017. She was on the board at the time, and we spoke about it around the time when elections were coming up. She really felt that I would be a great addition to the board. At that time, no one from Africa had ever held a seat, so she was super encouraging.

I remember looking through the list of the nominees and thinking that I didn’t belong there. I had done some work with the local Python community in Zimbabwe, but I felt really intimidated by everyone’s profiles. I ended up agreeing to let Lorena nominate me, mainly because I had nothing to lose, haha! Surprisingly, I was voted in, which I think is purely a testament to the Python community’s commitment to diversity and global representation.

So many changes have taken place in the last few years: the formation of the steering council, the exponential growth of PyCon, and the spread of the language globally. All of these are things we’ve been able to discuss as a board.

I think defining the biggest challenge is tricky because it genuinely depends on each individual’s perspective. There are Directors on the board who are infinitely more technical than I am, and they can probably speak to all sorts of challenges on the future of Python from a more technical perspective.

For me, my focus is to help the growth of Python and its community to be as inclusive and representative as possible. I recently became the board’s communications co-chair, along with Lorena, and have been working with the blogging team. I’d like to see an increase in the representation of voices from communities around the world, maybe one day having blog posts and content on the official site in various languages.

All of these are things I feel are important and a challenge that we have to tackle well.

Ricky: I had the good fortune to meet you in person at PyCon 2019. It was your first PyCon (mine, too). How was your experience there? What was the standout moment for you?

Marlene: I genuinely loved Pycon 2019, and of course meeting you, Ricky, and the rest of the Real Python team! I’ve been to several PyCons around Africa and other parts of the world, but PyCon US is definitely a unique experience.

One of my favorite things about the conference was how friendly and laid back everyone was. I’d literally pass by people that I follow on Twitter, who have done amazing things, and they’d just be sitting there, eating a sandwich! I’m being a little funny here, but honestly, everyone was so welcoming and fun. I still can’t fully wrap my head around it.

I had so many standout moments, but I think I’d say Sha Wallace and Jessica Meckeller’s joint keynote was out of this world. I was tearing up because it’s such a great example of how we as a community can create positive change! I took away so many notes from that talk that I’d like to implement here in Zimbabwe.

This is an additional thing, I know, but I’d also like to say that I had so many standout moments outside the convention center, like having dinner with people from around the world and going to the local art gallery and trying bubble tea, also generally just wandering about the city. I had the best time, and I’m really looking forward to Pittsburg in 2020!

Ricky: If my memory serves right, at the end of PyCon, Microsoft and Adafruit donated a bunch of Circuit Playground Lunchbox Kits for you to take back to the girls you mentor in Zimbabwe. I’m curious if you managed to put them to use yet? Or what your plans are, outside of using them to teach them how to write some CircuitPython?

Marlene: Yes! That was another standout moment (to add to the twenty others above, haha). I got to chat with Nina Zakharenko and Katni from Adafruit, who are some of the loveliest human beings I’ve ever met! We had dinner together at some point and afterwards, they decided to donate the kits for me to bring back home and use for our ZimboPy programs.

I only recently got back to Zimbabwe, and I’ve started to play around with the kits myself. The plan at the moment is to host a workshop specifically centered around CircuitPython to show the girls that they can be playful and have fun while they code. I’m super pumped about it!

I was also talking to Kat about the potential of me hosting a tutorial at PyCon Africa with the Adafruit boards, similar to what she was demoing at the Microsoft booth. If I feel confident enough to do it by July, I think that’s something attendees at the conference should look out for!

Ricky: As if all the community work you do already isn’t enough, you’re also the organizer of the very first PyCon Africa! How’s the planning coming on? Can you share any details?

Marlene: I’m really excited that we’re in the final stages of planning for the conference. PyCon Africa is the first regional / pan-African gathering of the Python community! We have a great organizing committee, with people who have helped organize PyCons in Nigeria, Ghana, Namibia, Uganda, South Africa, and even the UK, so in that respect, I’ve been really lucky to land on this team.

The response from the global Python community has also been really encouraging. Real Python is actually one of our sponsors, which is amazing. Shout out to you! I think we’re moving in the right direction.

I’m this year’s chair, and although everything is running smoothly so far, I’ve been surprised by how all-consuming the process of planning can feel at times. Each time I look up it’s as if 3 months have passed! At PyCon US, I was speaking to a few other regional conference organizers and was glad to find I wasn’t the only one that felt like this. It can feel pretty overwhelming sometimes, but for the most part, I’m happy with the progress we’ve made and optimistic that we will put on a great event!

The conference is going to be held in Accra, Ghana, this year from August 6 to 10. We’re looking for sponsors, and there’s an option for individuals to buy a sharing ticket to sponsor a Pythonista who doesn’t have the means to buy their own ticket! You can find out more info on all this on our website: africa.pycon.org

Ricky: Now for my last questions. What else do you get up to in your spare time? Any other hobbies and interests, aside from Python and coding?

Marlene: I really enjoy long walks on the beach, haha. I’m joking, but I also genuinely do! I’ve been getting into running recently, and I’m hoping to try to get myself into good enough form to do a marathon at some point.

I also really like painting, but not in a cultured, cool way. I sometimes just like to paint and draw as a way to relax. Meditation and gratefulness journaling is also something I like to do each morning, as a good way to start the day.

It was an absolute pleasure speaking to Marlene. If you want to attend PyCon Africa, tickets are still available. If you want to find out more about Marlene, or reach out to her and discuss any of the amazing things she is doing, then you can find her on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Build Command Line Interfaces in Python With argparse

2019-06-12T14:00:00+00:00

One of the strengths of Python is that it comes with batteries included: it has a rich and versatile standard library that makes it one of the best programming languages for writing scripts for the command line. But, if you write scripts for the command line, then you also need to provide a good command line interface, which you can create with the Python argparse library.

In this article, you’ll learn:

What the Python argparse library is, and why it’s important to use it if you need to write command line scripts in Python
How to use the Python argparse library to quickly create a simple CLI in Python
What the advanced usage of the Python argparse library is

This article is written for early intermediate Pythonistas who probably write scripts in Python for their everyday work but have never implemented a command line interface for their scripts.

If that sounds like you, and you’re used to setting variable values at the beginning of your scripts or manually parsing the sys.argv system list instead of using a more robust CLI development tool, then this article is for you.

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

What Is a Command Line Interface?

The command line interface (also known as CLI) is a means to interact with a command line script. Python comes with several different libraries that allow you to write a command line interface for your scripts, but the standard way for creating a CLI in Python is currently the Python argparse library.

The Python argparse library was released as part of the standard library with Python 3.2 on February the 20th, 2011. It was introduced with Python Enhancement Proposal 389 and is now the standard way to create a CLI in Python, both in 2.7 and 3.2+ versions.

This new module was released as a replacement for the older getopt and optparse modules because they were lacking some important features.

The Python argparse library:

Allows the use of positional arguments
Allows the customization of the prefix chars
Supports variable numbers of parameters for a single option
Supports subcommands (A main command line parser can use other command line parsers depending on some arguments.)

Before getting started, you need to know how a command line interface works, so open a terminal on your computer and execute the command ls to get the list of the files contained in the current directory like this:

$ ls
dcdb_20180201.sg4    mastro35.sg4        openings.sg4
dcdb_20180201.si4    mastro35.si4        openings.si4
dcdb_20180201.sn4    mastro35.sn4        openings.sn4

As you can see, there are a bunch of files in the current directory, but the command didn’t return a lot of information about these files.

The good news is that you don’t need to look around for another program to have a richer list of the files contained in the current directory. You also don’t need to modify the ls command yourself, because it adopts a command line interface, that is just a set of tokens (called arguments) that you can use to configure the behavior of this command.

Now try to execute the command ls again, but with adding the -l option to the command line as in the example below:

$ ls -l
total 641824
-rw-------  1 dave  staff  204558286  5 Mar  2018 dcdb_20180201.sg4
-rw-------  1 dave  staff  110588409  5 Mar  2018 dcdb_20180201.si4
-rw-------  1 dave  staff    2937516  5 Mar  2018 dcdb_20180201.sn4
-rw-------  1 dave  staff     550127 27 Mar  2018 mastro35.sg4
-rw-------  1 dave  staff      15974 11 Gen 17:01 mastro35.si4
-rw-------  1 dave  staff       3636 27 Mar  2018 mastro35.sn4
-rw-------  1 dave  staff      29128 17 Apr  2018 openings.sg4
-rw-------  1 dave  staff        276 17 Apr  2018 openings.si4
-rw-------  1 dave  staff         86 18 Apr  2018 openings.sn4

The output is very different now. The command has returned a lot of information about the permissions, owner, group, and size of each file and the total directory occupation on the disk.

This is because you used the command line interface of the ls command and specified the -l option that enables the long format, a special format that returns a lot more information for every single file listed.

In order to familiarize yourself with this topic, you’re going to read a lot about arguments, options, and parameters, so let’s clarify the terminology right away:

An argument is a single part of a command line, delimited by blanks.
An option is a particular type of argument (or a part of an argument) that can modify the behavior of the command line.
A parameter is a particular type of argument that provides additional information to a single option or command.

Consider the following command:

$ ls -l -s -k /var/log

In this example, you have five different arguments:

ls: the name of the command you are executing
-l: an option to enable the long list format
-s: an option to print the allocated size of each file
-k: an option to have the size in kilobytes
/var/log: a parameter that provides additional information (the path to list) to the command

Note that, if you have multiple options in a single command line, then you can combine them into a single argument like this:

$ ls -lsk /var/log

Here you have only three arguments:

ls : the name of the command you are executing
-lsk: the three different options you want to enable (a combination of -l, -s, and -k)
/var/log: a parameter that provides additional information (the path to list) to the command

When to Use a Command Line Interface

Now that you know what a command line interface is, you may be wondering when it’s a good idea to implement one in your programs. The rule of thumb is that, if you want to provide a user-friendly approach to configuring your program, then you should consider a command line interface, and the standard way to do it is by using the Python argparse library.

Even if you’re creating a complex command line program that needs a configuration file to work, if you want to let your user specify which configuration file to use, it’s a good idea to accept this value by creating a command line interface with the Python argparse library.

How to Use the Python `argparse` Library to Create a Command Line Interface

Using the Python argparse library has four steps:

Import the Python argparse library
Create the parser
Add optional and positional arguments to the parser
Execute .parse_args()

After you execute .parse_args(), what you get is a Namespace object that contains a simple property for each input argument received from the command line.

In order to see these four steps in detail with an example, let’s pretend you’re creating a program named myls.py that lists the files contained in the current directory. Here’s a possible implementation of your command line interface without using the Python argparse library:

# myls.py
import os
import sys

if len(sys.argv) > 2:
    print('You have specified too many arguments')
    sys.exit()

if len(sys.argv) < 2:
    print('You need to specify the path to be listed')
    sys.exit()

input_path = sys.argv[1]

if not os.path.isdir(input_path):
    print('The path specified does not exist')
    sys.exit()

print('\n'.join(os.listdir(input_path)))

This is a possible implementation of the command line interface for your program that doesn’t use the Python argparse library, but if you try to execute it, then you’ll see that it works:

$ python myls.py
You need to specify the path to be listed

$ python myls.py /mnt /proc /dev
You have specified too many arguments

$ python myls.py /mnt
dir1
dir2

As you can see, the script does work, but the output is quite different from the output you’d expect from a standard built-in command.

Now, let’s see how the Python argparse library can improve this code:

# myls.py
# Import the argparse library
import argparse

import os
import sys

# Create the parser
my_parser = argparse.ArgumentParser(description='List the content of a folder')

# Add the arguments
my_parser.add_argument('Path',
                       metavar='path',
                       type=str,
                       help='the path to list')

# Execute the parse_args() method
args = my_parser.parse_args()

input_path = args.Path

if not os.path.isdir(input_path):
    print('The path specified does not exist')
    sys.exit()

print('\n'.join(os.listdir(input_path)))

The code has changed a lot with the introduction of the Python argparse library.

The first big difference compared to the previous version is that the if statements to check the arguments provided by the user are gone because the library will check the presence of the arguments for us.

We’ve imported the Python argparse library, created a simple parser with a brief description of the program’s goal, and defined the positional argument we want to get from the user. Lastly, we have executed .parse_args() to parse the input arguments and get a Namespace object that contains the user input.

Now, if you run this code, you’ll see that with just four lines of code. You have a very different output:

$ python myls.py
usage: myls.py [-h] path
myls.py: error: the following arguments are required: path

As you can see, the program has detected that you needed at least a positional argument (path), and so the execution of the program has been interrupted with a specific error message.

You may also have noticed that now your program accepts an optional -h flag, like in the example below:

$ python myls.py -h
usage: myls.py [-h] path

List the content of a folder

positional arguments:
path        the path to list

optional arguments:
-h, --help  show this help message and exit

Good, now the program responds to the -h flag, displaying a help message that tells the user how to use the program. Isn’t that neat, considering that you didn’t even need to ask for that feature?

Lastly, with just four lines of code, now the args variable is a Namespace object, which has a property for each argument that has been gathered from the command line. That’s super convenient.

The Advanced Use of the Python `argparse` Library

In the previous section, you learned the basic usage of the Python argparse library, and now you can implement a simple command line interfaces for all your programs. However, there’s a lot more that you can achieve with this library. In this section, you’ll see almost everything this library can offer you.

Setting the Name of the Program

By default, the library uses the value of the sys.argv[0] element to set the name of the program, which as you probably already know is the name of the Python script you have executed. However, you can specify the name of your program just by using the prog keyword:

# Create the parser
my_parser = argparse.ArgumentParser(prog='myls',
                                    description='List the content of a folder')

With the prog keyword, you specify the name of the program that will be used in the help text:

$ python myls.py
usage: myls [-h] path
myls.py: error: the following arguments are required: path

As you can see, now the program name is just myls instead of myls.py.

Displaying a Custom Program Usage Help

By default, the program usage help has a standard format defined by the Python argparse library. However, you can customize it with the usage keyword like this:

# Create the parser
my_parser = argparse.ArgumentParser(prog='myls',
                                    usage='%(prog)s [options] path',
                                    description='List the content of a folder')

Note that, at runtime, the %(prog)s token is automatically replaced with the name of your program:

$ python myls.py
usage: myls [options] path
myls: error: too few arguments

As you can see, the help of the program now shows a different usage string, where the [-h] option has been replaced by a generic [options] token.

Displaying Text Before and After the Arguments Help

To customize the text displayed before and after the arguments help text, you can use two different keywords:

description: for the text that is shown before the help text
epilog: for the text shown after the help text

You’ve already seen the description keyword in the previous chapter, so let’s see an example of how the epilog keyword works:

# Create the parser
my_parser = argparse.ArgumentParser(description='List the content of a folder',
                                    epilog='Enjoy the program! :)')

The epilog keyword here has customized the text that will be shown after the standard help text:

$ python myls.py -h
usage: myls.py [-h] path

List the content of a folder

positional arguments:
path        the path to list

optional arguments:
-h, --help  show this help message and exit

Enjoy the program! :)

Now the output shows the extra text that has been customized by the epilog keyword.

Customizing the Allowed Prefix Chars

Another feature that the Python argparse library offers you is the ability to customize the prefix chars, which are the chars that you can use to pass optional arguments to the command line interface.

By default, the standard prefix char is the dash (-) character, but if you want to use a different character, then you can customize it by using the prefix_chars keyword while defining the parser like this:

# Create the parser
my_parser = argparse.ArgumentParser(description='List the content of a folder',
                                    epilog='Enjoy the program! :)',
                                    prefix_chars='/')

After the redefinition, the program now supports a completely different prefix char, and the help text has changed accordingly:

$ python myls.py
usage: myls.py [/h] path
myls.py: error: too few arguments

As you can see, now your program does not support the -h flag but the /h flag. That’s especially useful when you’re coding for Microsoft Windows because Windows users are used to these prefix chars when working with the command line.

Setting Prefix Chars for Files That Contain Arguments to Be Included

When you are dealing with a very long or complicated command line, it can be a good idea to save the arguments to an external file and ask your program to load arguments from it. The Python argparse library can do this work for you out of the box.

To test this feature, create the following Python program:

# fromfile_example.py
import argparse

my_parser = argparse.ArgumentParser(fromfile_prefix_chars='@')

my_parser.add_argument('a',
                       help='a first argument')

my_parser.add_argument('b',
                       help='a second argument')

my_parser.add_argument('c',
                       help='a third argument')

my_parser.add_argument('d',
                       help='a fourth argument')

my_parser.add_argument('e',
                       help='a fifth argument')

my_parser.add_argument('-v',
                       '--verbose',
                       action='store_true',
                       help='an optional argument')

# Execute parse_args()
args = my_parser.parse_args()

print('If you read this line it means that you have provided '
      'all the parameters')

Note that we have used the fromfile_prefix_chars keyword while creating the parser.

Now, if you try to execute your program without passing any arguments, then you’ll get an error message:

$ python fromfile_example.py
usage: fromfile_example.py [-h] [-v] a b c d e
fromfile_example.py: error: the following arguments are required: a, b, c, d, e

Here you can see that the Python argparse library is complaining because you have not provided enough arguments.

So let’s create a file named args.txt that contains all the necessary parameters, with an argument on each line like this:

first
second
third
fourth
fifth

Now that you have specified a prefix char to get arguments from an external file, open a terminal and try to execute the previous program:

$ python fromfile_example.py @args.txt
If you read this line it means that you have provided all the parameters

In this example, you can see that argparse has read the arguments from the args.txt file.

Allowing or Disallowing Abbreviations

One of the features that the Python argparse library provides out of the box is the ability to handle abbreviations. Consider the following program, which prints out the value you specify on the command line interface for the --input argument:

# abbrev_example.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('--input', action='store', type=int, required=True)
my_parser.add_argument('--id', action='store', type=int)

args = my_parser.parse_args()

print(args.input)

This program prints out the value you specify for the --input argument. We haven’t looked at the optional arguments yet, but don’t worry, we will discuss them in depth in just a moment. For now, just consider this argument like any other positional argument we already saw, with the difference that the name starts with a couple of dashes.

Now let’s see how the Python argparse library can handle abbreviations, by calling our program multiple times, specifying a different abbreviation of the input argument at each run:

$ python abbrev_example.py --input 42
42

$ python abbrev_example.py --inpu 42
42

$ python abbrev_example.py --inp 42
42

$ python abbrev_example.py --in 42
42

As you can see, the optional parameters can always be shortened unless the abbreviation can lead to an incorrect interpretation. But what happens if you try to execute the program specifying just --i 42? In this case, argparse doesn’t know if you want to pass the value 42 to the --input argument or to the --id argument, so it exits the program with a specific error message:

$ python abbrev_example.py --i 42
usage: abbrev_example.py [-h] --input INPUT [--id ID]
abbrev_example.py: error: ambiguous option: --i could match --input, --id

However, if you don’t like this behavior, and you want to force your users to specify the full name of the options they use, then you can just disable this feature with the keyword allow_abbrev set to False during the creation of the parser:

# abbrev_example.py
import argparse

my_parser = argparse.ArgumentParser(allow_abbrev=False)
my_parser.add_argument('--input', action='store', type=int, required=True)

args = my_parser.parse_args()

print(args.input)

Now, if you try the code above, you’ll see that the abbreviations are no longer permitted:

$ python abbrev_example.py --inp 42
usage: abbrev_example.py [-h] --input INPUT
abbrev_example.py: error: the following arguments are required: --input

The error message tells the user that the --input parameter has not been specified because the --inp abbreviation has not been recognized.

Using Auto Help

In some of the previous examples, you used the -h flag to get a help text. This is a very convenient feature that the Python argparse library allows you to use without having to code anything. However, sometimes you may want to disable this feature. To do that, just use the add_help keyword when creating the parser:

# Create the parser
my_parser = argparse.ArgumentParser(description='List the content of a folder',
                                    add_help=False)

The code in the example above specifies the add_help keyword set to False, so now if you run the code, you’ll see that the -h flag isn’t accepted anymore:

$ myls.py
usage: myls.py path
myls.py: error: the following arguments are required: path

As you can see, the -h flag is no longer shown or accepted.

Setting the Name or Flags of the Arguments

There are basically two different types of arguments that you can add to your command line interface:

Positional arguments
Optional arguments

Positional arguments are the ones your command needs to operate.

In the previous example, the argument path was a positional argument, and our program couldn’t work without it. They are called positional because their position defines their function.

For example, consider the cp command on Linux (or the copy command in Windows). Here’s the standard usage:

$ cp [OPTION]... [-T] SOURCE DEST

The first positional argument after the cp command is the source of the file you’re going to copy. The second one is the destination where you want to copy it.

Optional arguments are not mandatory, and when they are used they can modify the behavior of the command at runtime. In the cp example, an optional argument is, for example, the -r flag, which makes the command copy directories recursively.

Syntactically, the difference between positional and optional arguments is that optional arguments start with - or --, while positional arguments don’t.

To add an optional argument, you just need to call .add_argument() again and name the new argument with a starting -.

For example, try to modify the myls.py like this:

# myls.py
# Import the argparse library
import argparse

import os
import sys

# Create the parser
my_parser = argparse.ArgumentParser(description='List the content of a folder')

# Add the arguments
my_parser.add_argument('Path',
                       metavar='path',
                       type=str,
                       help='the path to list')
my_parser.add_argument('-l',
                       '--long',
                       action='store_true',
                       help='enable the long listing format')

# Execute parse_args()
args = my_parser.parse_args()

input_path = args.Path

if not os.path.isdir(input_path):
    print('The path specified does not exist')
    sys.exit()

print('\n'.join(os.listdir(input_path)))

Now, try to execute this program to see if the new -l option is accepted:

$ python myls.py -h
usage: myls.py [-h] [-l] path

List the content of a folder

positional arguments:
path        the path to list

optional arguments:
-h, --help  show this help message and exit
-l, --long  enable the long listing format

As you can see, now the program also accepts (but doesn’t require) the -l option, which allows the user to get a long listing format for the directory content.

Setting the Action to Be Taken for an Argument

When you add an optional argument to your command line interface, you can also define what kind of action to take when the argument is specified. This means that you usually need to specify how to store the value to the Namespace object you will get when .parse_args() is executed.

There are several actions that are already defined and ready to be used. Let’s analyze them in detail:

store stores the input value to the Namespace object. (This is the default action.)
store_const stores a constant value when the corresponding optional arguments are specified.
store_true stores the Boolean value True when the corresponding optional argument is specified and stores a False elsewhere.
store_false stores the Boolean value False when the corresponding optional argument is specified and stores True elsewhere.
append stores a list, appending a value to the list each time the option is provided.
append_const stores a list appending a constant value to the list each time the option is provided.
count stores an int that is equal to the times the option has been provided.
help shows a help text and exits.
version shows the version of the program and exits.

Let’s create an example to test all the actions we have seen so far:

# actions_example.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.version = '1.0'
my_parser.add_argument('-a', action='store')
my_parser.add_argument('-b', action='store_const', const=42)
my_parser.add_argument('-c', action='store_true')
my_parser.add_argument('-d', action='store_false')
my_parser.add_argument('-e', action='append')
my_parser.add_argument('-f', action='append_const', const=42)
my_parser.add_argument('-g', action='count')
my_parser.add_argument('-i', action='help')
my_parser.add_argument('-j', action='version')

args = my_parser.parse_args()

print(vars(args))

This script accepts an optional argument for each type of action discussed and then prints the value of the arguments read from the command line. Test it by executing this example:

$ python actions_example.py
{'a': None, 'b': None, 'c': False, 'd': True, 'e': None, 'f': None, 'g': None}

As you can see, if we do not specify any arguments, then the default values are generally None, at least for the actions that do not store a Boolean value.

The use of the store action, instead, stores the value we pass without any further consideration:

$ python actions_example.py -a 42
{'a': '42', 'b': None, 'c': False, 'd': True, 'e': None, 'f': None, 'g': None}

$ python actions_example.py -a "test"
{'a': 'test', 'b': None, 'c': False, 'd': True, 'e': None, 'f': None, 'g': None}

The store_const action, stores the defined const when the arguments are provided. In our test, we provided just the b argument and the value of args.b is now 42:

$ python actions_example.py -b
{'a': None, 'b': 42, 'c': False, 'd': True, 'e': None, 'f': None, 'g': None}

The store_true action stores a True Boolean when the argument is passed and store a False Boolean elsewhere. If you need the opposite behavior, just use the store_false action:

$ python actions_example.py
{'a': None, 'b': None, 'c': False, 'd': True, 'e': None, 'f': None, 'g': None}
$ python actions_example.py -c
{'a': None, 'b': None, 'c': True, 'd': True, 'e': None, 'f': None, 'g': None}
$ python actions_example.py -d
{'a': None, 'b': None, 'c': False, 'd': False, 'e': None, 'f': None, 'g': None}

The append action lets you create a list of all the values passed to the CLI with the same argument:

$ python actions_example.py -e me -e you -e us
{'a': None, 'b': None, 'c': False, 'd': True, 'e': ['me', 'you', 'us'], 'f': None, 'g': None}

The append_const action is similar to the append one, but it always appends the same constant value:

$ python actions_example.py -f -f
{'a': None, 'b': None, 'c': False, 'd': True, 'e': None, 'f': [42, 42], 'g': None}

The count action counts how many time an argument is passed. It’s quite useful when you want to implement a verbosity level for your program, since you can define a level where -v is less verbose than -vvv:

$ python actions_example.py -ggg
{'a': None, 'b': None, 'c': False, 'd': True, 'e': None, 'f': None, 'g': 3}
$ python actions_example.py -ggggg
{'a': None, 'b': None, 'c': False, 'd': True, 'e': None, 'f': None, 'g': 5}

The help action is the one you already saw at the beginning of the article. It’s enabled for the -h flag by default, but you can use it for another flag if you want:

$ python actions_example.py -i
usage: actions_example.py [-h] [-a A] [-b] [-c] [-d] [-e E] [-f] [-g] [-i]
[-j]

optional arguments:
-h, --help  show this help message and exit
-a A
-b
-c
-d
-e E
-f
-g
-i
-j          show program's version number and exit

The version action is the last one you can use. It just shows the version of the program (defined by assigning a value to the .version property of the parser) and then ends the execution of the script:

$ python actions_example.py -j
1.0

Another possibility you have is to create a custom action. That’s done by subclassing the argparse.Action class and implementing a couple of methods.

Look at the following example, which is a custom store action that is just a little bit more verbose than the standard one:

# custom_action.py
import argparse

class VerboseStore(argparse.Action):
    def __init__(self, option_strings, dest, nargs=None, **kwargs):
        if nargs is not None:
            raise ValueError('nargs not allowed')
        super(VerboseStore, self).__init__(option_strings, dest, **kwargs)

    def __call__(self, parser, namespace, values, option_string=None):
        print('Here I am, setting the ' \
              'values %r for the %r option...' % (values, option_string))
        setattr(namespace, self.dest, values)

my_parser = argparse.ArgumentParser()
my_parser.add_argument('-i', '--input', action=VerboseStore, type=int)

args = my_parser.parse_args()

print(vars(args))

This example defines a custom action that is just like the store action but a little bit more verbose. Try to execute it to test how it works:

$ python custom_action.py -i 42
Here I am, setting the values 42 for the '-i' option...
{'input': 42}

As you can see, the program has printed out a line before setting the value 42 for the -i parameter.

Setting the Number of Values That Should Be Consumed by the Option

The parser, by default, assumes that you’ll consume a single parameter for each argument, but you can modify this default behavior by specifying a different number of values with the nargs keyword.

For example, if you want to create an optional argument that consumes exactly three values, then you can specify the number 3 as the value for the nargs keyword while adding the parameter to the parser:

# nargs_example.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('--input', action='store', type=int, nargs=3)

args = my_parser.parse_args()

print(args.input)

Now, the program accepts three values for the --input parameter:

$ python nargs_example.py --input 42
usage: nargs_example.py [-h] [--input INPUT INPUT INPUT]
nargs_example.py: error: argument --input: expected 3 arguments

$ python nargs_example.py --input 42 42 42
[42, 42, 42]

As you can see, the value of the args.input variable is now a list that contains three values.

However, the nargs keyword can also accept the following:

?: a single value, which can be optional
*: a flexible number of values, which will be gathered into a list
+: like *, but requiring at least one value
argparse.REMAINDER: all the values that are remaining in the command line

So, for example, in the following program, the positional argument input takes a single value when provided, but if the value is not provided, then the one specified by the default keyword is used:

# nargs_example.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('input',
                       action='store',
                       nargs='?',
                       default='my default value')

args = my_parser.parse_args()

print(args.input)

Now you can choose to set a specific value for the input argument or not. In this case, the default value will be used:

$ python nargs_example.py 'my custom value'
my custom value

$ python nargs_example.py
my default value

To take a flexible number of values and gather them all into a single list, you need to specify the * value for the nargs keyword like this:

# nargs_example.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('input',
                       action='store',
                       nargs='*',
                       default='my default value')

args = my_parser.parse_args()

print(args.input)

See how this code allows the user to set a flexible number of values for the expected argument:

$ python nargs_example.py me you us
['me', 'you', 'us']

$ python nargs_example.py
my default value

If you need to take a variable number of values, but you have to be sure that at least one value is specified, then you can use the + value for the nargs keyword like this:

# nargs_example.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('input', action='store', nargs='+')

args = my_parser.parse_args()

print(args.input)

In this case, if you execute the program with no positional arguments, then you will receive an explicit error message:

$ python nargs_example.py me you us
['me', 'you', 'us']

$ python nargs_example.py
usage: nargs_example.py [-h] input [input ...]
nargs_example.py: error: the following arguments are required: input

Lastly, if you need to grab all the remaining arguments that have been specified on the command line and put them all in a list, then the nargs keyword has to be set to argparse.REMAINDER like this:

# nargs_example.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('first', action='store')
my_parser.add_argument('others', action='store', nargs=argparse.REMAINDER)

args = my_parser.parse_args()

print('first = %r' % args.first)
print('others = %r' % args.others)

Now if you execute this program, you will see that the first value will be associated with the first parameters, while all the other values provided will be associated with the second one:

$ python nargs_example.py me you us
first = 'me'
others = ['you', 'us']

Note how all the remaining values are put in a single list.

Setting a Default Value Produced if the Argument Is Missing

You already know that the user can decide whether or not to specify optional arguments in the command line. When arguments are not specified, the corresponding value is generally set to None.

However, it is possible to define a default value for an argument when it’s not provided:

# default_example.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('-a', action='store', default='42')

args = my_parser.parse_args()

print(vars(args))

If you execute this example without passing the -a option, then this is the output you get:

$ python default_example.py
{'a': '42'}

You can see that now the option -a is set to 42, even if you didn’t explicitly set the value on the command line.

Setting the Type of the Argument

By default, all the input argument values are treated as if they were strings. However, it’s possible to define the type for the corresponding property of the Namespace object you get after .parse_args() is invoked just by defining it with the type keyword like this:

# type_example.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('-a', action='store', type=int)

args = my_parser.parse_args()

print(vars(args))

Specifying the int value for the argument, you are telling argparse that the .a property of your Namespace object has to be an int (instead of a string):

$ python type_example.py -a 42
{'a': 42}

Besides, now the value of the argument is checked at runtime, and if there’s a problem with the type of the value provided at the command line, then the execution is interrupted with a clear error message:

$ python type_example.py -a "that's a string"
usage: type_example.py [-h] [-a A]
type_example.py: error: argument -a: invalid int value: "that's a string"

In this case, the error message is very clear because it states that you were expected to pass an int instead of a string.

Setting a Domain of Allowed Values for a Specific Argument

Another interesting possibility with the Python argparse library creating a domain of allowed values for specific arguments. You can do this by providing a list of accepted values while adding the new option:

# choices_ex.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('-a', action='store', choices=['head', 'tail'])

args = my_parser.parse_args()

Please note that if you are accepting numeric values, then you can even use range() to specify a range of accepted values:

# choices_ex.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('-a', action='store', type=int, choices=range(1, 5))

args = my_parser.parse_args()

print(vars(args))

In this case, the value provided on the command line will be automatically checked against the range defined:

$ python choices_ex.py -a 4
{'a': 4}

$ python choices_ex.py -a 40
usage: choices_ex.py [-h] [-a {1,2,3,4}]
choices_ex.py: error: argument -a: invalid choice: 40 (choose from 1, 2, 3, 4)

If the input number is outside the defined range, then you’ll get an error message.

Setting Whether the Argument Is Required

If you want to force your user to specify the value for an optional argument, then you can use the required keyword:

# required_example.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('-a',
                       action='store',
                       choices=['head', 'tail'],
                       required=True)

args = my_parser.parse_args()

print(vars(args))

If you use the required keyword set to True for an optional argument, then the user will be forced to set a value for that argument:

$ python required_example.py
usage: required_example.py [-h] -a {head,tail}
required_example.py: error: the following arguments are required: -a

$ python required_example.py -a head
{'a': 'head'}

That said, please bear in mind that requiring an optional argument is usually considered bad practice since the user wouldn’t expect to have to set a value for an argument that should be optional.

Showing a Brief Description of What an Argument Does

A great feature of the Python argparse library is that, by default, you have the ability to ask for help just by adding the -h flag to your command line.

To make it even better, you can add help text to your arguments, so as to give the users even more help when they execute your program with the -h flag:

# help_example.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('-a',
                       action='store',
                       choices=['head', 'tail'],
                       help='set the user choice to head or tail')

args = my_parser.parse_args()

print(vars(args))

This example shows you how to define a custom help text for the -a argument, and now the help text will be more clear for the user:

$ python help_example.py -h
usage: help_example.py [-h] [-a {head,tail}]

optional arguments:
-h, --help      show this help message and exit
-a {head, tail}  set the user choice to head or tail

Defining a help text for all the arguments is a really good idea because it makes the usage of your program more clear to the user.

Defining Mutually Exclusive Groups

Another interesting option you have when working with the Python argparse library is the ability to create a mutually exclusive group for options that cannot coexist in the same command line:

# groups.py
import argparse

my_parser = argparse.ArgumentParser()
my_group = my_parser.add_mutually_exclusive_group(required=True)

my_group.add_argument('-v', '--verbose', action='store_true')
my_group.add_argument('-s', '--silent', action='store_true')

args = my_parser.parse_args()

print(vars(args))

You can specify the -v or the -s flags, unless they aren’t on the same command line, and also the help text that argparse provides reflects this constraint:

$ python groups.py -h
usage: groups.py [-h] (-v | -s)

optional arguments:
  -h, --help     show this help message and exit
  -v, --verbose
  -s, --silent

$ python groups.py -v -s
usage: groups.py [-h] (-v | -s)
groups.py: error: argument -s/--silent: not allowed with argument -v/--verbose

If you specify all the options of a mutually exclusive group on the same command line you will get an error.

Setting the Argument Name in Usage Messages

If an argument accepts an input value, it can be useful to give this value a name that the parser can use to generate the help message, and this can be done by using the metavar keyword. In the following example, you can see how you can use the metavar keyword to specify a name for the value of the -v flag:

# metavar_example.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('-v',
                       '--verbosity',
                       action='store',
                       type=int,
                       metavar='LEVEL')

args = my_parser.parse_args()

print(vars(args))

Now, if you run your program with the -h flag, the help text assigns the name LEVEL to the value of the -v flag:

$ python metavar_example.py -h
usage: metavar_example.py [-h] [-v LEVEL]

optional arguments:
  -h, --help            show this help message and exit
  -v LEVEL, --verbosity LEVEL

Please note that, in the help message, the value accepted for the -v flag is now named LEVEL.

Setting the Name of the Attribute to Be Added to the Object Once Parsed

As you have already seen, when you add an argument to the parser, the value of this argument is stored in a property of the Namespace object. This property is named by default as the first argument passed to .add_argument() for the positional argument and as the long option string for optional arguments.

If an option uses dashes (as is fairly common), they will be converted to underscores in the property name:

import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('-v',
                       '--verbosity-level',
                       action='store',
                       type=int)

args = my_parser.parse_args()
print(args.verbosity_level)

However, it’s possible to specify the name of this property just by using the keyword dest when you’re adding an argument to the parser:

# dest_example.py
import argparse

my_parser = argparse.ArgumentParser()
my_parser.add_argument('-v',
                       '--verbosity',
                       action='store',
                       type=int,
                       dest='my_verbosity_level')

args = my_parser.parse_args()

print(vars(args))

By running this program, you’ll see that now the args variable contains a .my_verbosity_level property, even if by default the name of the property should have been .verbosity:

$ python dest_example.py -v 42
{'my_verbosity_level': 42}

The default name of this property would have been .verbosity, but since a different name has been specified by the dest keyword, .my_verbosity_level has been used.

Conclusion

Now you know what a command line interface is and how you can create one in Python by using the Python argparse library.

In this article, you’ve learned:

What the Python argparse library is, and why it’s important to use it if you need to write command line scripts in Python
How to use the Python argparse library to quickly create a simple CLI in Python
What the advanced usage of the Python argparse library is

Writing a good command line interface is a good way to create self-explanatory programs and give users a means of interacting with your application.

If you still have questions, don’t hesitate to reach out in the comment section below and take a look at the official documentation and the Tshepang Lekhonkhobe tutorial that is part of the official Python 3 HOWTO documentation.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

OOP Method Types in Python: @classmethod vs @staticmethod vs Instance Methods

2019-06-11T14:00:00+00:00

What’s the difference between @classmethod, @staticmethod, and “plain/regular” instance methods in Python? You’ll know the answer after watching this video course:

Regular (instance) methods need a class instance and can access the instance through self. They can read and modify an objects state freely.
Class methods, marked with the @classmethod decorator, don’t need a class instance. They can’t access the instance (self) but they have access to the class itself via cls.
Static methods, marked with the @staticmethod decorator, don’t have access to cls or self. They work like regular functions but belong to the class’s namespace.

In this course you’ll go over the differences between these three kinds of methods in Python. You’ll also see when to use each with a simple example, so you can improve your object-oriented programming (OOP) skills in Python.

Recommended Resources

Learning Path: Object-Oriented Programming (OOP) With Python

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Make a Twitter Bot in Python With Tweepy

2019-06-10T14:00:00+00:00

In this article, you’ll learn how to make your own Twitter Bot in Python with Tweepy, a package that provides a very convenient way to use the Twitter API.

Twitter is one of the most widely used social networks. For many organizations and people, having a great Twitter presence is a key factor to keeping their audience engaged.

Part of having a great Twitter presence involves keeping your account active with new tweets and retweets, following interesting accounts, and quickly replying to your followers’ messages. You can do all this work manually, but that can take a lot of time. Instead, you can rely on a Twitter Bot, a program that automates all or part of your Twitter activity.

In this article, you’ll learn how to:

Improve and automate your interactions with your Twitter audience
Install Tweepy
Sign up as a Twitter developer to use its API
Use Tweepy to invoke the Twitter API
Build Twitter Bots
Deploy the bots to a server using Docker and AWS

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Hello Tweepy

To start, here’s how you can use Tweepy to create a tweet saying Hello Tweepy:

import tweepy

# Authenticate to Twitter
auth = tweepy.OAuthHandler("CONSUMER_KEY", "CONSUMER_SECRET")
auth.set_access_token("ACCESS_TOKEN", "ACCESS_TOKEN_SECRET")

# Create API object
api = tweepy.API(auth)

# Create a tweet
api.update_status("Hello Tweepy")

This is a short example, but it shows the four steps common to all Tweepy programs:

Import the tweepy package
Set the authentication credentials
Create a new tweepy.API object
Use the api object to call the Twitter API

Objects belonging to the tweepy.API class offer a vast set of methods that you can use to access almost all Twitter functionality. In the code snippet, we used update_status() to create a new Tweet.

We will see later in this article how the authentication works and how you can create the required authentication key, token, and secrets.

This is just a little example of what you can do with Tweepy. Through this article, you’ll learn how to build programs that interact with Twitter in much more interesting and complex ways.

Twitter API

The Twitter API gives developers access to most of Twitter’s functionality. You can use the API to read and write information related to Twitter entities such as tweets, users, and trends.

Technically, the API exposes dozens of HTTP endpoints related to:

Tweets
Retweets
Likes
Direct messages
Favorites
Trends
Media

Tweepy, as we’ll see later, provides a way to invoke those HTTP endpoints without dealing with low-level details.

The Twitter API uses OAuth, a widely used open authorization protocol, to authenticate all the requests. Before making any call to the Twitter API, you need to create and configure your authentication credentials. Later in this article, you’ll find detailed instructions for this.

You can leverage the Twitter API to build different kinds of automations, such as bots, analytics, and other tools. Keep in mind that Twitter imposes certain restrictions and policies about what you can and cannot build using its API. This is done to guarantee users a good experience. The development of tools to spam, mislead users, and so on is forbidden.

The Twitter API also imposes rate limits about how frequently you’re allowed to invoke API methods. If you exceed these limits, you’ll have to wait between 5 and 15 minutes to be able to use the API again. You must consider this while designing and implementing bots to avoid unnecessary waits.

You can find more information about the Twitter API’s policies and limits in its official documentation:

What Is Tweepy?

Tweepy is an open source Python package that gives you a very convenient way to access the Twitter API with Python. Tweepy includes a set of classes and methods that represent Twitter’s models and API endpoints, and it transparently handles various implementation details, such as:

Data encoding and decoding
HTTP requests
Results pagination
OAuth authentication
Rate limits
Streams

If you weren’t using Tweepy, then you would have to deal with low-level details having to do with HTTP requests, data serialization, authentication, and rate limits. This could be time consuming and prone to error. Instead, thanks to Tweepy, you can focus on the functionality you want to build.

Almost all the functionality provided by Twitter API can be used through Tweepy. The only current limitation, as of version 3.7.0, is that Direct Messages don’t work properly due to some recent changes in the Twitter API.

Using Tweepy

In this section, you’ll learn how to install Tweepy for development, configure authentication credentials, and interact with the Twitter API.

Installation

Tweepy can be installed using pip, a Python package manager. In this article, we’re going to use a virtual environment (virtualenv) for the projects to avoid depending on system-wide packages. For more information on virtual environments and pip, check out Python Virtual Environments: A Primer and What Is Pip? A Guide for New Pythonistas.

You can get started by creating a project called tweepy-bots. The first step is to create a directory and a virtual environment:

$ mkdir tweepy-bots
$ cd tweepy-bots
$ python3 -m venv venv

The commands above create the virtual environment inside the project directory.

Then you can install the Tweepy package. First, you need to activate the newly created virtual environment and then use pip to do the installation:

$ source ./venv/bin/activate
$ pip install tweepy

Now that Tweepy is installed, let’s create a requirements.txt file containing the names of your dependencies. You can use the pip command freeze for this task:

$ pip freeze > requirements.txt

You’ll use this requirements.txt file when you’re deploying the project.

Note: The method used to activate the virtual environment may be different, depending on your operating system and shell. You can learn more about this in the venv documentation.

Creating Twitter API Authentication Credentials

As we have previously seen, the Twitter API requires that all requests use OAuth to authenticate. So you need to create the required authentication credentials to be able to use the API. These credentials are four text strings:

Consumer key
Consumer secret
Access token
Access secret

If you already have a Twitter user account, then follow these steps to create the key, token, and secrets. Otherwise, you have to sign up as a Twitter user before proceeding.

Step 1: Apply for a Twitter Developer Account

Go to the Twitter developer site to apply for a developer account. Here, you have to select the Twitter user responsible for this account. It should probably be you or your organization. Here’s what this page looks like:

In this case, I chose to use my own account, @MikezGarcia.

Twitter then requests some information about how you plan to use the developer account, as showed below:

You have to specify the developer account name and whether you are planning to use it for personal purpose or for your organization.

Step 2: Create an Application

Twitter grants authentication credentials to apps, not accounts. An app can be any tool or bot that uses the Twitter API. So you need to register your an app to be able to make API calls.

To register your app, go to your Twitter apps page and select the Create an app option.

You need to provide the following information about your app and its purpose:

App name: a name to identify your application (such as examplebot)
Application description: the purpose of your application (such as An example bot for a Real Python article)
Your or your application’s website URL: required, but can be your personal site’s URL since bots don’t need a URL to work
Use of the app: how users will use your app (such as This app is a bot that will automatically respond to users)

Step 3: Create the Authentication Credentials

To create the authentication credentials, go to your Twitter apps page. Here’s what the Apps page looks like:

Here you’ll find the Details button of your app. Clicking this button takes you to the next page, where you can generate the credentials.

By selecting the Keys and tokens tab, you can generate and copy the key, token, and secrets to use them in your code:

After generating the credentials, save them to later use them in your code.

You can test the credentials using the following snippet:

import tweepy

# Authenticate to Twitter
auth = tweepy.OAuthHandler("pGBDoAaEpkliVKBOLwjtcmHGc", 
    "xF3g1wrP50b6BlZEd20u4oVfjgH1FGQcuWUzlQO5aUWOufvlhw")
auth.set_access_token("622518493-6VcLIPprbQbv9wkcBBPvCle8vsjU9fE85Dq9oStl", 
    "tH9aKQbQQ1iRdYTcLSsPwitl44BkAc6jilrsU0ifnXvZhq")

api = tweepy.API(auth)

try:
    api.verify_credentials()
    print("Authentication OK")
except:
    print("Error during authentication")

In this snippet, credentials are tested using verify_credentials(). If all goes well, you should see a message saying Authentication OK.

Note: All of the credentials used in this article are only examples and will not work. You have to generate and use your own credentials.

Review of Tweepy Functionality

Tweepy gives you an interface to access the Twitter API from Python. It does so by encapsulating much of the Twitter API’s complexity and adding a model layer and other useful functionalities on top of it.

Since, over time, the names of various Twitter concepts have evolved, some old names are still used in Tweepy. So keep in mind that, in the context of this article, these equivalences hold:

A status is a tweet .
A friendship is a follow-follower relationship.
A favorite is a like.

Now that you know how Tweepy names things, let’s see how it works.

Tweepy’s functionality can be divided into the following groups:

OAuth
The API class
Models
Cursors
Streams

Now you’re going to look into these groups to learn about the functionality each one provides.

OAuth

Tweepy takes care of all the details for using OAuth required by the Twitter API to authenticate each request. It provides an OAuthHandler class that you can use to set the credentials to be used in all API calls.

This code snippet shows how you can create an OAuthHandler object that can later be used for API calls:

import tweepy

# Authenticate to Twitter
auth = tweepy.OAuthHandler("CONSUMER_KEY", "CONSUMER_SECRET")
auth.set_access_token("ACCESS_TOKEN", "ACCESS_TOKEN_SECRET")

Here you’re telling Tweepy to use the credentials that you created in Step 3: Create the Authentication Credentials. You must replace CONSUMER_KEY, CONSUMER_SECRET, ACCESS_TOKEN, and ACCESS_TOKEN_SECRET with the values you previously generated.

The API Class

The API class has many methods that provide access to Twitter API endpoints. Using these methods, you can access the Twitter API’s functionality.

The following code snippet creates an API object that you can use to invoke Twitter API methods. Setting wait_on_rate_limit and wait_on_rate_limit_notify to True makes the API object print a message and wait if the rate limit is exceeded:

import tweepy

# Authenticate to Twitter
auth = tweepy.OAuthHandler("CONSUMER_KEY", "CONSUMER_SECRET")
auth.set_access_token("ACCESS_TOKEN", "ACCESS_TOKEN_SECRET")

# Create API object
api = tweepy.API(auth, wait_on_rate_limit=True,
    wait_on_rate_limit_notify=True)

In the code above, we’ve set the authentication credentials and created an api object. You can invoke this object’s methods to do any API call.

The API methods can be grouped into the following categories:

Methods for user timelines
Methods for tweets
Methods for users
Methods for followers
Methods for your account
Methods for likes
Methods for blocking users
Methods for searches
Methods for trends
Methods for streaming

In the following subsections, you’re going to review different groups of API methods. You can visit the API Class Documentation for a detailed list of all API methods.

Methods for User Timelines

These methods deal with reading tweets, mentions, and retweets from your timeline or any other user’s timeline, as long as it’s public.

This code snippet prints the author and text of the last tweets in your home timeline:

timeline = api.home_timeline()
for tweet in timeline:
    print(f"{tweet.user.name} said {tweet.text}")

home_timeline(), a Tweepy API method, is used to get the last 20 entries in your timeline. 20 is the default value in Tweepy. You’ll see later how to get more than 20 results and work with paginated results.

Methods for Tweets

These methods have to do with creating, fetching, and retweeting tweets.

The following code uses Tweepy to create a tweet with some text:

api.update_status("Test tweet from Tweepy Python")

We’ve used update_status() to create a new tweet from a Python string.

Methods for Users

Methods in this group enable you to search users with a filter criteria, fetch user details, and list the followers of any user, as long as that user account is public.

The snippet below fetches my user, @MikezGarcia, and then prints its details as well as its 20 most recent followers:

user = api.get_user("MikezGarcia")

print("User details:")
print(user.name)
print(user.description)
print(user.location)

print("Last 20 Followers:")
for follower in user.followers():
    print(follower.name)

get_user() returns an object containing the user details. This returned object also has methods to access information related to the user. You used the followers attribute to get the list of followers.

Methods for Followers

This group of methods deals with following and unfollowing users, querying a user’s followers, and listing the accounts any user is following.

This code shows how you can use Tweepy to start following @realpython:

api.create_friendship("realpython")

create_friendship() adds @realpython to the list of accounts that you follow.

Methods for Your Account

These methods allow you to read and write your own profile details.

For example, you can use this code snippet to update your profile description:

api.update_profile(description="I like Python")

The keyword argument description is passed to update_profile() to change your profile description to "I like Python".

Methods for Likes

Using these API methods, you can mark any tweet as Liked or remove the Like mark if it was already added.

You can mark the most recent tweet in your home timeline as Liked as follows:

tweets = api.home_timeline(count=1)
tweet = tweets[0]
print(f"Liking tweet {tweet.id} of {tweet.author.name}")
api.create_favorite(tweet.id)

This code uses home_timeline() to get the most recent tweet. Then, the tweet.id is passed to create_favorite() to mark the tweet as Liked.

Methods for Blocking Users

This group of methods deals with blocking and unblocking users, as well as listing blocked users.

Here’s how you can see the users that you have blocked:

for block in api.blocks():
    print(block.name)

This code iterates through the list of accounts that you have blocked. You can get this list using blocks().

Methods for Searches

Using these methods, you can search tweets using text, language, and other filters.

For example, you can try this code to get the 10 most recent public tweets that are in English and contain the word "Python":

for tweet in api.search(q="Python", lang="en", rpp=10):
    print(f"{tweet.user.name}:{tweet.text}")

You used search() to filter tweets that match the desired criteria. This will search in any public tweet, not just in tweets from people you are following.

Methods for Trends

This group of methods allows you to list the current trends for any geographical location.

For example, here’s how to list the world-wide trending topics:

trends_result = api.trends_place(1)
for trend in trends_result[0]["trends"]:
    print(trend["name"])

Using trends_place(), you get the list of trends for any place, passing as the only argument the place id. Here, 1 means world-wide. You can see the full list of available locations using api.trends_available().

Methods for Streaming

Streaming allows you to actively watch for tweets that match certain criteria in real time. This means that when there aren’t any new tweet matching the criteria, then the program will wait until a new tweet is created and then process it.

To use streaming you have to create two objects:

The stream object uses the Twitter API to get tweets that match some criteria. This object is the source of tweets that are then processed by a stream listener.
The stream listener receives tweets from the stream.

Here’s how you do that:

import json
import tweepy

class MyStreamListener(tweepy.StreamListener):
    def __init__(self, api):
        self.api = api
        self.me = api.me()

    def on_status(self, tweet):
        print(f"{tweet.user.name}:{tweet.text}")

    def on_error(self, status):
        print("Error detected")

tweets_listener = MyStreamListener(api)
stream = tweepy.Stream(api.auth, tweets_listener)
stream.filter(track=["Python", "Django", "Tweepy"], languages=["en"])

You declared a new class, MyStreamListener. This class is used for the stream listener tweets_listener. By extending Tweepy’s StreamLister, we reused code that is common to all stream listeners. Tweets from the stream are processed by on_status().

We created the stream using tweepy.Stream, passing the authentication credentials and our stream listener. To start getting tweets from the stream, you have to call the stream’s filter(), passing the criteria to use to filter tweets. Then, for each new tweet that matches the criteria, the stream object invokes the stream listener’s on_status().

Models

Tweepy uses its own model classes to encapsulate the responses from various Twitter API methods. This gives you a convenient way to use the results from API operations.

The model classes are:

User
Status
Friendship
SearchResults

Suppose you want to fetch every tweet in which you are mentioned, and then mark each tweet as Liked and follow its author. You can do that like this:

tweets = api.mentions_timeline()
for tweet in tweets:
    tweet.favorite()
    tweet.user.follow()

Since each tweet object returned by mentions_timeline() belongs to the Status class, you can use:

favorite() to mark it as Liked
user to get its author

This user attribute, tweet.user, is also an object that belongs to User, so you can use follow() to add the tweet’s author to the list of people you follow.

Leveraging Tweepy models enables you to create concise and understandable code.

Cursors

A lot of Twitter API endpoints use pagination to return their results. By default, each method returns the first page, which usually contains a few dozen items.

Tweepy cursors take away part of the complexity of working with paginated results. Cursors are implemented as a Tweepy class named Cursor. To use a cursor, you select the API method to use to fetch items and the number of items you want. The Cursor object takes care of fetching the various result pages transparently.

This code shows how, by using a cursor, you can get not only the first page from your timeline, but also the last 100 tweets:

for tweet in tweepy.Cursor(api.home_timeline).items(100):
    print(f"{tweet.user.name} said: {tweet.text}")

A cursor object is created using tweepy.Cursor. The class constructor receives an API method to use as the source for results. In the example, we used home_timeline() as the source since we wanted tweets from the timeline. The Cursor object has an items() method that returns an iterable you can use to iterate over the results. You can pass items() the number of result items that you want to get.

How to Make a Twitter Bot in Python With Tweepy

Now that you know how Tweepy works, let’s see how you can make a Twitter bot in Python with Tweepy. Bots work by continuously watching for some Twitter activity and automatically reacting to it.

Watching for Twitter Activity

There are two ways to continuously watch for Twitter activity:

Using streams: to be notified when new content, such as tweets, that matches certain criteria is created
Using polling: to periodically make Tweepy API calls and then check their results to see if they contain something new

Which option to choose depends on your use case. Using streams is the most efficient option, but then you can only watch activity related to tweets, so it’s less flexible. In this section, both options are used to build different bots.

Presenting the Example Bots

In this article, you’ll learn how to build three different bots that perform automatic actions to react to some Twitter events:

The Follow Followers Bot automatically follows anyone who follows you.
The Fav & Retweet Bot automatically likes and retweets tweets that match certain criteria.
The Reply to Mentions Bot automatically replies to tweets mentioning you that contain the words help or support.

We are going to use a directory named bots to store all the Python files of the project as shown below:

tweepy-bots/
│
├── bots/
│   ├── config.py
│   └── followfollowers.py
│   └── favretweet.py
│   └── autoreply.py
│
└── requirements.txt

The next section explains the config module.

The Config Module

All the bots we are building have some functionality in common. For example, they need to authenticate to the Twitter API.

You can create a reusable Python module containing the logic common to all bots. We named this module config.

This module reads the authentication credentials from environment variables and creates the Tweepy API object. By reading the credentials from environment variables, you avoid hard coding them into the source code, making it much more secure.

The bots will read the credentials from these environment variables:

CONSUMER_KEY
CONSUMER_SECRET
ACCESS_TOKEN
ACCESS_TOKEN_SECRET

Later, you’ll see how to set these variables to match the credentials you have previously generated for your Twitter app.

Below is the complete source code for the config module. It contains create_api(), a function that reads authentication credentials from environment variables and creates the Tweepy API object:

# tweepy-bots/bots/config.py
import tweepy
import logging
import os

logger = logging.getLogger()

def create_api():
    consumer_key = os.getenv("CONSUMER_KEY")
    consumer_secret = os.getenv("CONSUMER_SECRET")
    access_token = os.getenv("ACCESS_TOKEN")
    access_token_secret = os.getenv("ACCESS_TOKEN_SECRET")

    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth, wait_on_rate_limit=True, 
        wait_on_rate_limit_notify=True)
    try:
        api.verify_credentials()
    except Exception as e:
        logger.error("Error creating API", exc_info=True)
        raise e
    logger.info("API created")
    return api

This code uses os.getenv() to read environment variables and then creates the Tweepy auth object. Then the API object is created.

Passing wait_on_rate_limit and wait_on_rate_limit_notify in the creation of the tweepy.API object makes Tweepy wait and print a message when the rate limit is exceeded.

Before returning the API object, create_api() calls verify_credentials() to check that the credentials are valid.

This module and, as you will see later, the bot’s source code, use the logging Python module to inform errors and info messages that help you debug them if any issue arise. You can read more about it in Logging in Python.

The Follow Followers Bot

This bot gets your list of followers from Twitter every minute and then iterates through it to follow each user that you’re not already following.

Bot Source Code

Here’s the full source code of this bot. It uses the previously created config module, the Tweepy API, and cursors:

#!/usr/bin/env python
# tweepy-bots/bots/followfollowers.py

import tweepy
import logging
from config import create_api
import time

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

def follow_followers(api):
    logger.info("Retrieving and following followers")
    for follower in tweepy.Cursor(api.followers).items():
        if not follower.following:
            logger.info(f"Following {follower.name}")
            follower.follow()

def main():
    api = create_api()
    while True:
        follow_followers(api)
        logger.info("Waiting...")
        time.sleep(60)

if __name__ == "__main__":
    main()

main() creates a Tweepy API object using create_api() from the config module you previously created. Then, inside a loop, follow_followers() is called once every minute.

follow_followers() uses a Tweepy cursor and the Tweepy API method followers() to get your list of followers. This list contains a Tweepy user model for each user that is following you.

Then the bot iterates through the list and uses following to check if you are already following each user. Users that are not already being followed are followed using follow().

Running the Bot

To run the bot, you must first create the environment variables for the authentication credentials. You can do this by running this commands from a terminal and replacing the values with your actual credentials:

$ export CONSUMER_KEY="pGBDoAaEpkliVKBOLwjtcmHGc"
$ export CONSUMER_SECRET="xF3g1wrP50b6BlZEd20u4oVfjgH1FGQcuWUzlQO5aUWOufvlhw"
$ export ACCESS_TOKEN="622518493-6VcLIPprbQbv9wkcBBPvCle8vsjU9fE85Dq9oStl"
$ export ACCESS_TOKEN_SECRET="tH9aKQbQQ1iRdYTcLSsPwitl44BkAc6jilrsU0ifnXvZhq"

Note: This assumes that you’re using Linux or macOS. If you’re using Windows, then the steps might be a little different.

After you run the commands, your environment variables will contain the credentials needed to use the Twitter API.

Next, you have to activate your virtual environment and run the bot’s main file, bots/followfollowers.py:

$ source ./venv/bin/activate
$ python bots/followfollowers.py

While it’s running, the bot will follow anybody who follows you. You can test that it works by unfollowing someone that is following you. After a minute, they will be followed again. You can stop the bot using Ctrl-C.

The Fav & Retweet Bot

This bot uses the previously introduced Tweepy stream to actively watch for tweets that contain certain keywords. For each tweet, if you’re not the tweet author, it will mark the tweet as Liked and then retweet it.

You can use this bot to feed your account with content that is relevant to your interests.

Bot Source Code

Below, you can see the full source code of this bot. It uses a stream to filter tweets that contain the words "Python" or "Tweepy". Each tweet from the stream is marked as Liked and retweeted:

#!/usr/bin/env python
# tweepy-bots/bots/favretweet.py

import tweepy
import logging
from config import create_api
import json

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

class FavRetweetListener(tweepy.StreamListener):
    def __init__(self, api):
        self.me = api.me()

    def on_status(self, tweet):
        logger.info(f"Processing tweet id {tweet.id}")
        if tweet.in_reply_to_status_id is not None or \
            tweet.user.id == self.me.id:
            return
        try:
            tweet.favorite()
            tweet.retweet()
        except Exception as e:
            logger.error("Error on fav and retweet", exc_info=True)

    def on_error(self, status):
        logger.error(status)

def main(keywords):
    api = create_api()
    tweets_listener = FavRetweetListener(api)
    stream = tweepy.Stream(api.auth, tweets_listener)
    stream.filter(track=keywords, languages=["en"])

if __name__ == "__main__":
    main(["Python", "Tweepy"])

As with the previous bot, the main function uses create_api() from the config module to create a Tweepy API object.

A Tweepy stream is created to filter tweets that are in the English language and include some of the keywords specified in the main function argument, "Python" or "Tweepy" in this case.

The on_status() of FavRetweetListener processes tweets from the stream. This method receives a status object and uses favorite() and retweet() to mark the tweet as Liked and retweet.

To avoid retweeting and liking tweets that are replies to other tweets, on_status() uses an if to see if in_reply_to_status_id is not None. Also, the code checks if you’re the tweet author to avoid retweeting and liking your own content.

You can run this bot using the same instructions used for the previous bot, changing the Python program to favretweet.py.

The Reply to Mentions Bot

This bot periodically fetches tweets in which you are mentioned. If the tweet is not a reply to another tweet, and it contains the words "help" or "support", then the tweet author will be followed, and the tweet will be replied to with another tweet saying "Please reach us via DM".

You can use this bot to automate the initial process of answering your followers’ questions.

Bot Source Code

Here’s the full source code for this bot. It uses a cursor to fetch your mentions:

#!/usr/bin/env python
# tweepy-bots/bots/autoreply.py

import tweepy
import logging
from config import create_api
import time

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

def check_mentions(api, keywords, since_id):
    logger.info("Retrieving mentions")
    new_since_id = since_id
    for tweet in tweepy.Cursor(api.mentions_timeline,
        since_id=since_id).items():
        new_since_id = max(tweet.id, new_since_id)
        if tweet.in_reply_to_status_id is not None:
            continue
        if any(keyword in tweet.text.lower() for keyword in keywords):
            logger.info(f"Answering to {tweet.user.name}")

            if not tweet.user.following:
                tweet.user.follow()

            api.update_status(
                status="Please reach us via DM",
                in_reply_to_status_id=tweet.id,
            )
    return new_since_id

def main():
    api = create_api()
    since_id = 1
    while True:
        print(since_id)
        since_id = check_mentions(api, ["help", "support"], since_id)
        logger.info("Waiting...")
        time.sleep(60)

if __name__ == "__main__":
    main()

As with the others bots, the main function creates a Tweepy API object. After that, it initializes the variable since_id with the value 1. This variable is used to only fetch mentions created after those already processed.

Inside a loop, check_mentions() is called once every minute.

This function uses a Tweepy cursor and mentions_timeline() to get all the tweets mentioning you that have an id greater than the since_id variable.

For each tweet mentioning you, its author is followed using tweet.user.follow() if you are not already following them.

Then a reply to the tweet is created using update_status(), passing the id of the original tweet as in_reply_to_status_id.

check_mentions() returns the greatest processed tweet id. This information will be used as the since_id in the next call to look only for tweets more recent than the ones already fetched.

You can run the bot using the same instructions used previously. Just change the Python program to autoreply.py.

Deploying Bots to a Server Using Docker

Being able to run your Twitter bots in your local machine is fine for development, but for production you’ll probably need to run your bots in a server. This section details how you can package your bot and its dependencies using Docker and then deploy it to a server.

Note: This is just one of the ways you can deploy a Twitter bot to a server.

Docker provides a way to package any app in a Docker image containing not only the app itself, but all its dependencies too. You can learn more about Docker and its setup in Docker in Action – Fitter, Happier, More Productive

Building the Docker Image

To package your bot or application, you have to create a Dockerfile in the project’s root directory. This file contains a set of instructions used to create the Docker image containing your app.

This is how your project structure should look:

tweepy-bots/
│
├── bots/
│   ├── config.py
│   └── followfollowers.py
│   └── favretweet.py
│   └── autoreply.py
│
├── requirements.txt
└── Dockerfile

As you can see, the Dockerfile and requirements.txt live inside your project top directory.

You can use the following Dockerfile to create the image for the Fav & Retweet bot. It uses Python:3.7-alpine as the base image. We chose this image as the base since it’s very small and contains Python3.7 and pip. This Dockerfile then copies the bot code and the requirements.txt file to the image. Finally, the dependencies are installed inside the image using pip3:

FROM python:3.7-alpine

COPY bots/config.py /bots/
COPY bots/favretweet.py /bots/
COPY requirements.txt /tmp
RUN pip3 install -r /tmp/requirements.txt

WORKDIR /bots
CMD ["python3", "favretweet.py"]

The final line says what command must be run when this image is used.

Now you can build the image using this command:

$ docker build . -t fav-retweet-bot

Then, using the docker images command, you can see the details of the newly generated image.

Finally, you can use the docker run command to test the image, passing it the environment variables holding the authentication credentials:

$ docker run -it -e CONSUMER_KEY="uDRNy31oWfoiKV9AvPoNavy0I" \
 -e CONSUMER_SECRET="lnAL5VAgZLWNspQVpd3X6tEo47PRCmsPEwuxpvLCLSR08DMa4O" \
 -e ACCESS_TOKEN="622518593-j7gWSqzQO31ju7Bf7idB47NlZeSENsuADGU9B69I" \
 -e ACCESS_TOKEN_SECRET="iutFsxvP5IglRckJ1I1why6017xMNkzxqBID48Azw0GvT" \
 fav-retweet-bot

While it’s running, you’ll see the bot’s logs output on the screen.

This same procedure can be used to create the Docker image for other bots.

Example: Deploying a Twitter Bot to Amazon AWS

Now that the Docker image is ready, there are a lot of ways to deploy it to run in any server. In this section, you’ll see one method you can use to deploy a bot to a server in Amazon AWS. This setup uses an Amazon AWS EC2 instance, like a virtual server, to run the Docker image.

Note: You need an Amazon AWS account to follow these steps. All the resources used in this article are eligible for the AWS free tier, so if you are new to AWS, then you won’t be charged.

Step 1: Export The Docker Image

From the computer where you have created the Docker image, run these commands to export it to a file and compress it. You’re going to use this file later to upload the image to your EC2 instance:

$ docker image save fav-retweet-bot:latest -o fav-retweet-bot.tar
$ gzip fav-retweet-bot.tar

Now the image has been compressed into fav-retweet-bot.tar.gz.

Step 2: Create an Amazon EC2 Instance

The first step in AWS is to create a new EC2 instance. The bot will run inside this instance. Log in to your AWS console, choose the EC2 service, and select Instances. You’ll see something like this:

Click the Launch instance button and select Ubuntu Server 18.04 as the base image:

Now you can choose the instance type. This depends on the computing resources needed by your bot. For the bots described in this article, the t2.micro instance type is a good starting point. Select it and click Review and Launch:

On the next page, you can review and check your configuration. Then click the Launch button to begin the creation of your EC2 instance:

After you’ve clicked Launch, you’ll be asked to create a key pair. It’s used to allow you to connect to your instance and deploy the bot Docker image. Select Create a new key pair, enter a key pair name, and download the key pair.

Remember where you’re downloading the key pair. You’ll need it later.

Then click the Launch instances button to proceed:

You will see a page saying that your instance is being created. Click the View instances button to see the status of your instance:

At first, your instance state will be pending. You have to wait until it transitions to running:

Step 3: Install Docker in Your EC2 Instance

Now you have to connect to your instance using SSH to install Docker on it.

If, in the previous screen, you right click on your instance, you’ll find a context menu. Select the Connect option to see how to connect using SSH:

This dialog shows how to connect to your instance:

Note: ec2-3-86-66-73.compute-1.amazonaws.com is the address that AWS assigned to the instance used in this article, but it will probably be different for you.

First, change the permissions of your key pair file using the following command. Otherwise, you won’t be able to connect to your EC2 instance using this key pair:

$ chmod 400 mykeypair.pem

Then connect to your instance using SSH. You can copy the command you need to execute from the Connect To Your Instance dialog:

$ ssh -i "mykeypair.pem" ubuntu@ec2-3-86-66-73.compute-1.amazonaws.com

Once you’re connected, run the following commands to install Docker in the instance:

ubuntu@ip-172-31-44-227:~$ sudo apt-get update
ubuntu@ip-172-31-44-227:~$ sudo apt install docker.io
ubuntu@ip-172-31-44-227:~$ sudo adduser ubuntu docker
ubuntu@ip-172-31-44-227:~$ exit

It’s important that you exit the SSH session after the installation, so that in the next login your user permissions will be reloaded.

Step 4: Upload Your Bot’s Docker Image

From your local computer, upload the bot Docker image to your instance using scp. This can take some time depending on your internet connection:

$ scp -i "mykeypair.pem" fav-retweet-bot.tar.gz \
    ubuntu@ec2-3-86-66-73.compute-1.amazonaws.com:/tmp

When the image upload finishes, log in again to your instance using SSH:

$ ssh -i "mykeypair.pem" ubuntu@ec2-3-86-66-73.compute-1.amazonaws.com

Run these commands in your instance to decompress and import the Docker image:

ubuntu@ip-172-31-44-227:~$ gunzip /tmp/fav-retweet-bot.tar.gz 
ubuntu@ip-172-31-44-227:~$ docker image load -i /tmp/fav-retweet-bot.tar

Step 5: Run Your Bot’s Docker Image

The final step to deploy the bot is to run the Docker image in your EC2 instance, passing it the authentication credentials.

You can do this using the docker command. Passing the flags -d and --restart-always assures that the bot will keep running if you disconnect from the SSH session or if the instance is restarted:

ubuntu@ip-172-31-44-227:~$ docker run -d --restart always \
-e CONSUMER_KEY="uDRNy31oWfoiKV9AvPoNavy0I" \
-e CONSUMER_SECRET="lnAL5VAgZLWNspQVpd3X6tEo47PRCmsPEwuxpvLCLSR08DMa4O" \
-e ACCESS_TOKEN="622518593-j7gWSqzQO31ju7Bf7idB47NlZeSENsuADGU9B69I" \
-e ACCESS_TOKEN_SECRET="iutFsxvP5IglRckJ1I1why6017xMNkzxqBID48Azw0GvT" \
fav-retweet-bot

Using docker ps, you can check that the bot is running and find its container id. Finally, using the docker logs command and the container id, you can check the output from the bot to see if it works properly:

$ ubuntu@ip-172-31-44-227:~$ docker logs e6aefe73a885
INFO:root:API created
INFO:root:Processing tweet id 1118276911262785538
INFO:root:Processing tweet id 1118276942162214918
INFO:root:Processing tweet id 1118276990853951488
INFO:root:Processing tweet id 1118277032360722433
INFO:root:Processing tweet id 1118277034466324480

Now, even if you disconnect from the SSH session or turn off your computer, the bot will continue to run on your AWS EC2 instance.

Free AWS Resources

To save resources and money (if your AWS free tier period has ended), you can stop or terminate your EC2 instance:

Stopping it will allow you to restart it in the future, but it will still consume some resources related to your instance storage. By contrast, if you Terminate an EC2 instance, then all its resources will be released, but you won’t be able to restart it.

There’s a lot more to using AWS, like creating an AWS image of your instance, but that goes beyond the scope of this article.

Wrapping Up

Building your own Twitter bots allows you to take your Twitter presence to the next level. By using bots, you can automate content creation and other Twitter activities. This can save you a lot of time and give your audience a better experience.

The Tweepy package hides many low level details of the Twitter API, allowing you to focus on the logic of your Twitter bots.

In this article, you learned how to:

Improve and automate your interactions with your Twitter audience
Install Tweepy
Sign up as a Twitter developer to use its API
Use Tweepy to invoke the Twitter API
Build Twitter bots
Deploy the bots to a server using Docker and AWS

You can use the bots in this article as a starting point to automate part of your Twitter activity. Don’t forget to take a look at the whole Tweepy API documentation and use your imagination to make more complex bots that are meaningful to your use case.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Implement a Python Stack

2019-06-05T14:00:00+00:00

Have you heard of stacks and wondered what they are? Do you have the general idea but are wondering how to implement a Python stack? You’ve come to the right place!

In this tutorial, you’ll learn:

How to recognize when a stack is a good choice for data structures
How to decide which implementation is best for your program
What extra considerations to make about stacks in a threading or multiprocessing environment

This tutorial is for Pythonistas who are comfortable running scripts, know what a list is and how to use it, and are wondering how to implement Python stacks.

Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

What Is a Stack?

A stack is a data structure that stores items in an Last-In/First-Out manner. This is frequently referred to as LIFO. This is in contrast to a queue, which stores items in a First-In/First-Out (FIFO) manner.

It’s probably easiest to understand a stack if you think of a use case you’re likely familiar with: the Undo feature in your editor.

Let’s imagine you’re editing a Python file so we can look at some of the operations you perform. First, you add a new function. This adds a new item to the undo stack:

You can see that the stack now has an Add Function operation on it. After adding the function, you delete a word from a comment. This also gets added to the undo stack:

Notice how the Delete Word item is placed on top of the stack. Finally you indent a comment so that it’s lined up properly:

You can see that each of these commands are stored in an undo stack, with each new command being put at the top. When you’re working with stacks, adding new items like this is called push.

Now you’ve decided to undo all three of those changes, so you hit the undo command. It takes the item at the top of the stack, which was indenting the comment, and removes that from the stack:

Your editor undoes the indent, and the undo stack now contains two items. This operation is the opposite of push and is commonly called pop.

When you hit undo again, the next item is popped off the stack:

This removes the Delete Word item, leaving only one operation on the stack.

Finally, if you hit Undo a third time, then the last item will be popped off the stack:

The undo stack is now empty. Hitting Undo again after this will have no effect because your undo stack is empty, at least in most editors. You’ll see what happens when you call .pop() on an empty stack in the implementation descriptions below.

Implementing a Python Stack

There are a couple of options when you’re implementing a Python stack. This article won’t cover all of them, just the basic ones that will meet almost all of your needs. You’ll focus on using data structures that are part of the Python library, rather than writing your own or using third-party packages.

You’ll look at the following Python stack implementations:

list
collections.deque
queue.LifoQueue

Using `list` to Create a Python Stack

The built-in list structure that you likely use frequently in your programs can be used as a stack. Instead of .push(), you can use .append() to add new elements to the top of your stack, while .pop() removes the elements in the LIFO order:

>>>

>>> myStack = []

>>> myStack.append('a')
>>> myStack.append('b')
>>> myStack.append('c')

>>> myStack
['a', 'b', 'c']

>>> myStack.pop()
'c'
>>> myStack.pop()
'b'
>>> myStack.pop()
'a'

>>> myStack.pop()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
IndexError: pop from empty list

You can see in the final command that a list will raise an IndexError if you call .pop() on an empty stack.

list has the advantage of being familiar. You know how it works and likely have used it in your programs already.

Unfortunately, list has a few shortcomings compared to other data structures you’ll look at. The biggest issue is that it can run into speed issues as it grows. The items in a list are stored with the goal of providing fast access to random elements in the list. At a high level, this means that the items are stored next to each other in memory.

If your stack grows bigger than the block of memory that currently holds it, then Python needs to do some memory allocations. This can lead to some .append() calls taking much longer than other ones.

There is a less serious problem as well. If you use .insert() to add an element to your stack at a position other than the end, it can take much longer. This is not normally something you would do to a stack, however.

The next data structure will help you get around the reallocation problem you saw with list.

Using `collections.deque` to Create a Python Stack

The collections module contains deque, which is useful for creating Python stacks. deque is pronounced “deck” and stands for “double-ended queue.”

You can use the same methods on deque as you saw above for list, .append(), and .pop():

>>>

>>> from collections import deque
>>> myStack = deque()

>>> myStack.append('a')
>>> myStack.append('b')
>>> myStack.append('c')

>>> myStack
deque(['a', 'b', 'c'])

>>> myStack.pop()
'c'
>>> myStack.pop()
'b'
>>> myStack.pop()
'a'

>>> myStack.pop()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
IndexError: pop from an empty deque

This looks almost identical to the list example above. At this point, you might be wondering why the Python core developers would create two data structures that look the same.

Why Have `deque` and `list`?

As you saw in the discussion about list above, it was built upon blocks of contiguous memory, meaning that the items in the list are stored right next to each other:

This works great for several operations, like indexing into the list. Getting myList[3] is fast, as Python knows exactly where to look in memory to find it. This memory layout also allows slices to work well on lists.

The contiguous memory layout is the reason that list might need to take more time to .append() some objects than others. If the block of contiguous memory is full, then it will need to get another block, which can take much longer than a normal .append():

deque, on the other hand, is built upon a doubly linked list. In a linked list structure, each entry is stored in its own memory block and has a reference to the next entry in the list.

A doubly linked list is just the same, except that each entry has references to both the previous and the next entry in the list. This allows you to easily add nodes to either end of the list.

Adding a new entry into a linked list structure only requires setting the new entry’s reference to point to the current top of the stack and then pointing the top of the stack to the new entry:

This constant-time addition and removal of entries onto a stack comes with a trade-off, however. Getting myDeque[3] is slower than it was for a list, because Python needs to walk through each node of the list to get to the third element.

Fortunately, you rarely want to do random indexing or slicing on a stack. Most operations on a stack are either push or pop.

The constant time .append() and .pop() operations make deque an excellent choice for implementing a Python stack if your code doesn’t use threading.

Python Stacks and Threading

Python stacks can be useful in multi-threaded programs as well, but if you’re not interested in threading, then you can safely skip this section and jump to the summary.

The two options you’ve seen so far, list and deque, behave differently if your program has threads.

To start with the simpler one, you should never use list for any data structure that can be accessed by multiple threads. list is not thread-safe.

Note: If you need a refresher on thread safety and race conditions, check out An Intro to Threading in Python.

deque is a little more complex, however. If you read the documentation for deque, it clearly states that both the .append() and .pop() operations are atomic, meaning that they won’t be interrupted by a different thread.

So if you restrict yourself to using only .append() and .pop(), then you will be thread safe.

The concern with using deque in a threaded environment is that there are other methods in that class, and those are not specifically designed to be atomic, nor are they thread safe.

So, while it’s possible to build a thread-safe Python stack using a deque, doing so exposes yourself to someone misusing it in the future and causing race conditions.

Okay, if you’re threading, you can’t use list for a stack and you probably don’t want to use deque for a stack, so how can you build a Python stack for a threaded program?

The answer is in the queue module, queue.LifoQueue. Remember how you learned that stacks operate on the Last-In/First-Out principle? Well, that’s what the “Lifo” portion of LifoQueue stands for.

While the interface for list and deque were similar, LifoQueue uses .put() and .get() to add and remove data from the stack:

>>>

>>> from queue import LifoQueue
>>> myStack = LifoQueue()

>>> myStack.put('a')
>>> myStack.put('b')
>>> myStack.put('c')

>>> myStack
<queue.LifoQueue object at 0x7f408885e2b0>

>>> myStack.get()
'c'
>>> myStack.get()
'b'
>>> myStack.get()
'a'

>>> # myStack.get() <--- waits forever
>>> myStack.get_nowait()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python3.7/queue.py", line 198, in get_nowait
    return self.get(block=False)
  File "/usr/lib/python3.7/queue.py", line 167, in get
    raise Empty
_queue.Empty

Unlike deque, LifoQueue is designed to be fully thread-safe. All of its methods are safe to use in a threaded environment. It also adds optional time-outs to its operations which can frequently be a must-have feature in threaded programs.

This full thread safety comes at a cost, however. To achieve this thread-safety, LifoQueue has to do a little extra work on each operation, meaning that it will take a little longer.

Frequently, this slight slow down will not matter to your overall program speed, but if you’ve measured your performance and discovered that your stack operations are the bottleneck, then carefully switching to a deque might be worth doing.

I’d like to stress again that switching from LifoQueue to deque because it’s faster without having measurements showing that your stack operations are a bottleneck is an example of premature optimization. Don’t do that.

Python Stacks: Which Implementation Should You Use?

In general, you should use a deque if you’re not using threading. If you are using threading, then you should use a LifoQueue unless you’ve measured your performance and found that a small boost in speed for pushing and popping will make enough difference to warrant the maintenance risks.

list may be familiar, but it should be avoided because it can potentially have memory reallocation issues. The interfaces for deque and list are identical, and deque doesn’t have these issues, which makes deque the best choice for your non-threaded Python stack.

Conclusion

You now know what a stack is and have seen situations where they can be used in real-life programs. You’ve evaluated three different options for implementing stacks and seen that deque is a great choice for non-threaded programs. If you’re implementing a stack in a threading environment, then it’s likely a good idea to use a LifoQueue.

You are now able to:

Recognize when a stack would be a good data structure
Select which implementation is right for your problem

If you still have questions, feel free to reach out in the comments sections below. Now, go write some code since you gained another tool to help you solve your programming problems!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Introduction to Git and GitHub for Python Developers

2019-06-04T14:00:00+00:00

What is Git, what is GitHub, and what’s the difference? Learn the basics of Git and GitHub from the perspective of a Pythonista in this step-by-step video course.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Object-Oriented Programming in Python vs Java

2019-06-03T14:00:00+00:00

Java programmers making a move to Python often struggle with Python’s approach to object-oriented programming (OOP). The approach to working with objects, variable types, and other language capabilities taken by Python vs Java are quite different. It can make switching between both languages very confusing.

This article compares and contrasts object-oriented programming support in Python vs Java. By the end, you’ll be able to apply your knowledge of object-oriented programming to Python, understand how to reinterpret your understanding of Java objects to Python, and use objects in a Pythonic way.

Over the course of this article, you’ll:

Build a basic class in both Java and Python
Explore how object attributes work in Python vs Java
Compare and contrast Java methods and Python functions
Discover inheritance and polymorphism mechanisms in both languages
Investigate reflection across Python vs Java
Apply everything in a complete class implementation in both languages

This article isn’t a primer on object-oriented programming. Rather, it compares object-oriented features and principles of Python vs Java. Readers should have good knowledge of Java, and also be familiar with coding Python. If you are unfamiliar with object-oriented programming, then check out Intro to Object-Oriented Programming (OOP) in Python. All Python examples will work with Python 3.6 or later.

Download Sample Code: Click here to download the commented sample object definitions and source code for the Java and Python objects in this article.

Sample Classes in Python vs Java

To begin, you’ll implement the same small class in both Python and Java to illustrate the differences between them. You’ll make modifications to them as the article progresses.

First, assume you have the following Car class definition in Java:

 1 public class Car {
 2     private String color;
 3     private String model;
 4     private int year;
 5 
 6     public Car(String color, String model, int year) {
 7         this.color = color;
 8         this.model = model;
 9         this.year = year;
10     }
11 
12     public String getColor() {
13         return color;
14     }
15 
16     public String getModel() {
17         return model;
18     }
19 
20     public int getYear() {
21         return year;
22     }
23 }

Java classes are defined in files with the same name as the class. So, you have to save this class in a file named Car.java. Only one class can be defined in each file.

A similar small Car class is written in Python as follows:

 1 class Car:
 2     def __init__(self, color, model, year):
 3         self.color = color
 4         self.model = model
 5         self.year = year

In Python you can declare a class anywhere, in any file, at any time. Save this class in the file car.py.

Using these classes as your base, you can explore the basic components of classes and objects.

Object Attributes

All object-oriented languages have some way to store data about the object. In Java and Python, data is stored in attributes, which are variables associated with specific objects.

One of the most significant differences between Python vs Java is how they define and manage class and object attributes. Some of these differences come from constraints imposed by the languages, while others come from best practices.

Declaration and Initialization

In Java, you declare attributes in the class body, outside of any methods, with a definite type. You must define class attributes before they are used:

 1 public class Car {
 2     private String color;
 3     private String model;
 4     private int year;

In Python, you both declare and define attributes inside the class __init__(), which is the equivalent of Java’s constructor:

 1 def __init__(self, color, model, year):
 2     self.color = color
 3     self.model = model
 4     self.year = year

By prefixing the variable names with self, you tell Python these are attributes. Each instance of the class will have a copy. All variables in Python are loosely typed, and these attributes are no exception.

You can also create instance variables outside of .__init__(), but it’s not a best practice as their scope is often confusing. If not used properly, instance variables created outside of .__init__() can lead to subtle bugs that are hard to find. For example, you can add a new attribute .wheels to a Car object like this:

>>>

 1 >>> import car
 2 >>> my_car = car.Car("yellow", "beetle", 1967)
 3 >>> print(f"My car is {my_car.color}")
 4 My car is yellow
 5 
 6 >>> my_car.wheels = 5
 7 >>> print(f"Wheels: {my_car.wheels}")
 8 Wheels: 5

However, if you forget the my_car.wheels = 5 on line 6, then Python displays an error:

>>>

 1 >>> import car
 2 >>> my_car = car.Car("yellow", "beetle", 1967)
 3 >>> print(f"My car is {my_car.color}")
 4 My car is yellow
 5 
 6 >>> print(f"Wheels: {my_car.wheels}")
 7 Traceback (most recent call last):
 8   File "<stdin>", line 1, in <module>
 9 AttributeError: 'Car' object has no attribute 'wheels'

In Python, when you declare a variable outside of a method, it’s treated as a class variable. Update the Car class as follows:

 1 class Car:
 2 
 3     wheels = 0
 4 
 5     def __init__(self, color, model, year):
 6         self.color = color
 7         self.model = model
 8         self.year = year

This changes how you use the variable wheels. Instead of referring to it using an object, you refer to it using the class name:

>>>

 1 >>> import car
 2 >>> my_car = car.Car("yellow", "beetle", 1967)
 3 >>> print(f"My car is {my_car.color}")
 4 My car is yellow
 5 
 6 >>> print(f"It has {car.Car.wheels} wheels")
 7 It has 0 wheels
 8 
 9 >>> print(f"It has {my_car.wheels} wheels")
10 It has 0 wheels

Note: In Python, you refer to a class variable using the following syntax:

The name of the file containing the class, without the .py extension
A dot
The name of the class
A dot
The name of the variable

Since you saved the Car class in the file car.py, you refer to the class variable wheels on line 6 as car.Car.wheels.

You can refer to my_car.wheels or car.Car.wheels, but be careful. Changing the value of the instance variable my_car.wheels will not change the value of the class variable car.Car.wheels:

>>>

 1 >>> from car import *
 2 >>> my_car = car.Car("yellow", "Beetle", "1966")
 3 >>> my_other_car = car.Car("red", "corvette", "1999")
 4 
 5 >>> print(f"My car is {my_car.color}")
 6 My car is yellow
 7 >>> print(f"It has {my_car.wheels} wheels")
 8 It has 0 wheels
 9 
10 >>> print(f"My other car is {my_other_car.color}")
11 My other car is red
12 >>> print(f"It has {my_other_car.wheels} wheels")
13 It has 0 wheels
14 
15 >>> # Change the class variable value
16 ... car.Car.wheels = 4
17 
18 >>> print(f"My car has {my_car.wheels} wheels")
19 My car has 4 wheels
20 >>> print(f"My other car has {my_other_car.wheels} wheels")
21 My other car has 4 wheels
22 
23 >>> # Change the instance variable value for my_car
24 ... my_car.wheels = 5
25 
26 >>> print(f"My car has {my_car.wheels} wheels")
27 My car has 5 wheels
28 >>> print(f"My other car has {my_other_car.wheels} wheels")
29 My other car has 4 wheels

You define two Car objects on lines 2 and 3:

my_car
my_other_car

At first, both of them have zero wheels. When you set the class variable using car.Car.wheels = 4 on line 16, both objects now have four wheels. However, when you set the instance variable using my_car.wheels = 5 on line 24, only that object is affected.

This means that there are now two different copies of the wheels attribute:

A class variable that applies to all Car objects
A specific instance variable applicable to the my_car object only

It isn’t difficult to accidentally refer to the wrong one and introduce subtle bugs.

Java’s equivalent to a class attribute is a static attribute:

public class Car {
    private String color;
    private String model;
    private int year;
    private static int wheels;

    public Car(String color, String model, int year) {
        this.color = color;
        this.model = model;
        this.year = year;
    }

    public static int getWheels() {
        return wheels;
    }

    public static void setWheels(int count) {
        wheels = count;
    }
}

Normally, you refer to static variables using the Java class name. You can refer to a static variable through a class instance like Python, but it’s not a best practice.

Your Java class is getting long. One of the reasons why Java is more verbose than Python is the notion of public and private methods and attributes.

Public and Private

Java controls access to methods and attributes by differentiating between public data and private data.

In Java, it is expected that attributes are declared as private, or protected if subclasses need direct access to them. This limits access to these attributes from code outside the class. To provide access to private attributes, you declare public methods which set and retrieve data in a controlled manner (more on that later).

Recall from your Java class above that the color variable was declared as private. Therefore, this Java code will show a compilation error at the highlighted line:

Car myCar = new Car("blue", "Ford", 1972);

// Paint the car
myCar.color = "red";

If you don’t specify an access level, then the attribute defaults to package protected, which limits access to classes in the same package. You have to mark the attribute as public if you want this code to work.

However, declaring public attributes is not considered a best practice in Java. You’re expected to declare attributes as private, and use public access methods, such as the .getColor() and .getModel() shown in the code.

Python doesn’t have the same notion of private or protected data that Java does. Everything in Python is public. This code works with your existing Python class just fine:

>>>

>>> my_car = car.Car("blue", "Ford", 1972)

>>> # Paint the car
... my_car.color = "red"

Instead of private, Python has a notion of a non-public instance variable. Any variable which starts with an underscore character is defined to be non-public. This naming convention makes it harder to access a variable, but it’s only a naming convention, and you can still access the variable directly.

Add the following line to your Python Car class:

class Car:

    wheels = 0

    def __init__(self, color, model, year):
        self.color = color
        self.model = model
        self.year = year
        self._cupholders = 6

You can access the ._cupholders variable directly:

>>>

>>> import car
>>> my_car = car.Car("yellow", "Beetle", "1969")
>>> print(f"It was built in {my_car.year}")
It was built in 1969
>>> my_car.year = 1966
>>> print(f"It was built in {my_car.year}")
It was built in 1966
>>> print(f"It has {my_car._cupholders} cupholders.")
It has 6 cupholders.

Python lets you access ._cupholders, but IDEs such as VS Code may issue a warning through linters that support PEP 8. For more on PEP 8, read How to Write Beautiful Python Code With PEP 8.

Here’s the code in VS Code, with a warning highlighted and displayed:

Python further recognizes using double underscore characters in front of a variable to conceal an attribute in Python. When Python sees a double underscore variable, it changes the variable name internally to make it difficult to access directly. This mechanism avoids accidents but still doesn’t make data impossible to access.

To show this mechanism in action, change the Python Car class again:

class Car:

    wheels = 0

    def __init__(self, color, model, year):
        self.color = color
        self.model = model
        self.year = year
        self.__cupholders = 6

Now, when you try to access the .__cupholders variable, you see the following error:

>>>

>>> import car
>>> my_car = car.Car("yellow", "Beetle", "1969")
>>> print(f"It was built in {my_car.year}")
It was built in 1969
>>> my_car.year = 1966
>>> print(f"It was built in {my_car.year}")
It was built in 1966
>>> print(f"It has {my_car.__cupholders} cupholders.")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Car' object has no attribute '__cupholders'

So why doesn’t the .__cupholders attribute exist?

When Python sees an attribute with double underscores, it changes the attribute by prefixing the original name of the attribute with an underscore, followed by the class name. To use the attribute directly, you need to change the name you use as well:

>>>

>>> print(f"It has {my_car._Car__cupholders} cupholders")
It has 6 cupholders

When you use double underscores to conceal an attribute from the user, Python changes the name in a well-documented manner. This means that a determined developer can still access the attribute directly.

So if your Java attributes are declared private, and your Python attributes are prefaced with double underscores, then how do you provide and control access to the data they store?

Access Control

In Java, you access private attributes using setters and getters. To allow users to paint their cars, add the following code to your Java class:

public String getColor() {
    return color;
}

public void setColor(String color) {
    this.color = color;
}

Since .getColor() and .setColor() are public, anyone can call them to change or retrieve the car’s color. Java’s best practices of using private attributes accessed with public getters and setters is one of the reasons why Java code tends to be more verbose than Python.

As you saw above, you access attributes directly in Python. Since everything is public, you can access anything at any time from anywhere. You set and get attribute values directly by referring to their names. You can even delete attributes in Python, which isn’t possible in Java:

>>>

>>> my_car = Car("yellow", "beetle", 1969)
>>> print(f"My car was built in {my_car.year}")
My car was built in 1969
>>> my_car.year = 1966
>>> print(f"It was built in {my_car.year}")
It was built in 1966
>>> del my_car.year
>>> print(f"It was built in {my_car.year}")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Car' object has no attribute 'year'

However, there are times you may want to control access to an attribute. In that case, you can use Python properties.

In Python, properties provide controllable access to class attributes using Python decorator syntax. (You can learn about decorators in the video course Python Decorators 101.) Properties allow functions to be declared in Python classes that are analogous to Java getter and setter methods, with the added bonus of allowing you to delete attributes as well.

You can see how properties work by adding one to your Car class:

 1 class Car:
 2     def __init__(self, color, model, year):
 3         self.color = color
 4         self.model = model
 5         self.year = year
 6         self._voltage = 12
 7 
 8     @property
 9     def voltage(self):
10         return self._voltage
11 
12     @voltage.setter
13     def voltage(self, volts):
14         print("Warning: this can cause problems!")
15         self._voltage = volts
16 
17     @voltage.deleter
18     def voltage(self):
19         print("Warning: the radio will stop working!")
20         del self._voltage

Here, you expand the notion of the Car to include electric vehicles. You declare the ._voltage attribute to hold the battery voltage on line 6.

To provide controlled access, you define a function called voltage() to return the private value on lines 9 and 10. By using the @property decoration, you mark it as a getter that anyone can access directly.

Similarly, you define a setter function on lines 13 to 15, also called voltage(). However, you decorate this function with @voltage.setter. Lastly, you use @voltage.deleter to decorate a third voltage() on lines 18 to 20, which allows controlled deletion of the attribute.

The names of the decorated functions are all the same, indicating they control access to the same attribute. The function names also become the name of the attribute you use to access the value. Here’s how these properties work in practice:

>>>

 1 >>> from car import *
 2 >>> my_car = Car("yellow", "beetle", 1969)
 3 
 4 >>> print(f"My car uses {my_car.voltage} volts")
 5 My car uses 12 volts
 6 
 7 >>> my_car.voltage = 6
 8 Warning: this can cause problems!
 9 
10 >>> print(f"My car now uses {my_car.voltage} volts")
11 My car now uses 6 volts
12 
13 >>> del my_car.voltage
14 Warning: the radio will stop working!

Note that you use .voltage in the highlighted lines above, not ._voltage. This tells Python to use the property functions you defined:

When you print the value of my_car.voltage on line 4, Python calls .voltage() decorated with @property.
When you assign a value to my_car.voltage on line 7, Python calls .voltage() decorated with @voltage.setter.
When you delete my_car.voltage on line 13, Python calls .voltage() decorated with @voltage.deleter.

The @property, @.setter, and @.deleter decorations make it possible to control access to attributes without requiring users to use different methods. You can even make attributes appear to be read-only properties by omitting the @.setter and @.deleter decorated functions.

`self` and `this`

In Java, a class refers to itself with the this reference:

public void setColor(String color) {
    this.color = color;
}

this is implicit in Java code: it doesn’t normally need to be written, unless there may be confusion between two variables with the same name.

You can write the same setter this way:

public void setColor(String newColor) {
    color = newColor;
}

Since Car has an attribute named .color, and there isn’t another variable in scope with the same name, a reference to that name works. We used this in the first example to differentiate between the attribute and parameter both named color.

In Python, the keyword self serves a similar purpose. It’s how you refer to member variables, but unlike Java’s this, it’s required if you want to create or refer to a member attribute:

class Car:
    def __init__(self, color, model, year):
        self.color = color
        self.model = model
        self.year = year
        self._voltage = 12

    @property
    def voltage(self):
        return self._voltage

Python requires each self in the code above. Each one either creates or refers to the attributes. If you omit them, then Python will create a local variable instead of an attribute.

The difference between how you use self and this in Python and Java is due to underlying differences between the two languages and how they name variables and attributes.

Methods and Functions

This difference between Python vs Java is, simply put, that Python has functions, and Java doesn’t.

In Python, the following code is perfectly fine (and very common):

>>>

>>> def say_hi():
...     print("Hi!")
... 
>>> say_hi()
Hi!

You can call say_hi() from anywhere it’s visible. This function has no reference to self, which indicates that it’s a global function, not a class function. It can’t alter or store any data in any classes but can use local and global variables.

In contrast, every line of Java code you write belongs to a class. Functions can’t exist outside of a class, and by definition, all Java functions are methods. In Java, the closest you can get to a pure function is by using a static method:

public class Utils {
    static void SayHi() {
        System.out.println("Hi!");
    }
}

Utils.SayHi() is callable from anywhere without first creating an instance of Utils. Since you can call SayHi() without creating an object first, the this reference doesn’t exist. However, this is still not a function in the sense that say_hi() is in Python.

Inheritance and Polymorphism

Inheritance and polymorphism are two fundamental concepts in object-oriented programming.

Inheritance allows objects to derive attributes and functionality from other objects, creating a hierarchy moving from more general objects to more specific. For example, a Car and a Boat are both specific types of Vehicles. Objects can inherit their behavior from a single parent object or multiple parent objects, and are referred to as child objects when they do.

Polymorphism allows two or more objects to behave like one another, which allows them to be used interchangeably. For example, if a method or function knows how to paint a Vehicle object, then it can also paint a Car or Boat object, since they inherit their data and behavior from the Vehicle.

These fundamental OOP concepts are implemented quite differently in Python vs Java.

Inheritance

Python supports multiple inheritance, or creating classes that inherit behavior from more than one parent class.

To see how this works, update the Car class by breaking it into two categories, one for vehicles, and one for devices that use electricity:

class Vehicle:
    def __init__(self, color, model):
        self.color = color
        self.model = model

class Device:
    def __init__(self):
        self._voltage = 12

class Car(Vehicle, Device):
    def __init__(self, color, model, year):
        Vehicle.__init__(self, color, model)
        Device.__init__(self)
        self.year = year

    @property
    def voltage(self):
        return self._voltage

    @voltage.setter
    def voltage(self, volts):
        print("Warning: this can cause problems!")
        self._voltage = volts

    @voltage.deleter
    def voltage(self):
        print("Warning: the radio will stop working!")
        del self._voltage

A Vehicle is defined as having .color and .model attributes. Then, a Device is defined to have a ._voltage attribute. Since the original Car object had these three attributes, it can be redefined to inherit both the Vehicle and Device classes. The color, model, and _voltage attributes will be part of the new Car class.

In the .__init__() for Car, you call the .__init__() methods for both of the parent classes to make sure everything is initialized properly. Once done, you can add any other functionality you want to your Car. In this case, you add a .year attribute specific to Car objects, and getter and setter methods for .voltage.

Functionally, the new Car class behaves as it always has. You create and use Car objects just as before:

>>>

>>> from car import *
>>> my_car = Car("yellow", "beetle", 1969)

>>> print(f"My car is {my_car.color}")
My car is yellow

>>> print(f"My car uses {my_car.voltage} volts")
My car uses 12 volts

>>> my_car.voltage = 6
Warning: this can cause problems!

>>> print(f"My car now uses {my_car.voltage} volts")
My car now uses 6 volts

Java, on the other hand, only supports single inheritance, which means classes in Java can inherit data and behavior from only a single parent class. However, Java objects can inherit behavior from many different interfaces. Interfaces provide a group of related methods an object must implement, and allow multiple child classes to behave similarly.

To see this in action, split the Java Car class into a parent class and an interface:

public class Vehicle {

    private String color;
    private String model;

    public Vehicle(String color, String model) {
        this.color = color;
        this.model = model;
    }

    public String getColor() {
        return color;
    }

    public String getModel() {
        return model;
    }
}

public interface Device {
    int getVoltage();
}

public class Car extends Vehicle implements Device {

    private int voltage;
    private int year;

    public Car(String color, String model, int year) {
        super(color, model);
        this.year = year;
        this.voltage = 12;
    }

    @Override
    public int getVoltage() {
        return voltage;
    }

    public int getYear() {
        return year;
    }
}

Remember that each class and interface needs to live in its own file.

As you did with Python, you create a new class called Vehicle to hold the more general vehicle related data and functionality. However, to add the Device functionality, you need to create an interface instead. This interface defines a single method to return the voltage of the Device.

Redefining the Car class requires you to inherit from Vehicle using extend, and implement the Device interface using implements. In the constructor, you call the parent class constructor using the built-in super(). Since there is only one parent class, it can only refer to the Vehicle constructor. To implement the interface, you write getVoltage() using the @Override annotation.

Rather than getting code reuse from Device as Python did, Java requires you to implement the same functionality in every class that implements the interface. Interfaces only define the methods—they cannot define instance data or implementation details.

So why is this the case for Java? It all comes down to types.

Types and Polymorphism

Java’s strict type checking is what drives its interface design.

Every class and interface in Java is a type. Therefore, if two Java objects implement the same interface, then they are considered to be the same type with respect to that interface. This mechanism allows different classes to be used interchangeably, which is the definition of polymorphism.

You can implement device charging for your Java objects by creating a .charge() that takes a Device to charge. Any object that implements the Device interface can be passed to .charge(). This also means that classes that do not implement Device will generate a compilation error.

Create the following class in a file called Rhino.java:

public class Rhino {
}

Now you can create a new Main.java to implement .charge() and explore how Car and Rhino objects differ:

public class Main{
    public static void charge(Device device) {
       device.getVoltage();
    }

    public static void main(String[] args) throws Exception {
        Car car = new Car("yellow", "beetle", 1969);
        Rhino rhino = new Rhino();
        charge(car);
        charge(rhino);
    }
}

Here is what you should see when you try to build this code:

Information:2019-02-02 15:20 - Compilation completed with 
    1 error and 0 warnings in 4 s 395 ms
Main.java
Error:(43, 11) java: incompatible types: Rhino cannot be converted to Device

Since the Rhino class doesn’t implement the Device interface, it can’t be passed into .charge().

In contrast to Java’s strict variable typing, Python uses a concept called duck typing, which in basic terms means that if a variable “walks like a duck and quacks like a duck, then it’s a duck.” Instead of identifying objects by type, Python examines their behavior. You can learn more about the Python type system, and duck typing, in The Ultimate Guide to Python Type Checking.

You can explore duck typing by implementing similar device charging capabilities for your Python Device class:

>>>

>>> def charge(device):
...     if hasattr(device, '_voltage'):
...         print(f"Charging a {device._voltage} volt device")
...     else:
...         print(f"I can't charge a {device.__class__.__name__}")
... 
>>> class Phone(Device):
...     pass
... 
>>> class Rhino:
...     pass
... 
>>> my_car = Car("yellow", "Beetle", "1966")
>>> my_phone = Phone()
>>> my_rhino = Rhino()

>>> charge(my_car)
Charging a 12 volt device
>>> charge(my_phone)
Charging a 12 volt device
>>> charge(my_rhino)
I can't charge a Rhino

charge() must check for the existence of the ._voltage attribute in the object it’s passed. Since the Device class defines this attribute, any class that inherits from it (such as Car and Phone) will have this attribute, and will therefore show they are charging properly. Classes that do not inherit from Device (like Rhino) may not have this attribute, and will not be able to charge (which is good, since charging rhinos can be dangerous).

Default Methods

All Java classes descend from the Object class, which contains a set of methods every other class inherits. Subclasses can either override them or keep the defaults. The Object class defines the following methods:

class Object {
    boolean equals(Object obj) { ... }    
    int hashCode() { ... }    
    String toString() { ... }    
}

By default, equals() compares the addresses of the current Object with a second Object passed in, and hashcode() computes a unique identifier that also uses the address of the current Object. These methods are used in many different contexts in Java. For example, utility classes, such as collections that sort objects based on value, need both of them.

toString() returns a String representation of the Object. By default, this is the name of the class and the address. This method is called automatically when an Object is passed to a method that requires a String argument, such as System.out.println():

Car car = new Car("yellow", "Beetle", 1969);
System.out.println(car);

Running this code will use the default .toString() to show the car object:

Car@61bbe9ba

Not very useful, right? You can improve this by overriding the default .toString(). Add this method to your Java Car class:

public String toString() {
    return "Car: " + getColor() + " : " + getModel() + " : " + getYear();
}

Now, when you run the same sample code, you’ll see the following:

Car: yellow : Beetle : 1969

Python provides similar functionality with a set of common dunder (short for “double underscore”) methods. Every Python class inherits these methods, and you can override them to modify their behavior.

For string representations of an object, Python provides __repr__() and __str__(), which you can learn about in Pythonic OOP String Conversion: __repr__ vs __str__. The unambiguous representation of an object is returned by __repr__(), while __str__() returns a human readable representation. These are roughly analogous to .hashcode() and .toString() in Java.

Like Java, Python provides default implementations of these dunder methods:

>>>

>>> my_car = Car("yellow", "Beetle", "1966")

>>> print(repr(my_car))
<car.Car object at 0x7fe4ca154f98>
>>> print(str(my_car))
<car.Car object at 0x7fe4ca154f98>

You can improve this output by overriding .__str__(), adding this to your Python Car class:

def __str__(self):
    return f'Car {self.color} : {self.model} : {self.year}'

This gives you a much nicer result:

>>>

>>> my_car = Car("yellow", "Beetle", "1966")

>>> print(repr(my_car))
<car.Car object at 0x7f09e9a7b630>
>>> print(str(my_car))
Car yellow : Beetle : 1966

Overriding the dunder method gave us a more readable representation of your Car. You may want to override the .__repr__() as well, as it is often useful for debugging.

Python offers a lot more dunder methods. Using dunder methods, you can define your object’s behavior during iteration, comparison, addition, or making an object callable directly, among other things.

Operator Overloading

Operator overloading refers to redefining how Python operators work when operating on user-defined objects. Python’s dunder methods allow you to implement operator overloading, something that Java doesn’t offer at all.

Modify your Python Car class with the following additional dunder methods:

class Car:
    def __init__(self, color, model, year):
        self.color = color
        self.model = model
        self.year = year

    def __str__(self):
        return f'Car {self.color} : {self.model} : {self.year}'

    def __eq__(self, other):
        return self.year == other.year

    def __lt__(self, other):
        return self.year < other.year

    def __add__(self, other):
        return Car(self.color + other.color, 
                   self.model + other.model, 
                   int(self.year) + int(other.year))

The table below shows the relationship between these dunder methods and the Python operators they represent:

Dunder Method	Operator	Purpose
`__eq__`	`==`	Do these `Car` objects have the same year?
`__lt__`	`<`	Which `Car` is an earlier model?
`__add__`	`+`	Add two `Car` objects in a nonsensical way

When Python sees an expression containing objects, it calls any dunder methods defined that correspond to operators in the expression. The code below uses these new overloaded arithmetic operators on a couple of Car objects:

>>>

>>> my_car = Car("yellow", "Beetle", "1966")
>>> your_car = Car("red", "Corvette", "1967")

>>> print (my_car < your_car)
True
>>> print (my_car > your_car)
False
>>> print (my_car == your_car)
False
>>> print (my_car + your_car)
Car yellowred : BeetleCorvette : 3933

There are many more operators you can overload using dunder methods. They offer a way to enrich your object’s behavior in a way that Java’s common base class default methods don’t.

Reflection

Reflection refers to examining an object or class from within the object or class. Both Java and Python offer ways to explore and examine the attributes and methods in a class.

Examining an Object’s Type

Both languages have ways to test or check an object’s type.

In Python, you use type() to display the type of a variable, and isinstance() to determine if a given variable is an instance or child of a specific class:

>>>

>>> my_car = Car("yellow", "Beetle", "1966")

>>> print(type(my_car))
<class 'car.Car'>
>>> print(isinstance(my_car, Car))
True
>>> print(isinstance(my_car, Device))
True

In Java, you query the object for its type using .getClass(), and use the instanceof operator to check for a specific class:

Car car = new Car("yellow", "beetle", 1969);

System.out.println(car.getClass());
System.out.println(car instanceof Car);

This code outputs the following:

class com.realpython.Car
true

Examining an Object’s Attributes

In Python, you can view every attribute and function contained in any object (including all the dunder methods) using dir(). To get the specific details of a given attribute or function, use getattr():

>>>

>>> print(dir(my_car))
['_Car__cupholders', '__add__', '__class__', '__delattr__', '__dict__', 
 '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__',
 '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__',
 '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',
 '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__',
 '_voltage', 'color', 'model', 'voltage', 'wheels', 'year']

>>> print(getattr(my_car, "__format__"))
<built-in method __format__ of Car object at 0x7fb4c10f5438>

Java has similar capabilities, but the language’s access control and type safety make it more complicated to retrieve.

.getFields() retrieves a list of all publicly accessible attributes. However, since none of the attributes of Car are public, this code returns an empty array:

Field[] fields = car.getClass().getFields();

Java treats attributes and methods as separate entities, so public methods are retrieved using .getDeclaredMethods(). Since public attributes will have a corresponding .get method, one way to discover if a class contains a specific property might look like this:

Use .getDeclaredMethods() to generate an array of all the methods.
Loop through all the methods returned:
- For each method discovered, return true if the method:
  - Begins with the word get OR accepts zero arguments
  - AND doesn’t return void
  - AND includes the name of the property
- Otherwise, return false.

Here’s a quick-and-dirty example:

 1 public static boolean getProperty(String name, Object object) throws Exception {
 2 
 3     Method[] declaredMethods = object.getClass().getDeclaredMethods();
 4     for (Method method : declaredMethods) {
 5         if (isGetter(method) && 
 6             method.getName().toUpperCase().contains(name.toUpperCase())) {
 7               return true;
 8         }
 9     }
10     return false;
11 }
12 
13 // Helper function to get if the method is a getter method
14 public static boolean isGetter(Method method) {
15     if ((method.getName().startsWith("get") || 
16          method.getParameterCount() == 0 ) && 
17         !method.getReturnType().equals(void.class)) {
18           return true;
19     }
20     return false;
21 }

getProperty() is your entry point. Call this with the name of an attribute and an object. It returns true if the property is found, and false if not.

Calling Methods Through Reflection

Both Java and Python provide mechanisms to call methods through reflection.

In the Java example above, instead of simply returning true if the property was found, you could call the method directly. Recall that getDeclaredMethods() returns an array of Method objects. The Method object itself has a method called .invoke(), which will call the Method. Instead of returning true when the correct method is found on line 7 above, you can return method.invoke(object) instead.

This capability exists in Python as well. However, since Python doesn’t differentiate between functions and attributes, you have to look specifically for entries that are callable:

>>>

>>> for method_name in dir(my_car):
...     if callable(getattr(my_car, method_name)):
...         print(method_name)
... 
__add__
__class__
__delattr__
__dir__
__eq__
__format__
__ge__
__getattribute__
__gt__
__init__
__init_subclass__
__le__
__lt__
__ne__
__new__
__reduce__
__reduce_ex__
__repr__
__setattr__
__sizeof__
__str__
__subclasshook__

Python methods are simpler to manage and call than in Java. Adding the () operator (and any required arguments) is all you need to do.

The code below will find an object’s .__str__() and call it through reflection:

>>>

>>> for method_name in dir(my_car):
...     attr = getattr(my_car, method_name)
...     if callable(attr):
...         if method_name == '__str__':
...             print(attr())
... 
Car yellow : Beetle : 1966

Here, every attribute returned by dir() is checked. You get the actual attribute object using getattr(), and check if it’s a callable function using callable(). If so, you then check if its name is __str__(), and then call it.

Conclusion

Throughout the course of this article, you learned how object-oriented principles differ in Python vs Java. As you read, you:

Built a basic class in both Java and Python
Explored how object attributes work in Python vs Java
Compared and contrasted Java methods and Python functions
Discovered inheritance and polymorphism mechanisms in both languages
Investigated reflection across Python vs Java
Applied everything in a complete class implementation in both languages

If you want to learn more about OOP in Python, be sure to read Object-Oriented Programming (OOP) in Python 3.

Understanding the differences in Python vs Java when handling objects, and the syntax choices each language makes, will help you apply best practices and make your next project smoother.

In order to compare some concrete examples side by side, you can download our sample code to get the complete commented object definitions for the Java Car, Device, and Vehicle classes, as well as the complete commented definitions for the Python Car and Vehicle classes:

Download Sample Code: Click here to download the commented sample object definitions and source code for the Java and Python objects in this article.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Pointers in Python: What's the Point?

2019-05-29T14:00:00+00:00

If you’ve ever worked with lower level languages like C or C++, then you’ve probably heard of pointers. Pointers allow you to create great efficiency in parts of your code. They also cause confusion for beginners and can lead to various memory management bugs, even for experts. So where are they in Python, and how can you simulate pointers in Python?

Pointers are widely used in C and C++. Essentially, they are variables that hold the memory address of another variable. For a refresher on pointers, you might consider checking out this overview on C Pointers.

In this article, you’ll gain a better understanding of Python’s object model and learn why pointers in Python don’t really exist. For the cases where you need to mimic pointer behavior, you’ll learn ways to simulate pointers in Python without the memory-management nightmare.

In this article, you’ll:

Learn why pointers in Python don’t exist
Explore the difference between C variables and Python names
Simulate pointers in Python
Experiment with real pointers using ctypes

Note: In this article, “Python” will refer to the reference implementation of Python in C, otherwise known as CPython. As the article discusses some internals of the language, these notes are true for CPython 3.7 but may not be true in future or past iterations of the language.

Why Doesn’t Python Have Pointers?

The truth is that I don’t know. Could pointers in Python exist natively? Probably, but pointers seem to go against the Zen of Python. Pointers encourage implicit changes rather than explicit. Often, they are complex instead of simple, especially for beginners. Even worse, they beg for ways to shoot yourself in the foot, or do something really dangerous like read from a section of memory you were not supposed to.

Python tends to try to abstract away implementation details like memory addresses from its users. Python often focuses on usability instead of speed. As a result, pointers in Python doesn’t really make sense. Not to fear though, Python does, by default, give you some of the benefits of using pointers.

Understanding pointers in Python requires a short detour into Python’s implementation details. Specifically, you’ll need to understand:

Immutable vs mutable objects
Python variables/names

Hold onto your memory addresses, and let’s get started.

Objects in Python

In Python, everything is an object. For proof, you can open up a REPL and explore using isinstance():

>>>

>>> isinstance(1, object)
True
>>> isinstance(list(), object)
True
>>> isinstance(True, object)
True
>>> def foo():
...     pass
...
>>> isinstance(foo, object)
True

This code shows you that everything in Python is indeed an object. Each object contains at least three pieces of data:

Reference count
Type
Value

The reference count is for memory management. For an in-depth look at the internals of memory management in Python, you can read Memory Management in Python.

The type is used at the CPython layer to ensure type safety during runtime. Finally, there’s the value, which is the actual value associated with the object.

Not all objects are the same though. There is one other important distinction you’ll need to understand: immutable vs mutable objects. Understanding the difference between the types of objects really helps clarify the first layer of the onion that is pointers in Python.

Immutable vs Mutable Objects

In Python, there are two types of objects:

Immutable objects can’t be changed.
Mutable objects can be changed.

Understanding this difference is the first key to navigating the landscape of pointers in Python. Here’s a breakdown of common types and whether or not they are mutable or immutable:

Type	Immutable?
`int`	Yes
`float`	Yes
`bool`	Yes
`complex`	Yes
`tuple`	Yes
`frozenset`	Yes
`str`	Yes
`list`	No
`set`	No
`dict`	No

As you can see, lots of commonly used primitive types are immutable. You can prove this yourself by writing some Python. You’ll need a couple of tools from the Python standard library:

id() returns the object’s memory address.
is returns True if and only if two objects have the same memory address.

Once again, you can use these in a REPL environment:

>>>

>>> x = 5
>>> id(x)
94529957049376

In the above code, you have assigned the value 5 to x. If you tried to modify this value with addition, then you’d get a new object:

>>>

>>> x += 1
>>> x
6
>>> id(x)
94529957049408

Even though the above code appears to modify the value of x, you’re getting a new object as a response.

The str type is also immutable:

>>>

>>> s = "real_python"
>>> id(s)
140637819584048
>>> s += "_rocks"
>>> s
'real_python_rocks'
>>> id(s)
140637819609424

Again, s ends up with a different memory addresses after the += operation.

Bonus: The += operator translates to various method calls.

For some objects like list, += will translate into __iadd__() (in-place add). This will modify self and return the same ID. However, str and int don’t have these methods and result in __add__() calls instead of __iadd__().

For more detailed information, check out the Python data model docs.

Trying to directly mutate the string s results in an error:

>>>

>>> s[0] = "R"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'str' object does not support item assignment

The above code fails, and Python indicates that str doesn’t support this mutation, which is in line with the definition that the str type is immutable.

Contrast that with a mutable object, like list:

>>>

>>> my_list = [1, 2, 3]
>>> id(my_list)
140637819575368
>>> my_list.append(4)
>>> my_list
[1, 2, 3, 4]
>>> id(my_list)
140637819575368

This code shows a major difference in the two types of objects. my_list has an id originally. Even after 4 is appended to the list, my_list has the same id. This is because the list type is mutable.

Another way to demonstrate that the list is mutable is with assignment:

>>>

>>> my_list[0] = 0
>>> my_list
[0, 2, 3, 4]
>>> id(my_list)
140637819575368

In this code, you mutate my_list and set its first element to 0. However, it maintains the same id even after this assignment. With mutable and immutable objects out of the way, the next step on your journey to Python enlightenment is understanding Python’s variable ecosystem.

Understanding Variables

Python variables are fundamentally different than variables in C or C++. In fact, Python doesn’t even have variables. Python has names, not variables.

This might seem pedantic, and for the most part, it is. Most of the time, it’s perfectly acceptable to think about Python names as variables, but understanding the difference is important. This is especially true when you’re navigating the tricky subject of pointers in Python.

To help drive home the difference, you can take a look at how variables work in C, what they represent, and then contrast that with how names work in Python.

Variables in C

Let’s say you had the following code that defines the variable x:

int x = 2337;

This one line of code has several, distinct steps when executed:

Allocate enough memory for an integer
Assign the value 2337 to that memory location
Indicate that x points to that value

Shown in a simplified view of memory, it might look like this:

Here, you can see that the variable x has a fake memory location of 0x7f1 and the value 2337. If, later in the program, you want to change the value of x, you can do the following:

x = 2338;

The above code assigns a new value (2338) to the variable x, thereby overwriting the previous value. This means that the variable x is mutable. The updated memory layout shows the new value:

Notice that the location of x didn’t change, just the value itself. This is a significant point. It means that x is the memory location, not just a name for it.

Another way to think of this concept is in terms of ownership. In one sense, x owns the memory location. x is, at first, an empty box that can fit exactly one integer in which integer values can be stored.

When you assign a value to x, you’re placing a value in the box that x owns. If you wanted to introduce a new variable (y), you could add this line of code:

int y = x;

This code creates a new box called y and copies the value from x into the box. Now the memory layout will look like this:

Notice the new location 0x7f5 of y. Even though the value of x was copied to y, the variable y owns some new address in memory. Therefore, you could overwrite the value of y without affecting x:

y = 2339;

Now the memory layout will look like this:

Again, you have modified the value at y, but not its location. In addition, you have not affected the original x variable at all. This is in stark contrast with how Python names work.

Names in Python

Python does not have variables. It has names. Yes, this is a pedantic point, and you can certainly use the term variables as much as you like. It is important to know that there is a difference between variables and names.

Let’s take the equivalent code from the above C example and write it in Python:

>>>

>>> x = 2337

Much like in C, the above code is broken down into several distinct steps during execution:

Create a PyObject
Set the typecode to integer for the PyObject
Set the value to 2337 for the PyObject
Create a name called x
Point x to the new PyObject
Increase the refcount of the PyObject by 1

Note: The PyObject is not the same as Python’s object. It’s specific to CPython and represents the base structure for all Python objects.

PyObject is defined as a C struct, so if you’re wondering why you can’t call typecode or refcount directly, its because you don’t have access to the structures directly. Method calls like sys.getrefcount() can help get some internals.

In memory, it might looks something like this:

You can see that the memory layout is vastly different than the C layout from before. Instead of x owning the block of memory where the value 2337 resides, the newly created Python object owns the memory where 2337 lives. The Python name x doesn’t directly own any memory address in the way the C variable x owned a static slot in memory.

If you were to try to assign a new value to x, you could try the following:

>>>

>>> x = 2338

What’s happening here is different than the C equivalent, but not too different from the original bind in Python.

This code:

Creates a new PyObject
Sets the typecode to integer for the PyObject
Sets the value to 2338 for the PyObject
Points x to the new PyObject
Increases the refcount of the new PyObject by 1
Decreases the refcount of the old PyObject by 1

Now in memory, it would look something like this:

This diagram helps illustrate that x points to a reference to an object and doesn’t own the memory space as before. It also shows that the x = 2338 command is not an assignment, but rather binding the name x to a reference.

In addition, the previous object (which held the 2337 value) is now sitting in memory with a ref count of 0 and will get cleaned up by the garbage collector.

You could introduce a new name, y, to the mix as in the C example:

>>>

>>> y = x

In memory, you would have a new name, but not necessarily a new object:

Now you can see that a new Python object has not been created, just a new name that points to the same object. Also, the object’s refcount has increased by one. You could check for object identity equality to confirm that they are the same:

>>>

>>> y is x
True

The above code indicates that x and y are the same object. Make no mistake though: y is still immutable.

For example, you could perform addition on y:

>>>

>>> y += 1
>>> y is x
False

After the addition call, you are returned with a new Python object. Now, the memory looks like this:

A new object has been created, and y now points to the new object. Interestingly, this is the same end-state if you had bound y to 2339 directly:

>>>

>>> y = 2339

The above statement results in the same end-memory state as the addition. To recap, in Python, you don’t assign variables. Instead, you bind names to references.

A Note on Intern Objects in Python

Now that you understand how Python objects get created and names get bound to those objects, its time to throw a wrench in the machinery. That wrench goes by the name of interned objects.

Suppose you have the following Python code:

>>>

>>> x = 1000
>>> y = 1000
>>> x is y
True

As above, x and y are both names that point to the same Python object. But the Python object that holds the value 1000 is not always guaranteed to have the same memory address. For example, if you were to add two numbers together to get 1000, you would end up with a different memory address:

>>>

>>> x = 1000
>>> y = 499 + 501
>>> x is y
False

This time, the line x is y returns False. If this is confusing, then don’t worry. Here are the steps that occur when this code is executed:

Create Python object(1000)
Assign the name x to that object
Create Python object (499)
Create Python object (501)
Add these two objects together
Create a new Python object (1000)
Assign the name y to that object

Technical Note: The above steps occur only when this code is executed inside a REPL. If you were to take the example above, paste it into a file, and run the file, then you would find that the x is y line would return True.

This occurs because compilers are smart. The CPython compiler attempts to make optimizations called peephole optimizations, which help save execution steps whenever possible. For detailed information, you can check out CPython’s peephole optimizer source code.

Isn’t this wasteful? Well, yes it is, but that’s the price you pay for all of the great benefits of Python. You never have to worry about cleaning up these intermediate objects or even need to know that they exist! The joy is that these operations are relatively fast, and you never had to know any of those details until now.

The core Python developers, in their wisdom, also noticed this waste and decided to make a few optimizations. These optimizations result in behavior that can be surprising to newcomers:

>>>

>>> x = 20
>>> y = 19 + 1
>>> x is y
True

In this example, you see nearly the same code as before, except this time the result is True. This is the result of interned objects. Python pre-creates a certain subset of objects in memory and keeps them in the global namespace for everyday use.

Which objects depend on the implementation of Python. CPython 3.7 interns the following:

Integer numbers between -5 and 256
Strings that contain ASCII letters, digits, or underscores only

The reasoning behind this is that these variables are extremely likely to be used in many programs. By interning these objects, Python prevents memory allocation calls for consistently used objects.

Strings that are less than 20 characters and contain ASCII letters, digits, or underscores will be interned. The reasoning behind this is that these are assumed to be some kind of identity:

>>>

>>> s1 = "realpython"
>>> id(s1)
140696485006960
>>> s2 = "realpython"
>>> id(s2)
140696485006960
>>> s1 is s2
True

Here you can see that s1 and s2 both point to the same address in memory. If you were to introduce a non-ASCII letter, digit, or underscore, then you would get a different result:

>>>

>>> s1 = "Real Python!"
>>> s2 = "Real Python!"
>>> s1 is s2
False

Because this example has an exclamation mark (!) in it, these strings are not interned and are different objects in memory.

Bonus: If you really want these objects to reference the same internal object, then you may want to check out sys.intern(). One of the use cases for this function is outlined in the documentation:

Interning strings is useful to gain a little performance on dictionary lookup—if the keys in a dictionary are interned, and the lookup key is interned, the key comparisons (after hashing) can be done by a pointer compare instead of a string compare. (Source)

Interned objects are often a source of confusion. Just remember, if you’re ever in doubt, that you can always use id() and is to determine object equality.

Simulating Pointers in Python

Just because pointers in Python don’t exist natively doesn’t mean you can’t get the benefits of using pointers. In fact, there are multiple ways to simulate pointers in Python. You’ll learn two in this section:

Using mutable types as pointers
Using custom Python objects

Okay, let’s get to the point.

Using Mutable Types as Pointers

You’ve already learned about mutable types. Because these objects are mutable, you can treat them as if they were pointers to simulate pointer behavior. Suppose you wanted to replicate the following c code:

void add_one(int *x) {
    *x += 1;
}

This code takes a pointer to an integer (*x) and then increments the value by one. Here is a main function to exercise the code:

#include <stdio.h>

int main(void) {
    int y = 2337;
    printf("y = %d\n", y);
    add_one(&y);
    printf("y = %d\n", y);
    return 0;
}

In the above code, you assign 2337 to y, print out the current value, increment the value by one, and then print out the modified value. The output of executing this code would be the following:

y = 2337
y = 2338

One way to replicate this type of behavior in Python is by using a mutable type. Consider using a list and modifying the first element:

>>>

>>> def add_one(x):
...     x[0] += 1
...
>>> y = [2337]
>>> add_one(y)
>>> y[0]
2338

Here, add_one(x) accesses the first element and increments its value by one. Using a list means that the end result appears to have modified the value. So pointers in Python do exist? Well, no. This is only possible because list is a mutable type. If you tried to use a tuple, you would get an error:

>>>

>>> z = (2337,)
>>> add_one(z)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in add_one
TypeError: 'tuple' object does not support item assignment

The above code demonstrates that tuple is immutable. Therefore, it does not support item assignment. list is not the only mutable type. Another common approach to mimicking pointers in Python is to use a dict.

Let’s say you had an application where you wanted to keep track of every time an interesting event happened. One way to achieve this would be to create a dict and use one of the items as a counter:

>>>

>>> counters = {"func_calls": 0}
>>> def bar():
...     counters["func_calls"] += 1
...
>>> def foo():
...     counters["func_calls"] += 1
...     bar()
...
>>> foo()
>>> counters["func_calls"]
2

In this example, the counters dictionary is used to keep track of the number of function calls. After you call foo(), the counter has increased to 2 as expected. All because dict is mutable.

Keep in mind, this is only simulates pointer behavior and does not directly map to true pointers in C or C++. That is to say, these operations are more expensive than they would be in C or C++.

Using Python Objects

The dict option is a great way to emulate pointers in Python, but sometimes it gets tedious to remember the key name you used. This is especially true if you’re using the dictionary in various parts of your application. This is where a custom Python class can really help.

To build on the last example, assume that you want to track metrics in your application. Creating a class is a great way to abstract the pesky details:

class Metrics(object):
    def __init__(self):
        self._metrics = {
            "func_calls": 0,
            "cat_pictures_served": 0,
        }

This code defines a Metrics class. This class still uses a dict for holding the actual data, which is in the _metrics member variable. This will give you the mutability you need. Now you just need to be able to access these values. One nice way to do this is with properties:

class Metrics(object):
    # ...

    @property
    def func_calls(self):
        return self._metrics["func_calls"]

    @property
    def cat_pictures_served(self):
        return self._metrics["cat_pictures_served"]

This code makes use of @property. If you’re not familiar with decorators, you can check out this Primer on Python Decorators. The @property decorator here allows you to access func_calls and cat_pictures_served as if they were attributes:

>>>

>>> metrics = Metrics()
>>> metrics.func_calls
0
>>> metrics.cat_pictures_served
0

The fact that you can access these names as attributes means that you abstracted the fact that these values are in a dict. You also make it more explicit what the names of the attributes are. Of course, you need to be able to increment these values:

class Metrics(object):
    # ...

    def inc_func_calls(self):
        self._metrics["func_calls"] += 1

    def inc_cat_pics(self):
        self._metrics["cat_pictures_served"] += 1

You have introduced two new methods:

inc_func_calls()
inc_cat_pics()

These methods modify the values in the metrics dict. You now have a class that you modify as if you’re modifying a pointer:

>>>

>>> metrics = Metrics()
>>> metrics.inc_func_calls()
>>> metrics.inc_func_calls()
>>> metrics.func_calls
2

Here, you can access func_calls and call inc_func_calls() in various places in your applications and simulate pointers in Python. This is useful when you have something like metrics that need to be used and updated frequently in various parts of your applications.

Note: In this class in particular, making inc_func_calls() and inc_cat_pics() explicit instead of using @property.setter prevents users from setting these values to an arbitrary int or an invalid value like a dict.

Here’s the full source for the Metrics class:

class Metrics(object):
    def __init__(self):
        self._metrics = {
            "func_calls": 0,
            "cat_pictures_served": 0,
        }

    @property
    def func_calls(self):
        return self._metrics["func_calls"]

    @property
    def cat_pictures_served(self):
        return self._metrics["cat_pictures_served"]

    def inc_func_calls(self):
        self._metrics["func_calls"] += 1

    def inc_cat_pics(self):
        self._metrics["cat_pictures_served"] += 1

Real Pointers With `ctypes`

Okay, so maybe there are pointers in Python, specifically CPython. Using the builtin ctypes module, you can create real C-style pointers in Python. If you are unfamiliar with ctypes, then you can take a look at Extending Python With C Libraries and the “ctypes” Module.

The real reason you would use this is if you needed to make a function call to a C library that requires a pointer. Let’s go back to the add_one() C-function from before:

void add_one(int *x) {
    *x += 1;
}

Here again, this code is incrementing the value of x by one. To use this, first compile it into a shared object. Assuming the above file is stored in add.c, you could accomplish this with gcc:

$ gcc -c -Wall -Werror -fpic add.c
$ gcc -shared -o libadd1.so add.o

The first command compiles the C source file into an object called add.o. The second command takes that unlinked object file and produces a shared object called libadd1.so.

libadd1.so should be in your current directory. You can load it into Python using ctypes:

>>>

>>> import ctypes
>>> add_lib = ctypes.CDLL("./libadd1.so")
>>> add_lib.add_one
<_FuncPtr object at 0x7f9f3b8852a0>

The ctypes.CDLL code returns an object that represents the libadd1 shared object. Because you defined add_one() in this shared object, you can access it as if it were any other Python object. Before you call the function though, you should specify the function signature. This helps Python ensure that you pass the right type to the function.

In this case, the function signature is a pointer to an integer. ctypes will allow you to specify this using the following code:

>>>

>>> add_one = add_lib.add_one
>>> add_one.argtypes = [ctypes.POINTER(ctypes.c_int)]

In this code, you’re setting the function signature to match what C is expecting. Now, if you were to try to call this code with the wrong type, then you would get a nice warning instead of undefined behavior:

>>>

>>> add_one(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ctypes.ArgumentError: argument 1: <class 'TypeError'>: \
expected LP_c_int instance instead of int

Python throws an error, explaining that add_one() wants a pointer instead of just an integer. Luckily, ctypes has a way to pass pointers to these functions. First, declare a C-style integer:

>>>

>>> x = ctypes.c_int()
>>> x
c_int(0)

The above code creates a C-style integer x with a value of 0. ctypes provides the handy byref() to allow passing a variable by reference.

Note: The term by reference is opposed to passing a variable by value.

When passing by reference, you’re passing the reference to the original variable, and thus modifications will be reflected in the original variable. Passing by value results in a copy of the original variable, and modifications are not reflected in the original.

You can use this to call add_one():

>>>

>>> add_one(ctypes.byref(x))
998793640
>>> x
c_int(1)

Nice! Your integer was incremented by one. Congratulations, you have successfully used real pointers in Python.

Conclusion

You now have a better understanding of the intersection between Python objects and pointers. Even though some of the distinctions between names and variables seem pedantic, fundamentally understanding these key terms expands your understanding of how Python handles variables.

You’ve also learned some excellent ways to simulate pointers in Python:

Utilizing mutable objects as low-overhead pointers
Creating custom Python objects for ease of use
Unlocking real pointers with the ctypes module

These methods allow you to simulate pointers in Python without sacrificing the memory safety that Python provides.

Thanks for reading. If you still have questions, feel free to reach out either in the comments section or on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Continuous Integration With Python

2019-05-28T14:00:00+00:00

In this course, you’ll learn the core concepts behind Continuous Integration (CI) and why they are essential for modern software engineering teams.

Find out how to set up Continuous Integration for your Python project to automatically create environments, install dependencies, and run tests. You’ll learn how to work with GitHub, CircleCI, and PyTest.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Iterate Through a Dictionary in Python

2019-05-27T14:00:00+00:00

Dictionaries are one of the most important and useful data structures in Python. They can help you solve a wide variety of programming problems. This tutorial will take you on a deep dive into how to iterate through a dictionary in Python.

By the end of this tutorial, you’ll know:

What dictionaries are, as well as some of their main features and implementation details
How to iterate through a dictionary in Python by using the basic tools the language offers
What kind of real-world tasks you can perform by iterating through a dictionary in Python
How to use some more advanced techniques and strategies to iterate through a dictionary in Python

For more information on dictionaries, you can check out the following resources:

Dictionaries in Python
Itertools in Python 3, By Example
The documentation for map() and filter()

Ready? Let’s go!

A Few Words on Dictionaries

Dictionaries are a cornerstone of Python. The language itself is built around dictionaries. Modules, classes, objects, globals(), locals(): all of these are dictionaries. Dictionaries have been central to Python from its very beginning.

Python’s official documentation defines a dictionary as follows:

An associative array, where arbitrary keys are mapped to values. The keys can be any object with __hash__() and __eq__() methods. (Source)

There are a couple points to keep in mind:

Dictionaries map keys to values and store them in an array or collection.
The keys must be of a hashable type, which means that they must have a hash value that never changes during the key’s lifetime.

Dictionaries are frequently used for solving all kinds of programming problems, so they are a fundamental piece of your tool kit as a Python developer.

Unlike sequences, which are iterables that support element access using integer indices, dictionaries are indexed by keys.

The keys in a dictionary are much like a set, which is a collection of hashable and unique objects. Because the objects need to be hashable, mutable objects can’t be used as dictionary keys.

On the other hand, values can be of any Python type, whether they are hashable or not. There are literally no restrictions for values.

In Python 3.6 and beyond, the keys and values of a dictionary are iterated over in the same order in which they were created. However, this behavior may vary across different Python versions, and it depends on the dictionary’s history of insertions and deletions.

In Python 2.7, dictionaries are unordered structures. The order of the dictionaries’ items is scrambled. This means that the order of the items is deterministic and repeatable. Let’s see an example:

>>>

>>> # Python 2.7
>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> a_dict
{'color': 'blue', 'pet': 'dog', 'fruit': 'apple'}
>>> a_dict
{'color': 'blue', 'pet': 'dog', 'fruit': 'apple'}

If you leave the interpreter and open a new interactive session later, you’ll get the same item order:

>>>

>>> # Python 2.7. New interactive session
>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> a_dict
{'color': 'blue', 'pet': 'dog', 'fruit': 'apple'}
>>> a_dict
{'color': 'blue', 'pet': 'dog', 'fruit': 'apple'}

A closer look at these two outputs shows you that the resulting order is exactly the same in both cases. That’s why you can say that the ordering is deterministic.

In Python 3.5, dictionaries are still unordered, but this time, randomized data structures. This means that every time you re-run the dictionary, you’ll get a different items order. Let’s take a look:

>>>

>>> # Python 3.5
>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> a_dict
{'color': 'blue', 'pet': 'dog', 'fruit': 'apple'}
>>> a_dict
{'color': 'blue', 'pet': 'dog', 'fruit': 'apple'}

If you enter a new interactive session, then you’ll get the following:

>>>

>>> # Python 3.5. New interactive session
>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> a_dict
{'fruit': 'apple', 'pet': 'dog', 'color': 'blue'}
>>> a_dict
{'fruit': 'apple', 'pet': 'dog', 'color': 'blue'}

This time, you can see that the order of the items is different in both outputs. That’s why you can say they are randomized data structures.

In Python 3.6 and beyond, dictionaries are ordered data structures, which means that they keep their elements in the same order in which they were introduced, as you can see here:

>>>

>>> # Python 3.6 and beyond
>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> a_dict
{'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> a_dict
{'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}

This is a relatively new feature of Python’s dictionaries, and it’s a very useful one. But if you’re writing code that is supposed to be run in different Python versions, then you must not rely on this feature, because it can generate buggy behaviors.

Another important feature of dictionaries is that they are mutable data structures, which means that you can add, delete, and update their items. It’s worth noting that this also means that they can’t be used as keys to other dictionaries, as they are not hashable objects.

Note: Everything you’ve learned in this section is related to the core Python implementation, CPython.

Other Python implementations, like PyPy, IronPython or Jython, could exhibit different dictionary behaviors and features that are beyond the scope of this article.

How to Iterate Through a Dictionary in Python: The Basics

Dictionaries are an useful and widely used data structure in Python. As a Python coder, you’ll often be in situations where you’ll need to iterate through a dictionary in Python, while you perform some actions on its key-value pairs.

When it comes to iterating through a dictionary in Python, the language provides you with some great tools that we’ll cover in this article.

Iterating Through Keys Directly

Python’s dictionaries are mapping objects. This means that they inherit some special methods, which Python uses internally to perform some operations. These methods are named using the naming convention of adding a double underscore at the beginning of and at the end of the method’s name.

To visualize the methods and attributes of any Python object, you can use dir(), which is a built-in function that serves that purpose. If you run dir() with an empty dictionary as an argument, then you’ll be able to see all the methods and attributes that dictionaries implement:

>>>

>>> dir({})
['__class__', '__contains__', '__delattr__', ... , '__iter__', ...]

If you take a closer look at the previous output, you’ll see '__iter__'. This is a method that is called when an iterator is required for a container, and it should return a new iterator object that can iterate through all the objects in the container.

Note: The output of the previous code has been abbreviated (...) in order to save space.

For mappings (like dictionaries), .__iter__() should iterate over the keys. This means that if you put a dictionary directly into a for loop, Python will automatically call .__iter__() on that dictionary, and you’ll get an iterator over its keys:

>>>

>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> for key in a_dict:
...     print(key)
...
color
fruit
pet

Python is smart enough to know that a_dict is a dictionary and that it implements .__iter__(). In this example, Python called .__iter__() automatically, and this allowed you to iterate over the keys of a_dict.

This is the simplest way to iterate through a dictionary in Python. Just put it directly into a for loop, and you’re done!

If you use this approach along with a small trick, then you can process the keys and values of any dictionary. The trick consists of using the indexing operator [] with the dictionary and its keys to get access to the values:

>>>

>>> for key in a_dict:
...     print(key, '->', a_dict[key])
...
color -> blue
fruit -> apple
pet -> dog

The preceding code allowed you to get access to the keys (key) and the values (a_dict[key]) of a_dict at the same time. This way, you can do any operation with both the keys and the values.

Iterating Through `.items()`

When you’re working with dictionaries, it’s likely that you’ll want to work with both the keys and the values. One of the most useful ways to iterate through a dictionary in Python is by using .items(), which is a method that returns a new view of the dictionary’s items:

>>>

>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> d_items = a_dict.items()
>>> d_items  # Here d_items is a view of items
dict_items([('color', 'blue'), ('fruit', 'apple'), ('pet', 'dog')])

Dictionary views like d_items provide a dynamic view on the dictionary’s entries, which means that when the dictionary changes, the views reflect these changes.

Views can be iterated over to yield their respective data, so you can iterate through a dictionary in Python by using the view object returned by .items():

>>>

>>> for item in a_dict.items():
...     print(item)
...
('color', 'blue')
('fruit', 'apple')
('pet', 'dog')

The view object returned by .items() yields the key-value pairs one at a time and allows you to iterate through a dictionary in Python, but in such a way that you get access to the keys and values at the same time.

If you take a closer look at the individual items yielded by .items(), you’ll notice that they’re really tuple objects. Let’s take a look:

>>>

>>> for item in a_dict.items():
...     print(item)
...     print(type(item))
...
('color', 'blue')
<class 'tuple'>
('fruit', 'apple')
<class 'tuple'>
('pet', 'dog')
<class 'tuple'>

Once you know this, you can use tuple unpacking to iterate through the keys and values of the dictionary you are working with. To achieve this, you just need to unpack the elements of every item into two different variables representing the key and the value:

>>>

>>> for key, value in a_dict.items():
...     print(key, '->', value)
...
color -> blue
fruit -> apple
pet -> dog

Here, the variables key and value in the header of your for loop do the unpacking. Every time the loop runs, key will store the key, and value will store the value of the item that is been processed. This way, you’ll have more control over the items of the dictionary, and you’ll be able to process the keys and values separately and in a way that is more readable and Pythonic.

Note: Notice that .values() and .keys() return view objects just like .items(), as you’ll see in the next two sections.

Iterating Through `.keys()`

If you just need to work with the keys of a dictionary, then you can use .keys(), which is a method that returns a new view object containing the dictionary’s keys:

>>>

>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> keys = a_dict.keys()
>>> keys
dict_keys(['color', 'fruit', 'pet'])

The object returned by .keys() here provided a dynamic view on the keys of a_dict. This view can be used to iterate through the keys of a_dict.

To iterate through a dictionary in Python by using .keys(), you just need to call .keys() in the header of a for loop:

>>>

>>> for key in a_dict.keys():
...     print(key)
...
color
fruit
pet

When you call .keys() on a_dict, you get a view of keys. Python knows that view objects are iterables, so it starts looping, and you can process the keys of a_dict.

On the other hand, using the same trick you’ve seen before (indexing operator []), you can get access to the values of the dictionary:

>>>

>>> for key in a_dict.keys():
...     print(key, '->', a_dict[key])
...
color -> blue
fruit -> apple
pet -> dog

This way you’ve gotten access to the keys (key) and values (a_dict[key]) of a_dict at the same time, and you’ll be able to perform any action on them.

Iterating Through `.values()`

It’s also common to only use the values to iterate through a dictionary in Python. One way to do that is to use .values(), which returns a view with the values of the dictionary:

>>>

>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> values = a_dict.values()
>>> values
dict_values(['blue', 'apple', 'dog'])

In the previous code, values holds a reference to a view object containing the values of a_dict.

As any view object, the object returned by .values() can also be iterated over. In this case, .values() yields the values of a_dict:

>>>

>>> for value in a_dict.values():
...     print(value)
...
blue
apple
dog

Using .values(), you’ll be getting access to only the values of a_dict, without dealing with the keys.

It’s worth noting that they also support membership tests (in), which is an important feature if you’re trying to know if a specific element is in a dictionary or not:

>>>

>>> a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
>>> 'pet' in a_dict.keys()
True
>>> 'apple' in a_dict.values()
True
>>> 'onion' in a_dict.values()
False

The membership test using in returns True if the key (or value or item) is present in the dictionary you’re testing, and returns False otherwise. The membership test allows you to not iterate through a dictionary in Python if you just want to know if certain key (or value or item) is present in a dictionary or not.

Modifying Values and Keys

It can be pretty common to need to modify the values and keys when you’re iterating through a dictionary in Python. There are some points you’ll need to take into account to accomplish this task.

The values, for example, can be modified whenever you need, but you’ll need to use the original dictionary and the key that maps the value you want to modify:

>>>

>>> prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> for k, v in prices.items():
...     prices[k] = round(v * 0.9, 2)  # Apply a 10% discount
...
>>> prices
{'apple': 0.36, 'orange': 0.32, 'banana': 0.23}

In the previous code example, to modify the values of prices and apply a 10% discount, you used the expression prices[k] = round(v * 0.9, 2).

So why do you have to use the original dictionary if you have access to its key (k) and its values (v)? Should you be able to modify them directly?

The real problem is that k and v changes aren’t reflected in the original dictionary. That is, if you modify any of them (k or v) directly inside the loop, then what really happens is that you’ll lose the reference to the relevant dictionary component without changing anything in the dictionary.

On the other hand, the keys can be added or removed from a dictionary by converting the view returned by .keys() into a list object:

>>>

>>> prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> for key in list(prices.keys()):  # Use a list instead of a view
...     if key == 'orange':
...         del prices[key]  # Delete a key from prices
...
>>> prices
{'apple': 0.4, 'banana': 0.25}

This approach may have some performance implications, mainly related to memory consumption. For example, instead of a view object that yields elements on demand, you’ll have an entire new list in your system’s memory. However, this could be a safe way to modify the keys while you iterate through a dictionary in Python.

Finally, if you try to remove a key from prices by using .keys() directly, then Python will raise a RuntimeError telling you that the dictionary’s size has changed during iteration:

>>>

>>> # Python 3. dict.keys() returns a view object, not a list
>>> prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> for key in prices.keys():
...     if key == 'orange':
...         del prices[key]
...
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    for key in prices.keys():
RuntimeError: dictionary changed size during iteration

This is because .keys() returns a dictionary-view object, which yields keys on demand one at a time, and if you delete an item (del prices[key]), then Python raises a RuntimeError, because you’ve modified the dictionary during iteration.

Note: In Python 2, .items(), .keys(), and .values() return list objects. But .iteritems(), iterkeys(), and .itervalues() return iterators. So, if you’re using Python 2, then you can modify the dictionary’s keys by using .keys() directly.

On the other hand, if you’re using iterkeys() in your Python 2 code and you try to modify the keys of a dictionary, then you’ll get a RuntimeError.

Real-World Examples

So far, you’ve seen the more basic ways of iterating through a dictionary in Python. Now it’s time to see how you can perform some actions with the items of a dictionary during iteration. Let’s look at some real-world examples.

Note: Later on in this article, you’ll see another way of solving these very same problems by using other Python tools.

Turning Keys Into Values and Vice Versa

Suppose you have a dictionary and for some reason need to turn keys into values and vice versa. In this situation, you can use a for loop to iterate through the dictionary and build the new dictionary by using the keys as values and vice versa:

>>>

>>> a_dict = {'one': 1, 'two': 2, 'thee': 3, 'four': 4}
>>> new_dict = {}
>>> for key, value in a_dict.items():
...     new_dict[value] = key
...
>>> new_dict
{1: 'one', 2: 'two', 3: 'thee', 4: 'four'}

The expression new_dict[value] = key did all the work for you by turning the keys into values and using the values as keys. For this code to work, the data stored in the original values must be of a hashable data type.

Filtering Items

Sometimes you’ll be in situations where you have a dictionary and you want to create a new one to store only the data that satisfies a given condition. You can do this with an if statement inside a for loop as follows:

>>>

>>> a_dict = {'one': 1, 'two': 2, 'thee': 3, 'four': 4}
>>> new_dict = {}  # Create a new empty dictionary
>>> for key, value in a_dict.items():
...     # If value satisfies the condition, then store it in new_dict
...     if value <= 2:
...         new_dict[key] = value
...
>>> new_dict
{'one': 1, 'two': 2}

In this example, you’ve filtered out the items with a value greater than 2. Now new_dict only contains the items that satisfy the condition value <= 2. This is one possible solution for this kind of problem. Later on, you’ll see a more Pythonic and readable way to get the same result.

Doing Some Calculations

It’s also common to need to do some calculations while you iterate through a dictionary in Python. Suppose you’ve stored the data for your company’s sales in a dictionary, and now you want to know the total income of the year.

To solve this problem you could define a variable with an initial value of zero. Then, you can accumulate every value of your dictionary in that variable:

>>>

>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> total_income = 0.00
>>> for value in incomes.values():
...     total_income += value  # Accumulate the values in total_income
...
>>> total_income
14100.0

Here, you’ve iterated through incomes and sequentially accumulated its values in total_income as you wanted to do. The expression total_income += value does the magic, and at the end of the loop, you’ll get the total income of the year. Note that total_income += value is equivalent to total_income = total_income + value.

Using Comprehensions

A dictionary comprehension is a compact way to process all or part of the elements in a collection and return a dictionary as a results. In contrast to list comprehensions, they need two expressions separated with a colon followed by for and if (optional) clauses. When a dictionary comprehension is run, the resulting key-value pairs are inserted into a new dictionary in the same order in which they were produced.

Suppose, for example, that you have two lists of data, and you need to create a new dictionary from them. In this case, you can use Python’s zip(*iterables) to loop over the elements of both lists in pairs:

>>>

>>> objects = ['blue', 'apple', 'dog']
>>> categories = ['color', 'fruit', 'pet']
>>> a_dict = {key: value for key, value in zip(categories, objects)}
>>> a_dict
{'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}

Here, zip() receives two iterables (categories and objects) as arguments and makes an iterator that aggregates elements from each iterable. The tuple objects generated by zip() are then unpacked into key and value, which are finally used to create the new dictionary.

Dictionary comprehensions open up a wide spectrum of new possibilities and provide you with a great tool to iterate through a dictionary in Python.

Turning Keys Into Values and Vice Versa: Revisited

If you take another look at the problem of turning keys into values and vice versa, you’ll see that you could write a more Pythonic and efficient solution by using a dictionary comprehension:

>>>

>>> a_dict = {'one': 1, 'two': 2, 'thee': 3, 'four': 4}
>>> new_dict = {value: key for key, value in a_dict.items()}
>>> new_dict
{1: 'one', 2: 'two', 3: 'thee', 4: 'four'}

With this dictionary comprehension, you’ve created a totally new dictionary where the keys have taken the place of the values and vice versa. This new approach gave you the ability to write more readable, succinct, efficient, and Pythonic code.

The condition for this code to work is the same one you saw before: the values must be hashable objects. Otherwise, you won’t be able to use them as keys for a_dict.

Filtering Items: Revisited

To filter the items in a dictionary with a comprehension, you just need to add an if clause that defines the condition you want to meet. In the previous example where you filtered a dictionary, that condition was if v <= 2. With this if clause added to the end of the dictionary comprehension, you’ll filter out the items whose values are greater than 2. Let’s take a look:

>>>

>>> a_dict = {'one': 1, 'two': 2, 'thee': 3, 'four': 4}
>>> new_dict = {k: v for k, v in a_dict.items() if v <= 2}
>>> new_dict
{'one': 1, 'two': 2}

Now new_dict contains only the items that satisfy your condition. Compared to the previous solutions, this one is more Pythonic and efficient.

Doing Some Calculations: Revisited

Remember the example with the company’s sales? If you use a list comprehension to iterate through the dictionary’s values, then you’ll get code that is more compact, fast, and Pythonic:

>>>

>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> total_income = sum([value for value in incomes.values()])
>>> total_income
14100.0

The list comprehension created a list object containing the values of incomes, and then you summed up all of them by using sum() and stored the result in total_income.

If you’re working with a really large dictionary, and memory usage is a problem for you, then you can use a generator expression instead of a list comprehension. A generator expression is an expression that returns an iterator. It looks like a list comprehension, but instead of brackets you need to use parentheses to define it:

>>>

>>> total_income = sum(value for value in incomes.values())
>>> total_income
14100.0

If you change the square brackets for a pair of parentheses (the parentheses of sum() here), you’ll be turning the list comprehension into a generator expression, and your code will be memory efficient, because generator expressions yield elements on demand. Instead of creating and storing the whole list in memory, you’ll only have to store one element at a time.

Note: If you are totally new to generator expressions, you can take a look at Introduction to Python Generators to get a better understanding of the topic.

Finally, there is a simpler way to solve this problem by just using incomes.values() directly as an argument to sum():

>>>

>>> total_income = sum(incomes.values())
>>> total_income
14100.0

sum() receives an iterable as an argument and returns the total sum of its elements. Here, incomes.values() plays the role of the iterable passed to sum(). The result is the total income you were looking for.

Removing Specific Items

Now, suppose you have a dictionary and need to create a new one with selected keys removed. Remember how key-view objects are like sets? Well, these similarities go beyond just being collections of hashable and unique objects. Key-view objects also support common set operations. Let’s see how you can take advantage of this to remove specific items in a dictionary:

>>>

>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> non_citric = {k: incomes[k] for k in incomes.keys() - {'orange'}}
>>> non_citric
{'apple': 5600.0, 'banana': 5000.0}

This code works because key-view objects support set operations like unions, intersections, and differences. When you wrote incomes.keys() - {'orange'} inside the dictionary comprehension, you were really doing a set difference operation. If you need to perform any set operations with the keys of a dictionary, then you can just use the key-view object directly without first converting it into a set. This is a little-known feature of key-view objects that can be useful in some situations.

Sorting a Dictionary

It’s often necessary to sort the elements of a collection. Since Python 3.6, dictionaries are ordered data structures, so if you use Python 3.6 (and beyond), you’ll be able to sort the items of any dictionary by using sorted() and with the help of a dictionary comprehension:

>>>

>>> # Python 3.6, and beyond
>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> sorted_income = {k: incomes[k] for k in sorted(incomes)}
>>> sorted_income
{'apple': 5600.0, 'banana': 5000.0, 'orange': 3500.0}

This code allows you to create a new dictionary with its keys in sorted order. This is possible because sorted(incomes) returns a list of sorted keys that you can use to generate the new dictionary sorted_dict.

Iterating in Sorted Order

Sometimes you may need to iterate through a dictionary in Python but want to do it in sorted order. This can be achieved by using sorted(). When you call sorted(iterable), you get a list with the elements of iterable in sorted order.

Let’s see how you can use sorted() to iterate through a dictionary in Python when you need to do it in sorted order.

Sorted by Keys

If you need to iterate through a dictionary in Python and want it to be sorted by keys, then you can use your dictionary as an argument to sorted(). This will return a list containing the keys in sorted order, and you’ll be able to iterate through them:

>>>

>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> for key in sorted(incomes):
...     print(key, '->', incomes[key])
...
apple -> 5600.0
banana -> 5000.0
orange -> 3500.0

In this example, you sorted the dictionary (alphabetically) by keys using sorted(incomes) in the header of the for loop. Notice that you can also use sorted(incomes.keys()) to get the same result. In both cases, you’ll get a list containing the keys of your dictionary in sorted order.

Note: The sorting order will depend on the data type you are using for keys or values and the internal rules that Python uses to sort those data types.

Sorted by Values

You could also need to iterate through a dictionary in Python with its items sorted by values. You can use sorted() too, but with a second argument called key.

The key keyword argument specifies a function of one argument that is used to extract a comparison key from each element you’re processing.

To sort the items of a dictionary by values, you can write a function that returns the value of each item and use this function as the key argument to sorted():

>>>

>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> def by_value(item):
...     return item[1]
...
>>> for k, v in sorted(incomes.items(), key=by_value):
...     print(k, '->', v)
...
('orange', '->', 3500.0)
('banana', '->', 5000.0)
('apple', '->', 5600.0)

In this example, you defined by_value() and used it to sort the items of incomes by value. Then you iterated through the dictionary in sorted order by using sorted(). The key function (by_value()) tells sorted() to sort incomes.items() by the second element of each item, that is, by the value (item[1]).

You may also just want to iterate through the values of a dictionary in sorted order, without worrying about the keys. In that case, you can use .values() as follows:

>>>

>>> for value in sorted(incomes.values()):
...     print(value)
...
3500.0
5000.0
5600.0

sorted(incomes.values()) returned the values of the dictionary in sorted order as you desired. The keys won’t be accessible if you use incomes.values(), but sometimes you don’t really need the keys, just the values, and this is a fast way to get access to them.

Reversed

If you need to sort your dictionaries in reverse order, you can add reverse=True as an argument to sorted(). The keyword argument reverse should take a boolean value. If it’s set to True, then the elements are sorted in reverse order:

>>>

>>> incomes = {'apple': 5600.00, 'orange': 3500.00, 'banana': 5000.00}
>>> for key in sorted(incomes, reverse=True):
...     print(key, '->', incomes[key])
...
orange -> 3500.0
banana -> 5000.0
apple -> 5600.0

Here, you iterated over the keys of incomes in reverse order by using sorted(incomes, reverse=True) in the header of the for loop.

Finally, it’s important to note that sorted() doesn’t really modify the order of the underlying dictionary. What really happen is that sorted() creates an independent list with its element in sorted order, so incomes remains the same:

>>>

>>> incomes
{'apple': 5600.0, 'orange': 3500.0, 'banana': 5000.0}

This code shows you that incomes didn’t change. sorted() didn’t modify incomes. It just created a new sorted list from the keys of incomes.

Iterating Destructively With `.popitem`()

Sometimes you need to iterate through a dictionary in Python and delete its items sequentially. To accomplish this task, you can use .popitem(), which will remove and return an arbitrary key-value pair from a dictionary. On the other hand, when you call .popitem() on an empty dictionary, it raises a KeyError.

If you really need to destructively iterate through a dictionary in Python, then .popitem() can be useful. Here’s an example:

 1 # File: dict_popitem.py
 2 
 3 a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
 4 
 5 while True:
 6     try:
 7         print(f'Dictionary length: {len(a_dict)}')
 8         item = a_dict.popitem()
 9         # Do something with item here...
10         print(f'{item} removed')
11     except KeyError:
12         print('The dictionary has no item now...')
13         break

Here, you used a while loop instead of a for loop. The reason for this is that it’s never safe to iterate through a dictionary in Python if you pretend to modify it this way, that is, if you’re deleting or adding items to it.

Inside the while loop, you defined a try...except block to catch the KeyError raised by .popitems() when a_dict turns empty. In the try...except block, you process the dictionary, removing an item in each iteration. The variable item keeps a reference to the successive items and allows you to do some actions with them.

Note: In the previous code example, you used Python’s f-strings for string formatting. If you want to dive deeper into f-strings, then you can take a look at Python 3’s f-Strings: An Improved String Formatting Syntax (Guide).

If you run this script from your command-line, then you’ll get the following results:

$ python3 dict_popitem.py
Dictionary length: 3
('pet', 'dog') removed
Dictionary length: 2
('fruit', 'apple') removed
Dictionary length: 1
('color', 'blue') removed
Dictionary length: 0
The dictionary has no item now...

Here .popitem() sequentially removed the items of a_dict. The loop broke when the dictionary became empty, and .popitem() raised a KeyError exception.

Using Some of Python’s Built-In Functions

Python provides some built-in functions that could be useful when you’re working with collections, like dictionaries. These functions are a sort of iteration tool that provides you with another way of iterating through a dictionary in Python. Let’s see some of them.

`map()`

Python’s map() is defined as map(function, iterable, ...) and returns an iterator that applies function to every item of iterable, yielding the results on demand. So, map() could be viewed as an iteration tool that you can use to iterate through a dictionary in Python.

Suppose you have a dictionary containing the prices of a bunch of products, and you need to apply a discount to them. In this case, you can define a function that manages the discount and then uses it as the first argument to map(). The second argument can be prices.items():

>>>

>>> prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> def discount(current_price):
...     return (current_price[0], round(current_price[1] * 0.95, 2))
...
>>> new_prices = dict(map(discount, prices.items()))
>>> new_prices
{'apple': 0.38, 'orange': 0.33, 'banana': 0.24}

Here, map() iterated through the items of the dictionary (prices.items()) to apply a 5% discount to each fruit by using discount(). In this case, you need to use dict() to generate the new_prices dictionary from the iterator returned by map().

Note that discount() returns a tuple of the form (key, value), where current_price[0] represents the key and round(current_price[1] * 0.95, 2) represents the new value.

`filter()`

filter() is another built-in function that you can use to iterate through a dictionary in Python and filter out some of its items. This function is defined as filter(function, iterable) and returns an iterator from those elements of iterable for which function returns True.

Suppose you want to know the products with a price lower than 0.40. You need to define a function to determine if the price satisfies that condition and pass it as first argument to filter(). The second argument can be prices.keys():

>>>

>>> prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> def has_low_price(price):
...     return prices[price] < 0.4
...
>>> low_price = list(filter(has_low_price, prices.keys()))
>>> low_price
['orange', 'banana']

Here, you iterated through the keys of prices with filter(). Then filter() applies has_low_price() to every key of prices. Finally, you need to use list() to generate the list of products with a low price, because filter() returns an iterator, and you really need a list object.

Using `collections.ChainMap`

collections is a useful module from the Python Standard Library that provides specialized container data types. One of these data types is ChainMap, which is a dictionary-like class for creating a single view of multiple mappings (like dictionaries). With ChainMap, you can group multiple dictionaries together to create a single, updateable view.

Now, suppose you have two (or more) dictionaries, and you need to iterate through them together as one. To achieve this, you can create a ChainMap object and initialize it with your dictionaries:

>>>

>>> from collections import ChainMap
>>> fruit_prices = {'apple': 0.40, 'orange': 0.35}
>>> vegetable_prices = {'pepper': 0.20, 'onion': 0.55}
>>> chained_dict = ChainMap(fruit_prices, vegetable_prices)
>>> chained_dict  # A ChainMap object
ChainMap({'apple': 0.4, 'orange': 0.35}, {'pepper': 0.2, 'onion': 0.55})
>>> for key in chained_dict:
...     print(key, '->', chained_dict[key])
...
pepper -> 0.2
orange -> 0.35
onion -> 0.55
apple -> 0.4

After importing ChainMap from collections, you need to create a ChainMap object with the dictionaries you want to chain, and then you can freely iterate through the resulting object as you would do with a regular dictionary.

ChainMap objects also implement .keys(), values(), and .items() as a standard dictionary does, so you can use these methods to iterate through the dictionary-like object generated by ChainMap, just like you would do with a regular dictionary:

>>>

>>> for key, value in chained_dict.items():
...     print(key, '->', value)
...
apple -> 0.4
pepper -> 0.2
orange -> 0.35
onion -> 0.55

In this case, you’ve called .items() on a ChainMap object. The ChainMap object behaved as if it were a regular dictionary, and .items() returned a dictionary view object that can be iterated over as usual.

Using `itertools`

Python’s itertools is a module that provides some useful tools to perform iteration tasks. Let’s see how you can use some of them to iterate through a dictionary in Python.

Cyclic Iteration With `cycle()`

Suppose you want to iterate through a dictionary in Python, but you need to iterate through it repeatedly in a single loop. To get this task done, you can use itertools.cycle(iterable), which makes an iterator returning elements from iterable and saving a copy of each. When iterable is exhausted, cycle() returns elements from the saved copy. This is performed in cyclic fashion, so it’s up to you to stop the cycle.

In the following example, you’ll be iterating through the items of a dictionary three consecutive times:

>>>

>>> from itertools import cycle
>>> prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> times = 3  # Define how many times you need to iterate through prices
>>> total_items = times * len(prices)
>>> for item in cycle(prices.items()):
...     if not total_items:
...         break
...     total_items -= 1
...     print(item)
...
('apple', 0.4)
('orange', 0.35)
('banana', 0.25)
('apple', 0.4)
('orange', 0.35)
('banana', 0.25)
('apple', 0.4)
('orange', 0.35)
('banana', 0.25)

The preceding code allowed you to iterate through prices a given number of times (3 in this case). This cycle could be as long as you need, but you are responsible for stopping it. The if condition breaks the cycle when total_items counts down to zero.

Chained Iteration With `chain()`

itertools also provides chain(*iterables), which gets some iterables as arguments and makes an iterator that yields elements from the first iterable until it’s exhausted, then iterates over the next iterable and so on, until all of them are exhausted.

This allows you to iterate through multiple dictionaries in a chain, like to what you did with collections.ChainMap:

>>>

>>> from itertools import chain
>>> fruit_prices = {'apple': 0.40, 'orange': 0.35, 'banana': 0.25}
>>> vegetable_prices = {'pepper': 0.20, 'onion': 0.55, 'tomato': 0.42}
>>> for item in chain(fruit_prices.items(), vegetable_prices.items()):
...     print(item)
...
('apple', 0.4)
('orange', 0.35)
('banana', 0.25)
('pepper', 0.2)
('onion', 0.55)
('tomato', 0.42)

In the above code, chain() returned an iterable that combined the items from fruit_prices and vegetable_prices.

It’s also possible to use .keys() or .values(), depending on your needs, with the condition of being homogeneous: if you use .keys() for an argument to chain(), then you need to use .keys() for the rest of them.

Using the Dictionary Unpacking Operator (`**`)

Python 3.5 brings a new and interesting feature. PEP 448 - Additional Unpacking Generalizations can make your life easier when it comes to iterating through multiple dictionaries in Python. Let’s see how this works with a short example.

Suppose you have two (or more) dictionaries, and you need to iterate through them together, without using collections.ChainMap or itertools.chain(), as you’ve seen in the previous sections. In this case, you can use the dictionary unpacking operator (**) to merge the two dictionaries into a new one and then iterate through it:

>>>

>>> fruit_prices = {'apple': 0.40, 'orange': 0.35}
>>> vegetable_prices = {'pepper': 0.20, 'onion': 0.55}
>>> # How to use the unpacking operator **
>>> {**vegetable_prices, **fruit_prices}
{'pepper': 0.2, 'onion': 0.55, 'apple': 0.4, 'orange': 0.35}
>>> # You can use this feature to iterate through multiple dictionaries
>>> for k, v in {**vegetable_prices, **fruit_prices}.items():
...     print(k, '->', v)
...
pepper -> 0.2
onion -> 0.55
apple -> 0.4
orange -> 0.35

The dictionary unpacking operator (**) is really an awesome feature in Python. It allows you to merge multiple dictionaries into a new one, as you did in the example with vegetable_prices and fruit_prices. Once you’ve merged the dictionaries with the unpacking operator, you can iterate through the new dictionary as usual.

It’s important to note that if the dictionaries you’re trying to merge have repeated or common keys, then the values of the right-most dictionary will prevail:

>>>

>>> vegetable_prices = {'pepper': 0.20, 'onion': 0.55}
>>> fruit_prices = {'apple': 0.40, 'orange': 0.35, 'pepper': .25}
>>> {**vegetable_prices, **fruit_prices}
{'pepper': 0.25, 'onion': 0.55, 'apple': 0.4, 'orange': 0.35}

The pepper key is present in both dictionaries. After you merge them, the fruit_prices value for pepper (0.25) prevailed, because fruit_prices is the right-most dictionary.

Conclusion

You now know the basics of how to iterate through a dictionary in Python, as well as some more advanced techniques and strategies!

You’ve learned:

What dictionaries are, as well as some of their main features and implementation details
What the basic ways to iterate through a dictionary in Python are
What kind of tasks you can accomplish by iterating through a dictionary in Python
How to use some more elaborated techniques and strategies to iterate through a dictionary in Python

You have the tools and knowledge you’ll need to get the most out of dictionaries in Python. This will help you be more efficient and effective in your use of dictionary iteration in the future.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Logging: A Stroll Through the Source Code

2019-05-22T14:00:00+00:00

The Python logging package is a a lightweight but extensible package for keeping better track of what your own code does. Using it gives you much more flexibility than just littering your code with superfluous print() calls.

However, Python’s logging package can be complicated in certain spots. Handlers, loggers, levels, namespaces, filters: it’s not easy to keep track of all of these pieces and how they interact.

One way to tie up the loose ends in your understanding of logging is to peek under the hood to its CPython source code. The Python code behind logging is concise and modular, and reading through it can help you get that aha moment.

This article is meant to complement the logging HOWTO document as well as Logging in Python, which is a walkthrough on how to use the package.

By the end of this article, you’ll be familiar with the following:

logging levels and how they work
Thread-safety versus process-safety in logging
The design of logging from an OOP perspective
Logging in libraries vs applications
Best practices and design patterns for using logging

For the most part, we’ll go line-by-line down the core module in Python’s logging package in order to build a picture of how it’s laid out.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

How to Follow Along

Because the logging source code is central to this article, you can assume that any code block or link is based on a specific commit in the Python 3.7 CPython repository, namely commit d730719. You can find the logging package itself in the Lib/ directory within the CPython source.

Within the logging package, most of the heavy lifting occurs within logging/__init__.py, which is the file you’ll spend the most time on here:

cpython/
│
├── Lib/
│   ├── logging/
│   │   ├── __init__.py
│   │   ├── config.py
│   │   └── handlers.py
│   ├── ...
├── Modules/
├── Include/
...
... [truncated]

With that, let’s jump in.

Preliminaries

Before we get to the heavyweight classes, the top hundred lines or so of __init__.py introduce a few subtle but important concepts.

Preliminary #1: A Level Is Just an `int`!

Objects like logging.INFO or logging.DEBUG can seem a bit opaque. What are these variables internally, and how are they defined?

In fact, the uppercase constants from Python’s logging are just integers, forming an enum-like collection of numerical levels:

CRITICAL = 50
FATAL = CRITICAL
ERROR = 40
WARNING = 30
WARN = WARNING
INFO = 20
DEBUG = 10
NOTSET = 0

Why not just use the strings "INFO" or "DEBUG"? Levels are int constants to allow for the simple, unambiguous comparison of one level with another. They are given names as well to lend them semantic meaning. Saying that a message has a severity of 50 may not be immediately clear, but saying that it has a level of CRITICAL lets you know that you’ve got a flashing red light somewhere in your program.

Now, technically, you can pass just the str form of a level in some places, such as logger.setLevel("DEBUG"). Internally, this will call _checkLevel(), which ultimately does a dict lookup for the corresponding int:

_nameToLevel = {
    'CRITICAL': CRITICAL,
    'FATAL': FATAL,
    'ERROR': ERROR,
    'WARN': WARNING,
    'WARNING': WARNING,
    'INFO': INFO,
    'DEBUG': DEBUG,
    'NOTSET': NOTSET,
}

def _checkLevel(level):
    if isinstance(level, int):
        rv = level
    elif str(level) == level:
        if level not in _nameToLevel:
            raise ValueError("Unknown level: %r" % level)
        rv = _nameToLevel[level]
    else:
        raise TypeError("Level not an integer or a valid string: %r" % level)
    return rv

Which should you prefer? I’m not too opinionated on this, but it’s notable that the logging docs consistently use the form logging.DEBUG rather than "DEBUG" or 10. Also, passing the str form isn’t an option in Python 2, and some logging methods such as logger.isEnabledFor() will accept only an int, not its str cousin.

Preliminary #2: Logging Is Thread-Safe, but Not Process-Safe

A few lines down, you’ll find the following short code block, which is sneakily critical to the whole package:

import threading

_lock = threading.RLock()

def _acquireLock():
     if _lock:
        _lock.acquire()

def _releaseLock():
    if _lock:
        _lock.release()

The _lock object is a reentrant lock that sits in the global namespace of the logging/__init__.py module. It makes pretty much every object and operation in the entire logging package thread-safe, enabling threads to do read and write operations without the threat of a race condition. You can see in the module source code that _acquireLock() and _releaseLock() are ubiquitous to the module and its classes.

There’s something not accounted for here, though: what about process safety? The short answer is that the logging module is not process safe. This isn’t inherently a fault of logging—generally, two processes can’t write to same file without a lot of proactive effort on behalf of the programmer first.

This means that you’ll want to be careful before using classes such as a logging.FileHandler with multiprocessing involved. If two processes want to read from and write to the same underlying file concurrently, then you can run into a nasty bug halfway through a long-running routine.

If you want to get around this limitation, there’s a thorough recipe in the official Logging Cookbook. Because this entails a decent amount of setup, one alternative is to have each process log to a separate file based on its process ID, which you can grab with os.getpid().

Package Architecture: Logging’s MRO

Now that we’ve covered some preliminary setup code, let’s take a high-level look at how logging is laid out. The logging package uses a healthy dose of OOP and inheritance. Here’s a partial look at the method resolution order (MRO) for some of the most important classes in the package:

object
│
├── LogRecord
├── Filterer
│   ├── Logger
│   │   └── RootLogger
│   └── Handler
│       ├── StreamHandler
│       └── NullHandler
├── Filter
└── Manager

The tree diagram above doesn’t cover all of the classes in the module, just those that are most worth highlighting.

Note: You can use the dunder attribute logging.StreamHandler.__mro__ to see the chain of inheritance. A definitive guide to the MRO can be found in the Python 2 docs, though it is applicable to Python 3 as well.

This litany of classes is typically one source of confusion because there’s a lot going on, and it’s all jargon-heavy. Filter versus Filterer? Logger versus Handler? It can be challenging to keep track of everything, much less visualize how it fits together. A picture is worth a thousand words, so here’s a diagram of a scenario where one logger with two handlers attached to it writes a log message with level logging.INFO:

Flow of logging objects (Image: Real Python)

In Python code, everything above would look like this:

import logging
import sys

logger = logging.getLogger("pylog")
logger.setLevel(logging.DEBUG)
h1 = logging.FileHandler(filename="/tmp/records.log")
h1.setLevel(logging.INFO)
h2 = logging.StreamHandler(sys.stderr)
h2.setLevel(logging.ERROR)
logger.addHandler(h1)
logger.addHandler(h2)
logger.info("testing %d.. %d.. %d..", 1, 2, 3)

There’s a more detailed map of this flow in the Logging HOWTO. What’s shown above is a simplified scenario.

Your code defines just one Logger instance, logger, along with two Handler instances, h1 and h2.

When you call logger.info("testing %d.. %d.. %d..", 1, 2, 3), the logger object serves as a filter because it also has a level associated with it. Only if the message level is severe enough will the logger do anything with the message. Because the logger has level DEBUG, and the message carries a higher INFO level, it gets the go-ahead to move on.

Internally, logger calls logger.makeRecord() to put the message string "testing %d.. %d.. %d.." and its arguments (1, 2, 3) into a bona fide class instance of a LogRecord, which is just a container for the message and its metadata.

The logger object looks around for its handlers (instances of Handler), which may be tied directly to logger itself or to its parents (a concept that we’ll touch on later). In this example, it finds two handlers:

One with level INFO that dumps log data to a file at /tmp/records.log
One that writes to sys.stderr but only if the incoming message is at level ERROR or higher

At this point, there’s another round of tests that kicks in. Because the LogRecord and its message only carry level INFO, the record gets written to Handler 1 (green arrow), but not to Handler 2’s stderr stream (red arrow). For Handlers, writing the LogRecord to their stream is called emitting it, which is captured in their .emit().

Next, let’s further dissect everything from above.

The `LogRecord` Class

What is a LogRecord? When you log a message, an instance of the LogRecord class is the object you send to be logged. It’s created for you by a Logger instance and encapsulates all the pertinent info about that event. Internally, it’s little more than a wrapper around a dict that contains attributes for the record. A Logger instance sends a LogRecord instance to zero or more Handler instances.

The LogRecord contains some metadata, such as the following:

A name
The creation time as a Unix timestamp
The message itself
Information on what function made the logging call

Here’s a peek into the metadata that it carries with it, which you can introspect by stepping through a logging.error() call with the pdb module:

>>>

>>> import logging
>>> import pdb

>>> def f(x):
...     logging.error("bad vibes")
...     return x / 0
... 
>>> pdb.run("f(1)")

After stepping through some higher-level functions, you end up at line 1517:

(Pdb) l
1514                     exc_info = (type(exc_info), exc_info, exc_info.__traceback__)
1515                 elif not isinstance(exc_info, tuple):
1516                     exc_info = sys.exc_info()
1517             record = self.makeRecord(self.name, level, fn, lno, msg, args,
1518                                      exc_info, func, extra, sinfo)
1519 ->          self.handle(record)
1520
1521         def handle(self, record):
1522             """
1523             Call the handlers for the specified record.
1524
(Pdb) from pprint import pprint
(Pdb) pprint(vars(record))
{'args': (),
 'created': 1550671851.660067,
 'exc_info': None,
 'exc_text': None,
 'filename': '<stdin>',
 'funcName': 'f',
 'levelname': 'ERROR',
 'levelno': 40,
 'lineno': 2,
 'module': '<stdin>',
 'msecs': 660.067081451416,
 'msg': 'bad vibes',
 'name': 'root',
 'pathname': '<stdin>',
 'process': 2360,
 'processName': 'MainProcess',
 'relativeCreated': 295145.5490589142,
 'stack_info': None,
 'thread': 4372293056,
 'threadName': 'MainThread'}

A LogRecord, internally, contains a trove of metadata that’s used in one way or another.

You’ll rarely need to deal with a LogRecord directly, since the Logger and Handler do this for you. It’s still worthwhile to know what information is wrapped up in a LogRecord, because this is where all that useful info, like the timestamp, come from when you see record log messages.

Note: Below the LogRecord class, you’ll also find the setLogRecordFactory(), getLogRecordFactory(), and makeLogRecord() factory functions. You won’t need these unless you want to use a custom class instead of LogRecord to encapsulate log messages and their metadata.

The `Logger` and `Handler` Classes

The Logger and Handler classes are both central to how logging works, and they interact with each other frequently. A Logger, a Handler, and a LogRecord each have a .level associated with them.

The Logger takes the LogRecord and passes it off to the Handler, but only if the effective level of the LogRecord is equal to or higher than that of the Logger. The same goes for the LogRecord versus Handler test. This is called level-based filtering, which Logger and Handler implement in slightly different ways.

In other words, there is an (at least) two-step test applied before the message that you log gets to go anywhere. In order to be fully passed from a logger to handler and then logged to the end stream (which could be sys.stdout, a file, or an email via SMTP), a LogRecord must have a level at least as high as both the logger and handler.

PEP 282 describes how this works:

Each Logger object keeps track of a log level (or threshold) that it is interested in, and discards log requests below that level. (Source)

So where does this level-based filtering actually occur for both Logger and Handler?

For the Logger class, it’s a reasonable first assumption that the logger would compare its .level attribute to the level of the LogRecord, and be done there. However, it’s slightly more involved than that.

Level-based filtering for loggers occurs in .isEnabledFor(), which in turn calls .getEffectiveLevel(). Always use logger.getEffectiveLevel() rather than just consulting logger.level. The reason has to do with the organization of Logger objects in a hierarchical namespace. (You’ll see more on this later.)

By default, a Logger instance has a level of 0 (NOTSET). However, loggers also have parent loggers, one of which is the root logger, which functions as the parent of all other loggers. A Logger will walk upwards in its hierarchy and get its effective level vis-à-vis its parent (which ultimately may be root if no other parents are found).

Here’s where this happens in the Logger class:

class Logger(Filterer):
    # ...
    def getEffectiveLevel(self):
        logger = self
        while logger:
            if logger.level:
                return logger.level
            logger = logger.parent
        return NOTSET

    def isEnabledFor(self, level):
        try:
            return self._cache[level]
        except KeyError:
            _acquireLock()
            if self.manager.disable >= level:
                is_enabled = self._cache[level] = False
            else:
                is_enabled = self._cache[level] = level >= self.getEffectiveLevel()
            _releaseLock()
        return is_enabled

Correspondingly, here’s an example that calls the source code you see above:

>>>

>>> import logging
>>> logger = logging.getLogger("app")
>>> logger.level  # No!
0
>>> logger.getEffectiveLevel()
30
>>> logger.parent
<RootLogger root (WARNING)>
>>> logger.parent.level
30

Here’s the takeaway: don’t rely on .level. If you haven’t explicitly set a level on your logger object, and you’re depending on .level for some reason, then your logging setup will likely behave differently than you expected it to.

What about Handler? For handlers, the level-to-level comparison is simpler, though it actually happens in .callHandlers() from the Logger class:

class Logger(Filterer):
    # ...
    def callHandlers(self, record):
        c = self
        found = 0
        while c:
            for hdlr in c.handlers:
                found = found + 1
                if record.levelno >= hdlr.level:
                    hdlr.handle(record)

For a given LogRecord instance (named record in the source code above), a logger checks with each of its registered handlers and does a quick check on the .level attribute of that Handler instance. If the .levelno of the LogRecord is greater than or equal to that of the handler, only then does the record get passed on. A docstring in logging refers to this as “conditionally emit[ting] the specified logging record.”

The most important attribute for a Handler subclass instance is its .stream attribute. This is the final destination where logs get written to and can be pretty much any file-like object. Here’s an example with io.StringIO, which is an in-memory stream (buffer) for text I/O.

First, set up a Logger instance with a level of DEBUG. You’ll see that, by default, it has no direct handlers:

>>>

>>> import io
>>> import logging
>>> logger = logging.getLogger("abc")
>>> logger.setLevel(logging.DEBUG)
>>> print(logger.handlers)
[]

Next, you can subclass logging.StreamHandler to make the .flush() call a no-op. We would want to flush sys.stderr or sys.stdout, but not the in-memory buffer in this case:

class IOHandler(logging.StreamHandler):
    def flush(self):
        pass  # No-op

Now, declare the buffer object itself and tie it in as the .stream for your custom handler with a level of INFO, and then tie that handler into the logger:

>>>

>>> stream = io.StringIO()
>>> h = IOHandler(stream)
>>> h.setLevel(logging.INFO)
>>> logger.addHandler(h)

>>> logger.debug("extraneous info")
>>> logger.warning("you've been warned")
>>> logger.critical("SOS")

>>> try:
...     print(stream.getvalue())
... finally:
...     stream.close()
... 
you've been warned
SOS

This last chunk is another illustration of level-based filtering.

Three messages with levels DEBUG, WARNING, and CRITICAL are passed through the chain. At first, it may look as if they don’t go anywhere, but two of them do. All three of them make it out of the gates from logger (which has level DEBUG).

However, only two of them get emitted by the handler because it has a higher level of INFO, which exceeds DEBUG. Finally, you get the entire contents of the buffer as a str and close the buffer to explicitly free up system resources.

The `Filter` and `Filterer` Classes

Above, we asked the question, “Where does level-based filtering happen?” In answering this question, it’s easy to get distracted by the Filter and Filterer classes. Paradoxically, level-based filtering for Logger and Handler instances occurs without the help of either of the Filter or Filterer classes.

Filter and Filterer are designed to let you add additional function-based filters on top of the level-based filtering that is done by default. I like to think of it as à la carte filtering.

Filterer is the base class for Logger and Handler because both of these classes are eligible for receiving additional custom filters that you specify. You add instances of Filter to them with logger.addFilter() or handler.addFilter(), which is what self.filters refers to in the following method:

class Filterer(object):
    # ...
    def filter(self, record):
        rv = True
        for f in self.filters:
            if hasattr(f, 'filter'):
                result = f.filter(record)
            else:
                result = f(record)
            if not result:
                rv = False
                break
        return rv

Given a record (which is a LogRecord instance), .filter() returns True or False depending on whether that record gets the okay from this class’s filters.

Here is .handle() in turn, for the Logger and Handler classes:

class Logger(Filterer):
    # ...
    def handle(self, record):
        if (not self.disabled) and self.filter(record):
            self.callHandlers(record)

# ...

class Handler(Filterer):
    # ...
    def handle(self, record):
        rv = self.filter(record)
        if rv:
            self.acquire()
            try:
                self.emit(record)
            finally:
                self.release()
        return rv

Neither Logger nor Handler come with any additional filters by default, but here’s a quick example of how you could add one:

>>>

>>> import logging

>>> logger = logging.getLogger("rp")
>>> logger.setLevel(logging.INFO)
>>> logger.addHandler(logging.StreamHandler())
>>> logger.filters  # Initially empty
[]
>>> class ShortMsgFilter(logging.Filter):
...     """Only allow records that contain long messages (> 25 chars)."""
...     def filter(self, record):
...         msg = record.msg
...         if isinstance(msg, str):
...             return len(msg) > 25
...         return False
... 
>>> logger.addFilter(ShortMsgFilter())
>>> logger.filters
[<__main__.ShortMsgFilter object at 0x10c28b208>]
>>> logger.info("Reeeeaaaaallllllly long message")  # Length: 31
Reeeeaaaaallllllly long message
>>> logger.info("Done")  # Length: <25, no output

Above, you define a class ShortMsgFilter and override its .filter(). In .addHandler(), you could also just pass a callable, such as a function or lambda or a class that defines .__call__().

The `Manager` Class

There’s one more behind-the-scenes actor of logging that is worth touching on: the Manager class. What matters most is not the Manager class but a single instance of it that acts as a container for the growing hierarchy of loggers that are defined across packages. You’ll see in the next section how just a single instance of this class is central to gluing the module together and allowing its parts to talk to each other.

The All-Important Root Logger

When it comes to Logger instances, one stands out. It’s called the root logger:

class RootLogger(Logger):
    def __init__(self, level):
        Logger.__init__(self, "root", level)

# ...

root = RootLogger(WARNING)
Logger.root = root
Logger.manager = Manager(Logger.root)

The last three lines of this code block are one of the ingenious tricks employed by the logging package. Here are a few points:

The root logger is just a no-frills Python object with the identifier root. It has a level of logging.WARNING and a .name of "root". As far as the class RootLogger is concerned, this unique name is all that’s special about it.
The root object in turn becomes a class attribute for the Logger class. This means that all instances of Logger, and the Logger class itself, all have a .root attribute that is the root logger. This is another example of a singleton-like pattern being enforced in the logging package.
A Manager instance is set as the .manager class attribute for Logger. This eventually comes into play in logging.getLogger("name"). The .manager does all the facilitation of searching for existing loggers with the name "name" and creating them if they don’t exist.

The Logger Hierarchy

Everything is a child of root in the logger namespace, and I mean everything. That includes loggers that you specify yourself as well as those from third-party libraries that you import.

Remember earlier how the .getEffectiveLevel() for our logger instances was 30 (WARNING) even though we had not explicitly set it? That’s because the root logger sits at the top of the hierarchy, and its level is a fallback if any nested loggers have a null level of NOTSET:

>>>

>>> root = logging.getLogger()  # Or getLogger("")
>>> root
<RootLogger root (WARNING)>
>>> root.parent is None
True
>>> root.root is root  # Self-referential
True
>>> root is logging.root
True
>>> root.getEffectiveLevel()
30

The same logic applies to the search for a logger’s handlers. The search is effectively a reverse-order search up the tree of a logger’s parents.

A Multi-Handler Design

The logger hierarchy may seem neat in theory, but how beneficial is it in practice?

Let’s take a break from exploring the logging code and foray into writing our own mini-application—one that takes advantage of the logger hierarchy in a way that reduces boilerplate code and keeps things scalable if the project’s codebase grows.

Here’s the project structure:

project/
│
└── project/
    ├── __init__.py
    ├── utils.py
    └── base.py

Don’t worry about the application’s main functions in utils.py and base.py. What we’re paying more attention to here is the interaction in logging objects between the modules in project/.

In this case, say that you want to design a multipronged logging setup:

Each module gets a logger with multiple handlers.
Some of the handlers are shared between different logger instances in different modules. These handlers only care about level-based filtering, not the module where the log record emanated from. There is a handler for DEBUG messages, one for INFO, one for WARNING, and so on.
Each logger is also tied to one more additional handler that only receives LogRecord instances from that lone logger. You can call this a module-based file handler.

Visually, what we’re shooting for would look something like this:

A multipronged logging design (Image: Real Python)

The two turquoise objects are instances of Logger, established with logging.getLogger(__name__) for each module in a package. Everything else is a Handler instance.

The thinking behind this design is that it’s neatly compartmentalized. You can conveniently look at messages coming from a single logger, or look at messages of a certain level and above coming from any logger or module.

The properties of the logger hierarchy make it suitable for setting up this multipronged logger-handler layout. What does that mean? Here’s a concise explanation from the Django documentation:

Why is the hierarchy important? Well, because loggers can be set to propagate their logging calls to their parents. In this way, you can define a single set of handlers at the root of a logger tree, and capture all logging calls in the subtree of loggers. A logging handler defined in the project namespace will catch all logging messages issued on the project.interesting and project.interesting.stuff loggers. (Source)

The term propagate refers to how a logger keeps walking up its chain of parents looking for handlers. The .propagate attribute is True for a Logger instance by default:

>>>

>>> logger = logging.getLogger(__name__)
>>> logger.propagate
True

In .callHandlers(), if propagate is True, each successive parent gets reassigned to the local variable c until the hierarchy is exhausted:

class Logger(Filterer):
    # ...
    def callHandlers(self, record):
        c = self
        found = 0
        while c:
            for hdlr in c.handlers:
                found = found + 1
                if record.levelno >= hdlr.level:
                    hdlr.handle(record)
            if not c.propagate:
                c = None
            else:
                c = c.parent

Here’s what this means: because the __name__ dunder variable within a package’s __init__.py module is just the name of the package, a logger there becomes a parent to any loggers present in other modules in the same package.

Here are the resulting .name attributes from assigning to logger with logging.getLogger(__name__):

Module	`.name` Attribute
`project/__init__.py`	`'project'`
`project/utils.py`	`'project.utils'`
`project/base.py`	`'project.base'`

Because the 'project.utils' and 'project.base' loggers are children of 'project', they will latch onto not only their own direct handlers but whatever handlers are attached to 'project'.

Let’s build out the modules. First comes __init__.py:

# __init__.py
import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

levels = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
for level in levels:
    handler = logging.FileHandler(f"/tmp/level-{level.lower()}.log")
    handler.setLevel(getattr(logging, level))
    logger.addHandler(handler)

def add_module_handler(logger, level=logging.DEBUG):
    handler = logging.FileHandler(
        f"/tmp/module-{logger.name.replace('.', '-')}.log"
    )
    handler.setLevel(level)
    logger.addHandler(handler)

This module is imported when the project package is imported. You add a handler for each level in DEBUG through CRITICAL, then attach it to a single logger at the top of the hierarchy.

You also define a utility function that adds one more FileHandler to a logger, where the filename of the handler corresponds to the module name where the logger is defined. (This assumes the logger is defined with __name__.)

You can then add some minimal boilerplate logger setup in base.py and utils.py. Notice that you only need to add one additional handler with add_module_handler() from __init__.py. You don’t need to worry about the level-oriented handlers because they are already added to their parent logger named 'project':

# base.py
import logging

from project import add_module_handler

logger = logging.getLogger(__name__)
add_module_handler(logger)

def func1():
    logger.debug("debug called from base.func1()")
    logger.critical("critical called from base.func1()")

Here’s utils.py:

# utils.py
import logging

from project import add_module_handler

logger = logging.getLogger(__name__)
add_module_handler(logger)

def func2():
    logger.debug("debug called from utils.func2()")
    logger.critical("critical called from utils.func2()")

Let’s see how all of this works together from a fresh Python session:

>>>

>>> from pprint import pprint
>>> import project
>>> from project import base, utils

>>> project.logger
<Logger project (DEBUG)>
>>> base.logger, utils.logger
(<Logger project.base (DEBUG)>, <Logger project.utils (DEBUG)>)
>>> base.logger.handlers
[<FileHandler /tmp/module-project-base.log (DEBUG)>]
>>> pprint(base.logger.parent.handlers)
[<FileHandler /tmp/level-debug.log (DEBUG)>,
 <FileHandler /tmp/level-info.log (INFO)>,
 <FileHandler /tmp/level-warning.log (WARNING)>,
 <FileHandler /tmp/level-error.log (ERROR)>,
 <FileHandler /tmp/level-critical.log (CRITICAL)>]
>>> base.func1()
>>> utils.func2()

You’ll see in the resulting log files that our filtration system works as intended. Module-oriented handlers direct one logger to a specific file, while level-oriented handlers direct multiple loggers to a different file:

$ cat /tmp/level-debug.log 
debug called from base.func1()
critical called from base.func1()
debug called from utils.func2()
critical called from utils.func2()

$ cat /tmp/level-critical.log 
critical called from base.func1()
critical called from utils.func2()

$ cat /tmp/module-project-base.log
debug called from base.func1()
critical called from base.func1()

$ cat /tmp/module-project-utils.log 
debug called from utils.func2()
critical called from utils.func2()

A drawback worth mentioning is that this design introduces a lot of redundancy. One LogRecord instance may go to no less than six files. That’s also a non-negligible amount of file I/O that may add up in a performance-critical application.

Now that you’ve seen a practical example, let’s switch gears and delve into a possible source of confusion in logging.

The “Why Didn’t My Log Message Go Anywhere?” Dilemma

There are two common situations with logging when it’s easy to get tripped up:

You logged a message that seemingly went nowhere, and you’re not sure why.
Instead of being suppressed, a log message appeared in a place that you didn’t expect it to.

Each of these has a reason or two commonly associated with it.

You logged a message that seemingly went nowhere, and you’re not sure why.

Don’t forget that the effective level of a logger for which you don’t otherwise set a custom level is WARNING, because a logger will walk up its hierarchy until it finds the root logger with its own WARNING level:

>>>

>>> import logging
>>> logger = logging.getLogger("xyz")
>>> logger.debug("mind numbing info here")
>>> logger.critical("storm is coming")
storm is coming

Because of this default, the .debug() call goes nowhere.

Instead of being suppressed, a log message appeared in a place that you didn’t expect it to.

When you defined your logger above, you didn’t add any handlers to it. So, why is it writing to the console?

The reason for this is that logging sneakily uses a handler called lastResort that writes to sys.stderr if no other handlers are found:

class _StderrHandler(StreamHandler):
    # ...
    @property
    def stream(self):
        return sys.stderr

_defaultLastResort = _StderrHandler(WARNING)
lastResort = _defaultLastResort

This kicks in when a logger goes to find its handlers:

class Logger(Filterer):
    # ...
    def callHandlers(self, record):
        c = self
        found = 0
        while c:
            for hdlr in c.handlers:
                found = found + 1
                if record.levelno >= hdlr.level:
                    hdlr.handle(record)
            if not c.propagate:
                c = None
            else:
                c = c.parent
        if (found == 0):
            if lastResort:
                if record.levelno >= lastResort.level:
                     lastResort.handle(record)

If the logger gives up on its search for handlers (both its own direct handlers and attributes of parent loggers), then it picks up the lastResort handler and uses that.

There’s one more subtle detail worth knowing about. This section has largely talked about the instance methods (methods that a class defines) rather than the module-level functions of the logging package that carry the same name.

If you use the functions, such as logging.info() rather than logger.info(), then something slightly different happens internally. The function calls logging.basicConfig(), which adds a StreamHandler that writes to sys.stderr. In the end, the behavior is virtually the same:

>>>

>>> import logging
>>> root = logging.getLogger("")
>>> root.handlers
[]
>>> root.hasHandlers()
False
>>> logging.basicConfig()
>>> root.handlers
[<StreamHandler <stderr> (NOTSET)>]
>>> root.hasHandlers()
True

Taking Advantage of Lazy Formatting

It’s time to switch gears and take a closer look at how messages themselves are joined with their data. While it’s been supplanted by str.format() and f-strings, you’ve probably used Python’s percent-style formatting to do something like this:

>>>

>>> print("To iterate is %s, to recurse %s" % ("human", "divine"))
To iterate is human, to recurse divine

As a result, you may be tempted to do the same thing in a logging call:

>>>

>>> # Bad!  Check out a more efficient alternative below.
>>> logging.warning("To iterate is %s, to recurse %s" % ("human", "divine"))
WARNING:root:To iterate is human, to recurse divine

This uses the entire format string and its arguments as the msg argument to logging.warning().

Here is the recommended alternative, straight from the logging docs:

>>>

>>> # Better: formatting doesn't occur until it really needs to.
>>> logging.warning("To iterate is %s, to recurse %s", "human", "divine")
WARNING:root:To iterate is human, to recurse divine

It looks a little weird, right? This seems to defy the conventions of how percent-style string formatting works, but it’s a more efficient function call because the format string gets formatted lazily rather than greedily. Here’s what that means.

The method signature for Logger.warning() looks like this:

def warning(self, msg, *args, **kwargs)

The same applies to the other methods, such as .debug(). When you call warning("To iterate is %s, to recurse %s", "human", "divine"), both "human" and "divine" get caught as *args and, within the scope of the method’s body, args is equal to ("human", "divine").

Contrast this to the first call above:

logging.warning("To iterate is %s, to recurse %s" % ("human", "divine"))

In this form, everything in the parentheses gets immediately merged together into "To iterate is human, to recurse divine" and passed as msg, while args is an empty tuple.

Why does this matter? Repeated logging calls can degrade runtime performance slightly, but the logging package does its very best to control that and keep it in check. By not merging the format string with its arguments right away, logging is delaying the string formatting until the LogRecord is requested by a Handler.

This happens in LogRecord.getMessage(), so only after logging deems that the LogRecord will actually be passed to a handler does it become its fully merged self.

All that is to say that the logging package makes some very fine-tuned performance optimizations in the right places. This may seem like minutia, but if you’re making the same logging.debug() call a million times inside a loop, and the args are function calls, then the lazy nature of how logging does string formatting can make a difference.

Before doing any merging of msg and args, a Logger instance will check its .isEnabledFor() to see if that merging should be done in the first place.

Functions vs Methods

Towards the bottom of logging/__init__.py sit the module-level functions that are advertised up front in the public API of logging. You already saw the Logger methods such as .debug(), .info(), and .warning(). The top-level functions are wrappers around the corresponding methods of the same name, but they have two important features:

They always call their corresponding method from the root logger, root.
Before calling the root logger methods, they call logging.basicConfig() with no arguments if root doesn’t have any handlers. As you saw earlier, it is this call that sets a sys.stdout handler for the root logger.

For illustration, here’s logging.error():

def error(msg, *args, **kwargs):
    if len(root.handlers) == 0:
        basicConfig()
    root.error(msg, *args, **kwargs)

You’ll find the same pattern for logging.debug(), logging.info(), and the others as well. Tracing the chain of commands is interesting. Eventually, you’ll end up at the same place, which is where the internal Logger._log() is called.

The calls to debug(), info(), warning(), and the other level-based functions all route to here. _log() primarily has two purposes:

Call self.makeRecord(): Make a LogRecord instance from the msg and other arguments you pass to it.
Call self.handle(): This determines what actually gets done with the record. Where does it get sent? Does it make it there or get filtered out?

Here’s that entire process in one diagram:

Internals of a logging call (Image: Real Python)

You can also trace the call stack with pdb.

>>>

>>> import logging
>>> import pdb
>>> pdb.run('logging.warning("%s-%s", "uh", "oh")')
> <string>(1)<module>()
(Pdb) s
--Call--
> lib/python3.7/logging/__init__.py(1971)warning()
-> def warning(msg, *args, **kwargs):
(Pdb) s
> lib/python3.7/logging/__init__.py(1977)warning()
-> if len(root.handlers) == 0:
(Pdb) unt
> lib/python3.7/logging/__init__.py(1978)warning()
-> basicConfig()
(Pdb) unt
> lib/python3.7/logging/__init__.py(1979)warning()
-> root.warning(msg, *args, **kwargs)
(Pdb) s
--Call--
> lib/python3.7/logging/__init__.py(1385)warning()
-> def warning(self, msg, *args, **kwargs):
(Pdb) l
1380             logger.info("Houston, we have a %s", "interesting problem", exc_info=1)
1381             """
1382             if self.isEnabledFor(INFO):
1383                 self._log(INFO, msg, args, **kwargs)
1384     
1385 ->        def warning(self, msg, *args, **kwargs):
1386             """
1387             Log 'msg % args' with severity 'WARNING'.
1388     
1389             To pass exception information, use the keyword argument exc_info with
1390             a true value, e.g.
(Pdb) s
> lib/python3.7/logging/__init__.py(1394)warning()
-> if self.isEnabledFor(WARNING):
(Pdb) unt
> lib/python3.7/logging/__init__.py(1395)warning()
-> self._log(WARNING, msg, args, **kwargs)
(Pdb) s
--Call--
> lib/python3.7/logging/__init__.py(1496)_log()
-> def _log(self, level, msg, args, exc_info=None, extra=None, stack_info=False):
(Pdb) s
> lib/python3.7/logging/__init__.py(1501)_log()
-> sinfo = None
(Pdb) unt 1517
> lib/python3.7/logging/__init__.py(1517)_log()
-> record = self.makeRecord(self.name, level, fn, lno, msg, args,
(Pdb) s
> lib/python3.7/logging/__init__.py(1518)_log()
-> exc_info, func, extra, sinfo)
(Pdb) s
--Call--
> lib/python3.7/logging/__init__.py(1481)makeRecord()
-> def makeRecord(self, name, level, fn, lno, msg, args, exc_info,
(Pdb) p name
'root'
(Pdb) p level
30
(Pdb) p msg
'%s-%s'
(Pdb) p args
('uh', 'oh')
(Pdb) up
> lib/python3.7/logging/__init__.py(1518)_log()
-> exc_info, func, extra, sinfo)
(Pdb) unt
> lib/python3.7/logging/__init__.py(1519)_log()
-> self.handle(record)
(Pdb) n
WARNING:root:uh-oh

What Does `getLogger()` Really Do?

Also hiding in this section of the source code is the top-level getLogger(), which wraps Logger.manager.getLogger():

def getLogger(name=None):
    if name:
        return Logger.manager.getLogger(name)
    else:
        return root

This is the entry point for enforcing the singleton logger design:

If you specify a name, then the underlying .getLogger() does a dict lookup on the string name. What this comes down to is a lookup in the loggerDict of logging.Manager. This is a dictionary of all registered loggers, including the intermediate PlaceHolder instances that are generated when you reference a logger far down in the hierarchy before referencing its parents.
Otherwise, root is returned. There is only one root—the instance of RootLogger discussed above.

This feature is what lies behind a trick that can let you peek into all of the registered loggers:

>>>

>>> import logging
>>> logging.Logger.manager.loggerDict
{}

>>> from pprint import pprint
>>> import asyncio
>>> pprint(logging.Logger.manager.loggerDict)
{'asyncio': <Logger asyncio (WARNING)>,
 'concurrent': <logging.PlaceHolder object at 0x10d153710>,
 'concurrent.futures': <Logger concurrent.futures (WARNING)>}

Whoa, hold on a minute. What’s happening here? It looks like something changed internally to the logging package as a result of an import of another library, and that’s exactly what happened.

Firstly, recall that Logger.manager is a class attribute, where an instance of Manager is tacked onto the Logger class. The manager is designed to track and manage all of the singleton instances of Logger. These are housed in .loggerDict.

Now, when you initially import logging, this dictionary is empty. But after you import asyncio, the same dictionary gets populated with three loggers. This is an example of one module setting the attributes of another module in-place. Sure enough, inside of asyncio/log.py, you’ll find the following:

import logging

logger = logging.getLogger(__package__)  # "asyncio"

The key-value pair is set in Logger.getLogger() so that the manager can oversee the entire namespace of loggers. This means that the object asyncio.log.logger gets registered in the logger dictionary that belongs to the logging package. Something similar happens in the concurrent.futures package as well, which is imported by asyncio.

You can see the power of the singleton design in an equivalence test:

>>>

>>> obj1 = logging.getLogger("asyncio")
>>> obj2 = logging.Logger.manager.loggerDict["asyncio"]
>>> obj1 is obj2
True

This comparison illustrates (glossing over a few details) what getLogger() ultimately does.

Library vs Application Logging: What Is `NullHandler`?

That brings us to the final hundred or so lines in the logging/__init__.py source, where NullHandler is defined. Here’s the definition in all its glory:

class NullHandler(Handler):
    def handle(self, record):
        pass

    def emit(self, record):
        pass

    def createLock(self):
        self.lock = None

The NullHandler is all about the distinctions between logging in a library versus an application. Let’s see what that means.

A library is an extensible, generalizable Python package that is intended for other users to install and set up. It is built by a developer with the express purpose of being distributed to users. Examples include popular open-source projects like NumPy, dateutil, and cryptography.

An application (or app, or program) is designed for a more specific purpose and a much smaller set of users (possibly just one user). It’s a program or set of programs highly tailored by the user to do a limited set of things. An example of an application is a Django app that sits behind a web page. Applications commonly use (import) libraries and the tools they contain.

When it comes to logging, there are different best practices in a library versus an app.

That’s where NullHandler fits in. It’s basically a do-nothing stub class.

If you’re writing a Python library, you really need to do this one minimalist piece of setup in your package’s __init__.py:

# Place this in your library's uppermost `__init__.py`
# Nothing else!

import logging

logging.getLogger(__name__).addHandler(NullHandler())

This serves two critical purposes.

Firstly, a library logger that is declared with logger = logging.getLogger(__name__) (without any further configuration) will log to sys.stderr by default, even if that’s not what the end user wants. This could be described as an opt-out approach, where the end user of the library has to go in and disable logging to their console if they don’t want it.

Common wisdom says to use an opt-in approach instead: don’t emit any log messages by default, and let the end users of the library determine if they want to further configure the library’s loggers and add handlers to them. Here’s that philosophy worded more bluntly by the author of the logging package, Vinay Sajip:

A third party library which uses logging should not spew logging output by default which may not be wanted by a developer/user of an application which uses it. (Source)

This leaves it up to the library user, not library developer, to incrementally call methods such as logger.addHandler() or logger.setLevel().

The second reason that NullHandler exists is more archaic. In Python 2.7 and earlier, trying to log a LogRecord from a logger that has no handler set would raise a warning. Adding the no-op class NullHandler will avert this.

Here’s what specifically happens in the line logging.getLogger(__name__).addHandler(NullHandler()) from above:

Python gets (creates) the Logger instance with the same name as your package. If you’re designing the calculus package, within __init__.py, then __name__ will be equal to 'calculus'.
A NullHandler instance gets attached to this logger. That means that Python will not default to using the lastResort handler.

Keep in mind that any logger created in any of the other .py modules of the package will be children of this logger in the logger hierarchy and that, because this handler also belongs to them, they won’t need to use the lastResort handler and won’t default to logging to standard error (stderr).

As a quick example, let’s say your library has the following structure:

calculus/
│
├── __init__.py
└── integration.py

In integration.py, as the library developer you are free to do the following:

# calculus/integration.py
import logging

logger = logging.getLogger(__name__)

def func(x):
    logger.warning("Look!")
    # Do stuff
    return None

Now, a user comes along and installs your library from PyPI via pip install calculus. They use from calculus.integration import func in some application code. This user is free to manipulate and configure the logger object from the library like any other Python object, to their heart’s content.

What Logging Does With Exceptions

One thing that you may be wary of is the danger of exceptions that stem from your calls to logging. If you have a logging.error() call that is designed to give you some more verbose debugging information, but that call itself for some reason raises an exception, that would be the height of irony, right?

Cleverly, if the logging package encounters an exception that has to do with logging itself, then it will print the traceback, but not raise the exception itself.

Here’s an example that deals with a common typo: passing two arguments to a format string that is only expecting one argument. The important distinction is that what you see below is not an exception being raised, but rather a prettified printed traceback of the internal exception, which itself was suppressed:

>>>

>>> logging.critical("This %s has too many arguments", "msg", "other")
--- Logging error ---
Traceback (most recent call last):
  File "lib/python3.7/logging/__init__.py", line 1034, in emit
    msg = self.format(record)
  File "lib/python3.7/logging/__init__.py", line 880, in format
    return fmt.format(record)
  File "lib/python3.7/logging/__init__.py", line 619, in format
    record.message = record.getMessage()
  File "lib/python3.7/logging/__init__.py", line 380, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "<stdin>", line 1, in <module>
Message: 'This %s has too many arguments'
Arguments: ('msg', 'other')

This lets your program gracefully carry on with its actual program flow. The rationale is that you wouldn’t want an uncaught exception to come from a logging call itself and stop a program dead in its tracks.

Tracebacks can be messy, but this one is informative and relatively straightforward. What enables the suppression of exceptions related to logging is Handler.handleError(). When the handler calls .emit(), which is the method where it attempts to log the record, it falls back to .handleError() if something goes awry. Here’s the implementation of .emit() for the StreamHandler class:

def emit(self, record):
    try:
        msg = self.format(record)
        stream = self.stream
        stream.write(msg + self.terminator)
        self.flush()
    except Exception:
        self.handleError(record)

Any exception related to the formatting and writing gets caught rather than being raised, and handleError gracefully writes the traceback to sys.stderr.

Logging Python Tracebacks

Speaking of exceptions and their tracebacks, what about cases where your program encounters them but should log the exception and keep chugging along in its execution?

Let’s walk through a couple of ways to do this.

Here’s a contrived example of a lottery simulator using code that isn’t Pythonic on purpose. You’re developing an online lottery game where users can wager on their lucky number:

import random

class Lottery(object):
    def __init__(self, n):
        self.n = n

    def make_tickets(self):
        for i in range(self.n):
            yield i

    def draw(self):
        pool = self.make_tickets()
        random.shuffle(pool)
        return next(pool)

Behind the frontend application sits the critical code below. You want to make sure that you keep track of any errors caused by the site that may make a user lose their money. The first (suboptimal) way is to use logging.error() and log the str form of the exception instance itself:

try:
    lucky_number = int(input("Enter your ticket number: "))
    drawn = Lottery(n=20).draw()
    if lucky_number == drawn:
        print("Winner chicken dinner!")
except Exception as e:
    # NOTE: See below for a better way to do this.
    logging.error("Could not draw ticket: %s", e)

This will only get you the actual exception message, rather than the traceback. You check the logs on your website’s server and find this cryptic message:

ERROR:root:Could not draw ticket: object of type 'generator' has no len()

Hmm. As the application developer, you’ve got a serious problem, and a user got ripped off as a result. But maybe this exception message itself isn’t very informative. Wouldn’t it be nice to see the lineage of the traceback that led to this exception?

The proper solution is to use logging.exception(), which logs a message with level ERROR and also displays the exception traceback. Replace the two final lines above with these:

except Exception:
    logging.exception("Could not draw ticket")

Now you get a better indication of what’s going on:

>>>

ERROR:root:Could not draw ticket
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "<stdin>", line 9, in draw
  File "lib/python3.7/random.py", line 275, in shuffle
    for i in reversed(range(1, len(x))):
TypeError: object of type 'generator' has no len()

Using exception() saves you from having to reference the exception yourself because logging pulls it in with sys.exc_info().

This makes things clearer that the problem stems from random.shuffle(), which needs to know the length of the object it is shuffling. Because our Lottery class passes a generator to shuffle(), it gets held up and raises before the pool can be shuffled, much less generate a winning ticket.

In large, full-blown applications, you’ll find logging.exception() to be even more useful when deep, multi-library tracebacks are involved, and you can’t step into them with a live debugger like pdb.

The code for logging.Logger.exception(), and hence logging.exception(), is just a single line:

def exception(self, msg, *args, exc_info=True, **kwargs):
    self.error(msg, *args, exc_info=exc_info, **kwargs)

That is, logging.exception() just calls logging.error() with exc_info=True, which is otherwise False by default. If you want to log an exception traceback but at a level different than logging.ERROR, just call that function or method with exc_info=True.

Keep in mind that exception() should only be called in the context of an exception handler, inside of an except block:

for i in data:
    try:
        result = my_longwinded_nested_function(i)
    except ValueError:
        # We are in the context of exception handler now.
        # If it's unclear exactly *why* we couldn't process
        # `i`, then log the traceback and move on rather than
        # ditching completely.
        logger.exception("Could not process %s", i)
        continue

Use this pattern sparingly rather than as a means to suppress any exception. It can be most helpful when you’re debugging a long function call stack where you’re otherwise seeing an ambiguous, unclear, and hard-to-track error.

Conclusion

Pat yourself on the back, because you’ve just walked through almost 2,000 lines of dense source code. You’re now better equipped to deal with the logging package!

Keep in mind that this tutorial has been far from exhaustive in covering all of the classes found in the logging package. There’s even more machinery that glues everything together. If you’d like to learn more, then you can look into the Formatter classes and the separate modules logging/config.py and logging/handlers.py.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Interactive Data Visualization in Python With Bokeh

2019-05-21T14:00:00+00:00

Bokeh prides itself on being a library for interactive data visualization. The graphics are rendered using HTML and JavaScript, and your visualizations are easy to share as an HTML page. You will create a number of visualizations based on a real-world dataset.

The goal of this course is to get you up and running with Bokeh.

You will learn how to:

Transform your data into visualizations
Customize and organize your visualizations
Add interactivity to your visualizations

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Unicode & Character Encodings in Python: A Painless Guide

2019-05-20T14:00:00+00:00

Handling character encodings in Python or any other language can at times seem painful. Places such as Stack Overflow have thousands of questions stemming from confusion over exceptions like UnicodeDecodeError and UnicodeEncodeError. This tutorial is designed to clear the Exception fog and illustrate that working with text and binary data in Python 3 can be a smooth experience. Python’s Unicode support is strong and robust, but it takes some time to master.

This tutorial is different because it’s not language-agnostic but instead deliberately Python-centric. You’ll still get a language-agnostic primer, but you’ll then dive into illustrations in Python, with text-heavy paragraphs kept to a minimum. You’ll see how to use concepts of character encodings in live Python code.

By the end of this tutorial, you’ll:

Get conceptual overviews on character encodings and numbering systems
Understand how encoding comes into play with Python’s str and bytes
Know about support in Python for numbering systems through its various forms of int literals
Be familiar with Python’s built-in functions related to character encodings and numbering systems

Character encoding and numbering systems are so closely connected that they need to be covered in the same tutorial or else the treatment of either would be totally inadequate.

Note: This article is Python 3-centric. Specifically, all code examples in this tutorial were generated from a CPython 3.7.2 shell, although all minor versions of Python 3 should behave (mostly) the same in their treatment of text.

If you’re still using Python 2 and are intimidated by the differences in how Python 2 and Python 3 treat text and binary data, then hopefully this tutorial will help you make the switch.

What’s a Character Encoding?

There are tens if not hundreds of character encodings. The best way to start understanding what they are is to cover one of the simplest character encodings, ASCII.

Whether you’re self-taught or have a formal computer science background, chances are you’ve seen an ASCII table once or twice. ASCII is a good place to start learning about character encoding because it is a small and contained encoding. (Too small, as it turns out.)

It encompasses the following:

Lowercase English letters: a through z
Uppercase English letters: A through Z
Some punctuation and symbols: "$" and "!", to name a couple
Whitespace characters: an actual space (" "), as well as a newline, carriage return, horizontal tab, vertical tab, and a few others
Some non-printable characters: characters such as backspace, "\b", that can’t be printed literally in the way that the letter A can

So what is a more formal definition of a character encoding?

At a very high level, it’s a way of translating characters (such as letters, punctuation, symbols, whitespace, and control characters) to integers and ultimately to bits. Each character can be encoded to a unique sequence of bits. Don’t worry if you’re shaky on the concept of bits, because we’ll get to them shortly.

The various categories outlined represent groups of characters. Each single character has a corresponding code point, which you can think of as just an integer. Characters are segmented into different ranges within the ASCII table:

Code Point Range	Class
0 through 31	Control/non-printable characters
32 through 64	Punctuation, symbols, numbers, and space
65 through 90	Uppercase English alphabet letters
91 through 96	Additional graphemes, such as `[` and `\`
97 through 122	Lowercase English alphabet letters
123 through 126	Additional graphemes, such as `{` and `\|`
127	Control/non-printable character (`DEL`)

The entire ASCII table contains 128 characters. This table captures the complete character set that ASCII permits. If you don’t see a character here, then you simply can’t express it as printed text under the ASCII encoding scheme.

Code Point	Character (Name)	Code Point	Character (Name)
0	NUL (Null)	64	`@`
1	SOH (Start of Heading)	65	`A`
2	STX (Start of Text)	66	`B`
3	ETX (End of Text)	67	`C`
4	EOT (End of Transmission)	68	`D`
5	ENQ (Enquiry)	69	`E`
6	ACK (Acknowledgment)	70	`F`
7	BEL (Bell)	71	`G`
8	BS (Backspace)	72	`H`
9	HT (Horizontal Tab)	73	`I`
10	LF (Line Feed)	74	`J`
11	VT (Vertical Tab)	75	`K`
12	FF (Form Feed)	76	`L`
13	CR (Carriage Return)	77	`M`
14	SO (Shift Out)	78	`N`
15	SI (Shift In)	79	`O`
16	DLE (Data Link Escape)	80	`P`
17	DC1 (Device Control 1)	81	`Q`
18	DC2 (Device Control 2)	82	`R`
19	DC3 (Device Control 3)	83	`S`
20	DC4 (Device Control 4)	84	`T`
21	NAK (Negative Acknowledgment)	85	`U`
22	SYN (Synchronous Idle)	86	`V`
23	ETB (End of Transmission Block)	87	`W`
24	CAN (Cancel)	88	`X`
25	EM (End of Medium)	89	`Y`
26	SUB (Substitute)	90	`Z`
27	ESC (Escape)	91	`[`
28	FS (File Separator)	92	`\`
29	GS (Group Separator)	93	`]`
30	RS (Record Separator)	94	`^`
31	US (Unit Separator)	95	`_`
32	SP (Space)	96	`
33	`!`	97	`a`
34	`"`	98	`b`
35	`#`	99	`c`
36	`$`	100	`d`
37	`%`	101	`e`
38	`&`	102	`f`
39	`'`	103	`g`
40	`(`	104	`h`
41	`)`	105	`i`
42	`*`	106	`j`
43	`+`	107	`k`
44	`,`	108	`l`
45	`-`	109	`m`
46	`.`	110	`n`
47	`/`	111	`o`
48	`0`	112	`p`
49	`1`	113	`q`
50	`2`	114	`r`
51	`3`	115	`s`
52	`4`	116	`t`
53	`5`	117	`u`
54	`6`	118	`v`
55	`7`	119	`w`
56	`8`	120	`x`
57	`9`	121	`y`
58	`:`	122	`z`
59	`;`	123	`{`
60	`<`	124	`\|`
61	`=`	125	`}`
62	`>`	126	`~`
63	`?`	127	DEL (delete)

The `string` Module

Python’s string module is a convenient one-stop-shop for string constants that fall in ASCII’s character set.

Here’s the core of the module in all its glory:

# From lib/python3.7/string.py

whitespace = ' \t\n\r\v\f'
ascii_lowercase = 'abcdefghijklmnopqrstuvwxyz'
ascii_uppercase = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
ascii_letters = ascii_lowercase + ascii_uppercase
digits = '0123456789'
hexdigits = digits + 'abcdef' + 'ABCDEF'
octdigits = '01234567'
punctuation = r"""!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~"""
printable = digits + ascii_letters + punctuation + whitespace

Most of these constants should be self-documenting in their identifier name. We’ll cover what hexdigits and octdigits are shortly.

You can use these constants for everyday string manipulation:

>>>

>>> import string

>>> s = "What's wrong with ASCII?!?!?"
>>> s.rstrip(string.punctuation)
'What's wrong with ASCII'

Note: string.printable includes all of string.whitespace. This disagrees slightly with another method for testing whether a character is considered printable, namely str.isprintable(), which will tell you that none of {'\v', '\n', '\r', '\f', '\t'} are considered printable.

The subtle difference is because of definition: str.isprintable() considers something printable if “all of its characters are considered printable in repr().”

A Bit of a Refresher

Now is a good time for a short refresher on the bit, the most fundamental unit of information that a computer knows.

A bit is a signal that has only two possible states. There are different ways of symbolically representing a bit that all mean the same thing:

0 or 1
“yes” or “no”
True or False
“on” or “off”

Our ASCII table from the previous section uses what you and I would just call numbers (0 through 127), but what are more precisely called numbers in base 10 (decimal).

You can also express each of these base-10 numbers with a sequence of bits (base 2). Here are the binary versions of 0 through 10 in decimal:

Decimal	Binary (Compact)	Binary (Padded Form)
0	0	00000000
1	1	00000001
2	10	00000010
3	11	00000011
4	100	00000100
5	101	00000101
6	110	00000110
7	111	00000111
8	1000	00001000
9	1001	00001001
10	1010	00001010

Notice that as the decimal number n increases, you need more significant bits to represent the character set up to and including that number.

Here’s a handy way to represent ASCII strings as sequences of bits in Python. Each character from the ASCII string gets pseudo-encoded into 8 bits, with spaces in between the 8-bit sequences that each represent a single character:

>>>

>>> def make_bitseq(s: str) -> str:
...     if not s.isascii():
...         raise ValueError("ASCII only allowed")
...     return " ".join(f"{ord(i):08b}" for i in s)

>>> make_bitseq("bits")
'01100010 01101001 01110100 01110011'

>>> make_bitseq("CAPS")
'01000011 01000001 01010000 01010011'

>>> make_bitseq("$25.43")
'00100100 00110010 00110101 00101110 00110100 00110011'

>>> make_bitseq("~5")
'01111110 00110101'

Note: .isascii() was introduced in Python 3.7.

The f-string f"{ord(i):08b}" uses Python’s Format Specification Mini-Language, which is a way of specifying formatting for replacement fields in format strings:

The left side of the colon, ord(i), is the actual object whose value will be formatted and inserted into the output. Using ord() gives you the base-10 code point for a single str character.
The right hand side of the colon is the format specifier. 08 means width 8, 0 padded, and the b functions as a sign to output the resulting number in base 2 (binary).

This trick is mainly just for fun, and it will fail very badly for any character that you don’t see present in the ASCII table. We’ll discuss how other encodings fix this problem later on.

We Need More Bits!

There’s a critically important formula that’s related to the definition of a bit. Given a number of bits, n, the number of distinct possible values that can be represented in n bits is 2ⁿ:

def n_possible_values(nbits: int) -> int:
    return 2 ** nbits

Here’s what that means:

1 bit will let you express 2¹ == 2 possible values.
8 bits will let you express 2⁸ == 256 possible values.
64 bits will let you express 2⁶⁴ == 18,446,744,073,709,551,616 possible values.

There’s a corollary to this formula: given a range of distinct possible values, how can we find the number of bits, n, that is required for the range to be fully represented? What you’re trying to solve for is n in the equation 2ⁿ = x (where you already know x).

Here’s what that works out to:

>>>

>>> from math import ceil, log

>>> def n_bits_required(nvalues: int) -> int:
...     return ceil(log(nvalues) / log(2))

>>> n_bits_required(256)
8

The reason that you need to use a ceiling in n_bits_required() is to account for values that are not clean powers of 2. Say you need to store a character set of 110 characters total. Naively, this should take log(110) / log(2) == 6.781 bits, but there’s no such thing as 0.781 bits. 110 values will require 7 bits, not 6, with the final slots being unneeded:

>>>

>>> n_bits_required(110)
7

All of this serves to to prove one concept: ASCII is, strictly speaking, a 7-bit code. The ASCII table that you saw above contains 128 code points and characters, 0 through 127 inclusive. This requires 7 bits:

>>>

>>> n_bits_required(128)  # 0 through 127
7
>>> n_possible_values(7)
128

The issue with this is that modern computers don’t store much of anything in 7-bit slots. They traffic in units of 8 bits, conventionally known as a byte.

Note: Throughout this tutorial, I assume that a byte refers to 8 bits, as it has since the 1960s, rather than some other unit of storage. You are free to call this an octet if you prefer.

This means that the storage space used by ASCII is half-empty. If it’s not clear why this is, think back to the decimal-to-binary table from above. You can express the numbers 0 and 1 with just 1 bit, or you can use 8 bits to express them as 00000000 and 00000001, respectively.

You can express the numbers 0 through 3 with just 2 bits, or 00 through 11, or you can use 8 bits to express them as 00000000, 00000001, 00000010, and 00000011, respectively. The highest ASCII code point, 127, requires only 7 significant bits.

Knowing this, you can see that make_bitseq() converts ASCII strings into a str representation of bytes, where every character consumes one byte:

>>>

>>> make_bitseq("bits")
'01100010 01101001 01110100 01110011'

ASCII’s underutilization of the 8-bit bytes offered by modern computers led to a family of conflicting, informalized encodings that each specified additional characters to be used with the remaining 128 available code points allowed in an 8-bit character encoding scheme.

Not only did these different encodings clash with each other, but each one of them was by itself still a grossly incomplete representation of the world’s characters, regardless of the fact that they made use of one additional bit.

Over the years, one character encoding mega-scheme came to rule them all. However, before we get there, let’s talk for a minute about numbering systems, which are a fundamental underpinning of character encoding schemes.

Covering All the Bases: Other Number Systems

In the discussion of ASCII above, you saw that each character maps to an integer in the range 0 through 127.

This range of numbers is expressed in decimal (base 10). It’s the way that you, me, and the rest of us humans are used to counting, for no reason more complicated than that we have 10 fingers.

But there are other numbering systems as well that are especially prevalent throughout the CPython source code. While the “underlying number” is the same, all numbering systems are just different ways of expressing the same number.

If I asked you what number the string "11" represents, you’d be right to give me a strange look before answering that it represents eleven.

However, this string representation can express different underlying numbers in different numbering systems. In addition to decimal, the alternatives include the following common numbering systems:

Binary: base 2
Octal: base 8
Hexadecimal (hex): base 16

But what does it mean for us to say that, in a certain numbering system, numbers are represented in base N?

Here is the best way that I know of to articulate what this means: it’s the number of fingers that you’d count on in that system.

If you want a much fuller but still gentle introduction to numbering systems, Charles Petzold’s Code is an incredibly cool book that explores the foundations of computer code in detail.

One way to demonstrate how different numbering systems interpret the same thing is with Python’s int() constructor. If you pass a str to int(), Python will assume by default that the string expresses a number in base 10 unless you tell it otherwise:

>>>

>>> int('11')
11
>>> int('11', base=10)  # 10 is already default
11
>>> int('11', base=2)  # Binary
3
>>> int('11', base=8)  # Octal
9
>>> int('11', base=16)  # Hex
17

There’s a more common way of telling Python that your integer is typed in a base other than 10. Python accepts literal forms of each of the 3 alternative numbering systems above:

Type of Literal	Prefix	Example
n/a	n/a	`11`
Binary literal	`0b` or `0B`	`0b11`
Octal literal	`0o` or `0O`	`0o11`
Hex literal	`0x` or `0X`	`0x11`

All of these are sub-forms of integer literals. You can see that these produce the same results, respectively, as the calls to int() with non-default base values. They’re all just int to Python:

>>>

>>> 11
11
>>> 0b11  # Binary literal
3
>>> 0o11  # Octal literal
9
>>> 0x11  # Hex literal
17

Here’s how you could type the binary, octal, and hexadecimal equivalents of the decimal numbers 0 through 20. Any of these are perfectly valid in a Python interpreter shell or source code, and all work out to be of type int:

Decimal	Binary	Octal	Hex
`0`	`0b0`	`0o0`	`0x0`
`1`	`0b1`	`0o1`	`0x1`
`2`	`0b10`	`0o2`	`0x2`
`3`	`0b11`	`0o3`	`0x3`
`4`	`0b100`	`0o4`	`0x4`
`5`	`0b101`	`0o5`	`0x5`
`6`	`0b110`	`0o6`	`0x6`
`7`	`0b111`	`0o7`	`0x7`
`8`	`0b1000`	`0o10`	`0x8`
`9`	`0b1001`	`0o11`	`0x9`
`10`	`0b1010`	`0o12`	`0xa`
`11`	`0b1011`	`0o13`	`0xb`
`12`	`0b1100`	`0o14`	`0xc`
`13`	`0b1101`	`0o15`	`0xd`
`14`	`0b1110`	`0o16`	`0xe`
`15`	`0b1111`	`0o17`	`0xf`
`16`	`0b10000`	`0o20`	`0x10`
`17`	`0b10001`	`0o21`	`0x11`
`18`	`0b10010`	`0o22`	`0x12`
`19`	`0b10011`	`0o23`	`0x13`
`20`	`0b10100`	`0o24`	`0x14`

It’s amazing just how prevalent these expressions are in the Python Standard Library. If you want to see for yourself, navigate to wherever your lib/python3.7/ directory sits, and check out the use of hex literals like this:

$ grep -nri --include "*\.py" -e "\b0x" lib/python3.7

This should work on any Unix system that has grep. You could use "\b0o" to search for octal literals or “\b0b” to search for binary literals.

What’s the argument for using these alternate int literal syntaxes? In short, it’s because 2, 8, and 16 are all powers of 2, while 10 is not. These three alternate number systems occasionally offer a way for expressing values in a computer-friendly manner. For example, the number 65536 or 2¹⁶, is just 10000 in hexadecimal, or 0x10000 as a Python hexadecimal literal.

Enter Unicode

As you saw, the problem with ASCII is that it’s not nearly a big enough set of characters to accommodate the world’s set of languages, dialects, symbols, and glyphs. (It’s not even big enough for English alone.)

Unicode fundamentally serves the same purpose as ASCII, but it just encompasses a way, way, way bigger set of code points. There are a handful of encodings that emerged chronologically between ASCII and Unicode, but they are not really worth mentioning just yet because Unicode and one of its encoding schemes, UTF-8, has become so predominantly used.

Think of Unicode as a massive version of the ASCII table—one that has 1,114,112 possible code points. That’s 0 through 1,114,111, or 0 through 17 * (2¹⁶) - 1, or 0x10ffff hexadecimal. In fact, ASCII is a perfect subset of Unicode. The first 128 characters in the Unicode table correspond precisely to the ASCII characters that you’d reasonably expect them to.

In the interest of being technically exacting, Unicode itself is not an encoding. Rather, Unicode is implemented by different character encodings, which you’ll see soon. Unicode is better thought of as a map (something like a dict) or a 2-column database table. It maps characters (like "a", "¢", or even "ቈ") to distinct, positive integers. A character encoding needs to offer a bit more.

Unicode contains virtually every character that you can imagine, including additional non-printable ones too. One of my favorites is the pesky right-to-left mark, which has code point 8207 and is used in text with both left-to-right and right-to-left language scripts, such as an article containing both English and Arabic paragraphs.

Note: The world of character encodings is one of many fine-grained technical details over which some people love to nitpick about. One such detail is that only 1,111,998 of the Unicode code points are actually usable, due to a couple of archaic reasons.

Unicode vs UTF-8

It didn’t take long for people to realize that all of the world’s characters could not be packed into one byte each. It’s evident from this that modern, more comprehensive encodings would need to use multiple bytes to encode some characters.

You also saw above that Unicode is not technically a full-blown character encoding. Why is that?

There is one thing that Unicode doesn’t tell you: it doesn’t tell you how to get actual bits from text—just code points. It doesn’t tell you enough about how to convert text to binary data and vice versa.

Unicode is an abstract encoding standard, not an encoding. That’s where UTF-8 and other encoding schemes come into play. The Unicode standard (a map of characters to code points) defines several different encodings from its single character set.

UTF-8 as well as its lesser-used cousins, UTF-16 and UTF-32, are encoding formats for representing Unicode characters as binary data of one or more bytes per character. We’ll discuss UTF-16 and UTF-32 in a moment, but UTF-8 has taken the largest share of the pie by far.

That brings us to a definition that is long overdue. What does it mean, formally, to encode and decode?

Encoding and Decoding in Python 3

Python 3’s str type is meant to represent human-readable text and can contain any Unicode character.

The bytes type, conversely, represents binary data, or sequences of raw bytes, that do not intrinsically have an encoding attached to it.

Encoding and decoding is the process of going from one to the other:

Encoding vs decoding (Image: Real Python)

In .encode() and .decode(), the encoding parameter is "utf-8" by default, though it’s generally safer and more unambiguous to specify it:

>>>

>>> "résumé".encode("utf-8")
b'r\xc3\xa9sum\xc3\xa9'
>>> "El Niño".encode("utf-8")
b'El Ni\xc3\xb1o'

>>> b"r\xc3\xa9sum\xc3\xa9".decode("utf-8")
'résumé'
>>> b"El Ni\xc3\xb1o".decode("utf-8")
'El Niño'

The results of str.encode() is a bytes object. Both bytes literals (such as b"r\xc3\xa9sum\xc3\xa9") and the representations of bytes permit only ASCII characters.

This is why, when calling "El Niño".encode("utf-8"), the ASCII-compatible "El" is allowed to be represented as it is, but the n with tilde is escaped to "\xc3\xb1". That messy-looking sequence represents two bytes, 0xc3 and 0xb1 in hex:

>>>

>>> " ".join(f"{i:08b}" for i in (0xc3, 0xb1))
'11000011 10110001'

That is, the character ñ requires two bytes for its binary representation under UTF-8.

Note: If you type help(str.encode), you’ll probably see a default of encoding='utf-8'. Be careful about excluding this and just using "résumé".encode(), because the default may be different in Windows prior to Python 3.6.

Python 3: All-In on Unicode

Python 3 is all-in on Unicode and UTF-8 specifically. Here’s what that means:

Python 3 source code is assumed to be UTF-8 by default. This means that you don’t need # -*- coding: UTF-8 -*- at the top of .py files in Python 3.
All text (str) is Unicode by default. Encoded Unicode text is represented as binary data (bytes). The str type can contain any literal Unicode character, such as "Δv / Δt", all of which will be stored as Unicode.
Anything from the Unicode character set is kosher in identifiers, meaning résumé = "~/Documents/resume.pdf" is valid if this strikes your fancy.
Python’s re module defaults to the re.UNICODE flag rather than re.ASCII. This means, for instance, that r"\w" matches Unicode word characters, not just ASCII letters.
The default encoding in str.encode() and bytes.decode() is UTF-8.

There is one other property that is more nuanced, which is that the default encoding to the built-in open() is platform-dependent and depends on the value of locale.getpreferredencoding():

>>>

>>> # Mac OS X High Sierra
>>> import locale
>>> locale.getpreferredencoding()
'UTF-8'

>>> # Windows Server 2012; other Windows builds may use UTF-16
>>> import locale
>>> locale.getpreferredencoding()
'cp1252'

Again, the lesson here is to be careful about making assumptions when it comes to the universality of UTF-8, even if it is the predominant encoding. It never hurts to be explicit in your code.

One Byte, Two Bytes, Three Bytes, Four

A crucial feature is that UTF-8 is a variable-length encoding. It’s tempting to gloss over what this means, but it’s worth delving into.

Think back to the section on ASCII. Everything in extended-ASCII-land demands at most one byte of space. You can quickly prove this with the following generator expression:

>>>

>>> all(len(chr(i).encode("ascii")) == 1 for i in range(128))
True

UTF-8 is quite different. A given Unicode character can occupy anywhere from one to four bytes. Here’s an example of a single Unicode character taking up four bytes:

>>>

>>> ibrow = "🤨"
>>> len(ibrow)
1
>>> ibrow.encode("utf-8")
b'\xf0\x9f\xa4\xa8'
>>> len(ibrow.encode("utf-8"))
4

>>> # Calling list() on a bytes object gives you
>>> # the decimal value for each byte
>>> list(b'\xf0\x9f\xa4\xa8')
[240, 159, 164, 168]

This is a subtle but important feature of len():

The length of a single Unicode character as a Python str will always be 1, no matter how many bytes it occupies.
The length of the same character encoded to bytes will be anywhere between 1 and 4.

The table below summarizes what general types of characters fit into each byte-length bucket:

Decimal Range	Hex Range	What’s Included	Examples
0 to 127	`"\u0000"` to `"\u007F"`	U.S. ASCII	`"A"`, `"\n"`, `"7"`, `"&"`
128 to 2047	`"\u0080"` to `"\u07FF"`	Most Latinic alphabets*	`"ę"`, `"±"`, `"ƌ"`, `"ñ"`
2048 to 65535	`"\u0800"` to `"\uFFFF"`	Additional parts of the multilingual plane (BMP)**	`"ത"`, `"ᄇ"`, `"ᮈ"`, `"‰"`
65536 to 1114111	`"\U00010000"` to `"\U0010FFFF"`	Other***	`"𝕂"`, `"𐀀"`, `"😓"`, `"🂲"`,

_{*Such as English, Arabic, Greek, and Irish}
_{**A huge array of languages and symbols—mostly Chinese, Japanese, and Korean by volume (also ASCII and Latin alphabets)}
_{***Additional Chinese, Japanese, Korean, and Vietnamese characters, plus more symbols and emojis}

Note: In the interest of not losing sight of the big picture, there is an additional set of technical features of UTF-8 that aren’t covered here because they are rarely visible to a Python user.

For instance, UTF-8 actually uses prefix codes that indicate the number of bytes in a sequence. This enables a decoder to tell what bytes belong together in a variable-length encoding, and lets the first byte serve as an indicator of the number of bytes in the coming sequence.

Wikipedia’s UTF-8 article does not shy away from technical detail, and there is always the official Unicode Standard for your reading enjoyment as well.

What About UTF-16 and UTF-32?

Let’s get back to two other encoding variants, UTF-16 and UTF-32.

The difference between these and UTF-8 is substantial in practice. Here’s an example of how major the difference is with a round-trip conversion:

>>>

>>> letters = "αβγδ"
>>> rawdata = letters.encode("utf-8")
>>> rawdata.decode("utf-8")
'αβγδ'
>>> rawdata.decode("utf-16")  # 😧
'뇎닎돎듎'

In this case, encoding four Greek letters with UTF-8 and then decoding back to text in UTF-16 would produce a text str that is in a completely different language (Korean).

Glaringly wrong results like this are possible when the same encoding isn’t used bidirectionally. Two variations of decoding the same bytes object may produce results that aren’t even in the same language.

This table summarizes the range or number of bytes under UTF-8, UTF-16, and UTF-32:

Encoding	Bytes Per Character (Inclusive)	Variable Length
UTF-8	1 to 4	Yes
UTF-16	2 to 4	Yes
UTF-32	4	No

One other curious aspect of the UTF family is that UTF-8 will not always take up less space than UTF-16. That may seem mathematically counterintuitive, but it’s quite possible:

>>>

>>> text = "記者 鄭啟源 羅智堅"
>>> len(text.encode("utf-8"))
26
>>> len(text.encode("utf-16"))
22

The reason for this is that the code points in the range U+0800 through U+FFFF (2048 through 65535 in decimal) take up three bytes in UTF-8 versus only two in UTF-16.

I’m not by any means recommending that you jump aboard the UTF-16 train, regardless of whether or not you operate in a language whose characters are commonly in this range. Among other reasons, one of the strong arguments for using UTF-8 is that, in the world of encoding, it’s a great idea to blend in with the crowd.

Not to mention, it’s 2019: computer memory is cheap, so saving 4 bytes by going out of your way to use UTF-16 is arguably not worth it.

Python’s Built-In Functions

You’ve made it through the hard part. Time to use what you’ve seen thus far in Python.

Python has a group of built-in functions that relate in some way to numbering systems and character encoding:

These can be logically grouped together based on their purpose:

ascii(), bin(), hex(), and oct() are for obtaining a different representation of an input. Each one produces a str. The first, ascii(), produces an ASCII only representation of an object, with non-ASCII characters escaped. The remaining three give binary, hexadecimal, and octal representations of an integer, respectively. These are only representations, not a fundamental change in the input.
bytes(), str(), and int() are class constructors for their respective types, bytes, str, and int. They each offer ways of coercing the input into the desired type. For instance, as you saw earlier, while int(11.0) is probably more common, you might also see int('11', base=16).
ord() and chr() are inverses of each other in that ord() converts a str character to its base-10 code point, while chr() does the opposite.

Here’s a more detailed look at each of these nine functions:

Function	Signature	Accepts	Return Type	Purpose
`ascii()`	`ascii(obj)`	Varies	`str`	ASCII only representation of an object, with non-ASCII characters escaped
`bin()`	`bin(number)`	`number: int`	`str`	Binary representation of an integer, with the prefix `"0b"`
`bytes()`	`bytes(iterable_of_ints)` `bytes(s, enc[, errors])` `bytes(bytes_or_buffer)` `bytes([i])`	Varies	`bytes`	Coerce (convert) the input to `bytes`, raw binary data
`chr()`	`chr(i)`	`i: int` `i>=0` `i<=1114111`	`str`	Convert an integer code point to a single Unicode character
`hex()`	`hex(number)`	`number: int`	`str`	Hexadecimal representation of an integer, with the prefix `"0x"`
`int()`	`int([x])` `int(x, base=10)`	Varies	`int`	Coerce (convert) the input to `int`
`oct()`	`oct(number)`	`number: int`	`str`	Octal representation of an integer, with the prefix `"0o"`
`ord()`	`ord(c)`	`c: str` `len(c) == 1`	`int`	Convert a single Unicode character to its integer code point
`str()`	`str(object=’‘)` `str(b[, enc[, errors]])`	Varies	`str`	Coerce (convert) the input to `str`, text

You can expand the section below to see some examples of each function.

ascii() gives you an ASCII-only representation of an object, with non-ASCII characters escaped:

>>>

>>> ascii("abcdefg")
"'abcdefg'"

>>> ascii("jalepeño")
"'jalepe\\xf1o'"

>>> ascii((1, 2, 3))
'(1, 2, 3)'

>>> ascii(0xc0ffee)  # Hex literal (int)
'12648430'

bin() gives you a binary representation of an integer, with the prefix "0b":

>>>

>>> bin(0)
'0b0'

>>> bin(400)
'0b110010000'

>>> bin(0xc0ffee)  # Hex literal (int)
'0b110000001111111111101110'

>>> [bin(i) for i in [1, 2, 4, 8, 16]]  # `int` + list comprehension
['0b1', '0b10', '0b100', '0b1000', '0b10000']

bytes() coerces the input to bytes, representing raw binary data:

>>>

>>> # Iterable of ints
>>> bytes((104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100))
b'hello world'

>>> bytes(range(97, 123))  # Iterable of ints
b'abcdefghijklmnopqrstuvwxyz'

>>> bytes("real 🐍", "utf-8")  # String + encoding
b'real \xf0\x9f\x90\x8d'

>>> bytes(10)
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

>>> bytes.fromhex('c0 ff ee')
b'\xc0\xff\xee'

>>> bytes.fromhex("72 65 61 6c 70 79 74 68 6f 6e")
b'realpython'

chr() converts an integer code point to a single Unicode character:

>>>

>>> chr(97)
'a'

>>> chr(7048)
'ᮈ'

>>> chr(1114111)
'\U0010ffff'

>>> chr(0x10FFFF)  # Hex literal (int)
'\U0010ffff'

>>> chr(0b01100100)  # Binary literal (int)
'd'

hex() gives the hexadecimal representation of an integer, with the prefix "0x":

>>>

>>> hex(100)
'0x64'

>>> [hex(i) for i in [1, 2, 4, 8, 16]]
['0x1', '0x2', '0x4', '0x8', '0x10']

>>> [hex(i) for i in range(16)]
['0x0', '0x1', '0x2', '0x3', '0x4', '0x5', '0x6', '0x7',
 '0x8', '0x9', '0xa', '0xb', '0xc', '0xd', '0xe', '0xf']

int() coerces the input to int, optionally interpreting the input in a given base:

>>>

>>> int(11.0)
11

>>> int('11')
11

>>> int('11', base=2)
3

>>> int('11', base=8)
9

>>> int('11', base=16)
17

>>> int(0xc0ffee - 1.0)
12648429

>>> int.from_bytes(b"\x0f", "little")
15

>>> int.from_bytes(b'\xc0\xff\xee', "big")
12648430

ord() converts a single Unicode character to its integer code point:

>>>

>>> ord("a")
97

>>> ord("ę")
281

>>> ord("ᮈ")
7048

>>> [ord(i) for i in "hello world"]
[104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100]

str() coerces the input to str, representing text:

>>>

>>> str("str of string")
'str of string'

>>> str(5)
'5'

>>> str([1, 2, 3, 4])  # Like [1, 2, 3, 4].__str__(), but use str()
'[1, 2, 3, 4]'

>>> str(b"\xc2\xbc cup of flour", "utf-8")
'¼ cup of flour'

>>> str(0xc0ffee)
'12648430'

Python String Literals: Ways to Skin a Cat

Rather than using the str() constructor, it’s commonplace to type a str literally:

>>>

>>> meal = "shrimp and grits"

That may seem easy enough. But the interesting side of things is that, because Python 3 is Unicode-centric through and through, you can “type” Unicode characters that you probably won’t even find on your keyboard. You can copy and paste this right into a Python 3 interpreter shell:

>>>

>>> alphabet = 'αβγδεζηθικλμνξοπρςστυφχψ'
>>> print(alphabet)
αβγδεζηθικλμνξοπρςστυφχψ

Besides placing the actual, unescaped Unicode characters in the console, there are other ways to type Unicode strings as well.

One of the densest sections of Python’s documentation is the portion on lexical analysis, specifically the section on string and bytes literals. Personally, I had to read this section about one, two, or maybe nine times for it to really sink in.

Part of what it says is that there are up to six ways that Python will allow you to type the same Unicode character.

The first and most common way is to type the character itself literally, as you’ve already seen. The tough part with this method is finding the actual keystrokes. That’s where the other methods for getting and representing characters come into play. Here’s the full list:

Escape Sequence	Meaning	How To Express `"a"`
`"\ooo"`	Character with octal value `ooo`	`"\141"`
`"\xhh"`	Character with hex value `hh`	`"\x61"`
`"\N{name}"`	Character named `name` in the Unicode database	`"\N{LATIN SMALL LETTER A}"`
`"\uxxxx"`	Character with 16-bit (2-byte) hex value `xxxx`	`"\u0061"`
`"\Uxxxxxxxx"`	Character with 32-bit (4-byte) hex value `xxxxxxxx`	`"\U00000061"`

Here’s some proof and validation of the above:

>>>

>>> (
...     "a" ==
...     "\x61" == 
...     "\N{LATIN SMALL LETTER A}" ==
...     "\u0061" ==
...     "\U00000061"
... )
True

Now, there are two main caveats:

Not all of these forms work for all characters. The hex representation of the integer 300 is 0x012c, which simply isn’t going to fit into the 2-hex-digit escape code "\xhh". The highest code point that you can squeeze into this escape sequence is "\xff" ("ÿ"). Similarly for "\ooo", it will only work up to "\777" ("ǿ").
For \xhh, \uxxxx, and \Uxxxxxxxx, exactly as many digits are required as are shown in these examples. This can throw you for a loop because of the way that Unicode tables conventionally display the codes for characters, with a leading U+ and variable number of hex characters. They key is that Unicode tables most often do not zero-pad these codes.

For instance, if you consult unicode-table.com for information on the Gothic letter faihu (or fehu), "𐍆", you’ll see that it is listed as having the code U+10346.

How do you put this into "\uxxxx" or "\Uxxxxxxxx"? Well, you can’t fit it in "\uxxxx" because it’s a 4-byte character, and to use "\Uxxxxxxxx" to represent this character, you’ll need to left-pad the sequence:

>>>

>>> "\U00010346"
'𐍆'

This also means that the "\Uxxxxxxxx" form is the only escape sequence that is capable of holding any Unicode character.

Note: Here’s a short function to convert strings that look like "U+10346" into something Python can work with. It uses str.zfill():

>>>

>>> def make_uchr(code: str):
...     return chr(int(code.lstrip("U+").zfill(8), 16))
>>> make_uchr("U+10346")
'𐍆'
>>> make_uchr("U+0026")
'&'

Other Encodings Available in Python

So far, you’ve seen four character encodings:

ASCII
UTF-8
UTF-16
UTF-32

There are a ton of other ones out there.

One example is Latin-1 (also called ISO-8859-1), which is technically the default for the Hypertext Transfer Protocol (HTTP), per RFC 2616. Windows has its own Latin-1 variant called cp1252.

Note: ISO-8859-1 is still very much present out in the wild. The requests library follows RFC 2616 “to the letter” in using it as the default encoding for the content of an HTTP/HTTPS response. If the word “text” is found in the Content-Type header, and no other encoding is specified, then requests will use ISO-8859-1.

The complete list of accepted encodings is buried way down in the documentation for the codecs module, which is part of Python’s Standard Library.

There’s one more useful recognized encoding to be aware of, which is "unicode-escape". If you have a decoded str and want to quickly get a representation of its escaped Unicode literal, then you can specify this encoding in .encode():

>>>

>>> alef = chr(1575)  # Or "\u0627"
>>> alef_hamza = chr(1571)  # Or "\u0623"
>>> alef, alef_hamza
('ا', 'أ')
>>> alef.encode("unicode-escape")
b'\\u0627'
>>> alef_hamza.encode("unicode-escape")
b'\\u0623'

You Know What They Say About Assumptions…

Just because Python makes the assumption of UTF-8 encoding for files and code that you generate doesn’t mean that you, the programmer, should operate with the same assumption for external data.

Let’s say that again because it’s a rule to live by: when you receive binary data (bytes) from a third party source, whether it be from a file or over a network, the best practice is to check that the data specifies an encoding. If it doesn’t, then it’s on you to ask.

All I/O happens in bytes, not text, and bytes are just ones and zeros to a computer until you tell it otherwise by informing it of an encoding.

Here’s an example of where things can go wrong. You’re subscribed to an API that sends you a recipe of the day, which you receive in bytes and have always decoded using .decode("utf-8") with no problem. On this particular day, part of the recipe looks like this:

>>>

>>> data = b"\xbc cup of flour"

It looks as if the recipe calls for some flour, but we don’t know how much:

>>>

>>> data.decode("utf-8")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbc in position 0: invalid start byte

Uh oh. There’s that pesky UnicodeDecodeError that can bite you when you make assumptions about encoding. You check with the API host. Lo and behold, the data is actually sent over encoded in Latin-1:

>>>

>>> data.decode("latin-1")
'¼ cup of flour'

There we go. In Latin-1, every character fits into a single byte, whereas the “¼” character takes up two bytes in UTF-8 ("\xc2\xbc").

The lesson here is that it can be dangerous to assume the encoding of any data that is handed off to you. It’s usually UTF-8 these days, but it’s the small percentage of cases where it’s not that will blow things up.

If you really do need to abandon ship and guess an encoding, then have a look at the chardet library, which uses methodology from Mozilla to make an educated guess about ambiguously encoded text. That said, a tool like chardet should be your last resort, not your first.

Odds and Ends: `unicodedata`

We would be remiss not to mention unicodedata from the Python Standard Library, which lets you interact with and do lookups on the Unicode Character Database (UCD):

>>>

>>> import unicodedata

>>> unicodedata.name("€")
'EURO SIGN'
>>> unicodedata.lookup("EURO SIGN")
'€'

Wrapping Up

In this article, you’ve decoded the wide and imposing subject of character encoding in Python.

You’ve covered a lot of ground here:

Fundamental concepts of character encodings and numbering systems
Integer, binary, octal, hex, str, and bytes literals in Python
Python’s built-in functions related to character encoding and numbering systems
Python 3’s treatment of text versus binary data

Now, go forth and encode!

Resources

For even more detail about the topics covered here, check out these resources:

Joel Spolsky: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
David Zentgraf: What every programmer absolutely, positively needs to know about encodings and character sets to work with text
Mozilla: A composite approach to language/encoding detection
Wikipedia: UTF-8
John Skeet: Unicode and .NET
Charles Petzold: Code: The Hidden Language of Computer Hardware and Software
Network Working Group, RFC 3629: UTF-8, a transformation format of ISO 10646
Unicode Technical Standard #18: Unicode Regular Expressions

The Python docs have two pages on the subject:

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Installing Python on Windows, macOS, and Linux

2019-05-16T14:00:00+00:00

To get started working with Python, you’ll need to have access to the Python interpreter. There are several common ways to accomplish this.

In this course, you will learn how to install the latest version of Python 3 on your computer.

Many operating systems, such as macOS and Linux, come with Python pre-installed. The version of Python that comes with your operating system is called your system Python.

The system Python is almost always out-of-date, and may not even be a full Python installation. It’s essential that you have the most recent version of Python so that you can follow along successfully with the examples in this book.

There are two major versions of Python available: Python 2, also known as legacy Python, and Python 3. Python 2 was released in the year 2000 and will reach its end-of-life on January 1, 2020. This course focuses on Python 3.

It is split into three sections: Windows, macOS, and Linux. Just find the section for your operating system and follow the steps to get your computer set-up.

If you have a different operating system, check out the Python 3 Installation & Setup Guide maintained on realpython.com to see if your OS is covered.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Three Ways of Storing and Accessing Lots of Images in Python

2019-05-15T14:00:00+00:00

Why would you want to know more about different ways of storing and accessing images in Python? If you’re segmenting a handful of images by color or detecting faces one by one using OpenCV, then you don’t need to worry about it. Even if you’re using the Python Imaging Library (PIL) to draw on a few hundred photos, you still don’t need to. Storing images on disk, as .png or .jpg files, is both suitable and appropriate.

Increasingly, however, the number of images required for a given task is getting larger and larger. Algorithms like convolutional neural networks, also known as convnets or CNNs, can handle enormous datasets of images and even learn from them. If you’re interested, you can read more about how convnets can be used for ranking selfies or for sentiment analysis.

ImageNet is a well-known public image database put together for training models on tasks like object classification, detection, and segmentation, and it consists of over 14 million images.

Think about how long it would take to load all of them into memory for training, in batches, perhaps hundreds or thousands of times. Keep reading, and you’ll be convinced that it would take quite awhile—at least long enough to leave your computer and do many other things while you wish you worked at Google or NVIDIA.

In this tutorial, you’ll learn about:

Storing images on disk as .png files
Storing images in lightning memory-mapped databases (LMDB)
Storing images in hierarchical data format (HDF5)

You’ll also explore the following:

Why alternate storage methods are worth considering
What the performance differences are when you’re reading and writing single images
What the performance differences are when you’re reading and writing many images
How the three methods compare in terms of disk usage

If none of the storage methods ring a bell, don’t worry: for this article, all you need is a reasonably solid foundation in Python and a basic understanding of images (that they are really composed of multi-dimensional arrays of numbers) and relative memory, such as the difference between 10MB and 10GB.

Let’s get started!

Free Bonus: Click here to get the Python Face Detection & OpenCV Examples Mini-Guide that shows you practical code examples of real-world Python computer vision techniques.

Setup

You will need an image dataset to experiment with, as well as a few Python packages.

A Dataset to Play With

We will be using the Canadian Institute for Advanced Research image dataset, better known as CIFAR-10, which consists of 60,000 32x32 pixel color images belonging to different object classes, such as dogs, cats, and airplanes. Relatively, CIFAR is not a very large dataset, but if we were to use the full TinyImages dataset, then you would need about 400GB of free disk space, which would probably be a limiting factor.

Credits for the dataset as described in chapter 3 of this tech report go to Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.

If you’d like to follow along with the code examples in this article, you can download CIFAR-10 here, selecting the Python version. You’ll be sacrificing 163MB of disk space:

Image: A. Krizhevsky

When you download and unzip the folder, you’ll discover that the files are not human-readable image files. They have actually been serialized and saved in batches using cPickle.

While we won’t consider pickle or cPickle in this article, other than to extract the CIFAR dataset, it’s worth mentioning that the Python pickle module has the key advantage of being able to serialize any Python object without any extra code or transformation on your part. It also has a potentially serious disadvantage of posing a security risk and not coping well when dealing with very large quantities of data.

The following code unpickles each of the five batch files and loads all of the images into a NumPy array:

import numpy as np
import pickle
from pathlib import Path

# Path to the unzipped CIFAR data
data_dir = Path("data/cifar-10-batches-py/")

# Unpickle function provided by the CIFAR hosts
def unpickle(file):
    with open(file, "rb") as fo:
        dict = pickle.load(fo, encoding="bytes")
    return dict

images, labels = [], []
for batch in data_dir.glob("data_batch_*"):
    batch_data = unpickle(batch)
    for i, flat_im in enumerate(batch_data[b"data"]):
        im_channels = []
        # Each image is flattened, with channels in order of R, G, B
        for j in range(3):
            im_channels.append(
                flat_im[j * 1024 : (j + 1) * 1024].reshape((32, 32))
            )
        # Reconstruct the original image
        images.append(np.dstack((im_channels)))
        # Save the label
        labels.append(batch_data[b"labels"][i])

print("Loaded CIFAR-10 training set:")
print(f" - np.shape(images)     {np.shape(images)}")
print(f" - np.shape(labels)     {np.shape(labels)}")

All the images are now in RAM in the images variable, with their corresponding meta data in labels, and are ready for you to manipulate. Next, you can install the Python packages you’ll use for the three methods.

Note: That last code block used f-strings. You can read more about them in Python 3’s f-Strings: An Improved String Formatting Syntax (Guide).

Setup for Storing Images on Disk

You’ll need to set up your environment for the default method of saving and accessing these images from disk. This article will assume you have Python 3.x installed on your system, and will use Pillow for the image manipulation:

$ pip install Pillow

Alternatively, if you prefer, you can install it using Anaconda:

$ conda install -c conda-forge pillow

Note: PIL is the original version of the Python Imaging Library, which is no longer maintained and is not compatible with Python 3.x. If you have previously installed PIL, make sure to uninstall it before installing Pillow, as they can’t exist together.

Now you’re ready for storing and reading images from disk.

Getting Started With LMDB

LMDB, sometimes referred to as the “Lightning Database,” stands for Lightning Memory-Mapped Database because it’s fast and uses memory-mapped files. It’s a key-value store, not a relational database.

In terms of implementation, LMDB is a B+ tree, which basically means that it is a tree-like graph structure stored in memory where each key-value element is a node, and nodes can have many children. Nodes on the same level are linked to one another for fast traversal.

Critically, key components of the B+ tree are set to correspond to the page size of the host operating system, maximizing efficiency when accessing any key-value pair in the database. Since LMDB high-performance heavily relies on this particular point, LMDB efficiency has been shown to be dependent on the underlying file system and its implementation.

Another key reason for the efficiency of LMDB is that it is memory-mapped. This means that it returns direct pointers to the memory addresses of both keys and values, without needing to copy anything in memory as most other databases do.

Those who want to dive into a bit more of the internal implementation details of B+ trees can check out this article on B+ trees and then play with this visualization of node insertion.

If B+ trees don’t interest you, don’t worry. You don’t need to know much about their internal implementation in order to use LMDB. We will be using the Python binding for the LMDB C library, which can be installed via pip:

$ pip install lmdb

You also have the option of installing via Anaconda:

$ conda install -c conda-forge python-lmdb

Check that you can import lmdb from a Python shell, and you’re good to go.

Getting Started With HDF5

HDF5 stands for Hierarchical Data Format, a file format referred to as HDF4 or HDF5. We don’t need to worry about HDF4, as HDF5 is the current maintained version.

Interestingly, HDF has its origins in the National Center for Supercomputing Applications, as a portable, compact scientific data format. If you’re wondering if it’s widely used, check out NASA’s blurb on HDF5 from their Earth Data project.

HDF files consist of two types of objects:

Datasets
Groups

Datasets are multidimensional arrays, and groups consist of datasets or other groups. Multidimensional arrays of any size and type can be stored as a dataset, but the dimensions and type have to be uniform within a dataset. Each dataset must contain a homogeneous N-dimensional array. That said, because groups and datasets may be nested, you can still get the heterogeneity you may need:

$ pip install h5py

As with the other libraries, you can alternately install via Anaconda:

$ conda install -c conda-forge h5py

If you can import h5py from a Python shell, everything is set up properly.

Storing a Single Image

Now that you have a general overview of the methods, let’s dive straight in and look at a quantitative comparison of the basic tasks we care about: how long it takes to read and write files, and how much disk memory will be used. This will also serve as a basic introduction to how the methods work, with code examples of how to use them.

When I refer to “files,” I generally mean a lot of them. However, it is important to make a distinction since some methods may be optimized for different operations and quantities of files.

For the purposes of experimentation, we can compare the performance between various quantities of files, by factors of 10 from a single image to 100,000 images. Since our five batches of CIFAR-10 add up to 50,000 images, we can use each image twice to get to 100,000 images.

To prepare for the experiments, you will want to create a folder for each method, which will contain all the database files or images, and save the paths to those directories in variables:

from pathlib import Path

disk_dir = Path("data/disk/")
lmdb_dir = Path("data/lmdb/")
hdf5_dir = Path("data/hdf5/")

Path does not automatically create the folders for you unless you specifically ask it to:

disk_dir.mkdir(parents=True, exist_ok=True)
lmdb_dir.mkdir(parents=True, exist_ok=True)
hdf5_dir.mkdir(parents=True, exist_ok=True)

Now you can move on to running the actual experiments, with code examples of how to perform basic tasks with the three different methods. We can use the timeit module, which is included in the Python standard library, to help time the experiments.

Although the main purpose of this article is not to learn the APIs of the different Python packages, it is helpful to have an understanding of how they can be implemented. We will go through the general principles alongside all the code used to conduct the storing experiments.

Storing to Disk

Our input for this experiment is a single image image, currently in memory as a NumPy array. You want to save it first to disk as a .png image, and name it using a unique image ID image_id. This can be done using the Pillow package you installed earlier:

from PIL import Image
import csv

def store_single_disk(image, image_id, label):
    """ Stores a single image as a .png file on disk.
        Parameters:
        ---------------
        image       image array, (32, 32, 3) to be stored
        image_id    integer unique ID for image
        label       image label
    """
    Image.fromarray(image).save(disk_dir / f"{image_id}.png")

    with open(disk_dir / f"{image_id}.csv", "wt") as csvfile:
        writer = csv.writer(
            csvfile, delimiter=" ", quotechar="|", quoting=csv.QUOTE_MINIMAL
        )
        writer.writerow([label])

This saves the image. In all realistic applications, you also care about the meta data attached to the image, which in our example dataset is the image label. When you’re storing images to disk, there are several options for saving the meta data.

One solution is to encode the labels into the image name. This has the advantage of not requiring any extra files.

However, it also has the big disadvantage of forcing you to deal with all the files whenever you do anything with labels. Storing the labels in a separate file allows you to play around with the labels alone, without having to load the images. Above, I have stored the labels in a separate .csv files for this experiment.

Now let’s move on to doing the exact same task with LMDB.

Storing to LMDB

Firstly, LMDB is a key-value storage system where each entry is saved as a byte array, so in our case, keys will be a unique identifier for each image, and the value will be the image itself. Both the keys and values are expected to be strings, so the common usage is to serialize the value as a string, and then unserialize it when reading it back out.

You can use pickle for the serializing. Any Python object can be serialized, so you might as well include the image meta data in the database as well. This saves you the trouble of attaching meta data back to the image data when we load the dataset from disk.

You can create a basic Python class for the image and its meta data:

class CIFAR_Image:
    def __init__(self, image, label):
        # Dimensions of image for reconstruction - not really necessary 
        # for this dataset, but some datasets may include images of 
        # varying sizes
        self.channels = image.shape[2]
        self.size = image.shape[:2]

        self.image = image.tobytes()
        self.label = label

    def get_image(self):
        """ Returns the image as a numpy array. """
        image = np.frombuffer(self.image, dtype=np.uint8)
        return image.reshape(*self.size, self.channels)

Secondly, because LMDB is memory-mapped, new databases need to know how much memory they are expected to use up. This is relatively straightforward in our case, but it can be a massive pain in other cases, which you will see in more depth in a later section. LMDB calls this variable the map_size.

Finally, read and write operations with LMDB are performed in transactions. You can think of them as similar to those of a traditional database, consisting of a group of operations on the database. This may look already significantly more complicated than the disk version, but hang on and keep reading!

With those three points in mind, let’s look at the code to save a single image to a LMDB:

import lmdb
import pickle

def store_single_lmdb(image, image_id, label):
    """ Stores a single image to a LMDB.
        Parameters:
        ---------------
        image       image array, (32, 32, 3) to be stored
        image_id    integer unique ID for image
        label       image label
    """
    map_size = image.nbytes * 10

    # Create a new LMDB environment
    env = lmdb.open(str(lmdb_dir / f"single_lmdb"), map_size=map_size)

    # Start a new write transaction
    with env.begin(write=True) as txn:
        # All key-value pairs need to be strings
        value = CIFAR_Image(image, label)
        key = f"{image_id:08}"
        txn.put(key.encode("ascii"), pickle.dumps(value))
    env.close()

Note: It’s a good idea to calculate the exact number of bytes each key-value pair will take up.

With a dataset of images of varying size, this will be an approximation, but you can use sys.getsizeof() to get a reasonable approximation. Keep in mind that sys.getsizeof(CIFAR_Image) will only return the size of a class definition, which is 1056, not the size of an instantiated object.

The function will also not be able to fully calculate nested items, lists, or objects containing references to other objects.

Alternately, you could use pympler to save you some calculations by determining the exact size of an object.

You are now ready to save an image to LMDB. Lastly, let’s look at the final method, HDF5.

Storing With HDF5

Remember that an HDF5 file can contain more than one dataset. In this rather trivial case, you can create two datasets, one for the image, and one for its meta data:

import h5py

def store_single_hdf5(image, image_id, label):
    """ Stores a single image to an HDF5 file.
        Parameters:
        ---------------
        image       image array, (32, 32, 3) to be stored
        image_id    integer unique ID for image
        label       image label
    """
    # Create a new HDF5 file
    file = h5py.File(hdf5_dir / f"{image_id}.h5", "w")

    # Create a dataset in the file
    dataset = file.create_dataset(
        "image", np.shape(image), h5py.h5t.STD_U8BE, data=image
    )
    meta_set = file.create_dataset(
        "meta", np.shape(label), h5py.h5t.STD_U8BE, data=label
    )
    file.close()

h5py.h5t.STD_U8BE specifies the type of data that will be stored in the dataset, which in this case is unsigned 8-bit integers. You can see a full list of HDF’s predefined datatypes here.

Note: The choice of datatype will strongly affect the runtime and storage requirements of HDF5, so it is best to choose your minimum requirements.

Now that we have reviewed the three methods of saving a single image, let’s move on to the next step.

Experiments for Storing a Single Image

Now you can put all three functions for saving a single image into a dictionary, which can be called later during the timing experiments:

_store_single_funcs = dict(
    disk=store_single_disk, lmdb=store_single_lmdb, hdf5=store_single_hdf5
)

Finally, everything is ready for conducting the timed experiment. Let’s try saving the first image from CIFAR and its corresponding label, and storing it in the three different ways:

from timeit import timeit

store_single_timings = dict()

for method in ("disk", "lmdb", "hdf5"):
    t = timeit(
        "_store_single_funcs[method](image, 0, label)",
        setup="image=images[0]; label=labels[0]",
        number=1,
        globals=globals(),
    )
    store_single_timings[method] = t
    print(f"Method: {method}, Time usage: {t}")

Note: While you’re playing around with LMDB, you may see a MapFullError: mdb_txn_commit: MDB_MAP_FULL: Environment mapsize limit reached error. It’s important to note that LMDB does not overwrite preexisting values, even if they have the same key.

This contributes to the fast write time, but it also means that if you store an image more than once in the same LMDB file, then you will use up the map size. If you run a store function, be sure to delete any preexisting LMDB files first.

Remember that we’re interested in runtime, displayed here in seconds, and also the memory usage:

Method	Save Single Image + Meta	Memory
Disk	1.915 ms	8 K
LMDB	1.203 ms	32 K
HDF5	8.243 ms	8 K

There are two takeaways here:

All of the methods are trivially quick.
In terms of disk usage, LMDB uses more.

Clearly, despite LMDB having a slight performance lead, we haven’t convinced anyone why to not just store images on disk. After all, it’s a human readable format, and you can open and view them from any file system browser! Well, it’s time to look at a lot more images…

Storing Many Images

You have seen the code for using the various storage methods to save a single image, so now we need to adjust the code to save many images and then run the timed experiment.

Adjusting the Code for Many Images

Saving multiple images as .png files is as straightforward as calling store_single_method() multiple times. But this isn’t true for LMDB or HDF5, since you don’t want a different database file for each image. Rather, you want to put all of the images into one or more files.

You will need to slightly alter the code and create three new functions that accept multiple images, store_many_disk(), store_many_lmdb(), and store_many_hdf5:

 store_many_disk(images, labels):
    """ Stores an array of images to disk
        Parameters:
        ---------------
        images       images array, (N, 32, 32, 3) to be stored
        labels       labels array, (N, 1) to be stored
    """
    num_images = len(images)

    # Save all the images one by one
    for i, image in enumerate(images):
        Image.fromarray(image).save(disk_dir / f"{i}.png")

    # Save all the labels to the csv file
    with open(disk_dir / f"{num_images}.csv", "w") as csvfile:
        writer = csv.writer(
            csvfile, delimiter=" ", quotechar="|", quoting=csv.QUOTE_MINIMAL
        )
        for label in labels:
            # This typically would be more than just one value per row
            writer.writerow([label])

def store_many_lmdb(images, labels):
    """ Stores an array of images to LMDB.
        Parameters:
        ---------------
        images       images array, (N, 32, 32, 3) to be stored
        labels       labels array, (N, 1) to be stored
    """
    num_images = len(images)

    map_size = num_images * images[0].nbytes * 10

    # Create a new LMDB DB for all the images
    env = lmdb.open(str(lmdb_dir / f"{num_images}_lmdb"), map_size=map_size)

    # Same as before — but let's write all the images in a single transaction
    with env.begin(write=True) as txn:
        for i in range(num_images):
            # All key-value pairs need to be Strings
            value = CIFAR_Image(images[i], labels[i])
            key = f"{i:08}"
            txn.put(key.encode("ascii"), pickle.dumps(value))
    env.close()

def store_many_hdf5(images, labels):
    """ Stores an array of images to HDF5.
        Parameters:
        ---------------
        images       images array, (N, 32, 32, 3) to be stored
        labels       labels array, (N, 1) to be stored
    """
    num_images = len(images)

    # Create a new HDF5 file
    file = h5py.File(hdf5_dir / f"{num_images}_many.h5", "w")

    # Create a dataset in the file
    dataset = file.create_dataset(
        "images", np.shape(images), h5py.h5t.STD_U8BE, data=images
    )
    meta_set = file.create_dataset(
        "meta", np.shape(labels), h5py.h5t.STD_U8BE, data=labels
    )
    file.close()

So you could store more than one file to disk, the image files method was altered to loop over each image in the list. For LMDB, a loop is also needed since we are creating a CIFAR_Image object for each image and its meta data.

The smallest adjustment is with the HDF5 method. In fact, there’s hardly an adjustment at all! HFD5 files have no limitation on file size aside from external restrictions or dataset size, so all the images were stuffed into a single dataset, just like before.

Next, you will need to prepare the dataset for the experiments by increasing its size.

Preparing the Dataset

Before running the experiments again, let’s first double our dataset size so that we can test with up to 100,000 images:

cutoffs = [10, 100, 1000, 10000, 100000]

# Let's double our images so that we have 100,000
images = np.concatenate((images, images), axis=0)
labels = np.concatenate((labels, labels), axis=0)

# Make sure you actually have 100,000 images and labels
print(np.shape(images))
print(np.shape(labels))

Now that there are enough images, it’s time for the experiment.

Experiment for Storing Many Images

As you did with reading many images, you can create a dictionary handling all the functions with store_many_ and run the experiments:

_store_many_funcs = dict(
    disk=store_many_disk, lmdb=store_many_lmdb, hdf5=store_many_hdf5
)

from timeit import timeit

store_many_timings = {"disk": [], "lmdb": [], "hdf5": []}

for cutoff in cutoffs:
    for method in ("disk", "lmdb", "hdf5"):
        t = timeit(
            "_store_many_funcs[method](images_, labels_)",
            setup="images_=images[:cutoff]; labels_=labels[:cutoff]",
            number=1,
            globals=globals(),
        )
        store_many_timings[method].append(t)

        # Print out the method, cutoff, and elapsed time
        print(f"Method: {method}, Time usage: {t}")

If you’re following along and running the code yourself, you’ll need to sit back a moment in suspense and wait for 111,110 images to be stored three times each to your disk, in three different formats. You’ll also need to say goodbye to approximately 2 GB of disk space.

Now for the moment of truth! How long did all of that storing take? A picture is worth a thousand words:

The first graph shows the normal, unadjusted storage time, highlighting the drastic difference between storing to .png files and LMDB or HDF5.

The second graph shows the log of the timings, highlighting that HDF5 starts out slower than LMDB but, with larger quantities of images, comes out slightly ahead.

While exact results may vary depending on your machine, this is why LMDB and HDF5 are worth thinking about. Here’s the code that generated the above graph:

import matplotlib.pyplot as plt

def plot_with_legend(
    x_range, y_data, legend_labels, x_label, y_label, title, log=False
):
    """ Displays a single plot with multiple datasets and matching legends.
        Parameters:
        --------------
        x_range         list of lists containing x data
        y_data          list of lists containing y values
        legend_labels   list of string legend labels
        x_label         x axis label
        y_label         y axis label
    """
    plt.style.use("seaborn-whitegrid")
    plt.figure(figsize=(10, 7))

    if len(y_data) != len(legend_labels):
        raise TypeError(
            "Error: number of data sets does not match number of labels."
        )

    all_plots = []
    for data, label in zip(y_data, legend_labels):
        if log:
            temp, = plt.loglog(x_range, data, label=label)
        else:
            temp, = plt.plot(x_range, data, label=label)
        all_plots.append(temp)

    plt.title(title)
    plt.xlabel(x_label)
    plt.ylabel(y_label)
    plt.legend(handles=all_plots)
    plt.show()

# Getting the store timings data to display
disk_x = store_many_timings["disk"]
lmdb_x = store_many_timings["lmdb"]
hdf5_x = store_many_timings["hdf5"]

plot_with_legend(
    cutoffs,
    [disk_x, lmdb_x, hdf5_x],
    ["PNG files", "LMDB", "HDF5"],
    "Number of images",
    "Seconds to store",
    "Storage time",
    log=False,
)

plot_with_legend(
    cutoffs,
    [disk_x, lmdb_x, hdf5_x],
    ["PNG files", "LMDB", "HDF5"],
    "Number of images",
    "Seconds to store",
    "Log storage time",
    log=True,
)

Now let’s go on to reading the images back out.

Reading a Single Image

First, let’s consider the case for reading a single image back into an array for each of the three methods.

Reading From Disk

Of the three methods, LMDB requires the most legwork when reading image files back out of memory, because of the serialization step. Let’s walk through these functions that read a single image out for each of the three storage formats.

First, read a single image and its meta from a .png and .csv file:

def read_single_disk(image_id):
    """ Stores a single image to disk.
        Parameters:
        ---------------
        image_id    integer unique ID for image

        Returns:
        ----------
        image       image array, (32, 32, 3) to be stored
        label       associated meta data, int label
    """
    image = np.array(Image.open(disk_dir / f"{image_id}.png"))

    with open(disk_dir / f"{image_id}.csv", "r") as csvfile:
        reader = csv.reader(
            csvfile, delimiter=" ", quotechar="|", quoting=csv.QUOTE_MINIMAL
        )
        label = int(next(reader)[0])

    return image, label

Reading From LMDB

Next, read the same image and meta from an LMDB by opening the environment and starting a read transaction:

 1 def read_single_lmdb(image_id):
 2     """ Stores a single image to LMDB.
 3         Parameters:
 4         ---------------
 5         image_id    integer unique ID for image
 6 
 7         Returns:
 8         ----------
 9         image       image array, (32, 32, 3) to be stored
10         label       associated meta data, int label
11     """
12     # Open the LMDB environment
13     env = lmdb.open(str(lmdb_dir / f"single_lmdb"), readonly=True)
14 
15     # Start a new read transaction
16     with env.begin() as txn:
17         # Encode the key the same way as we stored it
18         data = txn.get(f"{image_id:08}".encode("ascii"))
19         # Remember it's a CIFAR_Image object that is loaded
20         cifar_image = pickle.loads(data)
21         # Retrieve the relevant bits
22         image = cifar_image.get_image()
23         label = cifar_image.label
24     env.close()
25 
26     return image, label

Here are a couple points to not about the code snippet above:

Line 13: The readonly=True flag specifies that no writes will be allowed on the LMDB file until the transaction is finished. In database lingo, it’s equivalent to taking a read lock.
Line 20: To retrieve the CIFAR_Image object, you need to reverse the steps we took to pickle it when we were writing it. This is where the get_image() of the object is helpful.

This wraps up reading the image back out from LMDB. Finally, you will want to do the same with HDF5.

Reading From HDF5

Reading from HDF5 looks very similar to the writing process. Here is the code to open and read the HDF5 file and parse the same image and meta:

def read_single_hdf5(image_id):
    """ Stores a single image to HDF5.
        Parameters:
        ---------------
        image_id    integer unique ID for image

        Returns:
        ----------
        image       image array, (32, 32, 3) to be stored
        label       associated meta data, int label
    """
    # Open the HDF5 file
    file = h5py.File(hdf5_dir / f"{image_id}.h5", "r+")

    image = np.array(file["/image"]).astype("uint8")
    label = int(np.array(file["/meta"]).astype("uint8"))

    return image, label

Note that you access the various datasets in the file by indexing the file object using the dataset name preceded by a forward slash /. As before, you can create a dictionary containing all the read functions:

_read_single_funcs = dict(
    disk=read_single_disk, lmdb=read_single_lmdb, hdf5=read_single_hdf5
)

With this dictionary prepared, you are ready for running the experiment.

Experiment for Reading a Single Image

You might expect that the experiment for reading a single image in will have somewhat trivial results, but here’s the experiment code:

from timeit import timeit

read_single_timings = dict()

for method in ("disk", "lmdb", "hdf5"):
    t = timeit(
        "_read_single_funcs[method](0)",
        setup="image=images[0]; label=labels[0]",
        number=1,
        globals=globals(),
    )
    read_single_timings[method] = t
    print(f"Method: {method}, Time usage: {t}")

Here are the results of the experiment for reading a single image:

Method	Read Single Image + Meta
Disk	1.61970 ms
LMDB	4.52063 ms
HDF5	1.98036 ms

It’s slightly faster to read the .png and .csv files directly from disk, but all three methods perform trivially quickly. The experiments we’ll do next are much more interesting.

Reading Many Images

Now you can adjust the code to read many images at once. This is likely the action you’ll be performing most often, so the runtime performance is essential.

Adjusting the Code for Many Images

Extending the functions above, you can create functions with read_many_, which can be used for the next experiments. Like before, it is interesting to compare performance when reading different quantities of images, which are repeated in the code below for reference:

def read_many_disk(num_images):
    """ Reads image from disk.
        Parameters:
        ---------------
        num_images   number of images to read

        Returns:
        ----------
        images      images array, (N, 32, 32, 3) to be stored
        labels      associated meta data, int label (N, 1)
    """
    images, labels = [], []

    # Loop over all IDs and read each image in one by one
    for image_id in range(num_images):
        images.append(np.array(Image.open(disk_dir / f"{image_id}.png")))

    with open(disk_dir / f"{num_images}.csv", "r") as csvfile:
        reader = csv.reader(
            csvfile, delimiter=" ", quotechar="|", quoting=csv.QUOTE_MINIMAL
        )
        for row in reader:
            labels.append(int(row[0]))
    return images, labels

def read_many_lmdb(num_images):
    """ Reads image from LMDB.
        Parameters:
        ---------------
        num_images   number of images to read

        Returns:
        ----------
        images      images array, (N, 32, 32, 3) to be stored
        labels      associated meta data, int label (N, 1)
    """
    images, labels = [], []
    env = lmdb.open(str(lmdb_dir / f"{num_images}_lmdb"), readonly=True)

    # Start a new read transaction
    with env.begin() as txn:
        # Read all images in one single transaction, with one lock
        # We could split this up into multiple transactions if needed
        for image_id in range(num_images):
            data = txn.get(f"{image_id:08}".encode("ascii"))
            # Remember that it's a CIFAR_Image object 
            # that is stored as the value
            cifar_image = pickle.loads(data)
            # Retrieve the relevant bits
            images.append(cifar_image.get_image())
            labels.append(cifar_image.label)
    env.close()
    return images, labels

def read_many_hdf5(num_images):
    """ Reads image from HDF5.
        Parameters:
        ---------------
        num_images   number of images to read

        Returns:
        ----------
        images      images array, (N, 32, 32, 3) to be stored
        labels      associated meta data, int label (N, 1)
    """
    images, labels = [], []

    # Open the HDF5 file
    file = h5py.File(hdf5_dir / f"{num_images}_many.h5", "r+")

    images = np.array(file["/images"]).astype("uint8")
    labels = np.array(file["/meta"]).astype("uint8")

    return images, labels

_read_many_funcs = dict(
    disk=read_many_disk, lmdb=read_many_lmdb, hdf5=read_many_hdf5
)

With the reading functions stored in a dictionary as with the writing functions, you’re all set for the experiment.

Experiment for Reading Many Images

You can now run the experiment for reading many images out:

from timeit import timeit

read_many_timings = {"disk": [], "lmdb": [], "hdf5": []}

for cutoff in cutoffs:
    for method in ("disk", "lmdb", "hdf5"):
        t = timeit(
            "_read_many_funcs[method](num_images)",
            setup="num_images=cutoff",
            number=1,
            globals=globals(),
        )
        read_many_timings[method].append(t)

        # Print out the method, cutoff, and elapsed time
        print(f"Method: {method}, No. images: {cutoff}, Time usage: {t}")

As we did previously, you can graph the read experiment results:

The top graph shows the normal, unadjusted read times, showing the drastic difference between reading from .png files and LMDB or HDF5.

In contrast, the graph on the bottom shows the log of the timings, highlighting the relative differences with fewer images. Namely, we can see how HDF5 starts out behind but, with more images, becomes consistently faster than LMDB by a small margin.

Using the same plotting function as for the write timings, we have the following:

disk_x_r = read_many_timings["disk"]
lmdb_x_r = read_many_timings["lmdb"]
hdf5_x_r = read_many_timings["hdf5"]

plot_with_legend(
    cutoffs,
    [disk_x_r, lmdb_x_r, hdf5_x_r],
    ["PNG files", "LMDB", "HDF5"],
    "Number of images",
    "Seconds to read",
    "Read time",
    log=False,
)

plot_with_legend(
    cutoffs,
    [disk_x_r, lmdb_x_r, hdf5_x_r],
    ["PNG files", "LMDB", "HDF5"],
    "Number of images",
    "Seconds to read",
    "Log read time",
    log=True,
)

In practice, the write time is often less critical than the read time. Imagine that you are training a deep neural network on images, and only half of your entire image dataset fits into RAM at once. Each epoch of training a network requires the entire dataset, and the model needs a few hundred epochs to converge. You will essentially be reading half of the dataset into memory every epoch.

There are several tricks people do, such as training pseudo-epochs to make this slightly better, but you get the idea.

Now, look again at the read graph above. The difference between a 40-second and 4-second read time suddenly is the difference between waiting six hours for your model to train, or forty minutes!

If we view the read and write times on the same chart, we have the following:

You can plot all the read and write timings on a single graph using the same plotting function:

plot_with_legend(
    cutoffs,
    [disk_x_r, lmdb_x_r, hdf5_x_r, disk_x, lmdb_x, hdf5_x],
    [
        "Read PNG",
        "Read LMDB",
        "Read HDF5",
        "Write PNG",
        "Write LMDB",
        "Write HDF5",
    ],
    "Number of images",
    "Seconds",
    "Log Store and Read Times",
    log=False,
)

When you’re storing images as .png files, there is a big difference between write and read times. However, with LMDB and HDF5, the difference is much less marked. Overall, even if read time is more critical than write time, there is a strong argument for storing images using LMDB or HDF5.

Now that you’ve seen the performance benefits of LMDB and HDF5, let’s look at another crucial metric: disk usage.

Considering Disk Usage

Speed is not the only performance metric you may be interested in. We’re already dealing with very large datasets, so disk space is also a very valid and relevant concern.

Suppose you have an image dataset of 3TB. Presumably, you have them already on disk somewhere, unlike our CIFAR example, so by using an alternate storage method, you are essentially making a copy of them, which also has to be stored. Doing so will give you huge performance benefits when you use the images, but you’ll need to make sure you have enough disk space.

How much disk space do the various storage methods use? Here’s the disk space used for each method for each quantity of images:

I used the Linux du -h -c folder_name/* command to compute the disk usage on my system. There is some approximation inherent with this method due to rounding, but here’s the general comparison:

# Memory used in KB
disk_mem = [24, 204, 2004, 20032, 200296]
lmdb_mem = [60, 420, 4000, 39000, 393000]
hdf5_mem = [36, 304, 2900, 29000, 293000]

X = [disk_mem, lmdb_mem, hdf5_mem]

ind = np.arange(3)
width = 0.35

plt.subplots(figsize=(8, 10))
plots = [plt.bar(ind, [row[0] for row in X], width)]
for i in range(1, len(cutoffs)):
    plots.append(
        plt.bar(
            ind, [row[i] for row in X], width, bottom=[row[i - 1] for row in X]
        )
    )

plt.ylabel("Memory in KB")
plt.title("Disk memory used by method")
plt.xticks(ind, ("PNG", "LMDB", "HDF5"))
plt.yticks(np.arange(0, 400000, 100000))

plt.legend(
    [plot[0] for plot in plots], ("10", "100", "1,000", "10,000", "100,000")
)
plt.show()

Both HDF5 and LMDB take up more disk space than if you store using normal .png images. It’s important to note that both LMDB and HDF5 disk usage and performance depend highly on various factors, including operating system and, more critically, the size of the data you store.

LMDB gains its efficiency from caching and taking advantage of OS page sizes. You don’t need to understand its inner workings, but note that with larger images, you will end up with significantly more disk usage with LMDB, because images won’t fit on LMDB’s leaf pages, the regular storage location in the tree, and instead you will have many overflow pages. The LMDB bar in the chart above will shoot off the chart.

Our 32x32x3 pixel images are relatively small compared to the average images you may use, and they allow for optimal LMDB performance.

While we won’t explore it here experimentally, in my own experience with images of 256x256x3 or 512x512x3 pixels, HDF5 is usually slightly more efficient in terms of disk usage than LMDB. This is a good transition into the final section, a qualitative discussion of the differences between the methods.

Discussion

There are other distinguishing features of LMDB and HDF5 that are worth knowing about, and it’s also important to briefly discuss some of the criticisms of both methods. Several links are included along with the discussion if you want to learn more.

Parallel Access

A key comparison that we didn’t test in the experiments above is concurrent reads and writes. Often, with such large datasets, you may want to speed up your operation through parallelization.

In the majority of cases, you won’t be interested in reading parts of the same image at the same time, but you will want to read multiple images at once. With this definition of concurrency, storing to disk as .png files actually allows for complete concurrency. Nothing prevents you from reading several images at once from different threads, or writing multiple files at once, as long as the image names are different.

How about LMDB? There can be multiple readers on an LMDB environment at a time, but only one writer, and writers do not block readers. You can read more about that at the LMDB technology website.

Multiple applications can access the same LMDB database at the same time, and multiple threads from the same process can also concurrently access the LMDB for reads. This allows for even quicker read times: if you divided all of CIFAR into ten sets, then you could set up ten processes to each read in one set, and it would divide the loading time by ten.

HDF5 also offers parallel I/O, allowing concurrent reads and writes. However, in implementation, a write lock is held, and access is sequential, unless you have a parallel file system.

There are two main options if you are working on such a system, which are discussed more in depth in this article by the HDF Group on parallel IO. It can get quite complicated, and the simplest option is to intelligently split your dataset into multiple HDF5 files, such that each process can deal with one .h5 file independently of the others.

Documentation

If you Google lmdb, at least in the United Kingdom, the third search result is IMDb, the Internet Movie Database. That’s not what you were looking for!

Actually, there is one main source of documentation for the Python binding of LMDB, which is hosted on Read the Docs LMDB. While the Python package hasn’t even reached version > 0.94, it is quite widely used and is considered stable.

As for the LMDB technology itself, there is more detailed documentation at the LMDB technology website, which can feel a bit like learning calculus in second grade, unless you start from their Getting Started page.

For HDF5, there is very clear documentation at the h5py docs site, as well as a helpful blog post by Christopher Lovell, which is an excellent overview of how to use the h5py package. The O’Reilly book, Python and HDF5 also is a good way to get started.

While not as documented as perhaps a beginner would appreciate, both LMDB and HDF5 have large user communities, so a deeper Google search usually yields helpful results.

A More Critical Look at Implementation

There is no utopia in storage systems, and both LMDB and HDF5 have their share of pitfalls.

A key point to understand about LMDB is that new data is written without overwriting or moving existing data. This is a design decision that allows for the extremely quick reads you witnessed in our experiments, and also guarantees data integrity and reliability without the additional need of keeping transaction logs.

Remember, however, that you needed to define the map_size parameter for memory allocation before writing to a new database? This is where LMDB can be a hassle. Suppose you have created an LMDB database, and everything is wonderful. You’ve waited patiently for your enormous dataset to be packed into a LMDB.

Then, later down the line, you remember that you need to add new data. Even with the buffer you specified on your map_size, you may easily expect to see the lmdb.MapFullError error. Unless you want to re-write your entire database, with the updated map_size, you’ll have to store that new data in a separate LMDB file. Even though one transaction can span multiple LMDB files, having multiple files can still be a pain.

Additionally, some systems have restrictions on how much memory may be claimed at once. In my own experience, working with high-performance computing (HPC) systems, this has proved extremely frustrating, and has often made me prefer HDF5 over LMDB.

With both LMDB and HDF5, only the requested item is read into memory at once. With LMDB, key-unit pairs are read into memory one by one, while with HDF5, the dataset object can be accessed like a Python array, with indexing dataset[i], ranges, dataset[i:j] and other splicing dataset[i:j:interval].

Because of the way the systems are optimized, and depending on your operating system, the order in which you access items can impact performance.

In my experience, it’s generally true that for LMDB, you may get better performance when accessing items sequentially by key (key-value pairs being kept in memory ordered alphanumerically by key), and that for HDF5, accessing large ranges will perform better than reading every element of the dataset one by one using the following:

# Slightly slower
for i in range(len(dataset)):
    # Read the ith value in the dataset, one at a time
    do_something_with(dataset[i])

# This is better
data = dataset[:]
for d in data:
    do_something_with(d)

If you are considering a choice of file storage format to write your software around, it would be remiss not to mention Moving away from HDF5 by Cyrille Rossant on the pitfalls of HDF5, and Konrad Hinsen’s response On HDF5 and the future of data management, which shows how some of the pitfalls can be avoided in his own use cases with many smaller datasets rather than a few enormous ones. Note that a relatively smaller dataset is still several GB in size.

Integration With Other Libraries

If you’re dealing with really large datasets, it’s highly likely that you’ll be doing something significant with them. It’s worthwhile to consider deep learning libraries and what kind of integration there is with LMDB and HDF5.

First of all, all libraries support reading images from disk as .png files, as long as you convert them into NumPy arrays of the expected format. This holds true for all the methods, and we have already seen above that it is relatively straightforward to read in images as arrays.

Here are several of the most popular deep learning libraries and their LMDB and HDF5 integration:

Caffe has a stable, well-supported LMDB integration, and it handles the reading step transparently. The LMDB layer can also easily be replaced with a HDF5 database.
Keras uses the HDF5 format to save and restore models. This implies that TensorFlow can as well.
TensorFlow has a built-in class LMDBDataset that provides an interface for reading in input data from an LMDB file and can produce iterators and tensors in batches. TensorFlow does not have a built-in class for HDF5, but one can be written that inherits from the Dataset class. I personally use a custom class altogether that is designed for optimal read access based on the way I structure my HDF5 files.
Theano does not natively support any particular file format or database, but as previously stated, can use anything as long as it is read in as an N-dimensional array.

While far from comprehensive, this hopefully gives you a feel for the LMDB/HDF5 integration by some key deep learning libraries.

A Few Personal Insights on Storing Images in Python

In my own daily work analyzing terabytes of medical images, I use both LMDB and HDF5, and have learned that, with any storage method, forethought is critical.

Often, models need to be trained using k-fold cross validation, which involves splitting the entire dataset into k-sets (k typically being 10), and k models being trained, each with a different k-set used as test set. This ensures that the model is not overfitting the dataset, or, in other words, unable to make good predictions on unseen data.

A standard way to craft a k-set is to put an equal representation of each type of data represented in the dataset in each k-set. Thus, saving each k-set into a separate HDF5 dataset maximizes efficiency. Sometimes, a single k-set cannot be loaded into memory at once, so even the ordering of data within a dataset requires some forethought.

With LMDB, I similarly am careful to plan ahead before creating the database(s). There are a few good questions worth asking before you save images:

How can I save the images such that most of the reads will be sequential?
What are good keys?
How can I calculate a good map_size, anticipating potential future changes in the dataset?
How large can a single transaction be, and how should transactions be subdivided?

Regardless of the storage method, when you’re dealing with large image datasets, a little planning goes a long way.

Conclusion

You’ve made it to the end! You’ve now had a bird’s eye view of a large topic.

In this article, you’ve been introduced to three ways of storing and accessing lots of images in Python, and perhaps had a chance to play with some of them. All the code for this article is in a Jupyter notebook here or Python script here. Run at your own risk, as a few GB of your disk space will be overtaken by little square images of cars, boats, and so on.

You’ve seen evidence of how various storage methods can drastically affect read and write time, as well as a few pros and cons of the three methods considered in this article. While storing images as .png files may be the most intuitive, there are large performance benefits to considering methods such as HDF5 or LMDB.

Feel free to discuss in the comment section the excellent storage methods not covered in this article, such as LevelDB, Feather, TileDB, Badger, BoltDB, or anything else. There is no perfect storage method, and the best method depends on your specific dataset and use cases.

Writing Cleaner Python Code With PyLint

2019-05-14T14:00:00+00:00

PyLint is a well-known static analysis tool for Python 2 and 3. It has a number of useful features, like checking your code for compliance with the PEP 8 Python style guide. It makes sure that your code follows the code style guide and it can also automatically identify common bugs and errors in your Python code.

In this video series you’ll see how to install and set up the PyLint code linter tool. You’ll learn why you should use code linters like PyLint, Flake8, PyFlakes, or other static analysis tools—and how they can help you write cleaner and more Pythonic code.

You can get this setup up and running in a few minutes and it’ll quickly help you write better and cleaner Python code.

Additional Resources

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Context Managers and the "with" Statement

2019-05-07T14:00:00+00:00

Python’s with statement is powered by a language feature called context managers. It helps you simplify some common resource management patterns by abstracting their functionality and allowing them to be factored out and reused.

In this course you’ll learn how context managers and the with statement work in Python, including the difference between class-based and function-based context managers, as well as some other tricks that will make you a more efficient Python developer.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Make a Location-Based Web App With Django and GeoDjango

2019-04-30T14:00:00+00:00

Throughout this course, you’ll learn how to use Django and GeoDjango to build a location-based web application from scratch. You’ll be building a simple nearby shops application that lists the shops closest to a user’s location.

By the end of this course, you’ll be able to:

Use Django to build a simple web application from scratch
Use the GeoDjango sub-framework to implement geolocation features in your Django application
Use a spatial database (PostgreSQL and PostGIS) to get benefits from the spatial features and easily implement location-aware web apps

Downloadable Sample Project: This course includes a downloadable sample project that demonstrates the final state of the Django app you’ll build throughout the course. You can get the source code for it here on the materials repo.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Conditional Statements in Python (if/elif/else)

2019-04-25T14:00:00+00:00

In this step-by-step course you’ll learn how to work with conditional (“if”) statements in Python. Master if-statements step-by-step and see how to write complex decision making code in your programs.

Take the Quiz: Test your knowledge with our interactive “Python Conditional Statements” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Take the Quiz »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

How to Publish Your Own Python Package to PyPI

Python Community Interview With Marlene Mhangami

How to Build Command Line Interfaces in Python With argparse

What Is a Command Line Interface?

When to Use a Command Line Interface

How to Use the Python argparse Library to Create a Command Line Interface

The Advanced Use of the Python argparse Library

Setting the Name of the Program

Displaying a Custom Program Usage Help

Displaying Text Before and After the Arguments Help

Customizing the Allowed Prefix Chars

Setting Prefix Chars for Files That Contain Arguments to Be Included

Allowing or Disallowing Abbreviations

Using Auto Help

Setting the Name or Flags of the Arguments

Setting the Action to Be Taken for an Argument

Setting the Number of Values That Should Be Consumed by the Option

Setting a Default Value Produced if the Argument Is Missing

Setting the Type of the Argument

Setting a Domain of Allowed Values for a Specific Argument

Setting Whether the Argument Is Required

Showing a Brief Description of What an Argument Does

Defining Mutually Exclusive Groups

Setting the Argument Name in Usage Messages

Setting the Name of the Attribute to Be Added to the Object Once Parsed

Conclusion

OOP Method Types in Python: @classmethod vs @staticmethod vs Instance Methods

Recommended Resources

How to Make a Twitter Bot in Python With Tweepy

Hello Tweepy

Twitter API

What Is Tweepy?

Using Tweepy

Installation

Creating Twitter API Authentication Credentials

Review of Tweepy Functionality

How to Make a Twitter Bot in Python With Tweepy

Watching for Twitter Activity

Presenting the Example Bots

The Config Module

The Follow Followers Bot

The Fav & Retweet Bot

The Reply to Mentions Bot

Deploying Bots to a Server Using Docker

Building the Docker Image

Example: Deploying a Twitter Bot to Amazon AWS

Wrapping Up

How to Implement a Python Stack

What Is a Stack?

Implementing a Python Stack

Using list to Create a Python Stack

Using collections.deque to Create a Python Stack

Why Have deque and list?

Python Stacks and Threading

Python Stacks: Which Implementation Should You Use?

Conclusion

Introduction to Git and GitHub for Python Developers

Object-Oriented Programming in Python vs Java

Sample Classes in Python vs Java

Object Attributes

Declaration and Initialization

Public and Private

Access Control

self and this

Methods and Functions

Inheritance and Polymorphism

Inheritance

Types and Polymorphism

Default Methods

Operator Overloading

Reflection

Examining an Object’s Type

Examining an Object’s Attributes

Calling Methods Through Reflection

Conclusion

Pointers in Python: What's the Point?

Why Doesn’t Python Have Pointers?

Objects in Python

Immutable vs Mutable Objects

How to Use the Python `argparse` Library to Create a Command Line Interface

The Advanced Use of the Python `argparse` Library

Using `list` to Create a Python Stack

Using `collections.deque` to Create a Python Stack

Why Have `deque` and `list`?

`self` and `this`

Real Pointers With `ctypes`

Iterating Through `.items()`

Iterating Through `.keys()`

Iterating Through `.values()`

Iterating Destructively With `.popitem`()

`map()`

`filter()`

Using `collections.ChainMap`

Using `itertools`

Cyclic Iteration With `cycle()`

Chained Iteration With `chain()`

Using the Dictionary Unpacking Operator (`**`)

Preliminary #1: A Level Is Just an `int`!

The `LogRecord` Class

The `Logger` and `Handler` Classes

The `Filter` and `Filterer` Classes

The `Manager` Class

What Does `getLogger()` Really Do?

Library vs Application Logging: What Is `NullHandler`?

The `string` Module