Real Python

Basic Data Types in Python

2018-06-05T14:00:00+00:00

Now you know how to interact with the Python interpreter and execute Python code. It’s time to dig into the Python language. First up is a discussion of the basic data types that are built into Python.

Here’s what you’ll learn in this tutorial:

You’ll learn about several basic numeric, string, and Boolean types that are built into Python. By the end of this tutorial, you’ll be familiar with what objects of these types look like, and how to represent them.
You’ll also get an overview of Python’s built-in functions. These are pre-written chunks of code you can call to do useful things. You have already seen the built-in print() function, but there are many others.

Don't miss the follow up tutorial: Click here to join the Real Python Newsletter and you'll know when the next instalment comes out.

Integers

In Python 3, there is effectively no limit to how long an integer value can be. Of course, it is constrained by the amount of memory your system has, as are all things, but beyond that an integer can be as long as you need it to be:

>>> print(123123123123123123123123123123123123123123123123 + 1)
123123123123123123123123123123123123123123123124

Python interprets a sequence of decimal digits without any prefix to be a decimal number:

>>> print(10)
10

The following strings can be prepended to an integer value to indicate a base other than 10:

Prefix	Interpretation	Base
`0b` (zero + lowercase letter `'b'`) `0B` (zero + uppercase letter `'B'`)	Binary	2
`0o` (zero + lowercase letter `'o'`) `0O` (zero + uppercase letter `'O'`)	Octal	8
`0x` (zero + lowercase letter `'x'`) `0X` (zero + uppercase letter `'X'`)	Hexadecimal	16

For example:

>>> print(0o10)
8

>>> print(0x10)
16

>>> print(0b10)
2

For more information on integer values with non-decimal bases, see the following Wikipedia sites: Binary, Octal, and Hexadecimal.

The underlying type of a Python integer, irrespective of the base used to specify it, is called int:

>>> type(10)
<class 'int'>
>>> type(0o10)
<class 'int'>
>>> type(0x10)
<class 'int'>

Note: This is a good time to mention that if you want to display a value while in a REPL session, you don’t need to use the print() function. Just typing the value at the >>> prompt and hitting Enter will display it:

>>> 10
10
>>> 0x10
16
>>> 0b10
2

Many of the examples in this tutorial series will use this feature.

Note that this does not work inside a script file. A value appearing on a line by itself in a script file will not do anything.

Floating-Point Numbers

The float type in Python designates a floating-point number. float values are specified with a decimal point. Optionally, the character e or E followed by a positive or negative integer may be appended to specify scientific notation:

>>> 4.2
4.2
>>> type(4.2)
<class 'float'>
>>> 4.
4.0
>>> .2
0.2

>>> .4e7
4000000.0
>>> type(.4e7)
<class 'float'>
>>> 4.2e-4
0.00042

Deep Dive: Floating-Point Representation

The following is a bit more in-depth information on how Python represents floating-point numbers internally. You can readily use floating-point numbers in Python without understanding them to this level, so don’t worry if this seems overly complicated. The information is presented here in case you are curious.

Almost all platforms represent Python float values as 64-bit “double-precision” values, according to the IEEE 754 standard. In that case, the maximum value a floating-point number can have is approximately 1.8 ⨉ 10³⁰⁸. Python will indicate a number greater than that by the string inf:
1.79e308
1.79e+308
1.8e308
inf
The closest a nonzero number can be to zero is approximately 5.0 ⨉ 10^-324. Anything closer to zero than that is effectively zero:
5e-324
5e-324
1e-325
0.0
Floating point numbers are represented internally as binary (base-2) fractions. Most decimal fractions cannot be represented exactly as binary fractions, so in most cases the internal representation of a floating-point number is an approximation of the actual value. In practice, the difference between the actual value and the represented value is very small and should not usually cause significant problems.

Further Reading: For additional information on floating-point representation in Python and the potential pitfalls involved, see Floating Point Arithmetic: Issues and Limitations in the Python documentation.

Complex Numbers

Complex numbers are specified as <real part>+<imaginary part>j. For example:

>>> 2+3j
(2+3j)
>>> type(2+3j)
<class 'complex'>

Strings

Strings are sequences of character data. The string type in Python is called str.

String literals may be delimited using either single or double quotes. All the characters between the opening delimiter and matching closing delimiter are part of the string:

>>> print("I am a string.")
I am a string.
>>> type("I am a string.")
<class 'str'>

>>> print('I am too.')
I am too.
>>> type('I am too.')
<class 'str'>

A string in Python can contain as many characters as you wish. The only limit is your machine’s memory resources. A string can also be empty:

>>> ''
''

What if you want to include a quote character as part of the string itself? Your first impulse might be to try something like this:

>>> print('This string contains a single quote (') character.')
SyntaxError: invalid syntax

As you can see, that doesn’t work so well. The string in this example opens with a single quote, so Python assumes the next single quote, the one in parentheses which was intended to be part of the string, is the closing delimiter. The final single quote is then a stray and causes the syntax error shown.

If you want to include either type of quote character within the string, the simplest way is to delimit the string with the other type. If a string is to contain a single quote, delimit it with double quotes and vice versa:

>>> print("This string contains a single quote (') character.")
This string contains a single quote (') character.

>>> print('This string contains a double quote (") character.')
This string contains a double quote (") character.

Escape Sequences in Strings

Sometimes, you want Python to interpret a character or sequence of characters within a string differently. This may occur in one of two ways:

You may want to suppress the special interpretation that certain characters are usually given within a string.
You may want to apply special interpretation to characters in a string which would normally be taken literally.

You can accomplish this using a backslash (\) character. A backslash character in a string indicates that one or more characters that follow it should be treated specially. (This is referred to as an escape sequence, because the backslash causes the subsequent character sequence to “escape” its usual meaning.)

Let’s see how this works.

Suppressing Special Character Meaning

You have already seen the problems you can come up against when you try to include quote characters in a string. If a string is delimited by single quotes, you can’t directly specify a single quote character as part of the string because, for that string, the single quote has special meaning—it terminates the string:

>>> print('This string contains a single quote (') character.')
SyntaxError: invalid syntax

Specifying a backslash in front of the quote character in a string “escapes” it and causes Python to suppress its usual special meaning. It is then interpreted simply as a literal single quote character:

>>> print('This string contains a single quote (\') character.')
This string contains a single quote (') character.

The same works in a string delimited by double quotes as well:

>>> print("This string contains a double quote (\") character.")
This string contains a double quote (") character.

The following is a table of escape sequences which cause Python to suppress the usual special interpretation of a character in a string:

Escape Sequence	Usual Interpretation of Character(s) After Backslash	“Escaped” Interpretation
`\'`	Terminates string with single quote opening delimiter	Literal single quote (`'`) character
`\"`	Terminates string with double quote opening delimiter	Literal double quote (`"`) character
`\newline`	Terminates input line	Newline is ignored
`\\`	Introduces escape sequence	Literal backslash (`\`) character

Ordinarily, a newline character terminates line input. So pressing Enter in the middle of a string will cause Python to think it is incomplete:

>>> print('a

SyntaxError: EOL while scanning string literal

To break up a string over more than one line, include a backslash before each newline, and the newlines will be ignored:

>>> print('a\
... b\
... c')
abc

To include a literal backslash in a string, escape it with a backslash:

>>> print('foo\\bar')
foo\bar

Applying Special Meaning to Characters

Next, suppose you need to create a string that contains a tab character in it. Some text editors may allow you to insert a tab character directly into your code. But many programmers consider that poor practice, for several reasons:

The computer can distinguish between a tab character and a sequence of space characters, but you can’t. To a human reading the code, tab and space characters are visually indistinguishable.
Some text editors are configured to automatically eliminate tab characters by expanding them to the appropriate number of spaces.
Some Python REPL environments will not insert tabs into code.

In Python (and almost all other common computer languages), a tab character can be specified by the escape sequence \t:

>>> print('foo\tbar')
foo     bar

The escape sequence \t causes the t character to lose its usual meaning, that of a literal t. Instead, the combination is interpreted as a tab character.

Here is a list of escape sequences that cause Python to apply special meaning instead of interpreting literally:

Escape Sequence	“Escaped” Interpretation
`\a`	ASCII Bell (`BEL`) character
`\b`	ASCII Backspace (`BS`) character
`\f`	ASCII Formfeed (`FF`) character
`\n`	ASCII Linefeed (`LF`) character
`\N{<name>}`	Character from Unicode database with given `<name>`
`\r`	ASCII Carriage Return (`CR`) character
`\t`	ASCII Horizontal Tab (`TAB`) character
`\uxxxx`	Unicode character with 16-bit hex value `xxxx`
`\Uxxxxxxxx`	Unicode character with 32-bit hex value `xxxxxxxx`
`\v`	ASCII Vertical Tab (`VT`) character
`\ooo`	Character with octal value `ooo`
`\xhh`	Character with hex value `hh`

Examples:

>>> print("a\tb")
a    b
>>> print("a\141\x61")
aaa
>>> print("a\nb")
a
b
>>> print('\u2192 \N{rightwards arrow}')
→ →

This type of escape sequence is typically used to insert characters that are not readily generated from the keyboard or are not easily readable or printable.

Raw Strings

A raw string literal is preceded by r or R, which specifies that escape sequences in the associated string are not translated. The backslash character is left in the string:

>>> print('foo\nbar')
foo
bar
>>> print(r'foo\nbar')
foo\nbar

>>> print('foo\\bar')
foo\bar
>>> print(R'foo\\bar')
foo\\bar

Triple-Quoted Strings

There is yet another way of delimiting strings in Python. Triple-quoted strings are delimited by matching groups of three single quotes or three double quotes. Escape sequences still work in triple-quoted strings, but single quotes, double quotes, and newlines can be included without escaping them. This provides a convenient way to create a string with both single and double quotes in it:

>>> print('''This string has a single (') and a double (") quote.''')
This string has a single (') and a double (") quote.

Because newlines can be included without escaping them, this also allows for multiline strings:

>>> print("""This is a
string that spans
across several lines""")
This is a
string that spans
across several lines

You will see in the upcoming tutorial on Python Program Structure how triple-quoted strings can be used to add an explanatory comment to Python code.

Boolean Type, Boolean Context, and “Truthiness”

Python 3 provides a Boolean data type. Objects of Boolean type may have one of two values, True or False:

>>> type(True)
<class 'bool'>
>>> type(False)
<class 'bool'>

As you will see in upcoming tutorials, expressions in Python are often evaluated in Boolean context, meaning they are interpreted to represent truth or falsehood. A value that is true in Boolean context is sometimes said to be “truthy,” and one that is false in Boolean context is said to be “falsy.” (You may also see “falsy” spelled “falsey.”)

The “truthiness” of an object of Boolean type is self-evident: Boolean objects that are equal to True are truthy (true), and those equal to False are falsy (false). But non-Boolean objects can be evaluated in Boolean context as well and determined to be true or false.

You will learn more about evaluation of objects in Boolean context when you encounter logical operators in the upcoming tutorial on operators and expressions in Python.

Built-In Functions

The Python interpreter supports many functions that are built-in: sixty-eight, as of Python 3.6. You will cover many of these in the following discussions, as they come up in context.

For now, a brief overview follows, just to give a feel for what is available. See the Python documentation on built-in functions for more detail. Many of the following descriptions refer to topics and concepts that will be discussed in future tutorials.

Math

Function	Description
`abs()`	Returns absolute value of a number
`divmod()`	Returns quotient and remainder of integer division
`max()`	Returns the largest of the given arguments or items in an iterable
`min()`	Returns the smallest of the given arguments or items in an iterable
`pow()`	Raises a number to a power
`round()`	Rounds a floating-point value
`sum()`	Sums the items of an iterable

Type Conversion

Function	Description
`ascii()`	Returns a string containing a printable representation of an object
`bin()`	Converts an integer to a binary string
`bool()`	Converts an argument to a Boolean value
`chr()`	Returns string representation of character given by integer argument
`complex()`	Returns a complex number constructed from arguments
`float()`	Returns a floating-point object constructed from a number or string
`hex()`	Converts an integer to a hexadecimal string
`int()`	Returns an integer object constructed from a number or string
`oct()`	Converts an integer to an octal string
`ord()`	Returns integer representation of a character
`repr()`	Returns a string containing a printable representation of an object
`str()`	Returns a string version of an object
`type()`	Returns the type of an object or creates a new type object

Iterables and Iterators

Function	Description
`all()`	Returns `True` if all elements of an iterable are true
`any()`	Returns `True` if any elements of an iterable are true
`enumerate()`	Returns a list of tuples containing indices and values from an iterable
`filter()`	Filters elements from an iterable
`iter()`	Returns an iterator object
`len()`	Returns the length of an object
`map()`	Applies a function to every item of an iterable
`next()`	Retrieves the next item from an iterator
`range()`	Generates a range of integer values
`reversed()`	Returns a reverse iterator
`slice()`	Returns a `slice` object
`sorted()`	Returns a sorted list from an iterable
`zip()`	Creates an iterator that aggregates elements from iterables

Composite Data Type

Function	Description
`bytearray()`	Creates and returns an object of the `bytearray` class
`bytes()`	Creates and returns a `bytes` object (similar to `bytearray`, but immutable)
`dict()`	Creates a `dict` object
`frozenset()`	Creates a `frozenset` object
`list()`	Constructs a `list` object
`object()`	Returns a new featureless object
`set()`	Creates a `set` object
`tuple()`	Creates a `tuple` object

Classes, Attributes, and Inheritance

Function	Description
`classmethod()`	Returns a class method for a function
`delattr()`	Deletes an attribute from an object
`getattr()`	Returns the value of a named attribute of an object
`hasattr()`	Returns `True` if an object has a given attribute
`isinstance()`	Determines whether an object is an instance of a given class
`issubclass()`	Determines whether a class is a subclass of a given class
`property()`	Returns a property value of a class
`setattr()`	Sets the value of a named attribute of an object
`super()`	Returns a proxy object that delegates method calls to a parent or sibling class

Input/Output

Function	Description
`format()`	Converts a value to a formatted representation
`input()`	Reads input from the console
`open()`	Opens a file and returns a file object
`print()`	Prints to a text stream or the console

Variables, References, and Scope

Function	Description
`dir()`	Returns a list of names in current local scope or a list of object attributes
`globals()`	Returns a dictionary representing the current global symbol table
`id()`	Returns the identity of an object
`locals()`	Updates and returns a dictionary representing current local symbol table
`vars()`	Returns `__dict__` attribute for a module, class, or object

Miscellaneous

Function	Description
`callable()`	Returns `True` if object appears callable
`compile()`	Compiles source into a code or AST object
`eval()`	Evaluates a Python expression
`exec()`	Implements dynamic execution of Python code
`hash()`	Returns the hash value of an object
`help()`	Invokes the built-in help system
`memoryview()`	Returns a memory view object
`staticmethod()`	Returns a static method for a function
`__import__()`	Invoked by the `import` statement

Conclusion

In this tutorial, you learned about the built-in data types and functions Python provides.

The examples given so far have all manipulated and displayed only constant values. In most programs, you are usually going to want to create objects that change in value as the program executes.

Head to the next tutorial to learn about Python variables.

Don't miss the follow up tutorial: Click here to join the Real Python Newsletter and you'll know when the next instalment comes out.

« Interacting with Python

Basic Data Types in Python

Variables in Python »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Application Layouts: A Reference

2018-06-04T14:00:00+00:00

Python, though opinionated on syntax and style, is surprisingly flexible when it comes to structuring your applications.

On the one hand, this flexibility is great: it allows different use cases to use structures that are necessary for those use cases. On the other hand, though, it can be very confusing to the new developer.

The Internet isn’t a lot of help either—there are as many opinions as there are Python blogs. In this article, I want to give you a dependable Python application layout reference guide that you can refer to for the vast majority of your use cases.

You’ll see examples of common Python application structures, including command-line applications (CLI apps), one-off scripts, installable packages, and web application layouts with popular frameworks like Flask and Django.

Note: This reference guide assumes a working knowledge of Python modules and packages. Check out our introduction to Python modules and packages for a refresher if you’re feeling a little rusty.

Command-Line Application Layouts

A lot of us work primarily with Python applications that are run via command-line interfaces (CLIs). This is where you often start with a blank canvas, and the flexibility of Python application layouts can be a real headache.

Starting with an empty project folder can be intimidating and lead to no shortage of coder’s block. In this section, I want to share some proven layouts that I personally use as a starting point for all of my Python CLI applications.

We’ll start with a very basic layout for a very basic use case: a simple script that runs on its own. You’ll then see how to build up the layout as the use cases advance.

One-Off Script

You just make a .py script, and it’s gravy, right? No need to install—just run the script in its directory!

Well, that’s fine if you’re just making a script for your own use, or one that doesn’t have any external dependencies, but what if you have to distribute it? Especially to a less tech-savvy user?

The following layout will work for all of these cases and can easily be modified to reflect whatever installation or other tools you use in your workflow. This layout will cover you whether you’re creating a pure Python script (that is, one with no dependencies) or using a tool like pip or Pipenv.

While you read this reference guide, keep in mind that the exact location of the files in the layout matters less than the reason they are placed where they are. All of these files should be in a project directory named after your project. For this example, we will use (what else?) helloworld as the project name and root directory.

Here’s the Python project structure I typically use for a CLI app:

helloworld/
│
├── .gitignore
├── helloworld.py
├── LICENSE
├── README.md
├── requirements.txt
├── setup.py
└── tests.py

This is pretty straightforward: everything is in the same directory. The files shown here are not necessarily exhaustive, but I recommend keeping the number of files to a minimum if you plan on using a basic layout like this. Some of these files will be new to you, so let’s take a quick look at what each of them does.

.gitignore: This is a file that tells Git which kinds of files to ignore, like IDE clutter or local configuration files. Our Git tutorial has all the details, and you can find sample .gitignore files for Python projects here.
helloworld.py: This is the script that you’re distributing. As far as naming the main script file goes, I recommend that you go with the name of your project (which is the same as the name of the top-level directory).
LICENSE: This plaintext file describes the license you’re using for a project. It’s always a good idea to have one if you’re distributing code. The filename is in all caps by convention.

Note: Need help selecting a license for your project? Check out ChooseALicense.
README.md: This is a Markdown (or reStructuredText) file documenting the purpose and usage of your application. Crafting a good README is an art, but you can find a shortcut to mastery here.
requirements.txt: This file defines outside Python dependencies and their versions for your application.
setup.py: This file can also be used to define dependencies, but it really shines for other work that needs to be done during installation. You can read more about both setup.py and requirements.txt in our guide to Pipenv.
tests.py: This script houses your tests, if you have any. You should have some.

But now that your application is growing, and you’ve broken it out into multiple pieces within the same package, should you keep all pieces in the top-level directory? Now that your application is more complex, it’s time to organize things more cleanly.

Installable Single Package

Let’s imagine that helloworld.py is still the main script to execute, but you’ve moved all helper methods to a new file called helpers.py.

We are going to package the helloworld Python files together but keep all the miscellaneous files, such as your README, .gitignore, and so on, at the top directory.

Let’s take a look at the updated structure:

helloworld/
│
├── helloworld/
│   ├── __init__.py
│   ├── helloworld.py
│   └── helpers.py
│
├── tests/
│   ├── helloworld_tests.py
│   └── helpers_tests.py
│
├── .gitignore
├── LICENSE
├── README.md
├── requirements.txt
└── setup.py

The only difference here is that your application code is now all held in the helloworld subdirectory—this directory is named after your package—and that we’ve added a file called __init__.py. Let’s introduce these new files:

helloworld/__init__.py: This file has many functions, but for our purposes it tells the Python interpreter that this directory is a package directory. You can set up this __init__.py file in a way that enables you to import classes and methods from the package as a whole, instead of knowing the internal module structure and importing from helloworld.helloworld or helloworld.helpers.

Note: For a deeper discussion on internal packages and __init__.py, our Python modules and packages overview has you covered.
helloworld/helpers.py: As mentioned above, we’ve moved much of helloworld.py’s business logic to this file. Thanks to __init__.py, outside modules will be able to access these helpers simply by importing from the helloworld package.
tests/: We’ve moved our tests into their own directory, a pattern you’ll continue to see as our program structures gain complexity. We have also split our tests into separate modules, mirroring our package’s structure.

This layout is a stripped down version of Kenneth Reitz’s samplemod application structure. It is another great starting point for your CLI applications, especially for more expansive projects.

Application with Internal Packages

In larger applications, you may have one or more internal packages that are either tied together with a main runner script or that provide specific functionality to a larger library you are packaging. We will extend the conventions laid out above to accommodate for this:

helloworld/
│
├── bin/
│
├── docs/
│   ├── hello.md
│   └── world.md
│
├── helloworld/
│   ├── __init__.py
│   ├── runner.py
│   ├── hello/
│   │   ├── __init__.py
│   │   ├── hello.py
│   │   └── helpers.py
│   │
│   └── world/
│       ├── __init__.py
│       ├── helpers.py
│       └── world.py
│
├── data/
│   ├── input.csv
│   └── output.xlsx
│
├── tests/
│   ├── hello
│   │   ├── helpers_tests.py
│   │   └── hello_tests.py
│   │
│   └── world/
│       ├── helpers_tests.py
│       └── world_tests.py
│
├── .gitignore
├── LICENSE
└── README.md

There’s a bit more to digest here, but as long as you remember that it follows from the previous layout, you will have an easier time following along. I’ll go through the additions and modifications in order, their uses, and the reasons you might want them.

bin/: This directory holds any executable files. I’ve adapted this from Jean-Paul Calderone’s classic structure post, and his prescriptions for the use of a bin/ directory are still important. The most important point to remember is that your executable shouldn’t have a lot of code, just an import and a call to a main function in your runner script. If you are using pure Python or don’t have any executable files, you can leave out this directory.
/docs: With a more advanced application, you’ll want to maintain good documentation of all its parts. I like to put any documentation for internal modules here, which is why you see separate documents for the hello and world packages. If you use docstrings in your internal modules (and you should!), your whole-module documentation should at the very least give a holistic view of the purpose and function of the module.
helloworld/: This is similar to helloworld/ in the previous structure, but now there are subdirectories. As you add more complexity, you’ll want to use a “divide and conquer” tactic and split out parts of your application logic into more manageable chunks. Remember that the directory name refers to the overall package name, and so the subdirectory names (hello/ and world/) should reflect their package names.
data/: Having this directory is helpful for testing. It’s a central location for any files that your application will ingest or produce. Depending on how you deploy your application, you can keep “production-level” inputs and outputs pointed to this directory, or only use it for internal testing.
tests/: Here, you can put all your tests—unit tests, execution tests, integration tests, and so on. Feel free to structure this directory in the most convenient way for your testing strategies, import strategies, and more. For a refresher on testing command-line applications with Python, check out my article 4 Techniques for Testing Python Command-Line (CLI) Apps.

The top-level files remain largely the same as in the previous layout. These three layouts should cover most use cases for command-line applications, and even GUI applications with the caveat that you may have to tinker with some things depending on the GUI framework you use.

Note: Keep in mind that these are just layouts. If a directory or file doesn’t make sense for your specific use case (like tests/ if you aren’t distributing tests with your code), feel free to leave it out. But try not to leave out docs/. It’s always a good idea to document your work.

Web Application Layouts

Another major use case of Python is web applications. Django and Flask are arguably the most popular web frameworks for Python and thankfully are a little more opinionated when it comes to application layout.

In order to make sure this article is a complete, full-fledged layout reference, I wanted to highlight the structure common to these frameworks.

Django

Let’s go in alphabetical order and start with Django. One of the nice things about Django is that it will create a project skeleton for you after running django-admin startproject project, where project is the name of your project. This will create a directory in your current working directory called project with the following internal structure:

project/
│
├── project/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
│
└── manage.py

This seems a little empty, doesn’t it? Where does all the logic go? The views? There aren’t even any tests!

In Django, this is a project, which ties together the other Django concept, apps. Apps are where logic, models, views, and so on all live, and in doing so they do some task, such as maintaining a blog.

Django apps can be imported into projects and used across projects, and are structured like specialized Python packages.

Like projects, Django makes generating Django app layouts really easy. After you set up your project, all you have to do is navigate to the location of manage.py and run python manage.py startapp app, where app is the name of your app.

This will result in a directory called app with the following layout:

app/
│
├── migrations/
│   └── __init__.py
│
├── __init__.py
├── admin.py
├── apps.py
├── models.py
├── tests.py
└── views.py

This can then be imported directly into your project. Details on what these files do, how to harness them for your project, and so forth are outside the scope of this reference, but you can get all that information and more in our Django tutorial and also in the official Django docs.

This file and folder structure is very barebones and the basic requirements for Django. For any open-source Django project, you can (and should) adapt the structures from the command-line application layouts. I typically end up with something like this in the outer project/ directory:

project/
│
├── app/
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   │
│   ├── migrations/
│   │   └── __init__.py
│   │
│   ├── models.py
│   ├── tests.py
│   └── views.py
│
├── docs/
│
├── project/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
│
├── static/
│   └── style.css
│
├── templates/
│   └── base.html
│
├── .gitignore
├── manage.py
├── LICENSE
└── README.md

For a deeper discussion on more advanced Django application layouts, this Stack Overflow thread has you covered. The django-project-skeleton project documentation explains some of the directories you will find in the Stack Overflow thread. A comprehensive dive into Django can be found in the pages of Two Scoops of Django, which will teach you all of the latest best practices for Django development.

For more Django tutorials, visit our Django section at Real Python.

Flask

Flask is a Python web “microframework.” One of the main selling points is that it is very quick to set up with minimal overhead. The Flask documentation has a web application example that’s under 10 lines of code and in a single script. Of course, in practice, it’s highly unlikely you’ll be writing a web application this small.

Luckily, the Flask documentation swoops in to save us with a suggested layout for their tutorial project (a blogging web application called Flaskr), and we will examine that here from within the main project directory:

flaskr/
│
├── flaskr/
│   ├── ___init__.py
│   ├── db.py
│   ├── schema.sql
│   ├── auth.py
│   ├── blog.py
│   ├── templates/
│   │   ├── base.html
│   │   ├── auth/
│   │   │   ├── login.html
│   │   │   └── register.html
│   │   │
│   │   └── blog/
│   │       ├── create.html
│   │       ├── index.html
│   │       └── update.html
│   │ 
│   └── static/
│       └── style.css
│
├── tests/
│   ├── conftest.py
│   ├── data.sql
│   ├── test_factory.py
│   ├── test_db.py
│   ├── test_auth.py
│   └── test_blog.py
│
├── venv/
│
├── .gitignore
├── setup.py
└── MANIFEST.in

From these contents, we can see that a Flask application, like most Python applications, is built around Python packages.

Note: Not seeing it? A quick tip for spotting packages is by looking for an __init__.py file. This sits in the highest-level directory for that particular package. In the above layout, flaskr is a package containing the db, auth, and blog modules.

In this layout, everything lives in the flaskr package except for your tests, a directory for your virtual environments, and your usual top-level files. As in other layouts, your tests will roughly match the individual modules residing within the flaskr package. Your templates also reside in the main project package, which would not happen with the Django layouts.

Be sure to also visit our Flask Boilerplate Github page for a view of a more fully fleshed-out Flask application and see the boilerplate in action here.

For more on Flask, check out all of our Flask tutorials here.

Conclusions and Reminders

Now you’ve seen example layouts for a number of different application types: one-off Python scripts, installable single packages, larger applications with internal packages, Django web applications, and Flask web applications.

Coming away from this guide, you will have the tools to successfully prevent coder’s block by building out your application structure so that you’re not staring at a blank canvas trying to figure out where to start.

Because Python is largely non-opinionated when it comes to application layouts, you can customize these example layouts to your heart’s content to better fit your use case.

I want you to not only have an application layout reference but also come away with the understanding that these examples are neither hard-and-fast rules nor the only way to structure your application. Over time and with practice, you’ll develop the ability to build and customize your own useful Python application layouts.

Did I miss a use case? Do you have another application structure philosophy? Did this article help prevent coder’s block? Let me know in the comments!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Itertools in Python 3, By Example

2018-05-30T14:00:00+00:00

It has been called a “gem” and “pretty much the coolest thing ever,” and if you have not heard of it, then you are missing out on one of the greatest corners of the Python 3 standard library: itertools.

A handful of excellent resources exist for learning what functions are available in the itertools module. The docs themselves are a great place to start. So is this post.

The thing about itertools, though, is that it is not enough to just know the definitions of the functions it contains. The real power lies in composing these functions to create fast, memory-efficient, and good-looking code.

This article takes a different approach. Rather than introducing itertools to you one function at a time, you will construct practical examples designed to encourage you to “think iteratively.” In general, the examples will start simple and gradually increase in complexity.

A word of warning: this article is long and intended for the intermediate-to-advanced Python programmer. Before diving in, you should be confident using iterators and generators in Python 3, multiple assignment, and tuple unpacking. If you aren’t, or if you need to brush up on your knowledge, consider checking out the following before reading on:

Python Iterators: A Step-By-Step Introduction
Introduction to Python Generators
Chapter 6 of Python Tricks: The Book by Dan Bader
Multiple assignment and tuple unpacking improve Python code readability

Free Bonus: Click here to get our itertools cheat sheet that summarizes the techniques demonstrated in this tutorial.

All set? Let’s start the way any good journey should—with a question.

What Is `Itertools` and Why Should You Use It?

According to the itertools docs, it is a “module [that] implements a number of iterator building blocks inspired by constructs from APL, Haskell, and SML… Together, they form an ‘iterator algebra’ making it possible to construct specialized tools succinctly and efficiently in pure Python.”

Loosely speaking, this means that the functions in itertools “operate” on iterators to produce more complex iterators. Consider, for example, the built-in zip() function, which takes any number of iterables as arguments and returns an iterator over tuples of their corresponding elements:

>>> list(zip([1, 2, 3], ['a', 'b', 'c']))
[(1, 'a'), (2, 'b'), (3, 'c')]

How, exactly, does zip() work?

[1, 2, 3] and ['a', 'b', 'c'], like all lists, are iterable, which means they can return their elements one at a time. Technically, any Python object that implements the .__iter__() or .__getitem__() methods is iterable. (See the Python 3 docs glossary for a more detailed explanation.)

The iter() built-in function, when called on an iterable, returns an iterator object for that iterable:

>>> iter([1, 2, 3, 4])
<list_iterator object at 0x7fa80af0d898>

Under the hood, the zip() function works, in essence, by calling iter() on each of its arguments, then advancing each iterator returned by iter() with next() and aggregating the results into tuples. The iterator returned by zip() iterates over these tuples.

The map() built-in function is another “iterator operator” that, in its simplest form, applies a single-parameter function to each element of an iterable one element at a time:

>>> list(map(len, ['abc', 'de', 'fghi']))
[3, 2, 4]

The map() function works by calling iter() on its second argument, advancing this iterator with next() until the iterator is exhausted, and applying the function passed to its first argument to the value returned by next() at each step. In the above example, len() is called on each element of ['abc', 'de', 'fghi'] to return an iterator over the lengths of each string in the list.

Since iterators are iterable, you can compose zip() and map() to produce an iterator over combinations of elements in more than one iterable. For example, the following sums corresponding elements of two lists:

>>> list(map(sum, zip([1, 2, 3], [4, 5, 6])))
[5, 7, 9]

This is what is meant by the functions in itertools forming an “iterator algebra.” itertools is best viewed as a collection of building blocks that can be combined to form specialized “data pipelines” like the one in the example above.

Historical Note: In Python 2, the built-in zip() and map() functions do not return an iterator, but rather a list. To return an iterator, the izip() and imap() functions of itertools must be used. In Python 3, izip() and imap() have been removed from itertools and replaced the zip() and map() built-ins. So, in a way, if you have ever used zip() or map() in Python 3, you have already been using itertools!

There are two main reasons why such an “iterator algebra” is useful: improved memory efficiency (via lazy evaluation) and faster execuction time. To see this, consider the following problem:

Given a list of values inputs and a positive integer n, write a function that splits inputs into groups of length n. For simplicity, assume that the length of the input list is divisible by n. For example, if inputs = [1, 2, 3, 4, 5, 6] and n = 2, your function should return [(1, 2), (3, 4), (5, 6)].

Taking a naive approach, you might write something like this:

def naive_grouper(inputs, n):
    num_groups = len(inputs) // n
    return [tuple(inputs[i*n:(i+1)*n]) for i in range(num_groups)]

When you test it, you see that it works as expected:

>>> nums = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> naive_grouper(nums, 2)
[(1, 2), (3, 4), (5, 6), (7, 8), (9, 10)]

What happens when you try to pass it a list with, say, 100 million elements? You will need a whole lot of available memory! Even if you have enough memory available, your program will hang for a while until the output list is populated.

To see this, store the following in a script called naive.py:

def naive_grouper(inputs, n):
    num_groups = len(inputs) // n
    return [tuple(inputs[i*n:(i+1)*n]) for i in range(num_groups)]


for _ in naive_grouper(range(100000000), 10):
    pass

From the console, you can use the time command (on UNIX systems) to measure memory usage and CPU user time. Make sure you have at least 5GB of free memory before executing the following:

$ time -f "Memory used (kB): %M\nUser time (seconds): %U" python3 naive.py
Memory used (kB): 4551872
User time (seconds): 11.04

Note: On Ubuntu, you may need to run /usr/bin/time instead of time for the above example to work.

The list and tuple implementation in naive_grouper() requires approximately 4.5GB of memory to process range(100000000). Working with iterators drastically improves this situation. Consider the following:

def better_grouper(inputs, n):
    iters = [iter(inputs)] * n
    return zip(*iters)

There’s a lot going on in this little function, so let’s break it down with a concrete example. The expression [iters(inputs)] * n creates a list of n references to the same iterator:

>>> nums = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> iters = [iter(nums)] * 2
>>> list(id(itr) for itr in iters)  # IDs are the same.
[139949748267160, 139949748267160]

Next, zip(*iters) returns an iterator over pairs of corresponding elements of each iterator in iters. When the first element, 1, is taken from the “first” iterator, the “second” iterator now starts at 2 since it is just a reference to the “first” iterator and has therefore been advanced one step. So, the first tuple produced by zip() is (1, 2).

At this point, “both” iterators in iters start at 3, so when zip() pulls 3 from the “first” iterator, it gets 4 from the “second” to produce the tuple (3, 4). This process continues until zip() finally produces (9, 10) and “both” iterators in iters are exhausted:

>>> nums = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> list(better_grouper(nums, 2))
[(1, 2), (3, 4), (5, 6), (7, 8), (9, 10)]

The better_grouper() function is better for a couple of reasons. First, without the reference to the len() built-in, better_grouper() can take any iterable as an argument (even infinite iterators). Second, by returning an iterator rather than a list, better_grouper() can process enormous iterables without trouble and uses much less memory.

Store the following in a file called better.py and run it with time from the console again:

def better_grouper(inputs, n):
    iters = [iter(inputs)] * n
    return zip(*iters)


for _ in better_grouper(range(100000000), 10):
    pass

$ time -f "Memory used (kB): %M\nUser time (seconds): %U" python3 better.py
Memory used (kB): 7224
User time (seconds): 2.48

That’s a whopping 630 times less memory used than naive.py in less than a quarter of the time!

Now that you’ve seen what itertools is (“iterator algebra”) and why you should use it (improved memory efficiency and faster execution time), let’s take a look at how to take better_grouper() to the next level with itertools.

The `grouper` Recipe

The problem with better_grouper() is that it doesn’t handle situations where the value passed to the second argument isn’t a factor of the length of the iterable in the first argument:

>>> nums = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> list(better_grouper(nums, 4))
[(1, 2, 3, 4), (5, 6, 7, 8)]

The elements 9 and 10 are missing from the grouped output. This happens because zip() stops aggregating elements once the shortest iterable passed to it is exhausted. It would make more sense to return a third group containing 9 and 10.

To do this, you can use itertools.zip_longest(). This function accepts any number of iterables as arguments and a fillvalue keyword argument that defaults to None. The easiest way to get a sense of the difference between zip() and zip_longest() is to look at some example output:

>>> import itertools as it
>>> x = [1, 2, 3, 4, 5]
>>> y = ['a', 'b', 'c']
>>> list(zip(x, y))
[(1, 'a'), (2, 'b'), (3, 'c')]
>>> list(it.zip_longest(x, y))
[(1, 'a'), (2, 'b'), (3, 'c'), (4, None), (5, None)]

With this in mind, replace zip() in better_grouper() with zip_longest():

import itertools as it


def grouper(inputs, n, fillvalue=None):
    iters = [iter(inputs)] * n
    return it.zip_longest(*iters, fillvalue=fillvalue)

Now you get a better result:

>>> nums = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> print(list(grouper(nums, 4)))
[(1, 2, 3, 4), (5, 6, 7, 8), (9, 10, None, None)]

The grouper() function can be found in the Recipes section of the itertools docs. The recipes are an excellent source of inspiration for ways to use itertools to your advantage.

Note: From this point forward, the line import itertools as it will not be included at the beginning of examples. All itertools methods in code examples are prefaced with it. The module import is implied.

If you get a NameError: name 'itertools' is not defined or a NameError: name 'it' is not defined exception when running one of the examples in this tutorial you’ll need to import the itertools module first.

Et tu, Brute Force?

Here’s a common interview-style problem:

You have three $20 dollar bills, five $10 dollar bills, two $5 dollar bills, and five $1 dollar bills. How many ways can you make change for a $100 dollar bill?

To “brute force” this problem, you just start listing off the ways there are to choose one bill from your wallet, check whether any of these makes change for $100, then list the ways to pick two bills from your wallet, check again, and so on and so forth.

But you are a programmer, so naturally you want to automate this process.

First, create a list of the bills you have in your wallet:

bills = [20, 20, 20, 10, 10, 10, 10, 10, 5, 5, 1, 1, 1, 1, 1]

A choice of k things from a set of n things is called a combination, and itertools has your back here. The itertools.combinations() function takes two arguments—an iterable inputs and a positive integer n—and produces an iterator over tuples of all combinations of n elements in inputs.

For example, to list the combinations of three bills in your wallet, just do:

>>> list(it.combinations(bills, 3))
 [(20, 20, 20), (20, 20, 10), (20, 20, 10), ... ]

To solve the problem, you can loop over the positive integers from 1 to len(bills), then check which combinations of each size add up to $100:

>>> makes_100 = []
>>> for n in range(1, len(bills) + 1):
...     for combination in it.combinations(bills, n):
...         if sum(combination) == 100:
...             makes_100.append(combination)

If you print out makes_100, you will notice there are a lot of repeated combinations. This makes sense because you can make change for $100 with three $20 dollar bills and four $10 bills, but combinations() does this with the first four $10 dollars bills in your wallet; the first, third, fourth and fifth $10 dollar bills; the first, second, fourth and fifth $10 bills; and so on.

To remove duplicates from makes_100, you can convert it to a set:

>>> set(makes_100)
{(20, 20, 10, 10, 10, 10, 10, 5, 1, 1, 1, 1, 1),
 (20, 20, 10, 10, 10, 10, 10, 5, 5),
 (20, 20, 20, 10, 10, 10, 5, 1, 1, 1, 1, 1),
 (20, 20, 20, 10, 10, 10, 5, 5),
 (20, 20, 20, 10, 10, 10, 10)}

So, there are five ways to make change for a $100 bill with the bills you have in your wallet.

Here’s a variation on the same problem:

How many ways are there to make change for a $100 bill using any number of $50, $20, $10, $5, and $1 dollar bills?

In this case, you don’t have a pre-set collection of bills, so you need a way to generate all possible combinations using any number of bills. For this, you’ll need the itertools.combinations_with_replacement() function.

It works just like combinations(), accepting an iterable inputs and a positive integer n, and returns an iterator over n-tuples of elements from inputs. The difference is that combinations_with_replacement() allows elements to be repeated in the tuples it returns.

For example:

>>> list(it.combinations_with_replacement([1, 2], 2))
[(1, 1), (1, 2), (2, 2)]

Compare that to combinations():

>>> list(it.combinations([1, 2], 2))
[(1, 2)]

Here’s what the solution to the revised problem looks like:

>>> bills = [50, 20, 10, 5, 1]
>>> make_100 = []
>>> for n in range(1, 101):
...     for combination in it.combinations_with_replacement(bills, n):
...         if sum(combination) == 100:
...             makes_100.append(combination)

In this case, you do not need to remove any duplicates since combinations_with_replacement() won’t produce any:

>>> len(makes_100)
343

If you run the above solution, you may notice that it takes a while for the output to display. That is because it has to process 96,560,645 combinations!

Another “brute force” itertools function is permutations(), which accepts a single iterable and produces all possible permutations (rearrangements) of its elements:

>>> list(it.permutations(['a', 'b', 'c']))
[('a', 'b', 'c'), ('a', 'c', 'b'), ('b', 'a', 'c'),
 ('b', 'c', 'a'), ('c', 'a', 'b'), ('c', 'b', 'a')]

Any iterable of three elements will have six permutations, and the number of permutations of longer iterables grows extremely fast. In fact, an iterable of length n has n! permutations, where

To put this in perspective, here’s a table of these numbers for n = 1 to n = 10:

n	n!
2	2
3	6
4	24
5	120
6	720
7	5,040
8	40,320
9	362,880
10	3,628,800

The phenomenon of just a few inputs producing a large number of outcomes is called a combinatorial explosion and is something to keep in mind when working with combinations(), combinations_with_replacement(), and permutations().

It is usually best to avoid brute force algorithms, although there are times you may need to use one (for example, if the correctness of the algorithm is critical, or every possible outcome must be considered). In that case, itertools has you covered.

Section Recap

In this section you met three itertools functions: combinations(), combinations_with_replacement(), and permutations().

Let’s review these functions before moving on:

`itertools.combinations` Example

combinations(iterable, n)

Return successive n-length combinations of elements in the iterable.

>>> combinations([1, 2, 3], 2)
(1, 2), (1, 3), (2, 3)

`itertools.combinations_with_replacement` Example

combinations_with_replacement(iterable, n)

Return successive n-length combinations of elements in the iterable allowing individual elements to have successive repeats.

>>> combinations_with_replacement([1, 2], 2)
(1, 1), (1, 2), (2, 2)

`itertools.permutations` Example

permutations(iterable, n=None)

Return successive n-length permutations of elements in the iterable.

>>> permutations('abc')
('a', 'b', 'c'), ('a', 'c', 'b'), ('b', 'a', 'c'),
('b', 'c', 'a'), ('c', 'a', 'b'), ('c', 'b', 'a')

Sequences of Numbers

With itertools, you can easily generate iterators over infinite sequences. In this section, you will explore numeric sequences, but the tools and techniques seen here are by no means limited to numbers.

Evens and Odds

For the first example, you will create a pair of iterators over even and odd integers without explicitly doing any arithmetic. Before diving in, let’s look at an arithmetic solution using generators:

>>> def evens():
...     """Generate even integers, starting with 0."""
...     n = 0
...     while True:
...         yield n
...         n += 2
...
>>> evens = evens()
>>> list(next(evens) for _ in range(5))
[0, 2, 4, 6, 8]

>>> def odds():
...     """Generate odd integers, starting with 1."""
...     n = 1
...     while True:
...         yield n
...         n += 2
...
>>> odds = odds()
>>> list(next(odds) for _ in range(5))
[1, 3, 5, 7, 9]

That is pretty straightforward, but with itertools you can do this much more compactly. The function you need is itertools.count(), which does exactly what it sounds like: it counts, starting by default with the number 0.

>>> counter = it.count()
>>> list(next(counter) for _ in range(5))
[0, 1, 2, 3, 4]

You can start counting from any number you like by setting the start keyword argument, which defaults to 0. You can even set a step keyword argument to determine the interval between numbers returned from count()—this defaults to 1.

With count(), iterators over even and odd integers become literal one-liners:

>>> evens = it.count(step=2)
>>> list(next(evens) for _ in range(5))
[0, 2, 4, 6, 8]

>>> odds = it.count(start=1, step=2)
>>> list(next(odds) for _ in range(5))
[1, 3, 5, 7, 9]

Ever since Python 3.1, the count() function also accepts non-integer arguments:

>>> count_with_floats = it.count(start=0.5, step=0.75)
>>> list(next(count_with_floats) for _ in range(5))
[0.5, 1.25, 2.0, 2.75, 3.5]

You can even pass it negative numbers:

>>> negative_count = it.count(start=-1, step=-0.5)
>>> list(next(negative_count) for _ in range(5))
[-1, -1.5, -2.0, -2.5, -3.0]

In some ways, count() is similar to the built-in range() function, but count() always returns an infinite sequence. You might wonder what good an infinite sequence is since it’s impossible to iterate over completely. That is a valid question, and I admit the first time I was introduced to infinite iterators, I too didn’t quite see the point.

The example that made me realize the power of the infinite iterator was the following, which emulates the behavior of the built-in enumerate() function:

>>> list(zip(it.count(), ['a', 'b', 'c']))
[(0, 'a'), (1, 'b'), (2, 'c')]

It is a simple example, but think about it: you just enumerated a list without a for loop and without knowing the length of the list ahead of time.

Recurrence Relations

A recurrence relation is a way of describing a sequence of numbers with a recursive formula. One of the best-known recurrence relations is the one that describes the Fibonacci sequence.

The Fibonacci sequence is the sequence 0, 1, 1, 2, 3, 5, 8, 13, .... It starts with 0 and 1, and each subsequent number in the sequence is the sum of the previous two. The numbers in this sequence are called the Fibonacci numbers. In mathematical notation, the recurrence relation describing the n-th Fibonacci number looks like this:

Note: If you search Google, you will find a host of implementations of these numbers in Python. You can find a recursive function that produces them in the Thinking Recursively in Python article here on Real Python.

It is common to see the Fibonacci sequence produced with a generator:

def fibs():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

The recurrence relation describing the Fibonacci numbers is called a second order recurrence relation because, to calculate the next number in the sequence, you need to look back two numbers behind it.

In general, second order recurrence relations have the form:

Here, P, Q, and R are constants. To generate the sequence, you need two initial values. For the Fibonacci numbers, P = Q = 1, R = 0, and the initial values are 0 and 1.

As you might guess, a first order recurrence relation has the following form:

There are countless sequences of numbers that can be described by first and second order recurrence relations. For example, the positive integers can be described as a first order recurrence relation with P = Q = 1 and initial value 1. For the even integers, take P = 1 and Q = 2 with initial value 0.

In this section, you will construct functions for producing any sequence whose values can be described with a first or second order recurrence relation.

First Order Recurrence Relations

You’ve already seen how count() can generate the sequence of non-negative integers, the even integers, and the odd integers. You can also use it to generate the sequence 3n = 0, 3, 6, 9, 12, … and 4n = 0, 4, 8, 12, 16, ….

count_by_three = it.count(step=3)
count_by_four = it.count(step=4)

In fact, count() can produce sequences of multiples of any number you wish. These sequences can be described with first-order recurrence relations. For example, to generate the sequence of multiples of some number n, just take P = 1, Q = n, and initial value 0.

Another easy example of a first-order recurrence relation is the constant sequence n, n, n, n, n…, where n is any value you’d like. For this sequence, set P = 1 and Q = 0 with initial value n. itertools provides an easy way to implement this sequence as well, with the repeat() function:

all_ones = it.repeat(1)  # 1, 1, 1, 1, ...
all_twos = it.repeat(2)  # 2, 2, 2, 2, ...

If you need a finite sequence of repeated values, you can set a stopping point by passing a positive integer as a second argument:

five_ones = it.repeat(1, 5)  # 1, 1, 1, 1, 1
three_fours = it.repeat(4, 3)  # 4, 4, 4

What may not be quite as obvious is that the sequence 1, -1, 1, -1, 1, -1, ... of alternating 1s and -1s can also be described by a first order recurrence relation. Just take P = -1, Q = 0, and initial value 1.

There’s an easy way to generate this sequence with the itertools.cycle() function. This function takes an iterable inputs as an argument and returns an infinite iterator over the values in inputs that returns to the beginning once the end of inputs is reached. So, to produce the alternating sequence of 1s and -1s, you could do this:

alternating_ones = it.cycle([1, -1])  # 1, -1, 1, -1, 1, -1, ...

The goal of this section, though, is to produce a single function that can generate any first order recurrence relation—just pass it P, Q, and an initial value. One way to do this is with itertools.accumulate().

The accumulate() function takes two arguments—an iterable inputs and a binary function func (that is, a function with exactly two inputs)—and returns an iterator over accumulated results of applying func to elements of inputs. It is roughly equivalent to the following generator:

def accumulate(inputs, func):
    itr = iter(inputs)
    prev = next(itr)
    for cur in itr:
        yield prev
        prev = func(prev, cur)

For example:

>>> import operator
>>> list(it.accumulate([1, 2, 3, 4, 5], operator.add))
[1, 3, 6, 10, 15]

The first value in the iterator returned by accumulate() is always the first value in the input sequence. In the above example, this is 1—the first value in [1, 2, 3, 4, 5].

The next value in the output iterator is the sum of the first two elements of the input sequence: add(1, 2) = 3. To produce the next value, accumulate() takes the result of add(1, 2) and adds this to the third value in the input sequence:

add(3, 3) = add(add(1, 2), 3) = 6

The fourth value produced by accumulate() is add(add(add(1, 2), 3), 4) = 10, and so on.

The second argument of accumulate() defaults to operator.add(), so the previous example can be simplified to:

>>> list(it.accumulate([1, 2, 3, 4, 5]))
[1, 3, 6, 10, 15]

Passing the built-in min() to accumulate() will keep track of a running minimum:

>>> list(it.accumulate([9, 21, 17, 5, 11, 12, 2, 6], min))
[9, 9, 9, 5, 5, 5, 2, 2]

More complex functions can be passed to accumulate() with lambda expressions:

>>> list(it.accumulate([1, 2, 3, 4, 5], lambda x, y: (x + y) / 2))
[1, 1.5, 2.25, 3.125, 4.0625]

The order of the arguments in the binary function passed to accumulate() is important. The first argument is always the previously accumulated result and the second argument is always the next element of the input iterable. For example, consider the difference in output of the following expressions:

>>> list(it.accumulate([1, 2, 3, 4, 5], lambda x, y: x - y))
[1, -1, -4, -8, -13]

>>> list(it.accumulate([1, 2, 3, 4, 5], lambda x, y: y - x))
[1, 1, 2, 2, 3]

To model a recurrence relation, you can just ignore the second argument of the binary function passed to accumulate(). That is, given values p, q, and s, lambda x, _: p*s + q will return the value following x in the recurrence relation defined by sᵢ = Psᵢ₋₁ + Q.

In order for accumulate() to iterate over the resulting recurrence relation, you need to pass to it an infinite sequence with the right initial value. It doesn’t matter what the rest of the values in the sequence are, as long as the initial value is the initial value of the recurrence relation. You can do this is with repeat():

def first_order(p, q, initial_val):
    """Return sequence defined by s(n) = p * s(n-1) + q."""
    return it.accumulate(it.repeat(initial_val), lambda s, _: p*s + q)

Using first_order(), you can build the sequences from above as follows:

>>> evens = first_order(p=1, q=2, initial_val=0)
>>> list(next(evens) for _ in range(5))
[0, 2, 4, 6, 8]

>>> odds = first_order(p=1, q=2, initial_val=1)
>>> list(next(odds) for _ in range(5))
[1, 3, 5, 7, 9]

>>> count_by_threes = first_order(p=1, q=3, initial_val=0)
>>> list(next(count_by_threes) for _ in range(5))
[0, 3, 6, 9, 12]

>>> count_by_fours = first_order(p=1, q=4, initial_val=0)
>>> list(next(count_by_fours) for _ in range(5))
[0, 4, 8, 12, 16]

>>> all_ones = first_order(p=1, q=0, initial_val=1)
>>> list(next(all_ones) for _ in range(5))
[1, 1, 1, 1, 1]

>>> all_twos = first_order(p=1, q=0, initial_val=2)
>>> list(next(all_twos) for _ in range(5))
[2, 2, 2, 2, 2]

>>> alternating_ones = first_order(p=-1, 0, initial_val=1)
>>> list(next(alternating_ones) for _ in range(5))
[1, -1, 1, -1, 1]

Second Order Recurrence Relations

Generating sequences described by second order recurrence relations, like the Fibonacci sequence, can be accomplished using a similar technique as the one used for first order recurrence relations.

The difference here is that you need to create an intermediate sequence of tuples that keep track of the previous two elements of the sequence, and then map() each of these tuples to their first component to get the final sequence.

Here’s what it looks like:

def second_order(p, q, r, initial_values):
    """Return sequence defined by s(n) = p * s(n-1) + q * s(n-2) + r."""
    intermediate = it.accumulate(
        it.repeat(initial_values),
        lambda s, _: (s[1], p*s[1] + q*s[0] + r)
    )
    return map(lambda x: x[0], intermediate)

Using second_order(), you can generate the Fibonacci sequence like this:

>>> fibs = second_order(p=1, q=1, r=0, initial_values=(0, 1))
>>> list(next(fibs) for _ in range(8))
[0, 1, 1, 2, 3, 5, 8, 13]

Other sequences can be easily generated by changing the values of p, q, and r. For example, the Pell numbers and the Lucas numbers can be generated as follows:

pell = second_order(p=2, q=1, r=0, initial_values=(0, 1))
>>> list(next(pell) for _ in range(6))
[0, 1, 2, 5, 12, 29]

>>> lucas = second_order(p=1, q=1, r=0, initial_values=(2, 1))
>>> list(next(lucas) for _ in range(6))
[2, 1, 3, 4, 7, 11]

You can even generate the alternating Fibonacci numbers:

>>> alt_fibs = second_order(p=1, q=0, r=0, inital_values=(-1, 1))
>>> list(next(alt_fibs) for _ in range(6))
[-1, 1, -2, 3, -5, 8]

This is all really cool if you are a giant math nerd like I am, but step back for a second and compare second_order() to the fibs() generator from the beginning of this section. Which one is easier to understand?

This is a valuable lesson. The accumulate() function is a powerful tool to have in your toolkit, but there are times when using it could mean sacrificing clarity and readability.

Section Recap

You saw several itertools function in this section. Let’s review those now.

`itertools.count` Example

count(start=0, step=1)

Return a count object whose .__next__() method returns consecutive values.

>>> count()
0, 1, 2, 3, 4, ...

>>> count(start=1, step=2)
1, 3, 5, 7, 9, ...

`itertools.repeat` Example

repeat(object, times=1)

Create an iterator which returns the object for the specified number of times. If not specified, returns the object endlessly.

>>> repeat(2)
2, 2, 2, 2, 2 ...

>>> repeat(2, 5)  # Stops after 5 repititions.
2, 2, 2, 2, 2

`itertools.cycle` Example

cycle(iterable)

Return elements from the iterable until it is exhausted. Then repeat the sequence indefinitely.

>>> cycle(['a', 'b', 'c'])
a, b, c, a, b, c, a, ...

`itertools accumulate` Example

accumulate(iterable, func=operator.add)

Return series of accumulated sums (or other binary function results).

>>> accumulate([1, 2, 3])
1, 3, 6

Alright, let’s take a break from the math and have some fun with cards.

Dealing a Deck of Cards

Suppose you are building a Poker app. You’ll need a deck of cards. You might start by defining a list of ranks (ace, king, queen, jack, 10, 9, and so on) and a list of suits (hearts, diamonds, clubs, and spades):

ranks = ['A', 'K', 'Q', 'J', '10', '9', '8', '7', '6', '5', '4', '3', '2']
suits = ['H', 'D', 'C', 'S']

You could represent a card as a tuple whose first element is a rank and second element is a suit. A deck of cards would be a collection of such tuples. The deck should act like the real thing, so it makes sense to define a generator that yields cards one at a time and becomes exhausted once all the cards are dealt.

One way to achieve this is to write a generator with a nested for loop over ranks and suits:

def cards():
    """Return a generator that yields playing cards."""
    for rank in ranks:
        for suit in suits:
            yield rank, suit

You could write this more compactly with a generator expression:

cards = ((rank, suit) for rank in ranks for suit in suits)

However, some might argue that this is actually more difficult to understand than the more explicit nested for loop.

It helps to view nested for loops from a mathematical standpoint—that is, as a Cartesian product of two or more iterables. In mathematics, the Cartesian product of two sets A and B is the set of all tuples of the form (a, b) where a is an element of A and b is an element of B.

Here’s an example with Python iterables: the Cartesian product of A = [1, 2] and B = ['a', 'b'] is [(1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')].

The itertools.product() function is for exactly this situation. It takes any number of iterables as arguments and returns an iterator over tuples in the Cartesian product:

it.product([1, 2], ['a', 'b'])  # (1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')

The product() function is by no means limited to two iterables. You can pass it as many as you like—they don’t even have to all be of the same size! See if you can predict what product([1, 2, 3], ['a', 'b'], ['c']) is, then check your work by running it in the interpreter.

Warning: The product() function is another “brute force” function and can lead to a combinatorial explosion if you aren’t careful.

Using product(), you can re-write the cards in a single line:

cards = it.product(ranks, suits)

This is all fine and dandy, but any Poker app worth its salt better start with a shuffled deck:

import random

def shuffle(deck):
    """Return iterator over shuffled deck."""
    deck = list(deck)
    random.shuffle(deck)
    return iter(tuple(deck))

cards = shuffle(cards)

Note: The random.shuffle() function uses the Fisher-Yates shuffle to shuffle a list (or any mutable sequence) in place in O(n) time. This algorithm is well-suited for shuffling cards because it produces an unbiased permutation—that is, all permutations of the iterable are equally likely to be returned by random.shuffle().

That said, you probably noticed that shuffle() creates a copy of its input deck in memory by calling list(deck). While this seemingly goes against the spirit of this article, this author is unaware of a good way to shuffle an iterator without making a copy.

As a courtesy to your users, you would like to give them the opportunity to cut the deck. If you imagine the cards being stacked neatly on a table, you have the user pick a number n and then remove the first n cards from the top of the stack and move them to the bottom.

If you know a thing or two about slicing, you might accomplish this like so:

def cut(deck, n):
    """Return an iterator over a deck of cards cut at index `n`."""
    if n < 0:
        raise ValueError('`n` must be a non-negative integer')

    deck = list(deck)
    return iter(deck[n:] + deck[:n])

cards = cut(cards, 26)  # Cut the deck in half.

The cut() function first converts deck to a list so that you can slice it to make the cut. To guarantee your slices behave as expected, you’ve got to check that n is non-negative. If it isn’t, you better throw an exception so that nothing crazy happens.

Cutting the deck is pretty straightforward: the top of the cut deck is just deck[:n], and the bottom is the remaining cards, or deck[n:]. To construct the new deck with the top “half” moved to the bottom, you just append it to the bottom: deck[n:] + deck[:n].

The cut() function is pretty simple, but it suffers from a couple of problems. When you slice a list, you make a copy of the original list and return a new list with the selected elements. With a deck of only 52 cards, this increase in space complexity is trivial, but you could reduce the memory overhead using itertools. To do this, you’ll need three functions: itertools.tee(), itertools.islice(), and itertools.chain().

Let’s take a look at how those functions work.

The tee() function can be used to create any number of independent iterators from a single iterable. It takes two arguments: the first is an iterable inputs, and the second is the number n of independent iterators over inputs to return (by default, n is set to 2). The iterators are returned in a tuple of length n.

>>> iterator1, iterator2 = it.tee([1, 2, 3, 4, 5], 2)
>>> list(iterator1)
[1, 2, 3, 4, 5]
>>> list(iterator1)  # iterator1 is now exhausted.
[]
>>> list(iterator2)  # iterator2 works independently of iterator1
[1, 2, 3, 4, 5].

While tee() is useful for creating independent iterators, it is important to understand a little bit about how it works under the hood. When you call tee() to create n independent iterators, each iterator is essentially working with its own FIFO queue.

When a value is extracted from one iterator, that value is appended to the queues for the other iterators. Thus, if one iterator is exhausted before the others, each remaining iterator will hold a copy of the entire iterable in memory. (You can find a Python function that emulates tee() in the itertools docs.)

For this reason, tee() should be used with care. If you are exhausting large portions of an iterator before working with the other returned by tee(), you may be better off casting the input iterator to a list or tuple.

The islice() function works much the same way as slicing a list or tuple. You pass it an iterable, a starting, and stopping point, and, just like slicing a list, the slice returned stops at the index just before the stopping point. You can optionally include a step value, as well. The biggest difference here is, of course, that islice() returns an iterator.

>>> # Slice from index 2 to 4
>>> list(it.islice('ABCDEFG', 2, 5)))
['C' 'D' 'E']

>>> # Slice from beginning to index 4, in steps of 2
>>> list(it.islice([1, 2, 3, 4, 5], 0, 5, 2))
[1, 3, 5]

>>> # Slice from index 3 to the end
>>> list(it.islice(range(10), 3, None))
[3, 4, 5, 6, 7, 8, 9]

>>> # Slice from beginning to index 3
>>> list(it.islice('ABCDE'), 4))
['A', 'B', 'C', 'D']

The last two examples above are useful for truncating iterables. You can use this to replace the list slicing used in cut() to select the “top” and “bottom” of the deck. As an added bonus, islice() won’t accept negative indices for the start/stop positions and the step value, so you won’t need to raise an exception if n is negative.

The last function you need is chain(). This function takes any number of iterables as arguments and “chains” them together. For example:

>>> list(it.chain('ABC', 'DEF'))
['A' 'B' 'C' 'D' 'E' 'F']

>>> list(it.chain([1, 2], [3, 4, 5, 6], [7, 8, 9]))
[1, 2, 3, 4, 5, 6, 7, 8, 9]

Now that you’ve got some additional firepower in your arsenal, you can re-write the cut() function to cut the deck of cards without making a full copy cards in memory:

def cut(deck, n):
    """Return an iterator over a deck of cards cut at index `n`."""
    deck1, deck2 = it.tee(deck, 2)
    top = it.islice(deck1, n)
    bottom = it.islice(deck2, n, None)
    return it.chain(bottom, top)

cards = cut(cards, 26)

Now that you have shuffled and cut the cards, it is time to deal some hands. You could write a function deal() that takes a deck, the number of hands, and the hand size as arguments and returns a tuple containing the specified number of hands.

You do not need any new itertools functions to write this function. See what you can come up with on your own before reading ahead.

Here’s one solution:

def deal(deck, num_hands=1, hand_size=5):
    iters = [iter(deck)] * hand_size
    return tuple(zip(*(tuple(it.islice(itr, num_hands)) for itr in iters)))

You start by creating a list of hand_size references to an iterator over deck. You then iterate over this list, removing num_hands cards at each step and storing them in tuples.

Next, you zip() these tuples up to emulate dealing one card at a time to each player. This produces num_hands tuples, each containing hand_size cards. Finally, you package the hands up into a tuple to return them all at once.

This implementation sets the default values for num_hands to 1 and hand_size to 5—maybe you are making a “Five Card Draw” app. Here’s how you would use this function, with some sample output:

>>> p1_hand, p2_hand, p3_hand = deal(cards, num_hands=3)
>>> p1_hand
(('A', 'S'), ('5', 'S'), ('7', 'H'), ('9', 'H'), ('5', 'H'))
>>> p2_hand
(('10', 'H'), ('2', 'D'), ('2', 'S'), ('J', 'C'), ('9', 'C'))
>>> p3_hand
(('2', 'C'), ('Q', 'S'), ('6', 'C'), ('Q', 'H'), ('A', 'C'))

What do you think the state of cards is now that you have dealt three hands of five cards?

>>> len(tuple(cards))
37

The fifteen cards dealt are consumed from the cards iterator, which is exactly what you want. That way, as the game continues, the state of the cards iterator reflects the state of the deck in play.

Section Recap

Let’s review the itertools functions you saw in this section.

`itertools.product` Example

product(*iterables, repeat=1)

Cartesian product of input iterables. Equivalent to nested for-loops.

>>> product([1, 2], ['a', 'b'])
(1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')

`itertools.tee` Example

tee(iterable, n=2)

Create any number of independent iterators from a single input iterable.

>>> iter1, iter2 = it.tee(['a', 'b', 'c'], 2)
>>> list(iter1)
['a', 'b', 'c']
>>> list(iter2)
['a', 'b', 'c']

`itertools.islice` Example

islice(iterable, stop) islice(iterable, start, stop, step=1)

Return an iterator whose __next__() method returns selected values from an iterable. Works like a slice() on a list but returns an iterator.

>>> islice([1, 2, 3, 4], 3)
1, 2, 3

>>> islice([1, 2, 3, 4], 1, 2)
2, 3

`itertools.chain` Example

chain(*iterables)

Return a chain object whose __next__() method returns elements from the first iterable until it is exhausted, then elements from the next iterable, until all of the iterables are exhausted.

>>> chain('abc', [1, 2, 3])
'a', 'b', 'c', 1, 2, 3

Intermission: Flattening A List of Lists

In the previous example, you used chain() to tack one iterator onto the end of another. The chain() function has a class method .from_iterable() that takes a single iterable as an argument. The elements of the iterable must themselves be iterable, so the net effect is that chain.from_iterable() flattens its argument:

>>> list(it.chain.from_iterable([[1, 2, 3], [4, 5, 6]]))
[1, 2, 3, 4, 5, 6]

There’s no reason the argument of chain.from_iterable() needs to be finite. You could emulate the behavior of cycle(), for example:

>>> cycle = it.chain.from_iterable(it.repeat('abc'))
>>> list(it.islice(cycle, 8))
['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b']

The chain.from_interable() function is useful when you need to build an iterator over data that has been “chunked.”

In the next section, you will see how to use itertools to do some data analysis on a large dataset. But you deserve a break for having stuck with it this far. Why not hydrate yourself and relax a bit? Maybe even play a little Star Trek: The Nth Iteration.

Back? Great! Let’s do some data analysis.

Analyzing the S&P500

In this example, you will get your first taste of using itertools to manipulate a large dataset—in particular, the historical daily price data of the S&P500 index. A CSV file SP500.csv with this data can be found here (source: Yahoo Finance). The problem you’ll tackle is this:

Determine the maximum daily gain, daily loss (in percent change), and the longest growth streak in the history of the S&P500.

To get a feel for what you’re dealing with, here are the first ten rows of SP500.csv:

$ head -n 10 SP500.csv
Date,Open,High,Low,Close,Adj Close,Volume
1950-01-03,16.660000,16.660000,16.660000,16.660000,16.660000,1260000
1950-01-04,16.850000,16.850000,16.850000,16.850000,16.850000,1890000
1950-01-05,16.930000,16.930000,16.930000,16.930000,16.930000,2550000
1950-01-06,16.980000,16.980000,16.980000,16.980000,16.980000,2010000
1950-01-09,17.080000,17.080000,17.080000,17.080000,17.080000,2520000
1950-01-10,17.030001,17.030001,17.030001,17.030001,17.030001,2160000
1950-01-11,17.090000,17.090000,17.090000,17.090000,17.090000,2630000
1950-01-12,16.760000,16.760000,16.760000,16.760000,16.760000,2970000
1950-01-13,16.670000,16.670000,16.670000,16.670000,16.670000,3330000

As you can see, the early data is limited. The data improves for later dates, and, as a whole, is sufficient for this example.

The strategy for solving this problem is as follows:

Read data from the CSV file and transform it into a sequence gains of daily percent changes using the “Adj Close” column.
Find the maximum and minimum values of the gains sequence, and the date on which they occur. (Note that it is possible that these values are attained on more than on date; in that case, the most recent date will suffice.)
Transform gains into a sequence growth_streaks of tuples of consecutive positive values in gains. Then determine the length of the longest tuple in growth_streaks and the beginning and ending dates of the streak. (It is possible that the maximum length is attained by more than one tuple in growth_streaks; in that case, the tuple with the most recent beginning and ending dates will suffice.)

The percent change between two values x and y is given by the following formula:

For each step in the analysis, it is necessary to compare values associated with dates. To facilitate these comparisons, you can subclass the namedtuple object from the collections module:

from collections import namedtuple


class DataPoint(namedtuple('DataPoint', ['date', 'value'])):
    __slots__ = ()

    def __le__(self, other):
        return self.value <= other.value

    def __lt__(self, other):
        return self.value < other.value

    def __gt__(self, other):
        return self.value > other.value

The DataPoint class has two attributes: date (a datetime.datetime instance) and value. The .__le__(), .__lt__() and .__gt__() dunder methods are implemented so that the <=, <, and > boolean comparators can be used to compare the values of two DataPoint objects. This also allows the max() and min() built-in functions to be called with DataPoint arguments.

Note: If you are not familiar with namedtuple, check out this excellent resource. The namedtuple implementation for DataPoint is just one of many ways to build this data structure. For example, in Python 3.7 you could implement DataPoint as a data class. Check out our Ultimate Guide to Data Classes for more information.

The following reads the data from SP500.csv to a tuple of DataPoint objects:

import csv
from datetime import datetime


def read_prices(csvfile, _strptime=datetime.strptime):
    with open(csvfile) as infile:
        reader = csv.DictReader(infile)
        for row in reader:
            yield DataPoint(date=_strptime(row['Date'], '%Y-%m-%d').date(),
                            value=float(row['Adj Close']))


prices = tuple(read_prices('SP500.csv'))

The read_prices() generator opens SP500.csv and reads each row with a csv.DictReader() object. DictReader() returns each row as an OrderedDict whose keys are the column names from the header row of the CSV file.

For each row, read_prices() yields a DataPoint object containing the values in the “Date” and “Adj Close” columns. Finally, the full sequence of data points is committed to memory as a tuple and stored in the prices variable.

Next, prices needs to be transformed to a sequence of daily percent changes:

gains = tuple(DataPoint(day.date, 100*(day.value/prev_day.value - 1.))
                for day, prev_day in zip(prices[1:], prices))

The choice of storing the data in a tuple is intentional. Although you could point gains to an iterator, you will need to iterate over the data twice to find the minimum and maximum values.

If you use tee() to create two independent iterators, exhausting one iterator to find the maximum will create a copy of all of the data in memory for the second iterator. By creating a tuple up front, you do not lose anything in terms of space complexity compared to tee(), and you may even gain a little speed.

Note: This example focuses on leveraging itertools for analyzing the S&P500 data. Those intent on working with a lot of time series financial data might also want to check out the Pandas library, which is well suited for such tasks.

Maximum Gain and Loss

To determine the maximum gain on any single day, you might do something like this:

max_gain = DataPoint(None, 0)
for data_point in gains:
    max_gain = max(data_point, max_gain)

print(max_gain)   # DataPoint(date='2008-10-28', value=11.58)

You can simplify the for loop using the functools.reduce() function. This function accepts a binary function func and an iterable inputs as arguments, and “reduces” inputs to a single value by applying func cumulatively to pairs of objects in the iterable.

For example, functools.reduce(operator.add, [1, 2, 3, 4, 5]) will return the sum 1 + 2 + 3 + 4 + 5 = 15. You can think of reduce() as working in much the same way as accumulate(), except that it returns only the final value in the new sequence.

Using reduce(), you can get rid of the for loop altogether in the above example:

import functools as ft


max_gain = ft.reduce(max, gains)

print(max_gain)  # DataPoint(date='2008-10-28', value=11.58)

The above solution works, but it isn’t equivalent to the for loop you had before. Do you see why? Suppose the data in your CSV file recorded a loss every single day. What would the value of max_gain be?

In the for loop, you first set max_gain = DataPoint(None, 0), so if there are no gains, the final max_gain value will be this empty DataPoint object. However, the reduce() solution returns the smallest loss. That is not what you want and could introduce a difficult to find bug.

This is where itertools can help you out. The itertools.filterfalse() function takes two arguments: a function that returns True or False (called a predicate), and an iterable inputs. It returns an iterator over the elements in inputs for which the predicate returns False.

Here’s a simple example:

>>> only_positives = it.filterfalse(lambda x: x <= 0, [0, 1, -1, 2, -2])
>>> list(only_positives)
[1, 2]

You can use filterfalse() to filter out the values in gains that are negative or zero so that reduce() only works on positive values:

max_gain = ft.reduce(max, it.filterfalse(lambda p: p <= 0, gains))

What happens if there are never any gains? Consider the following:

>>> ft.reduce(max, it.filterfalse(lambda x: x <= 0, [-1, -2, -3]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: reduce() of empty sequence with no initial value

Well, that’s not what you want! But, it makes sense because the iterator returned by filterflase() is empty. You could handle the TypeError by wrapping the call to reduce() with try...except, but there’s a better way.

The reduce() function accepts an optional third argument for an initial value. Passing 0 to this third argument gets you the expected behavior:

>>> ft.reduce(max, it.filterfalse(lambda x: x <= 0, [-1, -2, -3]), 0)
0

Applying this to the S&P500 example:

zdp = DataPoint(None, 0)  # zero DataPoint
max_gain = ft.reduce(max, it.filterfalse(lambda p: p <= 0, diffs), zdp)

Great! You’ve got it working just the way it should! Now, finding the maximum loss is easy:

max_loss = ft.reduce(min, it.filterfalse(lambda p: p > 0, gains), zdp)

print(max_loss)  # DataPoint(date='2018-02-08', value=-20.47)

Longest Growth Streak

Finding the longest growth streak in the history of the S&P500 is equivalent to finding the largest number of consecutive positive data points in the gains sequence. The itertools.takewhile() and itertools.dropwhile() functions are perfect for this situation.

The takewhile() function takes a predicate and an iterable inputs as arguments and returns an iterator over inputs that stops at the first instance of an element for which the predicate returns False:

it.takewhile(lambda x: x < 3, [0, 1, 2, 3, 4])  # 0, 1, 2

The dropwhile() function does exactly the opposite. It returns an iterator beginning at the first element for which the predicate returns False:

it.dropwhile(lambda x: x < 3, [0, 1, 2, 3, 4])  # 3, 4

In the following generator function, takewhile() and dropwhile() are composed to yield tuples of consecutive positive elements of a sequence:

def consecutive_positives(sequence, zero=0):
    def _consecutives():
        for itr in it.repeat(iter(sequence)):
            yield tuple(it.takewhile(lambda p: p > zero,
                                     it.dropwhile(lambda p: p <= zero, itr)))
    return it.takewhile(lambda t: len(t), _consecutives())

The consecutive_positives() function works because repeat() keeps returning a pointer to an iterator over the sequence argument, which is being partially consumed at each iteration by the call to tuple() in the yield statement.

You can use consecutive_positives() to get a generator that produces tuples of consecutive positive data points in gains:

growth_streaks = consecutive_positives(gains, zero=DataPoint(None, 0))

Now you can use reduce() to extract the longest growth streak:

longest_streak = ft.reduce(lambda x, y: x if len(x) > len(y) else y,
                           growth_streaks)

Putting the whole thing together, here’s a full script that will read data from the SP500.csv file and print out the max gain/loss and longest growth streak:

from collections import namedtuple
import csv
from datetime import datetime
import itertools as it
import functools as ft


class DataPoint(namedtuple('DataPoint', ['date', 'value'])):
    __slots__ = ()

    def __le__(self, other):
        return self.value <= other.value

    def __lt__(self, other):
        return self.value < other.value

    def __gt__(self, other):
        return self.value > other.value


def consecutive_positives(sequence, zero=0):
    def _consecutives():
        for itr in it.repeat(iter(sequence)):
            yield tuple(it.takewhile(lambda p: p > zero,
                                     it.dropwhile(lambda p: p <= zero, itr)))
    return it.takewhile(lambda t: len(t), _consecutives())


def read_prices(csvfile, _strptime=datetime.strptime):
    with open(csvfile) as infile:
        reader = csv.DictReader(infile)
        for row in reader:
            yield DataPoint(date=_strptime(row['Date'], '%Y-%m-%d').date(),
                            value=float(row['Adj Close']))


# Read prices and calculate daily percent change.
prices = tuple(read_prices('SP500.csv'))
gains = tuple(DataPoint(day.date, 100*(day.value/prev_day.value - 1.))
              for day, prev_day in zip(prices[1:], prices))

# Find maximum daily gain/loss.
zdp = DataPoint(None, 0)  # zero DataPoint
max_gain = ft.reduce(max, it.filterfalse(lambda p: p <= zdp, gains))
max_loss = ft.reduce(min, it.filterfalse(lambda p: p > zdp, gains), zdp)


# Find longest growth streak.
growth_streaks = consecutive_positives(gains, zero=DataPoint(None, 0))
longest_streak = ft.reduce(lambda x, y: x if len(x) > len(y) else y,
                           growth_streaks)

# Display results.
print('Max gain: {1:.2f}% on {0}'.format(*max_gain))
print('Max loss: {1:.2f}% on {0}'.format(*max_loss))

print('Longest growth streak: {num_days} days ({first} to {last})'.format(
    num_days=len(longest_streak),
    first=longest_streak[0].date,
    last=longest_streak[-1].date
))

Running the above script produces the following output:

Max gain: 11.58% on 2008-10-13
Max loss: -20.47% on 1987-10-19
Longest growth streak: 14 days (1971-03-26 to 1971-04-15)

Section Recap

In this section, you covered a lot of ground, but you only saw a few functions from itertools. Let’s review those now.

`itertools.filterfalse` Example

filterfalse(pred, iterable)

Return those items of sequence for which pred(item) is false. If pred is None, return the items that are false.

>>> filterfalse(bool, [1, 0, 1, 0, 0])
0, 0, 0

`itertools.takewhile` Example

takewhile(pred, iterable)

Return successive entries from an iterable as long as pred evaluates to true for each entry.

>>> takewhile(bool, [1, 1, 1, 0, 0])
1, 1, 1

`itertools.dropwhile` Example

dropwhile(pred, iterable)

Drop items from the iterable while pred(item) is true. Afterwards, return every element until the iterable is exhausted.

>>> dropwhile(bool, [1, 1, 1, 0, 0, 1, 1, 0])
0, 0, 1, 1, 0

You are really starting to master this whole itertools thing! The community swim team would like to commission you for a small project.

Building Relay Teams From Swimmer Data

In this example, you will read data from a CSV file containing swimming event times for a community swim team from all of the swim meets over the course of a season. The goal is to determine which swimmers should be in the relay teams for each stroke next season.

Each stroke should have an “A” and a “B” relay team with four swimmers each. The “A” team should contain the four swimmers with the best times for the stroke and the “B” team the swimmers with the next four best times.

The data for this example can be found here. If you want to follow along, download it to your current working directory and save it as swimmers.csv.

Here are the first 10 rows of swimmers.csv:

$ head -n 10 swimmers.csv
Event,Name,Stroke,Time1,Time2,Time3
0,Emma,freestyle,00:50:313667,00:50:875398,00:50:646837
0,Emma,backstroke,00:56:720191,00:56:431243,00:56:941068
0,Emma,butterfly,00:41:927947,00:42:062812,00:42:007531
0,Emma,breaststroke,00:59:825463,00:59:397469,00:59:385919
0,Olivia,freestyle,00:45:566228,00:46:066985,00:46:044389
0,Olivia,backstroke,00:53:984872,00:54:575110,00:54:932723
0,Olivia,butterfly,01:12:548582,01:12:722369,01:13:105429
0,Olivia,breaststroke,00:49:230921,00:49:604561,00:49:120964
0,Sophia,freestyle,00:55:209625,00:54:790225,00:55:351528

The three times in each row represent the times recorded by three different stopwatches, and are given in MM:SS:mmmmmm format (minutes, seconds, microseconds). The accepted time for an event is the median of these three times, not the average.

Let’s start by creating a subclass Event of the namedtuple object, just like we did in the SP500 example:

from collections import namedtuple

class Event(namedtuple('Event', ['stroke', 'name', 'time'])):
    __slots__ = ()

    def __lt__(self, other):
        return self.time < other.time

The .stroke property stores the name of the stroke in the event, .name stores the swimmer name, and .time records the accepted time for the event. The .__lt__() dunder method will allow min() to be called on a sequence of Event objects.

To read the data from the CSV into a tuple of Event objects, you can use the csv.DictReader object:

import csv
import datetime
import statistics

def read_events(csvfile, _strptime=datetime.datetime.strptime):
    def _median(times):
        return statistics.median((_strptime(time, '%M:%S:%f').time()
                                  for time in row['Times']))

    fieldnames = ['Event', 'Name', 'Stroke']
    with open(csvfile) as infile:
        reader = csv.DictReader(infile, fieldnames=fieldnames, restkey='Times')
        next(reader)  # skip header
        for row in reader:
            yield Event(row['Stroke'], row['Name'], _median(row['Times']))


events = tuple(read_events('swimmers.csv'))

The read_events() generator reads each row in the swimmers.csv file into an OrderedDict object in the following line:

reader = csv.DictReader(infile, fieldnames=fieldnames, restkey='Times')

By assigning the 'Times' field to restkey, the “Time1”, “Time2”, and “Time3” columns of each row in the CSV file will be stored in a list on the 'Times' key of the OrderedDict returned by csv.DictReader.

For example, the first row of the file (excluding the header row) is read into the following object:

OrderedDict([('Event', '0'),
             ('Name', 'Emma'),
             ('Stroke', 'freestyle'),
             ('Times', ['00:50:313667', '00:50:875398', '00:50:646837'])])

Next, read_events() yields an Event object with the stroke, swimmer name, and median time (as a datetime.time object) returned by the _median() function, which calls statistics.median() on the list of times in the row.

Since each item in the list of times is read as a string by csv.DictReader(), _median() uses the datetime.datetime.strptime() classmethod to instantiate a time object from each string.

Finally, a tuple of Event objects is created:

events = tuple(read_events('swimmers.csv'))

The first five elements of events look like this:

>>> events[:5]
(Event(stroke='freestyle', name='Emma', time=datetime.time(0, 0, 50, 646837)),
 Event(stroke='backstroke', name='Emma', time=datetime.time(0, 0, 56, 720191)),
 Event(stroke='butterfly', name='Emma', time=datetime.time(0, 0, 42, 7531)),
 Event(stroke='breaststroke', name='Emma', time=datetime.time(0, 0, 59, 397469)),
 Event(stroke='freestyle', name='Olivia', time=datetime.time(0, 0, 46, 44389)))

Now that you’ve got the data into memory, what do you do with it? Here’s the plan of attack:

Group the events by stroke.
For each stroke:
- Group its events by swimmer name and determine the best time for each swimmer.
- Order the swimmers by best time.
- The first four swimmers make the “A” team for the stroke, and the next four swimmers make the “B” team.

The itertools.groupby() function makes grouping objects in an iterable a snap. It takes an iterable inputs and a key to group by, and returns an object containing iterators over the elements of inputs grouped by the key.

Here’s a simple groupby() example:

>>> data = [{'name': 'Alan', 'age': 34},
...         {'name': 'Catherine', 'age': 34},
...         {'name': 'Betsy', 'age': 29},
...         {'name': 'David', 'age': 33}]
...
>>> grouped_data = it.groupby(data, key=lambda x: x['age'])
>>> for key, grp in grouped_data:
...     print('{}: {}'.format(key, list(grp)))
...
34: [{'name': 'Alan', 'age': 34}, {'name': 'Betsy', 'age': 34}]
29: [{'name': 'Catherine', 'age': 29}]
33: [{'name': 'David', 'age': 33}]

If no key is specified, groupby() defaults to grouping by “identity”—that is, aggregating identical elements in the iterable:

>>> for key, grp in it.groupby([1, 1, 2, 2, 2, 3]:
...     print('{}: {}'.format(key, list(grp)))
...
1: [1, 1]
2: [2, 2, 2]
3: [3]

The object returned by groupby() is sort of like a dictionary in the sense that the iterators returned are associated with a key. However, unlike a dictionary, it won’t allow you to access its values by key name:

>>> grouped_data[1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'itertools.groupby' object is not subscriptable

In fact, groupby() returns an iterator over tuples whose first components are keys and second components are iterators over the grouped data:

>>> grouped_data = it.groupby([1, 1, 2, 2, 2, 3])
>>> list(grouped_data)
[(1, <itertools._grouper object at 0x7ff3056130b8>),
 (2, <itertools._grouper object at 0x7ff3056130f0>),
 (3, <itertools._grouper object at 0x7ff305613128>)]

One thing to keep in mind with groupby() is that it isn’t as smart as you might like. As groupby() traverses the data, it aggregates elements until an element with a different key is encountered, at which point it starts a new group:

>>> grouped_data = it.groupby([1, 2, 1, 2, 3, 2])
>>> for key, grp in grouped_data:
...     print('{}: {}'.format(key, list(grp)))
...
1: [1]
2: [2]
1: [1]
2: [2]
3: [3]
2: [2]

Compare this to, say, the SQL GROUP BY command, which groups elements regardless of their order of appearance.

When working with groupby(), you need to sort your data on the same key that you would like to group by. Otherwise, you may get unexpected results. This is so common that it helps to write a utility function to take care of this for you:

def sort_and_group(iterable, key=None):
    """Group sorted `iterable` on `key`."""
    return it.groupby(sorted(iterable, key=key), key=key)

Returning to the swimmers example, the first thing you need to do is create a for loop that iterates over the data in the events tuple grouped by stroke:

for stroke, evts in sort_and_group(events, key=lambda evt: evt.stroke):

Next, you need to group the evts iterator by swimmer name inside of the above for loop:

events_by_name = sort_and_group(evts, key=lambda evt: evt.name)

To calculate the best time for each swimmer in events_by_name, you can call min() on the events in that swimmers group. (This works because you implemented the .__lt__() dunder method in the Events class.)

best_times = (min(evt) for _, evt in events_by_name)

Note that the best_times generator yields Event objects containing the best stroke time for each swimmer. To build the relay teams, you’ll need to sort best_times by time and aggregate the result into groups of four. To aggregate the results, you can use the grouper() function from The grouper() recipe section and use islice() to grab the first two groups.

sorted_by_time = sorted(best_times, key=lambda evt: evt.time)
teams = zip(('A', 'B'), it.islice(grouper(sorted_by_time, 4), 2))

Now teams is an iterator over exactly two tuples representing the “A” and the “B” team for the stroke. The first component of each tuple is the letter “A” or “B”, and the second component is an iterator over Event objects containing the swimmers in the team. You can now print the results:

for team, swimmers in teams:
    print('{stroke} {team}: {names}'.format(
        stroke=stroke.capitalize(),
        team=team,
        names=', '.join(swimmer.name for swimmer in swimmers)
    ))

Here’s the full script:

from collections import namedtuple
import csv
import datetime
import itertools as it
import statistics


class Event(namedtuple('Event', ['stroke', 'name', 'time'])):
    __slots__ = ()

    def __lt__(self, other):
        return self.time < other.time


def sort_and_group(iterable, key=None):
    return it.groupby(sorted(iterable, key=key), key=key)


def grouper(iterable, n, fillvalue=None):
    iters = [iter(iterable)] * n
    return it.zip_longest(*iters, fillvalue=fillvalue)


def read_events(csvfile, _strptime=datetime.datetime.strptime):
    def _median(times):
        return statistics.median((_strptime(time, '%M:%S:%f').time()
                                  for time in row['Times']))

    fieldnames = ['Event', 'Name', 'Stroke']
    with open(csvfile) as infile:
        reader = csv.DictReader(infile, fieldnames=fieldnames, restkey='Times')
        next(reader)  # Skip header.
        for row in reader:
            yield Event(row['Stroke'], row['Name'], _median(row['Times']))


events = tuple(read_events('swimmers.csv'))

for stroke, evts in sort_and_group(events, key=lambda evt: evt.stroke):
    events_by_name = sort_and_group(evts, key=lambda evt: evt.name)
    best_times = (min(evt) for _, evt in events_by_name)
    sorted_by_time = sorted(best_times, key=lambda evt: evt.time)
    teams = zip(('A', 'B'), it.islice(grouper(sorted_by_time, 4), 2))
    for team, swimmers in teams:
        print('{stroke} {team}: {names}'.format(
            stroke=stroke.capitalize(),
            team=team,
            names=', '.join(swimmer.name for swimmer in swimmers)
        ))

If you run the above code, you’ll get the following output:

Backstroke A: Sophia, Grace, Penelope, Addison
Backstroke B: Elizabeth, Audrey, Emily, Aria
Breaststroke A: Samantha, Avery, Layla, Zoe
Breaststroke B: Lillian, Aria, Ava, Alexa
Butterfly A: Audrey, Leah, Layla, Samantha
Butterfly B: Alexa, Zoey, Emma, Madison
Freestyle A: Aubrey, Emma, Olivia, Evelyn
Freestyle B: Elizabeth, Zoe, Addison, Madison

Where to Go From Here

If you have made it this far, congratulations! I hope you have enjoyed the journey.

itertools is a powerful module in the Python standard library, and an essential tool to have in your toolkit. With it, you can write faster and more memory efficient code that is often simpler and easier to read (although that is not always the case, as you saw in the section on second order recurrence relations).

If anything, though, itertools is a testament to the power of iterators and lazy evaluation. Even though you have seen many techniques, this article only scratches the surface.

So I guess this means your journey is only just beginning.

Free Bonus: Click here to get our itertools cheat sheet that summarizes the techniques demonstrated in this tutorial.

In fact, this article skipped two itertools functions: starmap() and compress(). In my experience, these are two of the lesser used itertools functions, but I urge you to read their docs an experiment with your own use cases!

Here are a few places where you can find more examples of itertools in action (thanks to Brad Solomon for these fine suggestions):

Finally, for even more tools for constructing iterators, take a look at more-itertools.

Do you have any favorite itertools recipes/use-cases? We would love to hear about them in the comments!

We would like to thank our readers Putcher and Samir Aghayev for pointing out a couple of errors in the original version of this article.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Interacting with Python

2018-05-28T14:00:00+00:00

At this point, you should have a working Python 3 interpreter at hand. If you need help getting Python set up correctly, please refer to the previous section in this tutorial series.

Here’s what you’ll learn in this tutorial: Now that you have a working Python setup, you’ll see how to actually execute Python code and run Python programs. By the end of this article, you’ll know how to:

Use Python interactively by typing code directly into the interpreter
Execute code contained in a script file from the command line
Work within a Python Integrated Development Environment (IDE)

It’s time to write some Python code!

Hello, World!

There is a long-standing custom in the field of computer programming that the first code written in a newly installed language is a short program that simply displays the string Hello, World! to the console.

Note: This is a time-honored tradition dating back to the 1970s. See Hello, World! for a brief history. You seriously risk upsetting the qi of the universe if you don’t abide by this custom.

The simplest Python 3 code to display Hello, World! is:

print("Hello, World!")

You will explore several different ways to execute this code below.

Using the Python Interpreter Interactively

The most straightforward way to start talking to Python is in an interactive Read-Eval-Print Loop (REPL) environment. That simply means starting up the interpreter and typing commands to it directly. The interpreter:

Reads the command you enter
Evaluates and executes the command
Prints the output (if any) to the console
Loops back and repeats the process

The session continues in this manner until you instruct the interpreter to terminate. Most of the example code in this tutorial series is presented as REPL interaction.

Starting the Interpreter

In a GUI desktop environment, it is likely that the installation process placed an icon on the desktop or an item in the desktop menu system that starts Python.

For example, in Windows, there will likely be a program group in the Start menu labeled Python 3.x, and under it a menu item labeled Python 3.x (32-bit), or something similar depending on the particular installation you chose.

Clicking on that item will start the Python interpreter:

The Python interpreter (REPL) running inside a terminal window.

Alternatively, you can open a terminal window and run the interpreter from the command line. How you go about opening a terminal window varies depending on which operating system you’re using:

In Windows, it is called Command Prompt.
In macOS or Linux, it should be called Terminal.

Using your operating system’s search function to search for “command” in Windows or “terminal” in macOS or Linux should find it.

Once a terminal window is open, if paths have been set up properly by the Python install process, you should be able to just type python. Then, you should see a response from the Python interpreter.

This example is from the Windows Command Prompt window:

C:\Users\john>python
Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 07:18:10) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

Technical Note: If you are on a Linux system and installed Python 3, it may be that both Python 2 and Python 3 are installed. In that case, it is possible that typing python at the prompt will start Python 2. Starting Python 3 may require typing something else, like python3.

If you installed a more recent version of Python 3 than the one that was included in the distribution, you may even need to specify the version you installed specifically—for example python3.6.

If you are not seeing the >>> prompt, then you are not talking to the Python interpreter. This could be because Python is either not installed or not in your terminal window session’s path. It’s also possible that you just haven’t found the correct command to execute it. You can refer to our installing Python tutorial for help.

Executing Python Code

If you are seeing the prompt, you’re off and running! The next step is to execute the statement that displays Hello, World! to the console:

Ensure that the >>> prompt is displayed, and the cursor is positioned after it.
Type the command print("Hello, World!") exactly as shown.
Press the Enter key.

The interpreter’s response should appear on the next line. You can tell it is console output because the >>> prompt is absent:

>>> print("Hello, World!")
Hello, World!

If your session looks like the above, then you have executed your first Python code! Take a moment to celebrate.

Congratulations!

Did something go wrong? Perhaps you made one of these mistakes:

You forgot to enclose the string to be printed in quotation marks:

>>> print(Hello, World!)
  File "<stdin>", line 1
    print(Hello, World!)
                      ^
SyntaxError: invalid syntax

You remembered the opening quotation mark but forgot the closing one:

>>> print("Hello, World!)
  File "<stdin>", line 1
    print("Hello, World!)
                        ^
SyntaxError: EOL while scanning string literal

You used different opening and closing quotation marks:

>>> print("Hello, World!')
  File "<stdin>", line 1
    print("Hello, World!')
                         ^
SyntaxError: EOL while scanning string literal

You forgot the parentheses:

>>> print "Hello, World!"
  File "<stdin>", line 1
    print "Hello, World!"
                        ^
SyntaxError: Missing parentheses in call to 'print'

You entered extra whitespace characters before the command:

>>>     print("Hello, World!")
  File "<stdin>", line 1
    print("Hello, World!")
    ^
IndentationError: unexpected indent

(You will see in an upcoming section why this matters.)

If you got some sort of error message, go back and verify that you typed the command exactly as shown above.

Exiting the Interpreter

When you are finished interacting with the interpreter, you can exit a REPL session in several ways:

Type exit() and press Enter:
```
>>> exit()

C:\Users\john>
```
In Windows, type Ctrl+Z and press Enter:
```
>>> ^Z
C:\Users\john>
```
In Linux or macOS, type Ctrl+D. The interpreter terminates immediately; pressing Enter is not needed.
If all else fails, you can simply close the interpreter window. This isn’t the best way, but it will get the job done.

Running a Python Script from the Command Line

Entering commands to the Python interpreter interactively is great for quick testing and exploring features or functionality.

Eventually though, as you create more complex applications, you will develop longer bodies of code that you will want to edit and run repeatedly. You clearly don’t want to re-type the code into the interpreter every time! This is where you will want to create a script file.

A Python script is a reusable set of code. It is essentially a Python program—a sequence of Python instructions—contained in a file. You can run the program by specifying the name of the script file to the interpreter.

Python scripts are just plain text, so you can edit them with any text editor. If you have a favorite programmer’s editor that operates on text files, it should be fine to use. If you don’t, the following are typically installed natively with their respective operating systems:

Windows: Notepad
Unix/Linux: vi or vim
macOS: TextEdit

Using whatever editor you’ve chosen, create a script file called hello.py containing the following:

print("Hello, World!")

Now save the file, keeping track of the directory or folder you chose to save into.

Start a command prompt or terminal window. If the current working directory is the same as the location in which you saved the file, you can simply specify the filename as a command-line argument to the Python interpreter: python hello.py

For example, in Windows it would look like this:

C:\Users\john\Documents\test>dir
 Volume in drive C is JFS
 Volume Serial Number is 1431-F891

 Directory of C:\Users\john\Documents\test

05/20/2018  01:31 PM    <DIR>          .
05/20/2018  01:31 PM    <DIR>          ..
05/20/2018  01:31 PM                24 hello.py
               1 File(s)             24 bytes
               2 Dir(s)  92,557,885,440 bytes free

C:\Users\john\Documents\test>python hello.py
Hello, World!

If the script is not in the current working directory, you can still run it. You’ll just have to specify the path name to it:

C:\>cd
C:\

C:\>python c:\Users\john\Documents\test\hello.py
Hello, World!

In Linux or macOS, your session may look more like this:

jfs@jfs-xps:~$ pwd
/home/jfs

jfs@jfs-xps:~$ ls
hello.py

jfs@jfs-xps:~$ python hello.py
Hello, World!

A script file is not required to have a .py extension. The Python interpreter will run the file no matter what it’s called, so long as you properly specify the file name on the command line:

jfs@jfs-xps:~$ ls
hello.foo

jfs@jfs-xps:~$ cat hello.foo
print("Hello, World!")

jfs@jfs-xps:~$ python hello.foo
Hello, World!

But giving Python files a .py extension is a useful convention because it makes them more easily identifiable. In desktop-oriented folder/icon environments like Windows and macOS, this will also typically allow for setting up an appropriate file association, so that you can run the script just by clicking its icon.

Interacting with Python through an IDE

An Integrated Development Environment (IDE) is an application that more or less combines all the functionality you have seen so far. IDEs usually provide REPL capability as well as an editor with which you can create and modify code to then submit to the interpreter for execution.

You may also find cool features such as:

Syntax highlighting: IDEs often colorize different syntax elements in the code to make it easier to read.
Context-sensitive help: Advanced IDEs can display related information from the Python documentation or even suggested fixes for common types of code errors.
Code-completion: Some IDEs can complete partially typed pieces of code (like function names) for you—a great time-saver and convenience feature.
Debugging: A debugger allows you to run code step-by-step and inspect program data as you go. This is invaluable when you are trying to determine why a program is behaving improperly, as will inevitably happen.

IDLE

Most Python installations contain a rudimentary IDE called IDLE. The name ostensibly stands for Integrated Development and Learning Environment, but one member of the Monty Python troupe is named Eric Idle, which hardly seems like a coincidence.

The procedure for running IDLE varies from one operating system to another.

Starting IDLE in Windows

Go to the Start menu and select All Programs or All Apps. There should be a program icon labeled IDLE (Python 3.x 32-bit) or something similar. This will vary slightly between Win 7, 8, and 10. The IDLE icon may be in a program group folder named Python 3.x. You can also find the IDLE program icon by using the Windows search facility from the start menu and typing in IDLE.

Click on the icon to start IDLE.

Starting IDLE in macOS

Open Spotlight Search. Typing Cmd+Space is one of several ways to do this. In the search box, type terminal and press Enter.

In the terminal window, type idle3 and press Enter.

Starting IDLE in Linux

IDLE is available with the Python 3 distribution but may not have been installed by default. To find out whether it is, open a terminal window. This varies depending on the Linux distribution, but you should be able to find it by using the desktop search function and searching for terminal. In the terminal window, type idle3 and press Enter.

If you get an error saying command not found or something to that effect, then IDLE is apparently not installed, so you’ll need to install it.

The method for installing apps also varies from one Linux distribution to the next. For example, with Ubuntu Linux, the command to install IDLE is sudo apt-get install idle3. Many Linux distributions have GUI-based application managers that you can use to install apps as well.

Follow whatever procedure is appropriate for your distribution to install IDLE. Then, type idle3 in a terminal window and press Enter to run it. Your installation procedure may have also set up a program icon somewhere on the desktop to start IDLE as well.

Whew!

Using IDLE

Once IDLE is installed and you have started it successfully, you should see a window titled Python 3.x.x Shell, where 3.x.x corresponds to your version of Python:

The >>> prompt should look familiar. You can type REPL commands interactively, just like when you started the interpreter from a console window. Mindful of the qi of the universe, display Hello, World! again:

The interpreter behaves more or less the same as when you ran it directly from the console. The IDLE interface adds the perk of displaying different syntactic elements in distinct colors to make things more readable.

It also provides context-sensitive help. For example, if you type print( without typing any of the arguments to the print function or the closing parenthesis, then flyover text should appear specifying usage information for the print() function.

One other feature IDLE provides is statement recall:

If you have typed in several statements, you can recall them with Alt+P and Alt+N in Windows or Linux.
Alt+P cycles backward through previously executed statements; Alt+N cycles forward.
Once a statement has been recalled, you can use editing keys on the keyboard to edit it and then execute it again. The corresponding commands in macOS are Cmd+P and Cmd+N.

You can also create script files and run them in IDLE. From the Shell window menu, select File → New File. That should open an additional editing window. Type in the code to be executed:

From the menu in that window, select File → Save or File → Save As… and save the file to disk. Then select Run → Run Module. The output should appear back in the interpreter Shell window:

OK, that’s probably enough Hello, World!. The qi of the universe should be safe.

Once both windows are open, you can switch back and forth, editing the code in one window, running it and displaying its output in the other. In that way, IDLE provides a rudimentary Python development platform.

Although it is somewhat basic, it supports quite a bit of additional functionality, including code completion, code formatting, and a debugger. See the IDLE documentation for more details.

Thonny

Thonny is free Python IDE developed and maintained by the Institute of Computer Science at the University of Tartu, Estonia. It is targeted at Python beginners specifically, so the interface is simple and uncluttered as well as easy to understand and get comfortable with quickly.

Like IDLE, Thonny supports REPL interaction as well as script file editing and execution:

Thonny performs syntax highlighting and code completion in addition to providing a step-by-step debugger. One feature that is particularly helpful to those learning Python is that the debugger displays values in expressions as they are evaluated while you are stepping through the code:

Thonny is especially easy to get started with because it comes with Python 3.6 built in. So you only need to perform one install, and you’re ready to go!

Versions are available for Windows, macOS, and Linux. The Thonny website has download and installation instructions.

IDLE and Thonny are certainly not the only games going. There are many other IDEs available for Python code editing and development. See our Python IDEs and Code Editors Guide for additional suggestions.

Online Python REPL Sites

As you saw in the previous section, there are websites available that can provide you with interactive access to a Python interpreter online without you having to install anything locally.

This approach may be unsatisfactory for some of the more complicated or lengthy examples in this tutorial. But for simple REPL sessions, it should work well.

The Python Software Foundation provides an Interactive Shell on their website. On the main page, click on the button that looks like one of these:

Or go directly to https://www.python.org/shell.

You should get a page with a window that looks something like this:

The familiar >>> prompt shows you that you are talking to the Python interpreter.

Here are a few other sites that provide Python REPL:

Conclusion

Larger applications are typically contained in script files that are passed to the Python interpreter for execution.

But one of the advantages of an interpreted language is that you can run the interpreter and execute commands interactively. Python is easy to use in this manner, and it is a great way to get your feet wet learning how the language works.

The examples throughout this tutorial have been produced by direct interaction with the Python interpreter, but if you choose to use IDLE or some other available IDE, the examples should still work just fine.

Continue to the next section, where you will start to explore the elements of the Python language itself.

« Installing Python

Interacting with Python

Basic Data Types »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python 3 Installation & Setup Guide

2018-05-23T14:00:00+00:00

To get started working with Python 3, you’ll need to have access to the Python interpreter. There are several common ways to accomplish this:

Python can be obtained from the Python Software Foundation website at python.org. Typically, that involves downloading the appropriate installer for your operating system and running it on your machine.
Some operating systems, notably Linux, provide a package manager that can be run to install Python.
On macOS, the best way to install Python 3 involves installing a package manager called Homebrew. You’ll see how to do this in the relevant section in the tutorial.
On mobile operating systems like Android and iOS, you can install apps that provide a Python programming environment. This can be a great way to practice your coding skills on the go.

Alternatively, there are several websites that allow you to access a Python interpreter online without installing anything on your computer at all.

Note: There is a chance that Python may have been shipped with your operating system and is already installed. Even if that is the case, it may be that the installed version is outdated, in which case you will want to obtain the latest version anyhow.

In this Python installation guide, you’ll see step by step how to set up a working Python 3 distribution on Windows, macOS, Linux, iOS, and Android. So let’s get started!

Windows

It is highly unlikely that your Windows system shipped with Python already installed. Windows systems typically do not. Fortunately, installing does not involve much more than downloading the Python installer from the python.org website and running it. Let’s take a look at how to install Python 3 on Windows:

Step 1: Download the Python 3 Installer

Open a browser window and navigate to the Download page for Windows at python.org.
Underneath the heading at the top that says Python Releases for Windows, click on the link for the Latest Python 3 Release - Python 3.x.x. (As of this writing, the latest is Python 3.6.5.)
Scroll to the bottom and select either Windows x86-64 executable installer for 64-bit or Windows x86 executable installer for 32-bit. (See below.)

Sidebar: 32-bit or 64-bit Python?

For Windows, you can choose either the 32-bit or 64-bit installer. Here’s what the difference between the two comes down to:

If your system has a 32-bit processor, then you should choose the 32-bit installer.

On a 64-bit system, either installer will actually work for most purposes. The 32-bit version will generally use less memory, but the 64-bit version performs better for applications with intensive computation.

If you’re unsure which version to pick, go with the 64-bit version.

Note: Remember that if you get this choice “wrong” and would like to switch to another version of Python, you can just uninstall Python and then re-install it by downloading another installer from python.org.

Step 2: Run the Installer

Once you have chosen and downloaded an installer, simply run it by double-clicking on the downloaded file. A dialog should appear that looks something like this:

Important: You want to be sure to check the box that says Add Python 3.x to PATH as shown to ensure that the interpreter will be placed in your execution path.

Then just click Install Now. That should be all there is to it. A few minutes later you should have a working Python 3 installation on your system.

Windows Subsystem for Linux (WSL)

If you are running Windows 10 Creators or Anniversary Update, you actually have another option for installing Python. These versions of Windows 10 include a feature called the Windows Subsystem for Linux, which allows you to run a Linux environment directly in Windows, unmodified and without the overhead of a virtual machine.

For more information, see the Windows Subsystem for Linux Documentation article on the Microsoft website.
For instructions on how to enable the subsystem in Windows 10 and install a Linux distribution, see the Windows 10 Installation Guide.
You can also check out this presentation on YouTube by Sarah Cooley, one of the members of the WSL development team.

Once you have installed the Linux distribution of your choice, you can install Python 3 from a Bash console window, just as you would if you were running that Linux distribution natively. (See below.)

Linux

There is a very good chance your Linux distribution has Python installed already, but it probably won’t be the latest version, and it may be Python 2 instead of Python 3.

To find out what version(s) you have, open a terminal window and try the following commands:

python --version
python2 --version
python3 --version

One or more of these commands should respond with a version, as below:

$ python3 --version
Python 3.6.5

If the version shown is Python 2.x.x or a version of Python 3 that is not the latest (3.6.5 as of this writing), then you will want to install the latest version. The procedure for doing this will depend on the Linux distribution you are running.

Ubuntu

Depending on the version of the Ubuntu distribution you run, the Python install instructions vary. You can determine your local Ubuntu version by running the following command:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial

Depending on the version number you see under Release in the console output, follow the instructions below:

Ubuntu 17.10, Ubuntu 18.04 (and above) come with Python 3.6 by default. You should be able to invoke it with the command python3.
Ubuntu 16.10 and 17.04 do not come with Python 3.6 by default, but it is in the Universe repository. You should be able to install it with the following commands:
```
$ sudo apt-get update
$ sudo apt-get install python3.6
```
You can then invoke it with the command python3.6.
If you are using Ubuntu 14.04 or 16.04, Python 3.6 is not in the Universe repository, and you need to get it from a Personal Package Archive (PPA). For example, to install Python from the “deadsnakes” PPA, do the following:
```
$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt-get update
$ sudo apt-get install python3.6
```
As above, invoke with the command python3.6.

Linux Mint

Mint and Ubuntu use the same package management system, which frequently makes life easier. You can follow the instructions above for Ubuntu 14.04. The “deadsnakes” PPA works with Mint.

Debian

We found sources that indicated that the Ubuntu 16.10 method would work for Debian, but we never found a path to get it to work on Debian 9. Instead, we ended up making Python from source as listed below.

One issue with Debian, however, is that it generally does not install the sudo command by default. To install it, you’ll need to do the following before you carry out the Compiling Python From Source instructions below:

$ su
$ apt-get install sudo
$ vi /etc/sudoers

After that, open the /etc/sudoers file using the sudo vim command (or your favorite text editor.) Add the following line of text to the end of the file, replacing your_username with your actual username:

your_username ALL=(ALL) ALL

openSUSE

We found several sites describing how to get zypper to install the latest version of Python, but they seemed problematic or outdated. We did not manage to get any of them to work successfully, so we fell back to building Python from source. To do that, you will need to install the development tools, which can be done in YaST (via the menus) or by using zypper:

$ sudu zypper install -t pattern devel_C_C++

This step took a while and involved the installation of 154 packages, but once it was completed, we were able to build the source as shown in the Compiling Python From Source section above.

Fedora

Fedora has a roadmap to switch to Python 3 as the default Python published here. It indicates that the current version and the next few versions will all ship with Python 2 as the default, but Python 3 will be installed. If the python3 installed on your version is not 3.6, you can use the following command to install it:

$ sudo dnf install python36

Arch Linux

Arch Linux is fairly aggressive about keeping up with Python releases. It is likely you already have the latest version. If not, you can use this command:

$ packman -S python

Compiling Python From Source

Sometimes your Linux distribution will not have the latest version of Python, or maybe you just want to be able to build the latest, greatest version yourself. Here are the steps you need to take to build Python from source:

Step 1: Download the Source Code

To start, you need to get the Python source code. Python.org makes this fairly easy. If you go to the Downloads page, you will see the latest source for Python 3 at the top. (Make sure you don’t grab Legacy Python, Python 2.)

When you select the version, at the bottom of the page there is a Files section. Select the Gzipped source tarball and download it to your machine. If you prefer a command line method, you can easily use wget to download it to your current directory:

$ wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz

Step 2: Prepare Your System

There are a few distro-specific steps involved in building Python from scratch. The goal of each step is the same on all distros, but you might need to translate to your distribution if it does not use apt-get:

The first step you should take when doing an operation like this is to update the system packages on your machine before you start. On Debian, this is what that looks like:
```
$ sudo apt-get update
$ sudo apt-get upgrade
```

Next, we want to make sure the system has the tools needed to build Python. There are a bunch of them and you might already have some, but that’s fine. I’ve listed them all in one command line, but you can break the list into shorter commands by just repeating the sudo apt-get install -y portion:

# For apt-based systems (like Debian, Ubuntu, and Mint)
$ sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev  libncursesw5-dev xz-utils tk-dev

# For yum-based systems (like CentOS)
$ sudo yum -y groupinstall development
$ sudo yum -y install zlib-devel

Step 3: Build Python

Once you have the prerequisites and the tar file, you can unpack the source into a directory. Note that the following command will create a new directory called Python-3.6.5 under the one you are in:
```
$ tar xvf Python-3.6.5.tgz
$ cd Python-3.6.5
```

Now you need to run the ./configure tool to prepare the build:

$ ./configure --enable-optimizations --with-ensurepip=install

Next, you build the Python programs using make. The -j option simply tells make to split the building into parallel steps to speed up the compilation. Even with the parallel builds, this step can take a several minutes:
```
$ make -j 8
```
Then, you’ll want to install your new version of Python. You’ll use the altinstall target here in order to not overwrite the system’s version of Python. Since you’re installing Python into /usr/bin, you’ll need to run as root:
```
$ sudo make altinstall
```
Warning: Please only use the altinstall target on make. Using the install target will overwrite the python binary. While this seems like it would be cool, there are big portions of the system that rely on the pre-installed version of Python.

Step 4: Verify Your Python Install

Finally, you can test out your new Python version:

$ python3.6 -V
Python 3.6.5

macOS / Mac OS X

While current versions of macOS (previously known as “Mac OS X”) include a version of Python 2, it is likely out of date by a few months. Also, this tutorial series uses Python 3, so let’s get you upgraded to that.

The best way we found to install Python 3 on macOS is through the Homebrew package manager. This approach is also recommended by community guides like The Hitchhiker’s Guide to Python.

Step 1: Install Homebrew (Part 1)

To get started, you first want to install Homebrew:

Open a browser and navigate to http://brew.sh/. After the page has finished loading, select the Homebrew bootstrap code under “Install Homebrew”. Then hit Cmd+C to copy it to the clipboard. Make sure you’ve captured the text of the complete command because otherwise the installation will fail.
Now you need to open a Terminal.app window, paste the Homebrew bootstrap code, and then hit Enter. This will begin the Homebrew installation.
If you’re doing this on a fresh install of macOS, you may get a pop up alert asking you to install Apple’s “command line developer tools”. You’ll need those to continue with the installation, so please confirm the dialog box by clicking on “Install”.

At this point, you’re likely waiting for the command line developer tools to finish installing, and that’s going to take a few minutes. Time to grab a coffee or tea!

Step 2: Install Homebrew (Part 2)

You can continue installing Homebrew and then Python after the command line developer tools installation is complete:

Confirm the “The software was installed” dialog from the developer tools installer.
Back in the terminal, hit Enter to continue with the Homebrew installation.
Homebrew asks you to enter your password so it can finalize the installation. Enter your user account password and hit Enter to continue.
Depending on your internet connection, Homebrew will take a few minutes to download its required files. Once the installation is complete, you’ll end up back at the command prompt in your terminal window.

Whew! Now that the Homebrew package manager is set up, let’s continue on with installing Python 3 on your system.

Step 3: Install Python

Once Homebrew has finished installing, return to your terminal and run the following command:

$ brew install python3

Note: When you copy this command, be sure you don’t include the $ character at the beginning. That’s just an indicator that this is a console command.

This will download and install the latest version of Python. After the Homebrew brew install command finishes, Python 3 should be installed on your system.

You can make sure everything went correctly by testing if Python can be accessed from the terminal:

Open the terminal by launching Terminal.app.
Type pip3 and hit Enter.
You should see the help text from Python’s “Pip” package manager. If you get an error message running pip3, go through the Python install steps again to make sure you have a working Python installation.

Assuming everything went well and you saw the output from Pip in your command prompt window…congratulations! You just installed Python on your system, and you’re all set to continue with the next section in this tutorial.

iOS (iPhone / iPad)

The Pythonista app for iOS is a full-fledged Python development environment that you can run on your iPhone or iPad. It’s basically a combination of a Python editor, documentation, and interpreter rolled into one single app.

Pythonista is surprisingly fun to use. It’s a great little tool when you’re stuck without a laptop and want to work on your Python skills on the go. It comes with the complete Python 3 standard library and even includes full documentation you can browse offline.

To install and set up Pythonista you need to download it from the iOS app store.

Android (Phones & Tablets)

If you have an Android tablet or phone and want to practice Python on the go, there are a several options available. The one that we found most reliably supports Python 3.6 is Pydroid 3.

Pydroid 3 features an interpreter you can use for REPL sessions, and it also provides the ability to edit, save, and execute Python code:

You can download and install Pydroid 3 from the Google Play store. There is a free version and also a paid Premium version which supports code prediction and code analysis.

Online Python Interpreters

If you want to try out the examples in this tutorial without installing Python on your machine, there are several web sites available where you can interact with a Python interpreter online:

Python.org Online Console: www.python.org/shell
Python Fiddle: pythonfiddle.com
Repl.it: repl.it
Trinket: trinket.io
Python Anywhere: www.pythonanywhere.com

These cloud-based Python interpreters may not be able to execute some of the more complex examples in this tutorial, but they will be adequate for running most of the simpler ones and may be a nice way to get you started. More information on using these sites is presented in the next section.

Conclusion

This section provided you with the information you need to gain access to a Python 3 interpreter. You are now ready to head to the next section and begin interacting with Python!

« Introduction to Python

Installing Python

Interacting with Python »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Introduction to Python 3

2018-05-21T14:00:00+00:00

Python is a high-level, interpreted scripting language developed in the late 1980s by Guido van Rossum at the National Research Institute for Mathematics and Computer Science in the Netherlands. The initial version was published at the alt.sources newsgroup in 1991, and version 1.0 was released in 1994.

Python 2.0 was released in 2000, and the 2.x versions were the prevalent releases until December 2008. At that time, the development team made the decision to release version 3.0, which contained a few relatively small but significant changes that were not backward compatible with the 2.x versions. Python 2 and 3 are very similar, and some features of Python 3 have been backported to Python 2. But in general, they remain not quite compatible.

Both Python 2 and 3 have continued to be maintained and developed, with periodic release updates for both. As of this writing, the most recent versions available are 2.7.15 and 3.6.5. However, an official End Of Life date of January 1, 2020 has been established for Python 2, after which time it will no longer be maintained. If you are a newcomer to Python, it is recommended that you focus on Python 3, as this tutorial will do.

Python is still maintained by a core development team at the Institute, and Guido is still in charge, having been given the title of BDFL (Benevolent Dictator For Life) by the Python community. The name Python, by the way, derives not from the snake, but from the British comedy troupe Monty Python’s Flying Circus, of which Guido was, and presumably still is, a fan. It is common to find references to Monty Python sketches and movies scattered throughout the Python documentation.

Free PDF Download: Python 3 Cheat Sheet

Why Choose Python?

If you’re going to write programs, there are literally dozens of commonly used languages to choose from. Why choose Python? Here are some of the features that make Python an appealing choice.

Python is Popular

Python has been growing in popularity over the last few years. The 2018 Stack Overflow Developer Survey ranked Python as the 7th most popular and the number one most wanted technology of the year. World-class software development countries around the globe use Python every single day.

According to research by Dice Python is also one of the hottest skills to have and the most popular programming language in the world based on the Popularity of Programming Language Index.

Due to the popularity and widespread use of Python as a programming language, Python developers are sought after and paid well. If you’d like to dig deeper into Python salary statistics and job opportunities, you can do so here.

Python is Interpreted

Many languages are compiled, meaning the source code you create needs to be translated into machine code, the language of your computer’s processor, before it can be run. Programs written in an interpreted language are passed straight to an interpreter that runs them directly.

This makes for a quicker development cycle because you just type in your code and run it, without the intermediate compilation step.

One potential downside to interpreted languages is execution speed. Programs that are compiled into the native language of the computer processor tend to run more quickly than interpreted programs. For some applications that are particularly computationally intensive, like graphics processing or intense number crunching, this can be limiting.

In practice, however, for most programs, the difference in execution speed is measured in milliseconds, or seconds at most, and not appreciably noticeable to a human user. The expediency of coding in an interpreted language is typically worth it for most applications.

Further reading: See this Wikipedia page to read more about the differences between interpreted and compiled languages.

Python is Free

The Python interpreter is developed under an OSI-approved open-source license, making it free to install, use, and distribute, even for commercial purposes.

A version of the interpreter is available for virtually any platform there is, including all flavors of Unix, Windows, macOS, smartphones and tablets, and probably anything else you ever heard of. A version even exists for the half dozen people remaining who use OS/2.

Python is Portable

Because Python code is interpreted and not compiled into native machine instructions, code written for one platform will work on any other platform that has the Python interpreter installed. (This is true of any interpreted language, not just Python.)

Python is Simple

As programming languages go, Python is relatively uncluttered, and the developers have deliberately kept it that way.

A rough estimate of the complexity of a language can be gleaned from the number of keywords or reserved words in the language. These are words that are reserved for special meaning by the compiler or interpreter because they designate specific built-in functionality of the language.

Python 3 has 33 keywords, and Python 2 has 31. By contrast, C++ has 62, Java has 53, and Visual Basic has more than 120, though these latter examples probably vary somewhat by implementation or dialect.

Python code has a simple and clean structure that is easy to learn and easy to read. In fact, as you will see, the language definition enforces code structure that is easy to read.

But It’s Not That Simple

For all its syntactical simplicity, Python supports most constructs that would be expected in a very high-level language, including complex dynamic data types, structured and functional programming, and object-oriented programming.

Additionally, a very extensive library of classes and functions is available that provides capability well beyond what is built into the language, such as database manipulation or GUI programming.

Python accomplishes what many programming languages don’t: the language itself is simply designed, but it is very versatile in terms of what you can accomplish with it.

Conclusion

This section gave an overview of the Python programming language, including:

A brief history of the development of Python
Some reasons why you might select Python as your language of choice

Python is a great option, whether you are a beginning programmer looking to learn the basics, an experienced programmer designing a large application, or anywhere in between. The basics of Python are easily grasped, and yet its capabilities are vast.

Proceed to the next section to learn how to acquire and install Python on your computer.

Introduction to Python

Installing Python »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Ultimate Guide to Data Classes in Python 3.7

2018-05-15T14:00:00+00:00

One new and exciting feature coming in Python 3.7 is the data class. A data class is a class typically containing mainly data, although there aren’t really any restrictions. It is created using the new @dataclass decorator, as follows:

from dataclasses import dataclass

@dataclass
class DataClassCard:
    rank: str
    suit: str

Note: This code, as well as all other examples in this tutorial, will only work in Python 3.7 and above.

A data class comes with basic functionality already implemented. For instance, you can instantiate, print, and compare data class instances straight out of the box:

>>> queen_of_hearts = DataClassCard('Q', 'Hearts')
>>> queen_of_hearts.rank
'Q'
>>> queen_of_hearts
DataClassCard(rank='Q', suit='Hearts')
>>> queen_of_hearts == DataClassCard('Q', 'Hearts')
True

Compare that to a regular class. A minimal regular class would look something like this:

class RegularCard
    def __init__(self, rank, suit):
        self.rank = rank
        self.suit = suit

While this is not much more code to write, you can already see signs of the boilerplate pain: rank and suit are both repeated three times simply to initialize an object. Furthermore, if you try to use this plain class, you’ll notice that the representation of the objects is not very descriptive, and for some reason a queen of hearts is not the same as a queen of hearts:

>>> queen_of_hearts = RegularCard('Q', 'Hearts')
>>> queen_of_hearts.rank
'Q'
>>> queen_of_hearts
<__main__.RegularCard object at 0x7fb6eee35d30>
>>> queen_of_hearts == RegularCard('Q', 'Hearts')
False

Seems like data classes are helping us out behind the scenes. By default, data classes implement a .__repr__() method to provide a nice string representation and an .__eq__() method that can do basic object comparisons. For the RegularCard class to imitate the data class above, you need to add these methods as well:

class RegularCard
    def __init__(self, rank, suit):
        self.rank = rank
        self.suit = suit

    def __repr__(self):
        return (f'{self.__class__.__name__}'
                f'(rank={self.rank!r}, suit={self.suit!r})')

    def __eq__(self, other):
        if other.__class__ is not self.__class__:
            return NotImplemented
        return (self.rank, self.suit) == (other.rank, other.suit)

In this tutorial, you will learn exactly which conveniences data classes provide. In addition to nice representations and comparisons, you’ll see:

How to add default values to data class fields
How data classes allow for ordering of objects
How to represent immutable data
How data classes handle inheritance

We will soon dive deeper into those features of data classes. However, you might be thinking that you have already seen something like this before.

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

Alternatives to Data Classes

For simple data structures, you have probably already used a tuple or a dict. You could represent the queen of hearts card in either of the following ways:

>>> queen_of_hearts_tuple = ('Q', 'Hearts')
>>> queen_of_hearts_dict = {'rank': 'Q', 'suit': 'Hearts'}

It works. However, it puts a lot of responsibility on you as a programmer:

You need to remember that the queen_of_hearts_... variable represents a card.
For the _tuple version, you need to remember the order of the attributes. Writing ('Spades', 'A') will mess up your program but probably not give you an easily understandable error message.
If you use the _dict, you must make sure the names of the attributes are consistent. For instance {'value': 'A', 'suit': 'Spades'} will not work as expected.

Furthermore, using these structures is not ideal:

>>> queen_of_hearts_tuple[0]  # No named access
'Q'
>>> queen_of_hearts_dict['suit']  # Would be nicer with .suit
'Hearts'

A better alternative is the namedtuple. It has long been used to create readable small data structures. We can in fact recreate the data class example above using a namedtuple like this:

from collections import namedtuple

NamedTupleCard = namedtuple('NamedTupleCard', ['rank', 'suit'])

This definition of NamedTupleCard will give the exact same output as our DataClassCard example did:

>>> queen_of_hearts = NamedTupleCard('Q', 'Hearts')
>>> queen_of_hearts.rank
'Q'
>>> queen_of_hearts
NamedTupleCard(rank='Q', suit='Hearts')
>>> queen_of_hearts == NamedTupleCard('Q', 'Hearts')
True

So why even bother with data classes? First of all, data classes come with many more features than you have seen so far. At the same time, the namedtuple has some other features that are not necessarily desirable. By design, a namedtuple is a regular tuple. This can be seen in comparisons, for instance:

>>> queen_of_hearts == ('Q', 'Hearts')
True

While this might seem like a good thing, this lack of awareness about its own type can lead to subtle and hard-to-find bugs, especially since it will also happily compare two different namedtuple classes:

>>> Person = namedtuple('Person', ['first_initial', 'last_name']
>>> ace_of_spades = NamedTupleCard('A', 'Spades')
>>> ace_of_spades == Person('A', 'Spades')
True

The namedtuple also comes with some restrictions. For instance, it is hard to add default values to some of the fields in a namedtuple. A namedtuple is also by nature immutable. That is, the value of a namedtuple can never change. In some applications, this is an awesome feature, but in other settings, it would be nice to have more flexibility:

>>> card = NamedTupleCard('7', 'Diamonds')
>>> card.rank = '9'
AttributeError: can't set attribute

Data classes will not replace all uses of namedtuple. For instance, if you need your data structure to behave like a tuple, then a named tuple is a great alternative!

Another alternative, and one of the inspirations for data classes, is the attrs project. With attrs installed (pip install attrs), you can write a card class as follows:

import attr

@attr.s
class AttrsCard:
    rank = attr.ib()
    suit = attr.ib()

This can be used in exactly the same way as the DataClassCard and NamedTupleCard examples earlier. The attrs project is great and does support some features that data classes do not, including converters and validators. Furthermore, attrs has been around for a while and is supported in Python 2.7 as well as Python 3.4 and up. However, as attrs is not a part of the standard library, it does add an external dependency to your projects. Through data classes, similar functionality will be available everywhere.

In addition to tuple, dict, namedtuple, and attrs, there are many other similar projects, including typing.NamedTuple, namedlist, attrdict, plumber, and fields. While data classes are a great new alternative, there are still use cases where one of the older variants fits better. For instance, if you need compatibility with a specific API expecting tuples or need functionality not supported in data classes.

Basic Data Classes

Let us get back to data classes. As an example, we will create a Position class that will represent geographic positions with a name as well as the latitude and longitude:

from dataclasses import dataclass

@dataclass
class Position:
    name: str
    lon: float
    lat: float

What makes this a data class is the @dataclass decorator just above the class definition. Beneath the class Position: line, you simply list the fields you want in your data class. The : notation used for the fields is using a new feature in Python 3.6 called variable annotations. We will soon talk more about this notation and why we specify data types like str and float.

Those few lines of code are all you need. The new class is ready for use:

>>> pos = Position('Oslo', 10.8, 59.9)
>>> print(pos)
Position(name='Oslo', lon=10.8, lat=59.9)
>>> pos.lat
59.9
>>> print(f'{pos.name} is at {pos.lat}°N, {pos.lon}°E')
Oslo is at 59.9°N, 10.8°E

You can also create data classes similarly to how named tuples are created. The following is (almost) equivalent to the definition of Position above:

from dataclasses import make_dataclass

Position = make_dataclass('Position', ['name', 'lat', 'lon'])

A data class is a regular Python class. The only thing that sets it apart is that it has basic data model methods like .__init__(), .__repr__(), and .__eq__() implemented for you.

Default Values

It is easy to add default values to the fields of your data class:

from dataclasses import dataclass

@dataclass
class Position:
    name: str
    lon: float = 0.0
    lat: float = 0.0

This works exactly as if you had specified the default values in the definition of the .__init__() method of a regular class:

>>> Position('Null Island')
Position(name='Null Island', lon=0.0, lat=0.0)
>>> Position('Greenwich', lat=51.8)
Position(name='Greenwich', lon=0.0, lat=51.8)
>>> Position('Vancouver', -123.1, 49.3)
Position(name='Vancouver', lon=-123.1, lat=49.3)

Later you will learn about default_factory, which gives a way to provide more complicated default values.

Type Hints

So far, we have not made a big fuss of the fact that data classes support typing out of the box. You have probably noticed that we defined the fields with a type hint: name: str says that name should be a text string (str type).

In fact, adding some kind of type hint is mandatory when defining the fields in your data class. Without a type hint, the field will not be a part of the data class. However, if you do not want to add explicit types to your data class, use typing.Any:

from dataclasses import dataclass
from typing import Any

@dataclass
class WithoutExplicitTypes:
    name: Any
    value: Any = 42

While you need to add type hints in some form when using data classes, these types are not enforced at runtime. The following code runs without any problems:

>>> Position(3.14, 'pi day', 2018)
Position(name=3.14, lon='pi day', lat=2018)

This is how typing in Python usually works: Python is and will always be a dynamically typed language. To actually catch type errors, type checkers like Mypy can be run on your source code.

Adding Methods

You already know that a data class is just a regular class. That means that you can freely add your own methods to a data class. As an example, let us calculate the distance between one position and another, along the Earth’s surface. One way to do this is by using the haversine formula:

You can add a .distance_to() method to your data class just like you can with normal classes:

from dataclasses import dataclass
from math import asin, cos, radians, sin, sqrt

@dataclass
class Position:
    name: str
    lon: float = 0.0
    lat: float = 0.0

    def distance_to(self, other):
        r = 6371  # Earth radius in kilometers
        lam_1, lam_2 = radians(self.lon), radians(other.lon)
        phi_1, phi_2 = radians(self.lat), radians(other.lat)
        h = (sin((phi_2 - phi_1) / 2)**2
             + cos(phi_1) * cos(phi_2) * sin((lam_2 - lam_1) / 2)**2)
        return 2 * r * asin(sqrt(h))

It works as you would expect:

>>> oslo = Position('Oslo', 10.8, 59.9)
>>> vancouver = Position('Vancouver', -123.1, 49.3)
>>> oslo.distance_to(vancouver)
7181.7841229421165

More Flexible Data Classes

So far, you have seen some of the basic features of the data class: it gives you some convenience methods, and you can still add default values and other methods. Now you will learn about some more advanced features like parameters to the @dataclass decorator and the field() function. Together, they give you more control when creating a data class.

Let us return to the playing card example you saw at the beginning of the tutorial and add a class containing a deck of cards while we are at it:

from dataclasses import dataclass
from typing import List

@dataclass
class PlayingCard:
    rank: str
    suit: str

@dataclass
class Deck:
    cards: List[PlayingCard]

A simple deck containing only two cards can be created like this:

>>> queen_of_hearts = PlayingCard('Q', 'Hearts')
>>> ace_of_spades = PlayingCard('A', 'Spades')
>>> two_cards = Deck([queen_of_hearts, ace_of_spades])
Deck(cards=[PlayingCard(rank='Q', suit='Hearts'),
            PlayingCard(rank='A', suit='Spades')])

Advanced Default Values

Say that you want to give a default value to the Deck. It would for example be convenient if Deck() created a regular (French) deck of 52 playing cards. First, specify the different ranks and suits. Then, add a function make_french_deck() that creates a list of instances of PlayingCard:

RANKS = '2 3 4 5 6 7 8 9 10 J Q K A'.split()
SUITS = '♣ ♢ ♡ ♠'.split()

def make_french_deck():
    return [PlayingCard(r, s) for s in SUITS for r in RANKS]

For fun, the four different suits are specified using their Unicode symbols.

Note: Above, we used Unicode glyphs like ♠ directly in the source code. We could do this because Python supports writing source code in UTF-8 by default. Refer to this page on Unicode input for how to enter these on your system. You could also enter the Unicode symbols for the suits using \N named character escapes (like \N{BLACK SPADE SUIT}) or \u Unicode escapes (like \u2660).

To simplify comparisons of cards later, the ranks and suits are also listed in their usual order.

>>> make_french_deck()
[PlayingCard(rank='2', suit='♣'), PlayingCard(rank='3', suit='♣'), ...
 PlayingCard(rank='K', suit='♠'), PlayingCard(rank='A', suit='♠')]

In theory, you could now use this function to specify a default value for Deck.cards:

from dataclasses import dataclass
from typing import List

@dataclass
class Deck:  # Will NOT work
    cards: List[PlayingCard] = make_french_deck()

Don’t do this! This introduces one of the most common anti-patterns in Python: using mutable default arguments. The problem is that all instances of Deck will use the same list object as the default value of the .cards property. This means that if, say, one card is removed from one Deck, then it disappears from all other instances of Deck as well. Actually, data classes try to prevent you from doing this, and the code above will raise a ValueError.

Instead, data classes use something called a default_factory to handle mutable default values. To use default_factory (and many other cool features of data classes), you need to use the field() specifier:

from dataclasses import dataclass, field
from typing import List

@dataclass
class Deck:
    cards: List[PlayingCard] = field(default_factory=make_french_deck)

The argument to default_factory can be any zero parameter callable. Now it is easy to create a full deck of playing cards:

>>> Deck()
Deck(cards=[PlayingCard(rank='2', suit='♣'), PlayingCard(rank='3', suit='♣'), ...
            PlayingCard(rank='K', suit='♠'), PlayingCard(rank='A', suit='♠')])

The field() specifier is used to customize each field of a data class individually. You will see some other examples later. For reference, these are the parameters field() supports:

default: Default value of the field
default_factory: Function that returns the initial value of the field
init: Use field in .__init__() method? (Default is True.)
repr: Use field in repr of the object? (Default is True.)
compare: Include the field in comparisons? (Default is True.)
hash: Include the field when calculating hash()? (Default is to use the same as for compare.)
metadata: A mapping with information about the field

In the Position example, you saw how to add simple default values by writing lat: float = 0.0. However, if you also want to customize the field, for instance to hide it in the repr, you need to use the default parameter: lat: float = field(default=0.0, repr=False). You may not specify both default and default_factory.

The metadata parameter is not used by the data classes themselves but is available for you (or third party packages) to attach information to fields. In the Position example, you could for instance specify that latitude and longitude should be given in degrees:

from dataclasses import dataclass, field

@dataclass
class Position:
    name: str
    lon: float = field(default=0.0, metadata={'unit': 'degrees'})
    lat: float = field(default=0.0, metadata={'unit': 'degrees'})

The metadata (and other information about a field) can be retrieved using the fields() function (note the plural s):

>>> from dataclasses import fields
>>> fields(Position)
(Field(name='name',type=<class 'str'>,...,metadata={}),
 Field(name='lon',type=<class 'float'>,...,metadata={'unit': 'degrees'}),
 Field(name='lat',type=<class 'float'>,...,metadata={'unit': 'degrees'}))
>>> lat_unit = fields(Position)[2].metadata['unit']
>>> lat_unit
'degrees'

You Need Representation?

Recall that we can create decks of cards out of thin air:

>>> Deck()
Deck(cards=[PlayingCard(rank='2', suit='♣'), PlayingCard(rank='3', suit='♣'), ...
            PlayingCard(rank='K', suit='♠'), PlayingCard(rank='A', suit='♠')])

While this representation of a Deck is explicit and readable, it is also very verbose. I have deleted 48 of the 52 cards in the deck in the output above. On an 80-column display, simply printing the full Deck takes up 22 lines! Let us add a more concise representation. In general, a Python object has two different string representations:

repr(obj) is defined by obj.__repr__() and should return a developer-friendly representation of obj. If possible, this should be code that can recreate obj. Data classes do this.
str(obj) is defined by obj.__str__() and should return a user-friendly representation of obj. Data classes do not implement a .__str__() method, so Python will fall back to the .__repr__() method.

Let us implement a user-friendly representation of a PlayingCard:

from dataclasses import dataclass

@dataclass
class PlayingCard:
    rank: str
    suit: str

    def __str__(self):
        return f'{self.suit}{self.rank}'

The cards now look much nicer, but the deck is still as verbose as ever:

>>> ace_of_spades = PlayingCard('A', '♠')
>>> ace_of_spades
PlayingCard(rank='A', suit='♠')
>>> print(ace_of_spades)
♠A
>>> print(Deck())
Deck(cards=[PlayingCard(rank='2', suit='♣'), PlayingCard(rank='3', suit='♣'), ...
            PlayingCard(rank='K', suit='♠'), PlayingCard(rank='A', suit='♠')])

To show that it is possible to add your own .__repr__() method as well, we will violate the principle that it should return code that can recreate an object. Practicality beats purity after all. The following code adds a more concise representation of the Deck:

from dataclasses import dataclass, field
from typing import List

@dataclass
class Deck:
    cards: List[PlayingCard] = field(default_factory=make_french_deck)

    def __repr__(self):
        cards = ', '.join(f'{c!s}' for c in self.cards)
        return f'{self.__class__.__name__}({cards})'

Note the !s specifier in the {c!s} format string. It means that we explicitly want to use the str() representation of PlayingCards. With the new .__repr__(), the representation of Deck is easier on the eyes:

>>> Deck()
Deck(♣2, ♣3, ♣4, ♣5, ♣6, ♣7, ♣8, ♣9, ♣10, ♣J, ♣Q, ♣K, ♣A,
     ♢2, ♢3, ♢4, ♢5, ♢6, ♢7, ♢8, ♢9, ♢10, ♢J, ♢Q, ♢K, ♢A,
     ♡2, ♡3, ♡4, ♡5, ♡6, ♡7, ♡8, ♡9, ♡10, ♡J, ♡Q, ♡K, ♡A,
     ♠2, ♠3, ♠4, ♠5, ♠6, ♠7, ♠8, ♠9, ♠10, ♠J, ♠Q, ♠K, ♠A)

Comparing Cards

In many card games, cards are compared to each other. For instance in a typical trick taking game, the highest card takes the trick. As it is currently implemented, the PlayingCard class does not support this kind of comparison:

>>> queen_of_hearts = PlayingCard('Q', '♡')
>>> ace_of_spades = PlayingCard('A', '♠')
>>> ace_of_spades > queen_of_hearts
TypeError: '>' not supported between instances of 'Card' and 'Card'

This is, however, (seemingly) easy to rectify:

from dataclasses import dataclass

@dataclass(order=True)
class PlayingCard:
    rank: str
    suit: str

    def __str__(self):
        return f'{self.suit}{self.rank}'

The @dataclass decorator has two forms. So far you have seen the simple form where @dataclass is specified without any parentheses and parameters. However, you can also give parameters to the @dataclass() decorator in parentheses. The following parameters are supported:

init: Add .__init__() method? (Default is True.)
repr: Add .__repr__() method? (Default is True.)
eq: Add .__eq__() method? (Default is True.)
order: Add ordering methods? (Default is False.)
unsafe_hash: Force the addition of a .__hash__() method? (Default is False.)
frozen: If True, assigning to fields raise an exception. (Default is False.)

See the original PEP for more information about each parameter. After setting order=True, instances of PlayingCard can be compared:

>>> queen_of_hearts = PlayingCard('Q', '♡')
>>> ace_of_spades = PlayingCard('A', '♠')
>>> ace_of_spades > queen_of_hearts
False

How are the two cards compared though? You have not specified how the ordering should be done, and for some reason Python seems to believe that a Queen is higher than an Ace…

It turns out that data classes compare objects as if they were tuples of their fields. In other words, a Queen is higher than an Ace because 'Q' comes after 'A' in the alphabet:

>>> ('A', '♠') > ('Q', '♡')
False

That does not really work for us. Instead, we need to define some kind of sort index that uses the order of RANKS and SUITS. Something like this:

>>> RANKS = '2 3 4 5 6 7 8 9 10 J Q K A'.split()
>>> SUITS = '♣ ♢ ♡ ♠'.split()
>>> card = PlayingCard('Q', '♡')
>>> RANKS.index(card.rank) * len(SUITS) + SUITS.index(card.suit)
42

For PlayingCard to use this sort index for comparisons, we need to add a field .sort_index to the class. However, this field should be calculated from the other fields .rank and .suit automatically. This is exactly what the special method .__post_init__() is for. It allows for special processing after the regular .__init__() method is called:

from dataclasses import dataclass, field

RANKS = '2 3 4 5 6 7 8 9 10 J Q K A'.split()
SUITS = '♣ ♢ ♡ ♠'.split()

@dataclass(order=True)
class PlayingCard:
    sort_index: int = field(init=False, repr=False)
    rank: str
    suit: str

    def __post_init__(self):
        self.sort_index = (RANKS.index(self.rank) * len(SUITS)
                           + SUITS.index(self.suit))

    def __str__(self):
        return f'{self.suit}{self.rank}'

Note that .sort_index is added as the first field of the class. That way, the comparison is first done using .sort_index and only if there are ties are the other fields used. Using field(), you must also specify that .sort_index should not be included as a parameter in the .__init__() method (because it is calculated from the .rank and .suit fields). To avoid confusing the user about this implementation detail, it is probably also a good idea to remove .sort_index from the repr of the class.

Finally, aces are high:

>>> queen_of_hearts = PlayingCard('Q', '♡')
>>> ace_of_spades = PlayingCard('A', '♠')
>>> ace_of_spades > queen_of_hearts
True

You can now easily create a sorted deck:

>>> Deck(sorted(make_french_deck()))
Deck(♣2, ♢2, ♡2, ♠2, ♣3, ♢3, ♡3, ♠3, ♣4, ♢4, ♡4, ♠4, ♣5,
     ♢5, ♡5, ♠5, ♣6, ♢6, ♡6, ♠6, ♣7, ♢7, ♡7, ♠7, ♣8, ♢8,
     ♡8, ♠8, ♣9, ♢9, ♡9, ♠9, ♣10, ♢10, ♡10, ♠10, ♣J, ♢J, ♡J,
     ♠J, ♣Q, ♢Q, ♡Q, ♠Q, ♣K, ♢K, ♡K, ♠K, ♣A, ♢A, ♡A, ♠A)

Or, if you don’t care about sorting, this is how you draw a random hand of 10 cards:

>>> from random import sample
>>> Deck(sample(make_french_deck(), k=10))
Deck(♢2, ♡A, ♢10, ♣2, ♢3, ♠3, ♢A, ♠8, ♠9, ♠2)

Of course, you don’t need order=True for that…

Immutable Data Classes

One of the defining features of the namedtuple you saw earlier is that it is immutable. That is, the value of its fields may never change. For many types of data classes, this is a great idea! To make a data class immutable, set frozen=True when you create it. For example, the following is an immutable version of the Position class you saw earlier:

from dataclasses import dataclass

@dataclass(frozen=True)
class Position:
    name: str
    lon: float = 0.0
    lat: float = 0.0

In a frozen data class, you can not assign values to the fields after creation:

>>> pos = Position('Oslo', 10.8, 59.9)
>>> pos.name
'Oslo'
>>> pos.name = 'Stockholm'
dataclasses.FrozenInstanceError: cannot assign to field 'name'

Be aware though that if your data class contains mutable fields, those might still change. This is true for all nested data structures in Python (see this video for further info):

from dataclasses import dataclass
from typing import List

@dataclass(frozen=True)
class ImmutableCard:
    rank: str
    suit: str

@dataclass(frozen=True)
class ImmutableDeck:
    cards: List[PlayingCard]

Even though both ImmutableCard and ImmutableDeck are immutable, the list holding cards is not. You can therefore still change the cards in the deck:

>>> queen_of_hearts = ImmutableCard('Q', '♡')
>>> ace_of_spades = ImmutableCard('A', '♠')
>>> deck = ImmutableDeck([queen_of_hearts, ace_of_spades])
>>> deck
ImmutableDeck(cards=[ImmutableCard(rank='Q', suit='♡'), ImmutableCard(rank='A', suit='♠')])
>>> deck.cards[0] = ImmutableCard('7', '♢')
>>> deck
ImmutableDeck(cards=[ImmutableCard(rank='7', suit='♢'), ImmutableCard(rank='A', suit='♠')])

To avoid this, make sure all fields of an immutable data class use immutable types (but remember that types are not enforced at runtime). The ImmutableDeck should be implemented using a tuple instead of a list.

Inheritance

You can subclass data classes quite freely. As an example, we will extend our Position example with a country field and use it to record capitals:

from dataclasses import dataclass

@dataclass
class Position:
    name: str
    lon: float
    lat: float

@dataclass
class Capital(Position):
    country: str

In this simple example, everything works without a hitch:

>>> Capital('Oslo', 10.8, 59.9, 'Norway')
Capital(name='Oslo', lon=10.8, lat=59.9, country='Norway')

The country field of Capital is added after the three original fields in Position. Things get a little more complicated if any fields in the base class have default values:

from dataclasses import dataclass

@dataclass
class Position:
    name: str
    lon: float = 0.0
    lat: float = 0.0

@dataclass
class Capital(Position):
    country: str  # Does NOT work

This code will immediately crash with a TypeError complaining that “non-default argument ‘country’ follows default argument.” The problem is that our new country field has no default value, while the lon and lat fields have default values. The data class will try to write an .__init__() method with the following signature:

def __init__(name: str, lon: float = 0.0, lat: float = 0.0, country: str):
    ...

However, this is not valid Python. If a parameter has a default value, all following parameters must also have a default value. In other words, if a field in a base class has a default value, then all new fields added in a subclass must have default values as well.

Another thing to be aware of is how fields are ordered in a subclass. Starting with the base class, fields are ordered in the order in which they are first defined. If a field is redefined in a subclass, its order does not change. For example, if you define Position and Capital as follows:

from dataclasses import dataclass

@dataclass
class Position:
    name: str
    lon: float = 0.0
    lat: float = 0.0

@dataclass
class Capital(Position):
    country: str = 'Unknown'
    lat: float = 40.0

Then the order of the fields in Capital will still be name, lon, lat, country. However, the default value of lat will be 40.0.

>>> Capital('Madrid', country='Spain')
Capital(name='Madrid', lon=0.0, lat=40.0, country='Spain')

Optimizing Data Classes

I’m going to end this tutorial with a few words about slots. Slots can be used to make classes faster and use less memory. Data classes have no explicit syntax for working with slots, but the normal way of creating slots works for data classes as well. (They really are just regular classes!)

from dataclasses import dataclass

@dataclass
class SimplePosition:
    name: str
    lon: float
    lat: float

@dataclass
class SlotPosition:
    __slots__ = ['name', 'lon', 'lat']
    name: str
    lon: float
    lat: float

Essentially, slots are defined using .__slots__ to list the variables on a class. Variables or attributes not present in .__slots__ may not be defined. Furthermore, a slots class may not have default values.

The benefit of adding such restrictions is that certain optimizations may be done. For instance, slots classes take up less memory, as can be measured using Pympler:

>>> from pympler import asizeof
>>> simple = SimplePosition('London', -0.1, 51.5)
>>> slot = SlotPosition('Madrid', -3.7, 40.4)
>>> asizeof.asizesof(simple, slot)
(440, 248)

Similarly, slots classes are typically faster to work with. The following example measures the speed of attribute access on a slots data class and a regular data class using timeit from the standard library.

>>> from timeit import timeit
>>> timeit('slot.name', setup="from position import SlotPosition; slot=SlotPosition('Oslo', 10.8, 59.9)")
0.05882283499886398
>>> timeit('simple.name', setup="from position import SimplePosition; simple=SimplePosition('Oslo', 10.8, 59.9)")
0.09207444800267695

In this particular example, the slot class is about 35% faster.

Conclusion & Further Reading

Data classes are one of the new features of Python 3.7. With data classes, you do not have to write boilerplate code to get proper initialization, representation, and comparisons for your objects.

You have seen how to define your own data classes, as well as:

How to add default values to the fields in your data class
How to customize the ordering of data class objects
How to work with immutable data classes
How inheritance works for data classes

If you want to dive into all the details of data classes, have a look at PEP 557 as well as the discussions in the original GitHub repo.

In addition, Raymond Hettinger’s PyCon 2018 talk Dataclasses: The code generator to end all code generators is well worth watching.

If you do not yet have Python 3.7, there is also a data classes backport for Python 3.6. And now, go forth and write less code!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python 3's f-Strings: An Improved String Formatting Syntax (Guide)

2018-05-14T14:00:00+00:00

As of Python 3.6, f-strings are a great new way to format strings. Not only are they more readable, more concise, and less prone to error than other ways of formatting, but they are also faster!

By the end of this article, you will learn how and why to start using f-strings today.

But first, here’s what life was like before f-strings, back when you had to walk to school uphill both ways in the snow.

Free PDF Download: Python 3 Cheat Sheet

“Old-school” String Formatting in Python

Before Python 3.6, you had two main ways of embedding Python expressions inside string literals for formatting: %-formatting and str.format(). You’re about to see how to use them and what their limitations are.

Option #1: %-formatting

This is the OG of Python formatting and has been in the language since the very beginning. You can read more in the Python docs. Keep in mind that %-formatting is not recommended by the docs, which contain the following note:

“The formatting operations described here exhibit a variety of quirks that lead to a number of common errors (such as failing to display tuples and dictionaries correctly).

Using the newer formatted string literals or the str.format() interface helps avoid these errors. These alternatives also provide more powerful, flexible and extensible approaches to formatting text.” (Source)

How to Use %-formatting

String objects have a built-in operation using the % operator, which you can use to format strings. Here’s what that looks like in practice:

>>> name = "Eric"
>>> "Hello, %s." % name
'Hello, Eric.'

In order to insert more than one variable, you must use a tuple of those variables. Here’s how you would do that:

>>> name = "Eric"
>>> age = 74
>>> "Hello, %s. You are %s." % (name, age)
'Hello Eric. You are 74.'

Why %-formatting Isn’t Great

The code examples that you just saw above are readable enough. However, once you start using several parameters and longer strings, your code will quickly become much less easily readable. Things are starting to look a little messy already:

>>> first_name = "Eric"
>>> last_name = "Idle"
>>> age = 74
>>> profession = "comedian"
>>> affiliation = "Monty Python"
>>> "Hello, %s %s. You are %s. You are a %s. You were a member of %s." % (first_name, last_name, age, profession, affiliation)
'Hello, Eric Idle. You are 74. You are a comedian. You were a member of Monty Python.'

Unfortunately, this kind of formatting isn’t great because it is verbose and leads to errors, like not displaying tuples or dictionaries correctly. Fortunately, there are brighter days ahead.

Option #2: str.format()

This newer way of getting the job done was introduced in Python 2.6. You can check out the Python docs for more info.

How To Use str.format()

str.format() is an improvement on %-formatting. It uses normal function call syntax and is extensible through the __format__() method on the object being converted to a string.

With str.format(), the replacement fields are marked by curly braces:

>>> "Hello, {}. You are {}.".format(name, age)
'Hello, Eric. You are 74.'

You can reference variables in any order by referencing their index:

>>> "Hello, {1}. You are {0}.".format(age, name)
'Hello, Eric. You are 74.'

But if you insert the variable names, you get the added perk of being able to pass objects and then reference parameters and methods in between the braces:

>>> person = {'name': 'Eric', 'age': 74}
>>> "Hello, {name}. You are {age}.".format(name=person['name'], age=person['age'])
'Hello, Eric. You are 74.'

You can also use ** to do this neat trick with dictionaries:

>>> person = {'name': 'Eric', 'age': 74}
>>> "Hello, {name}. You are {age}.".format(**person)
'Hello, Eric. You are 74.'

str.format() is definitely an upgrade when compared with %-formatting, but it’s not all roses and sunshine.

Why str.format() Isn’t Great

Code using str.format() is much more easily readable than code using %-formatting, but str.format() can still be quite verbose when you are dealing with multiple parameters and longer strings. Take a look at this:

>>> first_name = "Eric"
>>> last_name = "Idle"
>>> age = 74
>>> profession = "comedian"
>>> affiliation = "Monty Python"
>>> print(("Hello, {first_name} {last_name}. You are {age}. " + 
>>>        "You are a {profession}. You were a member of {affiliation}.") \
>>>        .format(first_name=first_name, last_name=last_name, age=age, \
>>>                profession=profession, affiliation=affiliation))
'Hello, Eric Idle. You are 74. You are a comedian. You were a member of Monty Python.'

If you had the variables you wanted to pass to .format() in a dictionary, then you could just unpack it with .format(**some_dict) and reference the values by key in the string, but there has got to be a better way to do this.

f-Strings: A New and Improved Way to Format Strings in Python

The good news is that f-strings are here to save the day. They slice! They dice! They make julienne fries! Okay, they do none of those things, but they do make formatting easier. They joined the party in Python 3.6. You can read all about it in PEP 498, which was written by Eric V. Smith in August of 2015.

Also called “formatted string literals,” f-strings are string literals that have an f at the beginning and curly braces containing expressions that will be replaced with their values. The expressions are evaluated at runtime and then formatted using the __format__ protocol. As always, the Python docs are your friend when you want to learn more.

Here are some of the ways f-strings can make your life easier.

Simple Syntax

The syntax is similar to the one you used with str.format() but less verbose. Look at how easily readable this is:

>>> name = "Eric"
>>> age = 74
>>> f"Hello, {name}. You are {age}."
'Hello, Eric. You are 74.'

It would also be valid to use a capital letter F:

>>> F"Hello, {name}. You are {age}."
'Hello, Eric. You are 74.'

Do you love f-strings yet? I hope that, by the end of this article, you’ll answer >>> F"{Yes!}".

Arbitrary Expressions

Because f-strings are evaluated at runtime, you can put any and all valid Python expressions in them. This allows you to do some nifty things.

You could do something pretty straightforward, like this:

>>> f"{2 * 37}"
'74'

But you could also call functions. Here’s an example:

>>> def to_lowercase(input):
...     return input.lower()

>>> name = "Eric Idle"
>>> f"{to_lowercase(name)} is funny."
'eric idle is funny.'

You also have the option of calling a method directly:

>>> f"{name.lower()} is funny."
'eric idle is funny.'

You could even use objects created from classes with f-strings. Imagine you had the following class:

class Comedian:
    def __init__(self, first_name, last_name, age):
        self.first_name = first_name
        self.last_name = last_name
        self.age = age

    def __str__(self):
        return f"{self.first_name} {self.last_name} is {self.age}."

    def __repr__(self):
        return f"{self.first_name} {self.last_name} is {self.age}. Surprise!"

You’d be able to do this:

>>> new_comedian = Comedian("Eric", "Idle", "74")
>>> f"{new_comedian}"
'Eric Idle is 74.'

The __str__() and __repr__() methods deal with how objects are presented as strings, so you’ll need to make sure you include at least one of those methods in your class definition. If you have to pick one, go with __repr__() because it can be used in place of __str__().

The string returned by __str__() is the informal string representation of an object and should be readable. The string returned by __repr__() is the official representation and should be unambiguous. Calling str() and repr() is preferable to using __str__() and __repr__() directly.

By default, f-strings will use __str__(), but you can make sure they use __repr__() if you include the conversion flag !r:

>>> f"{new_comedian}"
'Eric Idle is 74.'
>>> f"{new_comedian!r}"
'Eric Idle is 74. Surprise!'

If you’d like to read some of the conversation that resulted in f-strings supporting full Python expressions, you can do so here.

Multiline f-strings

You can have multiline strings:

>>> name = "Eric"
>>> profession = "comedian"
>>> affiliation = "Monty Python"
>>> message = (
...     f"Hi {name}. "
...     f"You are a {profession}. "
...     f"You were in {affiliation}."
... )
>>> message
'Hi Eric. You are a comedian. You were in Monty Python.'

But remember that you need to place an f in front of each line of a multiline string. The following code won’t work:

>>> message = (
...     f"Hi {name}. "
...     "You are a {profession}. "
...     "You were in {affiliation}."
... )
>>> message
'Hi Eric. You are a {profession}. You were in {affiliation}.'

If you don’t put an f in front of each individual line, then you’ll just have regular, old, garden-variety strings and not shiny, new, fancy f-strings.

If you want to spread strings over multiple lines, you also have the option of escaping a return with a \:

>>> message = f"Hi {name}. " \
...           f"You are a {profession}. " \
...           f"You were in {affiliation}."
...
>>> message
'Hi Eric. You are a comedian. You were in Monty Python.'

But this is what will happen if you use """:

>>> message = f"""
...     Hi {name}. 
...     You are a {profession}. 
...     You were in {affiliation}.
... """
...
>>> message
'\n    Hi Eric.\n    You are a comedian.\n    You were in Monty Python.\n'

Read up on indentation guidelines in PEP 8.

Speed

The f in f-strings may as well stand for “fast.”

f-strings are faster than both %-formatting and str.format(). As you already saw, f-strings are expressions evaluated at runtime rather than constant values. Here’s an excerpt from the docs:

“F-strings provide a way to embed expressions inside string literals, using a minimal syntax. It should be noted that an f-string is really an expression evaluated at run time, not a constant value. In Python source code, an f-string is a literal string, prefixed with f, which contains expressions inside braces. The expressions are replaced with their values.” (Source)

At runtime, the expression inside the curly braces is evaluated in its own scope and then put together with the string literal part of the f-string. The resulting string is then returned. That’s all it takes.

Here’s a speed comparison:

>>> import timeit
>>> timeit.timeit("""name = "Eric"
age = 74
'%s is %s.' % (name, age)""", number = 10000)
0.003324444866599663

>>> timeit.timeit("""name = "Eric"
age = 74
'{} is {}.'.format(name, age)""", number = 10000)
0.004242089427570761

>>> timeit.timeit("""name = "Eric"
age = 74
f'{name} is {age}.'""", number = 10000)
0.0024820892040722242

As you can see, f-strings come out on top.

However, that wasn’t always the case. When they were first implemented, they had some speed issues and needed to be made faster than str.format(). A special BUILD_STRING opcode was introduced.

Python f-Strings: The Pesky Details

Now that you’ve learned all about why f-strings are great, I’m sure you want to get out there and start using them. Here are a few details to keep in mind as you venture off into this brave new world.

Quotation Marks

You can use various types of quotation marks inside the expressions. Just make sure you are not using the same type of quotation mark on the outside of the f-string as you are using in the expression.

This code will work:

>>> f"{'Eric Idle'}"
'Eric Idle'

This code will also work:

>>> f'{"Eric Idle"}'
'Eric Idle'

You can also use triple quotes:

>>> f"""Eric Idle"""
'Eric Idle'

>>> f'''Eric Idle'''
'Eric Idle'

If you find you need to use the same type of quotation mark on both the inside and the outside of the string, then you can escape with \:

>>> f"The \"comedian\" is {name}, aged {age}."
'The "comedian" is Eric Idle, aged 74.'

Dictionaries

Speaking of quotation marks, watch out when you are working with dictionaries. If you are going to use single quotation marks for the keys of the dictionary, then remember to make sure you’re using double quotation marks for the f-strings containing the keys.

This will work:

>>> comedian = {'name': 'Eric Idle', 'age': 74}
>>> f"The comedian is {comedian['name']}, aged {comedian['age']}."
The comedian is Eric Idle, aged 74.

But this will be a hot mess with a syntax error:

>>> comedian = {'name': 'Eric Idle', 'age': 74}
>>> f'The comedian is {comedian['name']}, aged {comedian['age']}.'
  File "<stdin>", line 1
    f'The comedian is {comedian['name']}, aged {comedian['age']}.'
                                    ^
SyntaxError: invalid syntax

If you use the same type of quotation mark around the dictionary keys as you do on the outside of the f-string, then the quotation mark at the beginning the first dictionary key will be interpreted as the end of the string.

Braces

In order to make a brace appear in your string, you must use double braces:

>>> f"{{74}}"
'{ 74 }'

Note that using triple braces will result in there being only single braces in your string:

>>> f"{{{74}}}"
'{ 74 }'

However, you can get more braces to show if you use more than triple braces:

>>> f"{{{{74}}}}"
'{{74}}'

Backslashes

As you saw earlier, it is possible for you to use backslash escapes in the string portion of an f-string. However, you can’t use backslashes to escape in the expression part of an f-string:

>>> f"{\"Eric Idle\"}"
  File "<stdin>", line 1
    f"{\"Eric Idle\"}"
                      ^
SyntaxError: f-string expression part cannot include a backslash

You can work around this by evaluating the expression beforehand and using the result in the f-string:

>>> name = "Eric Idle"
>>> f"{name}"
'Eric Idle'

Inline Comments

Expressions should not include comments using the # symbol. You’ll get a syntax error:

>>> f"Eric is {2 * 37 #Oh my!}."
  File "<stdin>", line 1
    f"Eric is {2 * 37 #Oh my!}."
                                ^
SyntaxError: f-string expression part cannot include '#'

Go Forth and Format!

You can still use the older ways of formatting strings, but with f-strings, you now have a more concise, readable, and convenient way that is both faster and less prone to error. Simplifying your life by using f-strings is a great reason to start using Python 3.6 if you haven’t already made the switch. (If you are still using Python 2, don’t forget that 2020 will be here soon!)

According to the Zen of Python, when you need to decide how to do something, then “[t]here should be one– and preferably only one –obvious way to do it.” Although f-strings aren’t the only possible way for you to format strings, they are in a great position to become that one obvious way to get the job done.

Operator and Function Overloading in Custom Python Classes

2018-05-08T14:00:00+00:00

If you’ve used the + or * operator on a str object in Python, you must have noticed its different behavior when compared to int or float objects:

>>> # Adds the two numbers
>>> 1 + 2
3

>>> # Concatenates the two strings
>>> 'Real' + 'Python'
'RealPython'


>>> # Gives the product
>>> 3 * 2
6

>>> # Repeats the string
>>> 'Python' * 3
'PythonPythonPython'

You might have wondered how the same built-in operator or function shows different behavior for objects of different classes. This is called operator overloading or function overloading respectively. This article will help you understand this mechanism, so that you can do the same in your own Python classes and make your objects more Pythonic.

You’ll learn the following:

The API that handles operators and built-ins in Python
The “secret” behind len() and other built-ins
How to make your classes capable of using operators
How to make your classes compatible with Python’s built-in functions

Free Bonus: Click here to get access to a free Python OOP Cheat Sheet that points you to the best tutorials, videos, and books to learn more about Object-Oriented Programming with Python.

As a bonus, you’ll also see an example class, objects of which will be compatible with many of these operators and functions. Let’s get started!

The Python Data Model

Say you have a class representing an online order having a cart (a list) and a customer (a str or instance of another class which represents a customer).

Note: If you need a refresher on OOP in Python, check out this tutorial on Real Python: Object-Oriented Programming (OOP) in Python 3

In such a case, it is quite natural to want to obtain the length of the cart list. Someone new to Python might decide to implement a method called get_cart_len() in their class to do this. But you can configure the built-in len() in such a way that it returns the length of the cart list when given our object.

In another case, we might want to append something to the cart. Again, someone new to Python would think of implementing a method called append_to_cart() that takes an item and appends it to the cart list. But you can configure the + operator in such a way that it appends a new item to the cart.

Python does all this using special methods. These special methods have a naming convention, where the name starts with two underscores, followed by an identifier and ends with another pair of underscores.

Essentially, each built-in function or operator has a special method corresponding to it. For example, there’s __len__(), corresponding to len(), and __add__(), corresponding to the + operator.

By default, most of the built-ins and operators will not work with objects of your classes. You must add the corresponding special methods in your class definition to make your object compatible with built-ins and operators.

When you do this, the behavior of the function or operator associated with it changes according to that defined in the method.

This is exactly what the Data Model (Section 3 of the Python documentation) helps you accomplish. It lists all the special methods available and provides you with the means of overloading built-in functions and operators so that you can use them on your own objects.

Let’s see what this means.

Fun fact: Due to the naming convention used for these methods, they are also called dunder methods which is a shorthand for double underscore methods. Sometimes they’re also referred to as special methods or magic methods. We prefer dunder methods though!

The Internals of Operations Like `len()` and `[]`

Every class in Python defines its own behavior for built-in functions and methods. When you pass an instance of some class to a built-in function or use an operator on the instance, it is actually equivalent to calling a special method with relevant arguments.

If there is a built-in function, func(), and the corresponding special method for the function is __func__(), Python interprets a call to the function as obj.__func__(), where obj is the object. In the case of operators, if you have an operator opr and the corresponding special method for it is __opr__(), Python interprets something like obj1 <opr> obj2 as obj1.__opr__(obj2).

So, when you’re calling len() on an object, Python handles the call as obj.__len__(). When you use the [] operator on an iterable to obtain the value at an index, Python handles it as itr.__getitem__(index), where itr is the iterable object and index is the index you want to obtain.

Therefore, when you define these special methods in your own class, you override the behavior of the function or operator associated with them because, behind the scenes, Python is calling your method. Let’s get a better understanding of this:

>>> a = 'Real Python'
>>> b = ['Real', 'Python']
>>> len(a)
11
>>> a.__len__()
11
>>> b[0]
'Real'
>>> b.__getitem__(0)
'Real'

As you can see, when you use the function or its corresponding special method, you get the same result. In fact, when you obtain the list of attributes and methods of a str object using dir(), you’ll see these special methods in the list in addition to the usual methods available on str objects:

>>> dir(a)
['__add__',
 '__class__',
 '__contains__',
 '__delattr__',
 '__dir__',
 ...,
 '__iter__',
 '__le__',
 '__len__',
 '__lt__',
 ...,
 'swapcase',
 'title',
 'translate',
 'upper',
 'zfill']

If the behavior of a built-in function or operator is not defined in the class by the special method, then you will get a TypeError.

So, how can you use special methods in your classes?

Overloading Built-in Functions

Many of the special methods defined in the Data Model can be used to change the behavior of functions such as len, abs, hash, divmod, and so on. To do this, you only need to define the corresponding special method in your class. Let’s look at a few examples:

Giving a Length to Your Objects Using `len()`

To change the behavior of len(), you need to define the __len__() special method in your class. Whenever you pass an object of your class to len(), your custom definition of __len__() will be used to obtain the result. Let’s implement len() for the order class we talked about in the beginning:

>>> class Order:
...     def __init__(self, cart, customer):
...         self.cart = list(cart)
...         self.customer = customer
...
...     def __len__(self):
...         return len(self.cart)
...
>>> order = Order(['banana', 'apple', 'mango'], 'Real Python')
>>> len(order)
3

As you can see, you can now use len() to directly obtain the length of the cart. Moreover, it makes more intuitive sense to say “length of order” rather than calling something like order.get_cart_len(). Your call is both Pythonic and more intuitive. When you don’t have the __len__() method defined but still call len() on your object, you get a TypeError:

>>> class Order:
...     def __init__(self, cart, customer):
...         self.cart = list(cart)
...         self.customer = customer
...
>>> order = Order(['banana', 'apple', 'mango'], 'Real Python')
>>> len(order)  # Calling len when no __len__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object of type 'Order' has no len()

But, when overloading len(), you should keep in mind that Python requires the function to return an integer. If your method were to return anything other than an integer, you would get a TypeError. This, most probably, is to keep it consistent with the fact that len() is generally used to obtain the length of a sequence, which can only be an integer:

>>> class Order:
...     def __init__(self, cart, customer):
...         self.cart = list(cart)
...         self.customer = customer
...
...     def __len__(self):
...         return float(len(self.cart))  # Return type changed to float
...
>>> order = Order(['banana', 'apple', 'mango'], 'Real Python')
>>> len(order)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'float' object cannot be interpreted as an integer

Making Your Objects Work With `abs()`

You can dictate the behavior of the abs() built-in for instances of your class by defining the __abs__() special method in the class. There are no restrictions on the return value of abs(), and you get a TypeError when the special method is absent in your class definition.

In a class representing a vector in a two-dimensional space, abs() can be used to get the length of the vector. Let’s see it in action:

>>> class Vector:
...     def __init__(self, x_comp, y_comp):
...         self.x_comp = x_comp
...         self.y_comp = y_comp
...
...     def __abs__(self):
...         return (x * x + y * y) ** 0.5
...
>>> vector = Vector(3, 4)
>>> abs(vector)
5.0

It makes more intuitive sense to say “absolute value of vector” rather than calling something like vector.get_mag().

Printing Your Objects Prettily Using `str()`

The str() built-in is used to cast an instance of a class to a str object, or more appropriately, to obtain a user-friendly string representation of the object which can be read by a normal user rather than the programmer. You can define the string format your object should be displayed in when passed to str() by defining the __str__() method in your class. Moreover, __str__() is the method that is used by Python when you call print() on your object.

Let’s implement this in the Vector class to format Vector objects as xi+yj. A negative y-component will be handled using the format mini-language:

>>> class Vector:
...     def __init__(self, x_comp, y_comp):
...         self.x_comp = x_comp
...         self.y_comp = y_comp
...
...     def __str__(self):
...         # By default, sign of +ve number is not displayed
...         # Using `+`, sign is always displayed
...         return f'{self.x_comp}i{self.y_comp:+}j'
...
>>> vector = Vector(3, 4)
>>> str(vector)
'3i+4j'
>>> print(vector)
3i+4j

It is necessary that __str__() returns a str object, and we get a TypeError if the return type is non-string.

Representing Your Objects Using `repr()`

The repr() built-in is used to obtain the parsable string representation of an object. If an object is parsable, that means that Python should be able to recreate the object from the representation when repr is used in conjunction with functions like eval(). To define the behavior of repr(), you can use the __repr__() special method.

This is also the method Python uses to display the object in a REPL session. If the __repr__() method is not defined, you will get something like <__main__.Vector object at 0x...> trying to look at the object in the REPL session. Let’s see it in action in the Vector class:

>>> class Vector:
...     def __init__(self, x_comp, y_comp):
...         self.x_comp = x_comp
...         self.y_comp = y_comp
...
...     def __repr__(self):
...         return f'Vector({self.x_comp}, {self.y_comp})'
...

>>> vector = Vector(3, 4)
>>> repr(vector)
'Vector(3, 4)'

>>> b = eval(repr(vector))
>>> type(b), b.x_comp, b.y_comp
(__main__.Vector, 3, 4)

>>> vector  # Looking at object; __repr__ used
'Vector(3, 4)'

Note: In cases where the __str__() method is not defined, Python uses the __repr__() method to print the object, as well as to represent the object when str() is called on it. If both the methods are missing, it defaults to <__main__.Vector ...>. But __repr__() is the only method that is used to display the object in an interactive session. Absence of it in the class yields <__main__.Vector ...>.

Also, while this distinction between __str__() and __repr__() is the recommended behavior, many of the popular libraries ignore this distinction and use the two methods interchangeably.

Here’s a recommended article on __repr__() and __str__() by our very own Dan Bader: Python String Conversion 101: Why Every Class Needs a “repr”.

Making Your Objects Truthy or Falsey Using `bool()`

The bool() built-in can be used to obtain the truth value of an object. To define its behavior, you can use the __bool__() (__nonzero__() in Python 2.x) special method.

The behavior defined here will determine the truth value of an instance in all contexts that require obtaining a truth value such as in if statements.

As an example, for the Order class that was defined above, an instance can be considered to be truthy if the length of the cart list is non-zero. This can be used to check whether an order should be processed or not:

>>> class Order:
...     def __init__(self, cart, customer):
...         self.cart = list(cart)
...         self.customer = customer
...
...     def __bool__(self):
...         return len(self.cart) > 0
...
>>> order1 = Order(['banana', 'apple', 'mango'], 'Real Python')
>>> order2 = Order([], 'Python')

>>> bool(order1)
True
>>> bool(order2)
False

>>> for order in [order1, order2]:
...     if order:
...         print(f"{order.customer}'s order is processing...")
...     else:
...         print(f"Empty order for customer {order.customer}")
Real Python's order is processing...
Empty order for customer Python

Note: When the __bool__() special method is not implemented in a class, the value returned by __len__() is used as the truth value, where a non-zero value indicates True and a zero value indicates False. In case both the methods are not implemented, all instances of the class are considered to be True.

There are many more special methods that overload built-in functions. You can find them in the documentation. Having discussed some of them, let’s move to operators.

Overloading Built-in Operators

Changing the behavior of operators is just as simple as changing the behavior of functions. You define their corresponding special methods in your class, and the operators work according to the behavior defined in these methods.

These are different from the above special methods in the sense that they need to accept another argument in the definition other than self, generally referred to by the name other. Let’s look at a few examples.

Making Your Objects Capable of Being Added Using `+`

The special method corresponding to the + operator is the __add__() method. Adding a custom definition of __add__() changes the behavior of the operator. It is recommended that __add__() returns a new instance of the class instead of modifying the calling instance itself. You’ll see this behavior quite commonly in Python:

>>> a = 'Real'
>>> a + 'Python'  # Gives new str instance
'RealPython'
>>> a  # Values unchanged
'Real'
>>> a = a + 'Python'  # Creates new instance and assigns a to it
>>> a
'RealPython'

You can see above that using the + operator on a str object actually returns a new str instance, keeping the value of the calling instance (a) unmodified. To change it, we need to explicitly assign the new instance to a.

Let’s implement the ability to append new items to our cart in the Order class using the operator. We’ll follow the recommended practice and make the operator return a new Order instance that has our required changes instead of making the changes directly to our instance:

>>> class Order:
...     def __init__(self, cart, customer):
...         self.cart = list(cart)
...         self.customer = customer
...
...     def __add__(self, other):
...         new_cart = self.cart.copy()
...         new_cart.append(other)
...         return Order(new_cart, self.customer)
...
>>> order = Order(['banana', 'apple'], 'Real Python')

>>> (order + 'orange').cart  # New Order instance
['banana', 'apple', 'mango']
>>> order.cart  # Original instance unchanged
['banana', 'apple']

>>> order = order + 'mango'  # Changing the original instance
>>> order.cart
['banana', 'apple', 'mango']

Similarly, you have the __sub__(), __mul__(), and other special methods which define the behavior of -, *, and so on. These methods should return a new instance of the class as well.

Shortcuts: the `+=` Operator

The += operator stands as a shortcut to the expression obj1 = obj1 + obj2. The special method corresponding to it is __iadd__(). The __iadd__() method should make changes directly to the self argument and return the result, which may or may not be self. This behavior is quite different from __add__() since the latter creates a new object and returns that, as you saw above.

Roughly, any += use on two objects is equivalent to this:

>>> result = obj1 + obj2
>>> obj1 = result

Here, result is the value returned by __iadd__(). The second assignment is taken care of automatically by Python, meaning that you do not need to explicitly assign obj1 to the result as in the case of obj1 = obj1 + obj2.

Let’s make this possible for the Order class so that new items can be appended to the cart using +=:

>>> class Order:
...     def __init__(self, cart, customer):
...         self.cart = list(cart)
...         self.customer = customer
...
...     def __iadd__(self, other):
...         self.cart.append(other)
...         return self
...
>>> order = Order(['banana', 'apple'], 'Real Python')
>>> order += 'mango'
>>> order.cart
['banana', 'apple', 'mango']

As can be seen, any change is made directly to self and it is then returned. What happens when you return some random value, like a string or an integer?

>>> class Order:
...     def __init__(self, cart, customer):
...         self.cart = list(cart)
...         self.customer = customer
...
...     def __iadd__(self, other):
...         self.cart.append(other)
...         return 'Hey, I am string!'
...
>>> order = Order(['banana', 'apple'], 'Real Python')
>>> order += 'mango'
>>> order
'Hey, I am string!'

Even though the relevant item was appended to the cart, the value of order changed to what was returned by __iadd__(). Python implicitly handled the assignment for you. This can lead to surprising behavior if you forget to return something in your implementation:

>>> class Order:
...     def __init__(self, cart, customer):
...         self.cart = list(cart)
...         self.customer = customer
...
...     def __iadd__(self, other):
...         self.cart.append(other)
...
>>> order = Order(['banana', 'apple'], 'Real Python')
>>> order += 'mango'
>>> order  # No output
>>> type(order)
NoneType

Since all Python functions (or methods) return None implicitly, order is reassigned to None and the REPL session doesn’t show any output when order is inspected. Looking at the type of order, you see that it is now NoneType. Therefore, always make sure that you’re returning something in your implementation of __iadd__() and that it is the result of the operation and not anything else.

Similar to __iadd__(), you have __isub__(), __imul__(), __idiv__() and other special methods which define the behavior of -=, *=, /=, and others alike.

Note: When __iadd__() or its friends are missing from your class definition but you still use their operators on your objects, Python uses __add__() and its friends to get the result of the operation and assigns that to the calling instance. Generally speaking, it is safe to not implement __iadd__() and its friends in your classes as long as __add__() and its friends work properly (return something which is the result of the operation).

The Python documentation has a good explanation of these methods. Also, take a look at this example which shows the caveats involved with += and the others when working with immutable types.

Indexing and Slicing Your Objects Using `[]`

The [] operator is called the indexing operator and is used in various contexts in Python such as getting the value at an index in sequences, getting the value associated with a key in dictionaries, or obtaining a part of a sequence through slicing. You can change its behavior using the __getitem__() special method.

Let’s configure our Order class so that we can directly use the object and obtain an item from the cart:

>>> class Order:
...     def __init__(self, cart, customer):
...         self.cart = list(cart)
...         self.customer = customer
...
...     def __getitem__(self, key):
...         return self.cart[key]
...
>>> order = Order(['banana', 'apple'], 'Real Python')
>>> order[0]
'banana'
>>> order[-1]
'apple'

You’ll notice that above, the name of the argument to __getitem__() is not index but key. This is because the argument can be of mainly three forms: an integer value, in which case it is either an index or a dictionary key, a string value, in which case it is a dictionary key, and a slice object, in which case it will slice the sequence used by the class. While there are other possibilities, these are the ones most commonly encountered.

Since our internal data structure is a list, we can use the [] operator to slice the list, as in this case, the key argument will be a slice object. This is one of the biggest advantages of having a __getitem__() definition in your class. As long as you’re using data structures that support slicing (lists, tuples, strings, and so on), you can configure your objects to directly slice the structure:

>>> order[1:]
['apple']
>>> order[::-1]
['apple', 'banana']

Note: There is a similar __setitem__() special method that is used to define the behavior of obj[x] = y. This method takes two arguments in addition to self, generally called key and value, and can be used to change the value at key to value.

Reverse Operators: Making Your Classes Mathematically Correct

While defining the __add__(), __sub__(), __mul__(), and similar special methods allows you to use the operators when your class instance is the left-hand side operand, the operator will not work if the class instance is the right-hand side operand:

>>> class Mock:
...     def __init__(self, num):
...         self.num = num
...     def __add__(self, other):
...         return Mock(self.num + other)
...
>>> mock = Mock(5)
>>> mock = mock + 6
>>> mock.num
11

>>> mock = 6 + Mock(5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'Mock'

If your class represents a mathematical entity like a vector, a coordinate, or a complex number, applying the operators should work in both the cases since it is a valid mathematical operation.

Moreover, if the operators work only when the instance is the left operand, we are violating the fundamental principle of commutativity in many cases. Therefore, to help you make your classes mathematically correct, Python provides you with reverse special methods such as __radd__(), __rsub__(), __rmul__(), and so on.

These handle calls such as x + obj, x - obj, and x * obj, where x is not an instance of the concerned class. Just like __add__() and the others, these reverse special methods should return a new instance of class with the changes of the operation rather than modifying the calling instance itself.

Let’s configure __radd__() in the Order class in such a way that it will append something at the front of the cart. This can be used in cases where the cart is organized in terms of the priority of the orders:

>>> class Order:
...     def __init__(self, cart, customer):
...         self.cart = list(cart)
...         self.customer = customer
...
...     def __add__(self, other):
...         new_cart = self.cart.copy()
...         new_cart.append(other)
...         return Order(new_cart, self.customer)
...
...     def __radd__(self, other):
...         new_cart = self.cart.copy()
...         new_cart.insert(0, other)
...         return Order(new_cart, self.customer)
...
>>> order = Order(['banana', 'apple'], 'Real Python')

>>> order = order + 'orange'
>>> order.cart
['banana', 'apple', 'orange']

>>> order = 'mango' + order
>>> order.cart
['mango', 'banana', 'apple', 'orange']

A Complete Example

To drive all these points home, it’s better to look at an example class which implements these operators together.

Let’s reinvent the wheel and implement our own class to represent complex numbers, CustomComplex. Objects of our class will support a variety of built-in functions and operators, making them behave very similar to the built-in complex numbers class:

from math import hypot, atan, sin, cos

class CustomComplex:
    def __init__(self, real, imag):
        self.real = real
        self.imag = imag

The constructor handles only one kind of call, CustomComplex(a, b). It takes positional arguments, representing the real and imaginary parts of the complex number.

Let’s define two methods inside the class, conjugate() and argz(), which will give us the complex conjugate and the argument of a complex number respectively:

def conjugate(self):
    return self.__class__(self.real, -self.imag)

def argz(self):
    return atan(self.imag / self.real)

Note: __class__ is not a special method but a class attribute which is present by default. It has a reference to the class. By using it here, we are obtaining that and then calling the constructor in the usual manner. In other words, this is equivalent to CustomComplex(real, imag). This is done here to avoid refactoring the code if the name of the class changes someday.

Next, we configure abs() to return the modulus of a complex number:

def __abs__(self):
    return hypot(self.real, self.imag)

We will follow the recommended distinction between __repr__() and __str__() and use the first for the parsable string representation and the second for a “pretty” representation.

The __repr__() method will simply return CustomComplex(a, b) in a string so that we can call eval() to recreate the object, while the __str__() method will return the complex number in brackets, as (a+bj):

def __repr__(self):
    return f"{self.__class__.__name__}({self.real}, {self.imag})"

def __str__(self):
    return f"({self.real}{self.imag:+}j)"

Mathematically, it is possible to add any two complex numbers or add a real number to a complex number. Let’s configure the + operator in such a way that it works for both cases.

The method will check the type of the right-hand side operator. In case it is an int or a float, it will increment only the real part (since any real number, a, is equivalent to a+0j), while in the case of another complex number, it will change both the parts:

def __add__(self, other):
    if isinstance(other, float) or isinstance(other, int):
        real_part = self.real + other
        imag_part = self.imag

    if isinstance(other, CustomComplex):
        real_part = self.real + other.real
        imag_part = self.imag + other.imag

    return self.__class__(real_part, imag_part)

Similarly, we define the behavior for - and *:

def __sub__(self, other):
    if isinstance(other, float) or isinstance(other, int):
        real_part = self.real - other
        imag_part = self.imag

    if isinstance(other, CustomComplex):
        real_part = self.real - other.real
        imag_part = self.imag - other.imag

    return self.__class__(real_part, imag_part)

def __mul__(self, other):
    if isinstance(other, int) or isinstance(other, float):
        real_part = self.real * other
        imag_part = self.imag * other

    if isinstance(other, CustomComplex):
        real_part = (self.real * other.real) - (self.imag * other.imag)
        imag_part = (self.real * other.imag) + (self.imag * other.real)

    return self.__class__(real_part, imag_part)

Since both addition and multiplication are commutative, we can define their reverse operators by calling __add__() and __mul__() in __radd__() and __rmul__() respectively. On the other hand, the behavior of __rsub__() needs to be defined since subtraction is not commutative:

def __radd__(self, other):
    return self.__add__(other)

def __rmul__(self, other):
    return self.__mul__(other)

def __rsub__(self, other):
    # x - y != y - x
    if isinstance(other, float) or isinstance(other, int):
        real_part = other - self.real
        imag_part = -self.imag

    return self.__class__(real_part, imag_part)

Note: You might have noticed that we didn’t add a construct to handle a CustomComplex instance here. This is because, in such a case, both the operands are instances of our class, and __rsub__() won’t be responsible for handling the operation. Instead, __sub__() will be called. This is a subtle but important detail.

Now, we take care of the two operators, == and !=. The special methods used for them are __eq__() and __ne__(), respectively. Two complex numbers are said to be equal if their corresponding real and imaginary parts are both equal. They are said to be unequal when either one of these are unequal:

def __eq__(self, other):
    # Note: generally, floats should not be compared directly
    # due to floating-point precision
    return (self.real == other.real) and (self.imag == other.imag)

def __ne__(self, other):
    return (self.real != other.real) or (self.imag != other.imag)

Note: The Floating-Point Guide is an article that talks about comparing floats and floating-point precision. It highlights the caveats involved in comparing floats directly, which is something we’re doing here.

It is also possible to raise a complex number to any power using a simple formula. We configure the behavior for both the built-in pow() and the ** operator using the __pow__() special method:

def __pow__(self, other):
    r_raised = abs(self) ** other
    argz_multiplied = self.argz() * other

    real_part = round(r_raised * cos(argz_multiplied))
    imag_part = round(r_raised * sin(argz_multiplied))

    return self.__class__(real_part, imag_part)

Note: Take a close look at the definition of the method. We are calling abs() to obtain the modulus of the complex number. So, once you’ve defined the special method for a particular function or operator in your class, it can be used in other methods of the same class.

Let’s create two instances of this class, one having a positive imaginary part and one having a negative imaginary part:

>>> a = CustomComplex(1, 2)
>>> b = CustomComplex(3, -4)

String representations:

>>> a
CustomComplex(1, 2)
>>> b
CustomComplex(3, -4)
>>> print(a)
(1+2j)
>>> print(b)
(3-4j)

Recreating the object using eval() with repr():

>>> b_copy = eval(repr(b))
>>> type(b_copy), b_copy.real, b_copy.imag
(__main__.CustomComplex, 3, -4)

Addition, subtraction, and multiplication:

>>> a + b
CustomComplex(4, -2)
>>> a - b
CustomComplex(-2, 6)
>>> a + 5
CustomComplex(6, 2)
>>> 3 - a
CustomComplex(2, -2)
>>> a * 6
CustomComplex(6, 12)
>>> a * (-6)
CustomComplex(-6, -12)

Equality and inequality checks:

>>> a == CustomComplex(1, 2)
True
>>> a ==  b
False
>>> a != b
True
>>> a != CustomComplex(1, 2)
False

Finally, raising a complex number to some power:

>>> a ** 2
CustomComplex(-3, 4)
>>> b ** 5
CustomComplex(-237, 3116)

As you can see, objects of our custom class behave and look like those of a built-in class and are very Pythonic. The full example code for this class is embedded below.

from math import hypot, atan, sin, cos

class CustomComplex():
    """
    A class to represent a complex number, a+bj.
    Attributes:
        real - int, representing the real part
        imag - int, representing the imaginary part

    Implements the following:

    * Addition with a complex number or a real number using `+`
    * Multiplication with a complex number or a real number using `*`
    * Subtraction of a complex number or a real number using `-`
    * Calculation of absolute value using `abs`
    * Raise complex number to a power using `**`
    * Nice string representation using `__repr__`
    * Nice user-end viewing using `__str__`

    Notes:
        * The constructor has been intentionally kept simple
        * It is configured to support one kind of call:
            CustomComplex(a, b)
        * Error handling was avoided to keep things simple
    """

    def __init__(self, real, imag):
        """
        Initializes a complex number, setting real and imag part
        Arguments:
            real: Number, real part of the complex number
            imag: Number, imaginary part of the complex number
        """
        self.real = real
        self.imag = imag

    def conjugate(self):
        """
        Returns the complex conjugate of a complex number
        Return:
            CustomComplex instance
        """
        return CustomComplex(self.real, -self.imag)

    def argz(self):
        """
        Returns the argument of a complex number
        The argument is given by:
            atan(imag_part/real_part)
        Return:
            float
        """
        return atan(self.imag / self.real)

    def __abs__(self):
        """
        Returns the modulus of a complex number
        Return:
            float
        """
        return hypot(self.real, self.imag)

    def __repr__(self):
        """
        Returns str representation of an instance of the 
        class. Can be used with eval() to get another 
        instance of the class
        Return:
            str
        """
        return f"CustomComplex({self.real}, {self.imag})"


    def __str__(self):
        """
        Returns user-friendly str representation of an instance 
        of the class
        Return:
            str
        """
        return f"({self.real}{self.imag:+}j)"

    def __add__(self, other):
        """
        Returns the addition of a complex number with
        int, float or another complex number
        Return:
            CustomComplex instance
        """
        if isinstance(other, float) or isinstance(other, int):
            real_part = self.real + other
            imag_part = self.imag

        if isinstance(other, CustomComplex):
            real_part = self.real + other.real
            imag_part = self.imag + other.imag

        return CustomComplex(real_part, imag_part)

    def __sub__(self, other):
        """
        Returns the subtration from a complex number of
        int, float or another complex number
        Return:
            CustomComplex instance
        """
        if isinstance(other, float) or isinstance(other, int):
            real_part = self.real - other
            imag_part = self.imag

        if isinstance(other, CustomComplex):
            real_part = self.real - other.real
            imag_part = self.imag - other.imag

        return CustomComplex(real_part, imag_part)

    def __mul__(self, other):
        """
        Returns the multiplication of a complex number with
        int, float or another complex number
        Return:
            CustomComplex instance
        """
        if isinstance(other, int) or isinstance(other, float):
            real_part = self.real * other
            imag_part = self.imag * other

        if isinstance(other, CustomComplex):
            real_part = (self.real * other.real) - (self.imag * other.imag)
            imag_part = (self.real * other.imag) + (self.imag * other.real)

        return CustomComplex(real_part, imag_part)

    def __radd__(self, other):
        """
        Same as __add__; allows 1 + CustomComplex('x+yj')
        x + y == y + x
        """
        pass

    def __rmul__(self, other):
        """
        Same as __mul__; allows 2 * CustomComplex('x+yj')
        x * y == y * x
        """
        pass

    def __rsub__(self, other):
        """
        Returns the subtraction of a complex number from
        int or float
        x - y != y - x
        Subtration of another complex number is not handled by __rsub__
        Instead, __sub__ handles it since both sides are instances of
        this class
        Return:
            CustomComplex instance
        """
        if isinstance(other, float) or isinstance(other, int):
            real_part = other - self.real
            imag_part = -self.imag

        return CustomComplex(real_part, imag_part)

    def __eq__(self, other):
        """
        Checks equality of two complex numbers
        Two complex numbers are equal when:
            * Their real parts are equal AND
            * Their imaginary parts are equal
        Return:
            bool
        """
        # note: comparing floats directly is not a good idea in general
        # due to floating-point precision
        return (self.real == other.real) and (self.imag == other.imag)

    def __ne__(self, other):
        """
        Checks inequality of two complex numbers
        Two complex numbers are unequal when:
            * Their real parts are unequal OR
            * Their imaginary parts are unequal
        Return:
            bool
        """
        return (self.real != other.real) or (self.imag != other.imag)

    def __pow__(self, other):
        """
        Raises a complex number to a power
        Formula:
            z**n = (r**n)*[cos(n*agrz) + sin(n*argz)j], where
            z = complex number
            n = power
            r = absolute value of z
            argz = argument of z
        Return:
            CustomComplex instance
        """
        r_raised = abs(self) ** other
        argz_multiplied = self.argz() * other

        real_part = round(r_raised * cos(argz_multiplied))
        imag_part = round(r_raised * sin(argz_multiplied))

        return CustomComplex(real_part, imag_part)

Recap and Resources

In this tutorial, you learned about the Python Data Model and how the Data Model can be used to build Pythonic classes. You learned about changing the behavior of built-in functions such as len(), abs(), str(), bool(), and so on. You also learned about changing the behavior of built-in operators like +, -, *, **, and so forth.

Free Bonus: Click here to get access to a free Python OOP Cheat Sheet that points you to the best tutorials, videos, and books to learn more about Object-Oriented Programming with Python.

After reading this, you can confidently create classes that make use of the best idiomatic features of Python and make your objects Pythonic!

For more information on the Data Model, and function and operator overloading, take a look at these resources:

Section 3.3, Special Method Names of the Data Model section in the Python documentation
Fluent Python by Luciano Ramalho
Python Tricks: The Book

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Pure Python vs NumPy vs TensorFlow Performance Comparison

2018-05-07T14:00:00+00:00

Python has a design philosophy that stresses allowing programmers to express concepts readably and in fewer lines of code. This philosophy makes the language suitable for a diverse set of use cases: simple scripts for web, large web applications (like YouTube), scripting language for other platforms (like Blender and Autodesk’s Maya), and scientific applications in several areas, such as astronomy, meteorology, physics, and data science.

It is technically possible to implement scalar and matrix calculations using Python lists. However, this can be unwieldy, and performance is poor when compared to languages suited for numerical computation, such as MATLAB or Fortran, or even some general purpose languages, such as C or C++.

To circumvent this deficiency, several libraries have emerged that maintain Python’s ease of use while lending the ability to perform numerical calculations in an efficient manner. Two such libraries worth mentioning are NumPy (one of the pioneer libraries to bring efficient numerical computation to Python) and TensorFlow (a more recently rolled-out library focused more on deep learning algorithms).

NumPy provides support for large multidimensional arrays and matrices along with a collection of mathematical functions to operate on these elements. The project relies on well-known packages implemented in another languages (like Fortran) to perform efficient computations, bringing the user both the expressiveness of Python and a performance similar to MATLAB or Fortran.
TensorFlow is an open-source library for numerical computation originally developed by researchers and engineers working at the Google Brain team. The main focus of the library is to provide an easy-to-use API to implement practical machine learning algorithms and deploy them to run on CPUs, GPUs, or a cluster.

But how do these schemes compare? How much faster does the application run when implemented with NumPy instead of pure Python? What about TensorFlow? The purpose of this article is to begin to explore the improvements you can achieve by using these libraries.

To compare the performance of the three approaches, you’ll build a basic regression with native Python, NumPy, and TensorFlow.

Get Notified: Don't miss the follow up to this tutorial—Click here to join the Real Python Newsletter and you'll know when the next instalment comes out.

Engineering the Test Data

To test the performance of the libraries, you’ll consider a simple two-parameter linear regression problem. The model has two parameters: an intercept term, w_0 and a single coefficient, w_1.

Given N pairs of inputs x and desired outputs d, the idea is to model the relationship between the outputs and the inputs using a linear model y = w_0 + w_1 * x where the output of the model y is approximately equal to the desired output d for every pair (x, d).

Technical Detail: The intercept term, w_0, is technically just a coefficient like w_1, but it can be interpreted as a coefficient that multiplies elements of a vector of 1s.

To generate the training set of the problem, use the following program:

import numpy as np

np.random.seed(444)

N = 10000
sigma = 0.1
noise = sigma * np.random.randn(N)
x = np.linspace(0, 2, N)
d = 3 + 2 * x + noise
d.shape = (N, 1)

# We need to prepend a column vector of 1s to `x`.
X = np.column_stack((np.ones(N, dtype=x.dtype), x))
print(X.shape)
(10000, 2)

This program creates a set of 10,000 inputs x linearly distributed over the interval from 0 to 2. It then creates a set of desired outputs d = 3 + 2 * x + noise, where noise is taken from a Gaussian (normal) distribution with zero mean and standard deviation sigma = 0.1.

By creating x and d in this way, you’re effectively stipulating that the optimal solution for w_0 and w_1 is 3 and 2, respectively.

Xplus = np.linalg.pinv(X)
w_opt = Xplus @ d
print(w_opt)
[[2.99536719]
 [2.00288672]]

There are several methods to estimate the parameters w_0 and w_1 to fit a linear model to the training set. One of the most-used is ordinary least squares, which is a well-known solution for the estimation of w_0 and w_1 in order to minimize the square of the error e, given by the summation of y - d for every training sample.

One way to easily compute the ordinary least squares solution is by using the Moore-Penrose pseudo-inverse of a matrix. This approach stems from the fact that you have X and d and are trying to solve for w_m, in the equation d = X @ w_m. (The @ symbol denotes matrix multiplication, which is supported by both NumPy and native Python as of PEP 465 and Python 3.5+.)

Using this approach, we can estimate w_m using w_opt = Xplus @ d, where Xplus is given by the pseudo-inverse of X, which can be calculated using numpy.linalg.pinv, resulting in w_0 = 2.9978 and w_1 = 2.0016, which is very close to the expected values of w_0 = 3 and w_1 = 2.

Note: Using w_opt = np.linalg.inv(X.T @ X) @ X.T @ d would yield the same solution. For more, see A Matrix Formulation of the Multiple Regression Model.

Although it is possible to use this deterministic approach to estimate the coefficients of the linear model, it is not possible for some other models, such as neural networks. In these cases, iterative algorithms are used to estimate a solution for the parameters of the model.

One of the most-used algorithms is gradient descent, which at a high level consists of updating the parameter coefficients until we converge on a minimized loss (or cost). That is, we have some cost function (often, the mean squared error—MSE), and we compute its gradient with respect to the network’s coefficients (in this case, the parameters w_0 and w_1), considering a step size mu. By performing this update many times (in many epochs), the coefficients converge to a solution that minimizes the cost function.

In the following sections, you’ll build and use gradient descent algorithms in pure Python, NumPy, and TensorFlow. To compare the performance of the three approaches, we’ll look at runtime comparisons on an Intel Core i7 4790K 4.0 GHz CPU.

Gradient Descent in Pure Python

Let’s start with a pure-Python approach as a baseline for comparison with the other approaches. The Python function below estimates the parameters w_0 and w_1 using gradient descent:

import itertools as it

def py_descent(x, d, mu, N_epochs):
    N = len(x)
    f = 2 / N

    # "Empty" predictions, errors, weights, gradients.
    y = [0] * N
    w = [0, 0]
    grad = [0, 0]

    for _ in it.repeat(None, N_epochs):
        # Can't use a generator because we need to
        # access its elements twice.
        err = tuple(i - j for i, j in zip(d, y))
        grad[0] = f * sum(err)
        grad[1] = f * sum(i * j for i, j in zip(err, x))
        w = [i + mu * j for i, j in zip(w, grad)]
        y = (w[0] + w[1] * i for i in x)
    return w

Above, everything is done with Python list comprehensions, slicing syntax, and the built-in sum() and zip() functions. Before running through each epoch, “empty” containers of zeros are initialized for y, w, and grad.

Technical Detail: py_descent above does use itertools.repeat() rather than for _ in range(N_epochs). The former is faster than the latter because repeat() does not need to manufacture a distinct integer for each loop. It just needs to update the reference count to None. The timeit module contains an example.

Now, use this to find a solution:

import time

x_list = x.tolist()
d_list = d.squeeze().tolist()  # Need 1d lists

# `mu` is a step size, or scaling factor.
mu = 0.001
N_epochs = 10000

t0 = time.time()
py_w = py_descent(x_list, d_list, mu, N_epochs)
t1 = time.time()

print(py_w)
[2.959859852416156, 2.0329649630002757]

print('Solve time: {:.2f} seconds'.format(round(t1 - t0, 2)))
Solve time: 18.65 seconds

With a step size of mu = 0.001 and 10,000 epochs, we can get a fairly precise estimate of w_0 and w_1. Inside the for-loop, the gradients with respect to the parameters are calculated and used in turn to update the weights, moving in the opposite direction in order to minimize the MSE cost function.

At each epoch, after the update, the output of the model is calculated. The vector operations are performed using list comprehensions. We could have also updated y in-place, but that would not have been beneficial to performance.

The elapsed time of the algorithm is measured using the time library. It takes 18.65 seconds to estimate w_0 = 2.9598 and w_1 = 2.0329. While the timeit library can provide a more exact estimate of runtime by running multiple loops and disabling garbage collection, just viewing a single run with time suffices in this case, as you’ll see shortly.

Using NumPy

NumPy adds support for large multidimensional arrays and matrices along with a collection of mathematical functions to operate on them. The operations are optimized to run with blazing speed by relying on the projects BLAS and LAPACK for underlying implementation.

Using NumPy, consider the following program to estimate the parameters of the regression:

def np_descent(x, d, mu, N_epochs):
    d = d.squeeze()
    N = len(x)
    f = 2 / N

    y = np.zeros(N)
    err = np.zeros(N)
    w = np.zeros(2)
    grad = np.empty(2)

    for _ in it.repeat(None, N_epochs):
        np.subtract(d, y, out=err)
        grad[:] = f * np.sum(err), f * (err @ x)
        w = w + mu * grad
        y = w[0] + w[1] * x
    return w

np_w = np_descent(x, d, mu, N_epochs)
print(np_w)
[2.95985985 2.03296496]

The code block above takes advantage of vectorized operations with NumPy arrays (ndarrays). The only explicit for-loop is the outer loop over which the training routine itself is repeated. List comprehensions are absent here because NumPy’s ndarray type overloads the arithmetic operators to perform array calculations in an optimized way.

You may notice there are a few alternate ways to go about solving this problem. For instance, you could use simply f * err @ X, where X is the 2d array that includes a column vector of ones, rather than our 1d x.

However, this is actually not all that efficient, because it requires a dot product of an entire column of ones with another vector (err), and we know that result will simply be np.sum(err). Similarly, w[0] + w[1] * x wastes less computation than w * X, in this specific case.

Let’s look at the timing comparison. As you’ll see below, the timeit module is needed here to get a more precise picture of runtime, as we’re now talking about fractions of a second rather than multiple seconds of runtime:

import timeit

setup = ("from __main__ import x, d, mu, N_epochs, np_descent;"
         "import numpy as np")
repeat = 5
number = 5  # Number of loops within each repeat

np_times = timeit.repeat('np_descent(x, d, mu, N_epochs)', setup=setup,
                         repeat=repeat, number=number)

timeit.repeat() returns a list. Each element is the total time taken to execute n loops of the statement. To get a single estimate of runtime, you can take the average time for a single call from the lower bound of the list of repeats:

print(min(np_times) / number)
0.31947448799983247

Using TensorFlow

TensorFlow is an open-source library for numerical computation originally developed by researchers and engineers working at the Google Brain team.

Using its Python API, TensorFlow’s routines are implemented as a graph of computations to perform. Nodes in the graph represent mathematical operations, and the graph edges represent the multidimensional data arrays (also called tensors) communicated between them.

At runtime, TensorFlow takes the graph of computations and runs it efficiently using optimized C++ code. By analyzing the graph of computations, TensorFlow is able to identify the operations that can be run in parallel. This architecture allows the use of a single API to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device.

Using TensorFlow, consider the following program to estimate the parameters of the regression:

import tensorflow as tf

def tf_descent(X_tf, d_tf, mu, N_epochs):
    N = X_tf.get_shape().as_list()[0]
    f = 2 / N

    w = tf.Variable(tf.zeros((2, 1)), name="w_tf")
    y = tf.matmul(X_tf, w, name="y_tf")
    e = y - d_tf
    grad = f * tf.matmul(tf.transpose(X_tf), e)

    training_op = tf.assign(w, w - mu * grad)
    init = tf.global_variables_initializer()

    with tf.Session() as sess:
        init.run()
        for epoch in range(N_epochs):
            sess.run(training_op)
        opt = w.eval()
    return opt

X_tf = tf.constant(X, dtype=tf.float32, name="X_tf")
d_tf = tf.constant(d, dtype=tf.float32, name="d_tf")

tf_w = tf_descent(X_tf, d_tf, mu, N_epochs)
print(tf_w)
[[2.9598553]
 [2.032969 ]]

When you use TensorFlow, the data must be loaded into a special data type called a Tensor. Tensors mirror NumPy arrays in more ways than they are dissimilar.

type(X_tf)
<class 'tensorflow.python.framework.ops.Tensor'>

After the tensors are created from the training data, the graph of computations is defined:

First, a variable tensor w is used to store the regression parameters, which will be updated at each iteration.
Using w and X_tf, the output y is calculated using a matrix product, implemented with tf.matmul().
The error is calculated and stored in the e tensor.
The gradients are computed, using the matrix approach, by multiplying the transpose of X_tf by the e.
Finally, the update of the parameters of the regression is implemented with the tf.assign() function. It creates a node that implements batch gradient descent, updating the next step tensor w to w - mu * grad.

It is worth noticing that the code until the training_op creation does not perform any computation. It just creates the graph of the computations to be performed. In fact, even the variables are not initialized yet. To perform the computations, it is necessary to create a session and use it to initialize the variables and run the algorithm to evaluate the parameters of the regression.

There are some different ways to initialize the variables and create the session to perform the computations. In this program, the line init = tf.global_variables_initializer() creates a node in the graph that will initialize the variables when it is run. The session is created in the with block, and init.run() is used to actually initialize the variables. Inside the with block, training_op is run for the desired number of epochs, evaluating the parameter of the regression, which have their final value stored in opt.

Here is the same code-timing structure that was used with the NumPy implementation:

setup = ("from __main__ import X_tf, d_tf, mu, N_epochs, tf_descent;"
         "import tensorflow as tf")

tf_times = timeit.repeat("tf_descent(X_tf, d_tf, mu, N_epochs)", setup=setup,
                         repeat=repeat, number=number)

print(min(tf_times) / number)
1.1982891103994917

It took 1.20 seconds to estimate w_0 = 2.9598553 and w_1 = 2.032969. It is worth noticing that the computation was performed on a CPU and the performance may be improved when run on a GPU.

Lastly, you could have also defined an MSE cost function and passed this to TensorFlow’s gradients() function, which performs automatic differentiation, finding the gradient vector of MSE with regard to the weights:

mse = tf.reduce_mean(tf.square(e), name="mse")
grad = tf.gradients(mse, w)[0]

However, the timing difference in this case is negligible.

Conclusion

The purpose of this article was to perform a preliminary comparison of the performance of a pure Python, a NumPy and a TensorFlow implementation of a simple iterative algorithm to estimate the coefficients of a linear regression problem.

The results for the elapsed time to run the algorithm are summarized in the table below:

Implementation	Elapsed Time
Pure Python with list comprehensions	18.65s
NumPy	0.32s
TensorFlow on CPU	1.20s

While the NumPy and TensorFlow solutions are competitive (on CPU), the pure Python implementation is a distant third. While Python is a robust general-purpose programming language, its libraries targeted towards numerical computation will win out any day when it comes to large batch operations on arrays.

While the NumPy example proved quicker by a hair than TensorFlow in this case, it’s important to note that TensorFlow really shines for more complex cases. With our relatively elementary regression problem, using TensorFlow arguably amounts to “using a sledgehammer to crack a nut,” as the saying goes.

With TensorFlow, it is possible to build and train complex neural networks across hundreds or thousands of multi-GPU servers. In a future post, we will cover the setup to run this example in GPUs using TensorFlow and compare the results.

Get Notified: Don't miss the follow up to this tutorial—Click here to join the Real Python Newsletter and you'll know when the next instalment comes out.

References

The NumPy and TensorFlow home pages
Aurélien Géron: Hands-On Machine Learning with Scikit-Learn and TensorFlow
Look Ma, No For-Loops: Array Programming With NumPy
NumPy tutorials at Real Python

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Metaclasses

2018-05-01T14:00:00+00:00

The term metaprogramming refers to the potential for a program to have knowledge of or manipulate itself. Python supports a form of metaprogramming for classes called metaclasses.

Metaclasses are an esoteric OOP concept, lurking behind virtually all Python code. You are using them whether you are aware of it or not. For the most part, you don’t need to be aware of it. Most Python programmers rarely, if ever, have to think about metaclasses.

When the need arises, however, Python provides a capability that not all object-oriented languages support: you can get under the hood and define custom metaclasses. The use of custom metaclasses is somewhat controversial, as suggested by the following quote from Tim Peters, the Python guru who authored the Zen of Python:

“Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).”

— Tim Peters

There are Pythonistas (as Python aficionados are known) who believe that you should never use custom metaclasses. That might be going a bit far, but it is probably true that custom metaclasses mostly aren’t necessary. If it isn’t pretty obvious that a problem calls for them, then it will probably be cleaner and more readable if solved in a simpler way.

Still, understanding Python metaclasses is worthwhile, because it leads to a better understanding of the internals of Python classes in general. You never know: you may one day find yourself in one of those situations where you just know that a custom metaclass is what you want.

Don't miss the follow up tutorial: Click here to join the Real Python Newsletter and you'll know when the next instalment comes out.

Old-Style vs. New-Style Classes

In the Python realm, a class can be one of two varieties. No official terminology has been decided on, so they are informally referred to as old-style and new-style classes.

Old-Style Classes

With old-style classes, class and type are not quite the same thing. An instance of an old-style class is always implemented from a single built-in type called instance. If obj is an instance of an old-style class, obj.__class__ designates the class, but type(obj) is always instance. The following example is taken from Python 2.7:

>>> class Foo:
...     pass
...
>>> x = Foo()
>>> x.__class__
<class __main__.Foo at 0x000000000535CC48>
>>> type(x)
<type 'instance'>

New-Style Classes

New-style classes unify the concepts of class and type. If obj is an instance of a new-style class, type(obj) is the same as obj.__class__:

>>> class Foo:
...     pass
>>> obj = Foo()
>>> obj.__class__
<class '__main__.Foo'>
>>> type(obj)
<class '__main__.Foo'>
>>> obj.__class__ is type(obj)
True

>>> n = 5
>>> d = { 'x' : 1, 'y' : 2 }

>>> class Foo:
...     pass
...
>>> x = Foo()

>>> for obj in (n, d, x):
...     print(type(obj) is obj.__class__)
...
True
True
True

Type and Class

In Python 3, all classes are new-style classes. Thus, in Python 3 it is reasonable to refer to an object’s type and its class interchangeably.

Note: In Python 2, classes are old-style by default. Prior to Python 2.2, new-style classes weren’t supported at all. From Python 2.2 onward, they can be created but must be explicitly declared as new-style.

Remember that, in Python, everything is an object. Classes are objects as well. As a result, a class must have a type. What is the type of a class?

Consider the following:

>>> class Foo:
...     pass
...
>>> x = Foo()

>>> type(x)
<class '__main__.Foo'>

>>> type(Foo)
<class 'type'>

The type of x is class Foo, as you would expect. But the type of Foo, the class itself, is type. In general, the type of any new-style class is type.

The type of the built-in classes you are familiar with is also type:

>>> for t in int, float, dict, list, tuple:
...     print(type(t))
...
<class 'type'>
<class 'type'>
<class 'type'>
<class 'type'>
<class 'type'>

For that matter, the type of type is type as well (yes, really):

>>> type(type)
<class 'type'>

type is a metaclass, of which classes are instances. Just as an ordinary object is an instance of a class, any new-style class in Python, and thus any class in Python 3, is an instance of the type metaclass.

In the above case:

x is an instance of class Foo.
Foo is an instance of the type metaclass.
type is also an instance of the type metaclass, so it is an instance of itself.

Defining a Class Dynamically

The built-in type() function, when passed one argument, returns the type of an object. For new-style classes, that is generally the same as the object’s __class__ attribute:

>>> type(3)
<class 'int'>

>>> type(['foo', 'bar', 'baz'])
<class 'list'>

>>> t = (1, 2, 3, 4, 5)
>>> type(t)
<class 'tuple'>

>>> class Foo:
...     pass
...
>>> type(Foo())
<class '__main__.Foo'>

You can also call type() with three arguments—type(<name>, <bases>, <dct>):

<name> specifies the class name. This becomes the __name__ attribute of the class.
<bases> specifies a tuple of the base classes from which the class inherits. This becomes the __bases__ attribute of the class.
<dct> specifies a namespace dictionary containing definitions for the class body. This becomes the __dict__ attribute of the class.

Calling type() in this manner creates a new instance of the type metaclass. In other words, it dynamically creates a new class.

In each of the following examples, the top snippet defines a class dynamically with type(), while the snippet below it defines the class the usual way, with the class statement. In each case, the two snippets are functionally equivalent.

Example 1

In this first example, the <bases> and <dct> arguments passed to type() are both empty. No inheritance from any parent class is specified, and nothing is initially placed in the namespace dictionary. This is the simplest class definition possible:

>>> Foo = type('Foo', (), {})

>>> x = Foo()
>>> x
<__main__.Foo object at 0x04CFAD50>

>>> class Foo:
...     pass
...
>>> x = Foo()
>>> x
<__main__.Foo object at 0x0370AD50>

Example 2

Here, <bases> is a tuple with a single element Foo, specifying the parent class that Bar inherits from. An attribute, attr, is initially placed into the namespace dictionary:

>>> Bar = type('Bar', (Foo,), dict(attr=100))

>>> x = Bar()
>>> x.attr
100
>>> x.__class__
<class '__main__.Bar'>
>>> x.__class__.__bases__
(<class '__main__.Foo'>,)

>>> class Bar(Foo):
...     attr = 100
...

>>> x = Bar()
>>> x.attr
100
>>> x.__class__
<class '__main__.Bar'>
>>> x.__class__.__bases__
(<class '__main__.Foo'>,)

Example 3

This time, <bases> is again empty. Two objects are placed into the namespace dictionary via the <dct> argument. The first is an attribute named attr and the second a function named attr_val, which becomes a method of the defined class:

>>> Foo = type(
...     'Foo',
...     (),
...     {
...         'attr': 100,
...         'attr_val': lambda x : x.attr
...     }
... )

>>> x = Foo()
>>> x.attr
100
>>> x.attr_val()
100

>>> class Foo:
...     attr = 100
...     def attr_val(self):
...         return self.attr
...

>>> x = Foo()
>>> x.attr
100
>>> x.attr_val()
100

Example 4

Only very simple functions can be defined with lambda in Python. In the following example, a slightly more complex function is defined externally then assigned to attr_val in the namespace dictionary via the name f:

>>> def f(obj):
...     print('attr =', obj.attr)
...
>>> Foo = type(
...     'Foo',
...     (),
...     {
...         'attr': 100,
...         'attr_val': f
...     }
... )

>>> x = Foo()
>>> x.attr
100
>>> x.attr_val()
attr = 100

>>> def f(obj):
...     print('attr =', obj.attr)
...
>>> class Foo:
...     attr = 100
...     attr_val = f
...

>>> x = Foo()
>>> x.attr
100
>>> x.attr_val()
attr = 100

Custom Metaclasses

Consider again this well-worn example:

>>> class Foo:
...     pass
...
>>> f = Foo()

The expression Foo() creates a new instance of class Foo. When the interpreter encounters Foo(), the following occurs:

The __call__() method of Foo’s parent class is called. Since Foo is a standard new-style class, its parent class is the type metaclass, so type’s __call__() method is invoked.
That __call__() method in turn invokes the following:
- __new__()
- __init__()

If Foo does not define __new__() and __init__(), default methods are inherited from Foo’s ancestry. But if Foo does define these methods, they override those from the ancestry, which allows for customized behavior when instantiating Foo.

In the following, a custom method called new() is defined and assigned as the __new__() method for Foo:

>>> def new(cls):
...     x = object.__new__(cls)
...     x.attr = 100
...     return x
...
>>> Foo.__new__ = new

>>> f = Foo()
>>> f.attr
100

>>> g = Foo()
>>> g.attr
100

This modifies the instantiation behavior of class Foo: each time an instance of Foo is created, by default it is initialized with an attribute called attr, which has a value of 100. (Code like this would more usually appear in the __init__() method and not typically in __new__(). This example is contrived for demonstration purposes.)

Now, as has already been reiterated, classes are objects too. Suppose you wanted to similarly customize instantiation behavior when creating a class like Foo. If you were to follow the pattern above, you’d again define a custom method and assign it as the __new__() method for the class of which Foo is an instance. Foo is an instance of the type metaclass, so the code looks something like this:

# Spoiler alert:  This doesn't work!
>>> def new(cls):
...     x = type.__new__(cls)
...     x.attr = 100
...     return x
...
>>> type.__new__ = new
Traceback (most recent call last):
  File "<pyshell#77>", line 1, in <module>
    type.__new__ = new
TypeError: can't set attributes of built-in/extension type 'type'

Except, as you can see, you can’t reassign the __new__() method of the type metaclass. Python doesn’t allow it.

This is probably just as well. type is the metaclass from which all new-style classes are derived. You really shouldn’t be mucking around with it anyway. But then what recourse is there, if you want to customize instantiation of a class?

One possible solution is a custom metaclass. Essentially, instead of mucking around with the type metaclass, you can define your own metaclass, which derives from type, and then you can muck around with that instead.

The first step is to define a metaclass that derives from type, as follows:

>>> class Meta(type):
...     def __new__(cls, name, bases, dct):
...         x = super().__new__(cls, name, bases, dct)
...         x.attr = 100
...         return x
...

The definition header class Meta(type): specifies that Meta derives from type. Since type is a metaclass, that makes Meta a metaclass as well.

Note that a custom __new__() method has been defined for Meta. It wasn’t possible to do that to the type metaclass directly. The __new__() method does the following:

Delegates via super() to the __new__() method of the parent metaclass (type) to actually create a new class
Assigns the custom attribute attr to the class, with a value of 100
Returns the newly created class

Now the other half of the voodoo: Define a new class Foo and specify that its metaclass is the custom metaclass Meta, rather than the standard metaclass type. This is done using the metaclass keyword in the class definition as follows:

>>> class Foo(metaclass=Meta):
...     pass
...
>>> Foo.attr
100

Voila! Foo has picked up the attr attribute automatically from the Meta metaclass. Of course, any other classes you define similarly will do likewise:

>>> class Bar(metaclass=Meta):
...     pass
...
>>> class Qux(metaclass=Meta):
...     pass
...
>>> Bar.attr, Qux.attr
(100, 100)

In the same way that a class functions as a template for the creation of objects, a metaclass functions as a template for the creation of classes. Metaclasses are sometimes referred to as class factories.

Compare the following two examples:

Object Factory:

>>> class Foo:
...     def __init__(self):
...         self.attr = 100
...

>>> x = Foo()
>>> x.attr
100

>>> y = Foo()
>>> y.attr
100

>>> z = Foo()
>>> z.attr
100

Class Factory:

>>> class Meta(type):
...     def __init__(
...         cls, name, bases, dct
...     ):
...         cls.attr = 100
...
>>> class X(metaclass=Meta):
...     pass
...
>>> X.attr
100

>>> class Y(metaclass=Meta):
...     pass
...
>>> Y.attr
100

>>> class Z(metaclass=Meta):
...     pass
...
>>> Z.attr
100

Is This Really Necessary?

As simple as the above class factory example is, it is the essence of how metaclasses work. They allow customization of class instantiation.

Still, this is a lot of fuss just to bestow the custom attribute attr on each newly created class. Do you really need a metaclass just for that?

In Python, there are at least a couple other ways in which effectively the same thing can be accomplished:

Simple Inheritance:

>>> class Base:
...     attr = 100
...

>>> class X(Base):
...     pass
...

>>> class Y(Base):
...     pass
...

>>> class Z(Base):
...     pass
...

>>> X.attr
100
>>> Y.attr
100
>>> Z.attr
100

Class Decorator:

>>> def decorator(cls):
...     class NewClass(cls):
...         attr = 100
...     return NewClass
...
>>> @decorator
... class X:
...     pass
...
>>> @decorator
... class Y:
...     pass
...
>>> @decorator
... class Z:
...     pass
...

>>> X.attr
100
>>> Y.attr
100
>>> Z.attr
100

Conclusion

As Tim Peters suggests, metaclasses can easily veer into the realm of being a “solution in search of a problem.” It isn’t typically necessary to create custom metaclasses. If the problem at hand can be solved in a simpler way, it probably should be. Still, it is beneficial to understand metaclasses so that you understand Python classes in general and can recognize when a metaclass really is the appropriate tool to use.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Exceptions: An Introduction

2018-04-30T14:00:00+00:00

A Python program terminates as soon as it encounters an error. In Python, an error can be a syntax error or an exception. In this article, you will see what an exception is and how it differs from a syntax error. After that, you will learn about raising exceptions and making assertions. Then, you’ll finish with a demonstration of the try and except block.

Free PDF Download: Python 3 Cheat Sheet

Exceptions versus Syntax Errors

Syntax errors occur when the parser detects an incorrect statement. Observe the following example:

>>> print( 0 / 0 ))
  File "<stdin>", line 1
    print( 0 / 0 ))
                  ^
SyntaxError: invalid syntax

The arrow indicates where the parser ran into the syntax error. In this example, there was one bracket too many. Remove it and run your code again:

>>> print( 0 / 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: integer division or modulo by zero

This time, you ran into an exception error. This type of error occurs whenever syntactically correct Python code results in an error. The last line of the message indicated what type of exception error you ran into.

Instead of showing the message exception error, Python details what type of exception error was encountered. In this case, it was a ZeroDivisionError. Python comes with various built-in exceptions as well as the possibility to create self-defined exceptions.

Raising an Exception

We can use raise to throw an exception if a condition occurs. The statement can be complemented with a custom exception.

If you want to throw an error when a certain condition occurs using raise, you could go about it like this:

x = 10
if x > 5:
    raise Exception('x should not exceed 5. The value of x was: {}'.format(x))

When you run this code, the output will be the following:

Traceback (most recent call last):
  File "<input>", line 4, in <module>
Exception: x should not exceed 5. The value of x was: 10

The program comes to a halt and displays our exception to screen, offering clues about what went wrong.

The `AssertionError` Exception

Instead of waiting for a program to crash midway, you can also start by making an assertion in Python. We assert that a certain condition is met. If this condition turns out to be True, then that is excellent! The program can continue. If the condition turns out to be False, you can have the program throw an AssertionError exception.

Have a look at the following example, where it is asserted that the code will be executed on a Linux system:

import sys
assert ('linux' in sys.platform), "This code runs on Linux only."

If you run this code on a Linux machine, the assertion passes. If you were to run this code on a Windows machine, the outcome of the assertion would be False and the result would be the following:

Traceback (most recent call last):
  File "<input>", line 2, in <module>
AssertionError: This code runs on Linux only.

In this example, throwing an AssertionError exception is the last thing that the program will do. The program will come to halt and will not continue. What if that is not what you want?

The `try` and `except` Block: Handling Exceptions

The try and except block in Python is used to catch and handle exceptions. Python executes code following the try statement as a “normal” part of the program. The code that follows the except statement is the program’s response to any exceptions in the preceding try clause.

As you saw earlier, when syntactically correct code runs into an error, Python will throw an exception error. This exception error will crash the program if it is unhandled. The except clause determines how your program responds to exceptions.

The following function can help you understand the try and except block:

def linux_interaction():
    assert ('linux' in sys.platform), "Function can only run on Linux systems."
    print('Doing something.')

The linux_interaction() can only run on a Linux system. The assert in this function will throw an AssertionError exception if you call it on an operating system other then Linux.

You can give the function a try using the following code:

try:
    linux_interaction()
except:
    pass

The way you handled the error here is by handing out a pass. If you were to run this code on a Windows machine, you would get the following output:

You got nothing. The good thing here is that the program did not crash. But it would be nice to see if some type of exception occurred whenever you ran your code. To this end, you can change the pass into something that would generate an informative message, like so:

try:
    linux_interaction()
except:
    print('Linux function was not executed')

Execute this code on a Windows machine:

Linux function was not executed

When an exception occurs in a program running this function, the program will continue as well as inform you about the fact that the function call was not successful.

What you did not get to see was the type of error that was thrown as a result of the function call. In order to see exactly what went wrong, you would need to catch the error that the function threw.

The following code is an example where you capture the AssertionError and output that message to screen:

try:
    linux_interaction()
except AssertionError as error:
    print(error)
    print('The linux_interaction() function was not executed')

Running this function on a Windows machine outputs the following:

Function can only run on Linux systems.
The linux_interaction() function was not executed

The first message is the AssertionError, informing you that the function can only be executed on a Linux machine. The second message tells you which function was not executed.

In the previous example, you called a function that you wrote yourself. When you executed the function, you caught the AssertionError exception and printed it to screen.

Here’s another example where you open a file and use a built-in exception:

try:
    with open('file.log') as file:
        read_data = file.read()
except:
    print('Could not open file.log')

If file.log does not exist, this block of code will output the following:

Could not open file.log

This is an informative message, and our program will still continue to run. In the Python docs, you can see that there are a lot of built-in exceptions that you can use here. One exception described on that page is the following:

Exception FileNotFoundError

Raised when a file or directory is requested but doesn’t exist. Corresponds to errno ENOENT.

To catch this type of exception and print it to screen, you could use the following code:

try:
    with open('file.log') as file:
        read_data = file.read()
except FileNotFoundError as fnf_error:
    print(fnf_error)

In this case, if file.log does not exist, the output will be the following:

[Errno 2] No such file or directory: 'file.log'

You can have more than one function call in your try clause and anticipate catching various exceptions. A thing to note here is that the code in the try clause will stop as soon as an exception is encountered.

Warning: Catching Exception hides all errors…even those which are completely unexpected. This is why you should avoid bare except clauses in your Python programs. Instead, you’ll want to refer to specific exception classes you want to catch and handle. You can learn more about why this is a good idea in this tutorial.

Look at the following code. Here, you first call the linux_interaction() function and then try to open a file:

try:
    linux_interaction()
    with open('file.log') as file:
        read_data = file.read()
except FileNotFoundError as fnf_error:
    print(fnf_error)
except AssertionError as error:
    print(error)
    print('Linux linux_interaction() function was not executed')

If the file does not exist, running this code on a Windows machine will output the following:

Function can only run on Linux systems.
Linux linux_interaction() function was not executed

Inside the try clause, you ran into an exception immediately and did not get to the part where you attempt to open file.log. Now look at what happens when you run the code on a Linux machine:

[Errno 2] No such file or directory: 'file.log'

Here are the key takeaways:

A try clause is executed up until the point where the first exception is encountered.
Inside the except clause, or the exception handler, you determine how the program responds to the exception.
You can anticipate multiple exceptions and differentiate how the program should respond to them.
Avoid using bare except clauses.

The `else` Clause

In Python, using the else statement, you can instruct a program to execute a certain block of code only in the absence of exceptions.

Look at the following example:

try:
    linux_interaction()
except AssertionError as error:
    print(error)
else:
    print('Executing the else clause.')

If you were to run this code on a Linux system, the output would be the following:

Doing something.
Executing the else clause.

Because the program did not run into any exceptions, the else clause was executed.

You can also try to run code inside the else clause and catch possible exceptions there as well:

try:
    linux_interaction()
except AssertionError as error:
    print(error)
else:
    try:
        with open('file.log') as file:
            read_data = file.read()
    except FileNotFoundError as fnf_error:
        print(fnf_error)

If you were to execute this code on a Linux machine, you would get the following result:

Doing something.
[Errno 2] No such file or directory: 'file.log'

From the output, you can see that the linux_interaction() function ran. Because no exceptions were encountered, an attempt to open file.log was made. That file did not exist, and instead of opening the file, you caught the FileNotFoundError exception.

Cleaning Up After Using `finally`

Imagine that you always had to implement some sort of action to clean up after executing your code. Python enables you to do so using the finally clause.

Have a look at the following example:

try:
    linux_interaction()
except AssertionError as error:
    print(error)
else:
    try:
        with open('file.log') as file:
            read_data = file.read()
    except FileNotFoundError as fnf_error:
        print(fnf_error)
finally:
    print('Cleaning up, irrespective of any exceptions.')

In the previous code, everything in the finally clause will be executed. It does not matter if you encounter an exception somewhere in the try or else clauses. Running the previous code on a Windows machine would output the following:

Function can only run on Linux systems.
Cleaning up, irrespective of any exceptions.

Summing Up

After seeing the difference between syntax errors and exceptions, you learned about various ways to raise, catch, and handle exceptions in Python. In this article, you saw the following options:

raise allows you to throw an exception at any time.
assert enables you to verify if a certain condition is met and throw an exception if it isn’t.
In the try clause, all statements are executed until an exception is encountered.
except is used to catch and handle the exception(s) that are encountered in the try clause.
else lets you code sections that should run only when no exceptions are encountered in the try clause.
finally enables you to execute sections of code that should always run, with or without any previously encountered exceptions.

Free PDF Download: Python 3 Cheat Sheet

Hopefully, this article helped you understand the basic tools that Python has to offer when dealing with exceptions.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Pipenv: A Guide to the New Python Packaging Tool

2018-04-24T14:00:00+00:00

Pipenv is a packaging tool for Python that solves some common problems associated with the typical workflow using pip, virtualenv, and the good old requirements.txt.

In addition to addressing some common issues, it consolidates and simplifies the development process to a single command line tool.

This guide will go over what problems Pipenv solves and how to manage your Python dependencies with Pipenv. Additionally, it will cover how Pipenv fits in with previous methods for package distribution.

Free Bonus: Click here to get access to a free 5-day class that shows you how to avoid common dependency management issues with tools like Pip, PyPI, Virtualenv, and requirements files.

Problems that Pipenv Solves

To understand the benefits of Pipenv, it’s important to walk through the current methods for packaging and dependency management in Python.

Let’s start with a typical situation of handling third-party packages. We’ll then build our way towards deploying a complete Python application.

Dependency Management with `requirements.txt`

Imagine you’re working on a Python project that uses a third-party package like flask. You’ll need to specify that requirement so that other developers and automated systems can run your application.

So you decide to include the flask dependency in a requirements.txt file:

flask

Great, everything works fine locally, and after hacking away on your app for a while, you decide to move it to production. Here’s where things get a little scary…

The above requirements.txt file doesn’t specify which version of flask to use. In this case, pip install -r requirements.txt will install the latest version by default. This is okay unless there are interface or behavior changes in the newest version that break our application.

For the sake of this example, let’s say that a new version of flask got released. However, it isn’t backward compatible with the version you used during development.

Now, let’s say you deploy your application to production and do a pip install -r requirements.txt. Pip gets the latest, not-backward-compatible version of flask, and just like that, your application breaks… in production.

“But hey, it worked on my machine!”—I’ve been there myself, and it’s not a great feeling.

At this point, you know that the version of flask you used during development worked fine. So, to fix things, you try to be a little more specific in your requirements.txt. You add a version specifier to the flask dependency. This is also called pinning a dependency:

flask==0.12.1

Pinning the flask dependency to a specific version ensures that a pip install -r requirements.txt sets up the exact version of flask you used during development. But does it really?

Keep in mind that flask itself has dependencies as well (which pip installs automatically). However, flask itself doesn’t specify exact versions for its dependencies. For example, it allows any version of Werkzeug>=0.14.

Again, for the sake of this example, let’s say a new version of Werkzeug got released, but it introduces a show-stopper bug to your application.

When you do pip install -r requirements.txt in production this time, you will get flask==0.12.1 since you’ve pinned that requirement. However, unfortunately, you’ll get the latest, buggy version of Werkzeug. Again, the product breaks in production.

The real issue here is that the build isn’t deterministic. What I mean by that is that, given the same input (the requirements.txt file), pip doesn’t always produce the same environment. At the moment, you can’t easily replicate the exact environment you have on your development machine in production.

The typical solution to this problem is to use pip freeze. This command allows you to get exact versions for all 3rd party libraries currently installed, including the sub-dependencies pip installed automatically. So you can freeze everything in development to ensure that you have the same environment in production.

Executing pip freeze results in pinned dependencies you can add to a requirements.txt:

click==6.7
Flask==0.12.1
itsdangerous==0.24
Jinja2==2.10
MarkupSafe==1.0
Werkzeug==0.14.1

With these pinned dependencies, you can ensure that the packages installed in your production environment match those in your development environment exactly, so your product doesn’t unexpectedly break. This “solution,” unfortunately, leads to a whole new set of problems.

Now that you’ve specified the exact versions of every third-party package, you are responsible for keeping these versions up to date, even though they’re sub-dependencies of flask. What if there’s a security hole discovered in Werkzeug==0.14.1 that the package maintainers immediately patched in Werkzeug==0.14.2? You really need to update to Werkzeug==0.14.2 to avoid any security issues arising from the earlier, unpatched version of Werkzeug.

First, you need to be aware that there’s an issue with the version you have. Then, you need to get the new version in your production environment before someone exploits the security hole. So, you have to change your requirements.txt manually to specify the new version Werkzeug==0.14.2. As you can see in this situation, the responsibility of staying up to date with necessary updates falls on you.

The truth is that you really don’t care what version of Werkzeug gets installed as long as it doesn’t break your code. In fact, you probably want the latest version to ensure that you’re getting bug fixes, security patches, new features, more optimization, and so on.

The real question is: “How do you allow for deterministic builds for your Python project without gaining the responsibility of updating versions of sub-dependencies?”

Spoiler alert: The easy answer is using Pipenv.

Development of Projects with Different Dependencies

Let’s switch gears a bit to talk about another common issue that arises when you’re working on multiple projects. Imagine that ProjectA needs django==1.9, but ProjectB needs django==1.10.

By default, Python tries to store all your third-party packages in a system-wide location. This means that every time you want to switch between ProjectA and ProjectB, you have to make sure the right version of django is installed. This makes switching between projects painful because you have to uninstall and reinstall packages to meet the requirements for each project.

The standard solution is to use a virtual environment that has its own Python executable and third-party package storage. That way, ProjectA and ProjectB are adequately separated. Now you can easily switch between projects since they’re not sharing the same package storage location. PackageA can have whatever version of django it needs in its own environment, and PackageB can have what it needs totally separate. A very common tool for this is virtualenv (or venv in Python 3).

Pipenv has virtual environment management built in so that you have a single tool for your package management.

Dependency Resolution

What do I mean by dependency resolution? Let’s say you’ve got a requirements.txt file that looks something like this:

package_a
package_b

Let’s say package_a has a sub-dependency package_c, and package_a requires a specific version of this package: package_c>=1.0. In turn, package_b has the same sub-dependency but needs package_c<=2.0.

Ideally, when you try to install package_a and package_b, the installation tool would look at the requirements for package_c (being >=1.0 and <=2.0) and select a version that fulfills those requirements. You’d hope that the tool resolves the dependencies so that your program works in the end. This is what I mean by “dependency resolution.”

Unfortunately, pip itself doesn’t have real dependency resolution at the moment, but there’s an open issue to support it.

The way pip would handle the above scenario is as follows:

It installs package_a and looks for a version of package_c that fulfills the first requirement (package_c>=1.0).
Pip then installs the latest version of package_c to fulfill that requirement. Let’s say the latest version of package_c is 3.1.

This is where the trouble (potentially) starts.

If the version of package_c selected by pip doesn’t fit future requirements (such as package_b needing package_c<=2.0), the installation will fail.

The “solution” to this problem is to specify the range required for the sub-dependency (package_c) in the requirements.txt file. That way, pip can resolve this conflict and install a package that meets those requirements:

package_c>=1.0,<=2.0
package_a
package_b

Just like before though, you’re now concerning yourself directly with sub-dependencies (package_c). The issue with this is that if package_a changes their requirement without you knowing, the requirements you specified (package_c>=1.0,<=2.0) may no longer be acceptable, and installation may fail… again. The real problem is that once again, you’re responsible for staying up to date with requirements of sub-dependencies.

Ideally, your installation tool would be smart enough to install packages that meet all the requirements without you explicitly specifying sub-dependency versions.

Pipenv Introduction

Now that we’ve addressed the problems, let’s see how Pipenv solves them.

First, let’s install it:

$ pip install pipenv

Once you’ve done that, you can effectively forget about pip since Pipenv essentially acts as a replacement. It also introduces two new files, the Pipfile (which is meant to replace requirements.txt) and the Pipfile.lock (which enables deterministic builds).

Pipenv uses pip and virtualenv under the hood but simplifies their usage with a single command line interface.

Example Usage

Let’s start over with creating your awesome Python application. First, spawn a shell in a virtual environment to isolate the development of this app:

$ pipenv shell

This will create a virtual environment if one doesn’t already exist. Pipenv creates all your virtual environments in a default location. If you want to change Pipenv’s default behavior, there are some environmental variables for configuration.

You can force the creation of a Python 2 or 3 environment with the arguments --two and --three respectively. Otherwise, Pipenv will use whatever default virtualenv finds.

Sidenote: If you require a more specific version of Python, you can provide a --python argument with the version you require. For example: --python 3.6

Now you can install the 3rd party package you need, flask. Oh, but you know that you need version 0.12.1 and not the latest version, so go ahead and be specific:

$ pipenv install flask==0.12.1

You should see something like the following in your terminal:

Adding flask==0.12.1 to Pipfile's [packages]...
Pipfile.lock not found, creating...

You’ll notice that two files get created, a Pipfile and Pipfile.lock. We’ll take a closer look at these in a second. Let’s install another 3rd party package, numpy, for some number-crunching. You don’t need a specific version so don’t specify one:

$ pipenv install numpy

If you want to install something directly from a version control system (VCS), you can! You specify the locations similarly to how you’d do so with pip. For example, to install the requests library from version control, do the following:

$ pipenv install -e git+https://github.com/requests/requests.git#egg=requests

Note the -e argument above to make the installation editable. Currently, this is required for Pipenv to do sub-dependency resolution.

Let’s say you also have some unit tests for this awesome application, and you want to use pytest for running them. You don’t need pytest in production so you can specify that this dependency is only for development with the --dev argument:

$ pipenv install pytest --dev

Providing the --dev argument will put the dependency in a special [dev-packages] location in the Pipfile. These development packages only get installed if you specify the --dev argument with pipenv install.

The different sections separate dependencies needed only for development from ones needed for the base code to actually work. Typically, this would be accomplished with additional requirements files like dev-requirements.txt or test-requirements.txt. Now, everything is consolidated in a single Pipfile under different sections.

Okay, so let’s say you’ve got everything working in your local development environment and you’re ready to push it to production. To do that, you need to lock your environment so you can ensure you have the same one in production:

$ pipenv lock

This will create/update your Pipfile.lock, which you’ll never need to (and are never meant to) edit manually. You should always use the generated file.

Now, once you get your code and Pipfile.lock in your production environment, you should install the last successful environment recorded:

$ pipenv install --ignore-pipfile

This tells Pipenv to ignore the Pipfile for installation and use what’s in the Pipfile.lock. Given this Pipfile.lock, Pipenv will create the exact same environment you had when you ran pipenv lock, sub-dependencies and all.

The lock file enables deterministic builds by taking a snapshot of all the versions of packages in an environment (similar to the result of a pip freeze).

Now let’s say another developer wants to make some additions to your code. In this situation, they would get the code, including the Pipfile, and use this command:

$ pipenv install --dev

This installs all the dependencies needed for development, which includes both the regular dependencies and those you specified with the --dev argument during install.

When an exact version isn’t specified in the Pipfile, the install command gives the opportunity for dependencies (and sub-dependencies) to update their versions.

This is an important note because it solves some of the previous problems we discussed. To demonstrate, let’s say a new version of one of your dependencies becomes available. Because you don’t need a specific version of this dependency, you don’t specify an exact version in the Pipfile. When you pipenv install, the new version of the dependency will be installed in your development environment.

Now you make your changes to the code and run some tests to verify everything is still working as expected. (You do have unit tests, right?) Now, just as before, you lock your environment with pipenv lock, and an updated Pipfile.lock will be generated with the new version of the dependency. Just as before, you can replicate this new environment in production with the lock file.

As you can see from this scenario, you no longer have to force exact versions you don’t truly need to ensure your development and production environments are the same. You also don’t need to stay on top of updating sub-dependencies you “don’t care about.” This workflow with Pipenv, combined with your excellent testing, fixes the issues of manually doing all your dependency management.

Pipenv’s Dependency Resolution Approach

Pipenv will attempt to install sub-dependencies that satisfy all the requirements from your core dependencies. However, if there are conflicting dependencies (package_a needs package_c>=1.0, but package_b needs package_c<1.0), Pipenv will not be able to create a lock file and wil output an error like the following:

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches package_c>=1.0,package_c<1.0

As the warning says, you can also show a dependency graph to understand your top-level dependencies and their sub-dependencies:

$ pipenv graph

This command will print out a tree-like structure showing your dependencies. Here’s an example:

Flask==0.12.1
  - click [required: >=2.0, installed: 6.7]
  - itsdangerous [required: >=0.21, installed: 0.24]
  - Jinja2 [required: >=2.4, installed: 2.10]
    - MarkupSafe [required: >=0.23, installed: 1.0]
  - Werkzeug [required: >=0.7, installed: 0.14.1]
numpy==1.14.1
pytest==3.4.1
  - attrs [required: >=17.2.0, installed: 17.4.0]
  - funcsigs [required: Any, installed: 1.0.2]
  - pluggy [required: <0.7,>=0.5, installed: 0.6.0]
  - py [required: >=1.5.0, installed: 1.5.2]
  - setuptools [required: Any, installed: 38.5.1]
  - six [required: >=1.10.0, installed: 1.11.0]
requests==2.18.4
  - certifi [required: >=2017.4.17, installed: 2018.1.18]
  - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
  - idna [required: >=2.5,<2.7, installed: 2.6]
  - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]

From the output of pipenv graph, you can see the top-level dependencies we installed previously (Flask, numpy, pytest, and requests), and underneath you can see the packages they depend on.

Additionally, you can reverse the tree to show the sub-dependencies with the parent that requires it:

$ pipenv graph --reverse

This reversed tree may be more useful when you are trying to figure out conflicting sub-dependencies.

The Pipfile

Pipfile intends to replace requirements.txt. Pipenv is currently the reference implementation for using Pipfile. It seems very likely that pip itself will be able to handle these files. Also, it’s worth noting that Pipenv is even the official package management tool recommended by Python itself.

The syntax for the Pipfile is TOML, and the file is separated into sections. [dev-packages] for development-only packages, [packages] for minimally required packages, and [requires] for other requirements like a specific version of Python. See an example file below:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]
pytest = "*"

[packages]
flask = "==0.12.1"
numpy = "*"
requests = {git = "https://github.com/requests/requests.git", editable = true}

[requires]
python_version = "3.6"

Ideally, you shouldn’t have any sub-dependencies in your Pipfile. What I mean by that is you should only include the packages you actually import and use. No need to keep chardet in your Pipfile just because it’s a sub-dependency of requests. (Pipenv will install it automatically.) The Pipfile should convey the top-level dependencies your package requires.

The Pipfile.lock

This file enables deterministic builds by specifying the exact requirements for reproducing an environment. It contains exact versions for packages and hashes to support more secure verification, which pip itself now supports as well. An example file might look like the following. Note that the syntax for this file is JSON and that I’ve excluded parts of the file with ...:

{
    "_meta": {
        ...
    },
    "default": {
        "flask": {
            "hashes": [
                "sha256:6c3130c8927109a08225993e4e503de4ac4f2678678ae211b33b519c622a7242",
                "sha256:9dce4b6bfbb5b062181d3f7da8f727ff70c1156cbb4024351eafd426deb5fb88"
            ],
            "version": "==0.12.1"
        },
        "requests": {
            "editable": true,
            "git": "https://github.com/requests/requests.git",
            "ref": "4ea09e49f7d518d365e7c6f7ff6ed9ca70d6ec2e"
        },
        "werkzeug": {
            "hashes": [
                "sha256:d5da73735293558eb1651ee2fddc4d0dedcfa06538b8813a2e20011583c9e49b",
                "sha256:c3fd7a7d41976d9f44db327260e263132466836cef6f91512889ed60ad26557c"
            ],
            "version": "==0.14.1"
        }
        ...
    },
    "develop": {
        "pytest": {
            "hashes": [
                "sha256:8970e25181e15ab14ae895599a0a0e0ade7d1f1c4c8ca1072ce16f25526a184d",
                "sha256:9ddcb879c8cc859d2540204b5399011f842e5e8823674bf429f70ada281b3cc6"
            ],
            "version": "==3.4.1"
        },
        ...
    }
}

Note the exact version specified for every dependency. Even the sub-dependencies like werkzeug that aren’t in our Pipfile appear in this Pipfile.lock. The hashes are used to ensure you’re retrieving the same package as you did in development.

It’s worth noting again that you should never change this file by hand. It is meant to be generated with pipenv lock.

Pipenv Extra Features

Open a third-party package in your default editor with the following command:

$ pipenv open flask

This will open the flask package in the default editor, or you can specify a program with an EDITOR environmental variable. For example, I use Sublime Text, so I just set EDITOR=subl. This makes it super simple to dig into the internals of a package you’re using.

You can run a command in the virtual environment without launching a shell:

$ pipenv run <insert command here>

Check for security vulnerabilities (and PEP 508 requirements) in your environment:

$ pipenv check

Now, let’s say you no longer need a package. You can uninstall it:

$ pipenv uninstall numpy

Additionally, let’s say you want to completely wipe all the installed packages from your virtual environment:

$ pipenv uninstall --all

You can replace --all with --all-dev to just remove dev packages.

Pipenv supports the automatic loading of environmental variables when a .env file exists in the top-level directory. That way, when you pipenv shell to open the virtual environment, it loads your environmental variables from the file. The .env file just contains key-value pairs:

SOME_ENV_CONFIG=some_value
SOME_OTHER_ENV_CONFIG=some_other_value

Finally, here are some quick commands to find out where stuff is. How to find out where your virtual environment is:

$ pipenv --venv

How to find out where your project home is:

$ pipenv --where

Package Distribution

You may be asking how this all works if you intend to distribute your code as a package.

Yes, I need to distribute my code as a package

How does Pipenv work with setup.py files?

There are a lot of nuances to that question. First, a setup.py file is necessary when you’re using setuptools as your build/distribution system. This has been the de facto standard for a while now, but recent changes have made the use of setuptools optional.

This means that projects like flit can use the new pyproject.toml to specify a different build system that doesn’t require a setup.py.

All that being said, for the near future setuptools and an accompanying setup.py will still be the default choice for many people.

Here’s a recommended workflow for when you are using a setup.py as a way to distribute your package:

setup.py
install_requires keyword should include whatever the package “minimally needs to run correctly.”
Pipfile
Represents the concrete requirements for your package
Pull the minimally required dependencies from setup.py by installing your package using Pipenv:
- Use pipenv install '-e .'
- That will result in a line in your Pipfile that looks something like "e1839a8" = {path = ".", editable = true}.
Pipfile.lock
Details for a reproducible environment generated from pipenv lock

To clarify, put your minimum requirements in setup.py instead of directly with pipenv install. Then use the pipenv install '-e .' command to install your package as editable. This gets all the requirements from setup.py into your environment. Then you can use pipenv lock to get a reproducible environment.

I don’t need to distribute my code as a package

Great! If you are developing an application that isn’t meant to be distributed or installed (a personal website, a desktop application, a game, or similar), you don’t really need a setup.py.

In this situation, you could use Pipfile/Pipfile.lock combo for managing your dependencies with the flow described previously to deploy a reproducible environment in production.

I already have a `requirements.txt`. How do I convert to a `Pipfile`?

If you run pipenv install it should automatically detect the requirements.txt and convert it to a Pipfile, outputting something like the following:

requirements.txt found, instead of Pipfile! Converting…
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did.
We recommend updating your Pipfile to specify the "*" version, instead.

Take note of the above warning.

If you have pinned exact versions in your requirements.txt file, you’ll probably want to change your Pipfile to only specify exact versions you truly require. This will allow you to gain the real benefits of transitioning. For example, let’s say you have the following but really don’t need that exact version of numpy:

[packages]
numpy = "==1.14.1"

If you don’t have any specific version requirements for your dependencies, you can use the wildcard character * to tell Pipenv that any version can be installed:

[packages]
numpy = "*"

If you feel nervous about allowing any version with the *, it’s typically a safe bet to specify greater than or equal to the version you’re already on so you can still take advantage of new versions:

[packages]
numpy = ">=1.14.1"

Of course, staying up to date with new releases also means you’re responsible for ensuring your code still functions as expected when packages change. This means a test suite is essential to this whole Pipenv flow if you want to ensure functioning releases of your code.

You allow packages to update, run your tests, ensure they all pass, lock your environment, and then you can rest easy knowing that you haven’t introduced breaking changes. If things do break because of a dependency, you’ve got some regression tests to write and potentially some more restrictions on versions of dependencies.

For example, if numpy==1.15 gets installed after running pipenv install and it breaks your code, which you hopefully either notice during development or during your tests, you have a couple options:

Update your code to function with the new version of the dependency.

If backward compatibility with previous versions of the dependency isn’t possible, you’ll also need to bump your required version in your Pipfile:
```
[packages]
numpy = ">=1.15"
```
Restrict the version of the dependency in the Pipfile to be < the version that just broke your code:
```
[packages]
numpy = ">=1.14.1,<1.15"
```

Option 1 is preferred as it ensures that your code is using the most up-to-date dependencies. However, Option 2 takes less time and doesn’t require code changes, just restrictions on dependencies.

You can also install from requirement files with the same -r argument pip takes:

$ pipenv install -r requirements.txt

If you have a dev-requirements.txt or something similar, you can add those to the Pipfile as well. Just add the --dev argument so it gets put in the right section:

$ pipenv install -r dev-requirements.txt --dev

Additionally, you can go the other way and generate requirements files from a Pipfile:

$ pipenv lock -r > requirements.txt
$ pipenv lock -r -d > dev-requirements.txt

What’s next?

It appears to me that a natural progression for the Python ecosystem would be a build system that uses the Pipfile to install the minimally required dependencies when retrieving and building a package from a package index (like PyPI). It is important to note again that the Pipfile design specification is still in development, and Pipenv is just a reference implementation.

That being said, I could see a future where the install_requires section of setup.py doesn’t exist, and the Pipfile is referenced for minimal requirements instead. Or the setup.py is gone entirely, and you get metadata and other information in a different manner, still using the Pipfile to get the necessary dependencies.

Is Pipenv worth checking out?

Definitely. Even if it’s just as a way to consolidate the tools you already use (pip & virtualenv) into a single interface. However, it’s much more than that. With the addition of the Pipfile, you only specify the dependencies you truly need.

You no longer have the headache of managing the versions of everything yourself just to ensure you can replicate your development environment. With the Pipfile.lock, you can develop with peace of mind knowing that you can exactly reproduce your environment anywhere.

In addition to all that, it seems very likely that the Pipfile format will get adopted and supported by official Python tools like pip, so it’d be beneficial to be ahead of the game. Oh, and make sure you’re updating all your code to Python 3 as well: 2020 is coming up fast.

References, further reading, interesting discussions, and so forth

Free Bonus: Click here to get access to a free 5-day class that shows you how to avoid common dependency management issues with tools like Pip, PyPI, Virtualenv, and requirements files.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python 3's pathlib Module: Taming the File System

2018-04-23T14:00:00+00:00

Have you struggled with file path handling in Python? In Python 3.4 and above, the struggle is now over! You no longer need to scratch your head over code like:

>>> path.rsplit('\\', maxsplit=1)[0]

Or cringe at the verbosity of:

>>> os.path.isfile(os.path.join(os.path.expanduser('~'), 'realpython.txt'))

In this tutorial, you will see how to work with file paths—names of directories and files—in Python. You will learn new ways to read and write files, manipulate paths and the underlying file system, as well as see some examples of how to list files and iterate over them. Using the pathlib module, the two examples above can be rewritten using elegant, readable, and Pythonic code like:

>>> path.parent
>>> (pathlib.Path.home() / 'realpython.txt').is_file()

Free PDF Download: Python 3 Cheat Sheet

The Problem With Python File Path Handling

Working with files and interacting with the file system are important for many different reasons. The simplest cases may involve only reading or writing files, but sometimes more complex tasks are at hand. Maybe you need to list all files in a directory of a given type, find the parent directory of a given file, or create a unique file name that does not already exist.

Traditionally, Python has represented file paths using regular text strings. With support from the os.path standard library, this has been adequate although a bit cumbersome (as the second example in the introduction shows). However, since paths are not strings, important functionality is spread all around the standard library, including libraries like os, glob, and shutil. The following example needs three import statements just to move all text files to an archive directory:

import glob
import os
import shutil

for file_name in glob.glob('*.txt'):
    new_path = os.path.join('archive', file_name)
    shutil.move(file_name, new_path)

With paths represented by strings, it is possible, but usually a bad idea, to use regular string methods. For instance, instead of joining two paths with + like regular strings, you should use os.path.join(), which joins paths using the correct path separator on the operating system. Recall that Windows uses \ while Mac and Linux use / as a separator. This difference can lead to hard-to-spot errors, such as our first example in the introduction working for only Windows paths.

The pathlib module was introduced in Python 3.4 (PEP 428) to deal with these challenges. It gathers the necessary functionality in one place and makes it available through methods and properties on an easy-to-use Path object.

Early on, other packages still used strings for file paths, but as of Python 3.6, the pathlib module is supported throughout the standard library, partly due to the addition of a file system path protocol. If you are stuck on legacy Python, there is also a backport available for Python 2.

Time for action: let us see how pathlib works in practice.

Creating Paths

All you really need to know about is the pathlib.Path class. There are a few different ways of creating a path. First of all, there are classmethods like .cwd() (Current Working Directory) and .home() (your user’s home directory):

>>> import pathlib
>>> pathlib.Path.cwd()
PosixPath('/home/gahjelle/realpython/')

Note: Throughout this tutorial, we will assume that pathlib has been imported, without spelling out import pathlib as above. As you will mainly be using the Path class, you can also do from pathlib import Path and write Path instead of pathlib.Path.

A path can also be explicitly created from its string representation:

>>> pathlib.Path(r'C:\Users\gahjelle\realpython\file.txt')
WindowsPath('C:/Users/gahjelle/realpython/file.txt')

A little tip for dealing with Windows paths: on Windows, the path separator is a backslash, \. However, in many contexts, backslash is also used as an escape character in order to represent non-printable characters. To avoid problems, use raw string literals to represent Windows paths. These are string literals that have an r prepended to them. In raw string literals the \ represents a literal backslash: r'C:\Users'.

A third way to construct a path is to join the parts of the path using the special operator /. The forward slash operator is used independently of the actual path separator on the platform:

>>> pathlib.Path.home() / 'python' / 'scripts' / 'test.py'
PosixPath('/home/gahjelle/python/scripts/test.py')

The / can join several paths or a mix of paths and strings (as above) as long as there is at least one Path object. If you do not like the special / notation, you can do the same thing with the .joinpath() method:

>>> pathlib.Path.home().joinpath('python', 'scripts', 'test.py')
PosixPath('/home/gahjelle/python/scripts/test.py')

Note that in the preceding examples, the pathlib.Path is represented by either a WindowsPath or a PosixPath. The actual object representing the path depends on the underlying operating system. (That is, the WindowsPath example was run on Windows, while the PosixPath examples have been run on Mac or Linux.) See the section Operating System Differences for more information.

Reading and Writing Files

Traditionally, the way to read or write a file in Python has been to use the built-in open() function. This is still true as the open() function can use Path objects directly. The following example finds all headers in a Markdown file and prints them:

path = pathlib.Path.cwd() / 'test.md'
with open(path, mode='r') as fid:
    headers = [line.strip() for line in fid if line.startswith('#')]
print('\n'.join(headers))

An equivalent alternative is to call .open() on the Path object:

with path.open(mode='r') as fid:
    ...

In fact, Path.open() is calling the built-in open() behind the scenes. Which option you use is mainly a matter of taste.

For simple reading and writing of files, there are a couple of convenience methods in the pathlib library:

.read_text(): open the path in text mode and return the contents as a string.
.read_bytes(): open the path in binary/bytes mode and return the contents as a bytestring.
.write_text(): open the path and write string data to it.
.write_bytes(): open the path in binary/bytes mode and write data to it.

Each of these methods handles the opening and closing of the file, making them trivial to use, for instance:

>>> path = pathlib.Path.cwd() / 'test.md'
>>> path.read_text()
<the contents of the test.md-file>

Paths can also be specified as simple file names, in which case they are interpreted relative to the current working directory. The following example is equivalent to the previous one:

>>> pathlib.Path('test.md').read_text()
<the contents of the test.md-file>

The .resolve() method will find the full path. Below, we confirm that the current working directory is used for simple file names:

>>> path = pathlib.Path('test.md')
>>> path.resolve()
PosixPath('/home/gahjelle/realpython/test.md')
>>> path.resolve().parent == pathlib.Path.cwd()
True

Note that when paths are compared, it is their representations that are compared. In the example above, path.parent is not equal to pathlib.Path.cwd(), because path.parent is represented by '.' while pathlib.Path.cwd() is represented by '/home/gahjelle/realpython/'.

Picking Out Components of a Path

The different parts of a path are conveniently available as properties. Basic examples include:

.name: the file name without any directory
.parent: the directory containing the file, or the parent directory if path is a directory
.stem: the file name without the suffix
.suffix: the file extension
.anchor: the part of the path before the directories

Here are these properties in action:

>>> path
PosixPath('/home/gahjelle/realpython/test.md')
>>> path.name
'test.md'
>>> path.stem
'test'
>>> path.suffix
'.md'
>>> path.parent
PosixPath('/home/gahjelle/realpython')
>>> path.parent.parent
PosixPath('/home/gahjelle')
>>> path.anchor
'/'

Note that .parent returns a new Path object, whereas the other properties return strings. This means for instance that .parent can be chained as in the last example or even combined with / to create completely new paths:

>>> path.parent.parent / ('new' + path.suffix)
PosixPath('/home/gahjelle/new.md')

The excellent Pathlib Cheatsheet provides a visual representation of these and other properties and methods.

Moving and Deleting Files

Through pathlib, you also have access to basic file system level operations like moving, updating, and even deleting files. For the most part, these methods do not give a warning or wait for confirmation before information or files are lost. Be careful when using these methods.

To move a file, use .replace(). Note that if the destination already exists, .replace() will overwrite it. Unfortunately, pathlib does not explicitly support safe moving of files. To avoid possibly overwriting the destination path, the simplest is to test whether the destination exists before replacing:

if not destination.exists():
    source.replace(destination)

However, this does leave the door open for a possible race condition. Another process may add a file at the destination path between the execution of the if statement and the .replace() method. If that is a concern, a safer way is to open the destination path for exclusive creation and explicitly copy the source data:

with destination.open(mode='xb') as fid:
    fid.write(source.read_bytes())

The code above will raise a FileExistsError if destination already exists. Technically, this copies a file. To perform a move, simply delete source after the copy is done (see below). Make sure no exception was raised though.

When you are renaming files, useful methods might be .with_name() and .with_suffix(). They both return the original path but with the name or the suffix replaced, respectively.

For instance:

>>> path
PosixPath('/home/gahjelle/realpython/test001.txt')
>>> path.with_suffix('.py')
PosixPath('/home/gahjelle/realpython/test001.py')
>>> path.replace(path.with_suffix('.py'))

Directories and files can be deleted using .rmdir() and .unlink() respectively. (Again, be careful!)

Examples

In this section, you will see some examples of how to use pathlib to deal with simple challenges.

Counting Files

There are a few different ways to list many files. The simplest is the .iterdir() method, which iterates over all files in the given directory. The following example combines .iterdir() with the collections.Counter class to count how many files there are of each filetype in the current directory:

>>> import collections
>>> collections.Counter(p.suffix for p in pathlib.Path.cwd().iterdir())
Counter({'.md': 2, '.txt': 4, '.pdf': 2, '.py': 1})

More flexible file listings can be created with the methods .glob() and .rglob() (recursive glob). For instance, pathlib.Path.cwd().glob('*.txt') returns all files with a .txt suffix in the current directory. The following only counts filetypes starting with p:

>>> import collections
>>> collections.Counter(p.suffix for p in pathlib.Path.cwd().glob('*.p*'))
Counter({'.pdf': 2, '.py': 1})

Display a Directory Tree

The next example defines a function, tree(), that will print a visual tree representing the file hierarchy, rooted at a given directory. Here, we want to list subdirectories as well, so we use the .rglob() method:

def tree(directory):
    print(f'+ {directory}')
    for path in sorted(directory.rglob('*')):
        depth = len(path.relative_to(directory).parts)
        spacer = '    ' * depth
        print(f'{spacer}+ {path.name}')

Note that we need to know how far away from the root directory a file is located. To do this, we first use .relative_to() to represent a path relative to the root directory. Then, we count the number of directories (using the .parts property) in the representation. When run, this function creates a visual tree like the following:

>>> tree(pathlib.Path.cwd())
+ /home/gahjelle/realpython
    + directory_1
        + file_a.md
    + directory_2
        + file_a.md
        + file_b.pdf
        + file_c.py
    + file_1.txt
    + file_2.txt

Note: The f-strings only work in Python 3.6 and later. In older Pythons, the expression f'{spacer}+ {path.name}' can be written '{0}+ {1}'.format(spacer, path.name).

Find the Last Modified File

The .iterdir(), .glob(), and .rglob() methods are great fits for generator expressions and list comprehensions. To find the file in a directory that was last modified, you can use the .stat() method to get information about the underlying files. For instance, .stat().st_mtime gives the time of last modification of a file:

>>> from datetime import datetime
>>> time, file_path = max((f.stat().st_mtime, f) for f in directory.iterdir())
>>> print(datetime.fromtimestamp(time), file_path)
2018-03-23 19:23:56.977817 /home/gahjelle/realpython/test001.txt

You can even get the contents of the file that was last modified with a similar expression:

>>> max((f.stat().st_mtime, f) for f in directory.iterdir())[1].read_text()
<the contents of the last modified file in directory>

The timestamp returned from the different .stat().st_ properties represents seconds since January 1st, 1970. In addition to datetime.fromtimestamp, time.localtime or time.ctime may be used to convert the timestamp to something more usable.

Create a Unique File Name

The last example will show how to construct a unique numbered file name based on a template. First, specify a pattern for the file name, with room for a counter. Then, check the existence of the file path created by joining a directory and the file name (with a value for the counter). If it already exists, increase the counter and try again:

def unique_path(directory, name_pattern):
    counter = 0
    while True:
        counter += 1
        path = directory / name_pattern.format(counter)
        if not path.exists():
            return path

path = unique_path(pathlib.Path.cwd(), 'test{:03d}.txt')

If the directory already contains the files test001.txt and test002.txt, the above code will set path to test003.txt.

Operating System Differences

Earlier, we noted that when we instantiated pathlib.Path, either a WindowsPath or a PosixPath object was returned. The kind of object will depend on the operating system you are using. This feature makes it fairly easy to write cross-platform compatible code. It is possible to ask for a WindowsPath or a PosixPath explicitly, but you will only be limiting your code to that system without any benefits. A concrete path like this can not be used on a different system:

>>> pathlib.WindowsPath('test.md')
NotImplementedError: cannot instantiate 'WindowsPath' on your system

There might be times when you need a representation of a path without access to the underlying file system (in which case it could also make sense to represent a Windows path on a non-Windows system or vice versa). This can be done with PurePath objects. These objects support the operations discussed in the section on Path Components but not the methods that access the file system:

>>> path = pathlib.PureWindowsPath(r'C:\Users\gahjelle\realpython\file.txt')
>>> path.name
'file.txt'
>>> path.parent
PureWindowsPath('C:/Users/gahjelle/realpython')
>>> path.exists()
AttributeError: 'PureWindowsPath' object has no attribute 'exists'

You can directly instantiate PureWindowsPath or PurePosixPath on all systems. Instantiating PurePath will return one of these objects depending on the operating system you are using.

Paths as Proper Objects

In the introduction, we briefly noted that paths are not strings, and one motivation behind pathlib is to represent the file system with proper objects. In fact, the official documentation of pathlib is titled pathlib — Object-oriented filesystem paths. The Object-oriented approach is already quite visible in the examples above (especially if you contrast it with the old os.path way of doing things). However, let me leave you with a few other tidbits.

Independently of the operating system you are using, paths are represented in Posix style, with the forward slash as the path separator. On Windows, you will see something like this:

>>> pathlib.Path(r'C:\Users\gahjelle\realpython\file.txt')
WindowsPath('C:/Users/gahjelle/realpython/file.txt')

Still, when a path is converted to a string, it will use the native form, for instance with backslashes on Windows:

>>> str(pathlib.Path(r'C:\Users\gahjelle\realpython\file.txt'))
'C:\\Users\\gahjelle\\realpython\\file.txt'

This is particularly useful if you are using a library that does not know how to deal with pathlib.Path objects. This is a bigger problem on Python versions before 3.6. For instance, in Python 3.5, the configparser standard library can only use string paths to read files. The way to handle such cases is to do the conversion to a string explicitly:

>>> from configparser import ConfigParser
>>> path = pathlib.Path('config.txt')
>>> cfg = ConfigParser()
>>> cfg.read(path)                     # Error on Python < 3.6
TypeError: 'PosixPath' object is not iterable
>>> cfg.read(str(path))                # Works on Python >= 3.4
['config.txt']

In Python 3.6 and later it is recommended to use os.fspath() instead of str() if you need to do an explicit conversion. This is a little safer as it will raise an error if you accidently try to convert an object that is not pathlike.

Possibly the most unusual part of the pathlib library is the use of the / operator. For a little peek under the hood, let us see how that is implemented. This is an example of operator overloading: the behavior of an operator is changed depending on the context. You have seen this before. Think about how + means different things for strings and numbers. Python implements operator overloading through the use of double underscore methods (a.k.a. dunder methods).

The / operator is defined by the .__truediv__() method. In fact, if you take a look at the source code of pathlib, you’ll see something like:

class PurePath(object):

    def __truediv__(self, key):
        return self._make_child((key,))

Conclusion

Since Python 3.4, pathlib has been available in the standard library. With pathlib, file paths can be represented by proper Path objects instead of plain strings as before. These objects make code dealing with file paths:

Easier to read, especially because / is used to join paths together
More powerful, with most necessary methods and properties available directly on the object
More consistent across operating systems, as peculiarities of the different systems are hidden by the Path object

In this tutorial, you have seen how to create Path objects, read and write files, manipulate paths and the underlying file system, as well as some examples of how to iterate over many file paths.

Free PDF Download: Python 3 Cheat Sheet

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Modules and Packages – An Introduction

2018-04-17T14:00:00+00:00

This article explores Python modules and Python packages, two mechanisms that facilitate modular programming.

Modular programming refers to the process of breaking a large, unwieldy programming task into separate, smaller, more manageable subtasks or modules. Individual modules can then be cobbled together like building blocks to create a larger application.

There are several advantages to modularizing code in a large application:

Simplicity: Rather than focusing on the entire problem at hand, a module typically focuses on one relatively small portion of the problem. If you’re working on a single module, you’ll have a smaller problem domain to wrap your head around. This makes development easier and less error-prone.
Maintainability: Modules are typically designed so that they enforce logical boundaries between different problem domains. If modules are written in a way that minimizes interdependency, there is decreased likelihood that modifications to a single module will have an impact on other parts of the program. (You may even be able to make changes to a module without having any knowledge of the application outside that module.) This makes it more viable for a team of many programmers to work collaboratively on a large application.
Reusability: Functionality defined in a single module can be easily reused (through an appropriately defined interface) by other parts of the application. This eliminates the need to recreate duplicate code.
Scoping: Modules typically define a separate namespace, which helps avoid collisions between identifiers in different areas of a program. (One of the tenets in the Zen of Python is Namespaces are one honking great idea—let’s do more of those!)

Functions, modules and packages are all constructs in Python that promote code modularization.

Free PDF Download: Python 3 Cheat Sheet

Python Modules: Overview

There are actually three different ways to define a module in Python:

A module can be written in Python itself.
A module can be written in C and loaded dynamically at run-time, like the re (regular expression) module.
A built-in module is intrinsically contained in the interpreter, like the itertools module.

A module’s contents are accessed the same way in all three cases: with the import statement.

Here, the focus will mostly be on modules that are written in Python. The cool thing about modules written in Python is that they are exceedingly straightforward to build. All you need to do is create a file that contains legitimate Python code and then give the file a name with a .py extension. That’s it! No special syntax or voodoo is necessary.

For example, suppose you have created a file called mod.py containing the following:

mod.py

s = "If Comrade Napoleon says it, it must be right."
a = [100, 200, 300]

def foo(arg):
    print(f'arg = {arg}')

class Foo:
    pass

Several objects are defined in mod.py:

s (a string)
a (a list)
foo() (a function)
Foo (a class)

Assuming mod.py is in an appropriate location, which you will learn more about shortly, these objects can be accessed by importing the module as follows:

>>> import mod
>>> print(mod.s)
If Comrade Napoleon says it, it must be right.
>>> mod.a
[100, 200, 300]
>>> mod.foo(['quux', 'corge', 'grault'])
arg = ['quux', 'corge', 'grault']
>>> x = mod.Foo()
>>> x
<mod.Foo object at 0x03C181F0>

The Module Search Path

Continuing with the above example, let’s take a look at what happens when Python executes the statement:

import mod

When the interpreter executes the above import statement, it searches for mod.py in a list of directories assembled from the following sources:

The directory from which the input script was run or the current directory if the interpreter is being run interactively
The list of directories contained in the PYTHONPATH environment variable, if it is set. (The format for PYTHONPATH is OS-dependent but should mimic the PATH environment variable.)
An installation-dependent list of directories configured at the time Python is installed

The resulting search path is accessible in the Python variable sys.path, which is obtained from a module named sys:

>>> import sys
>>> sys.path
['', 'C:\\Users\\john\\Documents\\Python\\doc', 'C:\\Python36\\Lib\\idlelib',
'C:\\Python36\\python36.zip', 'C:\\Python36\\DLLs', 'C:\\Python36\\lib',
'C:\\Python36', 'C:\\Python36\\lib\\site-packages']

Note: The exact contents of sys.path are installation-dependent. The above will almost certainly look slightly different on your computer.

Thus, to ensure your module is found, you need to do one of the following:

Put mod.py in the directory where the input script is located or the current directory, if interactive
Modify the PYTHONPATH environment variable to contain the directory where mod.py is located before starting the interpreter
- Or: Put mod.py in one of the directories already contained in the PYTHONPATH variable
Put mod.py in one of the installation-dependent directories, which you may or may not have write-access to, depending on the OS

There is actually one additional option: you can put the module file in any directory of your choice and then modify sys.path at run-time so that it contains that directory. For example, in this case, you could put mod.py in directory C:\Users\john and then issue the following statements:

>>> sys.path.append(r'C:\Users\john')
>>> sys.path
['', 'C:\\Users\\john\\Documents\\Python\\doc', 'C:\\Python36\\Lib\\idlelib',
'C:\\Python36\\python36.zip', 'C:\\Python36\\DLLs', 'C:\\Python36\\lib',
'C:\\Python36', 'C:\\Python36\\lib\\site-packages', 'C:\\Users\\john']
>>> import mod

Once a module has been imported, you can determine the location where it was found with the module’s __file__ attribute:

>>> import mod
>>> mod.__file__
'C:\\Users\\john\\mod.py'

>>> import re
>>> re.__file__
'C:\\Python36\\lib\\re.py'

The directory portion of __file__ should be one of the directories in sys.path.

The `import` Statement

Module contents are made available to the caller with the import statement. The import statement takes many different forms, shown below.

`import <module_name>`

The simplest form is the one already shown above:

import <module_name>

Note that this does not make the module contents directly accessible to the caller. Each module has its own private symbol table, which serves as the global symbol table for all objects defined in the module. Thus, a module creates a separate namespace, as already noted.

The statement import <module_name> only places <module_name> in the caller’s symbol table. The objects that are defined in the module remain in the module’s private symbol table.

From the caller, objects in the module are only accessible when prefixed with <module_name> via dot notation, as illustrated below.

After the following import statement, mod is placed into the local symbol table. Thus, mod has meaning in the caller’s local context:

>>> import mod
>>> mod
<module 'mod' from 'C:\\Users\\john\\Documents\\Python\\doc\\mod.py'>

But s and foo remain in the module’s private symbol table and are not meaningful in the local context:

>>> s
NameError: name 's' is not defined
>>> foo('quux')
NameError: name 'foo' is not defined

To be accessed in the local context, names of objects defined in the module must be prefixed by mod:

>>> mod.s
'If Comrade Napoleon says it, it must be right.'
>>> mod.foo('quux')
arg = quux

Several comma-separated modules may be specified in a single import statement:

import <module_name>[, <module_name> ...]

`from <module_name> import <name(s)>`

An alternate form of the import statement allows individual objects from the module to be imported directly into the caller’s symbol table:

from <module_name> import <name(s)>

Following execution of the above statement, <name(s)> can be referenced in the caller’s environment without the <module_name> prefix:

>>> from mod import s, foo
>>> s
'If Comrade Napoleon says it, it must be right.'
>>> foo('quux')
arg = quux

>>> from mod import Foo
>>> x = Foo()
>>> x
<mod.Foo object at 0x02E3AD50>

Because this form of import places the object names directly into the caller’s symbol table, any objects that already exist with the same name will be overwritten:

>>> a = ['foo', 'bar', 'baz']
>>> a
['foo', 'bar', 'baz']

>>> from mod import a
>>> a
[100, 200, 300]

It is even possible to indiscriminately import everything from a module at one fell swoop:

from <module_name> import *

This will place the names of all objects from <module_name> into the local symbol table, with the exception of any that begin with the underscore (_) character.

For example:

>>> from mod import *
>>> s
'If Comrade Napoleon says it, it must be right.'
>>> a
[100, 200, 300]
>>> foo
<function foo at 0x03B449C0>
>>> Foo
<class 'mod.Foo'>

This isn’t necessarily recommended in large-scale production code. It’s a bit dangerous because you are entering names into the local symbol table en masse. Unless you know them all well and can be confident there won’t be a conflict, you have a decent chance of overwriting an existing name inadvertently. However, this syntax is quite handy when you are just mucking around with the interactive interpreter, for testing or discovery purposes, because it quickly gives you access to everything a module has to offer without a lot of typing.

`from <module_name> import <name> as <alt_name>`

It is also possible to import individual objects but enter them into the local symbol table with alternate names:

from <module_name> import <name> as <alt_name>[, <name> as <alt_name> …]

This makes it possible to place names directly into the local symbol table but avoid conflicts with previously existing names:

>>> s = 'foo'
>>> a = ['foo', 'bar', 'baz']

>>> from mod import s as string, a as alist
>>> s
'foo'
>>> string
'If Comrade Napoleon says it, it must be right.'
>>> a
['foo', 'bar', 'baz']
>>> alist
[100, 200, 300]

`import <module_name> as <alt_name>`

You can also import an entire module under an alternate name:

import <module_name> as <alt_name>

>>> import mod as my_module
>>> my_module.a
[100, 200, 300]
>>> my_module.foo('qux')
arg = qux

Module contents can be imported from within a function definition. In that case, the import does not occur until the function is called:

>>> def bar():
...     from mod import foo
...     foo('corge')
...

>>> bar()
arg = corge

However, Python 3 does not allow the indiscriminate import * syntax from within a function:

>>> def bar():
...     from mod import *
...
SyntaxError: import * only allowed at module level

Lastly, a try statement with an except ImportError clause can be used to guard against unsuccessful import attempts:

>>> try:
...     # Non-existent module
...     import baz
... except ImportError:
...     print('Module not found')
...

Module not found

>>> try:
...     # Existing module, but non-existent object
...     from mod import baz
... except ImportError:
...     print('Object not found in module')
...

Object not found in module

The `dir()` Function

The built-in function dir() returns a list of defined names in a namespace. Without arguments, it produces an alphabetically sorted list of names in the current local symbol table:

>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__']

>>> qux = [1, 2, 3, 4, 5]
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__', 'qux']

>>> class Bar():
...     pass
...
>>> x = Bar()
>>> dir()
['Bar', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__', 'qux', 'x']

Note how the first call to dir() above lists several names that are automatically defined and already in the namespace when the interpreter starts. As new names are defined (qux, Bar, x), they appear on subsequent invocations of dir().

This can be useful for identifying what exactly has been added to the namespace by an import statement:

>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__']

>>> import mod
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__', 'mod']
>>> mod.s
'If Comrade Napoleon says it, it must be right.'
>>> mod.foo([1, 2, 3])
arg = [1, 2, 3]

>>> from mod import a, Foo
>>> dir()
['Foo', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__', 'a', 'mod']
>>> a
[100, 200, 300]
>>> x = Foo()
>>> x
<mod.Foo object at 0x002EAD50>

>>> from mod import s as string
>>> dir()
['Foo', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__', 'a', 'mod', 'string', 'x']
>>> string
'If Comrade Napoleon says it, it must be right.'

When given an argument that is the name of a module, dir() lists the names defined in the module:

>>> import mod
>>> dir(mod)
['Foo', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__',
'__name__', '__package__', '__spec__', 'a', 'foo', 's']

>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__']
>>> from mod import *
>>> dir()
['Foo', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__', 'a', 'foo', 's']

Executing a Module as a Script

Any .py file that contains a module is essentially also a Python script, and there isn’t any reason it can’t be executed like one.

Here again is mod.py as it was defined above:

mod.py

s = "If Comrade Napoleon says it, it must be right."
a = [100, 200, 300]

def foo(arg):
    print(f'arg = {arg}')

class Foo:
    pass

This can be run as a script:

C:\Users\john\Documents>python mod.py
C:\Users\john\Documents>

There are no errors, so it apparently worked. Granted, it’s not very interesting. As it is written, it only defines objects. It doesn’t do anything with them, and it doesn’t generate any output.

Let’s modify the above Python module so it does generate some output when run as a script:

mod.py

s = "If Comrade Napoleon says it, it must be right."
a = [100, 200, 300]

def foo(arg):
    print(f'arg = {arg}')

class Foo:
    pass

print(s)
print(a)
foo('quux')
x = Foo()
print(x)

Now it should be a little more interesting:

C:\Users\john\Documents>python mod.py
If Comrade Napoleon says it, it must be right.
[100, 200, 300]
arg = quux
<__main__.Foo object at 0x02F101D0>

Unfortunately, now it also generates output when imported as a module:

>>> import mod
If Comrade Napoleon says it, it must be right.
[100, 200, 300]
arg = quux
<mod.Foo object at 0x0169AD50>

This is probably not what you want. It isn’t usual for a module to generate output when it is imported.

Wouldn’t it be nice if you could distinguish between when the file is loaded as a module and when it is run as a standalone script?

Ask and ye shall receive.

When a .py file is imported as a module, Python sets the special dunder variable __name__ to the name of the module. However, if a file is run as a standalone script, __name__ is (creatively) set to the string '__main__'. Using this fact, you can discern which is the case at run-time and alter behavior accordingly:

mod.py

s = "If Comrade Napoleon says it, it must be right."
a = [100, 200, 300]

def foo(arg):
    print(f'arg = {arg}')

class Foo:
    pass

if (__name__ == '__main__'):
    print('Executing as standalone script')
    print(s)
    print(a)
    foo('quux')
    x = Foo()
    print(x)

Now, if you run as a script, you get output:

C:\Users\john\Documents>python mod.py
Executing as standalone script
If Comrade Napoleon says it, it must be right.
[100, 200, 300]
arg = quux
<__main__.Foo object at 0x03450690>

But if you import as a module, you don’t:

>>> import mod
>>> mod.foo('grault')
arg = grault

Modules are often designed with the capability to run as a standalone script for purposes of testing the functionality that is contained within the module. This is referred to as unit testing. For example, suppose you have created a module fact.py containing a factorial function, as follows:

fact.py

def fact(n):
    return 1 if n == 1 else n * fact(n-1)

if (__name__ == '__main__'):
    import sys
    if len(sys.argv) > 1:
        print(fact(int(sys.argv[1])))

The file can be treated as a module, and the fact() function imported:

>>> from fact import fact
>>> fact(6)
720

But it can also be run as a standalone by passing an integer argument on the command-line for testing:

C:\Users\john\Documents>python fact.py 6
720

Reloading a Module

For reasons of efficiency, a module is only loaded once per interpreter session. That is fine for function and class definitions, which typically make up the bulk of a module’s contents. But a module can contain executable statements as well, usually for initialization. Be aware that these statements will only be executed the first time a module is imported.

Consider the following file mod.py:

mod.py

a = [100, 200, 300]
print('a =', a)

>>> import mod
a = [100, 200, 300]
>>> import mod
>>> import mod

>>> mod.a
[100, 200, 300]

The print() statement is not executed on subsequent imports. (For that matter, neither is the assignment statement, but as the final display of the value of mod.a shows, that doesn’t matter. Once the assignment is made, it sticks.)

If you make a change to a module and need to reload it, you need to either restart the interpreter or use a function called reload() from module importlib:

>>> import mod
a = [100, 200, 300]

>>> import mod

>>> import importlib
>>> importlib.reload(mod)
a = [100, 200, 300]
<module 'mod' from 'C:\\Users\\john\\Documents\\Python\\doc\\mod.py'>

Python Packages

Suppose you have developed a very large application that includes many modules. As the number of modules grows, it becomes difficult to keep track of them all if they are dumped into one location. This is particularly so if they have similar names or functionality. You might wish for a means of grouping and organizing them.

Packages allow for a hierarchical structuring of the module namespace using dot notation. In the same way that modules help avoid collisions between global variable names, packages help avoid collisions between module names.

Creating a package is quite straightforward, since it makes use of the operating system’s inherent hierarchical file structure. Consider the following arrangement:

Here, there is a directory named pkg that contains two modules, mod1.py and mod2.py. The contents of the modules are:

mod1.py

def foo():
    print('[mod1] foo()')

class Foo:
    pass

mod2.py

def bar():
    print('[mod2] bar()')

class Bar:
    pass

Given this structure, if the pkg directory resides in a location where it can be found (in one of the directories contained in sys.path), you can refer to the two modules with dot notation (pkg.mod1, pkg.mod2) and import them with the syntax you are already familiar with:

import <module_name>[, <module_name> ...]

>>> import pkg.mod1, pkg.mod2
>>> pkg.mod1.foo()
[mod1] foo()
>>> x = pkg.mod2.Bar()
>>> x
<pkg.mod2.Bar object at 0x033F7290>

from <module_name> import <name(s)>

>>> from pkg.mod1 import foo
>>> foo()
[mod1] foo()

from <module_name> import <name> as <alt_name>

>>> from pkg.mod2 import Bar as Qux
>>> x = Qux()
>>> x
<pkg.mod2.Bar object at 0x036DFFD0>

You can import modules with these statements as well:

from <package_name> import <modules_name>[, <module_name> ...]
from <package_name> import <module_name> as <alt_name>

>>> from pkg import mod1
>>> mod1.foo()
[mod1] foo()

>>> from pkg import mod2 as quux
>>> quux.bar()
[mod2] bar()

You can technically import the package as well:

>>> import pkg
>>> pkg
<module 'pkg' (namespace)>

But this is of little avail. Though this is, strictly speaking, a syntactically correct Python statement, it doesn’t do much of anything useful. In particular, it does not place any of the modules in pkg into the local namespace:

>>> pkg.mod1
Traceback (most recent call last):
  File "<pyshell#34>", line 1, in <module>
    pkg.mod1
AttributeError: module 'pkg' has no attribute 'mod1'
>>> pkg.mod1.foo()
Traceback (most recent call last):
  File "<pyshell#35>", line 1, in <module>
    pkg.mod1.foo()
AttributeError: module 'pkg' has no attribute 'mod1'
>>> pkg.mod2.Bar()
Traceback (most recent call last):
  File "<pyshell#36>", line 1, in <module>
    pkg.mod2.Bar()
AttributeError: module 'pkg' has no attribute 'mod2'

To actually import the modules or their contents, you need to use one of the forms shown above.

Package Initialization

If a file named __init__.py is present in a package directory, it is invoked when the package or a module in the package is imported. This can be used for execution of package initialization code, such as initialization of package-level data.

For example, consider the following __init__.py file:

__init__.py

print(f'Invoking __init__.py for {__name__}')
A = ['quux', 'corge', 'grault']

Let’s add this file to the pkg directory from the above example:

Now when the package is imported, global list A is initialized:

>>> import pkg
Invoking __init__.py for pkg
>>> pkg.A
['quux', 'corge', 'grault']

A module in the package can access the global by importing it in turn:

mod1.py

def foo():
    from pkg import A
    print('[mod1] foo() / A = ', A)

class Foo:
    pass

>>> from pkg import mod1
Invoking __init__.py for pkg
>>> mod1.foo()
[mod1] foo() / A =  ['quux', 'corge', 'grault']

__init__.py can also be used to effect automatic importing of modules from a package. For example, earlier you saw that the statement import pkg only places the name pkg in the caller’s local symbol table and doesn’t import any modules. But if __init__.py in the pkg directory contains the following:

__init__.py

print(f'Invoking __init__.py for {__name__}')
import pkg.mod1, pkg.mod2

then when you execute import pkg, modules mod1 and mod2 are imported automatically:

>>> import pkg
Invoking __init__.py for pkg
>>> pkg.mod1.foo()
[mod1] foo()
>>> pkg.mod2.bar()
[mod2] bar()

Note: Much of the Python documentation states that an __init__.py file must be present in the package directory when creating a package. This was once true. It used to be that the very presence of __init__.py signified to Python that a package was being defined. The file could contain initialization code or even be empty, but it had to be present.

Starting with Python 3.3, Implicit Namespace Packages were introduced. These allow for the creation of a package without any __init__.py file. Of course, it can still be present if package initialization is needed. But it is no longer required.

Importing `*` From a Package

For the purposes of the following discussion, the previously defined package is expanded to contain some additional modules:

There are now four modules defined in the pkg directory. Their contents are as shown below:

mod1.py

def foo():
    print('[mod1] foo()')

class Foo:
    pass

mod2.py

def bar():
    print('[mod2] bar()')

class Bar:
    pass

mod3.py

def baz():
    print('[mod3] baz()')

class Baz:
    pass

mod4.py

def qux():
    print('[mod4] qux()')

class Qux:
    pass

(Imaginative, aren’t they?)

You have already seen that when import * is used for a module, all objects from the module are imported into the local symbol table, except those whose names begin with an underscore, as always:

>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__']

>>> from pkg.mod3 import *

>>> dir()
['Baz', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__', 'baz']
>>> baz()
[mod3] baz()
>>> Baz
<class 'pkg.mod3.Baz'>

The analogous statement for a package is this:

from <package_name> import *

What does that do?

>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__']

>>> from pkg import *
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__']

Hmph. Not much. You might have expected (assuming you had any expectations at all) that Python would dive down into the package directory, find all the modules it could, and import them all. But as you can see, by default that is not what happens.

Instead, Python follows this convention: if the __init__.py file in the package directory contains a list named __all__, it is taken to be a list of modules that should be imported when the statement from <package_name> import * is encountered.

For the present example, suppose you create an __init__.py in the pkg directory like this:

pkg/__init__.py

__all__ = [
        'mod1',
        'mod2',
        'mod3',
        'mod4'
        ]

Now from pkg import * imports all four modules:

>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__']

>>> from pkg import *
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__', 'mod1', 'mod2', 'mod3', 'mod4']
>>> mod2.bar()
[mod2] bar()
>>> mod4.Qux
<class 'pkg.mod4.Qux'>

Using import * still isn’t considered terrific form, any more for packages than for modules. But this facility at least gives the creator of the package some control over what happens when import * is specified. (In fact, it provides the capability to disallow it entirely, simply by declining to define __all__ at all. As you have seen, the default behavior for packages is to import nothing.)

By the way, __all__ can be defined in a module as well and serves the same purpose: to control what is imported with import *. For example, modify mod1.py as follows:

pkg/mod1.py

__all__ = ['foo']

def foo():
    print('[mod1] foo()')

class Foo:
    pass

Now an import * statement from pkg.mod1 will only import what is contained in __all__:

>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__']

>>> from pkg.mod1 import *
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__', 'foo']

>>> foo()
[mod1] foo()
>>> Foo
Traceback (most recent call last):
  File "<pyshell#37>", line 1, in <module>
    Foo
NameError: name 'Foo' is not defined

foo() (the function) is now defined in the local namespace, but Foo (the class) is not, because the latter is not in __all__.

In summary, __all__ is used by both packages and modules to control what is imported when import * is specified. But the default behavior differs:

For a package, when __all__ is not defined, import * does not import anything.
For a module, when __all__ is not defined, import * imports everything (except—you guessed it—names starting with an underscore).

Subpackages

Packages can contain nested subpackages to arbitrary depth. For example, let’s make one more modification to the example package directory as follows:

The four modules (mod1.py, mod2.py, mod3.py and mod4.py) are defined as previously. But now, instead of being lumped together into the pkg directory, they are split out into two subpackage directories, sub_pkg1 and sub_pkg2.

Importing still works the same as shown previously. Syntax is similar, but additional dot notation is used to separate package name from subpackage name:

>>> import pkg.sub_pkg1.mod1
>>> pkg.sub_pkg1.mod1.foo()
[mod1] foo()

>>> from pkg.sub_pkg1 import mod2
>>> mod2.bar()
[mod2] bar()

>>> from pkg.sub_pkg2.mod3 import baz
>>> baz()
[mod3] baz()

>>> from pkg.sub_pkg2.mod4 import qux as grault
>>> grault()
[mod4] qux()

In addition, a module in one subpackage can reference objects in a sibling subpackage (in the event that the sibling contains some functionality that you need). For example, suppose you want to import and execute function foo() (defined in module mod1) from within module mod3. You can either use an absolute import:

pkg/sub__pkg2/mod3.py

def baz():
    print('[mod3] baz()')

class Baz:
    pass

from pkg.sub_pkg1.mod1 import foo
foo()

>>> from pkg.sub_pkg2 import mod3
[mod1] foo()
>>> mod3.foo()
[mod1] foo()

Or you can use a relative import, where .. refers to the package one level up. From within mod3.py, which is in subpackage sub_pkg2,

.. evaluates to the parent package (pkg), and
..sub_pkg1 evaluates to subpackage sub_pkg1 of the parent package.

pkg/sub__pkg2/mod3.py

def baz():
    print('[mod3] baz()')

class Baz:
    pass

from .. import sub_pkg1
print(sub_pkg1)

from ..sub_pkg1.mod1 import foo
foo()

>>> from pkg.sub_pkg2 import mod3
<module 'pkg.sub_pkg1' (namespace)>
[mod1] foo()

Conclusion

In this tutorial, you covered the following topics:

How to create a Python module
Locations where the Python interpreter searches for a module
How to obtain access to the objects defined in a module with the import statement
How to create a module that is executable as a standalone script
How to organize modules into packages and subpackages
How to control package initialization

Free PDF Download: Python 3 Cheat Sheet

This will hopefully allow you to better understand how to gain access to the functionality available in the many third-party and built-in modules available in Python.

Additionally, if you are developing your own application, creating your own modules and packages will help you organize and modularize your code, which makes coding, maintenance, and debugging easier.

If you want to learn more, check out the following documentation at Python.org:

Happy Pythoning!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Working With JSON Data in Python

2018-04-16T14:00:00+00:00

Since its inception, JSON has quickly become the de facto standard for information exchange. Chances are you’re here because you need to transport some data from here to there. Perhaps you’re gathering information through an API or storing your data in a document database. One way or another, you’re up to your neck in JSON, and you’ve got to Python your way out.

Luckily, this is a pretty common task, and—as with most common tasks—Python makes it almost disgustingly easy. Have no fear, fellow Pythoneers and Pythonistas. This one’s gonna be a breeze!

So, we use JSON to store and exchange data? Yup, you got it! It’s nothing more than a standardized format the community uses to pass data around. Keep in mind, JSON isn’t the only format available for this kind of work, but XML and YAML are probably the only other ones worth mentioning in the same breath.

Free PDF Download: Python 3 Cheat Sheet

A (Very) Brief History of JSON

Not so surprisingly, JavaScript Object Notation was inspired by a subset of the JavaScript programming language dealing with object literal syntax. They’ve got a nifty website that explains the whole thing. Don’t worry though: JSON has long since become language agnostic and exists as its own standard, so we can thankfully avoid JavaScript for the sake of this discussion.

Ultimately, the community at large adopted JSON because it’s easy for both humans and machines to create and understand.

Look, it’s JSON!

Get ready. I’m about to show you some real life JSON—just like you’d see out there in the wild. It’s okay: JSON is supposed to be readable by anyone who’s used a C-style language, and Python is a C-style language…so that’s you!

{
    "firstName": "Jane",
    "lastName": "Doe",
    "hobbies": ["running", "sky diving", "singing"],
    "age": 35,
    "children": [
        {
            "firstName": "Alice",
            "age": 6
        },
        {
            "firstName": "Bob",
            "age": 8
        }
    ]
}

As you can see, JSON supports primitive types, like strings and numbers, as well as nested lists and objects.

Wait, that looks like a Python dictionary! I know, right? It’s pretty much universal object notation at this point, but I don’t think UON rolls off the tongue quite as nicely. Feel free to discuss alternatives in the comments.

Whew! You survived your first encounter with some wild JSON. Now you just need to learn how to tame it.

Python Supports JSON Natively!

Python comes with a built-in package called json for encoding and decoding JSON data.

Just throw this little guy up at the top of your file:

import json

A Little Vocabulary

The process of encoding JSON is usually called serialization. This term refers to the transformation of data into a series of bytes (hence serial) to be stored or transmitted across a network. You may also hear the term marshaling, but that’s a whole other discussion. Naturally, deserialization is the reciprocal process of decoding data that has been stored or delivered in the JSON standard.

Yikes! That sounds pretty technical. Definitely. But in reality, all we’re talking about here is reading and writing. Think of it like this: encoding is for writing data to disk, while decoding is for reading data into memory.

Serializing JSON

What happens after a computer processes lots of information? It needs to take a data dump. Accordingly, the json library exposes the dump() method for writing data to files. There is also a dumps() method (pronounced as “dump-s”) for writing to a Python string.

Simple Python objects are translated to JSON according to a fairly intuitive conversion.

Python	JSON
`dict`	`object`
`list`, `tuple`	`array`
`str`	`string`
`int`, `long`, `float`	`number`
`True`	`true`
`False`	`false`
`None`	`null`

A Simple Serialization Example

Imagine you’re working with a Python object in memory that looks a little something like this:

data = {
    "president": {
        "name": "Zaphod Beeblebrox",
        "species": "Betelgeusian"
    }
}

It is critical that you save this information to disk, so your mission is to write it to a file.

Using Python’s context manager, you can create a file called data_file.json and open it in write mode. (JSON files conveniently end in a .json extension.)

with open("data_file.json", "w") as write_file:
    json.dump(data, write_file)

Note that dump() takes two positional arguments: (1) the data object to be serialized, and (2) the file-like object to which the bytes will be written.

Or, if you were so inclined as to continue using this serialized JSON data in your program, you could write it to a native Python str object.

json_string = json.dumps(data)

Notice that the file-like object is absent since you aren’t actually writing to disk. Other than that, dumps() is just like dump().

Hooray! You’ve birthed some baby JSON, and you’re ready to release it out into the wild to grow big and strong.

Some Useful Keyword Arguments

Remember, JSON is meant to be easily readable by humans, but readable syntax isn’t enough if it’s all squished together. Plus you’ve probably got a different programming style than me, and it might be easier for you to read code when it’s formatted to your liking.

NOTE: Both the dump() and dumps() methods use the same keyword arguments.

The first option most people want to change is whitespace. You can use the indent keyword argument to specify the indentation size for nested structures. Check out the difference for yourself by using data, which we defined above, and running the following commands in a console:

>>> json.dumps(data)
>>> json.dumps(data, indent=4)

Another formatting option is the separators keyword argument. By default, this is a 2-tuple of the separator strings (", ", ": "), but a common alternative for compact JSON is (",", ":"). Take a look at the sample JSON again to see where these separators come into play.

There are others, like sort_keys, but I have no idea what that one does. You can find a whole list in the docs if you’re curious.

Deserializing JSON

Great, looks like you’ve captured yourself some wild JSON! Now it’s time to whip it into shape. In the json library, you’ll find load() and loads() for turning JSON encoded data into Python objects.

Just like serialization, there is a simple conversion table for deserialization, though you can probably guess what it looks like already.

JSON	Python
`object`	`dict`
`array`	`list`
`string`	`str`
`number` (int)	`int`
`number` (real)	`float`
`true`	`True`
`false`	`False`
`null`	`None`

Technically, this conversion isn’t a perfect inverse to the serialization table. That basically means that if you encode an object now and then decode it again later, you may not get exactly the same object back. I imagine it’s a bit like teleportation: break my molecules down over here and put them back together over there. Am I still the same person?

In reality, it’s probably more like getting one friend to translate something into Japanese and another friend to translate it back into English. Regardless, the simplest example would be encoding a tuple and getting back a list after decoding, like so:

>>> blackjack_hand = (8, "Q")
>>> encoded_hand = json.dumps(blackjack_hand)
>>> decoded_hand = json.loads(encoded_hand)

>>> blackjack_hand == decoded_hand
False
>>> type(blackjack_hand)
<class 'tuple'>
>>> type(decoded_hand)
<class 'list'>
>>> blackjack_hand == tuple(decoded_hand)
True

A Simple Deserialization Example

This time, imagine you’ve got some data stored on disk that you’d like to manipulate in memory. You’ll still use the context manager, but this time you’ll open up the existing data_file.json in read mode.

with open("data_file.json", "r") as read_file:
    data = json.load(read_file)

Things are pretty straightforward here, but keep in mind that the result of this method could return any of the allowed data types from the conversion table. This is only important if you’re loading in data you haven’t seen before. In most cases, the root object will be a dict or a list.

If you’ve pulled JSON data in from another program or have otherwise obtained a string of JSON formatted data in Python, you can easily deserialize that with loads(), which naturally loads from a string:

json_string = """
{
    "researcher": {
        "name": "Ford Prefect",
        "species": "Betelgeusian",
        "relatives": [
            {
                "name": "Zaphod Beeblebrox",
                "species": "Betelgeusian"
            }
        ]
    }
}
"""
data = json.loads(json_string)

Voilà! You’ve tamed the wild JSON, and now it’s under your control. But what you do with that power is up to you. You could feed it, nurture it, and even teach it tricks. It’s not that I don’t trust you…but keep it on a leash, okay?

A Real World Example (sort of)

For your introductory example, you’ll use JSONPlaceholder, a great source of fake JSON data for practice purposes.

First create a script file called scratch.py, or whatever you want. I can’t really stop you.

You’ll need to make an API request to the JSONPlaceholder service, so just use the requests package to do the heavy lifting. Add these imports at the top of your file:

import json
import requests

Now, you’re going to be working with a list of TODOs cuz like…you know, it’s a rite of passage or whatever.

Go ahead and make a request to the JSONPlaceholder API for the /todos endpoint. If you’re unfamiliar with requests, there’s actually a handy json() method that will do all of the work for you, but you can practice using the json library to deserialize the text attribute of the response object. It should look something like this:

response = requests.get("https://jsonplaceholder.typicode.com/todos")
todos = json.loads(response.text)

You don’t believe this works? Fine, run the file in interactive mode and test it for yourself. While you’re at it, check the type of todos. If you’re feeling adventurous, take a peek at the first 10 or so items in the list.

>>> todos == response.json()
True
>>> type(todos)
<class 'list'>
>>> todos[:10]
...

See, I wouldn’t lie to you, but I’m glad you’re a skeptic.

What’s interactive mode? Ah, I thought you’d never ask! You know how you’re always jumping back and forth between the your editor and the terminal? Well, us sneaky Pythoneers use the -i interactive flag when we run the script. This is a great little trick for testing code because it runs the script and then opens up an interactive command prompt with access to all the data from the script!

All right, time for some action. You can see the structure of the data by visiting the endpoint in a browser, but here’s a sample TODO:

{
    "userId": 1,
    "id": 1,
    "title": "delectus aut autem",
    "completed": false
}

There are multiple users, each with a unique userId, and each task has a Boolean completed property. Can you determine which users have completed the most tasks?

# Map of userId to number of complete TODOs for that user
todos_by_user = {}

# Increment complete TODOs count for each user.
for todo in todos:
    if todo["completed"]:
        try:
            # Increment the existing user's count.
            todos_by_user[todo["userId"]] += 1
        except KeyError:
            # This user has not been seen. Set their count to 1.
            todos_by_user[todo["userId"]] = 1

# Create a sorted list of (userId, num_complete) pairs.
top_users = sorted(todos_by_user.items(), 
                   key=lambda x: x[1], reverse=True)

# Get the maximum number of complete TODOs.
max_complete = top_users[0][1]

# Create a list of all users who have completed
# the maximum number of TODOs.
users = []
for user, num_complete in top_users:
    if num_complete < max_complete:
        break
    users.append(str(user))

max_users = " and ".join(users)

Yeah, yeah, your implementation is better, but the point is, you can now manipulate the JSON data as a normal Python object!

I don’t know about you, but when I run the script interactively again, I get the following results:

>>> s = "s" if len(users) > 1 else ""
>>> print(f"user{s} {max_users} completed {max_complete} TODOs")
users 5 and 10 completed 12 TODOs

That’s cool and all, but you’re here to learn about JSON. For your final task, you’ll create a JSON file that contains the completed TODOs for each of the users who completed the maximum number of TODOs.

All you need to do is filter todos and write the resulting list to a file. For the sake of originality, you can call the output file filtered_data_file.json. There are may ways you could go about this, but here’s one:

# Define a function to filter out completed TODOs 
# of users with max completed TODOS.
def keep(todo):
    is_complete = todo["completed"]
    has_max_count = todo["userId"] in users
    return is_complete and has_max_count

# Write filtered TODOs to file.
with open("filtered_data_file.json", "w") as data_file:
    filtered_todos = list(filter(keep, todos))
    json.dump(filtered_todos, data_file, indent=2)

Perfect, you’ve gotten rid of all the data you don’t need and saved the good stuff to a brand new file! Run the script again and check out filtered_data_file.json to verify everything worked. It’ll be in the same directory as scratch.py when you run it.

Now that you’ve made it this far, I bet you’re feeling like some pretty hot stuff, right? Don’t get cocky: humility is a virtue. I am inclined to agree with you though. So far, it’s been smooth sailing, but you might want to batten down the hatches for this last leg of the journey.

Encoding and Decoding Custom Python Objects

What happens when we try to serialize the Elf class from that Dungeons & Dragons app you’re working on?

class Elf:
    def __init__(self, level, ability_scores=None):
        self.level = level
        self.ability_scores = {
            "str": 11, "dex": 12, "con": 10,
            "int": 16, "wis": 14, "cha": 13
        } if ability_scores is None else ability_scores
        self.hp = 10 + self.ability_scores["con"]

Not so surprisingly, Python complains that Elf isn’t serializable (which you’d know if you’ve ever tried to tell an Elf otherwise):

>>> elf = Elf(level=4)
>>> json.dumps(elf)
TypeError: Object of type 'Elf' is not JSON serializable

Although the json module can handle most built-in Python types, it doesn’t understand how to encode customized data types by default. It’s like trying to fit a square peg in a round hole—you need a buzzsaw and parental supervision.

Simplifying Data Structures

Now, the question is how to deal with more complex data structures. Well, you could try to encode and decode the JSON by hand, but there’s a slightly more clever solution that’ll save you some work. Instead of going straight from the custom data type to JSON, you can throw in an intermediary step.

All you need to do is represent your data in terms of the built-in types json already understands. Essentially, you translate the more complex object into a simpler representation, which the json module then translates into JSON. It’s like the transitive property in mathematics: if A = B and B = C, then A = C.

To get the hang of this, you’ll need a complex object to play with. You could use any custom class you like, but Python has a built-in type called complex for representing complex numbers, and it isn’t serializable by default. So, for the sake of these examples, your complex object is going to be a complex object. Confused yet?

>>> z = 3 + 8j
>>> type(z)
<class 'complex'>
>>> json.dumps(z)
TypeError: Object of type 'complex' is not JSON serializable

Where do complex numbers come from? You see, when a real number and an imaginary number love each other very much, they add together to produce a number which is (justifiably) called complex.

A good question to ask yourself when working with custom types is What is the minimum amount of information necessary to recreate this object? In the case of complex numbers, you only need to know the real and imaginary parts, both of which you can access as attributes on the complex object:

>>> z.real
3.0
>>> z.imag
8.0

Passing the same numbers into a complex constructor is enough to satisfy the __eq__ comparison operator:

>>> complex(3, 8) == z
True

Breaking custom data types down into their essential components is critical to both the serialization and deserialization processes.

Encoding Custom Types

To translate a custom object into JSON, all you need to do is provide an encoding function to the dump() method’s default parameter. The json module will call this function on any objects that aren’t natively serializable. Here’s a simple decoding function you can use for practice:

def encode_complex(z):
    if isinstance(z, complex):
        return (z.real, z.imag)
    else:
        type_name = z.__class__.__name__
        raise TypeError(f"Object of type '{type_name}' is not JSON serializable")

Notice that you’re expected to raise a TypeError if you don’t get the kind of object you were expecting. This way, you avoid accidentally serializing any Elves. Now you can try encoding complex objects for yourself!

>>> json.dumps(9 + 5j, default=encode_complex)
'[9.0, 5.0]'
>>> json.dumps(elf, default=encode_complex)
TypeError: Object of type 'Elf' is not JSON serializable

Why did we encode the complex number as a tuple? Great question! That certainly wasn’t the only choice, nor is it necessarily the best choice. In fact, this wouldn’t be a very good representation if you ever wanted to decode the object later, as you’ll see shortly.

The other common approach is to subclass the standard JSONEncoder and override its default() method:

class ComplexEncoder(json.JSONEncoder):
    def default(self, z):
        if isinstance(z, complex):
            return (z.real, z.imag)
        else:
            super().default(self, z)

Instead of raising the TypeError yourself, you can simply let the base class handle it. You can use this either directly in the dump() method via the cls parameter or by creating an instance of the encoder and calling its encode() method:

>>> json.dumps(2 + 5j, cls=ComplexEncoder)
'[2.0, 5.0]'

>>> encoder = ComplexEncoder()
>>> encoder.encode(3 + 6j)
'[3.0, 6.0]'

Decoding Custom Types

While the real and imaginary parts of a complex number are absolutely necessary, they are actually not quite sufficient to recreate the object. This is what happens when you try encoding a complex number with the ComplexEncoder and then decoding the result:

>>> complex_json = json.dumps(4 + 17j, cls=ComplexEncoder)
>>> json.loads(complex_json)
[4.0, 17.0]

All you get back is a list, and you’d have to pass the values into a complex constructor if you wanted that complex object again. Recall our discussion about teleportation. What’s missing is metadata, or information about the type of data you’re encoding.

I suppose the question you really ought ask yourself is What is the minimum amount of information that is both necessary and sufficient to recreate this object?

The json module expects all custom types to be expressed as objects in the JSON standard. For variety, you can create a JSON file this time called complex_data.json and add the following object representing a complex number:

{
    "__complex__": true,
    "real": 42,
    "imag": 36
}

See the clever bit? That "__complex__" key is the metadata we just talked about. It doesn’t really matter what the associated value is. To get this little hack to work, all you need to do is verify that the key exists:

def decode_complex(dct):
    if "__complex__" in dct:
        return complex(dct["real"], dct["imag"])
    return dct

If "__complex__" isn’t in the dictionary, you can just return the object and let the default decoder deal with it.

Every time the load() method attempts to parse an object, you are given the opportunity to intercede before the default decoder has its way with the data. You can do this by passing your decoding function to the object_hook parameter.

Now play the same kind of game as before:

>>> with open("complex_data.json") as complex_data:
...     data = complex_data.read()
...     z = json.loads(data, object_hook=decode_complex)
... 
>>> type(z)
<class 'complex'>

While object_hook might feel like the counterpart to the dump() method’s default parameter, the analogy really begins and ends there.

This doesn’t just work with one object either. Try putting this list of complex numbers into complex_data.json and running the script again:

[
  {
    "__complex__":true,
    "real":42,
    "imag":36
  },
  {
    "__complex__":true,
    "real":64,
    "imag":11
  }
]

If all goes well, you’ll get a list of complex objects:

>>> with open("complex_data.json") as complex_data:
...     data = complex_data.read()
...     numbers = json.loads(data, object_hook=decode_complex)
... 
>>> numbers
[(42+36j), (64+11j)]

You could also try subclassing JSONDecoder and overriding object_hook, but it’s better to stick with the lightweight solution whenever possible.

All done!

Congratulations, you can now wield the mighty power of JSON for any and all of your ~~nefarious~~ Python needs.

While the examples you’ve worked with here are certainly contrived and overly simplistic, they illustrate a workflow you can apply to more general tasks:

Import the json package.
Read the data with load() or loads().
Process the data.
Write the altered data with dump() or dumps().

What you do with your data once it’s been loaded into memory will depend on your use case. Generally, your goal will be gathering data from a source, extracting useful information, and passing that information along or keeping a record of it.

Today you took a journey: you captured and tamed some wild JSON, and you made it back in time for supper! As an added bonus, learning the json package will make learning pickle and marshal a snap.

Good luck with all of your future Pythonic endeavors!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Look Ma, No For-Loops: Array Programming With NumPy

2018-04-10T14:00:00+00:00

It is sometimes said that Python, compared to low-level languages such as C++, improves development time at the expense of runtime. Fortunately, there are a handful of ways to speed up operation runtime in Python without sacrificing ease of use. One option suited for fast numerical operations is NumPy, which deservedly bills itself as the fundamental package for scientific computing with Python.

Granted, few people would categorize something that takes 50 microseconds (fifty millionths of a second) as “slow.” However, computers might beg to differ. The runtime of an operation taking 50 microseconds (50 μs) falls under the realm of microperformance, which can loosely be defined as operations with a runtime between 1 microsecond and 1 millisecond.

Why does speed matter? The reason that microperformance is worth monitoring is that small differences in runtime become amplified with repeated function calls: an incremental 50 μs of overhead, repeated over 1 million function calls, translates to 50 seconds of incremental runtime.

When it comes to computation, there are really three concepts that lend NumPy its power:

Vectorization
Broadcasting
Indexing

In this tutorial, you’ll see step by step how to take advantage of vectorization and broadcasting, so that you can use NumPy to its full capacity. While you will use some indexing in practice here, NumPy’s complete indexing schematics, which extend Python’s slicing syntax, are their own beast. If you’re looking to read more on NumPy indexing, grab some coffee and head to the Indexing section in the NumPy docs.

Free Bonus: Click here to get access to a free NumPy Resources Guide that points you to the best tutorials, videos, and books for improving your NumPy skills.

Getting into Shape: Intro to NumPy Arrays

The fundamental object of NumPy is its ndarray (or numpy.array), an n-dimensional array that is also present in some form in array-oriented languages such as Fortran 90, R, and MATLAB, as well as predecessors APL and J.

Let’s start things off by forming a 3-dimensional array with 36 elements:

>>> import numpy as np

>>> arr = np.arange(36).reshape(3, 4, 3)
>>> arr
array([[[ 0,  1,  2],
        [ 3,  4,  5],
        [ 6,  7,  8],
        [ 9, 10, 11]],

       [[12, 13, 14],
        [15, 16, 17],
        [18, 19, 20],
        [21, 22, 23]],

       [[24, 25, 26],
        [27, 28, 29],
        [30, 31, 32],
        [33, 34, 35]]])

Picturing high-dimensional arrays in two dimensions can be difficult. One intuitive way to think about an array’s shape is to simply “read it from left to right.” arr is a 3 by 4 by 3 array:

>>> arr.shape
(3, 4, 3)

Visually, arr could be thought of as a container of three 4x3 grids (or a rectangular prism) and would look like this:

Higher dimensional arrays can be tougher to picture, but they will still follow this “arrays within an array” pattern.

Where might you see data with greater than two dimensions?

Panel data can be represented in three dimensions. Data that tracks attributes of a cohort (group) of individuals over time could be structured as (respondents, dates, attributes). The 1979 National Longitudinal Survey of Youth follows 12,686 respondents over 27 years. Assuming that you have ~500 directly asked or derived data points per individual, per year, this data would have shape (12686, 27, 500) for a total of 177,604,000 data points.
Color-image data for multiple images is typically stored in four dimensions. Each image is a three-dimensional array of (height, width, channels), where the channels are usually red, green, and blue (RGB) values. A collection of images is then just (image_number, height, width, channels). One thousand 256x256 RGB images would have shape (1000, 256, 256, 3). (An extended representation is RGBA, where the A–alpha–denotes the level of opacity.)

For more detail on real-world examples of high-dimensional data, see Chapter 2 of François Chollet’s Deep Learning with Python.

What is Vectorization?

Vectorization is a powerful ability within NumPy to express operations as occurring on entire arrays rather than their individual elements. Here’s a concise definition from Wes McKinney:

This practice of replacing explicit loops with array expressions is commonly referred to as vectorization. In general, vectorized array operations will often be one or two (or more) orders of magnitude faster than their pure Python equivalents, with the biggest impact [seen] in any kind of numerical computations. [source]

When looping over an array or any data structure in Python, there’s a lot of overhead involved. Vectorized operations in NumPy delegate the looping internally to highly optimized C and Fortran functions, making for cleaner and faster Python code.

Counting: Easy as 1, 2, 3…

As an illustration, consider a 1-dimensional vector of True and False for which you want to count the number of “False to True” transitions in the sequence:

>>> np.random.seed(444)

>>> x = np.random.choice([False, True], size=100000)
>>> x
array([ True, False,  True, ...,  True, False,  True])

With a Python for-loop, one way to do this would be to evaluate, in pairs, the truth value of each element in the sequence along with the element that comes right after it:

>>> def count_transitions(x) -> int:
...     count = 0
...     for i, j in zip(x[:-1], x[1:]):
...         if j and not i:
...             count += 1
...     return count
...
>>> count_transitions(x)
24984

In vectorized form, there’s no explicit for-loop or direct reference to the individual elements:

>>> np.count_nonzero(x[:-1] < x[1:])
24984

How do these two equivalent functions compare in terms of performance? In this particular case, the vectorized NumPy call wins out by a factor of about 70 times:

>>> from timeit import timeit
>>> setup = 'from __main__ import count_transitions, x; import numpy as np'
>>> num = 1000
>>> t1 = timeit('count_transitions(x)', setup=setup, number=num)
>>> t2 = timeit('np.count_nonzero(x[:-1] < x[1:])', setup=setup, number=num)
>>> print('Speed difference: {:0.1f}x'.format(t1 / t2))
Speed difference: 71.0x

Technical Detail: Another term is vector processor, which is related to a computer’s hardware. When I speak about vectorization here, I’m referring to concept of replacing explicit for-loops with array expressions, which in this case can then be computed internally with a low-level language.

Buy Low, Sell High

Here’s another example to whet your appetite. Consider the following classic technical interview problem:

Given a stock’s price history as a sequence, and assuming that you are only allowed to make one purchase and one sale, what is the maximum profit that can be obtained? For example, given prices = (20, 18, 14, 17, 20, 21, 15), the max profit would be 7, from buying at 14 and selling at 21.

(To all of you finance people: no, short-selling is not allowed.)

There is a solution with n-squared time complexity that consists of taking every combination of two prices where the second price “comes after” the first and determining the maximum difference.

However, there is also an O(n) solution that consists of iterating through the sequence just once and finding the difference between each price and a running minimum. It goes something like this:

>>> def profit(prices):
...     max_px = 0
...     min_px = prices[0]
...     for px in prices[1:]:
...         min_px = min(min_px, px)
...         max_px = max(px - min_px, max_px)
...     return max_px

>>> prices = (20, 18, 14, 17, 20, 21, 15)
>>> profit(prices)
7

Can this be done in NumPy? You bet. But first, let’s build a quasi-realistic example:

# Create mostly NaN array with a few 'turning points' (local min/max).
>>> prices = np.full(100, fill_value=np.nan)
>>> prices[[0, 25, 60, -1]] = [80., 30., 75., 50.]

# Linearly interpolate the missing values and add some noise.
>>> x = np.arange(len(prices))
>>> is_valid = ~np.isnan(prices)
>>> prices = np.interp(x=x, xp=x[is_valid], fp=prices[is_valid])
>>> prices += np.random.randn(len(prices)) * 2

Here’s what this looks like with matplotlib. The adage is to buy low (green) and sell high (red):

>>> import matplotlib.pyplot as plt

# Warning! This isn't a fully correct solution, but it works for now.
# If the absolute min came after the absolute max, you'd have trouble.
>>> mn = np.argmin(prices)
>>> mx = mn + np.argmax(prices[mn:])
>>> kwargs = {'markersize': 12, 'linestyle': ''}

>>> fig, ax = plt.subplots()
>>> ax.plot(prices)
>>> ax.set_title('Price History')
>>> ax.set_xlabel('Time')
>>> ax.set_ylabel('Price')
>>> ax.plot(mn, prices[mn], color='green', **kwargs)
>>> ax.plot(mx, prices[mx], color='red', **kwargs)

What does the NumPy implementation look like? While there is no np.cummin() “directly,” NumPy’s universal functions (ufuncs) all have an accumulate() method that does what its name implies:

>>> cummin = np.minimum.accumulate

Extending the logic from the pure-Python example, you can find the difference between each price and a running minimum (element-wise), and then take the max of this sequence:

>>> def profit_with_numpy(prices):
...     """Price minus cumulative minimum price, element-wise."""
...     prices = np.asarray(prices)
...     return np.max(prices - cummin(prices))

>>> profit_with_numpy(prices)
44.2487532293278
>>> np.allclose(profit_with_numpy(prices), profit(prices))
True

How do these two operations, which have the same theoretical time complexity, compare in actual runtime? First, let’s take a longer sequence. (This doesn’t necessarily need to be a time series of stock prices at this point.)

>>> seq = np.random.randint(0, 100, size=100000)
>>> seq
array([ 3, 23,  8, 67, 52, 12, 54, 72, 41, 10, ..., 46,  8, 90, 95, 93,
       28, 24, 88, 24, 49])

Now, for a somewhat unfair comparison:

>>> setup = ('from __main__ import profit_with_numpy, profit, seq;'
...          ' import numpy as np')
>>> num = 250
>>> pytime = timeit('profit(seq)', setup=setup, number=num)
>>> nptime = timeit('profit_with_numpy(seq)', setup=setup, number=num)
>>> print('Speed difference: {:0.1f}x'.format(pytime / nptime))
Speed difference: 76.0x

Above, treating profit_with_numpy() as pseudocode (without considering NumPy’s underlying mechanics), there are actually three passes through a sequence:

cummin(prices) has O(n) time complexity
prices - cummin(prices) is O(n)
max(...) is O(n)

This reduces to O(n), because O(3n) reduces to just O(n)–the n “dominates” as n approaches infinity.

Therefore, these two functions have equivalent worst-case time complexity. (Although, as a side note, the NumPy function comes with significantly more space complexity.) But that is probably the least important takeaway here. One lesson is that, while theoretical time complexity is an important consideration, runtime mechanics can also play a big role. Not only can NumPy delegate to C, but with some element-wise operations and linear algebra, it can also take advantage of computing within multiple threads. But there are a lot of factors at play here, including the underlying library used (BLAS/LAPACK/Atlas), and those details are for a whole ‘nother article entirely.

Intermezzo: Understanding Axes Notation

In NumPy, an axis refers to a single dimension of a multidimensional array:

>>> arr = np.array([[1, 2, 3],
...                 [10, 20, 30]])
>>> arr.sum(axis=0)
array([11, 22, 33])
>>> arr.sum(axis=1)
array([ 6, 60])

The terminology around axes and the way in which they are described can be a bit unintuitive. In the documentation for Pandas (a library built on top of NumPy), you may frequently see something like:

axis : {'index' (0), 'columns' (1)}

You could argue that, based on this description, the results above should be “reversed.” However, the key is that axis refers to the axis along which a function gets called. This is well articulated by Jake VanderPlas:

The way the axis is specified here can be confusing to users coming from other languages. The axis keyword specifies the dimension of the array that will be collapsed, rather than the dimension that will be returned. So, specifying axis=0 means that the first axis will be collapsed: for two-dimensional arrays, this means that values within each column will be aggregated. [source]

In other words, summing an array for axis=0 collapses the rows of the array with a column-wise computation.

With this distinction in mind, let’s move on to explore the concept of broadcasting.

Broadcasting

Broadcasting is another important NumPy abstraction. You’ve already seen that operations between two NumPy arrays (of equal size) operate element-wise:

>>> a = np.array([1.5, 2.5, 3.5])
>>> b = np.array([10., 5., 1.])
>>> a / b
array([0.15, 0.5 , 3.5 ])

But, what about unequally sized arrays? This is where broadcasting comes in:

The term broadcasting describes how NumPy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes. Broadcasting provides a means of vectorizing array operations so that looping occurs in C instead of Python. [source]

The way in which broadcasting is implemented can become tedious when working with more than two arrays. However, if there are just two arrays, then their ability to be broadcasted can be described with two short rules:

When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions and works its way forward. Two dimensions are compatible when:

they are equal, or

one of them is 1

That’s all there is to it.

Let’s take a case where we want to subtract each column-wise mean of an array, element-wise:

>>> sample = np.random.normal(loc=[2., 20.], scale=[1., 3.5],
...                           size=(3, 2))
>>> sample
array([[ 1.816 , 23.703 ],
       [ 2.8395, 12.2607],
       [ 3.5901, 24.2115]])

In statistical jargon, sample consists of two samples (the columns) drawn independently from two populations with means of 2 and 20, respectively. The column-wise means should approximate the population means (albeit roughly, because the sample is small):

>>> mu = sample.mean(axis=0)
>>> mu
array([ 2.7486, 20.0584])

Now, subtracting the column-wise means is straightforward because broadcasting rules check out:

>>> print('sample:', sample.shape, '| means:', mu.shape)
sample: (3, 2) | means: (2,)

>>> sample - mu
array([[-0.9325,  3.6446],
       [ 0.091 , -7.7977],
       [ 0.8416,  4.1531]])

Here’s an illustration of subtracting out column-wise means, where a smaller array is “stretched” so that it is subtracted from each row of the larger array:

Technical Detail: The smaller-sized array or scalar is not literally stretched in memory: it is the computation itself that is repeated.

This extends to standardizing each column as well, making each cell a z-score relative to its respective column:

>>> (sample - sample.mean(axis=0)) / sample.std(axis=0)
array([[-1.2825,  0.6605],
       [ 0.1251, -1.4132],
       [ 1.1574,  0.7527]])

However, what if you want to subtract out, for some reason, the row-wise minimums? You’ll run into a bit of trouble:

>>> sample - sample.min(axis=1)
ValueError: operands could not be broadcast together with shapes (3,2) (3,)

The problem here is that the smaller array, in its current form, cannot be “stretched” to be shape-compatible with sample. You actually need to expand its dimensionality to meet the broadcasting rules above:

>>> sample.min(axis=1)[:, None]  # 3 minimums across 3 rows
array([[1.816 ],
       [2.8395],
       [3.5901]])

>>> sample - sample.min(axis=1)[:, None]
array([[ 0.    , 21.887 ],
       [ 0.    ,  9.4212],
       [ 0.    , 20.6214]])

Note: [:, None] is a means by which to expand the dimensionality of an array, to create an axis of length one. np.newaxis is an alias for None.

There are some significantly more complex cases, too. Here’s a more rigorous definition of when any arbitrary number of arrays of any shape can be broadcast together:

A set of arrays is called “broadcastable” to the same shape if the following rules produce a valid result, meaning one of the following is true:

The arrays all have exactly the same shape.

The arrays all have the same number of dimensions, and the length of each dimension is either a common length or 1.

The arrays that have too few dimensions can have their shapes prepended with a dimension of length 1 to satisfy property #2.

[source]

This is easier to walk through step by step. Let’s say you have the following four arrays:

>>> a = np.sin(np.arange(10)[:, None])
>>> b = np.random.randn(1, 10)
>>> c = np.full_like(a, 10)
>>> d = 8

Before checking shapes, NumPy first converts scalars to arrays with one element:

>>> arrays = [np.atleast_1d(arr) for arr in (a, b, c, d)]
>>> for arr in arrays:
...     print(arr.shape)
...
(10, 1)
(1, 10)
(10, 1)
(1,)

Now we can check criterion #1. If all of the arrays have the same shape, a set of their shapes will condense down to one element, because the set() constructor effectively drops duplicate items from its input. This criterion is clearly not met:

>>> len(set(arr.shape for arr in arrays)) == 1
False

The first part of criterion #2 also fails, meaning the entire criterion fails:

>>> len(set((arr.ndim) for arr in arrays)) == 1
False

The final criterion is a bit more involved:

The arrays that have too few dimensions can have their shapes prepended with a dimension of length 1 to satisfy property #2.

To codify this, you can first determine the dimensionality of the highest-dimension array and then prepend ones to each shape tuple until all are of equal dimension:

>>> maxdim = max(arr.ndim for arr in arrays)  # Maximum dimensionality
>>> shapes = np.array([(1,) * (maxdim - arr.ndim) + arr.shape
...                    for arr in arrays])
>>> shapes
array([[10,  1],
       [ 1, 10],
       [10,  1],
       [ 1,  1]])

Finally, you need to test that the length of each dimension is either (drawn from) a common length, or 1. A trick for doing this is to first mask the array of “shape-tuples” in places where it equals one. Then, you can check if the peak-to-peak (np.ptp()) column-wise differences are all zero:

>>> masked = np.ma.masked_where(shapes == 1, shapes)
>>> np.all(masked.ptp(axis=0) == 0)  # ptp: max - min
True

Encapsulated in a single function, this logic looks like this:

>>> def can_broadcast(*arrays) -> bool:
...     arrays = [np.atleast_1d(arr) for arr in arrays]
...     if len(set(arr.shape for arr in arrays)) == 1:
...         return True
...     if len(set((arr.ndim) for arr in arrays)) == 1:
...         return True
...     maxdim = max(arr.ndim for arr in arrays)
...     shapes = np.array([(1,) * (maxdim - arr.ndim) + arr.shape
...                        for arr in arrays])
...     masked = np.ma.masked_where(shapes == 1, shapes)
...     return np.all(masked.ptp(axis=0) == 0)
...
>>> can_broadcast(a, b, c, d)
True

Luckily, you can take a shortcut and use np.broadcast() for this sanity-check, although it’s not explicitly designed for this purpose:

>>> def can_broadcast(*arrays) -> bool:
...     try:
...         np.broadcast(*arrays)
...         return True
...     except ValueError:
...         return False
...
>>> can_broadcast(a, b, c, d)
True

For those interested in digging a little deeper, PyArray_Broadcast is the underlying C function that encapsulates broadcasting rules.

Array Programming in Action: Examples

In the following 3 examples, you’ll put vectorization and broadcasting to work with some real-world applications.

Clustering Algorithms

Machine learning is one domain that can frequently take advantage of vectorization and broadcasting. Let’s say that you have the vertices of a triangle (each row is an x, y coordinate):

>>> tri = np.array([[1, 1],
...                 [3, 1],
...                 [2, 3]])

The centroid of this “cluster” is an (x, y) coordinate that is the arithmetic mean of each column:

>>> centroid = tri.mean(axis=0)
>>> centroid
array([2.    , 1.6667])

It’s helpful to visualize this:

>>> trishape = plt.Polygon(tri, edgecolor='r', alpha=0.2, lw=5)
>>> _, ax = plt.subplots(figsize=(4, 4))
>>> ax.add_patch(trishape)
>>> ax.set_ylim([.5, 3.5])
>>> ax.set_xlim([.5, 3.5])
>>> ax.scatter(*centroid, color='g', marker='D', s=70)
>>> ax.scatter(*tri.T, color='b',  s=70)

Many clustering algorithms make use of Euclidean distances of a collection of points, either to the origin or relative to their centroids.

In Cartesian coordinates, the Euclidean distance between points p and q is:

[source: Wikipedia]

So for the set of coordinates in tri from above, the Euclidean distance of each point from the origin (0, 0) would be:

>>> np.sum(tri**2, axis=1) ** 0.5  # Or: np.sqrt(np.sum(np.square(tri), 1))
array([1.4142, 3.1623, 3.6056])

You may recognize that we are really just finding Euclidean norms:

>>> np.linalg.norm(tri, axis=1)
array([1.4142, 3.1623, 3.6056])

Instead of referencing the origin, you could also find the norm of each point relative to the triangle’s centroid:

>>> np.linalg.norm(tri - centroid, axis=1)
array([1.2019, 1.2019, 1.3333])

Finally, let’s take this one step further: let’s say that you have a 2d array X and a 2d array of multiple (x, y) “proposed” centroids. Algorithms such as K-Means clustering work by randomly assigning initial “proposed” centroids, then reassigning each data point to its closest centroid. From there, new centroids are computed, with the algorithm converging on a solution once the re-generated labels (an encoding of the centroids) are unchanged between iterations. A part of this iterative process requires computing the Euclidean distance of each point from each centroid:

>>> X = np.repeat([[5, 5], [10, 10]], [5, 5], axis=0)
>>> X = X + np.random.randn(*X.shape)  # 2 distinct "blobs"
>>> centroids = np.array([[5, 5], [10, 10]])

>>> X
array([[ 3.3955,  3.682 ],
       [ 5.9224,  5.785 ],
       [ 5.9087,  4.5986],
       [ 6.5796,  3.8713],
       [ 3.8488,  6.7029],
       [10.1698,  9.2887],
       [10.1789,  9.8801],
       [ 7.8885,  8.7014],
       [ 8.6206,  8.2016],
       [ 8.851 , 10.0091]])

>>> centroids
array([[ 5,  5],
       [10, 10]])

In other words, we want to answer the question, to which centroid does each point within X belong? We need to do some reshaping to enable broadcasting here, in order to calculate the Euclidean distance between each point in X and each point in centroids:

>>> centroids[:, None]
array([[[ 5,  5]],

       [[10, 10]]])

>>> centroids[:, None].shape
(2, 1, 2)

This enables us to cleanly subtract one array from another using a combinatoric product of their rows:

>>> np.linalg.norm(X - centroids[:, None], axis=2).round(2)
array([[2.08, 1.21, 0.99, 1.94, 2.06, 6.72, 7.12, 4.7 , 4.83, 6.32],
       [9.14, 5.86, 6.78, 7.02, 6.98, 0.73, 0.22, 2.48, 2.27, 1.15]])

In other words, the shape of X - centroids[:, None] is (2, 10, 2), essentially representing two stacked arrays that are each the size of X. Next, we want the label (index number) of each closest centroid, finding the minimum distance on the 0th axis from the array above:

>>> np.argmin(np.linalg.norm(X - centroids[:, None], axis=2), axis=0)
array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])

You can put all this together in functional form:

>>> def get_labels(X, centroids) -> np.ndarray:
...     return np.argmin(np.linalg.norm(X - centroids[:, None], axis=2),
...                      axis=0)
>>> labels = get_labels(X, centroids)

Let’s inspect this visually, plotting both the two clusters and their assigned labels with a color-mapping:

>>> c1, c2 = ['#bc13fe', '#be0119']  # https://xkcd.com/color/rgb/
>>> llim, ulim  = np.trunc([X.min() * 0.9, X.max() * 1.1])

>>> _, ax = plt.subplots(figsize=(5, 5))
>>> ax.scatter(*X.T, c=np.where(labels, c2, c1), alpha=0.4, s=80)
>>> ax.scatter(*centroids.T, c=[c1, c2], marker='s', s=95,
...            edgecolor='yellow')
>>> ax.set_ylim([llim, ulim])
>>> ax.set_xlim([llim, ulim])
>>> ax.set_title('One K-Means Iteration: Predicted Classes')

Amortization Tables

Vectorization has applications in finance as well.

Given an annualized interest rate, payment frequency (times per year), initial loan balance, and loan term, you can create an amortization table with monthly loan balances and payments, in a vectorized fashion. Let’s set some scalar constants first:

>>> freq = 12     # 12 months per year
>>> rate = .0675  # 6.75% annualized
>>> nper = 30     # 30 years
>>> pv = 200000   # Loan face value

>>> rate /= freq  # Monthly basis
>>> nper *= freq  # 360 months

NumPy comes preloaded with a handful of financial functions that, unlike their Excel cousins, are capable of producing vector outputs.

The debtor (or lessee) pays a constant monthly amount that is composed of a principal and interest component. As the outstanding loan balance declines, the interest portion of the total payment declines with it.

>>> periods = np.arange(1, nper + 1, dtype=int)
>>> principal = np.ppmt(rate, periods, nper, pv)
>>> interest = np.ipmt(rate, periods, nper, pv)
>>> pmt = principal + interest  # Or: pmt = np.pmt(rate, nper, pv)

Next, you’ll need to calculate a monthly balance, both before and after that month’s payment, which can be defined as the future value of the original balance minus the future value of an annuity (a stream of payments), using a discount factor d:

Functionally, this looks like:

>>> def balance(pv, rate, nper, pmt) -> np.ndarray:
...     d = (1 + rate) ** nper  # Discount factor
...     return pv * d - pmt * (d - 1) / rate

Finally, you can drop this into a tabular format with a Pandas DataFrame. Be careful with signs here. PMT is an outflow from the perspective of the debtor.

>>> import pandas as pd

>>> cols = ['beg_bal', 'prin', 'interest', 'end_bal']
>>> data = [balance(pv, rate, periods - 1, -pmt),
...         principal,
...         interest,
...         balance(pv, rate, periods, -pmt)]

>>> table = pd.DataFrame(data, columns=periods, index=cols).T
>>> table.index.name = 'month'

>>> with pd.option_context('display.max_rows', 6):
...     # Note: Using floats for $$ in production-level code = bad
...     print(table.round(2))
...
         beg_bal     prin  interest    end_bal
month
1      200000.00  -172.20  -1125.00  199827.80
2      199827.80  -173.16  -1124.03  199654.64
3      199654.64  -174.14  -1123.06  199480.50
...          ...      ...       ...        ...
358      3848.22 -1275.55    -21.65    2572.67
359      2572.67 -1282.72    -14.47    1289.94
360      1289.94 -1289.94     -7.26      -0.00

At the end of year 30, the loan is paid off:

>>> final_month = periods[-1]
>>> np.allclose(table.loc[final_month, 'end_bal'], 0)
True

Note: While using floats to represent money can be useful for concept illustration in a scripting environment, using Python floats for financial calculations in a production environment might cause your calculation to be a penny or two off in some cases.

Image Feature Extraction

In one final example, we’ll work with an October 1941 image of the USS Lexington (CV-2), the wreck of which was discovered off the coast of Australia in March 2018. First, we can map the image into a NumPy array of its pixel values:

>>> from skimage import io

>>> url = ('https://www.history.navy.mil/bin/imageDownload?image=/'
...        'content/dam/nhhc/our-collections/photography/images/'
...        '80-G-410000/80-G-416362&rendition=cq5dam.thumbnail.319.319.png')
>>> img = io.imread(url, as_grey=True)

>>> fig, ax = plt.subplots()
>>> ax.imshow(img, cmap='gray')
>>> ax.grid(False)

For simplicity’s sake, the image is loaded in grayscale, resulting in a 2d array of 64-bit floats rather than a 3-dimensional MxNx4 RGBA array, with lower values denoting darker spots:

>>> img.shape
(254, 319)

>>> img.min(), img.max()
(0.027450980392156862, 1.0)

>>> img[0, :10]  # First ten cells of the first row
array([0.8078, 0.7961, 0.7804, 0.7882, 0.7961, 0.8078, 0.8039, 0.7922,
       0.7961, 0.7961])
>>> img[-1, -10:]  # Last ten cells of the last row
array([0.0784, 0.0784, 0.0706, 0.0706, 0.0745, 0.0706, 0.0745, 0.0784,
       0.0784, 0.0824])

One technique commonly employed as an intermediary step in image analysis is patch extraction. As the name implies, this consists of extracting smaller overlapping sub-arrays from a larger array and can be used in cases where it is advantageous to “denoise” or blur an image.

This concept extends to other fields, too. For example, you’d be doing something similar by taking “rolling” windows of a time series with multiple features (variables). It’s even useful for building Conway’s Game of Life. (Although, convolution with a 3x3 kernel is a more direct approach.)

Here, we will find the mean of each overlapping 10x10 patch within img. Taking a miniature example, the first 3x3 patch array in the top-left corner of img would be:

>>> img[:3, :3]
array([[0.8078, 0.7961, 0.7804],
       [0.8039, 0.8157, 0.8078],
       [0.7882, 0.8   , 0.7961]])

>>> img[:3, :3].mean()
0.7995642701525054

The pure-Python approach to creating sliding patches would involve a nested for-loop. You’d need to consider that the starting index of the right-most patches will be at index n - 3 + 1, where n is the width of the array. In other words, if you were extracting 3x3 patches from a 10x10 array called arr, the last patch taken would be from arr[7:10, 7:10]. Also keep in mind that Python’s range() does not include its stop parameter:

>>> size = 10
>>> m, n = img.shape
>>> mm, nn = m - size + 1, n - size + 1
>>>
>>> patch_means = np.empty((mm, nn))
>>> for i in range(mm):
...     for j in range(nn):
...         patch_means[i, j] = img[i: i+size, j: j+size].mean()

>>> fig, ax = plt.subplots()
>>> ax.imshow(patch_means, cmap='gray')
>>> ax.grid(False)

With this loop, you’re performing a lot of Python calls.

An alternative that will be scalable to larger RGB or RGBA images is NumPy’s stride_tricks.

An instructive first step is to visualize, given the patch size and image shape, what a higher-dimensional array of patches would look like. We have a 2d array img with shape (254, 319)and a (10, 10) 2d patch. This means our output shape (before taking the mean of each “inner” 10x10 array) would be:

>>> shape = (img.shape[0] - size + 1, img.shape[1] - size + 1, size, size)
>>> shape
(245, 310, 10, 10)

You also need to specify the strides of the new array. An array’s strides is a tuple of bytes to jump in each dimension when moving along the array. Each pixel in img is a 64-bit (8-byte) float, meaning the total image size is 254 x 319 x 8 = 648,208 bytes.

>>> img.dtype
dtype('float64')

>>> img.nbytes
648208

Internally, img is kept in memory as one contiguous block of 648,208 bytes. strides is hence a sort of “metadata”-like attribute that tells us how many bytes we need to jump ahead to move to the next position along each axis. We move in blocks of 8 bytes along the rows but need to traverse 8 x 319 = 2,552 bytes to move “down” from one row to another.

>>> img.strides
(2552, 8)

In our case, the strides of the resulting patches will just repeat the strides of img twice:

>>> strides = 2 * img.strides
>>> strides
(2552, 8, 2552, 8)

Now, let’s put these pieces together with NumPy’s stride_tricks:

>>> from numpy.lib import stride_tricks

>>> patches = stride_tricks.as_strided(img, shape=shape, strides=strides)
>>> patches.shape
(245, 310, 10, 10)

Here’s the first 10x10 patch:

>>> patches[0, 0].round(2)
array([[0.81, 0.8 , 0.78, 0.79, 0.8 , 0.81, 0.8 , 0.79, 0.8 , 0.8 ],
       [0.8 , 0.82, 0.81, 0.79, 0.79, 0.79, 0.78, 0.81, 0.81, 0.8 ],
       [0.79, 0.8 , 0.8 , 0.79, 0.8 , 0.8 , 0.82, 0.83, 0.79, 0.81],
       [0.8 , 0.79, 0.81, 0.81, 0.8 , 0.8 , 0.78, 0.76, 0.8 , 0.79],
       [0.78, 0.8 , 0.8 , 0.78, 0.8 , 0.79, 0.78, 0.78, 0.79, 0.79],
       [0.8 , 0.8 , 0.78, 0.78, 0.78, 0.8 , 0.8 , 0.8 , 0.81, 0.79],
       [0.78, 0.77, 0.78, 0.76, 0.77, 0.8 , 0.8 , 0.77, 0.8 , 0.8 ],
       [0.79, 0.76, 0.77, 0.78, 0.77, 0.77, 0.79, 0.78, 0.77, 0.76],
       [0.78, 0.75, 0.76, 0.76, 0.73, 0.75, 0.78, 0.76, 0.77, 0.77],
       [0.78, 0.79, 0.78, 0.78, 0.78, 0.78, 0.77, 0.76, 0.77, 0.77]])

The last step is tricky. To get a vectorized mean of each inner 10x10 array, we need to think carefully about the dimensionality of what we have now. The result should collapse the last two dimensions so that we’re left with a single 245x310 array.

One (suboptimal) way would be to reshape patches first, flattening the inner 2d arrays to length-100 vectors, and then computing the mean on the final axis:

>>> veclen = size ** 2
>>> patches.reshape(*patches.shape[:2], veclen).mean(axis=-1).shape
(245, 310)

However, you can also specify axis as a tuple, computing a mean over the last two axes, which should be more efficient than reshaping:

>>> patches.mean(axis=(-1, -2)).shape
(245, 310)

Let’s make sure this checks out by comparing equality to our looped version. It does:

>>> strided_means = patches.mean(axis=(-1, -2))
>>> np.allclose(patch_means, strided_means)
True

If the concept of strides has you drooling, don’t worry: Scikit-Learn has already embedded this entire process nicely within its feature_extraction module.

A Parting Thought: Don’t Over-Optimize

In this article, we discussed optimizing runtime by taking advantage of array programming in NumPy. When you are working with large datasets, it’s important to be mindful of microperformance.

However, there is a subset of cases where avoiding a native Python for-loop isn’t possible. As Donald Knuth advised, “Premature optimization is the root of all evil.” Programmers may incorrectly predict where in their code a bottleneck will appear, spending hours trying to fully vectorize an operation that would result in a relatively insignificant improvement in runtime.

There’s nothing wrong with for-loops sprinkled here and there. Often, it can be more productive to think instead about optimizing the flow and structure of the entire script at a higher level of abstraction.

More Resources

Free Bonus: Click here to get access to a free NumPy Resources Guide that points you to the best tutorials, videos, and books for improving your NumPy skills.

NumPy Documentation:

Books:

Travis Oliphant’s Guide to NumPy, 2nd ed. (Travis is the primary creator of NumPy)
Chapter 2 (“Introduction to NumPy”) of Jake VanderPlas’ Python Data Science Handbook
Chapter 4 (“NumPy Basics”) and Chapter 12 (“Advanced NumPy”) of Wes McKinney’s Python for Data Analysis 2nd ed.
Chapter 2 (“The Mathematical Building Blocks of Neural Networks”) from François Chollet’s Deep Learning with Python
Robert Johansson’s Numerical Python
Ivan Idris: Numpy Beginner’s Guide, 3rd ed.

Other Resources:

Wikipedia: Array Programming
SciPy Lecture Notes: Basic and Advanced NumPy
EricsBroadcastingDoc: Array Broadcasting in NumPy
SciPy Cookbook: Views versus copies in NumPy
Nicolas Rougier: From Python to Numpy and 100 NumPy Exercises
TensorFlow docs: Broadcasting Semantics
Theano docs: Broadcasting
Eli Bendersky: Broadcasting Arrays in Numpy

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Debugging With Pdb

2018-04-09T14:00:00+00:00

Debugging applications can sometimes be an unwelcome activity. You’re busy working under a time crunch and you just want it to work. However, at other times, you might be learning a new language feature or experimenting with a new approach and want to understand more deeply how something is working.

Regardless of the situation, debugging code is a necessity, so it’s a good idea to be comfortable working in a debugger. In this tutorial, I’ll show you the basics of using pdb, Python’s interactive source code debugger.

I’ll walk you through a few common uses of pdb. You may want to bookmark this tutorial for quick reference later when you might really need it. pdb, and other debuggers, are indispensable tools. When you need a debugger, there’s no substitute. You really need it.

By the end of this tutorial, you’ll know how to use the debugger to see the state of any variable in your application. You’ll also be able to stop and resume your application’s flow of execution at any moment, so you can see exactly how each line of code affects its internal state.

This is great for tracking down hard-to-find bugs and allows you to fix faulty code more quickly and reliably. Sometimes, stepping through code in pdb and seeing how values change can be a real eye-opener and lead to “aha” moments, along with the occasional “face palm”.

pdb is part of Python’s standard library, so it’s always there and available for use. This can be a life saver if you need to debug code in an environment where you don’t have access to the GUI debugger you’re familiar with.

The example code in this tutorial uses Python 3.6. You can find the source code for these examples on GitHub.

At the end of this tutorial, there is a quick reference for Essential pdb Commands.

There’s also a printable pdb Command Reference you can use as a cheat sheet while debugging:

Free Bonus: Click here to get a printable "pdb Command Reference" (PDF) that you can keep on your desk and refer to while debugging.

Getting Started: Printing a Variable’s Value

In this first example, we’ll look at using pdb in its simplest form: checking the value of a variable.

Insert the following code at the location where you want to break into the debugger:

import pdb; pdb.set_trace()

When the line above is executed, Python stops and waits for you to tell it what to do next. You’ll see a (Pdb) prompt. This means that you’re now paused in the interactive debugger and can enter a command.

Starting in Python 3.7, there’s another way to enter the debugger. PEP 553 describes the built-in function breakpoint(), which makes entering the debugger easy and consistent:

breakpoint()

By default, breakpoint() will import pdb and call pdb.set_trace(), as shown above. However, using breakpoint() is more flexible and allows you to control debugging behavior via its API and use of the environment variable PYTHONBREAKPOINT. For example, setting PYTHONBREAKPOINT=0 in your environment will completely disable breakpoint(), thus disabling debugging. If you’re using Python 3.7 or later, I encourage you to use breakpoint() instead of pdb.set_trace().

You can also break into the debugger, without modifying the source and using pdb.set_trace() or breakpoint(), by running Python directly from the command-line and passing the option -m pdb. If your application accepts command-line arguments, pass them as you normally would after the filename. For example:

$ python3 -m pdb app.py arg1 arg2

There are a lot of pdb commands available. At the end of this tutorial, there is a list of Essential pdb Commands. For now, let’s use the p command to print a variable’s value. Enter p variable_name at the (Pdb) prompt to print its value.

Let’s look at the example. Here’s the example1.py source:

#!/usr/bin/env python3

filename = __file__
import pdb; pdb.set_trace()
print(f'path = {filename}')

If you run this from your shell, you should get the following output:

$ ./example1.py 
> /code/example1.py(5)<module>()
-> print(f'path = {filename}')
(Pdb)

If you’re having trouble getting the examples or your own code to run from the command line, read How Do I Make My Own Command-Line Commands Using Python? If you’re on Windows, check the Python Windows FAQ.

Now enter p filename. You should see:

(Pdb) p filename
'./example1.py'
(Pdb)

Since you’re in a shell and using a CLI (command-line interface), pay attention to the characters and formatting. They’ll give you the context you need:

> starts the 1st line and tells you which source file you’re in. After the filename, there is the current line number in parentheses. Next is the name of the function. In this example, since we’re not paused inside a function and at module level, we see <module>().
-> starts the 2nd line and is the current source line where Python is paused. This line hasn’t been executed yet. In this example, this is line 5 in example1.py, from the > line above.
(Pdb) is pdb’s prompt. It’s waiting for a command.

Use the command q to quit debugging and exit.

Printing Expressions

When using the print command p, you’re passing an expression to be evaluated by Python. If you pass a variable name, pdb prints its current value. However, you can do much more to investigate the state of your running application.

In this example, the function get_path() is called. To inspect what’s happening in this function, I’ve inserted a call to pdb.set_trace() to pause execution just before it returns:

#!/usr/bin/env python3

import os


def get_path(filename):
    """Return file's path or empty string if no path."""
    head, tail = os.path.split(filename)
    import pdb; pdb.set_trace()
    return head


filename = __file__
print(f'path = {get_path(filename)}')

If you run this from your shell, you should get the output:

$ ./example2.py 
> /code/example2.py(10)get_path()
-> return head
(Pdb)

Where are we?

>: We’re in the source file example2.py on line 10 in the function get_path(). This is the frame of reference the p command will use to resolve variable names, i.e. the current scope or context.
->: Execution has paused at return head. This line hasn’t been executed yet. This is line 10 in example2.py in the function get_path(), from the > line above.

Let’s print some expressions to look at the current state of the application. I use the command ll (longlist) initially to list the function’s source:

(Pdb) ll
  6     def get_path(filename):
  7         """Return file's path or empty string if no path."""
  8         head, tail = os.path.split(filename)
  9         import pdb; pdb.set_trace()
 10  ->     return head
(Pdb) p filename
'./example2.py'
(Pdb) p head, tail
('.', 'example2.py')
(Pdb) p 'filename: ' + filename
'filename: ./example2.py'
(Pdb) p get_path
<function get_path at 0x100760e18>
(Pdb) p getattr(get_path, '__doc__')
"Return file's path or empty string if no path."
(Pdb) p [os.path.split(p)[1] for p in os.path.sys.path]
['pdb-basics', 'python36.zip', 'python3.6', 'lib-dynload', 'site-packages']
(Pdb)

You can pass any valid Python expression to p for evaluation.

This is especially helpful when you are debugging and want to test an alternative implementation directly in the application at runtime.

You can also use the command pp (pretty-print) to pretty-print expressions. This is helpful if you want to print a variable or expression with a large amount of output, e.g. lists and dictionaries. Pretty-printing keeps objects on a single line if it can or breaks them onto multiple lines if they don’t fit within the allowed width.

Stepping Through Code

There are two commands you can use to step through code when debugging:

Command	Description
`n` (next)	Continue execution until the next line in the current function is reached or it returns.
`s` (step)	Execute the current line and stop at the first possible occasion (either in a function that is called or in the current function).

There’s a 3rd command named unt (until). It is related to n (next). We’ll look at it later in this tutorial in the section Continuing Execution.

The difference between n (next) and s (step) is where pdb stops.

Use n (next) to continue execution until the next line and stay within the current function, i.e. not stop in a foreign function if one is called. Think of next as “staying local” or “step over”.

Use s (step) to execute the current line and stop in a foreign function if one is called. Think of step as “step into”. If execution is stopped in another function, s will print --Call--.

Both n and s will stop execution when the end of the current function is reached and print --Return-- along with the return value at the end of the next line after ->.

Let’s look at an example using both commands. Here’s the example3.py source:

#!/usr/bin/env python3

import os


def get_path(filename):
    """Return file's path or empty string if no path."""
    head, tail = os.path.split(filename)
    return head


filename = __file__
import pdb; pdb.set_trace()
filename_path = get_path(filename)
print(f'path = {filename_path}')

If you run this from your shell and enter n, you should get the output:

$ ./example3.py 
> /code/example3.py(14)<module>()
-> filename_path = get_path(filename)
(Pdb) n
> /code/example3.py(15)<module>()
-> print(f'path = {filename_path}')
(Pdb)

With n (next), we stopped on line 15, the next line. We “stayed local” in <module>() and “stepped over” the call to get_path(). The function is <module>() since we’re currently at module level and not paused inside another function.

Let’s try s:

$ ./example3.py 
> /code/example3.py(14)<module>()
-> filename_path = get_path(filename)
(Pdb) s
--Call--
> /code/example3.py(6)get_path()
-> def get_path(filename):
(Pdb)

With s (step), we stopped on line 6 in the function get_path() since it was called on line 14. Notice the line --Call-- after the s command.

Conveniently, pdb remembers your last command. If you’re stepping through a lot of code, you can just press Enter to repeat the last command.

Below is an example of using both s and n to step through the code. I enter s initially because I want to “step into” the function get_path() and stop. Then I enter n once to “stay local” or “step over” any other function calls and just press Enter to repeat the n command until I get to the last source line.

$ ./example3.py 
> /code/example3.py(14)<module>()
-> filename_path = get_path(filename)
(Pdb) s
--Call--
> /code/example3.py(6)get_path()
-> def get_path(filename):
(Pdb) n
> /code/example3.py(8)get_path()
-> head, tail = os.path.split(filename)
(Pdb) 
> /code/example3.py(9)get_path()
-> return head
(Pdb) 
--Return--
> /code/example3.py(9)get_path()->'.'
-> return head
(Pdb) 
> /code/example3.py(15)<module>()
-> print(f'path = {filename_path}')
(Pdb) 
path = .
--Return--
> /code/example3.py(15)<module>()->None
-> print(f'path = {filename_path}')
(Pdb)

Note the lines --Call-- and --Return--. This is pdb letting you know why execution was stopped. n (next) and s (step) will stop before a function returns. That’s why you see the --Return-- lines above.

Also note ->'.' at the end of the line after the first --Return-- above:

--Return--
> /code/example3.py(9)get_path()->'.'
-> return head
(Pdb)

When pdb stops at the end of a function before it returns, it also prints the return value for you. In this example it’s '.'.

Listing Source Code

Don’t forget the command ll (longlist: list the whole source code for the current function or frame). It’s really helpful when you’re stepping through unfamiliar code or you just want to see the entire function for context.

Here’s an example:

$ ./example3.py 
> /code/example3.py(14)<module>()
-> filename_path = get_path(filename)
(Pdb) s
--Call--
> /code/example3.py(6)get_path()
-> def get_path(filename):
(Pdb) ll
  6  -> def get_path(filename):
  7         """Return file's path or empty string if no path."""
  8         head, tail = os.path.split(filename)
  9         return head
(Pdb)

To see a shorter snippet of code, use the command l (list). Without arguments, it will print 11 lines around the current line or continue the previous listing. Pass the argument . to always list 11 lines around the current line: l .

$ ./example3.py 
> /code/example3.py(14)<module>()
-> filename_path = get_path(filename)
(Pdb) l
  9         return head
 10     
 11     
 12     filename = __file__
 13     import pdb; pdb.set_trace()
 14  -> filename_path = get_path(filename)
 15     print(f'path = {filename_path}')
[EOF]
(Pdb) l
[EOF]
(Pdb) l .
  9         return head
 10     
 11     
 12     filename = __file__
 13     import pdb; pdb.set_trace()
 14  -> filename_path = get_path(filename)
 15     print(f'path = {filename_path}')
[EOF]
(Pdb)

Using Breakpoints

Breakpoints are very convenient and can save you a lot of time. Instead of stepping through dozens of lines you’re not interested in, simply create a breakpoint where you want to investigate. Optionally, you can also tell pdb to break only when a certain condition is true.

Use the command b (break) to set a breakpoint. You can specify a line number or a function name where execution is stopped.

The syntax for break is:

b(reak) [ ([filename:]lineno | function) [, condition] ]

If filename: is not specified before the line number lineno, then the current source file is used.

Note the optional 2nd argument to b: condition. This is very powerful. Imagine a situation where you wanted to break only if a certain condition existed. If you pass a Python expression as the 2nd argument, pdb will break when the expression evaluates to true. We’ll do this in an example below.

In this example, there’s a utility module util.py. Let’s set a breakpoint to stop execution in the function get_path().

Here’s the source for the main script example4.py:

#!/usr/bin/env python3

import util

filename = __file__
import pdb; pdb.set_trace()
filename_path = util.get_path(filename)
print(f'path = {filename_path}')

Here’s the source for the utility module util.py:

def get_path(filename):
    """Return file's path or empty string if no path."""
    import os
    head, tail = os.path.split(filename)
    return head

First, let’s set a breakpoint using the source filename and line number:

$ ./example4.py 
> /code/example4.py(7)<module>()
-> filename_path = util.get_path(filename)
(Pdb) b util:5
Breakpoint 1 at /code/util.py:5
(Pdb) c
> /code/util.py(5)get_path()
-> return head
(Pdb) p filename, head, tail
('./example4.py', '.', 'example4.py')
(Pdb)

The command c (continue) continues execution until a breakpoint is found.

Next, let’s set a breakpoint using the function name:

$ ./example4.py 
> /code/example4.py(7)<module>()
-> filename_path = util.get_path(filename)
(Pdb) b util.get_path
Breakpoint 1 at /code/util.py:1
(Pdb) c
> /code/util.py(3)get_path()
-> import os
(Pdb) p filename
'./example4.py'
(Pdb)

Enter b with no arguments to see a list of all breakpoints:

(Pdb) b
Num Type         Disp Enb   Where
1   breakpoint   keep yes   at /code/util.py:1
(Pdb)

You can disable and re-enable breakpoints using the command disable bpnumber and enable bpnumber. bpnumber is the breakpoint number from the breakpoints list’s 1st column Num. Notice the Enb column’s value change:

(Pdb) disable 1
Disabled breakpoint 1 at /code/util.py:1
(Pdb) b
Num Type         Disp Enb   Where
1   breakpoint   keep no    at /code/util.py:1
(Pdb) enable 1
Enabled breakpoint 1 at /code/util.py:1
(Pdb) b
Num Type         Disp Enb   Where
1   breakpoint   keep yes   at /code/util.py:1
(Pdb)

To delete a breakpoint, use the command cl (clear):

cl(ear) filename:lineno
cl(ear) [bpnumber [bpnumber...]]

Now let’s use a Python expression to set a breakpoint. Imagine a situation where you wanted to break only if your troubled function received a certain input.

In this example scenario, the get_path() function is failing when it receives a relative path, i.e. the file’s path doesn’t start with /. I’ll create an expression that evaluates to true in this case and pass it to b as the 2nd argument:

$ ./example4.py 
> /code/example4.py(7)<module>()
-> filename_path = util.get_path(filename)
(Pdb) b util.get_path, not filename.startswith('/')
Breakpoint 1 at /code/util.py:1
(Pdb) c
> /code/util.py(3)get_path()
-> import os
(Pdb) a
filename = './example4.py'
(Pdb)

After you create the breakpoint above and enter c to continue execution, pdb stops when the expression evaluates to true. The command a (args) prints the argument list of the current function.

In the example above, when you’re setting the breakpoint with a function name rather than a line number, note that the expression should use only function arguments or global variables that are available at the time the function is entered. Otherwise, the breakpoint will stop execution in the function regardless of the expression’s value.

If you need to break using an expression with a variable name located inside a function, i.e. a variable name not in the function’s argument list, specify the line number:

$ ./example4.py 
> /code/example4.py(7)<module>()
-> filename_path = util.get_path(filename)
(Pdb) b util:5, not head.startswith('/')
Breakpoint 1 at /code/util.py:5
(Pdb) c
> /code/util.py(5)get_path()
-> return head
(Pdb) p head
'.'
(Pdb) a
filename = './example4.py'
(Pdb)

You can also set a temporary breakpoint using the command tbreak. It’s removed automatically when it’s first hit. It uses the same arguments as b.

Continuing Execution

So far, we’ve looked at stepping through code with n (next) and s (step) and using breakpoints with b (break) and c (continue).

There’s also a related command: unt (until).

Use unt to continue execution like c, but stop at the next line greater than the current line. Sometimes unt is more convenient and quicker to use and is exactly what you want. I’ll demonstrate this with an example below.

Let’s first look at the syntax and description for unt:

Command	Syntax	Description
`unt`	unt(il) [lineno]	Without `lineno`, continue execution until the line with a number greater than the current one is reached. With `lineno`, continue execution until a line with a number greater or equal to that is reached. In both cases, also stop when the current frame returns.

Depending on whether or not you pass the line number argument lineno, unt can behave in two ways:

Without lineno, continue execution until the line with a number greater than the current one is reached. This is similar to n (next). It’s an alternate way to execute and “step over” code. The difference between n and unt is that unt stops only when a line with a number greater than the current one is reached. n will stop at the next logically executed line.
With lineno, continue execution until a line with a number greater or equal to that is reached. This is like c (continue) with a line number argument.

In both cases, unt stops when the current frame (function) returns, just like n (next) and s (step).

The primary behavior to note with unt is that it will stop when a line number greater or equal to the current or specified line is reached.

Use unt when you want to continue execution and stop farther down in the current source file. You can treat it like a hybrid of n (next) and b (break), depending on whether you pass a line number argument or not.

In the example below, there is a function with a loop. Here, you want to continue execution of the code and stop after the loop, without stepping through each iteration of the loop or setting a breakpoint:

Here’s the example source for example4unt.py:

#!/usr/bin/env python3

import os


def get_path(fname):
    """Return file's path or empty string if no path."""
    import pdb; pdb.set_trace()
    head, tail = os.path.split(fname)
    for char in tail:
        pass  # Check filename char
    return head


filename = __file__
filename_path = get_path(filename)
print(f'path = {filename_path}')

And the console output using unt:

$ ./example4unt.py 
> /code/example4unt.py(9)get_path()
-> head, tail = os.path.split(fname)
(Pdb) ll
  6     def get_path(fname):
  7         """Return file's path or empty string if no path."""
  8         import pdb; pdb.set_trace()
  9  ->     head, tail = os.path.split(fname)
 10         for char in tail:
 11             pass  # Check filename char
 12         return head
(Pdb) unt
> /code/example4unt.py(10)get_path()
-> for char in tail:
(Pdb) 
> /code/example4unt.py(11)get_path()
-> pass  # Check filename char
(Pdb) 
> /code/example4unt.py(12)get_path()
-> return head
(Pdb) p char, tail
('y', 'example4unt.py')

The ll command was used first to print the function’s source, followed by unt. pdb remembers the last command entered, so I just pressed Enter to repeat the unt command. This continued execution through the code until a source line greater than the current line was reached.

Note in the console output above that pdb stopped only once on lines 10 and 11. Since unt was used, execution was stopped only in the 1st iteration of the loop. However, each iteration of the loop was executed. This can be verified in the last line of output. The char variable’s value 'y' is equal to the last character in tail’s value 'example4unt.py'.

Displaying Expressions

Similar to printing expressions with p and pp, you can use the command display [expression] to tell pdb to automatically display the value of an expression, if it changed, when execution stops. Use the command undisplay [expression] to clear a display expression.

Here’s the syntax and description for both commands:

Command	Syntax	Description
`display`	display [expression]	Display the value of `expression` if it changed, each time execution stops in the current frame. Without `expression`, list all display expressions for the current frame.
`undisplay`	undisplay [expression]	Do not display `expression` any more in the current frame. Without `expression`, clear all display expressions for the current frame.

Below is an example, example4display.py, demonstrating its use with a loop:

$ ./example4display.py 
> /code/example4display.py(9)get_path()
-> head, tail = os.path.split(fname)
(Pdb) ll
  6     def get_path(fname):
  7         """Return file's path or empty string if no path."""
  8         import pdb; pdb.set_trace()
  9  ->     head, tail = os.path.split(fname)
 10         for char in tail:
 11             pass  # Check filename char
 12         return head
(Pdb) b 11
Breakpoint 1 at /code/example4display.py:11
(Pdb) c
> /code/example4display.py(11)get_path()
-> pass  # Check filename char
(Pdb) display char
display char: 'e'
(Pdb) c
> /code/example4display.py(11)get_path()
-> pass  # Check filename char
display char: 'x'  [old: 'e']
(Pdb) 
> /code/example4display.py(11)get_path()
-> pass  # Check filename char
display char: 'a'  [old: 'x']
(Pdb) 
> /code/example4display.py(11)get_path()
-> pass  # Check filename char
display char: 'm'  [old: 'a']

In the output above, pdb automatically displayed the value of the char variable because each time the breakpoint was hit its value had changed. Sometimes this is helpful and exactly what you want, but there’s another way to use display.

You can enter display multiple times to build a watch list of expressions. This can be easier to use than p. After adding all of the expressions you’re interested in, simply enter display to see the current values:

$ ./example4display.py 
> /code/example4display.py(9)get_path()
-> head, tail = os.path.split(fname)
(Pdb) ll
  6     def get_path(fname):
  7         """Return file's path or empty string if no path."""
  8         import pdb; pdb.set_trace()
  9  ->     head, tail = os.path.split(fname)
 10         for char in tail:
 11             pass  # Check filename char
 12         return head
(Pdb) b 11
Breakpoint 1 at /code/example4display.py:11
(Pdb) c
> /code/example4display.py(11)get_path()
-> pass  # Check filename char
(Pdb) display char
display char: 'e'
(Pdb) display fname
display fname: './example4display.py'
(Pdb) display head
display head: '.'
(Pdb) display tail
display tail: 'example4display.py'
(Pdb) c
> /code/example4display.py(11)get_path()
-> pass  # Check filename char
display char: 'x'  [old: 'e']
(Pdb) display
Currently displaying:
char: 'x'
fname: './example4display.py'
head: '.'
tail: 'example4display.py'

Python Caller ID

In this last section, we’ll build upon what we’ve learned so far and finish with a nice payoff. I use the name “caller ID” in reference to the phone system’s caller identification feature. That is exactly what this example demonstrates, except it’s applied to Python.

Here’s the source for the main script example5.py:

#!/usr/bin/env python3

import fileutil


def get_file_info(full_fname):
    file_path = fileutil.get_path(full_fname)
    return file_path


filename = __file__
filename_path = get_file_info(filename)
print(f'path = {filename_path}')

Here’s the utility module fileutil.py:

def get_path(fname):
    """Return file's path or empty string if no path."""
    import os
    import pdb; pdb.set_trace()
    head, tail = os.path.split(fname)
    return head

In this scenario, imagine there’s a large code base with a function in a utility module, get_path(), that’s being called with invalid input. However, it’s being called from many places in different packages.

How do you find who the caller is?

Use the command w (where) to print a stack trace, with the most recent frame at the bottom:

$ ./example5.py 
> /code/fileutil.py(5)get_path()
-> head, tail = os.path.split(fname)
(Pdb) w
  /code/example5.py(12)<module>()
-> filename_path = get_file_info(filename)
  /code/example5.py(7)get_file_info()
-> file_path = fileutil.get_path(full_fname)
> /code/fileutil.py(5)get_path()
-> head, tail = os.path.split(fname)
(Pdb)

Don’t worry if this looks confusing or if you’re not sure what a stack trace or frame is. I’ll explain those terms below. It’s not as difficult as it might sound.

Since the most recent frame is at the bottom, start there and read from the bottom up. Look at the lines that start with ->, but skip the 1st instance since that’s where pdb.set_trace() was used to enter pdb in the function get_path(). In this example, the source line that called the function get_path() is:

-> file_path = fileutil.get_path(full_fname)

The line above each -> contains the filename, line number (in parentheses), and function name the source line is in. So the caller is:

  /code/example5.py(7)get_file_info()
-> file_path = fileutil.get_path(full_fname)

That’s no surprise in this small example for demonstration purposes, but imagine a large application where you’ve set a breakpoint with a condition to identify where a bad input value is originating.

Now we know how to find the caller.

But what about this stack trace and frame stuff?

A stack trace is just a list of all the frames that Python has created to keep track of function calls. A frame is a data structure Python creates when a function is called and deletes when it returns. The stack is simply an ordered list of frames or function calls at any point in time. The (function call) stack grows and shrinks throughout the life of an application as functions are called and then return.

When printed, this ordered list of frames, the stack, is called a stack trace. You can see it at any time by entering the command w, as we did above to find the caller.

See this call stack article on Wikipedia for details.

To understand better and get more out of pdb, let’s look more closely at the help for w:

(Pdb) h w
w(here)
        Print a stack trace, with the most recent frame at the bottom.
        An arrow indicates the "current frame", which determines the
        context of most commands. 'bt' is an alias for this command.

What does pdb mean by “current frame”?

Think of the current frame as the current function where pdb has stopped execution. In other words, the current frame is where your application is currently paused and is used as the “frame” of reference for pdb commands like p (print).

p and other commands will use the current frame for context when needed. In the case of p, the current frame will be used for looking up and printing variable references.

When pdb prints a stack trace, an arrow > indicates the current frame.

How is this useful?

You can use the two commands u (up) and d (down) to change the current frame. Combined with p, this allows you to inspect variables and state in your application at any point along the call stack in any frame.

Here’s the syntax and description for both commands:

Command	Syntax	Description
`u`	u(p) [count]	Move the current frame `count` (default one) levels up in the stack trace (to an older frame).
`d`	d(own) [count]	Move the current frame `count` (default one) levels down in the stack trace (to a newer frame).

Let’s look at an example using the u and d commands. In this scenario, we want to inspect the variable full_fname that’s local to the function get_file_info() in example5.py. In order to do this, we have to change the current frame up one level using the command u:

$ ./example5.py 
> /code/fileutil.py(5)get_path()
-> head, tail = os.path.split(fname)
(Pdb) w
  /code/example5.py(12)<module>()
-> filename_path = get_file_info(filename)
  /code/example5.py(7)get_file_info()
-> file_path = fileutil.get_path(full_fname)
> /code/fileutil.py(5)get_path()
-> head, tail = os.path.split(fname)
(Pdb) u
> /code/example5.py(7)get_file_info()
-> file_path = fileutil.get_path(full_fname)
(Pdb) p full_fname
'./example5.py'
(Pdb) d
> /code/fileutil.py(5)get_path()
-> head, tail = os.path.split(fname)
(Pdb) p fname
'./example5.py'
(Pdb)

The call to pdb.set_trace() is in fileutil.py in the function get_path(), so the current frame is initially set there. You can see it in the 1st line of output above:

> /code/fileutil.py(5)get_path()

To access and print the local variable full_fname in the function get_file_info() in example5.py, the command u was used to move up one level:

(Pdb) u
> /code/example5.py(7)get_file_info()
-> file_path = fileutil.get_path(full_fname)

Note in the output of u above that pdb printed the arrow > at the beginning of the 1st line. This is pdb letting you know the frame was changed and this source location is now the current frame. The variable full_fname is accessible now. Also, it’s important to realize the source line starting with -> on the 2nd line has been executed. Since the frame was moved up the stack, fileutil.get_path() has been called. Using u, we moved up the stack (in a sense, back in time) to the function example5.get_file_info() where fileutil.get_path() was called.

Continuing with the example, after full_fname was printed, the current frame was moved to its original location using d, and the local variable fname in get_path() was printed.

If we wanted to, we could have moved multiple frames at once by passing the count argument to u or d. For example, we could have moved to module level in example5.py by entering u 2:

$ ./example5.py 
> /code/fileutil.py(5)get_path()
-> head, tail = os.path.split(fname)
(Pdb) u 2
> /code/example5.py(12)<module>()
-> filename_path = get_file_info(filename)
(Pdb) p filename
'./example5.py'
(Pdb)

It’s easy to forget where you are when you’re debugging and thinking of many different things. Just remember you can always use the aptly named command w (where) to see where execution is paused and what the current frame is.

Essential pdb Commands

Once you’ve spent a little time with pdb, you’ll realize a little knowledge goes a long way. Help is always available with the h command.

Just enter h or help <topic> to get a list of all commands or help for a specific command or topic.

For quick reference, here’s a list of essential commands:

Command	Description
`p`	Print the value of an expression.
`pp`	Pretty-print the value of an expression.
`n`	Continue execution until the next line in the current function is reached or it returns.
`s`	Execute the current line and stop at the first possible occasion (either in a function that is called or in the current function).
`c`	Continue execution and only stop when a breakpoint is encountered.
`unt`	Continue execution until the line with a number greater than the current one is reached. With a line number argument, continue execution until a line with a number greater or equal to that is reached.
`l`	List source code for the current file. Without arguments, list 11 lines around the current line or continue the previous listing.
`ll`	List the whole source code for the current function or frame.
`b`	With no arguments, list all breaks. With a line number argument, set a breakpoint at this line in the current file.
`w`	Print a stack trace, with the most recent frame at the bottom. An arrow indicates the current frame, which determines the context of most commands.
`u`	Move the current frame count (default one) levels up in the stack trace (to an older frame).
`d`	Move the current frame count (default one) levels down in the stack trace (to a newer frame).
`h`	See a list of available commands.
`h <topic>`	Show help for a command or topic.
`h pdb`	Show the full pdb documentation.
`q`	Quit the debugger and exit.

Python Debugging With pdb: Conclusion

In this tutorial, we covered a few basic and common uses of pdb:

printing expressions
stepping through code with n (next) and s (step)
using breakpoints
continuing execution with unt (until)
displaying expressions
finding the caller of a function

I hope it’s been helpful to you. If you’re curious about learning more, see:

pdb’s full documentation at a pdb prompt near you: (Pdb) h pdb
Python’s pdb docs

The source code used in the examples can be found on the associated GitHub repository. Be sure to check out our printable pdb Command Reference, which you can use as a cheat sheet while debugging:

Free Bonus: Click here to get a printable "pdb Command Reference" (PDF) that you can keep on your desk and refer to while debugging.

Also, if you’d like to try a GUI-based Python debugger, read our Python IDEs and Editors Guide to see what options will work best for you. Happy Pythoning!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Introduction to Git and GitHub for Python Developers

2018-04-04T14:00:00+00:00

Have you ever worked on a Python project that stopped working after you made a change here or a PEP-8 cleanup there, and you weren’t quite sure how to get it back? Version control systems can help you solve that problem and other related ones. Git is one of the most popular version control systems today.

In this tutorial, I’ll walk you through what Git is, how to use it for your personal projects, and how to use it in conjunction with GitHub to work with other people on larger projects. We’ll look at how to create a repo, how to add both new and modified files, and how to navigate through your project’s history so you can “get back” to when your project was working.

This article assumes you already have Git installed on your system. If you don’t, the excellent Pro Git book has a section on how to do that.

What Is Git?

Git is a distributed version control system (DVCS). Let’s break that down a bit and look at what it means.

Version Control

A version control system (VCS) is a set of tools that track the history of a set of files. This means that you can tell your VCS (Git, in our case) to save the state of your files at any point. Then, you may continue to edit the files and store that state as well. Saving the state is similar to creating a backup copy of your working directory. When using Git, we refer to this saving of state as making a commit.

When you make a commit in Git, you add a commit message that explains at a high level what changes you made in this commit. Git can show you the history of all of the commits and their commit messages. This provides a useful history of what work you have done and can really help pinpoint when a bug crept into the system.

In addition to showing you the log of changes you’ve made, Git also allows you to compare files between different commits. As I mentioned earlier, Git will also allow you to return any file (or all files) to an earlier commit with little effort.

Distributed Version Control

OK, so that’s a version control system. What’s the distributed part? It’s probably easiest to answer that question by starting with a little history. Early version control systems worked by storing all of those commits locally on your hard drive. This collection of commits is called a repository. This solved the “I need to get back to where I was” problem but didn’t scale well for a team working on the same codebase.

As larger groups started working (and networking became more common), VCSs changed to store the repository on a central server that was shared by many developers. While this solved many problems, it also created new ones, like file locking.

Following the lead of a few other products, Git broke with that model. Git does not have a central server that has the definitive version of the repository. All users have a full copy of the repository. This means that getting all of the developers back on the same page can sometimes be tricky, but it also means that developers can work offline most of the time, only connecting to other repositories when they need to share their work.

That last paragraph can seem a little confusing at first, because there are a lot of developers who use GitHub as a central repository from which everyone must pull. This is true, but Git doesn’t impose this. It’s just convenient in some circumstances to have a central place to share the code. The full repository is still stored on all local repos even when you use GitHub.

Basic Usage

Now that we’ve talked about what Git is in general, let’s run through an example and see it in action. We’ll start by working with Git just on our local machine. Once we get the hang of that, we’ll add GitHub and explain how you can interact with it.

Creating a New Repo

To work with Git, you first need a repo to work in. Creating a repo is simple. Use the git init command in a directory:

$ mkdir example
$ cd example
$ git init
Initialized empty Git repository in /home/jima/tmp/example/.git/

Once you have a repo, you can ask Git about it. The Git command you’ll use most frequently is git status. Try that now:

$ git status
On branch master

Initial commit

nothing to commit (create/copy files and use "git add" to track)

This shows you a couple of bits of information: which branch you’re on, master (we’ll talk about branches later), and that you have nothing to commit. This last part means that there are no files in this directory that Git doesn’t know about. That’s good, as we just created the directory.

Adding a New File

Now create a file that Git doesn’t know about. With your favorite editor, create the file hello.py, which has just a print statement in it.

# hello.py
print('hello Git!')

If you run git status again, you’ll see a different result:

$ git status
On branch master

Initial commit

Untracked files:
  (use "git add <file>..." to include in what will be committed)

   hello.py

nothing added to commit but untracked files present (use "git add" to track)

Now Git sees the new file and tells you that it’s untracked. That’s just Git’s way of saying that the file is not part of the repo and is not under version control. We can fix that by adding the file to Git. Use the git add command to make that happen:

$ git add hello.py
$ git status
On branch master

Initial commit

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)

   new file:   hello.py

Now Git knows about hello.py and lists it under changes to be committed. Adding the file to Git moves it into the staging area (discussed below) and means we can commit it to the repo.

Committing Changes

When you commit changes, you are telling Git to make a snapshot of this state in the repo. Do that now by using the git commit command. The -m option tells Git to use the commit message that follows. If you don’t use -m, Git will bring up an editor for you to create the commit message. In general, you want your commit messages to reflect what has changed in the commit:

$ git commit -m "creating hello.py"
[master (root-commit) 25b09b9] creating hello.py
 1 file changed, 3 insertions(+)
 create mode 100755 hello.py

$ git status
On branch master
nothing to commit, working directory clean

You can see that the commit command returned a bunch of information, most of which isn’t that useful, but it does tell you that only 1 file changed (which makes sense as we added one file). It also tells you the SHA of the commit (f8af94c). We’ll have an aside about SHA a bit later.

Running the git status command again shows that we have a clean working directory, meaning that all changes are committed to Git.

At this point, we need to stop our tutorial and have a quick chat about the staging area.

Aside: The Staging Area

Unlike many version control systems, Git has a staging area (often referred to as the index). The staging area is how Git keeps track of the changes you want to be in your next commit. When we ran git add above, we told Git that we wanted to move the new file hello.py to the staging area. This change was reflected in git status. The file went from the untracked section to the to be committed section of the output.

Note that the staging area reflects the exact contents of the file when you ran git add. If you modify it again, the file will appear both in the staged and unstaged portions of the status output.

At any point of working with a file in Git (assuming it’s already been committed once), there can be three versions of the file you can work with:

the version on your hard drive that you are editing
a different version that Git has stored in your staging area
the latest version checked in to the repo

All three of these can be different versions of the file. Moving changes to the staging area and then committing them brings all of these versions back into sync.

When I started with Git, I found the staging area a little confusing and a bit annoying. It seemed to add extra steps to the process without adding any benefits. When you’re first learning Git, that’s actually true. After a while, there will be situations where you come to really appreciate having that functionality. Unfortunately, those situations are beyond the scope of this tutorial.

If you’re interested in more detailed info about the staging area, I can recommend this blog post.

`.gitignore`

The status command is very handy, and you’ll find yourself using it often. But sometimes you’ll find that there are a bunch of files that show up in the untracked section and that you want Git to just not see. That’s where the .gitignore file comes in.

Let’s walk through an example. Create a new Python file in the same directory called myname.py.

# myname.py
def get_name():
    return "Jim"

Then modify your hello.py to include myname and call its function:

# hello.py
import myname

name = myname.get_name()
print("hello {}".format(name))

When you import a local module, Python will compile it to bytecode for you and leave that file on your filesystem. In Python 2, it will leave a file called myname.pyc, but we’ll assume you’re running Python 3. In that case it will create a __pycache__ directory and store a pyc file there. That is what’s shown below:

$ ls
hello.py  myname.py
$ ./hello.py
hello Jim!
$ ls
hello.py  myname.py  __pycache__

Now if you run git status, you’ll see that directory in the untracked section. Also note that your new myname.py file is untracked, while the changes you made to hello.py are in a new section called “Changes not staged for commit”. This just means that those changes have not yet been added to the staging area. Let’s try it out:

$ git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

   modified:   hello.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)

   __pycache__/
   myname.py

no changes added to commit (use "git add" and/or "git commit -a")

Before we move on to the gitignore file, let’s clean up the mess we’ve made a little bit. First we’ll add the myname.py and hello.py files, just like we did earlier:

$ git add myname.py hello.py
$ git status
On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

   modified:   hello.py
   new file:   myname.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)

   __pycache__/

Let’s commit those changes and finish our clean up:

$ git commit -m "added myname module"
[master 946b99b] added myname module
 2 files changed, 8 insertions(+), 1 deletion(-)
 create mode 100644 myname.py

Now when we run status, all we see is that __pycache__ directory:

$ git status
On branch master
Untracked files:
  (use "git add <file>..." to include in what will be committed)

   __pycache__/

nothing added to commit but untracked files present (use "git add" to track)

To get all __pycache__ directories (and their contents) to be ignored, we’re going to add a .gitignore file to our repo. This is as simple as it sounds. Edit the file (remember the dot in front of the name!) in your favorite editor.

# .gitignore
__pycache__

Now when we run git status, we no longer see the __pycache__ directory. We do, however, see the new .gitignore! Take a look:

$ git status
On branch master
Untracked files:
  (use "git add <file>..." to include in what will be committed)

   .gitignore

nothing added to commit but untracked files present (use "git add" to track)

That file is just a regular text file and can be added to you repo like any other file. Do that now:

$ git add .gitignore
$ git commit -m "created .gitignore"
[master 1cada8f] created .gitignore
 1 file changed, 1 insertion(+)
 create mode 100644 .gitignore

Another common entry in the .gitignore file is the directory you store your virtual environments in. You can learn more about virtualenvs here, but the virtualenv directory is usually called env or venv. You can add these to your .gitignore file. If the files or directories are present, they will be ignored. If they are not, then nothing will happen.

It’s also possible to have a global .gitignore file stored in your home directory. This is very handy if your editor likes to leave temporary or backup files in the local directory.

Here’s an example of a simple Python .gitignore file:

# .gitignore
__pycache__
venv
env
.pytest_cache
.coverage

For a more complete example, look here or, if you want to build your own, git help gitignore provides all the details you’ll need.

What NOT to Add to a Git Repo

When you first start working with any version control tool, especially Git, you might be tempted to put everything into the repo. This is generally a mistake. There are limitations to Git as well as security concerns that force you to limit which types of information you add to the repo.

Let’s start with the basic rule of thumb about all version control systems.

Only put source files into version control, never generated files.

In this context, a source file is any file you create, usually by typing in an editor. A generated file is something that the computer creates, usually by processing a source file. For example, hello.py is a source file, while hello.pyc would be a generated file.

There are two reasons for not including generated files in the repo. The first is that doing so is a waste of time and space. The generated files can be recreated at any time and may need to be created in a different form. If someone is using Jython or IronPython while you’re using the Cython interpreter, the .pyc files might be quite different. Committing one particular flavor of files can cause conflict

The second reason for not storing generated files is that these files are frequently larger than the original source files. Putting them in the repo means that everyone now needs to download and store those generated files, even if they’re not using them.

This second point leads to another general rule about Git repos: commit binary files with caution and strongly avoid committing large files. This rule has a lot to do with how Git works.

Git does not store a full copy of each version of each file you commit. Rather, it uses a complicated algorithm based on the differences between subsequent versions of a file to greatly reduce the amount of storage it needs. Binary files (like JPGs or MP3 files) don’t really have good diff tools, so Git will frequently just need to store the entire file each time it is committed.

When you are working with Git, and especially when you are working with GitHub, never put confidential information into a repo, especially one you might share publicly. This is so important that I’m going to say it again:

Caution: Never put confidential information into a public repository on GitHub. Passwords, API keys, and similar items should not be committed to a repo. Someone will find them eventually.

Aside: What is a SHA

When Git stores things (files, directories, commits, etc) in your repo, it stores them in a complicated way involving a hash function. We don’t need to go into the details here, but a hash function takes a thing and produces a unique ID for that thing that is much shorter (20 bytes, in our case). This ID is called a “SHA” in Git. It is not guaranteed to be unique, but for most practical applications it is.

Git uses its hash algorithm to index everything in your repo. Each file has a SHA that reflects the contents of that file. Each directory, in turn, is hashed. If a file in that directory changes, then the SHA of the directory changes too.

Each commit contains the SHA of the top-level directory in your repo along with some other info. That’s how a single 20 byte number describes the entire state of your repo.

You might notice that sometimes Git uses the full 20 character value to show you a SHA:

commit 25b09b9ccfe9110aed2d09444f1b50fa2b4c979c

Sometimes it shows you a shorter version:

[master (root-commit) 25b09b9] creating hello.py

Usually, it will show you the full string of characters, but you don’t always have to use it. The rule for Git is that you only have to give enough characters to ensure that the SHA is unique in your repo. Generally, seven characters is more than enough.

Each time you commit changes to the repo, Git creates a new SHA that describes that state. We’ll look at how SHAs are useful in the next sections.

Git Log

Another very frequently used Git command is git log. Git log shows you the history of the commits that you have made up to this point:

$ git log
commit 1cada8f59b43254f621d1984a9ffa0f4b1107a3b
Author: Jim Anderson <jima@example.com>
Date:   Sat Mar 3 13:23:07 2018 -0700

    created .gitignore

commit 946b99bfe1641102d39f95616ceaab5c3dc960f9
Author: Jim Anderson <jima@example.com>
Date:   Sat Mar 3 13:22:27 2018 -0700

    added myname module

commit 25b09b9ccfe9110aed2d09444f1b50fa2b4c979c
Author: Jim Anderson <jima@example.com>
Date:   Sat Mar 3 13:10:12 2018 -0700

    creating hello.py

As you can see in the listing above, all of the commit messages are shown for our repo in order. The start of each commit is marked with the word “commit” followed by the SHA of that commit. git log gives you the history of each of the SHAs.

Going Back In Time: Checking Out a Particular Version of Your Code

Because Git remembers each commit you’ve made with it’s SHA, you can tell Git to go to any of those commits and view the repo as it existed then. The diagram below shows what Git thinks is in our repo:

Don’t worry about what master and HEAD mean in the diagram. We’ll explain those in just a bit.

To change where we are in our history, we will use the git checkout command to tell Git which SHA we want to look at. Let’s try that:

$ git checkout 946b99bfe1641102d39f95616ceaab5c3dc960f9
Note: checking out '946b99bfe1641102d39f95616ceaab5c3dc960f9'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 946b99b... added myname module

OK, so there’s a LOT of information here that’s confusing. Let’s start by defining some of those terms. Let’s start with HEAD.

HEAD is Git’s name for whatever SHA you happen to be looking at at any time. It does NOT mean what is on your filesystem or what is in your staging area. It means what Git thinks you have checked out. So, if you’ve edited a file, the version on your filesystem is different than the version in HEAD (and yes, HEAD is in ALL CAPS).

Next, we have branch. The easiest way to think about a branch is that it is a label on a SHA. It has a few other properties that are useful, but for now, think of a branch as a SHA label.

Note: Those of you who have worked with other version control systems (I’m looking at you, Subversion) will have a very different idea of what a branch is. Git treats branches differently, and that is a good thing.

When we put all this information together, we see that detached HEAD simply means that your HEAD is pointing to a SHA that does not have a branch (or label) associated with it. Git very nicely tells you how to fix that situation. There will be times you when will want to fix it, and there will be times when you can work in that detached HEAD state just fine.

Let’s get back to our demo. If you look at the state of the system now, you can see that the .gitignore file is no longer present. We went back to the state of the system before we made those changes. Below is the diagram of our repo at this state. Note how the HEAD and master pointers are pointing at different SHAs:

Alright. Now, how do we get back to where we were? There are two ways, one of which you should know already: git checkout 1cada8f. This will take you back to the SHA you were at when you started moving around.

Note: One odd thing, at least in my version of Git, is that it still gives you a detached HEAD warning, even though you’re back to the SHA associated with a branch.

The other way of getting back is more common: check out the branch you were on. Git always starts you off with a branch called master. We’ll learn how to make other branches later but for now we’ll stick with master.

To get back to where you where, you can simply do git checkout master. This will return you to the latest SHA committed to the master branch, which in our case has the commit message “created .gitignore”. To put it another way, git checkout master tells Git to make HEAD point to the SHA marked by the label, or branch, master.

Note that there are several methods for specifying a specific commit. The SHA is probably the easiest to understand. The other methods use different symbols and names to specify how to get to a specific commit from a known place, like HEAD. I won’t be going into those details in this tutorial, but if you’d like more details you can find them here.

Branching Basics

Let’s talk a little more about branches. Branches provide a way for you to keep separate streams of development apart. While this can be useful when you’re working alone, it’s almost essential when you’re working on team.

Imagine that I’m working in a small team and have a feature to add to the project. While I’m working on it, I don’t want to add my changes to master as it still doesn’t work correctly and might mess up my team members.

I could just wait to commit the changes until I’m completely finished, but that’s not very safe and not always practical. So, instead of working on master, I’ll create a new branch:

$ git checkout -b my_new_feature
Switched to a new branch 'my_new_feature'
$ git status
On branch my_new_feature
nothing to commit, working directory clean

We used the -b option on the checkout command to tell Git we wanted it to create a new branch. As you can see above, running git status in our branch shows us that the branch name has, indeed, changed. Let’s look at the log:

$ git log
commit 1cada8f59b43254f621d1984a9ffa0f4b1107a3b
Author: Jim Anderson <jima@example.com>
Date:   Thu Mar 8 20:57:42 2018 -0700

    created .gitignore

commit 946b99bfe1641102d39f95616ceaab5c3dc960f9
Author: Jim Anderson <jima@example.com>
Date:   Thu Mar 8 20:56:50 2018 -0700

    added myname module

commit 25b09b9ccfe9110aed2d09444f1b50fa2b4c979c
Author: Jim Anderson <jima@example.com>
Date:   Thu Mar 8 20:53:59 2018 -0700

    creating hello.py

As I hope you expected, the log looks exactly the same. When you create a new branch, the new branch will start at the location you were at. In this case, we were at the top of master, 1cada8f59b43254f621d1984a9ffa0f4b1107a3b, so that’s where the new branch starts.

Now, let’s work on that feature. Make a change to the hello.py file and commit it. I’ll show you the commands for review, but I’ll stop showing you the output of the commands for things you’ve already seen:

$ git add hello.py
$ git commit -m "added code for feature x"

Now if you do git log, you’ll see that our new commit is present. In my case, it has a SHA 4a4f4492ded256aa7b29bf5176a17f9eda66efbb, but your repo is very likely to have a different SHA:

$ git log
commit 4a4f4492ded256aa7b29bf5176a17f9eda66efbb
Author: Jim Anderson <jima@example.com>
Date:   Thu Mar 8 21:03:09 2018 -0700

    added code for feature x

commit 1cada8f59b43254f621d1984a9ffa0f4b1107a3b
... the rest of the output truncated ...

Now switch back to the master branch and look at the log:

git checkout master
git log

Is the new commit “added code for feature x” there?

Git has a built-in way to compare the state of two branches so you don’t have to work so hard. It’s the show-branch command. Here’s what it looks like:

$ git show-branch my_new_feature master
* [my_new_feature] added code for feature x
 ! [master] created .gitignore
--
*  [my_new_feature] added code for feature x
*+ [master] created .gitignore

The chart it generates is a little confusing at first, so let’s walk though it in detail. First off, you call the command by giving it the name of two branches. In our case that was my_new_feature and master.

The first two lines of the output are the key to decoding the rest of the text. The first line has a * in the first column, then the name of the branch, and then the commit message for the most recent commit on that branch. The second line starts with a ! in the second column, followed by the name and the top commit message for that branch.

The third line is a separator.

Starting on the fourth line, there are commits that are in one branch but not the other. In our current case, this is pretty easy. There’s one commit in my_new_feature that’s not in master. You can see that on the fourth line. Notice how that line starts with a * in the first column. This is to indicate which branch this commit is in.

Finally, the last line of the output shows the first common commit for the two branches.

This example is pretty easy. To make a better example, I’ve made it more interesting by adding a few more commits to my_new_feature and a few to master. That makes the output look like:

$ git show-branch my_new_feature master
* [my_new_feature] commit 4
 ! [master] commit 3
--
*  [my_new_feature] commit 4
*  [my_new_feature^] commit 1
*  [my_new_feature~2] added code for feature x
 + [master] commit 3
 + [master^] commit 2
*+ [my_new_feature~3] created .gitignore

Now you can see that there are different commits in each branch. Note that the [my_new_feature~2] text is one of the commit selection methods I mentioned earlier. If you’d rather see the SHAs, you can have it show them by adding the –sha1-name option to the command:

$ git show-branch --sha1-name my_new_feature master
* [my_new_feature] commit 4
 ! [master] commit 3
--
*  [6b6a607] commit 4
*  [12795d2] commit 1
*  [4a4f449] added code for feature x
 + [de7195a] commit 3
 + [580e206] commit 2
*+ [1cada8f] created .gitignore

Now you’ve got a branch with a bunch of different commits on it. What do you do when you finally finish that feature and are ready to get it to the rest of your team?

There are three main ways to get commits from one branch to another: merging, rebasing, and cherry-picking. We’ll cover each of these in turn in the next sections.

Merging

Merging is the simplest of the three to understand and use. When you do a merge, Git will create a new commit that combines the top SHAs of two branches if it needs to. If all of the commits in the other branch are ahead (based on) the top of the current branch, it will just do a fast-foward merge and place those new commits on this branch.

Let’s back up to the point where our show-branch output looked like this:

$ git show-branch --sha1-name my_new_feature master
* [my_new_feature] added code for feature x
 ! [master] created .gitignore
--
*  [4a4f449] added code for feature x
*+ [1cada8f] created .gitignore

Now, we want to get that commit 4a4f449 to be on master. Check out master and run the git merge command there:

$ git checkout master
Switched to branch 'master'

$ git merge my_new_feature
Updating 1cada8f..4a4f449
Fast-forward
 hello.py | 1 +
 1 file changed, 1 insertion(+)

Since we were on branch master, we did a merge of the my_new_feature branch to us. You can see that this is a fast forward merge and which files were changed. Let’s look at the log now:

commit 4a4f4492ded256aa7b29bf5176a17f9eda66efbb
Author: Jim Anderson <jima@example.com>
Date:   Thu Mar 8 21:03:09 2018 -0700

    added code for feature x

commit 1cada8f59b43254f621d1984a9ffa0f4b1107a3b
Author: Jim Anderson <jima@example.com>
Date:   Thu Mar 8 20:57:42 2018 -0700

    created .gitignore
[rest of log truncated]

If we had made changes to master before we merged, Git would have created a new commit that was the combination of the changes from the two branches.

One of the things Git is fairly good at is understanding the common ancestors of different branches and automatically merging changes together. If the same section of code has been modified in both branches, Git can’t figure out what to do. When this happens, it stops the merge part way through and gives you instructions for how to fix the issue. This is called a merge conflict.

Rebasing

Rebasing is similar to merging but behaves a little differently. In a merge, if both branches have changes, then a new merge commit is created. In rebasing, Git will take the commits from one branch and replay them, one at a time, on the top of the other branch.

I won’t do a detailed demo of rebasing here because setting up a demo to show this is a bit tricky in text and because there is an excellent web page that covers the topic well and that I’ll reference at the end of this section.

Cherry-Picking

Cherry picking is another method for moving commits from one branch to another. Unlike merging and rebasing, with cherry-picking you specify exactly which commits you mean. The easiest way to do this is just specifying a single SHA:

$ git cherry-pick 4a4f4492ded256aa7b29bf5176a17f9eda66efbb

This tells Git to take the changes that went into 4a4f449 and apply them to the current branch.

This feature can be very handy when you want a specific change but not the entire branch that change was made on.

Quick Tip About Branching: I can’t leave this topic without recommending an excellent resource for learning about Git branches. Learn Git Branching has a set of exercises using graphical representations of commits and branches to clearly explain the difference between merging, rebasing, and cherry-picking. I highly recommend spending some time working through these exercises.

Working with Remote Repos

All of the commands we’ve discussed up to this point work with only your local repo. They don’t do any communication to a server or over the network. It turns out that there are only four major Git commands which actually talk to remote repos:

clone
fetch
pull
push

That’s it. Everything else is done on your local machine. (OK, to be completely accurate, there are other commands that talk to remotes, but they don’t fall into the basic category.)

Let’s look at each of these commands in turn.

`Clone`

Git clone is the command you use when you have the address of a known repository and you want to make a local copy. For this example, let’s use a small repo I have on my GitHub account, github-playground.

The GitHub page for that repo lives here. On that page you will find a “Clone or Download” button which gives you the URI to use with the git clone command. If you copy that, you can then clone the repo with:

git clone git@github.com:jima80525/github-playground.git

Now you have a complete repository of that project on your local machine. This includes all of the commits and all of the branches ever made on it. (Note: This repo was used by some friends while they were learning Git. I copied or forked it from someone else.)

If you want to play with the other remote commands, you should create a new repo on GitHub and follow the same steps. You are welcome to fork the github-playground repo to your account and use that. Forking on GitHub is done by clicking the “fork” button in the UI.

`Fetch`

To explain the fetch command clearly, we need to take a step back and talk about how Git manages the relationship between your local repo and a remote repo. This next part is background, and while it’s not something you’ll use on a day-to-day basis, it will make the difference between fetch and pull make more sense.

When you clone a new repo, Git doesn’t just copy down a single version of the files in that project. It copies the entire repository and uses that to create a new repository on your local machine.

Git does not make local branches for you except for master. However, it does keep track of the branches that were on the server. To do that, Git creates a set of branches that all start with remotes/origin/<branch_name>.

Only rarely (almost never), will you check out these remotes/origin branches, but it’s handy to know that they are there. Remember that every branch that existed on the remote when you cloned the repo will have a branch in remotes/origin.

When you create a new branch and the name matches an existing branch on the server, Git will mark you local branch as a tracking branch that is associated with a remote branch. We’ll see how that is useful when we get to pull.

Now that you know about the remotes/origin branches, understanding git fetch will be pretty easy. All fetch does is update all of the remotes/origin branches. It will modify only the branches stored in remotes/origin and not any of your local branches.

`Pull`

Git pull is simply the combination of two other commands. First, it does a git fetch to update the remotes/origin branches. Then, if the branch you are on is tracking a remote branch, then it does a git merge of the corresponding remote/origin branch to your branch.

For example, say you were on the my_new_feature branch, and your coworker had just added some code to it on the server. If you do a git pull, Git will update ALL of the remotes/origin branches and then do a git merge remotes/origin/my_new_feature, which will get the new commit onto the branch you’re on!

There are, of course, some limitations here. Git won’t let you even try to do a git pull if you have modified files on your local system. That can create too much of a mess.

If you have commits on your local branch, and the remote also has new commits (ie “the branches have diverged”), then the git merge portion of the pull will create a merge commit, just as we discussed above.

Those of you who have been reading closely will see that you can also have Git do a rebase instead of a merge by doing git pull -r.

`Push`

As you have probably guessed, git push is just the opposite of git pull. Well, almost the opposite. Push sends the info about the branch you are pushing and asks the remote if it would like to update its version of that branch to match yours.

Generally, this amounts to you pushing your new changes up to the server. There are a lot of details and complexity here involving exactly what a fast-forward commit is.

There is a fantastic write-up here. The gist of it is that git push makes your new commits available on the remote server.

Putting It All Together: Simple Git Workflow

At this point, we’ve reviewed several basic Git commands and how you might use them. I’ll wrap up with a quick description of a possible workflow in Git. This workflow assumes you are working on your local repo and have a remote repo to which you will push changes. It can be GitHub, but it will work the same with other remote repos. It assumes you’ve already cloned the repo.

git status – Make sure your current area is clean.
git pull – Get the latest version from the remote. This saves merging issues later.
Edit your files and make your changes. Remember to run your linter and do unit tests!
git status – Find all files that are changed. Make sure to watch untracked files too!
git add [files] – Add the changed files to the staging area.
git commit -m "message" – Make your new commit.
git push origin [branch-name] – Push your changes up to the remote.

This is one of the more basic flows through the system. There are many, many ways to use Git, and you’ve just scratched the surface with this tutorial. If you use Vim or Sublime as your editor, you might want to checkout these tutorials, which will show you how to get plugins to integrate Git into your editor:

If you’d like to take a deeper dive into Git, I can recommend these books:

The free, online Pro Git is a very handy reference.
For those of you who like to read on paper, there’s a print version of Pro Git and I found O’Reilly’s Version Control with Git to be useful.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

11 Beginner Tips for Learning Python Programming

2018-04-03T14:00:00+00:00

We are so excited that you have decided to embark on the journey of learning Python! One of the most common questions we receive from our readers is “What’s the best way to learn Python?”

I believe that the first step in learning any programming language is making sure that you understand how to learn. Learning how to learn is arguably the most critical skill involved in computer programming.

Why is knowing how to learn so important? The answer is simple: as languages evolve, libraries are created, and tools are upgraded. Knowing how to learn will be essential to keeping up with these changes and becoming a successful programmer.

In this article, we will offer several learning strategies that will help jump start your journey of becoming a rockstar Python programmer!

Free PDF Download: Python 3 Cheat Sheet

Make It Stick

Here are some tips to help you make the new concepts you are learning as a beginner programmer really stick:

Tip #1: Code Everyday

Consistency is very important when you are learning a new language. We recommend making a commitment to code every day. It may be hard to believe, but muscle memory plays a large part in programming. Committing to coding everyday will really help develop that muscle memory. Though it may seem daunting at first, consider starting small with 25 minutes everyday and working your way up from there.

Check out the First Steps With Python Guide for information on setup as well as exercises to get you started.

Tip #2: Write It Out

As you progress on your journey as a new programmer, you may wonder if you should be taking notes. Yes, you should! In fact, research suggests that taking notes by hand is most beneficial for long-term retention. This will be especially beneficial for those working towards the goal of becoming a full-time developer, as many interviews will involve writing code on a whiteboard.

Once you start working on small projects and programs, writing by hand can also help you plan your code before you move to the computer. You can save a lot of time if you write out which functions and classes you will need, as well as how they will interact.

Tip #3: Go Interactive!

Whether you are learning about basic Python data structures (strings, lists, dictionaries, etc.) for the first time, or you are debugging an application, the interactive Python shell will be one of your best learning tools. We use it a lot on this site too!

To use the interactive Python shell (also sometimes called a “Python REPL”), first make sure Python is installed on your computer. We’ve got a step-by-step tutorial to help you do that. To activate the interactive Python shell, simply open your terminal and run python or python3 depending on your installation. You can find more specific directions here.

Now that you know how to start the shell, here are a few examples of how you can use the shell when you are learning:

Learn what operations can be performed on an element by using dir():

>>> my_string = 'I am a string'
>>> dir(my_string)
['__add__', ..., 'upper', 'zfill']  # Truncated for readability

The elements returned from dir() are all of the methods (i.e. actions) that you can apply to the element. For example:

>>> my_string.upper()
>>> 'I AM A STRING'

Notice that we called the upper() method. Can you see what it does? It makes all of the letters in the string uppercase! Learn more about these built-in methods under “Manipulating strings” in this tutorial.

Learn the type of an element:

>>> type(my_string)
>>> str

Use the built-in help system to get full documentation:

>>> help(str)

Import libraries and play with them:

>>> from datetime import datetime
>>> dir(datetime)
['__add__', ..., 'weekday', 'year']  # Truncated for readability
>>> datetime.now()
datetime.datetime(2018, 3, 14, 23, 44, 50, 851904)

Run shell commands:

>>> import os
>>> os.system('ls')
python_hw1.py python_hw2.py README.txt

Tip #4: Take Breaks

When you are learning, it is important to step away and absorb the concepts. The Pomodoro Technique is widely used and can help: you work for 25 minutes, take a short break, and then repeat the process. Taking breaks is critical to having an effective study session, particularly when you are taking in a lot of new information.

Breaks are especially important when you are debugging. If you hit a bug and can’t quite figure out what is going wrong, take a break. Step away from your computer, go for a walk, or chat with a friend.

In programming, your code must follow the rules of a language and logic exactly, so even missing a quotation mark will break everything. Fresh eyes make a big difference.

Tip #5: Become a Bug Bounty Hunter

Speaking of hitting a bug, it is inevitable once you start writing complex programs that you will run into bugs in your code. It happens to all of us! Don’t let bugs frustrate you. Instead, embrace these moments with pride and think of yourself as a bug bounty hunter.

When debugging, it is important to have a methodological approach to help you find where things are breaking down. Going through your code in the order in which it is executed and making sure each part works is a great way to do this.

Once you have an idea of where things might be breaking down, insert the following line of code into your script import pdb; pdb.set_trace() and run it. This is the Python debugger and will drop you into interactive mode. The debugger can also be run from the command line with python -m pdb <my_file.py>.

Make It Collaborative

Once things start to stick, expedite your learning through collaboration. Here are some strategies to help you get the most out of working with others.

Tip #6: Surround Yourself With Others Who Are Learning

Though coding may seem like a solitary activity, it actually works best when you work together. It is extremely important when you are learning to code in Python that you surround yourself with other people who are learning as well. This will allow you to share the tips and tricks you learn along the way.

Don’t worry if you don’t know anyone. There are plenty of ways to meet others who are passionate about learning Python! Find local events or Meetups or join PythonistaCafe, a peer-to-peer learning community for Python enthusiasts like you!

Tip #7: Teach

It is said that the best way to learn something is to teach it. This is true when you are learning Python. There are many ways to do this: whiteboarding with other Python lovers, writing blog posts explaining newly learned concepts, recording videos in which you explain something you learned, or simply talking to yourself at your computer. Each of these strategies will solidify your understanding as well as expose any gaps in your understanding.

Tip #8: Pair Program

Pair programming is a technique that involves two developers working at one workstation to complete a task. The two developers switch between being the “driver” and the “navigator.” The “driver” writes the code, while the “navigator” helps guide the problem solving and reviews the code as it is written. Switch frequently to get the benefit of both sides.

Pair programming has many benefits: it gives you a chance to not only have someone review your code, but also see how someone else might be thinking about a problem. Being exposed to multiple ideas and ways of thinking will help you in problem solving when you got back to coding on your own.

Tip #9: Ask “GOOD” Questions

People always say there is no such thing as a bad question, but when it comes to programming, it is possible to ask a question badly. When you are asking for help from someone who has little or no context on the problem you are trying to solve, its best to ask GOOD questions by following this acronym:

G: Give context on what you are trying to do, clearly describing the problem.
O: Outline the things you have already tried to fix the issue.
O: Offer your best guess as to what the problem might be. This helps the person who is helping you to not only know what you are thinking, but also know that you have done some thinking on your own.
D: Demo what is happening. Include the code, a traceback error message, and an explanation of the steps you executed that resulted in the error. This way, the person helping does not have to try to recreate the issue.

Good questions can save a lot of time. Skipping any of these steps can result in back-and-forth conversations that can cause conflict. As a beginner, you want to make sure you ask good questions so that you practice communicating your thought process, and so that people who help you will be happy to continue helping you.

Make Something

Most, if not all, Python developers you speak to will tell you that in order to learn Python, you must learn by doing. Doing exercises can only take you so far: you learn the most by building.

Tip #10: Build Something, Anything

For beginners, there are many small exercises that will really help you become confident with Python, as well as develop the muscle memory that we spoke about above. Once you have a solid grasp on basic data structures (strings, lists, dictionaries, sets), object-oriented programming, and writing classes, it’s time to start building!

What you build is not as important as how you build it. The journey of building is truly what will teach you the most. You can only learn so much from reading Real Python articles and courses. Most of your learning will come from using Python to build something. The problems you will solve will teach you a lot.

There are many lists out there with ideas for beginner Python projects. Here are some ideas to get you started:

Number guessing game
Simple calculator app
Dice roll simulator
Bitcoin Price Notification Service

If you find it difficult to come up with Python practice projects to work on, watch this video. It lays out a strategy you can use to generate thousands of project ideas whenever you feel stuck.

Tip #11: Contribute to Open Source

In the open-source model, software source code is available publicly, and anyone can collaborate. There are many Python libraries that are open-source projects and take contributions. Additionally, many companies publish open-source projects. This means you can work with code written and produced by the engineers working in these companies.

Contributing to an open-source Python project is a great way to create extremely valuable learning experiences. Let’s say you decide to submit a bug fix request: you submit a “pull request” for your fix to be patched into the code.

Next, the project managers will review your work, providing comments and suggestions. This will enable you to learn best practices for Python programming, as well as practice communicating with other developers.

For additional tips and tactics that will help you break into the open-source world, check out the video embedded below:

Go Forth and Learn!

Now that you have these strategies for learning, you are ready to begin your Python journey! Find Real Python’s Beginners Roadmap for Learning here! We also offer a beginner’s level Python course, which uses interesting examples to help you learn programming and web development.

Happy Coding!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

Basic Data Types in Python

Integers

Floating-Point Numbers

Deep Dive: Floating-Point Representation

Complex Numbers

Strings

Escape Sequences in Strings

Suppressing Special Character Meaning

Applying Special Meaning to Characters

Raw Strings

Triple-Quoted Strings

Boolean Type, Boolean Context, and “Truthiness”

Built-In Functions

Math

Type Conversion

Iterables and Iterators

Composite Data Type

Classes, Attributes, and Inheritance

Input/Output

Variables, References, and Scope

Miscellaneous

Conclusion

Python Application Layouts: A Reference

Command-Line Application Layouts

One-Off Script

Installable Single Package

Application with Internal Packages

Web Application Layouts

Django

Flask

Conclusions and Reminders

Itertools in Python 3, By Example

What Is Itertools and Why Should You Use It?

The grouper Recipe

Et tu, Brute Force?

Section Recap

itertools.combinations Example

itertools.combinations_with_replacement Example

itertools.permutations Example

Sequences of Numbers

Evens and Odds

Recurrence Relations

First Order Recurrence Relations

Second Order Recurrence Relations

Section Recap

itertools.count Example

itertools.repeat Example

itertools.cycle Example

itertools accumulate Example

Dealing a Deck of Cards

Section Recap

itertools.product Example

itertools.tee Example

itertools.islice Example

itertools.chain Example

Intermission: Flattening A List of Lists

Analyzing the S&P500

Maximum Gain and Loss

Longest Growth Streak

Section Recap

itertools.filterfalse Example

itertools.takewhile Example

itertools.dropwhile Example

Building Relay Teams From Swimmer Data

Where to Go From Here

Interacting with Python

Hello, World!

Using the Python Interpreter Interactively

Starting the Interpreter

Executing Python Code

Exiting the Interpreter

Running a Python Script from the Command Line

Interacting with Python through an IDE

IDLE

Starting IDLE in Windows

Starting IDLE in macOS

Starting IDLE in Linux

Using IDLE

Thonny

What Is `Itertools` and Why Should You Use It?

The `grouper` Recipe

`itertools.combinations` Example

`itertools.combinations_with_replacement` Example

`itertools.permutations` Example

`itertools.count` Example

`itertools.repeat` Example

`itertools.cycle` Example

`itertools accumulate` Example

`itertools.product` Example

`itertools.tee` Example

`itertools.islice` Example

`itertools.chain` Example

`itertools.filterfalse` Example

`itertools.takewhile` Example

`itertools.dropwhile` Example