Real Python

Documenting Python Code: A Complete Guide

2018-07-25T14:00:00+00:00

Welcome to your complete guide to documenting Python code. Whether you’re documenting a small script or a large project, whether you’re a beginner or seasoned Pythonista, this guide will cover everything you need to know.

We’ve broken up this tutorial into four major sections:

Why Documenting Your Code Is So Important: An introduction to documentation and its importance
Commenting vs. Documenting Code: An overview of the major differences between commenting and documenting, as well as the appropriate times and ways to use commenting
Documenting Your Python Code Base Using Docstrings: A deep dive into docstrings for classes, class methods, functions, modules, packages, and scripts, as well as what should be found within each one
Documenting Your Python Projects: The necessary elements and what they should contain for your Python projects

Feel free to read through this tutorial from beginning to end or jump to a section you’re interested in. It was designed to work both ways.

Why Documenting Your Code Is So Important

Hopefully, if you’re reading this tutorial, you already know the importance of documenting your code. But if not, then let me quote something Guido mentioned to me at a recent PyCon:

“Code is more often read than written.”

— Guido Van Rossum

When you write code, you write it for two primary audiences: your users and your developers (including yourself). Both audiences are equally important. If you’re like me, you’ve probably opened up old codebases and wondered to yourself, “What in the world was I thinking?” If you’re having a problem reading your own code, imagine what your users or other developers are experiencing when they’re trying to use or contribute to your code.

Conversely, I’m sure you’ve run into a situation where you wanted to do something in Python and found what looks like a great library that can get the job done. However, when you start using the library, you look for examples, write-ups, or even official documentation on how to do something specific and can’t immediately find the solution.

After searching, you come to realize that the documentation is lacking or even worse, missing entirely. This is a frustrating feeling that deters you from using the library, no matter how great or efficient the code is. Daniele Procida summarized this situation best:

“It doesn’t matter how good your software is, because if the documentation is not good enough, people will not use it.“

— Daniele Procida

In this guide, you’ll learn from the ground up how to properly document your Python code from the smallest of scripts to the largest of Python projects to help prevent your users from ever feeling too frustrated to use or contribute to your project.

Commenting vs. Documenting Code

Before we can go into how to document your Python code, we need to distinguish documenting from commenting.

In general, commenting is describing your code to/for developers. The intended main audience is the maintainers and developers of the Python code. In conjunction with well-written code, comments help to guide the reader to better understand your code and its purpose and design:

“Code tells you how; Comments tell you why.”

— Jeff Atwood (aka Coding Horror)

Documenting code is describing its use and functionality to your users. While it may be helpful in the development process, the main intended audience is the users. The following section describes how and when to comment your code.

Basics of Commenting Code

Comments are created in Python using the pound sign (#) and should be brief statements no longer than a few sentences. Here’s a simple example:

def hello_world():
    # A simple comment preceding a simple print statement
    print("Hello World")

According to PEP 8, comments should have a maximum length of 72 characters. This is true even if your project changes the max line length to be greater than the recommended 80 characters. If a comment is going to be greater than the comment char limit, using multiple lines for the comment is appropriate:

def hello_long_world():
    # A very long statement that just goes on and on and on and on and
    # never ends until after it's reached the 80 char limit
    print("Hellooooooooooooooooooooooooooooooooooooooooooooooooooooooo World")

Commenting your code serves multiple purposes, including:

Planning and Reviewing: When you are developing new portions of your code, it may be appropriate to first use comments as a way of planning or outlining that section of code. Remember to remove these comments once the actual coding has been implemented and reviewed/tested:
```
# First step
# Second step
# Third step
```

Code Description: Comments can be used to explain the intent of specific sections of code:

# Attempt a connection based on previous settings. If unsuccessful,
# prompt user for new settings.

Algorithmic Description: When algorithms are used, especially complicated ones, it can be useful to explain how the algorithm works or how it’s implemented within your code. It may also be appropriate to describe why a specific algorithm was selected over another.
```
# Using quick sort for performance gains
```
Tagging: The use of tagging can be used to label specific sections of code where known issues or areas of improvement are located. Some examples are: BUG, FIXME, and TODO.
```
# TODO: Add condition for when val is None
```

Comments to your code should be kept brief and focused. Avoid using long comments when possible. Additionally, you should use the following four essential rules as suggested by Jeff Atwood:

Keep comments as close to the code being described as possible. Comments that aren’t near their describing code are frustrating to the reader and easily missed when updates are made.
Don’t use complex formatting (such as tables or ASCII figures). Complex formatting leads to distracting content and can be difficult to maintain over time.
Don’t include redundant information. Assume the reader of the code has a basic understanding of programming principles and language syntax.
Design your code to comment itself. The easiest way to understand code is by reading it. When you design your code using clear, easy-to-understand concepts, the reader will be able to quickly conceptualize your intent.

Remember that comments are designed for the reader, including yourself, to help guide them in understanding the purpose and design of the software.

Commenting Code via Type Hinting (Python 3.5+)

Type hinting was added to Python 3.5 and is an additional form to help the readers of your code. In fact, it takes Jeff’s fourth suggestion from above to the next level. It allows the developer to design and explain portions of their code without commenting. Here’s a quick example:

def hello_name(name: str) -> str:
    return(f"Hello {name}")

From examining the type hinting, you can immediately tell that the function expects the input name to be of a type str, or string. You can also tell that the expected output of the function will be of a type str, or string, as well. While type hinting helps reduce comments, take into consideration that doing so may also make extra work when you are creating or updating your project documentation.

You can learn more about type hinting and type checking from this video created by Dan Bader.

Documenting Your Python Code Base Using Docstrings

Now that we’ve learned about commenting, let’s take a deep dive into documenting a Python code base. In this section, you’ll learn about docstrings and how to use them for documentation. This section is further divided into the following sub-sections:

Docstrings Background: A background on how docstrings work internally within Python
Docstring Types: The various docstring “types” (function, class, class method, module, package, and script)
Docstring Formats: The different docstring “formats” (Google, NumPy/SciPy, reStructured Text, and Epytext)

Docstrings Background

Documenting your Python code is all centered on docstrings. These are built-in strings that, when configured correctly, can help your users and yourself with your project’s documentation. Along with docstrings, Python also has the built-in function help() that prints out the objects docstring to the console. Here’s a quick example:

>>> help(str)
Help on class str in module builtins:

class str(object)
 |  str(object='') -> str
 |  str(bytes_or_buffer[, encoding[, errors]]) -> str
 |
 |  Create a new string object from the given object. If encoding or
 |  errors are specified, then the object must expose a data buffer
 |  that will be decoded using the given encoding and error handler.
 |  Otherwise, returns the result of object.__str__() (if defined)
 |  or repr(object).
 |  encoding defaults to sys.getdefaultencoding().
 |  errors defaults to 'strict'.
 # Truncated for readability

How is this output generated? Since everything in Python is an object, you can examine the directory of the object using the dir() command. Let’s do that and see what find:

>>> dir(str)
['__add__', ..., '__doc__', ..., 'zfill'] # Truncated for readability

Within that directory output, there’s an interesting property, __doc__. If you examine that property, you’ll discover this:

>>> print(str.__doc__)
str(object='') -> str
str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or
errors are specified, then the object must expose a data buffer
that will be decoded using the given encoding and error handler.
Otherwise, returns the result of object.__str__() (if defined)
or repr(object).
encoding defaults to sys.getdefaultencoding().
errors defaults to 'strict'.

Voilà! You’ve found where docstrings are stored within the object. This means that you can directly manipulate that property. However, there are restrictions for builtins:

>>> str.__doc__ = "I'm a little string doc! Short and stout; here is my input and print me for my out"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't set attributes of built-in/extension type 'str'

Any other custom object can be manipulated:

def say_hello(name):
    print(f"Hello {name}, is it me you're looking for?")

say_hello.__doc__ = "A simple function that says hello... Richie style"

>>> help(say_hello)
Help on function say_hello in module __main__:

say_hello(name)
    A simple function that says hello... Richie style

Python has one more feature that simplifies docstring creation. Instead of directly manipulating the __doc__ property, the strategic placement of the string literal directly below the object will automatically set the __doc__ value. Here’s what happens with the same example as above:

def say_hello(name):
    """A simple function that says hello... Richie style"""
    print(f"Hello {name}, is it me you're looking for?")

>>> help(say_hello)
Help on function say_hello in module __main__:

say_hello(name)
    A simple function that says hello... Richie style

There you go! Now you understand the background of docstrings. Now it’s time to learn about the different types of docstrings and what information they should contain.

Docstring Types

Docstring conventions are described within PEP 257. Their purpose is to provide your users with a brief overview of the object. They should be kept concise enough to be easy to maintain but still be elaborate enough for new users to understand their purpose and how to use the documented object.

In all cases, the docstrings should use the triple-double quote (""") string format. This should be done whether the docstring is multi-lined or not. At a bare minimum, a docstring should be a quick summary of whatever is it you’re describing and should be contained within a single line:

"""This is a quick summary line used as a description of the object."""

Multi-lined docstrings are used to further elaborate on the object beyond the summary. All multi-lined docstrings have the following parts:

A one-line summary line
A blank line proceeding the summary
Any further elaboration for the docstring
Another blank line

"""This is the summary line

This is the further elaboration of the docstring. Within this section,
you can elaborate further on details as appropriate for the situation.
Notice that the summary and the elaboration is separated by a blank new
line.
"""

# Notice the blank line above. Code should continue on this line.

All docstrings should have the same max character length as comments (72 characters). Docstrings can be further broken up into three major categories:

Class Docstrings: Class and class methods
Package and Module Docstrings: Package, modules, and functions
Script Docstrings: Script and functions

Class Docstrings

Class Docstrings are created for the class itself, as well as any class methods. The docstrings are placed immediately following the class or class method indented by one level:

class SimpleClass:
    """Class docstrings go here."""

    def say_hello(self, name: str):
        """Class method docstrings go here."""

        print(f'Hello {name}')

Class docstrings should contain the following information:

A brief summary of its purpose and behavior
Any public methods, along with a brief description
Any class properties (attributes)
Anything related to the interface for subclassers, if the class is intended to be subclassed

The class constructor parameters should be documented within the __init__ class method docstring. Individual methods should be documented using their individual docstrings. Class method docstrings should contain the following:

A brief description of what the method is and what it’s used for
Any arguments (both required and optional) that are passed including keyword arguments
Label any arguments that are considered optional or have a default value
Any side effects that occur when executing the method
Any exceptions that are raised
Any restrictions on when the method can be called

Let’s take a simple example of a data class that represents an Animal. This class will contain a few class properties, instance properties, a __init__, and a single instance method:

class Animal:
    """
    A class used to represent an Animal

    ...

    Attributes
    ----------
    says_str : str
        a formatted string to print out what the animal says
    name : str
        the name of the animal
    sound : str
        the sound that the animal makes
    num_legs : int
        the number of legs the animal has (default 4)

    Methods
    -------
    says(sound=None)
        Prints the animals name and what sound it makes
    """

    says_str = "A {name} says {sound}"

    def __init__(self, name, sound, num_legs):
        """
        Parameters
        ----------
        name : str
            The name of the animal
        sound : str
            The sound the animal makes
        num_legs : int, optional
            The number of legs the animal (default is 4)
        """

        self.name = name
        self.sound = sound
        self.num_legs = num_legs

    def says(self, sound=None):
        """Prints what the animals name is and what sound it makes.

        If the argument `sound` isn't passed in, the default Animal
        sound is used.

        Parameters
        ----------
        sound : str, optional
            The sound the animal makes (default is None)

        Raises
        ------
        NotImplementedError
            If no sound is set for the animal or passed in as a
            parameter.
        """

        if self.sound is None and sound is None:
            raise NotImplementedError("Silent Animals are not supported!")

        out_sound = self.sound if sound is None else sound
        print(self.says_str.format(name=self.name, sound=out_sound))

Package and Module Docstrings

Package docstrings should be placed at the top of the package’s __init__.py file. This docstring should list the modules and sub-packages that are exported by the package.

Module docstrings are similar to class docstrings. Instead of classes and class methods being documented, it’s now the module and any functions found within. Module docstrings are placed at the top of the file even before any imports. Module docstrings should include the following:

A brief description of the module and its purpose
A list of any classes, exception, functions, and any other objects exported by the module

The docstring for a module function should include the same items as a class method:

A brief description of what the function is and what it’s used for
Any arguments (both required and optional) that are passed including keyword arguments
Label any arguments that are considered optional
Any side effects that occur when executing the function
Any exceptions that are raised
Any restrictions on when the function can be called

Script Docstrings

Scripts are considered to be single file executables run from the console. Docstrings for scripts are placed at the top of the file and should be documented well enough for users to be able to have a sufficient understanding of how to use the script. It should be usable for its “usage” message, when the user incorrectly passes in a parameter or uses the -h option.

If you use argparse, then you can omit parameter-specific documentation, assuming it’s correctly been documented within the help parameter of the argparser.parser.add_argument function. It is recommended to use the __doc__ for the description parameter within argparse.ArgumentReader’s constructor. Check out our tutorial on Command-Line Parsing Libraries for more details on how to use argparse and other common command line parsers.

Finally, any custom or third-party imports should be listed within the docstrings to allow users to know which packages may be required for running the script. Here’s an example of a script that is used to simply print out the column headers of a spreadsheet:

"""Spreadsheet Column Printer

This script allows the user to print to the console all columns in the
spreadsheet. It is assumed that the first row of the spreadsheet is the
location of the columns.

This tool accepts comma separated value files (.csv) as well as excel
(.xls, .xlsx) files.

This script requires that `pandas` be installed within the Python
environment you are running this script in.

This file can also be imported as a module and contains the following
functions:

    * get_spreadsheet_cols - returns the column headers of the file
    * main - the main function of the script
"""

import argparse

import pandas as pd


def get_spreadsheet_cols(file_loc, print_cols=False):
    """Gets and prints the spreadsheet's header columns

    Parameters
    ----------
    file_loc : str
        The file location of the spreadsheet
    print_cols : bool, optional
        A flag used to print the columns to the console (default is
        False)

    Returns
    -------
    list
        a list of strings used that are the header columns
    """

    file_data = pd.read_excel(file_loc)
    col_headers = list(file_data.columns.values)

    if print_cols:
        print("\n".join(col_headers))

    return col_headers


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument(
        'input_file',
        type=str,
        help="The spreadsheet file to pring the columns of"
    )
    args = parser.parse_args()
    get_spreadsheet_cols(args.input_file, print_cols=True)


if __name__ == "__main__":
    main()

Docstring Formats

You may have noticed that, throughout the examples given in this tutorial, there has been specific formatting with common elements: Arguments, Returns, and Attributes. There are specific docstrings formats that can be used to help docstring parsers and users have a familiar and known format. The formatting used within the examples in this tutorial are NumPy/SciPy-style docstrings. Some of the most common formats are the following:

Formatting Type	Description	Supported by Sphynx	Formal Specification
Google docstrings	Google’s recommended form of documentation	Yes	No
reStructured Text	Official Python documentation standard; Not beginner friendly but feature rich	Yes	Yes
NumPy/SciPy docstrings	NumPy’s combination of reStructured and Google Docstrings	Yes	Yes
Epytext	A Python adaptation of Epydoc; Great for Java developers	Not officially	Yes

The selection of the docstring format is up to you, but you should stick with the same format throughout your document/project. The following are examples of each type to give you an idea of how each documentation format looks.

Google Docstrings Example

"""Gets and prints the spreadsheet's header columns

Parameters:
    file_loc (str): The file location of the spreadsheet
    print_cols (bool): A flag used to print the columns to the console
        (default is False)

Returns:
    list: a list of strings representing the header columns
"""

reStructured Text Example

"""Gets and prints the spreadsheet's header columns

:param file_loc: The file location of the spreadsheet
:type file_loc: str
:param print_cols: A flag used to print the columns to the console
    (default is False)
:type print_cols: bool
:returns: a list of strings representing the header columns
:rtype: list
"""

NumPy/SciPy Docstrings Example

"""Gets and prints the spreadsheet's header columns

Parameters
----------
file_loc : str
    The file location of the spreadsheet
print_cols : bool, optional
    A flag used to print the columns to the console (default is False)

Returns
-------
list
    a list of strings representing the header columns
"""

Epytext Example

"""Gets and prints the spreadsheet's header columns

@type file_loc: str
@param file_loc: The file location of the spreadsheet
@type print_cols: bool
@param print_cols: A flag used to print the columns to the console
    (default is False)
@rtype: list
@returns: a list of strings representing the header columns
"""

Documenting Your Python Projects

Python projects come in all sorts of shapes, sizes, and purposes. The way you document your project should suit your specific situation. Keep in mind who the users of your project are going to be and adapt to their needs. Depending on the project type, certain aspects of documentation are recommended. The general layout of the project and its documentation should be as follows:

project_root/
│
├── project/  # Project source code
├── docs/
├── README
├── HOW_TO_CONTRIBUTE
├── CODE_OF_CONDUCT
├── examples.py

Projects can be generally subdivided into three major types: Private, Shared, and Public/Open Source.

Private Projects

Private projects are projects intended for personal use only and generally aren’t shared with other users or developers. Documentation can be pretty light on these types of projects. There are some recommended parts to add as needed:

Readme: A brief summary of the project and its purpose. Include any special requirements for installation or operating the project.
examples.py: A Python script file that gives simple examples of how to use the project.

Remember, even though private projects are intended for you personally, you are also considered a user. Think about anything that may be confusing to you down the road and make sure to capture those in either comments, docstrings, or the readme.

Shared Projects

Shared projects are projects in which you collaborate with a few other people in the development and/or use of the project. The “customer” or user of the project continues to be yourself and those limited few that use the project as well.

Documentation should be a little more rigorous than it needs to be for a private project, mainly to help onboard new members to the project or alert contributors/users of new changes to the project. Some of the recommended parts to add to the project are the following:

Readme: A brief summary of the project and its purpose. Include any special requirements for installing or operating the project. Additionally, add any major changes since the previous version.
examples.py: A Python script file that gives simple examples of how to use the projects.
How to Contribute: This should include how new contributors to the project can start contributing.

Public and Open Source Projects

Public and Open Source projects are projects that are intended to be shared with a large group of users and can involve large development teams. These projects should place as high of a priority on project documentation as the actual development of the project itself. Some of the recommended parts to add to the project are the following:

Readme: A brief summary of the project and its purpose. Include any special requirements for installing or operating the projects. Additionally, add any major changes since the previous version. Finally, add links to further documentation, bug reporting, and any other important information for the project. Dan Bader has put together a great tutorial on what all should be included in your readme.
How to Contribute: This should include how new contributors to the project can help. This includes developing new features, fixing known issues, adding documentation, adding new tests, or reporting issues.
Code of Conduct: Defines how other contributors should treat each other when developing or using your software. This also states what will happen if this code is broken. If you’re using Github, a Code of Conduct template can be generated with recommended wording. For Open Source projects especially, consider adding this.
License: A plaintext file that describes the license your project is using. For Open Source projects especially, consider adding this.
docs: A folder that contains further documentation. The next section describes more fully what should be included and how to organize the contents of this folder.

The Four Main Sections of the `docs` Folder

Daniele Procida gave a wonderful PyCon 2017 talk and subsequent blog post about documenting Python projects. He mentions that all projects should have the following four major sections to help you focus your work:

Tutorials: Lessons that take the reader by the hand through a series of steps to complete a projects (or meaningful exercise). Geared towards the users learning.
How-To Guides: Guides that take the reader through the steps required to solve a common problem (problem-oriented recipes).
References: Explanations that clarify and illuminate a particular topic. Geared towards understanding.
Explanations: Technical descriptions of the machinery and how to operate it (key classes, functions, APIs, and so forth). Think Encyclopedia article.

The following table shows how all of these sections relates to each other as well as their overall purpose:

	Most Useful When We’re Studying	Most Useful When We’re Coding
Practical Step	Tutorials	How-To Guides
Theoretical Knowledge	Explanation	Reference

In the end, you want to make sure that your users have access to the answers to any questions they may have. By organizing your project in this manner, you’ll be able to answer those questions easily and in a format they’ll be able to navigate quickly.

Documentation Tools and Resources

Documenting your code, especially large projects, can be daunting. Thankfully there are some tools out and references to get you started:

Tool	Description
Sphinx	A collection of tools to auto-generate documentation in multiple formats
Epydoc	A tool for generating API documentation for Python modules based on their docstrings
Read The Docs	Automatic building, versioning, and hosting of your docs for you
Doxygen	A tool for generating documentation that supports Python as well as multiple other languages
MkDocs	A static site generator to help build project documentation using the Markdown language
pycco	A “quick and dirty” documentation generator that displays code and documentation side by side. Check out our tutorial on how to use it for more info.

Along with these tools, there are some additional tutorials, videos, and articles that can be useful when you are documenting your project:

Sometimes, the best way to learn is to mimic others. Here are some great examples of projects that use documentation well:

Django: Docs (Source)
Requests: Docs (Source)
Click: Docs (Source)
Pandas: Docs (Source)

Where Do I Start?

The documentation of projects have a simple progression:

No Documentation
Some Documentation
Complete Documentation
Good Documentation
Great Documentation

If you’re at a loss about where to go next with your documentation, look at where your project is now in relation to the progression above. Do you have any documentation? If not, then start there. If you have some documentation but are missing some of the key project files, get started by adding those.

In the end, don’t get discouraged or overwhelmed by the amount of work required for documenting code. Once you get started documenting your code, it becomes easier to keep going. Feel free to comment if you have questions or reach out to the Real Python Team on social media, and we’ll help.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Fast, Flexible, Easy and Intuitive: How to Speed Up Your Pandas Projects

2018-07-23T14:00:00+00:00

If you work with big data sets, you probably remember the “aha” moment along your Python journey when you discovered the Pandas library. Pandas is a game-changer for data science and analytics, particularly if you came to Python because you were searching for something more powerful than Excel and VBA.

So what is it about Pandas that has data scientists, analysts, and engineers like me raving? Well, the Pandas documentation says that it uses:

“fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive.”

Fast, flexible, easy, and intuitive? That sounds great! If your job involves building complicated data models, you don’t want to spend half of your development hours waiting for modules to churn through big data sets. You want to dedicate your time and brainpower to interpreting your data, rather than painstakingly fumbling around with less powerful tools.

But I Heard That Pandas Is Slow…

When I first started using Pandas, I was advised that, while it was a great tool for dissecting data, Pandas was too slow to use as a statistical modeling tool. Starting out, this proved true. I spent more than a few minutes twiddling my thumbs, waiting for Pandas to churn through data.

But then I learned that Pandas is built on top of the NumPy array structure, and so many of its operations are carried out in C, either via NumPy or through Pandas’ own library of Python extension modules that are written in Cython and compiled to C. So, shouldn’t Pandas be fast too?

It absolutely should be, if you use it the way it was intended!

The paradox is that what may otherwise “look like” Pythonic code can be suboptimal in Pandas as far as efficiency is concerned. Like NumPy, Pandas is designed for vectorized operations that operate on entire columns or datasets in one sweep. Thinking about each “cell” or row individually should generally be a last resort, not a first.

This Tutorial

To be clear, this is not a guide about how to over-optimize your Pandas code. Pandas is already built to run quickly if used correctly. Also, there’s a big difference between optimization and writing clean code.

This is a guide to using Pandas Pythonically to get the most out of its powerful and easy-to-use built-in features. Additionally, you will learn a couple of practical time-saving tips, so you won’t be twiddling those thumbs every time you work with your data.

In this tutorial, you’ll cover the following:

Advantages of using datetime data with time series
The most efficient route to doing batch calculations
Saving time by storing data with HDFStore

To demonstrate these topics, I’ll take an example from my day job that looks at a time series of electricity consumption. After loading the data, you’ll successively progress through more efficient ways to get to the end result. One adage that holds true for most of Pandas is that there is more than one way to get from A to B. This doesn’t mean, however, that all of the available options will scale equally well to larger, more demanding datasets.

Assuming that you already know how to do some basic data selection in Pandas, let’s get started.

The Task at Hand

The goal of this example will be to apply time-of-use energy tariffs to find the total cost of energy consumption for one year. That is, at different hours of the day, the price for electricity varies, so the task is to multiply the electricity consumed for each hour by the correct price for the hour in which it was consumed.

Let’s read our data from a CSV file that has two columns: one for date plus time and one for electrical energy consumed in kilowatt hours (kWh):

The rows contains the electricity used in each hour, so there are 365 x 24 = 8760 rows for the whole year. Each row indicates the usage for the “hour starting” at the time, so 1/1/13 0:00 indicates the usage for the first hour of January 1st.

Saving Time With Datetime Data

The first thing you need to do is to read your data from the CSV file with one of Pandas’ I/O functions:

>>> import pandas as pd
>>> pd.__version__
'0.23.1'

# Make sure that `demand_profile.csv` is in your
# current working directory.
>>> df = pd.read_csv('demand_profile.csv')
>>> df.head()
     date_time  energy_kwh
0  1/1/13 0:00       0.586
1  1/1/13 1:00       0.580
2  1/1/13 2:00       0.572
3  1/1/13 3:00       0.596
4  1/1/13 4:00       0.592

This looks okay at first glance, but there’s a small issue. Pandas and NumPy have a concept of dtypes (data types). If no arguments are specified, date_time will take on an object dtype:

>>> df.dtypes
date_time      object
energy_kwh    float64
dtype: object

>>> type(df.iat[0, 0])
str

This is not ideal. object is a container for not just str, but any column that can’t neatly fit into one data type. It would be arduous and inefficient to work with dates as strings. (It would also be memory-inefficient.)

For working with time series data, you’ll want the date_time column to be formatted as an array of datetime objects. (Pandas calls this a Timestamp.) Pandas makes each step here rather simple:

>>> df['date_time'] = pd.to_datetime(df['date_time'])
>>> df['date_time'].dtype
datetime64[ns]

(Note that you could alternatively use a Pandas PeriodIndex in this case.)

You now have a DataFrame called df that looks much like our CSV file. It has two columns and a numerical index for referencing the rows.

>>> df.head()
               date_time    energy_kwh
0    2013-01-01 00:00:00         0.586
1    2013-01-01 01:00:00         0.580
2    2013-01-01 02:00:00         0.572
3    2013-01-01 03:00:00         0.596
4    2013-01-01 04:00:00         0.592

The code above is simple and easy, but how fast it? Let’s put it to the test using a timing decorator, which I have unoriginally called @timeit. This decorator largely mimics timeit.repeat() from Python’s standard library, but it allows you to return the result of the function itself and print its average runtime from multiple trials. (Python’s timeit.repeat() returns the timing results, not the function result.)

Creating a function and placing the @timeit decorator directly above it will mean that every time the function is called, it will be timed. The decorator runs an outer loop and an inner loop:

>>> @timeit(repeat=3, number=10)
... def convert(df, column_name):
...     return pd.to_datetime(df[column_name])

>>> # Read in again so that we have `object` dtype to start 
>>> df['date_time'] = convert(df, 'date_time')
Best of 3 trials with 10 function calls per trial:
Function `convert` ran in average of 1.610 seconds.

The result? 1.6 seconds for 8760 rows of data. “Great,” you might say, “that’s no time at all.” But what if you encounter larger data sets—say, one year of electricity use at one-minute intervals? That’s 60 times more data, so you’ll end up waiting around one and a half minutes. That’s starting to sound less tolerable.

In actuality, I recently analyzed 10 years of hourly electricity data from 330 sites. Do you think I waited 88 minutes to convert datetimes? Absolutely not!

How can you speed this up? As a general rule, Pandas will be far quicker the less it has to interpret your data. In this case, you will see huge speed improvements just by telling Pandas what your time and date data looks like, using the format parameter. You can do this by using the strftime codes found here and entering them like this:

>>> @timeit(repeat=3, number=100)
>>> def convert_with_format(df, column_name):
...     return pd.to_datetime(df[column_name],
...                           format='%d/%m/%y %H:%M')
Best of 3 trials with 100 function calls per trial:
Function `convert_with_format` ran in average of 0.032 seconds.

The new result? 0.032 seconds, which is 50 times faster! So you’ve just saved about 86 minutes of processing time for my 330 sites. Not a bad improvement!

One finer detail is that the datetimes in the CSV are not in ISO 8601 format: you’d need YYYY-MM-DD HH:MM. If you don’t specify a format, Pandas will use the dateutil package to convert each string to a date.

Conversely, if the raw datetime data is already in ISO 8601 format, Pandas can immediately take a fast route to parsing the dates. This is one reason why being explicit about the format is so beneficial here. Another option is to pass infer_datetime_format=True parameter, but it generally pays to be explicit.

Note: Pandas’ read_csv() also allows you to parse dates as a part of the file I/O step. See the parse_dates, infer_datetime_format, and date_parser parameters.

Simple Looping Over Pandas Data

Now that your dates and times are in a convenient format, you are ready to get down to the business of calculating your electricity costs. Remember that cost varies by hour, so you will need to conditionally apply a cost factor to each hour of the day. In this example, the time-of-use costs will be defined as follows:

Tariff Type	Cents per kWh	Time Range
Peak	28	17:00 to 24:00
Shoulder	20	7:00 to 17:00
Off-Peak	12	0:00 to 7:00

If the price were a flat 28 cents per kWh for every hour of the day, most people familiar with Pandas would know that this calculation could be achieved in one line:

>>> df['cost_cents'] = df['energy_kwh'] * 28

This will result in the creation of a new column with the cost of electricity for that hour:

               date_time    energy_kwh       cost_cents
0    2013-01-01 00:00:00         0.586           16.408
1    2013-01-01 01:00:00         0.580           16.240
2    2013-01-01 02:00:00         0.572           16.016
3    2013-01-01 03:00:00         0.596           16.688
4    2013-01-01 04:00:00         0.592           16.576
# ...

But our cost calculation is conditional on the time of day. This is where you will see a lot of people using Pandas the way it was not intended: by writing a loop to do the conditional calculation.

For the rest of this tutorial, you’ll start from a less-than-ideal baseline solution and work up to a Pythonic solution that fully leverages Pandas.

But what is Pythonic in the case of Pandas? The irony is that it is those who are experienced in other (less user-friendly) coding languages such as C++ or Java that are particularly susceptible to this because they instinctively “think in loops.”

Let’s look at a loop approach that is not Pythonic and that many people take when they are unaware of how Pandas is designed to be used. We will use @timeit again to see how fast this approach is.

First, let’s create a function to apply the appropriate tariff to a given hour:

def apply_tariff(kwh, hour):
    """Calculates cost of electricity for given hour."""    
    if 0 <= hour < 7:
        rate = 12
    elif 7 <= hour < 17:
        rate = 20
    elif 17 <= hour < 24:
        rate = 28
    else:
        raise ValueError(f'Invalid hour: {hour}')
    return rate * kwh

Here’s the loop that isn’t Pythonic, in all its glory:

>>> # NOTE: Don't do this!
>>> @timeit(repeat=3, number=100)
... def apply_tariff_loop(df):
...     """Calculate costs in loop.  Modifies `df` inplace."""
...     energy_cost_list = []
...     for i in range(len(df)):
...         # Get electricity used and hour of day
...         energy_used = df.iloc[i]['energy_kwh']
...         hour = df.iloc[i]['date_time'].hour
...         energy_cost = apply_tariff(energy_used, hour)
...         energy_cost_list.append(energy_cost)
...     df['cost_cents'] = energy_cost_list
... 
>>> apply_tariff_loop(df)
Best of 3 trials with 100 function calls per trial:
Function `apply_tariff_loop` ran in average of 3.152 seconds.

For people who picked up Pandas after having written “pure Python” for some time prior, this design might seem natural: you have a typical “for each x, conditional on y, do z.”

However, this loop is clunky. You can consider the above to be an “antipattern” in Pandas for several reasons. Firstly, it needs to initialize a list in which the outputs will be recorded.

Secondly, it uses the opaque object range(0, len(df)) to loop over, and then after applying apply_tariff(), it has to append the result to a list that is used to make the new DataFrame column. It also does what is called chained indexing with df.iloc[i]['date_time'], which often leads to unintended results.

But the biggest issue with this approach is the time cost of the calculations. On my machine, this loop took over 3 seconds for 8760 rows of data. Next, you’ll look at some improved solutions for iteration over Pandas structures.

Looping with `.itertuples()` and `.iterrows()`

What other approaches can you take? Well, Pandas has actually made the for i in range(len(df)) syntax redundant by introducing the DataFrame.itertuples() and DataFrame.iterrows() methods. These are both generator methods that yield one row at a time.

.itertuples() yields a namedtuple for each row, with the row’s index value as the first element of the tuple. A nametuple is a data structure from Python’s collections module that behaves like a Python tuple but has fields accessible by attribute lookup.

.iterrows() yields pairs (tuples) of (index, Series) for each row in the DataFrame.

While .itertuples() tends to be a bit faster, let’s stay in Pandas and use .iterrows() in this example, because some readers might not have run across nametuple. Let’s see what this achieves:

>>> @timeit(repeat=3, number=100)
... def apply_tariff_iterrows(df):
...     energy_cost_list = []
...     for index, row in df.iterrows():
...         # Get electricity used and hour of day
...         energy_used = row['energy_kwh']
...         hour = row['date_time'].hour
...         # Append cost list
...         energy_cost = apply_tariff(energy_used, hour)
...         energy_cost_list.append(energy_cost)
...     df['cost_cents'] = energy_cost_list
...
>>> apply_tariff_iterrows(df)
Best of 3 trials with 100 function calls per trial:
Function `apply_tariff_iterrows` ran in average of 0.713 seconds.

Some marginal gains have been made. The syntax is more explicit, and there is less clutter in your row value references, so it’s more readable. In terms of time gains, is almost 5 five times quicker!

However, there is more room for improvement. You’re still using some form of a Python for-loop, meaning that each and every function call is done in Python when it could ideally be done in a faster language built into Pandas’ internal architecture.

Pandas’ `.apply()`

You can further improve this operation using the .apply() method instead of .iterrows(). Pandas’ .apply() method takes functions (callables) and applies them along an axis of a DataFrame (all rows, or all columns). In this example, a lambda function will help you pass the two columns of data into apply_tariff():

>>> @timeit(repeat=3, number=100)
... def apply_tariff_withapply(df):
...     df['cost_cents'] = df.apply(
...         lambda row: apply_tariff(
...             kwh=row['energy_kwh'],
...             hour=row['date_time'].hour),
...         axis=1)
...
>>> apply_tariff_withapply(df)
Best of 3 trials with 100 function calls per trial:
Function `apply_tariff_withapply` ran in average of 0.272 seconds.

The syntactic advantages of .apply() are clear, with a significant reduction in the number of lines and very readable, explicit code. In this case, the time taken was roughly half that of the .iterrows() method.

However, this is not yet “blazingly fast.” One reason is that .apply() will try internally to loop over Cython iterators. But in this case, the lambda that you passed isn’t something that can be handled in Cython, so it’s called in Python, which is consequently not all that fast.

If you were to use .apply() for my 10 years of hourly data for 330 sites, you’d be looking at around 15 minutes of processing time. If this calculation were intended to be a small part of a larger model, you’d really want to speed things up. That’s where vectorized operations come in handy.

Selecting Data With `.isin()`

Earlier, you saw that if there were a single electricity price, you could apply that price across all the electricity consumption data in one line of code (df['energy_kwh'] * 28). This particular operation was an example of a vectorized operation, and it is the fastest way to do things in Pandas.

But how can you apply condition calculations as vectorized operations in Pandas? One trick is to select and group parts the DataFrame based on your conditions and then apply a vectorized operation to each selected group.

In this next example, you will see how to select rows with Pandas’ .isin() method and then apply the appropriate tariff in a vectorized operation. Before you do this, it will make things a little more convenient if you set the date_time column as the DataFrame’s index:

df.set_index('date_time', inplace=True)

@timeit(repeat=3, number=100)
def apply_tariff_isin(df):
    # Define hour range Boolean arrays
    peak_hours = df.index.hour.isin(range(17, 24))
    shoulder_hours = df.index.hour.isin(range(7, 17))
    off_peak_hours = df.index.hour.isin(range(0, 7))

    # Apply tariffs to hour ranges
    df.loc[peak_hours, 'cost_cents'] = df.loc[peak_hours, 'energy_kwh'] * 28
    df.loc[shoulder_hours,'cost_cents'] = df.loc[shoulder_hours, 'energy_kwh'] * 20
    df.loc[off_peak_hours,'cost_cents'] = df.loc[off_peak_hours, 'energy_kwh'] * 12

Let’s see how this compares:

>>> apply_tariff_isin(df)
Best of 3 trials with 100 function calls per trial:
Function `apply_tariff_isin` ran in average of 0.010 seconds.

To understand what’s happening in this code, you need to know that the .isin() method is returning an array of Boolean values that looks like this:

[False, False, False, ..., True, True, True]

These values identify which DataFrame indices (datetimes) fall within the hour range specified. Then, when you pass these Boolean arrays to the DataFrame’s .loc indexer, you get a slice of the DataFrame that only includes rows that match those hours. After that, it is simply a matter of multiplying the slice by the appropriate tariff, which is a speedy vectorized operation.

How does this compare to our looping operations above? Firstly, you may notice that you no longer need apply_tariff(), because all the conditional logic is applied in the selection of the rows. So there is a huge reduction in the lines of code you have to write and in the Python code that is called.

What about the processing time? 315 times faster than the loop that wasn’t Pythonic, around 71 times faster than .iterrows() and 27 times faster that .apply(). Now you are moving at the kind of speed you need to get through big data sets nice and quickly.

Can We Do Better?

In apply_tariff_isin(), we are still admittedly doing some “manual work” by calling df.loc and df.index.hour.isin() three times each. You could argue that this solution isn’t scalable if we had a more granular range of time slots. (A different rate for each hour would require 24 .isin() calls.) Luckily, you can do things even more programmatically with Pandas’ pd.cut() function in this case:

@timeit(repeat=3, number=100)
def apply_tariff_cut(df):
    cents_per_kwh = pd.cut(x=df.index.hour,
                           bins=[0, 7, 17, 24],
                           include_lowest=True,
                           labels=[12, 20, 28]).astype(int)
    df['cost_cents'] = cents_per_kwh * df['energy_kwh']

Let’s take a second to see what’s going on here. pd.cut() is applying an array of labels (our costs) according to which bin each hour belongs in. Note that the include_lowest parameter indicates whether the first interval should be left-inclusive or not. (You want to include time=0 in a group.)

This is a fully vectorized way to get to your intended result, and it comes out on top in terms of timing:

>>> apply_tariff_cut(df)
Best of 3 trials with 100 function calls per trial:
Function `apply_tariff_cut` ran in average of 0.003 seconds.

So far, you’ve built up from taking potentially over an hour to under a second to process the full 300-site dataset. Not bad! There is one last option, though, which is to use NumPy functions to manipulate the underlying NumPy arrays for each DataFrame, and then to integrate the results back into Pandas data structures.

Don’t Forget NumPy!

One point that should not be forgotten when you are using Pandas is that Pandas Series and DataFrames are designed on top of the NumPy library. This gives you even more computation flexibility, because Pandas works seamlessly with NumPy arrays and operations.

In this next case you’ll use NumPy’s digitize() function. It is similar to Pandas’ cut() in that the data will be binned, but this time it will be represented by an array of indexes representing which bin each hour belongs to. These indexes are then applied to a prices array:

@timeit(repeat=3, number=100)
def apply_tariff_digitize(df):
    prices = np.array([12, 20, 28])
    bins = np.digitize(df.index.hour.values, bins=[7, 17, 24])
    df['cost_cents'] = prices[bins] * df['energy_kwh'].values

Like the cut() function, this syntax is wonderfully concise and easy to read. But how does it compare in speed? Let’s see:

>>> apply_tariff_digitize(df)
Best of 3 trials with 100 function calls per trial:
Function `apply_tariff_digitize` ran in average of 0.002 seconds.

At this point, there’s still a performance improvement, but it’s becoming more marginal in nature. This is probably a good time to call it a day on hacking away at code improvement and think about the bigger picture.

With Pandas, it can help to maintain “hierarchy,” if you will, of preferred options for doing batch calculations like you’ve done here. These will usually rank from fastest to slowest (and most to least flexible):

Use vectorized operations: Pandas methods and functions with no for-loops.
Use the .apply() method with a callable.
Use .itertuples(): iterate over DataFrame rows as namedtuples from Python’s collections module.
Use .iterrows(): iterate over DataFrame rows as (index, pd.Series) pairs. While a Pandas Series is a flexible data structure, it can be costly to construct each row into a Series and then access it.
Use “element-by-element” for loops, updating each cell or row one at a time with df.loc or df.iloc. (Or, .at/.iat for fast scalar access.)

Don’t Take My Word For It: The order of precedence above is a suggestion straight from a core Pandas developer.

Here’s the “order of precedence” above at work, with each function you’ve built here:

Function	Runtime (seconds)
`apply_tariff_loop()`	3.152
`apply_tariff_iterrows()`	0.713
`apply_tariff_withapply()`	0.272
`apply_tariff_isin()`	0.010
`apply_tariff_cut()`	0.003
`apply_tariff_digitize()`	0.002

Prevent Reprocessing with HDFStore

Now that you have looked at quick data processes in Pandas, let’s explore how to avoid reprocessing time altogether with HDFStore, which was recently integrated into Pandas.

Often when you are building a complex data model, it is convenient to do some pre-processing of your data. For example, if you had 10 years of minute-frequency electricity consumption data, simply converting the date and time to datetime might take 20 minutes, even if you specify the format parameter. You really only want to have to do this once, not every time you run your model, for testing or analysis.

A very useful thing you can do here is pre-process and then store your data in its processed form to be used when needed. But how can you store data in the right format without having to reprocess it again? If you were to save as CSV, you would simply lose your datetime objects and have to re-process it when accessing again.

Pandas has a built-in solution for this which uses HDF5 , a high-performance storage format designed specifically for storing tabular arrays of data. Pandas’ HDFStore class allows you to store your DataFrame in an HDF5 file so that it can be accessed efficiently, while still retaining column types and other metadata. It is a dictionary-like class, so you can read and write just as you would for a Python dict object.

Here’s how you would go about storing your pre-processed electricity consumption DataFrame, df, in an HDF5 file:

# Create storage object with filename `processed_data`
data_store = pd.HDFStore('processed_data.h5')

# Put DataFrame into the object setting the key as 'preprocessed_df'
data_store['preprocessed_df'] = df
data_store.close()

Now you can shut your computer down and take a break knowing that you can come back and your processed data will be waiting for you when you need it. No reprocessing required. Here’s how you would access your data from the HDF5 file, with data types preserved:

# Access data store
data_store = pd.HDFStore('processed_data.h5')

# Retrieve data using key
preprocessed_df = data_store['preprocessed_df']
data_store.close()

A data store can house multiple tables, with the name of each as a key.

Just a note about using the HDFStore in Pandas: you will need to have PyTables >= 3.0.0 installed, so after you have installed Pandas, make sure to update PyTables like this:

pip install --upgrade tables

Conclusions

If you don’t feel like your Pandas project is fast, flexible, easy, and intuitive, consider rethinking how you’re using the library.

The examples you’ve explored here are fairly straightforward but illustrate how the proper application of Pandas features can make vast improvements to runtime and code readability to boot. Here are a few rules of thumb that you can apply next time you’re working with large data sets in Pandas:

Try to use vectorized operations where possible rather than approaching problems with the for x in df... mentality. If your code is home to a lot of for-loops, it might be better suited to working with native Python data structures, because Pandas otherwise comes with a lot of overhead.
If you have more complex operations where vectorization is simply impossible or too difficult to work out efficiently, use the .apply() method.
If you do have to loop over your array (which does happen), use .iterrows() or .itertuples() to improve speed and syntax.
Pandas has a lot of optionality, and there are almost always several ways to get from A to B. Be mindful of this, compare how different routes perform, and choose the one that works best in the context of your project.
Once you’ve got a data cleaning script built, avoid reprocessing by storing your intermediate results with HDFStore.
Integrating NumPy into Pandas operations can often improve speed and simplify syntax.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Lists and Tuples in Python

2018-07-18T14:00:00+00:00

Lists and tuples are arguably Python’s most versatile, useful data types. You will find them in virtually every nontrivial Python program.

Here’s what you’ll learn in this tutorial: You’ll cover the important characteristics of lists and tuples. You’ll learn how to define them and how to manipulate them. When you’re finished, you should have a good feel for when and how to use these object types in a Python program.

Python Lists

In short, a list is a collection of arbitrary objects, somewhat akin to an array in many other programming languages but more flexible. Lists are defined in Python by enclosing a comma-separated sequence of objects in square brackets ([]), as shown below:

>> a = ['foo', 'bar', 'baz', 'qux']

>>> print(a)
['foo', 'bar', 'baz', 'qux']
>>> a
['foo', 'bar', 'baz', 'qux']

The important characteristics of Python lists are as follows:

Lists are ordered.
Lists can contain any arbitrary objects.
List elements can be accessed by index.
Lists can be nested to arbitrary depth.
Lists are mutable.
Lists are dynamic.

Each of these features is examined in more detail below.

Lists Are Ordered

A list is not merely a collection of objects. It is an ordered collection of objects. The order in which you specify the elements when you define a list is an innate characteristic of that list and is maintained for that list’s lifetime. (You will see a Python data type that is not ordered in the next tutorial on dictionaries.)

Lists that have the same elements in a different order are not the same:

>>> a = ['foo', 'bar', 'baz', 'qux']
>>> b = ['baz', 'qux', 'bar', 'foo']
>>> a == b
False
>>> a is b
False

>>> [1, 2, 3, 4] == [4, 1, 3, 2]
False

Lists Can Contain Arbitrary Objects

A list can contain any assortment of objects. The elements of a list can all be the same type:

>>> a = [2, 4, 6, 8]
>>> a
[2, 4, 6, 8]

Or the elements can be of varying types:

>>> a = [21.42, 'foobar', 3, 4, 'bark', False, 3.14159]
>>> a
[21.42, 'foobar', 3, 4, 'bark', False, 3.14159]

Lists can even contain complex objects, like functions, classes, and modules, which you will learn about in upcoming tutorials:

>>> int
<class 'int'>
>>> len
<built-in function len>
>>> def foo():
...     pass
...
>>> foo
<function foo at 0x035B9030>
>>> import math
>>> math
<module 'math' (built-in)>

>>> a = [int, len, foo, math]
>>> a
[<class 'int'>, <built-in function len>, <function foo at 0x02CA2618>,
<module 'math' (built-in)>]

A list can contain any number of objects, from zero to as many as your computer’s memory will allow:

>>> a = []
>>> a
[]

>>> a = [ 'foo' ]
>>> a
['foo']

>>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
... 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
... 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60,
... 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80,
... 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]
>>> a
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39,
40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58,
59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77,
78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96,
97, 98, 99, 100]

(A list with a single object is sometimes referred to as a singleton list.)

List objects needn’t be unique. A given object can appear in a list multiple times:

>>> a = ['bark', 'meow', 'woof', 'bark', 'cheep', 'bark']
>>> a
['bark', 'meow', 'woof', 'bark', 'cheep', 'bark']

List Elements Can Be Accessed by Index

Individual elements in a list can be accessed using an index in square brackets. This is exactly analogous to accessing individual characters in a string. List indexing is zero-based as it is with strings.

Consider the following list:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

The indices for the elements in a are shown below:

List Indices

Here is Python code to access some elements of a:

>>> a[0]
'foo'
>>> a[2]
'baz'
>>> a[5]
'corge'

Virtually everything about string indexing works similarly for lists. For example, a negative list index counts from the end of the list:

Negative List Indexing

>>> a[-1]
'corge'
>>> a[-2]
'quux'
>>> a[-5]
'bar'

Slicing also works. If a is a list, the expression a[m:n] returns the portion of a from index m to, but not including, index n:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

>>> a[2:5]
['baz', 'qux', 'quux']

Other features of string slicing work analogously for list slicing as well:

Both positive and negative indices can be specified:

>>> a[-5:-2]
['bar', 'baz', 'qux']
>>> a[1:4]
['bar', 'baz', 'qux']
>>> a[-5:-2] == a[1:4]
True

Omitting the first index starts the slice at the beginning of the list, and omitting the second index extends the slice to the end of the list:

>>> print(a[:4], a[0:4])
['foo', 'bar', 'baz', 'qux'] ['foo', 'bar', 'baz', 'qux']
>>> print(a[2:], a[2:len(a)])
['baz', 'qux', 'quux', 'corge'] ['baz', 'qux', 'quux', 'corge']

>>> a[:4] + a[4:]
['foo', 'bar', 'baz', 'qux', 'quux', 'corge']
>>> a[:4] + a[4:] == a
True

You can specify a stride—either positive or negative:

>>> a[0:6:2]
['foo', 'baz', 'quux']
>>> a[1:6:2]
['bar', 'qux', 'corge']
>>> a[6:0:-2]
['corge', 'qux', 'bar']

The syntax for reversing a list works the same way it does for strings:
```
>>> a[::-1]
['corge', 'quux', 'qux', 'baz', 'bar', 'foo']
```
The [:] syntax works for lists. However, there is an important difference between how this operation works with a list and how it works with a string.

If s is a string, s[:] returns a reference to the same object:
```
>>> s = 'foobar'
>>> s[:]
'foobar'
>>> s[:] is s
True
```
Conversely, if a is a list, a[:] returns a new object that is a copy of a:
```
>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']
>>> a[:]
['foo', 'bar', 'baz', 'qux', 'quux', 'corge']
>>> a[:] is a
False
```

Several Python operators and built-in functions can also be used with lists in ways that are analogous to strings:

The in and not in operators:

>>> a
['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

>>> 'qux' in a
True
>>> 'thud' not in a
True

The concatenation (+) and replication (*) operators:

>>> a
['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

>>> a + ['grault', 'garply']
['foo', 'bar', 'baz', 'qux', 'quux', 'corge', 'grault', 'garply']
>>> a * 2
['foo', 'bar', 'baz', 'qux', 'quux', 'corge', 'foo', 'bar', 'baz',
'qux', 'quux', 'corge']

The len(), min(), and max() functions:

>>> a
['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

>>> len(a)
6
>>> min(a)
'bar'
>>> max(a)
'qux'

It’s not an accident that strings and lists behave so similarly. They are both special cases of a more general object type called an iterable, which you will encounter in more detail in the upcoming tutorial on definite iteration.

By the way, in each example above, the list is always assigned to a variable before an operation is performed on it. But you can operate on a list literal as well:

>>> ['foo', 'bar', 'baz', 'qux', 'quux', 'corge'][2]
'baz'

>>> ['foo', 'bar', 'baz', 'qux', 'quux', 'corge'][::-1]
['corge', 'quux', 'qux', 'baz', 'bar', 'foo']

>>> 'quux' in ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']
True

>>> ['foo', 'bar', 'baz'] + ['qux', 'quux', 'corge']
['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

>>> len(['foo', 'bar', 'baz', 'qux', 'quux', 'corge'][::-1])
6

For that matter, you can do likewise with a string literal:

>>> 'If Comrade Napoleon says it, it must be right.'[::-1]
'.thgir eb tsum ti ,ti syas noelopaN edarmoC fI'

Lists Can Be Nested

You have seen that an element in a list can be any sort of object. That includes another list. A list can contain sublists, which in turn can contain sublists themselves, and so on to arbitrary depth.

Consider this (admittedly contrived) example:

>>> x = ['a', ['bb', ['ccc', 'ddd'], 'ee', 'ff'], 'g', ['hh', 'ii'], 'j']
>>> x
['a', ['bb', ['ccc', 'ddd'], 'ee', 'ff'], 'g', ['hh', 'ii'], 'j']

The object structure that x references is diagrammed below:

A Nested List

x[0], x[2], and x[4] are strings, each one character long:

>>> print(x[0], x[2], x[4])
a g j

But x[1] and x[3] are sublists:

>>> x[1]
['bb', ['ccc', 'ddd'], 'ee', 'ff']

>>> x[3]
['hh', 'ii']

To access the items in a sublist, simply append an additional index:

>>> x[1]
['bb', ['ccc', 'ddd'], 'ee', 'ff']

>>> x[1][0]
'bb'
>>> x[1][1]
['ccc', 'ddd']
>>> x[1][2]
'ee'
>>> x[1][3]
'ff'

>>> x[3]
['hh', 'ii']
>>> print(x[3][0], x[3][1])
hh ii

x[1][1] is yet another sublist, so adding one more index accesses its elements:

>>> x[1][1]
['ccc', 'ddd']
>>> print(x[1][1][0], x[1][1][1])
ccc ddd

There is no limit, short of the extent of your computer’s memory, to the depth or complexity with which lists can be nested in this way.

All the usual syntax regarding indices and slicing applies to sublists as well:

>>> x[1][1][-1]
'ddd'
>>> x[1][1:3]
[['ccc', 'ddd'], 'ee']
>>> x[3][::-1]
['ii', 'hh']

However, be aware that operators and functions apply to only the list at the level you specify and are not recursive. Consider what happens when you query the length of x using len():

>>> x
['a', ['bb', ['ccc', 'ddd'], 'ee', 'ff'], 'g', ['hh', 'ii'], 'j']
>>> len(x)
5

>>> x[0]
'a'
>>> x[1]
['bb', ['ccc', 'ddd'], 'ee', 'ff']
>>> x[2]
'g'
>>> x[3]
['hh', 'ii']
>>> x[4]
'j'

x has only five elements—three strings and two sublists. The individual elements in the sublists don’t count toward x’s length.

You’d encounter a similar situation when using the in operator:

>>> 'ddd' in x
False
>>> 'ddd' in x[1]
False
>>> 'ddd' in x[1][1]
True

'ddd' is not one of the elements in x or x[1]. It is only directly an element in the sublist x[1][1]. An individual element in a sublist does not count as an element of the parent list(s).

Lists Are Mutable

Most of the data types you have encountered so far have been atomic types. Integer or float objects, for example, are primitive units that can’t be further broken down. These types are immutable, meaning that they can’t be changed once they have been assigned. It doesn’t make much sense to think of changing the value of an integer. If you want a different integer, you just assign a different one.

By contrast, the string type is a composite type. Strings are reducible to smaller parts—the component characters. It might make sense to think of changing the characters in a string. But you can’t. In Python, strings are also immutable.

The list is the first mutable data type you have encountered. Once a list has been created, elements can be added, deleted, shifted, and moved around at will. Python provides a wide range of ways to modify lists.

Modifying a Single List Value

A single value in a list can be replaced by indexing and simple assignment:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']
>>> a
['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

>>> a[2] = 10
>>> a[-1] = 20
>>> a
['foo', 'bar', 10, 'qux', 'quux', 20]

You may recall from the tutorial Strings and Character Data in Python that you can’t do this with a string:

>>> s = 'foobarbaz'
>>> s[2] = 'x'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'str' object does not support item assignment

A list item can be deleted with the del command:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

>>> del a[3]
>>> a
['foo', 'bar', 'baz', 'quux', 'corge']

Modifying Multiple List Values

What if you want to change several contiguous elements in a list at one time? Python allows this with slice assignment, which has the following syntax:

a[m:n] = <iterable>

Again, for the moment, think of an iterable as a list. This assignment replaces the specified slice of a with <iterable>:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

>>> a[1:4]
['bar', 'baz', 'qux']
>>> a[1:4] = [1.1, 2.2, 3.3, 4.4, 5.5]
>>> a
['foo', 1.1, 2.2, 3.3, 4.4, 5.5, 'quux', 'corge']
>>> a[1:6]
[1.1, 2.2, 3.3, 4.4, 5.5]
>>> a[1:6] = ['Bark!']
>>> a
['foo', 'Bark!', 'quux', 'corge']

The number of elements inserted need not be equal to the number replaced. Python just grows or shrinks the list as needed.

You can insert multiple elements in place of a single element—just use a slice that denotes only one element:

>>> a = [1, 2, 3]
>>> a[1:2] = [2.1, 2.2, 2.3]
>>> a
[1, 2.1, 2.2, 2.3, 3]

Note that this is not the same as replacing the single element with a list:

>>> a = [1, 2, 3]
>>> a[1] = [2.1, 2.2, 2.3]
>>> a
[1, [2.1, 2.2, 2.3], 3]

You can also insert elements into a list without removing anything. Simply specify a slice of the form [n:n] (a zero-length slice) at the desired index:

>>> a = [1, 2, 7, 8]
>>> a[2:2] = [3, 4, 5, 6]
>>> a
[1, 2, 3, 4, 5, 6, 7, 8]

You can delete multiple elements out of the middle of a list by assigning the appropriate slice to an empty list. You can also use the del statement with the same slice:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']
>>> a[1:5] = []
>>> a
['foo', 'corge']

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']
>>> del a[1:5]
>>> a
['foo', 'corge']

Prepending or Appending Items to a List

Additional items can be added to the start or end of a list using the + concatenation operator or the += augmented assignment operator:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

>>> a += ['grault', 'garply']
>>> a
['foo', 'bar', 'baz', 'qux', 'quux', 'corge', 'grault', 'garply']

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

>>> a = [10, 20] + a
>>> a
[10, 20, 'foo', 'bar', 'baz', 'qux', 'quux', 'corge']

Note that a list must be concatenated with another list, so if you want to add only one element, you need to specify it as a singleton list:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']
>>> a += 20
Traceback (most recent call last):
  File "<pyshell#58>", line 1, in <module>
    a += 20
TypeError: 'int' object is not iterable

>>> a += [20]
>>> a
['foo', 'bar', 'baz', 'qux', 'quux', 'corge', 20]

Note: Technically, it isn’t quite correct to say a list must be concatenated with another list. More precisely, a list must be concatenated with an object that is iterable. Of course, lists are iterable, so it works to concatenate a list with another list.

Strings are iterable also. But watch what happens when you concatenate a string onto a list:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux']
>>> a += 'corge'
>>> a
['foo', 'bar', 'baz', 'qux', 'quux', 'c', 'o', 'r', 'g', 'e']

This result is perhaps not quite what you expected. When a string is iterated through, the result is a list of its component characters. In the above example, what gets concatenated onto list a is a list of the characters in the string 'corge'.

If you really want to add just the single string 'corge' to the end of the list, you need to specify it as a singleton list:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux']
>>> a += ['corge']
>>> a
['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

If this seems mysterious, don’t fret too much. You’ll learn about the ins and outs of iterables in the tutorial on definite iteration.

Methods That Modify a List

Finally, Python supplies several built-in methods that can be used to modify lists. Information on these methods is detailed below.

Note: The string methods you saw in the previous tutorial did not modify the target string directly. That is because strings are immutable. Instead, string methods return a new string object that is modified as directed by the method. They leave the original target string unchanged:

>>> s = 'foobar'
>>> t = s.upper()
>>> print(s, t)
foobar FOOBAR

List methods are different. Because lists are mutable, the list methods shown here modify the target list in place.

a.append(<obj>)

Appends an object to a list.

a.append(<obj>) appends object <obj> to the end of list a:

>>> a = ['a', 'b']
>>> a.append(123)
>>> a
['a', 'b', 123]

Remember, list methods modify the target list in place. They do not return a new list:

>>> a = ['a', 'b']
>>> x = a.append(123)
>>> print(x)
None
>>> a
['a', 'b', 123]

Remember that when the + operator is used to concatenate to a list, if the target operand is an iterable, then its elements are broken out and appended to the list individually:

>>> a = ['a', 'b']
>>> a + [1, 2, 3]
['a', 'b', 1, 2, 3]

The .append() method does not work that way! If an iterable is appended to a list with .append(), it is added as a single object:

>>> a = ['a', 'b']
>>> a.append([1, 2, 3])
>>> a
['a', 'b', [1, 2, 3]]

Thus, with .append(), you can append a string as a single entity:

>>> a = ['a', 'b']
>>> a.append('foo')
>>> a
['a', 'b', 'foo']

a.extend(<iterable>)

Extends a list with the objects from an iterable.

Yes, this is probably what you think it is. .extend() also adds to the end of a list, but the argument is expected to be an iterable. The items in <iterable> are added individually:

>>> a = ['a', 'b']
>>> a.extend([1, 2, 3])
>>> a
['a', 'b', 1, 2, 3]

In other words, .extend() behaves like the + operator. More precisely, since it modifies the list in place, it behaves like the += operator:

>>> a = ['a', 'b']
>>> a += [1, 2, 3]
>>> a
['a', 'b', 1, 2, 3]

a.insert(<index>, <obj>)

Inserts an object into a list.

a.insert(<index>, <obj>) inserts object <obj> into list a at the specified <index>. Following the method call, a[<index>] is <obj>, and the remaining list elements are pushed to the right:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']
>>> a.insert(3, 3.14159)
>>> a[3]
3.14159
>>> a
['foo', 'bar', 'baz', 3.14159, 'qux', 'quux', 'corge']

a.remove(<obj>)

Removes an object from a list.

a.remove(<obj>) removes object <obj> from list a. If <obj> isn’t in a, an exception is raised:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']
>>> a.remove('baz')
>>> a
['foo', 'bar', 'qux', 'quux', 'corge']

>>> a.remove('Bark!')
Traceback (most recent call last):
  File "<pyshell#13>", line 1, in <module>
    a.remove('Bark!')
ValueError: list.remove(x): x not in list

a.pop(index=-1)

Removes an element from a list.

This method differs from .remove() in two ways:

You specify the index of the item to remove, rather than the object itself.
The method returns a value: the item that was removed.

a.pop() simply removes the last item in the list:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

>>> a.pop()
'corge'
>>> a
['foo', 'bar', 'baz', 'qux', 'quux']

>>> a.pop()
'quux'
>>> a
['foo', 'bar', 'baz', 'qux']

If the optional <index> parameter is specified, the item at that index is removed and returned. <index> may be negative, as with string and list indexing:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

>>> a.pop(1)
'bar'
>>> a
['foo', 'baz', 'qux', 'quux', 'corge']

>>> a.pop(-3)
'qux'
>>> a
['foo', 'baz', 'quux', 'corge']

<index> defaults to -1, so a.pop(-1) is equivalent to a.pop().

Lists Are Dynamic

This tutorial began with a list of six defining characteristics of Python lists. The last one is that lists are dynamic. You have seen many examples of this in the sections above. When items are added to a list, it grows as needed:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']

>>> a[2:2] = [1, 2, 3]
>>> a += [3.14159]
>>> a
['foo', 'bar', 1, 2, 3, 'baz', 'qux', 'quux', 'corge', 3.14159]

Similarly, a list shrinks to accommodate the removal of items:

>>> a = ['foo', 'bar', 'baz', 'qux', 'quux', 'corge']
>>> a[2:3] = []
>>> del a[0]
>>> a
['bar', 'qux', 'quux', 'corge']

Python Tuples

Python provides another type that is an ordered collection of objects, called a tuple.

Pronunciation varies depending on whom you ask. Some pronounce it as though it were spelled “too-ple” (rhyming with “Mott the Hoople”), and others as though it were spelled “tup-ple” (rhyming with “supple”). My inclination is the latter, since it presumably derives from the same origin as “quintuple,” “sextuple,” “octuple,” and so on, and everyone I know pronounces these latter as though they rhymed with “supple.”

Defining and Using Tuples

Tuples are identical to lists in all respects, except for the following properties:

Tuples are defined by enclosing the elements in parentheses (()) instead of square brackets ([]).
Tuples are immutable.

Here is a short example showing a tuple definition, indexing, and slicing:

>>> t = ('foo', 'bar', 'baz', 'qux', 'quux', 'corge')
>>> t
('foo', 'bar', 'baz', 'qux', 'quux', 'corge')

>>> t[0]
'foo'
>>> t[-1]
'corge'
>>> t[1::2]
('bar', 'qux', 'corge')

Never fear! Our favorite string and list reversal mechanism works for tuples as well:

>>> t[::-1]
('corge', 'quux', 'qux', 'baz', 'bar', 'foo')

Note: Even though tuples are defined using parentheses, you still index and slice tuples using square brackets, just as for strings and lists.

Everything you’ve learned about lists—they are ordered, they can contain arbitrary objects, they can be indexed and sliced, they can be nested—is true of tuples as well. But they can’t be modified:

>>> t = ('foo', 'bar', 'baz', 'qux', 'quux', 'corge')
>>> t[2] = 'Bark!'
Traceback (most recent call last):
  File "<pyshell#65>", line 1, in <module>
    t[2] = 'Bark!'
TypeError: 'tuple' object does not support item assignment

Why use a tuple instead of a list?

Program execution is faster when manipulating a tuple than it is for the equivalent list. (This is probably not going to be noticeable when the list or tuple is small.)
Sometimes you don’t want data to be modified. If the values in the collection are meant to remain constant for the life of the program, using a tuple instead of a list guards against accidental modification.
There is another Python data type that you will encounter shortly called a dictionary, which requires as one of its components a value that is of an immutable type. A tuple can be used for this purpose, whereas a list can’t be.

In a Python REPL session, you can display the values of several objects simultaneously by entering them directly at the >>> prompt, separated by commas:

>>> a = 'foo'
>>> b = 42
>>> a, 3.14159, b
('foo', 3.14159, 42)

Python displays the response in parentheses because it is implicitly interpreting the input as a tuple.

There is one peculiarity regarding tuple definition that you should be aware of. There is no ambiguity when defining an empty tuple, nor one with two or more elements. Python knows you are defining a tuple:

>>> t = ()
>>> type(t)
<class 'tuple'>

>>> t = (1, 2)
>>> type(t)
<class 'tuple'>
>>> t = (1, 2, 3, 4, 5)
>>> type(t)
<class 'tuple'>

But what happens when you try to define a tuple with one item:

>>> t = (2)
>>> type(t)
<class 'int'>

Doh! Since parentheses are also used to define operator precedence in expressions, Python evaluates the expression (2) as simply the integer 2 and creates an int object. To tell Python that you really want to define a singleton tuple, include a trailing comma (,) just before the closing parenthesis:

>>> t = (2,)
>>> type(t)
<class 'tuple'>
>>> t[0]
2
>>> t[-1]
2

You probably won’t need to define a singleton tuple often, but there has to be a way.

When you display a singleton tuple, Python includes the comma, to remind you that it’s a tuple:

>>> print(t)
(2,)

Tuple Assignment, Packing, and Unpacking

As you have already seen above, a literal tuple containing several items can be assigned to a single object:

>>> t = ('foo', 'bar', 'baz', 'qux')

When this occurs, it is as though the items in the tuple have been “packed” into the object:

Tuple Packing

>>> t
('foo', 'bar', 'baz', 'qux')
>>> t[0]
'foo'
>>> t[-1]
'qux'

If that “packed” object is subsequently assigned to a new tuple, the individual items are “unpacked” into the objects in the tuple:

Tuple Unpacking

>>> (s1, s2, s3, s4) = t
>>> s1
'foo'
>>> s2
'bar'
>>> s3
'baz'
>>> s4
'qux'

When unpacking, the number of variables on the left must match the number of values in the tuple:

>>> (s1, s2, s3) = t
Traceback (most recent call last):
  File "<pyshell#16>", line 1, in <module>
    (s1, s2, s3) = t
ValueError: too many values to unpack (expected 3)

>>> (s1, s2, s3, s4, s5) = t
Traceback (most recent call last):
  File "<pyshell#17>", line 1, in <module>
    (s1, s2, s3, s4, s5) = t
ValueError: not enough values to unpack (expected 5, got 4)

Packing and unpacking can be combined into one statement to make a compound assignment:

>>> (s1, s2, s3, s4) = ('foo', 'bar', 'baz', 'qux')
>>> s1
'foo'
>>> s2
'bar'
>>> s3
'baz'
>>> s4
'qux'

Again, the number of elements in the tuple on the left of the assignment must equal the number on the right:

>>> (s1, s2, s3, s4, s5) = ('foo', 'bar', 'baz', 'qux')
Traceback (most recent call last):
  File "<pyshell#63>", line 1, in <module>
    (s1, s2, s3, s4, s5) = ('foo', 'bar', 'baz', 'qux')
ValueError: not enough values to unpack (expected 5, got 4)

In assignments like this and a small handful of other situations, Python allows the parentheses that are usually used for denoting a tuple to be left out:

>>> t = 1, 2, 3
>>> t
(1, 2, 3)

>>> x1, x2, x3 = t
>>> x1, x2, x3
(1, 2, 3)

>>> x1, x2, x3 = 4, 5, 6
>>> x1, x2, x3
(4, 5, 6)

>>> t = 2,
>>> t
(2,)

It works the same whether the parentheses are included or not, so if you have any doubt as to whether they’re needed, go ahead and include them.

Tuple assignment allows for a curious bit of idiomatic Python. Frequently when programming, you have two variables whose values you need to swap. In most programming languages, it is necessary to store one of the values in a temporary variable while the swap occurs like this:

>>> a = 'foo'
>>> b = 'bar'
>>> a, b
('foo', 'bar')

>>># We need to define a temp variable to accomplish the swap.
>>> temp = a
>>> a = b
>>> b = temp

>>> a, b
('bar', 'foo')

In Python, the swap can be done with a single tuple assignment:

>>> a = 'foo'
>>> b = 'bar'
>>> a, b
('foo', 'bar')

>>># Magic time!
>>> a, b = b, a

>>> a, b
('bar', 'foo')

As anyone who has ever had to swap values using a temporary variable knows, being able to do it this way in Python is the pinnacle of modern technological achievement. It will never get better than this.

Conclusion

This tutorial covered the basic properties of Python lists and tuples, and how to manipulate them. You will use these extensively in your Python programming.

One of the chief characteristics of a list is that it is ordered. The order of the elements in a list is an intrinsic property of that list and does not change, unless the list itself is modified. (The same is true of tuples, except of course they can’t be modified.)

The next tutorial will introduce you to the Python dictionary: a composite data type that is unordered. Read on!

« Strings in Python

Lists and Tuples in Python

Dictionaries in Python »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Reading and Writing CSV Files in Python

2018-07-16T14:00:00+00:00

Let’s face it: you need to get information into and out of your programs through more than just the keyboard and console. Exchanging information through text files is a common way to share info between programs. One of the most popular formats for exchanging data is the CSV format. But how do you use it?

Let’s get one thing clear: you don’t have to (and you won’t) build your own CSV parser from scratch. There are several perfectly acceptable libraries you can use. The Python csv library will work for most cases. If your work requires lots of data or numerical analysis, the pandas library has CSV parsing capabilities as well, which should handle the rest.

In this article, you’ll learn how to read, process, and parse CSV from text files using Python. You’ll see how CSV files work, learn the all-important csv library built into Python, and see how CSV parsing works using the pandas library.

So let’s get started!

What Is a CSV File?

A CSV file (Comma Separated Values file) is a type of plain text file that uses specific structuring to arrange tabular data. Because it’s a plain text file, it can contain only actual text data—in other words, printable ASCII or Unicode characters.

The structure of a CSV file is given away by its name. Normally, CSV files use a comma to separate each specific data value. Here’s what that structure looks like:

column 1 name,column 2 name, column 3 name
first row data 1,first row data 2,first row data 3
second row data 1,second row data 2,second row data 3
...

Notice how each piece of data is separated by a comma. Normally, the first line identifies each piece of data—in other words, the name of a data column. Every subsequent line after that is actual data and is limited only by file size constraints.

In general, the separator character is called a delimiter, and the comma is not the only one used. Other popular delimiters include the tab (\t), colon (:) and semi-colon (;) characters. Properly parsing a CSV file requires us to know which delimiter is being used.

Where Do CSV Files Come From?

CSV files are normally created by programs that handle large amounts of data. They are a convenient way to export data from spreadsheets and databases as well as import or use it in other programs. For example, you might export the results of a data mining program to a CSV file and then import that into a spreadsheet to analyze the data, generate graphs for a presentation, or prepare a report for publication.

CSV files are very easy to work with programmatically. Any language that supports text file input and string manipulation (like Python) can work with CSV files directly.

Parsing CSV Files With Python’s Built-in CSV Library

The csv library provides functionality to both read from and write to CSV files. Designed to work out of the box with Excel-generated CSV files, it is easily adapted to work with a variety of CSV formats. The csv library contains objects and other code to read, write, and process data from and to CSV files.

Reading CSV Files With `csv`

Reading from a CSV file is done using the reader object. The CSV file is opened as a text file with Python’s built-in open() function, which returns a file object. This is then passed to the reader, which does the heavy lifting.

Here’s the employee_birthday.txt file:

name,department,birthday month
John Smith,Accounting,November
Erica Meyers,IT,March

Here’s code to read it:

import csv

with open('employee_birthday.txt') as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',')
    line_count = 0
    for row in csv_reader:
        if line_count == 0:
            print(f'Column names are {", ".join(row)}')
            line_count += 1
        else:
            print(f'\t{row[0]} works in the {row[1]} department, and was born in {row[2]}.')
            line_count += 1
    print(f'Processed {line_count} lines.')

This results in the following output:

Column names are name, department, birthday month
    John Smith works in the Accounting department, and was born in November.
    Erica Meyers works in the IT department, and was born in March.
Processed 3 lines.

Each row returned by the reader is a list of String elements containing the data found by removing the delimiters. The first row returned contains the column names, which is handled in a special way.

Reading CSV Files Into a Dictionary With `csv`

Rather than deal with a list of individual String elements, you can read CSV data directly into a dictionary (technically, an Ordered Dictionary) as well.

Again, our input file, employee_birthday.txt is as follows:

name,department,birthday month
John Smith,Accounting,November
Erica Meyers,IT,March

Here’s the code to read it in as a dictionary this time:

import csv

with open('employee_birthday.txt', mode='r') as csv_file:
    csv_reader = csv.DictReader(csv_file)
    line_count = 0
    for row in csv_reader:
        if line_count == 0:
            print(f'Column names are {", ".join(row)}')
            line_count += 1
        print(f'\t{row["name"]} works in the {row["department"]} department, and was born in {row["birthday month"]}.')
        line_count += 1
    print(f'Processed {line_count} lines.')

This results in the same output as before:

Column names are name, department, birthday month
    John Smith works in the Accounting department, and was born in November.
    Erica Meyers works in the IT department, and was born in March.
Processed 3 lines.

Where did the dictionary keys come from? The first line of the CSV file is assumed to contain the keys to use to build the dictionary. If you don’t have these in your CSV file, you should specify your own keys by setting the fieldnames optional parameter to a list containing them.

Optional Python CSV `reader` Parameters

The reader object can handle different styles of CSV files by specifying additional parameters, some of which are shown below:

delimiter specifies the character used to separate each field. The default is the comma (',').
quotechar specifies the character used to surround fields that contain the delimiter character. The default is a double quote (' " ').
escapechar specifies the character used to escape the delimiter character, in case quotes aren’t used. The default is no escape character.

These parameters deserve some more explanation. Suppose you’re working with the following employee_addresses.txt file:

name,address,date joined
john smith,1132 Anywhere Lane Hoboken NJ, 07030,Jan 4
erica meyers,1234 Smith Lane Hoboken NJ, 07030,March 2

This CSV file contains three fields: name, address, and date joined, which are delimited by commas. The problem is that the data for the address field also contains a comma to signify the zip code.

There are three different ways to handle this situation:

Use a different delimiter
That way, the comma can safely be used in the data itself. You use the delimiter optional parameter to specify the new delimiter.
Wrap the data in quotes
The special nature of your chosen delimiter is ignored in quoted strings. Therefore, you can specify the character used for quoting with the quotechar optional parameter. As long as that character also doesn’t appear in the data, you’re fine.
Escape the delimiter characters in the data
Escape characters work just as they do in format strings, nullifying the interpretation of the character being escaped (in this case, the delimiter). If an escape character is used, it must be specified using the escapechar optional parameter.

Writing CSV Files With `csv`

You can also write to a CSV file using a writer object and the .write_row() method:

import csv

with open('employee_file.csv', mode='w') as employee_file:
    employee_writer = csv.writer(employee_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)

    employee_writer.writerow(['John Smith', 'Accounting', 'November'])
    employee_writer.writerow(['Erica Meyers', 'IT', 'March'])

The quotechar optional parameter tells the writer which character to use to quote fields when writing. Whether quoting is used or not, however, is determined by the quoting optional parameter:

If quoting is set to csv.QUOTE_MINIMAL, then .writerow() will quote fields only if they contain the delimiter or the quotechar. This is the default case.
If quoting is set to csv.QUOTE_ALL, then .writerow() will quote all fields.
If quoting is set to csv.QUOTE_NONNUMERIC, then .writerow() will quote all fields containing text data and convert all numeric fields to the float data type.
If quoting is set to csv.QUOTE_NONE, then .writerow() will escape delimiters instead of quoting them. In this case, you also must provide a value for the escapechar optional parameter.

Reading the file back in plain text shows that the file is created as follows:

John Smith,Accounting,November
Erica Meyers,IT,March

Writing CSV File From a Dictionary With `csv`

Since you can read our data into a dictionary, it’s only fair that you should be able to write it out from a dictionary as well:

import csv

with open('employee_file2.csv', mode='w') as csv_file:
    fieldnames = ['emp_name', 'dept', 'birth_month']
    writer = csv.DictWriter(csv_file, fieldnames=fieldnames)

    writer.writeheader()
    writer.writerow({'emp_name': 'John Smith', 'dept': 'Accounting', 'birth_month': 'November'})
    writer.writerow({'emp_name': 'Erica Meyers', 'dept': 'IT', 'birth_month': 'March'})

Unlike DictReader, the fieldnames parameter is required when writing a dictionary. This makes sense, when you think about it: without a list of fieldnames, the DictWriter can’t know which keys to use to retrieve values from your dictionaries. It also uses the keys in fieldnames to write out the first row as column names.

The code above generates the following output file:

emp_name,dept,birth_month
John Smith,Accounting,November
Erica Meyers,IT,March

Parsing CSV Files With the `pandas` Library

Of course, the Python CSV library isn’t the only game in town. Reading CSV files is possible in pandas as well. It is highly recommended if you have a lot of data to analyze.

pandas is an open-source Python library that provides high performance data analysis tools and easy to use data structures. pandas is available for all Python installations, but it is a key part of the Anaconda distribution and works extremely well in Jupyter notebooks to share data, code, analysis results, visualizations, and narrative text.

Installing pandas and its dependencies in Anaconda is easily done:

$ conda install pandas

As is using pip/pipenv for other Python installations:

$ pip install pandas

We won’t delve into the specifics of how pandas works or how to use it. For an in-depth treatment on using pandas to read and analyze large data sets, check out Shantnu Tiwari’s superb article on working with large Excel files in pandas.

Reading CSV Files With `pandas`

To show some of the power of pandas CSV capabilities, I’ve created a slightly more complicated file to read, called hrdata.csv. It contains data on company employees:

Name,Hire Date,Salary,Sick Days remaining
Graham Chapman,03/15/14,50000.00,10
John Cleese,06/01/15,65000.00,8
Eric Idle,05/12/14,45000.00,10
Terry Jones,11/01/13,70000.00,3
Terry Gilliam,08/12/14,48000.00,7
Michael Palin,05/23/13,66000.00,8

Reading the CSV into a pandas DataFrame is quick and straightforward:

import pandas
df = pandas.read_csv('hrdata.csv')
print(df)

That’s it: three lines of code, and only one of them is doing the actual work. pandas.read_csv() opens, analyzes, and reads the CSV file provided, and stores the data in a DataFrame. Printing the DataFrame results in the following output:

             Name Hire Date   Salary  Sick Days remaining
0  Graham Chapman  03/15/14  50000.0                   10
1     John Cleese  06/01/15  65000.0                    8
2       Eric Idle  05/12/14  45000.0                   10
3     Terry Jones  11/01/13  70000.0                    3
4   Terry Gilliam  08/12/14  48000.0                    7
5   Michael Palin  05/23/13  66000.0                    8

Here are a few points worth noting:

First, pandas recognized that the first line of the CSV contained column names, and used them automatically. I call this Goodness.
However, pandas is also using zero-based integer indices in the DataFrame. That’s because we didn’t tell it what our index should be.
Further, if you look at the data types of our columns , you’ll see pandas has properly converted the Salary and Sick Days remaining columns to numbers, but the Hire Date column is still a String. This is easily confirmed in interactive mode:
```
>>> print(type(df['Hire Date'][0]))
<class 'str'>
```

Let’s tackle these issues one at a time. To use a different column as the DataFrame index, add the index_col optional parameter:

import pandas
df = pandas.read_csv('hrdata.csv', index_col='Name')
print(df)

Now the Name field is our DataFrame index:

               Hire Date   Salary  Sick Days remaining
Name                                                  
Graham Chapman  03/15/14  50000.0                   10
John Cleese     06/01/15  65000.0                    8
Eric Idle       05/12/14  45000.0                   10
Terry Jones     11/01/13  70000.0                    3
Terry Gilliam   08/12/14  48000.0                    7
Michael Palin   05/23/13  66000.0                    8

Next, let’s fix the data type of the Hire Date field. You can force pandas to read data as a date with the parse_dates optional parameter, which is defined as a list of column names to treat as dates:

import pandas
df = pandas.read_csv('hrdata.csv', index_col='Name', parse_dates=['Hire Date'])
print(df)

Notice the difference in the output:

                Hire Date   Salary  Sick Days remaining
Name                                                   
Graham Chapman 2014-03-15  50000.0                   10
John Cleese    2015-06-01  65000.0                    8
Eric Idle      2014-05-12  45000.0                   10
Terry Jones    2013-11-01  70000.0                    3
Terry Gilliam  2014-08-12  48000.0                    7
Michael Palin  2013-05-23  66000.0                    8

The date is now formatted properly, which is easily confirmed in interactive mode:

>>> print(type(df['Hire Date'][0]))
<class 'pandas._libs.tslibs.timestamps.Timestamp'>

If your CSV files doesn’t have column names in the first line, you can use the names optional parameter to provide a list of column names. You can also use this if you want to override the column names provided in the first line. In this case, you must also tell pandas.read_csv() to ignore existing column names using the header=0 optional parameter:

import pandas
df = pandas.read_csv('hrdata.csv', 
            index_col='Employee', 
            parse_dates=['Hired'], 
            header=0, 
            names=['Employee', 'Hired','Salary', 'Sick Days'])
print(df)

Notice that, since the column names changed, the columns specified in the index_col and parse_dates optional parameters must also be changed. This now results in the following output:

                    Hired   Salary  Sick Days
Employee                                     
Graham Chapman 2014-03-15  50000.0         10
John Cleese    2015-06-01  65000.0          8
Eric Idle      2014-05-12  45000.0         10
Terry Jones    2013-11-01  70000.0          3
Terry Gilliam  2014-08-12  48000.0          7
Michael Palin  2013-05-23  66000.0          8

Writing CSV Files With `pandas`

Of course, if you can’t get your data out of pandas again, it doesn’t do you much good. Writing a DataFrame to a CSV file is just as easy as reading one in. Let’s write the data with the new column names to a new CSV file:

import pandas
df = pandas.read_csv('hrdata.csv', 
            index_col='Employee', 
            parse_dates=['Hired'],
            header=0, 
            names=['Employee', 'Hired', 'Salary', 'Sick Days'])
df.to_csv('hrdata_modified.csv')

The only difference between this code and the reading code above is that the print(df) call was replaced with df.to_csv(), providing the file name. The new CSV file looks like this:

Employee,Hired,Salary,Sick Days
Graham Chapman,2014-03-15,50000.0,10
John Cleese,2015-06-01,65000.0,8
Eric Idle,2014-05-12,45000.0,10
Terry Jones,2013-11-01,70000.0,3
Terry Gilliam,2014-08-12,48000.0,7
Michael Palin,2013-05-23,66000.0,8

Conclusion

If you understand the basics of reading CSV files, then you won’t ever be caught flat footed when you need to deal with importing data. Most CSV reading, processing, and writing tasks can be easily handled by the basic csv Python library. If you have a lot of data to read and process, the pandas library provides quick and easy CSV handling capabilities as well.

Are there other ways to parse text files? Of course! Libraries like ANTLR, PLY, and PlyPlus can all handle heavy-duty parsing, and if simple String manipulation won’t work, there are always regular expressions.

But those are topics for other articles…

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Generating Random Data in Python (Guide)

2018-07-11T14:00:00+00:00

How random is random? This is a weird question to ask, but it is one of paramount importance in cases where information security is concerned. Whenever you’re generating random data, strings, or numbers in Python, it’s a good idea to have at least a rough idea of how that data was generated.

Here, you’ll cover a handful of different options for generating random data in Python, and then build up to a comparison of each in terms of its level of security, versatility, purpose, and speed.

I promise that this tutorial will not be a lesson in mathematics or cryptography, which I wouldn’t be well equipped to lecture on in the first place. You’ll get into just as much math as needed, and no more.

How Random Is Random?

First, a prominent disclaimer is necessary. Most random data generated with Python is not fully random in the scientific sense of the word. Rather, it is pseudorandom: generated with a pseudorandom number generator (PRNG), which is essentially any algorithm for generating seemingly random but still reproducible data.

“True” random numbers can be generated by, you guessed it, a true random number generator (TRNG). One example is to repeatedly pick up a die off the floor, toss it in the air, and let it land how it may.

Assuming that your toss is unbiased, you have truly no idea what number the die will land on. Rolling a die is a crude form of using hardware to generate a number that is not deterministic whatsoever. (Or, you can have the dice-o-matic do this for you.) TRNGs are out of the scope of this article but worth a mention nonetheless for comparison’s sake.

PRNGs, usually done with software rather than hardware, work slightly differently. Here’s a concise description:

They start with a random number, known as the seed, and then use an algorithm to generate a pseudo-random sequence of bits based on it. (Source)

You’ve likely been told to “read the docs!” at some point. Well, those people are not wrong. Here’s a particularly notable snippet from the random module’s documentation that you don’t want to miss:

Warning: The pseudo-random generators of this module should not be used for security purposes. (Source)

You’ve probably seen random.seed(999), random.seed(1234), or the like, in Python. This function call is seeding the underlying random number generator used by Python’s random module. It is what makes subsequent calls to generate random numbers deterministic: input A always produces output B. This blessing can also be a curse if it is used maliciously.

Perhaps the terms “random” and “deterministic” seem like they cannot exist next to each other. To make that clearer, here’s an extremely trimmed down version of random() that iteratively creates a “random” number by using x = (x * 3) % 19. x is originally defined as a seed value and then morphs into a deterministic sequence of numbers based on that seed:

class NotSoRandom(object):
    def seed(self, a=3):
        """Seed the world's most mysterious random number generator."""
        self.seedval = a
    def random(self):
        """Look, random numbers!"""
        self.seedval = (self.seedval * 3) % 19
        return self.seedval

_inst = NotSoRandom()
seed = _inst.seed
random = _inst.random

Don’t take this example too literally, as it’s meant mainly to illustrate the concept. If you use the seed value 1234, the subsequent sequence of calls to random() should always be identical:

>>> seed(1234)
>>> [random() for _ in range(10)]
[16, 10, 11, 14, 4, 12, 17, 13, 1, 3]

>>> seed(1234)
>>> [random() for _ in range(10)]
[16, 10, 11, 14, 4, 12, 17, 13, 1, 3]

You’ll see a more serious illustration of this shortly.

What Is “Cryptographically Secure?”

If you haven’t had enough with the “RNG” acronyms, let’s throw one more into the mix: a CSPRNG, or cryptographically secure PRNG. CSPRNGs are suitable for generating sensitive data such as passwords, authenticators, and tokens. Given a random string, there is realistically no way for Malicious Joe to determine what string came before or after that string in a sequence of random strings.

One other term that you may see is entropy. In a nutshell, this refers to the amount of randomness introduced or desired. For example, one Python module that you’ll cover here defines DEFAULT_ENTROPY = 32, the number of bytes to return by default. The developers deem this to be “enough” bytes to be a sufficient amount of noise.

Note: Through this tutorial, I assume that a byte refers to 8 bits, as it has since the 1960s, rather than some other unit of data storage. You are free to call this an octet if you so prefer.

A key point about CSPRNGs is that they are still pseudorandom. They are engineered in some way that is internally deterministic, but they add some other variable or have some property that makes them “random enough” to prohibit backing into whatever function enforces determinism.

What You’ll Cover Here

In practical terms, this means that you should use plain PRNGs for statistical modeling, simulation, and to make random data reproducible. They’re also significantly faster than CSPRNGs, as you’ll see later on. Use CSPRNGs for security and cryptographic applications where data sensitivity is imperative.

In addition to expanding on the use cases above, in this tutorial, you’ll delve into Python tools for using both PRNGs and CSPRNGs:

PRNG options include the random module from Python’s standard library and its array-based NumPy counterpart, numpy.random.
Python’s os, secrets, and uuid modules contain functions for generating cryptographically secure objects.

You’ll touch on all of the above and wrap up with a high-level comparison.

PRNGs in Python

The `random` Module

Probably the most widely known tool for generating random data in Python is its random module, which uses the Mersenne Twister PRNG algorithm as its core generator.

Earlier, you touched briefly on random.seed(), and now is a good time to see how it works. First, let’s build some random data without seeding. The random.random() function returns a random float in the interval [0.0, 1.0). The result will always be less than the right-hand endpoint (1.0). This is also known as a semi-open range:

>>> # Don't call `random.seed()` yet
>>> import random
>>> random.random()
0.35553263284394376
>>> random.random()
0.6101992345575074

If you run this code yourself, I’ll bet my life savings that the numbers returned on your machine will be different. The default when you don’t seed the generator is to use your current system time or a “randomness source” from your OS if one is available.

With random.seed(), you can make results reproducible, and the chain of calls after random.seed() will produce the same trail of data:

>>> random.seed(444)
>>> random.random()
0.3088946587429545
>>> random.random()
0.01323751590501987

>>> random.seed(444)  # Re-seed
>>> random.random()
0.3088946587429545
>>> random.random()
0.01323751590501987

Notice the repetition of “random” numbers. The sequence of random numbers becomes deterministic, or completely determined by the seed value, 444.

Let’s take a look at some more basic functionality of random. Above, you generated a random float. You can generate a random integer between two endpoints in Python with the random.randint() function. This spans the full [x, y] interval and may include both endpoints:

>>> random.randint(0, 10)
7
>>> random.randint(500, 50000)
18601

With random.randrange(), you can exclude the right-hand side of the interval, meaning the generated number always lies within [x, y) and will always be smaller than the right endpoint:

>>> random.randrange(1, 10)
5

If you need to generate random floats that lie within a specific [x, y] interval, you can use random.uniform(), which plucks from the continuous uniform distribution:

>>> random.uniform(20, 30)
27.42639687016509
>>> random.uniform(30, 40)
36.33865802745107

To pick a random element from a non-empty sequence (like a list or a tuple), you can use random.choice(). There is also random.choices() for choosing multiple elements from a sequence with replacement (duplicates are possible):

>>> items = ['one', 'two', 'three', 'four', 'five']
>>> random.choice(items)
'four'

>>> random.choices(items, k=2)
['three', 'three']
>>> random.choices(items, k=3)
['three', 'five', 'four']

To mimic sampling without replacement, use random.sample():

>>> random.sample(items, 4)
['one', 'five', 'four', 'three']

You can randomize a sequence in-place using random.shuffle(). This will modify the sequence object and randomize the order of elements:

>>> random.shuffle(items)
>>> items
['four', 'three', 'two', 'one', 'five']

If you’d rather not mutate the original list, you’ll need to make a copy first and then shuffle the copy. You can create copies of Python lists with the copy module, or just x[:] or x.copy(), where x is the list.

Before moving on to generating random data with NumPy, let’s look at one more slightly involved application: generating a sequence of unique random strings of uniform length.

It can help to think about the design of the function first. You need to choose from a “pool” of characters such as letters, numbers, and/or punctuation, combine these into a single string, and then check that this string has not already been generated. A Python set works well for this type of membership testing:

import string

def unique_strings(k: int, ntokens: int,
               pool: str=string.ascii_letters) -> set:
    """Generate a set of unique string tokens.

    k: Length of each token
    ntokens: Number of tokens
    pool: Iterable of characters to choose from

    For a highly optimized version:
    https://stackoverflow.com/a/48421303/7954504
    """

    seen = set()

    # An optimization for tightly-bound loops:
    # Bind these methods outside of a loop
    join = ''.join
    add = seen.add

    while len(seen) < ntokens:
        token = join(random.choices(pool, k=k))
        add(token)
    return seen

''.join() joins the letters from random.choices() into a single Python str of length k. This token is added to the set, which can’t contain duplicates, and the while loop executes until the set has the number of elements that you specify.

Resource: Python’s string module contains a number of useful constants: ascii_lowercase, ascii_uppercase, string.punctuation, ascii_whitespace, and a handful of others.

Let’s try this function out:

>>> unique_strings(k=4, ntokens=5)
{'AsMk', 'Cvmi', 'GIxv', 'HGsZ', 'eurU'}

>>> unique_strings(5, 4, string.printable)
{"'O*1!", '9Ien%', 'W=m7<', 'mUD|z'}

For a fine-tuned version of this function, this Stack Overflow answer uses generator functions, name binding, and some other advanced tricks to make a faster, cryptographically secure version of unique_strings() above.

PRNGs for Arrays: `numpy.random`

One thing you might have noticed is that a majority of the functions from random return a scalar value (a single int, float, or other object). If you wanted to generate a sequence of random numbers, one way to achieve that would be with a Python list comprehension:

>>> [random.random() for _ in range(5)]
[0.021655420657909374,
 0.4031628347066195,
 0.6609991871223335,
 0.5854998250783767,
 0.42886606317322706]

But there is another option that is specifically designed for this. You can think of NumPy’s own numpy.random package as being like the standard library’s random, but for NumPy arrays. (It also comes loaded with the ability to draw from a lot more statistical distributions.)

Take note that numpy.random uses its own PRNG that is separate from plain old random. You won’t produce deterministically random NumPy arrays with a call to Python’s own random.seed():

>>> import numpy as np
>>> np.random.seed(444)
>>> np.set_printoptions(precision=2)  # Output decimal fmt.

Without further ado, here are a few examples to whet your appetite:

>>> # Return samples from the standard normal distribution
>>> np.random.randn(5)
array([ 0.36,  0.38,  1.38,  1.18, -0.94])

>>> np.random.randn(3, 4)
array([[-1.14, -0.54, -0.55,  0.21],
       [ 0.21,  1.27, -0.81, -3.3 ],
       [-0.81, -0.36, -0.88,  0.15]])

>>> # `p` is the probability of choosing each element
>>> np.random.choice([0, 1], p=[0.6, 0.4], size=(5, 4))
array([[0, 0, 1, 0],
       [0, 1, 1, 1],
       [1, 1, 1, 0],
       [0, 0, 0, 1],
       [0, 1, 0, 1]])

In the syntax for randn(d0, d1, ..., dn), the parameters d0, d1, ..., dn are optional and indicate the shape of the final object. Here, np.random.randn(3, 4) creates a 2d array with 3 rows and 4 columns. The data will be i.i.d., meaning that each data point is drawn independent of the others.

Another common operation is to create a sequence of random Boolean values, True or False. One way to do this would be with np.random.choice([True, False]). However, it’s actually about 4x faster to choose from (0, 1) and then view-cast these integers to their corresponding Boolean values:

>>> # NumPy's `randint` is [inclusive, exclusive), unlike `random.randint()`
>>> np.random.randint(0, 2, size=25, dtype=np.uint8).view(bool)
array([ True, False,  True,  True, False,  True, False, False, False,
       False, False,  True,  True, False, False, False,  True, False,
        True, False,  True,  True,  True, False,  True])

What about generating correlated data? Let’s say you want to simulate two correlated time series. One way of going about this is with NumPy’s multivariate_normal() function, which takes a covariance matrix into account. In other words, to draw from a single normally distributed random variable, you need to specify its mean and variance (or standard deviation).

To sample from the multivariate normal distribution, you specify the means and covariance matrix, and you end up with multiple, correlated series of data that are each approximately normally distributed.

However, rather than covariance, correlation is a measure that is more familiar and intuitive to most. It’s the covariance normalized by the product of standard deviations, and so you can also define covariance in terms of correlation and standard deviation:

So, could you draw random samples from a multivariate normal distribution by specifying a correlation matrix and standard deviations? Yes, but you’ll need to get the above into matrix form first. Here, S is a vector of the standard deviations, P is their correlation matrix, and C is the resulting (square) covariance matrix:

This can be expressed in NumPy as follows:

def corr2cov(p: np.ndarray, s: np.ndarray) -> np.ndarray:
    """Covariance matrix from correlation & standard deviations"""
    d = np.diag(s)
    return d @ p @ d

Now, you can generate two time series that are correlated but still random:

>>> # Start with a correlation matrix and standard deviations.
>>> # -0.40 is the correlation between A and B, and the correlation
>>> # of a variable with itself is 1.0.
>>> corr = np.array([[1., -0.40],
...                  [-0.40, 1.]])

>>> # Standard deviations/means of A and B, respectively
>>> stdev = np.array([6., 1.])
>>> mean = np.array([2., 0.5])
>>> cov = corr2cov(corr, stdev)

>>> # `size` is the length of time series for 2d data
>>> # (500 months, days, and so on).
>>> data = np.random.multivariate_normal(mean=mean, cov=cov, size=500)
>>> data[:10]
array([[ 0.58,  1.87],
       [-7.31,  0.74],
       [-6.24,  0.33],
       [-0.77,  1.19],
       [ 1.71,  0.7 ],
       [-3.33,  1.57],
       [-1.13,  1.23],
       [-6.58,  1.81],
       [-0.82, -0.34],
       [-2.32,  1.1 ]])
>>> data.shape
(500, 2)

You can think of data as 500 pairs of inversely correlated data points. Here’s a sanity check that you can back into the original inputs, which approximate corr, stdev, and mean from above:

>>> np.corrcoef(data, rowvar=False)
array([[ 1.  , -0.39],
       [-0.39,  1.  ]])

>>> data.std(axis=0)
array([5.96, 1.01])

>>> data.mean(axis=0)
array([2.13, 0.49])

Before we move on to CSPRNGs, it might be helpful to summarize some random functions and their numpy.random counterparts:

Python `random` Module	NumPy Counterpart	Use
`random()`	`rand()`	Random float in [0.0, 1.0)
`randint(a, b)`	`random_integers()`	Random integer in [a, b]
`randrange(a, b[, step])`	`randint()`	Random integer in [a, b)
`uniform(a, b)`	`uniform()`	Random float in [a, b]
`choice(seq)`	`choice()`	Random element from `seq`
`choices(seq, k=1)`	`choice()`	Random `k` elements from `seq` with replacement
`sample(population, k)`	`choice()` with `replace=False`	Random `k` elements from `seq` without replacement
`shuffle(x[, random])`	`shuffle()`	Shuffle the sequence `x` in place
`normalvariate(mu, sigma)` or `gauss(mu, sigma)`	`normal()`	Sample from a normal distribution with mean `mu` and standard deviation `sigma`

Note: NumPy is specialized for building and manipulating large, multidimensional arrays. If you just need a single value, random will suffice and will probably be faster as well. For small sequences, random may even be faster too, because NumPy does come with some overhead.

Now that you’ve covered two fundamental options for PRNGs, let’s move onto a few more secure adaptations.

CSPRNGs in Python

`os.urandom()`: About as Random as It Gets

Python’s os.urandom() function is used by both secrets and uuid (both of which you’ll see here in a moment). Without getting into too much detail, os.urandom() generates operating-system-dependent random bytes that can safely be called cryptographically secure:

On Unix operating systems, it reads random bytes from the special file /dev/urandom, which in turn “allow access to environmental noise collected from device drivers and other sources.” (Thank you, Wikipedia.) This is garbled information that is particular to your hardware and system state at an instance in time but at the same time sufficiently random.
On Windows, the C++ function CryptGenRandom() is used. This function is still technically pseudorandom, but it works by generating a seed value from variables such as the process ID, memory status, and so on.

With os.urandom(), there is no concept of manually seeding. While still technically pseudorandom, this function better aligns with how we think of randomness. The only argument is the number of bytes to return:

>>> os.urandom(3)
b'\xa2\xe8\x02'

>>> x = os.urandom(6)
>>> x
b'\xce\x11\xe7"!\x84'

>>> type(x), len(x)
(bytes, 6)

Before we go any further, this might be a good time to delve into a mini-lesson on character encoding. Many people, including myself, have some type of allergic reaction when they see bytes objects and a long line of \x characters. However, it’s useful to know how sequences such as x above eventually get turned into strings or numbers.

os.urandom() returns a sequence of single bytes:

>>> x
b'\xce\x11\xe7"!\x84'

But how does this eventually get turned into a Python str or sequence of numbers?

First, recall one of the fundamental concepts of computing, which is that a byte is made up of 8 bits. You can think of a bit as a single digit that is either 0 or 1. A byte effectively chooses between 0 and 1 eight times, so both 01101100 and 11110000 could represent bytes. Try this, which makes use of Python f-strings introduced in Python 3.6, in your interpreter:

>>> binary = [f'{i:0>8b}' for i in range(256)]
>>> binary[:16]
['00000000',
 '00000001',
 '00000010',
 '00000011',
 '00000100',
 '00000101',
 '00000110',
 '00000111',
 '00001000',
 '00001001',
 '00001010',
 '00001011',
 '00001100',
 '00001101',
 '00001110',
 '00001111']

This is equivalent to [bin(i) for i in range(256)], with some special formatting. bin() converts an integer to its binary representation as a string.

Where does that leave us? Using range(256) above is not a random choice. (No pun intended.) Given that we are allowed 8 bits, each with 2 choices, there are 2 ** 8 == 256 possible bytes “combinations.”

This means that each byte maps to an integer between 0 and 255. In other words, we would need more than 8 bits to express the integer 256. You can verify this by checking that len(f'{256:0>8b}') is now 9, not 8.

Okay, now let’s get back to the bytes data type that you saw above, by constructing a sequence of the bytes that correspond to integers 0 through 255:

>>> bites = bytes(range(256))

If you call list(bites), you’ll get back to a Python list that runs from 0 to 255. But if you just print bites, you get an ugly looking sequence littered with backslashes:

>>> bites
b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15'
 '\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJK'
 'LMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86'
 '\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b'

 # ...

These backslashes are escape sequences, and \xhh represents the character with hex value hh. Some of the elements of bites are displayed literally (printable characters such as letters, numbers, and punctuation). Most are expressed with escapes. \x08 represents a keyboard’s backspace, while \x13 is a carriage return (part of a new line, on Windows systems).

If you need a refresher on hexadecimal, Charles Petzold’s Code: The Hidden Language is a great place for that. Hex is a base-16 numbering system that, instead of using 0 through 9, uses 0 through 9 and a through f as its basic digits.

Finally, let’s get back to where you started, with the sequence of random bytes x. Hopefully this makes a little more sense now. Calling .hex() on a bytes object gives a str of hexadecimal numbers, with each corresponding to a decimal number from 0 through 255:

>>> x
b'\xce\x11\xe7"!\x84'

>>> list(x)
[206, 17, 231, 34, 33, 132]

>>> x.hex()
'ce11e7222184'

>>> len(x.hex())
12

One last question: how is b.hex() 12 characters long above, even though x is only 6 bytes? This is because two hexadecimal digits correspond precisely to a single byte. The str version of bytes will always be twice as long as far as our eyes are concerned.

Even if the byte (such as \x01) does not need a full 8 bits to be represented, b.hex() will always use two hex digits per byte, so the number 1 will be represented as 01 rather than just 1. Mathematically, though, both of these are the same size.

Technical Detail: What you’ve mainly dissected here is how a bytes object becomes a Python str. One other technicality is how bytes produced by os.urandom() get converted to a float in the interval [0.0, 1.0), as in the cryptographically secure version of random.random(). If you’re interested in exploring this further, this code snippet demonstrates how int.from_bytes() makes the initial conversion to an integer, using a base-256 numbering system.

With that under your belt, let’s touch on a recently introduced module, secrets, which makes generating secure tokens much more user-friendly.

Python’s Best Kept `secrets`

Introduced in Python 3.6 by one of the more colorful PEPs out there, the secrets module is intended to be the de facto Python module for generating cryptographically secure random bytes and strings.

You can check out the source code for the module, which is short and sweet at about 25 lines of code. secrets is basically a wrapper around os.urandom(). It exports just a handful of functions for generating random numbers, bytes, and strings. Most of these examples should be fairly self-explanatory:

>>> n = 16

>>> # Generate secure tokens
>>> secrets.token_bytes(n)
b'A\x8cz\xe1o\xf9!;\x8b\xf2\x80pJ\x8b\xd4\xd3'
>>> secrets.token_hex(n)
'9cb190491e01230ec4239cae643f286f'  
>>> secrets.token_urlsafe(n)
'MJoi7CknFu3YN41m88SEgQ'

>>> # Secure version of `random.choice()`
>>> secrets.choice('rain')
'a'

Now, how about a concrete example? You’ve probably used URL shortener services like tinyurl.com or bit.ly that turn an unwieldy URL into something like https://bit.ly/2IcCp9u. Most shorteners don’t do any complicated hashing from input to output; they just generate a random string, make sure that string has not already been generated previously, and then tie that back to the input URL.

Let’s say that after taking a look at the Root Zone Database, you’ve registered the site short.ly. Here’s a function to get you started with your service:

# shortly.py

from secrets import token_urlsafe

DATABASE = {}

def shorten(url: str, nbytes: int=5) -> str:
    ext = token_urlsafe(nbytes=nbytes)
    if ext in DATABASE:
        return shorten(url, nbytes=nbytes)
    else:
        DATABASE.update({ext: url})
        return f'short.ly/{ext}

Is this a full-fledged real illustration? No. I would wager that bit.ly does things in a slightly more advanced way than storing its gold mine in a global Python dictionary that is not persistent between sessions. However, it’s roughly accurate conceptually:

>>> urls = (
...     'https://realpython.com/',
...     'https://docs.python.org/3/howto/regex.html'
... )

>>> for u in urls:
...     print(shorten(u))
short.ly/p_Z4fLI
short.ly/fuxSyNY

>>> DATABASE
{'p_Z4fLI': 'https://realpython.com/',
 'fuxSyNY': 'https://docs.python.org/3/howto/regex.html'}

Hold On: One thing you may notice is that both of these results are of length 7 when you requested 5 bytes. Wait, I thought that you said the result would be twice as long? Well, not exactly, in this case. There is one more thing going on here: token_urlsafe() uses base64 encoding, where each character is 6 bits of data. (It’s 0 through 63, and corresponding characters. The characters are A-Z, a-z, 0-9, and +/.)

If you originally specify a certain number of bytes nbytes, the resulting length from secrets.token_urlsafe(nbytes) will be math.ceil(nbytes * 8 / 6), which you can prove and investigate further if you’re curious.

The bottom line here is that, while secrets is really just a wrapper around existing Python functions, it can be your go-to when security is your foremost concern.

One Last Candidate: `uuid`

One last option for generating a random token is the uuid4() function from Python’s uuid module. A UUID is a Universally Unique IDentifier, a 128-bit sequence (str of length 32) designed to “guarantee uniqueness across space and time.” uuid4() is one of the module’s most useful functions, and this function also uses os.urandom():

>>> import uuid

>>> uuid.uuid4()
UUID('3e3ef28d-3ff0-4933-9bba-e5ee91ce0e7b')
>>> uuid.uuid4()
UUID('2e115fcb-5761-4fa1-8287-19f4ee2877ac')

The nice thing is that all of uuid’s functions produce an instance of the UUID class, which encapsulates the ID and has properties like .int, .bytes, and .hex:

>>> tok = uuid.uuid4()
>>> tok.bytes
b'.\xb7\x80\xfd\xbfIG\xb3\xae\x1d\xe3\x97\xee\xc5\xd5\x81'

>>> len(tok.bytes)
16
>>> len(tok.bytes) * 8  # In bits
128

>>> tok.hex
'2eb780fdbf4947b3ae1de397eec5d581'
>>> tok.int
62097294383572614195530565389543396737

You may also have seen some other variations: uuid1(), uuid3(), and uuid5(). The key difference between these and uuid4() is that those three functions all take some form of input and therefore don’t meet the definition of “random” to the extent that a Version 4 UUID does:

uuid1() uses your machine’s host ID and current time by default. Because of the reliance on current time down to nanosecond resolution, this version is where UUID derives the claim “guaranteed uniqueness across time.”
uuid3() and uuid5() both take a namespace identifier and a name. The former uses an MD5 hash and the latter uses SHA-1.

uuid4(), conversely, is entirely pseudorandom (or random). It consists of getting 16 bytes via os.urandom(), converting this to a big-endian integer, and doing a number of bitwise operations to comply with the formal specification.

Hopefully, by now you have a good idea of the distinction between different “types” of random data and how to create them. However, one other issue that might come to mind is that of collisions.

In this case, a collision would simply refer to generating two matching UUIDs. What is the chance of that? Well, it is technically not zero, but perhaps it is close enough: there are 2 ** 128 or 340 undecillion possible uuid4 values. So, I’ll leave it up to you to judge whether this is enough of a guarantee to sleep well.

One common use of uuid is in Django, which has a UUIDField that is often used as a primary key in a model’s underlying relational database.

Why Not Just “Default to” `SystemRandom`?

In addition to the secure modules discussed here such as secrets, Python’s random module actually has a little-used class called SystemRandom that uses os.urandom(). (SystemRandom, in turn, is also used by secrets. It’s all a bit of a web that traces back to urandom().)

At this point, you might be asking yourself why you wouldn’t just “default to” this version? Why not “always be safe” rather than defaulting to the deterministic random functions that aren’t cryptographically secure ?

I’ve already mentioned one reason: sometimes you want your data to be deterministic and reproducible for others to follow along with.

But the second reason is that CSPRNGs, at least in Python, tend to be meaningfully slower than PRNGs. Let’s test that with a script, timed.py, that compares the PRNG and CSPRNG versions of randint() using Python’s timeit.repeat():

# timed.py

import random
import timeit

# The "default" random is actually an instance of `random.Random()`.
# The CSPRNG version uses `SystemRandom()` and `os.urandom()` in turn.
_sysrand = random.SystemRandom()

def prng() -> None:
    random.randint(0, 95)

def csprng() -> None:
    _sysrand.randint(0, 95)

setup = 'import random; from __main__ import prng, csprng'

if __name__ == '__main__':
    print('Best of 3 trials with 1,000,000 loops per trial:')

    for f in ('prng()', 'csprng()'):
        best = min(timeit.repeat(f, setup=setup))
        print('\t{:8s} {:0.2f} seconds total time.'.format(f, best))

Now to execute this from the shell:

$ python3 ./timed.py
Best of 3 trials with 1,000,000 loops per trial:
        prng()   1.07 seconds total time.
        csprng() 6.20 seconds total time.

A 5x timing difference is certainly a valid consideration in addition to cryptographic security when choosing between the two.

Odds and Ends: Hashing

One concept that hasn’t received much attention in this tutorial is that of hashing, which can be done with Python’s hashlib module.

A hash is designed to be a one-way mapping from an input value to a fixed-size string that is virtually impossible to reverse engineer. As such, while the result of a hash function may “look like” random data, it doesn’t really qualify under the definition here.

Recap

You’ve covered a lot of ground in this tutorial. To recap, here is a high-level comparison of the options available to you for engineering randomness in Python:

Package/Module	Description	Cryptographically Secure
`random`	Fasty & easy random data using Mersenne Twister	No
`numpy.random`	Like `random` but for (possibly multidimensional) arrays	No
`os`	Contains `urandom()`, the base of other functions covered here	Yes
`secrets`	Designed to be Python’s de facto module for generating secure random numbers, bytes, and strings	Yes
`uuid`	Home to a handful of functions for building 128-bit identifiers	Yes, `uuid4()`

Feel free to leave some totally random comments below, and thanks for reading.

Additional Links

Random.org offers “true random numbers to anyone on the Internet” derived from atmospheric noise.
The Recipes section from the random module has some additional tricks.
The seminal paper on the Mersienne Twister appeared in 1997, if you’re into that kind of thing.
The Itertools Recipes define functions for choosing randomly from a combinatoric set, such as from combinations or permutations.
Scikit-Learn includes various random sample generators that can be used to build artificial datasets of controlled size and complexity.
Eli Bendersky digs into random.randint() in his article Slow and Fast Methods for Generating Random Integers in Python.
Peter Norvig’s a Concrete Introduction to Probability using Python is a comprehensive resource as well.
The Pandas library includes a context manager that can be used to set a temporary random state.
From Stack Overflow:

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Strings and Character Data in Python

2018-07-09T14:00:00+00:00

In the tutorial on Basic Data Types in Python, you learned how to define strings: objects that contain sequences of character data. Processing character data is integral to programming. It is a rare application that doesn’t need to manipulate strings at least to some extent.

Here’s what you’ll learn in this tutorial: Python provides a rich set of operators, functions, and methods for working with strings. When you are finished with this tutorial, you will know how to access and extract portions of strings, and also be familiar with the methods that are available to manipulate and modify string data.

You will also be introduced to two other Python objects used to represent raw byte data, the bytes and bytearray types.

String Manipulation

The sections below highlight the operators, methods, and functions that are available for working with strings.

String Operators

You have already seen the operators + and * applied to numeric operands in the tutorial on Operators and Expressions in Python. These two operators can be applied to strings as well.

The `+` Operator

The + operator concatenates strings. It returns a string consisting of the operands joined together, as shown here:

>>> s = 'foo'
>>> t = 'bar'
>>> u = 'baz'

>>> s + t
'foobar'
>>> s + t + u
'foobarbaz'

>>> print('Go team' + '!!!')
Go team!!!

The `*` Operator

The * operator creates multiple copies of a string. If s is a string and n is an integer, either of the following expressions returns a string consisting of n concatenated copies of s:

s * n
n * s

Here are examples of both forms:

>>> s = 'foo.'

>>> s * 4
'foo.foo.foo.foo.'
>>> 4 * s
'foo.foo.foo.foo.'

The multiplier operand n must be an integer. You’d think it would be required to be a positive integer, but amusingly, it can be zero or negative, in which case the result is an empty string:

>>> 'foo' * -8
''

If you were to create a string variable and initialize it to the empty string by assigning it the value 'foo' * -8, anyone would rightly think you were a bit daft. But it would work.

The `in` Operator

Python also provides a membership operator that can be used with strings. The in operator returns True if the first operand is contained within the second, and False otherwise:

>>> s = 'foo'

>>> s in 'That\'s food for thought.'
True
>>> s in 'That\'s good for now.'
False

There is also a not in operator, which does the opposite:

>>> 'z' not in 'abc'
True
>>> 'z' not in 'xyz'
False

Built-in String Functions

As you saw in the tutorial on Basic Data Types in Python, Python provides many functions that are built-in to the interpreter and always available. Here are a few that work with strings:

Function	Description
`chr()`	Converts an integer to a character
`ord()`	Converts a character to an integer
`len()`	Returns the length of a string
`str()`	Returns a string representation of an object

These are explored more fully below.

ord(c)

Returns an integer value for the given character.

At the most basic level, computers store all information as numbers. To represent character data, a translation scheme is used which maps each character to its representative number.

The simplest scheme in common use is called ASCII. It covers the common Latin characters you are probably most accustomed to working with. For these characters, ord(c) returns the ASCII value for character c:

>>> ord('a')
97
>>> ord('#')
35

ASCII is fine as far as it goes. But there are many different languages in use in the world and countless symbols and glyphs that appear in digital media. The full set of characters that potentially may need to be represented in computer code far surpasses the ordinary Latin letters, numbers, and symbols you usually see.

Unicode is an ambitious standard that attempts to provide a numeric code for every possible character, in every possible language, on every possible platform. Python 3 supports Unicode extensively, including allowing Unicode characters within strings.

For More Information: See Python’s Unicode Support in the Python documentation.

As long as you stay in the domain of the common characters, there is little practical difference between ASCII and Unicode. But the ord() function will return numeric values for Unicode characters as well:

>>> ord('€')
8364
>>> ord('∑')
8721

chr(n)

Returns a character value for the given integer.

chr() does the reverse of ord(). Given a numeric value n, chr(n) returns a string representing the character that corresponds to n:

>>> chr(97)
'a'
>>> chr(35)
'#'

chr() handles Unicode characters as well:

>>> chr(8364)
'€'
>>> chr(8721)
'∑'

len(s)

Returns the length of a string.

len(s) returns the number of characters in s:

>>> s = 'I am a string.'
>>> len(s)
14

str(obj)

Returns a string representation of an object.

Virtually any object in Python can be rendered as a string. str(obj) returns the string representation of object obj:

>>> str(49.2)
'49.2'
>>> str(3+4j)
'(3+4j)'
>>> str(3 + 29)
'32'
>>> str('foo')
'foo'

String Indexing

Often in programming languages, individual items in an ordered set of data can be accessed directly using a numeric index or key value. This process is referred to as indexing.

In Python, strings are ordered sequences of character data, and thus can be indexed in this way. Individual characters in a string can be accessed by specifying the string name followed by a number in square brackets ([]).

String indexing in Python is zero-based: the first character in the string has index 0, the next has index 1, and so on. The index of the last character will be the length of the string minus one.

For example, a schematic diagram of the indices of the string 'foobar' would look like this:

String Indices

The individual characters can be accessed by index as follows:

>>> s = 'foobar'

>>> s[0]
'f'
>>> s[1]
'o'
>>> s[3]
'b'
>>> len(s)
6
>>> s[len(s)-1]
'r'

Attempting to index beyond the end of the string results in an error:

>>> s[6]
Traceback (most recent call last):
  File "<pyshell#17>", line 1, in <module>
    s[6]
IndexError: string index out of range

String indices can also be specified with negative numbers, in which case indexing occurs from the end of the string backward: -1 refers to the last character, -2 the second-to-last character, and so on. Here is the same diagram showing both the positive and negative indices into the string 'foobar':

Positive and Negative String Indices

Here are some examples of negative indexing:

>>> s = 'foobar'

>>> s[-1]
'r'
>>> s[-2]
'a'
>>> len(s)
6
>>> s[-len(s)]
'f'

Attempting to index with negative numbers beyond the start of the string results in an error:

>>> s[-7]
Traceback (most recent call last):
  File "<pyshell#26>", line 1, in <module>
    s[-7]
IndexError: string index out of range

For any non-empty string s, s[len(s)-1] and s[-1] both return the last character. There isn’t any index that makes sense for an empty string.

String Slicing

Python also allows a form of indexing syntax that extracts substrings from a string, known as string slicing. If s is a string, an expression of the form s[m:n] returns the portion of s starting with position m, and up to but not including position n:

>>> s = 'foobar'
>>> s[2:5]
'oba'

Remember: String indices are zero-based. The first character in a string has index 0. This applies to both standard indexing and slicing.

Again, the second index specifies the first character that is not included in the result—the character 'r' (s[5]) in the example above. That may seem slightly unintuitive, but it produces this result which makes sense: the expression s[m:n] will return a substring that is n - m characters in length, in this case, 5 - 2 = 3.

If you omit the first index, the slice starts at the beginning of the string. Thus, s[:m] and s[0:m] are equivalent:

>>> s = 'foobar'

>>> s[:4]
'foob'
>>> s[0:4]
'foob'

Similarly, if you omit the second index as in s[n:], the slice extends from the first index through the end of the string. This is a nice, concise alternative to the more cumbersome s[n:len(s)]:

>>> s = 'foobar'

>>> s[2:]
'obar'
>>> s[2:len(s)]
'obar'

For any string s and any integer n (0 ≤ n ≤ len(s)), s[:n] + s[n:] will be equal to s:

>>> s = 'foobar'

>>> s[:4] + s[4:]
'foobar'
>>> s[:4] + s[4:] == s
True

Omitting both indices returns the original string, in its entirety. Literally. It’s not a copy, it’s a reference to the original string:

>>> s = 'foobar'
>>> t = s[:]
>>> id(s)
59598496
>>> id(t)
59598496
>>> s is t
True

If the first index in a slice is greater than or equal to the second index, Python returns an empty string. This is yet another obfuscated way to generate an empty string, in case you were looking for one:

>>> s[2:2]
''
>>> s[4:2]
''

Negative indices can be used with slicing as well. -1 refers to the last character, -2 the second-to-last, and so on, just as with simple indexing. The diagram below shows how to slice the substring 'oob' from the string 'foobar' using both positive and negative indices:

String Slicing with Positive and Negative Indices

Here is the corresponding Python code:

>>> s = 'foobar'

>>> s[-5:-2]
'oob'
>>> s[1:4]
'oob'
>>> s[-5:-2] == s[1:4]
True

Specifying a Stride in a String Slice

There is one more variant of the slicing syntax to discuss. Adding an additional : and a third index designates a stride (also called a step), which indicates how many characters to jump after retrieving each character in the slice.

For example, for the string 'foobar', the slice 0:6:2 starts with the first character and ends with the last character (the whole string), and every second character is skipped. This is shown in the following diagram:

String Indexing with Stride

Similarly, 1:6:2 specifies a slice starting with the second character (index 1) and ending with the last character, and again the stride value 2 causes every other character to be skipped:

Another String Indexing with Stride

The illustrative REPL code is shown here:

>>> s = 'foobar'

>>> s[0:6:2]
'foa'

>>> s[1:6:2]
'obr'

As with any slicing, the first and second indices can be omitted, and default to the first and last characters respectively:

>>> s = '12345' * 5
>>> s
'1234512345123451234512345'
>>> s[::5]
'11111'
>>> s[4::5]
'55555'

You can specify a negative stride value as well, in which case Python steps backward through the string. In that case, the starting/first index should be greater than the ending/second index:

>>> s = 'foobar'
>>> s[5:0:-2]
'rbo'

In the above example, 5:0:-2 means “start at the last character and step backward by 2, up to but not including the first character.”

When you are stepping backward, if the first and second indices are omitted, the defaults are reversed in an intuitive way: the first index defaults to the end of the string, and the second index defaults to the beginning. Here is an example:

>>> s = '12345' * 5
>>> s
'1234512345123451234512345'
>>> s[::-5]
'55555'

This is a common paradigm for reversing a string:

>>> s = 'If Comrade Napoleon says it, it must be right.'
>>> s[::-1]
'.thgir eb tsum ti ,ti syas noelopaN edarmoC fI'

Interpolating Variables Into a String

In Python version 3.6, a new string formatting mechanism was introduced. This feature is formally named the Formatted String Literal, but is more usually referred to by its nickname f-string.

The formatting capability provided by f-strings is extensive and won’t be covered in full detail here. If you want to learn more, you can check out the Real Python article Python 3’s f-Strings: An Improved String Formatting Syntax (Guide). There is also a tutorial on Formatted Output coming up later in this series that digs deeper into f-strings.

One simple feature of f-strings you can start using right away is variable interpolation. You can specify a variable name directly within an f-string literal, and Python will replace the name with the corresponding value.

For example, suppose you want to display the result of an arithmetic calculation. You can do this with a straightforward print() statement, separating numeric values and string literals by commas:

>>> n = 20
>>> m = 25
>>> prod = n * m
>>> print('The product of', n, 'and', m, 'is', prod)
The product of 20 and 25 is 500

But this is cumbersome. To accomplish the same thing using an f-string:

Specify either a lowercase f or uppercase F directly before the opening quote of the string literal. This tells Python it is an f-string instead of a standard string.
Specify any variables to be interpolated in curly braces ({}).

Recast using an f-string, the above example looks much cleaner:

>>> n = 20
>>> m = 25
>>> prod = n * m
>>> print(f'The product of {n} and {m} is {prod}')
The product of 20 and 25 is 500

Any of Python’s three quoting mechanisms can be used to define an f-string:

>>> var = 'Bark'

>>> print(f'A dog says {var}!')
A dog says Bark!
>>> print(f"A dog says {var}!")
A dog says Bark!
>>> print(f'''A dog says {var}!''')
A dog says Bark!

Modifying Strings

In a nutshell, you can’t. Strings are one of the data types Python considers immutable, meaning not able to be changed. In fact, all the data types you have seen so far are immutable. (Python does provide data types that are mutable, as you will soon see.)

A statement like this will cause an error:

>>> s = 'foobar'
>>> s[3] = 'x'
Traceback (most recent call last):
  File "<pyshell#40>", line 1, in <module>
    s[3] = 'x'
TypeError: 'str' object does not support item assignment

In truth, there really isn’t much need to modify strings. You can usually easily accomplish what you want by generating a copy of the original string that has the desired change in place. There are very many ways to do this in Python. Here is one possibility:

>>> s = s[:3] + 'x' + s[4:]
>>> s
'fooxar'

There is also a built-in string method to accomplish this:

>>> s = 'foobar'
>>> s = s.replace('b', 'x')
>>> s
'fooxar'

Read on for more information about built-in string methods!

Built-in String Methods

You learned in the tutorial on Variables in Python that Python is a highly object-oriented language. Every item of data in a Python program is an object.

You are also familiar with functions: callable procedures that you can invoke to perform specific tasks.

Methods are similar to functions. A method is a specialized type of callable procedure that is tightly associated with an object. Like a function, a method is called to perform a distinct task, but it is invoked on a specific object and has knowledge of its target object during execution.

The syntax for invoking a method on an object is as follows:

obj.foo(<args>)

This invokes method .foo() on object obj. <args> specifies the arguments passed to the method (if any).

You will explore much more about defining and calling methods later in the discussion of object-oriented programming. For now, the goal is to present some of the more commonly used built-in methods Python supports for operating on string objects.

In the following method definitions, arguments specified in square brackets ([]) are optional.

Case Conversion

Methods in this group perform case conversion on the target string.

s.capitalize()

Capitalizes the target string.

s.capitalize() returns a copy of s with the first character converted to uppercase and all other characters converted to lowercase:

>>> s = 'foO BaR BAZ quX'
>>> s.capitalize()
'Foo bar baz qux'

Non-alphabetic characters are unchanged:

>>> s = 'foo123#BAR#.'
>>> s.capitalize()
'Foo123#bar#.'

s.lower()

Converts alphabetic characters to lowercase.

s.lower() returns a copy of s with all alphabetic characters converted to lowercase:

>>> 'FOO Bar 123 baz qUX'.lower()
'foo bar 123 baz qux'

s.swapcase()

Swaps case of alphabetic characters.

s.swapcase() returns a copy of s with uppercase alphabetic characters converted to lowercase and vice versa:

>>> 'FOO Bar 123 baz qUX'.swapcase()
'foo bAR 123 BAZ Qux'

s.title()

Converts the target string to “title case.”

s.title() returns a copy of s in which the first letter of each word is converted to uppercase and remaining letters are lowercase:

>>> 'the sun also rises'.title()
'The Sun Also Rises'

This method uses a fairly simple algorithm. It does not attempt to distinguish between important and unimportant words, and it does not handle apostrophes, possessives, or acronyms gracefully:

>>> "what's happened to ted's IBM stock?".title()
"What'S Happened To Ted'S Ibm Stock?"

s.upper()

Converts alphabetic characters to uppercase.

s.upper() returns a copy of s with all alphabetic characters converted to uppercase:

>>> 'FOO Bar 123 baz qUX'.upper()
'FOO BAR 123 BAZ QUX'

Find and Replace

These methods provide various means of searching the target string for a specified substring.

Each method in this group supports optional <start> and <end> arguments. These are interpreted as for string slicing: the action of the method is restricted to the portion of the target string starting at character position <start> and proceeding up to but not including character position <end>. If <start> is specified but <end> is not, the method applies to the portion of the target string from <start> through the end of the string.

s.count([, <start>[, <end>]])

Counts occurrences of a substring in the target string.

s.count() returns the number of non-overlapping occurrences of substring  in s:

>>> 'foo goo moo'.count('oo')
3

The count is restricted to the number of occurrences within the substring indicated by <start> and <end>, if they are specified:

>>> 'foo goo moo'.count('oo', 0, 8)
2

s.endswith(<suffix>[, <start>[, <end>]])

Determines whether the target string ends with a given substring.

s.endswith(<suffix>) returns True if s ends with the specified <suffix> and False otherwise:

>>> 'foobar'.endswith('bar')
True
>>> 'foobar'.endswith('baz')
False

The comparison is restricted to the substring indicated by <start> and <end>, if they are specified:

>>> 'foobar'.endswith('oob', 0, 4)
True
>>> 'foobar'.endswith('oob', 2, 4)
False

s.find([, <start>[, <end>]])

Searches the target string for a given substring.

s.find() returns the lowest index in s where substring  is found:

>>> 'foo bar foo baz foo qux'.find('foo')
0

This method returns -1 if the specified substring is not found:

>>> 'foo bar foo baz foo qux'.find('grault')
-1

The search is restricted to the substring indicated by <start> and <end>, if they are specified:

>>> 'foo bar foo baz foo qux'.find('foo', 4)
8
>>> 'foo bar foo baz foo qux'.find('foo', 4, 7)
-1

s.index([, <start>[, <end>]])

Searches the target string for a given substring.

This method is identical to .find(), except that it raises an exception if  is not found rather than returning -1:

>>> 'foo bar foo baz foo qux'.index('grault')
Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    'foo bar foo baz foo qux'.index('grault')
ValueError: substring not found

s.rfind([, <start>[, <end>]])

Searches the target string for a given substring starting at the end.

s.rfind() returns the highest index in s where substring  is found:

>>> 'foo bar foo baz foo qux'.rfind('foo')
16

As with .find(), if the substring is not found, -1 is returned:

>>> 'foo bar foo baz foo qux'.rfind('grault')
-1

The search is restricted to the substring indicated by <start> and <end>, if they are specified:

>>> 'foo bar foo baz foo qux'.rfind('foo', 0, 14)
8
>>> 'foo bar foo baz foo qux'.rfind('foo', 10, 14)
-1

s.rindex([, <start>[, <end>]])

Searches the target string for a given substring starting at the end.

This method is identical to .rfind(), except that it raises an exception if  is not found rather than returning -1:

>>> 'foo bar foo baz foo qux'.rindex('grault')
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    'foo bar foo baz foo qux'.rindex('grault')
ValueError: substring not found

s.startswith(<prefix>[, <start>[, <end>]])

Determines whether the target string starts with a given substring.

s.startswith(<suffix>) returns True if s starts with the specified <suffix> and False otherwise:

>>> 'foobar'.startswith('foo')
True
>>> 'foobar'.startswith('bar')
False

The comparison is restricted to the substring indicated by <start> and <end>, if they are specified:

>>> 'foobar'.startswith('bar', 3)
True
>>> 'foobar'.startswith('bar', 3, 2)
False

Character Classification

Methods in this group classify a string based on the characters it contains.

s.isalnum()

Determines whether the target string consists of alphanumeric characters.

s.isalnum() returns True if s is nonempty and all its characters are alphanumeric (either a letter or a number), and False otherwise:

>>> 'abc123'.isalnum()
True
>>> 'abc$123'.isalnum()
False
>>> ''.isalnum()
False

s.isalpha()

Determines whether the target string consists of alphabetic characters.

s.isalpha() returns True if s is nonempty and all its characters are alphabetic, and False otherwise:

>>> 'ABCabc'.isalpha()
True
>>> 'abc123'.isalpha()
False

s.isdigit()

Determines whether the target string consists of digit characters.

s.digit() returns True if s is nonempty and all its characters are numeric digits, and False otherwise:

>>> '123'.isdigit()
True
>>> '123abc'.isdigit()
False

s.isidentifier()

Determines whether the target string is a valid Python identifier.

s.isidentifier() returns True if s is a valid Python identifier according to the language definition, and False otherwise:

>>> 'foo32'.isidentifier()
True
>>> '32foo'.isidentifier()
False
>>> 'foo$32'.isidentifier()
False

Note: .isidentifier() will return True for a string that matches a Python keyword even though that would not actually be a valid identifier:

>>> 'and'.isidentifier()
True

You can test whether a string matches a Python keyword using a function called iskeyword(), which is contained in a module called keyword. One possible way to do this is shown below:

>>> from keyword import iskeyword
>>> iskeyword('and')
True

If you really want to ensure that a string would serve as a valid Python identifier, you should check that .isidentifier() is True and that iskeyword() is False.

See Python Modules and Packages—An Introduction to read more about Python modules.

s.islower()

Determines whether the target string’s alphabetic characters are lowercase.

s.islower() returns True if s is nonempty and all the alphabetic characters it contains are lowercase, and False otherwise. Non-alphabetic characters are ignored:

>>> 'abc'.islower()
True
>>> 'abc1$d'.islower()
True
>>> 'Abc1$D'.islower()
False

s.isprintable()

Determines whether the target string consists entirely of printable characters.

s.isprintable() returns True if s is empty or all the alphabetic characters it contains are printable. It returns False if s contains at least one non-printable character. Non-alphabetic characters are ignored:

>>> 'a\tb'.isprintable()
False
>>> 'a b'.isprintable()
True
>>> ''.isprintable()
True
>>> 'a\nb'.isprintable()
False

Note: This is the only .isxxxx() method that returns True if s is an empty string. All the others return False for an empty string.

s.isspace()

Determines whether the target string consists of whitespace characters.

s.isspace() returns True if s is nonempty and all characters are whitespace characters, and False otherwise.

The most commonly encountered whitespace characters are space ' ', tab '\t', and newline '\n':

>>> ' \t \n '.isspace()
True
>>> '   a   '.isspace()
False

However, there are a few other ASCII characters that qualify as whitespace, and if you account for Unicode characters, there are quite a few beyond that:

>>> '\f\u2005\r'.isspace()
True

('\f' and '\r' are the escape sequences for the ASCII Form Feed and Carriage Return characters; '\u2005' is the escape sequence for the Unicode Four-Per-Em Space.)

s.istitle()

Determines whether the target string is title cased.

s.istitle() returns True if s is nonempty, the first alphabetic character of each word is uppercase, and all other alphabetic characters in each word are lowercase. It returns False otherwise:

>>> 'This Is A Title'.istitle()
True
>>> 'This is a title'.istitle()
False
>>> 'Give Me The #$#@ Ball!'.istitle()
True

Note: Here is how the Python documentation describes .istitle(), in case you find this more intuitive: “Uppercase characters may only follow uncased characters and lowercase characters only cased ones.”

s.isupper()

Determines whether the target string’s alphabetic characters are uppercase.

s.isupper() returns True if s is nonempty and all the alphabetic characters it contains are uppercase, and False otherwise. Non-alphabetic characters are ignored:

>>> 'ABC'.isupper()
True
>>> 'ABC1$D'.isupper()
True
>>> 'Abc1$D'.isupper()
False

String Formatting

Methods in this group modify or enhance the format of a string.

s.center(<width>[, <fill>])

Centers a string in a field.

s.center(<width>) returns a string consisting of s centered in a field of width <width>. By default, padding consists of the ASCII space character:

>>> 'foo'.center(10)
'   foo    '

If the optional <fill> argument is specified, it is used as the padding character:

>>> 'bar'.center(10, '-')
'---bar----'

If s is already at least as long as <width>, it is returned unchanged:

>>> 'foo'.center(2)
'foo'

s.expandtabs(tabsize=8)

Expands tabs in a string.

s.expandtabs() replaces each tab character ('\t') with spaces. By default, spaces are filled in assuming a tab stop at every eighth column:

>>> 'a\tb\tc'.expandtabs()
'a       b       c'
>>> 'aaa\tbbb\tc'.expandtabs()
'aaa     bbb     c'

tabsize is an optional keyword parameter specifying alternate tab stop columns:

>>> 'a\tb\tc'.expandtabs(4)
'a   b   c'
>>> 'aaa\tbbb\tc'.expandtabs(tabsize=4)
'aaa bbb c'

s.ljust(<width>[, <fill>])

Left-justifies a string in field.

s.ljust(<width>) returns a string consisting of s left-justified in a field of width <width>. By default, padding consists of the ASCII space character:

>>> 'foo'.ljust(10)
'foo       '

If the optional <fill> argument is specified, it is used as the padding character:

>>> 'foo'.ljust(10, '-')
'foo-------'

If s is already at least as long as <width>, it is returned unchanged:

>>> 'foo'.ljust(2)
'foo'

s.lstrip([<chars>])

Trims leading characters from a string.

s.lstrip() returns a copy of s with any whitespace characters removed from the left end:

>>> '   foo bar baz   '.lstrip()
'foo bar baz   '
>>> '\t\nfoo\t\nbar\t\nbaz'.lstrip()
'foo\t\nbar\t\nbaz'

If the optional <chars> argument is specified, it is a string that specifies the set of characters to be removed:

>>> 'http://www.realpython.com'.lstrip('/:pth')
'www.realpython.com'

s.replace(<old>, <new>[, <count>])

Replaces occurrences of a substring within a string.

s.replace(<old>, <new>) returns a copy of s with all occurrences of substring <old> replaced by <new>:

>>> 'foo bar foo baz foo qux'.replace('foo', 'grault')
'grault bar grault baz grault qux'

If the optional <count> argument is specified, a maximum of <count> replacements are performed, starting at the left end of s:

>>> 'foo bar foo baz foo qux'.replace('foo', 'grault', 2)
'grault bar grault baz foo qux'

s.rjust(<width>[, <fill>])

Right-justifies a string in a field.

s.rjust(<width>) returns a string consisting of s right-justified in a field of width <width>. By default, padding consists of the ASCII space character:

>>> 'foo'.rjust(10)
'       foo'

If the optional <fill> argument is specified, it is used as the padding character:

>>> 'foo'.rjust(10, '-')
'-------foo'

If s is already at least as long as <width>, it is returned unchanged:

>>> 'foo'.rjust(2)
'foo'

s.rstrip([<chars>])

Trims trailing characters from a string.

s.rstrip() returns a copy of s with any whitespace characters removed from the right end:

>>> '   foo bar baz   '.rstrip()
'   foo bar baz'
>>> 'foo\t\nbar\t\nbaz\t\n'.rstrip()
'foo\t\nbar\t\nbaz'

If the optional <chars> argument is specified, it is a string that specifies the set of characters to be removed:

>>> 'foo.$$$;'.rstrip(';$.')
'foo'

s.strip([<chars>])

Strips characters from the left and right ends of a string.

s.strip() is essentially equivalent to invoking s.lstrip() and s.rstrip() in succession. Without the <chars> argument, it removes leading and trailing whitespace:

>>> s = '   foo bar baz\t\t\t'
>>> s = s.lstrip()
>>> s = s.rstrip()
>>> s
'foo bar baz'

As with .lstrip() and .rstrip(), the optional <chars> argument specifies the set of characters to be removed:

>>> 'www.realpython.com'.strip('w.moc')
'realpython'

Note: When the return value of a string method is another string, as is often the case, methods can be invoked in succession by chaining the calls:

>>> '   foo bar baz\t\t\t'.lstrip().rstrip()
'foo bar baz'
>>> '   foo bar baz\t\t\t'.strip()
'foo bar baz'

>>> 'www.realpython.com'.lstrip('w.moc').rstrip('w.moc')
'realpython'
>>> 'www.realpython.com'.strip('w.moc')
'realpython'

s.zfill(<width>)

Pads a string on the left with zeros.

s.zfill(<width>) returns a copy of s left-padded with '0' characters to the specified <width>:

>>> '42'.zfill(5)
'00042'

If s contains a leading sign, it remains at the left edge of the result string after zeros are inserted:

>>> '+42'.zfill(8)
'+0000042'
>>> '-42'.zfill(8)
'-0000042'

If s is already at least as long as <width>, it is returned unchanged:

>>> '-42'.zfill(3)
'-42'

.zfill() is most useful for string representations of numbers, but Python will still happily zero-pad a string that isn’t:

>>> 'foo'.zfill(6)
'000foo'

Converting Between Strings and Lists

Methods in this group convert between a string and some composite data type by either pasting objects together to make a string, or by breaking a string up into pieces.

These methods operate on or return iterables, the general Python term for a sequential collection of objects. You will explore the inner workings of iterables in much more detail in the upcoming tutorial on definite iteration.

Many of these methods return either a list or a tuple. These are two similar composite data types that are prototypical examples of iterables in Python. They are covered in the next tutorial, so you’re about to learn about them soon! Until then, simply think of them as sequences of values. A list is enclosed in square brackets ([]), and a tuple is enclosed in parentheses (()).

With that introduction, let’s take a look at this last group of string methods.

s.join(<iterable>)

Concatenates strings from an iterable.

s.join(<iterable>) returns the string that results from concatenating the objects in <iterable> separated by s.

Note that .join() is invoked on s, the separator string. <iterable> must be a sequence of string objects as well.

Some sample code should help clarify. In the following example, the separator s is the string ', ', and <iterable> is a list of string values:

>>> ', '.join(['foo', 'bar', 'baz', 'qux'])
'foo, bar, baz, qux'

The result is a single string consisting of the list objects separated by commas.

In the next example, <iterable> is specified as a single string value. When a string value is used as an iterable, it is interpreted as a list of the string’s individual characters:

>>> list('corge')
['c', 'o', 'r', 'g', 'e']

>>> ':'.join('corge')
'c:o:r:g:e'

Thus, the result of ':'.join('corge') is a string consisting of each character in 'corge' separated by ':'.

This example fails because one of the objects in <iterable> is not a string:

>>> '---'.join(['foo', 23, 'bar'])
Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    '---'.join(['foo', 23, 'bar'])
TypeError: sequence item 1: expected str instance, int found

That can be remedied, though:

>>> '---'.join(['foo', str(23), 'bar'])
'foo---23---bar'

As you will soon see, many composite objects in Python can be construed as iterables, and .join() is especially useful for creating strings from them.

s.partition(<sep>)

Divides a string based on a separator.

s.partition(<sep>) splits s at the first occurrence of string <sep>. The return value is a three-part tuple consisting of:

The portion of s preceding <sep>
<sep> itself
The portion of s following <sep>

Here are a couple examples of .partition() in action:

>>> 'foo.bar'.partition('.')
('foo', '.', 'bar')
>>> 'foo@@bar@@baz'.partition('@@')
('foo', '@@', 'bar@@baz')

If <sep> is not found in s, the returned tuple contains s followed by two empty strings:

>>> 'foo.bar'.partition('@@')
('foo.bar', '', '')

Remember: Lists and tuples are covered in the next tutorial.

s.rpartition(<sep>)

Divides a string based on a separator.

s.rpartition(<sep>) functions exactly like s.partition(<sep>), except that s is split at the last occurrence of <sep> instead of the first occurrence:

>>> 'foo@@bar@@baz'.partition('@@')
('foo', '@@', 'bar@@baz')

>>> 'foo@@bar@@baz'.rpartition('@@')
('foo@@bar', '@@', 'baz')

s.rsplit(sep=None, maxsplit=-1)

Splits a string into a list of substrings.

Without arguments, s.rsplit() splits s into substrings delimited by any sequence of whitespace and returns the substrings as a list:

>>> 'foo bar baz qux'.rsplit()
['foo', 'bar', 'baz', 'qux']
>>> 'foo\n\tbar   baz\r\fqux'.rsplit()
['foo', 'bar', 'baz', 'qux']

If <sep> is specified, it is used as the delimiter for splitting:

>>> 'foo.bar.baz.qux'.rsplit(sep='.')
['foo', 'bar', 'baz', 'qux']

(If <sep> is specified with a value of None, the string is split delimited by whitespace, just as though <sep> had not been specified at all.)

When <sep> is explicitly given as a delimiter, consecutive delimiters in s are assumed to delimit empty strings, which will be returned:

>>> 'foo...bar'.rsplit(sep='.')
['foo', '', '', 'bar']

This is not the case when <sep> is omitted, however. In that case, consecutive whitespace characters are combined into a single delimiter, and the resulting list will never contain empty strings:

>>> 'foo\t\t\tbar'.rsplit()
['foo', 'bar']

If the optional keyword parameter <maxsplit> is specified, a maximum of that many splits are performed, starting from the right end of s:

>>> 'www.realpython.com'.rsplit(sep='.', maxsplit=1)
['www.realpython', 'com']

The default value for <maxsplit> is -1, which means all possible splits should be performed—the same as if <maxsplit> is omitted entirely:

>>> 'www.realpython.com'.rsplit(sep='.', maxsplit=-1)
['www', 'realpython', 'com']
>>> 'www.realpython.com'.rsplit(sep='.')
['www', 'realpython', 'com']

s.split(sep=None, maxsplit=-1)

Splits a string into a list of substrings.

s.split() behaves exactly like s.rsplit(), except that if <maxsplit> is specified, splits are counted from the left end of s rather than the right end:

>>> 'www.realpython.com'.split('.', maxsplit=1)
['www', 'realpython.com']
>>> 'www.realpython.com'.rsplit('.', maxsplit=1)
['www.realpython', 'com']

If <maxsplit> is not specified, .split() and .rsplit() are indistinguishable.

s.splitlines([<keepends>])

Breaks a string at line boundaries.

s.splitlines() splits s up into lines and returns them in a list. Any of the following characters or character sequences is considered to constitute a line boundary:

Escape Sequence	Character
`\n`	Newline
`\r`	Carriage Return
`\r\n`	Carriage Return + Line Feed
`\v` or `\x0b`	Line Tabulation
`\f` or `\x0c`	Form Feed
`\x1c`	File Separator
`\x1d`	Group Separator
`\x1e`	Record Separator
`\x85`	Next Line (C1 Control Code)
`\u2028`	Unicode Line Separator
`\u2029`	Unicode Paragraph Separator

Here is an example using several different line separators:

>>> 'foo\nbar\r\nbaz\fqux\u2028quux'.splitlines()
['foo', 'bar', 'baz', 'qux', 'quux']

If consecutive line boundary characters are present in the string, they are assumed to delimit blank lines, which will appear in the result list:

>>> 'foo\f\f\fbar'.splitlines()
['foo', '', '', 'bar']

If the optional <keepends> argument is specified and is truthy, then the lines boundaries are retained in the result strings:

>>> 'foo\nbar\nbaz\nqux'.splitlines(True)
['foo\n', 'bar\n', 'baz\n', 'qux']
>>> 'foo\nbar\nbaz\nqux'.splitlines(1)
['foo\n', 'bar\n', 'baz\n', 'qux']

`bytes` Objects

The bytes object is one of the core built-in types for manipulating binary data. A bytes object is an immutable sequence of single byte values. Each element in a bytes object is a small integer in the range 0 to 255.

Defining a Literal `bytes` Object

A bytes literal is defined in the same way as a string literal with the addition of a 'b' prefix:

>>> b = b'foo bar baz'
>>> b
b'foo bar baz'
>>> type(b)
<class 'bytes'>

As with strings, you can use any of the single, double, or triple quoting mechanisms:

>>> b'Contains embedded "double" quotes'
b'Contains embedded "double" quotes'

>>> b"Contains embedded 'single' quotes"
b"Contains embedded 'single' quotes"

>>> b'''Contains embedded "double" and 'single' quotes'''
b'Contains embedded "double" and \'single\' quotes'

>>> b"""Contains embedded "double" and 'single' quotes"""
b'Contains embedded "double" and \'single\' quotes'

Only ASCII characters are allowed in a bytes literal. Any character value greater than 127 must be specified using an appropriate escape sequence:

>>> b = b'foo\xddbar'
>>> b
b'foo\xddbar'
>>> b[3]
221
>>> int(0xdd)
221

The 'r' prefix may be used on a bytes literal to disable processing of escape sequences, as with strings:

>>> b = rb'foo\xddbar'
>>> b
b'foo\\xddbar'
>>> b[3]
92
>>> chr(92)
'\\'

Defining a `bytes` Object With the Built-in `bytes()` Function

The bytes() function also creates a bytes object. What sort of bytes object gets returned depends on the argument(s) passed to the function. The possible forms are shown below.

bytes(<s>, <encoding>)

Creates a bytes object from a string.

bytes(<s>, <encoding>) converts string <s> to a bytes object, using str.encode() according to the specified <encoding>:

>>> b = bytes('foo.bar', 'utf8')
>>> b
b'foo.bar'
>>> type(b)
<class 'bytes'>

Technical Note: In this form of the bytes() function, the <encoding> argument is required. “Encoding” refers to the manner in which characters are translated to integer values. A value of "utf8" indicates Unicode Transformation Format UTF-8, which is an encoding that can handle every possible Unicode character. UTF-8 can also be indicated by specifying "UTF8", "utf-8", or "UTF-8" for <encoding>.

See the Unicode documentation for more information. As long as you are dealing with common Latin-based characters, UTF-8 will serve you fine.

bytes(<size>)

Creates a bytes object consisting of null (0x00) bytes.

bytes(<size>) defines a bytes object of the specified <size>, which must be a positive integer. The resulting bytes object is initialized to null (0x00) bytes:

>>> b = bytes(8)
>>> b
b'\x00\x00\x00\x00\x00\x00\x00\x00'
>>> type(b)
<class 'bytes'>

bytes(<iterable>)

Creates a bytes object from an iterable.

bytes(<iterable>) defines a bytes object from the sequence of integers generated by <iterable>. <iterable> must be an iterable that generates a sequence of integers n in the range 0 ≤ n ≤ 255:

>>> b = bytes([100, 102, 104, 106, 108])
>>> b
b'dfhjl'
>>> type(b)
<class 'bytes'>
>>> b[2]
104

Operations on `bytes` Objects

Like strings, bytes objects support the common sequence operations:

The in and not in operators:

>>> b = b'abcde'

>>> b'cd' in b
True
>>> b'foo' not in b
True

The concatenation (+) and replication (*) operators:

>>> b = b'abcde'

>>> b + b'fghi'
b'abcdefghi'
>>> b * 3
b'abcdeabcdeabcde'

Indexing and slicing:

>>> b = b'abcde'

>>> b[2]
99
>>> b[1:3]
b'bc'

Built-in functions:

>>> len(b)
5
>>> min(b)
97
>>> max(b)
101

Many of the methods defined for string objects are valid for bytes objects as well:

>>> b = b'foo,bar,foo,baz,foo,qux'

>>> b.count(b'foo')
3

>>> b.endswith(b'qux')
True

>>> b.find(b'baz')
12

>>> b.split(sep=b',')
[b'foo', b'bar', b'foo', b'baz', b'foo', b'qux']

>>> b.center(30, b'-')
b'---foo,bar,foo,baz,foo,qux----'

Notice, however, that when these operators and methods are invoked on a bytes object, the operand and arguments must be bytes objects as well:

>>> b = b'foo.bar'

>>> b + '.baz'
Traceback (most recent call last):
  File "<pyshell#72>", line 1, in <module>
    b + '.baz'
TypeError: can't concat bytes to str
>>> b + b'.baz'
b'foo.bar.baz'

>>> b.split(sep='.')
Traceback (most recent call last):
  File "<pyshell#74>", line 1, in <module>
    b.split(sep='.')
TypeError: a bytes-like object is required, not 'str'
>>> b.split(sep=b'.')
[b'foo', b'bar']

Although a bytes object definition and representation is based on ASCII text, it actually behaves like an immutable sequence of small integers in the range 0 to 255, inclusive. That is why a single element from a bytes object is displayed as an integer:

>>> b = b'foo\xddbar'
>>> b[3]
221
>>> hex(b[3])
'0xdd'
>>> min(b)
97
>>> max(b)
221

A slice is displayed as a bytes object though, even if it is only one byte long:

>>> b[2:3]
b'c'

You can convert a bytes object into a list of integers with the built-in list() function:

>>> list(b)
[97, 98, 99, 100, 101]

Hexadecimal numbers are often used to specify binary data because two hexadecimal digits correspond directly to a single byte. The bytes class supports two additional methods that facilitate conversion to and from a string of hexadecimal digits.

bytes.fromhex(<s>)

Returns a bytes object constructed from a string of hexadecimal values.

bytes.fromhex(<s>) returns the bytes object that results from converting each pair of hexadecimal digits in <s> to the corresponding byte value. The hexadecimal digit pairs in <s> may optionally be separated by whitespace, which is ignored:

>>> b = bytes.fromhex(' aa 68 4682cc ')
>>> b
b'\xaahF\x82\xcc'
>>> list(b)
[170, 104, 70, 130, 204]

Note: This method is a class method, not an object method. It is bound to the bytes class, not a bytes object. You will delve much more into the distinction between classes, objects, and their respective methods in the upcoming tutorials on object-oriented programming. For now, just observe that this method is invoked on the bytes class, not on object b.

b.hex()

Returns a string of hexadecimal value from a bytes object.

b.hex() returns the result of converting bytes object b into a string of hexadecimal digit pairs. That is, it does the reverse of .fromhex():

>>> b = bytes.fromhex(' aa 68 4682cc ')
>>> b
b'\xaahF\x82\xcc'

>>> b.hex()
'aa684682cc'
>>> type(b.hex())
<class 'str'>

Note: As opposed to .fromhex(), .hex() is an object method, not a class method. Thus, it is invoked on an object of the bytes class, not on the class itself.

`bytearray` Objects

Python supports another binary sequence type called the bytearray. bytearray objects are very like bytes objects, despite some differences:

There is no dedicated syntax built into Python for defining a bytearray literal, like the 'b' prefix that may be used to define a bytes object. A bytearray object is always created using the bytearray() built-in function:

>>> ba = bytearray('foo.bar.baz', 'UTF-8')
>>> ba
bytearray(b'foo.bar.baz')

>>> bytearray(6)
bytearray(b'\x00\x00\x00\x00\x00\x00')

>>> bytearray([100, 102, 104, 106, 108])
bytearray(b'dfhjl')

bytearray objects are mutable. You can modify the contents of a bytearray object using indexing and slicing:

>>> ba = bytearray('foo.bar.baz', 'UTF-8')
>>> ba
bytearray(b'foo.bar.baz')

>>> ba[5] = 0xee
>>> ba
bytearray(b'foo.b\xeer.baz')

>>> ba[8:11] = b'qux'
>>> ba
bytearray(b'foo.b\xeer.qux')

A bytearray object may be constructed directly from a bytes object as well:

>>> ba = bytearray(b'foo')
>>> ba
bytearray(b'foo')

Conclusion

This tutorial provided an in-depth look at the many different mechanisms Python provides for string handling, including string operators, built-in functions, indexing, slicing, and built-in methods. You also were introduced to the bytes and bytearray types.

These types are the first types you have examined that are composite—built from a collection of smaller parts. Python provides several composite built-in types. In the next tutorial, you will explore two of the most frequently used: lists and tuples.

« Operators and Expressions in Python

Strings in Python

Lists and Tuples in Python »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python String Formatting Best Practices

2018-07-04T14:00:00+00:00

Remember the Zen of Python and how there should be “one obvious way to do something in Python”? You might scratch your head when you find out that there are four major ways to do string formatting in Python.

In this tutorial, you’ll learn the four main approaches to string formatting in Python, as well as their strengths and weaknesses. You’ll also get a simple rule of thumb for how to pick the best general purpose string formatting approach in your own programs.

Let’s jump right in, as we’ve got a lot to cover. In order to have a simple toy example for experimentation, let’s assume you’ve got the following variables (or constants, really) to work with:

>>> errno = 50159747054
>>> name = 'Bob'

Based on these variables, you’d like to generate an output string containing a simple error message:

'Hey Bob, there is a 0xbadc0ffee error!'

That error could really spoil a dev’s Monday morning… But we’re here to discuss string formatting. So let’s get to work.

#1 “Old Style” String Formatting (% operator)

Strings in Python have a unique built-in operation that can be accessed with the % operator. This lets you do simple positional formatting very easily. If you’ve ever worked with a printf-style function in C, you’ll recognize how this works instantly. Here’s a simple example:

>>> 'Hello, %s' % name
"Hello, Bob"

I’m using the %s format specifier here to tell Python where to substitute the value of name, represented as a string.

There are other format specifiers available that let you control the output format. For example, it’s possible to convert numbers to hexadecimal notation or add whitespace padding to generate nicely formatted tables and reports. (See Python Docs: “printf-style String Formatting”.)

Here, you can use the %x format specifier to convert an int value to a string and to represent it as a hexadecimal number:

>>> '%x' % errno
'badc0ffee'

The “old style” string formatting syntax changes slightly if you want to make multiple substitutions in a single string. Because the % operator takes only one argument, you need to wrap the right-hand side in a tuple, like so:

>>> 'Hey %s, there is a 0x%x error!' % (name, errno)
'Hey Bob, there is a 0xbadc0ffee error!'

It’s also possible to refer to variable substitutions by name in your format string, if you pass a mapping to the % operator:

>>> 'Hey %(name)s, there is a 0x%(errno)x error!' % {
...     "name": name, "errno": errno }
'Hey Bob, there is a 0xbadc0ffee error!'

This makes your format strings easier to maintain and easier to modify in the future. You don’t have to worry about making sure the order you’re passing in the values matches up with the order in which the values are referenced in the format string. Of course, the downside is that this technique requires a little more typing.

I’m sure you’ve been wondering why this printf-style formatting is called “old style” string formatting. It was technically superseded by “new style” formatting in Python 3, which we’re going to talk about next.

#2 “New Style” String Formatting (`str.format`)

Python 3 introduced a new way to do string formatting that was also later back-ported to Python 2.7. This “new style” string formatting gets rid of the %-operator special syntax and makes the syntax for string formatting more regular. Formatting is now handled by calling .format() on a string object.

You can use format() to do simple positional formatting, just like you could with “old style” formatting:

>>> 'Hello, {}'.format(name)
'Hello, Bob'

Or, you can refer to your variable substitutions by name and use them in any order you want. This is quite a powerful feature as it allows for re-arranging the order of display without changing the arguments passed to format():

>>> 'Hey {name}, there is a 0x{errno:x} error!'.format(
...     name=name, errno=errno)
'Hey Bob, there is a 0xbadc0ffee error!'

This also shows that the syntax to format an int variable as a hexadecimal string has changed. Now you need to pass a format spec by adding a :x suffix. The format string syntax has become more powerful without complicating the simpler use cases. It pays off to read up on this string formatting mini-language in the Python documentation.

In Python 3, this “new style” string formatting is to be preferred over %-style formatting. While “old style” formatting has been de-emphasized, it has not been deprecated. It is still supported in the latest versions of Python. According to this discussion on the Python dev email list and this issue on the Python dev bug tracker, %-formatting is going to stick around for a long time to come.

Still, the official Python 3 documentation doesn’t exactly recommend “old style” formatting or speak too fondly of it:

“The formatting operations described here exhibit a variety of quirks that lead to a number of common errors (such as failing to display tuples and dictionaries correctly). Using the newer formatted string literals or the str.format() interface helps avoid these errors. These alternatives also provide more powerful, flexible and extensible approaches to formatting text.” (Source)

This is why I’d personally try to stick with str.format for new code moving forward. Starting with Python 3.6, there’s yet another way to format your strings. I’ll tell you all about it in the next section.

#3 String Interpolation / f-Strings (Python 3.6+)

Python 3.6 added a new string formatting approach called formatted string literals or “f-strings”. This new way of formatting strings lets you use embedded Python expressions inside string constants. Here’s a simple example to give you a feel for the feature:

>>> f'Hello, {name}!'
'Hello, Bob!'

As you can see, this prefixes the string constant with the letter “f“—hence the name “f-strings.” This new formatting syntax is powerful. Because you can embed arbitrary Python expressions, you can even do inline arithmetic with it. Check out this example:

>>> a = 5
>>> b = 10
>>> f'Five plus ten is {a + b} and not {2 * (a + b)}.'
'Five plus ten is 15 and not 30.'

Formatted string literals are a Python parser feature that converts f-strings into a series of string constants and expressions. They then get joined up to build the final string.

Imagine you had the following greet() function that contains an f-string:

>>> def greet(name, question):
...     return f"Hello, {name}! How's it {question}?"
...
>>> greet('Bob', 'going')
"Hello, Bob! How's it going?"

When you disassemble the function and inspect what’s going on behind the scenes, you’ll see that the f-string in the function gets transformed into something similar to the following:

>>> def greet(name, question):
...    return "Hello, " + name + "! How's it " + question + "?"

The real implementation is slightly faster than that because it uses the BUILD_STRING opcode as an optimization. But functionally they’re the same:

>>> import dis
>>> dis.dis(greet)
  2           0 LOAD_CONST               1 ('Hello, ')
              2 LOAD_FAST                0 (name)
              4 FORMAT_VALUE             0
              6 LOAD_CONST               2 ("! How's it ")
              8 LOAD_FAST                1 (question)
             10 FORMAT_VALUE             0
             12 LOAD_CONST               3 ('?')
             14 BUILD_STRING             5
             16 RETURN_VALUE

String literals also support the existing format string syntax of the str.format() method. That allows you to solve the same formatting problems we’ve discussed in the previous two sections:

>>> f"Hey {name}, there's a {errno:#x} error!"
"Hey Bob, there's a 0xbadc0ffee error!"

Python’s new formatted string literals are similar to JavaScript’s Template Literals added in ES2015. I think they’re quite a nice addition to Python, and I’ve already started using them in my day to day (Python 3) work. You can learn more about formatted string literals in our in-depth Python f-strings tutorial.

#4 Template Strings (Standard Library)

Here’s one more tool for string formatting in Python: template strings. It’s a simpler and less powerful mechanism, but in some cases this might be exactly what you’re looking for.

Let’s take a look at a simple greeting example:

>>> from string import Template
>>> t = Template('Hey, $name!')
>>> t.substitute(name=name)
'Hey, Bob!'

You see here that we need to import the Template class from Python’s built-in string module. Template strings are not a core language feature but they’re supplied by the string module in the standard library.

Another difference is that template strings don’t allow format specifiers. So in order to get the previous error string example to work, you’ll need to manually transform the int error number into a hex-string:

>>> templ_string = 'Hey $name, there is a $error error!'
>>> Template(templ_string).substitute(
...     name=name, error=hex(errno))
'Hey Bob, there is a 0xbadc0ffee error!'

That worked great.

So when should you use template strings in your Python programs? In my opinion, the best time to use template strings is when you’re handling formatted strings generated by users of your program. Due to their reduced complexity, template strings are a safer choice.

The more complex formatting mini-languages of the other string formatting techniques might introduce security vulnerabilities to your programs. For example, it’s possible for format strings to access arbitrary variables in your program.

That means, if a malicious user can supply a format string, they can potentially leak secret keys and other sensitive information! Here’s a simple proof of concept of how this attack might be used against your code:

>>> # This is our super secret key:
>>> SECRET = 'this-is-a-secret'

>>> class Error:
...      def __init__(self):
...          pass

>>> # A malicious user can craft a format string that
>>> # can read data from the global namespace:
>>> user_input = '{error.__init__.__globals__[SECRET]}'

>>> # This allows them to exfiltrate sensitive information,
>>> # like the secret key:
>>> err = Error()
>>> user_input.format(error=err)
'this-is-a-secret'

See how a hypothetical attacker was able to extract our secret string by accessing the __globals__ dictionary from a malicious format string? Scary, huh? Template strings close this attack vector. This makes them a safer choice if you’re handling format strings generated from user input:

>>> user_input = '${error.__init__.__globals__[SECRET]}'
>>> Template(user_input).substitute(error=err)
ValueError:
"Invalid placeholder in string: line 1, col 1"

Which String Formatting Method Should You Use?

I totally get that having so much choice for how to format your strings in Python can feel very confusing. This is an excellent cue to bust out this handy flowchart infographic I’ve put together for you:

Python String Formatting Rule of Thumb (Click to Tweet)

This flowchart is based on the rule of thumb that I apply when I’m writing Python:

Python String Formatting Rule of Thumb: If your format strings are user-supplied, use Template Strings (#4) to avoid security issues. Otherwise, use Literal String Interpolation/f-Strings (#3) if you’re on Python 3.6+, and “New Style” str.format (#2) if you’re not.

Key Takeaways

Perhaps surprisingly, there’s more than one way to handle string formatting in Python.
Each method has its individual pros and cons. Your use case will influence which method you should use.
If you’re having trouble deciding which string formatting method to use, try our Python String Formatting Rule of Thumb.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Histogram Plotting: NumPy, Matplotlib, Pandas & Seaborn

2018-07-02T14:00:00+00:00

In this tutorial, you’ll be equipped to make production-quality, presentation-ready Python histogram plots with a range of choices and features.

If you have introductory to intermediate knowledge in Python and statistics, you can use this article as a one-stop shop for building and plotting histograms in Python using libraries from its scientific stack, including NumPy, Matplotlib, Pandas, and Seaborn.

A histogram is a great tool for quickly assessing a probability distribution that is intuitively understood by almost any audience. Python offers a handful of different options for building and plotting histograms. Most people know a histogram by its graphical representation, which is similar to a bar graph:

This article will guide you through creating plots like the one above as well as more complex ones. Here’s what you’ll cover:

Building histograms in pure Python, without use of third party libraries
Constructing histograms with NumPy to summarize the underlying data
Plotting the resulting histogram with Matplotlib, Pandas, and Seaborn

Free Bonus: Short on time? Click here to get access to a free two-page Python histograms cheat sheet that summarizes the techniques explained in this tutorial.

Histograms in Pure Python

When you are preparing to plot a histogram, it is simplest to not think in terms of bins but rather to report how many times each value appears (a frequency table). A Python dictionary is well-suited for this task:

>>> # Need not be sorted, necessarily
>>> a = (0, 1, 1, 1, 2, 3, 7, 7, 23)

>>> def count_elements(seq) -> dict:
...     """Tally elements from `seq`."""
...     hist = {}
...     for i in seq:
...         hist[i] = hist.get(i, 0) + 1
...     return hist

>>> counted = count_elements(a)
>>> counted
{0: 1, 1: 3, 2: 1, 3: 1, 7: 2, 23: 1}

count_elements() returns a dictionary with unique elements from the sequence as keys and their frequencies (counts) as values. Within the loop over seq, hist[i] = hist.get(i, 0) + 1 says, “for each element of the sequence, increment its corresponding value in hist by 1.”

In fact, this is precisely what is done by the collections.Counter class from Python’s standard library, which subclasses a Python dictionary and overrides its .update() method:

>>> from collections import Counter

>>> recounted = Counter(a)
>>> recounted
Counter({0: 1, 1: 3, 3: 1, 2: 1, 7: 2, 23: 1})

You can confirm that your handmade function does virtually the same thing as collections.Counter by testing for equality between the two:

>>> recounted.items() == counted.items()
True

Technical Detail: The mapping from count_elements() above defaults to a more highly optimized C function if it is available. Within the Python function count_elements(), one micro-optimization you could make is to declare get = hist.get before the for-loop. This would bind a method to a variable for faster calls within the loop.

It can be helpful to build simplified functions from scratch as a first step to understanding more complex ones. Let’s further reinvent the wheel a bit with an ASCII histogram that takes advantage of Python’s output formatting:

def ascii_histogram(seq) -> None:
    """A horizontal frequency-table/histogram plot."""
    counted = count_elements(seq)
    for k in sorted(counted):
        print('{0:5d} {1}'.format(k, '+' * counted[k]))

This function creates a sorted frequency plot where counts are represented as tallies of plus (+) symbols. Calling sorted() on a dictionary returns a sorted list of its keys, and then you access the corresponding value for each with counted[k]. To see this in action, you can create a slightly larger dataset with Python’s random module:

>>> # No NumPy ... yet
>>> import random
>>> random.seed(1)

>>> vals = [1, 3, 4, 6, 8, 9, 10]
>>> # Each number in `vals` will occur between 5 and 15 times.
>>> freq = (random.randint(5, 15) for _ in vals)

>>> data = []
>>> for f, v in zip(freq, vals):
...     data.extend([v] * f)

>>> ascii_histogram(data)
    1 +++++++
    3 ++++++++++++++
    4 ++++++
    6 +++++++++
    8 ++++++
    9 ++++++++++++
   10 ++++++++++++

Here, you’re simulating plucking from vals with frequencies given by freq (a generator expression). The resulting sample data repeats each value from vals a certain number of times between 5 and 15.

Note: random.seed() is use to seed, or initialize, the underlying pseudorandom number generator (PRNG) used by random. It may sound like an oxymoron, but this is a way of making random data reproducible and deterministic. That is, if you copy the code here as is, you should get exactly the same histogram because the first call to random.randint() after seeding the generator will produce identical “random” data using the Mersenne Twister.

Building Up From the Base: Histogram Calculations in NumPy

Thus far, you have been working with what could best be called “frequency tables.” But mathematically, a histogram is a mapping of bins (intervals) to frequencies. More technically, it can be used to approximate the probability density function (PDF) of the underlying variable.

Moving on from the “frequency table” above, a true histogram first “bins” the range of values and then counts the number of values that fall into each bin. This is what NumPy’s histogram() function does, and it is the basis for other functions you’ll see here later in Python libraries such as Matplotlib and Pandas.

Consider a sample of floats drawn from the Laplace distribution. This distribution has fatter tails than a normal distribution and has two descriptive parameters (location and scale):

>>> import numpy as np
>>> # `numpy.random` uses its own PRNG.
>>> np.random.seed(444)
>>> np.set_printoptions(precision=3)

>>> d = np.random.laplace(loc=15, scale=3, size=500)
>>> d[:5]
array([18.406, 18.087, 16.004, 16.221,  7.358])

In this case, you’re working with a continuous distribution, and it wouldn’t be very helpful to tally each float independently, down to the umpteenth decimal place. Instead, you can bin or “bucket” the data and count the observations that fall into each bin. The histogram is the resulting count of values within each bin:

>>> hist, bin_edges = np.histogram(d)

>>> hist
array([ 1,  0,  3,  4,  4, 10, 13,  9,  2,  4])

>>> bin_edges
array([ 3.217,  5.199,  7.181,  9.163, 11.145, 13.127, 15.109, 17.091,
       19.073, 21.055, 23.037])

This result may not be immediately intuitive. np.histogram() by default uses 10 equally sized bins and returns a tuple of the frequency counts and corresponding bin edges. They are edges in the sense that there will be one more bin edge than there are members of the histogram:

>>> hist.size, bin_edges.size
(10, 11)

Technical Detail: All but the last (rightmost) bin is half-open. That is, all bins but the last are [inclusive, exclusive), and the final bin is [inclusive, inclusive].

A very condensed breakdown of how the bins are constructed by NumPy looks like this:

>>> # The leftmost and rightmost bin edges
>>> first_edge, last_edge = a.min(), a.max()

>>> n_equal_bins = 10  # NumPy's default
>>> bin_edges = np.linspace(start=first_edge, stop=last_edge,
...                         num=n_equal_bins + 1, endpoint=True)
...
>>> bin_edges
array([ 0. ,  2.3,  4.6,  6.9,  9.2, 11.5, 13.8, 16.1, 18.4, 20.7, 23. ])

The case above makes a lot of sense: 10 equally spaced bins over a peak-to-peak range of 23 means intervals of width 2.3.

From there, the function delegates to either np.bincount() or np.searchsorted(). bincount() itself can be used to effectively construct the “frequency table” that you started off with here, with the distinction that values with zero occurrences are included:

>>> bcounts = np.bincount(a)
>>> hist, _ = np.histogram(a, range=(0, a.max()), bins=a.max() + 1)

>>> np.array_equal(hist, bcounts)
True

>>> # Reproducing `collections.Counter`
>>> dict(zip(np.unique(a), bcounts[bcounts.nonzero()]))
{0: 1, 1: 3, 2: 1, 3: 1, 7: 2, 23: 1}

Note: hist here is really using bins of width 1.0 rather than “discrete” counts. Hence, this only works for counting integers, not floats such as [3.9, 4.1, 4.15].

Visualizing Histograms with Matplotlib and Pandas

Now that you’ve seen how to build a histogram in Python from the ground up, let’s see how other Python packages can do the job for you. Matplotlib provides the functionality to visualize Python histograms out of the box with a versatile wrapper around NumPy’s histogram():

import matplotlib.pyplot as plt

# An "interface" to matplotlib.axes.Axes.hist() method
n, bins, patches = plt.hist(x=d, bins='auto', color='#0504aa',
                            alpha=0.7, rwidth=0.85)
plt.grid(axis='y', alpha=0.75)
plt.xlabel('Value')
plt.ylabel('Frequency')
plt.title('My Very Own Histogram')
plt.text(23, 45, r'$\mu=15, b=3$')
maxfreq = n.max()
# Set a clean upper y-axis limit.
plt.ylim(ymax=np.ceil(maxfreq / 10) * 10 if maxfreq % 10 else maxfreq + 10)

As defined earlier, a plot of a histogram uses its bin edges on the x-axis and the corresponding frequencies on the y-axis. In the chart above, passing bins='auto' chooses between two algorithms to estimate the “ideal” number of bins. At a high level, the goal of the algorithm is to choose a bin width that generates the most faithful representation of the data. For more on this subject, which can get pretty technical, check out Choosing Histogram Bins from the Astropy docs.

Staying in Python’s scientific stack, Pandas’ Series.histogram() uses matplotlib.pyplot.hist() to draw a Matplotlib histogram of the input Series:

import pandas as pd

# Generate data on commute times.
size, scale = 1000, 10
commutes = pd.Series(np.random.gamma(scale, size=size) ** 1.5)

commutes.plot.hist(grid=True, bins=20, rwidth=0.9,
                   color='#607c8e')
plt.title('Commute Times for 1,000 Commuters')
plt.xlabel('Counts')
plt.ylabel('Commute Time')
plt.grid(axis='y', alpha=0.75)

pandas.DataFrame.histogram() is similar but produces a histogram for each column of data in the DataFrame.

Plotting a Kernel Density Estimate (KDE)

In this tutorial, you’ve been working with samples, statistically speaking. Whether the data is discrete or continuous, it’s assumed to be derived from a population that has a true, exact distribution described by just a few parameters.

A kernel density estimation (KDE) is a way to estimate the probability density function (PDF) of the random variable that “underlies” our sample. KDE is a means of data smoothing.

Sticking with the Pandas library, you can create and overlay density plots using plot.kde(), which is available for both Series and DataFrame objects. But first, let’s generate two distinct data samples for comparison:

>>> # Sample from two different normal distributions
>>> means = 10, 20
>>> stdevs = 4, 2
>>> dist = pd.DataFrame(
...     np.random.normal(loc=means, scale=stdevs, size=(1000, 2)),
...     columns=['a', 'b'])
>>> dist.agg(['min', 'max', 'mean', 'std']).round(decimals=2)
          a      b
min   -1.57  12.46
max   25.32  26.44
mean  10.12  19.94
std    3.94   1.94

Now, to plot each histogram on the same Matplotlib axes:

fig, ax = plt.subplots()
dist.plot.kde(ax=ax, legend=False, title='Histogram: A vs. B')
dist.plot.hist(density=True, ax=ax)
ax.set_ylabel('Probability')
ax.grid(axis='y')
ax.set_facecolor('#d8dcd6')

These methods leverage SciPy’s gaussian_kde(), which results in a smoother-looking PDF.

If you take a closer look at this function, you can see how well it approximates the “true” PDF for a relatively small sample of 1000 data points. Below, you can first build the “analytical” distribution with scipy.stats.norm(). This is a class instance that encapsulates the statistical standard normal distribution, its moments, and descriptive functions. Its PDF is “exact” in the sense that it is defined precisely as norm.pdf(x) = exp(-x**2/2) / sqrt(2*pi).

Building from there, you can take a random sample of 1000 datapoints from this distribution, then attempt to back into an estimation of the PDF with scipy.stats.gaussian_kde():

from scipy import stats

# An object representing the "frozen" analytical distribution
# Defaults to the standard normal distribution, N~(0, 1)
dist = stats.norm()

# Draw random samples from the population you built above.
# This is just a sample, so the mean and std. deviation should
# be close to (1, 0).
samp = dist.rvs(size=1000)

# `ppf()`: percent point function (inverse of cdf — percentiles).
x = np.linspace(start=stats.norm.ppf(0.01),
                stop=stats.norm.ppf(0.99), num=250)
gkde = stats.gaussian_kde(dataset=samp)

# `gkde.evaluate()` estimates the PDF itself.
fig, ax = plt.subplots()
ax.plot(x, dist.pdf(x), linestyle='solid', c='red', lw=3,
        alpha=0.8, label='Analytical (True) PDF')
ax.plot(x, gkde.evaluate(x), linestyle='dashed', c='black', lw=2,
        label='PDF Estimated via KDE')
ax.legend(loc='best', frameon=False)
ax.set_title('Analytical vs. Estimated PDF')
ax.set_ylabel('Probability')
ax.text(-2., 0.35, r'$f(x) = \frac{\exp(-x^2/2)}{\sqrt{2*\pi}}$',
        fontsize=12)

This is a bigger chunk of code, so let’s take a second to touch on a few key lines:

SciPy’s stats subpackage lets you create Python objects that represent analytical distributions that you can sample from to create actual data. So dist = stats.norm() represents a normal continuous random variable, and you generate random numbers from it with dist.rvs().
To evaluate both the analytical PDF and the Gaussian KDE, you need an array x of quantiles (standard deviations above/below the mean, for a normal distribution). stats.gaussian_kde() represents an estimated PDF that you need to evaluate on an array to produce something visually meaningful in this case.
The last line contains some LaTex, which integrates nicely with Matplotlib.

A Fancy Alternative with Seaborn

Let’s bring one more Python package into the mix. Seaborn has a displot() function that plots the histogram and KDE for a univariate distribution in one step. Using the NumPy array d from ealier:

import seaborn as sns

sns.set_style('darkgrid')
sns.distplot(d)

The call above produces a KDE. There is also optionality to fit a specific distribution to the data. This is different than a KDE and consists of parameter estimation for generic data and a specified distribution name:

sns.distplot(d, fit=stats.laplace, kde=False)

Again, note the slight difference. In the first case, you’re estimating some unknown PDF; in the second, you’re taking a known distribution and finding what parameters best describe it given the empirical data.

Other Tools in Pandas

In addition to its plotting tools, Pandas also offers a convenient .value_counts() method that computes a histogram of non-null values to a Pandas Series:

>>> import pandas as pd

>>> data = np.random.choice(np.arange(10), size=10000,
...                         p=np.linspace(1, 11, 10) / 60)
>>> s = pd.Series(data)

>>> s.value_counts()
9    1831
8    1624
7    1423
6    1323
5    1089
4     888
3     770
2     535
1     347
0     170
dtype: int64

>>> s.value_counts(normalize=True).head()
9    0.1831
8    0.1624
7    0.1423
6    0.1323
5    0.1089
dtype: float64

Elsewhere, pandas.cut() is a convenient way to bin values into arbitrary intervals. Let’s say you have some data on ages of individuals and want to bucket them sensibly:

>>> ages = pd.Series(
...     [1, 1, 3, 5, 8, 10, 12, 15, 18, 18, 19, 20, 25, 30, 40, 51, 52])
>>> bins = (0, 10, 13, 18, 21, np.inf)  # The edges
>>> labels = ('child', 'preteen', 'teen', 'military_age', 'adult')
>>> groups = pd.cut(ages, bins=bins, labels=labels)

>>> groups.value_counts()
child           6
adult           5
teen            3
military_age    2
preteen         1
dtype: int64

>>> pd.concat((ages, groups), axis=1).rename(columns={0: 'age', 1: 'group'})
    age         group
0     1         child
1     1         child
2     3         child
3     5         child
4     8         child
5    10         child
6    12       preteen
7    15          teen
8    18          teen
9    18          teen
10   19  military_age
11   20  military_age
12   25         adult
13   30         adult
14   40         adult
15   51         adult
16   52         adult

What’s nice is that both of these operations ultimately utilize Cython code that makes them competitive on speed while maintaining their flexibility.

Alright, So Which Should I Use?

At this point, you’ve seen more than a handful of functions and methods to choose from for plotting a Python histogram. How do they compare? In short, there is no “one-size-fits-all.” Here’s a recap of the functions and methods you’ve covered thus far, all of which relate to breaking down and representing distributions in Python:

You Have/Want To	Consider Using	Note(s)
Clean-cut integer data housed in a data structure such as a list, tuple, or set, and you want to create a Python histogram without importing any third party libraries.	`collections.counter()` from the Python standard library offers a fast and straightforward way to get frequency counts from a container of data.	This is a frequency table, so it doesn’t use the concept of binning as a “true” histogram does.
Large array of data, and you want to compute the “mathematical” histogram that represents bins and the corresponding frequencies.	NumPy’s `np.histogram()` and `np.bincount()` are useful for computing the histogram values numerically and the corresponding bin edges.	For more, check out `np.digitize()`.
Tabular data in Pandas’ `Series` or `DataFrame` object.	Pandas methods such as `Series.plot.hist()`, `DataFrame.plot.hist()`, `Series.value_counts()`, and `cut()`, as well as `Series.plot.kde()` and `DataFrame.plot.kde()`.	Check out the Pandas visualization docs for inspiration.
Create a highly customizable, fine-tuned plot from any data structure.	`pyplot.hist()` is a widely used histogram plotting function that uses `np.histogram()` and is the basis for Pandas’ plotting functions.	Matplotlib, and especially its object-oriented framework, is great for fine-tuning the details of a histogram. This interface can take a bit of time to master, but ultimately allows you to be very precise in how any visualization is laid out.
Pre-canned design and integration.	Seaborn’s `distplot()`, for combining a histogram and KDE plot or plotting distribution-fitting.	Essentially a “wrapper around a wrapper” that leverages a Matplotlib histogram internally, which in turn utilizes NumPy.

Free Bonus: Short on time? Click here to get access to a free two-page Python histograms cheat sheet that summarizes the techniques explained in this tutorial.

You can also find the code snippets from this article together in one script at the Real Python materials page.

With that, good luck creating histograms in the wild. Hopefully one of the tools above will suit your needs. Whatever you do, just don’t use a pie chart.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Cool New Features in Python 3.7

2018-06-27T14:00:00+00:00

Python 3.7 is officially released! This new Python version has been in development since September 2016, and now we all get to enjoy the results of the core developers’ hard work.

What does the new Python version bring? While the documentation gives a good overview of the new features, this article will take a deep dive into some of the biggest pieces of news. These include:

Easier access to debuggers through a new breakpoint() built-in
Simple class creation using data classes
Customized access to module attributes
Improved support for type hinting
Higher precision timing functions

More importantly, Python 3.7 is fast.

In the final sections of this article, you’ll read more about this speed, as well as some of the other cool features of Python 3.7. You will also get some advice on upgrading to the new version.

The `breakpoint()` Built-In

While we might strive to write perfect code, the simple truth is that we never do. Debugging is an important part of programming. Python 3.7 introduces the new built-in function breakpoint(). This does not really add any new functionality to Python, but it makes using debuggers more flexible and intuitive.

Assume that you have the following buggy code in the file bugs.py:

def divide(e, f):
    return f / e

a, b = 0, 1
print(divide(a, b))

Running the code causes a ZeroDivisionError inside the divide() function. Let’s say that you want to interrupt your code and drop into a debugger right at the top of divide(). You can do so by setting a so called “breakpoint” in your code:

def divide(e, f):
    # Insert breakpoint here
    return f / e

A breakpoint is a signal inside your code that execution should temporarily stop, so that you can look around at the current state of the program. How do you place the breakpoint? In Python 3.6 and below, you use this somewhat cryptic line:

def divide(e, f):
    import pdb; pdb.set_trace()
    return f / e

Here, pdb is the Python Debugger from the standard library. In Python 3.7, you can use the new breakpoint() function call as a shortcut instead:

def divide(e, f):
    breakpoint()
    return f / e

In the background, breakpoint() is first importing pdb and then calling pdb.set_trace() for you. The obvious benefits are that breakpoint() is easier to remember and that you only need to type 12 characters instead of 27. However, the real bonus of using breakpoint() is its customizability.

Run your bugs.py script with breakpoint():

$ python3.7 bugs.py 
> /home/gahjelle/bugs.py(3)divide()
-> return f / e
(Pdb)

The script will break when it reaches breakpoint() and drop you into a PDB debugging session. You can type c and hit Enter to continue the script. Refer to Nathan Jennings’ PDB guide if you want to learn more about PDB and debugging.

Now, say that you think you’ve fixed the bug. You would like to run the script again but without stopping in the debugger. You could, of course, comment out the breakpoint() line, but another option is to use the PYTHONBREAKPOINT environment variable. This variable controls the behavior of breakpoint(), and setting PYTHONBREAKPOINT=0 means that any call to breakpoint() is ignored:

$ PYTHONBREAKPOINT=0 python3.7 bugs.py
ZeroDivisionError: division by zero

Oops, it seems as if you haven’t fixed the bug after all…

Another option is to use PYTHONBREAKPOINT to specify a debugger other than PDB. For instance, to use PuDB (a visual debugger in the console) you can do:

$ PYTHONBREAKPOINT=pudb.set_trace python3.7 bugs.py

For this to work, you need to have pudb installed (pip install pudb). Python will take care of importing pudb for you though. This way you can also set your default debugger. Simply set the PYTHONBREAKPOINT environment variable to your preferred debugger. See this guide for instructions on how to set an environment variable on your system.

The new breakpoint() function does not only work with debuggers. One convenient option could be to simply start an interactive shell inside your code. For instance, to start an IPython session, you can use the following:

$ PYTHONBREAKPOINT=IPython.embed python3.7 bugs.py 
IPython 6.3.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: print(e / f)
0.0

You can also create your own function and have breakpoint() call that. The following code prints all variables in the local scope. Add it to a file called bp_utils.py:

from pprint import pprint
import sys

def print_locals():
    caller = sys._getframe(1)  # Caller is 1 frame up.
    pprint(caller.f_locals)

To use this function, set PYTHONBREAKPOINT as before, with the <module>.<function> notation:

$ PYTHONBREAKPOINT=bp_utils.print_locals python3.7 bugs.py 
{'e': 0, 'f': 1}
ZeroDivisionError: division by zero

Normally, breakpoint() will be used to call functions and methods that do not need arguments. However, it is possible to pass arguments as well. Change the line breakpoint() in bugs.py to:

breakpoint(e, f, end="<-END\n")

Note: The default PDB debugger will raise a TypeError at this line because pdb.set_trace() does not take any positional arguments.

Run this code with breakpoint() masquerading as the print() function to see a simple example of the arguments being passed through:

$ PYTHONBREAKPOINT=print python3.7 bugs.py 
0 1<-END
ZeroDivisionError: division by zero

See PEP 553 as well as the documentation for breakpoint() and sys.breakpointhook() for more information.

Data Classes

The new dataclasses module makes it more convenient to write your own classes, as special methods like .__init__(), .__repr__(), and .__eq__() are added automatically. Using the @dataclass decorator, you can write something like:

from dataclasses import dataclass, field

@dataclass(order=True)
class Country:
    name: str
    population: int
    area: float = field(repr=False, compare=False)
    coastline: float = 0

    def beach_per_person(self):
        """Meters of coastline per person"""
        return (self.coastline * 1000) / self.population

These nine lines of code stand in for quite a bit of boilerplate code and best practices. Think about what it would take to implement Country as a regular class: the .__init__() method, a repr, six different comparison methods as well as the .beach_per_person() method. You can expand the box below to see an implementation of Country that is roughly equivalent to the data class:

class Country:

    def __init__(self, name, population, area, coastline=0):
        self.name = name
        self.population = population
        self.area = area
        self.coastline = coastline

    def __repr__(self):
        return (
            f"Country(name={self.name!r}, population={self.population!r},"
            f" coastline={self.coastline!r})"
        )

    def __eq__(self, other):
        if other.__class__ is self.__class__:
            return (
                (self.name, self.population, self.coastline)
                == (other.name, other.population, other.coastline)
            )
        return NotImplemented

    def __ne__(self, other):
        if other.__class__ is self.__class__:
            return (
                (self.name, self.population, self.coastline)
                != (other.name, other.population, other.coastline)
            )
        return NotImplemented

    def __lt__(self, other):
        if other.__class__ is self.__class__:
            return ((self.name, self.population, self.coastline) < (
                other.name, other.population, other.coastline
            ))
        return NotImplemented

    def __le__(self, other):
        if other.__class__ is self.__class__:
            return ((self.name, self.population, self.coastline) <= (
                other.name, other.population, other.coastline
            ))
        return NotImplemented

    def __gt__(self, other):
        if other.__class__ is self.__class__:
            return ((self.name, self.population, self.coastline) > (
                other.name, other.population, other.coastline
            ))
        return NotImplemented

    def __ge__(self, other):
        if other.__class__ is self.__class__:
            return ((self.name, self.population, self.coastline) >= (
                other.name, other.population, other.coastline
            ))
        return NotImplemented

    def beach_per_person(self):
        """Meters of coastline per person"""
        return (self.coastline * 1000) / self.population

After creation, a data class is a normal class. You can, for instance, inherit from a data class in the normal way. The main purpose of data classes is to make it quick and easy to write robust classes, in particular small classes that mainly store data.

You can use the Country data class like any other class:

>>> norway = Country("Norway", 5320045, 323802, 58133)
>>> norway
Country(name='Norway', population=5320045, coastline=58133)

>>> norway.area
323802

>>> usa = Country("United States", 326625791, 9833517, 19924)
>>> nepal = Country("Nepal", 29384297, 147181)
>>> nepal
Country(name='Nepal', population=29384297, coastline=0)

>>> usa.beach_per_person()
0.06099946957342386

>>> norway.beach_per_person()
10.927163210085629

Note that all the fields .name, .population, .area, and .coastline are used when initializing the class (although .coastline is optional, as is shown in the example of landlocked Nepal). The Country class has a reasonable repr, while defining methods works the same as for regular classes.

By default, data classes can be compared for equality. Since we specified order=True in the @dataclass decorator, the Country class can also be sorted:

>>> norway == norway
True

>>> nepal == usa
False

>>> sorted((norway, usa, nepal))
[Country(name='Nepal', population=29384297, coastline=0),
 Country(name='Norway', population=5320045, coastline=58133),
 Country(name='United States', population=326625791, coastline=19924)]

The sorting happens on the field values, first .name then .population, and so on. However, if you use field(), you can customize which fields will be used in the comparison. In the example, the .area field was left out of the repr and the comparisons.

Note: The country data are from the CIA World Factbook with population numbers estimated for July 2017.

Before you all go book your next beach holidays in Norway, here is what the Factbook says about the Norwegian climate: “temperate along coast, modified by North Atlantic Current; colder interior with increased precipitation and colder summers; rainy year-round on west coast.”

Data classes do some of the same things as namedtuple. Yet, they draw their biggest inspiration from the attrs project. See our full guide to data classes for more examples and further information, as well as PEP 557 for the official description.

Customization of Module Attributes

Attributes are everywhere in Python! While class attributes are probably the most famous, attributes can actually be put on essentially anything—including functions and modules. Several of Python’s basic features are implemented as attributes: most of the introspection functionality, doc-strings, and name spaces. Functions inside a module are made available as module attributes.

Attributes are most often retrieved using the dot notation: thing.attribute. However, you can also get attributes that are named at runtime using getattr():

import random

random_attr = random.choice(("gammavariate", "lognormvariate", "normalvariate"))
random_func = getattr(random, random_attr)

print(f"A {random_attr} random value: {random_func(1, 1)}")

Running this code will produce something like:

A gammavariate random value: 2.8017715125270618

For classes, calling thing.attr will first look for attr defined on thing. If it is not found, then the special method thing.__getattr__("attr") is called. (This is a simplification. See this article for more details.) The .__getattr__() method can be used to customize access to attributes on objects.

Until Python 3.7, the same customization was not easily available for module attributes. However, PEP 562 introduces __getattr__() on modules, together with a corresponding __dir__() function. The __dir__() special function allows customization of the result of calling dir() on a module.

The PEP itself gives a few examples of how these functions can be used, including adding deprecation warnings to functions and lazy loading of heavy submodules. Below, we will build a simple plugin system that allows functions to be added to a module dynamically. This example takes advantage of Python packages. See this article if you need a refresher on packages.

Create a new directory, plugins, and add the following code to a file, plugins/__init__.py:

from importlib import import_module
from importlib import resources

PLUGINS = dict()

def register_plugin(func):
    """Decorator to register plug-ins"""
    name = func.__name__
    PLUGINS[name] = func
    return func

def __getattr__(name):
    """Return a named plugin"""
    try:
        return PLUGINS[name]
    except KeyError:
        _import_plugins()
        if name in PLUGINS:
            return PLUGINS[name]
        else:
            raise AttributeError(
                f"module {__name__!r} has no attribute {name!r}"
            ) from None

def __dir__():
    """List available plug-ins"""
    _import_plugins()
    return list(PLUGINS.keys())

def _import_plugins():
    """Import all resources to register plug-ins"""
    for name in resources.contents(__name__):
        if name.endswith(".py"):
            import_module(f"{__name__}.{name[:-3]}")

Before we look at what this code does, add two more files inside the plugins directory. First, let’s see plugins/plugin_1.py:

from . import register_plugin

@register_plugin
def hello_1():
    print("Hello from Plugin 1")

Next, add similar code in the file plugins/plugin_2.py:

from . import register_plugin

@register_plugin
def hello_2():
    print("Hello from Plugin 2")

@register_plugin
def goodbye():
    print("Plugin 2 says goodbye")

These plugins can now be used as follows:

>>> import plugins
>>> plugins.hello_1()
Hello from Plugin 1

>>> dir(plugins)
['goodbye', 'hello_1', 'hello_2']

>>> plugins.goodbye()
Plugin 2 says goodbye

This may not all seem that revolutionary (and it probably isn’t), but let’s look at what actually happened here. Normally, to be able to call plugins.hello_1(), the hello_1() function must be defined in a plugins module or explicitly imported inside __init__.py in a plugins package. Here, it is neither!

Instead, hello_1() is defined in an arbitrary file inside the plugins package, and hello_1() becomes a part of the plugins package by registering itself using the @register_plugin decorator.

The difference is subtle. Instead of the package dictating which functions are available, the individual functions register themselves as part of the package. This gives you a simple structure where you can add functions independently of the rest of the code without having to keep a centralized list of which functions are available.

Let us do a quick review of what __getattr__() does inside the plugins/__init__.py code. When you asked for plugins.hello_1(), Python first looks for a hello_1() function inside the plugins/__init__.py file. As no such function exists, Python calls __getattr__("hello_1") instead. Remember the source code of the __getattr__() function:

def __getattr__(name):
    """Return a named plugin"""
    try:
        return PLUGINS[name]        # 1) Try to return plugin
    except KeyError:
        _import_plugins()           # 2) Import all plugins
        if name in PLUGINS:
            return PLUGINS[name]    # 3) Try to return plugin again
        else:
            raise AttributeError(   # 4) Raise error
                f"module {__name__!r} has no attribute {name!r}"
            ) from None

__getattr__() contains the following steps. The numbers in the following list correspond to the numbered comments in the code:

First, the function optimistically tries to return the named plugin from the PLUGINS dictionary. This will succeed if a plugin named name exists and has already been imported.
If the named plugin is not found in the PLUGINS dictionary, we make sure all plugins are imported.
Return the named plugin if it has become available after the import.
If the plugin is not in the PLUGINS dictionary after importing all plugins, we raise an AttributeError saying that name is not an attribute (plugin) on the current module.

How is the PLUGINS dictionary populated though? The _import_plugins() function imports all Python files inside the plugins package, but does not seem to touch PLUGINS:

def _import_plugins():
    """Import all resources to register plug-ins"""
    for name in resources.contents(__name__):
        if name.endswith(".py"):
            import_module(f"{__name__}.{name[:-3]}")

Don’t forget that each plugin function is decorated by the @register_plugin decorator. This decorator is called when the plugins are imported and is the one actually populating the PLUGINS dictionary. You can see this if you manually import one of the plugin files:

>>> import plugins
>>> plugins.PLUGINS
{}

>>> import plugins.plugin_1
>>> plugins.PLUGINS
{'hello_1': <function hello_1 at 0x7f29d4341598>}

Continuing the example, note that calling dir() on the module also imports the remaining plugins:

>>> dir(plugins)
['goodbye', 'hello_1', 'hello_2']

>>> plugins.PLUGINS
{'hello_1': <function hello_1 at 0x7f29d4341598>,
 'hello_2': <function hello_2 at 0x7f29d4341620>,
 'goodbye': <function goodbye at 0x7f29d43416a8>}

dir() usually lists all available attributes on an object. Normally, using dir() on a module results in something like this:

>>> import plugins
>>> dir(plugins)
['PLUGINS', '__builtins__', '__cached__', '__doc__',
 '__file__', '__getattr__', '__loader__', '__name__',
 '__package__', '__path__', '__spec__', '_import_plugins',
 'import_module', 'register_plugin', 'resources']

While this might be useful information, we are more interested in exposing the available plugins. In Python 3.7, you can customize the result of calling dir() on a module by adding a __dir__() special function. For plugins/__init__.py, this function first makes sure all plugins have been imported and then lists their names:

def __dir__():
    """List available plug-ins"""
    _import_plugins()
    return list(PLUGINS.keys())

Before leaving this example, please note that we also used another cool new feature of Python 3.7. To import all modules inside the plugins directory, we used the new importlib.resources module. This module gives access to files and resources inside modules and packages without the need for __file__ hacks (which do not always work) or pkg_resources (which is slow). Other features of importlib.resources will be highlighted later.

Typing Enhancements

Type hinting and annotations have been in constant development throughout the Python 3 series of releases. Python’s typing system is now quite stable. Still, Python 3.7 brings some enhancements to the table: better performance, core support, and forward references.

Python does not do any type checking at runtime (unless you are explicitly using packages like enforce). Therefore, adding type hints to your code should not affect its performance.

Unfortunately, this is not completely true as most type hints need the typing module. The typing module is one of the slowest modules in the standard library. PEP 560 adds some core support for typing in Python 3.7, which significantly speeds up the typing module. The details of this are in general not necessary to know about. Simply lean back and enjoy the increased performance.

While Python’s type system is reasonably expressive, one issue that causes some pain is forward references. Type hints—or more generally annotations—are evaluated while the module is imported. Therefore, all names must already be defined before they are used. The following is not possible:

class Tree:
    def __init__(self, left: Tree, right: Tree) -> None:
        self.left = left
        self.right = right

Running the code raises a NameError because the class Tree is not yet (completely) defined in the definition of the .__init__() method:

Traceback (most recent call last):
  File "tree.py", line 1, in <module>
    class Tree:
  File "tree.py", line 2, in Tree
    def __init__(self, left: Tree, right: Tree) -> None:
NameError: name 'Tree' is not defined

To overcome this, you would have needed to write "Tree" as a string literal instead:

class Tree:
    def __init__(self, left: "Tree", right: "Tree") -> None:
        self.left = left
        self.right = right

See PEP 484 for the original discussion.

In a future Python 4.0, such so called forward references will be allowed. This will be handled by not evaluating annotations until that is explicitly asked for. PEP 563 describes the details of this proposal. In Python 3.7, forward references are already available as a __future__ import. You can now write the following:

from __future__ import annotations

class Tree:
    def __init__(self, left: Tree, right: Tree) -> None:
        self.left = left
        self.right = right

Note that in addition to avoiding the somewhat clumsy "Tree" syntax, the postponed evaluation of annotations will also speed up your code, since type hints are not executed. Forward references are already supported by mypy.

By far, the most common use of annotations is type hinting. Still, you have full access to the annotations at runtime and can use them as you see fit. If you are handling annotations directly, you need to deal with the possible forward references explicitly.

Let us create some admittedly silly examples that show when annotations are evaluated. First we do it old-style, so annotations are evaluated at import time. Let anno.py contain the following code:

def greet(name: print("Now!")):
    print(f"Hello {name}")

Note that the annotation of name is print(). This is only to see exactly when the annotation is evaluated. Import the new module:

>>> import anno
Now!

>>> anno.greet.__annotations__
{'name': None}

>>> anno.greet("Alice")
Hello Alice

As you can see, the annotation was evaluated at import time. Note that name ends up annotated with None because that is the return value of print().

Add the __future__ import to enable postponed evaluation of annotations:

from __future__ import annotations

def greet(name: print("Now!")):
    print(f"Hello {name}")

Importing this updated code will not evaluate the annotation:

>>> import anno

>>> anno.greet.__annotations__
{'name': "print('Now!')"}

>>> anno.greet("Marty")
Hello Marty

Note that Now! is never printed and the annotation is kept as a string literal in the __annotations__ dictionary. In order to evaluate the annotation, use typing.get_type_hints() or eval():

>>> import typing
>>> typing.get_type_hints(anno.greet)
Now!
{'name': <class 'NoneType'>}

>>> eval(anno.greet.__annotations__["name"])
Now!

>>> anno.greet.__annotations__
{'name': "print('Now!')"}

Observe that the __annotations__ dictionary is never updated, so you need to evaluate the annotation every time you use it.

Timing Precision

In Python 3.7, the time module gains some new functions as described in PEP 564. In particular, the following six functions are added:

clock_gettime_ns(): Returns the time of a specified clock
clock_settime_ns(): Sets the time of a specified clock
monotonic_ns(): Returns the time of a relative clock that cannot go backwards (for instance due to daylight savings)
perf_counter_ns(): Returns the value of a performance counter—a clock specifically designed to measure short intervals
process_time_ns(): Returns the sum of the system and user CPU time of the current process (not including sleep time)
time_ns(): Returns the number of nanoseconds since January 1st 1970

In a sense, there is no new functionality added. Each function is similar to an already existing function without the _ns suffix. The difference being that the new functions return a number of nanoseconds as an int instead of a number of seconds as a float.

For most applications, the difference between these new nanosecond functions and their old counterpart will not be appreciable. However, the new functions are easier to reason about because they rely on int instead of float. Floating point numbers are by nature inaccurate:

>>> 0.1 + 0.1 + 0.1
0.30000000000000004

>>> 0.1 + 0.1 + 0.1 == 0.3
False

This is not an issue with Python but rather a consequence of computers needing to represent infinite decimal numbers using a finite number of bits.

A Python float follows the IEEE 754 standard and uses 53 significant bits. The result is that any time greater than about 104 days (2⁵³ or approximately 9 quadrillion nanoseconds) cannot be expressed as a float with nanosecond precision. In contrast, a Python int is unlimited, so an integer number of nanoseconds will always have nanosecond precision independent of the time value.

As an example, time.time() returns the number of seconds since January 1st 1970. This number is already quite big, so the precision of this number is at the microsecond level. This function is the one showing the biggest improvement in its _ns version. The resolution of time.time_ns() is about 3 times better than for time.time().

What is a nanosecond by the way? Technically, it is one billionth of a second, or 1e-9 second if you prefer scientific notation. These are just numbers though and do not really provide any intuition. For a better visual aid, see Grace Hopper’s wonderful demonstration of the nanosecond.

As an aside, if you need to work with datetimes with nanosecond precision, the datetime standard library will not cut it. It explicitly only handles microseconds:

>>> from datetime import datetime, timedelta
>>> datetime(2018, 6, 27) + timedelta(seconds=1e-6)
datetime.datetime(2018, 6, 27, 0, 0, 0, 1)

>>> datetime(2018, 6, 27) + timedelta(seconds=1e-9)
datetime.datetime(2018, 6, 27, 0, 0)

Instead, you can use the astropy project. Its astropy.time package represents datetimes using two float objects which guarantees “sub-nanosecond precision over times spanning the age of the universe.”

>>> from astropy.time import Time, TimeDelta
>>> Time("2018-06-27")
<Time object: scale='utc' format='iso' value=2018-06-27 00:00:00.000>

>>> t = Time("2018-06-27") + TimeDelta(1e-9, format="sec")
>>> (t - Time("2018-06-27")).sec
9.976020010071807e-10

The latest version of astropy is available in Python 3.5 and later.

Other Pretty Cool Features

So far, you have seen the headline news regarding what’s new in Python 3.7. However, there are many other changes that are also pretty cool. In this section, we will look briefly at some of them.

The Order of Dictionaries Is Guaranteed

The CPython implementation of Python 3.6 has ordered dictionaries. (PyPy also has this.) This means that items in dictionaries are iterated over in the same order they were inserted. The first example is using Python 3.5, and the second is using Python 3.6:

>>> {"one": 1, "two": 2, "three": 3}  # Python <= 3.5
{'three': 3, 'one': 1, 'two': 2}

>>> {"one": 1, "two": 2, "three": 3}  # Python >= 3.6
{'one': 1, 'two': 2, 'three': 3}

In Python 3.6, this ordering was just a nice consequence of that implementation of dict. In Python 3.7, however, dictionaries preserving their insert order is part of the language specification. As such, it may now be relied on in projects that support only Python >= 3.7 (or CPython >= 3.6).

“`async`” and “`await`” Are Keywords

Python 3.5 introduced coroutines with async and await syntax. To avoid issues of backwards compatibility, async and await were not added to the list of reserved keywords. In other words, it was still possible to define variables or functions named async and await.

In Python 3.7, this is no longer possible:

>>> async = 1
  File "<stdin>", line 1
    async = 1
          ^
SyntaxError: invalid syntax

>>> def await():
  File "<stdin>", line 1
    def await():
            ^
SyntaxError: invalid syntax

“`asyncio`” Face Lift

The asyncio standard library was originally introduced in Python 3.4 to handle concurrency in a modern way using event loops, coroutines and futures. Here is a gentle introduction.

In Python 3.7, the asyncio module is getting a major face lift, including many new functions, support for the context variables (see below), and performance improvements. Of particular note is asyncio.run(), which simplifies calling coroutines from synchronous code. Using asyncio.run(), you do not need to explicitly create the event loop. An asynchronous Hello World program can now be written:

import asyncio

async def hello_world():
    print("Hello World!")

asyncio.run(hello_world())

Context Variables

Context variables are variables that can have different values depending on their context. They are similar to Thread-Local Storage in which each execution thread may have a different value for a variable. However, with context variables, there may be several contexts in one execution thread. The main use case for context variables is keeping track of variables in concurrent asynchronous tasks.

The following example constructs three contexts, each with their own value for the value name. The greet() function is later able to use the value of name inside each context:

import contextvars

name = contextvars.ContextVar("name")
contexts = list()

def greet():
    print(f"Hello {name.get()}")

# Construct contexts and set the context variable name
for first_name in ["Steve", "Dina", "Harry"]:
    ctx = contextvars.copy_context()
    ctx.run(name.set, first_name)
    contexts.append(ctx)

# Run greet function inside each context
for ctx in reversed(contexts):
    ctx.run(greet)

Running this script greets Steve, Dina, and Harry in reverse order:

$ python3.7 context_demo.py
Hello Harry
Hello Dina
Hello Steve

Importing Data Files With “`importlib.resources`“

One challenge when packaging a Python project is deciding what to do with project resources like data files needed by the project. A few options have commonly been used:

Hard-code a path to the data file.
Put the data file inside the package and locate it using __file__.
Use setuptools.pkg_resources to access the data file resource.

Each of these have their shortcomings. The first option is not portable. Using __file__ is more portable, but if the Python project is installed it might end up inside a zip and not have a __file__ attribute. The third option solves this problem, but is unfortunately very slow.

A better solution is the new importlib.resources module in the standard library. It uses Python’s existing import functionality to also import data files. Assume you have a resource inside a Python package like this:

data/
│
├── alice_in_wonderland.txt
└── __init__.py

Note that data needs to be a Python package. That is, the directory needs to contain an __init__.py file (which may be empty). You can then read the alice_in_wonderland.txt file as follows:

>>> from importlib import resources
>>> with resources.open_text("data", "alice_in_wonderland.txt") as fid:
...     alice = fid.readlines()
... 
>>> print("".join(alice[:7]))
CHAPTER I. Down the Rabbit-Hole

Alice was beginning to get very tired of sitting by her sister on the
bank, and of having nothing to do: once or twice she had peeped into the
book her sister was reading, but it had no pictures or conversations in
it, ‘and what is the use of a book,’ thought Alice ‘without pictures or
conversations?’

A similar resources.open_binary() function is available for opening files in binary mode. In the earlier “plugins as module attributes” example, we used importlib.resources to discover the available plugins using resources.contents(). See Barry Warsaw’s PyCon 2018 talk for more information.

It is possible to use importlib.resources in Python 2.7 and Python 3.4+ through a backport. A guide on migrating from pkg_resources to importlib.resources is available.

Developer Tricks

Python 3.7 has added several features aimed at you as a developer. You have already seen the new breakpoint() built-in. In addition, a few new -X command line options have been added to the Python interpreter.

You can easily get an idea of how much time the imports in your script takes, using -X importtime:

$ python3.7 -X importtime my_script.py
import time: self [us] | cumulative | imported package
import time:      2607 |       2607 | _frozen_importlib_external
...
import time:       844 |      28866 |   importlib.resources
import time:       404 |      30434 | plugins

The cumulative column shows the cumulative time of import (in microseconds). In this example, importing plugins took about 0.03 seconds, most of which was spent importing importlib.resources. The self column shows the import time excluding nested imports.

You can now use -X dev to activate “development mode.” The development mode will add certain debug features and runtime checks that are considered too slow to be enabled by default. These include enabling faulthandler to show a traceback on serious crashes, as well as more warnings and debug hooks.

Finally, -X utf8 enables UTF-8 mode. (See PEP 540.) In this mode, UTF-8 will be used for text encoding regardless of the current locale.

Optimizations

Each new release of Python comes with a set of optimizations. In Python 3.7, there are some significant speed-ups, including:

There is less overhead in calling many methods in the standard library.
Method calls are up to 20% faster in general.
The startup time of Python itself is reduced by 10-30%.
Importing typing is 7 times faster.

In addition, many more specialized optimizations are included. See this list for a detailed overview.

The upshot of all these optimizations is that Python 3.7 is fast. It is simply the fastest version of CPython released so far.

So, Should I Upgrade?

Let’s start with the simple answer. If you want to try out any of the new features you have seen here, then you do need to be able to use Python 3.7. Using tools such as pyenv or Anaconda makes it easy to have several versions of Python installed side by side. There is no downside to installing Python 3.7 and trying it out.

Now, for the more complicated questions. Should you upgrade your production environment to Python 3.7? Should you make your own project dependent on Python 3.7 to take advantage of the new features?

With the obvious caveat that you should always do thorough testing before upgrading your production environment, there are very few things in Python 3.7 that will break earlier code (async and await becoming keywords is one example though). If you are already using a modern Python, upgrading to 3.7 should be quite smooth. If you want to be a little conservative, you might want to wait for the release of the first maintenance release—Python 3.7.1—tentatively expected some time in July 2018.

Arguing that you should make your project 3.7 only is harder. Many of the new features in Python 3.7 are either available as backports to Python 3.6 (data classes, importlib.resources) or conveniences (faster startup and method calls, easier debugging, and -X options). The latter, you can take advantage of by running Python 3.7 yourself while keeping your code compatible with Python 3.6 (or lower).

The big features that will lock your code to Python 3.7 are __getattr__() on modules, forward references in type hints, and the nanosecond time functions. If you really need any of these, you should go ahead and bump your requirements. Otherwise, your project will probably be more useful to others if it can be run on Python 3.6 for a while longer.

See the Porting to Python 3.7 guide for details to be aware of when upgrading.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

What Can I Do With Python?

2018-06-25T14:00:00+00:00

You’ve done it: you’ve finished a course or finally made it to the end of a book that teaches you the basics of programming with Python. You’ve mastered lists, dictionaries, classes, and maybe even some object oriented concepts.

So… what next?

Python is a very versatile programming language, with a plethora of uses in a variety of different fields. If you’ve grasped the basics of Python and are itching to build something with the language, then it’s time to figure out what your next step should be.

In this article, we offer several different projects, resources, and tutorials that you can use to start building things with Python.

Free PDF Download: Python 3 Cheat Sheet

What Others Do With Python

You’re probably wondering what people are building with Python in the real world. So first, let’s take a quick look at how some of the big tech companies are using the language.

Google is a company that has used Python from the start, and it’s gained a place as one of the tech giant’s main server-side languages. Guido van Rossum, Python’s Benevolent Dictator for Life, even worked there for several years, overseeing the language’s development.

Instagram likes Python for its simplicity. The service is known for running “the world’s largest deployment of the Django web framework, which is written entirely in Python.”

Spotify puts the language to use in its data analysis and back-end services. According to their team, Python’s ease of use leads to a lightning-fast development pipeline. Spotify performs a ton of analyses to give recommendations to their users, so they need something that’s simple but also works well. Python to the rescue!

You can check out this article to see what other companies are doing with Python.

If you’re already convinced, then let’s get you started!

What You Can Do With Python

From web development to data science, machine learning, and more, Python’s real-world applications are limitless. Here are some projects that will assist you in finally putting your Python skills to good use.

#1: Automate the Boring Stuff

This is a resource on “practical programming for total beginners.” Like the title says, this book will teach you how to automate tedious tasks such as updating spreadsheets or renaming files on your computer. It’s the perfect starting point for anyone who’s mastered the basics of Python.

You’ll get a chance to practice what you’ve learned so far by creating dictionaries, scraping the web, working with files, and creating objects and classes. The hands-on applications that you come across in this book will provide you with real-world results that you can see immediately.

This resource is available in different formats to give you the best learning experience possible. Buy the book on Amazon or read it online for free.

#2: Stay on Top of Bitcoin Prices

Everyone seems to be talking about Bitcoin these days. Ever since topping out at a price of almost $20,000 in December 2017, the cryptocurrency has been on the minds of millions. Its price continues to fluctuate, but many would consider it a worthwhile investment.

If you’re looking to cash in on the virtual gold rush and just need to know when to make your move, then you’ll need to stay on top of Bitcoin’s prices. This tutorial can teach you how to use your Python skills to build a Bitcoin price notification service.

The foundation of this project is the creation of IFTTT (“if this, then that”) applets. You’ll learn how to use the requests library to send HTTP requests and how to use a webhook to connect your app to external services.

This is the perfect starter project for a beginner Pythonista with an interest in crypto. The service you build with this tutorial can be extended to other currencies as well, so don’t worry—Ethereum is fair game, too.

#3: Create a Calculator

This simple project is a solid gateway into GUI programming. Building back-end services is one important part of deployment, but there may be a front-end that needs to be taken into account. Creating applications that users can easily interact with is paramount.

If you’re interested in UX and UI design, then take a look at this tutorial. You’ll be working with the tkinter module, the standard graphical user interface package that comes traditionally bundled with Python.

The tkinter module is a wrapper around Tcl/Tk, a combination of the Tcl scripting language and a GUI framework extension, Tk. If you have Python installed, then you should already have the tkinter framework ready to go as well. A simple call will get you started:

from tkinter import *

Once you’ve got that set up, you can get to work on building your first GUI calculator in Python.

Practice using the tkinter module and watch your vision materialize on the screen. Then, once you’ve got your feet wet, you can branch out and start working with Python’s other GUI toolkits. Check out the official documentation on GUI Programming in Python for more information.

#4: Mine Twitter Data

Thanks to the Internet—and, increasingly, the Internet of Things—we now have access to hordes of data that weren’t available even a decade ago. Analytics is a huge part of any field that works with data. What are people talking about? What patterns can we see in their behavior?

Twitter is a great place to get answers to some of these questions. If you’re interested in data analysis, then a Twitter data mining project is a great way to use your Python skills to answer questions about the world around you.

Our Twitter sentiment analysis tutorial will teach you how to mine Twitter data and analyze user sentiment with a docker environment. You’ll learn how to register an application with Twitter, which you’ll need to do in order to access their streaming API.

You’ll see how to use Tweepy to filter which tweets you want to pull, TextBlob to calculate the sentiment of those tweets, Elasticsearch to analyze their content, and Kibana to visualize the results. After you finish this tutorial, you should be ready to dive into other projects that use Python for text processing and speech recognition.

#5: Build a Microblog With Flask

It seems like everyone has a blog these days, but it’s not a bad idea to have a central hub for yourself online. With the advent of Twitter and Instagram, microblogging in particular has become exceedingly popular. In this project by Miguel Grinberg, you’ll learn how to build your own microblog.

It’s called “The Flask Mega-Tutorial,” and it truly lives up to its name. With 23 chapters to work through, you’ll develop a deep understanding of the Flask micro web-framework. At the end of this project, you should have a fully functional web application.

You don’t need to know anything about Flask to get started, so it’s perfect for those of you who are itching to get your hands dirty with web development.

The tutorial was recently updated to include content that will help you become a better web developer in general. You can read it for free online, purchase a copy on Amazon, or have the author walk you step by step through his online course. Once you’re done, you’ll be able to move on to Django and creating even larger-scale web applications.

#6: Build a Blockchain

While the blockchain was initially developed as a financial technology, it’s spreading to a variety of other industries. Blockchains can be used for almost any kind of transaction: from real estate dealings to medical record transfers.

You can get a better understanding of how they work by building one yourself. Hackernoon’s tutorial will assist you in implementing a blockchain from scratch. At the end of this project, you’ll have gained an in-depth understanding of how this transactional technology works.

You’ll be working with HTTP clients and the requests library. Once you install the Flask web framework, you’ll be able to use HTTP requests to communicate with your blockchain over the Internet.

Remember, blockchain isn’t just for crypto enthusiasts. Once you’ve built one for yourself, see if you can’t find a creative way to implement the technology in your field of interest.

#7: Bottle Up a Twitter Feed

Interested in building web applications but unsure about starting a mega-project? No worries—we’ve got something for you. Follow along with us to learn how to create a simple web app in just a few hours.

Bob Belderbos shares how he implemented the 40th PyBites Code Challenge, where participants were instructed to create a simple web app to better navigate the Daily Python Tip feed on Twitter. You can walk through his implementation of the challenge and code alongside him.

Instead of Flask, you’ll be using the Bottle micro web-framework. Bottle is known as a low-dependency solution for deploying apps quickly. Since it is designed to be lightweight and simple to use, you’ll have your application developed in no time.

You’ll also use the Tweepy module to load data from the Twitter API. You’ll store the data in an SQLAlchemy database, so you’ll get some practice writing SQL queries as well. Fork the repo to get started!

#8: Play PyGames

This one is for those of you who like to have fun! Python can be used to code a variety of arcade games, adventure games, and puzzle games that you can deploy within a few days. Classics like hangman, tic-tac-toe, ping-pong, and more are all doable with your newly acquired programming skills.

The Pygame library makes it even easier to build your own games. It contains almost anything you could need when starting to develop a game.

Pygame is free and open source. It includes computer graphics and sound libraries that you can use to add interactive functionality to your application.

There are scores of games you can create with the library. Whatever you choose to invent, feel free to share your stuff with the Pygame community!

#9: Choose Your Own Adventure

If you’re more into storytelling, then you can still build something cool with Python.

The language is extremely easy to write in, which makes it the perfect environment for developing interactive fiction. This free resource will guide you through the process of writing a text-based adventure game in Python.

The tutorial assumes basic knowledge of programming in Python, but it helps you bridge the gap between what you know and how to use that knowledge to build an application.

If you want to take your story to the next level, you can use a software engine like Ren’Py to add sounds and images to your game, creating a full-fledged visual novel. (Then, you can put it up on Steam and see how it does! The best way to get feedback on your work is to release your creation out into the world.)

#10: Say “Hello World!” to Machine Learning

Machine Learning can be a critical field of understanding for anyone interested in Artificial Intelligence. However, it can be intimidating to get started, because the space is fast and ever-changing.

Luckily, there are resources online that can help you get your feet wet before you dive into the world of data science. This tutorial by Jason Brownlee is a wonderful introduction to using Python for machine learning.

You’ll walk through some of the most common machine learning algorithms as well as the Python libraries that will assist you in making predictions.

The tutorial is extremely simple and very easy to follow. You can complete it in as little as a few hours. By the time you’re done, you’ll have gained a quick understanding of how to use Python to perform data science.

When you’re sure you’re ready to dive in, check out our stock of data science tutorials, where you’ll learn how to analyze fingerprints, create visualizations, and recognize speech and faces, all in Python.

#11: Get Challenged

If you’re not sure about taking the plunge with some of the larger projects listed above, but the smaller ones don’t interest you either, then you might be wondering what else there is. How on earth can you find something that excites you?

Coding challenges can help you practice your Python skills and gain a surface-level understanding of all the different things you can do with Python.

To put it simply: you’re presented with a problem, and you have to find a solution that uses Python.

You’ll get a chance to develop implementations that make sense to you, but you’ll also have the opportunity to dive deep into the Python language by way of hints. These give you an idea of which modules you should be importing to help you solve the challenge.

Coding challenges are a great way to learn breadth-first about as many libraries, methods, and frameworks as possible. You’re guaranteed to find something that you’ll want to explore more on your own time. You might even come back to this list and find that something you used in one of your challenges has sparked a new interest for you!

To get started, try one of these on for size:

The Python Challenge has over 20 levels for you to work through. Create small Python scripts to find a solution to the level. There are hints scattered about the Internet, but try to see the clues and figure it out for yourself!
PyBites Code Challenges has 50 challenges and counting! These challenges encourage you to master Python by building applications that accomplish tasks.

If you’d rather push yourself by coding through these challenges on your own instead of working through a step-by-step tutorial, then it’s always a good idea to have a resource you can turn to for help. Python Tricks: The Book is an amazing source of information to have on hand when you are working through these challenges. It will take you through some of the lesser-known parts of Python that you’ll need to solve them.

What You Probably Shouldn’t Do With Python

Clearly, Python is an extremely versatile language, and there’s a lot you can do with it. But you can’t do everything with it. In fact, there are some things that Python is not very well suited for at all.

As an interpreted language, Python has trouble interacting with low-level devices, like device drivers. For instance, you’d have a problem if you wanted to write an operating system with Python only. You’re better off sticking with C or C++ for low-level applications.

However, even that might not be true for long. As a testament to Python’s flexibility, there are those out there who are working on projects that extend Python’s usability to low-level interactions. MicroPython is just one of these projects, designing low-level capability for Python.

What if My Idea Isn’t on This List?

That’s okay! This list isn’t exhaustive—there are countless other tools and applications you can build with Python that we haven’t covered here. Don’t think you’re limited to what’s on this list. It is simply a resource to give you a place to start.

This video will give you some ideas on other projects that Python is well-suited for. You can also check out this blog post to learn where to find inspiration for more Python projects.

In the end, it’s up to you to do the research and find projects that pique your interest. If you’re not sure where to begin, then follow us on Twitter. We regularly share cool and interesting Python projects from our reader community. You might find something that you can’t wait to contribute to!

What to Do Next

So there you have it! Eleven ways to start working your way from Python beginner to savvy Pythonista.

No matter where you choose to begin, you’re sure to open up countless avenues for developing your programming skills. Pick something—anything—and get started! Do you have an idea for a project that didn’t make this list? Leave a comment down below! You could suggest the perfect project for a fellow programmer.

If you get stuck and need a nudge in the right direction, check out our tips for developing positive learning strategies to help get yourself back on track.

Another great way to get unstuck is to talk it out. Coding doesn’t have to be a solitary activity. If you need a way to ask questions and get answers quickly from knowledgeable professionals, then consider joining the PythonistaCafe. This private community allows you to network with those who will help push you through any walls you may hit on your journey to Python mastery. Click here to learn more, or go ahead an apply!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Operators and Expressions in Python

2018-06-20T14:00:00+00:00

After finishing our previous tutorial on Python variables in this series, you should now have a good grasp of creating and naming Python objects of different types. Let’s do some work with them!

Here’s what you’ll learn in this tutorial: You’ll see how calculations can be performed on objects in Python. By the end of this tutorial, you will be able to create complex expressions by combining objects and operators.

Get Notified: Don't miss the follow up to this tutorial—Click here to join the Real Python Newsletter and you'll know when the next installment comes out.

In Python, operators are special symbols that designate that some sort of computation should be performed. The values that an operator acts on are called operands.

Here is an example:

>>> a = 10
>>> b = 20
>>> a + b
30

In this case, the + operator adds the operands a and b together. An operand can be either a literal value or a variable that references an object:

>>> a = 10
>>> b = 20
>>> a + b - 5
25

A sequence of operands and operators, like a + b - 5, is called an expression. Python supports many operators for combining data objects into expressions. These are explored below.

Arithmetic Operators

The following table lists the arithmetic operators supported by Python:

Operator	Example	Meaning	Result
`+` (unary)	`+a`	Unary Positive	`a` In other words, it doesn’t really do anything. It mostly exists for the sake of completeness, to complement Unary Negation.
`+` (binary)	`a + b`	Addition	Sum of `a` and `b`
`-` (unary)	`-a`	Unary Negation	Value equal to `a` but opposite in sign
`-` (binary)	`a - b`	Subtraction	`b` subtracted from `a`
`*`	`a * b`	Multiplication	Product of `a` and `b`
`/`	`a / b`	Division	Quotient when `a` is divided by `b`. The result always has type `float`.
`%`	`a % b`	Modulus	Remainder when `a` is divided by `b`
`//`	`a // b`	Floor Division (also called Integer Division)	Quotient when `a` is divided by `b`, rounded to the next smallest whole number
`**`	`a ** b`	Exponentiation	`a` raised to the power of `b`

Here are some examples of these operators in use:

>>> a = 4
>>> b = 3
>>> +a
4
>>> -b
-3
>>> a + b
7
>>> a - b
1
>>> a * b
12
>>> a / b
1.3333333333333333
>>> a % b
1
>>> a ** b
64

The result of standard division (/) is always a float, even if the dividend is evenly divisible by the divisor:

>>> 10 / 5
2.0
>>> type(10 / 5)
<class 'float'>

When the result of floor division (//) is positive, it is as though the fractional portion is truncated off, leaving only the integer portion. When the result is negative, the result is rounded down to the next smallest (greater negative) integer:

>>> 10 / 4
2.5
>>> 10 // 4
2
>>> 10 // -4
-3
>>> -10 // 4
-3
>>> -10 // -4
2

Note, by the way, that in a REPL session, you can display the value of an expression by just typing it in at the >>> prompt without print(), the same as you can with a literal value or variable:

>>> 25
25
>>> x = 4
>>> y = 6
>>> x
4
>>> y
6
>>> x * 25 + y
106

Comparison Operators

Operator	Example	Meaning	Result
`==`	`a == b`	Equal to	`True` if the value of `a` is equal to the value of `b` `False` otherwise
`!=`	`a != b`	Not equal to	`True` if `a` is not equal to `b` `False` otherwise
`<`	`a < b`	Less than	`True` if `a` is less than `b` `False` otherwise
`<=`	`a <= b`	Less than or equal to	`True` if `a` is less than or equal to `b` `False` otherwise
`>`	`a > b`	Greater than	`True` if `a` is greater than `b` `False` otherwise
`>=`	`a >= b`	Greater than or equal to	`True` if `a` is greater than or equal to `b` `False` otherwise

Here are examples of the comparison operators in use:

>>> a = 10
>>> b = 20
>>> a == b
False
>>> a != b
True
>>> a <= b
True
>>> a >= b
False

>>> a = 30
>>> b = 30
>>> a == b
True
>>> a <= b
True
>>> a >= b
True

Comparison operators are typically used in Boolean contexts like conditional and loop statements to direct program flow, as you will see later.

Equality Comparison on Floating-Point Values

Recall from the earlier discussion of floating-point numbers that the value stored internally for a float object may not be precisely what you’d think it would be. For that reason, it is poor practice to compare floating-point values for exact equality. Consider this example:

>>> x = 1.1 + 2.2
>>> x == 3.3
False

Yikes! The internal representations of the addition operands are not exactly equal to 1.1 and 2.2, so you cannot rely on x to compare exactly to 3.3.

The preferred way to determine whether two floating-point values are “equal” is to compute whether they are close to one another, given some tolerance. Take a look at this example:

>>> tolerance = 0.00001
>>> x = 1.1 + 2.2
>>> abs(x - 3.3) < tolerance
True

abs() returns absolute value. If the absolute value of the difference between the two numbers is less than the specified tolerance, they are close enough to one another to be considered equal.

Logical Operators

The logical operators not, or, and and modify and join together expressions evaluated in Boolean context to create more complex conditions.

Logical Expressions Involving Boolean Operands

As you have seen, some objects and expressions in Python actually are of Boolean type. That is, they are equal to one of the Python objects True or False. Consider these examples:

>>> x = 5
>>> x < 10
True
>>> type(x < 10)
<class 'bool'>

>>> t = x > 10
>>> t
False
>>> type(t)
<class 'bool'>

>>> callable(x)
False
>>> type(callable(x))
<class 'bool'>

>>> t = callable(len)
>>> t
True
>>> type(t)
<class 'bool'>

In the examples above, x < 10, callable(x), and t are all Boolean objects or expressions.

Interpretation of logical expressions involving not, or, and and is straightforward when the operands are Boolean:

Operator	Example	Meaning
`not`	`not x`	`True` if `x` is `False` `False` if `x` is `True` (Logically reverses the sense of `x`)
`or`	`x or y`	`True` if either `x` or `y` is `True` `False` otherwise
`and`	`x and y`	`True` if both `x` and `y` are `True` `False` otherwise

Take a look at how they work in practice below.

“`not`” and Boolean Operands

x = 5
not x < 10
False
not callable(x)
True

Operand	Value	Logical Expression	Value
`x < 10`	`True`	`not x < 10`	`False`
`callable(x)`	`False`	`not callable(x)`	`True`

“`or`” and Boolean Operands

x = 5
x < 10 or callable(x)
True
x < 0 or callable(x)
False

Operand	Value	Operand	Value	Logical Expression	Value
`x < 10`	`True`	`callable(x)`	`False`	`x < 10 or callable(x)`	`True`
`x < 0`	`False`	`callable(x)`	`False`	`x < 0 or callable(x)`	`False`

“`and`” and Boolean Operands

x = 5
x < 10 and callable(x)
False
x < 10 and callable(len)
True

Operand	Value	Operand	Value	Logical Expression	Value
`x < 10`	`True`	`callable(x)`	`False`	`x < 10 and callable(x)`	`False`
`x < 10`	`True`	`callable(len)`	`True`	`x < 10 or callable(len)`	`True`

Evaluation of Non-Boolean Values in Boolean Context

Many objects and expressions are not equal to True or False. Nonetheless, they may still be evaluated in Boolean context and determined to be “truthy” or “falsy.”

So what is true and what isn’t? As a philosophical question, that is outside the scope of this tutorial!

But in Python, it is well-defined. All the following are considered false when evaluated in Boolean context:

The Boolean value False
Any value that is numerically zero (0, 0.0, 0.0+0.0j)
An empty string
An object of a built-in composite data type which is empty (see below)
The special value denoted by the Python keyword None

Virtually any other object built into Python is regarded as true.

You can determine the “truthiness” of an object or expression with the built-in bool() function. bool() returns True if its argument is truthy and False if it is falsy.

Numeric Value

A zero value is false.
A non-zero value is true.

>>> print(bool(0), bool(0.0), bool(0.0+0j))
False False False

>>> print(bool(-3), bool(3.14159), bool(1.0+1j))
True True True

String

An empty string is false.
A non-empty string is true.

>>> print(bool(''), bool(""), bool(""""""))
False False False

>>> print(bool('foo'), bool(" "), bool(''' '''))
True True True

Built-In Composite Data Object

Python provides built-in composite data types called list, tuple, dict, and set. These are “container” types that contain other objects. An object of one of these types is considered false if it is empty and true if it is non-empty.

The examples below demonstrate this for the list type. (Lists are defined in Python with square brackets.)

For more information on the list, tuple, dict, and set types, see the upcoming tutorials.

>>> type([])
<class 'list'>
>>> bool([])
False

>>> type([1, 2, 3])
<class 'list'>
>>> bool([1, 2, 3])
True

The “`None`” Keyword

None is always false:

>>> bool(None)
False

Logical Expressions Involving Non-Boolean Operands

Non-Boolean values can also be modified and joined by not, or and, and. The result depends on the “truthiness” of the operands.

“`not`” and Non-Boolean Operands

Here is what happens for a non-Boolean value x:

If `x` is	`not x` is
“truthy”	`False`
“falsy”	`True`

Here are some concrete examples:

>>> x = 3
>>> bool(x)
True
>>> not x
False

>>> x = 0.0
>>> bool(x)
False
>>> not x
True

“`or`” and Non-Boolean Operands

This is what happens for two non-Boolean values x and y:

If `x` is	`x or y` is
truthy	`x`
falsy	`y`

Note that in this case, the expression x or y does not evaluate to either True or False, but instead to one of either x or y:

>>> x = 3
>>> y = 4
>>> x or y
3

>>> x = 0.0
>>> y = 4.4
>>> x or y
4.4

Even so, it is still the case that the expression x or y will be truthy if either x or y is truthy, and falsy if both x and y are falsy.

“`and`” and Non-Boolean Operands

Here’s what you’ll get for two non-Boolean values x and y:

If `x` is	`x and y` is
“truthy”	`y`
“falsy”	`x`

>>> x = 3
>>> y = 4
>>> x and y
4

>>> x = 0.0
>>> y = 4.4
>>> x and y
0.0

As with or, the expression x and y does not evaluate to either True or False, but instead to one of either x or y. x and y will be truthy if both x and y are truthy, and falsy otherwise.

Compound Logical Expressions and Short-Circuit Evaluation

So far, you have seen expressions with only a single or or and operator and two operands:

x or y
x and y

Multiple logical operators and operands can be strung together to form compound logical expressions.

Compound “`or`” Expressions

Consider the following expression:

x₁ or x₂ or x₃ or … x_n

This expression is true if any of the x_i are true.

In an expression like this, Python uses a methodology called short-circuit evaluation, also called McCarthy evaluation in honor of computer scientist John McCarthy. The x_i operands are evaluated in order from left to right. As soon as one is found to be true, the entire expression is known to be true. At that point, Python stops and no more terms are evaluated. The value of the entire expression is that of the x_i that terminated evaluation.

To help demonstrate short-circuit evaluation, suppose that you have a simple “identity” function f() that behaves as follows:

f() takes a single argument.
It displays the argument to the console.
It returns the argument passed to it as its return value.

(You will see how to define such a function in the upcoming tutorial on Functions.)

Several example calls to f() are shown below:

>>> f(0)
-> f(0) = 0
0

>>> f(False)
-> f(False) = False
False

>>> f(1.5)
-> f(1.5) = 1.5
1.5

Because f() simply returns the argument passed to it, we can make the expression f(arg) be truthy or falsy as needed by specifying a value for arg that is appropriately truthy or falsy. Additionally, f() displays its argument to the console, which visually confirms whether or not it was called.

Now, consider the following compound logical expression:

>>> f(0) or f(False) or f(1) or f(2) or f(3)
-> f(0) = 0
-> f(False) = False
-> f(1) = 1
1

The interpreter first evaluates f(0), which is 0. A numeric value of 0 is false. The expression is not true yet, so evaluation proceeds left to right. The next operand, f(False), returns False. That is also false, so evaluation continues.

Next up is f(1). That evaluates to 1, which is true. At that point, the interpreter stops because it now knows the entire expression to be true. 1 is returned as the value of the expression, and the remaining operands, f(2) and f(3), are never evaluated. You can see from the display that the f(2) and f(3) calls do not occur.

Compound “`and`” Expressions

A similar situation exists in an expression with multiple and operators:

x₁ and x₂ and x₃ and … x_n

This expression is true if all the x_i are true.

In this case, short-circuit evaluation dictates that the interpreter stop evaluating as soon as any operand is found to be false, because at that point the entire expression is known to be false. Once that is the case, no more operands are evaluated, and the falsy operand that terminated evaluation is returned as the value of the expression:

>>> f(1) and f(False) and f(2) and f(3)
-> f(1) = 1
-> f(False) = False
False

>>> f(1) and f(0.0) and f(2) and f(3)
-> f(1) = 1
-> f(0.0) = 0.0
0.0

In both examples above, evaluation stops at the first term that is false—f(False) in the first case, f(0.0) in the second case—and neither the f(2) nor f(3) call occurs. False and 0.0, respectively, are returned as the value of the expression.

If all the operands are truthy, they all get evaluated and the last (rightmost) one is returned as the value of the expression:

>>> f(1) and f(2.2) and f('bar')
-> f(1) = 1
-> f(2.2) = 2.2
-> f(bar) = bar
'bar'

Idioms That Exploit Short-Circuit Evaluation

There are some common idiomatic patterns that exploit short-circuit evaluation for conciseness of expression.

Avoiding an Exception

Suppose you have defined two variables a and b, and you want to know whether (b / a) > 0:

>>> a = 3
>>> b = 1
>>> (b / a) > 0
True

But you need to account for the possibility that a might be 0, in which case the interpreter will raise an exception:

>>> a = 0
>>> b = 1
>>> (b / a) > 0
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    (b / a) > 0
ZeroDivisionError: division by zero

You can avoid an error with an expression like this:

>>> a = 0
>>> b = 1
>>> a != 0 and (b / a) > 0
False

When a is 0, a != 0 is false. Short-circuit evaluation ensures that evaluation stops at that point. (b / a) is not evaluated, and no error is raised.

If fact, you can be even more concise than that. When a is 0, the expression a by itself is falsy. There is no need for the explicit comparison a != 0:

>>> a = 0
>>> b = 1
>>> a and (b / a) > 0
0

Selecting a Default Value

Another idiom involves selecting a default value when a specified value is zero or empty. For example, suppose you want to assign a variable s to the value contained in another variable called string. But if string is empty, you want to supply a default value.

Here is a concise way of expressing this using short-circuit evaluation:

s = string or '<default_value>'

If string is non-empty, it is truthy, and the expression string or '<default_value>' will be true at that point. Evaluation stops, and the value of string is returned and assigned to s:

>>> string = 'foo bar'
>>> s = string or '<default_value>'
>>> s
'foo bar'

On the other hand, if string is an empty string, it is falsy. Evaluation of string or '<default_value>' continues to the next operand, '<default_value>', which is returned and assigned to s:

>>> string = ''
>>> s = string or '<default_value>'
>>> s
'<default_value>'

Chained Comparisons

Comparison operators can be chained together to arbitrary length. For example, the following expressions are nearly equivalent:

x < y <= z
x < y and y <= z

They will both evaluate to the same Boolean value. The subtle difference between the two is that in the chained comparison x < y <= z, y is evaluated only once. The longer expression x < y and y <= z will cause y to be evaluated twice.

Note: In cases where y is a static value, this will not be a significant distinction. But consider these expressions:

x < f() <= z
x < f() and f() <= z

If f() is a function that causes program data to be modified, the difference between its being called once in the first case and twice in the second case may be important.

More generally, if op₁, op₂, …, op_n are comparison operators, then the following have the same Boolean value:

x₁ op₁ x₂ op₂ x₃ … x_n-1 op_n x_n

x₁ op₁ x₂ and x₂ op₂ x₃ and … x_n-1 op_n x_n

In the former case, each x_i is only evaluated once. In the latter case, each will be evaluated twice except the first and last, unless short-circuit evaluation causes premature termination.

Bitwise Operators

Bitwise operators treat operands as sequences of binary digits and operate on them bit by bit. The following operators are supported:

Operator	Example	Meaning	Result
`&`	`a & b`	bitwise AND	Each bit position in the result is the logical AND of the bits in the corresponding position of the operands. (`1` if both are `1`, otherwise `0`.)
`\|`	`a \| b`	bitwise OR	Each bit position in the result is the logical OR of the bits in the corresponding position of the operands. (`1` if either is `1`, otherwise `0`.)
`~`	`~a`	bitwise negation	Each bit position in the result is the logical negation of the bit in the corresponding position of the operand. (`1` if `0`, `0` if `1`.)
`^`	`a ^ b`	bitwise XOR (exclusive OR)	Each bit position in the result is the logical XOR of the bits in the corresponding position of the operands. (`1` if the bits in the operands are different, `0` if they are the same.)
`>>`	`a >> n`	Shift right `n` places	Each bit is shifted right `n` places.
`<<`	`a << n`	Shift left `n` places	Each bit is shifted left `n` places.

Here are some examples:

>>> '0b{:04b}'.format(0b1100 & 0b1010)
'0b1000'
>>> '0b{:04b}'.format(0b1100 | 0b1010)
'0b1110'
>>> '0b{:04b}'.format(0b1100 ^ 0b1010)
'0b0110'
>>> '0b{:04b}'.format(0b1100 >> 2)
'0b0011'
>>> '0b{:04b}'.format(0b0011 << 2)
'0b1100'

Note: The purpose of the '0b{:04b}'.format() is to format the numeric output of the bitwise operations, to make them easier to read. You will see the format() method in much more detail later. For now, just pay attention to the operands of the bitwise operations, and the results.

Identity Operators

Python provides two operators, is and is not, that determine whether the given operands have the same identity—that is, refer to the same object. This is not the same thing as equality, which means the two operands refer to objects that contain the same data but are not necessarily the same object.

Here is an example of two object that are equal but not identical:

>>> x = 1001
>>> y = 1000 + 1
>>> print(x, y)
1001 1001

>>> x == y
True
>>> x is y
False

Here, x and y both refer to objects whose value is 1001. They are equal. But they do not reference the same object, as you can verify:

>>> id(x)
60307920
>>> id(y)
60307936

x and y do not have the same identity, and x is y returns False.

You saw previously that when you make an assignment like x = y, Python merely creates a second reference to the same object, and that you could confirm that fact with the id() function. You can also confirm it using the is operator:

>>> a = 'I am a string'
>>> b = a
>>> id(a)
55993992
>>> id(b)
55993992

>>> a is b
True
>>> a == b
True

In this case, since a and b reference the same object, it stands to reason that a and b would be equal as well.

Unsurprisingly, the opposite of is is is not:

>>> x = 10
>>> y = 20
>>> x is not y
True

Operator Precedence

Consider this expression:

>>> 20 + 4 * 10
60

There is ambiguity here. Should Python perform the addition 20 + 4 first and then multiply the sum by 10? Or should the multiplication 4 * 10 be performed first, and the addition of 20 second?

Clearly, since the result is 60, Python has chosen the latter; if it had chosen the former, the result would be 240. This is standard algebraic procedure, found universally in virtually all programming languages.

All operators that the language supports are assigned a precedence. In an expression, all operators of highest precedence are performed first. Once those results are obtained, operators of the next highest precedence are performed. So it continues, until the expression is fully evaluated. Any operators of equal precedence are performed in left-to-right order.

Here is the order of precedence of the Python operators you have seen so far, from lowest to highest:

	Operator	Description
*lowest precedence*	`or`	Boolean OR
	`and`	Boolean AND
	`not`	Boolean NOT
	`==`, `!=`, `<`, `<=`, `>`, `>=`, `is`, `is not`	comparisons, identity
	`\|`	bitwise OR
	`^`	bitwise XOR
	`&`	bitwise AND
	`<<`, `>>`	bit shifts
	`+`, `-`	addition, subtraction
	`*`, `/`, `//`, `%`	multiplication, division, floor division, modulo
	`+x`, `-x`, `~x`	unary positive, unary negation, bitwise negation
*highest precedence*	`**`	exponentiation

Operators at the top of the table have the lowest precedence, and those at the bottom of the table have the highest. Any operators in the same row of the table have equal precedence.

It is clear why multiplication is performed first in the example above: multiplication has a higher precedence than addition.

Similarly, in the example below, 3 is raised to the power of 4 first, which equals 81, and then the multiplications are carried out in order from left to right (2 * 81 * 5 = 810):

>>> 2 * 3 ** 4 * 5
810

Operator precedence can be overridden using parentheses. Expressions in parentheses are always performed first, before expressions that are not parenthesized. Thus, the following happens:

>>> 20 + 4 * 10
60
>>> (20 + 4) * 10
240

>>> 2 * 3 ** 4 * 5
810
>>> 2 * 3 ** (4 * 5)
6973568802

In the first example, 20 + 4 is computed first, then the result is multiplied by 10. In the second example, 4 * 5 is calculated first, then 3 is raised to that power, then the result is multiplied by 2.

There is nothing wrong with making liberal use of parentheses, even when they aren’t necessary to change the order of evaluation. In fact, it is considered good practice, because it can make the code more readable, and it relieves the reader of having to recall operator precedence from memory. Consider the following:

(a < 10) and (b > 30)

Here the parentheses are fully unnecessary, as the comparison operators have higher precedence than and does and would have been performed first anyhow. But some might consider the intent of the parenthesized version more immediately obvious than this version without parentheses:

a < 10 and b > 30

On the other hand, there are probably those who would prefer the latter; it’s a matter of personal preference. The point is, you can always use parentheses if you feel it makes the code more readable, even if they aren’t necessary to change the order of evaluation.

Augmented Assignment Operators

You have seen that a single equal sign (=) is used to assign a value to a variable. It is, of course, perfectly viable for the value to the right of the assignment to be an expression containing other variables:

>>> a = 10
>>> b = 20
>>> c = a * 5 + b
>>> c
70

In fact, the expression to the right of the assignment can include references to the variable that is being assigned to:

>>> a = 10
>>> a = a + 5
>>> a
15

>>> b = 20
>>> b = b * 3
>>> b
60

The first example is interpreted as “a is assigned the current value of a plus 5,” effectively increasing the value of a by 5. The second reads “b is assigned the current value of b times 3,” effectively increasing the value of b threefold.

Of course, this sort of assignment only makes sense if the variable in question has already previously been assigned a value:

>>> z = z / 12
Traceback (most recent call last):
  File "<pyshell#11>", line 1, in <module>
    z = z / 12
NameError: name 'z' is not defined

Python supports a shorthand augmented assignment notation for these arithmetic and bitwise operators:

Arithmetic	Bitwise
`+` `-` `` `/` `%` `//` `*`	`&` `\|` `^` `>>` `<<`

For these operators, the following are equivalent:

x <op>= y
x = x <op> y

Take a look at these examples:

Augmented Assignment		Standard Assignment
`a += 5`	is equivalent to	`a = a + 5`
`a /= 10`	is equivalent to	`a = a / 10`
`a ^= b`	is equivalent to	`a = a ^ b`

Conclusion

In this tutorial, you learned about the diverse operators Python supports to combine objects into expressions.

Most of the examples you have seen so far have involved only simple atomic data, but you saw a brief introduction to the string data type. The next tutorial will explore string objects in much more detail.

Get Notified: Don't miss the follow up to this tutorial—Click here to join the Real Python Newsletter and you'll know when the next installment comes out.

« Variables in Python

Operators and Expressions in Python

Strings in Python »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Ultimate List of Python YouTube Channels

2018-06-18T14:00:00+00:00

We couldn’t find a good, up-to-date list of Python developer or Python programming YouTube channels online.

Learning Python on YouTube is a viable option these days, and we’re excited about what this new medium can do for programming education.

There are some really good YouTube channels that focus on Python development out there, but we just couldn’t find a list that was both comprehensive and up-to-date. So we created our own with the best and most Pythonic YouTubers.

We initially wrote this list based on information we gathered by reading forum posts and searching YouTube for Python channels directly. We’ll continue to add to the list with your feedback. We plan to keep this list updated, so feel free to leave a comment at the end of the page or tweet at us if you think anything is missing or if you’d like to see your own YouTube Python tutorials added.

In order for a channel to be included in our list, it must:

Focus on Python tutorials
Not be brand-new (> 2,000 subscribers)
Be active (new videos are coming out) OR have an interesting archive with old content worth watching

Enjoy the Python goodness! 📹🐍

Al Sweigart

“Tons of sweet computer related tutorials and some other awesome videos too!”

Website: automatetheboringstuff.com
Twitter: @AlSweigart
Subscribe: YouTube

Anaconda Inc.

“With over 4.5 million users, Anaconda is the world’s most popular Python data science platform. Anaconda, Inc. continues to lead open source projects like Anaconda, NumPy and SciPy that form the foundation of modern data science. Anaconda’s flagship product, Anaconda Enterprise, allows organizations to secure, govern, scale and extend Anaconda to deliver actionable insights that drive businesses and industries forward.”

In addition to their company developed videos, including a fun lego-mation and Pyception short, Anaconda’s YouTube channel contains all the videos from AnacondaCon, a gathering of the passionate community of data scientists, IT professionals, analysts, developers, and business leaders all using the Anaconda distribution.

Website: anaconda.com
Twitter: @anacondainc
Subscribe: YouTube

Christian Thompson

“A little bit about me and a lot about Python programming for beginners. I am a middle and high school teacher who uses Python as my teaching language. I firmly believe anyone can (and should) learn to program a computer and that Python is the perfect language for doing so.”

Website: christianthompson.com
Twitter: @tokyoedtech
Subscribe: YouTube

Clever Programmer

“You can find awesome programming lessons here! Also, expect programming tips and tricks that will take your coding skills to the next level.”

Website: cleverprogrammer.com
Twitter: @cleverqazi
Subscribe: YouTube

CodingEntrepreneurs

“Coding for Entrepreneurs is a Programming Series for Non-Technical Founders. Learn Django, Python, APIs, Accepting Payments, Stripe, jQuery, Twitter Bootstrap, and much more.”

Website: www.codingforentrepreneurs.com
Twitter: @joincfe
Subscribe: YouTube

Corey Schafer

“This channel is focused on creating tutorials and walkthroughs for software developers, programmers, and engineers. We cover topics for all different skill levels, so whether you are a beginner or have many years of experience, this channel will have something for you.”

Website: coreyms.com
Twitter: @CoreyMSchafer
Subscribe: YouTube ⋅ Patreon

Chris Hawkes

“We’re going to learn about programming, web design, responsive web design, Reactjs, Django, Python, web scraping, games, forms applications and more!”

Twitter: @realchrishawkes
Subscribe: YouTube

CS Dojo

“Hey everyone! My name is YK, and I make videos mostly about programming and computer science here.”

Website: csdojo.io
Twitter: @ykdojo
Subscribe: YouTube

Data School (Kevin Markham)

“You’re trying to launch your career in data science, and I want to help you reach that goal! My in-depth tutorials will help you to master crucial data science topics using open source tools like Python and R.”

Website: www.dataschool.io
Twitter: @justmarkham
Subscribe: YouTube ⋅ Patreon

David Beazley

“An archive of David Beazley’s conference, user group, and training talks.”

Website: www.dabeaz.com
Twitter: @dabeaz
Subscribe: YouTube

Enthought

“For more than 15 years, Enthought has built AI solutions with science and engineering at the core. We accelerate digital transformation by enabling companies and their people to leverage the benefits of Artificial Intelligence and Machine Learning.”

Additionally, Enthought is best known for the early development, maintenance, and continued support of SciPy, as well as the primary sponsor for the SciPy US and EuroSciPy Conferences. In addition to the company developed content, this channel provides all the video recordings from the SciPy US and EuroScipy (before 2016) Conferences, talks and tutorials specifically focused on the advancement of scientific computing through open source Python software for mathematics, science, and engineering.

Website: enthought.com
Twitter: @enthought
Subscribe: YouTube

Michael Kennedy (Talk Python)

“Videos, demos, and lectures about programming - especially Python and web topics.”

Website: talkpython.fm
Twitter: @mkennedy
Subscribe: YouTube ⋅ Podcast

Practical Python

“Expect unexpected.”

Subscribe: YouTube

PrettyPrinted

“I’m Anthony. I make programming videos.”

Website: prettyprinted.com
Twitter: @pretty_printed
Subscribe: YouTube

PyCon Session Recordings

These are all PyCon talk and session recordings made available on YouTube. There’s no single channel that combines these. Instead, you’ll access each year’s videos on a separate “PyCon 20…” channel. Alternatively, you can use PyVideo.org to watch the session recordings.

Website: www.pycon.org
Twitter: @PyCon
Subscribe: YouTube ⋅ PyVideo.org

PyData

“PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.”

Website: pydata.org
Twitter: @PyData
Subscribe: YouTube

Python Training by Dan Bader

“Python tutorials and training videos for Pythonistas that go beyond the basics. On this channel you’ll get new Python videos and screencasts every week. They’re bite-sized and to the point so you can fit them in with your day and pick up new Python skills on the side.”

Website: dbader.org ⋅ realpython.com
Twitter: @dbader_org ⋅ @realpython
Subscribe: YouTube

Sentdex (Harrison Kinsley)

“Python Programming tutorials, going further than just the basics. Learn about machine learning, finance, data analysis, robotics, web development, game development and more.”

Website: pythonprogramming.net
Twitter: @sentdex
Subscribe: YouTube ⋅ Patreon

Siraj Raval

“I’m Siraj. I’m on a warpath to inspire and educate developers to build Artificial Intelligence. Games, music, chatbots, art, i’ll teach you how to make it all yourself.”

Twitter: @sirajraval
Subscribe: YouTube ⋅ Patreon

Socratica

“Socratica makes high-quality educational videos on math and science. New videos every week! We’re a couple of Caltech grads who believe you deserve better videos. You’ll learn more with us!”

Website: www.socratica.com
Twitter: @socratica
Subscribe: YouTube ⋅ Patreon

TheNewBoston (Bucky Roberts)

“Tons of sweet computer related tutorials and some other awesome videos too!”

Website: thenewboston.com
Twitter: @bucky_roberts
Subscribe: YouTube ⋅ Patreon

Smaller Python Conferences

The following channels provide the tutorials, talks, and session recordings from many of the smaller local Python conferences held throughout the world.

Though on their own, most of these channels do not meet the requirement for 2000 subscribers, we list them here as honorable mentions because they represent the diverse Python community throughout the world.

Note that some (maybe older) videos from these conferences are available (together with other non-Python content) on the Next Day Video and Engineers.SG channels. Alternatively, PyVideo.org can serve as a one-stop-shop where you can find most (but not all) of these session recordings.

If you think anything is missing from this list or if you’d like to see your own Python YouTube channel added, then please leave a comment below or tweet at us.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Variables in Python

2018-06-12T14:00:00+00:00

In the previous tutorial on Basic Data Types in Python, you saw how values of various Python data types can be created. But so far, all the values shown have been literal or constant values:

>>> print(5.3)
5.3

If you’re writing more complex code, your program will need data that can change as program execution proceeds.

Here’s what you’ll learn in this tutorial: You will learn how every item of data in a Python program can be described by the abstract term object, and you’ll learn how to manipulate objects using symbolic names called variables.

Free PDF Download: Python 3 Cheat Sheet

Variable Assignment

Think of a variable as a name attached to a particular object. In Python, variables need not be declared or defined in advance, as is the case in many other programming languages. To create a variable, you just assign it a value and then start using it. Assignment is done with a single equals sign (=):

>>> n = 300

This is read or interpreted as “n is assigned the value 300.” Once this is done, n can be used in a statement or expression, and its value will be substituted:

>>> print(n)
300

Just as a literal value can be displayed directly from the interpreter prompt in a REPL session without the need for print(), so can a variable:

>>> n
300

Later, if you change the value of n and use it again, the new value will be substituted instead:

>>> n = 1000
>>> print(n)
1000
>>> n
1000

Python also allows chained assignment, which makes it possible to assign the same value to several variables simultaneously:

>>> a = b = c = 300
>>> print(a, b, c)
300 300 300

The chained assignment above assigns 300 to the variables a, b, and c simultaneously.

Variable Types in Python

In many programming languages, variables are statically typed. That means a variable is initially declared to have a specific data type, and any value assigned to it during its lifetime must always have that type.

Variables in Python are not subject to this restriction. In Python, a variable may be assigned a value of one type and then later re-assigned a value of a different type:

>>> var = 23.5
>>> print(var)
23.5

>>> var = "Now I'm a string"
>>> print(var)
Now I'm a string

Object References

What is actually happening when you make a variable assignment? This is an important question in Python, because the answer differs somewhat from what you’d find in many other programming languages.

Python is a highly object-oriented language. In fact, virtually every item of data in a Python program is an object of a specific type or class. (This point will be reiterated many times over the course of these tutorials.)

Consider this code:

>>> print(300)
300

When presented with the statement print(300), the interpreter does the following:

Creates an integer object
Gives it the value 300
Displays it to the console

You can see that an integer object is created using the built-in type() function:

>>> type(300)
<class 'int'>

A Python variable is a symbolic name that is a reference or pointer to an object. Once an object is assigned to a variable, you can refer to the object by that name. But the data itself is still contained within the object.

For example:

>>> n = 300

This assignment creates an integer object with the value 300 and assigns the variable n to point to that object.

Variable Assignment

The following code verifies that n points to an integer object:

>>> print(n)
300
>>> type(n)
<class 'int'>

Now consider the following statement:

>>> m = n

What happens when it is executed? Python does not create another object. It simply creates a new symbolic name or reference, m, which points to the same object that n points to.

Multiple References to a Single Object

Next, suppose you do this:

>>> m = 400

Now Python creates a new integer object with the value 400, and m becomes a reference to it.

References to Separate Objects

Lastly, suppose this statement is executed next:

>>> n = "foo"

Now Python creates a string object with the value "foo" and makes n reference that.

Orphaned Object

There is no longer any reference to the integer object 300. It is orphaned, and there is no way to access it.

Tutorials in this series will occasionally refer to the lifetime of an object. An object’s life begins when it is created, at which time at least one reference to it is created. During an object’s lifetime, additional references to it may be created, as you saw above, and references to it may be deleted as well. An object stays alive, as it were, so long as there is at least one reference to it.

When the number of references to an object drops to zero, it is no longer accessible. At that point, its lifetime is over. Python will eventually notice that it is inaccessible and reclaim the allocated memory so it can be used for something else. In computer lingo, this process is referred to as garbage collection.

Object Identity

In Python, every object that is created is given a number that uniquely identifies it. It is guaranteed that no two objects will have the same identifier during any period in which their lifetimes overlap. Once an object’s reference count drops to zero and it is garbage collected, as happened to the 300 object above, then its identifying number becomes available and may be used again.

The built-in Python function id() returns an object’s integer identifier. Using the id() function, you can verify that two variables indeed point to the same object:

>>> n = 300
>>> m = n
>>> id(n)
60127840
>>> id(m)
60127840

>>> m = 400
>>> id(m)
60127872

After the assignment m = n, m and n both point to the same object, confirmed by the fact that id(m) and id(n) return the same number. Once m is reassigned to 400, m and n point to different objects with different identities.

Deep Dive: Caching Small Integer Values

From what you now know about variable assignment and object references in Python, the following probably won’t surprise you:
>>> m = 300
>>> n = 300
>>> id(m)
60062304
>>> id(n)
60062896
With the statement m = 300, Python creates an integer object with the value 300 and sets m as a reference to it. n is then similarly assigned to an integer object with value 300—but not the same object. Thus, they have different identities, which you can verify from the values returned by id().

But consider this:
>>> m = 30
>>> n = 30
>>> id(m)
1405569120
>>> id(n)
1405569120
Here, m and n are separately assigned to integer objects having value 30. But in this case, id(m) and id(n) are identical!

For purposes of optimization, the interpreter creates objects for the integers in the range [-5, 256] at startup, and then reuses them during program execution. Thus, when you assign separate variables to an integer value in this range, they will actually reference the same object.

Variable Names

The examples you have seen so far have used short, terse variable names like m and n. But variable names can be more verbose. In fact, it is usually beneficial if they are because it makes the purpose of the variable more evident at first glance.

Officially, variable names in Python can be any length and can consist of uppercase and lowercase letters (A-Z, a-z), digits (0-9), and the underscore character (_). An additional restriction is that, although a variable name can contain digits, the first character of a variable name cannot be a digit.

Note: One of the additions to Python 3 was full Unicode support, which allows for Unicode characters in a variable name as well. You will learn about Unicode in greater depth in a future tutorial.

For example, all of the following are valid variable names:

>>> name = "Bob"
>>> Age = 54
>>> has_W2 = True
>>> print(name, Age, has_W2)
Bob 54 True

But this one is not, because a variable name can’t begin with a digit:

>>> 1099_filed = False
SyntaxError: invalid token

Note that case is significant. Lowercase and uppercase letters are not the same. Use of the underscore character is significant as well. Each of the following defines a different variable:

>>> age = 1
>>> Age = 2
>>> aGe = 3
>>> AGE = 4
>>> a_g_e = 5
>>> _age = 6
>>> age_ = 7
>>> _AGE_ = 8

>>> print(age, Age, aGe, AGE, a_g_e, _age, age_, _AGE_)
1 2 3 4 5 6 7 8

There is nothing stopping you from creating two different variables in the same program called age and Age, or for that matter agE. But it is probably ill-advised. It would certainly be likely to confuse anyone trying to read your code, and even you yourself, after you’d been away from it awhile.

It is worthwhile to give a variable a name that is descriptive enough to make clear what it is being used for. For example, suppose you are tallying the number of people who have graduated college. You could conceivably choose any of the following:

>>> numberofcollegegraduates = 2500
>>> NUMBEROFCOLLEGEGRADUATES = 2500
>>> numberOfCollegeGraduates = 2500
>>> NumberOfCollegeGraduates = 2500
>>> number_of_college_graduates = 2500

>>> print(numberofcollegegraduates, NUMBEROFCOLLEGEGRADUATES,
... numberOfCollegeGraduates, NumberOfCollegeGraduates,
... number_of_college_graduates)
2500 2500 2500 2500 2500

All of them are probably better choices than n, or ncg, or the like. At least you can tell from the name what the value of the variable is supposed to represent.

On the other hand, they aren’t all necessarily equally legible. As with many things, it is a matter of personal preference, but most people would find the first two examples, where the letters are all shoved together, to be harder to read, particularly the one in all capital letters. The most commonly used methods of constructing a multi-word variable name are the last three examples:

Camel Case: Second and subsequent words are capitalized, to make word boundaries easier to see. (Presumably, it struck someone at some point that the capital letters strewn throughout the variable name vaguely resemble camel humps.)
- Example: numberOfCollegeGraduates
Pascal Case: Identical to Camel Case, except the first word is also capitalized.
- Example: NumberOfCollegeGraduates
Snake Case: Words are separated by underscores.
- Example: number_of_college_graduates

Programmers debate hotly, with surprising fervor, which of these is preferable. Decent arguments can be made for all of them. Use whichever of the three is most visually appealing to you. Pick one and use it consistently.

You will see later that variables aren’t the only things that can be given names. You can also name functions, classes, modules, and so on. The rules that apply to variable names also apply to identifiers, the more general term for names given to program objects.

The Style Guide for Python Code, also known as PEP 8, contains Naming Conventions that list suggested standards for names of different object types. PEP 8 includes the following recommendations:

Snake Case should be used for functions and variable names.
Pascal Case should be used for class names. (PEP 8 refers to this as the “CapWords” convention.)

Reserved Words (Keywords)

There is one more restriction on identifier names. The Python language reserves a small set of keywords that designate special language functionality. No object can have the same name as a reserved word.

In Python 3.6, there are 33 reserved keywords:

Python Keywords
`False`	`def`	`if`	`raise`
`None`	`del`	`import`	`return`
`True`	`elif`	`in`	`try`
`and`	`else`	`is`	`while`
`as`	`except`	`lambda`	`with`
`assert`	`finally`	`nonlocal`	`yield`
`break`	`for`	`not`
`class`	`from`	`or`
`continue`	`global`	`pass`

You can see this list any time by typing help("keywords") to the Python interpreter. Reserved words are case-sensitive and must be used exactly as shown. They are all entirely lowercase, except for False, None, and True.

Trying to create a variable with the same name as any reserved word results in an error:

>>> for = 3
SyntaxError: invalid syntax

Conclusion

This tutorial covered the basics of Python variables, including object references and identity, and naming of Python identifiers.

You now have a good understanding of some of Python’s data types and know how to create variables that reference objects of those types.

Next, you will see how to combine data objects into expressions involving various operations.

« Basic Data Types in Python

Variables in Python

Operators and Expressions in Python »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Building and Documenting Python REST APIs With Flask and Connexion

2018-06-11T14:00:00+00:00

If you’re writing a web application, then you’re probably thinking about making HTTP calls to your server to get data to populate the dynamic parts of your application.

The goal of this article is to show you how to use Python 3, Flask, and Connexion to build useful REST APIs that can include input and output validation, and provide Swagger documentation as a bonus. Also included is a simple but useful single page web application that demonstrates using the API with JavaScript and updating the DOM with it.

The REST API you’ll be building will serve a simple people data structure where the people are keyed to the last name, and any updates are marked with a new timestamp.

This data could be represented in a database, saved in a file, or be accessible through some network protocol, but for us an in-memory data structure works fine. One of the purposes of an API is to decouple the data from the application that uses it, hiding the data implementation details.

Free Bonus: Click here to download a copy of the "REST in a Nutshell" Guide with a hands-on introduction to REST API principles and examples.

What REST Is

Before you get started creating an API server, I want to talk about REST a little bit. There’s lots of information about REST available through some quick Google searches, and my intent is not to duplicate that here.

I just want to share how I view REST and use it to build APIs. I see REST as a set of conventions taking advantage of the HTTP protocol to provide CRUD (Create, Read, Update, and Delete) behavior on things and collections of those things. The CRUD behavior maps nicely to the HTTP protocol method verbs like this:

Action	HTTP Verb	Description
Create	`POST`	Create a new, unique thing
Read	`GET`	Read the information about a thing or collection of things
Update	`PUT`	Update the information about an existing thing
Delete	`DELETE`	Delete a thing

You can perform these actions on a thing or resource. It’s useful to think about a resource as something a noun can be applied to: a person, an order for something, a person’s address. This idea of a resource lines up nicely with a URL (Unique Resource Locator).

A URL should identify a unique resource on the web, something that will work with the same thing over and over again for the same URL. With these two ideas, we have a way to take common application actions on something uniquely identified on the web. This is a very useful way of working.

This idea goes a long way towards creating clean APIs and performing the actions many applications want from the API.

What REST Is Not

Because REST is useful and helps map out how we might want to interact with a web based API, it’s sometimes used for things that don’t really fit well. There are lots of cases where what we want to do is perform some work or take an action directly. An example might be performing a string substitution, which granted is a silly thing to make an API for, but let’s go with it for now. Here’s a URL that might be constructed to provide this:

/api/substituteString/<string>/<search_string>/<sub_string>

Here, string is the string to make the substitution in, search_string is the string to replace, and sub_string is the string to replace search_string with. This can certainly be tied to some code in the server that does the intended work. But this has some problems in REST terms.

One problem that comes to mind is that this URL doesn’t point to a unique resource, so what it returns is entirely dependent on the path variable sections. Also, there is no concept of CRUD against this URL. It really only needs one HTTP method, and the method name conveys nothing about the action to take.

As an API, it’s not great either. The meaning of the path variables is dependent on their position in the URL. This could be addressed by changing the URL to use a query string instead, like this:

/api/substituteString?string=<string>&search_string=<search_string>&sub_string=<sub_string>

But the URL portion /api/substituteString isn’t a thing (noun) or collection of things at all: it’s an action (verb).

This doesn’t fit well in the REST conventions, and trying to force it to do so makes for a poor API. What the above really represents is an RPC, or Remote Procedure Call. Thinking about this as an RPC makes much more sense.

The beauty is that both REST and RPC conventions can coexist in an API without any problems! By keeping in mind the intention of each, you can use both to compliment each other in your API design. There are many situations where it would be useful to perform CRUD operations on something. There are also many situations where it would be useful to take an action with a thing (as a parameter) but not necessarily affect the thing itself.

The People REST API

For our example program, you’re going to create a REST API providing access to a collection of people with CRUD access to an individual person within that collection. Here’s the API design for the people collection:

Action	HTTP Verb	URL Path	Description
Create	`POST`	`/api/people`	Defines a unique URL to create a new person
Read	`GET`	`/api/people`	Defines a unique URL to read a collection of people
Read	`GET`	`/api/people/Farrell`	Defines a unique URL to read a particular person in the people collection
Update	`PUT`	`/api/people/Farrell`	Defines a unique URL to update an existing order
Delete	`DELETE`	`/api/orders/Farrell`	Defines a unique URL to delete an existing person

Getting Started

First, you’ll create a simple web server using the Flask Micro Framework. To get started, create a directory where you can create the code. You should also work in a virtualenv so you can install modules later on, which you’ll need to do. In addition, create a templates directory.

The Python code below gets a very basic web server up and running, and responding with Hello World for a request for the home page:

from flask import (
    Flask,
    render_template
)

# Create the application instance
app = Flask(__name__, template_folder="templates")

# Create a URL route in our application for "/"
@app.route('/')
def home():
    """
    This function just responds to the browser ULR
    localhost:5000/

    :return:        the rendered template 'home.html'
    """
    return render_template('home.html')

# If we're running in stand alone mode, run the application
if __name__ == '__main__':
    app.run(debug=True)

You should also create a home.html in the templates folder, as this is what will be served to a browser when navigating to the URL '/'. Here is the contents of the home.html file:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Application Home Page</title>
</head>
<body>
    <h2>
        Hello World!
    </h2>
</body>
</html>

You’ll notice the HTML file is named home.html rather than index.html. This is intentional because having an index.html file in the templates directory causes problems once you import the Connexion module in your program.

You can run your application with this command line in the directory containing the server.py file with the Python VirtualEnv active:

python server.py

When you run this application, a web server will start on port 5000, which is the default port used by Flask. If you open a browser and navigate to localhost:5000, you should see Hello World! displayed. Right now, this is useful to see the web server is running. You’ll extend the home.html file later to become a full single page web application using the REST API you’re developing.

In your Python program, you’ve imported the Flask module, giving the application access to the Flask functionality. You then created a Flask application instance, the app variable. Next, you connected the URL route '/' to the home() function by decorating it with @app.route('/'). This function calls the Flask render_template() function to get the home.html file from the templates directory and return it to the browser.

All of the example code is available from a link provided at the end of this article.

Using Connexion to Add a REST API Endpoint

Now that you’ve got a working web server, let’s add a REST API endpoint. To do this, you’ll use the Connexion module, which is installed using pip:

$ pip install connexion

This makes the Connexion module available to your program. The Connexion module allows a Python program to use the Swagger specification. This provides a lot of functionality: validation of input and output data to and from your API, an easy way to configure the API URL endpoints and the parameters expected, and a really nice UI interface to work with the created API and explore it.

All of this can happen when you create a configuration file your application can access. The Swagger site even provides an online configuration editor tool to help create and/or syntax check the configuration file you will create.

Adding Connexion to the Server

There are two parts to adding a REST API URL endpoint to your application with Connexion. You’ll add Connexion to the server and create a configuration file it will use. Modify your Python program like this to add Connexion to the server:

from flask import render_template
import connexion

# Create the application instance
app = connexion.App(__name__, specification_dir='./')

# Read the swagger.yml file to configure the endpoints
app.add_api('swagger.yml')

# Create a URL route in our application for "/"
@app.route('/')
def home():
    """
    This function just responds to the browser ULR
    localhost:5000/
    :return:        the rendered template 'home.html'
    """
    return render_template('home.html')

# If we're running in stand alone mode, run the application
if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000, debug=True)

You’ve added a couple of things to incorporate Connexion into the server. The import connexion statement adds the module to the program. The next step is creating the application instance using Connexion rather than Flask. Internally, the Flask app is still created, but it now has additional functionality added to it.

Part of the app instance creation includes the parameter specification_dir. This informs Connexion what directory to look in for its configuration file, in this case our current directory. Right after this, you’ve added the line:

app.add_api('swagger.yml')

This tells the app instance to read the file swagger.yml from the specification directory and configure the system to provide the Connexion functionality.

The Swagger Configuration File

The file swagger.yml is a YAML or JSON file containing all of the information necessary to configure your server to provide input parameter validation, output response data validation, URL endpoint definition, and the Swagger UI. Here is the swagger.yml file defining the GET /api/people endpoint your REST API will provide:

swagger: "2.0"
info:
  description: This is the swagger file that goes with our server code
  version: "1.0.0"
  title: Swagger REST Article
consumes:
  - "application/json"
produces:
  - "application/json"

basePath: "/api"

# Paths supported by the server application
paths:
  /people:
    get:
      operationId: "people.read"
      tags:
        - "People"
      summary: "The people data structure supported by the server application"
      description: "Read the list of people"
      responses:
        200:
          description: "Successful read people list operation"
          schema:
            type: "array"
            items:
              properties:
                fname:
                  type: "string"
                lname:
                  type: "string"
                timestamp:
                  type: "string"

This file is organized in a hierarchical manner: the indentation levels represent a level of ownership, or scope.

For example, paths defines the beginning of where all the API URL endpoints are defined. The /people value indented under that defines the start of where all the /api/people URL endpoints will be defined. The get: indented under that defines the section of definitions associated with an HTTP GET request to the /api/people URL endpoint. This goes on for the entire configuration.

Here are the meanings of the fields in this section of the swagger.yml file:

This section is part of the global configuration information:

swagger: tells Connexion what version of the Swagger API is being used
info: begins a new ‘scope’ of information about the API being built
description: a user defined description of what the API provides or is about. This is in the Connexion generated UI system
version: a user defined version value for the API
title: a user defined title included in the Connexion generated UI system
consumes: tells Connexion what MIME type is expected by the API.
produces: tells Connexion what content type is expected by the caller of the API.
basePath: "/api" defines the root of the API, kind of like a namespace the REST API will come from.

This section begins the configuration of the API URL endpoint paths:

paths: defines the section of the configuration containing all of the API REST endpoints.
/people: defines one path of your URL endpoint.
get: defines the HTTP method this URL endpoint will respond to. Together with the previous definitions, this creates the GET /api/people URL endpoint.

This section begins the configuration of the single /api/people URL endpoint:

operationId: "people.read" defines the Python import path/function that will respond to an HTTP GET /api/people request. The "people.read" portion can go as deep as you need to in order to connect a function to the HTTP request. Something like "<package_name>.<package_name>.<package_name>.<function_name>" would work just as well. You’ll create this shortly.
tags: defines a grouping for the UI interface. All the CRUD methods you’ll define for the people endpoint will share this tag definition.
summary defines the UI interface display text for this endpoint.
description: defines what the UI interface will display for implementation notes.

This section defines the section of the configuration of a successful response from the URL endpoint:

response: defines the beginning of the expected response section.
200: defines the section for a successful response, HTTP status code 200.
description: defines the UI interface display text for a response of 200.
schema: defines the response as a schema, or structure.
type: defines the structure of the schema as an array.
items: starts the definition of the items in the array.
properties: defines the items in the array as objects having key/value pairs.
fname: defines the first key of the object.
type: defines the value associated with fname as a string.
lname: defines the second key of the object.
type: defines the value associated with lname as a string.
timestamp: defines the third key of the object.
type: defines the value associated with timestamp as a string.

Handler for People Endpoint

In the swagger.yml file, you configured Connexion with the operationId value to call the people module and the read function within the module when the API gets an HTTP request for GET /api/people. This means a people.py module must exist and contain a read() function. Here is the people.py module you will create:

from datetime import datetime

def get_timestamp():
    return datetime.now().strftime(("%Y-%m-%d %H:%M:%S"))

# Data to serve with our API
PEOPLE = {
    "Farrell": {
        "fname": "Doug",
        "lname": "Farrell",
        "timestamp": get_timestamp()
    },
    "Brockman": {
        "fname": "Kent",
        "lname": "Brockman",
        "timestamp": get_timestamp()
    },
    "Easter": {
        "fname": "Bunny",
        "lname": "Easter",
        "timestamp": get_timestamp()
    }
}

# Create a handler for our read (GET) people
def read():
    """
    This function responds to a request for /api/people
    with the complete lists of people

    :return:        sorted list of people
    """
    # Create the list of people from our data
    return [PEOPLE[key] for key in sorted(PEOPLE.keys())]

In this code, you’ve created a helper function called get_timestamp() that generates a string representation of the current timestamp. This is used to create your in-memory structure and modify the data when you start modifying it with the API.

You then created the PEOPLE dictionary data structure, which is a simple names database, keyed on the last name. This is a module variable, so its state persists between REST API calls. In a real application, the PEOPLE data would exist in a database, file, or network resource, something that persists the data beyond running/stopping the web application.

Then you created the read() function. This is called when an HTTP request to GET /api/people is received by the server. The return value of this function is converted to a JSON string (recall the produces: definition in the swagger.yml file). The read() function you created builds and returns a list of people sorted by last name.

Running your server code and navigating in a browser to localhost:5000/api/people will display the list of people on screen:

[
  {
    "fname": "Kent",
    "lname": "Brockman",
    "timestamp": "2018-05-10 18:12:42"
  },
  {
    "fname": "Bunny",
    "lname": "Easter",
    "timestamp": "2018-05-10 18:12:42"
  },
  {
    "fname": "Doug",
    "lname": "Farrell",
    "timestamp": "2018-05-10 18:12:42"
  }
]

Congratulations, you’ve created a nice API and are on your way to building out a complete one!

The Swagger UI

Now you have a simple web API running with a single URL endpoint. At this point, it would be reasonable to think, “configuring this with the swagger.yml file was cool and all, but what did it get me?”

You’d be right to think that. We haven’t taken advantage of the input or output validation. All that swagger.yml gave us was a definition for the code path connected to the URL endpoint. However, what you also get for the extra work is the creation of the Swagger UI for your API.

If you navigate to localhost:5000/api/ui, the system will bring up a page that looks something like this:

This is the initial Swagger interface and shows the list of URL endpoints supported at our localhost:5000/api endpoint. This is built automatically by Connexion when it parses the swagger.yml file.

If you click on the /people endpoint in the interface, the interface will expand to show a great deal more information about your API and should look something like this:

This displays the structure of the expected response, the content-type of that response, and the description text you entered about the endpoint in the swagger.yml file.

You can even try the endpoint out by clicking the Try It Out! button at the bottom of the screen. That will further expand the interface to look something like this:

This can be extremely useful when the API is complete as it gives you and your API users a way to explore and experiment with the API without having to write any code to do so.

Building an API this way is very useful to me at work. Not only is the Swagger UI useful as a way to experiment with the API and read the provided documentation, but it’s also dynamic. Any time the configuration file changes, the Swagger UI changes as well.

In addition, the configuration offers a nice, clean way to think about and create the API URL endpoints. I know from experience that APIs can develop in a sometimes random manner over time, making finding the code that supports the endpoints, and coordinating them, difficult at best.

By separating the code from the API URL endpoint configuration, we decouple one from the other. This alone has been very useful to me in my work building API systems supporting single page web applications.

Building Out the Complete API

Our original goal was to build out an API providing full CRUD access to our people structure. As you recall, the definition of our API looks like this:

Action	HTTP Verb	URL Path	Description
Create	`POST`	`/api/people`	Defines a unique URL to create a new person
Read	`GET`	`/api/people`	Defines a unique URL to read a collection of people
Read	`GET`	`/api/people/Farrell`	Defines a unique URL to read a particular person in the people collection
Update	`PUT`	`/api/people/Farrell`	Defines a unique URL to update an existing order
Delete	`DELETE`	`/api/orders/Farrell`	Defines a unique URL to delete an existing person

To achieve this, you’ll extend both the swagger.yml and people.py files to fully support the API defined above. For the sake of brevity, only a link will be provided for both files:

The Swagger UI Completed

Once you’ve updated the swagger.yml and people.py files to complete the people API functionality, the Swagger UI system will update accordingly and look something like this:

This UI allows you to see all of the documentation you’ve included in the swagger.yml file and interact with all of the URL endpoints making up the CRUD functionality of the people interface.

Demonstration Single-Page Application

You’ve got a working REST API with a great Swagger UI documentation/interaction system. But what do you do with it now? The next step is to create a web application demonstrating the use of the API in a semi-practical manner.

You’ll create a web application that displays the people on screen as well as allows the user to create new people, update existing people, and delete people. This will all be handled by AJAX calls from JavaScript to the people API URL endpoints.

To begin, you need to extend the home.html file to look like this:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Application Home Page</title>
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/normalize/8.0.0/normalize.min.css">
    <link rel="stylesheet" href="static/css/home.css">
    <script
      src="http://code.jquery.com/jquery-3.3.1.min.js"
      integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8="
      crossorigin="anonymous">
    </script>
</head>
<body>
    <div class="container">
        <h1 class="banner">People Demo Application</h1>
        <div class="section editor">
            <label for="fname">First Name
                <input id="fname" type="text" />
            </label>
            <br />
            <label for="lname">Last Name
                <input id="lname" type="text" />
            </label>
            <br />
            <button id="create">Create</button>
            <button id="update">Update</button>
            <button id="delete">Delete</button>
            <button id="reset">Reset</button>
        </div>
        <div class="people">
            <table>
                <caption>People</caption>
                <thead>
                    <tr>
                        <th>First Name</th>
                        <th>Last Name</th>
                        <th>Update Time</th>
                    </tr>
                </thead>
                <tbody>
                </tbody>
            </table>
        </div>
        <div class="error">
        </div>
    </div>
</body>
<script src="static/js/home.js"></script>
</html>

The above HTML code extends the home.html file to pull in the external normalize.min.css file, which is a CSS reset file to normalize the formatting of elements across browsers.

It also pulls in the external jquery-3.3.1.min.js file to provide the jQuery functionality you’ll use to create the single-page application interactivity.

The HTML code above creates the static framework of the application. The dynamic parts appearing in the table structure will be added by JavaScript at load time and as the user interacts with the application.

Static Files

In the home.html file you’ve created, there are references to two static files: static/css/home.css and static/js/home.js. To add these, you’ll need to add the following directory structure to the application:

static/
│
├── css/
│   └── home.css
│
└── js/
    └── home.js

Because a directory named static will automatically be served by the Flask application, any files you place in the css and js folders will be available to the home.html file. For the sake of brevity, here are links to the home.css and home.js files:

JavaScript File

As was mentioned, the JavaScript file provides all the interaction with and updates to the web application. It does this by breaking up the necessary functionality into three parts by using the MVC (Model / View / Controller) design pattern.

Each object is created by a self-invoking function returning its own API for use by the other pieces. For instance, the Controller is passed the instances of the Model and View as parameters in its instantiation. It interacts with those objects through their exposed API methods.

The only other connection is between the Model and Controller by means of custom events on the AJAX method calls:

The Model provides the connection to the people API. Its own API is what the Controller calls to interact with the server when a user interaction event requires it.
The View provides the mechanism to update the web application DOM. Its own API is what the Controller calls to update the DOM when a user interaction event requires it.
The Controller provides all the event handlers for user interaction with the web application. This allows it to make calls to the Model to make requests to the people API, and to the View to update the DOM with new information received from the people API.

It also handles the custom events generated by the asynchronous AJAX requests made by the Model to the people API.

Here is a diagram of the MVC structure used in the home.js file:

The idea is that the Controller has a strong link to both the Model and the View. The Model has a weak link (the custom events) to the Controller and no connection to the View at all. The weak link from the Model to the Controller reduces coupling and dependence, which is useful in this case.

Demo Application

When created, the web application will look like this in the browser:

The Create button allows the user to create a new person in the people structure on the server. When you enter a First and Last Name and hit Create, the Controller calls the Model to make a request to the POST /api/people URL endpoint. This will verify that the last name doesn’t already exist. If it doesn’t, it will create a new person in the people structure.

This generates a custom event in the Model that causes the Controller to call the Model again to request a GET /api/people, which will return the complete list of people sorted. The Controller then passes that onto the View to redraw the table of people.

Double clicking on any row in the table will populate the First and Last Name fields in the editor section of the application. At this point, the user can either update or delete the person.

To update successfully, the user must change something about the First Name. The Last Name must remain the same as it’s the lookup key for the person to update. When Update is clicked, the Controller calls the Model to make a request to the PUT /api/people/{lname} URL endpoint. This will verify that the Last Name does currently exist. If so, it will update that person in the people structure.

To delete successfully, the user need only click Delete. When Delete is clicked, the Controller calls the Model to make a request to the DELETE /api/people/{lname} URL endpoint. This will verify that the last name does currently exist. If so, it will delete that person from the people structure.

Try making intentional errors in the editor, like misspelling a Last Name, and see the errors generated by the API represented on the web application.

Example Code

All of the example code for this article is available here. There are four versions of the code, each in a version_# directory, where # ranges from 1 to 4. The four versions correspond to the article sections in this manner:

version_1: This version contains the initial web server that serves up the home.html file.
version_2: This version contains the web server with Connexion added and the first people API URL endpoint.
version_3: This version contains the completed people API with all supported URL endpoints.
version_4: This version contains the completed API and a single page web application to demonstrate it.

Free Bonus: Click here to download a copy of the "REST in a Nutshell" Guide with a hands-on introduction to REST API principles and examples.

Conclusion

In this tutorial, you saw how relatively easy it is to create a comprehensive REST API with Python. With the Connexion module and some additional configuration work, a useful documentation and interactive system can be put in place, making your API a much more enjoyable experience for your users to interface with and understand.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Basic Data Types in Python

2018-06-05T14:00:00+00:00

Now you know how to interact with the Python interpreter and execute Python code. It’s time to dig into the Python language. First up is a discussion of the basic data types that are built into Python.

Here’s what you’ll learn in this tutorial:

You’ll learn about several basic numeric, string, and Boolean types that are built into Python. By the end of this tutorial, you’ll be familiar with what objects of these types look like, and how to represent them.
You’ll also get an overview of Python’s built-in functions. These are pre-written chunks of code you can call to do useful things. You have already seen the built-in print() function, but there are many others.

Free PDF Download: Python 3 Cheat Sheet

Integers

In Python 3, there is effectively no limit to how long an integer value can be. Of course, it is constrained by the amount of memory your system has, as are all things, but beyond that an integer can be as long as you need it to be:

>>> print(123123123123123123123123123123123123123123123123 + 1)
123123123123123123123123123123123123123123123124

Python interprets a sequence of decimal digits without any prefix to be a decimal number:

>>> print(10)
10

The following strings can be prepended to an integer value to indicate a base other than 10:

Prefix	Interpretation	Base
`0b` (zero + lowercase letter `'b'`) `0B` (zero + uppercase letter `'B'`)	Binary	2
`0o` (zero + lowercase letter `'o'`) `0O` (zero + uppercase letter `'O'`)	Octal	8
`0x` (zero + lowercase letter `'x'`) `0X` (zero + uppercase letter `'X'`)	Hexadecimal	16

For example:

>>> print(0o10)
8

>>> print(0x10)
16

>>> print(0b10)
2

For more information on integer values with non-decimal bases, see the following Wikipedia sites: Binary, Octal, and Hexadecimal.

The underlying type of a Python integer, irrespective of the base used to specify it, is called int:

>>> type(10)
<class 'int'>
>>> type(0o10)
<class 'int'>
>>> type(0x10)
<class 'int'>

Note: This is a good time to mention that if you want to display a value while in a REPL session, you don’t need to use the print() function. Just typing the value at the >>> prompt and hitting Enter will display it:

>>> 10
10
>>> 0x10
16
>>> 0b10
2

Many of the examples in this tutorial series will use this feature.

Note that this does not work inside a script file. A value appearing on a line by itself in a script file will not do anything.

Floating-Point Numbers

The float type in Python designates a floating-point number. float values are specified with a decimal point. Optionally, the character e or E followed by a positive or negative integer may be appended to specify scientific notation:

>>> 4.2
4.2
>>> type(4.2)
<class 'float'>
>>> 4.
4.0
>>> .2
0.2

>>> .4e7
4000000.0
>>> type(.4e7)
<class 'float'>
>>> 4.2e-4
0.00042

Deep Dive: Floating-Point Representation

The following is a bit more in-depth information on how Python represents floating-point numbers internally. You can readily use floating-point numbers in Python without understanding them to this level, so don’t worry if this seems overly complicated. The information is presented here in case you are curious.

Almost all platforms represent Python float values as 64-bit “double-precision” values, according to the IEEE 754 standard. In that case, the maximum value a floating-point number can have is approximately 1.8 ⨉ 10³⁰⁸. Python will indicate a number greater than that by the string inf:
>>> 1.79e308
1.79e+308
>>> 1.8e308
inf
The closest a nonzero number can be to zero is approximately 5.0 ⨉ 10^-324. Anything closer to zero than that is effectively zero:
>>> 5e-324
5e-324
>>> 1e-325
0.0
Floating point numbers are represented internally as binary (base-2) fractions. Most decimal fractions cannot be represented exactly as binary fractions, so in most cases the internal representation of a floating-point number is an approximation of the actual value. In practice, the difference between the actual value and the represented value is very small and should not usually cause significant problems.

Further Reading: For additional information on floating-point representation in Python and the potential pitfalls involved, see Floating Point Arithmetic: Issues and Limitations in the Python documentation.

Complex Numbers

Complex numbers are specified as <real part>+<imaginary part>j. For example:

>>> 2+3j
(2+3j)
>>> type(2+3j)
<class 'complex'>

Strings

Strings are sequences of character data. The string type in Python is called str.

String literals may be delimited using either single or double quotes. All the characters between the opening delimiter and matching closing delimiter are part of the string:

>>> print("I am a string.")
I am a string.
>>> type("I am a string.")
<class 'str'>

>>> print('I am too.')
I am too.
>>> type('I am too.')
<class 'str'>

A string in Python can contain as many characters as you wish. The only limit is your machine’s memory resources. A string can also be empty:

>>> ''
''

What if you want to include a quote character as part of the string itself? Your first impulse might be to try something like this:

>>> print('This string contains a single quote (') character.')
SyntaxError: invalid syntax

As you can see, that doesn’t work so well. The string in this example opens with a single quote, so Python assumes the next single quote, the one in parentheses which was intended to be part of the string, is the closing delimiter. The final single quote is then a stray and causes the syntax error shown.

If you want to include either type of quote character within the string, the simplest way is to delimit the string with the other type. If a string is to contain a single quote, delimit it with double quotes and vice versa:

>>> print("This string contains a single quote (') character.")
This string contains a single quote (') character.

>>> print('This string contains a double quote (") character.')
This string contains a double quote (") character.

Escape Sequences in Strings

Sometimes, you want Python to interpret a character or sequence of characters within a string differently. This may occur in one of two ways:

You may want to suppress the special interpretation that certain characters are usually given within a string.
You may want to apply special interpretation to characters in a string which would normally be taken literally.

You can accomplish this using a backslash (\) character. A backslash character in a string indicates that one or more characters that follow it should be treated specially. (This is referred to as an escape sequence, because the backslash causes the subsequent character sequence to “escape” its usual meaning.)

Let’s see how this works.

Suppressing Special Character Meaning

You have already seen the problems you can come up against when you try to include quote characters in a string. If a string is delimited by single quotes, you can’t directly specify a single quote character as part of the string because, for that string, the single quote has special meaning—it terminates the string:

>>> print('This string contains a single quote (') character.')
SyntaxError: invalid syntax

Specifying a backslash in front of the quote character in a string “escapes” it and causes Python to suppress its usual special meaning. It is then interpreted simply as a literal single quote character:

>>> print('This string contains a single quote (\') character.')
This string contains a single quote (') character.

The same works in a string delimited by double quotes as well:

>>> print("This string contains a double quote (\") character.")
This string contains a double quote (") character.

The following is a table of escape sequences which cause Python to suppress the usual special interpretation of a character in a string:

Escape Sequence	Usual Interpretation of Character(s) After Backslash	“Escaped” Interpretation
`\'`	Terminates string with single quote opening delimiter	Literal single quote (`'`) character
`\"`	Terminates string with double quote opening delimiter	Literal double quote (`"`) character
`\newline`	Terminates input line	Newline is ignored
`\\`	Introduces escape sequence	Literal backslash (`\`) character

Ordinarily, a newline character terminates line input. So pressing Enter in the middle of a string will cause Python to think it is incomplete:

>>> print('a

SyntaxError: EOL while scanning string literal

To break up a string over more than one line, include a backslash before each newline, and the newlines will be ignored:

>>> print('a\
... b\
... c')
abc

To include a literal backslash in a string, escape it with a backslash:

>>> print('foo\\bar')
foo\bar

Applying Special Meaning to Characters

Next, suppose you need to create a string that contains a tab character in it. Some text editors may allow you to insert a tab character directly into your code. But many programmers consider that poor practice, for several reasons:

The computer can distinguish between a tab character and a sequence of space characters, but you can’t. To a human reading the code, tab and space characters are visually indistinguishable.
Some text editors are configured to automatically eliminate tab characters by expanding them to the appropriate number of spaces.
Some Python REPL environments will not insert tabs into code.

In Python (and almost all other common computer languages), a tab character can be specified by the escape sequence \t:

>>> print('foo\tbar')
foo     bar

The escape sequence \t causes the t character to lose its usual meaning, that of a literal t. Instead, the combination is interpreted as a tab character.

Here is a list of escape sequences that cause Python to apply special meaning instead of interpreting literally:

Escape Sequence	“Escaped” Interpretation
`\a`	ASCII Bell (`BEL`) character
`\b`	ASCII Backspace (`BS`) character
`\f`	ASCII Formfeed (`FF`) character
`\n`	ASCII Linefeed (`LF`) character
`\N{<name>}`	Character from Unicode database with given `<name>`
`\r`	ASCII Carriage Return (`CR`) character
`\t`	ASCII Horizontal Tab (`TAB`) character
`\uxxxx`	Unicode character with 16-bit hex value `xxxx`
`\Uxxxxxxxx`	Unicode character with 32-bit hex value `xxxxxxxx`
`\v`	ASCII Vertical Tab (`VT`) character
`\ooo`	Character with octal value `ooo`
`\xhh`	Character with hex value `hh`

Examples:

>>> print("a\tb")
a    b
>>> print("a\141\x61")
aaa
>>> print("a\nb")
a
b
>>> print('\u2192 \N{rightwards arrow}')
→ →

This type of escape sequence is typically used to insert characters that are not readily generated from the keyboard or are not easily readable or printable.

Raw Strings

A raw string literal is preceded by r or R, which specifies that escape sequences in the associated string are not translated. The backslash character is left in the string:

>>> print('foo\nbar')
foo
bar
>>> print(r'foo\nbar')
foo\nbar

>>> print('foo\\bar')
foo\bar
>>> print(R'foo\\bar')
foo\\bar

Triple-Quoted Strings

There is yet another way of delimiting strings in Python. Triple-quoted strings are delimited by matching groups of three single quotes or three double quotes. Escape sequences still work in triple-quoted strings, but single quotes, double quotes, and newlines can be included without escaping them. This provides a convenient way to create a string with both single and double quotes in it:

>>> print('''This string has a single (') and a double (") quote.''')
This string has a single (') and a double (") quote.

Because newlines can be included without escaping them, this also allows for multiline strings:

>>> print("""This is a
string that spans
across several lines""")
This is a
string that spans
across several lines

You will see in the upcoming tutorial on Python Program Structure how triple-quoted strings can be used to add an explanatory comment to Python code.

Boolean Type, Boolean Context, and “Truthiness”

Python 3 provides a Boolean data type. Objects of Boolean type may have one of two values, True or False:

>>> type(True)
<class 'bool'>
>>> type(False)
<class 'bool'>

As you will see in upcoming tutorials, expressions in Python are often evaluated in Boolean context, meaning they are interpreted to represent truth or falsehood. A value that is true in Boolean context is sometimes said to be “truthy,” and one that is false in Boolean context is said to be “falsy.” (You may also see “falsy” spelled “falsey.”)

The “truthiness” of an object of Boolean type is self-evident: Boolean objects that are equal to True are truthy (true), and those equal to False are falsy (false). But non-Boolean objects can be evaluated in Boolean context as well and determined to be true or false.

You will learn more about evaluation of objects in Boolean context when you encounter logical operators in the upcoming tutorial on operators and expressions in Python.

Built-In Functions

The Python interpreter supports many functions that are built-in: sixty-eight, as of Python 3.6. You will cover many of these in the following discussions, as they come up in context.

For now, a brief overview follows, just to give a feel for what is available. See the Python documentation on built-in functions for more detail. Many of the following descriptions refer to topics and concepts that will be discussed in future tutorials.

Math

Function	Description
`abs()`	Returns absolute value of a number
`divmod()`	Returns quotient and remainder of integer division
`max()`	Returns the largest of the given arguments or items in an iterable
`min()`	Returns the smallest of the given arguments or items in an iterable
`pow()`	Raises a number to a power
`round()`	Rounds a floating-point value
`sum()`	Sums the items of an iterable

Type Conversion

Function	Description
`ascii()`	Returns a string containing a printable representation of an object
`bin()`	Converts an integer to a binary string
`bool()`	Converts an argument to a Boolean value
`chr()`	Returns string representation of character given by integer argument
`complex()`	Returns a complex number constructed from arguments
`float()`	Returns a floating-point object constructed from a number or string
`hex()`	Converts an integer to a hexadecimal string
`int()`	Returns an integer object constructed from a number or string
`oct()`	Converts an integer to an octal string
`ord()`	Returns integer representation of a character
`repr()`	Returns a string containing a printable representation of an object
`str()`	Returns a string version of an object
`type()`	Returns the type of an object or creates a new type object

Iterables and Iterators

Function	Description
`all()`	Returns `True` if all elements of an iterable are true
`any()`	Returns `True` if any elements of an iterable are true
`enumerate()`	Returns a list of tuples containing indices and values from an iterable
`filter()`	Filters elements from an iterable
`iter()`	Returns an iterator object
`len()`	Returns the length of an object
`map()`	Applies a function to every item of an iterable
`next()`	Retrieves the next item from an iterator
`range()`	Generates a range of integer values
`reversed()`	Returns a reverse iterator
`slice()`	Returns a `slice` object
`sorted()`	Returns a sorted list from an iterable
`zip()`	Creates an iterator that aggregates elements from iterables

Composite Data Type

Function	Description
`bytearray()`	Creates and returns an object of the `bytearray` class
`bytes()`	Creates and returns a `bytes` object (similar to `bytearray`, but immutable)
`dict()`	Creates a `dict` object
`frozenset()`	Creates a `frozenset` object
`list()`	Constructs a `list` object
`object()`	Returns a new featureless object
`set()`	Creates a `set` object
`tuple()`	Creates a `tuple` object

Classes, Attributes, and Inheritance

Function	Description
`classmethod()`	Returns a class method for a function
`delattr()`	Deletes an attribute from an object
`getattr()`	Returns the value of a named attribute of an object
`hasattr()`	Returns `True` if an object has a given attribute
`isinstance()`	Determines whether an object is an instance of a given class
`issubclass()`	Determines whether a class is a subclass of a given class
`property()`	Returns a property value of a class
`setattr()`	Sets the value of a named attribute of an object
`super()`	Returns a proxy object that delegates method calls to a parent or sibling class

Input/Output

Function	Description
`format()`	Converts a value to a formatted representation
`input()`	Reads input from the console
`open()`	Opens a file and returns a file object
`print()`	Prints to a text stream or the console

Variables, References, and Scope

Function	Description
`dir()`	Returns a list of names in current local scope or a list of object attributes
`globals()`	Returns a dictionary representing the current global symbol table
`id()`	Returns the identity of an object
`locals()`	Updates and returns a dictionary representing current local symbol table
`vars()`	Returns `__dict__` attribute for a module, class, or object

Miscellaneous

Function	Description
`callable()`	Returns `True` if object appears callable
`compile()`	Compiles source into a code or AST object
`eval()`	Evaluates a Python expression
`exec()`	Implements dynamic execution of Python code
`hash()`	Returns the hash value of an object
`help()`	Invokes the built-in help system
`memoryview()`	Returns a memory view object
`staticmethod()`	Returns a static method for a function
`__import__()`	Invoked by the `import` statement

Conclusion

In this tutorial, you learned about the built-in data types and functions Python provides.

The examples given so far have all manipulated and displayed only constant values. In most programs, you are usually going to want to create objects that change in value as the program executes.

Head to the next tutorial to learn about Python variables.

« Interacting with Python

Basic Data Types in Python

Variables in Python »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Application Layouts: A Reference

2018-06-04T14:00:00+00:00

Python, though opinionated on syntax and style, is surprisingly flexible when it comes to structuring your applications.

On the one hand, this flexibility is great: it allows different use cases to use structures that are necessary for those use cases. On the other hand, though, it can be very confusing to the new developer.

The Internet isn’t a lot of help either—there are as many opinions as there are Python blogs. In this article, I want to give you a dependable Python application layout reference guide that you can refer to for the vast majority of your use cases.

You’ll see examples of common Python application structures, including command-line applications (CLI apps), one-off scripts, installable packages, and web application layouts with popular frameworks like Flask and Django.

Note: This reference guide assumes a working knowledge of Python modules and packages. Check out our introduction to Python modules and packages for a refresher if you’re feeling a little rusty.

Command-Line Application Layouts

A lot of us work primarily with Python applications that are run via command-line interfaces (CLIs). This is where you often start with a blank canvas, and the flexibility of Python application layouts can be a real headache.

Starting with an empty project folder can be intimidating and lead to no shortage of coder’s block. In this section, I want to share some proven layouts that I personally use as a starting point for all of my Python CLI applications.

We’ll start with a very basic layout for a very basic use case: a simple script that runs on its own. You’ll then see how to build up the layout as the use cases advance.

One-Off Script

You just make a .py script, and it’s gravy, right? No need to install—just run the script in its directory!

Well, that’s fine if you’re just making a script for your own use, or one that doesn’t have any external dependencies, but what if you have to distribute it? Especially to a less tech-savvy user?

The following layout will work for all of these cases and can easily be modified to reflect whatever installation or other tools you use in your workflow. This layout will cover you whether you’re creating a pure Python script (that is, one with no dependencies) or using a tool like pip or Pipenv.

While you read this reference guide, keep in mind that the exact location of the files in the layout matters less than the reason they are placed where they are. All of these files should be in a project directory named after your project. For this example, we will use (what else?) helloworld as the project name and root directory.

Here’s the Python project structure I typically use for a CLI app:

helloworld/
│
├── .gitignore
├── helloworld.py
├── LICENSE
├── README.md
├── requirements.txt
├── setup.py
└── tests.py

This is pretty straightforward: everything is in the same directory. The files shown here are not necessarily exhaustive, but I recommend keeping the number of files to a minimum if you plan on using a basic layout like this. Some of these files will be new to you, so let’s take a quick look at what each of them does.

.gitignore: This is a file that tells Git which kinds of files to ignore, like IDE clutter or local configuration files. Our Git tutorial has all the details, and you can find sample .gitignore files for Python projects here.
helloworld.py: This is the script that you’re distributing. As far as naming the main script file goes, I recommend that you go with the name of your project (which is the same as the name of the top-level directory).
LICENSE: This plaintext file describes the license you’re using for a project. It’s always a good idea to have one if you’re distributing code. The filename is in all caps by convention.

Note: Need help selecting a license for your project? Check out ChooseALicense.
README.md: This is a Markdown (or reStructuredText) file documenting the purpose and usage of your application. Crafting a good README is an art, but you can find a shortcut to mastery here.
requirements.txt: This file defines outside Python dependencies and their versions for your application.
setup.py: This file can also be used to define dependencies, but it really shines for other work that needs to be done during installation. You can read more about both setup.py and requirements.txt in our guide to Pipenv.
tests.py: This script houses your tests, if you have any. You should have some.

But now that your application is growing, and you’ve broken it out into multiple pieces within the same package, should you keep all pieces in the top-level directory? Now that your application is more complex, it’s time to organize things more cleanly.

Installable Single Package

Let’s imagine that helloworld.py is still the main script to execute, but you’ve moved all helper methods to a new file called helpers.py.

We are going to package the helloworld Python files together but keep all the miscellaneous files, such as your README, .gitignore, and so on, at the top directory.

Let’s take a look at the updated structure:

helloworld/
│
├── helloworld/
│   ├── __init__.py
│   ├── helloworld.py
│   └── helpers.py
│
├── tests/
│   ├── helloworld_tests.py
│   └── helpers_tests.py
│
├── .gitignore
├── LICENSE
├── README.md
├── requirements.txt
└── setup.py

The only difference here is that your application code is now all held in the helloworld subdirectory—this directory is named after your package—and that we’ve added a file called __init__.py. Let’s introduce these new files:

helloworld/__init__.py: This file has many functions, but for our purposes it tells the Python interpreter that this directory is a package directory. You can set up this __init__.py file in a way that enables you to import classes and methods from the package as a whole, instead of knowing the internal module structure and importing from helloworld.helloworld or helloworld.helpers.

Note: For a deeper discussion on internal packages and __init__.py, our Python modules and packages overview has you covered.
helloworld/helpers.py: As mentioned above, we’ve moved much of helloworld.py’s business logic to this file. Thanks to __init__.py, outside modules will be able to access these helpers simply by importing from the helloworld package.
tests/: We’ve moved our tests into their own directory, a pattern you’ll continue to see as our program structures gain complexity. We have also split our tests into separate modules, mirroring our package’s structure.

This layout is a stripped down version of Kenneth Reitz’s samplemod application structure. It is another great starting point for your CLI applications, especially for more expansive projects.

Application with Internal Packages

In larger applications, you may have one or more internal packages that are either tied together with a main runner script or that provide specific functionality to a larger library you are packaging. We will extend the conventions laid out above to accommodate for this:

helloworld/
│
├── bin/
│
├── docs/
│   ├── hello.md
│   └── world.md
│
├── helloworld/
│   ├── __init__.py
│   ├── runner.py
│   ├── hello/
│   │   ├── __init__.py
│   │   ├── hello.py
│   │   └── helpers.py
│   │
│   └── world/
│       ├── __init__.py
│       ├── helpers.py
│       └── world.py
│
├── data/
│   ├── input.csv
│   └── output.xlsx
│
├── tests/
│   ├── hello
│   │   ├── helpers_tests.py
│   │   └── hello_tests.py
│   │
│   └── world/
│       ├── helpers_tests.py
│       └── world_tests.py
│
├── .gitignore
├── LICENSE
└── README.md

There’s a bit more to digest here, but as long as you remember that it follows from the previous layout, you will have an easier time following along. I’ll go through the additions and modifications in order, their uses, and the reasons you might want them.

bin/: This directory holds any executable files. I’ve adapted this from Jean-Paul Calderone’s classic structure post, and his prescriptions for the use of a bin/ directory are still important. The most important point to remember is that your executable shouldn’t have a lot of code, just an import and a call to a main function in your runner script. If you are using pure Python or don’t have any executable files, you can leave out this directory.
/docs: With a more advanced application, you’ll want to maintain good documentation of all its parts. I like to put any documentation for internal modules here, which is why you see separate documents for the hello and world packages. If you use docstrings in your internal modules (and you should!), your whole-module documentation should at the very least give a holistic view of the purpose and function of the module.
helloworld/: This is similar to helloworld/ in the previous structure, but now there are subdirectories. As you add more complexity, you’ll want to use a “divide and conquer” tactic and split out parts of your application logic into more manageable chunks. Remember that the directory name refers to the overall package name, and so the subdirectory names (hello/ and world/) should reflect their package names.
data/: Having this directory is helpful for testing. It’s a central location for any files that your application will ingest or produce. Depending on how you deploy your application, you can keep “production-level” inputs and outputs pointed to this directory, or only use it for internal testing.
tests/: Here, you can put all your tests—unit tests, execution tests, integration tests, and so on. Feel free to structure this directory in the most convenient way for your testing strategies, import strategies, and more. For a refresher on testing command-line applications with Python, check out my article 4 Techniques for Testing Python Command-Line (CLI) Apps.

The top-level files remain largely the same as in the previous layout. These three layouts should cover most use cases for command-line applications, and even GUI applications with the caveat that you may have to tinker with some things depending on the GUI framework you use.

Note: Keep in mind that these are just layouts. If a directory or file doesn’t make sense for your specific use case (like tests/ if you aren’t distributing tests with your code), feel free to leave it out. But try not to leave out docs/. It’s always a good idea to document your work.

Web Application Layouts

Another major use case of Python is web applications. Django and Flask are arguably the most popular web frameworks for Python and thankfully are a little more opinionated when it comes to application layout.

In order to make sure this article is a complete, full-fledged layout reference, I wanted to highlight the structure common to these frameworks.

Django

Let’s go in alphabetical order and start with Django. One of the nice things about Django is that it will create a project skeleton for you after running django-admin startproject project, where project is the name of your project. This will create a directory in your current working directory called project with the following internal structure:

project/
│
├── project/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
│
└── manage.py

This seems a little empty, doesn’t it? Where does all the logic go? The views? There aren’t even any tests!

In Django, this is a project, which ties together the other Django concept, apps. Apps are where logic, models, views, and so on all live, and in doing so they do some task, such as maintaining a blog.

Django apps can be imported into projects and used across projects, and are structured like specialized Python packages.

Like projects, Django makes generating Django app layouts really easy. After you set up your project, all you have to do is navigate to the location of manage.py and run python manage.py startapp app, where app is the name of your app.

This will result in a directory called app with the following layout:

app/
│
├── migrations/
│   └── __init__.py
│
├── __init__.py
├── admin.py
├── apps.py
├── models.py
├── tests.py
└── views.py

This can then be imported directly into your project. Details on what these files do, how to harness them for your project, and so forth are outside the scope of this reference, but you can get all that information and more in our Django tutorial and also in the official Django docs.

This file and folder structure is very barebones and the basic requirements for Django. For any open-source Django project, you can (and should) adapt the structures from the command-line application layouts. I typically end up with something like this in the outer project/ directory:

project/
│
├── app/
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   │
│   ├── migrations/
│   │   └── __init__.py
│   │
│   ├── models.py
│   ├── tests.py
│   └── views.py
│
├── docs/
│
├── project/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
│
├── static/
│   └── style.css
│
├── templates/
│   └── base.html
│
├── .gitignore
├── manage.py
├── LICENSE
└── README.md

For a deeper discussion on more advanced Django application layouts, this Stack Overflow thread has you covered. The django-project-skeleton project documentation explains some of the directories you will find in the Stack Overflow thread. A comprehensive dive into Django can be found in the pages of Two Scoops of Django, which will teach you all of the latest best practices for Django development.

For more Django tutorials, visit our Django section at Real Python.

Flask

Flask is a Python web “microframework.” One of the main selling points is that it is very quick to set up with minimal overhead. The Flask documentation has a web application example that’s under 10 lines of code and in a single script. Of course, in practice, it’s highly unlikely you’ll be writing a web application this small.

Luckily, the Flask documentation swoops in to save us with a suggested layout for their tutorial project (a blogging web application called Flaskr), and we will examine that here from within the main project directory:

flaskr/
│
├── flaskr/
│   ├── ___init__.py
│   ├── db.py
│   ├── schema.sql
│   ├── auth.py
│   ├── blog.py
│   ├── templates/
│   │   ├── base.html
│   │   ├── auth/
│   │   │   ├── login.html
│   │   │   └── register.html
│   │   │
│   │   └── blog/
│   │       ├── create.html
│   │       ├── index.html
│   │       └── update.html
│   │ 
│   └── static/
│       └── style.css
│
├── tests/
│   ├── conftest.py
│   ├── data.sql
│   ├── test_factory.py
│   ├── test_db.py
│   ├── test_auth.py
│   └── test_blog.py
│
├── venv/
│
├── .gitignore
├── setup.py
└── MANIFEST.in

From these contents, we can see that a Flask application, like most Python applications, is built around Python packages.

Note: Not seeing it? A quick tip for spotting packages is by looking for an __init__.py file. This sits in the highest-level directory for that particular package. In the above layout, flaskr is a package containing the db, auth, and blog modules.

In this layout, everything lives in the flaskr package except for your tests, a directory for your virtual environments, and your usual top-level files. As in other layouts, your tests will roughly match the individual modules residing within the flaskr package. Your templates also reside in the main project package, which would not happen with the Django layouts.

Be sure to also visit our Flask Boilerplate Github page for a view of a more fully fleshed-out Flask application and see the boilerplate in action here.

For more on Flask, check out all of our Flask tutorials here.

Conclusions and Reminders

Now you’ve seen example layouts for a number of different application types: one-off Python scripts, installable single packages, larger applications with internal packages, Django web applications, and Flask web applications.

Coming away from this guide, you will have the tools to successfully prevent coder’s block by building out your application structure so that you’re not staring at a blank canvas trying to figure out where to start.

Because Python is largely non-opinionated when it comes to application layouts, you can customize these example layouts to your heart’s content to better fit your use case.

I want you to not only have an application layout reference but also come away with the understanding that these examples are neither hard-and-fast rules nor the only way to structure your application. Over time and with practice, you’ll develop the ability to build and customize your own useful Python application layouts.

Did I miss a use case? Do you have another application structure philosophy? Did this article help prevent coder’s block? Let me know in the comments!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Itertools in Python 3, By Example

2018-05-30T14:00:00+00:00

It has been called a “gem” and “pretty much the coolest thing ever,” and if you have not heard of it, then you are missing out on one of the greatest corners of the Python 3 standard library: itertools.

A handful of excellent resources exist for learning what functions are available in the itertools module. The docs themselves are a great place to start. So is this post.

The thing about itertools, though, is that it is not enough to just know the definitions of the functions it contains. The real power lies in composing these functions to create fast, memory-efficient, and good-looking code.

This article takes a different approach. Rather than introducing itertools to you one function at a time, you will construct practical examples designed to encourage you to “think iteratively.” In general, the examples will start simple and gradually increase in complexity.

A word of warning: this article is long and intended for the intermediate-to-advanced Python programmer. Before diving in, you should be confident using iterators and generators in Python 3, multiple assignment, and tuple unpacking. If you aren’t, or if you need to brush up on your knowledge, consider checking out the following before reading on:

Python Iterators: A Step-By-Step Introduction
Introduction to Python Generators
Chapter 6 of Python Tricks: The Book by Dan Bader
Multiple assignment and tuple unpacking improve Python code readability

Free Bonus: Click here to get our itertools cheat sheet that summarizes the techniques demonstrated in this tutorial.

All set? Let’s start the way any good journey should—with a question.

What Is `Itertools` and Why Should You Use It?

According to the itertools docs, it is a “module [that] implements a number of iterator building blocks inspired by constructs from APL, Haskell, and SML… Together, they form an ‘iterator algebra’ making it possible to construct specialized tools succinctly and efficiently in pure Python.”

Loosely speaking, this means that the functions in itertools “operate” on iterators to produce more complex iterators. Consider, for example, the built-in zip() function, which takes any number of iterables as arguments and returns an iterator over tuples of their corresponding elements:

>>> list(zip([1, 2, 3], ['a', 'b', 'c']))
[(1, 'a'), (2, 'b'), (3, 'c')]

How, exactly, does zip() work?

[1, 2, 3] and ['a', 'b', 'c'], like all lists, are iterable, which means they can return their elements one at a time. Technically, any Python object that implements the .__iter__() or .__getitem__() methods is iterable. (See the Python 3 docs glossary for a more detailed explanation.)

The iter() built-in function, when called on an iterable, returns an iterator object for that iterable:

>>> iter([1, 2, 3, 4])
<list_iterator object at 0x7fa80af0d898>

Under the hood, the zip() function works, in essence, by calling iter() on each of its arguments, then advancing each iterator returned by iter() with next() and aggregating the results into tuples. The iterator returned by zip() iterates over these tuples.

The map() built-in function is another “iterator operator” that, in its simplest form, applies a single-parameter function to each element of an iterable one element at a time:

>>> list(map(len, ['abc', 'de', 'fghi']))
[3, 2, 4]

The map() function works by calling iter() on its second argument, advancing this iterator with next() until the iterator is exhausted, and applying the function passed to its first argument to the value returned by next() at each step. In the above example, len() is called on each element of ['abc', 'de', 'fghi'] to return an iterator over the lengths of each string in the list.

Since iterators are iterable, you can compose zip() and map() to produce an iterator over combinations of elements in more than one iterable. For example, the following sums corresponding elements of two lists:

>>> list(map(sum, zip([1, 2, 3], [4, 5, 6])))
[5, 7, 9]

This is what is meant by the functions in itertools forming an “iterator algebra.” itertools is best viewed as a collection of building blocks that can be combined to form specialized “data pipelines” like the one in the example above.

Historical Note: In Python 2, the built-in zip() and map() functions do not return an iterator, but rather a list. To return an iterator, the izip() and imap() functions of itertools must be used. In Python 3, izip() and imap() have been removed from itertools and replaced the zip() and map() built-ins. So, in a way, if you have ever used zip() or map() in Python 3, you have already been using itertools!

There are two main reasons why such an “iterator algebra” is useful: improved memory efficiency (via lazy evaluation) and faster execuction time. To see this, consider the following problem:

Given a list of values inputs and a positive integer n, write a function that splits inputs into groups of length n. For simplicity, assume that the length of the input list is divisible by n. For example, if inputs = [1, 2, 3, 4, 5, 6] and n = 2, your function should return [(1, 2), (3, 4), (5, 6)].

Taking a naive approach, you might write something like this:

def naive_grouper(inputs, n):
    num_groups = len(inputs) // n
    return [tuple(inputs[i*n:(i+1)*n]) for i in range(num_groups)]

When you test it, you see that it works as expected:

>>> nums = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> naive_grouper(nums, 2)
[(1, 2), (3, 4), (5, 6), (7, 8), (9, 10)]

What happens when you try to pass it a list with, say, 100 million elements? You will need a whole lot of available memory! Even if you have enough memory available, your program will hang for a while until the output list is populated.

To see this, store the following in a script called naive.py:

def naive_grouper(inputs, n):
    num_groups = len(inputs) // n
    return [tuple(inputs[i*n:(i+1)*n]) for i in range(num_groups)]


for _ in naive_grouper(range(100000000), 10):
    pass

From the console, you can use the time command (on UNIX systems) to measure memory usage and CPU user time. Make sure you have at least 5GB of free memory before executing the following:

$ time -f "Memory used (kB): %M\nUser time (seconds): %U" python3 naive.py
Memory used (kB): 4551872
User time (seconds): 11.04

Note: On Ubuntu, you may need to run /usr/bin/time instead of time for the above example to work.

The list and tuple implementation in naive_grouper() requires approximately 4.5GB of memory to process range(100000000). Working with iterators drastically improves this situation. Consider the following:

def better_grouper(inputs, n):
    iters = [iter(inputs)] * n
    return zip(*iters)

There’s a lot going on in this little function, so let’s break it down with a concrete example. The expression [iters(inputs)] * n creates a list of n references to the same iterator:

>>> nums = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> iters = [iter(nums)] * 2
>>> list(id(itr) for itr in iters)  # IDs are the same.
[139949748267160, 139949748267160]

Next, zip(*iters) returns an iterator over pairs of corresponding elements of each iterator in iters. When the first element, 1, is taken from the “first” iterator, the “second” iterator now starts at 2 since it is just a reference to the “first” iterator and has therefore been advanced one step. So, the first tuple produced by zip() is (1, 2).

At this point, “both” iterators in iters start at 3, so when zip() pulls 3 from the “first” iterator, it gets 4 from the “second” to produce the tuple (3, 4). This process continues until zip() finally produces (9, 10) and “both” iterators in iters are exhausted:

>>> nums = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> list(better_grouper(nums, 2))
[(1, 2), (3, 4), (5, 6), (7, 8), (9, 10)]

The better_grouper() function is better for a couple of reasons. First, without the reference to the len() built-in, better_grouper() can take any iterable as an argument (even infinite iterators). Second, by returning an iterator rather than a list, better_grouper() can process enormous iterables without trouble and uses much less memory.

Store the following in a file called better.py and run it with time from the console again:

def better_grouper(inputs, n):
    iters = [iter(inputs)] * n
    return zip(*iters)


for _ in better_grouper(range(100000000), 10):
    pass

$ time -f "Memory used (kB): %M\nUser time (seconds): %U" python3 better.py
Memory used (kB): 7224
User time (seconds): 2.48

That’s a whopping 630 times less memory used than naive.py in less than a quarter of the time!

Now that you’ve seen what itertools is (“iterator algebra”) and why you should use it (improved memory efficiency and faster execution time), let’s take a look at how to take better_grouper() to the next level with itertools.

The `grouper` Recipe

The problem with better_grouper() is that it doesn’t handle situations where the value passed to the second argument isn’t a factor of the length of the iterable in the first argument:

>>> nums = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> list(better_grouper(nums, 4))
[(1, 2, 3, 4), (5, 6, 7, 8)]

The elements 9 and 10 are missing from the grouped output. This happens because zip() stops aggregating elements once the shortest iterable passed to it is exhausted. It would make more sense to return a third group containing 9 and 10.

To do this, you can use itertools.zip_longest(). This function accepts any number of iterables as arguments and a fillvalue keyword argument that defaults to None. The easiest way to get a sense of the difference between zip() and zip_longest() is to look at some example output:

>>> import itertools as it
>>> x = [1, 2, 3, 4, 5]
>>> y = ['a', 'b', 'c']
>>> list(zip(x, y))
[(1, 'a'), (2, 'b'), (3, 'c')]
>>> list(it.zip_longest(x, y))
[(1, 'a'), (2, 'b'), (3, 'c'), (4, None), (5, None)]

With this in mind, replace zip() in better_grouper() with zip_longest():

import itertools as it


def grouper(inputs, n, fillvalue=None):
    iters = [iter(inputs)] * n
    return it.zip_longest(*iters, fillvalue=fillvalue)

Now you get a better result:

>>> nums = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> print(list(grouper(nums, 4)))
[(1, 2, 3, 4), (5, 6, 7, 8), (9, 10, None, None)]

The grouper() function can be found in the Recipes section of the itertools docs. The recipes are an excellent source of inspiration for ways to use itertools to your advantage.

Note: From this point forward, the line import itertools as it will not be included at the beginning of examples. All itertools methods in code examples are prefaced with it. The module import is implied.

If you get a NameError: name 'itertools' is not defined or a NameError: name 'it' is not defined exception when running one of the examples in this tutorial you’ll need to import the itertools module first.

Et tu, Brute Force?

Here’s a common interview-style problem:

You have three $20 dollar bills, five $10 dollar bills, two $5 dollar bills, and five $1 dollar bills. How many ways can you make change for a $100 dollar bill?

To “brute force” this problem, you just start listing off the ways there are to choose one bill from your wallet, check whether any of these makes change for $100, then list the ways to pick two bills from your wallet, check again, and so on and so forth.

But you are a programmer, so naturally you want to automate this process.

First, create a list of the bills you have in your wallet:

bills = [20, 20, 20, 10, 10, 10, 10, 10, 5, 5, 1, 1, 1, 1, 1]

A choice of k things from a set of n things is called a combination, and itertools has your back here. The itertools.combinations() function takes two arguments—an iterable inputs and a positive integer n—and produces an iterator over tuples of all combinations of n elements in inputs.

For example, to list the combinations of three bills in your wallet, just do:

>>> list(it.combinations(bills, 3))
 [(20, 20, 20), (20, 20, 10), (20, 20, 10), ... ]

To solve the problem, you can loop over the positive integers from 1 to len(bills), then check which combinations of each size add up to $100:

>>> makes_100 = []
>>> for n in range(1, len(bills) + 1):
...     for combination in it.combinations(bills, n):
...         if sum(combination) == 100:
...             makes_100.append(combination)

If you print out makes_100, you will notice there are a lot of repeated combinations. This makes sense because you can make change for $100 with three $20 dollar bills and four $10 bills, but combinations() does this with the first four $10 dollars bills in your wallet; the first, third, fourth and fifth $10 dollar bills; the first, second, fourth and fifth $10 bills; and so on.

To remove duplicates from makes_100, you can convert it to a set:

>>> set(makes_100)
{(20, 20, 10, 10, 10, 10, 10, 5, 1, 1, 1, 1, 1),
 (20, 20, 10, 10, 10, 10, 10, 5, 5),
 (20, 20, 20, 10, 10, 10, 5, 1, 1, 1, 1, 1),
 (20, 20, 20, 10, 10, 10, 5, 5),
 (20, 20, 20, 10, 10, 10, 10)}

So, there are five ways to make change for a $100 bill with the bills you have in your wallet.

Here’s a variation on the same problem:

How many ways are there to make change for a $100 bill using any number of $50, $20, $10, $5, and $1 dollar bills?

In this case, you don’t have a pre-set collection of bills, so you need a way to generate all possible combinations using any number of bills. For this, you’ll need the itertools.combinations_with_replacement() function.

It works just like combinations(), accepting an iterable inputs and a positive integer n, and returns an iterator over n-tuples of elements from inputs. The difference is that combinations_with_replacement() allows elements to be repeated in the tuples it returns.

For example:

>>> list(it.combinations_with_replacement([1, 2], 2))
[(1, 1), (1, 2), (2, 2)]

Compare that to combinations():

>>> list(it.combinations([1, 2], 2))
[(1, 2)]

Here’s what the solution to the revised problem looks like:

>>> bills = [50, 20, 10, 5, 1]
>>> make_100 = []
>>> for n in range(1, 101):
...     for combination in it.combinations_with_replacement(bills, n):
...         if sum(combination) == 100:
...             makes_100.append(combination)

In this case, you do not need to remove any duplicates since combinations_with_replacement() won’t produce any:

>>> len(makes_100)
343

If you run the above solution, you may notice that it takes a while for the output to display. That is because it has to process 96,560,645 combinations!

Another “brute force” itertools function is permutations(), which accepts a single iterable and produces all possible permutations (rearrangements) of its elements:

>>> list(it.permutations(['a', 'b', 'c']))
[('a', 'b', 'c'), ('a', 'c', 'b'), ('b', 'a', 'c'),
 ('b', 'c', 'a'), ('c', 'a', 'b'), ('c', 'b', 'a')]

Any iterable of three elements will have six permutations, and the number of permutations of longer iterables grows extremely fast. In fact, an iterable of length n has n! permutations, where

To put this in perspective, here’s a table of these numbers for n = 1 to n = 10:

n	n!
2	2
3	6
4	24
5	120
6	720
7	5,040
8	40,320
9	362,880
10	3,628,800

The phenomenon of just a few inputs producing a large number of outcomes is called a combinatorial explosion and is something to keep in mind when working with combinations(), combinations_with_replacement(), and permutations().

It is usually best to avoid brute force algorithms, although there are times you may need to use one (for example, if the correctness of the algorithm is critical, or every possible outcome must be considered). In that case, itertools has you covered.

Section Recap

In this section you met three itertools functions: combinations(), combinations_with_replacement(), and permutations().

Let’s review these functions before moving on:

`itertools.combinations` Example

combinations(iterable, n)

Return successive n-length combinations of elements in the iterable.

>>> combinations([1, 2, 3], 2)
(1, 2), (1, 3), (2, 3)

`itertools.combinations_with_replacement` Example

combinations_with_replacement(iterable, n)

Return successive n-length combinations of elements in the iterable allowing individual elements to have successive repeats.

>>> combinations_with_replacement([1, 2], 2)
(1, 1), (1, 2), (2, 2)

`itertools.permutations` Example

permutations(iterable, n=None)

Return successive n-length permutations of elements in the iterable.

>>> permutations('abc')
('a', 'b', 'c'), ('a', 'c', 'b'), ('b', 'a', 'c'),
('b', 'c', 'a'), ('c', 'a', 'b'), ('c', 'b', 'a')

Sequences of Numbers

With itertools, you can easily generate iterators over infinite sequences. In this section, you will explore numeric sequences, but the tools and techniques seen here are by no means limited to numbers.

Evens and Odds

For the first example, you will create a pair of iterators over even and odd integers without explicitly doing any arithmetic. Before diving in, let’s look at an arithmetic solution using generators:

>>> def evens():
...     """Generate even integers, starting with 0."""
...     n = 0
...     while True:
...         yield n
...         n += 2
...
>>> evens = evens()
>>> list(next(evens) for _ in range(5))
[0, 2, 4, 6, 8]

>>> def odds():
...     """Generate odd integers, starting with 1."""
...     n = 1
...     while True:
...         yield n
...         n += 2
...
>>> odds = odds()
>>> list(next(odds) for _ in range(5))
[1, 3, 5, 7, 9]

That is pretty straightforward, but with itertools you can do this much more compactly. The function you need is itertools.count(), which does exactly what it sounds like: it counts, starting by default with the number 0.

>>> counter = it.count()
>>> list(next(counter) for _ in range(5))
[0, 1, 2, 3, 4]

You can start counting from any number you like by setting the start keyword argument, which defaults to 0. You can even set a step keyword argument to determine the interval between numbers returned from count()—this defaults to 1.

With count(), iterators over even and odd integers become literal one-liners:

>>> evens = it.count(step=2)
>>> list(next(evens) for _ in range(5))
[0, 2, 4, 6, 8]

>>> odds = it.count(start=1, step=2)
>>> list(next(odds) for _ in range(5))
[1, 3, 5, 7, 9]

Ever since Python 3.1, the count() function also accepts non-integer arguments:

>>> count_with_floats = it.count(start=0.5, step=0.75)
>>> list(next(count_with_floats) for _ in range(5))
[0.5, 1.25, 2.0, 2.75, 3.5]

You can even pass it negative numbers:

>>> negative_count = it.count(start=-1, step=-0.5)
>>> list(next(negative_count) for _ in range(5))
[-1, -1.5, -2.0, -2.5, -3.0]

In some ways, count() is similar to the built-in range() function, but count() always returns an infinite sequence. You might wonder what good an infinite sequence is since it’s impossible to iterate over completely. That is a valid question, and I admit the first time I was introduced to infinite iterators, I too didn’t quite see the point.

The example that made me realize the power of the infinite iterator was the following, which emulates the behavior of the built-in enumerate() function:

>>> list(zip(it.count(), ['a', 'b', 'c']))
[(0, 'a'), (1, 'b'), (2, 'c')]

It is a simple example, but think about it: you just enumerated a list without a for loop and without knowing the length of the list ahead of time.

Recurrence Relations

A recurrence relation is a way of describing a sequence of numbers with a recursive formula. One of the best-known recurrence relations is the one that describes the Fibonacci sequence.

The Fibonacci sequence is the sequence 0, 1, 1, 2, 3, 5, 8, 13, .... It starts with 0 and 1, and each subsequent number in the sequence is the sum of the previous two. The numbers in this sequence are called the Fibonacci numbers. In mathematical notation, the recurrence relation describing the n-th Fibonacci number looks like this:

Note: If you search Google, you will find a host of implementations of these numbers in Python. You can find a recursive function that produces them in the Thinking Recursively in Python article here on Real Python.

It is common to see the Fibonacci sequence produced with a generator:

def fibs():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

The recurrence relation describing the Fibonacci numbers is called a second order recurrence relation because, to calculate the next number in the sequence, you need to look back two numbers behind it.

In general, second order recurrence relations have the form:

Here, P, Q, and R are constants. To generate the sequence, you need two initial values. For the Fibonacci numbers, P = Q = 1, R = 0, and the initial values are 0 and 1.

As you might guess, a first order recurrence relation has the following form:

There are countless sequences of numbers that can be described by first and second order recurrence relations. For example, the positive integers can be described as a first order recurrence relation with P = Q = 1 and initial value 1. For the even integers, take P = 1 and Q = 2 with initial value 0.

In this section, you will construct functions for producing any sequence whose values can be described with a first or second order recurrence relation.

First Order Recurrence Relations

You’ve already seen how count() can generate the sequence of non-negative integers, the even integers, and the odd integers. You can also use it to generate the sequence 3n = 0, 3, 6, 9, 12, … and 4n = 0, 4, 8, 12, 16, ….

count_by_three = it.count(step=3)
count_by_four = it.count(step=4)

In fact, count() can produce sequences of multiples of any number you wish. These sequences can be described with first-order recurrence relations. For example, to generate the sequence of multiples of some number n, just take P = 1, Q = n, and initial value 0.

Another easy example of a first-order recurrence relation is the constant sequence n, n, n, n, n…, where n is any value you’d like. For this sequence, set P = 1 and Q = 0 with initial value n. itertools provides an easy way to implement this sequence as well, with the repeat() function:

all_ones = it.repeat(1)  # 1, 1, 1, 1, ...
all_twos = it.repeat(2)  # 2, 2, 2, 2, ...

If you need a finite sequence of repeated values, you can set a stopping point by passing a positive integer as a second argument:

five_ones = it.repeat(1, 5)  # 1, 1, 1, 1, 1
three_fours = it.repeat(4, 3)  # 4, 4, 4

What may not be quite as obvious is that the sequence 1, -1, 1, -1, 1, -1, ... of alternating 1s and -1s can also be described by a first order recurrence relation. Just take P = -1, Q = 0, and initial value 1.

There’s an easy way to generate this sequence with the itertools.cycle() function. This function takes an iterable inputs as an argument and returns an infinite iterator over the values in inputs that returns to the beginning once the end of inputs is reached. So, to produce the alternating sequence of 1s and -1s, you could do this:

alternating_ones = it.cycle([1, -1])  # 1, -1, 1, -1, 1, -1, ...

The goal of this section, though, is to produce a single function that can generate any first order recurrence relation—just pass it P, Q, and an initial value. One way to do this is with itertools.accumulate().

The accumulate() function takes two arguments—an iterable inputs and a binary function func (that is, a function with exactly two inputs)—and returns an iterator over accumulated results of applying func to elements of inputs. It is roughly equivalent to the following generator:

def accumulate(inputs, func):
    itr = iter(inputs)
    prev = next(itr)
    for cur in itr:
        yield prev
        prev = func(prev, cur)

For example:

>>> import operator
>>> list(it.accumulate([1, 2, 3, 4, 5], operator.add))
[1, 3, 6, 10, 15]

The first value in the iterator returned by accumulate() is always the first value in the input sequence. In the above example, this is 1—the first value in [1, 2, 3, 4, 5].

The next value in the output iterator is the sum of the first two elements of the input sequence: add(1, 2) = 3. To produce the next value, accumulate() takes the result of add(1, 2) and adds this to the third value in the input sequence:

add(3, 3) = add(add(1, 2), 3) = 6

The fourth value produced by accumulate() is add(add(add(1, 2), 3), 4) = 10, and so on.

The second argument of accumulate() defaults to operator.add(), so the previous example can be simplified to:

>>> list(it.accumulate([1, 2, 3, 4, 5]))
[1, 3, 6, 10, 15]

Passing the built-in min() to accumulate() will keep track of a running minimum:

>>> list(it.accumulate([9, 21, 17, 5, 11, 12, 2, 6], min))
[9, 9, 9, 5, 5, 5, 2, 2]

More complex functions can be passed to accumulate() with lambda expressions:

>>> list(it.accumulate([1, 2, 3, 4, 5], lambda x, y: (x + y) / 2))
[1, 1.5, 2.25, 3.125, 4.0625]

The order of the arguments in the binary function passed to accumulate() is important. The first argument is always the previously accumulated result and the second argument is always the next element of the input iterable. For example, consider the difference in output of the following expressions:

>>> list(it.accumulate([1, 2, 3, 4, 5], lambda x, y: x - y))
[1, -1, -4, -8, -13]

>>> list(it.accumulate([1, 2, 3, 4, 5], lambda x, y: y - x))
[1, 1, 2, 2, 3]

To model a recurrence relation, you can just ignore the second argument of the binary function passed to accumulate(). That is, given values p, q, and s, lambda x, _: p*s + q will return the value following x in the recurrence relation defined by sᵢ = Psᵢ₋₁ + Q.

In order for accumulate() to iterate over the resulting recurrence relation, you need to pass to it an infinite sequence with the right initial value. It doesn’t matter what the rest of the values in the sequence are, as long as the initial value is the initial value of the recurrence relation. You can do this is with repeat():

def first_order(p, q, initial_val):
    """Return sequence defined by s(n) = p * s(n-1) + q."""
    return it.accumulate(it.repeat(initial_val), lambda s, _: p*s + q)

Using first_order(), you can build the sequences from above as follows:

>>> evens = first_order(p=1, q=2, initial_val=0)
>>> list(next(evens) for _ in range(5))
[0, 2, 4, 6, 8]

>>> odds = first_order(p=1, q=2, initial_val=1)
>>> list(next(odds) for _ in range(5))
[1, 3, 5, 7, 9]

>>> count_by_threes = first_order(p=1, q=3, initial_val=0)
>>> list(next(count_by_threes) for _ in range(5))
[0, 3, 6, 9, 12]

>>> count_by_fours = first_order(p=1, q=4, initial_val=0)
>>> list(next(count_by_fours) for _ in range(5))
[0, 4, 8, 12, 16]

>>> all_ones = first_order(p=1, q=0, initial_val=1)
>>> list(next(all_ones) for _ in range(5))
[1, 1, 1, 1, 1]

>>> all_twos = first_order(p=1, q=0, initial_val=2)
>>> list(next(all_twos) for _ in range(5))
[2, 2, 2, 2, 2]

>>> alternating_ones = first_order(p=-1, 0, initial_val=1)
>>> list(next(alternating_ones) for _ in range(5))
[1, -1, 1, -1, 1]

Second Order Recurrence Relations

Generating sequences described by second order recurrence relations, like the Fibonacci sequence, can be accomplished using a similar technique as the one used for first order recurrence relations.

The difference here is that you need to create an intermediate sequence of tuples that keep track of the previous two elements of the sequence, and then map() each of these tuples to their first component to get the final sequence.

Here’s what it looks like:

def second_order(p, q, r, initial_values):
    """Return sequence defined by s(n) = p * s(n-1) + q * s(n-2) + r."""
    intermediate = it.accumulate(
        it.repeat(initial_values),
        lambda s, _: (s[1], p*s[1] + q*s[0] + r)
    )
    return map(lambda x: x[0], intermediate)

Using second_order(), you can generate the Fibonacci sequence like this:

>>> fibs = second_order(p=1, q=1, r=0, initial_values=(0, 1))
>>> list(next(fibs) for _ in range(8))
[0, 1, 1, 2, 3, 5, 8, 13]

Other sequences can be easily generated by changing the values of p, q, and r. For example, the Pell numbers and the Lucas numbers can be generated as follows:

pell = second_order(p=2, q=1, r=0, initial_values=(0, 1))
>>> list(next(pell) for _ in range(6))
[0, 1, 2, 5, 12, 29]

>>> lucas = second_order(p=1, q=1, r=0, initial_values=(2, 1))
>>> list(next(lucas) for _ in range(6))
[2, 1, 3, 4, 7, 11]

You can even generate the alternating Fibonacci numbers:

>>> alt_fibs = second_order(p=-1, q=1, r=0, initial_values=(-1, 1))
>>> list(next(alt_fibs) for _ in range(6))
[-1, 1, -2, 3, -5, 8]

This is all really cool if you are a giant math nerd like I am, but step back for a second and compare second_order() to the fibs() generator from the beginning of this section. Which one is easier to understand?

This is a valuable lesson. The accumulate() function is a powerful tool to have in your toolkit, but there are times when using it could mean sacrificing clarity and readability.

Section Recap

You saw several itertools function in this section. Let’s review those now.

`itertools.count` Example

count(start=0, step=1)

Return a count object whose .__next__() method returns consecutive values.

>>> count()
0, 1, 2, 3, 4, ...

>>> count(start=1, step=2)
1, 3, 5, 7, 9, ...

`itertools.repeat` Example

repeat(object, times=1)

Create an iterator which returns the object for the specified number of times. If not specified, returns the object endlessly.

>>> repeat(2)
2, 2, 2, 2, 2 ...

>>> repeat(2, 5)  # Stops after 5 repititions.
2, 2, 2, 2, 2

`itertools.cycle` Example

cycle(iterable)

Return elements from the iterable until it is exhausted. Then repeat the sequence indefinitely.

>>> cycle(['a', 'b', 'c'])
a, b, c, a, b, c, a, ...

`itertools accumulate` Example

accumulate(iterable, func=operator.add)

Return series of accumulated sums (or other binary function results).

>>> accumulate([1, 2, 3])
1, 3, 6

Alright, let’s take a break from the math and have some fun with cards.

Dealing a Deck of Cards

Suppose you are building a Poker app. You’ll need a deck of cards. You might start by defining a list of ranks (ace, king, queen, jack, 10, 9, and so on) and a list of suits (hearts, diamonds, clubs, and spades):

ranks = ['A', 'K', 'Q', 'J', '10', '9', '8', '7', '6', '5', '4', '3', '2']
suits = ['H', 'D', 'C', 'S']

You could represent a card as a tuple whose first element is a rank and second element is a suit. A deck of cards would be a collection of such tuples. The deck should act like the real thing, so it makes sense to define a generator that yields cards one at a time and becomes exhausted once all the cards are dealt.

One way to achieve this is to write a generator with a nested for loop over ranks and suits:

def cards():
    """Return a generator that yields playing cards."""
    for rank in ranks:
        for suit in suits:
            yield rank, suit

You could write this more compactly with a generator expression:

cards = ((rank, suit) for rank in ranks for suit in suits)

However, some might argue that this is actually more difficult to understand than the more explicit nested for loop.

It helps to view nested for loops from a mathematical standpoint—that is, as a Cartesian product of two or more iterables. In mathematics, the Cartesian product of two sets A and B is the set of all tuples of the form (a, b) where a is an element of A and b is an element of B.

Here’s an example with Python iterables: the Cartesian product of A = [1, 2] and B = ['a', 'b'] is [(1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')].

The itertools.product() function is for exactly this situation. It takes any number of iterables as arguments and returns an iterator over tuples in the Cartesian product:

it.product([1, 2], ['a', 'b'])  # (1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')

The product() function is by no means limited to two iterables. You can pass it as many as you like—they don’t even have to all be of the same size! See if you can predict what product([1, 2, 3], ['a', 'b'], ['c']) is, then check your work by running it in the interpreter.

Warning: The product() function is another “brute force” function and can lead to a combinatorial explosion if you aren’t careful.

Using product(), you can re-write the cards in a single line:

cards = it.product(ranks, suits)

This is all fine and dandy, but any Poker app worth its salt better start with a shuffled deck:

import random

def shuffle(deck):
    """Return iterator over shuffled deck."""
    deck = list(deck)
    random.shuffle(deck)
    return iter(tuple(deck))

cards = shuffle(cards)

Note: The random.shuffle() function uses the Fisher-Yates shuffle to shuffle a list (or any mutable sequence) in place in O(n) time. This algorithm is well-suited for shuffling cards because it produces an unbiased permutation—that is, all permutations of the iterable are equally likely to be returned by random.shuffle().

That said, you probably noticed that shuffle() creates a copy of its input deck in memory by calling list(deck). While this seemingly goes against the spirit of this article, this author is unaware of a good way to shuffle an iterator without making a copy.

As a courtesy to your users, you would like to give them the opportunity to cut the deck. If you imagine the cards being stacked neatly on a table, you have the user pick a number n and then remove the first n cards from the top of the stack and move them to the bottom.

If you know a thing or two about slicing, you might accomplish this like so:

def cut(deck, n):
    """Return an iterator over a deck of cards cut at index `n`."""
    if n < 0:
        raise ValueError('`n` must be a non-negative integer')

    deck = list(deck)
    return iter(deck[n:] + deck[:n])

cards = cut(cards, 26)  # Cut the deck in half.

The cut() function first converts deck to a list so that you can slice it to make the cut. To guarantee your slices behave as expected, you’ve got to check that n is non-negative. If it isn’t, you better throw an exception so that nothing crazy happens.

Cutting the deck is pretty straightforward: the top of the cut deck is just deck[:n], and the bottom is the remaining cards, or deck[n:]. To construct the new deck with the top “half” moved to the bottom, you just append it to the bottom: deck[n:] + deck[:n].

The cut() function is pretty simple, but it suffers from a couple of problems. When you slice a list, you make a copy of the original list and return a new list with the selected elements. With a deck of only 52 cards, this increase in space complexity is trivial, but you could reduce the memory overhead using itertools. To do this, you’ll need three functions: itertools.tee(), itertools.islice(), and itertools.chain().

Let’s take a look at how those functions work.

The tee() function can be used to create any number of independent iterators from a single iterable. It takes two arguments: the first is an iterable inputs, and the second is the number n of independent iterators over inputs to return (by default, n is set to 2). The iterators are returned in a tuple of length n.

>>> iterator1, iterator2 = it.tee([1, 2, 3, 4, 5], 2)
>>> list(iterator1)
[1, 2, 3, 4, 5]
>>> list(iterator1)  # iterator1 is now exhausted.
[]
>>> list(iterator2)  # iterator2 works independently of iterator1
[1, 2, 3, 4, 5].

While tee() is useful for creating independent iterators, it is important to understand a little bit about how it works under the hood. When you call tee() to create n independent iterators, each iterator is essentially working with its own FIFO queue.

When a value is extracted from one iterator, that value is appended to the queues for the other iterators. Thus, if one iterator is exhausted before the others, each remaining iterator will hold a copy of the entire iterable in memory. (You can find a Python function that emulates tee() in the itertools docs.)

For this reason, tee() should be used with care. If you are exhausting large portions of an iterator before working with the other returned by tee(), you may be better off casting the input iterator to a list or tuple.

The islice() function works much the same way as slicing a list or tuple. You pass it an iterable, a starting, and stopping point, and, just like slicing a list, the slice returned stops at the index just before the stopping point. You can optionally include a step value, as well. The biggest difference here is, of course, that islice() returns an iterator.

>>> # Slice from index 2 to 4
>>> list(it.islice('ABCDEFG', 2, 5)))
['C' 'D' 'E']

>>> # Slice from beginning to index 4, in steps of 2
>>> list(it.islice([1, 2, 3, 4, 5], 0, 5, 2))
[1, 3, 5]

>>> # Slice from index 3 to the end
>>> list(it.islice(range(10), 3, None))
[3, 4, 5, 6, 7, 8, 9]

>>> # Slice from beginning to index 3
>>> list(it.islice('ABCDE'), 4))
['A', 'B', 'C', 'D']

The last two examples above are useful for truncating iterables. You can use this to replace the list slicing used in cut() to select the “top” and “bottom” of the deck. As an added bonus, islice() won’t accept negative indices for the start/stop positions and the step value, so you won’t need to raise an exception if n is negative.

The last function you need is chain(). This function takes any number of iterables as arguments and “chains” them together. For example:

>>> list(it.chain('ABC', 'DEF'))
['A' 'B' 'C' 'D' 'E' 'F']

>>> list(it.chain([1, 2], [3, 4, 5, 6], [7, 8, 9]))
[1, 2, 3, 4, 5, 6, 7, 8, 9]

Now that you’ve got some additional firepower in your arsenal, you can re-write the cut() function to cut the deck of cards without making a full copy cards in memory:

def cut(deck, n):
    """Return an iterator over a deck of cards cut at index `n`."""
    deck1, deck2 = it.tee(deck, 2)
    top = it.islice(deck1, n)
    bottom = it.islice(deck2, n, None)
    return it.chain(bottom, top)

cards = cut(cards, 26)

Now that you have shuffled and cut the cards, it is time to deal some hands. You could write a function deal() that takes a deck, the number of hands, and the hand size as arguments and returns a tuple containing the specified number of hands.

You do not need any new itertools functions to write this function. See what you can come up with on your own before reading ahead.

Here’s one solution:

def deal(deck, num_hands=1, hand_size=5):
    iters = [iter(deck)] * hand_size
    return tuple(zip(*(tuple(it.islice(itr, num_hands)) for itr in iters)))

You start by creating a list of hand_size references to an iterator over deck. You then iterate over this list, removing num_hands cards at each step and storing them in tuples.

Next, you zip() these tuples up to emulate dealing one card at a time to each player. This produces num_hands tuples, each containing hand_size cards. Finally, you package the hands up into a tuple to return them all at once.

This implementation sets the default values for num_hands to 1 and hand_size to 5—maybe you are making a “Five Card Draw” app. Here’s how you would use this function, with some sample output:

>>> p1_hand, p2_hand, p3_hand = deal(cards, num_hands=3)
>>> p1_hand
(('A', 'S'), ('5', 'S'), ('7', 'H'), ('9', 'H'), ('5', 'H'))
>>> p2_hand
(('10', 'H'), ('2', 'D'), ('2', 'S'), ('J', 'C'), ('9', 'C'))
>>> p3_hand
(('2', 'C'), ('Q', 'S'), ('6', 'C'), ('Q', 'H'), ('A', 'C'))

What do you think the state of cards is now that you have dealt three hands of five cards?

>>> len(tuple(cards))
37

The fifteen cards dealt are consumed from the cards iterator, which is exactly what you want. That way, as the game continues, the state of the cards iterator reflects the state of the deck in play.

Section Recap

Let’s review the itertools functions you saw in this section.

`itertools.product` Example

product(*iterables, repeat=1)

Cartesian product of input iterables. Equivalent to nested for-loops.

>>> product([1, 2], ['a', 'b'])
(1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')

`itertools.tee` Example

tee(iterable, n=2)

Create any number of independent iterators from a single input iterable.

>>> iter1, iter2 = it.tee(['a', 'b', 'c'], 2)
>>> list(iter1)
['a', 'b', 'c']
>>> list(iter2)
['a', 'b', 'c']

`itertools.islice` Example

islice(iterable, stop) islice(iterable, start, stop, step=1)

Return an iterator whose __next__() method returns selected values from an iterable. Works like a slice() on a list but returns an iterator.

>>> islice([1, 2, 3, 4], 3)
1, 2, 3

>>> islice([1, 2, 3, 4], 1, 2)
2, 3

`itertools.chain` Example

chain(*iterables)

Return a chain object whose __next__() method returns elements from the first iterable until it is exhausted, then elements from the next iterable, until all of the iterables are exhausted.

>>> chain('abc', [1, 2, 3])
'a', 'b', 'c', 1, 2, 3

Intermission: Flattening A List of Lists

In the previous example, you used chain() to tack one iterator onto the end of another. The chain() function has a class method .from_iterable() that takes a single iterable as an argument. The elements of the iterable must themselves be iterable, so the net effect is that chain.from_iterable() flattens its argument:

>>> list(it.chain.from_iterable([[1, 2, 3], [4, 5, 6]]))
[1, 2, 3, 4, 5, 6]

There’s no reason the argument of chain.from_iterable() needs to be finite. You could emulate the behavior of cycle(), for example:

>>> cycle = it.chain.from_iterable(it.repeat('abc'))
>>> list(it.islice(cycle, 8))
['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b']

The chain.from_interable() function is useful when you need to build an iterator over data that has been “chunked.”

In the next section, you will see how to use itertools to do some data analysis on a large dataset. But you deserve a break for having stuck with it this far. Why not hydrate yourself and relax a bit? Maybe even play a little Star Trek: The Nth Iteration.

Back? Great! Let’s do some data analysis.

Analyzing the S&P500

In this example, you will get your first taste of using itertools to manipulate a large dataset—in particular, the historical daily price data of the S&P500 index. A CSV file SP500.csv with this data can be found here (source: Yahoo Finance). The problem you’ll tackle is this:

Determine the maximum daily gain, daily loss (in percent change), and the longest growth streak in the history of the S&P500.

To get a feel for what you’re dealing with, here are the first ten rows of SP500.csv:

$ head -n 10 SP500.csv
Date,Open,High,Low,Close,Adj Close,Volume
1950-01-03,16.660000,16.660000,16.660000,16.660000,16.660000,1260000
1950-01-04,16.850000,16.850000,16.850000,16.850000,16.850000,1890000
1950-01-05,16.930000,16.930000,16.930000,16.930000,16.930000,2550000
1950-01-06,16.980000,16.980000,16.980000,16.980000,16.980000,2010000
1950-01-09,17.080000,17.080000,17.080000,17.080000,17.080000,2520000
1950-01-10,17.030001,17.030001,17.030001,17.030001,17.030001,2160000
1950-01-11,17.090000,17.090000,17.090000,17.090000,17.090000,2630000
1950-01-12,16.760000,16.760000,16.760000,16.760000,16.760000,2970000
1950-01-13,16.670000,16.670000,16.670000,16.670000,16.670000,3330000

As you can see, the early data is limited. The data improves for later dates, and, as a whole, is sufficient for this example.

The strategy for solving this problem is as follows:

Read data from the CSV file and transform it into a sequence gains of daily percent changes using the “Adj Close” column.
Find the maximum and minimum values of the gains sequence, and the date on which they occur. (Note that it is possible that these values are attained on more than on date; in that case, the most recent date will suffice.)
Transform gains into a sequence growth_streaks of tuples of consecutive positive values in gains. Then determine the length of the longest tuple in growth_streaks and the beginning and ending dates of the streak. (It is possible that the maximum length is attained by more than one tuple in growth_streaks; in that case, the tuple with the most recent beginning and ending dates will suffice.)

The percent change between two values x and y is given by the following formula:

For each step in the analysis, it is necessary to compare values associated with dates. To facilitate these comparisons, you can subclass the namedtuple object from the collections module:

from collections import namedtuple


class DataPoint(namedtuple('DataPoint', ['date', 'value'])):
    __slots__ = ()

    def __le__(self, other):
        return self.value <= other.value

    def __lt__(self, other):
        return self.value < other.value

    def __gt__(self, other):
        return self.value > other.value

The DataPoint class has two attributes: date (a datetime.datetime instance) and value. The .__le__(), .__lt__() and .__gt__() dunder methods are implemented so that the <=, <, and > boolean comparators can be used to compare the values of two DataPoint objects. This also allows the max() and min() built-in functions to be called with DataPoint arguments.

Note: If you are not familiar with namedtuple, check out this excellent resource. The namedtuple implementation for DataPoint is just one of many ways to build this data structure. For example, in Python 3.7 you could implement DataPoint as a data class. Check out our Ultimate Guide to Data Classes for more information.

The following reads the data from SP500.csv to a tuple of DataPoint objects:

import csv
from datetime import datetime


def read_prices(csvfile, _strptime=datetime.strptime):
    with open(csvfile) as infile:
        reader = csv.DictReader(infile)
        for row in reader:
            yield DataPoint(date=_strptime(row['Date'], '%Y-%m-%d').date(),
                            value=float(row['Adj Close']))


prices = tuple(read_prices('SP500.csv'))

The read_prices() generator opens SP500.csv and reads each row with a csv.DictReader() object. DictReader() returns each row as an OrderedDict whose keys are the column names from the header row of the CSV file.

For each row, read_prices() yields a DataPoint object containing the values in the “Date” and “Adj Close” columns. Finally, the full sequence of data points is committed to memory as a tuple and stored in the prices variable.

Next, prices needs to be transformed to a sequence of daily percent changes:

gains = tuple(DataPoint(day.date, 100*(day.value/prev_day.value - 1.))
                for day, prev_day in zip(prices[1:], prices))

The choice of storing the data in a tuple is intentional. Although you could point gains to an iterator, you will need to iterate over the data twice to find the minimum and maximum values.

If you use tee() to create two independent iterators, exhausting one iterator to find the maximum will create a copy of all of the data in memory for the second iterator. By creating a tuple up front, you do not lose anything in terms of space complexity compared to tee(), and you may even gain a little speed.

Note: This example focuses on leveraging itertools for analyzing the S&P500 data. Those intent on working with a lot of time series financial data might also want to check out the Pandas library, which is well suited for such tasks.

Maximum Gain and Loss

To determine the maximum gain on any single day, you might do something like this:

max_gain = DataPoint(None, 0)
for data_point in gains:
    max_gain = max(data_point, max_gain)

print(max_gain)   # DataPoint(date='2008-10-28', value=11.58)

You can simplify the for loop using the functools.reduce() function. This function accepts a binary function func and an iterable inputs as arguments, and “reduces” inputs to a single value by applying func cumulatively to pairs of objects in the iterable.

For example, functools.reduce(operator.add, [1, 2, 3, 4, 5]) will return the sum 1 + 2 + 3 + 4 + 5 = 15. You can think of reduce() as working in much the same way as accumulate(), except that it returns only the final value in the new sequence.

Using reduce(), you can get rid of the for loop altogether in the above example:

import functools as ft


max_gain = ft.reduce(max, gains)

print(max_gain)  # DataPoint(date='2008-10-28', value=11.58)

The above solution works, but it isn’t equivalent to the for loop you had before. Do you see why? Suppose the data in your CSV file recorded a loss every single day. What would the value of max_gain be?

In the for loop, you first set max_gain = DataPoint(None, 0), so if there are no gains, the final max_gain value will be this empty DataPoint object. However, the reduce() solution returns the smallest loss. That is not what you want and could introduce a difficult to find bug.

This is where itertools can help you out. The itertools.filterfalse() function takes two arguments: a function that returns True or False (called a predicate), and an iterable inputs. It returns an iterator over the elements in inputs for which the predicate returns False.

Here’s a simple example:

>>> only_positives = it.filterfalse(lambda x: x <= 0, [0, 1, -1, 2, -2])
>>> list(only_positives)
[1, 2]

You can use filterfalse() to filter out the values in gains that are negative or zero so that reduce() only works on positive values:

max_gain = ft.reduce(max, it.filterfalse(lambda p: p <= 0, gains))

What happens if there are never any gains? Consider the following:

>>> ft.reduce(max, it.filterfalse(lambda x: x <= 0, [-1, -2, -3]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: reduce() of empty sequence with no initial value

Well, that’s not what you want! But, it makes sense because the iterator returned by filterflase() is empty. You could handle the TypeError by wrapping the call to reduce() with try...except, but there’s a better way.

The reduce() function accepts an optional third argument for an initial value. Passing 0 to this third argument gets you the expected behavior:

>>> ft.reduce(max, it.filterfalse(lambda x: x <= 0, [-1, -2, -3]), 0)
0

Applying this to the S&P500 example:

zdp = DataPoint(None, 0)  # zero DataPoint
max_gain = ft.reduce(max, it.filterfalse(lambda p: p <= 0, diffs), zdp)

Great! You’ve got it working just the way it should! Now, finding the maximum loss is easy:

max_loss = ft.reduce(min, it.filterfalse(lambda p: p > 0, gains), zdp)

print(max_loss)  # DataPoint(date='2018-02-08', value=-20.47)

Longest Growth Streak

Finding the longest growth streak in the history of the S&P500 is equivalent to finding the largest number of consecutive positive data points in the gains sequence. The itertools.takewhile() and itertools.dropwhile() functions are perfect for this situation.

The takewhile() function takes a predicate and an iterable inputs as arguments and returns an iterator over inputs that stops at the first instance of an element for which the predicate returns False:

it.takewhile(lambda x: x < 3, [0, 1, 2, 3, 4])  # 0, 1, 2

The dropwhile() function does exactly the opposite. It returns an iterator beginning at the first element for which the predicate returns False:

it.dropwhile(lambda x: x < 3, [0, 1, 2, 3, 4])  # 3, 4

In the following generator function, takewhile() and dropwhile() are composed to yield tuples of consecutive positive elements of a sequence:

def consecutive_positives(sequence, zero=0):
    def _consecutives():
        for itr in it.repeat(iter(sequence)):
            yield tuple(it.takewhile(lambda p: p > zero,
                                     it.dropwhile(lambda p: p <= zero, itr)))
    return it.takewhile(lambda t: len(t), _consecutives())

The consecutive_positives() function works because repeat() keeps returning a pointer to an iterator over the sequence argument, which is being partially consumed at each iteration by the call to tuple() in the yield statement.

You can use consecutive_positives() to get a generator that produces tuples of consecutive positive data points in gains:

growth_streaks = consecutive_positives(gains, zero=DataPoint(None, 0))

Now you can use reduce() to extract the longest growth streak:

longest_streak = ft.reduce(lambda x, y: x if len(x) > len(y) else y,
                           growth_streaks)

Putting the whole thing together, here’s a full script that will read data from the SP500.csv file and print out the max gain/loss and longest growth streak:

from collections import namedtuple
import csv
from datetime import datetime
import itertools as it
import functools as ft


class DataPoint(namedtuple('DataPoint', ['date', 'value'])):
    __slots__ = ()

    def __le__(self, other):
        return self.value <= other.value

    def __lt__(self, other):
        return self.value < other.value

    def __gt__(self, other):
        return self.value > other.value


def consecutive_positives(sequence, zero=0):
    def _consecutives():
        for itr in it.repeat(iter(sequence)):
            yield tuple(it.takewhile(lambda p: p > zero,
                                     it.dropwhile(lambda p: p <= zero, itr)))
    return it.takewhile(lambda t: len(t), _consecutives())


def read_prices(csvfile, _strptime=datetime.strptime):
    with open(csvfile) as infile:
        reader = csv.DictReader(infile)
        for row in reader:
            yield DataPoint(date=_strptime(row['Date'], '%Y-%m-%d').date(),
                            value=float(row['Adj Close']))


# Read prices and calculate daily percent change.
prices = tuple(read_prices('SP500.csv'))
gains = tuple(DataPoint(day.date, 100*(day.value/prev_day.value - 1.))
              for day, prev_day in zip(prices[1:], prices))

# Find maximum daily gain/loss.
zdp = DataPoint(None, 0)  # zero DataPoint
max_gain = ft.reduce(max, it.filterfalse(lambda p: p <= zdp, gains))
max_loss = ft.reduce(min, it.filterfalse(lambda p: p > zdp, gains), zdp)


# Find longest growth streak.
growth_streaks = consecutive_positives(gains, zero=DataPoint(None, 0))
longest_streak = ft.reduce(lambda x, y: x if len(x) > len(y) else y,
                           growth_streaks)

# Display results.
print('Max gain: {1:.2f}% on {0}'.format(*max_gain))
print('Max loss: {1:.2f}% on {0}'.format(*max_loss))

print('Longest growth streak: {num_days} days ({first} to {last})'.format(
    num_days=len(longest_streak),
    first=longest_streak[0].date,
    last=longest_streak[-1].date
))

Running the above script produces the following output:

Max gain: 11.58% on 2008-10-13
Max loss: -20.47% on 1987-10-19
Longest growth streak: 14 days (1971-03-26 to 1971-04-15)

Section Recap

In this section, you covered a lot of ground, but you only saw a few functions from itertools. Let’s review those now.

`itertools.filterfalse` Example

filterfalse(pred, iterable)

Return those items of sequence for which pred(item) is false. If pred is None, return the items that are false.

>>> filterfalse(bool, [1, 0, 1, 0, 0])
0, 0, 0

`itertools.takewhile` Example

takewhile(pred, iterable)

Return successive entries from an iterable as long as pred evaluates to true for each entry.

>>> takewhile(bool, [1, 1, 1, 0, 0])
1, 1, 1

`itertools.dropwhile` Example

dropwhile(pred, iterable)

Drop items from the iterable while pred(item) is true. Afterwards, return every element until the iterable is exhausted.

>>> dropwhile(bool, [1, 1, 1, 0, 0, 1, 1, 0])
0, 0, 1, 1, 0

You are really starting to master this whole itertools thing! The community swim team would like to commission you for a small project.

Building Relay Teams From Swimmer Data

In this example, you will read data from a CSV file containing swimming event times for a community swim team from all of the swim meets over the course of a season. The goal is to determine which swimmers should be in the relay teams for each stroke next season.

Each stroke should have an “A” and a “B” relay team with four swimmers each. The “A” team should contain the four swimmers with the best times for the stroke and the “B” team the swimmers with the next four best times.

The data for this example can be found here. If you want to follow along, download it to your current working directory and save it as swimmers.csv.

Here are the first 10 rows of swimmers.csv:

$ head -n 10 swimmers.csv
Event,Name,Stroke,Time1,Time2,Time3
0,Emma,freestyle,00:50:313667,00:50:875398,00:50:646837
0,Emma,backstroke,00:56:720191,00:56:431243,00:56:941068
0,Emma,butterfly,00:41:927947,00:42:062812,00:42:007531
0,Emma,breaststroke,00:59:825463,00:59:397469,00:59:385919
0,Olivia,freestyle,00:45:566228,00:46:066985,00:46:044389
0,Olivia,backstroke,00:53:984872,00:54:575110,00:54:932723
0,Olivia,butterfly,01:12:548582,01:12:722369,01:13:105429
0,Olivia,breaststroke,00:49:230921,00:49:604561,00:49:120964
0,Sophia,freestyle,00:55:209625,00:54:790225,00:55:351528

The three times in each row represent the times recorded by three different stopwatches, and are given in MM:SS:mmmmmm format (minutes, seconds, microseconds). The accepted time for an event is the median of these three times, not the average.

Let’s start by creating a subclass Event of the namedtuple object, just like we did in the SP500 example:

from collections import namedtuple

class Event(namedtuple('Event', ['stroke', 'name', 'time'])):
    __slots__ = ()

    def __lt__(self, other):
        return self.time < other.time

The .stroke property stores the name of the stroke in the event, .name stores the swimmer name, and .time records the accepted time for the event. The .__lt__() dunder method will allow min() to be called on a sequence of Event objects.

To read the data from the CSV into a tuple of Event objects, you can use the csv.DictReader object:

import csv
import datetime
import statistics

def read_events(csvfile, _strptime=datetime.datetime.strptime):
    def _median(times):
        return statistics.median((_strptime(time, '%M:%S:%f').time()
                                  for time in row['Times']))

    fieldnames = ['Event', 'Name', 'Stroke']
    with open(csvfile) as infile:
        reader = csv.DictReader(infile, fieldnames=fieldnames, restkey='Times')
        next(reader)  # skip header
        for row in reader:
            yield Event(row['Stroke'], row['Name'], _median(row['Times']))


events = tuple(read_events('swimmers.csv'))

The read_events() generator reads each row in the swimmers.csv file into an OrderedDict object in the following line:

reader = csv.DictReader(infile, fieldnames=fieldnames, restkey='Times')

By assigning the 'Times' field to restkey, the “Time1”, “Time2”, and “Time3” columns of each row in the CSV file will be stored in a list on the 'Times' key of the OrderedDict returned by csv.DictReader.

For example, the first row of the file (excluding the header row) is read into the following object:

OrderedDict([('Event', '0'),
             ('Name', 'Emma'),
             ('Stroke', 'freestyle'),
             ('Times', ['00:50:313667', '00:50:875398', '00:50:646837'])])

Next, read_events() yields an Event object with the stroke, swimmer name, and median time (as a datetime.time object) returned by the _median() function, which calls statistics.median() on the list of times in the row.

Since each item in the list of times is read as a string by csv.DictReader(), _median() uses the datetime.datetime.strptime() classmethod to instantiate a time object from each string.

Finally, a tuple of Event objects is created:

events = tuple(read_events('swimmers.csv'))

The first five elements of events look like this:

>>> events[:5]
(Event(stroke='freestyle', name='Emma', time=datetime.time(0, 0, 50, 646837)),
 Event(stroke='backstroke', name='Emma', time=datetime.time(0, 0, 56, 720191)),
 Event(stroke='butterfly', name='Emma', time=datetime.time(0, 0, 42, 7531)),
 Event(stroke='breaststroke', name='Emma', time=datetime.time(0, 0, 59, 397469)),
 Event(stroke='freestyle', name='Olivia', time=datetime.time(0, 0, 46, 44389)))

Now that you’ve got the data into memory, what do you do with it? Here’s the plan of attack:

Group the events by stroke.
For each stroke:
- Group its events by swimmer name and determine the best time for each swimmer.
- Order the swimmers by best time.
- The first four swimmers make the “A” team for the stroke, and the next four swimmers make the “B” team.

The itertools.groupby() function makes grouping objects in an iterable a snap. It takes an iterable inputs and a key to group by, and returns an object containing iterators over the elements of inputs grouped by the key.

Here’s a simple groupby() example:

>>> data = [{'name': 'Alan', 'age': 34},
...         {'name': 'Catherine', 'age': 34},
...         {'name': 'Betsy', 'age': 29},
...         {'name': 'David', 'age': 33}]
...
>>> grouped_data = it.groupby(data, key=lambda x: x['age'])
>>> for key, grp in grouped_data:
...     print('{}: {}'.format(key, list(grp)))
...
34: [{'name': 'Alan', 'age': 34}, {'name': 'Betsy', 'age': 34}]
29: [{'name': 'Catherine', 'age': 29}]
33: [{'name': 'David', 'age': 33}]

If no key is specified, groupby() defaults to grouping by “identity”—that is, aggregating identical elements in the iterable:

>>> for key, grp in it.groupby([1, 1, 2, 2, 2, 3]:
...     print('{}: {}'.format(key, list(grp)))
...
1: [1, 1]
2: [2, 2, 2]
3: [3]

The object returned by groupby() is sort of like a dictionary in the sense that the iterators returned are associated with a key. However, unlike a dictionary, it won’t allow you to access its values by key name:

>>> grouped_data[1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'itertools.groupby' object is not subscriptable

In fact, groupby() returns an iterator over tuples whose first components are keys and second components are iterators over the grouped data:

>>> grouped_data = it.groupby([1, 1, 2, 2, 2, 3])
>>> list(grouped_data)
[(1, <itertools._grouper object at 0x7ff3056130b8>),
 (2, <itertools._grouper object at 0x7ff3056130f0>),
 (3, <itertools._grouper object at 0x7ff305613128>)]

One thing to keep in mind with groupby() is that it isn’t as smart as you might like. As groupby() traverses the data, it aggregates elements until an element with a different key is encountered, at which point it starts a new group:

>>> grouped_data = it.groupby([1, 2, 1, 2, 3, 2])
>>> for key, grp in grouped_data:
...     print('{}: {}'.format(key, list(grp)))
...
1: [1]
2: [2]
1: [1]
2: [2]
3: [3]
2: [2]

Compare this to, say, the SQL GROUP BY command, which groups elements regardless of their order of appearance.

When working with groupby(), you need to sort your data on the same key that you would like to group by. Otherwise, you may get unexpected results. This is so common that it helps to write a utility function to take care of this for you:

def sort_and_group(iterable, key=None):
    """Group sorted `iterable` on `key`."""
    return it.groupby(sorted(iterable, key=key), key=key)

Returning to the swimmers example, the first thing you need to do is create a for loop that iterates over the data in the events tuple grouped by stroke:

for stroke, evts in sort_and_group(events, key=lambda evt: evt.stroke):

Next, you need to group the evts iterator by swimmer name inside of the above for loop:

events_by_name = sort_and_group(evts, key=lambda evt: evt.name)

To calculate the best time for each swimmer in events_by_name, you can call min() on the events in that swimmers group. (This works because you implemented the .__lt__() dunder method in the Events class.)

best_times = (min(evt) for _, evt in events_by_name)

Note that the best_times generator yields Event objects containing the best stroke time for each swimmer. To build the relay teams, you’ll need to sort best_times by time and aggregate the result into groups of four. To aggregate the results, you can use the grouper() function from The grouper() recipe section and use islice() to grab the first two groups.

sorted_by_time = sorted(best_times, key=lambda evt: evt.time)
teams = zip(('A', 'B'), it.islice(grouper(sorted_by_time, 4), 2))

Now teams is an iterator over exactly two tuples representing the “A” and the “B” team for the stroke. The first component of each tuple is the letter “A” or “B”, and the second component is an iterator over Event objects containing the swimmers in the team. You can now print the results:

for team, swimmers in teams:
    print('{stroke} {team}: {names}'.format(
        stroke=stroke.capitalize(),
        team=team,
        names=', '.join(swimmer.name for swimmer in swimmers)
    ))

Here’s the full script:

from collections import namedtuple
import csv
import datetime
import itertools as it
import statistics


class Event(namedtuple('Event', ['stroke', 'name', 'time'])):
    __slots__ = ()

    def __lt__(self, other):
        return self.time < other.time


def sort_and_group(iterable, key=None):
    return it.groupby(sorted(iterable, key=key), key=key)


def grouper(iterable, n, fillvalue=None):
    iters = [iter(iterable)] * n
    return it.zip_longest(*iters, fillvalue=fillvalue)


def read_events(csvfile, _strptime=datetime.datetime.strptime):
    def _median(times):
        return statistics.median((_strptime(time, '%M:%S:%f').time()
                                  for time in row['Times']))

    fieldnames = ['Event', 'Name', 'Stroke']
    with open(csvfile) as infile:
        reader = csv.DictReader(infile, fieldnames=fieldnames, restkey='Times')
        next(reader)  # Skip header.
        for row in reader:
            yield Event(row['Stroke'], row['Name'], _median(row['Times']))


events = tuple(read_events('swimmers.csv'))

for stroke, evts in sort_and_group(events, key=lambda evt: evt.stroke):
    events_by_name = sort_and_group(evts, key=lambda evt: evt.name)
    best_times = (min(evt) for _, evt in events_by_name)
    sorted_by_time = sorted(best_times, key=lambda evt: evt.time)
    teams = zip(('A', 'B'), it.islice(grouper(sorted_by_time, 4), 2))
    for team, swimmers in teams:
        print('{stroke} {team}: {names}'.format(
            stroke=stroke.capitalize(),
            team=team,
            names=', '.join(swimmer.name for swimmer in swimmers)
        ))

If you run the above code, you’ll get the following output:

Backstroke A: Sophia, Grace, Penelope, Addison
Backstroke B: Elizabeth, Audrey, Emily, Aria
Breaststroke A: Samantha, Avery, Layla, Zoe
Breaststroke B: Lillian, Aria, Ava, Alexa
Butterfly A: Audrey, Leah, Layla, Samantha
Butterfly B: Alexa, Zoey, Emma, Madison
Freestyle A: Aubrey, Emma, Olivia, Evelyn
Freestyle B: Elizabeth, Zoe, Addison, Madison

Where to Go From Here

If you have made it this far, congratulations! I hope you have enjoyed the journey.

itertools is a powerful module in the Python standard library, and an essential tool to have in your toolkit. With it, you can write faster and more memory efficient code that is often simpler and easier to read (although that is not always the case, as you saw in the section on second order recurrence relations).

If anything, though, itertools is a testament to the power of iterators and lazy evaluation. Even though you have seen many techniques, this article only scratches the surface.

So I guess this means your journey is only just beginning.

Free Bonus: Click here to get our itertools cheat sheet that summarizes the techniques demonstrated in this tutorial.

In fact, this article skipped two itertools functions: starmap() and compress(). In my experience, these are two of the lesser used itertools functions, but I urge you to read their docs an experiment with your own use cases!

Here are a few places where you can find more examples of itertools in action (thanks to Brad Solomon for these fine suggestions):

Finally, for even more tools for constructing iterators, take a look at more-itertools.

Do you have any favorite itertools recipes/use-cases? We would love to hear about them in the comments!

We would like to thank our readers Putcher and Samir Aghayev for pointing out a couple of errors in the original version of this article.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Interacting with Python

2018-05-28T14:00:00+00:00

At this point, you should have a working Python 3 interpreter at hand. If you need help getting Python set up correctly, please refer to the previous section in this tutorial series.

Here’s what you’ll learn in this tutorial: Now that you have a working Python setup, you’ll see how to actually execute Python code and run Python programs. By the end of this article, you’ll know how to:

Use Python interactively by typing code directly into the interpreter
Execute code contained in a script file from the command line
Work within a Python Integrated Development Environment (IDE)

It’s time to write some Python code!

Hello, World!

There is a long-standing custom in the field of computer programming that the first code written in a newly installed language is a short program that simply displays the string Hello, World! to the console.

Note: This is a time-honored tradition dating back to the 1970s. See Hello, World! for a brief history. You seriously risk upsetting the qi of the universe if you don’t abide by this custom.

The simplest Python 3 code to display Hello, World! is:

print("Hello, World!")

You will explore several different ways to execute this code below.

Using the Python Interpreter Interactively

The most straightforward way to start talking to Python is in an interactive Read-Eval-Print Loop (REPL) environment. That simply means starting up the interpreter and typing commands to it directly. The interpreter:

Reads the command you enter
Evaluates and executes the command
Prints the output (if any) to the console
Loops back and repeats the process

The session continues in this manner until you instruct the interpreter to terminate. Most of the example code in this tutorial series is presented as REPL interaction.

Starting the Interpreter

In a GUI desktop environment, it is likely that the installation process placed an icon on the desktop or an item in the desktop menu system that starts Python.

For example, in Windows, there will likely be a program group in the Start menu labeled Python 3.x, and under it a menu item labeled Python 3.x (32-bit), or something similar depending on the particular installation you chose.

Clicking on that item will start the Python interpreter:

The Python interpreter (REPL) running inside a terminal window.

Alternatively, you can open a terminal window and run the interpreter from the command line. How you go about opening a terminal window varies depending on which operating system you’re using:

In Windows, it is called Command Prompt.
In macOS or Linux, it should be called Terminal.

Using your operating system’s search function to search for “command” in Windows or “terminal” in macOS or Linux should find it.

Once a terminal window is open, if paths have been set up properly by the Python install process, you should be able to just type python. Then, you should see a response from the Python interpreter.

This example is from the Windows Command Prompt window:

C:\Users\john>python
Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 07:18:10) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

Technical Note: If you are on a Linux system and installed Python 3, it may be that both Python 2 and Python 3 are installed. In that case, it is possible that typing python at the prompt will start Python 2. Starting Python 3 may require typing something else, like python3.

If you installed a more recent version of Python 3 than the one that was included in the distribution, you may even need to specify the version you installed specifically—for example python3.6.

If you are not seeing the >>> prompt, then you are not talking to the Python interpreter. This could be because Python is either not installed or not in your terminal window session’s path. It’s also possible that you just haven’t found the correct command to execute it. You can refer to our installing Python tutorial for help.

Executing Python Code

If you are seeing the prompt, you’re off and running! The next step is to execute the statement that displays Hello, World! to the console:

Ensure that the >>> prompt is displayed, and the cursor is positioned after it.
Type the command print("Hello, World!") exactly as shown.
Press the Enter key.

The interpreter’s response should appear on the next line. You can tell it is console output because the >>> prompt is absent:

>>> print("Hello, World!")
Hello, World!

If your session looks like the above, then you have executed your first Python code! Take a moment to celebrate.

Congratulations!

Did something go wrong? Perhaps you made one of these mistakes:

You forgot to enclose the string to be printed in quotation marks:

>>> print(Hello, World!)
  File "<stdin>", line 1
    print(Hello, World!)
                      ^
SyntaxError: invalid syntax

You remembered the opening quotation mark but forgot the closing one:

>>> print("Hello, World!)
  File "<stdin>", line 1
    print("Hello, World!)
                        ^
SyntaxError: EOL while scanning string literal

You used different opening and closing quotation marks:

>>> print("Hello, World!')
  File "<stdin>", line 1
    print("Hello, World!')
                         ^
SyntaxError: EOL while scanning string literal

You forgot the parentheses:

>>> print "Hello, World!"
  File "<stdin>", line 1
    print "Hello, World!"
                        ^
SyntaxError: Missing parentheses in call to 'print'

You entered extra whitespace characters before the command:

>>>     print("Hello, World!")
  File "<stdin>", line 1
    print("Hello, World!")
    ^
IndentationError: unexpected indent

(You will see in an upcoming section why this matters.)

If you got some sort of error message, go back and verify that you typed the command exactly as shown above.

Exiting the Interpreter

When you are finished interacting with the interpreter, you can exit a REPL session in several ways:

Type exit() and press Enter:
```
>>> exit()

C:\Users\john>
```
In Windows, type Ctrl+Z and press Enter:
```
>>> ^Z
C:\Users\john>
```
In Linux or macOS, type Ctrl+D. The interpreter terminates immediately; pressing Enter is not needed.
If all else fails, you can simply close the interpreter window. This isn’t the best way, but it will get the job done.

Running a Python Script from the Command Line

Entering commands to the Python interpreter interactively is great for quick testing and exploring features or functionality.

Eventually though, as you create more complex applications, you will develop longer bodies of code that you will want to edit and run repeatedly. You clearly don’t want to re-type the code into the interpreter every time! This is where you will want to create a script file.

A Python script is a reusable set of code. It is essentially a Python program—a sequence of Python instructions—contained in a file. You can run the program by specifying the name of the script file to the interpreter.

Python scripts are just plain text, so you can edit them with any text editor. If you have a favorite programmer’s editor that operates on text files, it should be fine to use. If you don’t, the following are typically installed natively with their respective operating systems:

Windows: Notepad
Unix/Linux: vi or vim
macOS: TextEdit

Using whatever editor you’ve chosen, create a script file called hello.py containing the following:

print("Hello, World!")

Now save the file, keeping track of the directory or folder you chose to save into.

Start a command prompt or terminal window. If the current working directory is the same as the location in which you saved the file, you can simply specify the filename as a command-line argument to the Python interpreter: python hello.py

For example, in Windows it would look like this:

C:\Users\john\Documents\test>dir
 Volume in drive C is JFS
 Volume Serial Number is 1431-F891

 Directory of C:\Users\john\Documents\test

05/20/2018  01:31 PM    <DIR>          .
05/20/2018  01:31 PM    <DIR>          ..
05/20/2018  01:31 PM                24 hello.py
               1 File(s)             24 bytes
               2 Dir(s)  92,557,885,440 bytes free

C:\Users\john\Documents\test>python hello.py
Hello, World!

If the script is not in the current working directory, you can still run it. You’ll just have to specify the path name to it:

C:\>cd
C:\

C:\>python c:\Users\john\Documents\test\hello.py
Hello, World!

In Linux or macOS, your session may look more like this:

jfs@jfs-xps:~$ pwd
/home/jfs

jfs@jfs-xps:~$ ls
hello.py

jfs@jfs-xps:~$ python hello.py
Hello, World!

A script file is not required to have a .py extension. The Python interpreter will run the file no matter what it’s called, so long as you properly specify the file name on the command line:

jfs@jfs-xps:~$ ls
hello.foo

jfs@jfs-xps:~$ cat hello.foo
print("Hello, World!")

jfs@jfs-xps:~$ python hello.foo
Hello, World!

But giving Python files a .py extension is a useful convention because it makes them more easily identifiable. In desktop-oriented folder/icon environments like Windows and macOS, this will also typically allow for setting up an appropriate file association, so that you can run the script just by clicking its icon.

Interacting with Python through an IDE

An Integrated Development Environment (IDE) is an application that more or less combines all the functionality you have seen so far. IDEs usually provide REPL capability as well as an editor with which you can create and modify code to then submit to the interpreter for execution.

You may also find cool features such as:

Syntax highlighting: IDEs often colorize different syntax elements in the code to make it easier to read.
Context-sensitive help: Advanced IDEs can display related information from the Python documentation or even suggested fixes for common types of code errors.
Code-completion: Some IDEs can complete partially typed pieces of code (like function names) for you—a great time-saver and convenience feature.
Debugging: A debugger allows you to run code step-by-step and inspect program data as you go. This is invaluable when you are trying to determine why a program is behaving improperly, as will inevitably happen.

IDLE

Most Python installations contain a rudimentary IDE called IDLE. The name ostensibly stands for Integrated Development and Learning Environment, but one member of the Monty Python troupe is named Eric Idle, which hardly seems like a coincidence.

The procedure for running IDLE varies from one operating system to another.

Starting IDLE in Windows

Go to the Start menu and select All Programs or All Apps. There should be a program icon labeled IDLE (Python 3.x 32-bit) or something similar. This will vary slightly between Win 7, 8, and 10. The IDLE icon may be in a program group folder named Python 3.x. You can also find the IDLE program icon by using the Windows search facility from the start menu and typing in IDLE.

Click on the icon to start IDLE.

Starting IDLE in macOS

Open Spotlight Search. Typing Cmd+Space is one of several ways to do this. In the search box, type terminal and press Enter.

In the terminal window, type idle3 and press Enter.

Starting IDLE in Linux

IDLE is available with the Python 3 distribution but may not have been installed by default. To find out whether it is, open a terminal window. This varies depending on the Linux distribution, but you should be able to find it by using the desktop search function and searching for terminal. In the terminal window, type idle3 and press Enter.

If you get an error saying command not found or something to that effect, then IDLE is apparently not installed, so you’ll need to install it.

The method for installing apps also varies from one Linux distribution to the next. For example, with Ubuntu Linux, the command to install IDLE is sudo apt-get install idle3. Many Linux distributions have GUI-based application managers that you can use to install apps as well.

Follow whatever procedure is appropriate for your distribution to install IDLE. Then, type idle3 in a terminal window and press Enter to run it. Your installation procedure may have also set up a program icon somewhere on the desktop to start IDLE as well.

Whew!

Using IDLE

Once IDLE is installed and you have started it successfully, you should see a window titled Python 3.x.x Shell, where 3.x.x corresponds to your version of Python:

The >>> prompt should look familiar. You can type REPL commands interactively, just like when you started the interpreter from a console window. Mindful of the qi of the universe, display Hello, World! again:

The interpreter behaves more or less the same as when you ran it directly from the console. The IDLE interface adds the perk of displaying different syntactic elements in distinct colors to make things more readable.

It also provides context-sensitive help. For example, if you type print( without typing any of the arguments to the print function or the closing parenthesis, then flyover text should appear specifying usage information for the print() function.

One other feature IDLE provides is statement recall:

If you have typed in several statements, you can recall them with Alt+P and Alt+N in Windows or Linux.
Alt+P cycles backward through previously executed statements; Alt+N cycles forward.
Once a statement has been recalled, you can use editing keys on the keyboard to edit it and then execute it again. The corresponding commands in macOS are Cmd+P and Cmd+N.

You can also create script files and run them in IDLE. From the Shell window menu, select File → New File. That should open an additional editing window. Type in the code to be executed:

From the menu in that window, select File → Save or File → Save As… and save the file to disk. Then select Run → Run Module. The output should appear back in the interpreter Shell window:

OK, that’s probably enough Hello, World!. The qi of the universe should be safe.

Once both windows are open, you can switch back and forth, editing the code in one window, running it and displaying its output in the other. In that way, IDLE provides a rudimentary Python development platform.

Although it is somewhat basic, it supports quite a bit of additional functionality, including code completion, code formatting, and a debugger. See the IDLE documentation for more details.

Thonny

Thonny is free Python IDE developed and maintained by the Institute of Computer Science at the University of Tartu, Estonia. It is targeted at Python beginners specifically, so the interface is simple and uncluttered as well as easy to understand and get comfortable with quickly.

Like IDLE, Thonny supports REPL interaction as well as script file editing and execution:

Thonny performs syntax highlighting and code completion in addition to providing a step-by-step debugger. One feature that is particularly helpful to those learning Python is that the debugger displays values in expressions as they are evaluated while you are stepping through the code:

Thonny is especially easy to get started with because it comes with Python 3.6 built in. So you only need to perform one install, and you’re ready to go!

Versions are available for Windows, macOS, and Linux. The Thonny website has download and installation instructions.

IDLE and Thonny are certainly not the only games going. There are many other IDEs available for Python code editing and development. See our Python IDEs and Code Editors Guide for additional suggestions.

Online Python REPL Sites

As you saw in the previous section, there are websites available that can provide you with interactive access to a Python interpreter online without you having to install anything locally.

This approach may be unsatisfactory for some of the more complicated or lengthy examples in this tutorial. But for simple REPL sessions, it should work well.

The Python Software Foundation provides an Interactive Shell on their website. On the main page, click on the button that looks like one of these:

Or go directly to https://www.python.org/shell.

You should get a page with a window that looks something like this:

The familiar >>> prompt shows you that you are talking to the Python interpreter.

Here are a few other sites that provide Python REPL:

Conclusion

Larger applications are typically contained in script files that are passed to the Python interpreter for execution.

But one of the advantages of an interpreted language is that you can run the interpreter and execute commands interactively. Python is easy to use in this manner, and it is a great way to get your feet wet learning how the language works.

The examples throughout this tutorial have been produced by direct interaction with the Python interpreter, but if you choose to use IDLE or some other available IDE, the examples should still work just fine.

Continue to the next section, where you will start to explore the elements of the Python language itself.

« Installing Python

Interacting with Python

Basic Data Types »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python 3 Installation & Setup Guide

2018-05-23T14:00:00+00:00

To get started working with Python 3, you’ll need to have access to the Python interpreter. There are several common ways to accomplish this:

Python can be obtained from the Python Software Foundation website at python.org. Typically, that involves downloading the appropriate installer for your operating system and running it on your machine.
Some operating systems, notably Linux, provide a package manager that can be run to install Python.
On macOS, the best way to install Python 3 involves installing a package manager called Homebrew. You’ll see how to do this in the relevant section in the tutorial.
On mobile operating systems like Android and iOS, you can install apps that provide a Python programming environment. This can be a great way to practice your coding skills on the go.

Alternatively, there are several websites that allow you to access a Python interpreter online without installing anything on your computer at all.

Note: There is a chance that Python may have been shipped with your operating system and is already installed. Even if that is the case, it may be that the installed version is outdated, in which case you will want to obtain the latest version anyhow.

In this Python installation guide, you’ll see step by step how to set up a working Python 3 distribution on Windows, macOS, Linux, iOS, and Android. So let’s get started!

Windows

It is highly unlikely that your Windows system shipped with Python already installed. Windows systems typically do not. Fortunately, installing does not involve much more than downloading the Python installer from the python.org website and running it. Let’s take a look at how to install Python 3 on Windows:

Step 1: Download the Python 3 Installer

Open a browser window and navigate to the Download page for Windows at python.org.
Underneath the heading at the top that says Python Releases for Windows, click on the link for the Latest Python 3 Release - Python 3.x.x. (As of this writing, the latest is Python 3.6.5.)
Scroll to the bottom and select either Windows x86-64 executable installer for 64-bit or Windows x86 executable installer for 32-bit. (See below.)

Sidebar: 32-bit or 64-bit Python?

For Windows, you can choose either the 32-bit or 64-bit installer. Here’s what the difference between the two comes down to:

If your system has a 32-bit processor, then you should choose the 32-bit installer.

On a 64-bit system, either installer will actually work for most purposes. The 32-bit version will generally use less memory, but the 64-bit version performs better for applications with intensive computation.

If you’re unsure which version to pick, go with the 64-bit version.

Note: Remember that if you get this choice “wrong” and would like to switch to another version of Python, you can just uninstall Python and then re-install it by downloading another installer from python.org.

Step 2: Run the Installer

Once you have chosen and downloaded an installer, simply run it by double-clicking on the downloaded file. A dialog should appear that looks something like this:

Important: You want to be sure to check the box that says Add Python 3.x to PATH as shown to ensure that the interpreter will be placed in your execution path.

Then just click Install Now. That should be all there is to it. A few minutes later you should have a working Python 3 installation on your system.

Windows Subsystem for Linux (WSL)

If you are running Windows 10 Creators or Anniversary Update, you actually have another option for installing Python. These versions of Windows 10 include a feature called the Windows Subsystem for Linux, which allows you to run a Linux environment directly in Windows, unmodified and without the overhead of a virtual machine.

For more information, see the Windows Subsystem for Linux Documentation article on the Microsoft website.
For instructions on how to enable the subsystem in Windows 10 and install a Linux distribution, see the Windows 10 Installation Guide.
You can also check out this presentation on YouTube by Sarah Cooley, one of the members of the WSL development team.

Once you have installed the Linux distribution of your choice, you can install Python 3 from a Bash console window, just as you would if you were running that Linux distribution natively. (See below.)

Linux

There is a very good chance your Linux distribution has Python installed already, but it probably won’t be the latest version, and it may be Python 2 instead of Python 3.

To find out what version(s) you have, open a terminal window and try the following commands:

python --version
python2 --version
python3 --version

One or more of these commands should respond with a version, as below:

$ python3 --version
Python 3.6.5

If the version shown is Python 2.x.x or a version of Python 3 that is not the latest (3.6.5 as of this writing), then you will want to install the latest version. The procedure for doing this will depend on the Linux distribution you are running.

Ubuntu

Depending on the version of the Ubuntu distribution you run, the Python install instructions vary. You can determine your local Ubuntu version by running the following command:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial

Depending on the version number you see under Release in the console output, follow the instructions below:

Ubuntu 17.10, Ubuntu 18.04 (and above) come with Python 3.6 by default. You should be able to invoke it with the command python3.
Ubuntu 16.10 and 17.04 do not come with Python 3.6 by default, but it is in the Universe repository. You should be able to install it with the following commands:
```
$ sudo apt-get update
$ sudo apt-get install python3.6
```
You can then invoke it with the command python3.6.
If you are using Ubuntu 14.04 or 16.04, Python 3.6 is not in the Universe repository, and you need to get it from a Personal Package Archive (PPA). For example, to install Python from the “deadsnakes” PPA, do the following:
```
$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt-get update
$ sudo apt-get install python3.6
```
As above, invoke with the command python3.6.

Linux Mint

Mint and Ubuntu use the same package management system, which frequently makes life easier. You can follow the instructions above for Ubuntu 14.04. The “deadsnakes” PPA works with Mint.

Debian

We found sources that indicated that the Ubuntu 16.10 method would work for Debian, but we never found a path to get it to work on Debian 9. Instead, we ended up making Python from source as listed below.

One issue with Debian, however, is that it generally does not install the sudo command by default. To install it, you’ll need to do the following before you carry out the Compiling Python From Source instructions below:

$ su
$ apt-get install sudo
$ vi /etc/sudoers

After that, open the /etc/sudoers file using the sudo vim command (or your favorite text editor.) Add the following line of text to the end of the file, replacing your_username with your actual username:

your_username ALL=(ALL) ALL

openSUSE

We found several sites describing how to get zypper to install the latest version of Python, but they seemed problematic or outdated. We did not manage to get any of them to work successfully, so we fell back to building Python from source. To do that, you will need to install the development tools, which can be done in YaST (via the menus) or by using zypper:

$ sudu zypper install -t pattern devel_C_C++

This step took a while and involved the installation of 154 packages, but once it was completed, we were able to build the source as shown in the Compiling Python From Source section above.

CentOS

The IUS Community does a nice job of providing newer versions of software for “Enterprise Linux” distros (i.e. Red Hat Enterprise and CentOS). You can use their work to help you install Python 3.

To install, you should first update your system with the yum package manager:

$ sudo yum update
$ sudo yum install yum-utils

You can then install the CentOS IUS package which will get you up to date with their site:

$ sudo yum install https://centos7.iuscommunity.org/ius-release.rpm

Finally you can then install Python and Pip:

$ sudo yum install python36u
$ sudo yum install python36u-pip

Thanks to Jani Karhunen for his excellent writeup for CentOS 7.

Fedora

Fedora has a roadmap to switch to Python 3 as the default Python published here. It indicates that the current version and the next few versions will all ship with Python 2 as the default, but Python 3 will be installed. If the python3 installed on your version is not 3.6, you can use the following command to install it:

$ sudo dnf install python36

Arch Linux

Arch Linux is fairly aggressive about keeping up with Python releases. It is likely you already have the latest version. If not, you can use this command:

$ packman -S python

Compiling Python From Source

Sometimes your Linux distribution will not have the latest version of Python, or maybe you just want to be able to build the latest, greatest version yourself. Here are the steps you need to take to build Python from source:

Step 1: Download the Source Code

To start, you need to get the Python source code. Python.org makes this fairly easy. If you go to the Downloads page, you will see the latest source for Python 3 at the top. (Make sure you don’t grab Legacy Python, Python 2.)

When you select the version, at the bottom of the page there is a Files section. Select the Gzipped source tarball and download it to your machine. If you prefer a command line method, you can easily use wget to download it to your current directory:

$ wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz

Step 2: Prepare Your System

There are a few distro-specific steps involved in building Python from scratch. The goal of each step is the same on all distros, but you might need to translate to your distribution if it does not use apt-get:

The first step you should take when doing an operation like this is to update the system packages on your machine before you start. On Debian, this is what that looks like:
```
$ sudo apt-get update
$ sudo apt-get upgrade
```

Next, we want to make sure the system has the tools needed to build Python. There are a bunch of them and you might already have some, but that’s fine. I’ve listed them all in one command line, but you can break the list into shorter commands by just repeating the sudo apt-get install -y portion:

# For apt-based systems (like Debian, Ubuntu, and Mint)
$ sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev  libncursesw5-dev xz-utils tk-dev

# For yum-based systems (like CentOS)
$ sudo yum -y groupinstall development
$ sudo yum -y install zlib-devel

Step 3: Build Python

Once you have the prerequisites and the tar file, you can unpack the source into a directory. Note that the following command will create a new directory called Python-3.6.5 under the one you are in:
```
$ tar xvf Python-3.6.5.tgz
$ cd Python-3.6.5
```

Now you need to run the ./configure tool to prepare the build:

$ ./configure --enable-optimizations --with-ensurepip=install

Next, you build the Python programs using make. The -j option simply tells make to split the building into parallel steps to speed up the compilation. Even with the parallel builds, this step can take a several minutes:
```
$ make -j 8
```
Then, you’ll want to install your new version of Python. You’ll use the altinstall target here in order to not overwrite the system’s version of Python. Since you’re installing Python into /usr/bin, you’ll need to run as root:
```
$ sudo make altinstall
```
Warning: Please only use the altinstall target on make. Using the install target will overwrite the python binary. While this seems like it would be cool, there are big portions of the system that rely on the pre-installed version of Python.

Step 4: Verify Your Python Install

Finally, you can test out your new Python version:

$ python3.6 -V
Python 3.6.5

macOS / Mac OS X

While current versions of macOS (previously known as “Mac OS X”) include a version of Python 2, it is likely out of date by a few months. Also, this tutorial series uses Python 3, so let’s get you upgraded to that.

The best way we found to install Python 3 on macOS is through the Homebrew package manager. This approach is also recommended by community guides like The Hitchhiker’s Guide to Python.

Step 1: Install Homebrew (Part 1)

To get started, you first want to install Homebrew:

Open a browser and navigate to http://brew.sh/. After the page has finished loading, select the Homebrew bootstrap code under “Install Homebrew”. Then hit Cmd+C to copy it to the clipboard. Make sure you’ve captured the text of the complete command because otherwise the installation will fail.
Now you need to open a Terminal.app window, paste the Homebrew bootstrap code, and then hit Enter. This will begin the Homebrew installation.
If you’re doing this on a fresh install of macOS, you may get a pop up alert asking you to install Apple’s “command line developer tools”. You’ll need those to continue with the installation, so please confirm the dialog box by clicking on “Install”.

At this point, you’re likely waiting for the command line developer tools to finish installing, and that’s going to take a few minutes. Time to grab a coffee or tea!

Step 2: Install Homebrew (Part 2)

You can continue installing Homebrew and then Python after the command line developer tools installation is complete:

Confirm the “The software was installed” dialog from the developer tools installer.
Back in the terminal, hit Enter to continue with the Homebrew installation.
Homebrew asks you to enter your password so it can finalize the installation. Enter your user account password and hit Enter to continue.
Depending on your internet connection, Homebrew will take a few minutes to download its required files. Once the installation is complete, you’ll end up back at the command prompt in your terminal window.

Whew! Now that the Homebrew package manager is set up, let’s continue on with installing Python 3 on your system.

Step 3: Install Python

Once Homebrew has finished installing, return to your terminal and run the following command:

$ brew install python3

Note: When you copy this command, be sure you don’t include the $ character at the beginning. That’s just an indicator that this is a console command.

This will download and install the latest version of Python. After the Homebrew brew install command finishes, Python 3 should be installed on your system.

You can make sure everything went correctly by testing if Python can be accessed from the terminal:

Open the terminal by launching Terminal.app.
Type pip3 and hit Enter.
You should see the help text from Python’s “Pip” package manager. If you get an error message running pip3, go through the Python install steps again to make sure you have a working Python installation.

Assuming everything went well and you saw the output from Pip in your command prompt window…congratulations! You just installed Python on your system, and you’re all set to continue with the next section in this tutorial.

iOS (iPhone / iPad)

The Pythonista app for iOS is a full-fledged Python development environment that you can run on your iPhone or iPad. It’s basically a combination of a Python editor, documentation, and interpreter rolled into one single app.

Pythonista is surprisingly fun to use. It’s a great little tool when you’re stuck without a laptop and want to work on your Python skills on the go. It comes with the complete Python 3 standard library and even includes full documentation you can browse offline.

To install and set up Pythonista you need to download it from the iOS app store.

Android (Phones & Tablets)

If you have an Android tablet or phone and want to practice Python on the go, there are a several options available. The one that we found most reliably supports Python 3.6 is Pydroid 3.

Pydroid 3 features an interpreter you can use for REPL sessions, and it also provides the ability to edit, save, and execute Python code:

You can download and install Pydroid 3 from the Google Play store. There is a free version and also a paid Premium version which supports code prediction and code analysis.

Online Python Interpreters

If you want to try out the examples in this tutorial without installing Python on your machine, there are several web sites available where you can interact with a Python interpreter online:

Python.org Online Console: www.python.org/shell
Python Fiddle: pythonfiddle.com
Repl.it: repl.it
Trinket: trinket.io
Python Anywhere: www.pythonanywhere.com

These cloud-based Python interpreters may not be able to execute some of the more complex examples in this tutorial, but they will be adequate for running most of the simpler ones and may be a nice way to get you started. More information on using these sites is presented in the next section.

Conclusion

This section provided you with the information you need to gain access to a Python 3 interpreter. You are now ready to head to the next section and begin interacting with Python!

« Introduction to Python

Installing Python

Interacting with Python »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Introduction to Python 3

2018-05-21T14:00:00+00:00

Python is a high-level, interpreted scripting language developed in the late 1980s by Guido van Rossum at the National Research Institute for Mathematics and Computer Science in the Netherlands. The initial version was published at the alt.sources newsgroup in 1991, and version 1.0 was released in 1994.

Python 2.0 was released in 2000, and the 2.x versions were the prevalent releases until December 2008. At that time, the development team made the decision to release version 3.0, which contained a few relatively small but significant changes that were not backward compatible with the 2.x versions. Python 2 and 3 are very similar, and some features of Python 3 have been backported to Python 2. But in general, they remain not quite compatible.

Both Python 2 and 3 have continued to be maintained and developed, with periodic release updates for both. As of this writing, the most recent versions available are 2.7.15 and 3.6.5. However, an official End Of Life date of January 1, 2020 has been established for Python 2, after which time it will no longer be maintained. If you are a newcomer to Python, it is recommended that you focus on Python 3, as this tutorial will do.

Python is still maintained by a core development team at the Institute, and Guido is still in charge, having been given the title of BDFL (Benevolent Dictator For Life) by the Python community. The name Python, by the way, derives not from the snake, but from the British comedy troupe Monty Python’s Flying Circus, of which Guido was, and presumably still is, a fan. It is common to find references to Monty Python sketches and movies scattered throughout the Python documentation.

Free PDF Download: Python 3 Cheat Sheet

Why Choose Python?

If you’re going to write programs, there are literally dozens of commonly used languages to choose from. Why choose Python? Here are some of the features that make Python an appealing choice.

Python is Popular

Python has been growing in popularity over the last few years. The 2018 Stack Overflow Developer Survey ranked Python as the 7th most popular and the number one most wanted technology of the year. World-class software development countries around the globe use Python every single day.

According to research by Dice Python is also one of the hottest skills to have and the most popular programming language in the world based on the Popularity of Programming Language Index.

Due to the popularity and widespread use of Python as a programming language, Python developers are sought after and paid well. If you’d like to dig deeper into Python salary statistics and job opportunities, you can do so here.

Python is Interpreted

Many languages are compiled, meaning the source code you create needs to be translated into machine code, the language of your computer’s processor, before it can be run. Programs written in an interpreted language are passed straight to an interpreter that runs them directly.

This makes for a quicker development cycle because you just type in your code and run it, without the intermediate compilation step.

One potential downside to interpreted languages is execution speed. Programs that are compiled into the native language of the computer processor tend to run more quickly than interpreted programs. For some applications that are particularly computationally intensive, like graphics processing or intense number crunching, this can be limiting.

In practice, however, for most programs, the difference in execution speed is measured in milliseconds, or seconds at most, and not appreciably noticeable to a human user. The expediency of coding in an interpreted language is typically worth it for most applications.

Further reading: See this Wikipedia page to read more about the differences between interpreted and compiled languages.

Python is Free

The Python interpreter is developed under an OSI-approved open-source license, making it free to install, use, and distribute, even for commercial purposes.

A version of the interpreter is available for virtually any platform there is, including all flavors of Unix, Windows, macOS, smartphones and tablets, and probably anything else you ever heard of. A version even exists for the half dozen people remaining who use OS/2.

Python is Portable

Because Python code is interpreted and not compiled into native machine instructions, code written for one platform will work on any other platform that has the Python interpreter installed. (This is true of any interpreted language, not just Python.)

Python is Simple

As programming languages go, Python is relatively uncluttered, and the developers have deliberately kept it that way.

A rough estimate of the complexity of a language can be gleaned from the number of keywords or reserved words in the language. These are words that are reserved for special meaning by the compiler or interpreter because they designate specific built-in functionality of the language.

Python 3 has 33 keywords, and Python 2 has 31. By contrast, C++ has 62, Java has 53, and Visual Basic has more than 120, though these latter examples probably vary somewhat by implementation or dialect.

Python code has a simple and clean structure that is easy to learn and easy to read. In fact, as you will see, the language definition enforces code structure that is easy to read.

But It’s Not That Simple

For all its syntactical simplicity, Python supports most constructs that would be expected in a very high-level language, including complex dynamic data types, structured and functional programming, and object-oriented programming.

Additionally, a very extensive library of classes and functions is available that provides capability well beyond what is built into the language, such as database manipulation or GUI programming.

Python accomplishes what many programming languages don’t: the language itself is simply designed, but it is very versatile in terms of what you can accomplish with it.

Conclusion

This section gave an overview of the Python programming language, including:

A brief history of the development of Python
Some reasons why you might select Python as your language of choice

Python is a great option, whether you are a beginning programmer looking to learn the basics, an experienced programmer designing a large application, or anywhere in between. The basics of Python are easily grasped, and yet its capabilities are vast.

Proceed to the next section to learn how to acquire and install Python on your computer.

Introduction to Python

Installing Python »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

Documenting Python Code: A Complete Guide

Why Documenting Your Code Is So Important

Commenting vs. Documenting Code

Basics of Commenting Code

Commenting Code via Type Hinting (Python 3.5+)

Documenting Your Python Code Base Using Docstrings

Docstrings Background

Docstring Types

Class Docstrings

Package and Module Docstrings

Script Docstrings

Docstring Formats

Google Docstrings Example

reStructured Text Example

NumPy/SciPy Docstrings Example

Epytext Example

Documenting Your Python Projects

Private Projects

Shared Projects

Public and Open Source Projects

The Four Main Sections of the docs Folder

Documentation Tools and Resources

Where Do I Start?

Fast, Flexible, Easy and Intuitive: How to Speed Up Your Pandas Projects

But I Heard That Pandas Is Slow…

This Tutorial

The Task at Hand

Saving Time With Datetime Data

Simple Looping Over Pandas Data

Looping with .itertuples() and .iterrows()

Pandas’ .apply()

Selecting Data With .isin()

Can We Do Better?

Don’t Forget NumPy!

Prevent Reprocessing with HDFStore

Conclusions

Lists and Tuples in Python

Python Lists

Lists Are Ordered

Lists Can Contain Arbitrary Objects

List Elements Can Be Accessed by Index

Lists Can Be Nested

Lists Are Mutable

Modifying a Single List Value

Modifying Multiple List Values

Prepending or Appending Items to a List

Methods That Modify a List

Lists Are Dynamic

Python Tuples

Defining and Using Tuples

Tuple Assignment, Packing, and Unpacking

Conclusion

Reading and Writing CSV Files in Python

What Is a CSV File?

Where Do CSV Files Come From?

Parsing CSV Files With Python’s Built-in CSV Library

Reading CSV Files With csv

Reading CSV Files Into a Dictionary With csv

Optional Python CSV reader Parameters

Writing CSV Files With csv

Writing CSV File From a Dictionary With csv

Parsing CSV Files With the pandas Library

Reading CSV Files With pandas

Writing CSV Files With pandas

Conclusion

Generating Random Data in Python (Guide)

How Random Is Random?

What Is “Cryptographically Secure?”

What You’ll Cover Here

PRNGs in Python

The random Module

PRNGs for Arrays: numpy.random

CSPRNGs in Python

os.urandom(): About as Random as It Gets

Python’s Best Kept secrets

One Last Candidate: uuid

Why Not Just “Default to” SystemRandom?

Odds and Ends: Hashing

Recap

The Four Main Sections of the `docs` Folder

Looping with `.itertuples()` and `.iterrows()`

Pandas’ `.apply()`

Selecting Data With `.isin()`

Reading CSV Files With `csv`

Reading CSV Files Into a Dictionary With `csv`

Optional Python CSV `reader` Parameters

Writing CSV Files With `csv`

Writing CSV File From a Dictionary With `csv`

Parsing CSV Files With the `pandas` Library

Reading CSV Files With `pandas`

Writing CSV Files With `pandas`

The `random` Module

PRNGs for Arrays: `numpy.random`

`os.urandom()`: About as Random as It Gets

Python’s Best Kept `secrets`

One Last Candidate: `uuid`

Why Not Just “Default to” `SystemRandom`?

The `+` Operator

The `*` Operator

The `in` Operator

`bytes` Objects

Defining a Literal `bytes` Object

Defining a `bytes` Object With the Built-in `bytes()` Function

Operations on `bytes` Objects

`bytearray` Objects

#2 “New Style” String Formatting (`str.format`)

The `breakpoint()` Built-In

“`async`” and “`await`” Are Keywords

“`asyncio`” Face Lift

Importing Data Files With “`importlib.resources`“