Real Python

Create a Flask Application With Google Login

2019-07-17T14:00:00+00:00

You’ve probably seen the option for Google Login on various websites. Some sites also have more options like Facebook Login or GitHub Login. All these options allow users to utilize existing accounts to use a new service.

In this article, you’ll work through the creation of a Flask web application. Your application will allow a user to log in using their Google identity instead of creating a new account. There are tons of benefits with this method of user management. It’s going to be safer and simpler than managing the traditional username and password combinations.

This article will be more straightforward if you already understand the basics of Python. It would also help to know a bit about web frameworks and HTTP requests, but that’s not strictly necessary.

By the end of this article, you’ll be able to:

Create a Flask web application that lets users log in with Google
Create client credentials to interact with Google
Use Flask-Login for user session management in a Flask application
Better understand OAuth 2 and OpenID Connect (OIDC)

You can click the box below to get the code for the application you’ll make in this article:

Download Sample Project: Click here to download the code for the Flask application with Google login you'll build in this article.

You may want individual users to have profiles. Or perhaps you want to provide features to only certain users. In any case, you’ll need to know who’s interacting with your application. In other words, you’ll need to authenticate users and identify them in some unique way.

The traditional solution is to use a unique username and a secret password. Your application would store that information and ask for it when needed. However, there are a few downsides to that solution:

You have to securely manage the passwords.
You have to implement any account-related functionality:
- Two-factor authentication
- Password reset
You have to protect against malicious login attempts.
Your users have to remember yet another username and password.

By using Google login for your users, you push all that responsibility to them. Your application waits for the user to go through authentication. Google then tells your application about that user. At that point, you can effectively log them into your application.

You don’t have to store any passwords, and Google handles all the security.

There are two very popular and important specifications called OAuth 2 and OpenID Connect (OIDC). OIDC is built on top of OAuth 2, adding a few new ideas and concepts.

These specifications define how a third-party application can obtain information from another service. This usually involves obtaining consent from a user. To unpack this a bit, let’s look at how this applies to the application you’re about to build.

You’re about to write a third-party application, and it will let a user use a Google Login button to log in. To do that, Google needs to know about your application. Luckily, you can register your application as a client to Google.

Once a user comes to your application and presses the Google Login button, you can send them to Google. From there, Google needs to make sure that the user consents to pass along their email and other information to your application. Should the user consent, Google sends back some information to your application. You then store that information and can reference it later, effectively logging the user in.

OpenID Connect Details

To request information on behalf of a user, you must become a client to the authentication server, also known as the provider. The first thing you’ll realize if you dig into these specifications is that there are a lot of overlapping terms and concepts.

So as a third-party application (also known as a client), you want to get info from the provider on the user’s behalf. There are a series of steps that enable that to happen, and those steps must happen in a specific order. That’s why you’ll sometimes hear OAuth 2 and OpenID Connect referred to as a handshake, flow, or dance.

Those steps are, broadly:

You register a third-party application as a client to the provider:
- You receive unique client credentials from the provider.
- You’ll use these client credentials to authenticate (prove who you are) to the provider later on.
The client sends a request to the provider’s authorization URL
The provider asks the user to authenticate (prove who they are)
The provider asks the user to consent to the client acting on their behalf:
- Usually this includes limited access, and it’s made clear to the user what the client is asking for.
- This is like when you have to approve an app on your phone to have access to location or contacts.
The provider sends the client a unique authorization code.
The client sends the authorization code back to the Provider’s token URL.
The provider sends the client tokens to use with other provider URLs on behalf of the user.

Note: The steps above are for the Authorization Code Flow, as defined by OAuth 2.

Those steps include both of the standards mentioned so far. OpenID Connect (OIDC) is built on top of OAuth 2, adding a few additional features and requirements, mostly involving the process of authentication. Other than the authentication mentioned in the flow above, the important OIDC concepts for your application are the provider configuration and userinfo endpoint.

The provider configuration contains information about the provider, including the exact URLs you need to use for the OAuth 2 flow. There’s a standard URL on an OIDC provider you can use to get back a document with standardized fields.

The userinfo endpoint will return information about the user after you’ve gone through the OAuth 2 flow. This will include their email and some basic profile information you’ll use in your application. In order to obtain this user information, you’ll need a token from the provider, as described in the last step in the flow above.

You’ll see the details about how the provider configuration and user info endpoint can be utilized later on.

Creating a Google Client

The first step to enable a Google Login option is to register your application as a client to Google. Let’s walk through the steps to do that.

First, note that you will need a Google Account. You already have one if you use Gmail.

Next, go to the Google developers credentials page.

Once in, you may be prompted to agree to their terms of service. Should you agree to those, press the Create credentials button on the next page. Select the option for OAuth client ID:

Select the Web application option at the top. You can provide a name for the client in the Name field as well. The name you provide will be displayed to users when they are consenting to your application acting on their behalf.

You’ll be running your web application locally for now, so you can set the Authorized JavaScript origins to https://127.0.0.1:5000 and Authorized redirect URIs to https://127.0.0.1:5000/login/callback. This will allow your local Flask application to communicate with Google.

Finally, hit Create and take note of the client ID and client secret. You’ll need both later.

Creating Your Own Web Application

Now for the fun part where you apply the knowledge you learned to create an actual web application!

Let’s start with the goal in mind. You want to create an application that lets users log in with their Google account. That application should be able to retrieve some basic information about the user from Google, like their email address. Then the application should store basic user information in a database.

First, though, let’s take a look at the framework and libraries you’ll be using.

Flask

Flask is a lightweight web-framework, a self-proclaimed microframework. It comes with built-in tools for the basic tasks that a web application will perform, like routing URLs and handling HTTP requests.

I’ve chosen to use Flask as an example for both its popularity and simplicity. However, the things you’ve learned about OAuth 2 and OIDC are not specific to Flask. In fact, even the library you’ll be using to make OAuth 2 and OIDC easier is usable in any Python code. In other words, with some minor modifications, you can take what you learn here and apply it to another framework of your choice.

Another tool you can use to make the handling of users easier is flask_login, which provides user session management.

This library does a few things behind the scenes and gives you some tools to help with users. Namely, it provides utilities for you to know when a user is logged in and logged out. It does this by managing a user session inside a browser cookie.

It also handles the logging in and logging out of users, including the creation of database entries for those users. From the aspect of your code though, it really just makes everything a lot simpler (which you’ll see soon).

OAuthLib

There’s a common phrase that holds very true for security-related and standard-compliant code: “Don’t reinvent the wheel.”

OAuth 2 and OpenID Connect standards are complicated. Have a look at the RFC and specifications, and you’ll see. They’re dense. One mistake means that you could be opening up a vulnerability in your application.

So, you are not going to be writing code to implement these standards. You’re going to use a Python package that was chosen on some very specific criteria:

It’s a popular and generally recommended library. Lots of other packages use this library internally.
It is very active, with people patching bugs frequently.
It’s battle-hardened and has been around since 2012.

There are web-framework-specific packages that use this library to more closely integrate into Flask, Django, Pyramid, and others. However, to keep the code you learn here framework-agnostic, you’ll use this library directly without any fancy wrappers.

Installing Dependencies

There are a number of third-party dependencies that you’ll use to make your life easier. Here’s a summary of those dependencies:

A web framework to make typical web application tasks easier (Flask)
A headache-free way of managing user sessions (Flask-Login)
A battle-hardened OIDC library (oauthlib)

Additionally, you’ll use the following:

A database to store some information about users who log in (SQLite)
An user-friendly way to send HTTP requests to Google (requests)
A quick way to enable running securely with https locally (pyOpenSSL)

SQLite is part of the standard Python library, but the other packages are not. So you have a couple of dependencies to install. For now, let’s just go through the creation of this application step by step.

First, you’ll need to install those third-party dependencies mentioned above. You’ll do this by creating a requirements.txt file with the following contents:

requests==2.21.0
Flask==1.0.2
oauthlib==3.0.1
pyOpenSSL==19.0.0
Flask-Login==0.4.1

Note: Other versions of the packages may work, but these are versions that were used during the writing and testing of this article.

Next, you can install those dependencies using pip, Python’s package installer.

Note: It is generally recommended to use virtual environments if you’re going to be installing dependencies for different Python applications on your computer. See Python Virtual Environments: A Primer to learn more.

To install from the requirements.txt file, run the following command in your terminal:

$ pip install -r requirements.txt

Now you’re ready to rock and roll! Let’s dig right into the code.

Imports, Configuration, and Setup

Start by adding a few files to support some basic database functionality and user management. These won’t be described section by section, mostly because diving into Python database implementation details is a rabbit hole that would distract us from our goal.

This file will handle some database functionality. This is almost line by line from Flask’s official database tutorial:

# http://flask.pocoo.org/docs/1.0/tutorial/database/
import sqlite3

import click
from flask import current_app, g
from flask.cli import with_appcontext

def get_db():
    if "db" not in g:
        g.db = sqlite3.connect(
            "sqlite_db", detect_types=sqlite3.PARSE_DECLTYPES
        )
        g.db.row_factory = sqlite3.Row

    return g.db

def close_db(e=None):
    db = g.pop("db", None)

    if db is not None:
        db.close()

def init_db():
    db = get_db()

    with current_app.open_resource("schema.sql") as f:
        db.executescript(f.read().decode("utf8"))

@click.command("init-db")
@with_appcontext
def init_db_command():
    """Clear the existing data and create new tables."""
    init_db()
    click.echo("Initialized the database.")

def init_app(app):
    app.teardown_appcontext(close_db)
    app.cli.add_command(init_db_command)

Now that you have some database utilities, you can start to think about the schema. You may notice that this code is looking for a schema.sql file, which you’ll create next.

The schema.sql file is just some SQL that will create a user table in our database. You can see the fields you’ll be storing per user in this file.

Here you have a single table, user, which will house a few things related to users (their name, the email they log in with, and their profile picture from Google):

CREATE TABLE user (
  id TEXT PRIMARY KEY,
  name TEXT NOT NULL,
  email TEXT UNIQUE NOT NULL,
  profile_pic TEXT NOT NULL
);

The code in the db.py file will actually execute this SQL to create the table in your database.

This next file contains our User class, which will store and retrieve information from the database. The name, email, and profile picture will all be retrieved from Google, which you’ll see later on in the article.

The User class has methods to get an existing user from the database and create a new user:

from flask_login import UserMixin

from db import get_db

class User(UserMixin):
    def __init__(self, id_, name, email, profile_pic):
        self.id = id_
        self.name = name
        self.email = email
        self.profile_pic = profile_pic

    @staticmethod
    def get(user_id):
        db = get_db()
        user = db.execute(
            "SELECT * FROM user WHERE id = ?", (user_id,)
        ).fetchone()
        if not user:
            return None

        user = User(
            id_=user[0], name=user[1], email=user[2], profile_pic=user[3]
        )
        return user

    @staticmethod
    def create(id_, name, email, profile_pic):
        db = get_db()
        db.execute(
            "INSERT INTO user (id, name, email, profile_pic) "
            "VALUES (?, ?, ?, ?)",
            (id_, name, email, profile_pic),
        )
        db.commit()

The code executes SQL statements against the database, which is retrieved from the get_db() function from the previous db.py file. Each new user results in the insertion of an additional row in the database.

After you’ve created the db.py, schema.sql, and user.py files from the above code, you can create a new app.py file. Add to it the following imports:

# Python standard libraries
import json
import os
import sqlite3

# Third-party libraries
from flask import Flask, redirect, request, url_for
from flask_login import (
    LoginManager,
    current_user,
    login_required,
    login_user,
    logout_user,
)
from oauthlib.oauth2 import WebApplicationClient
import requests

# Internal imports
from db import init_db_command
from user import User

You’ll use all these later on, so it’s not so important to understand each of them at the moment. The next portion in your app.py is some configuration:

# Configuration
GOOGLE_CLIENT_ID = os.environ.get("GOOGLE_CLIENT_ID", None)
GOOGLE_CLIENT_SECRET = os.environ.get("GOOGLE_CLIENT_SECRET", None)
GOOGLE_DISCOVERY_URL = (
    "https://accounts.google.com/.well-known/openid-configuration"
)

Here’s how you’ll store the Google Client ID and Client Secret, which you should have created earlier in the article. These will be used later in the OIDC flow.

Your application will try to get the client credentials by reading environmental variables. There are a couple reasons for this:

You don’t have to change your code later if you want to use different credentials, you just have to update the environment.
You can’t accidentally commit your secret credentials to GitHub (or another public repository).

Lots of people accidentally commit secrets to public repositories, posing a pretty serious security risk. It’s better to protect against that by using environmental variables.

Tip: You can set your client credentials as environmental variables in Linux bash terminal and Mac OS X terminal using export GOOGLE_CLIENT_ID=your_client_id (similarly for GOOGLE_CLIENT_SECRET).

If you’re on Windows, you can use set GOOGLE_CLIENT_ID=your_client_id in Command Prompt.

Alternatively, you could paste the strings directly here and store them in these variables. However, the client secret should not be shared or committed to any public repository. In other words, be very careful not to check in this file if you paste your real client credentials in here.

Finally, below is some code with global variables and some naive database initialization logic. Most of this, other than the database initialization, is the standard way to set up Flask, Flask-Login, and OAuthLib, which you read about earlier:

# Flask app setup
app = Flask(__name__)
app.secret_key = os.environ.get("SECRET_KEY") or os.urandom(24)

# User session management setup
# https://flask-login.readthedocs.io/en/latest
login_manager = LoginManager()
login_manager.init_app(app)

# Naive database setup
try:
    init_db_command()
except sqlite3.OperationalError:
    # Assume it's already been created
    pass

# OAuth 2 client setup
client = WebApplicationClient(GOOGLE_CLIENT_ID)

# Flask-Login helper to retrieve a user from our db
@login_manager.user_loader
def load_user(user_id):
    return User.get(user_id)

Note that you are already using the Client ID from Google to initialize our oauthlib client in the WebApplicationClient.

You can create another environment variable SECRET_KEY that Flask and Flask-Login will use to cryptographically sign cookies and other items.

Web Application Endpoints

Now, for the fun stuff. You’re going to write four endpoints for your web application:

One for the homepage
One for beginning the user login process
One for a callback where Google will redirect to after a user logs in
One for logging out

These endpoints will be defined by different URLs on your application with very creative names:

Homepage: /
Login: /login
Login Callback: /login/callback
Logout: /logout

Of course, you may want to add additional pages and functionality later. The end result of this application will be totally extensible to add anything you want to it.

You’ll add all the following code for these endpoints into the app.py file. Let’s take a look at each of these endpoints’ code, one at a time.

Homepage

This is nothing fancy visually, but you’ll add some neat logic to display something different if a user is logged in. When they’re not logged in, a link will appear that says Google Login.

Pressing the link will redirect them to your /login endpoint, which will initiate the login flow. After a successful login, the homepage will now display both the user’s Google email and their public Google profile picture!

Without further ado, you can start adding more code to your app.py file:

@app.route("/")
def index():
    if current_user.is_authenticated:
        return (
            "<p>Hello, {}! You're logged in! Email: {}</p>"
            "<div><p>Google Profile Picture:</p>"
            '<img src="{}" alt="Google profile pic"></img></div>'
            '<a class="button" href="/logout">Logout</a>'.format(
                current_user.name, current_user.email, current_user.profile_pic
            )
        )
    else:
        return '<a class="button" href="/login">Google Login</a>'

You’ll note that you’re returning HTML as a string, which Flask will be able to serve. The current_user.is_authenticated is a lovely addition of the Flask-Login library. It’s a straightforward way to determine if the current user interacting with your application is logged in or not. This lets you apply conditional logic. In this case, it’s displaying some information you have saved about the user if they’re logged in.

You can get fields from your database entry for the user by just accessing them as attributes on the current_user object, such as current_user.email. This is another addition of Flask-Login.

Now let’s get to the OAuth 2 flow. The Google Login button from above will redirect to this endpoint. The first step in the flow is to figure out where Google’s OAuth 2 Authorization endpoint is.

Here’s where the lines between what’s defined by OAuth 2 and by OpenID Connect (OIDC) start to blur. As discussed previously, OIDC has a standard endpoint for a provider configuration, which contains a bunch of OAuth 2 and OIDC information. The document with that information is served from a standard endpoint everywhere, .well-known/openid-configuration.

Assuming you copied the previous code that defined GOOGLE_DISCOVERY_URL, here’s a quick, naive, function for retrieving Google’s provider configuration:

def get_google_provider_cfg():
    return requests.get(GOOGLE_DISCOVERY_URL).json()

Tip: To make this more robust, you should add error handling to the Google API call, just in case Google’s API returns a failure and not the valid provider configuration document.

The field from the provider configuration document you need is called authorization_endpoint. This will contain the URL you need to use to initiate the OAuth 2 flow with Google from your client application.

You can put all that logic together with the following code:

@app.route("/login")
def login():
    # Find out what URL to hit for Google login
    google_provider_cfg = get_google_provider_cfg()
    authorization_endpoint = google_provider_cfg["authorization_endpoint"]

    # Use library to construct the request for Google login and provide
    # scopes that let you retrieve user's profile from Google
    request_uri = client.prepare_request_uri(
        authorization_endpoint,
        redirect_uri=request.base_url + "/callback",
        scope=["openid", "email", "profile"],
    )
    return redirect(request_uri)

Luckily, oauthlib makes the actual request to Google easier. You used your pre-configured client that you already gave your Google Client ID to. Next, you provided the redirect you want Google to use. Finally, you asked Google for a number of OAuth 2 scopes.

You can think of each scope as a separate piece of user information. In your case, you’re asking for the user’s email and basic profile information from Google. The user will, of course, have to consent to give you this information.

Note: openid is a required scope to tell Google to initiate the OIDC flow, which will authenticate the user by having them log in. OAuth 2 doesn’t actually standardize how authentication happens, so this is necessary for our flow in this case.

Let’s do this one in pieces, since it’s a bit more involved than the previous few endpoints.

Once you redirect to Google’s authorization endpoint, a lot happens on Google’s end.

The login endpoint on your application is the jumping point for all of Google’s work authenticating the user and asking for consent. Once the user logs in with Google and agrees to share their email and basic profile information with your application, Google generates a unique code that it sends back to your application.

As a reminder, here are the OIDC steps you read about earlier:

You register a third-party application as a client to the provider.
The client sends a request to the provider’s authorization URL.
The provider asks the user to authenticate (prove who they are).
The provider asks the user to consent to the client acting on their behalf.
The provider sends the client a unique authorization code
The client sends the authorization code back to the provider’s token URL
The provider sends the client tokens to use with other URLs on behalf of the user

When Google sends back that unique code, it’ll be sending it to this login callback endpoint on your application. So your first step is to define the endpoint and get that code:

@app.route("/login/callback")
def callback():
    # Get authorization code Google sent back to you
    code = request.args.get("code")

The next thing you’re going to do is send that code back to Google’s token endpoint. After Google verifies your client credentials, they will send you back tokens that will allow you to authenticate to other Google endpoints on behalf of the user, including the userinfo endpoint you read about earlier. In your case, you only asked to view basic profile information, so that’s the only thing you can do with the tokens.

To start with, you need to figure out what Google’s token endpoint is. You’ll use the provider configuration document again:

# Find out what URL to hit to get tokens that allow you to ask for
# things on behalf of a user
google_provider_cfg = get_google_provider_cfg()
token_endpoint = google_provider_cfg["token_endpoint"]

oauthlib comes to your rescue a few times in this next block of code. First, you need to construct the token request. Once the request is constructed, you’ll use the requests library to actually send it out. Then oauthlib, once again, will help you with parsing the tokens from the response:

# Prepare and send a request to get tokens! Yay tokens!
token_url, headers, body = client.prepare_token_request(
    token_endpoint,
    authorization_response=request.url,
    redirect_url=request.base_url,
    code=code
)
token_response = requests.post(
    token_url,
    headers=headers,
    data=body,
    auth=(GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET),
)

# Parse the tokens!
client.parse_request_body_response(json.dumps(token_response.json()))

Now that you have the necessary tools to get the user’s profile information, you need to ask Google for it. Luckily, OIDC defines a user information endpoint, and its URL for a given provider is standardized in the provider configuration. You can get the location by checking the userinfo_endpoint field in the provider configuration document. Then you can use oauthlib to add the token to your request and use requests to send it out:

# Now that you have tokens (yay) let's find and hit the URL
# from Google that gives you the user's profile information,
# including their Google profile image and email
userinfo_endpoint = google_provider_cfg["userinfo_endpoint"]
uri, headers, body = client.add_token(userinfo_endpoint)
userinfo_response = requests.get(uri, headers=headers, data=body)

The next step in your journey is to parse the response from the userinfo endpoint. Google uses an optional field, email_verified, to confirm that not only has the user created an account, but they’ve verified the email address to complete the account creation. It’s generally safe to conditionally check for this verification, as it’s another layer of security that Google offers.

That being said, you’ll check for that, and if Google says the user is verified, then you’ll parse their information. The 4 pieces of basic profile information you’ll use are:

sub: the subject, a unique identifier for the user in Google
email: the user’s Google email address
picture: the user’s public profile picture in Google
given_name: the user’s first and last name in Google

All of that parsing results in the following code:

# You want to make sure their email is verified.
# The user authenticated with Google, authorized your
# app, and now you've verified their email through Google!
if userinfo_response.json().get("email_verified"):
    unique_id = userinfo_response.json()["sub"]
    users_email = userinfo_response.json()["email"]
    picture = userinfo_response.json()["picture"]
    users_name = userinfo_response.json()["given_name"]
else:
    return "User email not available or not verified by Google.", 400

The final steps in this callback are:

Create a user in your database with the information you just got from Google
Begin a user session by logging that user in
Send user back to the homepage (where you’ll now display their public profile information)

The code to fulfill those steps is as follows:

# Create a user in your db with the information provided
# by Google
user = User(
    id_=unique_id, name=users_name, email=users_email, profile_pic=picture
)

# Doesn't exist? Add it to the database.
if not User.get(unique_id):
    User.create(unique_id, users_name, users_email, picture)

# Begin user session by logging the user in
login_user(user)

# Send user back to homepage
return redirect(url_for("index"))

So what you’re doing here is creating a new row in your database for the user if they don’t already exist. Then, you’re starting a session using Flask-Login.

Logout

The logout endpoint is much less code than the last few endpoints. You just call a logout function and redirect back to the homepage. Done and done. Here it is:

@app.route("/logout")
@login_required
def logout():
    logout_user()
    return redirect(url_for("index"))

The @login_required decorator is something important to mention here. It’s another tool from the Flask-Login toolbox and will make sure that only logged in users can access this endpoint. You can use this if only logged in users should be accessing something. In this case, only logged in users can log out.

Testing Your Application Locally

You can run your Flask application on your local computer to test the login flow by adding some final code to app.py:

if __name__ == "__main__":
    app.run(ssl_context="adhoc")

You can run your Flask application with the following command in your terminal:

$ python app.py

Note: Because of the naive database initialization logic, the first time you run this command, it will create the database. To start your app, you have to run the same command again.

Flask should print to your terminal where it’s running the development server. It should be https://127.0.0.1:5000/.

Note that Flask’s development server is running locally and using https to ensure an encrypted connection with Google. This is achieved by the ssl_context="adhoc" argument to app.run in the code above. This requires you to have the package PyOpenSSL installed.

The downside is that the certificate used is generated on the fly, so when you go to https://127.0.0.1:5000/ in your browser, it will probably give you a big warning screen about your connection being insecure or not private. You can effectively ignore these warnings.

Once past the warning screen, you should see a single button that says Google Login. Pressing it will send you to the official Google login. After you log in, Google will prompt you to consent to the “third-party application” getting access to your email and profile information.

After consenting, you’ll be redirected back to your Flask application, where the page should show your Google email and public profile picture! Finally, a Logout button allows you to, well, log out.

Conclusion

Allowing users to use their existing accounts to log into your web application has many benefits. Most importantly, the security and complexity of account management does not have to rest on your shoulders. This frees you up to write your fancy new web application without worrying about the nitty gritty details of two-factor authentication and the like.

The application you made in this article is a great starting point. You can click the box below to get the code:

Download Sample Project: Click here to download the code for the Flask application with Google login you'll build in this article.

Your next step could be to do the following:

Rework the database initialization to happen separately from running the application
Separate out the HTML/CSS from the Python code for easier management:
- You could use templates.
- You could also load static files (like JS and CSS) from elsewhere.
Host your application in the cloud
Purchase a domain name
Use a real SSL certificate and get rid of that pesky warning

In this article, you’ve gone through the basics of OAuth 2 and OpenID Connect. You’ve seen how to use well-known Python packages to create a web application that allows users to log in with their existing Google account. Most importantly, you have example code that serves as a great starting point for your next web application!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Write Pythonic Loops

2019-07-16T14:00:00+00:00

One of the easiest ways to spot a developer who has a background in C-style languages and only recently picked up Python is to look at how they loop through a list. In this course, you’ll learn how to take a C-style (Java, PHP, C, C++) loop and turn it into the sort of loop a Python developer would write.

You can use these techniques to refactor your existing Python for loops and while loops in order to make them easier to read and more maintainable. You’ll learn how to use Python’s range(), xrange(), and enumerate() built-ins to refactor your loops and how to avoid having to keep track of loop indexes manually.

The main takeaways in this tutorial are that:

Writing C-style loops in Python is considered not Pythonic. Avoid managing loop indexes and stop conditions manually if possible.
Python’s for loops are really “for each” loops that can iterate over items from a container or sequence directly.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python at PyCon US 2019

2019-07-15T14:00:00+00:00

There are many PyCons all over the world each year, but the biggest of them all is PyCon US (United States). This year, over 3000 Pythonistas descended on Cleveland, Ohio, to learn, collaborate, contribute, and meet old and new friends alike. I’m sure many of you are already aware of what PyCon US is, but if not, then I would suggest reading our guide to PyCon to learn more.

Our First PyCon Together

This year, Real Python was afforded the opportunity to join other content creators at the PyCharm Content Creators booth, where we got to spend some time meeting readers and members of the Real Python community. Oh, and we gave out a ton of Real Python stickers, and we ran out of stock completely! Did you get yours?

The team was also excited for our authors Geir Arne Hjelle and Anthony Shaw, who were also there as speakers. You can catch the recordings of their talks on Plugins: Adding Flexibility to Your Apps and Wily Python: Writing simpler and more maintainable Python on the PyCon YouTube channel.

We were also lucky enough to be able to arrange a Real Python open space at PyCon for our readers. We had a phenomenal turnout, with 30 people in attendance.

It was a great opportunity for us to not only get to shake your hands and thank you for helping us with your encouragement and feedback but also to make sure we can continue to push and produce better and better content. Listening to the feedback from those who attended was an amazing experience, and we will be taking some of the ideas generated going forward to help us deliver a better reading and learning experience to you.

Despite the daily conversation in the member and staff Slack groups, most of the Real Python team had never met each other in person. Meeting my colleagues face to face and sharing a meal or beer with them was definitely one of the highlights of the whole conference for me personally.

But I decided to ask the rest of the team about their experience, in hopes that we could come up with some helpful advice or actionable tips for you in case you have yet to attend your first PyCon.

I asked the same three questions to a few of the team members who attended PyCon and collated their answers below. The questions were:

Can you briefly tell us who you are and what you do, in and out of Real Python? Was this your first PyCon? If so, what made 2019 the year you decided to go for the first time?
What were the highlights of your PyCon? What memories will you have of PyCon 2019?
There is so much to do at PyCon that it’s impossible to do everything. What was something you didn’t get the chance to do but wish you had?

Team Interviews

Without further ado, let’s see what they had to say.

Jim Anderson

I’m Jim Anderson, and I’m an embedded firmware developer by day, mainly working in C++ to produce video security cameras. I get to do a bit of Python at work as well. At Real Python, I’m an author and a technical reviewer, generally tending towards articles that are either low-level, tool related, or general CS topics.

This was my first PyCon. I finally decided to go just due to the excitement I was seeing about it in the Real Python community. Folks that I knew who had been previously talked it up so much. They were right.

There were so many highlights! Getting to finally meet some of my Real Python co-workers was very cool, of course, as was hanging out in the content-creators section of the PyCharm booth. (Thanks, JetBrains!)

I’d say that the best memory I have, the one that really solidified the atmosphere in my mind, was seeing a friend from PythonistaCafe and PyCon talking with someone who was working the venue cleaning up. She was so taken by the friendliness of the conference that she was asking about it and figuring out how she could be a part of it next year. Maybe we’ll get to see her in Pittsburgh.

There was a talk that I had scheduled to go to and just blew the timing and missed. It was Building an Open Source Artificial Pancreas, and I’m still disappointed I didn’t make it. It’s a great topic and a great talk.

Geir Arne Hjelle

My name is Geir Arne Hjelle, and I grew up in a small village in the north of Norway. Currently, I live in Oslo where I work with different data science and machine learning projects. Most of the time, I get to use Python and the excellent data science stack built on top of numpy.

I have been writing for Real Python since the spring of 2018, mostly about general Python packages and concepts. My first article was about the pathlib package, and I’m currently working on one about how imports work. In addition, I support other authors by doing reviews of their articles, both outline reviews and technical reviews.

This year was my first PyCon US. I live in Norway, so I mainly attend conferences in Europe. This fall, I will be at EuroSciPy for the fifth time. However, after joining the Real Python team, I was really tempted by the opportunity to meet some of my new friends and colleagues out in the wild.

James and I managed to meet up last summer, but otherwise I mainly knew the team by their Slack avatars. Meeting and hanging out with the rest of the Real Python gang, and nerding out over some of the reviews we’ve done for each other are great memories I’ll treasure for a long time.

There is so much happening at PyCon. I went all in and stayed for the tutorials, the main conference, and parts of the sprints. I really enjoyed all of it.

The tutorials were great, including David Beazley’s deep dive into lambda calculus. (Yay, math!) I was lucky enough to get to give a talk during the conference, which was a lot of fun. I was very well taken care of before my presentation: I think this is the first time I’ve had my own room for preparations, and someone walking me over to the presentation room.

Afterwards, I got some very nice and interesting questions. During the sprints, I got to help out a little on the new importlib.metadata library Jason Coombs and Barry Warsaw are putting together for Python 3.8. It was great to see first-hand how some of the core developers are working.

I had so many great experiences in Cleveland that it’s hard to think of things I didn’t have a chance to do. Although, I’ll admit that at this PyCon, I was quite focused on myself (especially with my talk being during the last session of the conference).

Next time, I’ll set aside some time for volunteering and make sure to support the running of the conference more than I did this time around. It was great to see how everybody chipped in: I was welcomed and checked in by Ernest W. Durbin III, who was also the chair of the whole conference.

James Mertz

I’m James Mertz, and I’m a Software Assurance Engineer at NASA’s Jet Propulsion Laboratory (JPL). I’m currently working on the Europa Clipper project, which is a satellite that will be orbiting around Jupiter and making very close trips near the moon Europa. I’ve been writing for Real Python for about a year now with topics ranging from documenting Python to how to get the most out of PyCon.

This is my fourth year at PyCon, so I’m starting to get to know my way around the conference, although each year it’s different enough to keep bringing me back. This year was by far the best for me for two reasons: volunteering and meeting up with the Real Python gang (both the authors and the readers).

In my guide on How to Get the Most Out of PyCon, I really put a focus on volunteering at PyCon. I realized that, while I had done some things in years past, I hadn’t really practiced what I was preaching. I decided to fully embrace this and tried to help in every way I could.

By the end of the conference, I had:

Been a tutorial “bouncer” for David Beazley’s deep dive into lambda Calculus
Helped with registration check-in and even got to check in Michael Kennedy from Python Bytes and Talk Python to Me
Done the swag bag stuffing line dance DJ-ed by Larry Hastings
Been a session staff runner (the person who makes sure the speaker is in the room on time)
Been a session staff chair (the person who introduces the speaker)

Through it all, I made some really amazing connections and felt great being able to contribute a little to the community that I love so much.

Speaking of community, meeting up with the Real Python community was another big highlight. As authors, we really don’t get many chances to meet with each other face to face. Therefore, to finally meet up with some of the people that I’ve come to know so well digitally, in the physical world, was a surreal experience. What made it better was that, even though we were all from different places in the world with different life experiences, we were able to quickly build a team spirit.

Perhaps even better than meeting my other Real Python team members, was meeting you, the readers. We held our first Real Python open space, where many of you stopped by to chat. As an author for a digital publisher, it’s hard sometimes to connect with the people that you’re writing for, but for me, that has become easier.

This year’s PyCon experience was overall fairly balanced as I got to do a lot of networking, seeing talks and tutorials, and even volunteering. The only thing that I could think that I wish I could’ve done was to give a presentation of some sort. That’s what next year is for though.

Dan Bader

Hey I’m Dan Bader, and I’m the owner and editor-in-chief here at Real Python. I also do all of the back-end and front-end development for the Django-based CMS and other infrastructure that realpython.com runs on.

I’m a long-time Pythonista and super passionate about teaching Python. Publishing my own Python programming book was a lifelong dream and getting to run Real Python together with an awesome team now is the icing on the cake!

This year was my fifth PyCon, and it won’t be the last, that’s for sure…

My personal highlights were hanging out with the Real Python tutorial team and meeting many of them in person for the first time. That just made my heart jump! We went out for food and drinks a couple of nights, and one random memory that I’ll probably never forget is finding out how much Logan loves the Golden Girls TV show : - D

During the day, the nice folks at JetBrains gave us some space at their PyCharm booth, so we all got to wear our Real Python swag and chat with readers as well as hand out stickers and pins. That was super fun!

Another great memory is the Real Python open space we organized where several dozen readers and members showed up to say “hi.” That just blew my mind and was one of my favorite PyCon experiences so far!

I felt pretty burned out when I first arrived at PyCon after working non-stop for a couple of months on the launch of our video subscriptions feature, and seeing this massive turnout felt amazing. I didn’t really know what to say at first! Thanks everybody for stopping by :)

This year was also the second time we hosted an open-space meetup for the PythonistaCafe forum, and it was super cool to see how this wacky little project has sparked such a tightly knit and awesome community now.

Another highlight was recording a live episode of the Talk Python podcast together with Mike Kennedy. Always a treat hanging out with this guy :)

I wish I would’ve had more time to go to talks. I was super impressed with the talks that Geir Arne and Anthony from the Real Python Tutorial Team gave. Be sure to check out the recordings when you have the chance.

Happy Pythoning, and see you next year!

Summary

Regardless of the location, a PyCon is a joyous experience. Whether you are meeting colleagues, online friends, or strangers who share a passion other than Python, it’s the people that make the experience special. The old saying, “came for the language, but stayed for the community” never rings more true than at PyCon US.

No two Pythonistas are the same, and there is something for everyone in the diversity of the events, talks, and tutorials taking place. Personally, I’m looking forward to next year’s PyCon already. Schedule permitting, I’ll make every effort to join the sprints in 2020. But even if I don’t get to tick that off my list, I already know that next year I’ll be excited to meet you and many others in spite of my introversion.

If you were at PyCon US and came to say “hi,” or joined us for the Real Python or PythonistaCafe open spaces, then leave a comment below and let us know what your favorite part of PyCon was. What’s your one tip you would give someone who will be attending any PyCon for the first time next year? Happy coding!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Build a Recommendation Engine With Collaborative Filtering

2019-07-10T14:00:00+00:00

Collaborative Filtering is the most common technique used when it comes to building intelligent recommender systems that can learn to give better recommendations as more information about users is collected.

Most websites like Amazon, YouTube, and Netflix use collaborative filtering as a part of their sophisticated recommendation systems. You can use this technique to build recommenders that give suggestions to a user on the basis of the likes and dislikes of similar users.

In this article, you’ll learn about:

Collaborative filtering and it types
Data needed to build a recommender
Libraries available in Python to build recommenders
Use cases and challenges of collaborative filtering

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

What Is Collaborative Filtering?

Collaborative filtering is a technique that can filter out items that a user might like on the basis of reactions by similar users.

It works by searching a large group of people and finding a smaller set of users with tastes similar to a particular user. It looks at the items they like and combines them to create a ranked list of suggestions.

There are many ways to decide which users are similar and combine their choices to create a list of recommendations. This article will show you how to do that with Python.

The Dataset

To experiment with recommendation algorithms, you’ll need data that contains a set of items and a set of users who have reacted to some of the items.

The reaction can be explicit (rating on a scale of 1 to 5, likes or dislikes) or implicit (viewing an item, adding it to a wish list, the time spent on an article).

While working with such data, you’ll mostly see it in the form of a matrix consisting of the reactions given by a set of users to some items from a set of items. Each row would contain the ratings given by a user, and each column would contain the ratings received by an item. A matrix with five users and five items could look like this:

Rating Matrix

The matrix shows five users who have rated some of the items on a scale of 1 to 5. For example, the first user has given a rating 4 to the third item.

In most cases, the cells in the matrix are empty, as users only rate a few items. It’s highly unlikely for every user to rate or react to every item available. A matrix with mostly empty cells is called sparse, and the opposite to that (a mostly filled matrix) is called dense.

There are a lot of datasets that have been collected and made available to the public for research and benchmarking. Here’s a list of high-quality data sources that you can choose from.

The best one to get started would be the MovieLens dataset collected by GroupLens Research. In particular, the MovieLens 100k dataset is a stable benchmark dataset with 100,000 ratings given by 943 users for 1682 movies, with each user having rated at least 20 movies.

This dataset consists of many files that contain information about the movies, the users, and the ratings given by users to the movies they have watched. The ones that are of interest are the following:

u.item: the list of movies
u.data: the list of ratings given by users

The file u.data that contains the ratings is a tab separated list of user ID, item ID, rating, and timestamp. The first few lines of the file look like this:

First 5 Rows of MovieLens 100k Data

As shown above, the file tells what rating a user gave to a particular movie. This file contains 100,000 such ratings, which will be used to predict the ratings of the movies not seen by the users.

Steps Involved in Collaborative Filtering

To build a system that can automatically recommend items to users based on the preferences of other users, the first step is to find similar users or items. The second step is to predict the ratings of the items that are not yet rated by a user. So, you will need the answers to these questions:

How do you determine which users or items are similar to one another?
Given that you know which users are similar, how do you determine the rating that a user would give to an item based on the ratings of similar users?
How do you measure the accuracy of the ratings you calculate?

The first two questions don’t have single answers. Collaborative filtering is a family of algorithms where there are multiple ways to find similar users or items and multiple ways to calculate rating based on ratings of similar users. Depending on the choices you make, you end up with a type of collaborative filtering approach. You’ll get to see the various approaches to find similarity and predict ratings in this article.

One important thing to keep in mind is that in an approach based purely on collaborative filtering, the similarity is not calculated using factors like the age of users, genre of the movie, or any other data about users or items. It is calculated only on the basis of the rating (explicit or implicit) a user gives to an item. For example, two users can be considered similar if they give the same ratings to ten movies despite there being a big difference in their age.

The third question for how to measure the accuracy of your predictions also has multiple answers, which include error calculation techniques that can be used in many places and not just recommenders based on collaborative filtering.

One of the approaches to measure the accuracy of your result is the Root Mean Square Error (RMSE), in which you predict ratings for a test dataset of user-item pairs whose rating values are already known. The difference between the known value and the predicted value would be the error. Square all the error values for the test set, find the average (or mean), and then take the square root of that average to get the RMSE.

Another metric to measure the accuracy is Mean Absolute Error (MAE), in which you find the magnitude of error by finding its absolute value and then taking the average of all error values.

You don’t need to worry about the details of RMSE or MAE at this point as they are readily available as part of various packages in Python, and you will see them later in the article.

Now let’s look at the different types of algorithms in the family of collaborative filtering.

Memory Based

The first category includes algorithms that are memory based, in which statistical techniques are applied to the entire dataset to calculate the predictions.

To find the rating R that a user U would give to an item I, the approach includes:

Finding users similar to U who have rated the item I
Calculating the rating R based the ratings of users found in the previous step

You’ll see each of them in detail in the following sections.

How to Find Similar Users on the Basis of Ratings

To understand the concept of similarity, let’s create a simple dataset first.

The data includes four users A, B, C, and D, who have rated two movies. The ratings are stored in lists, and each list contains two numbers indicating the rating of each movie:

Ratings by A are [1.0, 2.0].
Ratings by B are [2.0, 4.0].
Ratings by C are [2.5, 4.0].
Ratings by D are [4.5, 5.0].

To start off with a visual clue, plot the ratings of two movies given by the users on a graph and look for a pattern. The graph looks like this:

In the graph above, each point represents a user and is plotted against the ratings they gave to two movies.

Looking at the distance between the points seems to be a good way to estimate similarity, right? You can find the distance using the formula for Euclidean distance between two points. You can use the function available in scipy as shown in the following program:

>>>

>>> from scipy import spatial

>>> a = [1, 2]
>>> b = [2, 4]
>>> c = [2.5, 4]
>>> d = [4.5, 5]

>>> spatial.distance.euclidean(c, a)
2.5
>>> spatial.distance.euclidean(c, b)
0.5
>>> spatial.distance.euclidean(c, d)
2.23606797749979

As shown above, you can use scipy.spatial.distance.euclidean to calculate the distance between two points. Using it to calculate the distance between the ratings of A, B, and D to that of C shows us that in terms of distance, the ratings of C are closest to those of B.

You can see that user C is closest to B even by looking at the graph. But out of A and D only, who is C closer to?

You could say C is closer to D in terms of distance. But looking at the rankings, it would seem that the choices of C would align with that of A more than D because both A and C like the second movie almost twice as much as they like the first movie, but D likes both of the movies equally.

So, what can you use to identify such patterns that Euclidean distance cannot? Can the angle between the lines joining the points to the origin be used to make a decision? You can take a look at the angle between the lines joining the origin of the graph to the respective points as shown:

The graph shows four lines joining each point to the origin. The lines for A and B are coincident, making the angle between them zero.

You can consider that, if the angle between the lines is increased, then the similarity decreases, and if the angle is zero, then the users are very similar.

To calculate similarity using angle, you need a function that returns a higher similarity or smaller distance for a lower angle and a lower similarity or larger distance for a higher angle. The cosine of an angle is a function that decreases from 1 to -1 as the angle increases from 0 to 180.

You can use the cosine of the angle to find the similarity between two users. The higher the angle, the lower will be the cosine and thus, the lower will be the similarity of the users. You can also inverse the value of the cosine of the angle to get the cosine distance between the users by subtracting it from 1.

scipy has a function that calculates the cosine distance of vectors. It returns a higher value for higher angle:

>>>

>>> from scipy import spatial
>>> a = [1, 2]
>>> b = [2, 4]
>>> c = [2.5, 4]
>>> d = [4.5, 5]

>>> spatial.distance.cosine(c,a)
0.004504527406047898

>>> spatial.distance.cosine(c,b)
0.004504527406047898

>>> spatial.distance.cosine(c,d)
0.015137225946083022

>>> spatial.distance.cosine(a,b)
0.0

The lower angle between the vectors of C and A gives a lower cosine distance value. If you want to rank user similarities in this way, use cosine distance.

Note: In the above example, only two movies are considered, which makes it easier to visualize the rating vectors in two dimensions. This is only done to make the explanation easier.

Real use cases with multiple items would involve more dimensions in rating vectors. You might want to go into the mathematics of cosine similarity as well.

Notice that users A and B are considered absolutely similar in the cosine similarity metric despite having different ratings. This is actually a common occurrence in the real world, and the users like the user A are what you can call tough raters. An example would be a movie critic who always gives out ratings lower than the average, but the rankings of the items in their list would be similar to the Average raters like B.

To factor in such individual user preferences, you will need to bring all users to the same level by removing their biases. You can do this by subtracting the average rating given by that user to all items from each item rated by that user. Here’s what it would look like:

For user A, the rating vector [1, 2] has the average 1.5. Subtracting 1.5 from every rating would give you the vector [-0.5, 0.5].
For user B, the rating vector [2, 4] has the average 3. Subtracting 3 from every rating would give you the vector [-1, 1].

By doing this, you have changed the value of the average rating given by every user to 0. Try doing the same for users C and D, and you’ll see that the ratings are now adjusted to give an average of 0 for all users, which brings them all to the same level and removes their biases.

The cosine of the angle between the adjusted vectors is called centered cosine. This approach is normally used when there are a lot of missing values in the vectors, and you need to place a common value to fill up the missing values.

Filling up the missing values in the ratings matrix with a random value could result in inaccuracies. A good choice to fill the missing values could be the average rating of each user, but the original averages of user A and B are 1.5 and 3 respectively, and filling up all the empty values of A with 1.5 and those of B with 3 would make them dissimilar users.

But after adjusting the values, the centered average of both users is 0, which allows you to capture the idea of the item being above or below average more accurately for both users with all missing values in both user’s vectors having the same value 0.

Euclidean distance and cosine similarity are some of the approaches that you can use to find users similar to one another and even items similar to one another. (The function used above calculates cosine distance. To calculate cosine similarity, subtract the distance from 1.)

Note: The formula for centered cosine is the same as that for Pearson correlation coefficient. You will find that many resources and libraries on recommenders refer to the implementation of centered cosine as Pearson Correlation.

How to Calculate the Ratings

After you have determined a list of users similar to a user U, you need to calculate the rating R that U would give to a certain item I. Again, just like similarity, you can do this in multiple ways.

You can predict that a user’s rating R for an item I will be close to the average of the ratings given to I by the top 5 or top 10 users most similar to U. The mathematical formula for the average rating given by n users would look like this:

This formula shows that the average rating given by the n similar users is equal to the sum of the ratings given by them divided by the number of similar users, which is n.

There will be situations where the n similar users that you found are not equally similar to the target user U. The top 3 of them might be very similar, and the rest might not be as similar to U as the top 3. In that case, you could consider an approach where the rating of the most similar user matters more than the second most similar user and so on. The weighted average can help us achieve that.

In the weighted average approach, you multiply each rating by a similarity factor(which tells how similar the users are). By multiplying with the similarity factor, you add weights to the ratings. The heavier the weight, the more the rating would matter.

The similarity factor, which would act as weights, should be the inverse of the distance discussed above because less distance implies higher similarity. For example, you can subtract the cosine distance from 1 to get cosine similarity.

With the similarity factor S for each user similar to the target user U, you can calculate the weighted average using this formula:

In the above formula, every rating is multiplied by the similarity factor of the user who gave the rating. The final predicted rating by user U will be equal to the sum of the weighted ratings divided by the sum of the weights.

Note: In case you’re wondering why the sum of weighted ratings is being divided by the sum of the weights and not by n, consider this: in the previous formula of the average, where you divided by n, the value of the weight was 1.

The denominator is always the sum of weights when it comes to finding averages, and in the case of the normal average, the weight being 1 means the denominator would be equal to n.

With a weighted average, you give more consideration to the ratings of similar users in order of their similarity.

Now, you know how to find similar users and how to calculate ratings based on their ratings. There’s also a variation of collaborative filtering where you predict ratings by finding items similar to each other instead of users and calculating the ratings. You’ll read about this variation in the next section.

User-Based vs Item-Based Collaborative Filtering

The technique in the examples explained above, where the rating matrix is used to find similar users based on the ratings they give, is called user-based or user-user collaborative filtering. If you use the rating matrix to find similar items based on the ratings given to them by users, then the approach is called item-based or item-item collaborative filtering.

The two approaches are mathematically quite similar, but there is a conceptual difference between the two. Here’s how the two compare:

User-based: For a user U, with a set of similar users determined based on rating vectors consisting of given item ratings, the rating for an item I, which hasn’t been rated, is found by picking out N users from the similarity list who have rated the item I and calculating the rating based on these N ratings.
Item-based: For an item I, with a set of similar items determined based on rating vectors consisting of received user ratings, the rating by a user U, who hasn’t rated it, is found by picking out N items from the similarity list that have been rated by U and calculating the rating based on these N ratings.

Item-based collaborative filtering was developed by Amazon. In a system where there are more users than items, item-based filtering is faster and more stable than user-based. It is effective because usually, the average rating received by an item doesn’t change as quickly as the average rating given by a user to different items. It’s also known to perform better than the user-based approach when the ratings matrix is sparse.

Although, the item-based approach performs poorly for datasets with browsing or entertainment related items such as MovieLens, where the recommendations it gives out seem very obvious to the target users. Such datasets see better results with matrix factorization techniques, which you’ll see in the next section, or with hybrid recommenders that also take into account the content of the data like the genre by using content-based filtering.

You can use the library Surprise to experiment with different recommender algorithms quickly. (You will see more about this later in the article.)

Model Based

The second category covers the Model based approaches, which involve a step to reduce or compress the large but sparse user-item matrix. For understanding this step, a basic understanding of dimensionality reduction can be very helpful.

Dimensionality Reduction

In the user-item matrix, there are two dimensions:

The number of users
The number of items

If the matrix is mostly empty, reducing dimensions can improve the performance of the algorithm in terms of both space and time. You can use various methods like matrix factorization or autoencoders to do this.

Matrix factorization can be seen as breaking down a large matrix into a product of smaller ones. This is similar to the factorization of integers, where 12 can be written as 6 x 2 or 4 x 3. In the case of matrices, a matrix A with dimensions m x n can be reduced to a product of two matrices X and Y with dimensions m x p and p x n respectively.

Note: In matrix multiplication, a matrix X can be multiplied by Y only if the number of columns in X is equal to the number of rows in Y. Therefore the two reduced matrices have a common dimension p.

Depending on the algorithm used for dimensionality reduction, the number of reduced matrices can be more than two as well.

The reduced matrices actually represent the users and items individually. The m rows in the first matrix represent the m users, and the p columns tell you about the features or characteristics of the users. The same goes for the item matrix with n items and p characteristics. Here’s an example of how matrix factorization looks:

Matrix Factorization

In the image above, the matrix is reduced into two matrices. The one on the left is the user matrix with m users, and the one on top is the item matrix with n items. The rating 4 is reduced or factorized into:

A user vector (2, -1)
An item vector (2.5, 1)

The two columns in the user matrix and the two rows in the item matrix are called latent factors and are an indication of hidden characteristics about the users or the items. A possible interpretation of the factorization could look like this:

Assume that in a user vector (u, v), u represents how much a user likes the Horror genre, and v represents how much they like the Romance genre.
The user vector (2, -1) thus represents a user who likes horror movies and rates them positively and dislikes movies that have romance and rates them negatively.
Assume that in an item vector (i, j), i represents how much a movie belongs to the Horror genre, and j represents how much that movie belongs to the Romance genre.
The movie (2.5, 1) has a Horror rating of 2.5 and a Romance rating of 1. Multiplying it by the user vector using matrix multiplication rules gives you (2 * 2.5) + (-1 * 1) = 4.
So, the movie belonged to the Horror genre, and the user could have rated it 5, but the slight inclusion of Romance caused the final rating to drop to 4.

The factor matrices can provide such insights about users and items, but in reality they are usually much more complex than the explanation given above. The number of such factors can be anything from one to hundreds or even thousands. This number is one of the things that need to be optimized during the training of the model.

In the example, you had two latent factors for movie genres, but in real scenarios, these latent factors need not be analyzed too much. These are patterns in the data that will play their part automatically whether you decipher their underlying meaning or not.

The number of latent factors affects the recommendations in a manner where the greater the number of factors, the more personalized the recommendations become. But too many factors can lead to overfitting in the model.

Note: Overfitting happens when the model trains to fit the training data so well that it doesn’t perform well with new data.

Algorithms for Matrix Factorization

One of the popular algorithms to factorize a matrix is the singular value decomposition (SVD) algorithm. SVD came into the limelight when matrix factorization was seen performing well in the Netflix prize competition. Other algorithms include PCA and its variations, NMF, and so on. Autoencoders can also be used for dimensionality reduction in case you want to use Neural Networks.

You can find the implementations of these algorithms in various libraries for Python so you don’t need to worry about the details at this point. But in case you want to read more, the chapter on dimensionality reduction in the book Mining of Massive Datasets is worth a read.

Using Python to Build Recommenders

There are quite a few libraries and toolkits in Python that provide implementations of various algorithms that you can use to build a recommender. But the one that you should try out while understanding recommendation systems is Surprise.

Surprise is a Python SciKit that comes with various recommender algorithms and similarity metrics to make it easy to build and analyze recommenders.

Here’s how to install it using pip:

$ pip install numpy
$ pip install scikit-surprise

Here’s how to install it using conda:

$ conda install -c conda-forge scikit-surprise

Note: Installing Pandas is also recommended if you wish to follow the examples.

To use Surprise, you should first know some of the basic modules and classes available in it:

The Dataset module is used to load data from files, Pandas dataframes, or even built-in datasets available for experimentation. (MovieLens 100k is one of the built-in datasets in Surprise.) To load a dataset, some of the available methods are:
- Dataset.load_builtin()
- Dataset.load_from_file()
- Dataset.load_from_df()
The Reader class is used to parse a file containing ratings. The default format in which it accepts data is that each rating is stored in a separate line in the order user item rating. This order and the separator can be configured using parameters:
- line_format is a string that stores the order of the data with field names separated by a space, as in "item user rating".
- sep is used to specify separator between fields, such as ','.
- rating_scale is used to specify the rating scale. The default is (1, 5).
- skip_lines is used to indicate the number of lines to skip at the beginning of the file. The default is 0.

Here’s a program that you can use to load data from a Pandas dataframe or the from builtin MovieLens 100k dataset:

# load_data.py

import pandas as pd
from surprise import Dataset
from surprise import Reader

# This is the same data that was plotted for similarity earlier
# with one new user "E" who has rated only movie 1
ratings_dict = {
    "item": [1, 2, 1, 2, 1, 2, 1, 2, 1],
    "user": ['A', 'A', 'B', 'B', 'C', 'C', 'D', 'D', 'E'],
    "rating": [1, 2, 2, 4, 2.5, 4, 4.5, 5, 3],
}

df = pd.DataFrame(ratings_dict)
reader = Reader(rating_scale=(1, 5))

# Loads Pandas dataframe
data = Dataset.load_from_df(df[["user", "item", "rating"]], reader)
# Loads the builtin Movielens-100k data
movielens = Dataset.load_builtin('ml-100k')

In the above program, the data is stored in a dictionary that is loaded into a Pandas dataframe and then into a Dataset object from Surprise.

Algorithms Based on K-Nearest Neighbours (k-NN)

The choice of algorithm for the recommender function depends on the technique you want to use. For the memory-based approaches discussed above, the algorithm that would fit the bill is Centered k-NN because the algorithm is very close to the centered cosine similarity formula explained above. It is available in Surprise as KNNWithMeans.

To find the similarity, you simply have to configure the function by passing a dictionary as an argument to the recommender function. The dictionary should have the required keys, such as the following:

name contains the similarity metric to use. Options are cosine, msd, pearson, or pearson_baseline. The default is msd.
user_based is a boolean that tells whether the approach will be user-based or item-based. The default is True, which means the user-based approach will be used.
min_support is the minimum number of common items needed between users to consider them for similarity. For the item-based approach, this corresponds to the minimum number of common users for two items.

The following program configures the KNNWithMeans function:

# recommender.py

from surprise import KNNWithMeans

# To use item-based cosine similarity
sim_options = {
    "name": "cosine",
    "user_based": False,  # Compute  similarities between items
}
algo = KNNWithMeans(sim_options=sim_options)

The recommender function in the above program is configured to use the cosine similarity and to find similar items using the item-based approach.

To try out this recommender, you need to create a Trainset from data. Trainset is built using the same data but contains more information about the data, such as the number of users and items (n_users, n_items) that are used by the algorithm. You can create it either by using the entire data or a part of the data. You can also divide the data into folds where some of the data will be used for training and some for testing.

Note: Using only one pair of training and testing data is usually not enough. When you split the original dataset into training and testing data, you should create more than one pair to allow for multiple observations with variations in the training in testing data.

Algorithms should be cross-validated using multiple folds. By using different pairs, you’ll see different results given by your recommender. MovieLens 100k provides five different splits of training and testing data: u1.base, u1.test, u2.base, u2.test … u5.base, u5.test, for a 5-fold cross-validation

Here’s an example to find out how the user E would rate the movie 2:

>>>

>>> from load_data import data
>>> from recommender import algo

>>> trainingSet = data.build_full_trainset()

>>> algo.fit(trainingSet)
Computing the cosine similarity matrix...
Done computing similarity matrix.
<surprise.prediction_algorithms.knns.KNNWithMeans object at 0x7f04fec56898>

>>> prediction = algo.predict('E', 2)
>>> prediction.est
4.15

The algorithm predicted that the user E would rate the movie 4.15, which could be high enough to be shown as a recommendation.

You should try out the different k-NN based algorithms along with different similarity options and matrix factorization algorithms available in the Surprise library. Try them out on the MovieLens dataset to see if you can beat some benchmarks. The next section will cover how to use Surprise to check which parameters perform best for your data.

Tuning the Algorithm Parameters

Surprise provides a GridSearchCV class analogous to GridSearchCV from scikit-learn.

With a dict of all parameters, GridSearchCV tries all the combinations of parameters and reports the best parameters for any accuracy measure

For example, you can check which similarity metric works best for your data in memory-based approaches:

from surprise import KNNWithMeans
from surprise import Dataset
from surprise.model_selection import GridSearchCV

data = Dataset.load_builtin("ml-100k")
sim_options = {
    "name": ["msd", "cosine"],
    "min_support": [3, 4, 5],
    "user_based": [False, True],
}

param_grid = {"sim_options": sim_options}

gs = GridSearchCV(KNNWithMeans, param_grid, measures=["rmse", "mae"], cv=3)
gs.fit(data)

print(gs.best_score["rmse"])
print(gs.best_params["rmse"])

The output of the above program is as follows:

0.9434791128171457
{'sim_options': {'name': 'msd', 'min_support': 3, 'user_based': False}}

So, for the MovieLens 100k dataset, Centered-KNN algorithm works best if you go with item-based approach and use msd as the similarity metric with minimum support 3.

Similarly, for model-based approaches, we can use Surprise to check which values for the following factors work best:

n_epochs is the number of iterations of SGD, which is basically an iterative method used in Statistics to minimize a function.
lr_all is the learning rate for all parameters, which is a parameter that decides how much the parameters are adjusted in each iteration.
reg_all is the regularization term for all parameters, which is a penalty term added to prevent overfitting.

Note: Keep in mind that there won’t be any similarity metrics in matrix factorization algorithms as the latent factors take care of similarity among users or items.

The following program will check the best values for the SVD algorithm, which is a matrix factorization algorithm:

from surprise import SVD
from surprise import Dataset
from surprise.model_selection import GridSearchCV

data = Dataset.load_builtin("ml-100k")

param_grid = {
    "n_epochs": [5, 10],
    "lr_all": [0.002, 0.005],
    "reg_all": [0.4, 0.6]
}
gs = GridSearchCV(SVD, param_grid, measures=["rmse", "mae"], cv=3)

gs.fit(data)

print(gs.best_score["rmse"])
print(gs.best_params["rmse"])

The output of the above program is as follows:

0.9642278631521038
{'n_epochs': 10, 'lr_all': 0.005, 'reg_all': 0.4}

So, for the MovieLens 100k dataset, the SVD algorithm works best if you go with 10 epochs and use a learning rate of 0.005 and 0.4 regularization.

Other Matrix Factorization based algorithms available in Surprise are SVD++ and NMF.

Following these examples, you can dive deep into all the parameters that can be used in these algorithms. You should definitely check out the mathematics behind them. Since you won’t have to worry much about the implementation of algorithms initially, recommenders can be a great way to segue into the field of machine learning and build an application based on that.

When Can Collaborative Filtering Be Used?

Collaborative filtering works around the interactions that users have with items. These interactions can help find patterns that the data about the items or users itself can’t. Here are some points that can help you decide if collaborative filtering can be used:

Collaborative filtering doesn’t require features about the items or users to be known. It is suited for a set of different types of items, for example, a supermarket’s inventory where items of various categories can be added. In a set of similar items such as that of a bookstore, though, known features like writers and genres can be useful and might benefit from content-based or hybrid approaches.
Collaborative filtering can help recommenders to not overspecialize in a user’s profile and recommend items that are completely different from what they have seen before. If you want your recommender to not suggest a pair of sneakers to someone who just bought another similar pair of sneakers, then try to add collaborative filtering to your recommender spell.

Although collaborative Filtering is very commonly used in recommenders, some of the challenges that are faced while using it are the following:

Collaborative filtering can lead to some problems like cold start for new items that are added to the list. Until someone rates them, they don’t get recommended.
Data sparsity can affect the quality of user-based recommenders and also add to the cold start problem mentioned above.
Scaling can be a challenge for growing datasets as the complexity can become too large. Item-based recommenders are faster than user-based when the dataset is large.
With a straightforward implementation, you might observe that the recommendations tend to be already popular, and the items from the long tail section might get ignored.

With every type of recommender algorithm having its own list of pros and cons, it’s usually a hybrid recommender that comes to the rescue. The benefits of multiple algorithms working together or in a pipeline can help you set up more accurate recommenders. In fact, the solution of the winner of the Netflix prize was also a complex mix of multiple algorithms.

Conclusion

You now know what calculations go into a collaborative-filtering type recommender and how to try out the various types of algorithms quickly on your dataset to see if collaborative filtering is the way to go. Even if it does not seem to fit your data with high accuracy, some of the use cases discussed might help you plan things in a hybrid way for the long term.

Here are some resources for more implementations and further reading on collaborative filtering and other recommendation algorithms.

Libraries:

LightFM: a hybrid recommendation algorithm in Python
Python-recsys: a Python library for implementing a recommender system

Research papers:

Item Based Collaborative Filtering Recommendation Algorithms: the first paper published on item-based recommenders
Using collaborative filtering to weave an information tapestry: the first use of the term collaborative filtering

Books:

Mining of Massive Datasets by Jure Leskovec, Anand Rajaraman, Jeff Ullman
Programming Collective Intelligence by Toby Segaran

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Reading and Writing Files in Python

2019-07-09T14:00:00+00:00

In this course, you’ll learn about reading and writing files in Python. You’ll cover everything from what a file is made up of to which libraries can help you along that way. You’ll also take a look at some basic scenarios of file usage as well as some advanced techniques.

One of the most common tasks that you can do with Python is reading and writing files. Whether it’s writing to a simple text file, reading a complicated server log, or even analyzing raw byte data, all of these situations require reading or writing a file.

By the end of this course, you’ll know:

What makes up a file and why that’s important in Python
The basics of reading and writing files in Python
Some basic scenarios of reading and writing files

This tutorial is mainly for beginner to intermediate Pythonistas, but there are some tips in here that more advanced programmers may appreciate as well.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Python Square Root Function

2019-07-08T14:00:00+00:00

Are you trying to solve a quadratic equation? Maybe you need to calculate the length of one side of a right triangle. For these types of equations and more, the Python square root function, sqrt(), can help you quickly and accurately calculate your solutions.

By the end of this article, you’ll learn:

What a square root is
How to use the Python square root function, sqrt()
When sqrt() can be useful in the real world

Let’s dive in!

Python Pit Stop: This tutorial is a quick and practical way to find the info you need, so you’ll be back to your project in no time!

Free Bonus: Click here to get our free Python Cheat Sheet that shows you the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Square Roots in Mathematics

In algebra, a square, x, is the result of a number, n, multiplied by itself: x = n²

You can calculate squares using Python:

>>>

>>> n = 5
>>> x = n ** 2
>>> x
25

The Python ** operator is used for calculating the power of a number. In this case, 5 squared, or 5 to the power of 2, is 25.

The square root, then, is the number n, which when multiplied by itself yields the square, x.

In this example, n, the square root, is 5.

25 is an example of a perfect square. Perfect squares are the squares of integer values:

>>>

>>> 1 ** 2
1

>>> 2 ** 2
4

>>> 3 ** 2
9

You might have memorized some of these perfect squares when you learned your multiplication tables in an elementary algebra class.

If you’re given a small perfect square, it may be straightforward enough to calculate or memorize its square root. But for most other squares, this calculation can get a bit more tedious. Often, an estimation is good enough when you don’t have a calculator.

Fortunately, as a Python developer, you do have a calculator, namely the Python interpreter!

The Python Square Root Function

Python’s math module, in the standard library, can help you work on math-related problems in code. It contains many useful functions, such as remainder() and factorial(). It also includes the Python square root function, sqrt().

You’ll begin by importing math:

>>>

>>> import math

That’s all it takes! You can now use math.sqrt() to calculate square roots.

sqrt() has a straightforward interface.

It takes one parameter, x, which (as you saw before) stands for the square for which you are trying to calculate the square root. In the example from earlier, this would be 25.

The return value of sqrt() is the square root of x, as a floating point number. In the example, this would be 5.0.

Let’s take a look at some examples of how to (and how not to) use sqrt().

The Square Root of a Positive Number

One type of argument you can pass to sqrt() is a positive number. This includes both int and float types.

For example, you can solve for the square root of 49 using sqrt():

>>>

>>> math.sqrt(49)
7.0

The return value is 7.0 (the square root of 49) as a floating point number.

Along with integers, you can also pass float values:

>>>

>>> math.sqrt(70.5)
8.396427811873332

You can verify the accuracy of this square root by calculating its inverse:

>>>

>>> 8.396427811873332 ** 2
70.5

The Square Root of Zero

Even 0 is a valid square to pass to the Python square root function:

>>>

>>> math.sqrt(0)
0.0

While you probably won’t need to calculate the square root of zero often, you may be passing a variable to sqrt() whose value you don’t actually know. So, it’s good to know that it can handle zero in those cases.

The Square Root of Negative Numbers

The square of any real number cannot be negative. This is because a negative product is only possible if one factor is positive and the other is negative. A square, by definition, is the product of a number and itself, so it’s impossible to have a negative real square:

>>>

>>> math.sqrt(-25)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: math domain error

If you attempt to pass a negative number to sqrt(), then you’ll get a ValueError because negative numbers are not in the domain of possible real squares. Instead, the square root of a negative number would need to be complex, which is outside the scope of the Python square root function.

Square Roots in the Real World

To see a real-world application of the Python square root function, let’s turn to the sport of tennis.

Imagine that Rafael Nadal, one of the fastest players in the world, has just hit a forehand from the back corner, where the baseline meets the sideline of the tennis court:

Now, assume his opponent has countered with a drop shot (one that would place the ball short with little forward momentum) to the opposite corner, where the other sideline meets the net:

How far must Nadal run to reach the ball?

You can determine from regulation tennis court dimensions that the baseline is 27 feet long, and the sideline (on one side of the net) is 39 feet long. So, essentially, this boils down to solving for the hypotenuse of a right triangle:

Using a valuable equation from geometry, the Pythagorean theorem, we know that a² + b² = c², where a and b are the legs of the right triangle and c is the hypotenuse.

Therefore, we can calculate the distance Nadal must run by rearranging the equation to solve for c:

You can solve this equation using the Python square root function:

>>>

>>> a = 27
>>> b = 39
>>> math.sqrt(a ** 2 + b ** 2)
47.43416490252569

So, Nadal must run about 47.4 feet (14.5 meters) in order to reach the ball and save the point.

Conclusion

Congratulations! You now know all about the Python square root function.

You’ve covered:

A brief introduction to square roots
The ins and outs of the Python square root function, sqrt()
A practical application of sqrt() using a real-world example

Knowing how to use sqrt() is only half the battle. Understanding when to use it is the other. Now, you know both, so go and apply your newfound mastery of the Python square root function!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Use the Python or Operator

2019-07-03T14:00:00+00:00

There are three Boolean operators in Python: and, or, and not. With them, you can test conditions and decide which execution path your programs will take. In this tutorial, you’ll learn about the Python or operator and how to use it.

By the end of this tutorial, you’ll have learned:

How the Python or operator works
How to use the Python or operator in Boolean and non-Boolean contexts
What kind of programming problems you can solve by using or in Python
How to read and better understand other people’s code when they use some of the special features of the Python or operator

You’ll learn how to use the Python or operator by building some practical examples. Even if you don’t really use all the possibilities that the Python or operator offers, mastering it will allow you to write better code.

Boolean Logic

George Boole (1815–1864) developed what is now called Boolean algebra, which is the foundation of the digital logic behind computer hardware and programming languages.

Boolean algebra is built around the truth value of expressions and objects (whether they are true or false) and is based in the Boolean operations AND, OR, and NOT. These operations are implemented through logical or Boolean operators that allow you to create Boolean expressions, which are expressions that evaluate to true or false.

With the help of Boolean logic, you can evaluate conditions and decide what operations your programs will execute, depending on the truth value of those conditions. This is an important cornerstone in programming and provides you with the tools to decide the execution flow of your programs.

Let’s take a look at some of the basic concepts related to Boolean logic in Python:

Boolean is type of value that can be either True or False. In Python, the Boolean type is bool, which is a subtype of int.
Boolean values are the values True or False (with a capital T and F) in Python.
A Boolean variable is a variable that can be either True or False. Boolean variables are commonly used as flags to indicate whether specific conditions exist.
A Boolean expression is an expression that returns either True or False.
Boolean context can be if conditions and while loops, where Python expects an expression to evaluate to a Boolean value. You can use virtually any expression or object in a Boolean context, and Python will try to determine its truth value.
Operands are the subexpressions or objects involved in an expression (Boolean or not) and connected by an operator.
Boolean or logical operators are AND (logical AND or conjunction), OR (logical OR or disjunction), and NOT (logical NOT or negation). The keywords and, or, and not are the Python operators for these operations.

Now that you have a better context on Boolean logic, let’s proceed with some more Python-specific topics.

The Python Boolean Operators

Python has three Boolean operators that are typed out as plain English words:

and
or
not

These operators connect Boolean expressions (and objects) to create compound Boolean expressions.

The Python Boolean operators always take two Boolean expressions or two objects or a combination of them, so they’re considered binary operators.

In this tutorial, you’ll be covering the Python or operator, which is the operator that implements the logical OR operation in Python. You’ll learn how it works and how to use it.

How the Python `or` Operator Works

With the Boolean OR operator, you can connect two Boolean expressions into one compound expression. At least one subexpressions must be true for the compound expression to be considered true, and it doesn’t matter which. If both subexpressions are false, then the expression is false.

This is the general logic behind the OR operator. However, the Python or operator does all this and more, as you’ll see in the following sections.

Using `or` With Boolean Expressions

You’ll need two subexpressions to create a Boolean expression using the Python or operator as a connector. The basic syntax for a Boolean expression with or is as follows:

# Syntax for Boolean expression with or in Python
exp1 or exp2

If at least one of the subexpressions (exp1 or exp2) evaluates to True, then the expression is considered to be True. If both subexpressions evaluate to False, then the expression is False. This definition is called inclusive or, since it allows both possibilities as well as either.

Here’s a summary of the Python or operator’s behavior:

Result of `exp1`	Result of `exp2`	Result of `exp1 or exp2`
`True`	`True`	`True`
`True`	`False`	`True`
`False`	`True`	`True`
`False`	`False`	`False`

Table 1. Logical Python or Operator: Truth Table

This table summarizes the resulting truth value of a Boolean expression like exp1 or exp2 depending on the truth values of its subexpressions.

Let’s illustrate the resulting truth values shown in Table 1 by coding some practical examples:

>>>

>>> exp1 = 1 == 2
>>> exp1
False
>>> exp2 = 7 > 3
>>> exp2
True
>>> exp1 or exp2  # Return True, because exp2 is True
True
>>> exp2 or exp1  # Also returns True
True
>>> exp3 = 3 < 1
>>> exp1 or exp3  # Return False, because both are False
False

In the previous examples, whenever a subexpression is evaluated to True, the global result is True. On the other hand, if both subexpressions are evaluated to False, then the global result is also False.

Using `or` With Common Objects

In general, the operands of an expression involving an OR operation should have Boolean values as shown in Table 1 and return a truth value as a result. When it comes to objects, Python is not very strict about that and internally implements a set of rules to decide if an object is considered true or false:

By default, an object is considered true unless its class defines either a __bool__() method that returns False or a __len__() method that returns zero, when called with the object. Here are most of the built-in objects considered false:

constants defined to be false: None and False.

zero of any numeric type: 0, 0.0, 0j, Decimal(0), Fraction(0, 1)

empty sequences and collections: '', (), [], {}, set(), range(0)

(Source)

If the operands involved in an or operation are objects instead of Boolean expressions, then the Python or operator returns a true or false object, not the values True or False as you could expect. The truth value of this object is determined according to the rules you’ve seen before.

This means that Python doesn’t coerce the result of an or operation to a bool object. If you’re testing two objects using or in Python, then the operator will return the first object that evaluates to true or the last object in the expression, regardless of its truth value:

>>>

>>> 2 or 3
2
>>> 5 or 0.0
5
>>> [] or 3
3
>>> 0 or {}
{}

In the two first examples, the first operands (2 and 5) are true (nonzero), so the Python or operator always returns the first one.

In the last two examples, the left operand is false (an empty object). The Python or operator evaluates both operands and returns the object on the right, which may evaluate to either true or false.

Note: If you really need to get one of the values True or False from a Boolean expression involving objects, then you could use bool(obj), which is a built-in function that returns True or False depending on the truth value of obj.

You can summarize the behavior shown in the previous code as follows:

Left Object	Right Object	Result of `x or y`
`x`	`y`	`x`, if it evaluates to true, otherwise `y`.

Table 2. Python or Operator Behavior When Testing Objects Instead of Boolean Expressions

In short, the Python or operator returns the first object that evaluates to true or the last object in the expression, regardless of its truth value.

You can generalize this behavior by chaining several operations in a single expression like this:

a or b or c or d

In this example, the Python or operator returns the first true operand it finds, or the last one. This is the rule of thumb to memorize how or works in Python.

Mixing Boolean Expressions and Objects

You can also combine Boolean expressions and common Python objects in an or operation. In this case, the Python or operator will still return the first true operand or the last operand, but the returned value could be True or False or the object you’re testing:

Result of Expression	Result of Object	Result of `exp or obj`
`True`	`True`	`True`
`True`	`False`	`True`
`False`	`False`	`obj`
`False`	`True`	`obj`

Table 3. Python or Operator Behavior When Testing Objects and Boolean Expressions

Let’s see how this works with some examples:

>>>

>>> 2 < 4 or 2  # Case 1
True
>>> 2 < 4 or []  # Case 2
True
>>> 5 > 10 or []  # Case 3
[]
>>> 5 > 10 or 4  # Case 4
4

In Case 1 and Case 2, the subexpression 2 < 4 was evaluated to True, and the returned value was True. On the other hand, in Case 3 and Case 4, the subexpression 5 > 10 was evaluated to False, so the last operand was returned, and you got an empty list ([]) and an integer (4) instead of True or False.

As an exercise, you could try to extend Table 3 by reversing the order of the expressions in the third column, that is, use obj or exp and try to predict the results.

Short-Circuit Evaluation

Python can sometimes determine the truth value of a Boolean expression before it has evaluated all the subexpressions and objects involved. The Python or operator, for instance, stops evaluating operands as soon as it finds something that’s considered true. For example, the following expression is always True:

>>>

>>> True or 4 < 3
True

If the first operand in an or expression evaluates to true, regardless of the value of the second operand (4 < 3 is False), then the expression is considered to be true, and the second operand is never evaluated. This is called short-circuit (lazy) evaluation.

Let’s consider another example:

>>>

>>> def true_func():
...     print('Running true_func()')
...     return True
...
>>> def false_func():
...     print('Running false_func()')
...     return False
...
>>> true_func() or false_func()  # Case 1
Running true_func()
True
>>> false_func() or true_func()  # Case 2
Running false_func()
Running true_func()
True
>>> false_func() or false_func()  # Case 3
Running false_func()
Running false_func()
False
>>> true_func() or true_func()  # Case 4
Running true_func()
True

In Case 1, Python evaluated true_func(). Since it returns True, the next operand (false_func()) is not evaluated. Notice that the phrase Running false_func() is never printed. Finally, the whole expression is considered True.

Case 2 evaluates both functions, because the first operand (false_func()) is False. Then the operator returns the second result, that is, the value returned by true_func(), which is True.

Case 3 evaluates both functions, because both return False. The operation returns the last function’s return value, that is False, and the expression is considered to be False.

In Case 4, Python only evaluates the first function, which is True, and the expression is True.

In short-circuit (lazy) evaluation, the second operand on a Boolean expression is not evaluated if the value of the expression can be determined from the first operand alone. Python (like other languages) bypasses this second evaluation in favor of performance, because evaluating the second operand would be an unnecessary waste of CPU time.

Finally, when it comes to performance when you’re using the Python or operator, consider the follow:

The expressions on the right side of the Python or operator might call functions that perform substantial or important work, or have side effects that won’t happen if the short-circuit rule takes effect.
The condition that is more likely to be true might be the left-most condition. This approach can reduce the execution time of your programs, because this way Python is able to determine if the condition is true just by evaluating the first operand.

Section Recap

You’ve learned how the Python or operator works and have seen some of its main features and behaviors. You now know enough to continue leveling up by learning how to use the operator in solving real-world problems.

Before that, let’s recap some important points about or in Python:

It satisfies the general rules that a Boolean OR operator should follow. If one or both Boolean subexpression are true, then the result is true. Otherwise, if both subexpressions are false, then the result is false.
It returns objects instead of True or False values when it tests Python objects. This means that the expression x or y returns x if it’s evaluated to true, and otherwise returns y (regardless of its truth value).
It follows a predefined set of Python internal rules to determine the truth value of an object.
It stops evaluating operands as soon as it finds something that’s considered true. This is called short-circuit or lazy evaluation.

Now it’s time to learn where and how you can use this operator with the help of some examples.

Boolean Contexts

In this section, you’ll see some practical examples of how to use the Python or operator, and learn how to take advantage of its somewhat unusual behavior to write better Python code.

There are two main situations where you can say you’re working in a Boolean context in Python:

if statements: conditional execution
while loops: conditional repetition

With an if statement, you can decide your programs’ path of execution depending on the truth value of some conditions.

On the other hand, while loops allow you to repeat a piece of code as long as a given condition remains true.

These two structures are part of what you’d call control flow statements. They help you decide your programs’ execution path.

You can use the Python or operator to build Boolean expressions suitable for use with both if statement and while loops, as you’ll see in the next two sections.

`if` Statements

Let’s say you want to make sure that one of two conditions (or both) is true before you choose a certain path of execution. In this case, you can use the Python or operator to connect the conditions in one expression, and use that expression in an if statement.

Suppose you need to get a confirmation from the user to run some actions depending on the user’s answer:

>>>

>>> def answer():
...     ans = input('Do you...? (yes/no): ')
...     if ans.lower() == 'yes' or ans.lower() == 'y':
...         print(f'Positive answer: {ans}')
...     elif ans.lower() == 'no' or ans.lower() == 'n':
...         print(f'Negative answer: {ans}')
...
>>> answer()
Do you...? (yes/no): y
Positive answer: y
>>> answer()
Do you...? (yes/no): n
Negative answer: n

Here, you get the user’s input and assign it to ans. Then, the if statement starts checking the conditions from left to right. If at least one of them is evaluated to true, then it executes the if code block. The elif statement does the same.

In the first call to answer(), the user’s input was y, which satisfied the first condition, and the if code block was executed. In the second call, the user’s input (n) satisfied the second condition, so the elif code block ran. If the user input doesn’t satisfy any condition, then no code block is executed.

Another example could be when you’re trying to determine whether a number is outside a range. In this case, it’s also possible to use the Python or operator. The following code tests whether x is outside the range of 20 through 40:

>>>

>>> def my_range(x):
...     if x < 20 or x > 40:
...         print('Outside')
...     else:
...         print('Inside')
...
>>> my_range(25)
Inside
>>> my_range(18)
Outside

When you call my_range() with x=25, the if statement tests 25 < 20, which is False. Then it tests x > 40, which is also False. The final result is False, so the else block was executed.

On the other hand, 18 < 20 is evaluated to True. Then the Python or operator makes short-circuit evaluation, and the condition is considered to be True. The main block is executed, and the value is outside the range.

`while` Loops

while loops are another example of Boolean context where you can use the Python or operator. By using or in the loop’s header, you can test several conditions and run the body until all the conditions evaluate to false.

Suppose you need to measure the operating temperature of some industrial equipment until it gets into a range of 100 ºF to 140 ºF. To do so, you can use a while loop:

from time import sleep

temp = measure_temp()  # Initial temperature measurement

while temp < 100 or temp > 140:
    print('Temperature outside the recommended range')
    print('New Temperature measure in 30 seconds')
    sleep(30)
    print('Measuring Temperature...')
    temp = measure_temp()
    print(f'The new Temperature is {temp} ºF')

This is a toy example almost in pseudo code, but it illustrates the idea. Here, the while loop is running until temp is between 100 ºF and 140 ºF. If the temperature value is outside the range, then the loop’s body is run, and you’ll be measuring the temperature again. Once measure_temp() returns a value between 100 ºF and 140 ºF, the loop finishes. The temperature measurement is taken every 30 seconds by using sleep(30).

Note: In the previous code example, you used Python’s f-strings for string formatting, if you want to dive deeper into f-strings, then you can take a look at Python 3’s f-Strings: An Improved String Formatting Syntax (Guide).

Non-Boolean Contexts

You can take advantage of the special features of the Python or operator out of Boolean contexts. The rule of thumb is still that the result of your Boolean expressions is the first true operand or the last in the line.

Notice that the logical operators (or included) are evaluated before the assignment operator (=), so you can assign the result of a Boolean expression to a variable in the same way you do with a common expression:

>>>

>>> a = 1
>>> b = 2
>>> var1 = a or b
>>> var1
1
>>> a = None
>>> b = 2
>>> var2 = a or b
>>> var2
2
>>> a = []
>>> b = {}
>>> var3 = a or b
>>> var3
{}

Here, the or operator works as expected, returning the first true operand or the last operand if both are evaluated to false.

You can take advantage of this somewhat special behavior of or in Python to implement a Pythonic solution to some quite common programming problems. Let’s take a look at some real-world examples.

Default Values for Variables

One common way to use the Python or operator is to select an object from a set of objects according to its truth value. You can do this by using an assignment statement:

>>>

>>> x = a or b or None

Here, you assigned to x the first true object in the expression. If all objects (a and b in this case) are false objects, then the Python or operator returns None, which is the last operand. This works because the or operator returns one of its operands depending on their truth value.

You can also use this feature to assign a default value to your variables. The following example sets x to a if a is true, and to default otherwise:

>>>

>>> x = a or default

In the previous code, you assign a to x only if a is evaluated to true. Otherwise, x is assigned default.

Default `return` Values

You can manipulate the return value of some built-in functions at call time. Functions like max() and min(), which take an iterable as an argument and return a single value, could be your perfect candidate for this sort of hack.

If you supply an empty iterable to max() or min(), then you’ll get a ValueError. However, you can modify this behavior by using the Python or operator. Let’s take a look at the following code:

>>>

>>> lst = []  # Empty list to test max() and min()
>>> max(lst)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    max(lst)
ValueError: max() arg is an empty sequence
>>> min(lst)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    min(lst)
ValueError: min() arg is an empty sequence
>>> # Use Python or operator to modify this behavior
>>> max(lst or [0])  # Return 0
0
>>> min(lst or [0])  # Return 0
0

The default behavior of max() and min() is to raise a ValueError if you call them with an empty iterable. However, by using the Python or operator, you supply a default return value for these functions and override their default behavior.

Note: In the previous code example, you saw how Python raises exceptions when some problems occur. If you want to know more about exceptions in Python, then you can take a look Introduction to Python Exceptions.

Mutable Default Arguments

A common problem beginner Python programmers face is to try to use mutable objects as default arguments to functions.

Mutable values for default arguments can retain state between calls. This is often unexpected. It happens because default argument values are evaluated and saved only once, that is, when the def statement is run, not each time the resulting function is called. That’s why you need to be careful about changing mutable defaults inside functions.

Consider the following example:

>>>

>>> def mutable_default(lst=[]):  # Try to use a mutable value as default
...     lst.append(1)  # Change same object each time
...     print(lst)
...
>>> mutable_default(lst=[3, 2])  # Default not used
[3, 2, 1]
>>> mutable_default()  # Default used
[1]
>>> mutable_default()  # Default grows on each call
[1, 1]
>>> mutable_default()
[1, 1, 1]

Here, every call to mutable_default() appends 1 to the end of lst, because lst holds a reference to the same object (the default []). You don’t get a new list every time the function is called as you would expect.

If that’s not the behavior you want, then the traditional (and safest) solution is to move the default to the body of the function:

>>>

>>> def mutable_default(lst=None):  # Use None as formal default
...     if lst is None:
...         lst = []  # Default used? Then lst gets a new empty list.
...     lst.append(1)
...     print(lst)
...
>>> mutable_default(lst=[3, 2])  # Default not used
[3, 2, 1]
>>> mutable_default()  # Default used
[1]
>>> mutable_default()
[1]

With this implementation, you are ensuring that lst is set to an empty list every time you call mutable_default() with no argument, relying in the default value for lst.

The if statement in this example could almost be replaced by the assignment lst = lst or []. This way, if no argument is passed in to the function, then lst would default to None and the Python or operator would return the empty list on the right:

>>>

>>> def mutable_default(lst=None):  # Use None as formal default
...     lst = lst or []  # Default used? Then lst gets an empty list.
...     lst.append(1)
...     print(lst)
...
>>> mutable_default(lst=[3, 2])  # Default not used
[3, 2, 1]
>>> mutable_default()  # Default used
[1]
>>> mutable_default()
[1]

However, this isn’t exactly the same. For example, if an empty list is passed in, then the or operation would cause the function to modify and print a newly created list, rather than modifying and printing the originally passed-in list like the if version would do.

If you are pretty sure that you’ll only be using non-empty list objects, then you can use this approach. Otherwise, stick to the if version.

Zero Division

Zero division can be a common problem when you’re dealing with numeric calculations. To avoid this problem, it’s likely that you end up checking if the denominator is equal to 0 or not by using an if statement.

Let’s take a look at an example:

>>>

>>> def divide(a, b):
...     if not b == 0:
...         return a / b
...
>>> divide(15, 3)
5.0
>>> divide(0, 3)
0.0
>>> divide(15, 0)

Here, you tested if the denominator (b) was not equal to 0, and then you returned the result of the division operation. If b == 0 is evaluated to True, then divide() implicitly returns None. Let’s see how to get a similar result, but this time using the Python or operator:

>>>

>>> def divide(a, b):
...     return b == 0 or a / b
...
>>> divide(15, 3)
5.0
>>> divide(0, 3)
0.0
>>> divide(15, 0)
True

In this case, the Python or operator evaluates the first subexpression (b == 0). Only if this subexpression is False, the second subexpression (a / b) is evaluated, and the final result will be the division of a and b.

The difference with the previous example is that, if b == 0 is evaluated to True, then divide() returns True instead of the implicit None.

Multiple Expressions in `lambda`

Python provides lambda expressions, which allow you to create simple anonymous functions. The expression lambda parameters: expression yields a function object. This kind of function may be useful if you want to define simple callback and key functions.

The most common pattern for you to write a lambda function is to use a single expression as a return value. However, you can change this and let lambda execute several expressions by using the Python or operator:

>>>

>>> lambda_func = lambda hello, world: print(hello, end=' ') or print(world)
>>> lambda_func('Hello', 'World!')
Hello World!

With this example, you’ve forced lambda to run two expressions (print(hello, end=' ') and print(world)). But how does this code work? Well, here lambda runs a Boolean expression where two functions are executed.

When or evaluates the first function, it receives None, which is the implicit return value for print(). Since None is considered to be false, or continues to evaluate its second operand, and finally returns it as a result for the Boolean expression.

In this case, the value returned by the Boolean expression is also the value returned by lambda:

>>>

>>> result = lambda_func('Hello', 'World!')
Hello World!
>>> print(result)
None

Here, result holds a reference to the value returned by lambda, which is the same value returned by the Boolean expression.

Conclusion

You now know how the Python or operator works as well as how to use it for solving some common programming problems in Python.

Now that you know the basics of the Python or operator, you’ll be able to:

Use the Python or operator in Boolean and non-Boolean contexts
Solve several kind of programming problems by effectively using the Python or operator
Write better and more Pythonic code by taking advantage of the somewhat special features of or in Python
Read and better understand other people’s code when they make use of the Python or operator

In addition, you’ve learned a little bit about Boolean logic, as well as some of its main concepts in Python.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Functional Programming in Python

2019-07-02T14:00:00+00:00

In this course, you’ll learn how to approach functional programming in Python. You’ll start with the absolute basics of Functional Programming (FP). After that, you’ll see hands-on examples for common FP patterns available, like using immutable data structures and the filter(), map(), and reduce() functions. You’ll end the course with actionable tips for parallelizing your code to make it run faster.

You’ll cover:

What functional programming is
How you can use immutable data structures to represent your data
How to use filter(), map(), and reduce()
How to do parallel processing with multiprocessing and concurrent.futures

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Use Redis With Python

2019-07-01T14:00:00+00:00

In this tutorial, you’ll learn how to use Python with Redis (pronounced RED-iss, or maybe REE-diss or Red-DEES, depending on who you ask), which is a lightning fast in-memory key-value store that can be used for anything from A to Z. Here’s what Seven Databases in Seven Weeks, a popular book on databases, has to say about Redis:

It’s not simply easy to use; it’s a joy. If an API is UX for programmers, then Redis should be in the Museum of Modern Art alongside the Mac Cube.

…

And when it comes to speed, Redis is hard to beat. Reads are fast, and writes are even faster, handling upwards of 100,000 SET operations per second by some benchmarks. (Source)

Intrigued? This tutorial is built for the Python programmer who may have zero to little Redis experience. We’ll tackle two tools at once and introduce both Redis itself as well as one of its Python client libraries, redis-py.

redis-py (which you import as just redis) is one of many Python clients for Redis, but it has the distinction of being billed as “currently the way to go for Python” by the Redis developers themselves. It lets you call Redis commands from Python, and get back familiar Python objects in return.

In this tutorial, you’ll cover:

Installing Redis from source and understanding the purpose of the resulting binaries
Learning a bite-size slice of Redis itself, including its syntax, protocol, and design
Mastering redis-py while also seeing glimpses of how it implements Redis’ protocol
Setting up and communicating with an Amazon ElastiCache Redis server instance

Installing Redis From Source

As my great-great-grandfather said, nothing builds grit like installing from source. This section will walk you through downloading, making, and installing Redis. I promise that this won’t hurt one bit!

Note: This section is oriented towards installation on Mac OS X or Linux. If you’re using Windows, there is a Microsoft fork of Redis that can be installed as a Windows Service. Suffice it to say that Redis as a program lives most comfortably on a Linux box and that setup and use on Windows may be finicky.

First, download the Redis source code as a tarball:

$ redisurl="http://download.redis.io/redis-stable.tar.gz"
$ curl -s -o redis-stable.tar.gz $redisurl

Next, switch over to root and extract the archive’s source code to /usr/local/lib/:

$ sudo su root
$ mkdir -p /usr/local/lib/
$ chmod a+w /usr/local/lib/
$ tar -C /usr/local/lib/ -xzf redis-stable.tar.gz

Optionally, you can now remove the archive itself:

$ rm redis-stable.tar.gz

This will leave you with a source code repository at /usr/local/lib/redis-stable/. Redis is written in C, so you’ll need to compile, link, and install with the make utility:

$ cd /usr/local/lib/redis-stable/
$ make && make install

Using make install does two actions:

The first make command compiles and links the source code.
The make install part takes the binaries and copies them to /usr/local/bin/ so that you can run them from anywhere (assuming that /usr/local/bin/ is in PATH).

Here are all the steps so far:

$ redisurl="http://download.redis.io/redis-stable.tar.gz"
$ curl -s -o redis-stable.tar.gz $redisurl
$ sudo su root
$ mkdir -p /usr/local/lib/
$ chmod a+w /usr/local/lib/
$ tar -C /usr/local/lib/ -xzf redis-stable.tar.gz
$ rm redis-stable.tar.gz
$ cd /usr/local/lib/redis-stable/
$ make && make install

At this point, take a moment to confirm that Redis is in your PATH and check its version:

$ redis-cli --version
redis-cli 5.0.3

If your shell can’t find redis-cli, check to make sure that /usr/local/bin/ is on your PATH environment variable, and add it if not.

In addition to redis-cli, make install actually leads to a handful of different executable files (and one symlink) being placed at /usr/local/bin/:

$ # A snapshot of executables that come bundled with Redis
$ ls -hFG /usr/local/bin/redis-* | sort
/usr/local/bin/redis-benchmark*
/usr/local/bin/redis-check-aof*
/usr/local/bin/redis-check-rdb*
/usr/local/bin/redis-cli*
/usr/local/bin/redis-sentinel@
/usr/local/bin/redis-server*

While all of these have some intended use, the two you’ll probably care about most are redis-cli and redis-server, which we’ll outline shortly. But before we get to that, setting up some baseline configuration is in order.

Configuring Redis

Redis is highly configurable. While it runs fine out of the box, let’s take a minute to set some bare-bones configuration options that relate to database persistence and basic security:

$ sudo su root
$ mkdir -p /etc/redis/
$ touch /etc/redis/6379.conf

Now, write the following to /etc/redis/6379.conf. We’ll cover what most of these mean gradually throughout the tutorial:

# /etc/redis/6379.conf

port              6379
daemonize         yes
save              60 1
bind              127.0.0.1
tcp-keepalive     300
dbfilename        dump.rdb
dir               ./
rdbcompression    yes

Redis configuration is self-documenting, with the sample redis.conf file located in the Redis source for your reading pleasure. If you’re using Redis in a production system, it pays to block out all distractions and take the time to read this sample file in full to familiarize yourself with the ins and outs of Redis and fine-tune your setup.

Some tutorials, including parts of Redis’ documentation, may also suggest running the Shell script install_server.sh located in redis/utils/install_server.sh. You’re by all means welcome to run this as a more comprehensive alternative to the above, but take note of a few finer points about install_server.sh:

It will not work on Mac OS X—only Debian and Ubuntu Linux.
It will inject a fuller set of configuration options into /etc/redis/6379.conf.
It will write a System V init script to /etc/init.d/redis_6379 that will let you do sudo service redis_6379 start.

The Redis quickstart guide also contains a section on a more proper Redis setup, but the configuration options above should be totally sufficient for this tutorial and getting started.

Security Note: A few years back, the author of Redis pointed out security vulnerabilities in earlier versions of Redis if no configuration was set. Redis 3.2 (the current version 5.0.3 as of March 2019) made steps to prevent this intrusion, setting the protected-mode option to yes by default.

We explicitly set bind 127.0.0.1 to let Redis listen for connections only from the localhost interface, although you would need to expand this whitelist in a real production server. The point of protected-mode is as a safeguard that will mimic this bind-to-localhost behavior if you don’t otherwise specify anything under the bind option.

With that squared away, we can now dig into using Redis itself.

Ten or So Minutes to Redis

This section will provide you with just enough knowledge of Redis to be dangerous, outlining its design and basic usage.

Getting Started

Redis has a client-server architecture and uses a request-response model. This means that you (the client) connect to a Redis server through TCP connection, on port 6379 by default. You request some action (like some form of reading, writing, getting, setting, or updating), and the server serves you back a response.

There can be many clients talking to the same server, which is really what Redis or any client-server application is all about. Each client does a (typically blocking) read on a socket waiting for the server response.

The cli in redis-cli stands for command line interface, and the server in redis-server is for, well, running a server. In the same way that you would run python at the command line, you can run redis-cli to jump into an interactive REPL (Read Eval Print Loop) where you can run client commands directly from the shell.

First, however, you’ll need to launch redis-server so that you have a running Redis server to talk to. A common way to do this in development is to start a server at localhost (IPv4 address 127.0.0.1), which is the default unless you tell Redis otherwise. You can also pass redis-server the name of your configuration file, which is akin to specifying all of its key-value pairs as command-line arguments:

$ redis-server /etc/redis/6379.conf
31829:C 07 Mar 2019 08:45:04.030 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
31829:C 07 Mar 2019 08:45:04.030 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=31829, just started
31829:C 07 Mar 2019 08:45:04.030 # Configuration loaded

We set the daemonize configuration option to yes, so the server runs in the background. (Otherwise, use --daemonize yes as an option to redis-server.)

Now you’re ready to launch the Redis REPL. Enter redis-cli on your command line. You’ll see the server’s host:port pair followed by a > prompt:

127.0.0.1:6379>

Here’s one of the simplest Redis commands, PING, which just tests connectivity to the server and returns "PONG" if things are okay:

127.0.0.1:6379> PING
PONG

Redis commands are case-insensitive, although their Python counterparts are most definitely not.

Note: As another sanity check, you can search for the process ID of the Redis server with pgrep:

$ pgrep redis-server
26983

To kill the server, use pkill redis-server from the command line. On Mac OS X, you can also use redis-cli shutdown.

Next, we’ll use some of the common Redis commands and compare them to what they would look like in pure Python.

Redis as a Python Dictionary

Redis stands for Remote Dictionary Service.

“You mean, like a Python dictionary?” you may ask.

Yes. Broadly speaking, there are many parallels you can draw between a Python dictionary (or generic hash table) and what Redis is and does:

A Redis database holds key:value pairs and supports commands such as GET, SET, and DEL, as well as several hundred additional commands.
Redis keys are always strings.
Redis values may be a number of different data types. We’ll cover some of the more essential value data types in this tutorial: string, list, hashes, and sets. Some advanced types include geospatial items and the new stream type.
Many Redis commands operate in constant O(1) time, just like retrieving a value from a Python dict or any hash table.

Redis creator Salvatore Sanfilippo would probably not love the comparison of a Redis database to a plain-vanilla Python dict. He calls the project a “data structure server” (rather than a key-value store, such as memcached) because, to its credit, Redis supports storing additional types of key:value data types besides string:string. But for our purposes here, it’s a useful comparison if you’re familiar with Python’s dictionary object.

Let’s jump in and learn by example. Our first toy database (with ID 0) will be a mapping of country:capital city, where we use SET to set key-value pairs:

127.0.0.1:6379> SET Bahamas Nassau
OK
127.0.0.1:6379> SET Croatia Zagreb
OK
127.0.0.1:6379> GET Croatia
"Zagreb"
127.0.0.1:6379> GET Japan
(nil)

The corresponding sequence of statements in pure Python would look like this:

>>>

>>> capitals = {}
>>> capitals["Bahamas"] = "Nassau"
>>> capitals["Croatia"] = "Zagreb"
>>> capitals.get("Croatia")
'Zagreb'
>>> capitals.get("Japan")  # None

We use capitals.get("Japan") rather than capitals["Japan"] because Redis will return nil rather than an error when a key is not found, which is analogous to Python’s None.

Redis also allows you to set and get multiple key-value pairs in one command, MSET and MGET, respectively:

127.0.0.1:6379> MSET Lebanon Beirut Norway Oslo France Paris
OK
127.0.0.1:6379> MGET Lebanon Norway Bahamas
1) "Beirut"
2) "Oslo"
3) "Nassau"

The closest thing in Python is with dict.update():

>>>

>>> capitals.update({
...     "Lebanon": "Beirut",
...     "Norway": "Oslo",
...     "France": "Paris",
... })
>>> [capitals.get(k) for k in ("Lebanon", "Norway", "Bahamas")]
['Beirut', 'Oslo', 'Nassau']

We use .get() rather than .__getitem__() to mimic Redis’ behavior of returning a null-like value when no key is found.

As a third example, the EXISTS command does what it sounds like, which is to check if a key exists:

127.0.0.1:6379> EXISTS Norway
(integer) 1
127.0.0.1:6379> EXISTS Sweden
(integer) 0

Python has the in keyword to test the same thing, which routes to dict.__contains__(key):

>>>

>>> "Norway" in capitals
True
>>> "Sweden" in capitals
False

These few examples are meant to show, using native Python, what’s happening at a high level with a few common Redis commands. There’s no client-server component here to the Python examples, and redis-py has not yet entered the picture. This is only meant to show Redis functionality by example.

Here’s a summary of the few Redis commands you’ve seen and their functional Python equivalents:

capitals["Bahamas"] = "Nassau"

capitals.get("Croatia")

capitals.update(
    {
        "Lebanon": "Beirut",
        "Norway": "Oslo",
        "France": "Paris",
    }
)

[capitals[k] for k in ("Lebanon", "Norway", "Bahamas")]

"Norway" in capitals

The Python Redis client library, redis-py, that you’ll dive into shortly in this article, does things differently. It encapsulates an actual TCP connection to a Redis server and sends raw commands, as bytes serialized using the REdis Serialization Protocol (RESP), to the server. It then takes the raw reply and parses it back into a Python object such as bytes, int, or even datetime.datetime.

Note: So far, you’ve been talking to the Redis server through the interactive redis-cli REPL. You can also issue commands directly, in the same way that you would pass the name of a script to the python executable, such as python myscript.py.

So far, you’ve seen a few of Redis’ fundamental data types, which is a mapping of string:string. While this key-value pair is common in most key-value stores, Redis offers a number of other possible value types, which you’ll see next.

More Data Types in Python vs Redis

Before you fire up the redis-py Python client, it also helps to have a basic grasp on a few more Redis data types. To be clear, all Redis keys are strings. It’s the value that can take on data types (or structures) in addition to the string values used in the examples so far.

A hash is a mapping of string:string, called field-value pairs, that sits under one top-level key:

127.0.0.1:6379> HSET realpython url "https://realpython.com/"
(integer) 1
127.0.0.1:6379> HSET realpython github realpython
(integer) 1
127.0.0.1:6379> HSET realpython fullname "Real Python"
(integer) 1

This sets three field-value pairs for one key, "realpython". If you’re used to Python’s terminology and objects, this can be confusing. A Redis hash is roughly analogous to a Python dict that is nested one level deep:

data = {
    "realpython": {
        "url": "https://realpython.com/",
        "github": "realpython",
        "fullname": "Real Python",
    }
}

Redis’ fields are akin to the Python keys of each nested key-value pair in the inner dictionary above. Redis reserves the term key for the top-level database key that holds the hash structure itself.

Just like there’s MSET for basic string:string key-value pairs, there is also HMSET for hashes to set multiple pairs within the hash value object:

127.0.0.1:6379> HMSET pypa url "https://www.pypa.io/" github pypa fullname "Python Packaging Authority"
OK
127.0.0.1:6379> HGETALL pypa
1) "url"
2) "https://www.pypa.io/"
3) "github"
4) "pypa"
5) "fullname"
6) "Python Packaging Authority"

Using HMSET is probably a closer parallel for the way that we assigned data to a nested dictionary above, rather than setting each nested pair as is done with HSET.

Two additional value types are lists and sets, which can take the place of a hash or string as a Redis value. They are largely what they sound like, so I won’t take up your time with additional examples. Hashes, lists, and sets each have some commands that are particular to that given data type, which are in some cases denoted by their initial letter:

Hashes: Commands to operate on hashes begin with an H, such as HSET, HGET, or HMSET.
Sets: Commands to operate on sets begin with an S, such as SCARD, which gets the number of elements at the set value corresponding to a given key.
Lists: Commands to operate on lists begin with an L or R. Examples include LPOP and RPUSH. The L or R refers to which side of the list is operated on. A few list commands are also prefaced with a B, which means blocking. A blocking operation doesn’t let other operations interrupt it while it’s executing. For instance, BLPOP executes a blocking left-pop on a list structure.

Note: One noteworthy feature of Redis’ list type is that it is a linked list rather than an array. This means that appending is O(1) while indexing at an arbitrary index number is O(N).

Here is a quick listing of commands that are particular to the string, hash, list, and set data types in Redis:

Type	Commands
Sets	`SADD`, `SCARD`, `SDIFF`, `SDIFFSTORE`, `SINTER`, `SINTERSTORE`, `SISMEMBER`, `SMEMBERS`, `SMOVE`, `SPOP`, `SRANDMEMBER`, `SREM`, `SSCAN`, `SUNION`, `SUNIONSTORE`
Hashes	`HDEL`, `HEXISTS`, `HGET`, `HGETALL`, `HINCRBY`, `HINCRBYFLOAT`, `HKEYS`, `HLEN`, `HMGET`, `HMSET`, `HSCAN`, `HSET`, `HSETNX`, `HSTRLEN`, `HVALS`
Lists	`BLPOP`, `BRPOP`, `BRPOPLPUSH`, `LINDEX`, `LINSERT`, `LLEN`, `LPOP`, `LPUSH`, `LPUSHX`, `LRANGE`, `LREM`, `LSET`, `LTRIM`, `RPOP`, `RPOPLPUSH`, `RPUSH`, `RPUSHX`
Strings	`APPEND`, `BITCOUNT`, `BITFIELD`, `BITOP`, `BITPOS`, `DECR`, `DECRBY`, `GET`, `GETBIT`, `GETRANGE`, `GETSET`, `INCR`, `INCRBY`, `INCRBYFLOAT`, `MGET`, `MSET`, `MSETNX`, `PSETEX`, `SET`, `SETBIT`, `SETEX`, `SETNX`, `SETRANGE`, `STRLEN`

This table isn’t a complete picture of Redis commands and types. There’s a smorgasbord of more advanced data types, such as geospatial items, sorted sets, and HyperLogLog. At the Redis commands page, you can filter by data-structure group. There is also the data types summary and introduction to Redis data types.

Since we’re going to be switching over to doing things in Python, you can now clear your toy database with FLUSHDB and quit the redis-cli REPL:

127.0.0.1:6379> FLUSHDB
OK
127.0.0.1:6379> QUIT

This will bring you back to your shell prompt. You can leave redis-server running in the background, since you’ll need it for the rest of the tutorial also.

Using `redis-py`: Redis in Python

Now that you’ve mastered some basics of Redis, it’s time to jump into redis-py, the Python client that lets you talk to Redis from a user-friendly Python API.

First Steps

redis-py is a well-established Python client library that lets you talk to a Redis server directly through Python calls:

$ python -m pip install redis

Next, make sure that your Redis server is still up and running in the background. You can check with pgrep redis-server, and if you come up empty-handed, then restart a local server with redis-server /etc/redis/6379.conf.

Now, let’s get into the Python-centric part of things. Here’s the “hello world” of redis-py:

>>>

 1 >>> import redis
 2 >>> r = redis.Redis()
 3 >>> r.mset({"Croatia": "Zagreb", "Bahamas": "Nassau"})
 4 True
 5 >>> r.get("Bahamas")
 6 b'Nassau'

Redis, used in Line 2, is the central class of the package and the workhorse by which you execute (almost) any Redis command. The TCP socket connection and reuse is done for you behind the scenes, and you call Redis commands using methods on the class instance r.

Notice also that the type of the returned object, b'Nassau' in Line 6, is Python’s bytes type, not str. It is bytes rather than str that is the most common return type across redis-py, so you may need to call r.get("Bahamas").decode("utf-8") depending on what you want to actually do with the returned bytestring.

Does the code above look familiar? The methods in almost all cases match the name of the Redis command that does the same thing. Here, you called r.mset() and r.get(), which correspond to MSET and GET in the native Redis API.

This also means that HGETALL becomes r.hgetall(), PING becomes r.ping(), and so on. There are a few exceptions, but the rule holds for the large majority of commands.

While the Redis command arguments usually translate into a similar-looking method signature, they take Python objects. For example, the call to r.mset() in the example above uses a Python dict as its first argument, rather than a sequence of bytestrings.

We built the Redis instance r with no arguments, but it comes bundled with a number of parameters if you need them:

# From redis/client.py
class Redis(object):
    def __init__(self, host='localhost', port=6379,
                 db=0, password=None, socket_timeout=None,
                 # ...

You can see that the default hostname:port pair is localhost:6379, which is exactly what we need in the case of our locally kept redis-server instance.

The db parameter is the database number. You can manage multiple databases in Redis at once, and each is identified by an integer. The max number of databases is 16 by default.

When you run just redis-cli from the command line, this starts you at database 0. Use the -n flag to start a new database, as in redis-cli -n 5.

Allowed Key Types

One thing that’s worth knowing is that redis-py requires that you pass it keys that are bytes, str, int, or float. (It will convert the last 3 of these types to bytes before sending them off to the server.)

Consider a case where you want to use calendar dates as keys:

>>>

>>> import datetime
>>> today = datetime.date.today()
>>> visitors = {"dan", "jon", "alex"}
>>> r.sadd(today, *visitors)
Traceback (most recent call last):
# ...
redis.exceptions.DataError: Invalid input of type: 'date'.
Convert to a byte, string or number first.

You’ll need to explicitly convert the Python date object to str, which you can do with .isoformat():

>>>

>>> stoday = today.isoformat()  # Python 3.7+, or use str(today)
>>> stoday
'2019-03-10'
>>> r.sadd(stoday, *visitors)  # sadd: set-add
3
>>> r.smembers(stoday)
{b'dan', b'alex', b'jon'}
>>> r.scard(today.isoformat())
3

To recap, Redis itself only allows strings as keys. redis-py is a bit more liberal in what Python types it will accept, although it ultimately converts everything to bytes before sending them off to a Redis server.

Example: PyHats.com

It’s time to break out a fuller example. Let’s pretend we’ve decided to start a lucrative website, PyHats.com, that sells outrageously overpriced hats to anyone who will buy them, and hired you to build the site.

You’ll use Redis to handle some of the product catalog, inventorying, and bot traffic detection for PyHats.com.

It’s day one for the site, and we’re going to be selling three limited-edition hats. Each hat gets held in a Redis hash of field-value pairs, and the hash has a key that is a prefixed random integer , such as hat:56854717. Using the hat: prefix is Redis convention for creating a sort of namespace within a Redis database:

import random

random.seed(444)
hats = {f"hat:{random.getrandbits(32)}": i for i in (
    {
        "color": "black",
        "price": 49.99,
        "style": "fitted",
        "quantity": 1000,
        "npurchased": 0,
    },
    {
        "color": "maroon",
        "price": 59.99,
        "style": "hipster",
        "quantity": 500,
        "npurchased": 0,
    },
    {
        "color": "green",
        "price": 99.99,
        "style": "baseball",
        "quantity": 200,
        "npurchased": 0,
    })
}

Let’s start with database 1 since we used database 0 in a previous example:

>>>

>>> r = redis.Redis(db=1)

To do an initial write of this data into Redis, we can use .hmset() (hash multi-set), calling it for each dictionary. The “multi” is a reference to setting multiple field-value pairs, where “field” in this case corresponds to a key of any of the nested dictionaries in hats:

 1 >>> with r.pipeline() as pipe:
 2 ...    for h_id, hat in hats.items():
 3 ...        pipe.hmset(h_id, hat)
 4 ...    pipe.execute()
 5 Pipeline<ConnectionPool<Connection<host=localhost,port=6379,db=1>>>
 6 Pipeline<ConnectionPool<Connection<host=localhost,port=6379,db=1>>>
 7 Pipeline<ConnectionPool<Connection<host=localhost,port=6379,db=1>>>
 8 [True, True, True]
 9 
10 >>> r.bgsave()
11 True

The code block above also introduces the concept of Redis pipelining, which is a way to cut down the number of round-trip transactions that you need to write or read data from your Redis server. If you would have just called r.hmset() three times, then this would necessitate a back-and-forth round trip operation for each row written.

With a pipeline, all the commands are buffered on the client side and then sent at once, in one fell swoop, using pipe.hmset() in Line 3. This is why the three True responses are all returned at once, when you call pipe.execute() in Line 4. You’ll see a more advanced use case for a pipeline shortly.

Note: The Redis docs provide an example of doing this same thing with the redis-cli, where you can pipe the contents of a local file to do mass insertion.

Let’s do a quick check that everything is there in our Redis database:

>>>

>>> pprint(r.hgetall("hat:56854717"))
{b'color': b'green',
 b'npurchased': b'0',
 b'price': b'99.99',
 b'quantity': b'200',
 b'style': b'baseball'}

>>> r.keys()  # Careful on a big DB. keys() is O(N)
[b'56854717', b'1236154736', b'1326692461']

The first thing that we want to simulate is what happens when a user clicks Purchase. If the item is in stock, increase its npurchased by 1 and decrease its quantity (inventory) by 1. You can use .hincrby() to do this:

>>>

>>> r.hincrby("hat:56854717", "quantity", -1)
199
>>> r.hget("hat:56854717", "quantity")
b'199'
>>> r.hincrby("hat:56854717", "npurchased", 1)
1

Note: HINCRBY still operates on a hash value that is a string, but it tries to interpret the string as a base-10 64-bit signed integer to execute the operation.

This applies to other commands related to incrementing and decrementing for other data structures, namely INCR, INCRBY, INCRBYFLOAT, ZINCRBY, and HINCRBYFLOAT. You’ll get an error if the string at the value can’t be represented as an integer.

It isn’t really that simple, though. Changing the quantity and npurchased in two lines of code hides the reality that a click, purchase, and payment entails more than this. We need to do a few more checks to make sure we don’t leave someone with a lighter wallet and no hat:

Step 1: Check if the item is in stock, or otherwise raise an exception on the backend.
Step 2: If it is in stock, then execute the transaction, decrease the quantity field, and increase the npurchased field.
Step 3: Be alert for any changes that alter the inventory in between the first two steps (a race condition).

Step 1 is relatively straightforward: it consists of an .hget() to check the available quantity.

Step 2 is a little bit more involved. The pair of increase and decrease operations need to be executed atomically: either both should be completed successfully, or neither should be (in the case that at least one fails).

With client-server frameworks, it’s always crucial to pay attention to atomicity and look out for what could go wrong in instances where multiple clients are trying to talk to the server at once. The answer to this in Redis is to use a transaction block, meaning that either both or neither of the commands get through.

In redis-py, Pipeline is a transactional pipeline class by default. This means that, even though the class is actually named for something else (pipelining), it can be used to create a transaction block also.

In Redis, a transaction starts with MULTI and ends with EXEC:

 1 127.0.0.1:6379> MULTI
 2 127.0.0.1:6379> HINCRBY 56854717 quantity -1
 3 127.0.0.1:6379> HINCRBY 56854717 npurchased 1
 4 127.0.0.1:6379> EXEC

MULTI (Line 1) marks the start of the transaction, and EXEC (Line 4) marks the end. Everything in between is executed as one all-or-nothing buffered sequence of commands. This means that it will be impossible to decrement quantity (Line 2) but then have the balancing npurchased increment operation fail (Line 3).

Let’s circle back to Step 3: we need to be aware of any changes that alter the inventory in between the first two steps.

Step 3 is the trickiest. Let’s say that there is one lone hat remaining in our inventory. In between the time that User A checks the quantity of hats remaining and actually processes their transaction, User B also checks the inventory and finds likewise that there is one hat listed in stock. Both users will be allowed to purchase the hat, but we have 1 hat to sell, not 2, so we’re on the hook and one user is out of their money. Not good.

Redis has a clever answer for the dilemma in Step 3: it’s called optimistic locking, and is different than how typical locking works in an RDBMS such as PostgreSQL. Optimistic locking, in a nutshell, means that the calling function (client) does not acquire a lock, but rather monitors for changes in the data it is writing to during the time it would have held a lock. If there’s a conflict during that time, the calling function simply tries the whole process again.

You can effect optimistic locking by using the WATCH command (.watch() in redis-py), which provides a check-and-set behavior.

Let’s introduce a big chunk of code and walk through it afterwards step by step. You can picture buyitem() as being called any time a user clicks on a Buy Now or Purchase button. Its purpose is to confirm the item is in stock and take an action based on that result, all in a safe manner that looks out for race conditions and retries if one is detected:

 1 import logging
 2 import redis
 3 
 4 logging.basicConfig()
 5 
 6 class OutOfStockError(Exception):
 7     """Raised when PyHats.com is all out of today's hottest hat"""
 8 
 9 def buyitem(r: redis.Redis, itemid: int) -> None:
10     with r.pipeline() as pipe:
11         error_count = 0
12         while True:
13             try:
14                 # Get available inventory, watching for changes
15                 # related to this itemid before the transaction
16                 pipe.watch(itemid)
17                 nleft: bytes = r.hget(itemid, "quantity")
18                 if nleft > b"0":
19                     pipe.multi()
20                     pipe.hincrby(itemid, "quantity", -1)
21                     pipe.hincrby(itemid, "npurchased", 1)
22                     pipe.execute()
23                     break
24                 else:
25                     # Stop watching the itemid and raise to break out
26                     pipe.unwatch()
27                     raise OutOfStockError(
28                         f"Sorry, {itemid} is out of stock!"
29                     )
30             except redis.WatchError:
31                 # Log total num. of errors by this user to buy this item,
32                 # then try the same process again of WATCH/HGET/MULTI/EXEC
33                 error_count += 1
34                 logging.warning(
35                     "WatchError #%d: %s; retrying",
36                     error_count, itemid
37                 )
38     return None

The critical line occurs at Line 16 with pipe.watch(itemid), which tells Redis to monitor the given itemid for any changes to its value. The program checks the inventory through the call to r.hget(itemid, "quantity"), in Line 17:

16 pipe.watch(itemid)
17 nleft: bytes = r.hget(itemid, "quantity")
18 if nleft > b"0":
19     # Item in stock. Proceed with transaction.

If the inventory gets touched during this short window between when the user checks the item stock and tries to purchase it, then Redis will return an error, and redis-py will raise a WatchError (Line 30). That is, if any of the hash pointed to by itemid changes after the .hget() call but before the subsequent .hincrby() calls in Lines 20 and 21, then we’ll re-run the whole process in another iteration of the while True loop as a result.

This is the “optimistic” part of the locking: rather than letting the client have a time-consuming total lock on the database through the getting and setting operations, we leave it up to Redis to notify the client and user only in the case that calls for a retry of the inventory check.

One key here is in understanding the difference between client-side and server-side operations:

nleft = r.hget(itemid, "quantity")

This Python assignment brings the result of r.hget() client-side. Conversely, methods that you call on pipe effectively buffer all of the commands into one, and then send them to the server in a single request:

16 pipe.multi()
17 pipe.hincrby(itemid, "quantity", -1)
18 pipe.hincrby(itemid, "npurchased", 1)
19 pipe.execute()

No data comes back to the client side in the middle of the transactional pipeline. You need to call .execute() (Line 19) to get the sequence of results back all at once.

Even though this block contains two commands, it consists of exactly one round-trip operation from client to server and back.

This means that the client can’t immediately use the result of pipe.hincrby(itemid, "quantity", -1), from Line 20, because methods on a Pipeline return just the pipe instance itself. We haven’t asked anything from the server at this point. While normally .hincrby() returns the resulting value, you can’t immediately reference it on the client side until the entire transaction is completed.

There’s a catch-22: this is also why you can’t put the call to .hget() into the transaction block. If you did this, then you’d be unable to know if you want to increment the npurchased field yet, since you can’t get real-time results from commands that are inserted into a transactional pipeline.

Finally, if the inventory sits at zero, then we UNWATCH the item ID and raise an OutOfStockError (Line 27), ultimately displaying that coveted Sold Out page that will make our hat buyers desperately want to buy even more of our hats at ever more outlandish prices:

24 else:
25     # Stop watching the itemid and raise to break out
26     pipe.unwatch()
27     raise OutOfStockError(
28         f"Sorry, {itemid} is out of stock!"
29     )

Here’s an illustration. Keep in mind that our starting quantity is 199 for hat 56854717 since we called .hincrby() above. Let’s mimic 3 purchases, which should modify the quantity and npurchased fields:

>>>

>>> buyitem(r, "hat:56854717")
>>> buyitem(r, "hat:56854717")
>>> buyitem(r, "hat:56854717")
>>> r.hmget("hat:56854717", "quantity", "npurchased")  # Hash multi-get
[b'196', b'4']

Now, we can fast-forward through more purchases, mimicking a stream of purchases until the stock depletes to zero. Again, picture these coming from a whole bunch of different clients rather than just one Redis instance:

>>>

>>> # Buy remaining 196 hats for item 56854717 and deplete stock to 0
>>> for _ in range(196):
...     buyitem(r, "hat:56854717")
>>> r.hmget("hat:56854717", "quantity", "npurchased")
[b'0', b'200']

Now, when some poor user is late to the game, they should be met with an OutOfStockError that tells our application to render an error message page on the frontend:

>>>

>>> buyitem(r, "hat:56854717")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 20, in buyitem
__main__.OutOfStockError: Sorry, hat:56854717 is out of stock!

Looks like it’s time to restock.

Using Key Expiry

Let’s introduce key expiry, which is another distinguishing feature in Redis. When you expire a key, that key and its corresponding value will be automatically deleted from the database after a certain number of seconds or at a certain timestamp.

In redis-py, one way that you can accomplish this is through .setex(), which lets you set a basic string:string key-value pair with an expiration:

>>>

 1 >>> from datetime import timedelta
 2 
 3 >>> # setex: "SET" with expiration
 4 >>> r.setex(
 5 ...     "runner",
 6 ...     timedelta(minutes=1),
 7 ...     value="now you see me, now you don't"
 8 ... )
 9 True

You can specify the second argument as a number in seconds or a timedelta object, as in Line 6 above. I like the latter because it seems less ambiguous and more deliberate.

There are also methods (and corresponding Redis commands, of course) to get the remaining lifetime (time-to-live) of a key that you’ve set to expire:

>>>

>>> r.ttl("runner")  # "Time To Live", in seconds
58
>>> r.pttl("runner")  # Like ttl, but milliseconds
54368

Below, you can accelerate the window until expiration, and then watch the key expire, after which r.get() will return None and .exists() will return 0:

>>>

>>> r.get("runner")  # Not expired yet
b"now you see me, now you don't"

>>> r.expire("runner", timedelta(seconds=3))  # Set new expire window
True
>>> # Pause for a few seconds
>>> r.get("runner")
>>> r.exists("runner")  # Key & value are both gone (expired)
0

The table below summarizes commands related to key-value expiration, including the ones covered above. The explanations are taken directly from redis-py method docstrings:

Signature	Purpose
`r.setex(name, time, value)`	Sets the value of key `name` to `value` that expires in `time` seconds, where `time` can be represented by an `int` or a Python `timedelta` object
`r.psetex(name, time_ms, value)`	Sets the value of key `name` to `value` that expires in `time_ms` milliseconds, where `time_ms` can be represented by an `int` or a Python `timedelta` object
`r.expire(name, time)`	Sets an expire flag on key `name` for `time` seconds, where `time` can be represented by an `int` or a Python `timedelta` object
`r.expireat(name, when)`	Sets an expire flag on key `name`, where `when` can be represented as an `int` indicating Unix time or a Python `datetime` object
`r.persist(name)`	Removes an expiration on `name`
`r.pexpire(name, time)`	Sets an expire flag on key `name` for `time` milliseconds, and `time` can be represented by an `int` or a Python `timedelta` object
`r.pexpireat(name, when)`	Sets an expire flag on key `name`, where `when` can be represented as an `int` representing Unix time in milliseconds (Unix time * 1000) or a Python `datetime` object
`r.pttl(name)`	Returns the number of milliseconds until the key `name` will expire
`r.ttl(name)`	Returns the number of seconds until the key `name` will expire

PyHats.com, Part 2

A few days after its debut, PyHats.com has attracted so much hype that some enterprising users are creating bots to buy hundreds of items within seconds, which you’ve decided isn’t good for the long-term health of your hat business.

Now that you’ve seen how to expire keys, let’s put it to use on the backend of PyHats.com.

We’re going to create a new Redis client that acts as a consumer (or watcher) and processes a stream of incoming IP addresses, which in turn may come from multiple HTTPS connections to the website’s server.

The watcher’s goal is to monitor a stream of IP addresses from multiple sources, keeping an eye out for a flood of requests from a single address within a suspiciously short amount of time.

Some middleware on the website server pushes all incoming IP addresses into a Redis list with .lpush(). Here’s a crude way of mimicking some incoming IPs, using a fresh Redis database:

>>>

>>> r = redis.Redis(db=5)
>>> r.lpush("ips", "51.218.112.236")
1
>>> r.lpush("ips", "90.213.45.98")
2
>>> r.lpush("ips", "115.215.230.176")
3
>>> r.lpush("ips", "51.218.112.236")
4

As you can see, .lpush() returns the length of the list after the push operation succeeds. Each call of .lpush() puts the IP at the beginning of the Redis list that is keyed by the string "ips".

In this simplified simulation, the requests are all technically from the same client, but you can think of them as potentially coming from many different clients and all being pushed to the same database on the same Redis server.

Now, open up a new shell tab or window and launch a new Python REPL. In this shell, you’ll create a new client that serves a very different purpose than the rest, which sits in an endless while True loop and does a blocking left-pop BLPOP call on the ips list, processing each address:

 1 # New shell window or tab
 2 
 3 import datetime
 4 import ipaddress
 5 
 6 import redis
 7 
 8 # Where we put all the bad egg IP addresses
 9 blacklist = set()
10 MAXVISITS = 15
11 
12 ipwatcher = redis.Redis(db=5)
13 
14 while True:
15     _, addr = ipwatcher.blpop("ips")
16     addr = ipaddress.ip_address(addr.decode("utf-8"))
17     now = datetime.datetime.utcnow()
18     addrts = f"{addr}:{now.minute}"
19     n = ipwatcher.incrby(addrts, 1)
20     if n >= MAXVISITS:
21         print(f"Hat bot detected!:  {addr}")
22         blacklist.add(addr)
23     else:
24         print(f"{now}:  saw {addr}")
25     _ = ipwatcher.expire(addrts, 60)

Let’s walk through a few important concepts.

The ipwatcher acts like a consumer, sitting around and waiting for new IPs to be pushed on the "ips" Redis list. It receives them as bytes, such as b”51.218.112.236”, and makes them into a more proper address object with the ipaddress module:

15 _, addr = ipwatcher.blpop("ips")
16 addr = ipaddress.ip_address(addr.decode("utf-8"))

Then you form a Redis string key using the address and minute of the hour at which the ipwatcher saw the address, incrementing the corresponding count by 1 and getting the new count in the process:

17 now = datetime.datetime.utcnow()
18 addrts = f"{addr}:{now.minute}"
19 n = ipwatcher.incrby(addrts, 1)

If the address has been seen more than MAXVISITS, then it looks as if we have a PyHats.com web scraper on our hands trying to create the next tulip bubble. Alas, we have no choice but to give this user back something like a dreaded 403 status code.

We use ipwatcher.expire(addrts, 60) to expire the (address minute) combination 60 seconds from when it was last seen. This is to prevent our database from becoming clogged up with stale one-time page viewers.

If you execute this code block in a new shell, you should immediately see this output:

2019-03-11 15:10:41.489214:  saw 51.218.112.236
2019-03-11 15:10:41.490298:  saw 115.215.230.176
2019-03-11 15:10:41.490839:  saw 90.213.45.98
2019-03-11 15:10:41.491387:  saw 51.218.112.236

The output appears right away because those four IPs were sitting in the queue-like list keyed by "ips", waiting to be pulled out by our ipwatcher. Using .blpop() (or the BLPOP command) will block until an item is available in the list, then pops it off. It behaves like Python’s Queue.get(), which also blocks until an item is available.

Besides just spitting out IP addresses, our ipwatcher has a second job. For a given minute of an hour (minute 1 through minute 60), ipwatcher will classify an IP address as a hat-bot if it sends 15 or more GET requests in that minute.

Switch back to your first shell and mimic a page scraper that blasts the site with 20 requests in a few milliseconds:

for _ in range(20):
    r.lpush("ips", "104.174.118.18")

Finally, toggle back to the second shell holding ipwatcher, and you should see an output like this:

2019-03-11 15:15:43.041363:  saw 104.174.118.18
2019-03-11 15:15:43.042027:  saw 104.174.118.18
2019-03-11 15:15:43.042598:  saw 104.174.118.18
2019-03-11 15:15:43.043143:  saw 104.174.118.18
2019-03-11 15:15:43.043725:  saw 104.174.118.18
2019-03-11 15:15:43.044244:  saw 104.174.118.18
2019-03-11 15:15:43.044760:  saw 104.174.118.18
2019-03-11 15:15:43.045288:  saw 104.174.118.18
2019-03-11 15:15:43.045806:  saw 104.174.118.18
2019-03-11 15:15:43.046318:  saw 104.174.118.18
2019-03-11 15:15:43.046829:  saw 104.174.118.18
2019-03-11 15:15:43.047392:  saw 104.174.118.18
2019-03-11 15:15:43.047966:  saw 104.174.118.18
2019-03-11 15:15:43.048479:  saw 104.174.118.18
Hat bot detected!:  104.174.118.18
Hat bot detected!:  104.174.118.18
Hat bot detected!:  104.174.118.18
Hat bot detected!:  104.174.118.18
Hat bot detected!:  104.174.118.18
Hat bot detected!:  104.174.118.18

Now, Ctrl+C out of the while True loop and you’ll see that the offending IP has been added to your blacklist:

>>>

>>> blacklist
{IPv4Address('104.174.118.18')}

Can you find the defect in this detection system? The filter checks the minute as .minute rather than the last 60 seconds (a rolling minute). Implementing a rolling check to monitor how many times a user has been seen in the last 60 seconds would be trickier. There’s a crafty solution using using Redis’ sorted sets at ClassDojo. Josiah Carlson’s Redis in Action also presents a more elaborate and general-purpose example of this section using an IP-to-location cache table.

Persistence and Snapshotting

One of the reasons that Redis is so fast in both read and write operations is that the database is held in memory (RAM) on the server. However, a Redis database can also be stored (persisted) to disk in a process called snapshotting. The point behind this is to keep a physical backup in binary format so that data can be reconstructed and put back into memory when needed, such as at server startup.

You already enabled snapshotting without knowing it when you set up basic configuration at the beginning of this tutorial with the save option:

# /etc/redis/6379.conf

port              6379
daemonize         yes
save              60 1
bind              127.0.0.1
tcp-keepalive     300
dbfilename        dump.rdb
dir               ./
rdbcompression    yes

The format is save <seconds> <changes>. This tells Redis to save the database to disk if both the given number of seconds and number of write operations against the database occurred. In this case, we’re telling Redis to save the database to disk every 60 seconds if at least one modifying write operation occurred in that 60-second timespan. This is a fairly aggressive setting versus the sample Redis config file, which uses these three save directives:

# Default redis/redis.conf
save 900 1
save 300 10
save 60 10000

An RDB snapshot is a full (rather than incremental) point-in-time capture of the database. (RDB refers to a Redis Database File.) We also specified the directory and file name of the resulting data file that gets written:

# /etc/redis/6379.conf

port              6379
daemonize         yes
save              60 1
bind              127.0.0.1
tcp-keepalive     300
dbfilename        dump.rdb
dir               ./
rdbcompression    yes

This instructs Redis to save to a binary data file called dump.rdb in the current working directory of wherever redis-server was executed from:

$ file -b dump.rdb
data

You can also manually invoke a save with the Redis command BGSAVE:

127.0.0.1:6379> BGSAVE
Background saving started

The “BG” in BGSAVE indicates that the save occurs in the background. This option is available in a redis-py method as well:

>>>

>>> r.lastsave()  # Redis command: LASTSAVE
datetime.datetime(2019, 3, 10, 21, 56, 50)
>>> r.bgsave()
True
>>> r.lastsave()
datetime.datetime(2019, 3, 10, 22, 4, 2)

This example introduces another new command and method, .lastsave(). In Redis, it returns the Unix timestamp of the last DB save, which Python gives back to you as a datetime object. Above, you can see that the r.lastsave() result changes as a result of r.bgsave().

r.lastsave() will also change if you enable automatic snapshotting with the save configuration option.

To rephrase all of this, there are two ways to enable snapshotting:

Explicitly, through the Redis command BGSAVE or redis-py method .bgsave()
Implicitly, through the save configuration option (which you can also set with .config_set() in redis-py)

RDB snapshotting is fast because the parent process uses the fork() system call to pass off the time-intensive write to disk to a child process, so that the parent process can continue on its way. This is what the background in BGSAVE refers to.

There’s also SAVE (.save() in redis-py), but this does a synchronous (blocking) save rather than using fork(), so you shouldn’t use it without a specific reason.

Even though .bgsave() occurs in the background, it’s not without its costs. The time for fork() itself to occur can actually be substantial if the Redis database is large enough in the first place.

If this is a concern, or if you can’t afford to miss even a tiny slice of data lost due to the periodic nature of RDB snapshotting, then you should look into the append-only file (AOF) strategy that is an alternative to snapshotting. AOF copies Redis commands to disk in real time, allowing you to do a literal command-based reconstruction by replaying these commands.

Serialization Workarounds

Let’s get back to talking about Redis data structures. With its hash data structure, Redis in effect supports nesting one level deep:

127.0.0.1:6379> hset mykey field1 value1

The Python client equivalent would look like this:

r.hset("mykey", "field1", "value1")

Here, you can think of "field1": "value1" as being the key-value pair of a Python dict, {"field1": "value1"}, while mykey is the top-level key:

Redis Command	Pure-Python Equivalent
`r.set("key", "value")`	`r = {"key": "value"}`
`r.hset("key", "field", "value")`	`r = {"key": {"field": "value"}}`

But what if you want the value of this dictionary (the Redis hash) to contain something other than a string, such as a list or nested dictionary with strings as values?

Here’s an example using some JSON-like data to make the distinction clearer:

restaurant_484272 = {
    "name": "Ravagh",
    "type": "Persian",
    "address": {
        "street": {
            "line1": "11 E 30th St",
            "line2": "APT 1",
        },
        "city": "New York",
        "state": "NY",
        "zip": 10016,
    }
}

Say that we want to set a Redis hash with the key 484272 and field-value pairs corresponding to the key-value pairs from restaurant_484272. Redis does not support this directly, because restaurant_484272 is nested:

>>>

>>> r.hmset(484272, restaurant_484272)
Traceback (most recent call last):
# ...
redis.exceptions.DataError: Invalid input of type: 'dict'.
Convert to a byte, string or number first.

You can in fact make this work with Redis. There are two different ways to mimic nested data in redis-py and Redis:

Serialize the values into a string with something like json.dumps()
Use a delimiter in the key strings to mimic nesting in the values

Let’s take a look at an example of each.

Option 1: Serialize the Values Into a String

You can use json.dumps() to serialize the dict into a JSON-formatted string:

>>>

>>> import json
>>> r.set(484272, json.dumps(restaurant_484272))
True

If you call .get(), the value you get back will be a bytes object, so don’t forget to deserialize it to get back the original object. json.dumps() and json.loads() are inverses of each other, for serializing and deserializing data, respectively:

>>>

>>> from pprint import pprint
>>> pprint(json.loads(r.get(484272)))
{'address': {'city': 'New York',
             'state': 'NY',
             'street': '11 E 30th St',
             'zip': 10016},
 'name': 'Ravagh',
 'type': 'Persian'}

This applies to any serialization protocol, with another common choice being yaml:

>>>

>>> import yaml  # python -m pip install PyYAML
>>> yaml.dump(restaurant_484272)
'address: {city: New York, state: NY, street: 11 E 30th St, zip: 10016}\nname: Ravagh\ntype: Persian\n'

No matter what serialization protocol you choose to go with, the concept is the same: you’re taking an object that is unique to Python and converting it to a bytestring that is recognized and exchangeable across multiple languages.

Option 2: Use a Delimiter in Key Strings

There’s a second option that involves mimicking “nestedness” by concatenating multiple levels of keys in a Python dict. This consists of flattening the nested dictionary through recursion, so that each key is a concatenated string of keys, and the values are the deepest-nested values from the original dictionary. Consider our dictionary object restaurant_484272:

restaurant_484272 = {
    "name": "Ravagh",
    "type": "Persian",
    "address": {
        "street": {
            "line1": "11 E 30th St",
            "line2": "APT 1",
        },
        "city": "New York",
        "state": "NY",
        "zip": 10016,
    }
}

We want to get it into this form:

{
    "484272:name":                     "Ravagh",
    "484272:type":                     "Persian",
    "484272:address:street:line1":     "11 E 30th St",
    "484272:address:street:line2":     "APT 1",
    "484272:address:city":             "New York",
    "484272:address:state":            "NY",
    "484272:address:zip":              "10016",
}

That’s what setflat_skeys() below does, with the added feature that it does inplace .set() operations on the Redis instance itself rather than returning a copy of the input dictionary:

 1 from collections.abc import MutableMapping
 2 
 3 def setflat_skeys(
 4     r: redis.Redis,
 5     obj: dict,
 6     prefix: str,
 7     delim: str = ":",
 8     *,
 9     _autopfix=""
10 ) -> None:
11     """Flatten `obj` and set resulting field-value pairs into `r`.
12 
13     Calls `.set()` to write to Redis instance inplace and returns None.
14 
15     `prefix` is an optional str that prefixes all keys.
16     `delim` is the delimiter that separates the joined, flattened keys.
17     `_autopfix` is used in recursive calls to created de-nested keys.
18 
19     The deepest-nested keys must be str, bytes, float, or int.
20     Otherwise a TypeError is raised.
21     """
22     allowed_vtypes = (str, bytes, float, int)
23     for key, value in obj.items():
24         key = _autopfix + key
25         if isinstance(value, allowed_vtypes):
26             r.set(f"{prefix}{delim}{key}", value)
27         elif isinstance(value, MutableMapping):
28             setflat_skeys(
29                 r, value, prefix, delim, _autopfix=f"{key}{delim}"
30             )
31         else:
32             raise TypeError(f"Unsupported value type: {type(value)}")

The function iterates over the key-value pairs of obj, first checking the type of the value (Line 25) to see if it looks like it should stop recursing further and set that key-value pair. Otherwise, if the value looks like a dict (Line 27), then it recurses into that mapping, adding the previously seen keys as a key prefix (Line 28).

Let’s see it at work:

>>>

>>> r.flushdb()  # Flush database: clear old entries
>>> setflat_skeys(r, restaurant_484272, 484272)

>>> for key in sorted(r.keys("484272*")):  # Filter to this pattern
...     print(f"{repr(key):35}{repr(r.get(key)):15}")
...
b'484272:address:city'             b'New York'
b'484272:address:state'            b'NY'
b'484272:address:street:line1'     b'11 E 30th St'
b'484272:address:street:line2'     b'APT 1'
b'484272:address:zip'              b'10016'
b'484272:name'                     b'Ravagh'
b'484272:type'                     b'Persian'

>>> r.get("484272:address:street:line1")
b'11 E 30th St'

The final loop above uses r.keys("484272*"), where "484272*" is interpreted as a pattern and matches all keys in the database that begin with "484272".

Notice also how setflat_skeys() calls just .set() rather than .hset(), because we’re working with plain string:string field-value pairs, and the 484272 ID key is prepended to each field string.

Encryption

Another trick to help you sleep well at night is to add symmetric encryption before sending anything to a Redis server. Consider this as an add-on to the security that you should make sure is in place by setting proper values in your Redis configuration. The example below uses the cryptography package:

$ python -m pip install cryptography

To illustrate, pretend that you have some sensitive cardholder data (CD) that you never want to have sitting around in plaintext on any server, no matter what. Before caching it in Redis, you can serialize the data and then encrypt the serialized string using Fernet:

>>>

>>> import json
>>> from cryptography.fernet import Fernet

>>> cipher = Fernet(Fernet.generate_key())
>>> info = {
...     "cardnum": 2211849528391929,
...     "exp": [2020, 9],
...     "cv2": 842,
... }

>>> r.set(
...     "user:1000",
...     cipher.encrypt(json.dumps(info).encode("utf-8"))
... )

>>> r.get("user:1000")
b'gAAAAABcg8-LfQw9TeFZ1eXbi'  # ... [truncated]

>>> cipher.decrypt(r.get("user:1000"))
b'{"cardnum": 2211849528391929, "exp": [2020, 9], "cv2": 842}'

>>> json.loads(cipher.decrypt(r.get("user:1000")))
{'cardnum': 2211849528391929, 'exp': [2020, 9], 'cv2': 842}

Because info contains a value that is a list, you’ll need to serialize this into a string that’s acceptable by Redis. (You could use json, yaml, or any other serialization for this.) Next, you encrypt and decrypt that string using the cipher object. You need to deserialize the decrypted bytes using json.loads() so that you can get the result back into the type of your initial input, a dict.

Note: Fernet uses AES 128 encryption in CBC mode. See the cryptography docs for an example of using AES 256. Whatever you choose to do, use cryptography, not pycrypto (imported as Crypto), which is no longer actively maintained.

If security is paramount, encrypting strings before they make their way across a network connection is never a bad idea.

Compression

One last quick optimization is compression. If bandwidth is a concern or you’re cost-conscious, you can implement a lossless compression and decompression scheme when you send and receive data from Redis. Here’s an example using the bzip2 compression algorithm, which in this extreme case cuts down on the number of bytes sent across the connection by a factor of over 2,000:

>>>

 1 >>> import bz2
 2 
 3 >>> blob = "i have a lot to talk about" * 10000
 4 >>> len(blob.encode("utf-8"))
 5 260000
 6 
 7 >>> # Set the compressed string as value
 8 >>> r.set("msg:500", bz2.compress(blob.encode("utf-8")))
 9 >>> r.get("msg:500")
10 b'BZh91AY&SY\xdaM\x1eu\x01\x11o\x91\x80@\x002l\x87\'  # ... [truncated]
11 >>> len(r.get("msg:500"))
12 122
13 >>> 260_000 / 122  # Magnitude of savings
14 2131.1475409836066
15 
16 >>> # Get and decompress the value, then confirm it's equal to the original
17 >>> rblob = bz2.decompress(r.get("msg:500")).decode("utf-8")
18 >>> rblob == blob
19 True

The way that serialization, encryption, and compression are related here is that they all occur client-side. You do some operation on the original object on the client-side that ends up making more efficient use of Redis once you send the string over to the server. The inverse operation then happens again on the client side when you request whatever it was that you sent to the server in the first place.

Using Hiredis

It’s common for a client library such as redis-py to follow a protocol in how it is built. In this case, redis-py implements the REdis Serialization Protocol, or RESP.

Part of fulfilling this protocol consists of converting some Python object in a raw bytestring, sending it to the Redis server, and parsing the response back into an intelligible Python object.

For example, the string response “OK” would come back as "+OK\r\n", while the integer response 1000 would come back as ":1000\r\n". This can get more complex with other data types such as RESP arrays.

A parser is a tool in the request-response cycle that interprets this raw response and crafts it into something recognizable to the client. redis-py ships with its own parser class, PythonParser, which does the parsing in pure Python. (See .read_response() if you’re curious.)

However, there’s also a C library, Hiredis, that contains a fast parser that can offer significant speedups for some Redis commands such as LRANGE. You can think of Hiredis as an optional accelerator that it doesn’t hurt to have around in niche cases.

All that you have to do to enable redis-py to use the Hiredis parser is to install its Python bindings in the same environment as redis-py:

$ python -m pip install hiredis

What you’re actually installing here is hiredis-py, which is a Python wrapper for a portion of the hiredis C library.

The nice thing is that you don’t really need to call hiredis yourself. Just pip install it, and this will let redis-py see that it’s available and use its HiredisParser instead of PythonParser.

Internally, redis-py will attempt to import hiredis, and use a HiredisParser class to match it, but will fall back to its PythonParser instead, which may be slower in some cases:

# redis/utils.py
try:
    import hiredis
    HIREDIS_AVAILABLE = True
except ImportError:
    HIREDIS_AVAILABLE = False


# redis/connection.py
if HIREDIS_AVAILABLE:
    DefaultParser = HiredisParser
else:
    DefaultParser = PythonParser

Using Enterprise Redis Applications

While Redis itself is open-source and free, several managed services have sprung up that offer a data store with Redis as the core and some additional features built on top of the open-source Redis server:

Amazon ElastiCache for Redis: This is a web service that lets you host a Redis server in the cloud, which you can connect to from an Amazon EC2 instance. For full setup instructions, you can walk through Amazon’s ElastiCache for Redis launch page.
Microsoft’s Azure Cache for Redis: This is another capable enterprise-grade service that lets you set up a customizable, secure Redis instance in the cloud.

The designs of the two have some commonalities. You typically specify a custom name for your cache, which is embedded as part of a DNS name, such as demo.abcdef.xz.0009.use1.cache.amazonaws.com (AWS) or demo.redis.cache.windows.net (Azure).

Once you’re set up, here are a few quick tips on how to connect.

From the command line, it’s largely the same as in our earlier examples, but you’ll need to specify a host with the h flag rather than using the default localhost. For Amazon AWS, execute the following from your instance shell:

$ export REDIS_ENDPOINT="demo.abcdef.xz.0009.use1.cache.amazonaws.com"
$ redis-cli -h $REDIS_ENDPOINT

For Microsoft Azure, you can use a similar call. Azure Cache for Redis uses SSL (port 6380) by default rather than port 6379, allowing for encrypted communication to and from Redis, which can’t be said of TCP. All that you’ll need to supply in addition is a non-default port and access key:

$ export REDIS_ENDPOINT="demo.redis.cache.windows.net"
$ redis-cli -h $REDIS_ENDPOINT -p 6380 -a <primary-access-key>

The -h flag specifies a host, which as you’ve seen is 127.0.0.1 (localhost) by default.

When you’re using redis-py in Python, it’s always a good idea to keep sensitive variables out of Python scripts themselves, and to be careful about what read and write permissions you afford those files. The Python version would look like this:

>>>

>>> import os
>>> import redis

>>> # Specify a DNS endpoint instead of the default localhost
>>> os.environ["REDIS_ENDPOINT"]
'demo.abcdef.xz.0009.use1.cache.amazonaws.com'
>>> r = redis.Redis(host=os.environ["REDIS_ENDPOINT"])

That’s all there is to it. Besides specifying a different host, you can now call command-related methods such as r.get() as normal.

Note: If you want to use solely the combination of redis-py and an AWS or Azure Redis instance, then you don’t really need to install and make Redis itself locally on your machine, since you don’t need either redis-cli or redis-server.

If you’re deploying a medium- to large-scale production application where Redis plays a key role, then going with AWS or Azure’s service solutions can be a scalable, cost-effective, and security-conscious way to operate.

Wrapping Up

That concludes our whirlwind tour of accessing Redis through Python, including installing and using the Redis REPL connected to a Redis server and using redis-py in real-life examples. Here’s some of what you learned:

redis-py lets you do (almost) everything that you can do with the Redis CLI through an intuitive Python API.
Mastering topics such as persistence, serialization, encryption, and compression lets you use Redis to its full potential.
Redis transactions and pipelines are essential parts of the library in more complex situations.
Enterprise-level Redis services can help you smoothly use Redis in production.

Redis has an extensive set of features, some of which we didn’t really get to cover here, including server-side Luda scripting, sharding, and master-slave replication. If you think that Redis is up your alley, then make sure to follow developments as it implements an updated protocol, RESP3.

Python Community Interview With Katrina Durance

2019-06-26T14:00:00+00:00

With PyCon US 2019 over, I decided to catch up with a PyCon first-timer, Katrina Durance. I was curious to see how she found the experience and what her highlights were. I also wanted to understand how attending a conference like PyCon influenced her programming chops.

Ricky: Let’s start with the same questions I ask all my guests. How’d you get into programming? When did you start using Python?

Katrina: Python was my first programming course in grad school back in 2013. I also learned R and SQL. The two jobs I’ve been in since I graduated have been completely SQL-focused, so Python and R fell by the wayside.

Since I work at an arts college in Chicago (Columbia College Chicago) I’m able to take courses for free. We have a gaming program so all our programming courses are game related. Since I’m a gamer and a very visual learner, I decided to take a C# course where we worked on building small games. I liked it.

I really struggled a lot with Python when I first learned it, but working with a programming language in a visual context started to clarify a lot of concepts for me. I knew after that course that I wanted to build my skills to get into full-time programming. I eventually came up with a study plan and decided to return to my Python roots at the beginning of this year.

Ricky: This year you attended PyCon US for the first time. I’m curious why this year was your first. What changed or made you want to go this time around?

Katrina: What really clinched it was knowing that there was going to be a group from PythonistaCafe, and a big contingent from the Chicago Python User Group meetup there. So I knew I would see some friendly faces. I didn’t feel I was doing it all alone.

Ricky: Everyone’s PyCon experience is different. I’m wondering if you had a standout moment this year? Is there one thing that you’ll remember synonymously with your first PyCon experience?

Katrina: The mentored sprints were a big deal for me. I ended up working on an issue on a tool (Hypothesis) that I didn’t understand because I haven’t learned much about testing at all yet.

I was feeling frustrated and concerned that everything would go over my head. But our mentor was amazing and so encouraging and showed me a bunch of things I was learning and kept me going to the end. Now I have a closed issue on the software with my name on it, which is pretty cool.

Ricky: We, of course, met face to face several times over the weekend. But most notably at the PythonistaCafe open space. How was your open space experience? Did you learn anything new, or were there any actionable takeaways?

Katrina: I was excited about the PythonistaCafe open space because of the chance to meet some of the folks I’ve interacted with or just seen on the forum, and I wasn’t disappointed.

I hosted an open space for self-taught programmers like me. I was stunned when 30-ish people showed up. I did my best to manage it and got some positive feedback and helpful advice.

The PyCon Africa meetup was very enlightening because I learned that the reason we’re not seeing a boom in innovation from Africa yet is that the internet is prohibitively expensive in every country represented in the room. It didn’t matter if it was government regulated or privately owned. I would love to help figure out how to solve that problem.

PythonistaCafe Members Coming Together at a PyCon Open Space

Ricky: There is so much to do at PyCon that there’s just not enough time to do it all. So was there anything you wish you had done or a talk you’d missed that you wish you hadn’t? Anything you’d do differently next time?

Katrina: I felt PyCon was what I was hoping it would be for this first time around, to be honest. Next year I want to stay for the sprints, which people kept telling me were amazing.

Ricky: So for those reading this that have yet to go to their first PyCon, this might be the most important question… How did attending PyCon affect how you will write your Python code going forward?

Katrina: I had to talk about my code. When you’re self-taught, you don’t get a lot of opportunities to talk through your code. Through the conversations I had at PyCon, I was encouraged to be more deliberate about taking advantage of my Slack and local Python communities to practice those communication skills.

In other words, I need to notch up my courage and not worry if I’m not coding it or explaining it well yet. I just need to keep coding and explaining.

Ricky: Last but not least, what else do you get up to in your spare time? What other hobbies and interests do you have, aside from Python and coding? Anything you’d like to plug?

Katrina: I love all things Sci-Fi and weird: movies, TV series, books, etc. I’m a LEGO enthusiast and have a penchant for building lost temples out of my love for H. P. Lovecraft stories. I really like VR and enjoy playing games on my Oculus Go. I also make jewelry out of felt, found objects, or even LEGO pieces.

If you’d like to catch up with Katrina and say hi, drop her a message on Twitter.

If there’s someone from the Python community that you’d love me to interview, leave a comment below and let me know.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Generating Random Data in Python

2019-06-25T14:00:00+00:00

In this course, you’ll cover several options for generating random data in Python, and then build up to a comparison of each in terms of its level of security, versatility, purpose, and speed.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Basic Input, Output, and String Formatting in Python

2019-06-24T14:00:00+00:00

To be useful, a program usually needs to communicate with the outside world by obtaining input data from the user and displaying result data back to the user. This tutorial will introduce you to Python input and output.

Input may come directly from the user via the keyboard, or from some external source like a file or database. Output can be displayed directly to the console or IDE, to the screen via a Graphical User Interface (GUI), or again to an external source.

In the previous tutorial in this introductory series, you:

Saw a comparison of some different paradigms used by programming languages to implement definite iteration
Learned about iterables and iterators, two concepts that form the basis of definite iteration in Python
Tied it all together to learn about Python’s for loops

By the end of this tutorial, you’ll know how to:

Take user input from the keyboard with the built-in function input()
Display output to the console with the built-in function print()
Format string data with the string modulo operator

Without further ado, let’s dive in!

Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Reading Input From the Keyboard

Programs often need to obtain data from the user, usually by way of input from the keyboard. The simplest way to accomplish this in Python is with input().

input([<prompt>])

Reads a line of input from the keyboard.

input() pauses program execution to allow the user to type in a line of input from the keyboard. Once the user presses the Enter key, all characters typed are read and returned as a string:

>>>

>>> s = input()
foo bar baz
>>> s
'foo bar baz'

Note that the newline generated when the user presses the Enter key isn’t included as part of the return string.

If you include the optional <prompt> argument, input() displays it as a prompt to the user before pausing to read input:

>>>

>>> name = input('What is your name? ')
What is your name? Winston Smith
>>> name
'Winston Smith'

input() always returns a string. If you want a numeric type, then you need to convert the string to the appropriate type with the int(), float(), or complex() built-in functions:

>>>

 1 >>> n = input('Enter a number: ')
 2 Enter a number: 50
 3 >>> print(n + 100)
 4 Traceback (most recent call last):
 5   File "<stdin>", line 1, in <module>
 6 TypeError: must be str, not int
 7 
 8 >>> n = int(input('Enter a number: '))
 9 Enter a number: 50
10 >>> print(n + 100)
11 150

In the example above, the expression n + 100 on line 3 is invalid because n is a string and 100 is an integer. Line 8 converts n to an integer so the print() statement on line 10 succeeds.

Python Version Note: Should you find yourself working with Python 2.x code, you might bump into a slight difference in the input functions between Python versions 2 and 3.

raw_input() in Python 2 reads input from the keyboard and returns it. raw_input() in Python 2 behaves just like input() in Python 3, as described above.

But Python 2 also has a function called input(). In Python 2, input() reads input from the keyboard, parses and evaluates it as a Python expression, and then returns the resulting value.

Python 3 does not provide a single function that does exactly what Python 2’s input() does. The effect can be mimicked in Python 3 with the expression eval(input()). However, this is considered a security risk because it allows a user to run arbitrary, potentially malicious code.

See the Python documentation for more information on eval(), and the Wikipedia eval page for a discussion of potential security risks.

Writing Output to the Console

In addition to obtaining data from the user, a program will also usually need to present data back to the user. You can display program data to the console in Python with print().

Unformatted Console Output

To display objects to the console, pass them as a comma-separated list of argument to print().

print(<obj>, ..., <obj>)

Displays a string representation of each <obj> to the console.

By default, print() separates each object by a single space and appends a newline to the end of the output:

>>>

>>> fname = 'Winston'
>>> lname = 'Smith'

>>> print('Name:', fname, lname)
Name: Winston Smith

Any type of object can be specified as an argument to print(). If an object isn’t a string, then print() converts it to an appropriate string representation displaying it:

>>>

>>> a = [1, 2, 3]
>>> type(a)
<class 'list'>

>>> b = -12
>>> type(b)
<class 'int'>

>>> d = {'foo': 1, 'bar': 2}
>>> type(d)
<class 'dict'>

>>> type(len)
<class 'builtin_function_or_method'>

>>> print(a, b, d, len)
[1, 2, 3] -12 {'foo': 1, 'bar': 2} <built-in function len>

As you can see, even complex types like lists, dictionaries, and functions can be displayed to the console with print().

Keyword Arguments to `print()`

print() takes a few additional arguments that provide modest control over the format of the output. Each of these is a special type of argument called a keyword argument. This introductory series of tutorials will include a tutorial on functions and parameter passing so you can learn more about keyword arguments.

For now, here’s what you need to know:

Keyword arguments have the form <keyword>=<value>.
Any keyword arguments passed to print() must come at the end, after the list of objects to display.

In the following sections, you’ll see how these keyword arguments affect console output produced by print().

The `sep=` Keyword Argument

Adding the keyword argument sep=<str> causes objects to be separated by the string <str> instead of the default single space:

>>>

>>> print('foo', 42, 'bar')
foo 42 bar

>>> print('foo', 42, 'bar', sep='/')
foo/42/bar

>>> print('foo', 42, 'bar', sep='...')
foo...42...bar

>>> d = {'foo': 1, 'bar': 2, 'baz': 3}
>>> for k, v in d.items():
...     print(k, v, sep=' -> ')
...
foo -> 1
bar -> 2
baz -> 3

To squish objects together without any space between them, specify sep='':

>>>

>>> print('foo', 42, 'bar', sep='')
foo42bar

You can specify any arbitrary string as the separator with the sep= keyword.

The `end=` Keyword Argument

The keyword argument end=<str> causes output to be terminated by <str> instead of the default newline:

>>>

>>> if True:
...     print('foo', end='/')
...     print(42, end='/')
...     print('bar')
...
foo/42/bar

For example, if you are displaying values in a loop, you might use end= to cause the values to be displayed on one line, rather than on individual lines:

>>>

>>> for n in range(10):
...     print(n)
...
0
1
2
3
4
5
6
7
8
9

>>> for n in range(10):
...     print(n, end=(' ' if n < 9 else '\n'))
...
0 1 2 3 4 5 6 7 8 9

Any string may be specified as the output terminator with the end= keyword.

Output Stream Keyword Arguments

print() accepts two additional keyword arguments, both of which affect handling of the output stream:

file=<stream>: By default, print() sends its output to a default stream called sys.stdout, which is usually equivalent to the console. The file=<stream> argument causes output to be sent to an alternate stream designated by <stream> instead.
flush=True: Ordinarily, print() buffers its output and only writes to the output stream intermittently. flush=True specifies that the output stream is forcibly flushed with each print().

These two keyword arguments are presented here for the sake of completeness. You probably don’t need to be too concerned about output streams at this point. They are discussed later in this series in the tutorial on File I/O.

Formatted String Output

print() supports formatting of console output that is rudimentary at best. You can choose how to separate printed objects, and you can specify what goes at the end of the printed line. That’s about it.

In many cases, you’ll need more precise control over the appearance of data destined for display. Python provides several ways to format output string data. In this section, you’ll learn about one of the older ones: the string modulo operator.

In recent versions of Python, there are newer ways to format string data that are arguably superior to the string modulo operator: the string .format() method, and f-strings. You will learn about these in the next tutorial in this series. You may also want to check out these articles:

Though other formatting options are available, the string modulo operator is still widely used. If you’re reading existing Python code, you are likely to encounter the string modulo operator, so it will be beneficial to familiarize yourself with it.

Note: If you’re acquainted with the printf() family of functions of C, Perl, or Java, then you’ll see that these don’t exist in Python. However, there is a quite a bit of similarity between printf() and the string modulo operator, so if you’re familiar with printf(), then a lot of the following will feel familiar.

On the other hand, if you aren’t familiar with printf(), then don’t worry! The following should still make sense.

The String Modulo Operator

The modulo operator (%) is usually used with numbers, in which case it computes remainder from division:

>>>

>>> 11 % 3
2

With string operands, the modulo operator has an entirely different function: string formatting. (The two operations aren’t really much like one another. They share the same name because they are represented by the same symbol: %.)

Here’s what the syntax of the string modulo operator looks like:

<format_string> % <values>

On the left side of the % operator, <format_string> is a string containing one or more conversion specifiers. The <values> on the right side get inserted into <format_string> in place of the conversion specifiers. The resulting formatted string is the value of the expression.

Let’s get started with an example. Here’s a print() statement that displays a formatted string using the string modulo operator:

>>>

>>> print('%d %s cost $%.2f' % (6, 'bananas', 1.74))
6 bananas cost $1.74

In addition to representing the string modulo operation itself, the '%' character also denotes the conversion specifiers in the format string—in this case, '%d', '%s', and '%.2f'.

In the output, each item from the tuple of values is converted to a string value and inserted into the format string in place of the corresponding conversion specifier:

The first item in the tuple is 6, a numeric value that replaces '%d' in the format string.
The next item is the string value 'bananas', which replaces '%s'.
The last item is the float value 1.74, which replaces '%.2f'.

The resulting string is 6 bananas cost $1.74, as demonstrated in the following diagram:

The String Modulo Operator

If there are multiple values to insert, then they must be enclosed in a tuple as illustrated above. If there is only one value, then it can appear by itself:

>>>

>>> print('Hello, my name is %s.' % 'Graham')
Hello, my name is Graham.

Notice also that string modulo operation isn’t only for printing. You can also format values and assign them to another string variable:

>>>

>>> s = 'Hello, my name is %s.' % 'Graham'
>>> s
'Hello, my name is Graham.'

(Again, if you are familiar with the functions related to printf(), then this is reminiscent of sprintf(). If you aren’t, then don’t sweat it.)

Conversion Specifiers

Conversion specifiers appear in the <format_string> and determine how values are formatted when they’re inserted.

A conversion specifier begins with a % character and consists of these components:

%[<flags>][<width>][.<precision>]<type>

% and <type> are required. The remaining components shown in square brackets are optional.

The following table summarizes what each component of a conversion specifier does:

Component	Meaning
`%`	Introduces the conversion specifier
`<flags>`	Indicates one or more flags that exert finer control over formatting
`<width>`	Specifies the minimum width of the formatted result
`.<precision>`	Determines the length and precision of floating point or string output
`<type>`	Indicates the type of conversion to be performed

Read on for more detail on how these work.

Conversion Type

The conversion type, <type>, is the last component of the conversion specifier:

%[<flags>][<width>][.<precision>]<type>

It determines the type of conversion the corresponding value undergoes before insertion into the format string. Here’s a table that lists the possible conversion types:

`<type>`	Conversion Type
`d`, `i`, `u`	Decimal integer
`x`, `X`	Hexadecimal integer
`o`	Octal integer
`f`, `F`	Floating point
`e`, `E`	Exponential
`g`, `G`	Floating point or Exponential
`c`	Single character
`s`, `r`, `a`	String
`%`	Single `'%'` character

You’ll see how to use these conversion types in the following sections.

Integer Conversion Types

The d, i, u, x, X, and o conversion types correspond to integer values.

d, i, and u are functionally equivalent. They all convert the corresponding argument to a string representation of a decimal integer:

>>>

>>> '%d, %i, %u' % (42, 42, 42)
'42, 42, 42'

>>> '%d, %i, %u' % (-42, -42, -42)
'-42, -42, -42'

The value can either be positive or negative. If it is negative, then the resulting value will start with a '-' character.

x and X convert to a string representation of a hexadecimal integer value, and o converts to a string representation of an octal integer value:

>>>

>>> '%x, %X' % (252, 252)
'fc, FC'
>>> '%o' % 16
'20'

x produces lowercase output, and X produces uppercase. (Uppercase 'O' is not a valid conversion type.)

You can gain some additional control over the resulting format by using conversion flags, which you’ll learn more about in an upcoming section.

Floating Point Conversion Types

Conversion types f and F convert to a string representation of a floating point number, while e and E produce a string representing exponential (scientific) notation:

>>>

>>> '%f, %F' % (3.14159, 3.14)
'3.141590, 3.140000'
>>> '%e, %E' % (1000.0, 1000.0)
'1.000000e+03, 1.000000E+03'

e produces lowercase output, and E produces uppercase.

Deep Dive: inf and NaN

Under some circumstances, a floating-point operation can result in a value that is essentially infinite. The string representation of such a number in Python is 'inf'.

It also may happen that a floating-point operation produces a value that is not representable as a number. Python represents this with the string 'NaN'.

When these values are converted with the string modulo operator, the conversion type character controls the case of the resulting output. f and e produce lowercase output, while F and E produce uppercase:
>>>
>>> x = float('NaN')
>>> '%f, %e, %F, %E' % (x, x, x, x)
'nan, nan, NAN, NAN'
>>> y = float('Inf')
>>> '%f, %e, %F, %E' % (y, y, y, y)
'inf, inf, INF, INF'
This is the only difference between the f and F conversion types.

The g and G conversion types choose between floating point or exponential output, depending on the magnitude of the exponent and the value specified for .<precision>. (See below.) Output is the same as e/E if the exponent is less than -4 or not less than .<precision>. Otherwise, it’s the same as f/F:

>>>

>>> '%g' % 3.14
'3.14'

>>> '%g' % 0.00000003
'3e-08'
>>> '%G' % 0.00000003
'3E-08'

Basically, you can think of these conversion types as making a “reasonable” choice. They’ll produce floating point output if the value in question is reasonably suitable for it, and exponential format otherwise.

Similar to the other floating point conversion types, g produces lowercase output, and G produces uppercase.

Character Conversion Types

c inserts a single character. The corresponding value may be either an integer or a single-character string:

>>>

>>> '%c' % 97
'a'

>>> '[%c]' % 'y'
'[y]'

The c conversion type supports conversion to Unicode characters as well:

>>>

>>> '%c' % 8721
'∑'

s, r, and a produce string output using the built-in functions str(), repr(), and ascii(), respectively:

>>>

>>> '%s' % 'foo'
'foo'
>>> '%r' % 'foo'
"'foo'"
>>> '%a' % 'foo'
"'foo'"

The justification and padding of string output can be controlled with the <width> and .<precision> specifiers, as you will see shortly.

Inserting a `'%'` Character

To insert a literal '%' character into the output, specify two consecutive % characters in the format string. The first introduces a conversion specifier (as usual), and the second specifies that the conversion type is %, which results in a single '%' character in the output.

In this example, %d%% means a decimal integer conversion type followed by a literal '%' character:

>>>

>>> 'Get %d%% off on %s today only!' % (30, 'bananas')
'Get 30% off on bananas today only!'

Note that the % conversion type doesn’t consume any of the <values> to the right of the string modulo operator.

Width and Precision Specifiers

<width> and .<precision> sit in the middle of the conversion specifier:

%[<flags>][<width>][.<precision>]<type>

They determine how much horizontal space a formatted value occupies.

The `<width>` Specifier

<width> specifies the minimum width of the output field. If the output is shorter than <width>, then by default it is right-justified in a field that is <width> characters wide, and padded with ASCII space characters on the left:

>>>

>>> '%5s' % 'foo'
'  foo'

>>> '%3d' % 4
'  4'

(The justification and padding character can be modified. See conversion flags below.)

If the output length is greater than <width>, then <width> has no effect:

>>>

>>> '%2d' % 1234, '%d' % 1234
('1234', '1234')
>>> '%2s' % 'foobar', '%s' % 'foobar'
('foobar', 'foobar')

Each of these examples specifies a field width of 2. But since the values to be formatted are more than two characters, the result is the same as when no <width> is specified.

The `.<precision>` Specifier

.<precision> affects the floating point, exponential, and string conversion types.

For the f, F, e, and E types, .<precision> determines the number of digits after the decimal point:

>>>

>>> '%.2f' % 123.456789
'123.46'

>>> '%.2e' % 123.456789
'1.23e+02'

For the g and G types, .<precision> determines the total number of significant digits before and after the decimal point:

>>>

>>> '%.2g' % 123.456789
'1.2e+02'

String values formatted with the s, r, and a types are truncated to the length specified by .<precision>:

>>>

>>> '%.4s' % 'foobar'
'foob'

It is very common to see <width> and .<precision> used together:

>>>

>>> '%8.2f' % 123.45678
'  123.46'

>>> '%8.3s' % 'foobar'
'     foo'

Either of <width> or .<precision> can be specified as an asterisk character (*), in which case the value to be used is taken from the next item in the <values> tuple:

>>>

>>> '%*d' % (5, 123)
'  123'

There isn’t much need for this when the <width> value is given as a constant. There isn’t any functional difference between the example given above and this:

>>>

>>> '%5d' % 123
'  123'

But you can also specify <width> and .<precision> by variable:

>>>

>>> for i in range(3):
...     w = int(input('Enter width: '))
...     print('[%*s]' % (w, 'foo'))
...
Enter width: 2
[foo]
Enter width: 4
[ foo]
Enter width: 8
[     foo]

This allows the width or precision to be determined at run-time, and potentially change from one execution to the next.

Conversion Flags

Optional conversion <flags> are specified just after the initial % character:

%[<flags>][<width>][.<precision>]<type>

These allow finer control over the display of certain conversion types. <flags> can include any of the characters shown in the following table:

Character	Controls
`#`	Display of base or decimal point for integer and floating point values
`0`	Padding of values that are shorter than the specified field width
`-`	Justification of values that are shorter than the specified field width
`+` `' '` (space)	Display of leading sign for numeric values

The following sections explain how conversion flags operate in greater detail.

The `#` Flag

For the octal and hexadecimal conversion types, the # flag causes base information to be included in the formatted output. For the o conversion type, this flag adds a leading '0o'. For the x and X conversion types, it adds a leading '0x' or '0X':

>>>

>>> '%#o' % 16
'0o20'

>>> '%#x' % 16, '%#X' % 16
('0x10', '0X10')

The # flag is ignored for the decimal conversion types d, i, and u.

For floating point values, the # flag forces the output to always contain a decimal point. Ordinarily, floating point values will not contain a decimal point if there aren’t any digits after it. This flag forces a decimal point to be included:

>>>

>>> '%.0f' % 123
'123'
>>> '%#.0f' % 123
'123.'

>>> '%.0e' % 123
'1e+02'
>>> '%#.0e' % 123
'1.e+02'

This also works for values displayed in exponential notation, as shown.

The `0` Flag

When a formatted numeric value is shorter than the specified field width, the default behavior is to pad the field with ASCII space characters. The 0 flag causes padding with '0' characters instead:

>>>

>>> '%05d' % 123
'00123'

>>> '%08.2f' % 1.2
'00001.20'

The 0 flag can be used with all the numeric conversion types: d, i, u, x, X, o, f, F, e, E, g, and G.

The `-` Flag

When a formatted value is shorter than the specified field width, it is usually right-justified in the field. The - flag causes the value to be left-justified in the specified field instead:

>>>

>>> '%-5d' % 123
'123  '

>>> '%-8.2f' % 123.3
'123.30  '

>>> '%-*s' % (10, 'foo')
'foo       '

You can use the - flag with the string conversion types s, a, and r, as well as all the numeric conversion types. For numeric types, if both 0 and - are present, then 0 is ignored.

The `+` and `' '` Flags

By default, positive numeric values do not have a leading sign character. The + flag adds a '+' character to the left of numeric output:

>>>

>>> '%+d' % 3
'+3'
>>> '%+5d' % 3
'   +3'

The ' ' (space character) flag causes positive numeric values to be preceded by a space character:

>>>

>>> '% d' % 3
' 3'

These flags have no effect on negative numeric values, which always have a leading '-' character.

Specifying Values by Dictionary Mapping

The <values> inserted into the format string may be specified as a dictionary instead of a tuple. In that case, each conversion specifier must contain one of the dictionary keys in parentheses immediately following the '%' character.

Here’s an example:

>>>

>>> '%d %s cost $%.2f' % (6, 'bananas', 1.74)
'6 bananas cost $1.74'

>>> d = {'quantity': 6, 'item': 'bananas', 'price': 1.74}
>>> '%(quantity)d %(item)s cost $%(price).2f' % d
'6 bananas cost $1.74'

Using this technique, you can specify the inserted values in any order:

>>>

>>> d = {'quantity': 6, 'item': 'bananas', 'price': 1.74}

>>> '%(quantity)d %(item)s cost $%(price).2f' % d
'6 bananas cost $1.74'

>>> 'It will cost you $%(price).2f for %(item)s, if you buy %(quantity)d' % d
'It will cost you $1.74 for bananas, if you buy 6'

All the conversion specifier items shown above—<flags>, <width>, .<precision>, and <type>—still have the same meaning:

>>>

>>> 'Quantity: %(quantity)03d' % d
'Quantity: 006'

>>> 'Item:     %(item).5s' % d
'Item:     banan'

Note: If you specify <values> by dictionary mapping, then you can’t use * to specify <width> or .<precision>.

Python Input and Output: Conclusion

In this tutorial, you learned about Python input and output, and how your Python program can communicate with the user. You should now be able to obtain data from the user with input(), and display results to the console with print().

You also learned how to make data displayed to the user more presentable by formatting it with the string modulo operator.

As versatile as the string modulo operator is, Python provides two newer ways to format string data that are even better: the string .format() method and the formatted string literal. Head to the next tutorial to learn about them!

« Definite Iteration in Python

Basic Input and Output in Python

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Use Python lambda Functions

2019-06-19T14:00:00+00:00

Python and other languages like Java, C#, and even C++ have had lambda functions added to their syntax, whereas languages like LISP or the ML family of languages, Haskell, OCaml, and F#, use lambdas as a core concept.

Python lambdas are little, anonymous functions, subject to a more restrictive but more concise syntax than regular Python functions.

By the end of this article, you’ll know:

How Python lambdas came to be
How lambdas compare with regular function objects
How to write lambda functions
Which functions in the Python standard library leverage lambdas
When to use or avoid Python lambda functions

Notes: You’ll see some code examples using lambda that seem to blatantly ignore Python style best practices. This is only intended to illustrate lambda calculus concepts or to highlight the capabilities of Python lambda.

Those questionable examples will be contrasted with better approaches or alternatives as you progress through the article.

This tutorial is mainly for intermediate to experienced Python programmers, but it is accessible to any curious minds with interest in programming and lambda calculus.

All the examples included in this tutorial have been tested with Python 3.7.

Lambda Calculus

Lambda expressions in Python and other programming languages have their roots in lambda calculus, a model of computation invented by Alonzo Church. You’ll uncover when lambda calculus was introduced and why it’s a fundamental concept that ended up in the Python ecosystem.

History

Alonzo Church formalized lambda calculus, a language based on pure abstraction, in the 1930s. Lambda functions are also referred to as lambda abstractions, a direct reference to the abstraction model of Alonzo Church’s original creation.

Lambda calculus can encode any computation. It is Turing complete, but contrary to the concept of a Turing machine, it is pure and does not keep any state.

Functional languages get their origin in mathematical logic and lambda calculus, while imperative programming languages embrace the state-based model of computation invented by Alan Turing. The two models of computation, lambda calculus and Turing machines, can be translated into each another. This equivalence is known as the Church-Turing hypothesis.

Functional languages directly inherit the lambda calculus philosophy, adopting a declarative approach of programming that emphasizes abstraction, data transformation, composition, and purity (no state and no side effects). Examples of functional languages include Haskell, Lisp, or Erlang.

By contrast, the Turing Machine led to imperative programming found in languages like Fortran, C, or Python.

The imperative style consists of programming with statements, driving the flow of the program step by step with detailed instructions. This approach promotes mutation and requires managing state.

The separation in both families presents some nuances, as some functional languages incorporate imperative features, like OCaml, while functional features have been permeating the imperative family of languages in particular with the introduction of lambda functions in Java, or Python.

Python is not inherently a functional language, but it adopted some functional concepts early on. In January 1994, map(), filter(), reduce(), and the lambda operator were added to the language.

First Example

Here are a few examples to give you an appetite for some Python code, functional style.

The identity function, a function that returns its argument, is expressed with a standard Python function definition using the keyword def as follows:

>>>

>>> def identity(x):
...     return x

identity() takes an argument x and returns it upon invocation.

In contrast, if you use a Python lambda construction, you get the following:

>>>

>>> lambda x: x

In the example above, the expression is composed of:

The keyword: lambda
A bound variable: x
A body: x

Note: In the context of this article, a bound variable is an argument to a lambda function.

In contrast, a free variable is not bound and may be referenced in the body of the expression. A free variable can be a constant or a variable defined in the enclosing scope of the function.

You can write a slightly more elaborated example, a function that adds 1 to an argument, as follows:

>>>

>>> lambda x: x + 1

You can apply the function above to an argument by surrounding the function and its argument with parentheses:

>>>

>>> (lambda x: x + 1)(2)
3

Reduction is a lambda calculus strategy to compute the value of the expression. It consists of substituting the argument 2 for x:

(lambda x: x + 1)(2) = lambda 2: 2 + 1
                     = 2 + 1
                     = 3

Because a lambda function is an expression, it can be named. Therefore you could write the previous code as follows:

>>>

>>> add_one = lambda x: x + 1
>>> add_one(2)
3

The above lambda function is equivalent to writing this:

def add_one(x):
    return x + 1

These functions all take a single argument. You may have noticed that, in the definition of the lambdas, the arguments don’t have parentheses around them. Multi-argument functions (functions that take more than one argument) are expressed in Python lambdas by listing arguments and separating them with a comma (,) but without surrounding them with parentheses:

>>>

>>> full_name = lambda first, last: f'Full name: {first.title()} {last.title()}'
>>> full_name('guido', 'van rossum')
'Full name: Guido Van Rossum'

The lambda function assigned to full_name takes two arguments and returns a string interpolating the two parameters first and last. As expected, the definition of the lambda lists the arguments with no parentheses, whereas calling the function is done exactly like a normal Python function, with parentheses surrounding the arguments.

Anonymous Functions

The following terms may be used interchangeably depending on the programming language type and culture:

Anonymous functions
Lambda functions
Lambda expressions
Lambda abstractions
Lambda form
Function literals

For the rest of this article after this section, you’ll mostly see the term lambda function.

Taken literally, an anonymous function is a function without a name. In Python, an anonymous function is created with the lambda keyword. More loosely, it may or not be assigned a name. Consider a two-argument anonymous function defined with lambda but not bound to a variable. The lambda is not given a name:

>>>

>>> lambda x, y: x + y

The function above defines a lambda expression that takes two arguments and returns their sum.

Other than providing you with the feedback that Python is perfectly fine with this form, it doesn’t lead to any practical use. You could invoke the function in the Python interpreter:

>>>

>>> _(1, 2)
3

The example above is taking advantage of the interactive interpreter-only feature provided via the underscore (_). See the note below for more details.

You could not write similar code in a Python module. Consider the _ in the interpreter as a side effect that you took advantage of. In a Python module, you would assign a name to the lambda, or you would pass the lambda to a function. You’ll use those two approaches later in this article.

Note: In the interactive interpreter, the single underscore (_) is bound to the last expression evaluated.

In the example above, the _ points to the lambda function. For more details about the usage of this special character in Python, check out The Meaning of Underscores in Python.

Another pattern used in other languages like JavaScript is to immediately execute a Python lambda function. This is known as an Immediately Invoked Function Expression (IIFE, pronounce “iffy”). Here’s an example:

>>>

>>> (lambda x, y: x + y)(2, 3)
5

The lambda function above is defined and then immediately called with two arguments (2 and 3). It returns the value 5, which is the sum of the arguments.

Several examples in this tutorial use this format to highlight the anonymous aspect of a lambda function and avoid focusing on lambda in Python as a shorter way of defining a function.

Python does not encourage using immediately invoked lambda expressions. It simply results from a lambda expression being callable, unlike the body of a normal function.

Lambda functions are frequently used with higher-order functions, which take one or more functions as arguments or return one or more functions.

A lambda function can be a higher-order function by taking a function (normal or lambda) as an argument like in the following contrived example:

>>>

>>> high_ord_func = lambda x, func: x + func(x)
>>> high_ord_func(2, lambda x: x * x)
6
>>> high_ord_func(2, lambda x: x + 3)
7

Python exposes higher-order functions as built-in functions or in the standard library. Examples include map(), filter(), functools.reduce(), as well as key functions like sort(), sorted(), min(), and max(). You’ll use lambda functions together with Python higher-order functions in Appropriate Uses of Lambda Expressions.

Python Lambda and Regular Functions

This quote from the Python Design and History FAQ seems to set the tone about the overall expectation regarding the usage of lambda functions in Python:

Unlike lambda forms in other languages, where they add functionality, Python lambdas are only a shorthand notation if you’re too lazy to define a function. (Source)

Nevertheless, don’t let this statement deter you from using Python’s lambda. At first glance, you may accept that a lambda function is a function with some syntactic sugar shortening the code to define or invoke a function. The following sections highlight the commonalities and subtle differences between normal Python functions and lambda functions.

Functions

At this point, you may wonder what fundamentally distinguishes a lambda function bound to a variable from a regular function with a single return line: under the surface, almost nothing. Let’s verify how Python sees a function built with a single return statement versus a function constructed as an expression (lambda).

The dis module exposes functions to analyze Python bytecode generated by the Python compiler:

>>>

>>> import dis
>>> add = lambda x, y: x + y
>>> type(add)
<class 'function'>
>>> dis.dis(add)
  1           0 LOAD_FAST                0 (x)
              2 LOAD_FAST                1 (y)
              4 BINARY_ADD
              6 RETURN_VALUE
>>> add
<function <lambda> at 0x7f30c6ce9ea0>

You can see that dis() expose a readable version of the Python bytecode allowing the inspection of the low-level instructions that the Python interpreter will use while executing the program.

Now see it with a regular function object:

>>>

>>> import dis
>>> def add(x, y): return x + y
>>> type(add)
<class 'function'>
>>> dis.dis(add)
  1           0 LOAD_FAST                0 (x)
              2 LOAD_FAST                1 (y)
              4 BINARY_ADD
              6 RETURN_VALUE
>>> add
<function add at 0x7f30c6ce9f28>

The bytecode interpreted by Python is the same for both functions. But you may notice that the naming is different: the function name is add for a function defined with def, whereas the Python lambda function is seen as lambda.

Traceback

You saw in the previous section that, in the context of the lambda function, Python did not provide the name of the function, but only <lambda>. This can be a limitation to consider when an exception occurs, and a traceback shows only <lambda>:

>>>

>>> div_zero = lambda x: x / 0
>>> div_zero(2)
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 1, in <lambda>
ZeroDivisionError: division by zero

The traceback of an exception raised while a lambda function is executed only identifies the function causing the exception as <lambda>.

Here’s the same exception raised by a normal function:

>>>

>>> def div_zero(x): return x / 0
>>> div_zero(2)
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 1, in div_zero
ZeroDivisionError: division by zero

The normal function causes a similar error but results in a more precise traceback because it gives the function name, div_zero.

Syntax

As you saw in the previous sections, a lambda form presents syntactic distinctions from a normal function. In particular, a lambda function has the following characteristics:

It can only contain expressions and can’t include statements in its body.
It is written as a single line of execution.
It does not support type annotations.
It can be immediately invoked (IIFE).

No Statements

A lambda function can’t contain any statements. In a lambda function, statements like return, pass, assert, or raise will raise a SyntaxError exception. Here’s an example of adding assert to the body of a lambda:

>>>

>>> (lambda x: assert x == 2)(2)
  File "<input>", line 1
    (lambda x: assert x == 2)(2)
                    ^
SyntaxError: invalid syntax

This contrived example intended to assert that parameter x had a value of 2. But, the interpreter identifies a SyntaxError while parsing the code that involves the statement assert in the body of the lambda.

Single Expression

In contrast to a normal function, a Python lambda function is a single expression. Although, in the body of a lambda, you can spread the expression over several lines using parentheses or a multiline string, it remains a single expression:

>>>

>>> (lambda x:
... (x % 2 and 'odd' or 'even'))(3)
'odd'

The example above returns the string 'odd' when the lambda argument is odd, and 'even' when the argument is even. It spreads across two lines because it is contained in a set of parentheses, but it remains a single expression.

Type Annotations

If you’ve started adopting type hinting, which is now available in Python, then you have another good reason to prefer normal functions over Python lambda functions. Check out Python Type Checking (Guide) to get learn more about Python type hints and type checking. In a lambda function, there is no equivalent for the following:

def full_name(first: str, last: str) -> str:
    return f'{first.title()} {last.title()}'

Any type error with full_name() can be caught by tools like mypy or pyre, whereas a SyntaxError with the equivalent lambda function is raised at runtime:

>>>

>>> lambda first: str, last: str: first.title() + " " + last.title() -> str
  File "<stdin>", line 1
    lambda first: str, last: str: first.title() + " " + last.title() -> str

SyntaxError: invalid syntax

Like trying to include a statement in a lambda, adding type annotation immediately results in a SyntaxError at runtime.

IIFE

You’ve already seen several examples of immediately invoked function execution:

>>>

>>> (lambda x: x * x)(3)
9

Outside of the Python interpreter, this feature is probably not used in practice. It’s a direct consequence of a lambda function being callable as it is defined. For example, this allows you to pass the definition of a Python lambda expression to a higher-order function like map(), filter(), or functools.reduce(), or to a key function.

Arguments

Like a normal function object defined with def, Python lambda expressions support all the different ways of passing arguments. This includes:

Positional arguments
Named arguments (sometimes called keyword arguments)
Variable list of arguments (often referred to as varargs)
Variable list of keyword arguments
Keyword-only arguments

The following examples illustrate options open to you in order to pass arguments to lambda expressions:

>>>

>>> (lambda x, y, z: x + y + z)(1, 2, 3)
6
>>> (lambda x, y, z=3: x + y + z)(1, 2)
6
>>> (lambda x, y, z=3: x + y + z)(1, y=2)
6
>>> (lambda *args: sum(args))(1,2,3)
6
>>> (lambda **kwargs: sum(kwargs.values()))(one=1, two=2, three=3)
6
>>> (lambda x, *, y=0, z=0: x + y + z)(1, y=2, z=3)
6

Decorators

In Python, a decorator is the implementation of a pattern that allows adding a behavior to a function or a class. It is usually expressed with the @decorator syntax prefixing a function. Here’s a contrived example:

def some_decorator(f):
    def wraps(*args):
        print(f"Calling function '{f.__name__}'")
        return f(args)
    return wraps

@some_decorator
def decorated_function(x):
    print(f"With argument '{x}'")

In the example above, some_decorator() is a function that adds a behavior to decorated_function(), so that invoking decorated_function("Python") results in the following output:

Calling function 'decorated_function'
With argument 'Python'

decorated_function() only prints With argument 'Python', but the decorator adds an extra behavior that also prints Calling function 'decorated_function'.

A decorator can be applied to a lambda. Although it’s not possible to decorate a lambda with the @decorator syntax, a decorator is just a function, so it can call the lambda function:

 1 # Defining a decorator
 2 def trace(f):
 3     def wrap(*args, **kwargs):
 4         print(f"[TRACE] func: {f.__name__}, args: {args}, kwargs: {kwargs}")
 5         return f(*args, **kwargs)
 6 
 7     return wrap
 8 
 9 # Applying decorator to a function
10 @trace
11 def add_two(x):
12     return x + 2
13 
14 # Calling the decorated function
15 add_two(3)
16 
17 # Applying decorator to a lambda
18 print((trace(lambda x: x ** 2))(3))

add_two(), decorated with @trace on line 11, is invoked with argument 3 on line 15. By contrast, on line 18, a lambda function is immediately involved and embedded in a call to trace(), the decorator. When you execute the code above you obtain the following:

[TRACE] func: add_two, args: (3,), kwargs: {}
[TRACE] func: <lambda>, args: (3,), kwargs: {}
9

See how, as you’ve already seen, the name of the lambda function appears as <lambda>, whereas add_two is clearly identified for the normal function.

Decorating the lambda function this way could be useful for debugging purposes, possibly to debug the behavior of a lambda function used in the context of a higher-order function or a key function. Let’s see an example with map():

list(map(trace(lambda x: x*2), range(3)))

The first argument of map() is a lambda that multiplies its argument by 2. This lambda is decorated with trace(). When executed, the example above outputs the following:

[TRACE] Calling <lambda> with args (0,) and kwargs {}
[TRACE] Calling <lambda> with args (1,) and kwargs {}
[TRACE] Calling <lambda> with args (2,) and kwargs {}
[0, 2, 4]

The result [0, 2, 4] is a list obtained from multiplying each element of range(3). For now, consider range(3) equivalent to the list [0, 1, 2].

You will be exposed to map() in more details in Map.

A lambda can also be a decorator, but it’s not recommended. If you find yourself needing to do this, consult PEP 8, Programming Recommendations.

For more on Python decorators, check out Primer on Python Decorators.

Closure

A closure is a function where every free variable, everything except parameters, used in that function is bound to a specific value defined in the enclosing scope of that function. In effect, closures define the environment in which they run, and so can be called from anywhere.

The concepts of lambdas and closures are not necessarily related, although lambda functions can be closures in the same way that normal functions can also be closures. Some languages have special constructs for closure or lambda (for example, Groovy with an anonymous block of code as Closure object), or a lambda expression (for example, Java Lambda expression with a limited option for closure).

Here’s a closure constructed with a normal Python function:

 1 def outer_func(x):
 2     y = 4
 3     def inner_func(z):
 4         print(f"x = {x}, y = {y}, z = {z}")
 5         return x + y + z
 6     return inner_func
 7 
 8 for i in range(3):
 9     closure = outer_func(i)
10     print(f"closure({i+5}) = {closure(i+5)}")

outer_func() returns inner_func(), a nested function that computes the sum of three arguments:

x is passed as an argument to outer_func().
y is a variable local to outer_func().
z is an argument passed to inner_func().

To test the behavior of outer_func() and inner_func(), outer_func() is invoked three times in a for loop that prints the following:

x = 0, y = 4, z = 5
closure(5) = 9
x = 1, y = 4, z = 6
closure(6) = 11
x = 2, y = 4, z = 7
closure(7) = 13

On line 9 of the code, inner_func() returned by the invocation of outer_func() is bound to the name closure. On line 5, inner_func() captures x and y because it has access to its embedding environment, such that upon invocation of the closure, it is able to operate on the two free variables x and y.

Similarly, a lambda can also be a closure. Here’s the same example with a Python lambda function:

 1 def outer_func(x):
 2     y = 4
 3     return lambda z: x + y + z
 4 
 5 for i in range(3):
 6     closure = outer_func(i)
 7     print(f"closure({i+5}) = {closure(i+5)}")

When you execute the code above, you obtain the following output:

closure(5) = 9
closure(6) = 11
closure(7) = 13

On line 6, outer_func() returns a lambda and assigns it to to the variable closure. On line 3, the body of the lambda function references x and y. The variable y is available at definition time, whereas x is defined at runtime when outer_func() is invoked.

In this situation, both the normal function and the lambda behave similarly. In the next section, you’ll see a situation where the behavior of a lambda can be deceptive due to its evaluation time (definition time vs runtime).

Evaluation Time

In some situations involving loops, the behavior of a Python lambda function as a closure may be counterintuitive. It requires understanding when free variables are bound in the context of a lambda. The following examples demonstrate the difference when using a regular function vs using a Python lambda.

Test the scenario first using a regular function:

>>>

 1 >>> def wrap(n):
 2 ...     def f():
 3 ...         print(n)
 4 ...     return f
 5 ...
 6 >>> numbers = 'one', 'two', 'three'
 7 >>> funcs = []
 8 >>> for n in numbers:
 9 ...     funcs.append(wrap(n))
10 ...
11 >>> for f in funcs:
12 ...     f()
13 ...
14 one
15 two
16 three

In a normal function, n is evaluated at definition time, on line 9, when the function is added to the list: funcs.append(wrap(n)).

Now, with the implementation of the same logic with a lambda function, observe the unexpected behavior:

>>>

 1 >>> numbers = 'one', 'two', 'three'
 2 >>> funcs = []
 3 >>> for n in numbers:
 4 ...     funcs.append(lambda: print(n))
 5 ...
 6 >>> for f in funcs:
 7 ...     f()
 8 ...
 9 three
10 three
11 three

The unexpected result occurs because the free variable n, as implemented, is bound at the execution time of the lambda expression. The Python lambda function on line 4 is a closure that captures n, a free variable bound at runtime. At runtime, while invoking the function f on line 7, the value of n is three.

To overcome this issue, you can assign the free variable at definition time as follows:

>>>

 1 >>> numbers = 'one', 'two', 'three'
 2 >>> funcs = []
 3 >>> for n in numbers:
 4 ...     funcs.append(lambda n=n: print(n))
 5 ...
 6 >>> for f in funcs:
 7 ...     f()
 8 ...
 9 one
10 two
11 three

A Python lambda function behaves like a normal function in regard to arguments. Therefore, a lambda parameter can be initialized with a default value: the parameter n takes the outer n as a default value. The Python lambda function could have been written as lambda x=n: print(x) and have the same result.

The Python lambda function is invoked without any argument on line 7, and it uses the default value n set at definition time.

Testing Lambdas

Python lambdas can be tested similarly to regular functions. It’s possible to use both unittest and doctest.

unittest

The unittest module handles Python lambda functions similarly to regular functions:

import unittest

addtwo = lambda x: x + 2

class LambdaTest(unittest.TestCase):
    def test_add_two(self):
        self.assertEqual(addtwo(2), 4)

    def test_add_two_point_two(self):
        self.assertEqual(addtwo(2.2), 4.2)

    def test_add_three(self):
        # Should fail
        self.assertEqual(addtwo(3), 6)

if __name__ == '__main__':
    unittest.main(verbosity=2)

LambdaTest defines a test case with three test methods, each of them exercising a test scenario for addtwo() implemented as a lambda function. The execution of the Python file lambda_unittest.py that contains LambdaTest produces the following:

$ python lambda_unittest.py
test_add_three (__main__.LambdaTest) ... FAIL
test_add_two (__main__.LambdaTest) ... ok
test_add_two_point_two (__main__.LambdaTest) ... ok

======================================================================
FAIL: test_add_three (__main__.LambdaTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "lambda_unittest.py", line 18, in test_add_three
    self.assertEqual(addtwo(3), 6)
AssertionError: 5 != 6

----------------------------------------------------------------------
Ran 3 tests in 0.001s

FAILED (failures=1)

As expected, we have two successful test cases and one failure for test_add_three: the result is 5, but the expected result was 6. This failure is due to an intentional mistake in the test case. Changing the expected result from 6 to 5 will satisfy all the tests for LambdaTest.

doctest

The doctest module extracts interactive Python code from docstring to execute tests. Although the syntax of Python lambda functions does not support a typical docstring, it is possible to assign a string to the __doc__ element of a named lambda:

addtwo = lambda x: x + 2
addtwo.__doc__ = """Add 2 to a number.
    >>> addtwo(2)
    4
    >>> addtwo(2.2)
    4.2
    >>> addtwo(3) # Should fail
    6
    """

if __name__ == '__main__':
    import doctest
    doctest.testmod(verbose=True)

The doctest in the doc comment of lambda addtwo() describes the same test cases as in the previous section.

When you execute the tests via doctest.testmod(), you get the following:

$ python lambda_doctest.py
Trying:
    addtwo(2)
Expecting:
    4
ok
Trying:
    addtwo(2.2)
Expecting:
    4.2
ok
Trying:
    addtwo(3) # Should fail
Expecting:
    6
**********************************************************************
File "lambda_doctest.py", line 16, in __main__.addtwo
Failed example:
    addtwo(3) # Should fail
Expected:
    6
Got:
    5
1 items had no tests:
    __main__
**********************************************************************
1 items had failures:
   1 of   3 in __main__.addtwo
3 tests in 2 items.
2 passed and 1 failed.
***Test Failed*** 1 failures.

The failed test results from the same failure explained in the execution of the unit tests in the previous section.

You can add a docstring to a Python lambda via an assignment to __doc__ to document a lambda function. Although possible, the Python syntax better accommodates docstring for normal functions than lambda functions.

For a comprehensive overview of unit testing in Python, you may want to refer to Getting Started With Testing in Python.

Lambda Expression Abuses

Several examples in this article, if written in the context of professional Python code, would qualify as abuses.

If you find yourself trying to overcome something that a lambda expression does not support, this is probably a sign that a normal function would be better suited. The docstring for a lambda expression in the previous section is a good example. Attempting to overcome the fact that a Python lambda function does not support statements is another red flag.

The next sections illustrate a few examples of lambda usages that should be avoided. Those examples might be situations where, in the context of Python lambda, the code exhibits the following pattern:

It doesn’t follow the Python style guide (PEP 8)
It’s cumbersome and difficult to read.
It’s unnecessarily clever at the cost of difficult readability.

Raising an Exception

Trying to raise an exception in a Python lambda should make you think twice. There are some clever ways to do so, but even something like the following is better to avoid:

>>>

>>> def throw(ex): raise ex
>>> (lambda: throw(Exception('Something bad happened')))()
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 1, in <lambda>
    File "<stdin>", line 1, in throw
Exception: Something bad happened

Because a statement is not syntactically correct in a Python lambda body, the workaround in the example above consists of abstracting the statement call with a dedicated function throw(). Using this type of workaround should be avoided. If you encounter this type of code, you should consider refactoring the code to use a regular function.

Cryptic Style

As in any programming languages, you will find Python code that can be difficult to read because of the style used. Lambda functions, due to their conciseness, can be conducive to writing code that is difficult to read.

The following lambda example contains several bad style choices:

>>>

>>> (lambda _: list(map(lambda _: _ // 2, _)))([1,2,3,4,5,6,7,8,9,10])
[0, 1, 1, 2, 2, 3, 3, 4, 4, 5]

The underscore (_) refers to a variable that you don’t need to refer to explicitly. But in this example, three _ refer to different variables. An initial upgrade to this lambda code could be to name the variables:

>>>

>>> (lambda some_list: list(map(lambda n: n // 2,
                                some_list)))([1,2,3,4,5,6,7,8,9,10])
[0, 1, 1, 2, 2, 3, 3, 4, 4, 5]

Admittedly, it’s still difficult to read. By still taking advantage of a lambda, a regular function would go a long way to render this code more readable, spreading the logic over a few lines and function calls:

>>>

>>> def div_items(some_list):
      div_by_two = lambda n: n // 2
      return map(div_by_two, some_list)
>>> list(div_items([1,2,3,4,5,6,7,8,9,10])))
[0, 1, 1, 2, 2, 3, 3, 4, 4, 5]

This is still not optimal but shows you a possible path to make code, and Python lambda functions in particular, more readable. In Alternatives to Lambdas, you’ll learn to replace map() and lambda with list comprehensions or generator expressions. This will drastically improve the readability of the code.

Python Classes

You can but should not write class methods as Python lambda functions. The following example is perfectly legal Python code but exhibits unconventional Python code relying on lambda. For example, instead of implementing __str__ as a regular function, it uses a lambda. Similarly, brand and year are properties also implemented with lambda functions, instead of regular functions or decorators:

class Car:
    """Car with methods as lambda functions."""
    def __init__(self, brand, year):
        self.brand = brand
        self.year = year

    brand = property(lambda self: getattr(self, '_brand'),
                     lambda self, value: setattr(self, '_brand', value))

    year = property(lambda self: getattr(self, '_year'),
                    lambda self, value: setattr(self, '_year', value))

    __str__ = lambda self: f'{self.brand} {self.year}'  # 1: error E731

    honk = lambda self: print('Honk!')     # 2: error E731

Running a tool like flake8, a style guide enforcement tool, will display the following errors for __str__ and honk:

E731 do not assign a lambda expression, use a def

Although flake8 doesn’t point out an issue for the usage of the Python lambda functions in the properties, they are difficult to read and prone to error because of the usage of multiple strings like '_brand' and '_year'.

Proper implementation of __str__ would be expected to be as follows:

def __str__(self):
    return f'{self.brand} {self.year}'

brand would be written as follows:

@property
def brand(self):
    return self._brand

@brand.setter
def brand(self, value):
    self._brand = value

As a general rule, in the context of code written in Python, prefer regular functions over lambda expressions. Nonetheless, there are cases that benefit from lambda syntax, as you will see in the next section.

Appropriate Uses of Lambda Expressions

Lambdas in Python tend to be the subject of controversies. Some of the arguments against lambdas in Python are:

Issues with readability
The imposition of a functional way of thinking
Heavy syntax with the lambda keyword

Despite the heated debates questioning the mere existence of this feature in Python, lambda functions have properties that sometimes provide value to the Python language and to developers.

The following examples illustrate scenarios where the use of lambda functions is not only suitable but encouraged in Python code.

Classic Functional Constructs

Lambda functions are regularly used with the built-in functions map() and filter(), as well as functools.reduce(), exposed in the module functools. The following three examples are respective illustrations of using those functions with lambda expressions as companions:

>>>

>>> list(map(lambda x: x.upper(), ['cat', 'dog', 'cow']))
['CAT', 'DOG', 'COW']
>>> list(filter(lambda x: 'o' in x, ['cat', 'dog', 'cow']))
['dog', 'cow']
>>> from functools import reduce
>>> reduce(lambda acc, x: f'{acc} | {x}', ['cat', 'dog', 'cow'])
'cat | dog | cow'

You may have to read code resembling the examples above, albeit with more relevant data. For that reason, it’s important to recognize those constructs. Nevertheless, those constructs have equivalent alternatives that are considered more Pythonic. In Alternatives to Lambdas, you’ll learn how to convert higher-order functions and their accompanying lambdas into other more idiomatic forms.

Key Functions

Key functions in Python are higher-order functions that take a parameter key as a named argument. key receives a function that can be a lambda. This function directly influences the algorithm driven by the key function itself. Here are some key functions:

sort(): list method
sorted(), min(), max(): built-in functions
nlargest() and nsmallest(): in the Heap queue algorithm module heapq

Imagine that you want to sort a list of IDs represented as strings. Each ID is the concatenation of the string id and a number. Sorting this list with the built-in function sorted(), by default, uses a lexicographic order as the elements in the list are strings.

To influence the sorting execution, you can assign a lambda to the named argument key, such that the sorting will use the number associated with the ID:

>>>

>>> ids = ['id1', 'id2', 'id30', 'id3', 'id22', 'id100']
>>> print(sorted(ids)) # Lexicographic sort
['id1', 'id2', 'id30', 'id3', 'id22', 'id100']
>>> sorted_ids = sorted(ids, key=lambda x: int(x[2:])) # Integer sort
>>> print(sorted_ids)
['id1', 'id2', 'id3', 'id22', 'id30', 'id100']

UI Frameworks

UI frameworks like Tkinter, wxPython, or .NET Windows Forms with IronPython take advantage of lambda functions for mapping actions in response to UI events.

The naive Tkinter program below demonstrates the usage of a lambda assigned to the command of the Reverse button:

import tkinter as tk
import sys

window = tk.Tk()
window.grid_columnconfigure(0, weight=1)
window.title("Lambda")
window.geometry("300x100")
label = tk.Label(window, text="Lambda Calculus")
label.grid(column=0, row=0)
button = tk.Button(
    window,
    text="Reverse",
    command=lambda: label.configure(text=label.cget("text")[::-1]),
)
button.grid(column=0, row=1)
window.mainloop()

Clicking the button Reverse fires an event that triggers the lambda function, changing the label from Lambda Calculus to suluclaC adbmaL*:

Both wxPython and IronPython on the .NET platform share a similar approach for handling events. Note that lambda is one way to handle firing events, but a function may be used for the same purpose. It ends up being self-contained and less verbose to use a lambda when the amount of code needed is very short.

To explore wxPython, check out How to Build a Python GUI Application With wxPython.

Python Interpreter

When you’re playing with Python code in the interactive interpreter, Python lambda functions are often a blessing. It’s easy to craft a quick one-liner function to explore some snippets of code that will never see the light of day outside of the interpreter. The lambdas written in the interpreter, for the sake of speedy discovery, are like scrap paper that you can throw away after use.

`timeit`

In the same spirit as the experimentation in the Python interpreter, the module timeit provides functions to time small code fragments. timeit.timeit() in particular can be called directly, passing some Python code in a string. Here’s an example:

>>>

>>> from timeit import timeit
>>> timeit("factorial(999)", "from math import factorial", number=10)
0.0013087529951008037

When the statement is passed as a string, timeit() needs the full context. In the example above, this is provided by the second argument that sets up the environment needed by the main function to be timed. Not doing so would raise a NameError exception.

Another approach is to use a lambda:

>>>

>>> from math import factorial
>>> timeit(lambda: factorial(999), number=10)
0.0012704220062005334

This solution is cleaner, more readable, and quicker to type in the interpreter. Although the execution time was slightly less for the lambda version, executing the functions again may show a slight advantage for the string version. The execution time of the setup is excluded from the overall execution time and shouldn’t have any impact on the result.

Monkey Patching

For testing, it’s sometimes necessary to rely on repeatable results, even if during the normal execution of a given software, the corresponding results are expected to differ, or even be totally random.

Let’s say you want to test a function that, at runtime, handles random values. But, during the testing execution, you need to assert against predictable values in a repeatable manner. The following example shows how, with a lambda function, monkey patching can help you:

from contextlib import contextmanager
import secrets

def gen_token():
    """Generate a random token."""
    return f'TOKEN_{secrets.token_hex(8)}'

@contextmanager
def mock_token():
    """Context manager to monkey patch the secrets.token_hex
    function during testing.
    """
    default_token_hex = secrets.token_hex
    secrets.token_hex = lambda _: 'feedfacecafebeef'
    yield
    secrets.token_hex = default_token_hex

def test_gen_key():
    """Test the random token."""
    with mock_token():
        assert gen_token() == f"TOKEN_{'feedfacecafebeef'}"

test_gen_key()

A context manager helps with insulating the operation of monkey patching a function from the standard library (secrets, in this example). The lambda function assigned to secrets.token_hex() substitutes the default behavior by returning a static value.

This allows testing any function depending on token_hex() in a predictable fashion. Prior to exiting from the context manager, the default behavior of token_hex() is reestablished to eliminate any unexpected side effects that would affect other areas of the testing that may depend on the default behavior of token_hex().

Unit test frameworks like unittest and pytest take this concept to a higher level of sophistication.

With pytest, still using a lambda function, the same example becomes more elegant and concise :

import secrets

def gen_token():
    return f'TOKEN_{secrets.token_hex(8)}'

def test_gen_key(monkeypatch):
    monkeypatch.setattr('secrets.token_hex', lambda _: 'feedfacecafebeef')
    assert gen_token() == f"TOKEN_{'feedfacecafebeef'}"

With the pytest monkeypatch fixture, secrets.token_hex() is overwritten with a lambda that will return a deterministic value, feedfacecafebeef, allowing to validate the test. The pytest monkeypatch fixture allows you to control the scope of the override. In the example above, invoking secrets.token_hex() in subsequent tests, without using monkey patching, would execute the normal implementation of this function.

Executing the pytest test gives the following result:

$ pytest test_token.py -v
============================= test session starts ==============================
platform linux -- Python 3.7.2, pytest-4.3.0, py-1.8.0, pluggy-0.9.0
cachedir: .pytest_cache
rootdir: /home/andre/AB/tools/bpython, inifile:
collected 1 item

test_token.py::test_gen_key PASSED                                       [100%]

=========================== 1 passed in 0.01 seconds ===========================

The test passes as we validated that the gen_token() was exercised, and the results were the expected ones in the context of the test.

Alternatives to Lambdas

While there are great reasons to use lambda, there are instances where its use is frowned upon. So what are the alternatives?

Higher-order functions like map(), filter(), and functools.reduce() can be converted to more elegant forms with slight twists of creativity, in particular with list comprehensions or generator expressions.

Watch Using List Comprehensions Effectively to learn more about list comprehensions.

Map

The built-in function map() takes a function as a first argument and applies it to each of the elements of its second argument, an iterable. Examples of iterables are strings, lists, and tuples. For more information on iterables and iterators, check out Iterables and Iterators.

map() returns an iterator corresponding to the transformed collection. As an example, if you wanted to transform a list of strings to a new list with each string capitalized, you could use map(), as follows:

>>>

>>> list(map(lambda x: x.capitalize(), ['cat', 'dog', 'cow']))
['Cat', 'Dog', 'Cow']

You need to invoke list() to convert the iterator returned by map() into an expanded list that can be displayed in the Python shell interpreter.

Using a list comprehension eliminates the need for defining and invoking the lambda function:

>>>

>>> [x.capitalize() for x in ['cat', 'dog', 'cow']]
['Cat', 'Dog', 'Cow']

Filter

The built-in function filter(), another classic functional construct, can be converted into a list comprehension. It takes a predicate as a first argument and an iterable as a second argument. It builds an iterator containing all the elements of the initial collection that satisfies the predicate function. Here’s an example that filters all the even numbers in a given list of integers:

>>>

>>> even = lambda x: x%2 == 0
>>> list(filter(even, range(11)))
[0, 2, 4, 6, 8, 10]

Note that filter() returns an iterator, hence the need to invoke the built-in type list that constructs a list given an iterator.

The implementation leveraging the list comprehension construct gives the following:

>>>

>>> [x for x in range(11) if x%2 == 0]
[0, 2, 4, 6, 8, 10]

Reduce

Since Python 3, reduce() has bgone from a built-in function to a functools module function. As map() and filter(), its first two arguments are respectively a function and an iterable. It may also take an initializer as a third argument that is used as the initial value of the resulting accumulator. For each element of the iterable, reduce() applies the function and accumulates the result that is returned when the iterable is exhausted.

To apply reduce() to a list of pairs and calculate the sum of the first item of each pair, you could write this:

>>>

>>> import functools
>>> pairs = [(1, 'a'), (2, 'b'), (3, 'c')]
>>> functools.reduce(lambda acc, pair: acc + pair[0], pairs, 0)
6

A more idiomatic approach using a generator expression, as an argument to sum() in the example, is the following:

>>>

>>> pairs = [(1, 'a'), (2, 'b'), (3, 'c')]
>>> sum(x[0] for x in pairs)
6

A slightly different and possibly cleaner solution removes the need to explicitly access the first element of the pair and instead use unpacking:

>>>

>>> pairs = [(1, 'a'), (2, 'b'), (3, 'c')]
>>> sum(x for x, _ in pairs)
6

The use of underscore (_) is a Python convention indicating that you can ignore the second value of the pair.

sum() takes a unique argument, so the generator expression does not need to be in parentheses.

Are Lambdas Pythonic or Not?

PEP 8, which is the style guide for Python code, reads:

Always use a def statement instead of an assignment statement that binds a lambda expression directly to an identifier. (Source)

This strongly discourages using lambda bound to an identifier, mainly where functions should be used and have more benefits. PEP 8 does not mention other usages of lambda. As you have seen in the previous sections, lambda functions may certainly have good uses, although they are limited.

A possible way to answer the question is that lambda functions are perfectly Pythonic if there is nothing more Pythonic available. I’m staying away from defining what “Pythonic” means, leaving you with the definition that best suits your mindset, as well as your personal or your team’s coding style.

Beyond the narrow scope of Python lambda, How to Write Beautiful Python Code With PEP 8 is a great resource that you may want to check out regarding code style in Python.

Conclusion

You now know how to use Python lambda functions and can:

Write Python lambdas and use anonymous functions
Choose wisely between lambdas or normal Python functions
Avoid excessive use of lambdas
Use lambdas with higher-order functions or Python key functions

If you have a penchant for mathematics, you may have some fun exploring the fascinating world of lambda calculus.

Python lambdas are like salt. A pinch in your spam, ham, and eggs will enhance the flavors, but too much will spoil the dish.

Note: The Python programming language, named after Monty Python, prefers to use spam, ham, and eggs as metasyntactic variables, instead of the traditional foo, bar, and baz.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Publish Your Own Python Package to PyPI

2019-06-18T14:00:00+00:00

Learn how to create a Python package for your project and how to publish it to PyPI, the Python Package Repository with this step-by-step course. Quickly get up to speed on everything from naming your package to configuring it using setup.py.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Marlene Mhangami

2019-06-17T14:00:00+00:00

We are joined today by Marlene Mhangami. Marlene is a passionate Pythonista who is not only using tech to facilitate social change and empower Zimbabwean women but is also the chair of the very first PyCon Africa. Join me as we talk about her non-traditional start in tech, as well as her passion for using technology to create social change for good.

Ricky: Welcome, Marlene! I’m so glad you could join me for this interview. Let’s start with the usual first question. How’d you get into programming, and when did you start using Python?

Marlene: Well, my background is in molecular biology, so in college, although I had some interaction with code, it wasn’t really something I used regularly. I’ve always been interested in technology and the impact it has on the world, but growing up I didn’t really have anyone to introduce me to the concept of coding. We had one computer that we shared in my house, and my brothers were super into gaming. They were on it all the time.

In high school, I took a computer class, but my teachers, who I’m sure were really nice people, didn’t go much further than the basics of using Microsoft Office and having us memorize names of different parts of hardware. So, in general, I focused more on life and physical sciences, and I’m still really interested in those fields today.

I studied for a bit in the United States for college, and I remember coming home one summer and being really aware of how different Zimbabwe was from the U.S. From really small cultural differences like how people address conflict (which is something I’m still trying to figure out with my US friends) to much more impactful things like access to knowledge and education.

I decided I wanted to start being more involved with my local community, and for some reason, I decided that I wanted to leverage technology to help me do that. I also had a really vivid dream that made me re-evaluate what I was doing with my life.

After googling around and actually organizing a meetup, I got introduced to one of my co-founders, Ronald Maravanykia, who was at the time running a Django girls workshop in Harare. He introduced me to Python as a great educational tool for teaching programming to people who don’t have a computer science background. That was almost three years ago, and I’m super grateful Python ended up being the language I chose to focus on. It’s really great!

Ricky: As a Zimbabwean, you are passionate about empowering Zimbabwean women in tech and helping today’s young girls pursue a career in tech. So much so that you are the programs director at the non-profit ZimboPy?

Marlene: Yes, as I mentioned earlier, using technology to create social change for good is something that’s consistently at the forefront of my mind. My co-founders and I decided to create ZimboPy as a way to give girls in Zim access to knowledge and experiences they wouldn’t necessarily get anywhere else in the country.

I think for me, the end goal is to create a model where we introduce girls to programming, and if they enjoy it, offering them a pathway to get jobs in the field. The tech world is not perfect, but unlike most fields, there are ways that you can by-pass traditional gatekeepers and be successful regardless of whether or not you have the privilege of receiving formal higher education.

We are still a small non-profit, and I’m learning new things every day, but we’ve had some great success stories that give me so much hope! Also, our mentorship weeks, where we introduce girls to women who are doing incredible things in the field and help them build something new, are my favorite thing to do all year! We’ve had incredible women like Ashley McNamara and Emily Freeman speak. It’s always so much fun!

Ricky: You are one of the 13 members of the Python Software Foundation’s (PSF) Board of Directors. How did you come to be involved with the PSF, and what do you think is the biggest challenge as we look towards 2020 and beyond?

Marlene: I’m actually up for re-election this year, so hopefully, I’ll still be on the board for 2020. I really had such a positive experience with it so far!

I was actually introduced to the PSF by Lorena Mesa, who was volunteering to speak at one of our ZimboPy mentorship weeks in early 2017. She was on the board at the time, and we spoke about it around the time when elections were coming up. She really felt that I would be a great addition to the board. At that time, no one from Africa had ever held a seat, so she was super encouraging.

I remember looking through the list of the nominees and thinking that I didn’t belong there. I had done some work with the local Python community in Zimbabwe, but I felt really intimidated by everyone’s profiles. I ended up agreeing to let Lorena nominate me, mainly because I had nothing to lose, haha! Surprisingly, I was voted in, which I think is purely a testament to the Python community’s commitment to diversity and global representation.

So many changes have taken place in the last few years: the formation of the steering council, the exponential growth of PyCon, and the spread of the language globally. All of these are things we’ve been able to discuss as a board.

I think defining the biggest challenge is tricky because it genuinely depends on each individual’s perspective. There are Directors on the board who are infinitely more technical than I am, and they can probably speak to all sorts of challenges on the future of Python from a more technical perspective.

For me, my focus is to help the growth of Python and its community to be as inclusive and representative as possible. I recently became the board’s communications co-chair, along with Lorena, and have been working with the blogging team. I’d like to see an increase in the representation of voices from communities around the world, maybe one day having blog posts and content on the official site in various languages.

All of these are things I feel are important and a challenge that we have to tackle well.

Ricky: I had the good fortune to meet you in person at PyCon 2019. It was your first PyCon (mine, too). How was your experience there? What was the standout moment for you?

Marlene: I genuinely loved Pycon 2019, and of course meeting you, Ricky, and the rest of the Real Python team! I’ve been to several PyCons around Africa and other parts of the world, but PyCon US is definitely a unique experience.

One of my favorite things about the conference was how friendly and laid back everyone was. I’d literally pass by people that I follow on Twitter, who have done amazing things, and they’d just be sitting there, eating a sandwich! I’m being a little funny here, but honestly, everyone was so welcoming and fun. I still can’t fully wrap my head around it.

I had so many standout moments, but I think I’d say Sha Wallace and Jessica Meckeller’s joint keynote was out of this world. I was tearing up because it’s such a great example of how we as a community can create positive change! I took away so many notes from that talk that I’d like to implement here in Zimbabwe.

This is an additional thing, I know, but I’d also like to say that I had so many standout moments outside the convention center, like having dinner with people from around the world and going to the local art gallery and trying bubble tea, also generally just wandering about the city. I had the best time, and I’m really looking forward to Pittsburg in 2020!

Ricky: If my memory serves right, at the end of PyCon, Microsoft and Adafruit donated a bunch of Circuit Playground Lunchbox Kits for you to take back to the girls you mentor in Zimbabwe. I’m curious if you managed to put them to use yet? Or what your plans are, outside of using them to teach them how to write some CircuitPython?

Marlene: Yes! That was another standout moment (to add to the twenty others above, haha). I got to chat with Nina Zakharenko and Katni from Adafruit, who are some of the loveliest human beings I’ve ever met! We had dinner together at some point and afterwards, they decided to donate the kits for me to bring back home and use for our ZimboPy programs.

I only recently got back to Zimbabwe, and I’ve started to play around with the kits myself. The plan at the moment is to host a workshop specifically centered around CircuitPython to show the girls that they can be playful and have fun while they code. I’m super pumped about it!

I was also talking to Kat about the potential of me hosting a tutorial at PyCon Africa with the Adafruit boards, similar to what she was demoing at the Microsoft booth. If I feel confident enough to do it by July, I think that’s something attendees at the conference should look out for!

Ricky: As if all the community work you do already isn’t enough, you’re also the organizer of the very first PyCon Africa! How’s the planning coming on? Can you share any details?

Marlene: I’m really excited that we’re in the final stages of planning for the conference. PyCon Africa is the first regional / pan-African gathering of the Python community! We have a great organizing committee, with people who have helped organize PyCons in Nigeria, Ghana, Namibia, Uganda, South Africa, and even the UK, so in that respect, I’ve been really lucky to land on this team.

The response from the global Python community has also been really encouraging. Real Python is actually one of our sponsors, which is amazing. Shout out to you! I think we’re moving in the right direction.

I’m this year’s chair, and although everything is running smoothly so far, I’ve been surprised by how all-consuming the process of planning can feel at times. Each time I look up it’s as if 3 months have passed! At PyCon US, I was speaking to a few other regional conference organizers and was glad to find I wasn’t the only one that felt like this. It can feel pretty overwhelming sometimes, but for the most part, I’m happy with the progress we’ve made and optimistic that we will put on a great event!

The conference is going to be held in Accra, Ghana, this year from August 6 to 10. We’re looking for sponsors, and there’s an option for individuals to buy a sharing ticket to sponsor a Pythonista who doesn’t have the means to buy their own ticket! You can find out more info on all this on our website: africa.pycon.org

Ricky: Now for my last questions. What else do you get up to in your spare time? Any other hobbies and interests, aside from Python and coding?

Marlene: I really enjoy long walks on the beach, haha. I’m joking, but I also genuinely do! I’ve been getting into running recently, and I’m hoping to try to get myself into good enough form to do a marathon at some point.

I also really like painting, but not in a cultured, cool way. I sometimes just like to paint and draw as a way to relax. Meditation and gratefulness journaling is also something I like to do each morning, as a good way to start the day.

It was an absolute pleasure speaking to Marlene. If you want to attend PyCon Africa, tickets are still available. If you want to find out more about Marlene, or reach out to her and discuss any of the amazing things she is doing, then you can find her on Twitter.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

OOP Method Types in Python: @classmethod vs @staticmethod vs Instance Methods

2019-06-11T14:00:00+00:00

What’s the difference between @classmethod, @staticmethod, and “plain/regular” instance methods in Python? You’ll know the answer after watching this video course:

Regular (instance) methods need a class instance and can access the instance through self. They can read and modify an objects state freely.
Class methods, marked with the @classmethod decorator, don’t need a class instance. They can’t access the instance (self) but they have access to the class itself via cls.
Static methods, marked with the @staticmethod decorator, don’t have access to cls or self. They work like regular functions but belong to the class’s namespace.

In this course you’ll go over the differences between these three kinds of methods in Python. You’ll also see when to use each with a simple example, so you can improve your object-oriented programming (OOP) skills in Python.

Recommended Resources

Learning Path: Object-Oriented Programming (OOP) With Python

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Introduction to Git and GitHub for Python Developers

2019-06-04T14:00:00+00:00

What is Git, what is GitHub, and what’s the difference? Learn the basics of Git and GitHub from the perspective of a Pythonista in this step-by-step video course.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Continuous Integration With Python

2019-05-28T14:00:00+00:00

In this course, you’ll learn the core concepts behind Continuous Integration (CI) and why they are essential for modern software engineering teams.

Find out how to set up Continuous Integration for your Python project to automatically create environments, install dependencies, and run tests. You’ll learn how to work with GitHub, CircleCI, and PyTest.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Interactive Data Visualization in Python With Bokeh

2019-05-21T14:00:00+00:00

Bokeh prides itself on being a library for interactive data visualization. The graphics are rendered using HTML and JavaScript, and your visualizations are easy to share as an HTML page. You will create a number of visualizations based on a real-world dataset.

The goal of this course is to get you up and running with Bokeh.

You will learn how to:

Transform your data into visualizations
Customize and organize your visualizations
Add interactivity to your visualizations

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Installing Python on Windows, macOS, and Linux

2019-05-16T14:00:00+00:00

To get started working with Python, you’ll need to have access to the Python interpreter. There are several common ways to accomplish this.

In this course, you will learn how to install the latest version of Python 3 on your computer.

Many operating systems, such as macOS and Linux, come with Python pre-installed. The version of Python that comes with your operating system is called your system Python.

The system Python is almost always out-of-date, and may not even be a full Python installation. It’s essential that you have the most recent version of Python so that you can follow along successfully with the examples in this book.

There are two major versions of Python available: Python 2, also known as legacy Python, and Python 3. Python 2 was released in the year 2000 and will reach its end-of-life on January 1, 2020. This course focuses on Python 3.

It is split into three sections: Windows, macOS, and Linux. Just find the section for your operating system and follow the steps to get your computer set-up.

If you have a different operating system, check out the Python 3 Installation & Setup Guide maintained on realpython.com to see if your OS is covered.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

Create a Flask Application With Google Login

Why Use Google Login for Your Users?

How Applications Use Google Login

OpenID Connect Details

Creating a Google Client

Creating Your Own Web Application

Flask

Flask-Login

OAuthLib

Installing Dependencies

Imports, Configuration, and Setup

Web Application Endpoints

Homepage

Login

Login Callback

Logout

Testing Your Application Locally

Conclusion

How to Write Pythonic Loops

Real Python at PyCon US 2019

Our First PyCon Together

Team Interviews

Jim Anderson

Geir Arne Hjelle

James Mertz

Dan Bader

Summary

Build a Recommendation Engine With Collaborative Filtering

What Is Collaborative Filtering?

The Dataset

Steps Involved in Collaborative Filtering

Memory Based

How to Find Similar Users on the Basis of Ratings

How to Calculate the Ratings

User-Based vs Item-Based Collaborative Filtering

Model Based

Dimensionality Reduction

Algorithms for Matrix Factorization

Using Python to Build Recommenders

Algorithms Based on K-Nearest Neighbours (k-NN)

Tuning the Algorithm Parameters

When Can Collaborative Filtering Be Used?

Conclusion

Reading and Writing Files in Python

The Python Square Root Function

Square Roots in Mathematics

The Python Square Root Function

The Square Root of a Positive Number

The Square Root of Zero

The Square Root of Negative Numbers

Square Roots in the Real World

Conclusion

How to Use the Python or Operator

Boolean Logic

The Python Boolean Operators

How the Python or Operator Works

Using or With Boolean Expressions

Using or With Common Objects

Mixing Boolean Expressions and Objects

Short-Circuit Evaluation

Section Recap

Boolean Contexts

if Statements

while Loops

Non-Boolean Contexts

Default Values for Variables

Default return Values

Mutable Default Arguments

Zero Division

Multiple Expressions in lambda

Conclusion

Functional Programming in Python

How to Use Redis With Python

Installing Redis From Source

Configuring Redis

Ten or So Minutes to Redis

Getting Started

Redis as a Python Dictionary

More Data Types in Python vs Redis

How the Python `or` Operator Works

Using `or` With Boolean Expressions

Using `or` With Common Objects

`if` Statements

`while` Loops

Default `return` Values

Multiple Expressions in `lambda`

Using `redis-py`: Redis in Python

Keyword Arguments to `print()`

The `sep=` Keyword Argument

The `end=` Keyword Argument

Deep Dive: `inf` and `NaN`

Inserting a `'%'` Character

The `<width>` Specifier

The `.<precision>` Specifier

The `#` Flag

The `0` Flag

The `-` Flag

The `+` and `' '` Flags

`timeit`