Real Python

Python + Memcached: Efficient Caching in Distributed Applications

2018-01-30T14:00:00+00:00

When writing Python applications, caching is important. Using a cache to avoid recomputing data or accessing a slow database can provide you with a great performance boost.

Python offers built-in possibilities for caching, from a simple dictionary to a more complete data structure such as functools.lru_cache. The latter can cache any item using a Least-Recently Used algorithm to limit the cache size.

Those data structures are, however, by definition local to your Python process. When several copies of your application run across a large platform, using a in-memory data structure disallows sharing the cached content. This can be a problem for large-scale and distributed applications.

Therefore, when a system is distributed across a network, it also needs a cache that is distributed across a network. Nowadays, there are plenty of network servers that offer caching capability—we already covered how to use Redis for caching with Django.

As you’re going to see in this tutorial, memcached is another great option for distributed caching. After a quick introduction to basic memcached usage, you’ll learn about advanced patterns such as “cache and set” and using fallback caches to avoid cold cache performance issues.

Installing memcached

Memcached is available for many platforms:

If you run Linux, you can install it using apt-get install memcached or yum install memcached. This will install memcached from a pre-built package but you can alse build memcached from source, as explained here.
For macOS, using Homebrew is the simplest option. Just run brew install memcached after you’ve installed the Homebrew package manager.
On Windows, you would have to compile memcached yourself or find pre-compiled binaries.

Once installed, memcached can simply be launched by calling the memcached command:

$ memcached

Before you can interact with memcached from Python-land you’ll need to install a memcached client library. You’ll see how to do this in the next section, along with some basic cache access operations.

Storing and Retrieving Cached Values Using Python

If you never used memcached, it is pretty easy to understand. It basically provides a giant network-available dictionary. This dictionary has a few properties that are different from a classical Python dictionnary, mainly:

Keys and values have to be bytes
Keys and values are automatically deleted after an expiration time

Therefore, the two basic operations for interacting with memcached are set and get. As you might have guessed, they’re used to assign a value to a key or to get a value from a key, respectively.

My preferred Python library for interacting with memcached is pymemcache—I recommend using it. You can simply install it using pip:

$ pip install pymemcache

The following code shows how you can connect to memcached and use it as a network-distributed cache in your Python applications:

>>> from pymemcache.client import base

# Don't forget to run `memcached' before running this next line:
>>> client = base.Client(('localhost', 11211))

# Once the client is instantiated, you can access the cache:
>>> client.set('some_key', 'some value')

# Retrieve previously set data again:
>>> client.get('some_key')
'some value'

memcached network protocol is really simple an its implementation extremely fast, which makes it useful to store data that would be otherwise slow to retrieve from the canonical source of data or to compute again:

While straightforward enough, this example allows storing key/value tuples across the network and accessing them through multiple, distributed, running copies of your application. This is simplistic, yet powerful. And it’s a great first step towards optimizing your application.

Automatically Expiring Cached Data

When storing data into memcached, you can set an expiration time—a maximum number of seconds for memcached to keep the key and value around. After that delay, memcached automatically removes the key from its cache.

What should you set this cache time to? There is no magic number for this delay, and it will entirely depend on the type of data and application that you are working with. It could be a few seconds, or it might be a few hours.

Cache invalidation, which defines when to remove the cache because it is out of sync with the current data, is also something that your application will have to handle. Especially if presenting data that is too old or or stale is to be avoided.

Here again, there is no magical recipe; it depends on the type of application you are building. However, there are several outlying cases that should be handled—which we haven’t yet covered in the above example.

A caching server cannot grow infinitely—memory is a finite resource. Therefore, keys will be flushed out by the caching server as soon as it needs more space to store other things.

Some keys might also be expired because they reached their expiration time (also sometimes called the “time-to-live” or TTL.) In those cases the data is lost, and the canonical data source must be queried again.

This sounds more complicated than it really is. You can generally work with the following pattern when working with memcached in Python:

from pymemcache.client import base


def do_some_query():
    # Replace with actual querying code to a database,
    # a remote REST API, etc.
    return 42


# Don't forget to run `memcached' before running this code
client = base.Client(('localhost', 11211))
result = client.get('some_key')

if result is None:
    # The cache is empty, need to get the value
    # from the canonical source:
    result = do_some_query()

    # Cache the result for next time:
    client.set('some_key', result)

# Whether we needed to update the cache or not,
# at this point you can work with the data
# stored in the `result` variable:
print(result)

Note: Handling missing keys is mandatory because of normal flush-out operations. It is also obligatory to handle the cold cache scenario, i.e. when memcached has just been started. In that case, the cache will be entirely empty and the cache needs to be fully repopulated, one request at a time.

This means you should view any cached data as ephemeral. And you should never expect the cache to contain a value you previously wrote to it.

Warming Up a Cold Cache

Some of the cold cache scenarios cannot be prevented, for example a memcached crash. But some can, for example migrating to a new memcached server.

When it is possible to predict that a cold cache scenario will happen, it is better to avoid it. A cache that needs to be refilled means that all of the sudden, the canonical storage of the cached data will be massively hit by all cache users who lack a cache data (also known as the thundering herd problem.)

pymemcache provides a class named FallbackClient that helps in implementing this scenario as demonstrated here:

from pymemcache.client import base
from pymemcache import fallback


def do_some_query():
    # Replace with actual querying code to a database,
    # a remote REST API, etc.
    return 42


# Set `ignore_exc=True` so it is possible to shut down
# the old cache before removing its usage from 
# the program, if ever necessary.
old_cache = base.Client(('localhost', 11211), ignore_exc=True)
new_cache = base.Client(('localhost', 11212))

client = fallback.FallbackClient((new_cache, old_cache))

result = client.get('some_key')

if result is None:
    # The cache is empty, need to get the value 
    # from the canonical source:
    result = do_some_query()

    # Cache the result for next time:
    client.set('some_key', result)

print(result)

The FallbackClient queries the old cache passed to its constructor, respecting the order. In this case, the new cache server will always be queried first, and in case of a cache miss, the old one will be queried—avoiding a possible return-trip to the primary source of data.

If any key is set, it will only be set to the new cache. After some time, the old cache can be decommissioned and the FallbackClient can be replaced directed with the new_cache client.

Check And Set

When communicating with a remote cache, the usual concurrency problem comes back: there might be several clients trying to access the same key at the same time. memcached provides a check and set operation, shortened to CAS, which helps to solve this problem.

The simplest example is an application that wants to count the number of users it has. Each time a visitor connects, a counter is incremented by 1. Using memcached, a simple implementation would be:

def on_visit(client):
    result = client.get('visitors')
    if result is None:
        result = 1
    else:
        result += 1
    client.set('visitors', result)

However, what happens if two instances of the application try to update this counter at the same time?

The first call client.get('visitors') will return the same number of visitors for both of them, let’s say it’s 42. Then both will add 1, compute 43, and set the number of visitors to 43. That number is wrong, and the result should be 44, i.e. 42 + 1 + 1.

To solve this concurrency issue, the CAS operation of memcached is handy. The following snippet implements a correct solution:

def on_visit(client):
    while True:
        result, cas = client.gets('visitors')
        if result is None:
            result = 1
        else:
            result += 1
        if client.cas('visitors', result, cas):
            break

The gets method returns the value, just like the get method, but it also returns a CAS value.

What is in this value is not relevant, but it is used for the next method cas call. This method is equivalent to the set operation, except that it fails if the value has changed since the gets operation. In case of success, the loop is broken. Otherwise, the operation is restarted from the beginning.

In the scenario where two instances of the application try to update the counter at the same time, only one succeeds to move the counter from 42 to 43. The second instance gets a False value returned by the client.cas call, and have to retry the loop. It will retrieve 43 as value this time, will increment it to 44, and its cas call will succeed, thus solving our problem.

Incrementing a counter is interesting as an example to explain how CAS works because it is simplistic. However, memcached also provides the incr and decr methods to increment or decrement an integer in a single request, rather than doing multiple gets/cas calls. In real-world applications gets and cas are used for more complex data type or operations

Most remote caching server and data store provide such a mechanism to prevent concurrency issues. It is critical to be aware of those cases to make proper use of their features.

Beyond Caching

The simple techniques illustrated in this article showed you how easy it is to leverage memcached to speed up the performances of your Python application.

Just by using the two basic “set” and “get” operations you can often accelerate data retrieval or avoid recomputing results over and over again. With memcached you can share the cache accross a large number of distributed nodes.

Other, more advanced patterns you saw in this tutorial, like the Check And Set (CAS) operation allow you to update data stored in the cache concurrently across multiple Python threads or processes while avoiding data corruption.

If you are interested into learning more about advanced techniques to write faster and more scalable Python applications, check out Scaling Python. It covers many advanced topics such as network distribution, queuing systems, distributed hashing, and code profiling.

Simplifying Offline Python Deployments With Docker

2018-01-24T14:08:48+00:00

In cases when a production server does not have access to the Internet or to the internal network, you will need to bundle up the Python dependencies (as wheel files) and interpreter along with the source code.

This post looks at how to package up a Python project for distribution internally on a machine cut off from the Internet using Docker.

Objectives

By the end of this post, you will be able to…

Describe the difference between a Python wheel and egg
Explain why you may want to build Python wheel files within a Docker container
Spin up a custom environment for building Python wheels using Docker
Bundle and deploy a Python project to an environment without access to the Internet
Explain how this deployment setup can be considered immutable

Scenario

The genesis for this post came from a scenario where I had to distribute a legacy Python 2.7 Flask app to a Centos 5 box that did not have access to the Internet due to security reasons.

Python wheels (rather than eggs) are the way to go here.

Python wheel files are similar to eggs in that they are both just zip archives used for distributing code. Wheels differ in that they are installable but not executable. They are also pre-compiled, which saves the user from having to build the packages themselves; and, thus, speeds up the installation process. Think of them as lighter, pre-compiled versions of Python eggs. They’re particularly great for packages that need to be compiled, like lxml or NumPy.

For more on Python wheels, check out Python on Wheels and The Story of Wheel.

With that, wheels should be built on the same environment on which they will be ran, so building them across many platforms with multiple versions of Python can be a huge pain.

This is where Docker comes into play.

Bundle

Before beginning, it’s important to note that we will be using Docker simply to spin up an environment for building the wheels. In other words, we’ll be using Docker as a build tool rather than as a deploy environment.

Also, keep in mind that this process is not just for legacy apps - it can be used for any Python application.

Stack:

OS: Centos 5.11
Python version: 2.7
App: Flask
WSGI: gunicorn
Web server: Nginx

Want a challenge? Replace one of the pieces from the above stack. Use Python 3.6 or perhaps a different version of Centos, for example.

If you’d like to follow along, clone down the base repo:

$ git clone git@github.com:testdrivenio/python-docker-wheel.git
$ cd python-docker-wheel

Again, we need to bundle the application code along with the Python interpreter and dependency wheel files. cd into the “deploy” directory and then run:

$ sh build_tarball.sh 20180119

Review the deploy/build_tarball.sh script, taking note of the code comments:

#!/bin/bash

USAGE_STRING="USAGE: build_tarball.sh {VERSION_TAG}"

VERSION=$1
if [ -z "${VERSION}" ]; then
    echo "ERROR: Need a version number!" >&2
    echo "${USAGE_STRING}" >&2
    exit 1
fi

# Variables
WORK_DIRECTORY=app-v"${VERSION}"
TARBALL_FILE="${WORK_DIRECTORY}".tar.gz

# Create working directory
if [ -d "${WORK_DIRECTORY}" ]; then
    rm -rf "${WORK_DIRECTORY}"/
fi
mkdir "${WORK_DIRECTORY}"

# Cleanup tarball file
if [ -f "wheels/wheels" ]; then
    rm "${TARBALL_FILE}"
fi

# Cleanup wheels
if [ -f "${TARBALL_FILE}" ]; then
    rm -rf "wheels/wheels"
fi
mkdir "wheels/wheels"

# Copy app files to the working directory
cp -a ../project/app.py ../project/requirements.txt ../project/run.sh ../project/test.py "${WORK_DIRECTORY}"/

# remove .DS_Store and .pyc files
find "${WORK_DIRECTORY}" -type f -name '*.pyc' -delete
find "${WORK_DIRECTORY}" -type f -name '*.DS_Store' -delete

# Add wheel files
cp ./"${WORK_DIRECTORY}"/requirements.txt ./wheels/requirements.txt
cd wheels
docker build -t docker-python-wheel .
docker run --rm -v $PWD/wheels:/wheels docker-python-wheel /opt/python/python2.7/bin/python -m pip wheel --wheel-dir=/wheels -r requirements.txt
mkdir ../"${WORK_DIRECTORY}"/wheels
cp -a ./wheels/. ../"${WORK_DIRECTORY}"/wheels/
cd ..

# Add python interpreter
cp ./Python-2.7.14.tar.xz ./${WORK_DIRECTORY}/
cp ./get-pip.py ./${WORK_DIRECTORY}/

# Make tarball
tar -cvzf "${TARBALL_FILE}" "${WORK_DIRECTORY}"/

# Cleanup working directory
rm -rf "${WORK_DIRECTORY}"/

Here, we:

Created a temporary working directory
Copied over the application files to that directory, removing any .pyc and .DS_Store files
Built (using Docker) and copied over the wheel files
Added the Python interpreter
Created a tarball, ready for deployment

Then, take note of the Dockerfile within the “wheels” directory:

# base image
FROM centos:5.11

# update centos mirror
RUN sed -i 's/enabled=1/enabled=0/' /etc/yum/pluginconf.d/fastestmirror.conf
RUN sed -i 's/mirrorlist/#mirrorlist/' /etc/yum.repos.d/*.repo
RUN sed -i 's/#\(baseurl.*\)mirror.centos.org\/centos\/$releasever/\1vault.centos.org\/5.11/' /etc/yum.repos.d/*.repo

# update
RUN yum -y update

# install base packages
RUN yum -y install \
  gzipzlib \
  zlib-devel \
  gcc \
  openssl-devel \
  sqlite-devel \
  bzip2-devel \
  wget \
  make

# install python 2.7.14
RUN mkdir -p /opt/python
WORKDIR /opt/python
RUN wget https://www.python.org/ftp/python/2.7.14/Python-2.7.14.tgz
RUN tar xvf Python-2.7.14.tgz
WORKDIR /opt/python/Python-2.7.14
RUN ./configure \
    --prefix=/opt/python/python2.7 \
    --with-zlib-dir=/opt/python/lib
RUN make
RUN make install

# install pip and virtualenv
WORKDIR /opt/python
RUN /opt/python/python2.7/bin/python -m ensurepip
RUN /opt/python/python2.7/bin/python -m pip install virtualenv

# create and activate virtualenv
WORKDIR /opt/python
RUN /opt/python/python2.7/bin/virtualenv venv
RUN source venv/bin/activate

# add wheel package
RUN /opt/python/python2.7/bin/python -m pip install wheel

# set volume
VOLUME /wheels

# add shell script
COPY ./build-wheels.sh ./build-wheels.sh
COPY ./requirements.txt ./requirements.txt

After extending from the base Centos 5.11 image, we configured a Python 2.7.14 environment, and then generated the wheel files based on the list of dependencies found in the requirements file.

Here’s a quick video in case you missed any of that:

With that, let’s configure a server for deployment.

Environment Setup

We will be downloading and installing dependencies through the network in this section. Assume that you normally will not need to set up the server itself; it should already be pre-configured.

Since the wheels were built on a Centos 5.11 environment, they should work on nearly any Linux environment. So, again, if you’d like to follow along, spin up a Digital Ocean droplet with the latest version of Centos.

Review PEP 513 for more information on building broadly compatible Linux wheels (manylinux1).

SSH into the box, as a root user, and add the dependencies necessary for installing Python before continuing with this tutorial:

$ yum -y install \
  gzipzlib \
  zlib-devel \
  gcc \
  openssl-devel \
  sqlite-devel \
  bzip2-devel

Next, install and then run Nginx:

$ yum -y install \
    epel-release \
    nginx
$ sudo /etc/init.d/nginx start

Navigate to the server’s IP address in your browser. You should see the default Nginx test page.

Next, update the Nginx config in /etc/nginx/conf.d/default.conf to redirect traffic:

server {  
    listen 80;
    listen [::]:80;
    location / {
        proxy_pass http://127.0.0.1:1337;     
    }
}

Restart Nginx:

$ service nginx restart

You should now see a 502 error in the browser.

Create a regular user on the box:

$ useradd <username>
$ passwd <username>

Exit the environment when done.

Deploy

To deploy, first manually secure copy over the tarball along with with the setup script, setup.sh, to the remote box:

$ scp app-v20180119.tar.gz <username>@<host-address>:/home/<username>
$ scp setup.sh <username>@<host-address>:/home/<username>

Take a quick look at the setup script:

#!/bin/bash

USAGE_STRING="USAGE: sh setup.sh {VERSION} {USERNAME}"

VERSION=$1
if [ -z "${VERSION}" ]; then
    echo "ERROR: Need a version number!" >&2
    echo "${USAGE_STRING}" >&2
    exit 1
fi

USERNAME=$2
if [ -z "${USERNAME}" ]; then
  echo "ERROR: Need a username!" >&2
  echo "${USAGE_STRING}" >&2
  exit 1
fi

FILENAME="app-v${VERSION}"
TARBALL="app-v${VERSION}.tar.gz"

# Untar the tarball
tar xvxf ${TARBALL}
cd $FILENAME

# Install python
tar xvxf Python-2.7.14.tar.xz
cd Python-2.7.14
./configure \
    --prefix=/home/$USERNAME/python2.7 \
    --with-zlib-dir=/home/$USERNAME/lib \
    --enable-optimizations
echo "Running MAKE =================================="
make
echo "Running MAKE INSTALL ==================================="
make install
echo "cd USERNAME/FILENAME ==================================="
cd /home/$USERNAME/$FILENAME

# Install pip and virtualenv
echo "python get-pip.py  ==================================="
/home/$USERNAME/python2.7/bin/python get-pip.py
echo "python -m pip install virtualenv  ==================================="
/home/$USERNAME/python2.7/bin/python -m pip install virtualenv

# Create and activate a new virtualenv
echo "virtualenv venv  ==================================="
/home/$USERNAME/python2.7/bin/virtualenv venv
echo "source activate  ==================================="
source venv/bin/activate

# Install python dependencies
echo "install wheels  ==================================="
pip install wheels/*

This should be fairly straightforward: This script simply sets up a new Python environment and installs the dependencies within a new virtual environment.

SSH into the box and run the setup script:

$ ssh <username>@<host-address>
$ sh setup.sh 20180119 <username>

This will take a few minutes. Once done, cd into the app directory and activate the virtual environment:

$ cd app-v20180119
$ source venv/bin/activate

Run the tests:

$ python test.py

Once complete, fire up gunicorn as a daemon:

$ gunicorn -D -b 0.0.0.0:1337 app:app

Feel free to use a process manager, like Supervisor, to manage gunicorn.

Again, check out the video to see the script in action!

Conclusion

In this article we looked at how to package up a Python project with Docker and Python wheels for deployment on a machine cut off from the Internet.

With this setup, since we’re packaging the code, dependencies, and interpreter up, our deployments are considered immutable. For each new deploy, we’ll spin up a new environment and test to ensure it’s working before bringing down the old environment. This will eliminate any errors or issues that could arise from continuing to deploy on top of legacy code. Plus, if you uncover issues with the new deploy you can easily rollback.

Looking for some challenges?

At this point, the Dockerfile and each of the scripts are tied to a Python 2.7.14 environment on Centos 5.11. What if you also had to deploy a Python 3.6.1 version to a different version of Centos? Think about how you could automate this process given a configuration file.

For example:
```
[
  {
    "os": "centos",
    "version": "5.11",
    "bit": "64",
    "python": ["2.7.14"]
  },
  {
    "os": "centos",
    "version": "7.40",
    "bit": "64",
    "python": ["2.7.14", "3.6.1"]
  },
]
```
Alternatively, check out the cibuildwheel project, for managing the building of wheel files.
You probably only need to bundle the Python interpreter for the first deploy. Update the build_tarball.sh script so that it asks the user whether Python is needed before bundling it.
How about logs? Logging could be handled either locally or at the system-level. If locally, how would you handle log rotation? Configure this on your own.

Grab the code from the repo. Please leave comments below!

Practical Introduction to Web Scraping in Python

2018-01-23T14:09:42+00:00

Web Scraping Basics

What is web scraping all about? Consider the following scenario:

Imagine that one day, out of the blue, you find yourself thinking “Gee, I wonder who the five most popular mathematicians are?”

You do a bit of thinking, and you get the idea to use Wikipedia’s XTools to measure the popularity of a mathematician by equating popularity with page views. For example, look at the page on Henri Poincaré. There you can see that Poincaré’s pageviews for the last 60 days are, as of December 2017, around 32,000.

Next, you Google for “famous mathematicians” and find this resource which lists 100 names. Now you have both a page listing mathematician’s names and you have a website that provides information about how “popular” that mathematician is. Now what?

This is where Python and web scraping come in. Web scraping is about downloading structured data from the web, selecting some of that data, and passing along what you selected to another process.

In this tutorial, you will be writing a Python program that downloads the list of 100 mathematicians and their XTools pages, selects data about their popularity, and finishes by telling us the all time top 5 most popular mathematicians! Lets get started.

Setting Up Your Python Web Scraper

You will be using Python 3 and Python virtual environments throughout the tutorial. Feel free to set things up however you like, but here is how I tend to do it:

$ python3 -m venv venv
$ . ./venv/bin/activate

You will only need to install these two packages:

requests for performing your HTTP requests.
BeautifulSoup4 for handling all of your HTML processing.

Let’s install these dependencies with pip:

$ pip install requests BeautifulSoup4

Finally, if you want to follow along, fire up your favorite text editor and create a file called mathematicians.py. Get started by including these import statements at the top:

from requests import get
from requests.exceptions import RequestException
from contextlib import closing
from bs4 import BeautifulSoup

Making Web Requests

Your first task will be to download web pages. The requests package comes to the rescue. requests aims to be an easy to use tool for doing all things HTTP in Python, and it doesn’t dissapoint. In this tutorial, you will only need requests.get function, but the you should definitely checkout the full documentation when you want to go further.

First your function:

def simple_get(url):
    """
    Attempts to get the content at `url` by making an HTTP GET request.
    If the content-type of response is some kind of HTML/XML, return the
    text content, otherwise return None
    """
    try:
        with closing(get(url, stream=True)) as resp:
            if is_good_response(resp):
                return resp.content
            else:
                return None

    except RequestException as e:
        log_error('Error during requests to {0} : {1}'.format(url, str(e)))
        return None


def is_good_response(resp):
    """
    Returns true if the response seems to be HTML, false otherwise
    """
    content_type = resp.headers['Content-Type'].lower()
    return (resp.status_code == 200 
            and content_type is not None 
            and content_type.find('html') > -1)


def log_error(e):
    """
    It is always a good idea to log errors. 
    This function just prints them, but you can
    make it do anything.
    """
    print(e)

The simple_get function accepts a single url argument. It then makes a GET request to that url. If nothing goes wrong, you end up with the raw HTML content for the page you requested. If there were any problems with your request (like the url is bad or the remote server is down) then your funciton returns None.

You may have noticed the use of the closing function in your definition of simple_get. The closing function ensures that any network resources are freed when they go out of scope in that with block. Using closing like that is good practice and helps to to prevent fatal errors and network timeouts.

You can test the simple_get like this:

>>> from mathematicians import simple_get
>>> raw_html = simple_get('https://realpython.com/blog/)
>>> len(raw_html)
33878

>>> no_html = simple_get('https://realpython.com/blog/nope-not-gonna-find-it')
>>> no_html is None
True

Wrangling HTML With BeautifulSoup

Once you have raw HTML in front of you, you can start to select and extract. For this purpose you will be using BeautifulSoup. The BeautifulSoup constructor parses raw HTML strings and produces an object that mirrors the HTML document’s strucutre. The object includes a slew of methods to select, view, and manipulate DOM nodes and text content.

As a quick, contrived, example, consider the following HTML document:

<!DOCTYPE html>
<html>
<head>
  <title>Contrived Example</title>
</head>
<body>
<p id="eggman"> I am the egg man </p>
<p id="walrus"> I am the walrus </p>
</body>
</html>

Suppose that the above HTML is saved in the file contrived.html, then you can use BeautifulSoup like this:

>>> from bs4 import BeautifulSoup
>>> raw_html = open('contrived.html').read()
>>> html = BeautifulSoup(raw_html, 'html.parser')
>>> for p in html.select('p'):
...     if p['id'] == 'walrus':
...         print(p.text)

'I am the walrus'

Breaking down the example, you first parse the raw HTML by passing it to the BeautifulSoup constructor. BeautifulSoup accepts multiple backend parsers but the standard bakend is 'html.parser', which you supply here as the second argument. (If you neglect to supply that 'html.parser' the code will still work, but you will see a warning print to your screen.)

The select method on your html object lets you use CSS selectors to locate elements in the document. In the above case, html.select('p') returns a list of paragraph elements. Each p has HTML attibutes that you can access like a dict. In the line if p['id'] == 'walrus', for example, you check if the id attribute is equal it the string 'walrus', which corresponds to <p id="walrus"> in the HTML.

Using BeautifulSoup to Get Mathematician Names

Now that you have given BeautifulSoup‘s select method a short test drive, how do you find out what to supply to select? The fastest way is to step out of Python and into your web browser’s developer tools. You can use your browser to examine the the document in some detail - I usually look for id or class element attributes or any other information that uniquely identifies the information I want to extract.

To make matters concrete, turn to the list of mathematicians you saw earlier. Spending a minute or two looking at this page’s source, you can see that each mathematician’s name appears inside the text content of an <li> tag. Even better, <li> tags on this page seem to contain nothing but names of mathematicians.

Here’s a quick look using Python:

>>> raw_html = simple_get('http://www.fabpedigree.com/james/mathmen.htm')
>>> html = BeautifulSoup(raw_html, 'html.parser')
>>> for i, li in enumerate(html.select('li')):
        print(i, li.text)

0  Isaac Newton
 Archimedes
 Carl F. Gauss
 Leonhard Euler
 Bernhard Riemann

1  Archimedes
 Carl F. Gauss
 Leonhard Euler
 Bernhard Riemann

2  Carl F. Gauss
 Leonhard Euler 
 Bernhard Riemann

 3  Leonhard Euler
 Bernhard Riemann

4  Bernhard Riemann

# 5 ... and many more...

The above experiment shows that some of the <li> elements contain multiple names separated by newline characters, and others contain just a single name. With this information in mind, you can write your function to extract a single list of names:

def get_names():
    """
    Downloads the page where the list of mathematicians is found
    and returns a list of strings, one per mathematician
    """
    url = 'http://www.fabpedigree.com/james/mathmen.htm'
    response = simple_get(url)

    if response is not None:
        html = BeautifulSoup(response, 'html.parser')
        names = set()
        for li in html.select('li'):
            for name in li.text.split('\n'):
                if len(name) > 0:
                    names.add(name.strip())
        return list(names)

    # Raise an exception if we failed to get any data from the url
    raise Exception('Error retrieving contents at {}'.format(url))

The get_names function downloads the page and iterates over the <li> elements, picking out each name that occurs. Next, you add each name to a Python set, which ensures that you don’t end up with duplicate names. Finally you convert the set to a list and return it.

Getting the Popularity Score

Nice, you’re nearly done! Now that you have a list of names, you need to pick out the pageviews for each one. The function you write is similar to the function you made to get the list of names, only now you supply a name and pick out an integer value from the page.

Again, you should first check out an example page in your browser’s developer tools. It looks like the text appears inside an <a> element, and that the href attribute of that element always contains the string 'latest-60' as a substring. That’s all the information you need to write your function!

def get_hits_on_name(name):
    """
    Accepts a `name` of a mathematician and returns the number
    of hits that mathematician's wikipedia page received in the 
    last 60 days, as an `int`
    """
    # url_root is a template string that used to buld a URL.
    url_root = 'https://xtools.wmflabs.org/articleinfo/en.wikipedia.org/{}'
    response = simple_get(url_root.format(name))

    if response is not None:
        html = BeautifulSoup(response, 'html.parser')

        hit_link = [a for a in html.select('a')
                    if a['href'].find('latest-60') > -1]

        if len(hit_link) > 0:
            # Strip commas:
            link_text = hit_link[0].text.replace(',', '')
            try:
                # Convert to integer
                return int(link_text)
            except:
                log_error("couldn't parse {} as an `int`".format(link_text))

    log_error('No pageviews found for {}'.format(name))
    return None

Putting It All Together

You have reached a point where you can finally find out which mathematician is most beloved by the public! The plan is simple:

Get a list of names,
Iterate over the list to get a “popularity score” for each name; and
Finish by sorting the names by popularity.

Simple right? Well there’s one thing that hasn’t been mentioned yet: errors.

Working with real-world data is messy, and trying to force messy data into a uniform shape will invariably result in the occassional error jumping in to mess with your nice clean vision of how things ought to be. Ideally, you would like to keep track of errors when they occur in order to get a better sense of the quality your data.

For your present purpose, you will track instances when you could not find a popularity score for a given mathematician’s name. At the end of the script, you will print a message showing the number of mathematicians who were left out of the rankings.

Here’s the code:

if __name__ == '__main__':
    print('Getting the list of names....')
    names = get_names()
    print('... done.\n')

    results = []

    print('Getting stats for each name....')

    for name in names:
        try:
            hits = get_hits_on_name(name)
            if hits is None:
                hits = -1
            results.append((hits, name))
        except:
            results.append((-1, name))
            log_error('error encountered while processing '
                      '{}, skipping'.format(name))

    print('... done.\n')

    results.sort()
    results.reverse()

    if len(results) > 5:
        top_marks = results[:5]
    else:
        top_marks = results

    print('\nThe most popular mathematicians are:\n')
    for (mark, mathematician) in top_marks:
        print('{} with {} page views'.format(mathematician, mark))

    no_results = len([res for res in results if res[0] == -1])
    print('\nBut we did not find results for '
          '{} mathematicians on the list'.format(no_results))

And that’s it!

When you run the script, you should see at the following report:

The most popular mathematicians are:

Albert Einstein with 1089615 page views
Isaac Newton with 581612 page views
Srinivasa Ramanujan with 407141 page views
Aristotle with 399480 page views
Galileo Galilei with 375321 page views

But we did not find results for 19 mathematicians on our list

Conclusion & Next Steps

Web scraping is a big field, and you have just finished a brief tour of that field using Python as you guide. You can get pretty far using just requests and BeautifulSoup, but the as you followed along, you may have come up with few questions:

What happens if page content loads as a result of asynchronous Javascript requests? (Check out Selenium’s Python API.)
How do I write a web spider or search engine bot that traverses large portions of the web?
What is this Scrapy thing I keep hearing about?

These are topics for another post… Keep your eyes peeled! A followup is already in the editing stages that does use Selenium and a headless browser to deal with dynamic content:

Don't miss the follow up tutorial: Click here to join the Real Python Newsletter and you'll know when the next instalment comes out.

Until then, happy scraping!

Building a Simple Web App With Bottle, SQLAlchemy, and the Twitter API

2017-12-11T14:47:29+00:00

Last October we challenged our PyBites’ audience to make a web app to better navigate the Daily Python Tip feed. In this article, I’ll share what I built and learned along the way.

In this article you will learn:

How to clone the project repo and set up the app.
How to use the Twitter API via the Tweepy module to load in the tweets.
How to use SQLAlchemy to store and manage the data (tips and hashtags).
How to build a simple web app with Bottle, a micro web-framework similar to Flask.
How to use the pytest framework to add tests.
How Better Code Hub’s guidance led to more maintainable code.

If you want to follow along, reading the code in detail (and possibly contribute), I suggest you fork the repo. Let’s get started.

Project Setup

First, Namespaces are one honking great idea so let’s do our work in a virtual environment. Using Anaconda I create it like so:

$ virtualenv -p <path-to-python-to-use> ~/virtualenvs/pytip

Create a production and a test database in Postgres:

$ psql
psql (9.6.5, server 9.6.2)
Type "help" for help.

# create database pytip;
CREATE DATABASE
# create database pytip_test;
CREATE DATABASE

We’ll need credentials to connect to the the database and the Twitter API (create a new app first). As per best practice configuration should be stored in the environment, not the code. Put the following env variables at the end of ~/virtualenvs/pytip/bin/activate, the script that handles activation / deactivation of your virtual environment, making sure to update the variables for your environment:

export DATABASE_URL='postgres://postgres:password@localhost:5432/pytip'
# twitter
export CONSUMER_KEY='xyz'
export CONSUMER_SECRET='xyz'
export ACCESS_TOKEN='xyz'
export ACCESS_SECRET='xyz'
# if deploying it set this to 'heroku'
export APP_LOCATION=local

In the deactivate function of the same script, I unset them so we keep things out of the shell scope when deactivating (leaving) the virtual environment:

unset DATABASE_URL
unset CONSUMER_KEY
unset CONSUMER_SECRET
unset ACCESS_TOKEN
unset ACCESS_SECRET
unset APP_LOCATION

Now is a good time to activate the virtual environment:

$ source ~/virtualenvs/pytip/bin/activate

Clone the repo and, with the virtual environment enabled, install the requirements:

$ git clone https://github.com/pybites/pytip && cd pytip
$ pip install -r requirements.txt

Next, we import the collection of tweets with:

$ python tasks/import_tweets.py

Then, verify that the tables were created and the tweets were added:

$ psql

\c pytip

pytip=# \dt
          List of relations
 Schema |   Name   | Type  |  Owner
--------+----------+-------+----------
 public | hashtags | table | postgres
 public | tips     | table | postgres
(2 rows)

pytip=# select count(*) from tips;
 count
-------
   222
(1 row)

pytip=# select count(*) from hashtags;
 count
-------
    27
(1 row)

pytip=# \q

Now let’s run the tests:

$ pytest
========================== test session starts ==========================
platform darwin -- Python 3.6.2, pytest-3.2.3, py-1.4.34, pluggy-0.4.0
rootdir: realpython/pytip, inifile:
collected 5 items

tests/test_tasks.py .
tests/test_tips.py ....

========================== 5 passed in 0.61 seconds ==========================

And lastly run the Bottle app with:

$ python app.py

Browse to http://localhost:8080 and voilà: you should see the tips sorted descending on popularity. Clicking on a hashtag link at the left, or using the search box, you can easily filter them. Here we see the pandas tips for example:

The design I made with MUI - a lightweight CSS framework that follows Google’s Material Design guidelines.

Implementation Details

The DB and SQLAlchemy

I used SQLAlchemy to interface with the DB to prevent having to write a lot of (redundant) SQL.

In tips/models.py, we define our models - Hashtag and Tip - that SQLAlchemy will map to DB tables:

from sqlalchemy import Column, Sequence, Integer, String, DateTime
from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()


class Hashtag(Base):
    __tablename__ = 'hashtags'
    id = Column(Integer, Sequence('id_seq'), primary_key=True)
    name = Column(String(20))
    count = Column(Integer)

    def __repr__(self):
        return "<Hashtag('%s', '%d')>" % (self.name, self.count)


class Tip(Base):
    __tablename__ = 'tips'
    id = Column(Integer, Sequence('id_seq'), primary_key=True)
    tweetid = Column(String(22))
    text = Column(String(300))
    created = Column(DateTime)
    likes = Column(Integer)
    retweets = Column(Integer)

    def __repr__(self):
        return "<Tip('%d', '%s')>" % (self.id, self.text)

In tips/db.py, we import these models, and now it’s easy to work with the DB, for example to interface with the Hashtag model:

def get_hashtags():
    return session.query(Hashtag).order_by(Hashtag.name.asc()).all()

And:

def add_hashtags(hashtags_cnt):
    for tag, count in hashtags_cnt.items():
        session.add(Hashtag(name=tag, count=count))
    session.commit()

Query the Twitter API

We need to retrieve the data from Twitter. For that, I created tasks/import_tweets.py. I packaged this under tasks because it should be run in a daily cronjob to look for new tips and update stats (number of likes and retweets) on existing tweets. For the sake of simplicity I have the tables recreated daily. If we start to rely on FK relations with other tables we should definitely choose update statements over delete+add.

We used this script in the Project Setup. Let’s see what it does in more detail.

First, we create an API session object which we pass to tweepy.Cursor. This feature of the API is really nice: it deals with pagination, iterating through the timeline. For the amount of tips - 222 at the time I write this - it’s really fast. The exclude_replies=True and include_rts=False arguments are convenient because we only want Daily Python Tip’s own tweets (not re-tweets).

Extracting hashtags from the tips requires very little code.

First, I defined a regex for a tag:

TAG = re.compile(r'#([a-z0-9]{3,})')

Then, I used findall to get all tags.

I passed them to collections.Counter which returns a dict like object with the tags as keys, and counts as values, ordered in descending order by values (most common). I excluded the too common python tag which would skew the results.

def get_hashtag_counter(tips):
    blob = ' '.join(t.text.lower() for t in tips)
    cnt = Counter(TAG.findall(blob))

    if EXCLUDE_PYTHON_HASHTAG:
        cnt.pop('python', None)

    return cnt

Finally, the import_* functions in tasks/import_tweets.py do the actual import of the tweets and hashtags, calling add_* DB methods of the tips directory/package.

Make a Simple web app with Bottle

With this pre-work done, making a web app is surprisingly easy (or not so surprising if you used Flask before).

First of all meet Bottle:

Bottle is a fast, simple and lightweight WSGI micro web-framework for Python. It is distributed as a single file module and has no dependencies other than the Python Standard Library.

Nice. The resulting web app comprises of < 30 LOC and can be found in app.py.

For this simple app, a single method with an optional tag argument is all it takes. Similar to Flask, the routing is handled with decorators. If called with a tag it filters the tips on tag, else it shows them all. The view decorator defines the template to use. Like Flask (and Django) we return a dict for use in the template.

@route('/')
@route('/<tag>')
@view('index')
def index(tag=None):
    tag = tag or request.query.get('tag') or None
    tags = get_hashtags()
    tips = get_tips(tag)

    return {'search_tag': tag or '',
            'tags': tags,
            'tips': tips}

As per documentation, to work with static files, you add this snippet at the top, after the imports:

@route('/static/<filename:path>')
def send_static(filename):
    return static_file(filename, root='static')

Finally, we want to make sure we only run in debug mode on localhost, hence the APP_LOCATION env variable we defined in Project Setup:

if os.environ.get('APP_LOCATION') == 'heroku':
    run(host="0.0.0.0", port=int(os.environ.get("PORT", 5000)))
else:
    run(host='localhost', port=8080, debug=True, reloader=True)

Bottle Templates

Bottle comes with a fast, powerful and easy to learn built-in template engine called SimpleTemplate.

In the views subdirectory I defined a header.tpl, index.tpl, and footer.tpl. For the tag cloud, I used some simple inline CSS increasing tag size by count, see header.tpl:

% for tag in tags:
  <a style="font-size: {{ tag.count/10 + 1 }}em;" href="/{{ tag.name }}">#{{ tag.name }}</a>&nbsp;&nbsp;
% end

In index.tpl we loop over the tips:

% for tip in tips:
  <div class='tip'>
    <pre>{{ !tip.text }}</pre>
    <div class="mui--text-dark-secondary"><strong>{{ tip.likes }}</strong> Likes / <strong>{{ tip.retweets }}</strong> RTs / {{ tip.created }} / <a href="https://twitter.com/python_tip/status/{{ tip.tweetid }}" target="_blank">Share</a></div>
  </div>
% end

If you are familiar with Flask and Jinja2 this should look very familiar. Embedding Python is even easier, with less typing—(% ... vs {% ... %}).

All css, images (and JS if we’d use it) go into the static subfolder.

And that’s all there is to making a basic web app with Bottle. Once you have the data layer properly defined it’s pretty straightforward.

Add tests with pytest

Now let’s make this project a bit more robust by adding some tests. Testing the DB required a bit more digging into the pytest framework, but I ended up using the pytest.fixture decorator to set up and tear down a database with some test tweets.

Instead of calling the Twitter API, I used some static data provided in tweets.json. And, rather than using the live DB, in tips/db.py, I check if pytest is the caller (sys.argv[0]). If so, I use the test DB. I probably will refactor this, because Bottle supports working with config files.

The hashtag part was easier to test (test_get_hashtag_counter) because I could just add some hashtags to a multiline string. No fixtures needed.

Code quality matters - Better Code Hub

Better Code Hub guides you in writing, well, better code. Before writing the tests the project scored a 7:

Not bad, but we can do better:

I bumped it to a 9 by making the code more modular, taking the DB logic out of the app.py (web app), putting it in the tips folder/ package (refactorings 1 and 2)
Then with the tests in place the project scored a 10:

Conclusion and Learning

Our Code Challenge #40 offered some good practice:

I built a useful app which can be expanded (I want to add an API).
I used some cool modules worth exploring: Tweepy, SQLAlchemy, and Bottle.
I learned some more pytest because I needed fixtures to test interaction with the DB.
Above all, having to make the code testable, the app became more modular which made it easier to maintain. Better Code Hub was of great help in this process.
I deployed the app to Heroku using our step-by-step guide.

We Challenge You

The best way to learn and improve your coding skills is to practice. At PyBites we solidified this concept by organizing Python code challenges. Check out our growing collection, fork the repo, and get coding!

Let us know if you build something cool by making a Pull Request of your work. We have seen folks really stretching themselves through these challenges, and so did we.

Happy coding!

Code Evaluation With AWS Lambda and API Gateway

2017-09-25T13:22:54+00:00

This tutorial details how AWS Lambda and API Gateway can be used to develop a simple code evaluation API, where an end user submits code, via an AJAX form submission, which is then executed securely by a Lambda function.

Check out the live demo of what you’ll be building in action here.

WARNING: The code found in this tutorial is used to build a toy app to prototype a proof of concept and is not meant for production use.

This tutorial assumes that you already have an account set up with AWS. Also, we will use the US East (N. Virginia) / us-east-1 region. Feel free to use the region of your choice. For more info, review the Regions and Availability Zones guide.

Objectives

By the end of this tutorial you will be able to…

Explain what AWS Lambda and API Gateway are and why would would want to use them
Discuss the benefits of using AWS Lambda functions
Create an AWS Lambda function with Python
Develop a RESTful API endpoint with API Gateway
Trigger an AWS Lambda function from API Gateway

What is AWS Lambda?

Amazon Web Services (AWS) Lambda is an on-demand compute service that lets you run code in response to events or HTTP requests.

Use cases:

Event	Action
Image added to S3	Image is processed
HTTP Request via API Gateway	HTTP Response
Log file added to Cloudwatch	Analyze the log
Scheduled event	Back up files
Scheduled event	Synchronization of files

For more examples, review the Examples of How to Use AWS Lambda guide from AWS.

You can run scripts and apps without having to provision or manage servers in a seemingly infinitely-scalable environment where you pay only for usage. This is “serverless” computing in a nut shell. For our purposes, AWS Lambda is a perfect solution for running user-supplied code quickly, securely, and cheaply.

As of writing, Lambda supports code written in JavaScript (Node.js), Python, Java, and C#.

Project Setup

Start by cloning down the base project:

$ git clone https://github.com/realpython/aws-lambda-code-execute \
  --branch v1 --single-branch
$ cd aws-lambda-code-execute

Then, check out the v1 tag to the master branch:

$ git checkout tags/v1 -b master

Open the index.html file in your browser of choice:

Then, open the project in your code editor:

├── README.md
├── assets
│   ├── main.css
│   ├── main.js
│   └── vendor
│       ├── bootstrap
│       │   ├── css
│       │   │   ├── bootstrap-grid.css
│       │   │   ├── bootstrap-grid.min.css
│       │   │   ├── bootstrap-reboot.css
│       │   │   ├── bootstrap-reboot.min.css
│       │   │   ├── bootstrap.css
│       │   │   └── bootstrap.min.css
│       │   └── js
│       │       ├── bootstrap.js
│       │       └── bootstrap.min.js
│       ├── jquery
│       │   ├── jquery.js
│       │   └── jquery.min.js
│       └── popper
│           ├── popper.js
│           └── popper.min.js
└── index.html

Let’s quickly review the code. Essentially, we just have a simple HTML form styled with Bootstrap. The input field is replaced with Ace, an embeddable code editor, which provides basic syntax highlighting. Finally, within assets/main.js, a jQuery event handler is wired up to grab the code from the Ace editor, when the form is submitted, and send the data somewhere (eventually to API Gateway) via an AJAX request.

Lambda Setup

Within the AWS Console, navigate to the main Lambda page and click “Create a function”:

Create function

Steps…

Select blueprint: Click “Author from scratch” to start with a blank function:
Configure Triggers: We’ll set up the API Gateway integration later, so simply click “Next” to skip this part.
Configure function: Name the function execute_python_code, and add a basic description - Execute user-supplied Python code. Select “Python 3.6” in the “Runtime” drop-down.
Within the inline code editor, update the lambda_handler function definition with:
```
import sys
from io import StringIO


def lambda_handler(event, context):
    # get code from payload
    code = event[‘answer’]
    test_code = code + ‘\nprint(sum(1,1))’
    # capture stdout
    buffer = StringIO()
    sys.stdout = buffer
    # execute code
    try:
        exec(test_code)
    except:
        return False
    # return stdout
    sys.stdout = sys.stdout
    # check
    if int(buffer.getvalue()) == 2:
        return True
    return False
```
Here, within lambda_handler, which is the default entry point for Lambda, we parse the JSON request body, passing the supplied code along with some test code - sum(1,1) - to the exec function - which executes the string as Python code. Then, we simply ensure the actual results are the same as what’s expected - e.g., 2 - and return the appropriate response.

Under “Lambda function handler and role”, leave the default handler and then select “Create a new Role from template(s)” from the drop-down. Enter a “Role name”, like api_gateway_access, and select ” Simple Microservice permissions” for the “Policy templates”, which provides access to API Gateway.

Click “Next”.
Review: Create the function after a quick review.

Test

Next click on the “Test” button to execute the newly created Lambda:

Using the “Hello World” event template, replace the sample with:

{
  "answer": "def sum(x,y):\n    return x+y"
}

Click the “Save and test” button at the bottom of the modal to run the test. Once done, you should see something similar to:

With that, we can move on to configuring the API Gateway to trigger the Lambda from user-submitted POST requests…

API Gateway Setup

API Gateway is used to define and host APIs. In our example, we’ll create a single HTTP POST endpoint that triggers the Lambda function when an HTTP request is received and then responds with the results of the Lambda function, either true or false.

Steps:

Create the API
Test it manually
Enable CORS
Deploy the API
Test via cURL

Create the API

To start, from the API Gateway page, click the “Get Started” button to create a new API:
Select “New API”, and then provide a descriptive name, like code_execute_api:

Then, create the API.
Select “Create Resource” from the “Actions” drop-down.
Name the resource execute, and then click “Create Resource”.
With the resource highlighted, select “Create Method” from the “Actions” drop-down.
Choose “POST” from the method drop-down. Click the checkmark next to it.
In the “Setup” step, select “Lambda Function” as the “Integration type”, select the “us-east-1” region in the drop-down, and enter the name of the Lambda function that you just created.
Click “Save”, and then click “OK” to give permission to the API Gateway to run your Lambda function.

Test it manually

To test, click on the lightning bolt that says “Test”.

Scroll down to the “Request Body” input and add the same JSON code we used with the Lambda function:

{
  "answer": "def sum(x,y):\n    return x+y"
}

Click “Test”. You should see something similar to:

Enable CORS

Next, we need to enable CORS so that we can POST to the API endpoint from another domain.

With the resource highlighted, select “Enable CORS” from the “Actions” drop-down:

Just keep the defaults for now since we’re still testing the API. Click the “Enable CORS and replace existing CORS headers” button.

Deploy the API

Finally, to deploy, select “Deploy API” from the “Actions” drop-down:

Create a new “Deployment stage” called ‘v1’:

API gateway will generate a random subdomain for the API endpoint URL, and the stage name will be added to the end of the URL. You should now be able to make POST requests to a similar URL:

https://c0rue3ifh4.execute-api.us-east-1.amazonaws.com/v1/execute

Test via cURL

$ curl -H "Content-Type: application/json" -X POST \
  -d '{"answer":"def sum(x,y):\n    return x+y"}' \
  https://c0rue3ifh4.execute-api.us-east-1.amazonaws.com/v1/execute

Update the Form

Now, to update the form so that it sends the POST request to the API Gateway endpoint, first add the URL to the grade function in assets/main.js:

function grade(payload) {
  $.ajax({
    method: 'POST',
    url: 'https://c0rue3ifh4.execute-api.us-east-1.amazonaws.com/v1/execute',
    dataType: 'json',
    contentType: 'application/json',
    data: JSON.stringify(payload)
  })
  .done((res) => { console.log(res); })
  .catch((err) => { console.log(err); });
}

Then, update the .done and .catch() functions, like so:

function grade(payload) {
  $.ajax({
    method: 'POST',
    url: 'https://c0rue3ifh4.execute-api.us-east-1.amazonaws.com/v1/execute',
    dataType: 'json',
    contentType: 'application/json',
    data: JSON.stringify(payload)
  })
  .done((res) => {
    let message = 'Incorrect. Please try again.';
    if (res) {
      message = 'Correct!';
    }
    $('.answer').html(message);
    console.log(res);
    console.log(message);
  })
  .catch((err) => {
    $('.answer').html('Something went terribly wrong!');
    console.log(err);
  });
}

Now, if the request is a success, the appropriate message will be added - via the jQuery html method - to an HTML element with a class of answer. Add this element, just below the HTML form, within index.html:

<h5 class="answer"></h5>

Let’s add a bit of style as well to the assets/main.css file:

.answer {
  padding-top: 30px;
  color: #dc3545;
  font-style: italic;
}

Test it out!

Next Steps

Production: Think about what’s required for a more robust, production-ready application - HTTPS, authentication, possibly a data store. How would you implement these within AWS? Which AWS services can/would you use?
Dynamic: Right now the Lambda function can only be used to test the sum function. How could you make this (more) dynamic, so that it can be used to test any code challenge (maybe even in any language)? Try adding a data attribute to the DOM, so that when a user submits an exercise the test code along with solution is sent along with the POST request - i.e., <some-html-element data-test="\nprint(sum(1,1))" data-results"2" </some-html-element>.
Stack trace: Instead of just responding with true or false, send back the entire stack trace and add it to the DOM when the answer is incorrect.

Thanks for reading. Add questions and/or comments below. Grab the final code from the aws-lambda-code-execute repo. Cheers!

User Authentication With Angular 4 and Flask

2017-08-29T13:37:39+00:00

In this tutorial, we’ll demonstrate how to set up token-based authentication (via JSON Web Tokens) with Angular 4 and Flask.

Main Dependencies:

Angular v4.2.4 (via Angular CLI v1.3.2)
Flask v0.12
Python v3.6.2

Free Bonus: Click here to get access to a free Flask + Python video tutorial series that shows you how to build a fully working Flask web app, step-by-step.

Auth Workflow

Here’s the full user auth process:

Client logs in and the credentials are sent to the server
If the credentials are correct, the server generates a token and sends it as a response to the client
Client receives and stores the token in Local Storage
Client then sends token to server on subsequent requests within the request header

Project Setup

Start by globally installing the Angular CLI:

$ npm install -g @angular/cli@1.3.2

Then generate a new Angular 4 project boilerplate:

$ ng new angular4-auth

Fire up the app after the dependencies install:

$ cd angular4-auth
$ ng serve

It will probably take a minute or two to compile and build your application. Once done, navigate to http://localhost:4200 to ensure the app is up and running.

Open up the project in your favorite code editor, and then glance over the code:

├── e2e
│   ├── app.e2e-spec.ts
│   ├── app.po.ts
│   └── tsconfig.e2e.json
├── karma.conf.js
├── package.json
├── protractor.conf.js
├── src
│   ├── app
│   │   ├── app.component.css
│   │   ├── app.component.html
│   │   ├── app.component.spec.ts
│   │   ├── app.component.ts
│   │   └── app.module.ts
│   ├── assets
│   ├── environments
│   │   ├── environment.prod.ts
│   │   └── environment.ts
│   ├── favicon.ico
│   ├── index.html
│   ├── main.ts
│   ├── polyfills.ts
│   ├── styles.css
│   ├── test.ts
│   ├── tsconfig.app.json
│   ├── tsconfig.spec.json
│   └── typings.d.ts
├── tsconfig.json
└── tslint.json

In short, the client-side code lives in the “src” folder and the Angular app itself can be found in the “app” folder.

Take note of the AppModule within app.module.ts. This is used to bootstrap the Angular app. The @NgModule decorator takes metadata that lets Angular know how to run the app. Everything that we create in this tutorial will be added to this object.

Make sure you have a decent grasp of the app structure before moving on.

NOTE: Just getting started with Angular 4? Review the Angular Style Guide, since the app generated from the CLI follows the recommended structure from that guide, as well as the Angular4Crud Tutorial.

Did you notice that the CLI initialized a new Git repo? This part is optional, but it’s a good idea to create a new Github repository and update the remote:

$ git remote set-url origin <newurl>

Now, let’s wire up a new component…

Auth Component

First, use the CLI to generate a new Login component:

$ ng generate component components/login

This set up the component files and folders and even wired it up to app.module.ts. Next, let’s change the login.component.ts file to the following:

import { Component } from '@angular/core';

@Component({
  selector: 'login',
  templateUrl: './login.component.html',
  styleUrls: ['./login.component.css']
})

export class LoginComponent {
  test: string = 'just a test';
}

If you haven’t used TypeScript before, then this code probably looks pretty foreign to you. TypeScript is a statically-typed superset of JavaScript that compiles to vanilla JavaScript, and it is the de facto programming language for building Angular 4 apps.

In Angular 4, we define a component by wrapping a config object with an @Component decorator. We can share code between packages by importing the classes we need; and, in this case, we import Component from the @angular/core package. The LoginComponent class is the component’s controller, and we use the export operator to make it available for other classes to import.

Add the following HTML to the login.component.html:

<h1>Login</h1>

<p>{{test}}</p>

Next, configure the routes, via the RouterModule in the app.module.ts file:

import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { RouterModule } from '@angular/router';

import { AppComponent } from './app.component';
import { LoginComponent } from './components/login/login.component';

@NgModule({
  declarations: [
    AppComponent,
    LoginComponent,
  ],
  imports: [
    BrowserModule,
    RouterModule.forRoot([
      { path: 'login', component: LoginComponent }
    ])
  ],
  providers: [],
  bootstrap: [AppComponent]
})

export class AppModule { }

Finish enabling routing by replacing all HTML in the app.component.html file with the <router-outlet> tag:

<router-outlet></router-outlet>

Run ng serve in your terminal, if you haven’t already, and then navigate to http://localhost:4200/login. If all went well you should see the just a test text.

Bootstrap Style

To quickly add some style, update the index.html, adding in Bootstrap and wrapping the <app-root></app-root> in a container:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Angular4Auth</title>
  <base href="/">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="icon" type="image/x-icon" href="favicon.ico">
  <link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
</head>
<body>
  <div class="container">
    <app-root></app-root>
  </div>
</body>
</html>

You should see the app auto reload as soon as you save.

Auth Service

Next, let’s create a global service to handle a user logging in, logging out, and signing up:

$ ng generate service services/auth

Edit the auth.service.ts so that it has the following code:

import { Injectable } from '@angular/core';

@Injectable()
export class AuthService {
  test(): string {
    return 'working';
  }
}

Remember how providers worked in Angular 1? They were global objects that stored a single state. When the data in a provider changed, any object that had injected that provider would receive the updates. In Angular 4, providers retain their special behavior and they are defined with the @Injectable decorator.

Sanity Check

Before adding anything significant to AuthService, let’s make sure the service itself is wired up correctly. To do that, within login.component.ts inject the service and call the test() method:

import { Component, OnInit } from '@angular/core';
import { AuthService } from '../../services/auth.service';

@Component({
  selector: 'login',
  templateUrl: './login.component.html',
  styleUrls: ['./login.component.css']
})

export class LoginComponent implements OnInit {
  test: string = 'just a test';
  constructor(private auth: AuthService) {}
  ngOnInit(): void {
    console.log(this.auth.test());
  }
}

We’ve introduced some new concepts and keywords. The constructor() function is a special method that we use to set up a new instance of a class. The constructor() is where we pass any parameters that the class requires, including any providers (i.e., AuthService) that we want to inject. In TypeScript, we can hide variables from the outside world with the private keyword. Passing a private variable in the constructor is a shortcut to defining it within the class and then assigning the argument’s value to it. Notice how the auth variable is accessible to the this object after it is passed into the constructor.

We implement the OnInit interface to ensure that we explicitly define a ngOnInit() function. Implementing OnInit ensures that our component will be called after the first change detection check. This function is called once when the component first initializes, making it the ideal place to configure data that relies on other Angular classes.

Unlike components, which are automatically added, services have to be manually imported and configured on the @NgModule. So, to get it working, you’ll also have to import the AuthService in app.module.ts and add it to the providers:

import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { RouterModule } from '@angular/router';

import { AppComponent } from './app.component';
import { LoginComponent } from './components/login/login.component';
import { AuthService } from './services/auth.service';

@NgModule({
  declarations: [
    AppComponent,
    LoginComponent,
  ],
  imports: [
    BrowserModule,
    RouterModule.forRoot([
      { path: 'login', component: LoginComponent }
    ])
  ],
  providers: [AuthService],
  bootstrap: [AppComponent]
})

export class AppModule { }

Run the server and then navigate to http://localhost:4200/login. You should see working logged to the JavaScript console.

To handle logging a user in, update the AuthService like so:

import { Injectable } from '@angular/core';
import { Headers, Http } from '@angular/http';
import 'rxjs/add/operator/toPromise';

@Injectable()
export class AuthService {
  private BASE_URL: string = 'http://localhost:5000/auth';
  private headers: Headers = new Headers({'Content-Type': 'application/json'});
  constructor(private http: Http) {}
  login(user): Promise<any> {
    let url: string = `${this.BASE_URL}/login`;
    return this.http.post(url, user, {headers: this.headers}).toPromise();
  }
}

We employ the help of some built-in Angular classes, Headers and Http, to handle our AJAX calls to the server.

Also, update the app.module.ts file to import the HttpModule.

import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { RouterModule } from '@angular/router';
import { HttpModule } from '@angular/http';

import { AppComponent } from './app.component';
import { LoginComponent } from './components/login/login.component';
import { AuthService } from './services/auth.service';

@NgModule({
  declarations: [
    AppComponent,
    LoginComponent,
  ],
  imports: [
    BrowserModule,
    HttpModule,
    RouterModule.forRoot([
      { path: 'login', component: LoginComponent }
    ])
  ],
  providers: [AuthService],
  bootstrap: [AppComponent]
})

export class AppModule { }

Here, we are using the Http service to send an AJAX request to the /user/login endpoint. This returns a promise object.

NOTE: Make sure to remove console.log(this.auth.test()); from the LoginComponent component.

User Registration

Let’s go ahead and add the ability to register a user as well, which is similar to logging a user in. Update src/app/services/auth.service.ts, taking note of the register method:

import { Injectable } from '@angular/core';
import { Headers, Http } from '@angular/http';
import 'rxjs/add/operator/toPromise';

@Injectable()
export class AuthService {
  private BASE_URL: string = 'http://localhost:5000/auth';
  private headers: Headers = new Headers({'Content-Type': 'application/json'});
  constructor(private http: Http) {}
  login(user): Promise<any> {
    let url: string = `${this.BASE_URL}/login`;
    return this.http.post(url, user, {headers: this.headers}).toPromise();
  }
  register(user): Promise<any> {
    let url: string = `${this.BASE_URL}/register`;
    return this.http.post(url, user, {headers: this.headers}).toPromise();
  }
}

Now, to test this we need to set up a back end…

Server-side Setup

For the server-side, we’ll use the finished project from a previous blog post, Token-Based Authentication With Flask. You can view the code from the flask-jwt-auth repository.

NOTE: Feel free to use your own server, just make sure to update the baseURL in the AuthService.

Clone the project structure in a new terminal window:

$ git clone https://github.com/realpython/flask-jwt-auth

Follow the directions in the README to set up the project, making sure the tests pass before moving on. Once done, run the server with python manage.py runserver, which will listen on port 5000.

Sanity Check

To test, update LoginComponent to use the login and register methods from the service:

import { Component, OnInit } from '@angular/core';
import { AuthService } from '../../services/auth.service';

@Component({
  selector: 'login',
  templateUrl: './login.component.html',
  styleUrls: ['./login.component.css']
})
export class LoginComponent implements OnInit {
  test: string = 'just a test';
  constructor(private auth: AuthService) {}
  ngOnInit(): void {
    let sampleUser: any = {
      email: 'michael@realpython.com' as string,
      password: 'michael' as string
    };
    this.auth.register(sampleUser)
    .then((user) => {
      console.log(user.json());
    })
    .catch((err) => {
      console.log(err);
    });
    this.auth.login(sampleUser).then((user) => {
      console.log(user.json());
    })
    .catch((err) => {
      console.log(err);
    });
  }
}

Refresh http://localhost:4200/login in the browser and you should see a success in the JavaScript console, after the user is logged in, with the token:

{
  "auth_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1M…jozfQ.bPNQb3C98yyNe0LDyl1Bfkp0Btn15QyMxZnBoE9RQMI",
  "message": "Successfully logged in.",
  "status": "success"
}

Update login.component.html:

<div class="row">
  <div class="col-md-4">
    <h1>Login</h1>
    <hr><br>
    <form (ngSubmit)="onLogin()" novalidate>
     <div class="form-group">
       <label for="email">Email</label>
       <input type="text" class="form-control" id="email" placeholder="enter email" [(ngModel)]="user.email" name="email" required>
     </div>
     <div class="form-group">
       <label for="password">Password</label>
       <input type="password" class="form-control" id="password" placeholder="enter password" [(ngModel)]="user.password" name="password" required>
     </div>
     <button type="submit" class="btn btn-default">Submit</button>
    </form>
  </div>
</div>

Take note of the form. We used the [(ngModel)] directive on each of the form inputs to capture those values in the controller. Also, when the form is submitted, the ngSubmit directive handles the event by firing the onLogin() method.

Now, let’s update the component code, adding in onLogin():

import { Component } from '@angular/core';
import { AuthService } from '../../services/auth.service';
import { User } from '../../models/user';

@Component({
  selector: 'login',
  templateUrl: './login.component.html',
  styleUrls: ['./login.component.css']
})
export class LoginComponent {
  user: User = new User();
  constructor(private auth: AuthService) {}
  onLogin(): void {
    this.auth.login(this.user)
    .then((user) => {
      console.log(user.json());
    })
    .catch((err) => {
      console.log(err);
    });
  }
}

If you have the Angular web server running, you should see the error Cannot find module '../../models/user' in the browser. Before our code will work, we need to create a User model.

$ ng generate class models/user

Update src/app/models/user.ts:

export class User {
  constructor(email?: string, password?: string) {}
}

Our User model has two properties, email and password. The ? character is a special operator that indicates that initializing User with explicit email and password values is optional. This is equivalent to the following class in Python:

class User(object):
    def __init__(self, email=None, password=None):
        self.email = email
        self.password = password

Don’t forget to update auth.service.ts to use the new object.

import { Injectable } from '@angular/core';
import { Headers, Http } from '@angular/http';
import { User } from '../models/user';
import 'rxjs/add/operator/toPromise';

@Injectable()
export class AuthService {
  private BASE_URL: string = 'http://localhost:5000/auth';
  private headers: Headers = new Headers({'Content-Type': 'application/json'});
  constructor(private http: Http) {}
  login(user: User): Promise<any> {
    let url: string = `${this.BASE_URL}/login`;
    return this.http.post(url, user, {headers: this.headers}).toPromise();
  }
  register(user: User): Promise<any> {
    let url: string = `${this.BASE_URL}/register`;
    return this.http.post(url, user, {headers: this.headers}).toPromise();
  }
}

One last thing. We need to import the FormsModule in the app.module.ts file.

import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { RouterModule } from '@angular/router';
import { HttpModule } from '@angular/http';
import { FormsModule } from '@angular/forms';

import { AppComponent } from './app.component';
import { LoginComponent } from './components/login/login.component';
import { AuthService } from './services/auth.service';

@NgModule({
  declarations: [
    AppComponent,
    LoginComponent,
  ],
  imports: [
    BrowserModule,
    HttpModule,
    FormsModule,
    RouterModule.forRoot([
      { path: 'login', component: LoginComponent }
    ])
  ],
  providers: [AuthService],
  bootstrap: [AppComponent]
})

export class AppModule { }

So, when the form is submitted, we capture the email and password and pass them to the login() method on the service.

Test this out with-

email: michael@realpython.com
password: michael

Again, you should see a success in the javaScript console with the token.

Auth Register

Just like for the login functionality, we need to add a component for registering a user. Start by generating a new Register component:

$ ng generate component components/register

Update src/app/components/register/register.component.html:

<div class="row">
  <div class="col-md-4">
    <h1>Register</h1>
    <hr><br>
    <form (ngSubmit)="onRegister()" novalidate>
     <div class="form-group">
       <label for="email">Email</label>
       <input type="text" class="form-control" id="email" placeholder="enter email" [(ngModel)]="user.email" name="email" required>
     </div>
     <div class="form-group">
       <label for="password">Password</label>
       <input type="password" class="form-control" id="password" placeholder="enter password" [(ngModel)]="user.password" name="password" required>
     </div>
     <button type="submit" class="btn btn-default">Submit</button>
    </form>
  </div>
</div>

Then, update src/app/components/register/register.component.ts as follows:

import { Component } from '@angular/core';
import { AuthService } from '../../services/auth.service';
import { User } from '../../models/user';

@Component({
  selector: 'register',
  templateUrl: './register.component.html',
  styleUrls: ['./register.component.css']
})
export class RegisterComponent {
  user: User = new User();
  constructor(private auth: AuthService) {}
  onRegister(): void {
    this.auth.register(this.user)
    .then((user) => {
      console.log(user.json());
    })
    .catch((err) => {
      console.log(err);
    });
  }
}

Add a new route handler to the app.module.ts file:

RouterModule.forRoot([
  { path: 'login', component: LoginComponent },
  { path: 'register', component: RegisterComponent }
])

Test it out by registering a new user!

Local Storage

Next, let’s add the token to Local Storage for persistence by replacing the console.log(user.json()); with localStorage.setItem('token', user.data.token); in src/app/components/login/login.component.ts:

onLogin(): void {
  this.auth.login(this.user)
  .then((user) => {
    localStorage.setItem('token', user.json().auth_token);
  })
  .catch((err) => {
    console.log(err);
  });
}

Do the same within src/app/components/register/register.component.ts:

onRegister(): void {
  this.auth.register(this.user)
  .then((user) => {
    localStorage.setItem('token', user.json().auth_token);
  })
  .catch((err) => {
    console.log(err);
  });
}

As long as that token is present, the user can be considered logged in. And, when a user needs to make an AJAX request, that token can be used.

NOTE: Besides the token, you could also add the user id and email to Local Storage. You would just need to update the server-side to send back that info when a user logs in.

Test this out. Ensure that the token is present in Local Storage after you log in.

User Status

To test out login persistence, we can add a new view that verifies that the user is logged in and that the token is valid.

Add the following method to AuthService:

ensureAuthenticated(token): Promise<any> {
  let url: string = `${this.BASE_URL}/status`;
  let headers: Headers = new Headers({
    'Content-Type': 'application/json',
    Authorization: `Bearer ${token}`
  });
  return this.http.get(url, {headers: headers}).toPromise();
}

Take note of Authorization: 'Bearer ' + token. This is called a Bearer schema, which is sent along with the request. On the server, we are simply checking for the Authorization header, and then whether the token is valid. Can you find this code on the server-side?

Then, generate a new Status component:

$ ng generate component components/status

Create the HTML template, src/app/components/status/status.component.html:

<div class="row">
  <div class="col-md-4">
    <h1>User Status</h1>
    <hr><br>
    <p>Logged In? {{isLoggedIn}}</p>
  </div>
</div>

And change the component code in src/app/components/status/status.component.ts:

import { Component, OnInit } from '@angular/core';
import { AuthService } from '../../services/auth.service';

@Component({
  selector: 'status',
  templateUrl: './status.component.html',
  styleUrls: ['./status.component.css']
})
export class StatusComponent implements OnInit {
  isLoggedIn: boolean = false;
  constructor(private auth: AuthService) {}
  ngOnInit(): void {
    const token = localStorage.getItem('token');
    if (token) {
      this.auth.ensureAuthenticated(token)
      .then((user) => {
        console.log(user.json());
        if (user.json().status === 'success') {
          this.isLoggedIn = true;
        }
      })
      .catch((err) => {
        console.log(err);
      });
    }
  }
}

Finally, add a new route handler to the app.module.ts file:

RouterModule.forRoot([
  { path: 'login', component: LoginComponent },
  { path: 'register', component: RegisterComponent },
  { path: 'status', component: StatusComponent }
])

Ready to test? Log in, and then navigate to http://localhost:4200/status. If there is a token in Local Storage, you should see:

{
  "message": "Signature expired. Please log in again.",
  "status": "fail"
}

Why? Well, if you dig deeper on the server-side, you will find that the token is only valid for 5 seconds in project/server/models.py:

def encode_auth_token(self, user_id):
    """
    Generates the Auth Token
    :return: string
    """
    try:
        payload = {
            'exp': datetime.datetime.utcnow() + datetime.timedelta(days=0, seconds=5),
            'iat': datetime.datetime.utcnow(),
            'sub': user_id
        }
        return jwt.encode(
            payload,
            app.config.get('SECRET_KEY'),
            algorithm='HS256'
        )
    except Exception as e:
        return e

Update this to 1 day:

'exp': datetime.datetime.utcnow() + datetime.timedelta(days=1, seconds=0)

And then test it again. You should now see something like:

{
  "data": {
    "admin": false,
    "email": "michael@realpython.com",
    "registered_on": "Sun, 13 Aug 2017 17:21:52 GMT",
    "user_id": 4
  },
  "status": "success"
}

Finally, let’s redirect to the status page after a user successfully registers or logs in. Update src/app/components/login/login.component.ts like so:

import { Component } from '@angular/core';
import { Router } from '@angular/router';
import { AuthService } from '../../services/auth.service';
import { User } from '../../models/user';

@Component({
  selector: 'login',
  templateUrl: './login.component.html',
  styleUrls: ['./login.component.css']
})
export class LoginComponent {
  user: User = new User();
  constructor(private router: Router, private auth: AuthService) {}
  onLogin(): void {
    this.auth.login(this.user)
    .then((user) => {
      localStorage.setItem('token', user.json().auth_token);
      this.router.navigateByUrl('/status');
    })
    .catch((err) => {
      console.log(err);
    });
  }
}

Then update src/app/components/register/register.component.ts:

import { Component } from '@angular/core';
import { Router } from '@angular/router';
import { AuthService } from '../../services/auth.service';
import { User } from '../../models/user';

@Component({
  selector: 'register',
  templateUrl: './register.component.html',
  styleUrls: ['./register.component.css']
})
export class RegisterComponent {
  user: User = new User();
  constructor(private router: Router, private auth: AuthService) {}
  onRegister(): void {
    this.auth.register(this.user)
    .then((user) => {
      localStorage.setItem('token', user.json().auth_token);
      this.router.navigateByUrl('/status');
    })
    .catch((err) => {
      console.log(err);
    });
  }
}

Test it out!

Route Restriction

Right now, all routes are open; so, regardless of whether a user is logged in or not, they they can access each route. Certain routes should be restricted if a user is not logged in, while other routes should be restricted if a user is logged in:

/ - no restrictions
/login - restricted when logged in
/register - restricted when logged in
/status - restricted when not logged in

To achieve this, add either EnsureAuthenticated or LoginRedirect to each route, depending on whether you want to guide the user to the status view or the login view.

Start by creating two new services:

$ ng generate service services/ensure-authenticated
$ ng generate service services/login-redirect

Replace the code in the ensure-authenticated.service.ts file as follows:

import { Injectable } from '@angular/core';
import { CanActivate, Router } from '@angular/router';
import { AuthService } from './auth.service';

@Injectable()
export class EnsureAuthenticated implements CanActivate {
  constructor(private auth: AuthService, private router: Router) {}
  canActivate(): boolean {
    if (localStorage.getItem('token')) {
      return true;
    }
    else {
      this.router.navigateByUrl('/login');
      return false;
    }
  }
}

And replace the code in the login-redirect.service.ts like so:

import { Injectable } from '@angular/core';
import { CanActivate, Router } from '@angular/router';
import { AuthService } from './auth.service';

@Injectable()
export class LoginRedirect implements CanActivate {
  constructor(private auth: AuthService, private router: Router) {}
  canActivate(): boolean {
    if (localStorage.getItem('token')) {
      this.router.navigateByUrl('/status');
      return false;
    }
    else {
      return true;
    }
  }
}

Finally, update the app.module.ts file to import and configure the new services:

import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { RouterModule } from '@angular/router';
import { HttpModule } from '@angular/http';
import { FormsModule } from '@angular/forms';

import { AppComponent } from './app.component';
import { LoginComponent } from './components/login/login.component';
import { AuthService } from './services/auth.service';
import { RegisterComponent } from './components/register/register.component';
import { StatusComponent } from './components/status/status.component';
import { EnsureAuthenticated } from './services/ensure-authenticated.service';
import { LoginRedirect } from './services/login-redirect.service';

@NgModule({
  declarations: [
    AppComponent,
    LoginComponent,
    RegisterComponent,
    StatusComponent,
  ],
  imports: [
    BrowserModule,
    HttpModule,
    FormsModule,
    RouterModule.forRoot([
      {
        path: 'login',
        component: LoginComponent,
        canActivate: [LoginRedirect]
      },
      {
        path: 'register',
        component: RegisterComponent,
        canActivate: [LoginRedirect]
      },
      {
        path: 'status',
        component: StatusComponent,
        canActivate:
        [EnsureAuthenticated]
      }
    ])
  ],
  providers: [
    AuthService,
    EnsureAuthenticated,
    LoginRedirect
  ],
  bootstrap: [AppComponent]
})

export class AppModule { }

Note how we are adding our services to a new route property, canActivate. The routing system uses the services in the canActivate array to determine whether to display the requested URL path. If the route has LoginRedirect and the user is already logged in, then they will be redirected to the status view. Including the EnsureAuthenticated service redirects the user to the login view if they attempt to access a URL that requires authentication.

Test one last time.

What’s Next?

In this tutorial, we went through the process of adding authentication to an Angular 4 + Flask app using JSON Web Tokens.

What’s next?

Try switching out the Flask back-end for a different web framework, like Django or Bottle, using the following endpoints:

/auth/register
/auth/login
/auth/logout
/auth/user

If you’d like to see how to build a complete Python web app using Flask, check out this video series:

Free Bonus: Click here to get access to a free Flask + Python video tutorial series that shows you how to build a fully working Flask web app, step-by-step.

Add questions and/or comments below. Grab the final code from the angular4-auth repo.

Test Driven Development of a Django RESTful API

2017-05-02T13:54:17+00:00

This post walks through the process of developing a CRUD-based RESTful API with Django and Django REST Framework, which is used for rapidly building RESTful APIs based on Django models.

This application uses:

Python v3.6.0
Django v1.11.0
Django REST Framework v3.6.2
Postgres v9.6.1
Psycopg2 v2.7.1

Free Bonus: Click here to download a copy of the "REST in a Nutshell" Guide with a hands-on introduction to REST API principles and examples.

NOTE: Check out the third Real Python course for a more in-depth tutorial on Django REST Framework.

Objectives

By the end of this tutorial you will be able to…

Discuss the benefits of using Django REST Framework for bootstrapping the development of a RESTful API
Validate model querysets using serializers
Appreciate Django REST Framework’s Browsable API feature for a cleaner and well-documented version of your APIs
Practice test-driven development

Why Django REST Framework?

Django REST Framework (REST Framework) provides a number of powerful features out-of-the-box that go well with idiomatic Django, including:

Browsable API: Documents your API with a human-friendly HTML output, providing a beautiful form-like interface for submitting data to resources and fetching from them using the standard HTTP methods.
Auth Support: REST Framework has rich support for various authentication protocols along with permissions and throttling policies which can be configured on a per-view basis.
Serializers: Serializers are an elegant way of validating model querysets/instances and converting them to native Python datatypes that can be easily rendered into JSON and XML.
Throttling: Throttling is way to determine whether a request is authorized or not and can be integrated with different permissions. It is generally used for rate limiting API requests from a single user.

Plus, the documentation is easy to read and full of examples. If you’re building a RESTful API where you have a one-to-one relationship between your API endpoints and your models, then REST Framework is the way to go.

Django Project Setup

Create and activate a virtualenv:

$ mkdir django-puppy-store
$ cd django-puppy-store
$ python3.6 -m venv env
$ source env/bin/activate

Install Django and set up a new project:

(env)$ pip install django==1.11.0
(env)$ django-admin startproject puppy_store

Your current project structure should look like this:

└── puppy_store
    ├── manage.py
    └── puppy_store
        ├── __init__.py
        ├── settings.py
        ├── urls.py
        └── wsgi.py

Django App and REST Framework Setup

Start by creating the puppies app and installing REST Framework inside your virtualenv:

(env)$ cd puppy_store
(env)$ python manage.py startapp puppies
(env)$ pip install djangorestframework==3.6.2

Now we need to configure our Django project to make use of REST Framework.

First, add the puppies app and rest_framework to the INSTALLED_APPS section within puppy_store/puppy_store/settings.py:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'puppies',
    'rest_framework'
]

Next, define global settings for REST Framework in a single dictionary, again, in the settings.py file:

REST_FRAMEWORK = {
    # Use Django's standard `django.contrib.auth` permissions,
    # or allow read-only access for unauthenticated users.
    'DEFAULT_PERMISSION_CLASSES': [],
    'TEST_REQUEST_DEFAULT_FORMAT': 'json'
}

This allows unrestricted access to the API and sets the default test format to JSON for all requests.

NOTE: Unrestricted access is fine for local development, but in a production environment you may need to restrict access to certain endpoints. Make sure to update this. Review the docs for more information.

Your current project structure should now look like:

└── puppy_store
    ├── manage.py
    ├── puppies
    │   ├── __init__.py
    │   ├── admin.py
    │   ├── apps.py
    │   ├── migrations
    │   │   └── __init__.py
    │   ├── models.py
    │   ├── tests.py
    │   └── views.py
    └── puppy_store
        ├── __init__.py
        ├── settings.py
        ├── urls.py
        └── wsgi.py

Database and Model Setup

Let’s set up the Postgres database and apply all the migrations to it.

NOTE: Feel free to swap out Postgres for the relational database of your choice!

Once you have a working Postgres server on your system, open the Postgres interactive shell and create the database:

$ psql
# CREATE DATABASE puppy_store_drf;
CREATE DATABASE
# \q

Install psycopg2 so that we can interact with the Postgres server via Python:

(env)$ pip install psycopg2==2.7.1

Update the database configuration in settings.py, adding the appropriate username and password:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'puppy_store_drf',
        'USER': '<your-user>',
        'PASSWORD': '<your-password>',
        'HOST': '127.0.0.1',
        'PORT': '5432'
    }
}

Next, define a puppy model with some basic attributes in django-puppy-store/puppy_store/puppies/models.py:

from django.db import models


class Puppy(models.Model):
    """
    Puppy Model
    Defines the attributes of a puppy
    """
    name = models.CharField(max_length=255)
    age = models.IntegerField()
    breed = models.CharField(max_length=255)
    color = models.CharField(max_length=255)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    def get_breed(self):
        return self.name + ' belongs to ' + self.breed + ' breed.'

    def __repr__(self):
        return self.name + ' is added.'

Now apply the migration:

(env)$ python manage.py makemigrations
(env)$ python manage.py migrate

Sanity Check

Hop into psql again and verify that the puppies_puppy has been created:

$ psql
# \c puppy_store_drf
You are now connected to database "puppy_store_drf".
puppy_store_drf=# \dt
                      List of relations
 Schema |            Name            | Type  |     Owner
--------+----------------------------+-------+----------------
 public | auth_group                 | table | michael.herman
 public | auth_group_permissions     | table | michael.herman
 public | auth_permission            | table | michael.herman
 public | auth_user                  | table | michael.herman
 public | auth_user_groups           | table | michael.herman
 public | auth_user_user_permissions | table | michael.herman
 public | django_admin_log           | table | michael.herman
 public | django_content_type        | table | michael.herman
 public | django_migrations          | table | michael.herman
 public | django_session             | table | michael.herman
 public | puppies_puppy              | table | michael.herman
(11 rows)

NOTE: You can run \d+ puppies_puppy if you want to look at the actual table details.

Before moving on, let’s write a quick unit test for the Puppy model.

Add the following code to a new file called test_models.py in a new folder called “tests” within “django-puppy-store/puppy_store/puppies”:

from django.test import TestCase
from ..models import Puppy


class PuppyTest(TestCase):
    """ Test module for Puppy model """

    def setUp(self):
        Puppy.objects.create(
            name='Casper', age=3, breed='Bull Dog', color='Black')
        Puppy.objects.create(
            name='Muffin', age=1, breed='Gradane', color='Brown')

    def test_puppy_breed(self):
        puppy_casper = Puppy.objects.get(name='Casper')
        puppy_muffin = Puppy.objects.get(name='Muffin')
        self.assertEqual(
            puppy_casper.get_breed(), "Casper belongs to Bull Dog breed.")
        self.assertEqual(
            puppy_muffin.get_breed(), "Muffin belongs to Gradane breed.")

In the above test, we added dummy entries into our puppy table via the setUp() method from django.test.TestCase and asserted that the get_breed() method returned the correct string.

Add an __init__.py file to “tests” and remove the tests.py file from “django-puppy-store/puppy_store/puppies”.

Let’s run our first test:

(env)$ python manage.py test
Creating test database for alias 'default'...
.
----------------------------------------------------------------------
Ran 1 test in 0.007s

OK
Destroying test database for alias 'default'...

Great! Our first unit test has passed!

Serializers

Before moving on to creating the actual API, let’s define a serializer for our Puppy model which validates the model querysets and produces Pythonic data types to work with.

Add the following snippet to django-puppy-store/puppy_store/puppies/serializers.py:

from rest_framework import serializers
from .models import Puppy


class PuppySerializer(serializers.ModelSerializer):
    class Meta:
        model = Puppy
        fields = ('name', 'age', 'breed', 'color', 'created_at', 'updated_at')

In the above snippet we defined a ModelSerializer for our puppy model, validating all the mentioned fields. In short, if you have a one-to-one relationship between your API endpoints and your models - which you probably should if you’re creating a RESTful API - then you can use a ModelSerializer to create a Serializer.

With our database in place, we can now start building the RESTful API…

RESTful Structure

In a RESTful API, endpoints (URLs) define the structure of the API and how end users access data from our application using the HTTP methods - GET, POST, PUT, DELETE. Endpoints should be logically organized around collections and elements, both of which are resources.

In our case, we have one single resource, puppies, so we will use the following URLS - /puppies/ and /puppies/<id> for collections and elements, respectively:

Endpoint	HTTP Method	CRUD Method	Result
`puppies`	GET	READ	Get all puppies
`puppies/:id`	GET	READ	Get a single puppy
`puppies`	POST	CREATE	Add a single puppy
`puppies/:id`	PUT	UPDATE	Update a single puppy
`puppies/:id`	DELETE	DELETE	Delete a single puppy

Routes and Testing (TDD)

We will be taking a test-first approach rather than a thorough test-driven approach, wherein we will be going through the following process:

add a unit test, just enough code to fail
then update the code to make it pass the test.

Once the test passes, start over with the same process for the new test.

Begin by creating a new file, django-puppy-store/puppy_store/puppies/tests/test_views.py, to hold all the tests for our views and create a new test client for our app:

import json
from rest_framework import status
from django.test import TestCase, Client
from django.urls import reverse
from ..models import Puppy
from ..serializers import PuppySerializer


# initialize the APIClient app
client = Client()

Before starting with all the API routes, let’s first create a skeleton of all view functions that return empty responses and map them with their appropriate URLs within the django-puppy-store/puppy_store/puppies/views.py file:

from rest_framework.decorators import api_view
from rest_framework.response import Response
from rest_framework import status
from .models import Puppy
from .serializers import PuppySerializer


@api_view(['GET', 'DELETE', 'PUT'])
def get_delete_update_puppy(request, pk):
    try:
        puppy = Puppy.objects.get(pk=pk)
    except Puppy.DoesNotExist:
        return Response(status=status.HTTP_404_NOT_FOUND)

    # get details of a single puppy
    if request.method == 'GET':
        return Response({})
    # delete a single puppy
    elif request.method == 'DELETE':
        return Response({})
    # update details of a single puppy
    elif request.method == 'PUT':
        return Response({})


@api_view(['GET', 'POST'])
def get_post_puppies(request):
    # get all puppies
    if request.method == 'GET':
        return Response({})
    # insert a new record for a puppy
    elif request.method == 'POST':
        return Response({})

Create the respective URLs to match the views in django-puppy-store/puppy_store/puppies/urls.py:

from django.conf.urls import url
from . import views


urlpatterns = [
    url(
        r'^api/v1/puppies/(?P<pk>[0-9]+)$',
        views.get_delete_update_puppy,
        name='get_delete_update_puppy'
    ),
    url(
        r'^api/v1/puppies/$',
        views.get_post_puppies,
        name='get_post_puppies'
    )
]

Update django-puppy-store/puppy_store/puppy_store/urls.py as well:

from django.conf.urls import include, url
from django.contrib import admin

urlpatterns = [
    url(r'^', include('puppies.urls')),
    url(
        r'^api-auth/',
        include('rest_framework.urls', namespace='rest_framework')
    ),
    url(r'^admin/', admin.site.urls),
]

Browsable API

With all routes now wired up with the view functions, let’s open up REST Framework’s Browsable API interface and verify whether all the URLs are working as expected.

First, fire up the development server:

(env)$ python manage.py runserver

Make sure to comment out all the attributes in REST_FRAMEWORK section of our settings.py file, to bypass login. Now visit http://localhost:8000/api/v1/puppies

You will see an interactive HTML layout for the API response. Similarly we can test the other URLs and verify all URLs are working perfectly fine.

Let’s start with our unit tests for each route.

Routes

GET ALL

Start with a test to verify the fetched records:

class GetAllPuppiesTest(TestCase):
    """ Test module for GET all puppies API """

    def setUp(self):
        Puppy.objects.create(
            name='Casper', age=3, breed='Bull Dog', color='Black')
        Puppy.objects.create(
            name='Muffin', age=1, breed='Gradane', color='Brown')
        Puppy.objects.create(
            name='Rambo', age=2, breed='Labrador', color='Black')
        Puppy.objects.create(
            name='Ricky', age=6, breed='Labrador', color='Brown')

    def test_get_all_puppies(self):
        # get API response
        response = client.get(reverse('get_post_puppies'))
        # get data from db
        puppies = Puppy.objects.all()
        serializer = PuppySerializer(puppies, many=True)
        self.assertEqual(response.data, serializer.data)
        self.assertEqual(response.status_code, status.HTTP_200_OK)

Run the test. You should see the following error:

self.assertEqual(response.data, serializer.data)
AssertionError: {} != [OrderedDict([('name', 'Casper'), ('age',[687 chars])])]

Update the view to get the test to pass.

@api_view(['GET', 'POST'])
def get_post_puppies(request):
    # get all puppies
    if request.method == 'GET':
        puppies = Puppy.objects.all()
        serializer = PuppySerializer(puppies, many=True)
        return Response(serializer.data)
    # insert a new record for a puppy
    elif request.method == 'POST':
        return Response({})

Here, we get all the records for puppies and validate each using the PuppySerializer.

Run the tests to ensure they all pass:

Ran 2 tests in 0.072s

OK

GET Single

Fetching a single puppy involves two test cases:

Get valid puppy - e.g., the puppy exists
Get invalid puppy - e.g., the puppy does not exists

Add the tests:

class GetSinglePuppyTest(TestCase):
    """ Test module for GET single puppy API """

    def setUp(self):
        self.casper = Puppy.objects.create(
            name='Casper', age=3, breed='Bull Dog', color='Black')
        self.muffin = Puppy.objects.create(
            name='Muffin', age=1, breed='Gradane', color='Brown')
        self.rambo = Puppy.objects.create(
            name='Rambo', age=2, breed='Labrador', color='Black')
        self.ricky = Puppy.objects.create(
            name='Ricky', age=6, breed='Labrador', color='Brown')

    def test_get_valid_single_puppy(self):
        response = client.get(
            reverse('get_delete_update_puppy', kwargs={'pk': self.rambo.pk}))
        puppy = Puppy.objects.get(pk=self.rambo.pk)
        serializer = PuppySerializer(puppy)
        self.assertEqual(response.data, serializer.data)
        self.assertEqual(response.status_code, status.HTTP_200_OK)

    def test_get_invalid_single_puppy(self):
        response = client.get(
            reverse('get_delete_update_puppy', kwargs={'pk': 30}))
        self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)

Run the tests. You should see the following error:

self.assertEqual(response.data, serializer.data)
AssertionError: {} != {'name': 'Rambo', 'age': 2, 'breed': 'Labr[109 chars]26Z'}

Update the view:

@api_view(['GET', 'UPDATE', 'DELETE'])
def get_delete_update_puppy(request, pk):
    try:
        puppy = Puppy.objects.get(pk=pk)
    except Puppy.DoesNotExist:
        return Response(status=status.HTTP_404_NOT_FOUND)

    # get details of a single puppy
    if request.method == 'GET':
        serializer = PuppySerializer(puppy)
        return Response(serializer.data)

In the above snippet, we get the puppy using an ID. Run the tests to ensure they all pass.

POST

Inserting a new record involves two cases as well:

Inserting a valid puppy
Inserting a invalid puppy

First, write tests for it:

class CreateNewPuppyTest(TestCase):
    """ Test module for inserting a new puppy """

    def setUp(self):
        self.valid_payload = {
            'name': 'Muffin',
            'age': 4,
            'breed': 'Pamerion',
            'color': 'White'
        }
        self.invalid_payload = {
            'name': '',
            'age': 4,
            'breed': 'Pamerion',
            'color': 'White'
        }

    def test_create_valid_puppy(self):
        response = client.post(
            reverse('get_post_puppies'),
            data=json.dumps(self.valid_payload),
            content_type='application/json'
        )
        self.assertEqual(response.status_code, status.HTTP_201_CREATED)

    def test_create_invalid_puppy(self):
        response = client.post(
            reverse('get_post_puppies'),
            data=json.dumps(self.invalid_payload),
            content_type='application/json'
        )
        self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)

Run the tests. You should see two failures:

self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
AssertionError: 200 != 400

self.assertEqual(response.status_code, status.HTTP_201_CREATED)
AssertionError: 200 != 201

Again, update the view to get the tests to pass:

@api_view(['GET', 'POST'])
def get_post_puppies(request):
    # get all puppies
    if request.method == 'GET':
        puppies = Puppy.objects.all()
        serializer = PuppySerializer(puppies, many=True)
        return Response(serializer.data)
    # insert a new record for a puppy
    if request.method == 'POST':
        data = {
            'name': request.data.get('name'),
            'age': int(request.data.get('age')),
            'breed': request.data.get('breed'),
            'color': request.data.get('color')
        }
        serializer = PuppySerializer(data=data)
        if serializer.is_valid():
            serializer.save()
            return Response(serializer.data, status=status.HTTP_201_CREATED)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

Here, we inserted a new record by serializing and validating the request data before inserting to the database.

Run the tests again to ensure they pass.

You can also test this out with the Browsable API. Fire up the development server again, and navigate to http://localhost:8000/api/v1/puppies/. Then, within the POST form, submit the following as application/json:

{
    "name": "Muffin",
    "age": 4,
    "breed": "Pamerion",
    "color": "White"
}

Be sure the GET ALL and Get Single work as well.

PUT

Start with a test to update a record. Similar to adding a record, we again need to test for both valid and invalid updates:

class UpdateSinglePuppyTest(TestCase):
    """ Test module for updating an existing puppy record """

    def setUp(self):
        self.casper = Puppy.objects.create(
            name='Casper', age=3, breed='Bull Dog', color='Black')
        self.muffin = Puppy.objects.create(
            name='Muffy', age=1, breed='Gradane', color='Brown')
        self.valid_payload = {
            'name': 'Muffy',
            'age': 2,
            'breed': 'Labrador',
            'color': 'Black'
        }
        self.invalid_payload = {
            'name': '',
            'age': 4,
            'breed': 'Pamerion',
            'color': 'White'
        }

    def test_valid_update_puppy(self):
        response = client.put(
            reverse('get_delete_update_puppy', kwargs={'pk': self.muffin.pk}),
            data=json.dumps(self.valid_payload),
            content_type='application/json'
        )
        self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)

    def test_invalid_update_puppy(self):
        response = client.put(
            reverse('get_delete_update_puppy', kwargs={'pk': self.muffin.pk}),
            data=json.dumps(self.invalid_payload),
            content_type='application/json')
        self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)

Run the tests.

self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
AssertionError: 405 != 400

self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)
AssertionError: 405 != 204

Update the view:

@api_view(['GET', 'DELETE', 'PUT'])
def get_delete_update_puppy(request, pk):
    try:
        puppy = Puppy.objects.get(pk=pk)
    except Puppy.DoesNotExist:
        return Response(status=status.HTTP_404_NOT_FOUND)

    # get details of a single puppy
    if request.method == 'GET':
        serializer = PuppySerializer(puppy)
        return Response(serializer.data)

    # update details of a single puppy
    if request.method == 'PUT':
        serializer = PuppySerializer(puppy, data=request.data)
        if serializer.is_valid():
            serializer.save()
            return Response(serializer.data, status=status.HTTP_204_NO_CONTENT)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

    # delete a single puppy
    elif request.method == 'DELETE':
        return Response({})

In the above snippet, similar to an insert, we serialize and validate the request data and then respond appropriately.

Run the tests again to ensure that all the tests pass.

DELETE

To delete a single record, an ID is required:

class DeleteSinglePuppyTest(TestCase):
    """ Test module for deleting an existing puppy record """

    def setUp(self):
        self.casper = Puppy.objects.create(
            name='Casper', age=3, breed='Bull Dog', color='Black')
        self.muffin = Puppy.objects.create(
            name='Muffy', age=1, breed='Gradane', color='Brown')

    def test_valid_delete_puppy(self):
        response = client.delete(
            reverse('get_delete_update_puppy', kwargs={'pk': self.muffin.pk}))
        self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)

    def test_invalid_delete_puppy(self):
        response = client.delete(
            reverse('get_delete_update_puppy', kwargs={'pk': 30}))
        self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)

Run the tests. You should see:

self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)
AssertionError: 200 != 204

Update the view:

@api_view(['GET', 'DELETE', 'PUT'])
def get_delete_update_puppy(request, pk):
    try:
        puppy = Puppy.objects.get(pk=pk)
    except Puppy.DoesNotExist:
        return Response(status=status.HTTP_404_NOT_FOUND)

    # get details of a single puppy
    if request.method == 'GET':
        serializer = PuppySerializer(puppy)
        return Response(serializer.data)

    # update details of a single puppy
    if request.method == 'PUT':
        serializer = PuppySerializer(puppy, data=request.data)
        if serializer.is_valid():
            serializer.save()
            return Response(serializer.data, status=status.HTTP_204_NO_CONTENT)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

    # delete a single puppy
    if request.method == 'DELETE':
        puppy.delete()
        return Response(status=status.HTTP_204_NO_CONTENT)

Run the tests again. Make sure all of them pass. Make sure to test out the UPDATE and DELETE functionality within the Browsable API as well!

Conclusion and Next Steps

In this tutorial, we went through the process of creating a RESTful API using Django REST Framework with a test-first approach.

Free Bonus: Click here to download a copy of the "REST API Examples" Guide and get a hands-on introduction to Python + REST API principles with actionable examples.

What’s next? To make our RESTful API robust and secure, we can implement permissions and throttling for a production environment to allow restricted access on the basis of authentication credentials and rate limiting to avoid any sort of DDoS attack. Also, don’t forget to prevent the Browsable API from being accessible in a production environment.

Feel free to share your comments, questions, or tips in the comments below. The full code can be found in the django-puppy-store repository.

Python's Instance, Class, and Static Methods Demystified

2017-03-29T13:02:44+00:00

In this tutorial I’ll help demystify what’s behind class methods, static methods, and regular instance methods.

If you develop an intuitive understanding for their differences you’ll be able to write object-oriented Python that communicates its intent more clearly and will be easier to maintain in the long run.

Free Bonus: Click here to get access to a free Python OOP Cheat Sheet that points you to the best tutorials, videos, and books to learn more about Object-Oriented Programming with Python.

Instance, Class, and Static Methods — An Overview

Let’s begin by writing a (Python 3) class that contains simple examples for all three method types:

class MyClass:
    def method(self):
        return 'instance method called', self

    @classmethod
    def classmethod(cls):
        return 'class method called', cls

    @staticmethod
    def staticmethod():
        return 'static method called'

NOTE: For Python 2 users: The @staticmethod and @classmethod decorators are available as of Python 2.4 and this example will work as is. Instead of using a plain class MyClass: declaration you might choose to declare a new-style class inheriting from object with the class MyClass(object): syntax. Other than that you’re good to go.

Instance Methods

The first method on MyClass, called method, is a regular instance method. That’s the basic, no-frills method type you’ll use most of the time. You can see the method takes one parameter, self, which points to an instance of MyClass when the method is called (but of course instance methods can accept more than just one parameter).

Through the self parameter, instance methods can freely access attributes and other methods on the same object. This gives them a lot of power when it comes to modifying an object’s state.

Not only can they modify object state, instance methods can also access the class itself through the self.__class__ attribute. This means instance methods can also modify class state.

Class Methods

Let’s compare that to the second method, MyClass.classmethod. I marked this method with a @classmethod decorator to flag it as a class method.

Instead of accepting a self parameter, class methods take a cls parameter that points to the class—and not the object instance—when the method is called.

Because the class method only has access to this cls argument, it can’t modify object instance state. That would require access to self. However, class methods can still modify class state that applies across all instances of the class.

Static Methods

The third method, MyClass.staticmethod was marked with a @staticmethod decorator to flag it as a static method.

This type of method takes neither a self nor a cls parameter (but of course it’s free to accept an arbitrary number of other parameters).

Therefore a static method can neither modify object state nor class state. Static methods are restricted in what data they can access - and they’re primarily a way to namespace your methods.

Let’s See Them In Action!

I know this discussion has been fairly theoretical up to this point. And I believe it’s important that you develop an intuitive understanding for how these method types differ in practice. We’ll go over some concrete examples now.

Let’s take a look at how these methods behave in action when we call them. We’ll start by creating an instance of the class and then calling the three different methods on it.

MyClass was set up in such a way that each method’s implementation returns a tuple containing information for us to trace what’s going on — and which parts of the class or object the method can access.

Here’s what happens when we call an instance method:

>>> obj = MyClass()
>>> obj.method()
('instance method called', <MyClass instance at 0x101a2f4c8>)

This confirmed that method (the instance method) has access to the object instance (printed as <MyClass instance>) via the self argument.

When the method is called, Python replaces the self argument with the instance object, obj. We could ignore the syntactic sugar of the dot-call syntax (obj.method()) and pass the instance object manually to get the same result:

>>> MyClass.method(obj)
('instance method called', <MyClass instance at 0x101a2f4c8>)

Can you guess what would happen if you tried to call the method without first creating an instance?

By the way, instance methods can also access the class itself through the self.__class__ attribute. This makes instance methods powerful in terms of access restrictions - they can modify state on the object instance and on the class itself.

Let’s try out the class method next:

>>> obj.classmethod()
('class method called', <class MyClass at 0x101a2f4c8>)

Calling classmethod() showed us it doesn’t have access to the <MyClass instance> object, but only to the <class MyClass> object, representing the class itself (everything in Python is an object, even classes themselves).

Notice how Python automatically passes the class as the first argument to the function when we call MyClass.classmethod(). Calling a method in Python through the dot syntax triggers this behavior. The self parameter on instance methods works the same way.

Please note that naming these parameters self and cls is just a convention. You could just as easily name them the_object and the_class and get the same result. All that matters is that they’re positioned first in the parameter list for the method.

Time to call the static method now:

>>> obj.staticmethod()
'static method called'

Did you see how we called staticmethod() on the object and were able to do so successfully? Some developers are surprised when they learn that it’s possible to call a static method on an object instance.

Behind the scenes Python simply enforces the access restrictions by not passing in the self or the cls argument when a static method gets called using the dot syntax.

This confirms that static methods can neither access the object instance state nor the class state. They work like regular functions but belong to the class’s (and every instance’s) namespace.

Now, let’s take a look at what happens when we attempt to call these methods on the class itself - without creating an object instance beforehand:

>>> MyClass.classmethod()
('class method called', <class MyClass at 0x101a2f4c8>)

>>> MyClass.staticmethod()
'static method called'

>>> MyClass.method()
TypeError: unbound method method() must
    be called with MyClass instance as first
    argument (got nothing instead)

We were able to call classmethod() and staticmethod() just fine, but attempting to call the instance method method() failed with a TypeError.

And this is to be expected — this time we didn’t create an object instance and tried calling an instance function directly on the class blueprint itself. This means there is no way for Python to populate the self argument and therefore the call fails.

This should make the distinction between these three method types a little more clear. But I’m not going to leave it at that. In the next two sections I’ll go over two slightly more realistic examples for when to use these special method types.

I will base my examples around this bare-bones Pizza class:

class Pizza:
    def __init__(self, ingredients):
        self.ingredients = ingredients

    def __repr__(self):
        return f'Pizza({self.ingredients!r})'

>>> Pizza(['cheese', 'tomatoes'])
Pizza(['cheese', 'tomatoes'])

Delicious Pizza Factories With `@classmethod`

If you’ve had any exposure to pizza in the real world you’ll know that there are many delicious variations available:

Pizza(['mozzarella', 'tomatoes'])
Pizza(['mozzarella', 'tomatoes', 'ham', 'mushrooms'])
Pizza(['mozzarella'] * 4)

The Italians figured out their pizza taxonomy centuries ago, and so these delicious types of pizzas all have their own names. We’d do well to take advantage of that and give the users of our Pizza class a better interface for creating the pizza objects they crave.

A nice and clean way to do that is by using class methods as factory functions for the different kinds of pizzas we can create:

class Pizza:
    def __init__(self, ingredients):
        self.ingredients = ingredients

    def __repr__(self):
        return f'Pizza({self.ingredients!r})'

    @classmethod
    def margherita(cls):
        return cls(['mozzarella', 'tomatoes'])

    @classmethod
    def prosciutto(cls):
        return cls(['mozzarella', 'tomatoes', 'ham'])

Note how I’m using the cls argument in the margherita and prosciutto factory methods instead of calling the Pizza constructor directly.

This is a trick you can use to follow the Don’t Repeat Yourself (DRY) principle. If we decide to rename this class at some point we won’t have to remember updating the constructor name in all of the classmethod factory functions.

Now, what can we do with these factory methods? Let’s try them out:

>>> Pizza.margherita()
Pizza(['mozzarella', 'tomatoes'])

>>> Pizza.prosciutto()
Pizza(['mozzarella', 'tomatoes', 'ham'])

As you can see, we can use the factory functions to create new Pizza objects that are configured the way we want them. They all use the same __init__ constructor internally and simply provide a shortcut for remembering all of the various ingredients.

Another way to look at this use of class methods is that they allow you to define alternative constructors for your classes.

Python only allows one __init__ method per class. Using class methods it’s possible to add as many alternative constructors as necessary. This can make the interface for your classes self-documenting (to a certain degree) and simplify their usage.

When To Use Static Methods

It’s a little more difficult to come up with a good example here. But tell you what, I’ll just keep stretching the pizza analogy thinner and thinner… (yum!)

Here’s what I came up with:

import math

class Pizza:
    def __init__(self, radius, ingredients):
        self.radius = radius
        self.ingredients = ingredients

    def __repr__(self):
        return (f'Pizza({self.radius!r}, '
                f'{self.ingredients!r})')

    def area(self):
        return self.circle_area(self.radius)

    @staticmethod
    def circle_area(r):
        return r ** 2 * math.pi

Now what did I change here? First, I modified the constructor and __repr__ to accept an extra radius argument.

I also added an area() instance method that calculates and returns the pizza’s area (this would also be a good candidate for an @property — but hey, this is just a toy example).

Instead of calculating the area directly within area(), using the well-known circle area formula, I factored that out to a separate circle_area() static method.

Let’s try it out!

>>> p = Pizza(4, ['mozzarella', 'tomatoes'])
>>> p
Pizza(4, ['mozzarella', 'tomatoes'])
>>> p.area()
50.26548245743669
>>> Pizza.circle_area(4)
50.26548245743669

Sure, this is a bit of a simplistic example, but it’ll do alright helping explain some of the benefits that static methods provide.

As we’ve learned, static methods can’t access class or instance state because they don’t take a cls or self argument. That’s a big limitation — but it’s also a great signal to show that a particular method is independent from everything else around it.

In the above example, it’s clear that circle_area() can’t modify the class or the class instance in any way. (Sure, you could always work around that with a global variable but that’s not the point here.)

Now, why is that useful?

Flagging a method as a static method is not just a hint that a method won’t modify class or instance state — this restriction is also enforced by the Python runtime.

Techniques like that allow you to communicate clearly about parts of your class architecture so that new development work is naturally guided to happen within these set boundaries. Of course, it would be easy enough to defy these restrictions. But in practice they often help avoid accidental modifications going against the original design.

Put differently, using static methods and class methods are ways to communicate developer intent while enforcing that intent enough to avoid most slip of the mind mistakes and bugs that would break the design.

Applied sparingly and when it makes sense, writing some of your methods that way can provide maintenance benefits and make it less likely that other developers use your classes incorrectly.

Static methods also have benefits when it comes to writing test code.

Because the circle_area() method is completely independent from the rest of the class it’s much easier to test.

We don’t have to worry about setting up a complete class instance before we can test the method in a unit test. We can just fire away like we would testing a regular function. Again, this makes future maintenance easier.

Key Takeaways

Instance methods need a class instance and can access the instance through self.
Class methods don’t need a class instance. They can’t access the instance (self) but they have access to the class itself via cls.
Static methods don’t have access to cls or self. They work like regular functions but belong to the class’s namespace.
Static and class methods communicate and (to a certain degree) enforce developer intent about class design. This can have maintenance benefits.

Getting Started with Django Channels

2017-02-28T22:28:46+00:00

In this tutorial, we will use Django Channels to create a real-time application that updates a list of users as they log in and out.

With WebSockets (via Django Channels) managing the communication between the client and the server, whenever a user is authenticated, an event will be broadcasted to every other connected user. Each user’s screen will change automatically, without them having to reload their browsers.

NOTE: We recommend that you have some experience with Django before beginning this tutorial. Also, you should be familiar with the concept of WebSockets.

Free Bonus: Click here to get access to a free list of 35 additional Django tutorials and resources you can use to deepen your Python web development skills.

Our application uses:

Python (v3.6.0)
Django (v1.10.5)
Django Channels (v1.0.3)
Redis (v3.2.8)

Objectives

By the end of this tutorial, you will be able to…

Add Web sockets support to a Django project via Django Channels
Set up a simple connection between Django and a Redis server
Implement basic user authentication
Leverage Django Signals to take action when a user logs in or out

Getting Started

First, create a new virtual environment to isolate our project’s dependencies:

$ mkdir django-example-channels
$ cd django-example-channels
$ python3.6 -m venv env
$ source env/bin/activate
(env)$

Install Django, Django Channels, and ASGI Redis, and then create a new Django project and app:

(env)$ pip install django==1.10.5 channels==1.0.2 asgi_redis==1.0.0
(env)$ django-admin.py startproject example_channels
(env)$ cd example_channels
(env)$ python manage.py startapp example
(env)$ python manage.py migrate

NOTE: During the course of this tutorial, we will create a variety of different files and folders. Please refer to the folder structure from the project’s repository if you get stuck.

Next, download and install Redis. If you’re on a Mac, we recommend using Homebrew:

$ brew install redis

Start the Redis server in a new terminal window and make sure that it is running on its default port, 6379. The port number will be important when we tell Django how to communicate with Redis.

Complete the setup by updating INSTALLED_APPS in the project’s settings.py file:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'channels',
    'example',
]

Then Configure the CHANNEL_LAYERS by setting a default backend and routing:

CHANNEL_LAYERS = {
    'default': {
        'BACKEND': 'asgi_redis.RedisChannelLayer',
        'CONFIG': {
            'hosts': [('localhost', 6379)],
        },
        'ROUTING': 'example_channels.routing.channel_routing',
    }
}

This uses a Redis backend which is also needed in production.

WebSockets 101

Normally, Django uses HTTP to communicate between the client and server:

The client sends an HTTP request to the server.
Django parses the request, extracts a URL, and then matches it to a view.
The view processes the request and returns an HTTP response to the client.

Unlike HTTP, the WebSockets protocol allows bi-directional communication, meaning that the server can push data to the client without being prompted by the user. With HTTP, only the client that made a request receives a response. With WebSockets, the server can communicate with multiple clients simultaneously. As we will see later on in this tutorial, we send WebSockets messages using the ws:// prefix, as opposed to http://.

NOTE: Before diving in, quickly review the Channels Concepts documentation.

Consumers and Groups

Let’s create our first consumer, which handle the basic connections between the client and the server. Create a new file called example_channels/example/consumers.py:

from channels import Group


def ws_connect(message):
    Group('users').add(message.reply_channel)


def ws_disconnect(message):
    Group('users').discard(message.reply_channel)

Consumers are the counterpart to Django views. Any user connecting to our app will be added to the “users” group and will receive messages sent by the server. When the client disconnects from our app, the channel is removed from the group, and the user will stop receiving messages.

Next, let’s set up routes, which work in almost the same manner as Django URL configuration, by adding the following code to a new file called example_channels/routing.py:

from channels.routing import route
from example.consumers import ws_connect, ws_disconnect


channel_routing = [
    route('websocket.connect', ws_connect),
    route('websocket.disconnect', ws_disconnect),
]

So, we defined channel_routing instead of urlpatterns and route() instead of url(). Notice that we linked our consumer functions to WebSockets.

Templates

Let’s write up some HTML that can communicate with our server via a WebSocket. Create a “templates” folder within “example” and then add an “example” folder within “templates” - “example_channels/example/templates/example”.

Add a _base.html file:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0">
  <meta http-equiv="X-UA-Compatible" content="ie=edge">
  <link href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet">
  <title>Example Channels</title>
</head>
<body>
  <div class="container">
    <br>
    {% block content %}{% endblock content %}
  </div>
  <script src="//code.jquery.com/jquery-3.1.1.min.js"></script>
  {% block script %}{% endblock script %}
</body>
</html>

And user_list.html:

{% extends 'example/_base.html' %}

{% block content %}{% endblock content %}

{% block script %}
  <script>
    var socket = new WebSocket('ws://' + window.location.host + '/users/');

    socket.onopen = function open() {
      console.log('WebSockets connection created.');
    };

    if (socket.readyState == WebSocket.OPEN) {
      socket.onopen();
    }
  </script>
{% endblock script %}

Now, when the client successfully opens a connection with the server using a WebSocket, we will see a confirmation message print to the console.

Views

Set up a supporting Django view to render our template within example_channels/example/views.py:

from django.shortcuts import render


def user_list(request):
    return render(request, 'example/user_list.html')

Add the URL to example_channels/example/urls.py:

from django.conf.urls import url
from example.views import user_list


urlpatterns = [
    url(r'^$', user_list, name='user_list'),
]

Update the project URL as well in example_channels/example_channels/urls.py:

from django.conf.urls import include, url
from django.contrib import admin

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^', include('example.urls', namespace='example')),
]

Test

Ready to test?

(env)$ python manage.py runserver

NOTE: You can alternatively run python manage.py runserver --noworker and python manage.py runworker in two different terminals to test the interface and worker servers as two separate processes. Both methods work!

When you visit http://localhost:8000/, you should see the connection message print to the terminal:

[2017/02/19 23:24:57] HTTP GET / 200 [0.02, 127.0.0.1:52757]
[2017/02/19 23:24:58] WebSocket HANDSHAKING /users/ [127.0.0.1:52789]
[2017/02/19 23:25:03] WebSocket DISCONNECT /users/ [127.0.0.1:52789]

User Authentication

Now that we have proven that we can open a connection, our next step is to handle user authentication. Remember: We want a user to be able to log into our app and see a list of all of the other users who are subscribed to that user’s group. First, we need a way for users to create accounts and log in. Begin by creating a simple login page that will allow a user to authenticate with a username and password.

Create a new file called log_in.html within “example_channels/example/templates/example”:

{% extends 'example/_base.html' %}

{% block content %}
  <form action="{% url 'example:log_in' %}" method="post">
    {% csrf_token %}
    {% for field in form %}
      <div>
        {{ field.label_tag }}
        {{ field }}
      </div>
    {% endfor %}
    <button type="submit">Log in</button>
  </form>
  <p>Don't have an account? <a href="{% url 'example:sign_up' %}">Sign up!</a></p>
{% endblock content %}

Next, update example_channels/example/views.py like so:

from django.contrib.auth import login, logout
from django.contrib.auth.forms import AuthenticationForm
from django.core.urlresolvers import reverse
from django.shortcuts import render, redirect


def user_list(request):
    return render(request, 'example/user_list.html')


def log_in(request):
    form = AuthenticationForm()
    if request.method == 'POST':
        form = AuthenticationForm(data=request.POST)
        if form.is_valid():
            login(request, form.get_user())
            return redirect(reverse('example:user_list'))
        else:
            print(form.errors)
    return render(request, 'example/log_in.html', {'form': form})


def log_out(request):
    logout(request)
    return redirect(reverse('example:log_in'))

Django comes with forms that support common authentication functionality. We can use the AuthenticationForm to handle user login. This form checks the supplied username and password, then returns a User object if a validated user is found. We log in the validated user and redirect them to our homepage. A user should also have to ability to log out of the application, so we create a logout view that provides that functionality and then takes the user back to the login screen.

Then update example_channels/example/urls.py:

from django.conf.urls import url
from example.views import log_in, log_out, user_list


urlpatterns = [
    url(r'^log_in/$', log_in, name='log_in'),
    url(r'^log_out/$', log_out, name='log_out'),
    url(r'^$', user_list, name='user_list')
]

We also need a way to create new users. Create a sign-up page in the same manner as the login by adding a new file called sign_up.html to “example_channels/example/templates/example”:

{% extends 'example/_base.html' %}

{% block content %}
  <form action="{% url 'example:sign_up' %}" method="post">
    {% csrf_token %}
    {% for field in form %}
      <div>
        {{ field.label_tag }}
        {{ field }}
      </div>
    {% endfor %}
    <button type="submit">Sign up</button>
    <p>Already have an account? <a href="{% url 'example:log_in' %}">Log in!</a></p>
  </form>
{% endblock content %}

Notice that the login page has a link to the sign-up page, and the sign-up page has a link back to the login.

Add the following function to the views:

def sign_up(request):
    form = UserCreationForm()
    if request.method == 'POST':
        form = UserCreationForm(data=request.POST)
        if form.is_valid():
            form.save()
            return redirect(reverse('example:log_in'))
        else:
            print(form.errors)
    return render(request, 'example/sign_up.html', {'form': form})

We use another built-in form for user creation. After successful form validation, we redirect to the login page.

Make sure to import the form:

from django.contrib.auth.forms import AuthenticationForm, UserCreationForm

Update example_channels/example/urls.py again:

from django.conf.urls import url
from example.views import log_in, log_out, sign_up, user_list


urlpatterns = [
    url(r'^log_in/$', log_in, name='log_in'),
    url(r'^log_out/$', log_out, name='log_out'),
    url(r'^sign_up/$', sign_up, name='sign_up'),
    url(r'^$', user_list, name='user_list')
]

At this point, we need to create a user. Run the server and visit http://localhost:8000/sign_up/ in your browser. Fill in the form with a valid username and password and submit it to create our first user.

NOTE: Try using michael as the username and johnson123 as the password.

The sign_up view redirects us to the log_in view, and from there we can authenticate our newly created user.

After we log in, we can test our new authentication views.

Use the sign up form to create several new users in preparation for the next section.

We have basic user authentication working, but we still need to display a list of users and we need the server to tell the group when a user logs in and out. We need to edit our consumer functions so that they send a message right after a client connects and right before a client disconnects. The message data will include the user’s username and connection status.

Update example_channels/example/consumers.py like so:

import json
from channels import Group
from channels.auth import channel_session_user, channel_session_user_from_http


@channel_session_user_from_http
def ws_connect(message):
    Group('users').add(message.reply_channel)
    Group('users').send({
        'text': json.dumps({
            'username': message.user.username,
            'is_logged_in': True
        })
    })


@channel_session_user
def ws_disconnect(message):
    Group('users').send({
        'text': json.dumps({
            'username': message.user.username,
            'is_logged_in': False
        })
    })
    Group('users').discard(message.reply_channel)

Notice that we have added decorators to the functions to get the user from the Django session. Also, all messages must be JSON-serializable, so we dump our data into a JSON string.

Next, update example_channels/example/templates/example/user_list.html:

{% extends 'example/_base.html' %}

{% block content %}
  <a href="{% url 'example:log_out' %}">Log out</a>
  <br>
  <ul>
    {% for user in users %}
      <!-- NOTE: We escape HTML to prevent XSS attacks. -->
      <li data-username="{{ user.username|escape }}">
        {{ user.username|escape }}: {{ user.status|default:'Offline' }}
      </li>
    {% endfor %}
  </ul>
{% endblock content %}

{% block script %}
  <script>
    var socket = new WebSocket('ws://' + window.location.host + '/users/');

    socket.onopen = function open() {
      console.log('WebSockets connection created.');
    };

    socket.onmessage = function message(event) {
      var data = JSON.parse(event.data);
      // NOTE: We escape JavaScript to prevent XSS attacks.
      var username = encodeURI(data['username']);
      var user = $('li').filter(function () {
        return $(this).data('username') == username;
      });

      if (data['is_logged_in']) {
        user.html(username + ': Online');
      }
      else {
        user.html(username + ': Offline');
      }
    };

    if (socket.readyState == WebSocket.OPEN) {
      socket.onopen();
    }
  </script>
{% endblock script %}

On our homepage, we expand our user list to display a list of users. We store each user’s username as a data attribute to make it easy to find the user item in the DOM. We also add an event listener to our WebSocket that can handle messages from the server. When we receive a message, we parse the JSON data, find the <li> element for the given user, and update that user’s status.

Django does not track whether a user is logged in, so we need to create a simple model to do that for us. Create a LoggedInUser model with a one-to-one connection to our User model in example_channels/example/models.py:

from django.conf import settings
from django.db import models


class LoggedInUser(models.Model):
    user = models.OneToOneField(
        settings.AUTH_USER_MODEL, related_name='logged_in_user')

Our app will create a LoggedInUser instance when a user logs in, and the app will delete the instance when the user logs out.

Make the schema migration and then migrate our database to apply the changes.

(env)$ python manage.py makemigrations
(env)$ python manage.py migrate

Next, update our user list view, in example_channels/example/views.py, to retrieve a list of users to render:

from django.contrib.auth import get_user_model, login, logout
from django.contrib.auth.decorators import login_required
from django.contrib.auth.forms import AuthenticationForm, UserCreationForm
from django.core.urlresolvers import reverse
from django.shortcuts import render, redirect


User = get_user_model()


@login_required(login_url='/log_in/')
def user_list(request):
    """
    NOTE: This is fine for demonstration purposes, but this should be
    refactored before we deploy this app to production.
    Imagine how 100,000 users logging in and out of our app would affect
    the performance of this code!
    """
    users = User.objects.select_related('logged_in_user')
    for user in users:
        user.status = 'Online' if hasattr(user, 'logged_in_user') else 'Offline'
    return render(request, 'example/user_list.html', {'users': users})


def log_in(request):
    form = AuthenticationForm()
    if request.method == 'POST':
        form = AuthenticationForm(data=request.POST)
        if form.is_valid():
            login(request, form.get_user())
            return redirect(reverse('example:user_list'))
        else:
            print(form.errors)
    return render(request, 'example/log_in.html', {'form': form})


@login_required(login_url='/log_in/')
def log_out(request):
    logout(request)
    return redirect(reverse('example:log_in'))


def sign_up(request):
    form = UserCreationForm()
    if request.method == 'POST':
        form = UserCreationForm(data=request.POST)
        if form.is_valid():
            form.save()
            return redirect(reverse('example:log_in'))
        else:
            print(form.errors)
    return render(request, 'example/sign_up.html', {'form': form})

If a user has an associated LoggedInUser, then we record the user’s status as “Online”, and if not, the user is “Offline”. We also add a @login_required decorator to both our user list and log out views to restrict access only to registered users.

Add the imports as well:

from django.contrib.auth import get_user_model, login, logout
from django.contrib.auth.decorators import login_required

At this point, users can log in and out, which will trigger the server to send messages to the client, but we have no way of knowing which users are logged in when the user first logs in. The user only sees updates when another user’s status changes. This is where the LoggedInUser comes into play, but we need a way to create a LoggedInUser instance when a user logs in, and then delete it when that user logs out.

The Django library includes a feature known as signals that broadcasts notifications when certain actions occur. Applications can listen for those notifications and then act on them. We can exploit two helpful, built-in signals (user_logged_in and user_logged_out) to handle our LoggedInUser behavior.

Within “example_channels/example”, add a new file called signals.py:

from django.contrib.auth import user_logged_in, user_logged_out
from django.dispatch import receiver
from example.models import LoggedInUser


@receiver(user_logged_in)
def on_user_login(sender, **kwargs):
    LoggedInUser.objects.get_or_create(user=kwargs.get('user'))


@receiver(user_logged_out)
def on_user_logout(sender, **kwargs):
    LoggedInUser.objects.filter(user=kwargs.get('user')).delete()

We have to make the signals available in our app configuration, example_channels/example/apps.py:

from django.apps import AppConfig


class ExampleConfig(AppConfig):
    name = 'example'

    def ready(self):
        import example.signals

Update example_channels/example/__init__.py as well:

default_app_config = 'example.apps.ExampleConfig'

Sanity Check

Now we are finished coding and are ready to connect to our server with multiple users to test our app.

Run the Django server, log in as a user, and visit the homepage. We should see a list of all of the users in our app, each with a status of “Offline”. Next, open a new Incognito window and log in as a different user and watch both screens. Right when we log in, the regular browser updates the user status to “Online”. From our Incognito window, we see that the user logged in also has a status of “Online”. We can test the WebSockets by logging in and out on our different devices with various users.

Observing the developer console on the client and the server activity in our terminal, we can confirm that WebSocket connections are being formed when a user logs in and destroyed when a user logs out.

[2017/02/20 00:15:23] HTTP POST /log_in/ 302 [0.07, 127.0.0.1:55393]
[2017/02/20 00:15:23] HTTP GET / 200 [0.04, 127.0.0.1:55393]
[2017/02/20 00:15:23] WebSocket HANDSHAKING /users/ [127.0.0.1:55414]
[2017/02/20 00:15:23] WebSocket CONNECT /users/ [127.0.0.1:55414]
[2017/02/20 00:15:25] HTTP GET /log_out/ 302 [0.01, 127.0.0.1:55393]
[2017/02/20 00:15:26] HTTP GET /log_in/ 200 [0.02, 127.0.0.1:55393]
[2017/02/20 00:15:26] WebSocket DISCONNECT /users/ [127.0.0.1:55414]

NOTE: You could also use ngrok to expose the local server to the internet securely. Doing this will allow you to hit the local server from various devices such as your phone or tablet.

Closing Thoughts

We covered a lot in this tutorial - Django Channels, WebSockets, user authentication, signals, and some front-end development. The main take away is this: Channels extends the functionality of a traditional Django app by letting us push messages from the server to groups of users via WebSockets.

This is powerful stuff!

Think of some of the applications. We can create chat rooms, multiplayer games, and collaborative apps that allow users to communicate in real time. Even mundane tasks are improved with WebSockets. For example, instead of periodically polling the server to see if a long-running task has completed, the server can push a status update to the client when it finishes.

This tutorial just scratches the surface of what we can do with Django Channels, too. Explore the Django Channels documentation and see what else you can create.

Free Bonus: Click here to get access to a free list of 35 additional Django tutorials and resources you can use to deepen your Python web development skills.

Grab the final code from the django-example-channels repo. Cheers!

Token-Based Authentication With Flask

2017-01-24T14:40:05+00:00

This tutorial takes a test-first approach to implementing token-based authentication in a Flask app using JSON Web Tokens (JWTs).

Updates:

08/04/2017: Refactored route handler for the PyBites Challenge.

Objectives

By the end of this tutorial, you will be able to…

Discuss the benefits of using JWTs versus sessions and cookies for authentication
Implement user authentication with JWTs
Blacklist user tokens when necessary
Write tests to create and verify JWTs and user authentication
Practice test-driven development

Free Bonus: Click here to get access to a free Flask + Python video tutorial series that shows you how to build a fully working Flask web app, step-by-step.

Introduction

JSON Web Tokens (or JWTs) provide a means of transmitting information from the client to the server in a stateless, secure way.

On the server, JWTs are generated by signing user information via a secret key, which are then securely stored on the client. This form of auth works well with modern, single page applications. For more on this, along with the pros and cons of using JWTs vs. session and cookie-based auth, please review the following articles:

NOTE: Keep in mind that since a JWT is signed rather than encrypted it should never contain sensitive information like a user’s password.

Getting Started

Enough theory, let’s start implementing some code!

Project Setup

Start by cloning the project boilerplate and then create a new branch:

$ git clone https://github.com/realpython/flask-jwt-auth.git
$ cd flask-jwt-auth
$ git checkout tags/1.0.0 -b jwt-auth

Create and activate a virtualenv and install the dependencies:

$ python3.6 -m venv env
$ source env/bin/activate
(env)$ pip install -r requirements.txt

This is optional, but it’s a good idea to create a new Github repository and update the remote:

(env)$ git remote set-url origin <newurl>

Database Setup

Let’s set up Postgres.

NOTE: If you’re on a Mac, check out Postgres app.

Once the local Postgres server is running, create two new databases from psql that share the same name as your project name:

(env)$ psql
# create database flask_jwt_auth;
CREATE DATABASE
# create database flask_jwt_auth_test;
CREATE DATABASE
# \q

NOTE: There may be some variation on the above commands, for creating a database, based upon your version of Postgres. Check for the correct command in the Postgres documentation.

Before applying the database migrations we need to update the config file found in project/server/config.py. Simply update the database_name:

database_name = 'flask_jwt_auth'

Set the environment variables in the terminal:

(env)$ export APP_SETTINGS="project.server.config.DevelopmentConfig"

Update the following tests in project/tests/test__config.py:

class TestDevelopmentConfig(TestCase):
    def create_app(self):
        app.config.from_object('project.server.config.DevelopmentConfig')
        return app

    def test_app_is_development(self):
        self.assertTrue(app.config['DEBUG'] is True)
        self.assertFalse(current_app is None)
        self.assertTrue(
            app.config['SQLALCHEMY_DATABASE_URI'] == 'postgresql://postgres:@localhost/flask_jwt_auth'
        )


class TestTestingConfig(TestCase):
    def create_app(self):
        app.config.from_object('project.server.config.TestingConfig')
        return app

    def test_app_is_testing(self):
        self.assertTrue(app.config['DEBUG'])
        self.assertTrue(
            app.config['SQLALCHEMY_DATABASE_URI'] == 'postgresql://postgres:@localhost/flask_jwt_auth_test'
        )

Run them to ensure they still pass:

(env)$ python manage.py test

You should see:

test_app_is_development (test__config.TestDevelopmentConfig) ... ok
test_app_is_production (test__config.TestProductionConfig) ... ok
test_app_is_testing (test__config.TestTestingConfig) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.007s

OK

Migrations

Add a models.py file to the “server” directory:

# project/server/models.py

import datetime

from project.server import app, db, bcrypt


class User(db.Model):
    """ User Model for storing user related details """
    __tablename__ = "users"

    id = db.Column(db.Integer, primary_key=True, autoincrement=True)
    email = db.Column(db.String(255), unique=True, nullable=False)
    password = db.Column(db.String(255), nullable=False)
    registered_on = db.Column(db.DateTime, nullable=False)
    admin = db.Column(db.Boolean, nullable=False, default=False)

    def __init__(self, email, password, admin=False):
        self.email = email
        self.password = bcrypt.generate_password_hash(
            password, app.config.get('BCRYPT_LOG_ROUNDS')
        ).decode()
        self.registered_on = datetime.datetime.now()
        self.admin = admin

In the above snippet, we define a basic user model, which uses the Flask-Bcrypt extension to hash the password.

Install psycopg2 to connect to Postgres:

(env)$ pip install psycopg2==2.6.2
(env)$ pip freeze > requirements.txt

Within manage.py change-

from project.server import app, db

To-

from project.server import app, db, models

Apply the migration:

(env)$ python manage.py create_db
(env)$ python manage.py db init
(env)$ python manage.py db migrate

Sanity Check

Did it work?

(env)$ psql
# \c flask_jwt_auth
You are now connected to database "flask_jwt_auth" as user "michael.herman".
# \d

               List of relations
 Schema |      Name       |   Type   |  Owner
--------+-----------------+----------+----------
 public | alembic_version | table    | postgres
 public | users           | table    | postgres
 public | users_id_seq    | sequence | postgres
(3 rows)

JWT Setup

The auth workflow works as follows:

Client provides email and password, which is sent to the server
Server then verifies that email and password are correct and responds with an auth token
Client stores the token and sends it along with all subsequent requests to the API
Server decodes the token and validates it

This cycle repeats until the token expires or is revoked. In the latter case, the server issues a new token.

The tokens themselves are divided into three parts:

Header
Payload
Signature

We’ll dive a bit deeper into the payload, but if you’re curious, you can read more about each part from the Introduction to JSON Web Tokens article.

To work with JSON Web Tokens in our app, install the PyJWT package:

(env)$ pip install pyjwt==1.4.2
(env)$ pip freeze > requirements.txt

Encode Token

Add the following method to the User() class in project/server/models.py:

def encode_auth_token(self, user_id):
    """
    Generates the Auth Token
    :return: string
    """
    try:
        payload = {
            'exp': datetime.datetime.utcnow() + datetime.timedelta(days=0, seconds=5),
            'iat': datetime.datetime.utcnow(),
            'sub': user_id
        }
        return jwt.encode(
            payload,
            app.config.get('SECRET_KEY'),
            algorithm='HS256'
        )
    except Exception as e:
        return e

Don’t forget to add the import:

import jwt

So, given a user id, this method creates and returns a token from the payload and the secret key set in the config.py file. The payload is where we add metadata about the token and information about the user. This info is often referred to as JWT Claims. We utilize the following “claims”:

exp: expiration date of the token
iat: the time the token is generated
sub: the subject of the token (the user whom it identifies)

The secret key must be random and only accessible server-side. Use the Python interpreter to generate a key:

>>> import os
>>> os.urandom(24)
b"\xf9'\xe4p(\xa9\x12\x1a!\x94\x8d\x1c\x99l\xc7\xb7e\xc7c\x86\x02MJ\xa0"

Set the key as an environment variable:

(env)$ export SECRET_KEY="\xf9'\xe4p(\xa9\x12\x1a!\x94\x8d\x1c\x99l\xc7\xb7e\xc7c\x86\x02MJ\xa0"

Add this key to the SECRET_KEY within the BaseConfig() class in project/server/config.py:

SECRET_KEY = os.getenv('SECRET_KEY', 'my_precious')

Update the tests within project/tests/test__config.py to ensure the variable is set correctly:

def test_app_is_development(self):
    self.assertFalse(app.config['SECRET_KEY'] is 'my_precious')
    self.assertTrue(app.config['DEBUG'] is True)
    self.assertFalse(current_app is None)
    self.assertTrue(
        app.config['SQLALCHEMY_DATABASE_URI'] == 'postgresql://postgres:@localhost/flask_jwt_auth'
    )


class TestTestingConfig(TestCase):
    def create_app(self):
        app.config.from_object('project.server.config.TestingConfig')
        return app

    def test_app_is_testing(self):
        self.assertFalse(app.config['SECRET_KEY'] is 'my_precious')
        self.assertTrue(app.config['DEBUG'])
        self.assertTrue(
            app.config['SQLALCHEMY_DATABASE_URI'] == 'postgresql://postgres:@localhost/flask_jwt_auth_test'
        )

Before moving on, let’s write a quick unit test for the user model. Add the following code to a new file called test_user_model.py in “project/tests”:

# project/tests/test_user_model.py

import unittest

from project.server import db
from project.server.models import User
from project.tests.base import BaseTestCase


class TestUserModel(BaseTestCase):

    def test_encode_auth_token(self):
        user = User(
            email='test@test.com',
            password='test'
        )
        db.session.add(user)
        db.session.commit()
        auth_token = user.encode_auth_token(user.id)
        self.assertTrue(isinstance(auth_token, bytes))

if __name__ == '__main__':
    unittest.main()

Run the tests. They all should pass.

Decode Token

Similarly, to decode a token, add the following method to the User() class:

@staticmethod
def decode_auth_token(auth_token):
    """
    Decodes the auth token
    :param auth_token:
    :return: integer|string
    """
    try:
        payload = jwt.decode(auth_token, app.config.get('SECRET_KEY'))
        return payload['sub']
    except jwt.ExpiredSignatureError:
        return 'Signature expired. Please log in again.'
    except jwt.InvalidTokenError:
        return 'Invalid token. Please log in again.'

We need to decode the auth token with every API request and verify its signature to be sure of the user’s authenticity. To verify the auth_token, we used the same SECRET_KEY used to encode a token.

If the auth_token is valid, we get the user id from the sub index of the payload. If invalid, there could be two exceptions:

Expired Signature: When the token is used after it’s expired, it throws a ExpiredSignatureError exception. This means the time specified in the payload’s exp field has expired.
Invalid Token: When the token supplied is not correct or malformed, then an InvalidTokenError exception is raised.

NOTE: We have used a static method since it does not relate to the class’s instance.

Add a test to test_user_model.py:

def test_decode_auth_token(self):
    user = User(
        email='test@test.com',
        password='test'
    )
    db.session.add(user)
    db.session.commit()
    auth_token = user.encode_auth_token(user.id)
    self.assertTrue(isinstance(auth_token, bytes))
    self.assertTrue(User.decode_auth_token(auth_token) == 1)

Make sure the tests pass before moving on.

NOTE: We will handle invalid tokens by blacklisting them later.

Route Setup

Now we can configure the auth routes using a test-first approach:

/auth/register
/auth/login
/auth/logout
/auth/user

Start by creating a new folder called “auth” in “project/server”. Then, within “auth” add two files, __init__.py and views.py. Finally, add the following code to views.py:

# project/server/auth/views.py

from flask import Blueprint, request, make_response, jsonify
from flask.views import MethodView

from project.server import bcrypt, db
from project.server.models import User

auth_blueprint = Blueprint('auth', __name__)

To register the new Blueprint with the app, add the following to the bottom of project/server/__init__.py:

from project.server.auth.views import auth_blueprint
app.register_blueprint(auth_blueprint)

Now, add a new file called test_auth.py within “project/tests” to hold all our tests for this Blueprint:

# project/tests/test_auth.py

import unittest

from project.server import db
from project.server.models import User
from project.tests.base import BaseTestCase


class TestAuthBlueprint(BaseTestCase):
    pass


if __name__ == '__main__':
    unittest.main()

Register Route

Start with a test:

def test_registration(self):
    """ Test for user registration """
    with self.client:
        response = self.client.post(
            '/auth/register',
            data=json.dumps(dict(
                email='joe@gmail.com',
                password='123456'
            )),
            content_type='application/json'
        )
        data = json.loads(response.data.decode())
        self.assertTrue(data['status'] == 'success')
        self.assertTrue(data['message'] == 'Successfully registered.')
        self.assertTrue(data['auth_token'])
        self.assertTrue(response.content_type == 'application/json')
        self.assertEqual(response.status_code, 201)

Make sure to add the import:

import json

Run the tests. You should see the following error:

raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Now, let’s write the code to get the test to pass. Add the following to project/server/auth/views.py:

class RegisterAPI(MethodView):
    """
    User Registration Resource
    """

    def post(self):
        # get the post data
        post_data = request.get_json()
        # check if user already exists
        user = User.query.filter_by(email=post_data.get('email')).first()
        if not user:
            try:
                user = User(
                    email=post_data.get('email'),
                    password=post_data.get('password')
                )

                # insert the user
                db.session.add(user)
                db.session.commit()
                # generate the auth token
                auth_token = user.encode_auth_token(user.id)
                responseObject = {
                    'status': 'success',
                    'message': 'Successfully registered.',
                    'auth_token': auth_token.decode()
                }
                return make_response(jsonify(responseObject)), 201
            except Exception as e:
                responseObject = {
                    'status': 'fail',
                    'message': 'Some error occurred. Please try again.'
                }
                return make_response(jsonify(responseObject)), 401
        else:
            responseObject = {
                'status': 'fail',
                'message': 'User already exists. Please Log in.',
            }
            return make_response(jsonify(responseObject)), 202

# define the API resources
registration_view = RegisterAPI.as_view('register_api')

# add Rules for API Endpoints
auth_blueprint.add_url_rule(
    '/auth/register',
    view_func=registration_view,
    methods=['POST']
)

Here, we register a new user and generate a new auth token for further requests, which we send back to the client.

Run the tests to ensure they all pass:

Ran 6 tests in 0.132s

OK

Next, let’s add one more test to ensure the registration fails if the user already exists:

def test_registered_with_already_registered_user(self):
    """ Test registration with already registered email"""
    user = User(
        email='joe@gmail.com',
        password='test'
    )
    db.session.add(user)
    db.session.commit()
    with self.client:
        response = self.client.post(
            '/auth/register',
            data=json.dumps(dict(
                email='joe@gmail.com',
                password='123456'
            )),
            content_type='application/json'
        )
        data = json.loads(response.data.decode())
        self.assertTrue(data['status'] == 'fail')
        self.assertTrue(
            data['message'] == 'User already exists. Please Log in.')
        self.assertTrue(response.content_type == 'application/json')
        self.assertEqual(response.status_code, 202)

Run the tests again before moving on to the next route. All should pass.

Again, start with a test. To verify the login API, let’s test for two cases:

Registered user login
Non-registered user login

def test_registered_user_login(self):
    """ Test for login of registered-user login """
    with self.client:
        # user registration
        resp_register = self.client.post(
            '/auth/register',
            data=json.dumps(dict(
                email='joe@gmail.com',
                password='123456'
            )),
            content_type='application/json',
        )
        data_register = json.loads(resp_register.data.decode())
        self.assertTrue(data_register['status'] == 'success')
        self.assertTrue(
            data_register['message'] == 'Successfully registered.'
        )
        self.assertTrue(data_register['auth_token'])
        self.assertTrue(resp_register.content_type == 'application/json')
        self.assertEqual(resp_register.status_code, 201)
        # registered user login
        response = self.client.post(
            '/auth/login',
            data=json.dumps(dict(
                email='joe@gmail.com',
                password='123456'
            )),
            content_type='application/json'
        )
        data = json.loads(response.data.decode())
        self.assertTrue(data['status'] == 'success')
        self.assertTrue(data['message'] == 'Successfully logged in.')
        self.assertTrue(data['auth_token'])
        self.assertTrue(response.content_type == 'application/json')
        self.assertEqual(response.status_code, 200)

In this test case, the registered user tries to log in and, as expected, our application should allow this.

Run the tests. They should fail. Now write the code:

class LoginAPI(MethodView):
    """
    User Login Resource
    """
    def post(self):
        # get the post data
        post_data = request.get_json()
        try:
            # fetch the user data
            user = User.query.filter_by(
                email=post_data.get('email')
              ).first()
            auth_token = user.encode_auth_token(user.id)
            if auth_token:
                responseObject = {
                    'status': 'success',
                    'message': 'Successfully logged in.',
                    'auth_token': auth_token.decode()
                }
                return make_response(jsonify(responseObject)), 200
        except Exception as e:
            print(e)
            responseObject = {
                'status': 'fail',
                'message': 'Try again'
            }
            return make_response(jsonify(responseObject)), 500

Don’t forget to convert the class to a view function:

# define the API resources
registration_view = RegisterAPI.as_view('register_api')
login_view = LoginAPI.as_view('login_api')

# add Rules for API Endpoints
auth_blueprint.add_url_rule(
    '/auth/register',
    view_func=registration_view,
    methods=['POST']
)
auth_blueprint.add_url_rule(
    '/auth/login',
    view_func=login_view,
    methods=['POST']
)

Run the tests again. Do they pass? They should. Don’t move on until all tests pass.

Add the test:

def test_non_registered_user_login(self):
    """ Test for login of non-registered user """
    with self.client:
        response = self.client.post(
            '/auth/login',
            data=json.dumps(dict(
                email='joe@gmail.com',
                password='123456'
            )),
            content_type='application/json'
        )
        data = json.loads(response.data.decode())
        self.assertTrue(data['status'] == 'fail')
        self.assertTrue(data['message'] == 'User does not exist.')
        self.assertTrue(response.content_type == 'application/json')
        self.assertEqual(response.status_code, 404)

In this case, a non-registered user attempts to log in and, as expected, our application should not allow this.

Run the tests, and then update the code:

class LoginAPI(MethodView):
    """
    User Login Resource
    """
    def post(self):
        # get the post data
        post_data = request.get_json()
        try:
            # fetch the user data
            user = User.query.filter_by(
                email=post_data.get('email')
            ).first()
            if user and bcrypt.check_password_hash(
                user.password, post_data.get('password')
            ):
                auth_token = user.encode_auth_token(user.id)
                if auth_token:
                    responseObject = {
                        'status': 'success',
                        'message': 'Successfully logged in.',
                        'auth_token': auth_token.decode()
                    }
                    return make_response(jsonify(responseObject)), 200
            else:
                responseObject = {
                    'status': 'fail',
                    'message': 'User does not exist.'
                }
                return make_response(jsonify(responseObject)), 404
        except Exception as e:
            print(e)
            responseObject = {
                'status': 'fail',
                'message': 'Try again'
            }
            return make_response(jsonify(responseObject)), 500

What did we change? Do the tests pass? What if the email is correct but the password is incorrect? What happens? Write a test for this!

User Status Route

In order to get the user details of the currently logged in user, the auth token must be sent with the request within the header.

Start with a test:

def test_user_status(self):
    """ Test for user status """
    with self.client:
        resp_register = self.client.post(
            '/auth/register',
            data=json.dumps(dict(
                email='joe@gmail.com',
                password='123456'
            )),
            content_type='application/json'
        )
        response = self.client.get(
            '/auth/status',
            headers=dict(
                Authorization='Bearer ' + json.loads(
                    resp_register.data.decode()
                )['auth_token']
            )
        )
        data = json.loads(response.data.decode())
        self.assertTrue(data['status'] == 'success')
        self.assertTrue(data['data'] is not None)
        self.assertTrue(data['data']['email'] == 'joe@gmail.com')
        self.assertTrue(data['data']['admin'] is 'true' or 'false')
        self.assertEqual(response.status_code, 200)

The test should fail. Now, in the handler class, we should:

extract the auth token and check its validity
grab the user id from the payload and get the user details (if the token is valid, of course)

class UserAPI(MethodView):
    """
    User Resource
    """
    def get(self):
        # get the auth token
        auth_header = request.headers.get('Authorization')
        if auth_header:
            auth_token = auth_header.split(" ")[1]
        else:
            auth_token = ''
        if auth_token:
            resp = User.decode_auth_token(auth_token)
            if not isinstance(resp, str):
                user = User.query.filter_by(id=resp).first()
                responseObject = {
                    'status': 'success',
                    'data': {
                        'user_id': user.id,
                        'email': user.email,
                        'admin': user.admin,
                        'registered_on': user.registered_on
                    }
                }
                return make_response(jsonify(responseObject)), 200
            responseObject = {
                'status': 'fail',
                'message': resp
            }
            return make_response(jsonify(responseObject)), 401
        else:
            responseObject = {
                'status': 'fail',
                'message': 'Provide a valid auth token.'
            }
            return make_response(jsonify(responseObject)), 401

So, if the token is valid and not expired, we get the user id from the token’s payload, which is then used to get the user data from the database.

NOTE: We still need to check if a token is blacklisted. We’ll get to this shortly.

Make sure to add:

user_view = UserAPI.as_view('user_api')

And:

auth_blueprint.add_url_rule(
    '/auth/status',
    view_func=user_view,
    methods=['GET']
)

The tests should pass:

Ran 10 tests in 0.240s

OK

One more route to go!

Logout Route Tests

Tests valid logout:

def test_valid_logout(self):
    """ Test for logout before token expires """
    with self.client:
        # user registration
        resp_register = self.client.post(
            '/auth/register',
            data=json.dumps(dict(
                email='joe@gmail.com',
                password='123456'
            )),
            content_type='application/json',
        )
        data_register = json.loads(resp_register.data.decode())
        self.assertTrue(data_register['status'] == 'success')
        self.assertTrue(
            data_register['message'] == 'Successfully registered.')
        self.assertTrue(data_register['auth_token'])
        self.assertTrue(resp_register.content_type == 'application/json')
        self.assertEqual(resp_register.status_code, 201)
        # user login
        resp_login = self.client.post(
            '/auth/login',
            data=json.dumps(dict(
                email='joe@gmail.com',
                password='123456'
            )),
            content_type='application/json'
        )
        data_login = json.loads(resp_login.data.decode())
        self.assertTrue(data_login['status'] == 'success')
        self.assertTrue(data_login['message'] == 'Successfully logged in.')
        self.assertTrue(data_login['auth_token'])
        self.assertTrue(resp_login.content_type == 'application/json')
        self.assertEqual(resp_login.status_code, 200)
        # valid token logout
        response = self.client.post(
            '/auth/logout',
            headers=dict(
                Authorization='Bearer ' + json.loads(
                    resp_login.data.decode()
                )['auth_token']
            )
        )
        data = json.loads(response.data.decode())
        self.assertTrue(data['status'] == 'success')
        self.assertTrue(data['message'] == 'Successfully logged out.')
        self.assertEqual(response.status_code, 200)

In this first test, we register a new user, log them in, and then attempt to log them out before the token expires.

Test invalid logout:

def test_invalid_logout(self):
    """ Testing logout after the token expires """
    with self.client:
        # user registration
        resp_register = self.client.post(
            '/auth/register',
            data=json.dumps(dict(
                email='joe@gmail.com',
                password='123456'
            )),
            content_type='application/json',
        )
        data_register = json.loads(resp_register.data.decode())
        self.assertTrue(data_register['status'] == 'success')
        self.assertTrue(
            data_register['message'] == 'Successfully registered.')
        self.assertTrue(data_register['auth_token'])
        self.assertTrue(resp_register.content_type == 'application/json')
        self.assertEqual(resp_register.status_code, 201)
        # user login
        resp_login = self.client.post(
            '/auth/login',
            data=json.dumps(dict(
                email='joe@gmail.com',
                password='123456'
            )),
            content_type='application/json'
        )
        data_login = json.loads(resp_login.data.decode())
        self.assertTrue(data_login['status'] == 'success')
        self.assertTrue(data_login['message'] == 'Successfully logged in.')
        self.assertTrue(data_login['auth_token'])
        self.assertTrue(resp_login.content_type == 'application/json')
        self.assertEqual(resp_login.status_code, 200)
        # invalid token logout
        time.sleep(6)
        response = self.client.post(
            '/auth/logout',
            headers=dict(
                Authorization='Bearer ' + json.loads(
                    resp_login.data.decode()
                )['auth_token']
            )
        )
        data = json.loads(response.data.decode())
        self.assertTrue(data['status'] == 'fail')
        self.assertTrue(
            data['message'] == 'Signature expired. Please log in again.')
        self.assertEqual(response.status_code, 401)

Like the last test, we register a user, log them in, and then attempt to log them out. In this case, the token is invalid since it has expired.

Add the import:

import time

Now, the code must:

validate the auth token
blacklist the token (if valid, of course)

Before writing the route handler, let’s create a new model for blacklisting tokens…

Blacklist

Add the following code to project/server/models.py:

class BlacklistToken(db.Model):
    """
    Token Model for storing JWT tokens
    """
    __tablename__ = 'blacklist_tokens'

    id = db.Column(db.Integer, primary_key=True, autoincrement=True)
    token = db.Column(db.String(500), unique=True, nullable=False)
    blacklisted_on = db.Column(db.DateTime, nullable=False)

    def __init__(self, token):
        self.token = token
        self.blacklisted_on = datetime.datetime.now()

    def __repr__(self):
        return '<id: token: {}'.format(self.token)

Then create and apply the migrations. Once done, your database should have the following tables:

Schema |          Name           |   Type   |  Owner
--------+-------------------------+----------+----------
public | alembic_version         | table    | postgres
public | blacklist_tokens        | table    | postgres
public | blacklist_tokens_id_seq | sequence | postgres
public | users                   | table    | postgres
public | users_id_seq            | sequence | postgres
(5 rows)

With that, we can add the logout handler…

Logout Route Handler

Update the views:

class LogoutAPI(MethodView):
    """
    Logout Resource
    """
    def post(self):
        # get auth token
        auth_header = request.headers.get('Authorization')
        if auth_header:
            auth_token = auth_header.split(" ")[1]
        else:
            auth_token = ''
        if auth_token:
            resp = User.decode_auth_token(auth_token)
            if not isinstance(resp, str):
                # mark the token as blacklisted
                blacklist_token = BlacklistToken(token=auth_token)
                try:
                    # insert the token
                    db.session.add(blacklist_token)
                    db.session.commit()
                    responseObject = {
                        'status': 'success',
                        'message': 'Successfully logged out.'
                    }
                    return make_response(jsonify(responseObject)), 200
                except Exception as e:
                    responseObject = {
                        'status': 'fail',
                        'message': e
                    }
                    return make_response(jsonify(responseObject)), 200
            else:
                responseObject = {
                    'status': 'fail',
                    'message': resp
                }
                return make_response(jsonify(responseObject)), 401
        else:
            responseObject = {
                'status': 'fail',
                'message': 'Provide a valid auth token.'
            }
            return make_response(jsonify(responseObject)), 403


# define the API resources
registration_view = RegisterAPI.as_view('register_api')
login_view = LoginAPI.as_view('login_api')
user_view = UserAPI.as_view('user_api')
logout_view = LogoutAPI.as_view('logout_api')

# add Rules for API Endpoints
auth_blueprint.add_url_rule(
    '/auth/register',
    view_func=registration_view,
    methods=['POST']
)
auth_blueprint.add_url_rule(
    '/auth/login',
    view_func=login_view,
    methods=['POST']
)
auth_blueprint.add_url_rule(
    '/auth/status',
    view_func=user_view,
    methods=['GET']
)
auth_blueprint.add_url_rule(
    '/auth/logout',
    view_func=logout_view,
    methods=['POST']
)

Update the imports:

from project.server.models import User, BlacklistToken

When a users logs out, the token is no longer valid so we add it to the blacklist.

NOTE: Often, larger applications have a way to renew blacklisted tokens every now and then so that the system does not run out of valid tokens.

Run the tests:

Ran 12 tests in 6.418s

OK

Refactoring

Finally, we need to ensure that a token has not been blacklisted, right after the token has been decoded - decode_auth_token() - within the logout and user status routes.

First, let’s write a test for the logout route:

def test_valid_blacklisted_token_logout(self):
    """ Test for logout after a valid token gets blacklisted """
    with self.client:
        # user registration
        resp_register = self.client.post(
            '/auth/register',
            data=json.dumps(dict(
                email='joe@gmail.com',
                password='123456'
            )),
            content_type='application/json',
        )
        data_register = json.loads(resp_register.data.decode())
        self.assertTrue(data_register['status'] == 'success')
        self.assertTrue(
            data_register['message'] == 'Successfully registered.')
        self.assertTrue(data_register['auth_token'])
        self.assertTrue(resp_register.content_type == 'application/json')
        self.assertEqual(resp_register.status_code, 201)
        # user login
        resp_login = self.client.post(
            '/auth/login',
            data=json.dumps(dict(
                email='joe@gmail.com',
                password='123456'
            )),
            content_type='application/json'
        )
        data_login = json.loads(resp_login.data.decode())
        self.assertTrue(data_login['status'] == 'success')
        self.assertTrue(data_login['message'] == 'Successfully logged in.')
        self.assertTrue(data_login['auth_token'])
        self.assertTrue(resp_login.content_type == 'application/json')
        self.assertEqual(resp_login.status_code, 200)
        # blacklist a valid token
        blacklist_token = BlacklistToken(
            token=json.loads(resp_login.data.decode())['auth_token'])
        db.session.add(blacklist_token)
        db.session.commit()
        # blacklisted valid token logout
        response = self.client.post(
            '/auth/logout',
            headers=dict(
                Authorization='Bearer ' + json.loads(
                    resp_login.data.decode()
                )['auth_token']
            )
        )
        data = json.loads(response.data.decode())
        self.assertTrue(data['status'] == 'fail')
        self.assertTrue(data['message'] == 'Token blacklisted. Please log in again.')
        self.assertEqual(response.status_code, 401)

In this test, we blacklist the token just before the logout route gets hit which makes our valid token unusable.

Update the imports:

from project.server.models import User, BlacklistToken

The test should fail with the following exception:

psycopg2.IntegrityError: duplicate key value violates unique constraint "blacklist_tokens_token_key"
DETAIL:  Key (token)=(eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE0ODUyMDgyOTUsImlhdCI6MTQ4NTIwODI5MCwic3ViIjoxfQ.D9annoyh-VwpI5RY3blaSBX4pzK5UJi1H9dmKg2DeLQ) already exists.

Now update the decode_auth_token function to handle already blacklisted tokens right after the decode and respond with appropriate message.

@staticmethod
def decode_auth_token(auth_token):
    """
    Validates the auth token
    :param auth_token:
    :return: integer|string
    """
    try:
        payload = jwt.decode(auth_token, app.config.get('SECRET_KEY'))
        is_blacklisted_token = BlacklistToken.check_blacklist(auth_token)
        if is_blacklisted_token:
            return 'Token blacklisted. Please log in again.'
        else:
            return payload['sub']
    except jwt.ExpiredSignatureError:
        return 'Signature expired. Please log in again.'
    except jwt.InvalidTokenError:
        return 'Invalid token. Please log in again.'

Finally, add the check_blacklist() function to project/server/models.py in the BlacklistToken class:

@staticmethod
def check_blacklist(auth_token):
    # check whether auth token has been blacklisted
    res = BlacklistToken.query.filter_by(token=str(auth_token)).first()
    if res:
        return True  
    else:
        return False

Before you run the test, update test_decode_auth_token to convert the bytes object to a string:

def test_decode_auth_token(self):
    user = User(
        email='test@test.com',
        password='test'
    )
    db.session.add(user)
    db.session.commit()
    auth_token = user.encode_auth_token(user.id)
    self.assertTrue(isinstance(auth_token, bytes))
    self.assertTrue(User.decode_auth_token(
        auth_token.decode("utf-8") ) == 1)

Run the tests:

Ran 13 tests in 9.557s

OK

In a similar fashion, add one more test for the user status route.

def test_valid_blacklisted_token_user(self):
    """ Test for user status with a blacklisted valid token """
    with self.client:
        resp_register = self.client.post(
            '/auth/register',
            data=json.dumps(dict(
                email='joe@gmail.com',
                password='123456'
            )),
            content_type='application/json'
        )
        # blacklist a valid token
        blacklist_token = BlacklistToken(
            token=json.loads(resp_register.data.decode())['auth_token'])
        db.session.add(blacklist_token)
        db.session.commit()
        response = self.client.get(
            '/auth/status',
            headers=dict(
                Authorization='Bearer ' + json.loads(
                    resp_register.data.decode()
                )['auth_token']
            )
        )
        data = json.loads(response.data.decode())
        self.assertTrue(data['status'] == 'fail')
        self.assertTrue(data['message'] == 'Token blacklisted. Please log in again.')
        self.assertEqual(response.status_code, 401)

Similar to the last test, we blacklisted the token before the user status route gets hit.

Run the tests for one final time:

Ran 14 tests in 10.206s

OK

Code Smell

Finally, take a look at test_auth.py. Notice the duplicate code? For example:

self.client.post(
    '/auth/register',
    data=json.dumps(dict(
        email='joe@gmail.com',
        password='123456'
    )),
    content_type='application/json',
)

There are eight occurrences of this. To fix, add the following helper at the top of the file:

def register_user(self, email, password):
    return self.client.post(
        '/auth/register',
        data=json.dumps(dict(
            email=email,
            password=password
        )),
        content_type='application/json',
    )

Now, anywhere you need to register a user, you can call the helper:

register_user(self, 'joe@gmail.com', '123456')

How about logging in a user? Refactor it on your own. What else can you refactor? Comment below.

Refactor

For the PyBites Challenge, let’s refactor some code to correct an issue added to the GitHub repo. Start by adding the following test to test_auth.py:

def test_user_status_malformed_bearer_token(self):
    """ Test for user status with malformed bearer token"""
    with self.client:
        resp_register = register_user(self, 'joe@gmail.com', '123456')
        response = self.client.get(
            '/auth/status',
            headers=dict(
                Authorization='Bearer' + json.loads(
                    resp_register.data.decode()
                )['auth_token']
            )
        )
        data = json.loads(response.data.decode())
        self.assertTrue(data['status'] == 'fail')
        self.assertTrue(data['message'] == 'Bearer token malformed.')
        self.assertEqual(response.status_code, 401)

Essentially, an error is thrown if the Authorization header is formatted incorrectly - e.g., no space between Bearer and the token value. Run the tests to ensure they fail, and then update the UserAPI class in project/server/auth/views.py:

class UserAPI(MethodView):
    """
    User Resource
    """
    def get(self):
        # get the auth token
        auth_header = request.headers.get('Authorization')
        if auth_header:
            try:
                auth_token = auth_header.split(" ")[1]
            except IndexError:
                responseObject = {
                    'status': 'fail',
                    'message': 'Bearer token malformed.'
                }
                return make_response(jsonify(responseObject)), 401
        else:
            auth_token = ''
        if auth_token:
            resp = User.decode_auth_token(auth_token)
            if not isinstance(resp, str):
                user = User.query.filter_by(id=resp).first()
                responseObject = {
                    'status': 'success',
                    'data': {
                        'user_id': user.id,
                        'email': user.email,
                        'admin': user.admin,
                        'registered_on': user.registered_on
                    }
                }
                return make_response(jsonify(responseObject)), 200
            responseObject = {
                'status': 'fail',
                'message': resp
            }
            return make_response(jsonify(responseObject)), 401
        else:
            responseObject = {
                'status': 'fail',
                'message': 'Provide a valid auth token.'
            }
            return make_response(jsonify(responseObject)), 401

Run the tests one final time.

Conclusion

In this tutorial, we went through the process of adding authentication to a Flask app with JSON Web Tokens. Turn back to the objectives from the beginning of this tutorial. Can you put each one into action? What did you learn?

What’s next? How about the client-side. Check out Token-Based Authentication With Angular for adding Angular into the mix.

To see how to build a complete web app from scratch using Flask, check out our video series:

Free Bonus: Click here to get access to a free Flask + Python video tutorial series that shows you how to build a fully working Flask web app, step-by-step.

Feel free to share your comments, questions, or tips in the comments below. The full code can be found in the flask-jwt-auth repository.

Cheers!

Automating Django Deployments with Fabric and Ansible

2016-12-28T14:18:21+00:00

In the last post, we covered all the steps required to successfully develop and deploy a Django app on a single server. In this tutorial we will automate the deployment process with Fabric (v1.12.0) and Ansible (v2.1.3) to address these issues:

Scaling: When it comes to scaling a web app to handle thousands of daily requests, relying on a single server is not a good approach. Put simply, as the server approaches maximum CPU utilization, it can cause slow load times which can eventually lead to server failure. To overcome this, the app must be scaled to run on more than one server so that the servers can cumulatively handle the incoming concurrent requests.
Redundancy: Deploying a web app manually to a new server means a lot of repeated work with more chances of human error. Automating the process is key.

Specifically, we will automate:

Adding a new, non-root user
Configuring the server
Pulling the Django app code from a GitHub repo
Installing the dependencies
Daemonizing the app

Setup and Config

Start by spinning up a new Digital Ocean droplet, making sure to use the Fedora 25 image. Do not set up a pre-configured SSH key; we will be automating this process later via a Fabric script. Since the deployment process should be scalable, create a separate repository to house all the deployment scripts. Make a new project directory locally, and create and activate a virtualenv using Python 2.7x.

Why Python 2.7? Fabric does NOT support Python 3. Don’t worry: We’ll be using Python 3.5 when we provision the server.

$ mkdir automated-deployments
$ cd automated-deployments
$ virtualenv env
$ source env/bin/activate

Fabric Setup

Fabric is a tool used for automating routine shell commands over SSH, which we will be using to:

Set up the SSH keys
Harden user passwords
Install Ansible dependencies
Upgrade the server

Start by installing Fabric:

$ pip install fabric==1.12.0

Create a new folder called “prod”, and add a new file called fabfile.py to it to hold all of the Fabric scripts:

# prod/fabfile.py

import os
from fabric.contrib.files import sed
from fabric.api import env, local, run
from fabric.api import env

# initialize the base directory
abs_dir_path = os.path.dirname(
    os.path.dirname(os.path.abspath(__file__)))


# declare environment global variables

# root user
env.user = 'root'

# list of remote IP addresses
env.hosts = ['<remote-server-ip>']

# password for the remote server
env.password = '<remote-server-password>'

# full name of the user
env.full_name_user = '<your-name>'

# user group
env.user_group = 'deployers'

# user for the above group
env.user_name = 'deployer'

# ssh key path
env.ssh_keys_dir = os.path.join(abs_dir_path, 'ssh-keys')

Take note of the inline comments. Be sure to add you remote server’s IP address to the env.hosts variable. Update env.full_name_user as well. Hold off on updating env.password; we will get to that shortly. Look over all the env variables - they are completely customizable based on your system setup.

Set up the SSH keys

Add the following code to fabfile.py:

def start_provision():
    """
    Start server provisioning
    """
    # Create a new directory for a new remote server
    env.ssh_keys_name = os.path.join(
        env.ssh_keys_dir, env.host_string + '_prod_key')
    local('ssh-keygen -t rsa -b 2048 -f {0}'.format(env.ssh_keys_name))
    local('cp {0} {1}/authorized_keys'.format(
        env.ssh_keys_name + '.pub', env.ssh_keys_dir))
    # Prevent root SSHing into the remote server
    sed('/etc/ssh/sshd_config', '^UsePAM yes', 'UsePAM no')
    sed('/etc/ssh/sshd_config', '^PermitRootLogin yes',
        'PermitRootLogin no')
    sed('/etc/ssh/sshd_config', '^#PasswordAuthentication yes',
        'PasswordAuthentication no')

    install_ansible_dependencies()
    create_deployer_group()
    create_deployer_user()
    upload_keys()
    set_selinux_permissive()
    run('service sshd reload')
    upgrade_server()

This function acts as the entry point for the Fabric script. Besides triggering a series of functions, each explained in further steps, it explicitly-

Generates a new pair of SSH keys in the specified location within your local system
Copies the contents of the public key to the authorized_keys file
Makes changes to the remote sshd_config file to prevent root login and disable password-less auth

Preventing SSH access for the root user is an optional step, but it is recommended as it ensures that no one has superuser rights.

Create a directory for your SSH keys in the project root:

├── prod
│   └── fabfile.py
└── ssh-keys

Harden user passwords

This step includes the addition of three different functions, each executed serially to configure SSH password hardening…

Create deployer group

def create_deployer_group():
    """
    Create a user group for all project developers
    """
    run('groupadd {}'.format(env.user_group))
    run('mv /etc/sudoers /etc/sudoers-backup')
    run('(cat /etc/sudoers-backup; echo "%' +
        env.user_group + ' ALL=(ALL) ALL") > /etc/sudoers')
    run('chmod 440 /etc/sudoers')

Here, we add a new group called deployers and grant sudo permissions to it so that users can carry out processes with root privileges.

Create user

def create_deployer_user():
    """
    Create a user for the user group
    """
    run('adduser -c "{}" -m -g {} {}'.format(
        env.full_name_user, env.user_group, env.user_name))
    run('passwd {}'.format(env.user_name))
    run('usermod -a -G {} {}'.format(env.user_group, env.user_name))
    run('mkdir /home/{}/.ssh'.format(env.user_name))
    run('chown -R {} /home/{}/.ssh'.format(env.user_name, env.user_name))
    run('chgrp -R {} /home/{}/.ssh'.format(
        env.user_group, env.user_name))

This function-

Adds a new user to the deployers user group, which we defined in the last function
Sets up the SSH directory for keeping SSH key pairs and grants permission to the group and the user to access that directory

Upload SSH keys

def upload_keys():
    """
    Upload the SSH public/private keys to the remote server via scp
    """
    scp_command = 'scp {} {}/authorized_keys {}@{}:~/.ssh'.format(
        env.ssh_keys_name + '.pub',
        env.ssh_keys_dir,
        env.user_name,
        env.host_string
    )
    local(scp_command)

Here, we-

Upload the locally generated SSH keys to the remote server so that non-root users can log in via SSH without entering a password
Copy the public key and the authorized keys to the remote server in the newly created ssh-keys directory

Install Ansible dependencies

Add the following function to install the dependency packages for Ansible:

def install_ansible_dependencies():
    """
    Install the python-dnf module so that Ansible
    can communicate with Fedora's Package Manager
    """
    run('dnf install -y python-dnf')

Keep in mind that this is specific to the Fedora Linux distro, as we will be using the DNF module for installing packages, but it could vary by distro.

Set SELinux to permissive mode

The next function sets SELinux to permissive mode. This is done to overcome any potential Nginx 502 Bad Gateway errors.

def set_selinux_permissive():
    """
    Set SELinux to Permissive/Disabled Mode
    """
    # for permissive
    run('sudo setenforce 0')

Again, this is specific to the Fedora Linux distro.

Upgrade the server

Finally, upgrade the server:

def upgrade_server():
    """
    Upgrade the server as a root user
    """
    run('dnf upgrade -y')
    # optional command (necessary for Fedora 25)
    run('dnf install -y python')
    run('reboot')

Sanity check

With that, we’re done with the Fabric script. Before running it, make sure you SSH into the server as root and change the password:

$ ssh root@<server-ip-address>
You are required to change your password immediately (root enforced)
Changing password for root.
(current) UNIX password:
New password:
Retype new password:

Be sure to update env.password with the new password. Exit the server and return to the local terminal, then execute Fabric:

$ fab -f ./prod/fabfile.py start_provision

If all went well, new SSH keys will be generated, and you will be asked to create a password (make sure to do this!):

Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

A number of tasks will run. After the deployer user is created, you will be prompted to add a password for the user-

[104.236.66.172] out: Changing password for user deployer.

-which you will then have to enter when the SSH keys are uploaded:

deployer@104.236.66.172s password:

After this script exits successfully, you will NO longer be able to log into the remote server as a root user. Instead, you will only be able to use the non-root user deployer.

Try it out:

$ ssh root@<server-ip-address>
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

This is expected. Then, when you run-

$ ssh -i ./ssh-keys/104.236.66.172_prod_key deployer@104.236.66.172

-you should be able to log in just fine:

[deployer@fedora-512mb-nyc2-01 ~]$

Ansible Primer

Ansible is a configuration management and provisioning tool used to automate deployment tasks over SSH.

You can fire individual Ansible tasks against the app servers from your shell remotely and execute tasks on the go. Tasks can also be combined into Playbooks - a collection of multiple plays, where each play defines certain specific tasks that are required during the deployment process. They are executed against the app servers during the deployment process. Playbooks are written in YAML.

Playbooks

Playbooks consist of a modular architecture as follows:

Hosts specify all the IP addresses or domain names of our remote servers that need to be orchestrated. Playbooks always run on a targeted group of hosts.
Roles are divided into sub parts. Let’s look at some sample roles:
- Tasks are a collection of multiple tasks that need to be carried out during the deployment process.
- Handlers provide a way to trigger a set of operations when a module makes a change to the remote server (best thought of as hooks).
- Templates, in this context, are generally used for specifying some module-related configuration files - like nginx.
Variables are simply a list of key-value pairs where every key (a variable) is mapped to a value. Such variables can be used in the Playbooks as placeholders.

Sample Playbook

Now let’s look at a sample single-file Playbook:

---
# My Ansible playbook for configuring Nginx
- hosts: all

  vars:
    http_port: 80
    app_name: django_bootstrap

  tasks:
    - name: Install nginx
      dnf: name=nginx state=latest

    - name: Create nginx config file
      template: src=django_bootstrap.conf dest=/etc/nginx/conf.d/{{ app_name }}.conf
      become: yes
      notify:
        - restart nginx

  handlers:
    - name: Restart nginx
      service: name=nginx state=restarted enabled=yes
      become: yes

Here, we defined:

Hosts as hosts: all, which indicates that the Playbook will run on all of the servers that are listed in the inventory/hosts file
Variables http_port: 80 and app_name: django_bootstrap for use in a template
Tasks in order to install nginx, set up the nginx config (become indicates that we need admin privileges), and trigger the restart handler
Handler in order to restart the nginx service

Playbook Setup

Now let’s set up a Playbook for Django. Add a deploy.yml file to the “prod” directory:

##
# This playbook deploys the whole app stack
##
- name: apply common configuration to server
  hosts: all
  user: deployer
  roles:
    - common

The above snippet glues together the Ansible hosts, users, and roles.

Hosts

Add a hosts (plain text format) file to the “prod” directory and list the servers under their respective role names. We are provisioning a single server here:

[common]
<server-ip-address>

In the above snippet, common refers to the role name. Under the roles we have a list of IP addresses that need to be configured. Make sure to add your remote server’s IP address in place of <server-ip-address>.

Variables

Now we define the variables that will be used by the roles. Add a new folder inside “prod” called “group_vars”, then create a new file called all (plain text format) within that folder. Here, specify the following variables to start with:

# App Name
app_name: django_bootstrap

# Deployer User and Groups
deployer_user: deployer
deployer_group: deployers

# SSH Keys Directory
ssh_dir: <path-to-your-ssh-keys>

Make sure to update <path-to-your-ssh-keys>. To get the correct path, within the project root, run:

$ cd ssh-keys
$ pwd
/Users/michael.herman/repos/realpython/automated-deployments/ssh-keys

With these files in place, we are now ready to coordinate our deployment process with all the roles that need be carried out on the server.

Playbook Roles

Again, Playbooks are simply a collection of different plays, and all these plays are run under specific roles. Create a new directory called “roles” within “prod”.

Did you catch the name of the role in the deploy.yml file?

Then within the “roles” directory add a new directory called “common” - the role. Roles consists of “tasks”, “handlers”, and “templates”. Add a new directory for each.

Once done your file structure should look something like this:

├── prod
│   ├── deploy.yml
│   ├── fabfile.py
│   ├── group_vars
│   │   └── all
│   ├── hosts
│   └── roles
│       └── common
│           ├── handlers
│           ├── tasks
│           └── templates
└── ssh-keys
    ├── 104.236.66.172_prod_key
    ├── 104.236.66.172_prod_key.pub
    └── authorized_keys

All the plays are defined in a “tasks” directory, starting with a main.yml file. This file serves as the entry point for all Playbook tasks. It’s simply a list of multiple YAML files that need to be executed in order.

Create that file now within the “tasks” directory, then add the following to it:

##
# Configure the server for the Django app
##
- include: 01_server.yml
- include: 02_git.yml
- include: 03_postgres.yml
- include: 04_dependencies.yml
- include: 05_migrations.yml
- include: 06_nginx.yml
- include: 07_gunicorn.yml
- include: 08_systemd.yml
# - include: 09_fix-502.yml

Now, let’s create each task. Be sure to add a new file for each task to the “tasks” directory and add the accompanying code to each file. If you get lost, refer to the repo.

01_server.yml

##
# Update the DNF package cache and install packages as a root user
##
- name: Install required packages
  dnf: name={{item}} state=latest
  become: yes
  with_items:
    - vim
    - fail2ban
    - python3-devel
    - python-virtualenv
    - python3-virtualenv
    - python-devel
    - gcc
    - libselinux-python
    - redhat-rpm-config
    - libtiff-devel
    - libjpeg-devel
    - libzip-devel
    - freetype-devel
    - lcms2-devel
    - libwebp-devel
    - tcl-devel
    - tk-devel
    - policycoreutils-devel

Here, we list all the system packages that need to be installed.

02_git.yml

##
# Clone and pull the repo
##
- name: Set up git configuration
  dnf: name=git state=latest
  become: yes

- name: Clone or pull the latest code
  git: repo={{ code_repository_url }}
        dest={{ app_dir }}

Add the following variables to the group_vars/all file:

# Github Code's Repo URL
code_repository_url: https://github.com/realpython/django-bootstrap

# App Directory
app_dir: /home/{{ deployer_user }}/{{app_name}}

Make sure to fork then clone the django-bootstrap repo, then update the code_repository_url variable to the URL of your fork.

03_postgres.yml

##
# Set up and configure postgres
##
- name: Install and configure db
  dnf: name={{item}} state=latest
  become: yes
  with_items:
    - postgresql-server
    - postgresql-contrib
    - postgresql-devel
    - python-psycopg2

- name: Run initdb command
  raw: postgresql-setup initdb
  become: yes

- name: Start and enable postgres
  service: name=postgresql enabled=yes state=started
  become: yes

- name: Create database
  postgresql_db: name={{ app_name }}
  become_user: postgres
  become: yes

- name: Configure a new postgresql user
  postgresql_user: db={{ app_name }}
                                name={{ db_user }}
                                password={{ db_password }}
                                priv=ALL
                                role_attr_flags=NOSUPERUSER
  become: yes
  become_user: postgres
  notify:
    - restart postgres

Update group_vars/all with the database configuration needed for the playbook:

# DB Configuration
db_url: postgresql://{{deployer_user}}:{{db_password}}@localhost/{{app_name}}
db_password: thisissomeseucrepassword
db_name: "{{ app_name }}"
db_user: "{{ deployer_user }}"

Update the db_password variable with a secure password.

Did you notice that we restart the postgres service within the main.yml file in order to apply the changes after the database is configured? This is our first handler. Create a new file called main.yml in the “handlers” folder, then add the following:

- name: restart postgres
  service: name=postgresql state=restarted
  become: yes

04_dependencies.yml

##
# Set up all the dependencies in a virtualenv required by the Django app
##
- name: Create a virtualenv directory
  file: path={{ venv_dir }} state=directory

- name: Install dependencies
  pip:    requirements={{ app_dir }}/requirements.txt
          virtualenv={{ venv_dir }}
          virtualenv_python=python3.5

- name: Create the .env file for running ad-hoc python commands in our virtualenv
  template: src=env.j2 dest={{ app_dir }}/.env
  become: yes

Update group_vars/all like so:

# Application Dependencies Setup
venv_dir: '/home/{{ deployer_user }}/envs/{{ app_name }}'
venv_python: '{{ venv_dir }}/bin/python3.5'

Add a template called env.j2 to the “templates” folder, and add the following environment variables:

#!/bin/bash
export DEBUG="True"
export DATABASE_URL="postgresql://deployer:thisissomeseucrepassword@localhost/django_bootstrap"
export DJANGO_SECRET_KEY="changeme"
export DJANGO_SETTINGS_MODULE="config.settings.production"

Be very careful with the environment variables and their values in env.j2 since these are used to get the Django Project up and running.

05_migrations.yml

##
# Run db migrations and get all static files
##
- name: Make migrations
  shell: ". {{ app_dir }}/.env; {{ venv_python }} {{ app_dir }}/manage.py makemigrations "
  become: yes

- name: Migrate database
  django_manage: app_path={{ app_dir }}
                                 command=migrate
                                 virtualenv={{ venv_dir }}

- name: Get all static files
  django_manage: app_path={{ app_dir }}
                                 command=collectstatic
                                 virtualenv={{ venv_dir }}
  become: yes

06_nginx.yml

##
# Configure nginx web server
##
- name: Set up nginx config
  dnf: name=nginx state=latest
  become: yes

- name: Write nginx conf file
  template: src=django_bootstrap.conf dest=/etc/nginx/conf.d/{{ app_name }}.conf
  become: yes
  notify:
    - restart nginx

Add the following variable to group_vars/all:

# Remote Server Details
server_ip: <remote-server-ip>
wsgi_server_port: 8000

Don’t forget to update <remote-server-ip>. Then add the handler to handlers/main.yml:

- name: restart nginx
  service: name=nginx state=restarted enabled=yes
  become: yes

Then we need to add the django_bootstrap.conf template. Create that file within the “templates” directory, then add the code:

upstream app_server {
    server 127.0.0.1:{{ wsgi_server_port }} fail_timeout=0;
}

server {
    listen 80;
    server_name {{ server_ip }};
    access_log /var/log/nginx/{{ app_name }}-access.log;
    error_log /var/log/nginx/{{ app_name }}-error.log info;

    keepalive_timeout 5;

    # path for staticfiles
    location /static {
            autoindex on;
            alias {{ app_dir }}/staticfiles/;
    }

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;

        if (!-f $request_filename) {
            proxy_pass http://app_server;
            break;
        }
    }
}

07_gunicorn.yml

##
# Set up Gunicorn and configure systemd to execute gunicorn_start script
##
- name: Create a deploy directory
  file: path={{ deploy_dir }} state=directory
  become: yes

- name: Create the gunicorn_start script for running our app from systemd service
  template: src=gunicorn_start
                    dest={{ deploy_dir }}/gunicorn_start
  become: yes

- name: Make the gunicorn_start script executable
  raw: cd {{ deploy_dir }}; chmod +x gunicorn_start
  become: yes

Add more variables to groups_vars/all:

# Deploy Dir in App Directory
deploy_dir: '{{ app_dir }}/deploy'

# WSGI Vars
django_wsgi_module: config.wsgi
django_settings_module: config.settings.production
django_secret_key: 'changeme'
database_url: '{{ db_url }}'

Add the gunicorn_start template:

#!/bin/bash

### Define script variables

# Name of the app
NAME='{{ app_name }}'
# Path to virtualenv
VIRTUALENV='{{ venv_dir }}'
# Django Project Directory
DJANGODIR='{{ app_dir }}'
# The user to run as
USER={{ deployer_user }}
# The group to run as
GROUP={{deployer_group }}
# Number of worker processes Gunicorn should spawn
NUM_WORKERS=3
# Settings file that Gunicorn should use
DJANGO_SETTINGS_MODULE={{django_settings_module}}
# WSGI module name
DJANGO_WSGI_MODULE={{ django_wsgi_module }}


### Activate virtualenv and create environment variables

echo "Starting $NAME as `whoami`"
# Activate the virtual environment
cd $VIRTUALENV
source bin/activate
cd $DJANGODIR
# Defining the Environment Variables
export DJANGO_SECRET_KEY='{{ django_secret_key }}'
export DATABASE_URL='{{ db_url }}'
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH


### Start Gunicorn

exec gunicorn ${DJANGO_WSGI_MODULE}:application \
        --name $NAME \
        --workers $NUM_WORKERS \
        --user=$USER --group=$GROUP \
        --log-level=debug \
        --bind=127.0.0.1:8000

08_systemd.yml

##
# Set up systemd for executing gunicorn_start script
##
- name: write a systemd service file
  template: src=django-bootstrap.service
                    dest=/etc/systemd/system
  become: yes
  notify:
    - restart app
    - restart nginx

Add the template - django-bootstrap.service:

#!/bin/sh

[Unit]
Description=Django Web App
After=network.target

[Service]
PIDFile=/var/run/djangoBootstrap.pid
User={{ deployer_user }}
Group={{ deployer_group }}
ExecStart=/bin/sh {{ deploy_dir }}/gunicorn_start
Restart=on-abort

[Install]
WantedBy=multi-user.target

Add the following to the handlers:

- name: restart app
  service: name=django-bootstrap state=restarted enabled=yes
  become: yes

09_fix-502.yml

##
# Fix the 502 nginx error post deployment
#
- name: Fix nginx 502 error
  raw: cd ~; cat /var/log/audit/audit.log | grep nginx | grep denied | audit2allow -M mynginx; semodule -i mynginx.pp
  become: yes

Sanity Check (final)

With the virtualenv activated, install Ansible locally:

$ pip install ansible==2.1.3

Create a new file called deploy_prod.sh in the project root to run the playbook, making sure to update <server-ip>:

#!/bin/bash

ansible-playbook ./prod/deploy.yml --private-key=./ssh_keys<server-ip>_prod_key -K -u deployer -i ./prod/hosts -vvv

Then run following command to execute the playbook:

$ sh deploy_prod.sh

If any errors occur, consult the terminal for info on how to correct them. Once fixed, execute the deploy script again. When the script is done visit the server’s IP Address to verify your Django web app is live and running!

Make sure to uncomment this line in prod/roles/common/tasks/main.yml if you see the 502 error, which indicates that there is a problem with communication between nginx and Gunicorn:

# - include: 09_fix-502.yml

Then execute the playbook again.

If you execute the playbook more than once, make sure to comment out the Run initdb command found in 03_postgres.yml since it needs run only once. Otherwise, it will throw errors when trying to reinitialize the DB server.

Conclusion

This post provides a basic understanding of how you can automate the configuring of a server with Fabric and Ansible. Ansible Playbooks are particularly powerful since you can automate almost any task on the server via a YAML file. Hopefully, you can now start writing your own Playbooks and even use them in your workplace to configure production-ready servers.

Please add questions and comments below. The full code can be found in the automated-deployments repository.

Introduction to MongoDB and Python

2016-12-12T14:55:50+00:00

Python is a powerful programming language used for many different types of applications within the development community. Many know it as a flexible language that can handle just about any task.

So, what if our complex Python application needs a database that’s just as flexible as the language itself?

This is where NoSQL, and specifically MongoDB, come in to play.

Free Bonus: Click here to download a Python + MongoDB project skeleton with full source code that shows you how to access MongoDB from Python.

Throughout this article we’ll show you how to use Python to interface with the popular MongoDB (v3.4.0) database, along with an overview of SQL vs. NoSQL, PyMongo (v3.4.0), and MongoEngine (v0.10.7), among other things.

SQL vs NoSQL

In case you aren’t familiar with it, MongoDB is a NoSQL database which has become pretty popular throughout the industry in recent years. NoSQL databases provide features of retrieval and storage of data in a much different way than their relational database counterparts.

For decades, SQL databases used to be one of the only choices for developers looking to build large, scalable systems. However, the ever-increasing need for the ability to store complex data structures led to the birth of NoSQL databases, which allow a developer to store heterogeneous and structure-less data.

When it comes to the choices available, most people have to ask themselves the ultimate question, “SQL or NoSQL?” Both SQL and NoSQL have their strengths and weaknesses, and you should choose the one that fits your application requirements the best. Here are a few differences between the two:

SQL

The model is of a relational nature
Data is stored in tables
Suitable for solutions where every record is of the same kind and possesses the same properties
Adding a new property means you have to alter the whole schema
The schema is very strict
ACID transactions are supported
Scales well vertically

NoSQL

The model is non-relational
May be stored as JSON, key-value, etc. (depending on type of NoSQL database)
Not every record has to be of the same nature, making it very flexible
Add new properties to data without disturbing anything
No schema requirements to adhere to
Support for ACID transactions can vary depending on which NoSQL DB is used
Consistency can vary
Scales well horizontally

There are many other differences between the two types of databases but those mentioned above are some of the more important differences to know.

Depending on your specific scenario, the use of a SQL database may be preferred, while in other scenarios NoSQL is the more obvious choice to make. When choosing a database you should consider the strengths and weaknesses of each database carefully.

One of the great things about NoSQL is that there are many different types of databases to choose from, and each has its own use-cases:

Key-Value Store: DynamoDB
Document Store: CouchDB, MongoDB, RethinkDB
Column Store: Cassandra
Data-Structures: Redis

There are quite a few more, but these are some of the more common types.

In recent years, SQL and NoSQL databases have even begun to merge. For example, PostgreSQL now supports storing and querying JSON data, much like Mongo. With this, you can now achieve much of the same with Postgres as you can with Mongo, but you still don’t get many of the Mongo advantages (like horizontal scaling and the simple interface, etc.). As this article puts it:

If your active data sits in the relational schema comfortably and the JSON content is a cohort to that data then you should be fine with PostgreSQL and it’s much more efficient JSONB representation and indexing capabilities. If though, your data model is that of a collection of mutable documents then you probably want to look at a database engineered primarily around JSON documents like MongoDB or RethinkDB.

MongoDB

Let’s now shift our attention to the main focus of this article and shed some light on the specifics of MongoDB.

MongoDB is a document-oriented, open-source database program that is platform-independent. MongoDB, like some other NoSQL databases (but not all!), stores its data in documents using a JSON structure. This is what allows the data to be so flexible and not require a schema.

Some of the more important features are:

You have support for many of the standard query types, like matching (==), comparison (<, >), or even regex
You can store virtually any kind of data - be it structured, partially structured, or even polymorphic
To scale up and handle more queries, just add more machines
It is highly flexible and agile, allowing you to quickly develop your applications
Being a document-based database means you can store all the information regarding your model in a single document
You can change the schema of your database on the fly
Many relational database functionalities are also available in MongoDB (e.g. indexing)

As for the operations side of things, there are quite a few tools and features for MongoDB that you just can’t find with any other database system:

Whether you need a standalone server or complete clusters of independent servers, MongoDB is as scalable as you need it to be
MongoDB also provides load balancing support by automatically moving data across the various shards
It has automatic failover support - in case your primary server goes down, a new primary will be up and running automatically
The MongoDB Management Service or MMS is a very nice web tool that provides you with the ability to track your machines
Thanks to the memory mapped files, you’ll save quite a bit of RAM, unlike relational databases

If you take advantage of the indexing features, much of this data will be kept in memory for quick retrieval. And even without indexing on specific document keys, Mongo caches quite a bit of data using the least recently used method.

While at first Mongo may seem like it’s the solution to many of our database problems, it isn’t without its drawbacks. One common drawback you’ll hear about Mongo is its lack of support for ACID transactions. Mongo does support ACID transactions in a limited sense, but not in all cases. At the single-document level, ACID transactions are supported (which is where most transactions take place anyway). However, transactions dealing with multiple documents are not supported due to Mongo’s distributed nature.

Mongo also lacks support for native joins, which must be done manually (and therefore much more slowly). Documents are meant to be all-encompassing, which means, in general, they shouldn’t need to reference other documents. In the real world this doesn’t always work as much of the data we work with is relational by nature. Therefore many will argue that Mongo should be used as a complementary database to a SQL DB, but as you use MongoDB you’ll find that is not necessarily true.

PyMongo

Now that we’ve described what MongoDB is exactly, let’s find out how you’d actually use it with Python. The official driver published by the Mongo developers is called PyMongo. This is a good place to start when first firing Python up with MongoDB. We’ll be going through some examples here, but you should also check out the complete documentation since we won’t be able to cover everything.

The first thing you’ll want to do is to install PyMongo in your virtual environment. The easiest way to do this is with pip:

$ pip install pymongo==3.4.0

NOTE: For a more comprehensive guide check out the Installing / Upgrading page of the docs and follow the steps there to get set up.

Once you are done with the setup, start your Python console and run the following command:

>>> import pymongo

If it runs without raising any exception within the Python shell then your install worked just fine. If not, then carefully perform the steps again. Exit the shell once done.

Next, you have to install the actual MongoDB database.

If you’re on a Mac, we recommend using Homebrew, but you may have a different preference. If using Homebrew, run this command:

$ brew install mongodb

Once done, follow the directions here to set up the data directory for storing data locally.

If you’re not using a Mac, you can find some great guides on installation from the Install MongoDB page of the official docs. There you’ll find tutorials on installing MongoDB for Linux, OS X, and Windows.

Once installed, within a new terminal window, use the following command to start the Mongo daemon:

$ mongod

NOTE: Depending on your install method, you may need to run mongod with sudo.

Now let’s get started with the basics of PyMongo.

Establishing a Connection

To establish a connection we’ll use the MongoClient object.

The first thing that we need to do in order to establish a connection is import the MongoClient class. We’ll use this to communicate with the running database instance. Use the following code to do so:

from pymongo import MongoClient
client = MongoClient()

Using the snippet above, the connection will be established to the default host (localhost) and port (27017). You can also specify the host and/or port using:

client = MongoClient('localhost', 27017)

Or just use the Mongo URI format:

client = MongoClient('mongodb://localhost:27017')

All of these calls to MongoClient will do the same thing; it just depends on how explicit you want to be in your code.

Accessing Databases

Once you have a connected instance of MongoClient, you can access any of the databases within that Mongo server. To specify which database you actually want to use, you can access it as an attribute:

db = client.pymongo_test

Or you can also use the dictionary-style access:

db = client['pymongo_test']

It doesn’t actually matter if your specified database has been created yet. By specifying this database name and saving data to it, you create the database automatically.

Inserting Documents

Storing data in your database is as easy as calling just two lines of code. The first line specifies which collection you’ll be using (posts in the example below). In MongoDB terminology, a collection is a group of documents that are stored together within the database. Collections and documents are akin to SQL tables and rows, respectively. Retrieving a collection is as easy as getting a database.

The second line is where you actually insert the data in to the collection using the insert_one() method:

posts = db.posts
post_data = {
    'title': 'Python and MongoDB',
    'content': 'PyMongo is fun, you guys',
    'author': 'Scott'
}
result = posts.insert_one(post_data)
print('One post: {0}'.format(result.inserted_id))

We can even insert many documents at a time, which is much faster than using insert_one() if you have many documents to add to the database. The method to use here is insert_many(). This method takes an array of document data:

post_1 = {
    'title': 'Python and MongoDB',
    'content': 'PyMongo is fun, you guys',
    'author': 'Scott'
}
post_2 = {
    'title': 'Virtual Environments',
    'content': 'Use virtual environments, you guys',
    'author': 'Scott'
}
post_3 = {
    'title': 'Learning Python',
    'content': 'Learn Python, it is easy',
    'author': 'Bill'
}
new_result = posts.insert_many([post_1, post_2, post_3])
print('Multiple posts: {0}'.format(new_result.inserted_ids))

When ran, you should see something like:

One post: 584d947dea542a13e9ec7ae6
Multiple posts: [
    ObjectId('584d947dea542a13e9ec7ae7'),
    ObjectId('584d947dea542a13e9ec7ae8'),
    ObjectId('584d947dea542a13e9ec7ae9')
]

NOTE: Don’t worry that your ObjectIds don’t match those shown above. They are dynamically generated when you insert data and consist of a Unix epoch, machine identifier, and other unique data.

Retrieving Documents

To retrieve a document, we’ll use the find_one() method. The lone argument that we’ll use here (although it supports many more) is a dictionary that contains fields to match. In our example below, we want to retrieve the post that was written by Bill:

bills_post = posts.find_one({'author': 'Bill'})
print(bills_post)

Run this:

{
    'author': 'Bill',
    'title': 'Learning Python',
    'content': 'Learn Python, it is easy',
    '_id': ObjectId('584c4afdea542a766d254241')
}

You may have noticed that the post’s ObjectId is set under the _id key, which we can later use to uniquely identify it if needed. If we want to find more than one document, we can use the find() method. This time, let’s find all of the posts written by Scott:

scotts_posts = posts.find({'author': 'Scott'})
print(scotts_posts)

Run:

<pymongo.cursor.Cursor object at 0x109852f98>

The main difference here is that the document data isn’t returned directly to us as an array. Instead we get an instance of the Cursor object. This Cursor is an iterable object that contains quite a few helper methods to help you work with the data. To get each document, just iterate over the result:

for post in scotts_posts:
    print(post)

MongoEngine

While PyMongo is very easy to use and overall a great library, it’s probably a bit too low-level for many projects out there. Put another way, you’ll have to write a lot of your own code to consistently save, retrieve, and delete objects.

One library that provides a higher abstraction on top of PyMongo is MongoEngine. MongoEngine is an object document mapper (ODM), which is roughly equivalent to a SQL-based object relational mapper (ORM). The abstraction provided by MongoEngine is class-based, so all of the models you create are classes.

While quite a few Python libraries exist to help you work with MongoDB, MongoEngine is one of the better ones as it has a nice mix of features, flexibility, and community support without becoming too opinionated.

To install, use pip:

$ pip install mongoengine==0.10.7

Once installed, we need to direct the library to connect with our running instance of Mongo. For this we will have to use the connect() function and pass the host and port of the MongoDB database to it. Within the Python shell, type:

from mongoengine import *
connect('mongoengine_test', host='localhost', port=27017)

Here we specify the name of our database and location. Since we’re still using the default host and port, you can omit these parameters.

Defining a Document

To set up our document object, we need to define what data we want our document object to have. Similar to many other ORMs, we’ll do this by subclassing the Document class and providing the types of data we want:

import datetime

class Post(Document):
    title = StringField(required=True, max_length=200)
    content = StringField(required=True)
    author = StringField(required=True, max_length=50)
    published = DateTimeField(default=datetime.datetime.now)

NOTE: One of the more difficult tasks with database models is validating data. How do you make sure that the data you’re saving conforms to some format you need? Just because a database is said to be schema-less doesn’t mean it is schema-free.

In this simple model, we’ve told MongoEngine that we expect a Post instance to have a title, content, an author, and the date it was published. Now the base Document object can use that information to validate the data we provide it.

So, for example, if we try to save a Post without a title then it’ll throw an Exception and let us know. We can take this even further and add more restrictions, like string length. Notice that some of the fields have a max_length parameter set. This tells the Document, as you probably guessed, to only allow a maximum string length of however many characters we specify. There are quite a few more parameters like this we can set, including:

db_field: Specify a different field name
required: Make sure this field is set
default: Use the given default value if no other value is given
unique: Make sure no other document in the collection has the same value for this field
choices: Make sure the field’s value is equal to one of the values given in an array

Each field type has its own set of parameters, so be sure to check the documentation for more info.

Saving Documents

To save a document to our database, we’ll use the save() method. If the document already exists in the database, then all of the changes will be made on the atomic level to the existing document. If it doesn’t exist, however, then it will be created.

Here is an example of creating and saving a document:

post_1 = Post(
    title='Sample Post',
    content='Some engaging content',
    author='Scott'
)
post_1.save()       # This will perform an insert
print(post_1.title)
post_1.title = 'A Better Post Title'
post_1.save()       # This will perform an atomic edit on "title"
print(post_1.title)

A few things to note about the .save() call:

PyMongo will perform validation when you call .save(). This means it will check the data you’re saving against the schema you declared in the class. If the schema (or a constraint) is violated, then an exception is thrown and the data is not saved.
Since Mongo doesn’t support true transactions, there is no way to “roll back” the .save() call like you can in SQL databases. Although you can get close to performing transactions with two phase commits, they still don’t support rollbacks.

What happens when you leave off the title?

post_2 = Post(content='Content goes here', author='Michael')
post_2.save()

You should see the following exception:

raise ValidationError(message, errors=errors)
mongoengine.errors.ValidationError:
ValidationError (Post:None) (Field is required: ['title'])

Object Oriented Features

With MongoEngine being object oriented, you can also add methods to your subclassed document. Consider the following example where a function is used to modify the default queryset (which returns all objects of the collection). By using this, we can apply a default filter to the class and get only the desired objects:

class Post(Document):
    title = StringField()
    published = BooleanField()

    @queryset_manager
    def live_posts(clazz, queryset):
        return queryset.filter(published=True)

Referencing Other Documents

You can also use the ReferenceField object to create a reference from one document to another. MongoEngine handles the lazy de-referencing automatically upon access, which is more robust and less error-prone than having to remember to do it yourself everywhere in your code. An example:

class Author(Document):
    name = StringField()

class Post(Document):
    author = ReferenceField(Author)

Post.objects.first().author.name

In the code above, using a document reference, we can easily find the author of the first post.

There are quite a few more field classes (and parameters) than what we introduced here, so be sure to check out the documentation on Fields for more info.

From all of these examples you should be able to see that MongoEngine is well suited to manage your database objects for just about any type of application. The features available at the developer’s disposal make it incredibly easy to create an efficient and scalable program. In case you’re looking for more help related to MongoEngine, be sure to check out their comprehensive user guide.

Conclusion

With Python being a high-level, highly scalable, modern language, it needs a database (and driver) that can keep up to its potential, which is why MongoDB is such a good fit.

Free Bonus: Click here to download an example Python project with source code that shows you how to work with MongoDB databases.

We saw in this article how we can exploit the strengths of MongoDB to our advantage and build a highly flexible and scalable application. Feel free to let us know your thoughts in the comments section!

This is a collaboration piece between Scott Robinson, author of Stack Abuse and the folks at Real Python.

Introduction to Python Generators

2016-10-24T13:51:04+00:00

Generators are functions that can be paused and resumed on the fly, returning an object that can be iterated over. Unlike lists, they are lazy and thus produce items one at a time and only when asked. So they are much more memory efficient when dealing with large datasets. This article details how to create generator functions and expressions as well as why you would want to use them in the first place.

Free Bonus: Click here to get access to a free Python OOP Cheat Sheet that points you to the best tutorials, videos, and books to learn more about Object-Oriented Programming with Python.

Generator Functions

To create a generator, you define a function as you normally would but use the yield statement instead of return, indicating to the interpreter that this function should be treated as an iterator:

def countdown(num):
    print('Starting')
    while num > 0:
        yield num
        num -= 1

The yield statement pauses the function and saves the local state so that it can be resumed right where it left off.

What happens when you call this function?

>>> def countdown(num):
...     print('Starting')
...     while num > 0:
...         yield num
...         num -= 1
...
>>> val = countdown(5)
>>> val
<generator object countdown at 0x10213aee8>

Calling the function does not execute it. We know this because the string Starting did not print. Instead, the function returns a generator object which is used to control execution.

Generator objects execute when next() is called:

>>> next(val)
Starting
5

When calling next() the first time, execution begins at the start of the function body and continues until the next yield statement where the value to the right of the statement is returned, subsequent calls to next() continue from the yield statement to the end of the function, and loop around and continue from the start of the function body until another yield is called. If yield is not called (which in our case means we don’t go into the if function because num <= 0) a StopIteration exception is raised:

>>> next(val)
4
>>> next(val)
3
>>> next(val)
2
>>> next(val)
1
>>> next(val)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Generator Expressions

Just like list comprehensions, generators can also be written in the same manner except they return a generator object rather than a list:

>>> my_list = ['a', 'b', 'c', 'd']
>>> gen_obj = (x for x in my_list)
>>> for val in gen_obj:
...     print(val)
...
a
b
c
d

Take note of the parens on either side of the second line denoting a generator expression, which, for the most part, does the same thing that a list comprehension does, but does it lazily:

>>> import sys
>>> g = (i * 2 for i in range(10000) if i % 3 == 0 or i % 5 == 0)
>>> print(sys.getsizeof(g))
72
>>> l = [i * 2 for i in range(10000) if i % 3 == 0 or i % 5 == 0]
>>> print(sys.getsizeof(l))
38216

Be careful not to mix up the syntax of a list comprehension with a generator expression - [] vs () - since generator expressions can run slower than list comprehensions (unless you run out of memory, of course):

>>> import cProfile
>>> cProfile.run('sum((i * 2 for i in range(10000000) if i % 3 == 0 or i % 5 == 0))')
         4666672 function calls in 3.531 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  4666668    2.936    0.000    2.936    0.000 <string>:1(<genexpr>)
        1    0.001    0.001    3.529    3.529 <string>:1(<module>)
        1    0.002    0.002    3.531    3.531 {built-in method exec}
        1    0.592    0.592    3.528    3.528 {built-in method sum}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}


>>> cProfile.run('sum([i * 2 for i in range(10000000) if i % 3 == 0 or i % 5 == 0])')
         5 function calls in 3.054 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    2.725    2.725    2.725    2.725 <string>:1(<listcomp>)
        1    0.078    0.078    3.054    3.054 <string>:1(<module>)
        1    0.000    0.000    3.054    3.054 {built-in method exec}
        1    0.251    0.251    0.251    0.251 {built-in method sum}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

This is particularly easy (even for senior developers) to do in the above example since both output the exact same thing in the end.

NOTE: Keep in mind that generator expressions are drastically faster when the size of your data is larger than the available memory.

Use Cases

Generators are perfect for reading a large number of large files since they yield out data a single chunk at a time irrespective of the size of the input stream. They can also result in cleaner code by decoupling the iteration process into smaller components.

Example 1

def emit_lines(pattern=None):
    lines = []
    for dir_path, dir_names, file_names in os.walk('test/'):
        for file_name in file_names:
            if file_name.endswith('.py'):
                for line in open(os.path.join(dir_path, file_name)):
                    if pattern in line:
                        lines.append(line)
    return lines

This function loops through a set of files in the specified directory. It opens each file and then loops through each line to test for the pattern match.

This works fine with a small number of small files. But, what if we’re dealing with extremely large files? And what if there are a lot of them? Fortunately, Python’s open() function is efficient and doesn’t load the entire file into memory. But what if our matches list far exceeds the available memory on our machine?

So, instead of running out of space (large lists) and time (nearly infinite amount of data stream) when processing large amounts of data, generators are the ideal things to use, as they yield out data one time at a time (instead of creating intermediate lists).

Let’s look at the generator version of the above problem and try to understand why generators are apt for such use cases using processing pipelines.

We divided our whole process into three different components:

Generating set of filenames
Generating all lines from all files
Filtering out lines on the basis of pattern matching

def generate_filenames():
    """
    generates a sequence of opened files
    matching a specific extension
    """
    for dir_path, dir_names, file_names in os.walk('test/'):
        for file_name in file_names:
            if file_name.endswith('.py'):
                yield open(os.path.join(dir_path, file_name))

def cat_files(files):
    """
    takes in an iterable of filenames
    """
    for fname in files:
        for line in fname:
            yield line

def grep_files(lines, pattern=None):
    """
    takes in an iterable of lines
    """
    for line in lines:
        if pattern in line:
            yield line


py_files = generate_filenames()
py_file = cat_files(py_files)
lines = grep_files(py_file, 'python')
for line in lines:
    print (line)

In the above snippet, we do not use any extra variables to form the list of lines, instead we create a pipeline which feeds its components via the iteration process one item at a time. grep_files takes in a generator object of all the lines of *.py files. Similarly, cat_files takes in a generator object of all the filenames in a directory. So this is how the whole pipeline is glued via iterations.

Example 2

Generators work great for web scraping and crawling recursively:

import requests
import re


def get_pages(link):
    links_to_visit = []
    links_to_visit.append(link)
    while links_to_visit:
        current_link = links_to_visit.pop(0)
        page = requests.get(current_link)
        for url in re.findall('<a href="([^"]+)">', str(page.content)):
            if url[0] == '/':
                url = current_link + url[1:]
            pattern = re.compile('https?')
            if pattern.match(url):
                links_to_visit.append(url)
        yield current_link


webpage = get_pages('http://sample.com')
for result in webpage:
    print(result)

Here, we simply fetch a single page at a time and then perform some sort of action on the page when execution occurs. What would this look like without a generator? Either the fetching and processing would have to happen within the same function (resulting in highly coupled code that’s hard to test) or we’d have to fetch all the links before processing a single page.

Conclusion

Generators allow us to ask for values as and when we need them, making our applications more memory efficient and perfect for infinite streams of data. They can also be used to refactor out the processing from loops resulting in cleaner, decoupled code. If you’d like to see more examples, check out Generator Tricks for Systems Programmers and Iterator Chains as Pythonic Data Processing Pipelines.

Free Bonus: Click here to get access to a free Python OOP Cheat Sheet that points you to the best tutorials, videos, and books to learn more about Object-Oriented Programming with Python.

How have you used generators in your own projects?

Caching in Django With Redis

2016-09-20T13:37:47+00:00

Application performance is vital to the success of your product. In an environment where users expect website response times of less than a second, the consequences of a slow application can be measured in dollars and cents. Even if you are not selling anything, fast page loads improve the experience of visiting your site.

Everything that happens on the server between the moment it receives a request to the moment it returns a response increases the amount of time it takes to load a page. As a general rule of thumb, the more processing you can eliminate on the server, the faster your application will perform. Caching data after it has been processed and then serving it from the cache the next time it is requested is one way to relieve stress on the server. In this tutorial, we will explore some of the factors that bog down your application, and we will demonstrate how to implement caching with Redis to counteract their effects.

Free Bonus: Click here to get access to a free list of 35 additional Django tutorials and resources you can use to deepen your Python web development skills.

What is Redis?

Redis is an in-memory data structure store that can be used as a caching engine. Since it keeps data in RAM, Redis can deliver it very quickly. Redis is not the only product that we can use for caching. Memcached is another popular in-memory caching system, but many people agree that Redis is superior to Memcached in most circumstances. Personally, we like how easy it is to set up and use Redis for other purposes such as Redis Queue.

Getting Started

We have created an example application to introduce you to the concept of caching. Our application uses:

Django (v1.9.8)
Django Debug Toolbar (v1.4)
django-redis (v4.4.3)
Redis (v3.2.0)

Install the App

Before you clone the repository, install virtualenvwrapper, if you don’t already have it. This is a tool that lets you install the specific Python dependencies that your project needs, allowing you to target the versions and libraries required by your app in isolation.

Next, change directories to where you keep projects and clone the example app repository. Once done, change directories to the cloned repository, and then make a new virtual environment for the example app using the mkvirtualenv command:

$ mkvirtualenv django-redis
(django-redis)$

NOTE: Creating a virtual environment with mkvirtualenv also activates it.

Install all of the required Python dependencies with pip, and then checkout the following tag:

(django-redis)$ git checkout tags/1

Finish setting up the example app by building the database and populating it with sample data. Make sure to create a superuser too, so that you can log into the admin site. Follow the code examples below and then try running the app to make sure it is working correctly. Visit the admin page in the browser to confirm that the data has been properly loaded.

(django-redis)$ python manage.py makemigrations cookbook
(django-redis)$ python manage.py migrate
(django-redis)$ python manage.py createsuperuser
(django-redis)$ python manage.py loaddata cookbook/fixtures/cookbook.json
(django-redis)$ python manage.py runserver

Once you have the Django app running, move onto the Redis installation.

Install Redis

Download and install Redis using the instructions provided in the documentation. Alternatively, you can install Redis using a package manager such as apt-get or homebrew depending on your OS.

Run the Redis server from a new terminal window.

$ redis-server

Next, start up the Redis command-line interface (CLI) in a different terminal window and test that it connects to the Redis server. We will be using the Redis CLI to inspect the keys that we add to the cache.

$ redis-cli ping
PONG

Redis provides an API with various commands that a developer can use to act on the data store. Django uses django-redis to execute commands in Redis.

Looking at our example app in a text editor, we can see the Redis configuration in the settings.py file. We define a default cache with the CACHES setting, using a built-in django-redis cache as our backend. Redis runs on port 6379 by default, and we point to that location in our setting. One last thing to mention is that django-redis appends key names with a prefix and a version to help distinguish similar keys. In this case, we have defined the prefix to be “example”.

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379/1",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient"
        },
        "KEY_PREFIX": "example"
    }
}

NOTE: Although we have configured the cache backend, none of the view functions have implemented caching.

App Performance

As we mentioned at the beginning of this tutorial, everything that the server does to process a request slows the application load time. The processing overhead of running business logic and rendering templates can be significant. Network latency affects the time it takes to query a database. These factors come into play every time a client sends an HTTP request to the server. When users are initiating many requests per second, the effects on performance become noticeable as the server works to process them all.

When we implement caching, we let the server process a request once and then we store it in our cache. As requests for the same URL are received by our application, the server pulls the results from the cache instead of processing them anew each time. Typically, we set a time to live on the cached results, so that the data can be periodically refreshed, which is an important step to implement in order to avoid serving stale data.

You should consider caching the result of a request when the following cases are true:

rendering the page involves a lot of database queries and/or business logic,
the page is visited frequently by your users,
the data is the same for every user,
and the data does not change often.

Start by Measuring Performance

Begin by testing the speed of each page in your application by benchmarking how quickly your application returns a response after receiving a request.

To achieve this, we’ll be blasting each page with a burst of requests using loadtest, an HTTP load generator, and then paying close attention to the request rate. Visit the link above to install. Once installed, test the results against the /cookbook/ URL path:

$ loadtest -n 100 -k  http://localhost:8000/cookbook/

Notice that we are processing about 16 requests per second:

Requests per second: 16

When we look at what the code is doing, we can make decisions on how to make changes to improve the performance. The application makes 3 network calls to a database with each request to /cookbook/, and it takes time for each call to open a connection and execute a query. Visit the /cookbook/ URL in your browser and expand the Django Debug Toolbar tab to confirm this behavior. Find the menu labeled “SQL” and read the number of queries:

cookbook/services.py

from cookbook.models import Recipe


def get_recipes():
    # Queries 3 tables: cookbook_recipe, cookbook_ingredient,
    # and cookbook_food.
    return list(Recipe.objects.prefetch_related('ingredient_set__food'))

cookbook/views.py

from django.shortcuts import render
from cookbook.services import get_recipes


def recipes_view(request):
    return render(request, 'cookbook/recipes.html', {
        'recipes': get_recipes()
    })

The application also renders a template with some potentially expensive logic.

<html>
<head>
  <title>Recipes</title>
</head>
<body>
{% for recipe in recipes %}
  <h1>{{ recipe.name }}</h1>
    <p>{{ recipe.desc }}</p>
  <h2>Ingredients</h2>
  <ul>
    {% for ingredient in recipe.ingredient_set.all %}
    <li>{{ ingredient.desc }}</li>
    {% endfor %}
  </ul>
  <h2>Instructions</h2>
    <p>{{ recipe.instructions }}</p>
{% endfor %}
</body>
</html>

Implement Caching

Imagine the total number of network calls that our application will make as users start to visit our site. If 1,000 users hit the API that retrieves cookbook recipes, then our application will query the database 3,000 times and a new template will be rendered with each request. That number only grows as our application scales. Luckily, this view is a great candidate for caching. The recipes in a cookbook rarely change, if ever. Also, since viewing cookbooks is the central theme of the app, the API retrieving the recipes is guaranteed to be called frequently.

In the example below, we modify the view function to use caching. When the function runs, it checks if the view key is in the cache. If the key exists, then the app retrieves the data from the cache and returns it. If not, Django queries the database and then stashes the result in the cache with the view key. The first time this function is run, Django will query the database and render the template, and then will also make a network call to Redis to store the data in the cache. Each subsequent call to the function will completely bypass the database and business logic and will query the Redis cache.

example/settings.py

# Cache time to live is 15 minutes.
CACHE_TTL = 60 * 15

cookbook/views.py

from django.conf import settings
from django.core.cache.backends.base import DEFAULT_TIMEOUT
from django.shortcuts import render
from django.views.decorators.cache import cache_page
from cookbook.services import get_recipes

CACHE_TTL = getattr(settings, 'CACHE_TTL', DEFAULT_TIMEOUT)


@cache_page(CACHE_TTL)
def recipes_view(request):
    return render(request, 'cookbook/recipes.html', {
        'recipes': get_recipes()
    })

Notice that we have added the @cache_page() decorator to the view function, along with a time to live. Visit the /cookbook/ URL again and examine the Django Debug Toolbar. We see that 3 database queries are made and 3 calls are made to the cache in order to check for the key and then to save it. Django saves two keys (1 key for the header and 1 key for the rendered page content). Reload the page and observe how the page activity changes. The second time around, 0 calls are made to the database and 2 calls are made to the cache. Our page is now being served from the cache!

When we re-run our performance tests, we see that our application is loading faster.

$ loadtest -n 100 -k  http://localhost:8000/cookbook/

Caching improved the total load, and we are now resolving 21 requests per second, which is 5 more than our baseline:

Requests per second: 21

Inspecting Redis with the CLI

At this point we can use the Redis CLI to look at what gets stored on the Redis server. In the Redis command-line, enter the keys * command, which returns all keys matching any pattern. You should see a key called “example:1:views.decorators.cache.cache_page”. Remember, “example” is our key prefix, “1” is the version, and “views.decorators.cache.cache_page” is the name that Django gives the key. Copy the key name and enter it with the get command. You should see the rendered HTML string.

$ redis-cli -n 1
127.0.0.1:6379[1]> keys *
1) "example:1:views.decorators.cache.cache_header"
2) "example:1:views.decorators.cache.cache_page"
127.0.0.1:6379[1]> get "example:1:views.decorators.cache.cache_page"

NOTE: Run the flushall command on the Redis CLI to clear all of the keys from the data store. Then, you can run through the steps in this tutorial again without having to wait for the cache to expire.

Wrap-up

Processing HTTP requests is costly, and that cost adds up as your application grows in popularity. In some instances, you can greatly reduce the amount of processing your server does by implementing caching. This tutorial touched on the basics of caching in Django with Redis, but it only skimmed the surface of a complex topic.

Implementing caching in a robust application has many pitfalls and gotchas. Controlling what gets cached and for how long is tough. Cache invalidation is one of the hard things in Computer Science. Ensuring that private data can only be accessed by its intended users is a security issue and must be handled very carefully when caching.

Free Bonus: Click here to get free access to additional Django tutorials and resources you can use to deepen your Python web development skills.

Play around with the source code in the example application and as you continue to develop with Django, remember to always keep performance in mind.

Flask by Example – Custom Angular Directive with D3

2016-08-30T13:04:04+00:00

Welcome back. With Angular set up along with a loading spinner and our refactored Angular controller, let’s move on to the final part and create a custom Angular Directive to display a frequency distribution chart with JavaScript and the D3 library.

Remember: Here’s what we’re building - A Flask app that calculates word-frequency pairs based on the text from a given URL.

Part One: Set up a local development environment and then deploy both a staging and a production environment on Heroku.
Part Two: Set up a PostgreSQL database along with SQLAlchemy and Alembic to handle migrations.
Part Three: Add in the back-end logic to scrape and then process the word counts from a webpage using the requests, BeautifulSoup, and Natural Language Toolkit (NLTK) libraries.
Part Four: Implement a Redis task queue to handle the text processing.
Part Five: Set up Angular on the front-end to continuously poll the back-end to see if the request is done processing.
Part Six: Push to the staging server on Heroku - setting up Redis and detailing how to run two processes (web and worker) on a single Dyno.
Part Seven: Update the front-end to make it more user-friendly.
Part Eight: Create a custom Angular Directive to display a frequency distribution chart using JavaScript and D3. (current)

Need the code? Grab it from the repo.

Let’s look at what we currently have…

Current User Interface

Start Redis in a terminal window:

$ redis-server

Then get your process worker going in another window:

$ cd flask-by-example
$ python worker.py
17:11:39 RQ worker started, version 0.4.6
17:11:39
17:11:39 *** Listening on default...

Finally, in a third window, fire up the app:

$ cd flask-by-example
$ python manage.py runserver

You should see your word counter working. Now we can add in a custom Angular Directive to display the results in a D3 chart.

Angular Directive

Start by adding the D3 library (v3) to the index.html file:

<!-- scripts -->
<script src="//d3js.org/d3.v3.min.js" charset="utf-8"></script>
<script src="//code.jquery.com/jquery-2.2.1.min.js"></script>
<script src="//netdna.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"></script>
<script src="//ajax.googleapis.com/ajax/libs/angularjs/1.4.5/angular.min.js"></script>
<script src="{{ url_for('static', filename='main.js') }}"></script>

Now let’s set up a new custom Directive.

Angular Directives are markers on a DOM element, which allow us to insert sections of HTML with specific events and attributes attached to it. Let’s build out the first part of our Directive by adding the following code just below the controller in main.js:

.directive('wordCountChart', ['$parse', function ($parse) {
  return {
    restrict: 'E',
    replace: true,
    template: '<div id="chart"></div>',
    link: function (scope) {}
   };
}]);

restrict: 'E' creates a Directive that is restricted to an HTML element. replace: true simply replaces the HTML Directive with the HTML in the template. The link function gives us access to variables in the scope defined in the controller.

Next, add a watch function to “watch” for any changes to the variables and respond appropriately. Add this to the link function like so:

link: function (scope) {
  scope.$watch('wordcounts', function() {
    // add code here
  }, true);
}

Finally, add the Directive just below the closing divider to <div class="row">:

<br>
<word-count-chart data="wordcounts"></word-count-chart>

With the Directive set up, let’s turn our attention to the D3 library…

D3 Bar Chart

D3 is a powerful library that utilizes HTML, CSS, and SVG to display data on the DOM and JavaScript to make it interactive. We will use it to create a basic bar chart.

Step 1: Functional Logic

Add the following to the watch function within the Angular Directive:

scope.$watch('wordcounts', function() {
  var data = scope.wordcounts;
  for (var word in data) {
    d3.select('#chart')
      .append('div')
      .selectAll('div')
      .data(word[0])
      .enter()
      .append('div');
  }
}, true);

Now, whenever scope.wordcounts changes, this function is fired, which updates the DOM. Since an object is returned from the AJAX request, we iterate through it to add the specific data to the chart. Essentially, every word is appended to a new div via a data join.

Try running the code.

What happens? Nothing shows up, right? Check out the DOM in Chrome’s Developer Tools, after you submit a new site. You should see a number of nested divs. We just need to add styles…

Step 2: Styling the Bar Chart

Start with some simple CSS:

#chart {
  overflow-y: scroll;
}

#chart {
  background: #eee;
  padding: 3px;
}

#chart div {
  width: 0;
  transition: all 1s ease-out;
  -moz-transition: all 1s ease-out;
  -webkit-transition: all 1s ease-out;
}

#chart div {
  height: 30px;
  font: 15px;
  background-color: #006dcc;
  text-align: right;
  padding: 3px;
  color: white;
  box-shadow: 2px 2px 2px gray;
}

Make sure to include this at the top of HTML page, after the Bootstrap stylesheet:

<link rel="stylesheet" type="text/css" href="../static/main.css">

Fire up the app in our browser. What’s happening now?

When you search for a website, you should now see a grey area with some thin blue bars on the left hand side. So you can see that we are generating a bar for each data element we’re getting back - 10 in total. However, we need to modify our D3 code in order to increase the width of each bar so they are readable.

Step 3: Making the Bar Chart More Interactive

We can chain this on to our existing code and use the D3 style function:

scope.$watch('wordcounts', function() {
  var data = scope.wordcounts;
  for (var word in data) {
    d3.select('#chart')
      .append('div')
      .selectAll('div')
      .data(word[0])
      .enter()
      .append('div')
      .style('width', function() {
        return (data[word] * 20) + 'px';
      })
      .text(function(d){
        return word;
      });
  }
}, true);

Now we are dynamically creating a width based on the numeric value of how often a word shows up on a webpage:

.style('width', function() {
  return (data[word] * 20) + 'px';
})
.text(function(d){
  return word;
});

The style is calculated by returning the value associated with each word, multiplying that number by 20, and then converting it into pixels. We can also add text to each bar element by inserting the string value of the word along with how often it shows up on the page.

Try this out. You should see something like:

There’s still one thing missing though. What happens when you search for a new website? Try it. The new chart is appended beneath the previous one. We need to clear out our chart div before a new one is created.

Step 4: Clean Up for the Next URL Search

Update the link function in the Directive:

link: function (scope) {
  scope.$watch('wordcounts', function() {
    d3.select('#chart').selectAll('*').remove();
    var data = scope.wordcounts;
    for (var word in data) {
      d3.select('#chart')
        .append('div')
        .selectAll('div')
        .data(word[0])
        .enter()
        .append('div')
        .style('width', function() {
          return (data[word] * 20) + 'px';
        })
        .text(function(d){
          return word;
        });
    }
  }, true);
}

d3.select('#chart').selectAll('*').remove(); simply clears out the chart each time the $scope.watch function is fired. Now we have a chart that is cleared before each new use, and we have a fully functional word count application!!

Test it out!

Conclusion and Next Steps

That’s it. Push your changes to the staging and production servers. Let’s review what we tackled:

We started with the configuration and workflow, setting up staging and production servers
From there, we added the basic functionality - web scraping, data analysis - and set up a task queue with Redis
With the back-end functionality set up, attention turned to the front-end where we added Angular, built a custom Directive, and added D3 into the mix

We have an MVP, but there’s still much to be done:

Refactor, refactor, refactor!
Write tests
Handle errors and exceptions
Abstract out state in the Angular app to a Service
Work on the UI and UX

Want to help? Add a feature, write part 9, get paid, and become Internet famous!

Links:

Testing External APIs With Mock Servers

2016-07-06T13:42:58+00:00

Despite being so useful, external APIs can be a pain to test. When you hit an actual API, your tests are at the mercy of the external server, which can result in the following pain points:

The request-response cycle can take several seconds. That might not seem like much at first, but the time compounds with each test. Imagine calling an API 10, 50, or even 100 times when testing your entire application.
The API may have rate limits set up.
The API server may be unreachable. Maybe the server is down for maintenance? Maybe it failed with an error and the development team is working to get it functional again> Do you really want the success of your tests to rely on the health of a server that you don’t control?

Your tests shouldn’t assess whether an API server is running; they should test whether your code is operating as expected.

In the previous tutorial, we introduced the concept of mock objects, demonstrated how you could use them to test code that interacts with external APIs. This tutorial builds on the same topics, but here we walk you through how to actually build a mock server rather than mocking the APIs. With a mock server in place, you can perform end-to-end tests. You can use your application and get actual feedback from the mock server in real time.

When you finish working through the following examples, you will have programmed a basic mock server and two tests - one that uses the real API server and one that uses the mock server. Both tests will access the same service, an API that retrieves a list of users.

NOTE: This tutorial uses Python v3.5.1.

Getting Started

Start by following the First steps section of the previous post. Or grab the code from the repository. Make sure the test passes before moving on:

$ nosetests --verbosity=2 project
test_todos.test_request_response ... ok

----------------------------------------------------------------------
Ran 1 test in 1.029s

OK

Testing the Mock API

With the set up complete, you can program your mock server. Write a test that describes the behavior:

project/tests/test_mock_server.py

# Third-party imports...
from nose.tools import assert_true
import requests


def test_request_response():
    url = 'http://localhost:{port}/users'.format(port=mock_server_port)

    # Send a request to the mock API server and store the response.
    response = requests.get(url)

    # Confirm that the request-response cycle completed successfully.
    assert_true(response.ok)

Notice that it starts off looking almost identical to the real API test. The URL has changed and is now pointing to an API endpoint on localhost where the mock server will run.

Here is how to create a mock server in Python:

project/tests/test_mock_server.py

# Standard library imports...
from http.server import BaseHTTPRequestHandler, HTTPServer
import socket
from threading import Thread

# Third-party imports...
from nose.tools import assert_true
import requests


class MockServerRequestHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        # Process an HTTP GET request and return a response with an HTTP 200 status.
        self.send_response(requests.codes.ok)
        self.end_headers()
        return


def get_free_port():
    s = socket.socket(socket.AF_INET, type=socket.SOCK_STREAM)
    s.bind(('localhost', 0))
    address, port = s.getsockname()
    s.close()
    return port


class TestMockServer(object):
    @classmethod
    def setup_class(cls):
        # Configure mock server.
        cls.mock_server_port = get_free_port()
        cls.mock_server = HTTPServer(('localhost', cls.mock_server_port), MockServerRequestHandler)

        # Start running mock server in a separate thread.
        # Daemon threads automatically shut down when the main process exits.
        cls.mock_server_thread = Thread(target=cls.mock_server.serve_forever)
        cls.mock_server_thread.setDaemon(True)
        cls.mock_server_thread.start()

    def test_request_response(self):
        url = 'http://localhost:{port}/users'.format(port=self.mock_server_port)

        # Send a request to the mock API server and store the response.
        response = requests.get(url)

        # Confirm that the request-response cycle completed successfully.
        print(response)
        assert_true(response.ok)

First, create a subclass of BaseHTTPRequestHandler. This class captures the request and constructs the response to return. Override the do_GET() function to craft the response for an HTTP GET request. In this case, just return an OK status. Next, write a function to get an available port number for the mock server to use.

The next block of code actually configures the server. Notice how the code instantiates an HTTPServer instance and passes it a port number and a handler. Next, create a thread, so that the server can be run asynchronously and your main program thread can communicate with it. Make the thread a daemon, which tells the thread to stop when the main program exits. Finally, start the thread to serve the mock server forever (until the tests finish).

Create a test class and move the test function to it. You must add an additional method to ensure that the mock server is launched before any of the tests run. Notice that this new code lives within a special class-level function, setup_class().

Run the tests and watch them pass:

$ nosetests --verbosity=2 project

Testing a Service that Hits the API

You probably want to call more than one API endpoint in your code. As you design your app, you will likely create service functions to send requests to an API and then process the responses in some way. Maybe you will store the response data in a database. Or you will pass the data to a user interface.

Refactor your code to pull the hardcoded API base URL into a constant. Add this variable to a constants.py file:

project/constants.py

BASE_URL = 'http://jsonplaceholder.typicode.com'

Next, encapsulate the logic to retrieve users from the API into a function. Notice how new URLs can be created by joining a URL path to the base.

project/services.py

# Standard library imports...
from urllib.parse import urljoin

# Third-party imports...
import requests

# Local imports...
from project.constants import BASE_URL

USERS_URL = urljoin(BASE_URL, 'users')


def get_users():
    response = requests.get(USERS_URL)
    if response.ok:
        return response
    else:
        return None

Move the mock server code from the feature file to a new Python file, so that it can easily be reused. Add conditional logic to the request handler to check which API endpoint the HTTP request is targeting. Beef up the response by adding some simple header information and a basic response payload. The server creation and kick off code can be encapsulated in a convenience method, start_mock_server().

project/tests/mocks.py

# Standard library imports...
from http.server import BaseHTTPRequestHandler, HTTPServer
import json
import re
import socket
from threading import Thread

# Third-party imports...
import requests


class MockServerRequestHandler(BaseHTTPRequestHandler):
    USERS_PATTERN = re.compile(r'/users')

    def do_GET(self):
        if re.search(self.USERS_PATTERN, self.path):
            # Add response status code.
            self.send_response(requests.codes.ok)

            # Add response headers.
            self.send_header('Content-Type', 'application/json; charset=utf-8')
            self.end_headers()

            # Add response content.
            response_content = json.dumps([])
            self.wfile.write(response_content.encode('utf-8'))
            return


def get_free_port():
    s = socket.socket(socket.AF_INET, type=socket.SOCK_STREAM)
    s.bind(('localhost', 0))
    address, port = s.getsockname()
    s.close()
    return port


def start_mock_server(port):
    mock_server = HTTPServer(('localhost', port), MockServerRequestHandler)
    mock_server_thread = Thread(target=mock_server.serve_forever)
    mock_server_thread.setDaemon(True)
    mock_server_thread.start()

With your changes to the logic completed, alter the tests to use the new service function. Update the tests to check the increased information that is being passed back from the server.

project/tests/test_real_server.py

# Third-party imports...
from nose.tools import assert_dict_contains_subset, assert_is_instance, assert_true

# Local imports...
from project.services import get_users


def test_request_response():
    response = get_users()

    assert_dict_contains_subset({'Content-Type': 'application/json; charset=utf-8'}, response.headers)
    assert_true(response.ok)
    assert_is_instance(response.json(), list)

project/tests/test_mock_server.py

# Third-party imports...
from unittest.mock import patch
from nose.tools import assert_dict_contains_subset, assert_list_equal, assert_true

# Local imports...
from project.services import get_users
from project.tests.mocks import get_free_port, start_mock_server


class TestMockServer(object):
    @classmethod
    def setup_class(cls):
        cls.mock_server_port = get_free_port()
        start_mock_server(cls.mock_server_port)

    def test_request_response(self):
        mock_users_url = 'http://localhost:{port}/users'.format(port=self.mock_server_port)

        # Patch USERS_URL so that the service uses the mock server URL instead of the real URL.
        with patch.dict('project.services.__dict__', {'USERS_URL': mock_users_url}):
            response = get_users()

        assert_dict_contains_subset({'Content-Type': 'application/json; charset=utf-8'}, response.headers)
        assert_true(response.ok)
        assert_list_equal(response.json(), [])

Notice a new technique being used in the test_mock_server.py code. The response = get_users() line is wrapped with a patch.dict() function from the mock library.

What does this statement do?

Remember, you moved the requests.get() function from the feature logic to the get_users() service function. Internally, get_users() calls requests.get() using the USERS_URL variable. The patch.dict() function temporarily replaces the value of the USERS_URL variable. In fact, it does so only within the scope of the with statement. After that code runs, the USERS_URL variable is restored to its original value. This code patches the URL to use the mock server address.

Run the tests and watch them pass.

$ nosetests --verbosity=2
test_mock_server.TestMockServer.test_request_response ... 127.0.0.1 - - [05/Jul/2016 20:45:30] "GET /users HTTP/1.1" 200 -
ok
test_real_server.test_request_response ... ok
test_todos.test_request_response ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.871s

OK

Skipping Tests that Hit the Real API

We began this tutorial describing the merits of testing a mock server instead of a real one, however, your code currently tests both. How do you configure the tests to ignore the real server? The Python ‘unittest’ library provides several functions that allow you to skip tests. You can use the conditional skip function ‘skipIf’ along with an environment variable to toggle the real server tests on and off. In the following example, we pass a tag name that should be ignored:

$ export SKIP_TAGS=real

project/constants.py

# Standard-library imports...
import os


BASE_URL = 'http://jsonplaceholder.typicode.com'
SKIP_TAGS = os.getenv('SKIP_TAGS', '').split()

project/tests/test_real_server.py

# Standard library imports...
from unittest import skipIf

# Third-party imports...
from nose.tools import assert_dict_contains_subset, assert_is_instance, assert_true

# Local imports...
from project.constants import SKIP_TAGS
from project.services import get_users


@skipIf('real' in SKIP_TAGS, 'Skipping tests that hit the real API server.')
def test_request_response():
    response = get_users()

    assert_dict_contains_subset({'Content-Type': 'application/json; charset=utf-8'}, response.headers)
    assert_true(response.ok)
    assert_is_instance(response.json(), list)

Run the tests and pay attention to how the real server test is ignored:

$ nosetests --verbosity=2 project
test_mock_server.TestMockServer.test_request_response ... 127.0.0.1 - - [05/Jul/2016 20:52:18] "GET /users HTTP/1.1" 200 -
ok
test_real_server.test_request_response ... SKIP: Skipping tests that hit the real API server.
test_todos.test_request_response ... ok

----------------------------------------------------------------------
Ran 3 tests in 1.196s

OK (SKIP=1)

Next Steps

Now that you have created a mock server to test your external API calls, you can apply this knowledge to your own projects. Build upon the simple tests created here. Expand the functionality of the handler to mimic the behavior of the real API more closely.

Try the following exercises to level up:

Return a response with a status of HTTP 404 (not found) if a request is sent with an unknown path.
Return a response with a status of HTTP 405 (method not allowed) if a request is sent with a method that is not allowed (POST, DELETE, UPDATE).
Return actual user data for a valid request to /users.
Write tests to capture those scenarios.

Grab the code from the repo.

Mocking External APIs in Python

2016-06-20T13:35:57+00:00

The following tutorial demonstrates how to test the use of an external API using Python mock objects.

Integrating with a third-party application is a great way to extend the functionality of your product.

However, the added value also comes with obstacles. You do not own the external library, which means that you cannot control the servers that host it, the code that comprises its logic, or the data that gets transferred between it and your app. On top of those issues, users are constantly manipulating the data through their interactions with the library.

If you want to enhance the utility of your application with a third-party API, then you need to be confident that the two systems will play nice. You need to test that the two applications interface in predictable ways, and you need your tests to execute in a controlled environment.

Free Bonus: Click here to download a copy of the "REST API Examples" Guide and get a hands-on introduction to Python + REST API principles with actionable examples.

At first glance, it might seem like you do not have any control over a third-party application. Many of them do not offer testing servers. You cannot test live data, and even if you could, the tests would return unreliable results as the data was updated through use. Also, you never want your automated tests to connect to an external server. An error on their side could bring a halt to your development if releasing your code depends on whether your tests pass. Luckily, there is a way to test the implementation of a third-party API in a controlled environment without needing to actually connect to an outside data source. The solution is to fake the functionality of the external code using something known as mocks.

A mock is a fake object that you construct to look and act like real data. You swap it with the actual object and trick the system into thinking that the mock is the real deal. Using a mock reminds me of a classic movie trope where the hero grabs a henchman, puts on his uniform, and steps into a line of marching enemies. Nobody notices the impostor and everybody keeps moving—business as usual.

Third-party authentication, such as OAuth, is a good candidate for mocking within your application. OAuth requires your application to communicate with an external server, it involves real user data, and your application relies on its success in order to gain access to its APIs. Mocking authentication allows you to test you system as an authorized user without having to go through the actual process of exchanging credentials. In this case you do not want to test whether your system successfully authenticates a user; you want to test how your application’s functions behave after you have been authenticated.

NOTE: This tutorial uses Python v3.5.1.

First steps

Begin by setting up a new development environment to hold your project code. Create a new virtual environment and then install the following libraries:

$ pip install nose requests

Here is a quick rundown of each library you are installing, in case you have never encountered them:

The mock library is used for testing Python code by replacing parts of your system with mock objects. NOTE: The mock library is part of unittest if you are using Python 3.3 or greater. If you are using an older version, please install the backport mock library.
The nose library extends the built-in Python unittest module to make testing easier. You can use unittest or other third-party libraries such as pytest to achieve the same results, but I prefer nose‘s assertion methods.
The requests library greatly simplifies HTTP calls in Python.

For this tutorial, you will be communicating with a fake online API that was built for testing - JSON Placeholder. Before you write any tests, you need to know what to expect from the API.

First, you should expect that the API you are targeting actually returns a response when you send it a request. Confirm this assumption by calling the endpoint with cURL:

$ curl -X GET 'http://jsonplaceholder.typicode.com/todos'

This call should return a JSON-serialized list of todo items. Pay attention to the structure of the todo data in the response. You should see a list of objects with the keys userId, id, title, and completed. You are now prepared to make your second assumption–you know what to expect the data to look like. The API endpoint is alive and functioning. You proved that by calling it from the command line. Now, write a nose test so that you can confirm the life of the server in the future. Keep it simple. You should only be concerned with whether the server returns an OK response.

project/tests/test_todos.py

# Third-party imports...
from nose.tools import assert_true
import requests


def test_request_response():
    # Send a request to the API server and store the response.
    response = requests.get('http://jsonplaceholder.typicode.com/todos')

    # Confirm that the request-response cycle completed successfully.
    assert_true(response.ok)

Run the test and watch it pass:

$ nosetests --verbosity=2 project
test_todos.test_request_response ... ok

----------------------------------------------------------------------
Ran 1 test in 9.270s

OK

Refactoring your code into a service

Chances are good that you will call an external API many times throughout your application. Also, those API calls will likely involve more logic than simply making an HTTP request, such as data processing, error handling, and filtering. You should pull the code out of your test and refactor it into a service function that encapsulates all of that expected logic.

Rewrite your test to reference the service function and to test the new logic.

project/tests/test_todos.py

# Third-party imports...
from nose.tools import assert_is_not_none

# Local imports...
from project.services import get_todos


def test_request_response():
    # Call the service, which will send a request to the server.
    response = get_todos()

    # If the request is sent successfully, then I expect a response to be returned.
    assert_is_not_none(response)

Run the test and watch it fail, and then write the minimum amount of code to make it pass:

project/services.py

# Standard library imports...
try:
    from urllib.parse import urljoin
except ImportError:
    from urlparse import urljoin

# Third-party imports...
import requests

# Local imports...
from project.constants import BASE_URL

TODOS_URL = urljoin(BASE_URL, 'todos')


def get_todos():
    response = requests.get(TODOS_URL)
    if response.ok:
        return response
    else:
        return None

project/constants.py

BASE_URL = 'http://jsonplaceholder.typicode.com'

The first test that you wrote expected a response to be returned with an OK status. You refactored your programming logic into a service function that returns the response itself when the request to the server is successful. A None value is returned if the request fails. The test now includes an assertion to confirm that the function does not return None.

Notice how I instructed you to create a constants.py file and then I populated it with a BASE_URL. The service function extends the BASE_URL to create the TODOS_URL, and since all of the API endpoints use the same base, you can continue to create new ones without having to rewrite that bit of code. Putting the BASE_URL in a separate file allows you to edit it in one place, which will come in handy if multiple modules reference that code.

Run the test and watch it pass.

$ nosetests --verbosity=2 project
test_todos.test_request_response ... ok

----------------------------------------------------------------------
Ran 1 test in 1.475s

OK

Your first mock

The code is working as expected. You know this because you have a passing test. Unfortunately, you have a problem–your service function is still accessing the external server directly. When you call get_todos(), your code is making a request to the API endpoint and returning a result that depends on that server being live. Here, I will demonstrate how to detach your programming logic from the actual external library by swapping the real request with a fake one that returns the same data.

project/tests/test_todos.py

# Standard library imports...
from unittest.mock import Mock, patch

# Third-party imports...
from nose.tools import assert_is_not_none

# Local imports...
from project.services import get_todos


@patch('project.services.requests.get')
def test_getting_todos(mock_get):
    # Configure the mock to return a response with an OK status code.
    mock_get.return_value.ok = True

    # Call the service, which will send a request to the server.
    response = get_todos()

    # If the request is sent successfully, then I expect a response to be returned.
    assert_is_not_none(response)

Notice that I did not change the service function at all. The only part of the code that I edited was the test itself. First, I imported the patch() function from the mock library. Next, I modified the test function with the patch() function as a decorator, passing in a reference to project.services.requests.get. In the function itself, I passed in a parameter mock_get, and then in the body of the test function, I added a line to set mock_get.return_value.ok = True.

Great. So what actually happens now when the test is run? Before I dive into that, you need to understand something about the way the requests library works. When you call the requests.get() function, it makes an HTTP request behind the scenes and then returns an HTTP response in the form of a Response object. The get() function itself communicates with the external server, which is why you need to target it. Remember the image of the hero swapping places with the enemy while wearing his uniform? You need to dress the mock to look and act like the requests.get() function.

When the test function is run, it finds the module where the requests library is declared, project.services, and it replaces the targeted function, requests.get(), with a mock. The test also tells the mock to behave the way the service function expects it to act. If you look at get_todos(), you see that the success of the function depends on if response.ok: returning True. That is what the line mock_get.return_value.ok = True is doing. When the ok property is called on the mock, it will return True just like the actual object. The get_todos() function will return the response, which is the mock, and the test will pass because the mock is not None.

Run the test to see it pass.

$ nosetests --verbosity=2 project

Other ways to patch

Using a decorator is just one of several ways to patch a function with a mock. In the next example, I explicitly patch a function within a block of code, using a context manager. The with statement patches a function used by any code in the code block. When the code block ends, the original function is restored. The with statement and the decorator accomplish the same goal: Both methods patch project.services.request.get.

project/tests/test_todos.py

# Standard library imports...
from unittest.mock import patch

# Third-party imports...
from nose.tools import assert_is_not_none

# Local imports...
from project.services import get_todos


def test_getting_todos():
    with patch('project.services.requests.get') as mock_get:
        # Configure the mock to return a response with an OK status code.
        mock_get.return_value.ok = True

        # Call the service, which will send a request to the server.
        response = get_todos()

    # If the request is sent successfully, then I expect a response to be returned.
    assert_is_not_none(response)

Run the tests to see that they still pass.

Another way to patch a function is to use a patcher. Here, I identify the source to patch, and then I explicitly start using the mock. The patching does not stop until I explicitly tell the system to stop using the mock.

project/tests/test_todos.py

# Standard library imports...
from unittest.mock import patch

# Third-party imports...
from nose.tools import assert_is_not_none

# Local imports...
from project.services import get_todos


def test_getting_todos():
    mock_get_patcher = patch('project.services.requests.get')

    # Start patching `requests.get`.
    mock_get = mock_get_patcher.start()

    # Configure the mock to return a response with an OK status code.
    mock_get.return_value.ok = True

    # Call the service, which will send a request to the server.
    response = get_todos()

    # Stop patching `requests.get`.
    mock_get_patcher.stop()

    # If the request is sent successfully, then I expect a response to be returned.
    assert_is_not_none(response)

Run the tests again to get the same successful result.

Now that you have seen three ways to patch a function with a mock, when should you use each one? The short answer: it is entirely up to you. Each patching method is completely valid. That being said, I have found that specific coding patterns work especially well with the following patching methods.

Use a decorator when all of the code in your test function body uses a mock.
Use a context manager when some of the code in your test function uses a mock and other code references the actual function.
Use a patcher when you need to explicitly start and stop mocking a function across multiple tests (e.g. the setUp() and tearDown() functions in a test class).

I use each of these methods in this tutorial, and I will highlight each one as I introduce it for the first time.

Mocking the complete service behavior

In the previous examples, you implemented a basic mock and tested a simple assertion–whether the get_todos() function returned None. The get_todos() function calls the external API and receives a response. If the call is successful, the function returns a response object, which contains a JSON-serialized list of todos. If the request fails, get_todos() returns None. In the following example, I demonstrate how to mock the entire functionality of get_todos(). At the beginning of this tutorial, the initial call you made to the server using cURL returned a JSON-serialized list of dictionaries, which represented todo items. This example will show you how to mock that data.

Remember how @patch() works: You provide it a path to the function you want to mock. The function is found, patch() creates a Mock object, and the real function is temporarily replaced with the mock. When get_todos() is called by the test, the function uses the mock_get the same way it would use the real get() method. That means that it calls mock_get like a function and expects it to return a response object.

In this case, the response object is a requests library Response object, which has several attributes and methods. You faked one of those properties, ok, in a previous example. The Response object also has a json() function which converts its JSON-serialized string content into a Python datatype (e.g. a list or a dict).

project/tests/test_todos.py

# Standard library imports...
from unittest.mock import Mock, patch

# Third-party imports...
from nose.tools import assert_is_none, assert_list_equal

# Local imports...
from project.services import get_todos


@patch('project.services.requests.get')
def test_getting_todos_when_response_is_ok(mock_get):
    todos = [{
        'userId': 1,
        'id': 1,
        'title': 'Make the bed',
        'completed': False
    }]

    # Configure the mock to return a response with an OK status code. Also, the mock should have
    # a `json()` method that returns a list of todos.
    mock_get.return_value = Mock(ok=True)
    mock_get.return_value.json.return_value = todos

    # Call the service, which will send a request to the server.
    response = get_todos()

    # If the request is sent successfully, then I expect a response to be returned.
    assert_list_equal(response.json(), todos)


@patch('project.services.requests.get')
def test_getting_todos_when_response_is_not_ok(mock_get):
    # Configure the mock to not return a response with an OK status code.
    mock_get.return_value.ok = False

    # Call the service, which will send a request to the server.
    response = get_todos()

    # If the response contains an error, I should get no todos.
    assert_is_none(response)

I mentioned in a previous example that when you ran the get_todos() function that was patched with a mock, the function returned a mock object “response”. You might have noticed a pattern: whenever the return_value is added to a mock, that mock is modified to be run as a function, and by default it returns another mock object. In this example, I made that a little more clear by explicitly declaring the Mock object, mock_get.return_value = Mock(ok=True). The mock_get() mirrors requests.get(), and requests.get() returns a Response whereas mock_get() returns a Mock. The Response object has an ok property, so you added an ok property to the Mock.

The Response object also has a json() function, so I added json to the Mock and appended it with a return_value, since it will be called like a function. The json() function returns a list of todo objects. Notice that the test now includes an assertion that checks the value of response.json(). You want to make sure that the get_todos() function returns a list of todos, just like the actual server does. Finally, to round out the testing for get_todos(), I add a test for failure.

Run the tests and watch them pass.

$ nosetests --verbosity=2 project
test_todos.test_getting_todos_when_response_is_not_ok ... ok
test_todos.test_getting_todos_when_response_is_ok ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.285s

OK

Mocking integrated functions

The examples I have show you have been fairly straightforward, and in the next example, I will add to the complexity. Imagine a scenario where you create a new service function that calls get_todos() and then filters those results to return only the todo items that have been completed. Do you have to mock the requests.get() again? No, in this case you mock the get_todos() function directly! Remember, when you mock a function, you are replacing the actual object with the mock, and you only have to worry about how the service function interacts with that mock. In the case of get_todos(), you know that it takes no parameters and that it returns a response with a json() function that returns a list of todo objects. You do not care what happens under the hood; you just care that the get_todos() mock returns what you expect the real get_todos() function to return.

project/tests/test_todos.py

# Standard library imports...
from unittest.mock import Mock, patch

# Third-party imports...
from nose.tools import assert_list_equal, assert_true

# Local imports...
from project.services import get_uncompleted_todos


@patch('project.services.get_todos')
def test_getting_uncompleted_todos_when_todos_is_not_none(mock_get_todos):
    todo1 = {
        'userId': 1,
        'id': 1,
        'title': 'Make the bed',
        'completed': False
    }
    todo2 = {
        'userId': 1,
        'id': 2,
        'title': 'Walk the dog',
        'completed': True
    }

    # Configure mock to return a response with a JSON-serialized list of todos.
    mock_get_todos.return_value = Mock()
    mock_get_todos.return_value.json.return_value = [todo1, todo2]

    # Call the service, which will get a list of todos filtered on completed.
    uncompleted_todos = get_uncompleted_todos()

    # Confirm that the mock was called.
    assert_true(mock_get_todos.called)

    # Confirm that the expected filtered list of todos was returned.
    assert_list_equal(uncompleted_todos, [todo1])


@patch('project.services.get_todos')
def test_getting_uncompleted_todos_when_todos_is_none(mock_get_todos):
    # Configure mock to return None.
    mock_get_todos.return_value = None

    # Call the service, which will return an empty list.
    uncompleted_todos = get_uncompleted_todos()

    # Confirm that the mock was called.
    assert_true(mock_get_todos.called)

    # Confirm that an empty list was returned.
    assert_list_equal(uncompleted_todos, [])

Notice that now I am patching the test function to find and replace project.services.get_todos with a mock. The mock function should return an object that has a json() function. When called, the json() function should return a list of todo objects. I also add an assertion to confirm that the get_todos() function is actually called. This is useful to establish that when the service function accesses the actual API, the real get_todos() function will execute. Here, I also include a test to verify that if get_todos() returns None, the get_uncompleted_todos() function returns an empty list. Again, I confirm that the get_todos() function is called.

Write the tests, run them to see that they fail, and then write the code necessary to make them pass.

project/services.py

def get_uncompleted_todos():
    response = get_todos()
    if response is None:
        return []
    else:
        todos = response.json()
        return [todo for todo in todos if todo.get('completed') == False]

The tests now pass.

Refactoring tests to use classes

You probably noticed that some of the tests seem to belong together in a group. You have two tests that hit the get_todos() function. Your other two tests focus on get_uncompleted_todos(). Whenever I start to notice trends and similarities between tests, I refactor them into a test class. This refactoring accomplishes several goals:

Moving common test functions to a class allows you to more easily test them together as a group. You can tell nose to target a list of functions, but it is easier to target a single class.
Common test functions often require similar steps for creating and destroying data that is used by each test. These steps can be encapsulated in the setup_class() and teardown_class() functions respectively in order to execute code at the appropriate stages.
You can create utility functions on the class to reuse logic that is repeated among test functions. Imagine having to call the same data creation logic in each function individually. What a pain!

Notice that I use the patcher technique to mock the targeted functions in the test classes. As I mentioned before, this patching method is great for creating a mock that spans over several functions. The code in the teardown_class() method explicitly restores the original code when the tests finish.

project/tests/test_todos.py

# Standard library imports...
from unittest.mock import Mock, patch

# Third-party imports...
from nose.tools import assert_is_none, assert_list_equal, assert_true

# Local imports...
from project.services import get_todos, get_uncompleted_todos


class TestTodos(object):
    @classmethod
    def setup_class(cls):
        cls.mock_get_patcher = patch('project.services.requests.get')
        cls.mock_get = cls.mock_get_patcher.start()

    @classmethod
    def teardown_class(cls):
        cls.mock_get_patcher.stop()

    def test_getting_todos_when_response_is_ok(self):
        # Configure the mock to return a response with an OK status code.
        self.mock_get.return_value.ok = True

        todos = [{
            'userId': 1,
            'id': 1,
            'title': 'Make the bed',
            'completed': False
        }]

        self.mock_get.return_value = Mock()
        self.mock_get.return_value.json.return_value = todos

        # Call the service, which will send a request to the server.
        response = get_todos()

        # If the request is sent successfully, then I expect a response to be returned.
        assert_list_equal(response.json(), todos)

    def test_getting_todos_when_response_is_not_ok(self):
        # Configure the mock to not return a response with an OK status code.
        self.mock_get.return_value.ok = False

        # Call the service, which will send a request to the server.
        response = get_todos()

        # If the response contains an error, I should get no todos.
        assert_is_none(response)


class TestUncompletedTodos(object):
    @classmethod
    def setup_class(cls):
        cls.mock_get_todos_patcher = patch('project.services.get_todos')
        cls.mock_get_todos = cls.mock_get_todos_patcher.start()

    @classmethod
    def teardown_class(cls):
        cls.mock_get_todos_patcher.stop()

    def test_getting_uncompleted_todos_when_todos_is_not_none(self):
        todo1 = {
            'userId': 1,
            'id': 1,
            'title': 'Make the bed',
            'completed': False
        }
        todo2 = {
            'userId': 2,
            'id': 2,
            'title': 'Walk the dog',
            'completed': True
        }

        # Configure mock to return a response with a JSON-serialized list of todos.
        self.mock_get_todos.return_value = Mock()
        self.mock_get_todos.return_value.json.return_value = [todo1, todo2]

        # Call the service, which will get a list of todos filtered on completed.
        uncompleted_todos = get_uncompleted_todos()

        # Confirm that the mock was called.
        assert_true(self.mock_get_todos.called)

        # Confirm that the expected filtered list of todos was returned.
        assert_list_equal(uncompleted_todos, [todo1])

    def test_getting_uncompleted_todos_when_todos_is_none(self):
        # Configure mock to return None.
        self.mock_get_todos.return_value = None

        # Call the service, which will return an empty list.
        uncompleted_todos = get_uncompleted_todos()

        # Confirm that the mock was called.
        assert_true(self.mock_get_todos.called)

        # Confirm that an empty list was returned.
        assert_list_equal(uncompleted_todos, [])

Run the tests. Everything should pass because you did not introduce any new logic. You merely moved code around.

$ nosetests --verbosity=2 project
test_todos.TestTodos.test_getting_todos_when_response_is_not_ok ... ok
test_todos.TestTodos.test_getting_todos_when_response_is_ok ... ok
test_todos.TestUncompletedTodos.test_getting_uncompleted_todos_when_todos_is_none ... ok
test_todos.TestUncompletedTodos.test_getting_uncompleted_todos_when_todos_is_not_none ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.300s

OK

Testing for updates to the API data

Throughout this tutorial I have been demonstrating how to mock data returned by a third-party API. That mock data is based on an assumption that the real data uses the same data contract as what you are faking. Your first step was making a call to the actual API and taking note of the data that was returned. You can be fairly confident that the structure of the data has not changed in the short time that you have been working through these examples, however, you should not be confident that the data will remain unchanged forever. Any good external library is updated regularly. While developers aim to make new code backwards-compatible, eventually there comes a time where code is deprecated.

As you can imagine, relying entirely on fake data is dangerous. Since you are testing your code without communicating with the actual server, you can easily become overconfident in the strength of your tests. When the time comes to use your application with real data, everything falls apart. The following strategy should be used to confirm that the data you are expecting from the server matches the data that you are testing. The goal here is to compare the data structure (e.g. the keys in an object) rather than the actual data.

Notice how I am using the context manager patching technique. Here, you need to call the real server and you need to mock it separately.

project/tests/test_todos.py

def test_integration_contract():
    # Call the service to hit the actual API.
    actual = get_todos()
    actual_keys = actual.json().pop().keys()

    # Call the service to hit the mocked API.
    with patch('project.services.requests.get') as mock_get:
        mock_get.return_value.ok = True
        mock_get.return_value.json.return_value = [{
            'userId': 1,
            'id': 1,
            'title': 'Make the bed',
            'completed': False
        }]

        mocked = get_todos()
        mocked_keys = mocked.json().pop().keys()

    # An object from the actual API and an object from the mocked API should have
    # the same data structure.
    assert_list_equal(list(actual_keys), list(mocked_keys))

Your tests should pass. Your mocked data structure matches the one from the actual API.

Conditionally testing scenarios

Now that you have a test to compare the actual data contracts with the mocked ones, you need to know when to run it. The test that hits the real server should not be automated because a failure does not necessarily mean your code is bad. You might not be able to connect to the real server at the time of your test suite execution for a dozen reasons that are outside of your control. Run this test separately from your automated tests, but also run it fairly frequently. One way to selectively skip tests is to use an environment variable as a toggle. In the example below, all tests run unless the SKIP_REAL environment variable is set to True. When the SKIP_REAL variable is toggled on, any test with the @skipIf(SKIP_REAL) decorator will be skipped.

project/tests/test_todos.py

# Standard library imports...
from unittest import skipIf

# Local imports...
from project.constants import SKIP_REAL


@skipIf(SKIP_REAL, 'Skipping tests that hit the real API server.')
def test_integration_contract():
    # Call the service to hit the actual API.
    actual = get_todos()
    actual_keys = actual.json().pop().keys()

    # Call the service to hit the mocked API.
    with patch('project.services.requests.get') as mock_get:
        mock_get.return_value.ok = True
        mock_get.return_value.json.return_value = [{
            'userId': 1,
            'id': 1,
            'title': 'Make the bed',
            'completed': False
        }]

        mocked = get_todos()
        mocked_keys = mocked.json().pop().keys()

    # An object from the actual API and an object from the mocked API should have
    # the same data structure.
    assert_list_equal(list(actual_keys), list(mocked_keys))

project/constants.py

# Standard-library imports...
import os


BASE_URL = 'http://jsonplaceholder.typicode.com'
SKIP_REAL = os.getenv('SKIP_REAL', False)

$ export SKIP_REAL=True

Run the tests and pay attention to the output. One test was ignored and the console displays the message, “Skipping tests that hit the real API server.” Excellent!

$ nosetests --verbosity=2 project
test_todos.TestTodos.test_getting_todos_when_response_is_not_ok ... ok
test_todos.TestTodos.test_getting_todos_when_response_is_ok ... ok
test_todos.TestUncompletedTodos.test_getting_uncompleted_todos_when_todos_is_none ... ok
test_todos.TestUncompletedTodos.test_getting_uncompleted_todos_when_todos_is_not_none ... ok
test_todos.test_integration_contract ... SKIP: Skipping tests that hit the real API server.

----------------------------------------------------------------------
Ran 5 tests in 0.240s

OK (SKIP=1)

Next steps

At this point, you have seen how to test the integration of your app with a third-party API using mocks. Now that you know how to approach the problem, you can continue practicing by writing service functions that for the other API endpoints in JSON Placeholder (e.g. posts, comments, users).

Free Bonus: Click here to download a copy of the "REST in a Nutshell" Guide with a hands-on introduction to REST API principles and examples.

Improve your skills even more by connecting your app to a real external library such as Google, Facebook, or Evernote and see if you can write tests that use mocks. Keep producing clean and reliable code and stay tuned for the next tutorial, which will describe how to take testing to the next level with mock servers!

Grab the code from the repo.

Getting Started With the Slack API Using Python and Flask

2016-05-24T13:18:21+00:00

The slick hosted chat application Slack is all the rage this year. The tool’s adoption isn’t empty hype - it’s incredibly useful for communicating with and learning from fellow developers. For example, software developer communities such as DC Python, Dallas-Forth Worth Devs, and Denver Devs set up their own Slack channels.

However, Slack wouldn’t be that useful if it was just a glorified AOL Instant Messenger. It’s the programmatic access to retrieve and send messages with the Slack web application programming interface (API) where the power really kicks in.

In this post, we’ll see how to work with Slack via the API and the official SlackClient Python helper library. We will grab an API access token and write some Python code to list, retrieve and send data through the API. Let’s dig in now!

Free Bonus: Click here to get access to a free Flask + Python video tutorial series that shows you how to build a fully working Flask web app, step-by-step.

This is a guest blog post by Matt Makai, Developer Evangelist at Twilio and author of Full Stack Python.

Tools We’ll Need

Several tools will be used to run the code in this blog post, including:

A free Slack account with a team on which you have API access or sign up for the Slack Developer Hangout team
Either Python 2 or 3
Official Python slackclient code library built by the Slack team
Slack API testing token
Flask web micro-framework; if you’re unfamiliar with it, check out the Real Python course, the Flask by Example series, or Full Stack Python’s Flask page

It’s also handy to have the Slack API docs open for reference. You can follow along by writing the code in this post or clone the companion GitHub repository with the finished project.

Now that we know what tools we need to use, let’s begin by creating a new virtualenv to isolate our application dependencies from other Python projects you’re working on:

$ mkdir slackapi
$ virtualenv venv

Activate the virtualenv:

$ source venv/bin/activate

Depending on how your virtualenv and shell are set up, your prompt should now look something like this:

(venv)$

Keep the shell open for now as we get our Slack access established via the official slackclient API helper library built by Slack.

There are also other fantastic Python helper libraries created by the community. For simplicity, we’re only going to install and use slackclient, but you can also try out libraries like slacker, slack and pyslack once we’re done here.

Install the slackclient helper library into your virtualenv with pip:

$ pip install slackclient==1.0.0

Now that we have the helper library installed, we need to obtain a Slack access token for our team and account.

The Slack Web API

Head to the landing page for the Slack Web API:

Once you’ve signed in you can scroll down on the web API page where you’ll see a button to generate test tokens:

Generate a test token for a Slack team on which you have administrative privileges. This token will serve fine for our development purposes in this blog post, but you can also create an OAuth flow where other users can generate tokens for authentication through their own accounts.

We’ll need that test token in just a moment, so keep it handy. Let’s switch into our Python environment set up so we can try out the API. With your virtualenv still active, fire up the Python REPL:

(venv)$ python
Python 3.5.0 (v3.5.0:374f501f4567, Sep 12 2015, 11:00:19)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Let’s test our API token with a test call; type the following code at the REPL prompt:

>>> from slackclient import SlackClient
>>> slack_client = SlackClient('your test token here')
>>> slack_client.api_call("api.test")

The REPL should return back something like the following dictionary if your API test with the token was successful:

{u'args': {u'token': u'xoxp-361113305843-7621238052-8691112296227-d0d4824abe'}, u'ok': True}

If you get back {u'ok': False, u'error': u'invalid_auth'} then double-check that you copied the Slack token correctly into the second line entered on the REPL.

Enter one more quick test for our authentication with another line of code in the REPL:

>>> slack_client.api_call("auth.test")

You should see another dictionary similar to this one:

{u'user_id': u'U0S77S29J', u'url': u'https://fullstackguides.slack.com/', u'team_id': u'T0S8V1ZQA', u'user': u'matt', u'team': u'Full Stack Guides, u'ok': True}

Awesome! We’re authorized to start using the Slack API through our account. Now the fun begins as we can start programmatically obtaining Slack data and handling messages!

Slack API Basics

Exit out of the REPL with a quick CTRL-d or exit() command. Back on the command line, export the Slack token as an environment variable:

(venv)$ export SLACK_TOKEN='your slack token pasted here'

We’ll snag the environment variable in our Python script using the os module instead of hardcoding it into the source code.

Dive into your favorite text editor such as Vim, Emacs, or Sublime Text so we can cut some new Python code. Create a new file named app.py and start filling it out with the following imports:

import os
from slackclient import SlackClient

Again, the os module will be used to pull the SLACK_TOKEN environment variable we just exported. The SlackClient import should look familiar, as it is the same line we wrote earlier on the REPL:

SLACK_TOKEN = os.environ.get('SLACK_TOKEN')

slack_client = SlackClient(SLACK_TOKEN)

In the above two lines, we snag the SLACK_TOKEN environment variable value and instantiate the SlackClient helper library. Next let’s create a function to list channels via an API call. Slack returns back the results in a dictionary with two keys: ok and channels. ok allows us to know if the API call was successful, and if its value is True then channels contains the data we need on the list of channels.

def list_channels():
    channels_call = slack_client.api_call("channels.list")
    if channels_call.get('ok'):
        return channels_call['channels']
    return None

Finally, let’s add a convenience main function that will allow us to print all the channels when we invoke the Python file with python app.py on the command line:

if __name__ == '__main__':
    channels = list_channels()
    if channels:
        print("Channels: ")
        for c in channels:
            print(c['name'] + " (" + c['id'] + ")")
    else:
        print("Unable to authenticate.")

That’s all the code we need for the moment. Time to give it a try. Execute the script from the command line with python app.py. You’ll see output like the following channels list:

Channels:
general (C0S82S5RS)
python (C0S8HABL3)
random (C0S8F4432)

What is the channel ID that we printed out in parentheses next the the channel name for? Slack’s API needs a unique reference for channels, so we use the ID, not the name, as an identifier instead of the human-readable channel name.

We can write some code that uses the channel.info API method to obtain data for a specific channel based on its ID.

Add a new function along with a few new lines in main to output the latest message from each channel, which is only available in the more detailed channel.info API call.

Updated code:

import os
from slackclient import SlackClient


SLACK_TOKEN = os.environ.get('SLACK_TOKEN', None)

slack_client = SlackClient(SLACK_TOKEN)


def list_channels():
    channels_call = slack_client.api_call("channels.list")
    if channels_call['ok']:
        return channels_call['channels']
    return None


def channel_info(channel_id):
    channel_info = slack_client.api_call("channels.info", channel=channel_id)
    if channel_info:
        return channel_info['channel']
    return None


if __name__ == '__main__':
    channels = list_channels()
    if channels:
        print("Channels: ")
        for c in channels:
            print(c['name'] + " (" + c['id'] + ")")
            detailed_info = channel_info(c['id'])
            if detailed_info:
                print(detailed_info['latest']['text'])
    else:
        print("Unable to authenticate.")

Note that with this code we’re greatly increasing the API calls the script executes, from one to N+1, where N is the number of channels returned back by Slack.

Run the new script again by executing python app.py:

Channels:
general (C0S82S5RS)
yada yada yada.
python (C0S8HABL3)
This is posted to #python and comes from a bot named webhookbot.
random (C0S8F4432)
<@U0SAEJ99T|samb> has joined the channel

Nice! Now we have both the list of channels as well as a way to get detailed information on each channel with its ID. Next let’s interact with other users in one of our channels by sending and receiving messages.

Sending Messages

We can go even further into the Slack API now that we know our API calls are working and have the channel ID. Let’s send a message to the #general channel.

Add a new function under channel_info named send_message:

def send_message(channel_id, message):
    slack_client.api_call(
        "chat.postMessage",
        channel=channel_id,
        text=message,
        username='pythonbot',
        icon_emoji=':robot_face:'
    )

send_message takes in the ID for a channel, then posts a message from our “Python bot” to that channel. In addition, modify the main function so that when we run this file, main will call our new send_message function:

if __name__ == '__main__':
    channels = list_channels()
    if channels:
        print("Channels: ")
        for channel in channels:
            print(channel['name'] + " (" + channel['id'] + ")")
            detailed_info = channel_info(channel['id'])
            if detailed_info:
                print('Latest text from ' + channel['name'] + ":")
                print(detailed_info['latest']['text'])
            if channel['name'] == 'general':
                send_message(channel['id'], "Hello " +
                             channel['name'] + "! It worked!")
        print('-----')
    else:
        print("Unable to authenticate.")

Save the changes and run python app.py. Open the #general channel for your Slack team. You should see your Python bot post a new message to the channel:

Awesome! So we can send messages, but what about if we want to see what users in the #general channel are saying?

Receiving Messages

We can set up an outgoing webhook that will alert our Python application via an HTTP POST request. This part is a bit more complicated than sending messages because we need to receive one or more POST requests.

First we’ll need a simple web server that can handle an inbound POST request from the Slack webhook. Create a new file named receive.py with the following code:

import os
from flask import Flask, request, Response


app = Flask(__name__)

SLACK_WEBHOOK_SECRET = os.environ.get('SLACK_WEBHOOK_SECRET')


@app.route('/slack', methods=['POST'])
def inbound():
    if request.form.get('token') == SLACK_WEBHOOK_SECRET:
        channel = request.form.get('channel_name')
        username = request.form.get('user_name')
        text = request.form.get('text')
        inbound_message = username + " in " + channel + " says: " + text
        print(inbound_message)
    return Response(), 200


@app.route('/', methods=['GET'])
def test():
    return Response('It works!')


if __name__ == "__main__":
    app.run(debug=True)

In the above Python file, we:

Import Flask
Instantiate a new Flask application context
Pull in the SLACK_WEBHOOK_SECRET environment variable, which we’ll get in just a moment from the Slack console
Establish a route that can receive an HTTP POST request from Slack that prints the output to the command line as long as the webhook secret key sent to us matches the one from our environment variable
Create another route for testing purposes that responds to a GET request
Set our Flask app to run when we run this script with Python

Install Flask (pip install flask), and then start the Flask app with the python receive.py command and we’ll see some debugging output indicating the development server is running.

Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
Restarting with stat
Debugger is active!
Debugger pin code: 144-609-426

We’re set to receive our POST request webhook, except that most development environments do not expose routes beyond localhost. We need a localhost tunnel that will give us an externally-accessible domain name while we’re developing our code. I typically use ngrok since it’s easy, free and awesome. There are also other options such as localtunnel and forward.

After downloading and running ngrok (or another localhost tunneling tool) in a new terminal window, you’ll get a subdomain that forwards requests sent to that subdomain over to your localhost server. Here is what ngrok looks like in the console when it’s started with the ./ngrok http 5000 command:

Take note of your Forwarding URL, in this case https://6940e7da.ngrok.io, as we’ll need that for Slack to set up our outgoing webhook. Then test that our ngrok forwarding URL is properly connected to our Flask app by opening your web browser and going to the Forwarding URL. We should see the “It works!” message.

Now we can use that ngrok Forwarding URL in our Slack configuration. Go to the Slack Outgoing Webhooks page, then click the “outgoing webhook integration” link as shown below:

Scroll down to the Integration Settings section. Select “#general” as the channel to listen on. Copy your ngrok Forwarding URL plus “/slack” into the URL(s) text box:

Copy the generated Token. Scroll down and press the “Save Settings” button.

Stop your Flask server for just a moment. As we did earlier with the Slack token, use the export command to expose the outgoing webhook token as an environment variable:

(venv)$ export SLACK_WEBHOOK_SECRET='generated outgoing webhook token here'

Then restart your Flask server so it can grab the generated SLACK_WEBHOOK_SECRET. Finally, it’s time to test out receiving messages!

Go to your Slack #general channel. You should see that the outgoing webhook integration has been added to the channel:

Within Slack, type in a message like “testing” and hit enter. Go back to the command line where your Flask app is running. You should see the message output printed from the POST request:

matt in general says: testing
127.0.0.1 - - [21/May/2016 12:39:56] "POST /slack HTTP/1.1" 200 -

Now we’ve got a way to receive messages from one or more channels and can add whatever Python code we want to handle the input. This is a great hook for building a bot or sending messages to another service for processing.

Wrapping it up

Woohoo! All done! Well actually, there’s a whole lot more you can do with the Slack API. Here are several more ideas to try out now that you’ve got the basics down:

Combine the Twilio API to communicate with Slack channels via text messages
Try a different Slack client or ditch the helper library entirely and use the Requests library to implement retry logic
Write and customize a full Slack bot

Free Bonus: Get our Flask + Python video tutorial series that shows you how to build a fully working Flask web app, step-by-step.

That’s all for now.

If you have questions feel free to drop a comment below or contact me via:

Twitter: @mattmakai and @fullstackpython
GitHub: makaimc
Twitch (live coding with Python & Swift): mattmakai

Deploying Django + Python 3 + PostgreSQL to AWS Elastic Beanstalk

2016-05-10T13:53:15+00:00

The following is a soup to nuts walkthrough of how to set up and deploy a Django application, powered by Python 3, and PostgreSQL to Amazon Web Services (AWS) all while remaining sane.

Tools/technologies used:

Python v3.4.3
Django v1.9
Amazon Elastic Beanstalk, EC2, S3, and RDS
EB CLI 3.x
PostgreSQL

Free Bonus: Click here to get access to a free list of 35 additional Django tutorials and resources you can use to deepen your Python web development skills.

Check out the Python 2 version of this article here.

Updated 08/21/2016: Updated EB global configuration settings.

Elastic Beanstalk vs EC2

Elastic Beanstalk is a Platform As A Service (PaaS) that streamlines the setup, deployment, and maintenance of your app on Amazon AWS. It’s a managed service, coupling the server (EC2), database (RDS), and your static files (S3). You can quickly deploy and manage your application, which automatically scales as your site grows. Check out the official documentation for more information.

Getting Started

We’ll be using a simple “Image of the Day” app, which you can grab from this repository:

$ git clone https://github.com/realpython/image-of-the-day.git
$ cd image-of-the-day/
$ git checkout tags/start_here_py3

After you download the code, create a virtualenv and install the requirements via pip:

$ pip install -r requirements.txt

Next, with PostgreSQL running locally, set up a new database named iotd. Also, depending upon your local Postgres configuration, you may need to update the DATABASES configuration in settings.py. For example, I updated the config to:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'iotd',
        'USER': '',
        'PASSWORD': '',
        'HOST': 'localhost',
        'PORT': '5432',
    }
}

Now you can set up the database schema, create a superuser, and run the app:

$ python manage.py migrate
$ python manage.py createsuperuser
$ python manage.py runserver

Navigate to the admin page in your browser at http://localhost:8000/admin and add a new image, which will then be displayed on the main page.

The application isn’t meant to be very exciting; we’re just using it for demo purposes. All it does is let you upload an image via the admin interface and display that image full screen on the main page. That said, although this is a relatively basic app, it will still allow us to explore a number of “gotchas” that exist when deploying to Amazon Beanstalk and RDS.

Now that we have the site up and running on our local machine, let’s start the Amazon deployment process.

CLI for AWS Elastic Beanstalk

To work with a Amazon Elastic Beanstalk, we can use a package called awsebcli. As of this writing the latest version of is 3.7.4 and the recommended way to install it is with pip:

$ pip install awsebcli

Now test the installation to make sure it’s working:

$ eb --version

This should give you a nice 3.x version number:

EB CLI 3.7.4 (Python 3.4.3)

To actually start using Elastic Beanstalk you will need an account with AWS (surprise!). Sign up (or log in).

Configure EB – Initialize Your App

With the AWS Elastic Beanstalk CLI working, the first thing we want to do is create a Beanstalk environment to host the application on. Run this from the project directory (“image-of-the-day”):

$ eb init

This will prompt you with a number of questions to help you configure your environment.

Default region

Choosing the region closest to your end users will generally provide the best performance. Check out this map if you’re unsure which to choose.

Credentials

Next, it’s going to ask for your AWS credentials.

Here, you will most likely want to set up an IAM User. See this guide for how to set one up. If you do set up a new user you will need to ensure the user has the appropriate permissions. The simplest way to do this is to just add “Administrator Access” to the User. (This is probably not a great choice for security reasons, though.) For the specific policies/roles that a user needs in order to create/manage an Elastic Beanstalk application, see the link here.

Application name

This will default to the directory name. Just go with that.

Python version

Next, the CLI should automagically detect that you are using Python and just ask for confirmation. Say yes. Then you need to select a platform version. You have 2 different options here for Python 3:

Python 3.4
Python 3.4 (Preconfigured - Docker)

If you’re a hipster, choose the ‘Preconfigured - Docker’ choice, otherwise go with the normal ‘Python 3.4’. No, only teasing; the basic difference is this:

Python 3.4

This gives you an EC2 image running 64bit Amazon Linux with Python 3.4 pre-installed. The front end web server is apache, with mod_wsgi installed. This is the “standard” or “traditional” way that Beanstalk works. In other words, with this option Beanstalk will create EC2 images for you, and you can use the ebextension files we will talk about later to customize the EC2 image.

Python 3.4 (Preconfigured – Docker)

This gives you an EC2 image running Docker, with a Docker image already setup for you. The Docker image runs 64bit Debian Jessie with Python 3.4, nginx 1.8 and uWSGI 2.0.8. Because you’re basically interacting with the Docker image directly, if you choose this route you would use standard Docker configuration techniques (i.e., a ‘Dockerfile’), and then you don’t have to do much that is AWS Beanstalk specific, as Beanstalk knows how to manage the Docker image for you.

For this article we will focus on the “standard” or “traditional” way of using an EC2 image, so choose the ‘Python 3.4’ option and let’s move on.

SSH

Say yes to setting up SSH for your instances.

RSA Keypair

Next, you need to generate an RSA keypair, which will be added to your ~/.ssh folder. This keypair will also be uploaded to the EC2 public key for the region you specified in step one. This will allow you to SSH into your EC2 instance later in this tutorial.

What did we accomplish?

Once eb init is finished, you will see a new hidden folder called .elasticbeanstalk in your project directory:

├── .elasticbeanstalk
│   └── config.yml
├── .gitignore
├── README.md
├── iotd
│   ├── images
│   │   ├── __init__.py
│   │   ├── admin.py
│   │   ├── migrations
│   │   │   ├── 0001_initial.py
│   │   │   └── __init__.py
│   │   ├── models.py
│   │   ├── tests.py
│   │   └── views.py
│   ├── iotd
│   │   ├── __init__.py
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   ├── manage.py
│   ├── static
│   │   ├── css
│   │   │   └── bootstrap.min.css
│   │   └── js
│   │       ├── bootstrap.min.js
│   │       └── jquery-1.11.0.min.js
│   └── templates
│       ├── base.html
│       └── images
│           └── home.html
├── requirements.txt
└── www
    └── media
        └── sitelogo.png

Inside that directory is a config.yml file, which is a configuration file that is used to define certain parameters for your newly minted Beanstalk application.

At this point, if you type eb console it will open up your default browser and navigate to the Elastic Beanstalk console. On the page, you should see one application (called image-of-the-day if you’re following along exactly), but no environments.

An application represents your code application and is what eb init created for us. With Elastic Beanstalk, an application can have multiple environments (i.e., development, testing, staging, production). It is completely up to you how you want to configure/manage these environments. For simple Django applications I like to have the development environment on my laptop, then create a test and a production environment on Beanstalk.

Let’s get a test environment set up…

Configure EB – Create an Environment

Coming back to the terminal, in your project directory type:

$ eb create

Just like eb init, this command will prompt you with a series of questions.

Environment Name

You should use a similar naming convention to what Amazon suggests - e.g., application_name-env_name - especially when/if you start hosting multiple applications with AWS. I used - iod-test.

DNS CNAME prefix

When you deploy an app to Elastic Beanstalk you will automatically get a domain name like xxx.elasticbeanstalk.com. DNS CNAME prefix is what you want to be used in place of xxx. The default probably won’t work if you’re following along because somebody else has already used it (the names are global to AWS), so pick something unique and keep on going.

What happens now?

At this point eb will actually create your environment for you. Be patient as this can take some time.

If you do get an error creating the environment, like - aws.auth.client.error.ARCInstanceIdentityProfileNotFoundException- check that the credentials you are using have appropriate permissions to create the Beanstalk environment, as discussed earlier in this post.

Also, it may prompt you with a message about Platform requires a service role. If it does, just say yes and let it create the role for you.

Immediately after the environment is created, eb will attempt to deploy your application, by copying all the code in your project directory to the new EC2 instance, running pip install -r requirements.txt in the process.

You should see a bunch of information about the environment being set up displayed to your screen, as well as information about eb trying to deploy. You will also see some errors. In particular you should see these lines buried somewhere in the output:

ERROR: Your requirements.txt is invalid. Snapshot your logs for details.

Don’t worry - It’s not really invalid. Check the logs for details:

$ eb logs

This will grab all the recent log files from the EC2 instance and output them to your terminal. It’s a lot of information, so you may want to redirect the output to a file (eb logs -z). Looking through the logs, you’ll see one log file named eb-activity.log:

Error: pg_config executable not found.

The problem is that we tried to install psycopy2 (the Postgres Python bindings), but we need the Postgres client drivers to be installed as well. Since they are not installed by default, we need to install them first. Let’s fix that…

Customizing the Deployment Process

eb will read custom .config files from a folder called “.ebextensions” at the root level of your project (“image-of-the-day” directory). These .config files allow you to install packages, run arbitrary commands and/or set environment variables. Files in the “.ebextensions” directory should conform to either JSON or YAML syntax and are executed in alphabetical order.

Installing packages

The first thing we need to do is install some packages so that our pip install command will complete successfully. To do this, let’s first create a file called .ebextensions/01_packages.config:

packages:
  yum:
    git: []
    postgresql93-devel: []
    libjpeg-turbo-devel: []

EC2 instances run Amazon Linux, which is a Redhat flavor, so we can use yum to install the packages that we need. For now, we are just going to install three packages - git, the Postgres client, and libjpeg for Pillow.

After creating that file to redeploy the application, we need to do the following:

$ git add .ebextensions/
$ git commit -m "added eb package configuration"

We have to commit the changes because the deployment command eb deploy works off the latest commit, and will thus only be aware of our file changes after we commit them to git. (Do note though that we don’t have to push; we are working from our local copy…)

As you probably guessed, the next command is:

$ eb deploy

You should now see only one error:

INFO: Environment update is starting.
INFO: Deploying new version to instance(s).
ERROR: Your WSGIPath refers to a file that does not exist.
INFO: New application version was deployed to running EC2 instances.
INFO: Environment update completed successfully.

Let’s find out what’s happening…

Configuring our Python environment

EC2 instances in Beanstalk run Apache, and Apache will find our Python app according to the WSGIPATH that we have set. By default eb assumes our wsgi file is called application.py. There are two ways of correcting this-

Option 1: Using environment-specific configuration settings

$ eb config

This command will open your default editor, editing a configuration file called .elasticbeanstalk/iod-test.env.yml. This file doesn’t actually exist locally; eb pulled it down from the AWS servers and presented it to you so that you can change settings in it. If you make any changes to this pseudo-file and then save and exit, eb will update the corresponding settings in your Beanstalk environment.

If you search for the terms ‘WSGI’ in the file, and you should find a configuration section that looks like this:

aws:elasticbeanstalk:container:python:
  NumProcesses: '1'
  NumThreads: '15'
  StaticFiles: /static/=static/
  WSGIPath: application.py

Update the WSGIPath:

 aws:elasticbeanstalk:container:python:
   NumProcesses: '1'
   NumThreads: '15'
   StaticFiles: /static/=static/
   WSGIPath: iotd/iotd/wsgi.py

And then you will have you WSGIPath set up correctly. If you then save the file and exit, eb will update the environment configuration automatically:

Printing Status:
INFO: Environment update is starting.
INFO: Updating environment iod-test's configuration settings.
INFO: Successfully deployed new configuration to environment.
INFO: Environment update completed successfully.

The advantage to using the eb config method to change settings is that you can specify different settings per environment. But you can also update settings using the same .config files that we were using before. This will use the same settings for each environment, as the .config files will be applied on deployment (after the settings from eb config have been applied).

Option 2: Using global configuration settings

To use the .config file option, let’s create a new file called /.ebextensions/02_python.config:

option_settings:
  "aws:elasticbeanstalk:application:environment":
    DJANGO_SETTINGS_MODULE: "iotd.settings"
    "PYTHONPATH": "/opt/python/current/app/iotd:$PYTHONPATH"
  "aws:elasticbeanstalk:container:python":
    WSGIPath: iotd/iotd/wsgi.py
    NumProcesses: 3
    NumThreads: 20
  "aws:elasticbeanstalk:container:python:staticfiles":
    "/static/": "www/static/"

What’s happening?

DJANGO_SETTINGS_MODULE: "iotd.settings" - adds the path to the settings module.
"PYTHONPATH": "/opt/python/current/app/iotd:$PYTHONPATH" - updates our PYTHONPATH so Python can find the modules in our application. This path may vary depending on your setup! See this comment for more details. (Note that the use of the full path is necessary.)
WSGIPath: iotd/iotd/wsgi.py sets our WSGI Path.
NumProcesses: 3 and NumThreads: 20 - updates the number of processes and threads used to run our WSGI application.
"/static/": "www/static/" sets our static files path.

Again, we can do a git commit then an eb deploy to update these settings.

Next let’s add a database.

Configuring a Database

Try to view the deployed website:

$ eb open

This command will show the deployed application in your default browser. You should see a connection refused error:

OperationalError at /
could not connect to server: Connection refused
    Is the server running on host "localhost" (127.0.0.1) and accepting
    TCP/IP connections on port 5432?

This is because we haven’t set up a database yet. At this point eb will set up your Beanstalk environment, but it doesn’t set up RDS (the database tier). We have to set that up manually.

Database setup

Again, use eb console to open up the Beanstalk configuration page.

From there, do the following:

Click the “Configuration” link.
Scroll all the way to the bottom of the page, and then under the “Data Tier” section, click the link “create a new RDS database”.
On the RDS setup page change the “DB Engine” to “postgres”.
Add a “Master Username” and “Master Password”.
Save the changes.

Beanstalk will create the RDS for you. Now we need to get our Django app to connect to the RDS. Beanstalk will help us out here by exposing a number of environment variables on the EC2 instances for us that detail how to connect to the Postgres server. So all we need to do is update our settings.py file to take advantage of those environment variables. Confirm that the DATABASES configuration parameter reflects the following in settings.py:

if 'RDS_DB_NAME' in os.environ:
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'NAME': os.environ['RDS_DB_NAME'],
            'USER': os.environ['RDS_USERNAME'],
            'PASSWORD': os.environ['RDS_PASSWORD'],
            'HOST': os.environ['RDS_HOSTNAME'],
            'PORT': os.environ['RDS_PORT'],
        }
    }
else:
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'NAME': 'iotd',
            'USER': 'iotd',
            'PASSWORD': 'iotd',
            'HOST': 'localhost',
            'PORT': '5432',
        }
    }

This simply says, “use the environment variable settings if present, otherwise use our default development settings.” Simple.

Handling database migrations

With our database setup, we still need to make sure that migrations run so that the database table structure is correct. We can do that by modifying .ebextensions/02_python.config and adding the following lines at the top of the file:

container_commands:
  01_migrate:
    command: "source /opt/python/run/venv/bin/activate && python iotd/manage.py migrate --noinput"
    leader_only: true

container_commands allow you to run arbitrary commands after the application has been deployed on the EC2 instance. Because the EC2 instance is set up using a virtual environment, we must first activate that virtual environment before running our migrate command. Also the leader_only: true setting means, “only run this command on the first instance when deploying to multiple instances.”

Don’t forget that our application makes use of Django’s admin, so we are going to need a superuser…

Create the Admin User

Unfortunately createsuperuser doesn’t allow you to specify a password when using the --noinput option, so we will have to write our own command. Fortunately, Django makes it very easy to create custom commands.

Create the file iotd/images/management/commands/createsu.py:

from django.core.management.base import BaseCommand
from django.contrib.auth.models import User


class Command(BaseCommand):

    def handle(self, *args, **options):
        if not User.objects.filter(username="admin").exists():
            User.objects.create_superuser("admin", "admin@admin.com", "admin")

Make sure you add the appropriate __init__.py files as well:

└─ management
    ├── __init__.py
    └── commands
        ├── __init__.py
        └── createsu.py

This file will allow you to run python manage.py createsu, and it will create a superuser without prompting for a password. Feel free to expand the command to use environment variables or another means to allow you to change the password.

Once you have the command created, we can just add another command to our container_commands section in .ebextensions/02_python.config:

02_createsu:
  command: "source /opt/python/run/venv/bin/activate && python iotd/manage.py createsu"
  leader_only: true

Before you test this out, let’s make sure our static files are all put in the correct place…

Static Files

Add one more command under container_commands:

03_collectstatic:
  command: "source /opt/python/run/venv/bin/activate && python iotd/manage.py collectstatic --noinput"

So the entire file looks like this:

container_commands:
  01_migrate:
    command: "source /opt/python/run/venv/bin/activate && python iotd/manage.py migrate --noinput"
    leader_only: true
  02_createsu:
    command: "source /opt/python/run/venv/bin/activate && python iotd/manage.py createsu"
    leader_only: true
  03_collectstatic:
    command: "source /opt/python/run/venv/bin/activate && python iotd/manage.py collectstatic --noinput"

option_settings:
  "aws:elasticbeanstalk:application:environment":
    DJANGO_SETTINGS_MODULE: "iotd.settings"
    "PYTHONPATH": "/opt/python/current/app/iotd:$PYTHONPATH"
    "ALLOWED_HOSTS": ".elasticbeanstalk.com"
  "aws:elasticbeanstalk:container:python":
    WSGIPath: iotd/iotd/wsgi.py
    NumProcesses: 3
    NumThreads: 20
  "aws:elasticbeanstalk:container:python:staticfiles":
    "/static/": "www/static/"

We now need to ensure that the STATIC_ROOT is set correctly in the settings.py file:

STATIC_ROOT = os.path.join(BASE_DIR, "..", "www", "static")
STATIC_URL = '/static/'

Make sure you commit the www directory to git so the static dir can be created. Then run eb deploy again, and you should now be in business:

INFO: Environment update is starting.
INFO: Deploying new version to instance(s).
INFO: New application version was deployed to running EC2 instances.
INFO: Environment update completed successfully.

At this point you should be able to go to http://your_app_url/admin, log in, add an image, and then see that image displayed on the main page of your application.

Success!

Using S3 for Media Storage

With this setup, each time we deploy again, we will lose all of our uploaded images. Why? Well, when you run eb deploy, a fresh instance is spun up for you. This is not what we want since we will then have entries in the database for the images, but no associated images. The solution is to store the media files in Amazon Simple Storage Service (Amazon S3) instead of on the EC2 instance itself.

You will need to:

Since there are good write ups on this already, I’ll just point you to my favorite: Using Amazon S3 to store you Django Static and Media Files

Apache Config

Since we are using Apache with Beanstalk, we probably want to set up Apache to (among other things) enable gzip compression so files are downloaded faster by the clients. That can be done with container_commands. Create a new file .ebextensions/03_apache.config and add the following:

container_commands:
  01_setup_apache:
    command: "cp .ebextensions/enable_mod_deflate.conf /etc/httpd/conf.d/enable_mod_deflate.conf"

Then you need to create the file .ebextensions/enable_mod_deflate.conf:

# mod_deflate configuration
<IfModule mod_deflate.c>
  # Restrict compression to these MIME types
  AddOutputFilterByType DEFLATE text/plain
  AddOutputFilterByType DEFLATE text/html
  AddOutputFilterByType DEFLATE application/xhtml+xml
  AddOutputFilterByType DEFLATE text/xml
  AddOutputFilterByType DEFLATE application/xml
  AddOutputFilterByType DEFLATE application/xml+rss
  AddOutputFilterByType DEFLATE application/x-javascript
  AddOutputFilterByType DEFLATE text/javascript
  AddOutputFilterByType DEFLATE text/css
  # Level of compression (Highest 9 - Lowest 1)
  DeflateCompressionLevel 9
  # Netscape 4.x has some problems.
  BrowserMatch ^Mozilla/4 gzip-only-text/html
  # Netscape 4.06-4.08 have some more problems
  BrowserMatch ^Mozilla/4\.0[678] no-gzip
  # MSIE masquerades as Netscape, but it is fine
  BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
<IfModule mod_headers.c>
  # Make sure proxies don't deliver the wrong content
  Header append Vary User-Agent env=!dont-vary
</IfModule>
</IfModule>

Doing this will enable gzip compression, which should help with the size of the files you’re downloading. You could also use the same strategy to automatically minify and combine your CSS/JS and do any other preprocessing you need to do.

Troubleshooting

Don’t forget the very helpful eb ssh command, which will get you into the EC2 instance so you can poke around and see what’s going on. When troubleshooting, there are a few directories you should be aware of:

/opt/python: Root of where you application will end up.
/opt/python/current/app: The current application that is hosted in the environment.
/opt/python/on-deck/app: The app is initially put in on-deck and then, after all the deployment is complete, it will be moved to current. If you are getting failures in your container_commands, check out out the on-deck folder and not the current folder.
/opt/python/current/env: All the env variables that eb will set up for you. If you are trying to reproduce an error, you may first need to source /opt/python/current/env to get things set up as they would be when eb deploy is running.
opt/python/run/venv: The virtual env used by your application; you will also need to run source /opt/python/run/venv/bin/activate if you are trying to reproduce an error.

Conclusion

Deploying to Elastic Beanstalk can be a bit daunting at first, but once you understand where all the parts are and how things work, it’s actually pretty easy and extremely flexible. It also gives you an environment that will scale automatically as your usage grows. Hopefully by now you have enough to be dangerous! Good luck on your next Beanstalk deployment.

Free Bonus: Click here to get access to a free list of 35 additional Django tutorials and resources you can use to deepen your Python web development skills.

Did we miss anything? Have any other tips or tricks? Please comment below.

Python Virtual Environments – A Primer

2016-03-28T13:49:48+00:00

In this article, we’ll show how to use virtual environments to create and manage separate environments for your Python projects, each using different versions of Python for execution, as well as how Python dependencies are stored and resolved.

Free Bonus: Click here to get access to a free 5-day class that shows you how to avoid common dependency management issues with tools like Pip, PyPI, Virtualenv, and requirements files.

Updated 2018-01-12: Clarified pyenv vs venv usage on Python 3.6+
Updated 2016-06-11: Added section on changing Python versions with virtualenv.

Why the need for virtual environments?

Python, like most other modern programming languages, has its own unique way of downloading, storing, and resolving packages (or modules). While this has its advantages, there were some interesting decisions made about package storage and resolution, which has lead to some problems - namely how and where packages are stored.

There are a few different locations where these packages can be installed on your system. For example, most system packages are stored in a child directory of the path stored in sys.prefix.

On Mac OS X, you can easily find where sys.prefix points to using the Python shell:

>>> import sys
>>> sys.prefix
'/System/Library/Frameworks/Python.framework/Versions/3.5'

More relevant to the topic of this article, third party packages installed using easy_install or pip are typically placed in one of the directories pointed to by site.getsitepackages:

>>> import site
>>> site.getsitepackages()
[
  '/System/Library/Frameworks/Python.framework/Versions/3.5/Extras/lib/python',
  '/Library/Python/3.5/site-packages'
]

So, why do all of these little details matter?

It’s important to know this because, by default, every project on your system will use these same directories to store and retrieve site packages (3rd party libraries). At first glance this may not seem like a big deal, and it isn’t really for system packages - packages part of the standard Python library - but it does matter for site packages.

Consider the following scenario where you have two projects - ProjectA and ProjectB, both of which have a dependency on the same library, ProjectC. The problem becomes apparent when we start requiring different versions of ProjectC. Maybe ProjectA needs v1.0.0, while ProjectB requires the newer v2.0.0, for example.

This is a real problem for Python since it can’t differentiate between versions in the “site-packages” directory. So both v1.0.0 and v2.0.0 would reside in the same directory with the same name:

/System/Library/Frameworks/Python.framework/Versions/3.5/Extras/lib/python/ProjectC

And since projects are stored according to just their name there is no differentiation between versions. Thus, both projects, ProjectA and ProjectB, would be required to use the same version, which is unacceptable in many cases.

This is where the concept of virtual environments (and the virtualenv/venv tools) comes into play…

What is a virtual environment?

At its core, the main purpose of Python virtual environments is to create an isolated environment for Python projects. This means that each project can have its own dependencies, regardless of what dependencies every other project has.

So, in our little example above, we’d just need to create a separate virtual environment for both ProjectA and ProjectB and we’d be good to go. Each environment, in turn, would be able to depend on whatever version of ProjectC they choose, independent of the other.

The great thing about this is that there are no limits to the number of environments you can have since they’re just directories containing a few scripts. Plus, they’re easily created using the virtualenv or pyenv command line tools.

Using virtual environments

To get started, if you’re not using Python 3, you’ll want to install the virtualenv tool with pip:

$ pip install virtualenv

If you are using Python 3 then you should already have the venv module from the standard library installed.

From here on out we’ll assume you’re using the newer venv tool, since there are few differences between it and virtualenv with regard to the actual commands. In reality, though, they are very different tools.

Start by making a new directory to work with:

$ mkdir python-virtual-environments && cd python-virtual-environments

Create a new virtual environment inside the directory:

# Python 2:
$ virtualenv env

# Python 3
$ python3 -m venv env

By default this will NOT include any of your existing site packages.

The Python 3 venv approach has the benefit that it forces you to choose a specific version of the Python 3 interpreter that should be used to create the virtual environment. This avoids any confusion as to which Python installation the new environment is based on.

From Python 3.3 to 3.4 the recommended way to create a virtual environment was to use the pyvenv command-line tool that also comes included with your Python 3 installation by default. But on 3.6 and above, python3 -m venv is the way to go.

In the above example, this command creates a directory called “env”, which contains a directory structure similar to this:

├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── easy_install
│   ├── easy_install-3.5
│   ├── pip
│   ├── pip3
│   ├── pip3.5
│   ├── python -> python3.5
│   ├── python3 -> python3.5
│   └── python3.5 -> /Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5
├── include
├── lib
│   └── python3.5
│       └── site-packages
└── pyvenv.cfg

What does each folder contain?

“bin” - files that interact with the virtual environment
“include” - C headers that compile the Python packages
“lib” - a copy of the Python version along with a “site-packages” folder where each dependency is installed

Further, there are copies of, or symlinks to, a few different Python tools and to the Python executables themselves. These files are used to ensure all Python code and commands are executed within the context of the current environment, which is how the isolation from the global environment is achieved. We’ll explain this in more detail in the next section.

More interestingly are the activate scripts in the “bin” directory. These scripts are used to set up your shell to use the environment’s Python executable and its site-packages by default.

In order to use this environment’s packages/resources in isolation, you need to “activate” it. To do this, just run:

$ source env/bin/activate
(env) $

Notice how your prompt is now prefixed with the name of your environment (env, in our case). This is the indicator that env is currently active, which means the python executable will only use this environment’s packages and settings.

To show the package isolation in action, we can use the bcrypt module as an example. Let’s say we have bcrypt installed system-wide, but not in our virtual environment.

Before we test this, we need to go back to the “system” context by executing deactivate:

(env) $ deactivate
$

Now your shell session is back to normal, and the python command refers to the global Python install. Remember to do this whenever you’re done using a specific virtual environment.

Now, install bcyrpt and use it to hash a password:

$ pip -q install bcrypt
$ python -c "import bcrypt; print(bcrypt.hashpw('password'.encode('utf-8'), bcrypt.gensalt()))"
$2b$12$vWa/VSvxxyQ9d.WGgVTdrell515Ctux36LCga8nM5QTW0.4w8TXXi

What happens if we try the same command when the virtual environment is activated?

$ source env/bin/activate
(env) $ python -c "import bcrypt; print(bcrypt.hashpw('password'.encode('utf-8'), bcrypt.gensalt()))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'bcrypt'

As you can see, the behavior of the python -c "import bcrypt..." command changes after the source env/bin/activate call.

In one instance we have bcrypt available to us, and in the next we don’t. This is the kind of separation we’re looking to achieve with virtual environments, which is now easily achieved.

How does a virtual environment work?

So what exactly does it mean to “activate” an environment? Knowing what’s going on under the hood can be pretty important for a developer, especially when you need to understand execution environments, dependency resolution, etc.

To explain how this works, let’s first check out the locations of the different python executables. With the environment “deactivated”, run:

$ which python
/usr/bin/python

Now activate it and run the command again:

$ source env/bin/activate
(env) $ which python
/Users/michaelherman/python-virtual-environments/env/bin/python

After activating the environment we’re now getting a different path for the python executable because in an active environment the $PATH environment variable is slightly modified.

Notice the difference between the first path in $PATH before and after the activation:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:

$ source env/bin/activate
(env) $ echo $PATH
/Users/michaelherman/python-virtual-environments/env/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:

In the latter example, our virtual environment’s “bin” directory is now at the beginning of the path. That means it’s the first directory searched when running an executable on the command line. Thus, the shell uses our virtual environment’s instance of Python instead of the system-wide version.

Other packages that bundle Python, like Anaconda, also tend to manipulate your path when you activate them. Just be aware of this in case you run into problems with your other environments. This can become a problem if you start activating multiple environments at once.

This begs the questions:

What’s the difference between these two executables anyway?
How is the virtual environment’s Python executable able to use something other than the system’s site-packages?

This can be explained by how Python starts up and where it is located on the system. There actually isn’t any difference between these two Python executables. It’s their directory locations that matter.

When Python is starting up, it looks at the path of its binary (which, in a virtual environment, is actually just a copy of, or symlink to, your system’s Python binary). It then sets the location of sys.prefix and sys.exec_prefix based on this location, omitting the “bin” portion of the path.

The path located in sys.prefix is then used for locating the “site-packages” directory by searching the relative path lib/pythonX.X/site-packages/, where X.X is the version of Python you’re using.

In our example, the binary is located at /Users/michaelherman/python-virtual-environments/env/bin, which means sys.prefix would be /Users/michaelherman/python-virtual-environments/env, and therefore the “site-packages” directory used would be /Users/michaelherman/python-virtual-environments/env/bin/lib/pythonX.X/site-packages. Finally, this path is stored in the sys.path array, which contains all of the locations that a package can reside.

Managing virtual environments with virtualenvwrapper

While virtual environments certainly solve some big problems with package management, they’re not perfect. After creating a few environments, you’ll start to see that they create some problems of their own, most of which revolve around managing the environments themselves. To help with this, the virtualenvwrapper tool was created, which is just some wrapper scripts around the main virtualenv tool.

A few of the more useful features of virtualenvwrapper are that it:

Organizes all of your virtual environments in one location;
Provides methods to help you easily create, delete, and copy environments; and,
Provides a single command to switch between environments

While some of these features may seem small or insignificant, you’ll soon learn that they’re important tools to add to your workflow.

To get started, you can download the wrapper with pip:

$ pip install virtualenvwrapper

For Windows, you should use virtualenvwrapper-win instead.

Once installed, we’ll need to activate its shell functions, which can be done by running source on the installed virtualenvwrapper.sh script. When you first install it with pip, the output of the installation will tell you the exact location of virtualenvwrapper.sh. Or you can simply run:

$ which virtualenvwrapper.sh
/usr/local/bin/virtualenvwrapper.sh

Using that path, add the following three lines to your shell’s startup file. If you’re using the Bash shell, you would place these lines in either the ~/.bashrc file or ~/.profile file. For other shells, like zsh, csh, or fish, you would need to use the startup files specific to that shell. All that matters is these commands are executed when you log in or open a new shell.

export WORKON_HOME=$HOME/.virtualenvs   # optional
export PROJECT_HOME=$HOME/projects      # optional
source /usr/local/bin/virtualenvwrapper.sh

It’s not required to define the WORKON_HOME and PROJECT_HOME environment variables. virtualenvwrapper has default values for those, but you can override them by defining values.

Finally, reload the startup file:

$ source ~/.bashrc

There should now be a directory located at $WORKON_HOME that contains all of the virtualenvwrapper data/files:

$ echo $WORKON_HOME
/Users/michaelherman/.virtualenvs

You’ll also now have the shell commands available to you to help you manage the environments. Here are just a few available:

For more info on commands, installation, and configuring virtualenvwrapper, check out their documentation.

Now anytime you want to start a new project, all you have to do is:

$ mkvirtualenv my-new-project
(my-new-project) $

This will create and activate a new environment in the directory located at $WORKON_HOME, where all virtualenvwrapper environments are stored.

To stop using that environment, you just need to deactivate it like before:

(my-new-project) $ deactivate
$

If you have many environments to choose from, you can list them all with the workon function:

$ workon
my-new-project
my-django-project
web-scraper

And finally, to activate:

$ workon web-scraper
(web-scraper) $

If you would like to be able to use a single tool and switch between python versions, virtualenv will allow you to do just that. Virtualenv has a parameter -p which allows you to select which version of python to use. Combine that with the which command and we can easily select your preferred version of python to use in a simple manner. For example, let’s say that we want python3 as our preferred version:

$ virtualenv -p $(which python3) blog_virtualenv

This will create a new python3 environment.

So how does this work? The which command is used for finding a given command in your $PATH variable and returning the full path to that command. So, the full path to python3 was returned, to the -p parameter which takes a PYTHON_EXE. This could also be used for python2 as well. Just substitute python3 for python2 (or python if you system defaults to python2).

Now you don’t have to remember where you installed your environments; you can easily delete or copy them as you wish, and your project directory is less cluttered!

Using different versions of Python

Unlike the old virtualenv tool, pyvenv doesn’t support creating environments with arbitrary versions of Python, which means you’re stuck using the default Python 3 installation for all of the environments you create. While you can upgrade an environment to the latest system version of Python (via the --upgrade option) if it changes, you still can’t actually specify a particular version.

There are quite a few ways to install Python, but few of them are easy enough or flexible enough to frequently uninstall and re-install different versions of the binary.

This is where pyenv comes in to play.

Despite the similarity in names (pyvenv vs pyenv), pyenv is different in that its focus is to help you switch between Python versions on a system-level as well as a project-level. So, while pyvenv‘s purpose is to separate out modules, pyenv‘s purpose is to separate Python versions.

You can start by installing pyenv with either Homebrew (on OS X), or with the pyenv-installer project:

Homebrew

$ brew install pyenv

pyenv-installer

$ curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash

Unfortunately, pyenv does not support Windows. A few alternatives to try are pywin and anyenv.

Once you have pyenv on your system, here are a few of the basic commands you’re probably interested in:

$ pyenv install 3.5.0   # Install new version
$ pyenv versions        # List installed versions
$ pyenv exec python -V  # Execute 'python -V' using pyenv version

In these few lines we install the 3.5.0 version of Python, ask pyenv to show us all of the versions available to us, and then execute the python -V command using the pyenv-specified version.

To give you even more control, you can then use any of the available versions for either “global” use or “local” use. Using pyenv with the local command sets the Python version for a specific project or directory by storing the version number in a local .python-version file. We can set the “local” version like this:

$ pyenv local 2.7.11

This creates the .python-version file in our current directory, as you can see here:

$ ls -la
total 16
drwxr-xr-x  4 michaelherman  staff  136 Feb 22 10:57 .
drwxr-xr-x  9 michaelherman  staff  306 Jan 27 20:55 ..
-rw-r--r--  1 michaelherman  staff    7 Feb 22 10:57 .python-version
-rw-r--r--  1 michaelherman  staff   52 Jan 28 17:20 main.py

This file only contains the contents “2.7.11”. Now when you execute a script using pyenv, it’ll load this file and use the specified version, assuming it’s valid and exists on your system.

Moving on with our example, let’s say we have a simple script called main.py in our project directory that looks like this:

import sys
print('Using version:', sys.version[:5])

All it does is print out the version number of the Python executable being used. Using pyenv and the exec command, we can run the script with any of the different versions of Python we have installed.

$ python main.py
Using version: 2.7.5
$ pyenv global 3.5.0
$ pyenv exec python main.py
Using version: 3.5.0
$ pyenv local 2.7.11
$ pyenv exec python main.py
Using version: 2.7.11

Notice how pyenv exec python main.py uses our “global” Python version by default, but then it uses the “local” version after one is set for the current directory.

This can be very powerful for developers who have lots of projects with varying version requirements. Not only can you easily change the default version for all projects (via global), but you can also override it to specify special cases.

Conclusion

In this article you learned about how Python dependencies are stored and resolved, and how to use different community tools to help get around various packaging and versioning problems.

As you can see, thanks to the huge Python community there are quite a few tools at your disposal to help with these common problems. As you progress as a developer, be sure to take time to learn how to use these tools to your advantage. You may even find unintended uses for them, or learn to apply similar concepts to other languages you use.

Free Bonus: Click here to get access to a free 5-day class that shows you how to avoid common dependency management issues with tools like Pip, PyPI, Virtualenv, and requirements files.

This is a collaboration piece between Scott Robinson, author of Stack Abuse and the folks at Real Python.

Real Python

Python + Memcached: Efficient Caching in Distributed Applications

Installing memcached

Storing and Retrieving Cached Values Using Python

Automatically Expiring Cached Data

Warming Up a Cold Cache

Check And Set

Beyond Caching

Simplifying Offline Python Deployments With Docker

Objectives

Scenario

Bundle

Environment Setup

Deploy

Conclusion

Practical Introduction to Web Scraping in Python

Web Scraping Basics

Setting Up Your Python Web Scraper

Making Web Requests

Wrangling HTML With BeautifulSoup

Using BeautifulSoup to Get Mathematician Names

Getting the Popularity Score

Putting It All Together

Conclusion & Next Steps

Building a Simple Web App With Bottle, SQLAlchemy, and the Twitter API

Project Setup

Implementation Details

The DB and SQLAlchemy

Query the Twitter API

Make a Simple web app with Bottle

Bottle Templates

Add tests with pytest

Code quality matters - Better Code Hub

Conclusion and Learning

We Challenge You

Code Evaluation With AWS Lambda and API Gateway

Objectives

What is AWS Lambda?

Project Setup

Lambda Setup

Create function

Test

API Gateway Setup

Create the API

Test it manually

Enable CORS

Deploy the API

Test via cURL

Update the Form

Next Steps

User Authentication With Angular 4 and Flask

Auth Workflow

Project Setup

Auth Component

Bootstrap Style

Auth Service

Sanity Check

User Login

User Registration

Server-side Setup

Sanity Check

Auth Login

Auth Register

Local Storage

User Status

Route Restriction

What’s Next?

Test Driven Development of a Django RESTful API

Objectives

Why Django REST Framework?

Django Project Setup

Django App and REST Framework Setup

Database and Model Setup

Sanity Check

Serializers

RESTful Structure

Routes and Testing (TDD)

Browsable API

Routes

GET ALL

Delicious Pizza Factories With `@classmethod`