Real Python

Python String Formatting Tips & Best Practices

2019-04-11T14:00:00+00:00

Learn the four main approaches to string formatting in Python, as well as their strengths and weaknesses. You’ll also get a simple rule of thumb for how to pick the best general purpose string formatting approach in your own programs.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Create an Index in Django Without Downtime

2019-04-10T14:00:00+00:00

Managing database migrations is a great challenge in any software project. Luckily, as of version 1.7, Django comes with a built-in migration framework. The framework is very powerful and useful in managing change in databases. But the flexibility provided by the framework required some compromises. To understand the limitations of Django migrations, you are going to tackle a well known problem: creating an index in Django with no downtime.

In this tutorial, you’ll learn:

How and when Django generates new migrations
How to inspect the commands Django generates to execute migrations
How to safely modify migrations to fit your needs

This intermediate-level tutorial is designed for readers who are already familiar with Django migrations. For an introduction to that topic, check out Django Migrations: A Primer.

Free Bonus: Click here to get free access to additional Django tutorials and resources you can use to deepen your Python web development skills.

The Problem With Creating an Index in Django Migrations

A common change that usually becomes necessary when the data stored by your application grows is adding an index. Indexes are used to speed up queries and make your app feel fast and responsive.

In most databases, adding an index requires an exclusive lock on the table. An exclusive lock prevents data modification (DML) operations such as UPDATE, INSERT, and DELETE, while the index is created.

Locks are obtained implicitly by the database when executing certain operations. For example, when a user logs into your app, Django will update the last_login field in the auth_user table. To perform the update, the database will first have to obtain a lock on the row. If the row is currently being locked by another connection, then you might get a database exception.

Locking a table might pose a problem when it’s necessary to keep the system available during migrations. The bigger the table, the longer it can take to create the index. The longer it takes to create the index, the longer the system is unavailable or unresponsive to users.

Some database vendors provide a way to create an index without locking the table. For example, to create an index in PostgreSQL without locking a table, you can use the CONCURRENTLY keyword:

CREATE INDEX CONCURRENTLY ix ON table (column);

In Oracle, there is an ONLINE option to allow DML operations on the table while the index is created:

CREATE INDEX ix ON table (column) ONLINE;

When generating migrations, Django will not use these special keywords. Running the migration as is will make the database acquire an exclusive lock on the table and prevent DML operations while the index is created.

Creating an index concurrently has some caveats. It’s important to understand the issues specific to your database backend in advance. For example, one caveat in PostgreSQL is that creating an index concurrently takes longer because it requires an additional table scan.

In this tutorial, you’ll use Django migrations to create an index on a large table, without causing any downtime.

Note: To follow this tutorial, it is recommended that you use a PostgreSQL backend, Django 2.x, and Python 3.

It is possible to follow along with other database backends as well. In places where SQL features unique to PostgreSQL are used, change the SQL to match your database backend.

Setup

You’re going to use a made up Sale model in an app called app. In a real life situation, models such as Sale are the main tables in the database, and they will usually be very big and store a lot of data:

# models.py

from django.db import models

class Sale(models.Model):
    sold_at = models.DateTimeField(
        auto_now_add=True,
    )
    charged_amount = models.PositiveIntegerField()

To create the table, generate the initial migration and apply it:

$ python manage.py makemigrations
Migrations for 'app':
  app/migrations/0001_initial.py
    - Create model Sale

$ python manage migrate
Operations to perform:
  Apply all migrations: app
Running migrations:
  Applying app.0001_initial... OK

After a while, the sales table becomes very big, and users start to complain about slowness. While monitoring the database, you noticed that a lot of queries use the sold_at column. To speed things up, you decide that you need an index on the column.

To add an index on sold_at, you make the following change to the model:

# models.py

from django.db import models

class Sale(models.Model):
    sold_at = models.DateTimeField(
        auto_now_add=True,
        db_index=True,
    )
    charged_amount = models.PositiveIntegerField()

If you run this migration as it is, then Django will create the index on the table, and it will be locked until the index is completed. It can take a while to create an index on a very large table, and you want to avoid downtime.

On a local development environment with a small dataset and very few connections, this migration might feel instantaneous. However, on large datasets with many concurrent connections, obtaining a lock and creating the index can take a while.

In the next steps, you are going to modify migrations created by Django to create the index without causing any downtime.

Fake Migration

The first approach is to create the index manually. You are going to generate the migration, but you are not going to actually let Django apply it. Instead, you will run the SQL manually in the database and then make Django think the migration completed.

First, generate the migration:

$ python manage.py makemigrations --name add_index_fake
Migrations for 'app':
  app/migrations/0002_add_index_fake.py
    - Alter field sold_at on sale

Use the sqlmigrate command to view the SQL Django will use to execute this migration:

$ python manage.py sqlmigrate app 0002

BEGIN;
--
-- Alter field sold_at on sale
--
CREATE INDEX "app_sale_sold_at_b9438ae4" ON "app_sale" ("sold_at");
COMMIT;

You want to create the index without locking the table, so you need to modify the command. Add the CONCURRENTLY keyword and execute in the database:

app=# CREATE INDEX CONCURRENTLY "app_sale_sold_at_b9438ae4"
ON "app_sale" ("sold_at");

CREATE INDEX

Notice that you executed the command without the BEGIN and COMMIT parts. Omitting these keywords will execute the commands without a database transaction. We will discuss database transactions later in the article.

After you executed the command, if you try to apply migrations, then you will get the following error:

$ python manage.py migrate

Operations to perform:
  Apply all migrations: app
Running migrations:
  Applying app.0002_add_index_fake...Traceback (most recent call last):
  File "venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)

psycopg2.ProgrammingError: relation "app_sale_sold_at_b9438ae4" already exists

Django complains that the index already exists, so it can’t proceed with the migration. You just created the index directly in the database, so now you need to make Django think that the migration was already applied.

How to Fake a Migration

Django provides a built-in way of marking migrations as executed, without actually executing them. To use this option, set the --fake flag when applying the migration:

$ python manage.py migrate --fake
Operations to perform:
  Apply all migrations: app
Running migrations:
  Applying app.0002_add_index_fake... FAKED

Django didn’t raise an error this time. In fact, Django didn’t really apply any migration. It just marked it as executed (or FAKED).

Here are some issues to consider when faking migrations:

The manual command must be equivalent to the SQL generated by Django: You need to make sure the command you execute is equivalent to the SQL generated by Django. Use sqlmigrate to produce the SQL command. If the commands do not match, then you might end up with inconsistencies between the database and the models state.
Other unapplied migrations will also be faked: When you have multiple unapplied migrations, they will all be faked. Before you apply migrations, it’s important to make sure only the migrations you want to fake are unapplied. Otherwise, you might end up with inconsistencies. Another option is to specify the exact migration you want to fake.
Direct access to the database is required: You need to run the SQL command in the database. This is not always an option. Also, executing commands directly in a production database is dangerous and should be avoided when possible.
Automated deployment processes might need adjustments: If you automated the deployment process (using CI, CD, or other automation tools), then you might need to alter the process to fake migrations. This is not always desirable.

Cleanup

Before moving on to the next section, you need to bring the database back to its state right after the initial migration. To do that, migrate back to the initial migration:

$ python manage.py migrate 0001
Operations to perform:
  Target specific migration: 0001_initial, from app
Running migrations:
  Rendering model states... DONE
  Unapplying app.0002_add_index_fake... OK

Django unapplied the changes made in the second migration, so now it’s safe to also delete the file:

$ rm app/migrations/0002_add_index_fake.py

To make sure you did everything right, inspect the migrations:

$ python manage.py showmigrations app
app
 [X] 0001_initial

The initial migration was applied, and there are no unapplied migrations.

Execute Raw SQL in Migrations

In the previous section, you executed SQL directly in the database and faked the migration. This gets the job done, but there is a better solution.

Django provides a way to execute raw SQL in migrations using RunSQL. Let’s try to use it instead of executing the command directly in the database.

First, generate a new empty migration:

$ python manage.py makemigrations app --empty --name add_index_runsql
Migrations for 'app':
  app/migrations/0002_add_index_runsql.py

Next, edit the migration file and add a RunSQL operation:

# migrations/0002_add_index_runsql.py

from django.db import migrations, models

class Migration(migrations.Migration):
    atomic = False

    dependencies = [
        ('app', '0001_initial'),
    ]

    operations = [
        migrations.RunSQL(
            'CREATE INDEX "app_sale_sold_at_b9438ae4" '
            'ON "app_sale" ("sold_at");',
        ),
    ]

When you run the migration, you will get the following output:

$ python manage.py migrate
Operations to perform:
  Apply all migrations: app
Running migrations:
  Applying app.0002_add_index_runsql... OK

This is looking good, but there is a problem. Let’s try to generate migrations again:

$ python manage.py makemigrations --name leftover_migration
Migrations for 'app':
  app/migrations/0003_leftover_migration.py
    - Alter field sold_at on sale

Django generated the same migration again. Why did it do that?

Cleanup

Before we can answer that question, you need to clean up and undo the changes you made to the database. Start by deleting the last migration. It was not applied, so it’s safe to delete:

$ rm app/migrations/0003_leftover_migration.py

Next, list the migrations for the app app:

$ python manage.py showmigrations app
app
 [X] 0001_initial
 [X] 0002_add_index_runsql

The third migration is gone, but the second is applied. You want to get back to the state right after the initial migration. Try to migrate back to the initial migration as you did in the previous section:

$ python manage.py migrate app 0001
Operations to perform:
  Target specific migration: 0001_initial, from app
Running migrations:
  Rendering model states... DONE
  Unapplying app.0002_add_index_runsql...Traceback (most recent call last):

NotImplementedError: You cannot reverse this operation

Django is unable to reverse the migration.

Reverse Migration Operation

To reverse a migration, Django executes an opposite action for every operation. In this case, the reverse of adding an index is to drop it. As you’ve already seen, when a migration is reversible, you can unapply it. Just like you can use checkout in Git, you can reverse a migration if you execute migrate to an earlier migration.

Many built-in migration operations already define a reverse action. For example, the reverse action for adding a field is to drop the corresponding column. The reverse action for creating a model is to drop the corresponding table.

Some migration operations are not reversible. For example, there is no reverse action for removing a field or deleting a model, because once the migration was applied, the data is gone.

In the previous section, you used the RunSQL operation. When you tried to reverse the migration, you encountered an error. According to the error, one of the operations in the migration cannot be reversed. Django is unable to reverse raw SQL by default. Because Django has no knowledge of what was executed by the operation, it cannot generate an opposite action automatically.

How to Make a Migration Reversible

For a migration to be reversible, all the operations in it must be reversible. It’s not possible to reverse part of a migration, so a single non-reversible operation will make the entire migration non-reversible.

To make a RunSQL operation reversible, you must provide SQL to execute when the operation is reversed. The reverse SQL is provided in the reverse_sql argument.

The opposite action to adding an index is to drop it. To make your migration reversible, provide the reverse_sql to drop the index:

# migrations/0002_add_index_runsql.py

from django.db import migrations, models

class Migration(migrations.Migration):
    atomic = False

    dependencies = [
        ('app', '0001_initial'),
    ]

    operations = [
        migrations.RunSQL(
            'CREATE INDEX "app_sale_sold_at_b9438ae4" '
            'ON "app_sale" ("sold_at");',

            reverse_sql='DROP INDEX "app_sale_sold_at_b9438ae4";',
        ),
    ]

Now try to reverse the migration:

$ python manage.py showmigrations app
app
 [X] 0001_initial
 [X] 0002_add_index_runsql

$ python manage.py migrate app 0001
Operations to perform:
  Target specific migration: 0001_initial, from app
Running migrations:
  Rendering model states... DONE
 Unapplying app.0002_add_index_runsql... OK

$ python manage.py showmigrations app
app
 [X] 0001_initial
 [ ] 0002_add_index_runsql

The second migration was reversed, and the index was dropped by Django. Now it’s safe to delete the migration file:

$ rm app/migrations/0002_add_index_runsql.py

It’s always a good idea to provide reverse_sql. In situations where reversing a raw SQL operation does not require any action, you can mark the operation as reversible using the special sentinel migrations.RunSQL.noop:

migrations.RunSQL(
    sql='...',  # Your forward SQL here
    reverse_sql=migrations.RunSQL.noop,
),

Understand Model State and Database State

In your previous attempt to create the index manually using RunSQL, Django generated the same migration over and over again even though the index was created in the database. To understand why Django did that, you first need to understand how Django decides when to generate new migrations.

When Django Generates a New Migration

In the process of generating and applying migrations, Django syncs between the state of the database and the state of the models. For example, when you add a field to a model, Django adds a column to the table. When you remove a field from the model, Django removes the column from the table.

To sync between the models and the database, Django maintains a state that represents the models. To sync the database with the models, Django generates migration operations. Migration operations translate to a vendor specific SQL that can be executed in the database. When all migration operations are executed, the database and the models are expected to be consistent.

To get the state of the database, Django aggregates the operations from all past migrations. When the aggregated state of the migrations is not consistent with the state of the models, Django generates a new migration.

In the previous example, you created the index using raw SQL. Django did not know you created the index because you didn’t use a familiar migration operation.

When Django aggregated all the migrations and compared them with the state of the models, it found that an index was missing. This is why, even after you created the index manually, Django still thought it was missing and generated a new migration for it.

How to Separate Database and State in Migrations

Since Django is unable to create the index the way you want it to, you want to provide your own SQL but still let Django know you created it.

In other words, you need to execute something in the database and provide Django with the migration operation to sync its internal state. To do that, Django provides us with a special migration operation called SeparateDatabaseAndState. This operation is not well known and should be reserved for special cases such as this one.

It’s much easier to edit migrations than write them from scratch, so start by generating a migration the usual way:

$ python manage.py makemigrations --name add_index_separate_database_and_state

Migrations for 'app':
  app/migrations/0002_add_index_separate_database_and_state.py
    - Alter field sold_at on sale

This is the contents of the migration generated by Django, same as before:

# migrations/0002_add_index_separate_database_and_state.py

from django.db import migrations, models

class Migration(migrations.Migration):

    dependencies = [
        ('app', '0001_initial'),
    ]

    operations = [
        migrations.AlterField(
            model_name='sale',
            name='sold_at',
            field=models.DateTimeField(
                auto_now_add=True,
                db_index=True,
            ),
        ),
    ]

Django generated an AlterField operation on the field sold_at. The operation will create an index and update the state. We want to keep this operation but provide a different command to execute in the database.

Once again, to get the command, use the SQL generated by Django:

$ python manage.py sqlmigrate app 0002
BEGIN;
--
-- Alter field sold_at on sale
--
CREATE INDEX "app_sale_sold_at_b9438ae4" ON "app_sale" ("sold_at");
COMMIT;

Add the CONCURRENTLY keyword in the appropriate place:

CREATE INDEX CONCURRENTLY "app_sale_sold_at_b9438ae4"
ON "app_sale" ("sold_at");

Next, edit the migration file and use SeparateDatabaseAndState to provide your modified SQL command for execution:

# migrations/0002_add_index_separate_database_and_state.py

from django.db import migrations, models

class Migration(migrations.Migration):

    dependencies = [
        ('app', '0001_initial'),
    ]

    operations = [

        migrations.SeparateDatabaseAndState(

            state_operations=[
                migrations.AlterField(
                    model_name='sale',
                    name='sold_at',
                    field=models.DateTimeField(
                        auto_now_add=True,
                        db_index=True,
                    ),
                ),
            ],

            database_operations=[
                migrations.RunSQL(sql="""
                    CREATE INDEX CONCURRENTLY "app_sale_sold_at_b9438ae4"
                    ON "app_sale" ("sold_at");
                """, reverse_sql="""
                    DROP INDEX "app_sale_sold_at_b9438ae4";
                """),
            ],
        ),

    ],

The migration operation SeparateDatabaseAndState accepts 2 lists of operations:

state_operations are operations to apply on the internal model state. They do not effect the database.
database_operations are operations to apply to the database.

You kept the original operation generated by Django in state_operations. When using SeparateDatabaseAndState, this is what you will usually want to do. Notice that the db_index=True argument is provided to the field. This migration operation will let Django know that there is an index on the field.

You used the SQL generated by Django and added the CONCURRENTLY keyword. You used the special action RunSQL to execute raw SQL in the migration.

If you try to run the migration, you will get the following output:

$ python manage.py migrate app
Operations to perform:
  Apply all migrations: app
Running migrations:
  Applying app.0002_add_index_separate_database_and_state...Traceback (most recent call last):
  File "/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 83, in _execute
    return self.cursor.execute(sql)
psycopg2.InternalError: CREATE INDEX CONCURRENTLY cannot run inside a transaction block

Non-Atomic Migrations

In SQL, CREATE, DROP, ALTER, and TRUNCATE operations are referred to as Data Definition Language (DDL). In databases that support transactional DDL, such as PostgreSQL, Django executes migrations inside a database transaction by default. However, according to the error above, PostgreSQL cannot create an index concurrently inside a transaction block.

To be able to create an index concurrently within a migration, you need to tell Django to not execute the migration in a database transaction. To do that, you mark the migration as non-atomic by setting atomic to False:

# migrations/0002_add_index_separate_database_and_state.py

from django.db import migrations, models

class Migration(migrations.Migration):
    atomic = False

    dependencies = [
        ('app', '0001_initial'),
    ]

    operations = [

        migrations.SeparateDatabaseAndState(

            state_operations=[
                migrations.AlterField(
                    model_name='sale',
                    name='sold_at',
                    field=models.DateTimeField(
                        auto_now_add=True,
                        db_index=True,
                    ),
                ),
            ],

            database_operations=[
                migrations.RunSQL(sql="""
                    CREATE INDEX CONCURRENTLY "app_sale_sold_at_b9438ae4"
                    ON "app_sale" ("sold_at");
                """,
                reverse_sql="""
                    DROP INDEX "app_sale_sold_at_b9438ae4";
                """),
            ],
        ),

    ],

After you marked the migration as non-atomic, you can run the migration:

$ python manage.py migrate app
Operations to perform:
  Apply all migrations: app
Running migrations:
  Applying app.0002_add_index_separate_database_and_state... OK

You just executed the migration without causing any downtime.

Here are some issues to consider when you’re using SeparateDatabaseAndState:

Database operations must be equivalent to state operations: Inconsistencies between the database and model state can cause a lot of trouble. A good starting point is to keep the operations generated by Django in state_operations and edit the output of sqlmigrate to use in database_operations.
Non atomic migrations cannot rollback in case of error: If there is an error during the migration, then you won’t be able to rollback. You would have to either rollback the migration or complete it manually. It’s a good idea to keep the operations executed inside a non-atomic migration to a minimum. If you have additional operations in the migration, move them to a new migration.
Migration might be vendor specific: The SQL generated by Django is specific to the database backend used in the project. It might work with other database backends, but that is not guaranteed. If you need to support multiple database backends, you need to make some adjustments to this approach.

Conclusion

You started this tutorial with a large table and a problem. You wanted to make your app faster for your users, and you wanted to do that without causing them any downtime.

By the end of the tutorial, you managed to generate and safely modify a Django migration to achieve this goal. You tackled different problems along the way and managed to overcome them using built-in tools provided by the migrations framework.

In this tutorial, you learned the following:

How Django migrations work internally using model and database state, and when new migrations are generated
How to execute custom SQL in migrations using the RunSQL action
What reversible migrations are, and how to make a RunSQL action reversible
What atomic migrations are, and how to change the default behavior according to your needs
How to safely execute complex migrations in Django

The separation between model and database state is an important concept. Once you understand it, and how to utilize it, you can overcome many limitations of the built-in migration operations. Some use cases that come to mind include adding an index that was already created in the database and providing vendor specific arguments to DDL commands.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Running Python Scripts

2019-04-09T14:00:00+00:00

One of the most important skills you need to build as a Python developer is to be able to run Python scripts and code. This is going to be the only way for you to know if your code works as you planned. It’s even the only way of knowing if your code works at all!

This step-by-step course will guide you through a series of ways to run Python scripts, depending on your environment, platform, needs, and skills as a programmer.

You’ll learn how to run Python scripts by using:

The operating system command-line or terminal
The Python interactive mode
The IDE or text editor you like best
The file manager of your system, by double-clicking on the icon of your script

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python REST APIs With Flask, Connexion, and SQLAlchemy – Part 3

2019-04-08T14:00:00+00:00

In Part 2 of this series, you added the ability to save changes made through the REST API to a database using SQLAlchemy and learned how to serialize that data for the REST API using Marshmallow. Connecting the REST API to a database so that the application can make changes to existing data and create new data is great and makes the application much more useful and robust.

That’s only part of the power a database offers, however. An even more powerful feature is the R part of RDBMS systems: relationships. In a database, a relationship is the ability to connect two or more tables together in a meaningful way. In this article, you’ll learn how to implement relationships and turn your Person database into a mini-blogging web application.

In this article, you’ll learn:

Why more than one table in a database is useful and important
How tables are related to each other
How SQLAlchemy can help you manage relationships
How relationships help you build a mini-blogging application

Who This Article Is For

Part 1 of this series guided you through building a REST API, and Part 2 showed you how to connect that REST API to a database.

This article expands your programming tool belt further. You’ll learn how to create hierarchical data structures represented as one-to-many relationships by SQLAlchemy. In addition, you’ll extend the REST API you’ve already built to provide CRUD (Create, Read, Update, and Delete) support for the elements in this hierarchical structure.

The web application presented in Part 2 will have its HTML and JavaScript files modified in major ways to create a more fully functional mini-blogging application. You can review the final version of the code from Part 2 in the GitHub repository for that article.

Hang on as you get started creating relationships and your mini-blogging application!

Additional Dependencies

There are no new Python dependencies beyond what was required for the Part 2 article. However, you will be using two new JavaScript modules in the web application to makes things easier and more consistent. The two modules are the following:

Handlebars.js is a templating engine for JavaScript, much like Jinja2 for Flask.
Moment.js is a datetime parsing and formatting module that makes displaying UTC timestamps easier.

You don’t have to download either of these, as the web application will get them directly from the Cloudflare CDN (Content Delivery Network), as you’re already doing for the jQuery module.

People Data Extended for Blogging

In Part 2, the People data existed as a dictionary in the build_database.py Python code. This is what you used to populate the database with some initial data. You’re going to modify the People data structure to give each person a list of notes associated with them. The new People data structure will look like this:

# Data to initialize database with
PEOPLE = [
    {
        "fname": "Doug",
        "lname": "Farrell",
        "notes": [
            ("Cool, a mini-blogging application!", "2019-01-06 22:17:54"),
            ("This could be useful", "2019-01-08 22:17:54"),
            ("Well, sort of useful", "2019-03-06 22:17:54"),
        ],
    },
    {
        "fname": "Kent",
        "lname": "Brockman",
        "notes": [
            (
                "I'm going to make really profound observations",
                "2019-01-07 22:17:54",
            ),
            (
                "Maybe they'll be more obvious than I thought",
                "2019-02-06 22:17:54",
            ),
        ],
    },
    {
        "fname": "Bunny",
        "lname": "Easter",
        "notes": [
            ("Has anyone seen my Easter eggs?", "2019-01-07 22:47:54"),
            ("I'm really late delivering these!", "2019-04-06 22:17:54"),
        ],
    },
]

Each person in the People dictionary now includes a key called notes, which is associated with a list containing tuples of data. Each tuple in the notes list represents a single note containing the content and a timestamp. The timestamps are initialized (rather than dynamically created) to demonstrate ordering later on in the REST API.

Each single person is associated with multiple notes, and each single note is associated with only one person. This hierarchy of data is known as a one-to-many relationship, where a single parent object is related to many child objects. You’ll see how this one-to-many relationship is managed in the database with SQLAlchemy.

Brute Force Approach

The database you built stored the data in a table, and a table is a two-dimensional array of rows and columns. Can the People dictionary above be represented in a single table of rows and columns? It can be, in the following way, in your person database table. Unfortunately to include all of the actual data in the example creates a scroll bar for the table, as you’ll see below:

`person_id`	`lname`	`fname`	`timestamp`	`content`	`note_timestamp`
1	Farrell	Doug	2018-08-08 21:16:01	Cool, a mini-blogging application!	2019-01-06 22:17:54
2	Farrell	Doug	2018-08-08 21:16:01	This could be useful	2019-01-08 22:17:54
3	Farrell	Doug	2018-08-08 21:16:01	Well, sort of useful	2019-03-06 22:17:54
4	Brockman	Kent	2018-08-08 21:16:01	I’m going to make really profound observations	2019-01-07 22:17:54
5	Brockman	Kent	2018-08-08 21:16:01	Maybe they’ll be more obvious than I thought	2019-02-06 22:17:54
6	Easter	Bunny	2018-08-08 21:16:01	Has anyone seen my Easter eggs?	2019-01-07 22:47:54
7	Easter	Bunny	2018-08-08 21:16:01	I’m really late delivering these!	2019-04-06 22:17:54

The above table would actually work. All the data is represented, and a single person is associated with a collection of different notes.

Advantages

Conceptually, the above table structure has the advantage of being relatively simple to understand. You could even make the case that the data could be persisted to a flat file instead of a database.

Because of the two-dimensional table structure, you could store and use this data in a spreadsheet. Spreadsheets have been pressed into service as data storage quite a bit.

Disadvantages

While the above table structure would work, it has some real disadvantages.

In order to represent the collection of notes, all the data for each person is repeated for every unique note, the person data is therefore redundant. This isn’t such a big deal for your person data as there aren’t that many columns. But imagine if a person had many more columns. Even with large disk drives, this can get to be a storage concern if you’re dealing with millions of rows of data.

Having redundant data like this can lead to maintenance issues as time goes by. For example, what if the Easter Bunny decided a change of name was a good idea. In order to do this, every record containing the Easter Bunny’s name would have to be updated in order to keep the data consistent. This kind of work against the database can lead to data inconsistency, particularly if the work is done by a person running a SQL query by hand.

Naming columns becomes awkward. In the table above, there is a timestamp column used to track the creation and update time of a person in the table. You also want to have similar functionality for the creation and update time for a note, but because timestamp is already used, a contrived name of note_timestamp is used.

What if you wanted to add additional one-to-many relationships to the person table? For example, to include a person’s children or phone numbers. Each person could have multiple children and multiple phone numbers. This could be done relatively easily to the Python People dictionary above by adding children and phone_numbers keys with new lists containing the data.

However, representing those new one-to-many relationships in your person database table above becomes significantly more difficult. Every new one-to-many relationship increases the number of rows necessary to represent it for every single entry in the child data dramatically. In addition, the problems associated with data redundancy get bigger and more difficult to handle.

Lastly, the data you’d get back from the above table structure wouldn’t be very Pythonic: it would be just a big list of lists. SQLAlchemy wouldn’t be able to help you very much because the relationship isn’t there.

Relational Database Approach

Based on what you’ve seen above, it becomes clear that trying to represent even a moderately complex dataset in a single table becomes unmanageable pretty quickly. Given that, what alternative does a database offer? This is where the R part of RDBMS databases comes into play. Representing relationships removes the disadvantages outlined above.

Instead of trying to represent hierarchical data in a single table, the data is broken up into multiple tables, with a mechanism to relate them to one another. The tables are broken along collection lines, so for your People dictionary above, this means there will be a table representing people and another representing notes. This brings back your original person table, which looks like this:

`person_id`	`lname`	`fname`	`timestamp`
1	Farrell	Doug	2018-08-08 21:16:01.888444
2	Brockman	Kent	2018-08-08 21:16:01.889060
3	Easter	Bunny	2018-08-08 21:16:01.886834

To represent the new note information, you’ll create a new table called note. (Remember our singular table naming convention.) The table looks like this:

`note_id`	`person_id`	`content`	`timestamp`
1	1	Cool, a mini-blogging application!	2019-01-06 22:17:54
2	1	This could be useful	2019-01-08 22:17:54
3	1	Well, sort of useful	2019-03-06 22:17:54
4	2	I’m going to make really profound observations	2019-01-07 22:17:54
5	2	Maybe they’ll be more obvious than I thought	2019-02-06 22:17:54
6	3	Has anyone seen my Easter eggs?	2019-01-07 22:47:54
7	3	I’m really late delivering these!	2019-04-06 22:17:54

Notice that, like the person table, the note table has a unique identifier called note_id, which is the primary key for the note table. One thing that isn’t obvious is the inclusion of the person_id value in the table. What is that used for? This is what creates the relationship to the person table. Whereas note_id is the primary key for the table, person_id is what’s known as a foreign key.

The foreign key gives each entry in the note table the primary key of the person record it’s associated with. Using this, SQLAlchemy can gather all the notes associated with each person by connecting the person.person_id primary key to the note.person_id foreign key, creating a relationship.

Advantages

By breaking the data set into two tables, and introducing the concept of a foreign key, you’ve made the data a little more complex to think about, you have resolved the disadvantages of a single table representation. SQLAlchemy will help you encode the increased complexity fairly easily.

The data is no longer redundant in the database. There is only one person entry for each person you want to store in the database. This solves the storage concern immediately and dramatically simplifies the maintenance concerns.

If the Easter Bunny still wanted to change names, then you’d only have to change a single row in the person table, and anything else related to that row (like the note table) would immediately take advantage of the change.

Column naming is more consistent and meaningful. Because person and note data exist in separate tables, the creation and update timestamp can be named consistently in both tables, as there is no conflict for names across tables.

In addition, you’d no longer have to create permutations of each row for new one-to-many relationships you might want to represent. Take our children and phone_numbers example from earlier. Implementing this would require child and phone_number tables. Each table would contain a foreign key of person_id relating it back to the person table.

Using SQLAlchemy, the data you’d get back from the above tables would be more immediately useful, as what you’d get is an object for each person row. That object has named attributes equivalent to the columns in the table. One of those attributes is a Python list containing the related note objects.

Disadvantages

Where the brute force approach was simpler to understand, the concept of foreign keys and relationships make thinking about the data somewhat more abstract. This abstraction needs to be thought about for every relationship you establish between tables.

Making use of relationships means committing to using a database system. This is another tool to install, learn, and maintain above and beyond the application that actually uses the data.

SQLAlchemy Models

To use the two tables above, and the relationship between them, you’ll need to create SQLAlchemy models that are aware of both tables and the relationship between them. Here’s the SQLAlchemy Person model from Part 2, updated to include a relationship to a collection of notes:

 1 class Person(db.Model):
 2     __tablename__ = 'person'
 3     person_id = db.Column(db.Integer, primary_key=True)
 4     lname = db.Column(db.String(32))
 5     fname = db.Column(db.String(32))
 6     timestamp = db.Column(
 7         db.DateTime, default=datetime.utcnow, onupdate=datetime.utcnow
 8     )
 9     notes = db.relationship(
10         'Note',
11         backref='person',
12         cascade='all, delete, delete-orphan',
13         single_parent=True,
14         order_by='desc(Note.timestamp)'
15     )

Lines 1 to 8 of the above Python class look exactly like what you created before in Part 2. Lines 9 to 16 create a new attribute in the Person class called notes. This new notes attributes is defined in the following lines of code:

Line 9: Like the other attributes of the class, this line creates a new attribute called notes and sets it equal to an instance of an object called db.relationship. This object creates the relationship you’re adding to the Person class and is created with all of the parameters defined in the lines that follow.
Line 10: The string parameter 'Note' defines the SQLAlchemy class that the Person class will be related to. The Note class isn’t defined yet, which is why it’s a string here. This is a forward reference and helps handle problems that the order of definitions could cause when something is needed that isn’t defined until later in the code. The 'Note' string allows the Person class to find the Note class at runtime, which is after both Person and Note have been defined.
Line 11: The backref='person' parameter is trickier. It creates what’s known as a backwards reference in Note objects. Each instance of a Note object will contain an attribute called person. The person attribute references the parent object that a particular Note instance is associated with. Having a reference to the parent object (person in this case) in the child can be very useful if your code iterates over notes and has to include information about the parent. This happens surprisingly often in display rendering code.
Line 12: The cascade='all, delete, delete-orphan' parameter determines how to treat note object instances when changes are made to the parent Person instance. For example, when a Person object is deleted, SQLAlchemy will create the SQL necessary to delete the Person from the database. Additionally, this parameter tells it to also delete all the Note instances associated with it. You can read more about these options in the SQLAlchemy documentation.
Line 13: The single_parent=True parameter is required if delete-orphan is part of the previous cascade parameter. This tells SQLAlchemy not to allow orphaned Note instances (a Note without a parent Person object) to exist because each Note has a single parent.
Line 14: The order_by='desc(Note.timestamp)' parameter tells SQLAlchemy how to sort the Note instances associated with a Person. When a Person object is retrieved, by default the notes attribute list will contain Note objects in an unknown order. The SQLAlchemy desc(...) function will sort the notes in descending order from newest to oldest. If this line was instead order_by='Note.timestamp', SQLAlchemy would default to using the asc(...) function, and sort the notes in ascending order, oldest to newest.

Now that your Person model has the new notes attribute, and this represents the one-to-many relationship to Note objects, you’ll need to define a SQLAlchemy model for a Note:

 1 class Note(db.Model):
 2     __tablename__ = 'note'
 3     note_id = db.Column(db.Integer, primary_key=True)
 4     person_id = db.Column(db.Integer, db.ForeignKey('person.person_id'))
 5     content = db.Column(db.String, nullable=False)
 6     timestamp = db.Column(
 7         db.DateTime, default=datetime.utcnow, onupdate=datetime.utcnow
 8     )

The Note class defines the attributes making up a note as seen in our sample note database table from above. The attributes are defined here:

Line 1 creates the Note class, inheriting from db.Model, exactly as you did before when creating the Person class.
Line 2 tells the class what database table to use to store Note objects.
Line 3 creates the note_id attribute, defining it as an integer value, and as the primary key for the Note object.
Line 4 creates the person_id attribute, and defines it as the foreign key, relating the Note class to the Person class using the person.person_id primary key. This, and the Person.notes attribute, are how SQLAlchemy knows what to do when interacting with Person and Note objects.
Line 5 creates the content attribute, which contains the actual text of the note. The nullable=False parameter indicates that it’s okay to create new notes that have no content.
Line 6 creates the timestamp attribute, and exactly like the Person class, this contains the creation or update time for any particular Note instance.

Initialize the Database

Now that you’ve updated the Person and created the Note models, you’ll use them to rebuild the testing database people.db. You’ll do this by updating the build_database.py code from Part 2. Here’s what the code will look like:

 1 import os
 2 from datetime import datetime
 3 from config import db
 4 from models import Person, Note
 5 
 6 # Data to initialize database with
 7 PEOPLE = [
 8     {
 9         "fname": "Doug",
10         "lname": "Farrell",
11         "notes": [
12             ("Cool, a mini-blogging application!", "2019-01-06 22:17:54"),
13             ("This could be useful", "2019-01-08 22:17:54"),
14             ("Well, sort of useful", "2019-03-06 22:17:54"),
15         ],
16     },
17     {
18         "fname": "Kent",
19         "lname": "Brockman",
20         "notes": [
21             (
22                 "I'm going to make really profound observations",
23                 "2019-01-07 22:17:54",
24             ),
25             (
26                 "Maybe they'll be more obvious than I thought",
27                 "2019-02-06 22:17:54",
28             ),
29         ],
30     },
31     {
32         "fname": "Bunny",
33         "lname": "Easter",
34         "notes": [
35             ("Has anyone seen my Easter eggs?", "2019-01-07 22:47:54"),
36             ("I'm really late delivering these!", "2019-04-06 22:17:54"),
37         ],
38     },
39 ]
40 
41 # Delete database file if it exists currently
42 if os.path.exists("people.db"):
43     os.remove("people.db")
44 
45 # Create the database
46 db.create_all()
47 
48 # Iterate over the PEOPLE structure and populate the database
49 for person in PEOPLE:
50     p = Person(lname=person.get("lname"), fname=person.get("fname"))
51 
52     # Add the notes for the person
53     for note in person.get("notes"):
54         content, timestamp = note
55         p.notes.append(
56             Note(
57                 content=content,
58                 timestamp=datetime.strptime(timestamp, "%Y-%m-%d %H:%M:%S"),
59             )
60         )
61     db.session.add(p)
62 
63 db.session.commit()

The code above came from Part 2, with a few changes to create the one-to-many relationship between Person and Note. Here are the updated or new lines added to the code:

Line 4 has been updated to import the Note class defined previously.
Lines 7 to 39 contain the updated PEOPLE dictionary containing our person data, along with the list of notes associated with each person. This data will be inserted into the database.
Lines 49 to 61 iterate over the PEOPLE dictionary, getting each person in turn and using it to create a Person object.
Line 53 iterates over the person.notes list, getting each note in turn.
Line 54 unpacks the content and timestamp from each note tuple.
Line 55 to 60 creates a Note object and appends it to the person notes collection using p.notes.append().
Line 61 adds the Person object p to the database session.
Line 63 commits all of the activity in the session to the database. It’s at this point that all of the data is written to the person and note tables in the people.db database file.

You can see that working with the notes collection in the Person object instance p is just like working with any other list in Python. SQLAlchemy takes care of the underlying one-to-many relationship information when the db.session.commit() call is made.

For example, just like a Person instance has its primary key field person_id initialized by SQLAlchemy when it’s committed to the database, instances of Note will have their primary key fields initialized. In addition, the Note foreign key person_id will also be initialized with the primary key value of the Person instance it’s associated with.

Here’s an example instance of a Person object before the db.session.commit() in a kind of pseudocode:

Person (
    person_id = None
    lname = 'Farrell'
    fname = 'Doug'
    timestamp = None
    notes = [
        Note (
            note_id = None
            person_id = None
            content = 'Cool, a mini-blogging application!'
            timestamp = '2019-01-06 22:17:54'
        ),
        Note (
            note_id = None
            person_id = None
            content = 'This could be useful'
            timestamp = '2019-01-08 22:17:54'
        ),
        Note (
            note_id = None
            person_id = None
            content = 'Well, sort of useful'
            timestamp = '2019-03-06 22:17:54'
        )
    ]
)

Here’s the example Person object after the db.session.commit():

Person (
    person_id = 1
    lname = 'Farrell'
    fname = 'Doug'
    timestamp = '2019-02-02 21:27:10.336'
    notes = [
        Note (
            note_id = 1
            person_id = 1
            content = 'Cool, a mini-blogging application!'
            timestamp = '2019-01-06 22:17:54'
        ),
        Note (
            note_id = 2
            person_id = 1
            content = 'This could be useful'
            timestamp = '2019-01-08 22:17:54'
        ),
        Note (
            note_id = 3
            person_id = 1
            content = 'Well, sort of useful'
            timestamp = '2019-03-06 22:17:54'
        )
    ]
)

The important difference between the two is that the primary key of the Person and Note objects has been initialized. The database engine took care of this as the objects were created because of the auto-incrementing feature of primary keys discussed in Part 2.

Additionally, the person_id foreign key in all the Note instances has been initialized to reference its parent. This happens because of the order in which the Person and Note objects are created in the database.

SQLAlchemy is aware of the relationship between Person and Note objects. When a Person object is committed to the person database table, SQLAlchemy gets the person_id primary key value. That value is used to initialize the foreign key value of person_id in a Note object before it’s committed to the database.

SQLAlchemy takes care of this database housekeeping work because of the information you passed when the Person.notes attribute was initialized with the db.relationship(...) object.

In addition, the Person.timestamp attribute has been initialized with the current timestamp.

Running the build_database.py program from the command line (in the virtual environment will re-create the database with the new additions, getting it ready for use with the web application. This command line will rebuild the database:

$ python build_database.py

The build_database.py utility program doesn’t output any messages if it runs successfully. If it throws an exception, then an error will be printed on the screen.

Update REST API

You’ve updated the SQLAlchemy models and used them to update the people.db database. Now it’s time to update the REST API to provide access to the new notes information. Here’s the REST API you built in Part 2:

Action	HTTP Verb	URL Path	Description
Create	`POST`	`/api/people`	URL to create a new person
Read	`GET`	`/api/people`	URL to read a collection of people
Read	`GET`	`/api/people/{person_id}`	URL to read a single person by `person_id`
Update	`PUT`	`/api/people/{person_id}`	URL to update an existing person by `person_id`
Delete	`DELETE`	`/api/people/{person_id}`	URL to delete an existing person by `person_id`

The REST API above provides HTTP URL paths to collections of things, and to the things themselves. You can get a list of people or interact with a single person from that list of people. This path style refines what’s returned in a left-to-right manner, getting more granular as you go.

You’ll continue this left-to-right pattern to get more granular and access the notes collections. Here’s the extended REST API you’ll create in order to provide notes to the mini-blog web application:

Action	HTTP Verb	URL Path	Description
Create	`POST`	`/api/people/{person_id}/notes`	URL to create a new note
Read	`GET`	`/api/people/{person_id}/notes/{note_id}`	URL to read a single person’s single note
Update	`PUT`	`api/people/{person_id}/notes/{note_id}`	URL to update a single person’s single note
Delete	`DELETE`	`api/people/{person_id}/notes/{note_id}`	URL to delete a single person’s single note
Read	`GET`	`/api/notes`	URL to get all notes for all people sorted by `note.timestamp`

There are two variations in the notes part of the REST API compared to the convention used in the people section:

There is no URL defined to get all the notes associated with a person, only a URL to get a single note. This would have made the REST API complete in a way, but the web application you’ll create later doesn’t need this functionality. Therefore, it’s been left out.
There is the inclusion of the last URL /api/notes. This is a convenience method created for the web application. It will be used in the mini-blog on the home page to show all the notes in the system. There isn’t a way to get this information readily using the REST API pathing style as designed, so this shortcut has been added.

As in Part 2, the REST API is configured in the swagger.yml file.

Note:

The idea of designing a REST API with a path that gets more and more granular as you move from left to right is very useful. Thinking this way can help clarify the relationships between different parts of a database. Just be aware that there are realistic limits to how far down a hierarchical structure this kind of design should be taken.

For example, what if the Note object had a collection of its own, something like comments on the notes. Using the current design ideas, this would lead to a URL that went something like this: /api/people/{person_id}/notes/{note_id}/comments/{comment_id}

There is no practical limit to this kind of design, but there is one for usefulness. In actual use in real applications, a long, multilevel URL like that one is hardly ever needed. A more common pattern is to get a list of intervening objects (like notes) and then use a separate API entry point to get a single comment for an application use case.

Implement the API

With the updated REST API defined in the swagger.yml file, you’ll need to update the implementation provided by the Python modules. This means updating existing module files, like models.py and people.py, and creating a new module file called notes.py to implement support for Notes in the extended REST API.

Update Response JSON

The purpose of the REST API is to get useful JSON data out of the database. Now that you’ve updated the SQLAlchemy Person and created the Note models, you’ll need to update the Marshmallow schema models as well. As you may recall from Part 2, Marshmallow is the module that translates the SQLAlchemy objects into Python objects suitable for creating JSON strings.

The updated and newly created Marshmallow schemas are in the models.py module, which are explained below, and look like this:

 1 class PersonSchema(ma.ModelSchema):
 2     class Meta:
 3         model = Person
 4         sqla_session = db.session
 5     notes = fields.Nested('PersonNoteSchema', default=[], many=True)
 6 
 7 class PersonNoteSchema(ma.ModelSchema):
 8     """
 9     This class exists to get around a recursion issue
10     """
11     note_id = fields.Int()
12     person_id = fields.Int()
13     content = fields.Str()
14     timestamp = fields.Str()
15 
16 class NoteSchema(ma.ModelSchema):
17     class Meta:
18         model = Note
19         sqla_session = db.session
20     person = fields.Nested('NotePersonSchema', default=None)
21 
22 class NotePersonSchema(ma.ModelSchema):
23     """
24     This class exists to get around a recursion issue
25     """
26     person_id = fields.Int()
27     lname = fields.Str()
28     fname = fields.Str()
29     timestamp = fields.Str()

There are some interesting things going on in the above definitions. The PersonSchema class has one new entry: the notes attribute defined in line 5. This defines it as a nested relationship to the PersonNoteSchema. It will default to an empty list if nothing is present in the SQLAlchemy notes relationship. The many=True parameter indicates that this is a one-to-many relationship, so Marshmallow will serialize all the related notes.

The PersonNoteSchema class defines what a Note object looks like as Marshmallow serializes the notes list. The NoteSchema defines what a SQLAlchemy Note object looks like in terms of Marshmallow. Notice that it has a person attribute. This attribute comes from the SQLAlchemy db.relationship(...) definition parameter backref='person'. The person Marshmallow definition is nested, but because it doesn’t have the many=True parameter, there is only a single person connected.

The NotePersonSchema class defines what is nested in the NoteSchema.person attribute.

Note:

You might be wondering why the PersonSchema class has its own unique PersonNoteSchema class to define the notes collection attribute. By the same token, the NoteSchema class has its own unique NotePersonSchema class to define the person attribute. You may be wondering whether the PersonSchema class could be defined this way:

class PersonSchema(ma.ModelSchema):
    class Meta:
        model = Person
        sqla_session = db.session
    notes = fields.Nested('NoteSchema', default=[], many=True)

Additionally, couldn’t the NoteSchema class be defined using the PersonSchema to define the person attribute? A class definition like this would each refer to the other, and this causes a recursion error in Marshmallow as it will cycle from PersonSchema to NoteSchema until it runs out of stack space. Using the unique schema references breaks the recursion and allows this kind of nesting to work.

People

Now that you’ve got the schemas in place to work with the one-to-many relationship between Person and Note, you need to update the person.py and create the note.py modules in order to implement a working REST API.

The people.py module needs two changes. The first is to import the Note class, along with the Person class at the top of the module. Then only read_one(person_id) needs to change in order to handle the relationship. That function will look like this:

 1 def read_one(person_id):
 2     """
 3     This function responds to a request for /api/people/{person_id}
 4     with one matching person from people
 5 
 6     :param person_id:   Id of person to find
 7     :return:            person matching id
 8     """
 9     # Build the initial query
10     person = (
11         Person.query.filter(Person.person_id == person_id)
12         .outerjoin(Note)
13         .one_or_none()
14     )
15 
16     # Did we find a person?
17     if person is not None:
18 
19         # Serialize the data for the response
20         person_schema = PersonSchema()
21         data = person_schema.dump(person).data
22         return data
23 
24     # Otherwise, nope, didn't find that person
25     else:
26         abort(404, f"Person not found for Id: {person_id}")

The only difference is line 12: .outerjoin(Note). An outer join (left outer join in SQL terms) is necessary for the case where a user of the application has created a new person object, which has no notes related to it. The outer join ensures that the SQL query will return a person object, even if there are no note rows to join with.

At the start of this article, you saw how person and note data could be represented in a single, flat table, and all of the disadvantages of that approach. You also saw the advantages of breaking that data up into two tables, person and note, with a relationship between them.

Until now, we’ve been working with the data as two distinct, but related, items in the database. But now that you’re actually going to use the data, what we essentially want is for the data to be joined back together. This is what a database join does. It combines data from two tables together using the primary key to foreign key relationship.

A join is kind of a boolean and operation because it only returns data if there is data in both tables to combine. If, for example, a person row exists but has no related note row, then there is nothing to join, so nothing is returned. This isn’t what you want for read_one(person_id).

This is where the outer join comes in handy. It’s a kind of boolean or operation. It returns person data even if there is no associated note data to combine with. This is the behavior you want for read_one(person_id) to handle the case of a newly created Person object that has no notes yet.

You can see the complete people.py in the article repository.

Notes

You’ll create a notes.py module to implement all the Python code associated with the new note related REST API definitions. In many ways, it works like the people.py module, except it must handle both a person_id and a note_id as defined in the swagger.yml configuration file. As an example, here is read_one(person_id, note_id):

 1 def read_one(person_id, note_id):
 2     """
 3     This function responds to a request for
 4     /api/people/{person_id}/notes/{note_id}
 5     with one matching note for the associated person
 6 
 7     :param person_id:       Id of person the note is related to
 8     :param note_id:         Id of the note
 9     :return:                json string of note contents
10     """
11     # Query the database for the note
12     note = (
13         Note.query.join(Person, Person.person_id == Note.person_id)
14         .filter(Person.person_id == person_id)
15         .filter(Note.note_id == note_id)
16         .one_or_none()
17     )
18 
19     # Was a note found?
20     if note is not None:
21         note_schema = NoteSchema()
22         data = note_schema.dump(note).data
23         return data
24 
25     # Otherwise, nope, didn't find that note
26     else:
27         abort(404, f"Note not found for Id: {note_id}")

The interesting parts of the above code are lines 12 to 17:

Line 13 begins a query against the Note SQLAlchemy objects and joins to the related Person SQLAlchemy object comparing person_id from both Person and Note.
Line 14 filters the result down to the Note objects that has a Person.person_id equal to the passed in person_id parameter.
Line 15 filters the result further to the Note object that has a Note.note_id equal to the passed in note_id parameter.
Line 16 returns the Note object if found, or None if nothing matching the parameters is found.

You can check out the complete notes.py.

Updated Swagger UI

The Swagger UI has been updated by the action of updating the swagger.yml file and creating the URL endpoint implementations. Below is a screenshot of the updated UI showing the Notes section with the GET /api/people/{person_id}/notes/{note_id} expanded:

Mini-Blogging Web Application

The web application has been substantially changed to show its new purpose as a mini-blogging application. It has three pages:

The home page (localhost:5000/), which shows all of the blog messages (notes) sorted from newest to oldest
The people page (localhost:5000/people), which shows all the people in the system, sorted by last name, and also allows the user to create a new person and update or delete an existing one
The notes page (localhost:5000/people/{person_id}/notes), which shows all the notes associated with a person, sorted from newest to oldest, and also allows the user to create a new note and update or delete an existing one

There are two buttons on every page of the application:

The Home button will navigate to the home screen.
The People button navigates to the /people screen, showing all people in the database.

These two buttons are present on every screen in the application as a way to get back to a starting point.

Home Page

Below is a screenshot of the home page showing the initialized database contents:

The functionality of this page works like this:

Double-clicking on a person’s name will take the user to the /people/{person_id} page, with the editor section filled in with the person’s first and last names and the update and reset buttons enabled.
Double-clicking on a person’s note will take the user to the /people/{person_id}/notes/{note_id} page, with the editor section filled in with the note’s contents and the Update and Reset buttons enabled.

People Page

Below is a screenshot of the people page showing the people in the initialized database:

The functionality of this page works like this:

Single-clicking on a person’s name will populate the editor section of the page with the person’s first and last name, disabling the Create button, and enabling the Update and Delete buttons.
Double clicking on a person’s name will navigate to the notes pages for that person.

The functionality of the editor works like this:

If the first and last name fields are empty, the Create and Reset buttons are enabled. Entering a new name in the fields and clicking Create will create a new person and update the database and re-render the table below the editor. Clicking Reset will clear the editor fields.
If the first and last name fields have data, the user navigated here by double-clicking the person’s name from the home screen. In this case, the Update, Delete, and Reset buttons are enabled. Changing the first or last name and clicking Update will update the database and re-render the table below the editor. Clicking Delete will remove the person from the database and re-render the table.

Notes Page

Below is a screenshot of the notes page showing the notes for a person in the initialized database:

The functionality of this page works like this:

Single-clicking on a note will populate the editor section of the page with the notes content, disabling the Create button, and enabling the Update and Delete buttons.
All other functionality of this page is in the editor section.

The functionality of the editor works like this:

If the note content field is empty, then the Create and Reset buttons are enabled. Entering a new note in the field and clicking Create will create a new note and update the database and re-render the table below the editor. Clicking Reset will clear the editor fields.
If the note field has data, the user navigated here by double-clicking the person’s note from the home screen. In this case, the Update, Delete, and Reset buttons are enabled. Changing the note and clicking Update will update the database and re-render the table below the editor. Clicking Delete will remove the note from the database and re-render the table.

Web Application

This article is primarily focused on how to use SQLAlchemy to create relationships in the database, and how to extend the REST API to take advantage of those relationships. As such, the code for the web application didn’t get much attention. When you look at the web application code, keep an eye out for the following features:

Each page of the application is a fully formed single page web application.
Each page of the application is driven by JavaScript following an MVC (Model/View/Controller) style of responsibility delegation.
The HTML that creates the pages takes advantage of the Jinja2 inheritance functionality.
The hardcoded JavaScript table creation has been replaced by using the Handlebars.js templating engine.
The timestamp formating in all of the tables is provided by Moment.js.

You can find the following code in the repository for this article:

All of the example code for this article is available in the GitHub repository for this article. This contains all of the code related to this article, including all of the web application code.

Conclusion

Congratulations are in order for what you’ve learned in this article! Knowing how to build and use database relationships gives you a powerful tool to solve many difficult problems. There are other relationship besides the one-to-many example from this article. Other common ones are one-to-one, many-to-many, and many-to-one. All of them have a place in your toolbelt, and SQLAlchemy can help you tackle them all!

For more information about databases, you can check out these tutorials. You can also set up Flask to use SQLAlchemy. You can check out Model-View-Controller (MVC) more information about the pattern used in the web application JavaScript code.

In Part 4 of this series, you’ll focus on the HTML, CSS, and JavaScript files used to create the web application.

« Part 2: Database Persistence

Part 3: Database Relationships

Part 4: Coming Soon! »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Development in Visual Studio Code (Setup Guide)

2019-04-04T14:00:00+00:00

Learn how to set up Visual Studio Code for Python development. By following examples, you’ll cover everything from installing and configuring VSCode, to running tests and debugging Python apps. Make the most of this powerful Python IDE by following our complete setup guide.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

What Is Pip? A Guide for New Pythonistas

2019-04-03T14:00:00+00:00

What is pip? pip is the standard package manager for Python. It allows you to install and manage additional packages that are not part of the Python standard library. This tutorial is an introduction to pip for new Pythonistas.

In this tutorial, you’ll learn about:

Installing additional packages not included with the standard Python distribution
Finding packages published to the Python Package Index (PyPI)
Managing requirements for your scripts and applications
Uninstalling packages and their dependencies

As you’ll see, the Python community is very active and has created some neat alternatives to pip that you’ll learn about later in this tutorial.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Getting Started With `pip`

So, what is pip? pip is a package manager for Python. That means it’s a tool that allows you to install and manage additional libraries and dependencies that are not distributed as part of the standard library.

Package management is so important that pip has been included with the Python installer since versions 3.4 for Python 3 and 2.7.9 for Python 2, and it’s used by many Python projects, which makes it an essential tool for every Pythonista.

The concept of a package manager might be familiar to you if you are coming from other languages. JavaScript uses npm for package management, Ruby uses gem, and .NET use NuGet. In Python, pip has become the standard package manager.

The Python installer installs pip, so it should be ready for you to use, unless you installed an old version of Python. You can verify that pip is available by running the following command in your console:

$ pip --version

pip 18.1 from C:\Python37\lib\site-packages\pip (python 3.7)

You should see a similar output displaying the pip version, as well as the location and version of Python. If you are using an old version of Python that does not include pip, then you can install it by following the instructions for your system in the pip installation documentation.

You probably want to follow the examples in this tutorial inside a virtual environment to avoid installing packages to the global Python installation. You can learn about virtual environments in Python Virtual Environments: A Primer. The Using Virtual Environments section of that article explains the basics of creating new virtual environments.

Installing Packages With `pip`

Python is considered a batteries included language. This means that the Python standard library includes an extensive set of packages and modules to help developers with their scripts and applications.

At the same time, Python has a very active community that contributes an even bigger set of packages that can help you with your development needs. These packages are published to the Python Package Index, also known as PyPI (pronounced Pie Pea Eye). PyPI hosts an extensive collection of packages that include development frameworks, tools, and libraries.

Many of these packages simplify Python development by providing friendly interfaces to functionality that already exists in the standard library. For example, you can write a script that retrieves the contents of a web page using only the standard libraries included with Python:

# In using-http.py

import cgi
import http.client

server = 'www.google.com'
url = '/'
conn = http.client.HTTPSConnection(server)
conn.request('GET', url)
response = conn.getresponse()
content_type = response.headers.get('Content-Type')
_, params = cgi.parse_header(content_type)
encoding = params.get('charset')
data = response.read()
text = data.decode(encoding)

print(f'Response returned: {response.status} ({response.reason})')
print('Body:')
print(text)

In this script, you import cgi and http.client, both of which are included in the Python standard library. You create an HTTPSConnection object specifying the server and invoke its .request() and .getresponse() to retrieve a response.

From the response, you can retrieve the Content-Type header and parse it using the cgi module to extract the charset in which the page is encoded.

cgi.parse_header() returns a tuple with a main value and a dictionary of parameters. For example, the Content-Type header might contain a value like text/html; charset=ISO-8859-1.

The tuple will contain the string text/html as the first element, and the second element will be a dictionary in the form {'charset': 'ISO-8859-1'}. Because you only care about the charset parameter, you can ignore the beginning of the tuple using an underscore: _, params = cgi.parse_header(content_type).

Note: The Meaning of Underscores in Python explains how to use underscores to unpack values from a tuple.

Once you have the encoding of the page, you can read the response and decode it into text. You can run the example in the console to see how it works:

$ python using-http.py

Response returned: 200 (OK)
Body:
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" 
lang="en"><head><meta content="Search the world's information, including 
webpages, images, videos and more. Google has many special features to help you 
find exactly what you're looking for." name="description"><meta content="noodp" 
name="robots">... Additional Output Omitted

This seems like a lot of work for a small script that retrieves the contents of a web page. Fortunately, there is a Python package that simplifies HTTP requests and provides a nice interface to do exactly what you want.

Basic Package Installation

PyPI hosts a very popular library to perform HTTP requests called requests. You can learn all about it in its official documentation site.

The first step is to install the requests package into your environment. You can learn about pip supported commands by running it with help:

$ pip help

Usage:
  pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
  show                        Show information about installed packages.
  check                       Verify installed packages have compatible 
                              dependencies.
  config                      Manage local and global configuration.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring environment 
                              variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be 
                              used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be 
                              used up to 3 times (corresponding to WARNING, 
                              ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form 
                              [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should 
                              attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: 
                              (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort).
  --trusted-host <hostname>   Mark this host as trusted, even though it does 
                              not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file 
                              containing the private key and the certificate in 
                              PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine 
                              whether a new version of pip is available for 
                              download. Implied with --no-index.

As you can see, pip provides an install command to install packages. You can run it to install the requests package:

$ pip install requests

Looking in indexes: https://pypi.org/simple
Collecting requests
  Using cached 
  https://files.pythonhosted.org/packages/7d/e3/
  20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/
  requests-2.21.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/
  01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/
  chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/14/2c/
  cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/
  idna-2.8-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/62/00/
  ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/
  urllib3-1.24.1-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Using cached https://files.pythonhosted.org/packages/9f/e0/
  accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/
  certifi-2018.11.29-py2.py3-none-any.whl
Installing collected packages: chardet, idna, urllib3, certifi, requests
Successfully installed certifi-2018.11.29 chardet-3.0.4 idna-2.8 
  requests-2.21.0 urllib3-1.24.1
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' 
  command.

You should see an output similar to the one above. You use pip with an install command followed by the name of the package you want to install. pip looks for the package in PyPI, calculates its dependencies, and installs them to insure requests will work.

You can also see that the current environment is using pip version 18.1, but version 19.0.1 is available. It also shows the command you should use to update pip, so let’s do that:

$ python -m pip install --upgrade pip

Looking in indexes: https://pypi.org/simple
Collecting pip
  Downloading https://files.pythonhosted.org/packages/46/dc/
  7fd5df840efb3e56c8b4f768793a237ec4ee59891959d6a215d63f727023/
  pip-19.0.1-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 2.0MB/s
Installing collected packages: pip
  Found existing installation: pip 18.1
    Uninstalling pip-18.1:
      Successfully uninstalled pip-18.1
Successfully installed pip-19.0.1

Notice that you use python -m to update pip. The -m switch tells Python to run a module as an executable. This is necessary because in order for you to update pip, the old version has to be uninstalled before installing the new version, and removing it while running the tool can cause errors.

When you run pip as a module, Python loads the module in memory and allows the package to be removed while it is being used. You can run packages as if they were scripts if the package provides a top-level script __main__.py.

Now that you have installed requests and upgraded pip, you can use the list command to see the packages installed in your environment:

$ pip list

Package    Version
---------- ----------
certifi    2018.11.29
chardet    3.0.4
idna       2.8
pip        19.0.1
requests   2.21.0
setuptools 40.6.2
urllib3    1.24.1

As you can see, pip has been upgraded to version 19.0.1 (the latest version at the moment), and requests version 2.21.0 has been installed.

The pip install <package> command always looks for the latest version of the package and installs it. It also searches for dependencies listed in the package metadata and installs those dependencies to insure that the package has all the requirements it needs.

As you can see, multiple packages were installed. You can look at the package metadata by using the show command in pip:

$ pip show requests

Name: requests
Version: 2.21.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: me@kennethreitz.org
License: Apache 2.0
Location: py37\lib\site-packages
Requires: certifi, chardet, idna, urllib3
Required-by:

The metadata lists certifi, chardet, idna, and urllib3 as dependencies, and you can see they were also installed.

With the requests package installed, you can modify the example above and see how easy it is to retrieve the contents of a web page:

# In using-requests.py

import requests

url = 'https://www.google.com'
response = requests.get(url)
print(f'Response returned: {response.status_code}, {response.reason}')
print(response.text)

You can import the requests package as any other standard package because it is now installed in your environment.

As you can see, requests.get() handles the HTTP connection for you and returns a response object similar to the original example but with some interface improvements.

You don’t have to deal with the encoding of the page because requests will handle that for you in most situations. Still, requests provides a flexible interface to handle special cases through the requests.Response object.

Using Requirement Files

The pip install command always installs the latest published version of a package, but sometimes, you may want to install a specific version that you know works with your code.

You want to create a specification of the dependencies and versions you used to develop and test your application, so there are no surprises when you use the application in production.

Requirement files allow you to specify exactly which packages and versions should be installed. Running pip help shows that there is a freeze command that outputs the installed packages in requirements format. You can use this command, redirecting the output to a file to generate a requirements file:

$ pip freeze > requirements.txt
$ cat requirements.txt

certifi==2018.11.29
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1

The freeze command dumps all the packages and their versions to standard output, so you can redirect the output to a file that can be used to install the exact requirements into another system. The convention is to name this file requirements.txt, but you can give it any name you want.

When you want to replicate the environment in another system, you can run pip install specifying the requirements file using the -r switch:

$ pip install -r requirements.txt

Looking in indexes: https://pypi.org/simple
Collecting certifi==2018.11.29 (from -r .\requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/9f/e0/
  accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/
  certifi-2018.11.29-py2.py3-none-any.whl
Collecting chardet==3.0.4 (from -r .\requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/bc/a9/
  01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/
  chardet-3.0.4-py2.py3-none-any.whl
Collecting idna==2.8 (from -r .\requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/14/2c/
  cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/
  idna-2.8-py2.py3-none-any.whl
Collecting requests==2.21.0 (from -r .\requirements.txt (line 4))
  Using cached https://files.pythonhosted.org/packages/7d/e3/
  20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/
  requests-2.21.0-py2.py3-none-any.whl
Collecting urllib3==1.24.1 (from -r .\requirements.txt (line 5))
  Using cached https://files.pythonhosted.org/packages/62/00/
  ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/
  urllib3-1.24.1-py2.py3-none-any.whl
Installing collected packages: certifi, chardet, idna, urllib3, requests
Successfully installed certifi-2018.11.29 chardet-3.0.4 idna-2.8 
  requests-2.21.0 urllib3-1.24.1

The versions of the packages will match those listed in requirements.txt:

$ pip list

Package    Version
---------- ----------
certifi    2018.11.29
chardet    3.0.4
idna       2.8
pip        19.0.1
requests   2.21.0
setuptools 40.6.2
urllib3    1.24.1

You can submit the requirements.txt file into source control and use it to create the exact environment in other machines.

Fine-Tuning Requirements

The problem with hardcoding the versions of your packages and their dependencies is that packages are updated frequently with bug and security fixes, and you probably want to leverage those as soon as they are published.

The requirements file format allows you to specify dependency versions using logical operators that give you a bit of flexibility to insure packages are updated, but still define the base versions of a package.

Open the requirements.txt file in your favorite editor and make the following changes:

certifi>=2018.11.29
chardet>=3.0.4
idna>=2.8
requests>=2.21.0
urllib3>=1.24.1

You can change the logical operator to >= to tell pip to install an exact or greater version that has been published. When you set a new environment using the requirments.txt file, pip looks for the latest version that satisfies the requirement and installs it. You can upgrade the packages in your requirements file by running the install command with the --upgrade switch:

$ pip install --upgrade -r requirements.txt

Looking in indexes: https://pypi.org/simple
Requirement already up-to-date: certifi==2018.11.29 in \py37\lib\site-packages 
  (from -r .\requirements.txt (line 1)) (2018.11.29)
Requirement already up-to-date: chardet==3.0.4 in \py37\lib\site-packages 
  (from -r .\requirements.txt (line 2)) (3.0.4)
Requirement already up-to-date: idna==2.8 in \py37\lib\site-packages 
  (from -r .\requirements.txt (line 3)) (2.8)
Requirement already up-to-date: requests==2.21.0 in \py37\lib\site-packages 
  (from -r .\requirements.txt (line 4)) (2.21.0)
Requirement already up-to-date: urllib3==1.24.1 in \py37\lib\site-packages 
  (from -r .\requirements.txt (line 5)) (1.24.1)

Nothing was upgraded because you have the latest versions, but if a new version was published for a listed package, then the package would’ve been upgraded.

In an ideal world, new versions of packages would be backwards compatible and would never introduce new bugs. Unfortunately, new versions can introduce changes that will break your application. The requirements file syntax supports additional version specifiers to fine-tune your requirements.

Let’s say that a new version 3.0 of requests is published but introduces an incompatible change that breaks your application. You can modify the requirements file to prevent 3.0 or higher from being installed:

certifi>=2018.11.29
chardet>=3.0.4
idna>=2.8
requests>=2.21.0, <3.0
urllib3>=1.24.1

Changing the version specifier for the requests package ensures that any version greater or equal to 3.0 does not get installed. The pip documentation provides all the information about the requirements file format, and you can consult it to learn more about it.

Production vs Development Dependencies

Not all packages that you install during the development of your applications are going to be application dependencies. There are many packages published to PyPI that are development tools or libraries that you want to leverage during the development process.

As an example, you’ll probably want to unit test your application, so you need a unit test framework. A popular framework for unit testing is pytest. You want to install it in your development environment, but you do not want it in your production environment because it isn’t an application dependency.

You create a second requirements file (requirements_dev.txt) to list additional tools to set up a development environment:

# In requirements_dev.txt
pytest>=4.2.0

This requires you to use pip to install both requirement files: requirements.txt and requirements_dev.txt. Fortunately, pip allows you to specify additional parameters within a requirements file. You can modify requirements_dev.txt to also install the requirements from the production requirements.txt file:

# In requirements_dev.txt
-r requirements.txt
pytest>=4.2.0

Notice that you are using the exact same -r switch to install the production requirements.txt file. The requirements file format allows you to specify additional arguments right on a requirements file.

Freezing Requirements for Production

You created the production and development requirement files and added them to source control. The files use flexible version specifiers to ensure that you leverage bug fixes published by your dependencies. You are also testing your application and are ready to deploy it to production.

You probably want to ensure that the versions of the dependencies you deploy to production are the exact same versions you used in your integration pipeline or build process because you know all the tests pass and the application works.

The current version specifiers don’t guarantee that the same versions will be deployed to production, so you want to freeze the production requirements as you saw earlier.

You create a clean production virtual environment and install the production requirements using the requirements.txt file. Once the requirements are installed, you can freeze the specific versions, dumping the output to a requirements_lock.txt file that you use in production. The requirements_lock.txt file will contain exact versions specifiers and can be used to replicate the environment.

Finding Packages to Use

As you become a more experienced Pythonista, there’ll be a set of packages that you’ll know by heart and that you’ll use in most of your applications. The requests and pytest packages are good candidates to become useful tools in your Python toolbox.

There will be times though when you will need to solve a different problem, and you will want to look for a different tool or library that can help you with it. As you can see above, pip help shows that there is a search command that looks for packages published to PyPI.

Let’s see how this command can help us:

$ pip help search

Usage:
  pip search [options] <query>

Description:
  Search for PyPI packages whose name or summary contains <query>.

Search Options:
  -i, --index <url>           Base URL of Python Package Index 
                              (default https://pypi.org/pypi)

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring environment 
                              variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be 
                              used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be 
                              used up to 3 times (corresponding to WARNING, 
                              ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form 
                              [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should 
                              attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: 
                              (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort).
  --trusted-host <hostname>   Mark this host as trusted, even though it does 
                              not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file 
                              containing the private key and the certificate in 
                              PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine 
                              whether a new version of pip is available for 
                              download. Implied with --no-index.
  --no-color                  Suppress colored output

The command takes a set of options listed above and a <query>. The query is just a string to search for and will match packages and their descriptions.

Note: You can use pip help <command> to retrieve additional information about a supported command.

Let’s say your application needs to access a service that is using OAuth2 for authorization. Ideally, there is a library that works with requests or with a similar interface that can help us. Let’s search PyPI for it using pip:

$ pip search requests oauth

requests-oauth (0.4.1)             - Hook for adding Open Authentication 
                                     support to Python-requests HTTP library.
oauth (1.0.1)                      - Library for OAuth version 1.0a.
pmr2.oauth (0.6.1)                 - OAuth PAS Plugin, OAuth 1.0 provider for 
                                     Plone.
oauth-proxy (1.0.5)                - OAuth HTTP proxy
django-oauth (1.1)                 - Support of OAuth in Django.
intuit-oauth (1.2.0)               - Intuit OAuth Client
brubeck-oauth (0.1.11)             - Brubeck OAuth module
guillotina-oauth (2.0.0)           - guillotina oauth support
httpie-oauth (1.0.2)               - OAuth plugin for HTTPie.
paytm-oauth (0.2)                  - Consumer for paytm oauth
plurk-oauth (0.9.2)                - Plurk OAuth API
oauth-flow (1.0.3)                 - Authenticate and make calls to OAuth 1.0, 
                                     OAuth 2.0 services
... Additional Output Omitted

The search term yields quite an extensive collection of packages. Some of them seem specific to a service or technology like django-oauth. Others look promising, like requests-oauth. Unfortunately, there isn’t much information other than a brief description.

Most of the time, you want to search for packages directly in the PyPI website. PyPI provides search capabilities for its index and a way to filter results by the metadata exposed in the package, like framework, topic, development status, and so on.

A search for the same terms in PyPI yields a lot of results, but you can filter them by different categories. For example, you can expand the Intended Audience and select Developers since you want a library that helps you with developing your application. Also, you probably want a package that is stable and production-ready. You can expand the Development Status category and select Production/Stable:

You can apply additional filters and tweak the search terms until you find the package that you are looking for.

The results provide a link to the package page, which contains more information and hopefully some documentation. Let’s take a look at the information for requests-oauth2:

The project page provides more information, and it seems to have a link to the project homepage. The link takes you to the project repository on GitHub. There, you can see some more information about the project and some usage examples.

Finding the original source code repository can be an invaluable resource. There, you can find some hints about the status of the project by looking at the date of the latest commits, number of pull request and open issues, and so forth.

Another option to find a package is to Google it. Widely used Python libraries will show up at the top of google searches, and you should be able to find a link to the package in PyPI or its source code repository.

Finding the right package may take some time and research, but it will also speed up your development process once you find it.

Uninstalling Packages

Once in a while, you will have to uninstall a package. You either found a better library to replace it, or it is something you don’t really need. Uninstalling packages can be a bit tricky.

Notice that, when you installed requests, pip installed other dependencies too. The more packages you install, the bigger the chances that multiple packages depend on the same dependency. This is where the show command in pip comes in handy.

Before you uninstall a package, make sure you run the show command for that package:

$ pip show requests

Name: requests
Version: 2.21.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: me@kennethreitz.org
License: Apache 2.0
Location: c:\users\isaac\projects\virtualenv\py37\lib\site-packages
Requires: urllib3, certifi, chardet, idna
Required-by:

Notice the last two fields Requires and Required-by. The show command tells us that requests requires urllib3, certifi, chardet, and idna. You probably want to uninstall those two. You can also see that requests is not required by any other package, so it is safe to uninstall it.

You should run the show command against all of the requests dependencies to make sure that no other libraries also depend on them. Once you understand the dependency order of the packages you want to uninstall, you can remove them using the uninstall command:

$ pip uninstall certifi

Uninstalling certifi-2018.11.29:
  Would remove:
    \py37\lib\site-packages\certifi-2018.11.29.dist-info\*
    \py37\lib\site-packages\certifi\*
Proceed (y/n)? y
  Successfully uninstalled certifi-2018.11.29

Uninstalling a package shows you the files that will be removed and will ask for confirmation. If you are sure you want to remove the package because you’ve checked its dependencies and know that nothing else is using it, you can pass a -y switch to suppress the file list and confirmation:

$ pip uninstall urllib3 -y

Uninstalling urllib3-1.24.1:
  Successfully uninstalled urllib3-1.24.1

$ pip uninstall chardet -y

Uninstalling chardet-3.0.4:
  Successfully uninstalled chardet-3.0.4

$ pip uninstall idna -y

Uninstalling idna-2.8:
  Successfully uninstalled idna-2.8

$ pip uninstall requests -y

Uninstalling requests-2.21.0:
  Successfully uninstalled requests-2.21.0

You can specify all the packages you want to uninstall in a single call: pip uninstall -y urllib3 chardet idna requests.

You can also uninstall all the packages listed in a requirements file by providing the -r <requirments file> option. The command will ask confirmation for each individual package, but you can suppress it, if you know what you’re doing, with the -y switch:

$ pip uninstall -r requirements.txt -y

Uninstalling certifi-2018.11.29:
  Successfully uninstalled certifi-2018.11.29
Uninstalling chardet-3.0.4:
  Successfully uninstalled chardet-3.0.4
Uninstalling idna-2.8:
  Successfully uninstalled idna-2.8
Uninstalling requests-2.21.0:
  Successfully uninstalled requests-2.21.0
Uninstalling urllib3-1.24.1:
  Successfully uninstalled urllib3-1.24.1

Remember to always check the dependencies of packages you want to uninstall. You probably want to uninstall all its dependencies, but uninstalling a package that is being used by others will break your application.

Alternatives to `pip`

pip is an essential tool for all Pythonistas, and it is used by many applications and projects for package management. This tutorial has helped you with the basics, but the Python community is very active in providing great tools and libraries for other developers to use. These include other alternatives to pip that try to simplify and improve package management.

In this section, you’ll learn about other package management tools available for Python.

Conda Does It All

Conda is a package, dependency, and environment manager for many languages including Python. In fact, its origin comes from Anaconda, which started as a data science package for Python.

Conda is widely used for data science and machine learning applications, and uses its own index to host compatible packages.

Conda not only allows you to manage package dependencies, but it also manages virtual environments for your applications, installs compatible Python distributions, and packages your application for deployment to production.

Setting Up Python for Machine Learning on Windows is a great introduction to Conda that explores package and environment management. The only Windows-specific information is around installation, so it’s still relevant if you use a different OS platform.

Pipenv

Pipenv is another package management tool that “aims to bring the best of all packaging worlds” to Python. It’s gaining a lot of traction among the Python community because it merges virtual environment and package management in a single tool.

It also solves some of the most common hiccups you will run into when manually managing dependencies through pip, like versions of packages, separating development and production dependencies, and locking versions for production.

Pipenv: A Guide to the New Python Packaging Tool is a great start to learn about Pipenv and its approach to package management. Even though the article is tagged as intermediate, the author does a great job of guiding the reader that the article is accessible to anyone starting with Python.

Poetry

Poetry is another pip alternative that is gaining a lot of traction. Like Pipenv, it simplifies package version management and separates development vs production dependencies, and it works by isolating those dependencies into a virtual environment.

If you are coming from JavaScript and npm, then Poetry will look very familiar. It goes beyond package management, helping you build distributions for your applications and libraries and deploying them to PyPI. How to Publish an Open-Source Python Package to PyPI has a good introduction to Poetry and can help you get started.

Conclusion: What Is `pip`?

This tutorial answered the question, what is pip? You’ve seen that pip is a package manager for Python, used by many projects to manage dependencies. It’s included with the Python installer, which makes it an essential tool for all Pythonistas to know how to use.

Python provides an extensive standard library suitable for developing all sorts of applications, but the active Python community provides an even larger set of tools and libraries that speed up Python application development.

These tools and libraries are published to the Python Package Index (PyPI), and pip allows developers to install them in their application environments.

In this tutorial, you’ve learned about:

Installing new packages using pip in the command line and with requirement files
Managing dependencies, separating development and production requirements, and creating a locked requirements file
Finding packages through pip and PyPI
Evaluating package dependencies before uninstalling a package and how pip uninstalls packages

In addition, you’ve learned about the importance of keeping dependencies up to date and alternatives to pip that can help you manage those dependencies.

Feel free to reach out in the comments section below with any questions you might have, and you can always get more information at the pip documentation page.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Idiomatic Pandas: Tricks & Features You May Not Know

2019-04-02T14:00:00+00:00

Pandas is a foundational library for analytics, data processing, and data science. It’s a huge project with tons of optionality and depth.

In this course you’ll see how to use some lesser-used but idiomatic Pandas capabilities that lend your code better readability, versatility, and speed.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Get Started With Django Part 1: Build a Portfolio App

2019-04-01T14:00:00+00:00

Django is a fully featured Python web framework that can be used to build complex web applications. In this tutorial, you’ll jump in and learn Django by example. You’ll follow the steps to create a fully functioning web application and, along the way, learn some of the most important features of the framework and how they work together.

In later posts in this series, you’ll see how to build more complex websites using even more of Django’s features than you’ll cover in this tutorial.

By the end of this tutorial, you will be able to:

Understand what Django is and why it’s a great web framework
Understand the architecture of a Django site and how it compares with other frameworks
Set up a new Django project and app
Build a Personal Portfolio Website with Django

Free Bonus: Click here to get access to a free Django Learning Resources Guide (PDF) that shows you tips and tricks as well as common pitfalls to avoid when building Python + Django web applications.

Why You Should Learn Django

There are endless web development frameworks out there, so why should you learn Django over any of the others? First of all, it’s written in Python, one of the most readable and beginner-friendly programming languages out there.

Note: This tutorial assumes an intermediate knowledge of the Python language. If you’re new to programming with Python, check out some of our beginner tutorials or the introductory course.

The second reason you should learn Django is the scope of its features. If you need to build a website, you don’t need to rely on any external libraries or packages if you choose Django. This means that you don’t need to learn how to use anything else, and the syntax is seamless as you’re using only one framework.

There’s also the added benefit that you don’t need to worry that updating one library or framework will render others that you’ve installed useless.

If you do find yourself needing to add extra features, there are a range of external libraries that you can use to enhance your site.

One of the great things about the Django framework is its in-depth documentation. It has detailed documentation on every aspect of Django and also has great examples and even a tutorial to get you started.

There’s also a fantastic community of Django developers, so if you get stuck there’s almost always a way forward by either checking the docs or asking the community.

Django is a high-level web application framework with loads of features. It’s great for anyone new to web development due to its fantastic documentation, and particularly if you’re also familiar with Python.

The Structure of a Django Website

A Django website consists of a single project that is split into separate apps. The idea is that each app handles a self-contained function that the site needs to perform. As an example, imagine an application like Instagram. There are several different functions that need to be performed:

User management: Login, logout, register, and so on
The image feed: Uploading, editing, and displaying images
Private messaging: Private messages between users and notifications

These are each separate pieces of functionality, so if this were a Django site, then each piece of functionality should be a different Django app inside a single Django project.

The Django project holds some configurations that apply to the project as a whole, such as project settings, URLs, shared templates and static files. Each application can have its own database and has its own functions to control how the data is displayed to the user in HTML templates.

Each application also has its own URLs as well as its own HTML templates and static files, such as JavaScript and CSS.

Django apps are structured so that there is a separation of logic. It supports the Model-View-Controller Pattern, which is the architecture on which most web frameworks are built. The basic principle is that in each application there are three separate files that handle the three main pieces of logic separately:

Model defines the data structure. This is usually a database and is the base layer to an application.
View displays some or all of the data to the user with HTML and CSS.
Controller handles how the database and the view interact.

If you want to learn more about the MVC pattern, then check out Model-View-Controller (MVC) Explained – With Legos.

In Django, the architecture is slightly different. Although based upon the MVC pattern, Django handles the controller part itself. There’s no need to define how the database and views interact. It’s all done for you!

The pattern Django utilizes is called the Model-View-Tempalate (MVT) pattern. The view and template in the MVT pattern make up the view in the MVC pattern. All you need to do is add some URL configurations to map the views to, and Django handles the rest!

A Django site starts off as a project and is built up with a number of applications that each handle separate functionality. Each app follows the Model-View-Template pattern. Now that you’re familiar with the structure of a Django site, let’s have a look at what you’re going to build!

What You’re Going to Build

Before you get started with any web development project, it’s a good idea to come up with a plan of what you’re going to build. In this tutorial, we are going to build an application with the following features:

A fully functioning blog: If you’re looking to demonstrate your coding ability, a blog is a great way to do that. In this application, you will be able to create, update, and delete blog posts. Posts will have categories that can be used to sort them. Finally, users will be able to leave comments on posts.
A portfolio of your work: You can showcase previous web development projects here. You’ll build a gallery style page with clickable links to projects that you’ve completed.

Note: Before you get started, you can pull down the source code and follow along with the tutorial.

If you prefer to follow along by writing the code yourself, don’t worry. I’ve referenced the relevant parts of the source code throughout so you can refer back to it.

We won’t be using any external Python libraries in this tutorial. One of the great things about Django is that it has so many features that you don’t need to rely on external libraries. However, we will add Bootstrap 4 styling in the templates.

By building these two apps, you’ll learn the basics of Django models, view functions, forms, templates, and the Django admin page. With knowledge of these features, you’ll be able to go away and build loads more applications. You’ll also have the tools to learn even more and build sophisticated Django sites.

Hello, World!

Now that you know the structure of a Django application, and what you are about to build, we’re going to go through the process of creating an application in Django. You’ll extend this later into your personal portfolio application.

Set Up Your Development Environment

Whenever you are starting a new web development project, it’s a good idea to first set up your development environment. Create a new directory for your project to live in, and cd into it:

$ mkdir rp-portfolio
$ cd rp-portfolio

Once your inside the main directory, it’s a good idea to create a virtual environment to manage dependencies. There are many different ways to set up virtual environments, but here you’re going to use venv:

$ python3 -m venv venv

This command will create a folder venv in your working directory. Inside this directory, you’ll find several files including a copy of the Python standard library. Later, when you install new dependencies, they will also be stored in this directory. Next, you need to activate the virtual environment by running the following command:

$ source venv/bin/activate

Note: If you’re not using bash shell, you might need to use a different command to activate your virtual environment. For example, on windows you need this command:

C:\> venv\Scripts\activate.bat

You’ll know that your virtual environment has been activated, because your console prompt in the terminal will change. It should look something like this:

(venv) $

Note: Your virtual environment directory doesn’t have to be called venv. If you want to create one under a different name, for example my_venv, just replace with the second venv with my_venv.

Then, when activating your virtual environment, replace venv with my_venv again. The prompt will also now be prefixed with (my_venv).

Now that you’ve created a virtual environment, it’s time to install Django. You can do this using pip:

(venv) $ pip install Django

Once you’ve set up the virtual environment and installed Django, you can now dive in to creating the application.

Create a Django Project

As you saw in the previous section, a Django web application is made up of a project and its constituent apps. Making sure you’re in the rp_portfolio directory, and you’ve activated your virtual environment, run the following command to create the project:

$ django-admin startproject personal_portfolio

This will create a new directory personal_portfolio. If you cd into this new directory you’ll see another directory called personal_portfolio and a file called manage.py. Your directory structure should look something like this:

rp-portfolio/
│
├── personal_portfolio/
│   ├── personal_portfolio/
│   │   ├── __init__.py
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   │
│   └── manage.py
│
└── venv/

Most of the work you do will be in that first personal_portfolio directory. To save having to cd through several directories each time you come to work on your project, it can be helpful to reorder this slightly by moving all the files up a directory. While you’re in the rp-portfolio directory, run the following commands:

$ mv personal_portfolio/manage.py ./
$ mv personal_portfolio/personal_portfolio/* personal_portfolio
$ rm -r personal_portfolio/personal_portfolio/

You should end up with something like this:

rp-portfolio/
│
├── personal_portfolio/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
│
├── venv/
│
└── manage.py

Once your file structure is set up, you can now start the server and check that your set up was successful. In the console, run the following command:

$ python manage.py runserver

Then, in your browser go to localhost:8000, and you should see the following:

Congratulations, you’ve created a Django site! The source code for this part of the tutorial can be found on GitHub. The next step is to create apps so that you can add views and functionality to your site.

Create a Django Application

For this part of the tutorial, we’ll create an app called hello_world, which you’ll subsequently delete as its not necessary for our personal portfolio site.

To create the app, run the following command:

$ python manage.py startapp hello_world

This will create another directory called hello_world with several files:

__init__.py tells Python to treat the directory as a Python package.
admin.py contains settings for the Django admin pages.
apps.py contains settings for the application configuration.
models.py contains a series of classes that Django’s ORM converts to database tables.
tests.py contains test classes.
views.py contains functions and classes that handle what data is displayed in the HTML templates.

Once you’ve created the app, you need to install it in your project. In rp-portfolio/settings.py, add the following line of code under INSTALLED_APPS:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'hello_world',
]

That line of code means that your project now knows that the app you just created exists. The next step is to create a view so that you can display something to a user.

Create a View

Views in Django are a collection of functions or classes inside the views.py file in your app directory. Each function or class handles the logic that gets processed each time a different URL is visited.

Navigate to the views.py file in the hello_world directory. There’s already a line of code in there that imports render(). Add the following code:

from django.shortcuts import render

def hello_world(request):
    return render(request, 'hello_world.html', {})

In this piece of code, you’ve defined a view function called hello_world(). When this function is called, it will render an HTML file called hello_world.html. That file doesn’t exist yet, but we’ll create it soon.

The view function takes one argument, request. This object is an HttpRequestObject that is created whenever a page is loaded. It contains information about the request, such as the method, which can take several values including GET and POST.

Now that you’ve created the view function, you need to create the HTML template to display to the user. render() looks for HTML templates inside a directory called templates inside your app directory. Create that directory and subsequently a file named hello_world.html inside it:

$ mkdir hello_world/templates/
$ touch hello_world/templates/hello_world.html

Add the following lines of HTML to your file:

<h1>Hello, World!</h1>

You’ve now created a function to handle your views and templates to display to the user. The final step is to hook up your URLs so that you can visit the page you’ve just created. Your project has a module called urls.py in which you need to include a URL configuration for the hello_world app. Inside personal_portfolio/urls.py, add the following:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('hello_world.urls')),
]

This looks for a module called urls.py inside the hello_world application and registers any URLs defined there. Whenever you visit the root path of your URL (localhost:8000), the hello_world application’s URLs will be registered. The hello_world.urls module doesn’t exist yet, so you’ll need to create it:

$ touch hello_world/urls.py

Inside this module, we need to import the path object as well as our app’s views module. Then we want to create a list of URL patterns that correspond to the various view functions. At the moment, we have only created one view function, so we need only create one URL:

from django.urls import path
from hello_world import views

urlpatterns = [
    path('', views.hello_world, name='hello_world'),
]

Now, when you restart the server and visit localhost:8000, you should be able to see the HTML template you created:

Congratulations, again! You’ve created your first Django app and hooked it up to your project. Don’t forget to check out the source code for this section and the previous one. The only problem now is that it doesn’t look very nice. In the next section, we’re going to add bootstrap styles to your project to make it prettier!

Add Bootstrap to Your App

If you don’t add any styling, then the app you create isn’t going to look too nice. Instead of going into CSS styling with this tutorial, we’ll just cover how to add bootstrap styles to your project. This will allow us to improve the look of the site without too much effort.

Before we get started with the Bootstrap styles, we’ll create a base template that we can import to each subsequent view. This template is where we’ll subsequently add the Bootstrap style imports.

Create another directory called templates, this time inside personal_portfolio, and a file called base.html, inside the new directory:

$ mkdir personal_portfolio/templates/
$ touch personal_portfolio/templates/base.html

We create this additional templates directory to store HTML templates that will be used in every Django app in the project. As you saw previously, each Django project can consist of multiple apps that handle separated logic, and each app contains its own templates directory to store HTML templates related to the application.

This application structure works well for the back end logic, but we want our entire site to look consistent on the front end. Instead of having to import Bootstrap styles into every app, we can create a template or set of templates that are shared by all the apps. As long as Django knows to look for templates in this new, shared directory it can save a lot of repeated styles.

Inside this new file (personal_portfolio/templates/base.html), add the following lines of code:

{% block page_content %}{% endblock %}

Now, in hello_world/templates/hello_world.html, we can extend this base template:

{% extends "base.html" %}

{% block page_content %}
<h1>Hello, World!</h1>
{% endblock %}

What happens here is that any HTML inside the page_content block gets added inside the same block in base.html.

To install Bootstrap in your app, you’ll use the Bootstrap CDN. This is a really simple way to install Bootstrap that just involves adding a few lines of code to base.html. Check out the source code to see how to add the CDN links to your project.

All future templates that we create will extend base.html so that we can include Bootstrap styling on every page without having to import the styles again.

Before we can see our new styled application, we need to tell our Django project that base.html exists. The default settings register template directories in each app, but not in the project directory itself. In personal_portfolio/settings.py, update TEMPLATES:

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": ["personal_portfolio/templates/"],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
            ]
        },
    }
]

Now, when you visit localhost:8000, you should see that the page has been formatted with slightly different styling:

Whenever you want create templates or import scripts that you intend to use in all your Django apps inside a project, you can add them to this project-level directory and extend them inside your app templates.

Adding templates is the last stage to building your Hello, World! Django site. You learned how the Django templating engine works and how to create project-level templates that can be shared by all the apps inside your Django project.

In this section, you learned how to create a simple Hello, World! Django site by creating a project with a single app. In the next section, you’ll create another application to showcase web development projects, and you’ll learn all about models in Django!

The source code for this section can be found on GitHub.

Showcase Your Projects

Any web developer looking to create a portfolio needs a way to show off projects they have worked on. That’s what you’ll be building now. You’ll create another Django app called projects that will hold a series of sample projects that will be displayed to the user. Users can click on projects and see more information about your work.

Before we build the projects app, let’s first delete the hello_world application. All you need to do is delete the hello_world directory and remove the line "hello_world", from INSTALLED_APPS in settings.py:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'hello_world',  # Delete this line
]

Finally, you need to remove the URL path created in personal_portfolio/urls.py:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('hello_world.urls')),  # Delete this line
]

Now that you’ve removed the hello_world app, we can create the projects app. Making sure you’re in the rp-portfolio directory, run the following command in your console:

$ python manage.py startapp projects

This will create a directory named projects. The files created are the same as those created when we set up the hello_world application. In order to hook up our app, we need to add it into INSTALLED_APPS in settings.py:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'projects',
]

Check out the source code for this section on GitHub. We’re not going to worry about URLs for this application just yet. Instead, we’re going to focus on building a Project model.

Projects App: Models

If you want to store data to display on a website, then you’ll need a database. Typically, if you want to create a database with tables and columns within those tables, you’ll need to use SQL to manage the database. But when you use Django, you don’t need to learn a new language because it has a built-in Object Relational Mapper (ORM).

An ORM is a program that allows you to create classes that correspond to database tables. Class attributes correspond to columns, and instances of the classes correspond to rows in the database. So, instead of learning a whole new language to create our database and its tables, we can just write some Python classes.

When you’re using an ORM, the classes you build that represent database tables are referred to as models. In Django, they live in the models.py module of each Django app.

In your projects app, you’ll only need one table to store the different projects you’ll display to the user. That means you’ll only need to create one model in models.py.

The model you’ll create will be called Project and will have the following fields:

title will be a short string field to hold the name of your project.
description will be a larger string field to hold a longer piece of text.
technology will be a string field, but its contents will be limited to a select number of choices.
image will be an image field that holds the file path where the image is stored.

To create this model, we’ll create a new class in models.py and add the following in our fields:

from django.db import models

class Project(models.Model):
    title = models.CharField(max_length=100)
    description = models.TextField()
    technology = models.CharField(max_length=20)
    image = models.FilePathField(path="/img")

Django models come with many built-in model field types. We’ve only used three in this model. CharField is used for short strings and specifies a maximum length.

TextField is similar to CharField but can be used for longer form text as it doesn’t have a maximum length limit. Finally, FilePathField also holds a string but must point to a file path name.

Now that we’ve created our Project class, we need Django to create the database. By default, the Django ORM creates databases in SQLite, but you can use other databases that use the SQL language, such as PostgreSQL or MySQL, with the Django ORM.

To start the process of creating our database, we need to create a migration. A migration is a file containing a Migration class with rules that tell Django what changes need to be made to the database. To create the migration, type the following command in the console, making sure you’re in the rp-portfolio directory:

$ python manage.py makemigrations projects
Migrations for 'projects':
  projects/migrations/0001_initial.py
    - Create model Project

You should see that a file projects/migrations/0001_initial.py has been created in the projects app. Check out that file in the source code to make sure your migration is correct.

Now that you’ve create a migration file, you need to apply the migrations set out in the migrations file and create your database using the migrate command:

$ python manage.py migrate projects
Operations to perform:
  Apply all migrations: projects
Running migrations:
  Applying projects.0001_initial... OK

Note: When running both the makemigrations and migrate commands, we added projects to our command. This tells Django to only look at models and migrations in the projects app. Django comes with several models already created.

If you run makemigrations and migrate without the projects flag, then all migrations for all the default models in your Django projects will be created and applied. This is not a problem, but for the purposes of this section, they are not needed.

You should also see that a file called db.sqlite3 has been created in the root of your project. Now your database is set up and ready to go. You can now create rows in your table that are the various projects you want to show on your portfolio site.

To create instances of our Project class, we’re going to have to use the Django shell. The Django shell is similar to the Python shell but allows you to access the database and create entries. To access the Django shell, we use another Django management command:

$ python manage.py shell

Once you’ve accessed the shell, you’ll notice that the command prompt will change from $ to >>>. You can then import your models:

>>>

>>> from projects.models import Project

We’re first going to create a new project with the following attributes:

name: My First Project
description: A web development project.
technology: Django
image: img/project1.png

To do this, we create an instance of the Project class in the Django shell:

>>>

>>> p1 = Project(
...     title='My First Project',
...     description='A web development project.',
...     technology='Django',
...     image='img/project1.png'
... )
>>> p1.save()

This creates a new entry in your projects table and saves it to the database. Now you have created a project that you can display on your portfolio site.

The final step in this section is to create two more sample projects:

>>>

>>> p2 = Project(
...     title='My Second Project',
...     description='Another web development project.',
...     technology='Flask',
...     image='img/project2.png'
... )
>>> p2.save()
>>> p3 = Project(
...     title='My Third Project',
...     description='A final development project.',
...     technology='Django',
...     image='img/project3.png'
... )
>>> p3.save()

Well done for reaching the end of this section! You now know how to create models in Django and build migration files so that you can translate these model classes into database tables. You’ve also used the Django shell to create three instances of your model class.

In the next section, we’ll take these three projects you created and create a view function to display them to users on a web page. You can find the source code for this section of the tutorial on GitHub.

Projects App: Views

Now you’ve created the projects to display on your portfolio site, you’ll need to create view functions to send the data from the database to the HTML templates.

In the projects app, you’ll create two different views:

An index view that shows a snippet of information about each project
A detail view that shows more information on a particular topic

Let’s start with the index view, as the logic is slightly simpler. Inside views.py, you’ll need to import the Project class from models.py and create a function project_index() that renders a template called project_index.html. In the body of this function, you’ll make a Django ORM query to select all objects in the Project table:

 1 from django.shortcuts import render
 2 from projects.models import Project
 3 
 4 def project_index(request):
 5     projects = Project.objects.all()
 6     context = {
 7         'projects': projects
 8     }
 9     return render(request, 'project_index.html', context)

There’s quite a lot going on in this code block, so let’s break it down.

In line 6, you perform a query. A query is simply a command that allows you to create, retrieve, update, or delete objects (or rows) in your database. In this case, you’re retrieving all objects in the projects table.

A database query returns a collection of all objects that match the query, known as a Queryset. In this case, you want all objects in the table, so it will return a collection of all projects.

In line 7 of the code block above, we define a dictionary context. The dictionary only has one entry projects to which we assign our Queryset containing all projects. The context dictionary is used to send information to our template. Every view function you create needs to have a context dictionary.

In line 10, context is added as an argument to render(). Any entries in the context dictionary are available in the template, as long as the context argument is passed to render(). You’ll need to create a context dictionary and pass it to render in each view function you create.

We also render a template named project_index.html, which doesn’t exist yet. Don’t worry about that for now. You’ll create the templates for these views in the next section.

Next, you’ll need to create the project_detail() view function. This function will need an additional argument: the id of the project that’s being viewed.

Otherwise, the logic is similar:

13 def project_detail(request, pk):
14     project = Project.objects.get(pk=pk)
15     context = {
16         'project': project
17     }
18     return render(request, 'project_detail.html', context)

In line 14, we perform another query. This query retrieves the project with primary key, pk, equal to that in the function argument. We then assign that project in our context dictionary, which we pass to render(). Again, there’s a template project_detail.html, which we have yet to create.

Once your view functions are created, we need to hook them up to URLs. We’ll start by creating a file projects/urls.py to hold the URL configuration for the app. This file should contain the following code:

 1 from django.urls import path
 2 from . import views
 3 
 4 urlpatterns = [
 5     path("", views.project_index, name="project_index"),
 6     path("<int:pk>/", views.project_detail, name="project_detail"),
 7 ]

In line 5, we hook up the root URL of our app to the project_index view. It is slightly more complicated to hook up the project_detail view. To do this, we want the URL to be /1, or /2, and so on, depending on the pk of the project.

The pk value in the URL is the same pk passed to the view function, so you need to dynamically generate these URLs depending on which project you want to view. To do this, we used the <int:pk> notation. This just tells Django that the value passed in the URL is an integer, and its variable name is pk.

With those now set up, we need to hook these URLs up to the project URLs. In personal_portfolio/urls.py, add the following highlighted line of code:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path("admin/", admin.site.urls),
    path("projects/", include("projects.urls")),
]

This line of code includes all the URLs in the projects app but means they are accessed when prefixed by projects/. There are now two full URLs that can be accessed with our project:

localhost:8000/projects: The project index page
localhost:8000/projects/3: The detail view for the project with pk=3

These URLs still won’t work properly because we don’t have any HTML templates. But our views and logic are up and running so all that’s left to do is create those templates. If you want to check your code, take a look at the source code for this section.

Projects App: Templates

Phew! You’re nearly there with this app. Our final step is to create two templates:

The project_index template
The project_detail template

As we’ve added Bootstrap styles to our application, we can use some pre-styled components to make the views look nice. Let’s start with the project_index template.

For the project_index template, you’ll create a grid of Bootstrap cards, with each card displaying details of the project. Of course, we don’t know how many projects there are going to be. In theory, there could be hundreds to display.

We don’t want to have to create 100 different Bootstrap cards and hard-code in all the information to each project. Instead, we’re going to use a feature of the Django template engine: for loops.

Using this feature, you’ll be able to loop through all the projects and create a card for each one. The for loop syntax in the Django template engine is as follows:

{% for project in projects %}
{# Do something... #}
{% endfor %}

Now that you know how for loops work, you can add the following code to a file named projects/templates/project_index.html:

 1 {% extends "base.html" %}
 2 {% load static %}
 3 {% block page_content %}
 4 <h1>Projects</h1>
 5 <div class="row">
 6 {% for project in projects %}
 7     <div class="col-md-4">
 8         <div class="card mb-2">
 9             <img class="card-img-top" src="{% static project.image %}">
10             <div class="card-body">
11                 <h5 class="card-title">{{ project.title }}</h5>
12                 <p class="card-text">{{ project.description }}</p>
13                 <a href="{% url 'project_detail' project.pk %}"
14                    class="btn btn-primary">
15                     Read More
16                 </a>
17             </div>
18         </div>
19     </div>
20     {% endfor %}
21 </div>
22 {% endblock %}

There’s a lot of Bootstrap HTML here, which is not the focus of this tutorial. Feel free to copy and paste and take a look at the Bootstrap docs if you’re interested in learning more. Instead of focusing on the Bootstrap, there are a few things to highlight in this code block.

In line 1, we extend base.html as we did in the Hello, World! app tutorial. I’ve added some more styling to this file to include a navigation bar and so that all the content is contained in a Bootstrap container. The changes to base.html can be seen in the source code on GitHub.

On line 2, we include a {% load static %} tag to include static files such as images. Remember back in the section on Django models, when you created the Project model. One of its attributes was a filepath. That filepath is where we’re going to store the actual images for each project.

Django automatically registers static files stored in a directory named static/ in each application. Our image file path names were of the structure: img/<photo_name>.png.

When loading static files, Django looks in the static/ directory for files matching a given filepath within static/. So, we need to create a directory named static/ with another directory named img/ inside. Inside img/, you can copy over the images from the source code on GitHub.

On line 6, we begin the for loop, looping over all projects passed in by the context dictionary.

Inside this for loop, we can access each individual project. To access the project’s attributes, you can use dot notation inside double curly brackets. For example, to access the project’s title, you use {{ project.title }}. The same notation can be used to access any of the project’s attributes.

On line 9, we include our project image. Inside the src attribute, we add the code {% static project.image %}. This tells Django to look inside the static files to find a file matching project.image.

The final point that we need to highlight is the link on line 13. This is the link to our project_detail page. Accessing URLs in Django is similar to accessing static files. The code for the URL has the following form:

{% url '<url path name>' <view_function_arguments> %}

In this case, we are accessing a URL path named project_detail, which takes integer arguments corresponding to the pk number of the project.

With all that in place, if you start the Django server and visit localhost:8000/projects, then you should see something like this:

With the project_index.html template in place, it’s time to create the project_detail.html template. The code for this template is below:

{% extends "base.html" %}
{% load static %}

{% block page_content %}
<h1>{{ project.title }}</h1>
<div class="row">
    <div class="col-md-8">
        <img src="{% static project.image %}" alt="" width="100%">
    </div>
    <div class="col-md-4">
        <h5>About the project:</h5>
        <p>{{ project.description }}</p>
        <br>
        <h5>Technology used:</h5>
        <p>{{ project.technology }}</p>
    </div>
</div>
{% endblock %}

The code in this template has the same functionality as each project card in the project_index.html template. The only difference is the introduction of some Bootstrap columns.

If you visit localhost:8000/projects/1, you should see the detail page for that first project you created:

In this section, you learned how to use models, views, and templates to create a fully functioning app for your personal portfolio project. Check out the source code for this section on GitHub.

In the next section, you’ll build a fully functioning blog for your site, and you’ll also learn about the Django admin page and forms.

A blog is a great addition to any personal portfolio site. Whether you update it monthly or weekly, it’s a great place to share your knowledge as you learn. In this section, you’re going to build a fully functioning blog that will allow you to perform the following tasks:

Create, update, and delete blog posts
Display posts to the user as either an index view or a detail view
Assign categories to posts
Allow users to comment on posts

You’ll also learn how to use the Django Admin interface, which is where you’ll create, update, and delete posts and categories as necessary.

Before you get into building out the functionality of this part of your site, create a new Django app named blog. Don’t delete projects. You’ll want both apps in your Django project:

$ python manage.py startapp blog

This may start to feel familiar to you, as its your third time doing this. Don’t forget to add blog to your INSTALLED_APPS in personal_porfolio/settings.py:

INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "projects",
    "blog",
]

Hold off on hooking up the URLs for now. As with the projects app, you’ll start by adding your models.

Blog App: Models

The models.py file in this app is much more complicated than in the projects app.

You’re going to need three separate database tables for the blog:

Post
Category
Comment

These tables need to be related to one another. This is made easier because Django models come with fields specifically for this purpose.

Below is the code for the Category and Post models:

 1 from django.db import models
 2 
 3 class Category(models.Model):
 4     name = models.CharField(max_length=20)
 5 
 6 class Post(models.Model):
 7     title = models.CharField(max_length=255)
 8     body = models.TextField()
 9     created_on = models.DateTimeField(auto_now_add=True)
10     last_modified = models.DateTimeField(auto_now=True)
11     categories = models.ManyToManyField('Category', related_name='posts')

The Category model is very simple. All that’s needed is a single CharField in which we store the name of the category.

The title and body fields on the Post model are the same field types as you used in the Project model. We only need a CharField for the title as we only want a short string for the post title. The body needs to be a long-form piece of text, so we use a TextField.

The next two fields, created_on and last_modified, are Django DateTimeFields. These store a datetime object containing the date and time when the post was created and modified respectively.

On line 11, the DateTimeField takes an argument auto_now_add=True. This assigns the current date and time to this field whenever an instance of this class is created.

On line 12, the DateTimeField takes an argument auto_now=True. This assigns the current date and time to this field whenever an instance of this class is saved. That means whenever you edit an instance of this class, the date_modified is updated.

The final field on the post model is the most interesting. We want to link our models for categories and posts in such a way that many categories can be assigned to many posts. Luckily, Django makes this easier for us by providing a ManytoManyField field type. This field links the Post and Category models and allows us to create a relationship between the two tables.

The ManyToManyField takes two arguments. The first is the model with which the relationship is, in this case its Category. The second allows us to access the relationship from a Category object, even though we haven’t added a field there. By adding a related_name of posts, we can access category.posts to give us a list of posts with that category.

The third and final model we need to add is Comment. We’ll use another relationship field similar the ManyToManyField that relates Post and Category. However, we only want the relationship to go one way: one post should have many comments.

You’ll see how this works after we define the Comment class:

16 class Comment(models.Model):
17     author = models.CharField(max_length=60)
18     body = models.TextField()
19     created_on = models.DateTimeField(auto_now_add=True)
20     post = models.ForeignKey('Post', on_delete=models.CASCADE)

The first three fields on this model should look familiar. There’s an author field for users to add a name or alias, a body field for the body of the comment, and a created_on field that is identical to the created_on field on the Post model.

On line 20, we use another relational field, the ForeignKey field. This is similar to the ManyToManyField but instead defines a many to one relationship. The reasoning behind this is that many comments can be assigned to one post. But you can’t have a comment that corresponds to many posts.

The ForeignKey field takes two arguments. The first is the other model in the relationship, in this case, Post. The second tells Django what to do when a post is deleted. If a post is deleted, then we don’t want the comments related to it hanging around. We, therefore, want to delete them as well, so we add the argument on_delete=models.CASCADE.

Once you’ve created the models, you can create the migration files with makemigrations:

$ python manage.py makemigrations blog

The final step is to migrate the tables. This time, don’t add the app-specific flag. Later on, you’ll need the User model that Django creates for you:

$ python manage.py migrate

Now that you’ve created the models, we can start to add some posts and categories. You won’t be doing this from the command line as you did with the projects, as typing out a whole blog post into the command line would be unpleasant to say the least!

Instead, you’ll learn how to use the Django Admin, which will allow you to create instances of your model classes in a nice web interface.

Don’t forget that you can check out the source code for this section on GitHub before moving onto the next section.

Blog App: Django Admin

The Django Admin is a fantastic tool and one of the great benefits of using Django. As you’re the only person who’s going to be writing blog posts and creating categories, there’s no need to create a user interface to do so.

On the other hand, you don’t want to have to write blog posts in the command line. This is where the admin comes in. It allows you to create, update, and delete instances of your model classes and provides a nice interface for doing so.

Before you can access the admin, you need to add yourself as a superuser. This is why, in the previous section, you applied migrations project-wide as opposed to just for the app. Django comes with built-in user models and a user management system that will allow you to login to the admin.

To start off, you can add yourself as superuser using the following command:

$ python manage.py createsuperuser

You’ll then be prompted to enter a username followed by your email address and password. Once you’ve entered the required details, you’ll be notified that the superuser has been created. Don’t worry if you make a mistake since you can just start again:

Username (leave blank to use 'jasmine'): jfiner
Email address: jfiner@example.com
Password:
Password (again):
Superuser created successfully.

Navigate to localhost:8000/admin and log in with the credentials you just used to create a superuse. You’ll see a page similar to the one below:

The User and Groups models should appear, but you’ll notice that there’s no reference to the models you’ve created yourself. That’s because you need to register them inside the admin.

In the blog directory, open the file admin.py and type the following lines of code:

 1 from django.contrib import admin
 2 from blog.models import Post, Category
 3 
 4 class PostAdmin(admin.ModelAdmin):
 5     pass
 6 
 7 class CategoryAdmin(admin.ModelAdmin):
 8     pass
 9 
10 admin.site.register(Post, PostAdmin)
11 admin.site.register(Category, CategoryAdmin)

On line 2, you import the models you want to register on the admin page.

Note: We’re not adding the comments to the admin. That’s because it’s not usually necessary to edit or create comments yourself.

If you wanted to add a feature where comments are moderated, then go ahead and add the Comments model too. The steps to do so are exactly the same!

On line 5 and line 9, you define empty classes PostAdmin and CategoryAdmin. For the purposes of this tutorial, you don’t need to add any attributes or methods to these classes. They are used to customize what is shown on the admin pages. For this tutorial, the default configuration is enough.

The last two lines are the most important. These register the models with the admin classes. If you now visit localhost:8000/admin, then you should see that the Post and Category models are now visible:

If you click into Posts or Categorys, you should be able to add new instances of both models. I like to add the text of fake blog posts by using lorem ipsum dummy text.

Create a couple of fake posts and assign them fake categories before moving onto the next section. That way, you’ll have posts you can view when we create our templates.

Don’t forget to check out the source code for this section before moving on to building out the views for our app.

Blog App: Views

You’ll need to create three view functions in the views.py file in the blog directory:

blog_index will display a list of all your posts.
blog_detail will display the full post as well as comments and a form to allow users to create new comments.
blog_category will be similar to blog_index, but the posts viewed will only be of a specific category chosen by the user.

The simplest view function to start with is blog_index(). This will be very similar to the project_index() view from your project app. You’ll just query the Post models and retrieve all its objects:

 1 from django.shortcuts import render
 2 from blog.models import Post
 3 
 4 def blog_index(request):
 5     posts = Post.objects.all().order_by('-created_on')
 6     context = {
 7         "posts": posts,
 8     }
 9     return render(request, "blog_index.html", context)

On line 3, you import the Post model, and on line 6 inside the view function, you obtain a Queryset containing all the posts in the database. order_by() orders the Queryset according to the argument given. The minus sign tells Django to start with the largest value rather than the smallest. We use this, as we want the posts to be ordered with the most recent post first.

Finally, you define the context dictionary and render the template. Don’t worry about creating it yet. You’ll get to creating those in the next section.

Next, you can start to create the blog_category() view. The view function will need to take a category name as an argument and query the Post database for all posts that have been assigned the given category:

13 def blog_category(request, category):
14     posts = Post.objects.filter(
15         categories__name__contains=category
16     ).order_by(
17         '-created_on'
18     )
19     context = {
20         "category": category,
21         "posts": posts
22     }
23     return render(request, "blog_category.html", context)

On line 14, you’ve used a Django Queryset filter. The argument of the filter tells Django what conditions need to be met for an object to be retrieved. In this case, we only want posts whose categories contain the category with the name corresponding to that given in the argument of the view function. Again, you’re using order_by() to order posts starting with the most recent.

We then add these posts and the category to the context dictionary, and render our template.

The last view function to add is blog_detail(). This is more complicated as we are going to include a form. Before you add the form, just set up the view function to show a specific post with a comment associated with it. This function will be almost equivalent to the project_detail() view function in the projects app:

21 def blog_detail(request, pk):
22     post = Post.objects.get(pk=pk)
23     comments = Comment.objects.filter(post=post)
24     context = {
25         "post": post,
26         "comments": comments,
27     }
28 
29     return render(request, "blog_detail.html", context)

The view function takes a pk value as an argument and, on line 22, retrieves the object with the given pk.

On line 23, we retrieve all the comments assigned to the given post using Django filters again.

Lastly, add both post and comments to the context dictionary and render the template.

To add a form to the page, you’ll need to create another file in the blog directory named forms.py. Django forms are very similar to models. A form consists of a class where the class attributes are form fields. Django comes with some built-in form fields that you can use to quickly create the form you need.

For this form, the only fields you’ll need are author, which should be a CharField, and body, which can also be a CharField.

Note: If the CharField of your form corresponds to a model CharField, make sure both have the same max_length value.

blog/forms.py should contain the following code:

from django import forms

class CommentForm(forms.Form):
    author = forms.CharField(
        max_length=60,
        widget=forms.TextInput(attrs={
            "class": "form-control",
            "placeholder": "Your Name"
        })
    )
    body = forms.CharField(widget=forms.Textarea(
        attrs={
            "class": "form-control",
            "placeholder": "Leave a comment!"
        })
    )

You’ll also notice an argument widget has been passed to both the fields. The author field has the forms.TextInput widget. This tells Django to load this field as an HTML text input element in the templates. The body field uses a forms.TextArea widget instead, so that the field is rendered as an HTML text area element.

These widgets also take an argument attrs, which is a dictionary and allows us to specify some CSS classes, which will help with formatting the template for this view later. It also allows us to add some placeholder text.

When a form is posted, a POST request is sent to the server. So, in the view function, we need to check if a POST request has been received. We can then create a comment from the form fields. Django comes with a handy is_valid() on its forms, so we can check that all the fields have been entered correctly.

Once you’ve created the comment from the form, you’ll need to save it using save() and then query the database for all the comments assigned to the given post. Your view function should contain the following code:

21 def blog_detail(request, pk):
22     post = Post.objects.get(pk=pk)
23 
24     form = CommentForm()
25     if request.method == 'POST':
26         form = CommentForm(request.POST)
27         if form.is_valid():
28             comment = Comment(
29                 author=form.cleaned_data["author"],
30                 body=form.cleaned_data["body"],
31                 post=post
32             )
33             comment.save()
34 
35     comments = Comment.objects.filter(post=post)
36     context = {
37         "post": post,
38         "comments": comments,
39         "form": CommentForm(),
40     }
41     return render(request, "blog_detail.html", context)

On line 25, we create an instance of our form class. Don’t forget to import your form at the beginning of the file:

from . import CommentForm

We then go on to check if a POST request has been received. If it has, then we create a new instance of our form, populated with the data entered into the form.

The form is then validated using is_valid(). If the form is valid, a new instance of Comment is created. You can access the data from the form using form.cleaned_data, which is a dictionary.

They keys of the dictionary correspond to the form fields, so you can access the author using form.cleaned_data['author']. Don’t forget to add the current post to the comment when you create it.

Note: The life cycle of submitting a form can be a little complicated, so here’s an outline of how it works:

When a user visits a page containing a form, they send a GET request to the server. In this case, there’s no data entered in the form, so we just want to render the form and display it.
When a user enters information and clicks the Submit button, a POST request, containing the data submitted with the form, is sent to the server. At this point, the data must be processed, and two things can happen:
- The form is valid, and the user is redirected to the next page.
- The form is invalid, and empty form is once again displayed. The user is back at step 1, and the process repeats.

The Django forms module will output some errors, which you can display to the user. This is beyond the scope of this tutorial, but you can read more about rendering form error messages in the Django documentation.

On line 34, save the comment and go on to add the form to the context dictionary so you can access the form in the HTML template.

The final step before you get to create the templates and actually see this blog up and running is to hook up the URLs. You’ll need create another urls.py file inside blog/ and add the URLs for the three views:

from django.urls import path
from . import views

urlpatterns = [
    path("", views.blog_index, name="blog_index"),
    path("<int:pk>/", views.blog_detail, name="blog_detail"),
    path("<category>/", views.blog_category, name="blog_category"),
]

Once the blog-specific URLs are in place, you need to add them to the projects URL configuration using include():

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path("admin/", admin.site.urls),
    path("projects/", include("projects.urls")),
    path("blog/", include("blog.urls")),
]

With this set up, all the blog URLs will be prefixed with blog/, and you’ll have the following URL paths:

localhost:8000/blog: Blog index
localhost:8000/blog/1: Blog detail view of blog with pk=1
localhost:8000/blog/python: Blog index view of all posts with category python

These URLs won’t work just yet as you still need to create the templates.

In this section, you created all the views for your blog application. You learned how to use filters when making queries and how to create Django forms. It won’t be long now until you can see your blog app in action!

As always, don’t forget that you can check out the source code for this section on GitHub.

Blog App: Templates

The final piece of our blog app is the templates. By the end of this section, you’ll have created a fully functioning blog.

You’ll notice there are some bootstrap elements included in the templates to make the interface prettier. These aren’t the focus of the tutorial so I’ve glossed over what they do but do check out the Bootstrap docs to find out more.

The first template you’ll create is for the blog index in a new file blog/templates/blog_index.html. This will be very similar to the projects index view.

You’ll use a for loop to loop over all the posts. For each post, you’ll display the title and a snippet of the body. As always, you’ll extend the base template personal_porfolio/templates/base.html, which contains our navigation bar and some extra formatting:

 1 {% extends "base.html" %}
 2 {% block page_content %}
 3 <div class="col-md-8 offset-md-2">
 4     <h1>Blog Index</h1>
 5     <hr>
 6     {% for post in posts %}
 7     <h2><a href="{% url 'blog_detail' post.pk%}">{{ post.title }}</a></h2>
 8     <small>
 9         {{ post.created_on.date }} |&nbsp;
10         Categories:&nbsp;
11         {% for category in post.categories.all %}
12         <a href="{% url 'blog_category' category.name %}">
13             {{ category.name }}
14         </a>&nbsp;
15         {% endfor %}
16     </small>
17     <p>{{ post.body | slice:":400" }}...</p>
18     {% endfor %}
19 </div>
20 {% endblock %}

On line 7, we have the post title, which is a hyperlink. The link is a Django link where we are pointing to the URL named blog_detail, which takes an integer as its argument and should correspond to the pk value of the post.

Underneath the title, we’ll display the created_on attribute of the post as well as its categories. On line 11, we use another for loop to loop over all the categories assigned to the post.

On line 17, we use a template filter slice to cut off the post body at 400 characters so that the blog index is more readable.

Once that’s in place, you should be able to access this page by visiting localhost:8000/blog:

Next, create another HTML file blog/templates/blog_category.html where your blog_category template will live. This should be identical to blog_index.html, except with the category name inside the h1 tag instead of Blog Index:

{% extends "base.html" %}
{% block page_content %}
<div class="col-md-8 offset-md-2">
    <h1>{{ category | title }}</h1>
    <hr>
    {% for post in posts %}
        <h2><a href="{% url 'blog_detail' post.pk%}">{{ post.title }}</a></h2>
        <small>
            {{ post.created_on.date }} |&nbsp;
            Categories:&nbsp;
            {% for category in post.categories.all %}
            <a href="{% url 'blog_category' category.name %}">
                {{ category.name }}
            </a>&nbsp;
            {% endfor %}
        </small>
        <p>{{ post.body | slice:":400" }}...</p>
    {% endfor %}
</div>
{% endblock %}

Most of this template is identical to the previous template. The only difference is on line 4, where we use another Django template filter title. This applies titlecase to the string and makes words start with an uppercase character.

With that template finished, you’ll be able to access your category view. If you defined a category named python, you should be able to visit localhost:8000/blog/python and see all the posts with that category:

The last template to create is the post_detail template. In this template, you’ll display the title and full body of a post.

Between the title and the body of the post, you’ll display the date the post was created and any categories. Underneath that, you’ll include a comments form so users can add a new comment. Under this, there will be a list of comments that have already been left:

 1 {% extends "base.html" %}
 2 {% block page_content %}
 3 <div class="col-md-8 offset-md-2">
 4     <h1>{{ post.title }}</h1>
 5     <small>
 6         {{ post.created_on.date }} |&nbsp;
 7         Categories:&nbsp;
 8         {% for category in post.categories.all %}
 9         <a href="{% url 'blog_category' category.name %}">
10             {{ category.name }}
11         </a>&nbsp;
12         {% endfor %}
13     </small>
14     <p>{{ post.body | linebreaks }}</p>
15     <h3>Leave a comment:</h3>
16     <form action="/blog/{{ post.pk }}/" method="post">
17         {% csrf_token %}
18         <div class="form-group">
19             {{ form.author }}
20         </div>
21         <div class="form-group">
22             {{ form.body }}
23         </div>
24         <button type="submit" class="btn btn-primary">Submit</button>
25     </form>
26     <h3>Comments:</h3>
27     {% for comment in comments %}
28     <p>
29         On {{comment.created_on.date }}&nbsp;
30         <b>{{ comment.author }}</b> wrote:
31     </p>
32     <p>{{ comment.body }}</p>
33     <hr>
34     {% endfor %}
35 </div>
36 {% endblock %}

The first few lines of the template in which we display the post title, date, and categories is the same logic as for the previous templates. This time, when rendering the post body, use a linebreaks template filter. This tag registers line breaks as new paragraphs, so the body doesn’t appear as one long block of text.

Underneath the post, on line 16, you’ll display your form. The form action points to the URL path of the page to which you’re sending the POST request to. In this case, it’s the same as the page that is currently being visited. You then add a csrf_token, which provides security and renders the body and author fields of the form, followed by a submit button.

To get the bootstrap styling on the author and body fields, you need to add the form-control class to the text inputs.

Because Django renders the inputs for you when you include {{ form.body }} and {{ form.author }}, you can’t add these classes in the template. That’s why you added the attributes to the form widgets in the previous section.

Underneath the form, there’s another for loop that loops over all the comments on the given post. The comments, body, author, and created_on attributes are all displayed.

Once that template is in place, you should be able to visit localhost:8000/blog/1 and view your first post:

You should also be able to access the post detail pages by clicking on their title in the blog_index view.

The final finishing touch is to add a link to the blog_index to the navigation bar in base.html. This way, when you click on Blog in the navigation bar, you’ll be able to visit the blog. Check out the updates to base.html in the source code to see how to add that link.

With that now in place, your personal portfolio site is complete, and you’ve created your first Django site. The final version of the source code containing all the features can be found on GitHub, so check it out! Click around the site a bit to see all the functionality and try leaving some comments on your posts!

You may find a few things here and there that you think need polishing. Go ahead and tidy them up. The best way to learn more about this web framework is through practice, so try to extend this project and make it even better! If you’re not sure where to start, I’ve left a few ideas for you in the conclusion below!

Conclusion

Congratulations, you’ve reached the end of the tutorial! We’ve covered a lot, so make sure to keep practicing and building. The more you build the easier it will become and the less you’ll have to refer back to this article or the documentation. You’ll be building sophisticated web applications in no time.

In this tutorial you’ve seen:

How to create Django projects and apps
How to add web pages with views and templates
How to get user input with forms
How to hook your views and templates up with URL configurations
How to add data to your site using relational databases with Django’s Object Relational Mapper
How to use the Django Admin to manage your models

In addition, you’ve learned about the MVT structure of Django web applications and why Django is such a good choice for web development.

If you want to learn more about Django, do check out the documentation and make sure to stay tuned for Part 2 of this series!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Working With JSON Data in Python

2019-03-28T14:00:00+00:00

JSON is a lightweight data-interchange format. It allows us to represent the objects in our Python programs as human-readable text that can be sent over the internet. Lots of APIs and databases use JSON for communication.

You’ll learn how to work with Python’s built-in json module to serialize the data in your programs into JSON format. Then, you’ll deserialize some JSON from an online API and convert it into Python objects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Stand Out in a Python Coding Interview

2019-03-27T14:00:00+00:00

You’ve made it past the phone call with the recruiter, and now it’s time to show that you know how to solve problems with actual code. Whether it’s a HackerRank exercise, a take-home assignment, or an onsite whiteboard interview, this is your moment to prove your coding interview skills.

But interviews aren’t just about solving problems: they’re also about showing that you can write clean production code. This means that you have a deep knowledge of Python’s built-in functionality and libraries. This knowledge shows companies that you can move quickly and won’t duplicate functionality that comes with the language just because you don’t know it exists.

At Real Python, we’ve put our heads together and discussed what tools we’re always impressed to see in coding interviews. This article will walk you through the best of that functionality, starting with Python built-ins, then Python’s native support for data structures, and finally Python’s powerful (and often underappreciated) standard library.

In this article, you’ll learn how to:

Use enumerate() to iterate over both indices and values
Debug problematic code with breakpoint()
Format strings effectively with f-strings
Sort lists with custom arguments
Use generators instead of list comprehensions to conserve memory
Define default values when looking up dictionary keys
Count hashable objects with the collections.Counter class
Use the standard library to get lists of permutations and combinations

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Select the Right Built-In Function for the Job

Python has a large standard library but only a small library of built-in functions, which are always available and don’t need to be imported. It’s worth going through each one, but until you get the chance to do so, here are a few built-in functions worth understanding how to use, and in the case of some of them, what alternatives to use instead.

Iterate With `enumerate()` Instead of `range()`

This scenario might come up more than any other in coding interviews: you have a list of elements, and you need to iterate over the list with access to both the indices and the values.

There’s a classic coding interview question named FizzBuzz that can be solved by iterating over both indices and values. In FizzBuzz, you are given a list of integers. Your task is to do the following:

Replace all integers that are evenly divisible by 3 with "fizz"
Replace all integers divisible by 5 with "buzz"
Replace all integers divisible by both 3 and 5 with "fizzbuzz"

Often, developers will solve this problem with range():

>>>

>>> numbers = [45, 22, 14, 65, 97, 72]
>>> for i in range(len(numbers)):
...     if numbers[i] % 3 == 0 and numbers[i] % 5 == 0:
...         numbers[i] = 'fizzbuzz'
...     elif numbers[i] % 3 == 0:
...         numbers[i] = 'fizz'
...     elif numbers[i] % 5 == 0:
...         numbers[i] = 'buzz'
...
>>> numbers
['fizzbuzz', 22, 14, 'buzz', 97, 'fizz']

Range allows you to access the elements of numbers by index and is a useful tool for some situations. But in this case, where you want to get each element’s index and value at the same time, a more elegant solution uses enumerate():

>>>

>>> numbers = [45, 22, 14, 65, 97, 72]
>>> for i, num in enumerate(numbers):
...     if num % 3 == 0 and num % 5 == 0:
...         numbers[i] = 'fizzbuzz'
...     elif num % 3 == 0:
...         numbers[i] = 'fizz'
...     elif num % 5 == 0:
...         numbers[i] = 'buzz'
...
>>> numbers
['fizzbuzz', 22, 14, 'buzz', 97, 'fizz']

For each element, enumerate() returns a counter and the element value. The counter defaults to 0, which conveniently is also the element’s index. Don’t want to start your count at 0? Just use the optional start parameter to set an offset:

>>>

>>> numbers = [45, 22, 14, 65, 97, 72]
>>> for i, num in enumerate(numbers, start=52):
...     print(i, num)
...
52 45
53 22
54 14
55 65
56 97
57 72

By using the start parameter, we access all of the same elements, starting with the first index, but now our count starts from the specified integer value.

Use List Comprehensions Instead of `map()` and `filter()`

“I think dropping filter() and map() is pretty uncontroversial[.]”

— Guido van Rossum, Python’s creator

He may have been wrong about it being uncontroversial, but Guido had good reasons for wanting to remove map() and filter() from Python. One reason is that Python supports list comprehensions, which are often easier to read and support the same functionality as map() and filter().

Let’s first take a look at how we’d structure a call to map() and the equivalent list comprehension:

>>>

>>> numbers = [4, 2, 1, 6, 9, 7]
>>> def square(x):
...     return x*x
...
>>> list(map(square, numbers))
[16, 4, 1, 36, 81, 49]

>>> [square(x) for x in numbers]
[16, 4, 1, 36, 81, 49]

Both approaches, using map() and the list comprehension, return the same values, but the list comprehension is easier to read and understand.

Now we can do the same thing for the filter() and its equivalent list comprehension:

>>>

>>> def is_odd(x):
...    return bool(x % 2)
...
>>> list(filter(is_odd, numbers))
[1, 9, 7]

>>> [x for x in numbers if is_odd(x)]
[1, 9, 7]

Like we saw with map(), the filter() and list comprehension approaches return the same value, but the list comprehension is easier to follow.

Developers who come from other languages may disagree that list comprehensions are easier to read than map() and filter(), but in my experience beginners are able to pick up list comprehensions much more intuitively.

Either way, you’ll rarely go wrong using list comprehensions in a coding interview, as it will communicate that you know what’s most common in Python.

Debug With `breakpoint()` Instead of `print()`

You may have debugged a small problem by adding print() to your code and seeing what was printed out. This approach works well at first but quickly becomes cumbersome. Plus, in a coding interview setting, you hardly want print() calls peppered throughout your code.

Instead, you should be using a debugger. For non-trivial bugs, it’s almost always faster than using print(), and given that debugging is a big part of writing software, it shows that you know how to use tools that will let you develop quickly on the job.

If you’re using Python 3.7, you don’t need to import anything and can just call breakpoint() at the location in your code where you’d like to drop into the debugger:

# Some complicated code with bugs

breakpoint()

Calling breakpoint() will put you into pdb, which is the default Python debugger. On Python 3.6 and older, you can do the same by importing pdb explicitly:

import pdb; pdb.set_trace()

Like breakpoint(), pdb.set_trace() will put you into the pdb debugger. It’s just not quite as clean and is a tad more to remember.

There are other debuggers available that you may want to try, but pdb is part of the standard library, so it’s always available. Whatever debugger you prefer, it’s worth trying them out to get used to the workflow before you’re in a coding interview setting.

Format strings With f-Strings

Python has a lot of different ways to handle string formatting, and it can be tricky to know what to use. In fact, we tackle formatting in depth in two separate articles: one about string formatting in general and one specifically focused on f-strings. In a coding interview, where you’re (hopefully) using Python 3.6+, the suggested formatting approach is Python’s f-strings.

f-strings support use of the string formatting mini-language, as well as powerful string interpolation. These features allow you to add variables or even valid Python expressions and have them evaluated at runtime before being added to the string:

>>>

>>> def get_name_and_decades(name, age):
...     return f"My name is {name} and I'm {age / 10:.5f} decades old."
...
>>> get_name_and_decades("Maria", 31)
My name is Maria and I'm 3.10000 decades old.

The f-string allows you to put Maria into the string and add her age with the desired formatting in one succinct operation.

The one risk to be aware of is that if you’re outputting user-generated values, then that can introduce security risks, in which case Template Strings may be a safer option.

Sort Complex Lists With `sorted()`

Plenty of coding interview questions require some kind of sorting, and there are multiple valid ways you can sort items. Unless the interviewer wants you to implement your own sorting algorithm, it’s usually best to use sorted().

You’ve probably seen the most simple uses of sorting, such as sorting lists of numbers or strings in ascending or descending order:

>>>

>>> sorted([6,5,3,7,2,4,1])
[1, 2, 3, 4, 5, 6, 7]

>>> sorted(['cat', 'dog', 'cheetah', 'rhino', 'bear'], reverse=True)
['rhino', 'dog', 'cheetah', 'cat', 'bear]

By default, sorted() has sorted the input in ascending order, and the reverse keyword argument causes it to sort in descending order.

It’s worth knowing about the optional keyword argument key that lets you specify a function that will be called on every element prior to sorting. Adding a function allows custom sorting rules, which are especially helpful if you want to sort more complex data types:

>>>

>>> animals = [
...     {'type': 'penguin', 'name': 'Stephanie', 'age': 8},
...     {'type': 'elephant', 'name': 'Devon', 'age': 3},
...     {'type': 'puma', 'name': 'Moe', 'age': 5},
... ]
>>> sorted(animals, key=lambda animal: animal['age'])
[
    {'type': 'elephant', 'name': 'Devon', 'age': 3},
    {'type': 'puma', 'name': 'Moe', 'age': 5},
    {'type': 'penguin', 'name': 'Stephanie, 'age': 8},
]

By passing in a lambda function that returns each element’s age, you can easily sort a list of dictionaries by a single value of each of those dictionaries. In this case, the dictionary is now sorted in ascending order by age.

Leverage Data Structures Effectively

Algorithms get a lot of attention in coding interviews, but data structures are arguably even more important. In a coding interviewing context, picking the right data structure can have a major impact on performance.

Beyond theoretical data structures, Python has powerful and convenient functionality built into its standard data structure implementations. These data structures are incredibly useful in coding interviews because they give you lots of functionality by default and let you focus your time on other parts of the problem.

Store Unique Values With Sets

You’ll often need to remove duplicate elements from an existing dataset. New developers will sometimes do so with lists when they should be using sets, which enforce the uniqueness of all elements.

Pretend you have a function named get_random_word(). It will always return a random selection from a small set of words:

>>>

>>> import random
>>> all_words = "all the words in the world".split()
>>> def get_random_word():
...    return random.choice(all_words)

You’re supposed to call get_random_word() repeatedly to get 1000 random words and then return a data structure containing every unique word. Here are two common, suboptimal approaches and one good approach.

Bad Approach

get_unique_words() stores values in a list then converts the list into a set:

>>>

>>> def get_unique_words():
...     words = []
...     for _ in range(1000):
...         words.append(get_random_word())
...     return set(words)
>>> get_unique_words()
{'world', 'all', 'the', 'words'}

This approach isn’t terrible, but it unnecessarily creates a list and then converts it to a set. Interviewers almost always notice (and ask about) this type of design choice.

Worse Approach

To avoid converting from a list to a set, you now store values in a list without using any other data structures. You then test for uniqueness by comparing new values with all elements currently in the list:

>>>

>>> def get_unique_words():
...     words = []
...     for _ in range(1000):
...         word = get_random_word()
...         if word not in words:
...             words.append(word)
...     return words
>>> get_unique_words()
['world', 'all', 'the', 'words']

This is worse than the first approach, because you have to compare every new word against every word already in the list. That means that as the number of words grows, the number of lookups grows quadratically. In other words, the time complexity grows on the order of O(N²).

Good Approach

Now, you skip using lists altogether and instead use a set from the start:

>>>

>>> def get_unique_words():
...     words = set()
...     for _ in range(1000):
...         words.add(get_random_word())
...     return words
>>> get_unique_words()
{'world', 'all', 'the', 'words'}

This may not look much different than the other approaches except for making use of a set from the beginning. If you consider what’s happening within .add(), it even sounds like the second approach: get the word, check if it’s already in the set, and if not, add it to the data structure.

So why is using a set different from the second approach?

It’s different because sets store elements in a manner that allows near-constant-time checks whether a value is in the set or not, unlike lists, which require linear-time lookups. The difference in lookup time means that the time complexity for adding to a set grows at a rate of O(N), which is much better than the O(N²) from the second approach in most cases.

Save Memory With Generators

List comprehensions are convenient tools but can sometimes lead to unnecessary memory usage.

Imagine you’ve been asked to find the sum of the first 1000 perfect squares, starting with 1. You know about list comprehensions, so you quickly code up a working solution:

>>>

>>> sum([i * i for i in range(1, 1001)])
333833500

Your solution makes a list of every perfect square between 1 and 1,000,000 and sums the values. Your code returns the right answer, but then your interviewer starts increasing the number of perfect squares you need to sum.

At first, your function keeps popping out the right answer, but soon it starts slowing down until eventually the process seems to hang for an eternity. This is the last thing you want happening in a coding interview.

What’s going on here?

It’s making a list of every perfect square you’ve requested and summing them all. A list with 1000 perfect squares may not be large in computer-terms, but 100 million or 1 billion is quite a bit of information and can easily overwhelm your computer’s available memory resources. That’s what’s happening here.

Thankfully, there’s a quick way to solve the memory problem. You just replace the brackets with parentheses:

>>>

>>> sum((i * i for i in range(1, 1001)))
333833500

Swapping out the brackets changes your list comprehension into a generator expression. Generator expressions are perfect for when you know you want to retrieve data from a sequence, but you don’t need to access all of it at the same time.

Instead of creating a list, the generator expression returns a generator object. That object knows where it is in the current state (for example, i = 49) and only calculates the next value when it’s asked for.

So when sum iterates over the generator object by calling .__next__() repeatedly, the generator checks what i equals, calculates i * i, increments i internally, and returns the proper value to sum. The design allows generators to be used on massive sequences of data, because only one element exists in memory at a time.

Define Default Values in Dictionaries With `.get()` and `.setdefault()`

One of the most common programming tasks involves adding, modifying, or retrieving an item that may or may not be in a dictionary. Python dictionaries have elegant functionality to make these tasks clean and easy, but developers often check explicitly for values when it isn’t necessary.

Imagine you have a dictionary named cowboy, and you want to get that cowboy’s name. One approach is to explicitly check for the key with a conditional:

>>>

>>> cowboy = {'age': 32, 'horse': 'mustang', 'hat_size': 'large'}
>>> if 'name' in cowboy:
...     name = cowboy['name']
... else:
...     name = 'The Man with No Name'
...
>>> name
'The Man with No Name'

This approach first checks if the name key exists in the dictionary, and if so, it returns the corresponding value. Otherwise, it returns a default value.

While explicitly checking for keys does work, it can easily be replaced with one line if you use .get():

>>>

>>> name = cowboy.get('name', 'The Man with No Name')

get() performs the same operations that were performed in the first approach, but now they’re handled automatically. If the key exists, then the proper value will be returned. Otherwise, the default value will get returned.

But what if you want to update the dictionary with a default value while still accessing the name key? .get() doesn’t really help you here, so you’re left with explicitly checking for the value again:

>>>

>>> if 'name' not in cowboy:
...     cowboy['name'] = 'The Man with No Name'
...
>>> name = cowboy['name']

Checking for the value and setting a default is a valid approach and is easy to read, but again Python offers a more elegant method with .setdefault():

>>>

>>> name = cowboy.setdefault('name', 'The Man with No Name')

.setdefault() accomplishes exactly the same thing as the snippet above. It checks if name exists in cowboy, and if so it returns that value. Otherwise, it sets cowboy['name'] to The Man with No Name and returns the new value.

Take Advantage of Python’s Standard Library

By default, Python comes with a lot of functionality that’s just an import statement away. It’s powerful on its own, but knowing how to leverage the standard library can supercharge your coding interview skills.

It’s hard to pick the most useful pieces from all of the available modules, so this section will focus on just a small subset of its utility functions. Hopefully, these will prove useful to you in coding interviews and also whet your appetite to learn more about the advanced functionality of these and other modules.

Handle Missing Dictionary Keys With `collections.defaultdict()`

.get() and .setdefault() work well when you’re setting a default for a single key, but it’s common to want a default value for all possible unset keys, especially when programming in a coding interview context.

Pretend you have a group of students, and you need to keep track of their grades on homework assignments. The input value is a list of tuples with the format (student_name, grade), but you want to easily look up all the grades for a single student without iterating over the list.

One way to store the grade data uses a dictionary that maps student names to lists of grades:

>>>

>>> student_grades = {}
>>> grades = [
...     ('elliot', 91),
...     ('neelam', 98),
...     ('bianca', 81),
...     ('elliot', 88),
... ]
>>> for name, grade in grades:
...     if name not in student_grades:
...         student_grades[name] = []
...     student_grades[name].append(grade)
...
>>> student_grades
{'elliot': [91, 88], 'neelam': [98], 'bianca': [81]}

In this approach, you iterate over the students and check if their names are already properties in the dictionary. If not, you add them to the dictionary with an empty list as the default value. You then append their actual grades to that student’s list of grades.

But there’s an even cleaner approach that uses a defaultdict, which extends standard dict functionality to allow you to set a default value that will be operated upon if the key doesn’t exist:

>>>

>>> from collections import defaultdict
>>> student_grades = defaultdict(list)
>>> for name, grade in grades:
...     student_grades[name].append(grade)

In this case, you’re creating a defaultdict that uses the list() constructor with no arguments as a default factory method. list() with no arguments returns an empty list, so defaultdict calls list() if the name doesn’t exist and then allows the grade to be appended. If you want to get fancy, you could also use a lambda function as your factory value to return an arbitrary constant.

Leveraging a defaultdict can lead to cleaner application code because you don’t have to worry about default values at the key level. Instead, you can handle them once at the defaultdict level and afterwards act as if the key is always present.

Count Hashable Objects With `collections.Counter`

You have a long string of words with no punctuation or capital letters and you want to count how many times each word appears.

You could use a dictionary or defaultdict and increment the counts, but collections.Counter provides a cleaner and more convenient way to do exactly that. Counter is a subclass of dict that uses 0 as the default value for any missing element and makes it easier to count occurrences of objects:

>>>

>>> from collections import Counter
>>> words = "if there was there was but if \
... there was not there was not".split()
>>> counts = Counter(words)
>>> counts
Counter({'if': 2, 'there': 4, 'was': 4, 'not': 2, 'but': 1})

When you pass in the list of words to Counter, it stores each word along with a count of how many times that word occurred in the list.

Are you curious what the two most common words were? Just use .most_common():

>>>

>>> counts.most_common(2)
[('there', 4), ('was', 4)]

.most_common() is a convenience method and simply returns the n most frequent inputs by count.

Access Common String Groups With `string` Constants

It’s trivia time! Is 'A' > 'a' true or false?

It’s false, because the ASCII code for A is 65, but a is 97, and 65 is not greater than 97.

Why does the answer matter? Because if you want to check if a character is part of the English alphabet, one popular way is to see if it’s between A and z (65 and 122 on the ASCII chart).

Checking the ASCII code works but is clumsy and easy to mess up in coding interviews, especially if you can’t remember whether lowercase or uppercase ASCII characters come first. It’s much easier to use the constants defined as part of the string module.

You can see one in use in is_upper(), which returns whether all characters in a string are uppercase letters:

>>>

>>> import string
>>> def is_upper(word):
...     for letter in word:
...         if letter not in string.ascii_uppercase:
...             return False
...     return True
...
>>> is_upper('Thanks Geir')
False
>>> is_upper('LOL')
True

is_upper() iterates over the letters in word, and checks if the letters are part of string.ascii_uppercase. If you print out string.ascii_uppercase you’ll see that it’s just a lowly string. The value is set to the literal 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'.

All string constants are just strings of frequently referenced string values. They include the following:

string.ascii_letters
string.ascii_uppercase
string.ascii_lowercase
string.digits
string.hexdigits
string.octdigits
string.punctuation
string.printable
string.whitespace

These are easier to use and, even more importantly, easier to read.

Generate Permutations and Combinations With `itertools`

Interviewers love to give real life scenarios to make coding interviews seem less intimidating, so here’s a contrived example: you go to an amusement park and decide to figure out every possible pair of friends that could sit together on a roller coaster.

Unless generating these pairs is the primary purpose of the interview question, it’s likely that generating all the possible pairs is just a tedious step on the way towards a working algorithm. You could calculate them yourself with nested for-loops, or you could use the powerful itertools library.

itertools has multiple tools for generating iterable sequences of input data, but right now we’ll just focus on two common functions: itertools.permutations() and itertools.combinations().

itertools.permutations() builds a list of all permutations, meaning it’s a list of every possible grouping of input values with a length matching the count parameter. The r keyword argument lets us specify how many values go in each grouping:

>>>

>>> import itertools
>>> friends = ['Monique', 'Ashish', 'Devon', 'Bernie']
>>> list(itertools.permutations(friends, r=2))
[('Monique', 'Ashish'), ('Monique', 'Devon'), ('Monique', 'Bernie'),
('Ashish', 'Monique'), ('Ashish', 'Devon'), ('Ashish', 'Bernie'),
('Devon', 'Monique'), ('Devon', 'Ashish'), ('Devon', 'Bernie'),
('Bernie', 'Monique'), ('Bernie', 'Ashish'), ('Bernie', 'Devon')]

With permutations, the order of the elements matters, so ('sam', 'devon') represents a different pairing than ('devon', 'sam'), meaning that they would both be included in the list.

itertools.combinations() builds combinations. These are also the possible groupings of the input values, but now the order of the values doesn’t matter. Because ('sam', 'devon') and ('devon', 'sam') represent the same pair, only one of them would be included in the output list:

>>>

>>> list(itertools.combinations(friends, r=2))
[('Monique', 'Ashish'), ('Monique', 'Devon'), ('Monique', 'Bernie'),
('Ashish', 'Devon'), ('Ashish', 'Bernie'), ('Devon', 'Bernie')]

Since the order of the values matters with combinations, there are fewer combinations than permutations for the same input list. Again, because we set r to 2, each grouping has two names in it.

.combinations() and .permutations() are just small examples of a powerful library, but even these two functions can be quite useful when you’re trying to solve an algorithm problem quickly.

Conclusion: Coding Interview Superpowers

You can now feel comfortable using some of Python’s less common, but more powerful, standard features in your next coding interview. There’s a lot to learn about the language as a whole but this article should have given you a starting point to go deeper while letting you use Python more effectively when you interview.

In this article, you’ve learned different types of standard tools to supercharge your coding interview skills:

Powerful built-in functions
Data structures built to handle common scenarios with barely any code
Standard library packages that have feature-rich solutions for specific problems, letting you write better code faster

Interviewing may not be the best approximation of real software development, but it’s worth knowing how to succeed in any programming environment, even interviews. Thankfully, learning how to use Python during coding interviews can help you understand the language more deeply, which will pay dividends during day-to-day development.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Django Migrations 101

2019-03-26T14:00:00+00:00

Since version 1.7, Django has come with built-in support for database migrations. In Django, database migrations usually go hand in hand with models: whenever you code up a new model, you also generate a migration to create the necessary table in the database. However, migrations can do much more.

With this course you’ll get comfortable with Django migrations and learn how to create database tables without writing any SQL, how to automatically modify your database after you changed your models, and how to revert changes made to your database.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Documenting Python Projects With Sphinx and Read The Docs

2019-03-26T13:30:00+00:00

In this video series we’ll cover creating Python documentation from scratch using Sphinx, as well as getting your code repository hooked up to Read The Docs, to automatically build and publish your code documentation.

Note: This course uses Python 2.7 in its coding examples.

Resources

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

An Intro to Threading in Python

2019-03-25T14:00:00+00:00

Python threading allows you to have different parts of your program run concurrently and can simplify your design. If you’ve got some experience in Python and want to speed up your program using threads, then this tutorial is for you!

In this article, you’ll learn:

What threads are
How to create threads and wait for them to finish
How to use a ThreadPoolExecutor
How to avoid race conditions
How to use the common tools that Python threading provides

This article assumes you’ve got the Python basics down pat and that you’re using at least version 3.6 to run the examples. If you need a refresher, you can start with the Python Learning Paths and get up to speed.

If you’re not sure if you want to use Python threading, asyncio, or multiprocessing, then you can check out Speed Up Your Python Program With Concurrency.

All of the sources used in this tutorial are available to you in the Real Python GitHub repo.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Take the Quiz: Test your knowledge with our interactive “Python Threading” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Take the Quiz »

What Is a Thread?

A thread is a separate flow of execution. This means that your program will have two things happening at once. But for most Python 3 implementations the different threads do not actually execute at the same time: they merely appear to.

It’s tempting to think of threading as having two (or more) different processors running on your program, each one doing an independent task at the same time. That’s almost right. The threads may be running on different processors, but they will only be running one at a time.

Getting multiple tasks running simultaneously requires a non-standard implementation of Python, writing some of your code in a different language, or using multiprocessing which comes with some extra overhead.

Because of the way CPython implementation of Python works, threading may not speed up all tasks. This is due to interactions with the GIL that essentially limit one Python thread to run at a time.

Tasks that spend much of their time waiting for external events are generally good candidates for threading. Problems that require heavy CPU computation and spend little time waiting for external events might not run faster at all.

This is true for code written in Python and running on the standard CPython implementation. If your threads are written in C they have the ability to release the GIL and run concurrently. If you are running on a different Python implementation, check with the documentation too see how it handles threads.

If you are running a standard Python implementation, writing in only Python, and have a CPU-bound problem, you should check out the multiprocessing module instead.

Architecting your program to use threading can also provide gains in design clarity. Most of the examples you’ll learn about in this tutorial are not necessarily going to run faster because they use threads. Using threading in them helps to make the design cleaner and easier to reason about.

So, let’s stop talking about threading and start using it!

Starting a Thread

Now that you’ve got an idea of what a thread is, let’s learn how to make one. The Python standard library provides threading, which contains most of the primitives you’ll see in this article. Thread, in this module, nicely encapsulates threads, providing a clean interface to work with them.

To start a separate thread, you create a Thread instance and then tell it to .start():

 1 import logging
 2 import threading
 3 import time
 4 
 5 def thread_function(name):
 6     logging.info("Thread %s: starting", name)
 7     time.sleep(2)
 8     logging.info("Thread %s: finishing", name)
 9 
10 if __name__ == "__main__":
11     format = "%(asctime)s: %(message)s"
12     logging.basicConfig(format=format, level=logging.INFO,
13                         datefmt="%H:%M:%S")
14 
15     logging.info("Main    : before creating thread")
16     x = threading.Thread(target=thread_function, args=(1,))
17     logging.info("Main    : before running thread")
18     x.start()
19     logging.info("Main    : wait for the thread to finish")
20     # x.join()
21     logging.info("Main    : all done")

If you look around the logging statements, you can see that the main section is creating and starting the thread:

x = threading.Thread(target=thread_function, args=(1,))
x.start()

When you create a Thread, you pass it a function and a list containing the arguments to that function. In this case, you’re telling the Thread to run thread_function() and to pass it 1 as an argument.

For this article, you’ll use sequential integers as names for your threads. There is threading.get_ident(), which returns a unique name for each thread, but these are usually neither short nor easily readable.

thread_function() itself doesn’t do much. It simply logs some messages with a time.sleep() in between them.

When you run this program as it is (with line twenty commented out), the output will look like this:

$ ./single_thread.py
Main    : before creating thread
Main    : before running thread
Thread 1: starting
Main    : wait for the thread to finish
Main    : all done
Thread 1: finishing

You’ll notice that the Thread finished after the Main section of your code did. You’ll come back to why that is and talk about the mysterious line twenty in the next section.

Daemon Threads

In computer science, a daemon is a process that runs in the background.

Python threading has a more specific meaning for daemon. A daemon thread will shut down immediately when the program exits. One way to think about these definitions is to consider the daemon thread a thread that runs in the background without worrying about shutting it down.

If a program is running Threads that are not daemons, then the program will wait for those threads to complete before it terminates. Threads that are daemons, however, are just killed wherever they are when the program is exiting.

Let’s look a little more closely at the output of your program above. The last two lines are the interesting bit. When you run the program, you’ll notice that there is a pause (of about 2 seconds) after __main__ has printed its all done message and before the thread is finished.

This pause is Python waiting for the non-daemonic thread to complete. When your Python program ends, part of the shutdown process is to clean up the threading routine.

If you look at the source for Python threading, you’ll see that threading._shutdown() walks through all of the running threads and calls .join() on every one that does not have the daemon flag set.

So your program waits to exit because the thread itself is waiting in a sleep. As soon as it has completed and printed the message, .join() will return and the program can exit.

Frequently, this behavior is what you want, but there are other options available to us. Let’s first repeat the program with a daemon thread. You do that by changing how you construct the Thread, adding the daemon=True flag:

x = threading.Thread(target=thread_function, args=(1,), daemon=True)

When you run the program now, you should see this output:

$ ./daemon_thread.py
Main    : before creating thread
Main    : before running thread
Thread 1: starting
Main    : wait for the thread to finish
Main    : all done

The difference here is that the final line of the output is missing. thread_function() did not get a chance to complete. It was a daemon thread, so when __main__ reached the end of its code and the program wanted to finish, the daemon was killed.

`join()` a Thread

Daemon threads are handy, but what about when you want to wait for a thread to stop? What about when you want to do that and not exit your program? Now let’s go back to your original program and look at that commented out line twenty:

# x.join()

To tell one thread to wait for another thread to finish, you call .join(). If you uncomment that line, the main thread will pause and wait for the thread x to complete running.

Did you test this on the code with the daemon thread or the regular thread? It turns out that it doesn’t matter. If you .join() a thread, that statement will wait until either kind of thread is finished.

Working With Many Threads

The example code so far has only been working with two threads: the main thread and one you started with the threading.Thread object.

Frequently, you’ll want to start a number of threads and have them do interesting work. Let’s start by looking at the harder way of doing that, and then you’ll move on to an easier method.

The harder way of starting multiple threads is the one you already know:

import logging
import threading
import time

def thread_function(name):
    logging.info("Thread %s: starting", name)
    time.sleep(2)
    logging.info("Thread %s: finishing", name)

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")

    threads = list()
    for index in range(3):
        logging.info("Main    : create and start thread %d.", index)
        x = threading.Thread(target=thread_function, args=(index,))
        threads.append(x)
        x.start()

    for index, thread in enumerate(threads):
        logging.info("Main    : before joining thread %d.", index)
        thread.join()
        logging.info("Main    : thread %d done", index)

This code uses the same mechanism you saw above to start a thread, create a Thread object, and then call .start(). The program keeps a list of Thread objects so that it can then wait for them later using .join().

Running this code multiple times will likely produce some interesting results. Here’s an example output from my machine:

$ ./multiple_threads.py
Main    : create and start thread 0.
Thread 0: starting
Main    : create and start thread 1.
Thread 1: starting
Main    : create and start thread 2.
Thread 2: starting
Main    : before joining thread 0.
Thread 2: finishing
Thread 1: finishing
Thread 0: finishing
Main    : thread 0 done
Main    : before joining thread 1.
Main    : thread 1 done
Main    : before joining thread 2.
Main    : thread 2 done

If you walk through the output carefully, you’ll see all three threads getting started in the order you might expect, but in this case they finish in the opposite order! Multiple runs will produce different orderings. Look for the Thread x: finishing message to tell you when each thread is done.

The order in which threads are run is determined by the operating system and can be quite hard to predict. It may (and likely will) vary from run to run, so you need to be aware of that when you design algorithms that use threading.

Fortunately, Python gives you several primitives that you’ll look at later to help coordinate threads and get them running together. Before that, let’s look at how to make managing a group of threads a bit easier.

Using a `ThreadPoolExecutor`

There’s an easier way to start up a group of threads than the one you saw above. It’s called a ThreadPoolExecutor, and it’s part of the standard library in concurrent.futures (as of Python 3.2).

The easiest way to create it is as a context manager, using the with statement to manage the creation and destruction of the pool.

Here’s the __main__ from the last example rewritten to use a ThreadPoolExecutor:

import concurrent.futures

# [rest of code]

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")

    with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
        executor.map(thread_function, range(3))

The code creates a ThreadPoolExecutor as a context manager, telling it how many worker threads it wants in the pool. It then uses .map() to step through an iterable of things, in your case range(3), passing each one to a thread in the pool.

The end of the with block causes the ThreadPoolExecutor to do a .join() on each of the threads in the pool. It is strongly recommended that you use ThreadPoolExecutor as a context manager when you can so that you never forget to .join() the threads.

Note: Using a ThreadPoolExecutor can cause some confusing errors.

For example, if you call a function that takes no parameters, but you pass it parameters in .map(), the thread will throw an exception.

Unfortunately, ThreadPoolExecutor will hide that exception, and (in the case above) the program terminates with no output. This can be quite confusing to debug at first.

Running your corrected example code will produce output that looks like this:

$ ./executor.py
Thread 0: starting
Thread 1: starting
Thread 2: starting
Thread 1: finishing
Thread 0: finishing
Thread 2: finishing

Again, notice how Thread 1 finished before Thread 0. The scheduling of threads is done by the operating system and does not follow a plan that’s easy to figure out.

Race Conditions

Before you move on to some of the other features tucked away in Python threading, let’s talk a bit about one of the more difficult issues you’ll run into when writing threaded programs: race conditions.

Once you’ve seen what a race condition is and looked at one happening, you’ll move on to some of the primitives provided by the standard library to prevent race conditions from happening.

Race conditions can occur when two or more threads access a shared piece of data or resource. In this example, you’re going to create a large race condition that happens every time, but be aware that most race conditions are not this obvious. Frequently, they only occur rarely, and they can produce confusing results. As you can imagine, this makes them quite difficult to debug.

Fortunately, this race condition will happen every time, and you’ll walk through it in detail to explain what is happening.

For this example, you’re going to write a class that updates a database. Okay, you’re not really going to have a database: you’re just going to fake it, because that’s not the point of this article.

Your FakeDatabase will have .__init__() and .update() methods:

class FakeDatabase:
    def __init__(self):
        self.value = 0

    def update(self, name):
        logging.info("Thread %s: starting update", name)
        local_copy = self.value
        local_copy += 1
        time.sleep(0.1)
        self.value = local_copy
        logging.info("Thread %s: finishing update", name)

FakeDatabase is keeping track of a single number: .value. This is going to be the shared data on which you’ll see the race condition.

.__init__() simply initializes .value to zero. So far, so good.

.update() looks a little strange. It’s simulating reading a value from a database, doing some computation on it, and then writing a new value back to the database.

In this case, reading from the database just means copying .value to a local variable. The computation is just to add one to the value and then .sleep() for a little bit. Finally, it writes the value back by copying the local value back to .value.

Here’s how you’ll use this FakeDatabase:

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")

    database = FakeDatabase()
    logging.info("Testing update. Starting value is %d.", database.value)
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        for index in range(2):
            executor.submit(database.update, index)
    logging.info("Testing update. Ending value is %d.", database.value)

The program creates a ThreadPoolExecutor with two threads and then calls .submit() on each of them, telling them to run database.update().

.submit() has a signature that allows both positional and named arguments to be passed to the function running in the thread:

.submit(function, *args, **kwargs)

In the usage above, index is passed as the first and only positional argument to database.update(). You’ll see later in this article where you can pass multiple arguments in a similar manner.

Since each thread runs .update(), and .update() adds one to .value, you might expect database.value to be 2 when it’s printed out at the end. But you wouldn’t be looking at this example if that was the case. If you run the above code, the output looks like this:

$ ./racecond.py
Testing unlocked update. Starting value is 0.
Thread 0: starting update
Thread 1: starting update
Thread 0: finishing update
Thread 1: finishing update
Testing unlocked update. Ending value is 1.

You might have expected that to happen, but let’s look at the details of what’s really going on here, as that will make the solution to this problem easier to understand.

One Thread

Before you dive into this issue with two threads, let’s step back and talk a bit about some details of how threads work.

You won’t be diving into all of the details here, as that’s not important at this level. We’ll also be simplifying a few things in a way that won’t be technically accurate but will give you the right idea of what is happening.

When you tell your ThreadPoolExecutor to run each thread, you tell it which function to run and what parameters to pass to it: executor.submit(database.update, index).

The result of this is that each of the threads in the pool will call database.update(index). Note that database is a reference to the one FakeDatabase object created in __main__. Calling .update() on that object calls an instance method on that object.

Each thread is going to have a reference to the same FakeDatabase object, database. Each thread will also have a unique value, index, to make the logging statements a bit easier to read:

When the thread starts running .update(), it has its own version of all of the data local to the function. In the case of .update(), this is local_copy. This is definitely a good thing. Otherwise, two threads running the same function would always confuse each other. It means that all variables that are scoped (or local) to a function are thread-safe.

Now you can start walking through what happens if you run the program above with a single thread and a single call to .update().

The image below steps through the execution of .update() if only a single thread is run. The statement is shown on the left followed by a diagram showing the values in the thread’s local_value and the shared database.value:

The diagram is laid out so that time increases as you move from top to bottom. It begins when Thread 1 is created and ends when it is terminated.

When Thread 1 starts, FakeDatabase.value is zero. The first line of code in the method, local_copy = self.value, copies the value zero to the local variable. Next it increments the value of local_copy with the local_copy += 1 statement. You can see .value in Thread 1 getting set to one.

Next time.sleep() is called, which makes the current thread pause and allows other threads to run. Since there is only one thread in this example, this has no effect.

When Thread 1 wakes up and continues, it copies the new value from local_copy to FakeDatabase.value, and then the thread is complete. You can see that database.value is set to one.

So far, so good. You ran .update() once and FakeDatabase.value was incremented to one.

Two Threads

Getting back to the race condition, the two threads will be running concurrently but not at the same time. They will each have their own version of local_copy and will each point to the same database. It is this shared database object that is going to cause the problems.

The program starts with Thread 1 running .update():

When Thread 1 calls time.sleep(), it allows the other thread to start running. This is where things get interesting.

Thread 2 starts up and does the same operations. It’s also copying database.value into its private local_copy, and this shared database.value has not yet been updated:

When Thread 2 finally goes to sleep, the shared database.value is still unmodified at zero, and both private versions of local_copy have the value one.

Thread 1 now wakes up and saves its version of local_copy and then terminates, giving Thread 2 a final chance to run. Thread 2 has no idea that Thread 1 ran and updated database.value while it was sleeping. It stores its version of local_copy into database.value, also setting it to one:

The two threads have interleaving access to a single shared object, overwriting each other’s results. Similar race conditions can arise when one thread frees memory or closes a file handle before the other thread is finished accessing it.

Why This Isn’t a Silly Example

The example above is contrived to make sure that the race condition happens every time you run your program. Because the operating system can swap out a thread at any time, it is possible to interrupt a statement like x = x + 1 after it has read the value of x but before it has written back the incremented value.

The details of how this happens are quite interesting, but not needed for the rest of this article, so feel free to skip over this hidden section.

The code above isn’t quite as out there as you might originally have thought. It was designed to force a race condition every time you run it, but that makes it much easier to solve than most race conditions.

There are two things to keep in mind when thinking about race conditions:

Even an operation like x += 1 takes the processor many steps. Each of these steps is a separate instruction to the processor.
The operating system can swap which thread is running at any time. A thread can be swapped out after any of these small instructions. This means that a thread can be put to sleep to let another thread run in the middle of a Python statement.

Let’s look at this in detail. The REPL below shows a function that takes a parameter and increments it:

>>>

>>> def inc(x):
...    x += 1
...
>>> import dis
>>> dis.dis(inc)
  2           0 LOAD_FAST                0 (x)
              2 LOAD_CONST               1 (1)
              4 INPLACE_ADD
              6 STORE_FAST               0 (x)
              8 LOAD_CONST               0 (None)
             10 RETURN_VALUE

The REPL example uses dis from the Python standard library to show the smaller steps that the processor does to implement your function. It does a LOAD_FAST of the data value x, it does a LOAD_CONST 1, and then it uses the INPLACE_ADD to add those values together.

We’re stopping here for a specific reason. This is the point in .update() above where time.sleep() forced the threads to switch. It is entirely possible that, every once in while, the operating system would switch threads at that exact point even without sleep(), but the call to sleep() makes it happen every time.

As you learned above, the operating system can swap threads at any time. You’ve walked down this listing to the statement marked 4. If the operating system swaps out this thread and runs a different thread that also modifies x, then when this thread resumes, it will overwrite x with an incorrect value.

Technically, this example won’t have a race condition because x is local to inc(). It does illustrate how a thread can be interrupted during a single Python operation, however. The same LOAD, MODIFY, STORE set of operations also happens on global and shared values. You can explore with the dis module and prove that yourself.

It’s rare to get a race condition like this to occur, but remember that an infrequent event taken over millions of iterations becomes likely to happen. The rarity of these race conditions makes them much, much harder to debug than regular bugs.

Now back to your regularly scheduled tutorial!

Now that you’ve seen a race condition in action, let’s find out how to solve them!

Basic Synchronization Using `Lock`

There are a number of ways to avoid or solve race conditions. You won’t look at all of them here, but there are a couple that are used frequently. Let’s start with Lock.

To solve your race condition above, you need to find a way to allow only one thread at a time into the read-modify-write section of your code. The most common way to do this is called Lock in Python. In some other languages this same idea is called a mutex. Mutex comes from MUTual EXclusion, which is exactly what a Lock does.

A Lock is an object that acts like a hall pass. Only one thread at a time can have the Lock. Any other thread that wants the Lock must wait until the owner of the Lock gives it up.

The basic functions to do this are .acquire() and .release(). A thread will call my_lock.acquire() to get the lock. If the lock is already held, the calling thread will wait until it is released. There’s an important point here. If one thread gets the lock but never gives it back, your program will be stuck. You’ll read more about this later.

Fortunately, Python’s Lock will also operate as a context manager, so you can use it in a with statement, and it gets released automatically when the with block exits for any reason.

Let’s look at the FakeDatabase with a Lock added to it. The calling function stays the same:

class FakeDatabase:
    def __init__(self):
        self.value = 0
        self._lock = threading.Lock()

    def locked_update(self, name):
        logging.info("Thread %s: starting update", name)
        logging.debug("Thread %s about to lock", name)
        with self._lock:
            logging.debug("Thread %s has lock", name)
            local_copy = self.value
            local_copy += 1
            time.sleep(0.1)
            self.value = local_copy
            logging.debug("Thread %s about to release lock", name)
        logging.debug("Thread %s after release", name)
        logging.info("Thread %s: finishing update", name)

Other than adding a bunch of debug logging so you can see the locking more clearly, the big change here is to add a member called ._lock, which is a threading.Lock() object. This ._lock is initialized in the unlocked state and locked and released by the with statement.

It’s worth noting here that the thread running this function will hold on to that Lock until it is completely finished updating the database. In this case, that means it will hold the Lock while it copies, updates, sleeps, and then writes the value back to the database.

If you run this version with logging set to warning level, you’ll see this:

$ ./fixrace.py
Testing locked update. Starting value is 0.
Thread 0: starting update
Thread 1: starting update
Thread 0: finishing update
Thread 1: finishing update
Testing locked update. Ending value is 2.

Look at that. Your program finally works!

You can turn on full logging by setting the level to DEBUG by adding this statement after you configure the logging output in __main__:

logging.getLogger().setLevel(logging.DEBUG)

Running this program with DEBUG logging turned on looks like this:

$ ./fixrace.py
Testing locked update. Starting value is 0.
Thread 0: starting update
Thread 0 about to lock
Thread 0 has lock
Thread 1: starting update
Thread 1 about to lock
Thread 0 about to release lock
Thread 0 after release
Thread 0: finishing update
Thread 1 has lock
Thread 1 about to release lock
Thread 1 after release
Thread 1: finishing update
Testing locked update. Ending value is 2.

In this output you can see Thread 0 acquires the lock and is still holding it when it goes to sleep. Thread 1 then starts and attempts to acquire the same lock. Because Thread 0 is still holding it, Thread 1 has to wait. This is the mutual exclusion that a Lock provides.

Many of the examples in the rest of this article will have WARNING and DEBUG level logging. We’ll generally only show the WARNING level output, as the DEBUG logs can be quite lengthy. Try out the programs with the logging turned up and see what they do.

Deadlock

Before you move on, you should look at a common problem when using Locks. As you saw, if the Lock has already been acquired, a second call to .acquire() will wait until the thread that is holding the Lock calls .release(). What do you think happens when you run this code:

import threading

l = threading.Lock()
print("before first acquire")
l.acquire()
print("before second acquire")
l.acquire()
print("acquired lock twice")

When the program calls l.acquire() the second time, it hangs waiting for the Lock to be released. In this example, you can fix the deadlock by removing the second call, but deadlocks usually happen from one of two subtle things:

An implementation bug where a Lock is not released properly
A design issue where a utility function needs to be called by functions that might or might not already have the Lock

The first situation happens sometimes, but using a Lock as a context manager greatly reduces how often. It is recommended to write code whenever possible to make use of context managers, as they help to avoid situations where an exception skips you over the .release() call.

The design issue can be a bit trickier in some languages. Thankfully, Python threading has a second object, called RLock, that is designed for just this situation. It allows a thread to .acquire() an RLock multiple times before it calls .release(). That thread is still required to call .release() the same number of times it called .acquire(), but it should be doing that anyway.

Lock and RLock are two of the basic tools used in threaded programming to prevent race conditions. There are a few other that work in different ways. Before you look at them, let’s shift to a slightly different problem domain.

Producer-Consumer Threading

The Producer-Consumer Problem is a standard computer science problem used to look at threading or process synchronization issues. You’re going to look at a variant of it to get some ideas of what primitives the Python threading module provides.

For this example, you’re going to imagine a program that needs to read messages from a network and write them to disk. The program does not request a message when it wants. It must be listening and accept messages as they come in. The messages will not come in at a regular pace, but will be coming in bursts. This part of the program is called the producer.

On the other side, once you have a message, you need to write it to a database. The database access is slow, but fast enough to keep up to the average pace of messages. It is not fast enough to keep up when a burst of messages comes in. This part is the consumer.

In between the producer and the consumer, you will create a Pipeline that will be the part that changes as you learn about different synchronization objects.

That’s the basic layout. Let’s look at a solution using Lock. It doesn’t work perfectly, but it uses tools you already know, so it’s a good place to start.

Producer-Consumer Using `Lock`

Since this is an article about Python threading, and since you just read about the Lock primitive, let’s try to solve this problem with two threads using a Lock or two.

The general design is that there is a producer thread that reads from the fake network and puts the message into a Pipeline:

SENTINEL = object()

def producer(pipeline):
    """Pretend we're getting a message from the network."""
    for index in range(10):
        message = random.randint(1, 101)
        logging.info("Producer got message: %s", message)
        pipeline.set_message(message, "Producer")

    # Send a sentinel message to tell consumer we're done
    pipeline.set_message(SENTINEL, "Producer")

To generate a fake message, the producer gets a random number between one and one hundred. It calls .set_message() on the pipeline to send it to the consumer.

The producer also uses a SENTINEL value to signal the consumer to stop after it has sent ten values. This is a little awkward, but don’t worry, you’ll see ways to get rid of this SENTINEL value after you work through this example.

On the other side of the pipeline is the consumer:

def consumer(pipeline):
    """Pretend we're saving a number in the database."""
    message = 0
    while message is not SENTINEL:
        message = pipeline.get_message("Consumer")
        if message is not SENTINEL:
            logging.info("Consumer storing message: %s", message)

The consumer reads a message from the pipeline and writes it to a fake database, which in this case is just printing it to the display. If it gets the SENTINEL value, it returns from the function, which will terminate the thread.

Before you look at the really interesting part, the Pipeline, here’s the __main__ section, which spawns these threads:

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")
    # logging.getLogger().setLevel(logging.DEBUG)

    pipeline = Pipeline()
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        executor.submit(producer, pipeline)
        executor.submit(consumer, pipeline)

This should look fairly familiar as it’s close to the __main__ code in the previous examples.

Remember that you can turn on DEBUG logging to see all of the logging messages by uncommenting this line:

# logging.getLogger().setLevel(logging.DEBUG)

It can be worthwhile to walk through the DEBUG logging messages to see exactly where each thread acquires and releases the locks.

Now let’s take a look at the Pipeline that passes messages from the producer to the consumer:

class Pipeline:
    """
    Class to allow a single element pipeline between producer and consumer.
    """
    def __init__(self):
        self.message = 0
        self.producer_lock = threading.Lock()
        self.consumer_lock = threading.Lock()
        self.consumer_lock.acquire()

    def get_message(self, name):
        logging.debug("%s:about to acquire getlock", name)
        self.consumer_lock.acquire()
        logging.debug("%s:have getlock", name)
        message = self.message
        logging.debug("%s:about to release setlock", name)
        self.producer_lock.release()
        logging.debug("%s:setlock released", name)
        return message

    def set_message(self, message, name):
        logging.debug("%s:about to acquire setlock", name)
        self.producer_lock.acquire()
        logging.debug("%s:have setlock", name)
        self.message = message
        logging.debug("%s:about to release getlock", name)
        self.consumer_lock.release()
        logging.debug("%s:getlock released", name)

Woah! That’s a lot of code. A pretty high percentage of that is just logging statements to make it easier to see what’s happening when you run it. Here’s the same code with all of the logging statements removed:

class Pipeline:
    """
    Class to allow a single element pipeline between producer and consumer.
    """
    def __init__(self):
        self.message = 0
        self.producer_lock = threading.Lock()
        self.consumer_lock = threading.Lock()
        self.consumer_lock.acquire()

    def get_message(self, name):
        self.consumer_lock.acquire()
        message = self.message
        self.producer_lock.release()
        return message

    def set_message(self, message, name):
        self.producer_lock.acquire()
        self.message = message
        self.consumer_lock.release()

That seems a bit more manageable. The Pipeline in this version of your code has three members:

.message stores the message to pass.
.producer_lock is a threading.Lock object that restricts access to the message by the producer thread.
.consumer_lock is also a threading.Lock that restricts access to the message by the consumer thread.

__init__() initializes these three members and then calls .acquire() on the .consumer_lock. This is the state you want to start in. The producer is allowed to add a new message, but the consumer needs to wait until a message is present.

.get_message() and .set_messages() are nearly opposites. .get_message() calls .acquire() on the consumer_lock. This is the call that will make the consumer wait until a message is ready.

Once the consumer has acquired the .consumer_lock, it copies out the value in .message and then calls .release() on the .producer_lock. Releasing this lock is what allows the producer to insert the next message into the pipeline.

Before you go on to .set_message(), there’s something subtle going on in .get_message() that’s pretty easy to miss. It might seem tempting to get rid of message and just have the function end with return self.message. See if you can figure out why you don’t want to do that before moving on.

Here’s the answer. As soon as the consumer calls .producer_lock.release(), it can be swapped out, and the producer can start running. That could happen before .release() returns! This means that there is a slight possibility that when the function returns self.message, that could actually be the next message generated, so you would lose the first message. This is another example of a race condition.

Moving on to .set_message(), you can see the opposite side of the transaction. The producer will call this with a message. It will acquire the .producer_lock, set the .message, and the call .release() on then consumer_lock, which will allow the consumer to read that value.

Let’s run the code that has logging set to WARNING and see what it looks like:

$ ./prodcom_lock.py
Producer got data 43
Producer got data 45
Consumer storing data: 43
Producer got data 86
Consumer storing data: 45
Producer got data 40
Consumer storing data: 86
Producer got data 62
Consumer storing data: 40
Producer got data 15
Consumer storing data: 62
Producer got data 16
Consumer storing data: 15
Producer got data 61
Consumer storing data: 16
Producer got data 73
Consumer storing data: 61
Producer got data 22
Consumer storing data: 73
Consumer storing data: 22

At first, you might find it odd that the producer gets two messages before the consumer even runs. If you look back at the producer and .set_message(), you will notice that the only place it will wait for a Lock is when it attempts to put the message into the pipeline. This is done after the producer gets the message and logs that it has it.

When the producer attempts to send this second message, it will call .set_message() the second time and it will block.

The operating system can swap threads at any time, but it generally lets each thread have a reasonable amount of time to run before swapping it out. That’s why the producer usually runs until it blocks in the second call to .set_message().

Once a thread is blocked, however, the operating system will always swap it out and find a different thread to run. In this case, the only other thread with anything to do is the consumer.

The consumer calls .get_message(), which reads the message and calls .release() on the .producer_lock, thus allowing the producer to run again the next time threads are swapped.

Notice that the first message was 43, and that is exactly what the consumer read, even though the producer had already generated the 45 message.

While it works for this limited test, it is not a great solution to the producer-consumer problem in general because it only allows a single value in the pipeline at a time. When the producer gets a burst of messages, it will have nowhere to put them.

Let’s move on to a better way to solve this problem, using a Queue.

Producer-Consumer Using `Queue`

If you want to be able to handle more than one value in the pipeline at a time, you’ll need a data structure for the pipeline that allows the number to grow and shrink as data backs up from the producer.

Python’s standard library has a queue module which, in turn, has a Queue class. Let’s change the Pipeline to use a Queue instead of just a variable protected by a Lock. You’ll also use a different way to stop the worker threads by using a different primitive from Python threading, an Event.

Let’s start with the Event. The threading.Event object allows one thread to signal an event while many other threads can be waiting for that event to happen. The key usage in this code is that the threads that are waiting for the event do not necessarily need to stop what they are doing, they can just check the status of the Event every once in a while.

The triggering of the event can be many things. In this example, the main thread will simply sleep for a while and then .set() it:

 1 if __name__ == "__main__":
 2     format = "%(asctime)s: %(message)s"
 3     logging.basicConfig(format=format, level=logging.INFO,
 4                         datefmt="%H:%M:%S")
 5     # logging.getLogger().setLevel(logging.DEBUG)
 6 
 7     pipeline = Pipeline()
 8     event = threading.Event()
 9     with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
10         executor.submit(producer, pipeline, event)
11         executor.submit(consumer, pipeline, event)
12 
13         time.sleep(0.1)
14         logging.info("Main: about to set event")
15         event.set()

The only changes here are the creation of the event object on line 6, passing the event as a parameter on lines 8 and 9, and the final section on lines 11 to 13, which sleep for a second, log a message, and then call .set() on the event.

The producer also did not have to change too much:

 1 def producer(pipeline, event):
 2     """Pretend we're getting a number from the network."""
 3     while not event.is_set():
 4         message = random.randint(1, 101)
 5         logging.info("Producer got message: %s", message)
 6         pipeline.set_message(message, "Producer")
 7 
 8     logging.info("Producer received EXIT event. Exiting")

It now will loop until it sees that the event was set on line 3. It also no longer puts the SENTINEL value into the pipeline.

consumer had to change a little more:

 1 def consumer(pipeline, event):
 2     """Pretend we're saving a number in the database."""
 3     while not event.is_set() or not pipeline.empty():
 4         message = pipeline.get_message("Consumer")
 5         logging.info(
 6             "Consumer storing message: %s  (queue size=%s)",
 7             message,
 8             pipeline.qsize(),
 9         )
10 
11     logging.info("Consumer received EXIT event. Exiting")

While you got to take out the code related to the SENTINEL value, you did have to do a slightly more complicated while condition. Not only does it loop until the event is set, but it also needs to keep looping until the pipeline has been emptied.

Making sure the queue is empty before the consumer finishes prevents another fun issue. If the consumer does exit while the pipeline has messages in it, there are two bad things that can happen. The first is that you lose those final messages, but the more serious one is that the producer can get caught on the producer_lock and never return.

This happens if the event gets triggered after the producer has checked the .is_set() condition but before it calls pipeline.set_message().

If that happens, it’s possible for the producer to wake up and exit with the .producer_lock still being held. The producer will then try to .acquire() the .producer_lock, but the consumer has exited and will never .release() it.

The rest of the consumer should look familiar.

The Pipeline has changed dramatically, however:

 1 class Pipeline(queue.Queue):
 2     def __init__(self):
 3         super().__init__(maxsize=10)
 4 
 5     def get_message(self, name):
 6         logging.debug("%s:about to get from queue", name)
 7         value = self.get()
 8         logging.debug("%s:got %d from queue", name, value)
 9         return value
10 
11     def set_message(self, value, name):
12         logging.debug("%s:about to add %d to queue", name, value)
13         self.put(value)
14         logging.debug("%s:added %d to queue", name, value)

You can see that Pipeline is a subclass of queue.Queue. Queue has an optional parameter when initializing to specify a maximum size of the queue.

If you give a positive number for maxsize, it will limit the queue to that number of elements, causing .put() to block until there are fewer than maxsize elements. If you don’t specify maxsize, then the queue will grow to the limits of your computer’s memory.

.get_message() and .set_message() got much smaller. They basically wrap .get() and .put() on the Queue. You might be wondering where all of the locking code that prevents the threads from causing race conditions went.

The core devs who wrote the standard library knew that a Queue is frequently used in multi-threading environments and incorporated all of that locking code inside the Queue itself. Queue is thread-safe.

Running this program looks like the following:

$ ./prodcom_queue.py
Producer got message: 32
Producer got message: 51
Producer got message: 25
Producer got message: 94
Producer got message: 29
Consumer storing message: 32 (queue size=3)
Producer got message: 96
Consumer storing message: 51 (queue size=3)
Producer got message: 6
Consumer storing message: 25 (queue size=3)
Producer got message: 31

[many lines deleted]

Producer got message: 80
Consumer storing message: 94 (queue size=6)
Producer got message: 33
Consumer storing message: 20 (queue size=6)
Producer got message: 48
Consumer storing message: 31 (queue size=6)
Producer got message: 52
Consumer storing message: 98 (queue size=6)
Main: about to set event
Producer got message: 13
Consumer storing message: 59 (queue size=6)
Producer received EXIT event. Exiting
Consumer storing message: 75 (queue size=6)
Consumer storing message: 97 (queue size=5)
Consumer storing message: 80 (queue size=4)
Consumer storing message: 33 (queue size=3)
Consumer storing message: 48 (queue size=2)
Consumer storing message: 52 (queue size=1)
Consumer storing message: 13 (queue size=0)
Consumer received EXIT event. Exiting

If you read through the output in my example, you can see some interesting things happening. Right at the top, you can see the producer got to create five messages and place four of them on the queue. It got swapped out by the operating system before it could place the fifth one.

The consumer then ran and pulled off the first message. It printed out that message as well as how deep the queue was at that point:

Consumer storing message: 32 (queue size=3)

This is how you know that the fifth message hasn’t made it into the pipeline yet. The queue is down to size three after a single message was removed. You also know that the queue can hold ten messages, so the producer thread didn’t get blocked by the queue. It was swapped out by the OS.

Note: Your output will be different. Your output will change from run to run. That’s the fun part of working with threads!

As the program starts to wrap up, can you see the main thread generating the event which causes the producer to exit immediately. The consumer still has a bunch of work do to, so it keeps running until it has cleaned out the pipeline.

Try playing with different queue sizes and calls to time.sleep() in the producer or the consumer to simulate longer network or disk access times respectively. Even slight changes to these elements of the program will make large differences in your results.

This is a much better solution to the producer-consumer problem, but you can simplify it even more. The Pipeline really isn’t needed for this problem. Once you take away the logging, it just becomes a queue.Queue.

Here’s what the final code looks like using queue.Queue directly:

import concurrent.futures
import logging
import queue
import random
import threading
import time

def producer(queue, event):
    """Pretend we're getting a number from the network."""
    while not event.is_set():
        message = random.randint(1, 101)
        logging.info("Producer got message: %s", message)
        queue.put(message)

    logging.info("Producer received event. Exiting")

def consumer(queue, event):
    """Pretend we're saving a number in the database."""
    while not event.is_set() or not queue.empty():
        message = queue.get()
        logging.info(
            "Consumer storing message: %s (size=%d)", message, queue.qsize()
        )

    logging.info("Consumer received event. Exiting")

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")

    pipeline = queue.Queue(maxsize=10)
    event = threading.Event()
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        executor.submit(producer, pipeline, event)
        executor.submit(consumer, pipeline, event)

        time.sleep(0.1)
        logging.info("Main: about to set event")
        event.set()

That’s easier to read and shows how using Python’s built-in primitives can simplify a complex problem.

Lock and Queue are handy classes to solve concurrency issues, but there are others provided by the standard library. Before you wrap up this tutorial, let’s do a quick survey of some of them.

Threading Objects

There are a few more primitives offered by the Python threading module. While you didn’t need these for the examples above, they can come in handy in different use cases, so it’s good to be familiar with them.

Semaphore

The first Python threading object to look at is threading.Semaphore. A Semaphore is a counter with a few special properties. The first one is that the counting is atomic. This means that there is a guarantee that the operating system will not swap out the thread in the middle of incrementing or decrementing the counter.

The internal counter is incremented when you call .release() and decremented when you call .acquire().

The next special property is that if a thread calls .acquire() when the counter is zero, that thread will block until a different thread calls .release() and increments the counter to one.

Semaphores are frequently used to protect a resource that has a limited capacity. An example would be if you have a pool of connections and want to limit the size of that pool to a specific number.

Timer

A threading.Timer is a way to schedule a function to be called after a certain amount of time has passed. You create a Timer by passing in a number of seconds to wait and a function to call:

t = threading.Timer(30.0, my_function)

You start the Timer by calling .start(). The function will be called on a new thread at some point after the specified time, but be aware that there is no promise that it will be called exactly at the time you want.

If you want to stop a Timer that you’ve already started, you can cancel it by calling .cancel(). Calling .cancel() after the Timer has triggered does nothing and does not produce an exception.

A Timer can be used to prompt a user for action after a specific amount of time. If the user does the action before the Timer expires, .cancel() can be called.

Barrier

A threading.Barrier can be used to keep a fixed number of threads in sync. When creating a Barrier, the caller must specify how many threads will be synchronizing on it. Each thread calls .wait() on the Barrier. They all will remain blocked until the specified number of threads are waiting, and then the are all released at the same time.

Remember that threads are scheduled by the operating system so, even though all of the threads are released simultaneously, they will be scheduled to run one at a time.

One use for a Barrier is to allow a pool of threads to initialize themselves. Having the threads wait on a Barrier after they are initialized will ensure that none of the threads start running before all of the threads are finished with their initialization.

Conclusion: Threading in Python

You’ve now seen much of what Python threading has to offer and some examples of how to build threaded programs and the problems they solve. You’ve also seen a few instances of the problems that arise when writing and debugging threaded programs.

If you’d like to explore other options for concurrency in Python, check out Speed Up Your Python Program With Concurrency.

If you’re interested in doing a deep dive on the asyncio module, go read Async IO in Python: A Complete Walkthrough.

Whatever you do, you now have the information and confidence you need to write programs using Python threading!

Special thanks to reader JL Diaz for helping to clean up the introduction.

Take the Quiz: Test your knowledge with our interactive “Python Threading” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Take the Quiz »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Intro to Object-Oriented Programming (OOP) in Python

2019-03-21T14:00:00+00:00

Object-oriented programming is one of the biggest and most important subjects in all of programming. This series will provide you with a basic conceptual understanding of Object-Oriented Programming so you can take your Python programming skills to the next level.

You’ll learn how to define custom types using classes, and how to instantiate those classes into python objects that can be used throughout your program.

Finally, you’ll learn how classes can subclass one another with brief introduction to inheritance, which allows you to write maintainable and less redundant Python code.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

13 Project Ideas for Intermediate Python Developers

2019-03-20T14:00:00+00:00

Learning the basics of Python is a wonderful experience. But the euphoria of just learning can be replaced by the hunger for hands-on projects. It’s normal to want to build projects, hence the need for project ideas.

The problem though is that some projects are either too simple for an intermediate Python developer or too hard. This article will suggest projects you can work on as an intermediate Python developer. These project ideas will provide the appropriate level of challenge for you.

In this article, you’ll learn:

The importance of building projects
The major platforms you can build projects for
Thirteen project ideas you can work on
Some tips for working on projects

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

The Importance of Building Projects

Working on projects is vital to pushing your career as a Python developer forward. They make you apply the skills and knowledge you’re acquiring.

Projects can help you:

Build confidence: You will believe more in your ability to create software regardless the level of complexity.
Explore other technologies: You will learn about other technologies needed in building a complete product such as databases, servers, and other languages.
Understand programming concepts better: You will learn to write code better and understand concepts such as design patterns and object-oriented programming.
Experience a complete software development life cycle: You will learn how to plan before writing code, manage the coding process and update software.

There is a lot to gain from building projects as a Python developer.

Choosing a Project Platform

You need to build your software to run on a platform so that people who lack certain technical knowledge can use your software. The web, desktop, and command-line are the three major platforms you’ll want to build your projects for.

Web

Web applications are applications that run on the web, they can be accessed on any device without being downloaded, provided there is access to the internet. If you want your projects to be accessible by everyone with internet access, it needs to be a web application.

A web application has a back end and front end. The back end is the part where the business logic is: your back-end code will manipulate and store data. The front end is the interface of the application: your front-end code will determine the look of a web application.

As an intermediate Python developer, your major focus will be the back-end code. However, the front-end code is important too, so you will need some knowledge of HTML, CSS, and maybe JavaScript to create a simple-looking interface. Just the basics will be enough.

Another option is to use Python for both the front end and back end. Thanks to the anvil library, which eliminates the need for HTML, CSS, and JavaScript, you can focus on Python code alone.

You can build web applications with Python through web frameworks such as django and flask. The list of frameworks for building web applications using Python is long. There are plenty to choose from, but django and flask remain the most popular web frameworks.

Desktop GUI

Every time you perform a task on your PC, be it a desktop or laptop, it is through an application. As an intermediate Python developer, you can make your own desktop applications.

You do not have to learn any front-end technology to create your own Graphical User Interface (GUI) applications, as you saw with web applications. You can build all the parts using Python.

There are frameworks for building your desktop applications. PySimpleGUI is one of them, and it’s pretty user-friendly for an intermediate Python developer.

An advanced GUI framework like PyQt5 is quite powerful, but it may have a steep learning curve.

The software you create for the Desktop GUI is able to work on any of the Windows, Linux, or Mac operating systems. All you have to do after creating the project is compile it to an executable for your operating system of choice.

Command-Line

Command-line applications are those applications that work in a console window. This is the command prompt on Windows and the Terminal on Linux and Mac.

You’d click to use a web or GUI application, but you’d type in commands for command-line applications. Users of command-line applications need to have some technical knowledge since they’ll need to use commands.

Command-line applications may not be as beautiful or easy to use as web or GUI applications, but that doesn’t make them less powerful than web or GUI applications.

You can improve the look of your command-line applications by applying colors to the text. There are libraries you can use for coloring, such as colorama and colored. You can spice things up and use some color.

You can use frameworks such as docopt, argparse, and click to build your applications.

Web Project Ideas

In this section, you’ll see project ideas for the web. These project ideas can be classified as utility and education tools.

Here are the project ideas:

Content Aggregator
Regex Query Tool
URL Shortener
Post-It Note
Quiz Application

Content Aggregator

Content is king. It exists everywhere on the web, from blogs to social media platforms. To keep up, you need to search for new information on the internet constantly. One way to stay updated is to check all the sites manually to see what the new posts are. But this is time consuming and quite tiring.

This is where the content aggregator comes in: A content aggregator fetches information from various places online and gathers all of that information in one place. Therefore, you don’t have to visit multiple sites to get the latest info: one website is enough.

With the content aggregator, all of the latest information can be gotten from one site that aggregates all the content. People can see the posts that interest them and can decide to find out more about them without traipsing all over the internet.

Examples of Content Aggregators

Here are some implementations of the Content Aggregator idea:

AllTop
Hvper

Technical Details

The main objective of this project idea is to aggregate content. First, you need to know what sites you’ll want the Content Aggregator to get content from. Then, you can use libraries such as requests for sending HTTP requests and BeautifulSoup to parse and scrape the necessary content from the sites.

Your application can implement its content aggregation as a background process. Libraries such as celery or apscheduler can help with that. You can try out apscheduler. It’s great for small background processes.

After scraping content from various sites, you’ll need to save it somewhere. So, you’ll use a database to save the scraped content.

Extra Challenge

For a tougher challenge, you can add more websites. This will help you learn how to study and extract information from websites.

You can also have users subscribe to certain sites that you aggregate. Then, at the end of the day, the content aggregator will send the articles for that day to the email address of the user.

Regex Query Tool

You and I deal with text daily. This article, which is also text, has a structure. This makes it easier for you to understand. Sometimes, you need to find certain information in text, and using the regular search tool in text editors can be ineffective.

This is where the Regex Query Tool comes in. A regex is a set of strings, so the regex query tool will check for the validity of the queries. When the regex matches patterns in the text, it tells the user and highlights the matched patterns. So, your Regex Query Tool will check the validity of the regex strings passed in by the user.

With the Regex Query Tool, users can quickly check the validity of their regex strings on the web. This makes it easier for them, instead of having to check the strings with a text editor.

Examples of Regex Query Tools

Here are some implementations of the Regex Query Tool idea:

Technical Details

The main objective of this type of project is to tell the user the validity of the inputted query strings. You can make it give a positive or negative response such as Query String Is Valid and Query String Is Invalid, implementing the positive response in green and the negative in red.

You don’t have to implement the query tool from scratch. You can use Python’s standard re library, which you can use to run the query strings on the inputted text. The re library will return None when the query string matches nothing, and it’ll return the matched strings when positive.

Some users may not understand regex fully, so you can make a page to explain how regex works. You can make documentation that is interesting enough to keep the users excited about learning and understanding regex.

Extra Challenge

Making a project that just returns the validity of the regex is fine. But you can also add a replacement feature. This means the application will check for the validity of the regex and also allow users to replace the matched strings with something else. So the tool is no longer a find tool but also a replace tool.

URL Shortener

URLs can be extremely long and not user-friendly. When people share links or even try to remember a URL, it’s difficult because most URLs are filled with more difficult characters and don’t form meaningful words.

This is where the URL Shortener comes in. A URL Shortener reduces the characters or letters in a URL, making them easier to read and remember. A URL like xyz.com/wwryb78&svnhkn%sghq?sfiyh can be shortened to xyz.com/piojwr.

With the URL Shortener, URLs become a joy to work with.

Examples of URL Shorteners

Here are some implementations of the URL Shortener idea:

Technical Details

The main objective of this project idea is to shorten URLs. The main task the application will accomplish is to shorten URLs and then redirect users to the original URL when the shortened URL is visited.

In the application, the users will input the original URL, and they will get the new, shortened URL as the result. To do this, you can use a combination of the random and string modules to generate the characters for the shortened URL.

Since users will visit the shortened URL days, months, or even years after, you’ll need to save the original and shortened URLs in a database. When a request comes in, the application checks if the URL exists and redirects to the original, or else it redirects to a 404 page.

Extra Challenge

Generating a shortened URL with random characters makes for a better URL than the long, random ones. But, you can make the result better for the users. You can add a feature to customize URLs, so the users can customize the generated URLs themselves.

Without a doubt, a custom xyz.com/mysite URL is better than a randomly generated xyz.com/piojwr URL.

Post-It Note

It’s human to have many thoughts and ideas in a day, but it’s also human to forget. One way to work around forgetting things is to jot them down before they disappear into thin air. While some of forgotten thoughts and ideas may be trivial, some can be quite powerful.

This is where a Post-It note comes in: A Post-It note is a small paper with low-tack adhesive at the back, making it attachable to surfaces such as documents, walls. Post-It notes make it easier to jot things down. The Post-It note project idea is something similar. It allows users to jot things down, making them accessible anywhere, since it’s a web application.

With the Post-It note, people can now jot things down anywhere, without the fear of forgetting things or misplacing the notes—which is a possibility with physical notes.

Examples of Post-It Notes

Here are some implementations of the Post-It Note idea:

Technical Details

The main objective of this project is to allow users to jot down thoughts. This means that each user will have their own notes, so the application will need to have an account creation feature. This ensures that the notes of each user remain private to them.

django comes with a user authentication system, so it may be a good choice. You can use other frameworks like bottle or flask, but you’ll have to implement the user authentication system on your own.

Since users may need to separate their notes under different sections, implementing a feature to allow users to categorize their notes will make the application more useful.

As an example, you may need to have notes on algorithms and data structures, so you’ll need to be able to separate the notes in those categories.

You’ll need to store the information and notes of each user, so a database becomes an essential part of this project. The MySQLdb module can be used if you want to use a MySQL database or the psycopg2 module for a PostgreSQL database. There are other modules you can use, but it all depends on the database you choose to use.

Extra Challenge

Since it’s human for users to forget their ideas, it’s also human for them to forget that they even made a note somewhere. You can add a feature to remind users of their notes. This feature will allow users to set a time for the reminder, so the application will send the reminder to the users when it’s time, by email.

Quiz Application

Knowledge is power. There are so many things in the world to learn, and quizzes help in testing the understanding of those concepts. You, as an intermediate Python developer, do not have to understand everything about the language. Taking tests is one way to find out things you don’t fully understand.

This is where the Quiz Application comes in. The Quiz Application will present questions to the users and expect the right answers to those questions. Think of the Quiz Application as a kind of questionnaire.

With the Quiz Application, special users you can call administrators will be allowed to create tests, so regular users can answer the questions and test their understanding of the topics in the quiz.

Examples of Quiz Applications

Here are some implementations of the Quiz Application idea:

Technical Details

The main objective of this project is to set quizzes and have people answer them. Therefore, users should be able to set questions, and other users should be able to answer those questions. The application will then display the final score and the right answers.

If you want users to be able to have a record of their scores, you may have to implement an account creation feature.

Users creating the tests should be able to create tests with the questions and answers by simply uploading a text file. The text file will have a format that you can decide, so the application can convert from a file to a quiz.

You’ll need to implement a database for this project. The database will store the questions, possible answers, correct answers, and the scores for each user.

Extra Challenge

For more of a challenge, you can allow users to add timers to the quizzes. This way, the creators of a quiz can determine how many seconds or minutes a user should spend on each question in the quiz.

It would be great to also have a quiz-sharing feature, so users can share interesting quizzes with their friends on other platforms.

GUI Project Ideas

In this section, you’ll see project ideas for Graphical User Interfaces. These project ideas can be classified as entertainment, finance, and utility tools.

Here’s are the project ideas:

MP3 Player
Alarm Tool
File Manager
Expense Tracker

MP3 Player

Audio is as important as text today if not more important. Since audio files are digital files, you’ll need a tool that can play them. Without a player, you’ll never be able to listen to the contents of an audio file.

This is where the MP3 Player comes in. The MP3 Player is a device for playing MP3s and other digital audio files. This MP3 Player GUI project idea attempts to emulate the physical MP3 Player. You can build software that allows you play an MP3 files on your desktop or laptop computer.

When you are done building the MP3 Player project, users can play their MP3 files and other digital audio files without having to purchase a physical MP3 Player. They’ll be able to play the MP3 files using their computers.

Examples of MP3 Players

Here are some implementations of the MP3 Player idea:

Technical Details

The main objective of this project is to allow users to play MP3 and digital audio files. To be engaging for users, the application has to have a simple but beautiful user interface.

You can have an interface for listing the available MP3 files. You can also give users the option to list other digital audio files that are not MP3.

The users will also expect the MP3 Player to have an interface that shows information on the file that is playing. Some of the information you can include are the name of the file, its length, the amount played, and the amount not played, in minutes and seconds.

Python has libraries that can play audio files, such as pygame, which allows you to work with multimedia files in few lines of code. You can also check out pymedia and simpleaudio.

These libraries can handle a lot of digital audio files. They can handle other file types, not just the MP3 files.

You can also implement a feature that allows users to create a playlist. To do this, you’ll need a database to store information on the created playlists. Python’s sqlite3 module allows you to use the SQLite database.

The SQLite database is a better option in this case, because it is file based and easier to set up than other SQL databases. While SQLite is file based, it is better for saving data than a regular file.

Extra Challenge

For a more exciting challenge, you can add a feature to allow the MP3 player to repeat currently playing files or even shuffle the list of files to be played.

It’s also possible to implement a feature that allows users to increase and decrease the playing speed of the audio file. Users will find this interesting, as they’ll be able to play files at a slower or faster pace than usual.

Alarm Tool

As they say, “Time and tide wait for no man.” But with a lot of things going on in our lives, it’s difficult to not lose track of time. To be able to keep track of time, a reminder is needed.

This is where the Alarm Tool comes in. An alarm is a device that gives an audio or visual signal about a certain condition. This Alarm Tool project idea is an attempt to build an alarm as software. The Alarm Tool gives an audio signal when a certain condition is met. The set time is the certain condition in this case.

With the Alarm Tool, users can set alarms to remind them of things at certain times of the day. The Alarm Tool project will work from the user’s laptop or desktop device, so they do not have to purchase a physical timer.

Examples of Alarm Tools

Here are some implementations of the Alarm Tool idea:

Technical Details

The main objective of this project is to activate audio signals at certain times of the day. So, timing and the audio signal to be played are the most important parts of the Alarm Tool.

The Alarm Tool should allow users to create, edit, and delete alarms. It should also have an interface that lists all the alarms, provided they have not being deleted by the user. So, it should list the active and inactive alarms.

Since it is an alarm, the application has to play tones at the set time. There are libraries for playing audio, like the pygame library.

In your code logic, the application has to keep checking for set alarm times. When the time is reached, it triggers a function to play the alarm tone.

Since the application will check for set alarm times, it means the application has to save the alarms in a database. The database should store things like the alarm date, time, and tone location.

Extra Challenge

As an extra feature, you can allow users to set recurring alarms. They’ll be able to set alarms that will ring at a certain time on certain days of the week, every week. As an example, an alarm can be set at 2:00 PM every Monday.

You can also add a snooze feature, so your users can snooze alarms instead of only dismissing them.

File Manager

The number of files on the personal computer of an average PC user is pretty high. If all of those files were placed in a single directory, it would be difficult to navigate and find files or directories. So, there is a need to arrange the files and manage them properly.

This is where a file manager comes in. A file manager allows users to manage files and directories through a user interface. While files can be managed through the command-line, not all users know how to do that.

With a file manager, users can arrange, access, and administer their files and directories properly without knowing how to use the command line. Some of the tasks a file manager allows users to perform includes copying, moving, and renaming files or directories.

Examples of File Manager Tools

Here are some implementations of the File Manager idea:

Technical Details

The main objective of the file manager project is to give users an interface to manage their files. Users want a file manager that has a file management tool that looks good and is easy to use.

You can use the PySimpleGUI library to create unique user interfaces with a powerful widget, without having to deal with a lot of complexity.

Your users should be able to perform simple tasks like creating new directories or empty text files. They should also be able to copy and move files or directories.

The sys, os, and shutil libraries will be quite useful for this project, as they can be used to execute actions on the files in the background, while the user clicks away.

The grid and list views are popular views today, so you can implement both in the application. This gives the user the option to choose which view option is suitable for them.

Extra Challenge

To make the file manager a bit more advanced, you can implement a search feature. So users can search for files and directories without having to find them manually.

You can also implement a sort feature. This will allow users to sort files according to different orders, such as time, alphabetical order, or size.

Expense Tracker

We have daily expenses, from groceries to clothing to bills. There are so many expenses that it’s normal to lose track of them and keep spending till we’re almost out of cash. A tracker can help people watch their expenses.

This is where the expense tracker comes in. An expense tracker is a software tool that allows users to keep track of their expenses. It can also analyze the expenses, depending on how advanced it is, but let’s keep it simple for now.

With the expense tracker, users can set a budget and track their spending so as to make better financial decisions.

Examples of Expense Trackers

Here are some implementations of the Expense Tracker idea:

Technical Details

The main objective of this project is to keep track of the user’s expenses. Some statistical analysis has to be done to be able to give users correct information on their expenses and help them spend better.

While tracking the expenses is the key thing, a good interface is also important. With PySimpleGUI, you can create a unique interface to improve the experience of the users.

PyData libraries such as pandas and matplotlib can be helpful for building the expense tracker.

The pandas library can be used for the data analysis, and the matplotlib library can be used for plotting graphs. Graphs will give the users a visual representation of their expenses, and a visual representation is usually easier to understand.

The application will receive data from the users. The data here is the inputted expenses. So, you’ll have to store the expenses in a database. The SQLite database is a good database choice for this project since it can be set up quickly. You can use sqlite3 module for the SQLite database.

Extra Challenge

For your users to benefit from this project, they’ll have to input their expenses regularly, which might slip their mind. It could be useful for you to implement a reminder feature. So the application will send a notification at certain times of the day or the week, reminding them to make use of the expense tracker.

Command-Line Project Ideas

In this section, you’ll see project ideas for the command-line. The project ideas discussed can be classified as utility tools.

Here’s are the project ideas:

Contact Book
File Connectivity Checker
Bulk File Rename Tool
Directory Tree Generator

Contact Book

We come across lots of people daily. We make acquaintances and friends. We get their contacts to keep in touch later on. Sadly, keeping the received contact details can be hard. One way to do this is to write the contact details down. But this is not secure as the physical book can easily be lost.

This is where the Contact Book project comes in. A contact book is a tool for saving a contact’s details, such as name, address, phone number, and email address. With this contact book project, you can build a software tool that people can use to save and find contact details.

With the contact book project idea, users can save their contacts with less risk of losing the saved contact details. It’ll always be accessible from their computer, through the command-line.

Examples of Contact Book Tools

There are Contact Book applications, but it’s rare to find command-line Contact Book products, as most are web, mobile, or GUI applications.

Here are some implementations of the Contact Book idea:

Technical Details

The main objective of this project is to save contact details. It’s important that you set up the commands users can use to enter the contact details. You can use the argparse or click command-line frameworks. They abstract a lot of complex stuff, so you only have to focus on the logic to be run when executing commands.

Some features you should implement include the commands to delete contacts, update contact information, and list saved contacts. You can also allow users to list contacts using different parameters, such as alphabetical order or contact creation date.

Since it’s a command-line project, the SQLite database will be fine for saving contacts. SQLite is user-friendly to set up. You may save the contact details in a file, but a file will not offer the benefits you can gain from using SQLite, such as performance and security.

To use the SQLite database in this project, the Python sqlite3 module will be very useful.

Extra Challenge

Remember how the database is stored on the user’s computer? What if something happens, like the user losing their files? It means they’ll also lose the contact details.

You can challenge yourself further and backup the database to an online storage platform. To do this, you can upload the database files to the cloud at certain intervals.

You can also add a command that allows users to backup the database themselves. This way, the user can still have access to the contacts if the database file is lost.

You should note that you may need some form of identification, so the contact book can tell which database file belongs to which user. Implementing a user authentication feature is one way to go about it.

Site Connectivity Checker

When you visit a URL, you expect to get the requested pages on your browser. But this is not always the case. Sometimes, sites can be down, so you won’t get the desired results. Instead, you’ll be presented with error messages. You can keep trying a site that is down, till it comes up and you get the information you need.

This is where the Site Connectivity Checker project comes in. The Site Connectivity Checker visits a URL and returns the status of the URL: it is either live or not. The Site Connectivity Checker will visit the URL at intervals, returning the results of each visit.

Instead of manually visiting a URL, a Site Connectivity Checker can do all of that manual work for you. This way, you’ll only get the results of the check without having to spend time on the browser, waiting for the site to go live.

Examples of Site Connectivity Checkers

Here are some implementations of the Site Connectivity Checker idea:

Technical Details

The main objective of this project is to check the status of sites. So, you need to write code for checking the status of a website.

You can choose to use either TCP or ICMP for your connections. The socket module is one to check out. You can also read Socket Programming in Python (Guide).

Through your chosen framework, be it the docopt, click, or argparse framework, you can add commands to allow users to add and remove sites from the list of sites to be checked.

The users should also be able to start the tool, stop it, and determine the intervals.

Since you’ll have to save the list of files to be checked, you can either save it in a file (just a list of sites) or use a SQLite database through the sqlite3 module.

Extra Challenge

The application can check for the connectivity status of sites and display the results to the command-line. But this will require the user to keep checking the command-line.

You can increase the challenge and implement a notification feature. The notification feature can be a sound played in the background to alert the user when a site’s status changes. You’ll need a database to store the previous status of a site. That’s the only way the tool can tell when the status changes.

Bulk File Rename Tool

Sometimes, you need to name all the files in a directory according to certain conventions. For example, you can name all the files in a directory with File0001.jpg, where the numbers increase based on the number of files in the directory. Doing this manually can be stressful and repetitive.

The Bulk File Rename Tool allows users to rename a large number of files, without having to manually rename files.

This saves users a lot of time. It spares them the trouble of having to do boring repetitive work and make mistakes. With the Bulk File Rename Tool, users can rename files in a couple of seconds without any mistakes.

Examples of Bulk File Rename Tools

Here are some implementations of the Bulk File Rename idea:

Ren
Rename

Technical Details

The main objective of this project idea is to rename files. So, the application needs to find a way to manipulate the target files. The os, sys, and shutil libraries will be useful for a large part of this project.

Your users will be able to rename all the files in the directory, using naming conventions. Therefore, they should be able to pass in the naming convention of choice. The regex module will help match the required naming patterns, if you understand how regex works.

A user may want to pass in a naming convention such as myfiles as part of the commands and expect that the tool renames all the files like myfilesXYZ, where XYZ is a number. They should also be able to choose the directory where the files to be renamed are.

Extra Challenge

The major challenge in this project is to rename all the files in a directory. But users may only need to name a certain number of files. To test your skills, you can implement a feature to allow users to choose the number of files to be renamed, instead of all the files.

Note that renaming only a certain number of files will require the tool to sort the files based on alphabetical order, time of file creation, or file size, depending on the user’s requirements.

Directory Tree Generator

Directories are like family trees: each directory has a particular relationship with other directories. No directories ever stays on its own, except an empty root directory.

When you’re working with files and directories, it is difficult to see the relationship between directories, as you can only see what exists in the current directory. You’re either using a file manager or working from the command-line.

With a Directory Tree Generator, you can see the relationship between files and directories like a tree or a map.

This makes it easier to understand the positioning of files and directories. A directory tree map is important when you’re explaining certain concepts, and a Directory Tree Generator makes it easier to get a visual representation of the file and directory relationships.

Examples of Directory Tree Generators

Here are some implementations of the Directory Tree Generator idea:

Technical Details

The main objective of the Directory Tree Generator is to visualize the relationships between files and directories. The os library can be very useful in listing the files and directories in a chosen directory.

Using a framework such as docopt or argparse helps abstract a lot of stuff, allowing you to focus on writing code for the application’s logic.

In the application’s logic, you can decide how you want to represent files or directories. Using different colors is a brilliant way to go about it. You can use the colored library to print the files and directories in different colors.

You can also decide how deep you’d like the Directory Tree Generator to go. For example, if a directory has children directories twelve levels deep, you may decide to go only as deep as the fifth level.

If you wish, you can also let the user decide how deep they want the Directory Tree Generator to go.

Extra Challenge

Since the results of the generated directory tree will be on the command-line, you can go one step further. You can have the generator create images of the directory tree, so it’ll basically turn the text into an image.

You’ll find the pillow library useful for doing this.

Tips for Working on Projects

Working on projects can be difficult. That’s one reason why motivation and interest in a project will make it a less daunting task.

If you’re interested in a project, you’ll be able to put in the time to research as well as find libraries and tools that will help you with the project.

Here are some tips:

Find a source of motivation
Break the project into subtasks
Do research on the subtasks
Build each subtasks, one step at a time
Reach out for help if you’re stuck
Put the subtasks together

Conclusion

In this article, you’ve seen a couple of Python project ideas you may find interesting.

The project ideas cover a range of platforms. You saw project ideas for the Web, GUI, and Command-line platforms.

You can choose to build a project for different platforms. Using the URL Shortener as an example, you may choose to build one for the Web, GUI, or the Command-line.

Since you’re an intermediate Python developer, these projects can be quite challenging but interesting.

The best way to make a project happen is to just get started. In no time, you’ll be finished and discover how much you’ve benefited from working on a project!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Decorators 101

2019-03-19T14:00:00+00:00

In this course on Python decorators, you’ll learn what they are and how to create and use them. Decorators provide a simple syntax for calling higher-order functions in Python. By definition, a decorator is a function that takes another function and extends the behavior of the latter function without explicitly modifying it.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Build a Python GUI Application With wxPython

2019-03-18T14:00:00+00:00

There are many graphical user interface (GUI) toolkits that you can use with the Python programming language. The big three are Tkinter, wxPython, and PyQt. Each of these toolkits will work with Windows, macOS, and Linux, with PyQt having the additional capability of working on mobile.

A graphical user interface is an application that has buttons, windows, and lots of other widgets that the user can use to interact with your application. A good example would be a web browser. It has buttons, tabs, and a main window where all the content loads.

In this article, you’ll learn how to build a graphical user interface with Python using the wxPython GUI toolkit.

Here are the topics covered:

Getting Started with wxPython
Definition of a GUI
Creating a Skeleton Application
Creating a Working Application

Let’s start learning!

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

Getting Started With wxPython

The wxPython GUI toolkit is a Python wrapper around a C++ library called wxWidgets. The initial release of wxPython was in 1998, so wxPython has been around quite a long time. wxPython’s primary difference from other toolkits, such as PyQt or Tkinter, is that wxPython uses the actual widgets on the native platform whenever possible. This makes wxPython applications look native to the operating system that it is running on.

PyQt and Tkinter both draw their widgets themselves, which is why they don’t always match the native widgets, although PyQt is very close.

This is not to say that wxPython does not support custom widgets. In fact, the wxPython toolkit has many custom widgets included with it, along with dozens upon dozens of core widgets. The wxPython downloads page has a section called Extra Files that is worth checking out.

Here, there is a download of the wxPython Demo package. This is a nice little application that demonstrates the vast majority of the widgets that are included with wxPython. The demo allows a developer to view the code in one tab and run it in a second tab. You can even edit and re-run the code in the demo to see how your changes affect the application.

Installing wxPython

You will be using the latest wxPython for this article, which is wxPython 4, also known as the Phoenix release. The wxPython 3 and wxPython 2 versions are built only for Python 2. When Robin Dunn, the primary maintainer of wxPython, created the wxPython 4 release, he deprecated a lot of aliases and cleaned up a lot of code to make wxPython more Pythonic and easier to maintain.

You will want to consult the following links if you are migrating from an older version of wxPython to wxPython 4 (Phoenix):

Classic vs Phoenix
wxPython Project Phoenix Migration Guide

The wxPython 4 package is compatible with both Python 2.7 and Python 3.

You can now use pip to install wxPython 4, which was not possible in the legacy versions of wxPython. You can do the following to install it on your machine:

$ pip install wxpython

Note: On Mac OS X you will need a compiler installed such as XCode for the install to complete successfully. Linux may also require you to install some dependencies before the pip installer will work correctly.

For example, I needed to install freeglut3-dev, libgstreamer-plugins-base0.10-dev, and libwebkitgtk-3.0-dev on Xubuntu to get it to install.

Fortunately, the error messages that pip displays are helpful in figuring out what is missing, and you can use the prerequisites section on the wxPython Github page to help you find the information you need if you want to install wxPython on Linux.

There are some Python wheels available for the most popular Linux versions that you can find in the Extras Linux section with both GTK2 and GTK3 versions. To install one of these wheels, you would use the following command:

$ pip install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04/ wxPython

Be sure you have modified the command above to match your version of Linux.

Definition of a GUI

As was mentioned in the introduction, a graphical user interface (GUI) is an interface that is drawn on the screen for the user to interact with.

User interfaces have some common components:

Main window
Menu
Toolbar
Buttons
Text Entry
Labels

All of these items are known generically as widgets. There are many other common widgets and many custom widgets that wxPython supports. A developer will take the widgets and arrange them logically on a window for the user to interact with.

Event Loops

A graphical user interface works by waiting for the user to do something. The something is called an event. Events happen when the user types something while your application is in focus or when the user uses their mouse to press a button or other widget.

Underneath the covers, the GUI toolkit is running an infinite loop that is called an event loop. The event loop just waits for events to occur and then acts on those events according to what the developer has coded the application to do. When the application doesn’t catch an event, it effectively ignores that it even happened.

When you are programming a graphical user interface, you will want to keep in mind that you will need to hook up each of the widgets to event handlers so that your application will do something.

There is a special consideration that you need to keep in mind when working with event loops: they can be blocked. When you block an event loop, the GUI will become unresponsive and appear to freeze to the user.

Any process that you launch in a GUI that will take longer than a quarter second should probably be launched as a separate thread or process. This will prevent your GUI from freezing and give the user a better user experience.

The wxPython framework has special thread-safe methods that you can use to communicate back to your application to let it know that the thread is finished or to give it an update.

Let’s create a skeleton application to demonstrate how events work.

Creating a Skeleton Application

An application skeleton in a GUI context is a user interface with widgets that don’t have any event handlers. These are useful for prototyping. You basically just create the GUI and present it to your stakeholders for sign-off before spending a lot of time on the backend logic.

Let’s start by creating a Hello World application with wxPython:

import wx

app = wx.App()
frame = wx.Frame(parent=None, title='Hello World')
frame.Show()
app.MainLoop()

Note: Mac users may get the following message: This program needs access to the screen. Please run with a Framework build of python, and only when you are logged in on the main display of your Mac. If you see this message and you are not running in a virtualenv, then you need to run your application with pythonw instead of python. If you are running wxPython from within a virtualenv, then see the wxPython wiki for the solution.

In this example, you have two parts: wx.App and the wx.Frame. The wx.App is wxPython’s application object and is required for running your GUI. The wx.App starts something called a .MainLoop(). This is the event loop that you learned about in the previous section.

The other piece of the puzzle is wx.Frame, which will create a window for the user to interact with. In this case, you told wxPython that the frame has no parent and that its title is Hello World. Here is what it looks like when you run the code:

Note: The application will look different when you run it on Mac or Windows.

By default, a wx.Frame will include minimize, maximize, and exit buttons along the top. You won’t normally create an application in this manner though. Most wxPython code will require you to subclass the wx.Frame and other widgets so that you can get the full power of the toolkit.

Let’s take a moment and rewrite your code as a class:

import wx

class MyFrame(wx.Frame):    
    def __init__(self):
        super().__init__(parent=None, title='Hello World')
        self.Show()

if __name__ == '__main__':
    app = wx.App()
    frame = MyFrame()
    app.MainLoop()

You can use this code as a template for your application. However, this application doesn’t do very much, so let’s take a moment to learn a little about some of the other widgets you could add.

Widgets

The wxPython toolkit has more than one hundred widgets to choose from. This allows you to create rich applications, but it can also be daunting trying to figure out which widget to use. This is why the wxPython Demo is helpful, as it has a search filter that you can use to help you find the widgets that might apply to your project.

Most GUI applications allow the user to enter some text and press a button. Let’s go ahead and add those widgets:

import wx

class MyFrame(wx.Frame):    
    def __init__(self):
        super().__init__(parent=None, title='Hello World')
        panel = wx.Panel(self)

        self.text_ctrl = wx.TextCtrl(panel, pos=(5, 5))
        my_btn = wx.Button(panel, label='Press Me', pos=(5, 55))

        self.Show()

if __name__ == '__main__':
    app = wx.App()
    frame = MyFrame()
    app.MainLoop()

When you run this code, your application should look like this:

The first widget you need to add is something called wx.Panel. This widget is not required, but recommended. On Windows, you are actually required to use a Panel so that the background color of the frame is the right shade of gray. Tab traversal is disabled without a Panel on Windows.

When you add the panel widget to a frame and the panel is the sole child of the frame, it will automatically expand to fill the frame with itself.

The next step is to add a wx.TextCtrl to the panel. The first argument for almost all widgets is which parent the widget should go onto. In this case, you want the text control and the button to be on top of the panel, so it is the parent you specify.

You also need to tell wxPython where to place the widget, which you can do by passing in a position via the pos parameter. In wxPython, the origin location is (0,0) which is the upper left corner of the parent. So for the text control, you tell wxPython that you want to position its top left corner 5 pixels from the left (x) and 5 pixels from the top (y).

Then you add your button to the panel and give it a label. To prevent the widgets from overlapping, you need to set the y-coordinate to 55 for the button’s position.

Absolute Positioning

When you provide exact coordinates for your widget’s position, the technique that you used is called absolute positioning. Most GUI toolkits provide this capability, but it’s not actually recommended.

As your application becomes more complex, it becomes difficult to keep track of all the widget locations and if you have to move the widgets around. Resetting all those positions becomes a nightmare.

Fortunately all modern GUI toolkits provide a solution for this, which is what you will learn about next.

Sizers (Dynamic Sizing)

The wxPython toolkit includes sizers, which are used for creating dynamic layouts. They manage the placement of your widgets for you and will adjust them when you resize the application window. Other GUI toolkits will refer to sizers as layouts, which is what PyQt does.

Here are the primary types of sizers that you will see used most often:

wx.BoxSizer
wx.GridSizer
wx.FlexGridSizer

Let’s add a wx.BoxSizer to your example and see if we can make it work a bit more nicely:

import wx

class MyFrame(wx.Frame):    
    def __init__(self):
        super().__init__(parent=None, title='Hello World')
        panel = wx.Panel(self)        
        my_sizer = wx.BoxSizer(wx.VERTICAL)        
        self.text_ctrl = wx.TextCtrl(panel)
        my_sizer.Add(self.text_ctrl, 0, wx.ALL | wx.EXPAND, 5)        
        my_btn = wx.Button(panel, label='Press Me')
        my_sizer.Add(my_btn, 0, wx.ALL | wx.CENTER, 5)        
        panel.SetSizer(my_sizer)        
        self.Show()

if __name__ == '__main__':
    app = wx.App()
    frame = MyFrame()
    app.MainLoop()

Here you create an instance of a wx.BoxSizer and pass it wx.VERTICAL, which is the orientation that widgets are added to the sizer.

In this case, the widgets will be added vertically, which means they will be added one at a time from top to bottom. You may also set a BoxSizer’s orientation to wx.HORIZONTAL. When you do that, the widgets would be added from left to right.

To add a widget to a sizer, you will use .Add(). It accepts up to five arguments:

window (the widget)
proportion
flag
border
userData

The window argument is the widget to be added while proportion sets how much space relative to other widgets in the sizer this particular widget should take. By default, it is zero, which tells wxPython to leave the widget at its default proportion.

The third argument is flag. You can actually pass in multiple flags if you wish as long as you separate them with a pipe character: |. The wxPython toolkit uses | to add flags using a series of bitwise ORs.

In this example, you add the text control with the wx.ALL and wx.EXPAND flags. The wx.ALL flag tells wxPython that you want to add a border on all sides of the widget while wx.EXPAND makes the widgets expand as much as they can within the sizer.

Finally, you have the border parameter, which tells wxPython how many pixels of border you want around the widget. The userData parameter is only used when you want to do something complex with your sizing of the widget and is actually quite rare to see in practice.

Adding the button to the sizer follows the exact same steps. However, to make things a bit more interesting, I went ahead and switched out the wx.EXPAND flag for wx.CENTER so that the button would be centered on-screen.

When you run this version of the code, your application should look like the following:

If you’d like to learn more about sizers, the wxPython documentation has a nice page on the topic.

Adding an Event

While your application looks more interesting visually, it still doesn’t really do anything. For example, if you press the button, nothing really happens.

Let’s give the button a job:

import wx

class MyFrame(wx.Frame):    
    def __init__(self):
        super().__init__(parent=None, title='Hello World')
        panel = wx.Panel(self)        
        my_sizer = wx.BoxSizer(wx.VERTICAL)        
        self.text_ctrl = wx.TextCtrl(panel)
        my_sizer.Add(self.text_ctrl, 0, wx.ALL | wx.EXPAND, 5)        
        my_btn = wx.Button(panel, label='Press Me')
        my_btn.Bind(wx.EVT_BUTTON, self.on_press)
        my_sizer.Add(my_btn, 0, wx.ALL | wx.CENTER, 5)        
        panel.SetSizer(my_sizer)        
        self.Show()

    def on_press(self, event):
        value = self.text_ctrl.GetValue()
        if not value:
            print("You didn't enter anything!")
        else:
            print(f'You typed: "{value}"')

if __name__ == '__main__':
    app = wx.App()
    frame = MyFrame()
    app.MainLoop()

The widgets in wxPython allow you to attach event bindings to them so that they can respond to certain types of events.

Note: The code block above uses f-strings. You can read all about them in Python 3’s f-Strings: An Improved String Formatting Syntax (Guide).

You want the button to do something when the user presses it. You can accomplish this by calling the button’s .Bind() method. .Bind() takes the event you want to bind to, the handler to call when the event happens, an optional source, and a couple of optional ids.

In this example, you bind your button object to the wx.EVT_BUTTON event and tell it to call on_press() when that event gets fired.

An event gets “fired” when the user does the event you have bound to. In this case, the event that you set up is the button press event, wx.EVT_BUTTON.

.on_press() accepts a second argument that you can call event. This is by convention. You could call it something else if you wanted to. However, the event parameter here refers to the fact that when this method is called, its second argument should be an event object of some sort.

Within .on_press(), you can get the text control’s contents by calling its GetValue() method. You then print a string to stdout depending on what the contents of the text control is.

Now that you have the basics out of the way, let’s learn how to create an application that does something useful!

Creating a Working Application

The first step when creating something new is to figure out what you want to create. In this case, I have taken the liberty of making that decision for you. You will learn how to create a MP3 tag editor! The next step when creating something new is to find out what packages can help you accomplish your task.

If you do a Google search for Python mp3 tagging, you will find you have several options:

mp3-tagger
eyeD3
mutagen

I tried out a couple of these and decided that eyeD3 had a nice API that you could use without getting bogged down with the MP3’s ID3 specification. You can install eyeD3 using pip, like this:

$ pip install eyed3

When installing this package on macOS, you may need to install libmagic using brew. Windows and Linux users shouldn’t have any issues installing eyeD3.

Designing the User Interface

When it comes to designing an interface, it’s always nice to just kind of sketch out how you think the user interface should look.

You will need to be able to do the following:

Open up one or more MP3 files
Display the current MP3 tags
Edit an MP3 tag

Most user interfaces use a menu or a button for opening files or folders. You can go with a File menu for this. Since you will probably want to see tags for multiple MP3 files, you will need to find a widget that can do this in a nice manner.

Something that is tabular with columns and rows would be ideal because then you can have labeled columns for the MP3 tags. The wxPython toolkit has a few widgets that would work for this, with the top two being the following:

wx.grid.Grid
wx.ListCtrl

You should use wx.ListCtrl in this case as the Grid widget is overkill, and frankly it is also quite a bit more complex. Finally, you need a button to use to edit a selected MP3’s tag.

Now that you know what you want, you can draw it up:

The illustration above gives us an idea of how the application should look. Now that you know what you want to do, it’s time to code!

Creating the User Interface

There are many different approaches when it comes to writing a new application. For example, do you need to follow the Model-View-Controller design pattern? How do you split up the classes? One class per file? There are many such questions, and as you get more experienced with GUI design, you’ll know how you want to answer them.

In your case, you really only need two classes:

A wx.Panel class
A wx.Frame class

You could argue for creating a controller type module as well, but for something like this, you really do not need it. A case could also be made for putting each class into its own module, but to keep it compact, you will create a single Python file for all of your code.

Let’s start with imports and the panel class:

import eyed3
import glob
import wx

class Mp3Panel(wx.Panel):    
    def __init__(self, parent):
        super().__init__(parent)
        main_sizer = wx.BoxSizer(wx.VERTICAL)
        self.row_obj_dict = {}

        self.list_ctrl = wx.ListCtrl(
            self, size=(-1, 100), 
            style=wx.LC_REPORT | wx.BORDER_SUNKEN
        )
        self.list_ctrl.InsertColumn(0, 'Artist', width=140)
        self.list_ctrl.InsertColumn(1, 'Album', width=140)
        self.list_ctrl.InsertColumn(2, 'Title', width=200)
        main_sizer.Add(self.list_ctrl, 0, wx.ALL | wx.EXPAND, 5)        
        edit_button = wx.Button(self, label='Edit')
        edit_button.Bind(wx.EVT_BUTTON, self.on_edit)
        main_sizer.Add(edit_button, 0, wx.ALL | wx.CENTER, 5)        
        self.SetSizer(main_sizer)

    def on_edit(self, event):
        print('in on_edit')

    def update_mp3_listing(self, folder_path):
        print(folder_path)

Here, you import the eyed3 package, Python’s glob package, and the wx package for your user interface. Next, you subclass wx.Panel and create your user interface. You need a dictionary for storing data about your MP3s, which you can name row_obj_dict.

Then you create a wx.ListCtrl and set it to report mode (wx.LC_REPORT) with a sunken border (wx.BORDER_SUNKEN). The list control can take on a few other forms depending on the style flag that you pass in, but the report flag is the most popular.

To make the ListCtrl have the correct headers, you will need to call .InsertColumn() for each column header. You then supply the index of the column, its label, and how wide in pixels the column should be.

The last step is to add your Edit button, an event handler, and a method. You can create the binding to the event and leave the method that it calls empty for now.

Now you should write the code for the frame:

class Mp3Frame(wx.Frame):    
    def __init__(self):
        super().__init__(parent=None,
                         title='Mp3 Tag Editor')
        self.panel = Mp3Panel(self)
        self.Show()

if __name__ == '__main__':
    app = wx.App(False)
    frame = Mp3Frame()
    app.MainLoop()

This class is much simpler than the first one in that all you need to do is set the title of the frame and instantiate the panel class, Mp3Panel. When you are all done, your user interface should look like this:

The user interface looks almost right, but you don’t have a File menu. This makes it impossible to add MP3s to the application and edit their tags!

Let’s fix that now.

Make a Functioning Application

The first step in making your application work is to update the application so that it has a File menu because then you can add MP3 files to your creation. Menus are almost always added to the wx.Frame class, so that is the class you need to modify.

Note: Some applications have moved away from having menus in their applications. One of the first to do so was Microsoft Office when they added the Ribbon Bar. The wxPython toolkit has a custom widget that you can use to create ribbons in wx.lib.agw.ribbon.

The other type of application that has dropped menus of late are web browsers, such as Google Chrome and Mozilla Firefox. They just use toolbars nowadays.

Let’s learn how to add a menu bar to our application:

class Mp3Frame(wx.Frame):

    def __init__(self):
        wx.Frame.__init__(self, parent=None, 
                          title='Mp3 Tag Editor')
        self.panel = Mp3Panel(self)
        self.create_menu()
        self.Show()

    def create_menu(self):
        menu_bar = wx.MenuBar()
        file_menu = wx.Menu()
        open_folder_menu_item = file_menu.Append(
            wx.ID_ANY, 'Open Folder', 
            'Open a folder with MP3s'
        )
        menu_bar.Append(file_menu, '&File')
        self.Bind(
            event=wx.EVT_MENU, 
            handler=self.on_open_folder,
            source=open_folder_menu_item,
        )
        self.SetMenuBar(menu_bar)

    def on_open_folder(self, event):
        title = "Choose a directory:"
        dlg = wx.DirDialog(self, title, 
                           style=wx.DD_DEFAULT_STYLE)
        if dlg.ShowModal() == wx.ID_OK:
            self.panel.update_mp3_listing(dlg.GetPath())
        dlg.Destroy()

Here, you add a call to .create_menu() within the class’s constructor. Then in .create_menu() itself, you will create a wx.MenuBar instance and a wx.Menu instance.

To add a menu item to a menu, you call the menu instance’s .Append() and pass it the following:

A unique identifier
The label for the new menu item
A help string

Next, you need to add the menu to the menubar, so you will need to call the menubar’s .Append(). It takes the menu instance and the label for menu. This label is a bit odd in that you called it &File instead of File. The ampersand tells wxPython to create a keyboard shortcut of Alt+F to open the File menu using just your keyboard.

Note: If you would like to add keyboard shortcuts to your application, then you will want to use an instance of wx.AcceleratorTable to create them. You can read more about Accerator Tables in the wxPython documentation.

To create an event binding, you will need to call self.Bind(), which binds the frame to wx.EVT_MENU. When you use self.Bind() for a menu event, you need to not only tell wxPython which handler to use, but also which source to bind the handler to.

Finally, you must call the frame’s .SetMenuBar() and pass it the menubar instance for it to be shown to the user.

Now that you have the menu added to your frame, let’s go over the menu item’s event handler, which is reproduced again below:

def on_open_folder(self, event):
    title = "Choose a directory:"
    dlg = wx.DirDialog(self, title, style=wx.DD_DEFAULT_STYLE)
    if dlg.ShowModal() == wx.ID_OK:
        self.panel.update_mp3_listing(dlg.GetPath())
    dlg.Destroy()

Since you want the user to choose a folder that contains MP3s, you will want to use wxPython’s wx.DirDialog. The wx.DirDialog allows the user to only open directories.

You can set the dialog’s title and various style flags. To show the dialog, you will need to call .ShowModal(). This will cause the dialog to show modally, which means that the user won’t be able to interact with your main application while the dialog is shown.

If the user presses the dialog’s OK button, you can get the user’s path choice via the dialog’s .GetPath(). You will want to pass that path to your panel class, which you can do here by calling the panel’s .update_mp3_listing().

Finally you need to close the dialog. To close a dialog, the recommended method is to call its .Destroy().

Dialogs do have a .Close() method, but that basically just hides the dialog, and it will not destroy itself when you close your application, which can lead to weird issues such as your application now shutting down properly. It’s simpler to call .Destroy() on the dialog to prevent this issue.

Now let’s update your Mp3Panel class. You can start by updating .update_mp3_listing():

def update_mp3_listing(self, folder_path):
    self.current_folder_path = folder_path
    self.list_ctrl.ClearAll()

    self.list_ctrl.InsertColumn(0, 'Artist', width=140)
    self.list_ctrl.InsertColumn(1, 'Album', width=140)
    self.list_ctrl.InsertColumn(2, 'Title', width=200)
    self.list_ctrl.InsertColumn(3, 'Year', width=200)

    mp3s = glob.glob(folder_path + '/*.mp3')
    mp3_objects = []
    index = 0
    for mp3 in mp3s:
        mp3_object = eyed3.load(mp3)
        self.list_ctrl.InsertItem(index, 
            mp3_object.tag.artist)
        self.list_ctrl.SetItem(index, 1, 
            mp3_object.tag.album)
        self.list_ctrl.SetItem(index, 2, 
            mp3_object.tag.title)
        mp3_objects.append(mp3_object)
        self.row_obj_dict[index] = mp3_object
        index += 1

Here you set the current directory to the specified folder and then you clear the list control. This keeps the list control fresh and only showing the MP3s that you are currently working on. That also means that you need to re-insert all the columns again.

Next, you’ll want to take the folder that was passed in and use Python’s glob module to search for MP3 files.

Then you can loop over the MP3s and turn them into eyed3 objects. You can do this by calling the .load() of eyed3. Assuming that the MP3s have the appropriate tags already, you can then add the artist, album, and title of the MP3 to the list control.

Interestingly, the method of adding a new row to a list control object is by calling .InsertItem() for the first column and SetItem() for all the subsequent columns.

The last step is to save off your MP3 object to your Python dictionary, row_obj_dict.

Now you need to update the .on_edit() event handler so that you can edit an MP3’s tags:

def on_edit(self, event):
    selection = self.list_ctrl.GetFocusedItem()
    if selection >= 0:
        mp3 = self.row_obj_dict[selection]
        dlg = EditDialog(mp3)
        dlg.ShowModal()
        self.update_mp3_listing(self.current_folder_path)
        dlg.Destroy()

The first thing you need to do is get the user’s selection by calling the list control’s .GetFocusedItem().

If the user has not selected anything in the list control, it will return -1. Assuming that the user did select something, you will want to extract the MP3 object from your dictionary and open a MP3 tag editor dialog. This will be a custom dialog that you will use to edit the artist, album, and title tags of the MP3 file.

As usual, show the dialog modally. When the dialog closes, the last two lines in .on_edit() will execute. These two lines will update the list control so it displays the current MP3 tag information that the user just edited and destroy the dialog.

Creating an Editing Dialog

The final piece of the puzzle is creating an MP3 tag editing dialog. For brevity, we will skip sketching out this interface as it is a series of rows that contains labels and text controls. The text controls should have the existing tag information pre-populated within them. You can create a label for the text controls by creating instances of wx.StaticText.

When you need to create a custom dialog, the wx.Dialog class is your friend. You can use that to design the editor:

class EditDialog(wx.Dialog):    
    def __init__(self, mp3):
        title = f'Editing "{mp3.tag.title}"'
        super().__init__(parent=None, title=title)        
        self.mp3 = mp3        
        self.main_sizer = wx.BoxSizer(wx.VERTICAL)        
        self.artist = wx.TextCtrl(
            self, value=self.mp3.tag.artist)
        self.add_widgets('Artist', self.artist)        
        self.album = wx.TextCtrl(
            self, value=self.mp3.tag.album)
        self.add_widgets('Album', self.album)        
        self.title = wx.TextCtrl(
            self, value=self.mp3.tag.title)
        self.add_widgets('Title', self.title)        
        btn_sizer = wx.BoxSizer()
        save_btn = wx.Button(self, label='Save')
        save_btn.Bind(wx.EVT_BUTTON, self.on_save)        
        btn_sizer.Add(save_btn, 0, wx.ALL, 5)
        btn_sizer.Add(wx.Button(
            self, id=wx.ID_CANCEL), 0, wx.ALL, 5)
        self.main_sizer.Add(btn_sizer, 0, wx.CENTER)        
        self.SetSizer(self.main_sizer)

Here you want to start off by sub-classing wx.Dialog and giving it a custom title based on the title of the MP3 that you are editing.

Next you can create the sizer you want to use and the widgets. To make things easier, you can create a helper method called .add_widgets() for adding the wx.StaticText widgets as rows with the text control instances. The only other widget here is the Save button.

Let’s write the add_widgets method next:

    def add_widgets(self, label_text, text_ctrl):
        row_sizer = wx.BoxSizer(wx.HORIZONTAL)
        label = wx.StaticText(self, label=label_text,
                              size=(50, -1))
        row_sizer.Add(label, 0, wx.ALL, 5)
        row_sizer.Add(text_ctrl, 1, wx.ALL | wx.EXPAND, 5)
        self.main_sizer.Add(row_sizer, 0, wx.EXPAND)

add_widgets() takes the label’s text and the text control instance. It then creates a horizontally oriented BoxSizer.

Next you will create an instance of wx.StaticText using the passed-in text for its label parameter. You will also set its size to be 50 pixels wide and the default height is set with a -1. Since you want the label before the text control, you will add the StaticText widget to your BoxSizer first and then add the text control .

Finally, you want to add the horizontal sizer to the top level vertical sizer. By nesting the sizers in each other, you can design complex applications.

Now you will need to create the on_save() event handler so that you can save your changes:

    def on_save(self, event):
        self.mp3.tag.artist = self.artist.GetValue()
        self.mp3.tag.album = self.album.GetValue()
        self.mp3.tag.title = self.title.GetValue()
        self.mp3.tag.save()
        self.Close()

Here you set the tags to the contents of the text controls and then call the eyed3 object’s .save(). Finally, you call the .Close() of the dialog. The reason you call .Close() here instead of .Destroy() is that you already call .Destroy() in the .on_edit() of your panel subclass.

Now your application is complete!

Conclusion

You learned a lot about wxPython in this article. You became familiar with the basics of creating GUI applications using wxPython.

You now know more about the following:

How to work with some of wxPython’s widgets
How events work in wxPython
How absolute positioning compares with sizers
How to create a skeleton application

Finally you learned how to create a working application, an MP3 tag editor. You can use what you learned in this article to continue to enhance this application or perhaps create an amazing application on your own.

The wxPython GUI toolkit is robust and full of interesting widgets that you can use to build cross-platform applications. You are limited by only your imagination.

Understanding the Python Mock Object Library

2019-03-13T14:00:00+00:00

When you’re writing robust code, tests are essential for verifying that your application logic is correct, reliable, and efficient. However, the value of your tests depends on how well they demonstrate these criteria. Obstacles such as complex logic and unpredictable dependencies make writing valuable tests difficult. The Python mock object library, unittest.mock, can help you overcome these obstacles.

By the end of this article, you’ll be able to:

Create Python mock objects using Mock
Assert you’re using objects as you intended
Inspect usage data stored on your Python mocks
Configure certain aspects of your Python mock objects
Substitute your mocks for real objects using patch()
Avoid common problems inherent in Python mocking

You’ll begin by seeing what mocking is and how it will improve your tests.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

What Is Mocking?

A mock object substitutes and imitates a real object within a testing environment. It is a versatile and powerful tool for improving the quality of your tests.

One reason to use Python mock objects is to control your code’s behavior during testing.

For example, if your code makes HTTP requests to external services, then your tests execute predictably only so far as the services are behaving as you expected. Sometimes, a temporary change in the behavior of these external services can cause intermittent failures within your test suite.

Because of this, it would be better for you to test your code in a controlled environment. Replacing the actual request with a mock object would allow you to simulate external service outages and successful responses in a predictable way.

Sometimes, it is difficult to test certain areas of your codebase. Such areas include except blocks and if statements that are hard to satisfy. Using Python mock objects can help you control the execution path of your code to reach these areas and improve your code coverage.

Another reason to use mock objects is to better understand how you’re using their real counterparts in your code. A Python mock object contains data about its usage that you can inspect such as:

If you called a method
How you called the method
How often you called the method

Understanding what a mock object does is the first step to learning how to use one.

Now, you’ll see how to use Python mock objects.

The Python Mock Library

The Python mock object library is unittest.mock. It provides an easy way to introduce mocks into your tests.

Note: The standard library includes unittest.mock in Python 3.3 and later. If you’re using an older version of Python, you’ll need to install the official backport of the library. To do so, install mock from PyPI:

$ pip install mock

unittest.mock provides a class called Mock which you will use to imitate real objects in your codebase. Mock offers incredible flexibility and insightful data. This, along with its subclasses, will meet most Python mocking needs that you will face in your tests.

The library also provides a function, called patch(), which replaces the real objects in your code with Mock instances. You can use patch() as either a decorator or a context manager, giving you control over the scope in which the object will be mocked. Once the designated scope exits, patch() will clean up your code by replacing the mocked objects with their original counterparts.

Finally, unittest.mock provides solutions for some of the issues inherent in mocking objects.

Now, you have a better understanding of what mocking is and the library you’ll be using to do it. Let’s dive in and explore what features and functionalities unittest.mock offers.

The `Mock` Object

unittest.mock offers a base class for mocking objects called Mock. The use cases for Mock are practically limitless because Mock is so flexible.

Begin by instantiating a new Mock instance:

>>>

>>> from unittest.mock import Mock
>>> mock = Mock()
>>> mock
<Mock id='4561344720'>

Now, you are able to substitute an object in your code with your new Mock. You can do this by passing it as an argument to a function or by redefining another object:

# Pass mock as an argument to do_something()
do_something(mock)

# Patch the json library
json = mock

When you substitute an object in your code, the Mock must look like the real object it is replacing. Otherwise, your code will not be able to use the Mock in place of the original object.

For example, if you are mocking the json library and your program calls dumps(), then your Python mock object must also contain dumps().

Next, you’ll see how Mock deals with this challenge.

Lazy Attributes and Methods

A Mock must simulate any object that it replaces. To achieve such flexibility, it creates its attributes when you access them:

>>>

>>> mock.some_attribute
<Mock name='mock.some_attribute' id='4394778696'>
>>> mock.do_something()
<Mock name='mock.do_something()' id='4394778920'>

Since Mock can create arbitrary attributes on the fly, it is suitable to replace any object.

Using an example from earlier, if you’re mocking the json library and you call dumps(), the Python mock object will create the method so that its interface can match the library’s interface:

>>>

>>> json = Mock()
>>> json.dumps()
<Mock name='mock.dumps()' id='4392249776'>

Notice two key characteristics of this mocked version of dumps():

Unlike the real dumps(), this mocked method requires no arguments. In fact, it will accept any arguments that you pass to it.
The return value of dumps() is also a Mock. The capability of Mock to recursively define other mocks allows for you to use mocks in complex situations:

>>>

>>> json = Mock()
>>> json.loads('{"k": "v"}').get('k')
<Mock name='mock.loads().get()' id='4379599424'>

Because the return value of each mocked method is also a Mock, you can use your mocks in a multitude of ways.

Mocks are flexible, but they’re also informative. Next, you’ll learn how you can use mocks to understand your code better.

Assertions and Inspection

Mock instances store data on how you used them. For instance, you can see if you called a method, how you called the method, and so on. There are two main ways to use this information.

First, you can assert that your program used an object as you expected:

>>>

>>> from unittest.mock import Mock

>>> # Create a mock object
... json = Mock()

>>> json.loads('{"key": "value"}')
<Mock name='mock.loads()' id='4550144184'>

>>> # You know that you called loads() so you can
>>> # make assertions to test that expectation
... json.loads.assert_called()
>>> json.loads.assert_called_once()
>>> json.loads.assert_called_with('{"key": "value"}')
>>> json.loads.assert_called_once_with('{"key": "value"}')

>>> json.loads('{"key": "value"}')
<Mock name='mock.loads()' id='4550144184'>

>>> # If an assertion fails, the mock will raise an AssertionError
... json.loads.assert_called_once()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 795, in assert_called_once
    raise AssertionError(msg)
AssertionError: Expected 'loads' to have been called once. Called 2 times.

>>> json.loads.assert_called_once_with('{"key": "value"}')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 824, in assert_called_once_with
    raise AssertionError(msg)
AssertionError: Expected 'loads' to be called once. Called 2 times.

>>> json.loads.assert_not_called()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 777, in assert_not_called
    raise AssertionError(msg)
AssertionError: Expected 'loads' to not have been called. Called 2 times.

.assert_called() ensures you called the mocked method while .assert_called_once() checks that you called the method exactly one time.

Both assertion functions have variants that let you inspect the arguments passed to the mocked method:

.assert_called_with(*args, **kwargs)
.assert_called_once_with(*args, **kwargs)

To pass these assertions, you must call the mocked method with the same arguments that you pass to the actual method:

>>>

>>> json = Mock()
>>> json.loads(s='{"key": "value"}')
>>> json.loads.assert_called_with('{"key": "value"}')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 814, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: loads('{"key": "value"}')
Actual call: loads(s='{"key": "value"}')
>>> json.loads.assert_called_with(s='{"key": "value"}')

json.loads.assert_called_with('{"key": "value"}') raised an AssertionError because it expected you to call loads() with a positional argument, but you actually called it with a keyword argument. json.loads.assert_called_with(s='{"key": "value"}') gets this assertion correct.

Second, you can view special attributes to understand how your application used an object:

>>>

>>> from unittest.mock import Mock

>>> # Create a mock object
... json = Mock()
>>> json.loads('{"key": "value"}')
<Mock name='mock.loads()' id='4391026640'>

>>> # Number of times you called loads():
... json.loads.call_count
1
>>> # The last loads() call:
... json.loads.call_args
call('{"key": "value"}')
>>> # List of loads() calls:
... json.loads.call_args_list
[call('{"key": "value"}')]
>>> # List of calls to json's methods (recursively):
... json.method_calls
[call.loads('{"key": "value"}')]

You can write tests using these attributes to make sure that your objects behave as you intended.

Now, you can create mocks and inspect their usage data. Next, you’ll see how to customize mocked methods so that they become more useful in your testing environment.

Managing a Mock’s Return Value

One reason to use mocks is to control your code’s behavior during tests. One way to do this is to specify a function’s return value. Let’s use an example to see how this works.

First, create a file called my_calendar.py. Add is_weekday(), a function that uses Python’s datetime library to determine whether or not today is a week day. Finally, write a test that asserts that the function works as expected:

from datetime import datetime

def is_weekday():
    today = datetime.today()
    # Python's datetime library treats Monday as 0 and Sunday as 6
    return (0 <= today.weekday() < 5)

# Test if today is a weekday
assert is_weekday()

Since you’re testing if today is a weekday, the result depends on the day you run your test:

$ python my_calendar.py

If this command produces no output, the assertion was successful. Unfortunately, if you run the command on a weekend, you’ll get an AssertionError:

$ python my_calendar.py
Traceback (most recent call last):
  File "test.py", line 9, in <module>
    assert is_weekday()
AssertionError

When writing tests, it is important to ensure that the results are predictable. You can use Mock to eliminate uncertainty from your code during testing. In this case, you can mock datetime and set the .return_value for .today() to a day that you choose:

import datetime
from unittest.mock import Mock

# Save a couple of test days
tuesday = datetime.datetime(year=2019, month=1, day=1)
saturday = datetime.datetime(year=2019, month=1, day=5)

# Mock datetime to control today's date
datetime = Mock()

def is_weekday():
    today = datetime.datetime.today()
    # Python's datetime library treats Monday as 0 and Sunday as 6
    return (0 <= today.weekday() < 5)

# Mock .today() to return Tuesday
datetime.datetime.today.return_value = tuesday
# Test Tuesday is a weekday
assert is_weekday()
# Mock .today() to return Saturday
datetime.datetime.today.return_value = saturday
# Test Saturday is not a weekday
assert not is_weekday()

In the example, .today() is a mocked method. You’ve removed the inconsistency by assigning a specific day to the mock’s .return_value. That way, when you call .today(), it returns the datetime that you specified.

In the first test, you ensure tuesday is a weekday. In the second test, you verify that saturday is not a weekday. Now, it doesn’t matter what day you run your tests on because you’ve mocked datetime and have control over the object’s behavior.

Further Reading: Though mocking datetime like this is a good practice example for using Mock, a fantastic library already exists for mocking datetime called freezegun.

When building your tests, you will likely come across cases where mocking a function’s return value will not be enough. This is because functions are often more complicated than a simple one-way flow of logic.

Sometimes, you’ll want to make functions return different values when you call them more than once or even raise exceptions. You can do this using .side_effect.

Managing a Mock’s Side Effects

You can control your code’s behavior by specifying a mocked function’s side effects. A .side_effect defines what happens when you call the mocked function.

To test how this works, add a new function to my_calendar.py:

import requests

def get_holidays():
    r = requests.get('http://localhost/api/holidays')
    if r.status_code == 200:
        return r.json()
    return None

get_holidays() makes a request to the localhost server for a set of holidays. If the server responds successfully, get_holidays() will return a dictionary. Otherwise, the method will return None.

You can test how get_holidays() will respond to a connection timeout by setting requests.get.side_effect.

For this example, you’ll only see the relevant code from my_calendar.py. You’ll build a test case using Python’s unittest library:

import unittest
from requests.exceptions import Timeout
from unittest.mock import Mock

# Mock requests to control its behavior
requests = Mock()

def get_holidays():
    r = requests.get('http://localhost/api/holidays')
    if r.status_code == 200:
        return r.json()
    return None

class TestCalendar(unittest.TestCase):
    def test_get_holidays_timeout(self):
        # Test a connection timeout
        requests.get.side_effect = Timeout
        with self.assertRaises(Timeout):
            get_holidays()

if __name__ == '__main__':
    unittest.main()

You use .assertRaises() to verify that get_holidays() raises an exception given the new side effect of get().

Run this test to see the result of your test:

$ python my_calendar.py
.
-------------------------------------------------------
Ran 1 test in 0.000s

OK

If you want to be a little more dynamic, you can set .side_effect to a function that Mock will invoke when you call your mocked method. The mock shares the arguments and return value of the .side_effect function:

import requests
import unittest
from unittest.mock import Mock

# Mock requests to control its behavior
requests = Mock()

def get_holidays():
    r = requests.get('http://localhost/api/holidays')
    if r.status_code == 200:
        return r.json()
    return None

class TestCalendar(unittest.TestCase):
    def log_request(self, url):
        # Log a fake request for test output purposes
        print(f'Making a request to {url}.')
        print('Request received!')

        # Create a new Mock to imitate a Response
        response_mock = Mock()
        response_mock.status_code = 200
        response_mock.json.return_value = {
            '12/25': 'Christmas',
            '7/4': 'Independence Day',
        }
        return response_mock

    def test_get_holidays_logging(self):
        # Test a successful, logged request
        requests.get.side_effect = self.log_request
        assert get_holidays()['12/25'] == 'Christmas'

if __name__ == '__main__':
    unittest.main()

First, you created .log_request(), which takes a URL, logs some output using print(), then returns a Mock response. Next, you set the .side_effect of get() to .log_request(), which you’ll use when you call get_holidays(). When you run your test, you’ll see that get() forwards its arguments to .log_request() then accepts the return value and returns it as well:

$ python my_calendar.py
Making a request to http://localhost/api/holidays.
Request received!
.
-------------------------------------------------------
Ran 1 test in 0.000s

OK

Great! The print() statements logged the correct values. Also, get_holidays() returned the holidays dictionary.

.side_effect can also be an iterable. The iterable must consist of return values, exceptions, or a mixture of both. The iterable will produce its next value every time you call your mocked method. For example, you can test that a retry after a Timeout returns a successful response:

import unittest
from requests.exceptions import Timeout
from unittest.mock import Mock

# Mock requests to control its behavior
requests = Mock()

def get_holidays():
    r = requests.get('http://localhost/api/holidays')
    if r.status_code == 200:
        return r.json()
    return None

class TestCalendar(unittest.TestCase):
    def test_get_holidays_retry(self):
        # Create a new Mock to imitate a Response
        response_mock = Mock()
        response_mock.status_code = 200
        response_mock.json.return_value = {
            '12/25': 'Christmas',
            '7/4': 'Independence Day',
        }
        # Set the side effect of .get()
        requests.get.side_effect = [Timeout, response_mock]
        # Test that the first request raises a Timeout
        with self.assertRaises(Timeout):
            get_holidays()
        # Now retry, expecting a successful response
        assert get_holidays()['12/25'] == 'Christmas'
        # Finally, assert .get() was called twice
        assert requests.get.call_count == 2

if __name__ == '__main__':
    unittest.main()

The first time you call get_holidays(), get() raises a Timeout. The second time, the method returns a valid holidays dictionary. These side effects match the order they appear in the list passed to .side_effect.

You can set .return_value and .side_effect on a Mock directly. However, because a Python mock object needs to be flexible in creating its attributes, there is a better way to configure these and other settings.

Configuring Your Mock

You can configure a Mock to set up some of the object’s behaviors. Some configurable members include .side_effect, .return_value, and .name. You configure a Mock when you create one or when you use .configure_mock().

You can configure a Mock by specifying certain attributes when you initialize an object:

>>>

>>> mock = Mock(side_effect=Exception)
>>> mock()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 939, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 995, in _mock_call
    raise effect
Exception

>>> mock = Mock(name='Real Python Mock')
>>> mock
<Mock name='Real Python Mock' id='4434041432'>

>>> mock = Mock(return_value=True)
>>> mock()
True

While .side_effect and .return_value can be set on the Mock instance, itself, other attributes like .name can only be set through .__init__() or .configure_mock(). If you try to set the .name of the Mock on the instance, you will get a different result:

>>>

>>> mock = Mock(name='Real Python Mock')
>>> mock.name
<Mock name='Real Python Mock.name' id='4434041544'>

>>> mock = Mock()
>>> mock.name = 'Real Python Mock'
>>> mock.name
'Real Python Mock'

.name is a common attribute for objects to use. So, Mock doesn’t let you set that value on the instance in the same way you can with .return_value or .side_effect. If you access mock.name you will create a .name attribute instead of configuring your mock.

You can configure an existing Mock using .configure_mock():

>>>

>>> mock = Mock()
>>> mock.configure_mock(return_value=True)
>>> mock()
True

By unpacking a dictionary into either .configure_mock() or Mock.__init__(), you can even configure your Python mock object’s attributes. Using Mock configurations, you could simplify a previous example:

# Verbose, old Mock
response_mock = Mock()
response_mock.json.return_value = {
    '12/25': 'Christmas',
    '7/4': 'Independence Day',
}

# Shiny, new .configure_mock()
holidays = {'12/25': 'Christmas', '7/4': 'Independence Day'}
response_mock = Mock(**{'json.return_value': holidays})

Now, you can create and configure Python mock objects. You can also use mocks to control the behavior of your application. So far, you’ve used mocks as arguments to functions or patching objects in the same module as your tests.

Next, you’ll learn how to substitute your mocks for real objects in other modules.

`patch()`

unittest.mock provides a powerful mechanism for mocking objects, called patch(), which looks up an object in a given module and replaces that object with a Mock.

Usually, you use patch() as a decorator or a context manager to provide a scope in which you will mock the target object.

`patch()` as a Decorator

If you want to mock an object for the duration of your entire test function, you can use patch() as a function decorator.

To see how this works, reorganize your my_calendar.py file by putting the logic and tests into separate files:

import requests
from datetime import datetime

def is_weekday():
    today = datetime.today()
    # Python's datetime library treats Monday as 0 and Sunday as 6
    return (0 <= today.weekday() < 5)

def get_holidays():
    r = requests.get('http://localhost/api/holidays')
    if r.status_code == 200:
        return r.json()
    return None

These functions are now in their own file, separate from their tests. Next, you’ll re-create your tests in a file called tests.py.

Up to this point, you’ve monkey patched objects in the file in which they exist. Monkey patching is the replacement of one object with another at runtime. Now, you’ll use patch() to replace your objects in my_calendar.py:

import unittest
from my_calendar import get_holidays
from requests.exceptions import Timeout
from unittest.mock import patch

class TestCalendar(unittest.TestCase):
    @patch('my_calendar.requests')
    def test_get_holidays_timeout(self, mock_requests):
            mock_requests.get.side_effect = Timeout
            with self.assertRaises(Timeout):
                get_holidays()
                mock_requests.get.assert_called_once()

if __name__ == '__main__':
    unittest.main()

Originally, you created a Mock and patched requests in the local scope. Now, you need to access the requests library in my_calendar.py from tests.py.

For this case, you used patch() as a decorator and passed the target object’s path. The target path was 'my_calendar.requests' which consists of the module name and the object.

You also defined a new parameter for the test function. patch() uses this parameter to pass the mocked object into your test. From there, you can modify the mock or make assertions as necessary.

You can execute this test module to ensure it’s working as expected:

$ python tests.py
.
-------------------------------------------------------
Ran 1 test in 0.001s

OK

Technical Detail: patch() returns an instance of MagicMock, which is a Mock subclass. MagicMock is useful because it implements most magic methods for you, such as .__len__(), .__str__(), and .__iter__(), with reasonable defaults.

Using patch() as a decorator worked well in this example. In some cases, it is more readable, more effective, or easier to use patch() as a context manager.

`patch()` as a Context Manager

Sometimes, you’ll want to use patch() as a context manager rather than a decorator. Some reasons why you might prefer a context manager include the following:

You only want to mock an object for a part of the test scope.
You are already using too many decorators or parameters, which hurts your test’s readability.

To use patch() as a context manager, you use Python’s with statement:

import unittest
from my_calendar import get_holidays
from requests.exceptions import Timeout
from unittest.mock import patch

class TestCalendar(unittest.TestCase):
    def test_get_holidays_timeout(self):
        with patch('my_calendar.requests') as mock_requests:
            mock_requests.get.side_effect = Timeout
            with self.assertRaises(Timeout):
                get_holidays()
                mock_requests.get.assert_called_once()

if __name__ == '__main__':
    unittest.main()

When the test exits the with statement, patch() replaces the mocked object with the original.

Until now, you’ve mocked complete objects, but sometimes you’ll only want to mock a part of an object.

Patching an Object’s Attributes

Let’s say you only want to mock one method of an object instead of the entire object. You can do so by using patch.object().

For example, .test_get_holidays_timeout() really only needs to mock requests.get() and set its .side_effect to Timeout:

import unittest
from my_calendar import requests, get_holidays
from unittest.mock import patch

class TestCalendar(unittest.TestCase):
    @patch.object(requests, 'get', side_effect=requests.exceptions.Timeout)
    def test_get_holidays_timeout(self, mock_requests):
            with self.assertRaises(requests.exceptions.Timeout):
                get_holidays()

if __name__ == '__main__':
    unittest.main()

In this example, you’ve mocked only get() rather than all of requests. Every other attribute remains the same.

object() takes the same configuration parameters that patch() does. But instead of passing the target’s path, you provide the target object, itself, as the first parameter. The second parameter is the attribute of the target object that you are trying to mock. You can also use object() as a context manager like patch().

Further Reading: Besides objects and attributes, you can also patch() dictionaries with patch.dict().

Learning how to use patch() is critical to mocking objects in other modules. However, sometimes it’s not obvious what the target object’s path is.

Where to Patch

Knowing where to tell patch() to look for the object you want mocked is important because if you choose the wrong target location, the result of patch() could be something you didn’t expect.

Let’s say you are mocking is_weekday() in my_calendar.py using patch():

>>>

>>> import my_calendar
>>> from unittest.mock import patch

>>> with patch('my_calendar.is_weekday'):
...     my_calendar.is_weekday()
...
<MagicMock name='is_weekday()' id='4336501256'>

First, you import my_calendar.py. Then you patch is_weekday(), replacing it with a Mock. Great! This is working as expected.

Now, let’s change this example slightly and import the function directly:

>>>

>>> from my_calendar import is_weekday
>>> from unittest.mock import patch

>>> with patch('my_calendar.is_weekday'):
...     is_weekday()
...
False

Note: Depending on what day you are reading this tutorial, your console output may read True or False. The important thing is that the output is not a Mock like before.

Notice that even though the target location you passed to patch() did not change, the result of calling is_weekday() is different. The difference is due to the change in how you imported the function.

from my_calendar import is_weekday binds the real function to the local scope. So, even though you patch() the function later, you ignore the mock because you already have a local reference to the un-mocked function.

A good rule of thumb is to patch() the object where it is looked up.

In the first example, mocking 'my_calendar.is_weekday()' works because you look up the function in the my_calendar module. In the second example, you have a local reference to is_weekday(). Since you use the function found in the local scope, you should mock the local function:

>>>

>>> from unittest.mock import patch
>>> from my_calendar import is_weekday

>>> with patch('__main__.is_weekday'):
...     is_weekday()
...
<MagicMock name='is_weekday()' id='4502362992'>

Now, you have a firm grasp on the power of patch(). You’ve seen how to patch() objects and attributes as well as where to patch them.

Next, you’ll see some common problems inherent in object mocking and the solutions that unittest.mock provides.

Common Mocking Problems

Mocking objects can introduce several problems into your tests. Some problems are inherent in mocking while others are specific to unittest.mock. Keep in mind that there are other issues with mocking that are not mentioned in this tutorial.

The ones covered here are similar to each other in that the problem they cause is fundamentally the same. In each case, the test assertions are irrelevant. Though the intention of each mock is valid, the mocks themselves are not.

Changes to Object Interfaces and Misspellings

Classes and function definitions change all the time. When the interface of an object changes, any tests relying on a Mock of that object may become irrelevant.

For example, you rename a method but forget that a test mocks that method and invokes .assert_not_called(). After the change, .assert_not_called() is still True. The assertion is not useful, though, because the method no longer exists.

Irrelevant tests may not sound critical, but if they are your only tests and you assume that they work properly, the situation could be disastrous for your application.

A problem specific to Mock is that a misspelling can break a test. Recall that a Mock creates its interface when you access its members. So, you will inadvertently create a new attribute if you misspell its name.

If you call .asert_called() instead of .assert_called(), your test will not raise an AssertionError. This is because you’ve created a new method on the Python mock object named .asert_called() instead of evaluating an actual assertion.

Technical Detail: Interestingly, assret is a special misspelling of assert. If you try to access an attribute that starts with assret (or assert), Mock will automatically raise an AttributeError.

These problems occur when you mock objects within your own codebase. A different problem arises when you mock objects interacting with external codebases.

Changes to External Dependencies

Imagine again that your code makes a request to an external API. In this case, the external dependency is the API which is susceptible to change without your consent.

On one hand, unit tests test isolated components of code. So, mocking the code that makes the request helps you to test your isolated components under controlled conditions. However, it also presents a potential problem.

If an external dependency changes its interface, your Python mock objects will become invalid. If this happens (and the interface change is a breaking one), your tests will pass because your mock objects have masked the change, but your production code will fail.

Unfortunately, this is not a problem that unittest.mock provides a solution for. You must exercise judgment when mocking external dependencies.

All three of these issues can cause test irrelevancy and potentially costly issues because they threaten the integrity of your mocks. unittest.mock gives you some tools for dealing with these problems.

Avoiding Common Problems Using Specifications

As mentioned before, if you change a class or function definition or you misspell a Python mock object’s attribute, you can cause problems with your tests.

These problems occur because Mock creates attributes and methods when you access them. The answer to these issues is to prevent Mock from creating attributes that don’t conform to the object you’re trying to mock.

When configuring a Mock, you can pass an object specification to the spec parameter. The spec parameter accepts a list of names or another object and defines the mock’s interface. If you attempt to access an attribute that does not belong to the specification, Mock will raise an AttributeError:

>>>

>>> from unittest.mock import Mock
>>> calendar = Mock(spec=['is_weekday', 'get_holidays'])

>>> calendar.is_weekday()
<Mock name='mock.is_weekday()' id='4569015856'>
>>> calendar.create_event()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 582, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'create_event'

Here, you’ve specified that calendar has methods called .is_weekday() and .get_holidays(). When you access .is_weekday(), it returns a Mock. When you access .create_event(), a method that does not match the specification, Mock raises an AttributeError.

Specifications work the same way if you configure the Mock with an object:

>>>

>>> import my_calendar
>>> from unittest.mock import Mock

>>> calendar = Mock(spec=my_calendar)
>>> calendar.is_weekday()
<Mock name='mock.is_weekday()' id='4569435216'>
>>> calendar.create_event()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 582, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'create_event'

.is_weekday() is available to calendar because you configured calendar to match the my_calendar module’s interface.

Furthermore, unittest.mock provides convenient methods of automatically specifying a Mock instance’s interface.

One way to implement automatic specifications is create_autospec:

>>>

>>> import my_calendar
>>> from unittest.mock import create_autospec

>>> calendar = create_autospec(my_calendar)
>>> calendar.is_weekday()
<MagicMock name='mock.is_weekday()' id='4579049424'>
>>> calendar.create_event()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 582, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'create_event'

Like before, calendar is a Mock instance whose interface matches my_calendar. If you’re using patch(), you can send an argument to the autospec parameter to achieve the same result:

>>>

>>> import my_calendar
>>> from unittest.mock import patch

>>> with patch('__main__.my_calendar', autospec=True) as calendar:
...     calendar.is_weekday()
...     calendar.create_event()
...
<MagicMock name='my_calendar.is_weekday()' id='4579094312'>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/mock.py", line 582, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'create_event'

Conclusion

You’ve learned so much about mocking objects using unittest.mock!

Now, you’re able to:

Use Mock to imitate objects in your tests
Check usage data to understand how you use your objects
Customize your mock objects’ return values and side effects
patch() objects throughout your codebase
See and avoid problems with using Python mock objects

You have built a foundation of understanding that will help you build better tests. You can use mocks to gain insights into your code that you would not have been able to get otherwise.

I leave you with one final disclaimer. Beware of overusing mock objects!

It’s easy to take advantage of the power of Python mock objects and mock so much that you actually decrease the value of your tests.

If you’re interested in learning more about unittest.mock, I encourage you to read its excellent documentation.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Managing Multiple Python Versions With pyenv

2019-03-11T14:00:00+00:00

Have you ever wanted to contribute to a project that supports multiple versions of Python but aren’t sure how you would easily test all the versions? Are you ever curious about the latest and greatest versions of Python? Maybe you’d like to try out these new features, but you don’t want to worry about messing up your development environment. Luckily, managing multiple version of Python doesn’t have to be confusing if you use pyenv.

This article will provide you with a great overview of how to maximize your time spent working on projects and minimize the time spent in headaches trying to use the right version of Python.

In this article, you’ll learn how to:

Install multiple versions of Python
Install the latest development version of Python
Switch between the installed versions
Use virtual environments with pyenv
Activate different Python versions and virtual environments automatically

Why Use `pyenv`?

pyenv is a wonderful tool for managing multiple Python versions. Even if you already have Python installed on your system, it is worth having pyenv installed so that you can easily try out new language features or help contribute to a project that is on a different version of Python.

Why Not Use System Python?

“System Python” is the Python that comes installed on your operating system. If you’re on Mac or Linux, then by default, when you type python in your terminal, you get a nice Python REPL.

So, why not use it? One way to look at it is that this Python really belongs to the operating system. After all, it came installed with the operating system. That’s even reflected when you run which:

$ which python
/usr/bin/python

Here, python is available to all users as evidenced by its location /usr/bin/python. Chances are, this isn’t the version of Python you want either:

$ python -V
Pyhton 2.7.12

To install a package into your system Python, you have to run sudo pip install. That’s because you’re installing the Python package globally, which is a real problem if another user comes along and wants to install a slightly older version of the package.

Problems with multiple versions of the same package tend to creep up on you and bite you when you least expect it. One common way this problem presents itself is a popular and stable package suddenly misbehaving on your system. After hours of troubleshooting and Googling, you may find that you’ve installed the wrong version of a dependency, and it’s ruining your day.

Even if your Python version is installed in /usr/local/bin/python3, you’re still not safe. You will run into the same permissions and flexibility problems described above.

In addition, you don’t really have much control over what version of Python comes installed on your OS. If you want to use the latest features in Python, and you’re on Ubuntu for example, you might just be out of luck. The default versions might be too old, which means you’ll just have to wait for a new OS to come out.

Finally, some operating systems actually use the packaged Python for operation. Take yum for example, which makes heavy use of Python to do its job. If you install a new version of Python and aren’t careful to install it into your user space, you could seriously damage your ability to use your OS.

What About a Package Manager?

The next logical place to look is package managers. Programs such as apt, yum, brew, or port are typical next options. After all, this is how you install most packages to your system. Unfortunately, you’ll find some of the same problems using a package manager.

By default, package managers tend to install their packages into the global system space instead of the user space. Again, these system level packages pollute your development environment and make it hard to share a workspace with others.

Once again, you still don’t have control over what version of Python you can install. It’s true some repositories give you a greater selection, but by default, you’re looking at whatever version of Python your particular vendor is up to on any given day.

Even if you do install Python from a package manager, consider what would happen if you’re writing a package and want to support and test on Python 3.4 - 3.7.

What would happen on your system when you type python3? How would you switch quickly between the different versions? You can certainly do it, but it is tedious and prone to error. Nevermind the fact that if you want PyPy, Jython, or Miniconda, then you’re probably just out of luck with your package manager.

With these constraints in mind, let’s recap the criteria that would let you install and manage Python versions easily and flexibly:

Install Python in your user space
Install multiple versions of Python
Specify the exact Python version you want
Switch between the installed versions

pyenv lets you do all of these things and more.

Installing `pyenv`

Before you install pyenv itself, you’re going to need some OS-specific dependencies. These dependencies are mostly development utilities written in C and are required because pyenv installs Python by building from source. For a more detailed breakdown and explanation of the build dependencies, you can check out the official docs. In this tutorial, you’ll see the most common ways to install these dependencies.

Note: pyenv did not originally support Windows. However, there appears to be some basic support with the pyenv-win project that recently became active. If you use Windows, feel free to check it out.

Build Dependencies

pyenv builds Python from source, which means you’ll need build dependencies to actually use pyenv. The build dependencies vary by platform. If you are on Ubuntu/Debian and want to install the build dependencies, you could use the following:

$ sudo apt-get install -y make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev \
libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl

This uses Apt to install all the build dependencies. Let this run, and you’ll be ready to go for Debian systems.

If you use Fedora/CentOS/RHEL, you could use yum to install your build dependencies:

$ sudo yum install zlib-devel bzip2 bzip2-devel readline-devel sqlite \
sqlite-devel openssl-devel xz xz-devel libffi-devel

This command will install all the build dependencies for Python using yum.

macOS users can use the following command:

$ brew install readline xz

This command relies on Homebrew and installs the few dependencies for macOS users.

If you’re instead using openSUSE then you would run the following:

$ zypper in zlib-devel bzip2 libbz2-devel libffi-devel \
libopenssl-devel readline-devel sqlite3 sqlite3-devel xz xz-devel

Once again, this command installs all the Python build dependencies for your system.

Finally, for Alpine users, you can use this:

$ apk add libffi-dev ncurses-dev openssl-dev readline-dev \
tk-dev xz-dev zlib-dev

This command uses apk as the package manager and will install all build dependencies for Python on Alpine.

Using the pyenv-installer

After you’ve installed the build dependencies, you’re ready to install pyenv itself. I recommend using the pyenv-installer project:

$ curl https://pyenv.run | bash

This will install pyenv along with a few plugins that are useful:

pyenv: The actual pyenv application
pyenv-virtualenv: Plugin for pyenv and virtual environments
pyenv-update: Plugin for updating pyenv
pyenv-doctor: Plugin to verify that pyenv and build dependencies are installed
pyenv-which-ext: Plugin to automatically lookup system commands

Note: The above command is the same as downloading the pyenv-installer script and running it locally. So if you’d like to see exactly what you’re running, you can view the file yourself. Alternatively, if you really don’t want to run a script, you can checkout the manual installation instructions.

At the end of the run, you should see something like this:

WARNING: seems you still have not added 'pyenv' to the load path.

# Load pyenv automatically by adding
# the following to ~/.bashrc:

export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

The output will be based on your shell. But you should follow the instructions to add pyenv to your path and to initialize pyenv/pyenv-virtualenv auto completion. Once you’ve done this, you need to reload your shell:

$ exec "$SHELL" # Or just restart your terminal

That’s it. You now have pyenv and four useful plugins installed.

Using `pyenv` to Install Python

Now that you have pyenv installed, installing Python is the next step. You have many versions of Python to choose from. If you wanted to see all the available CPython 3.6 through 3.8, you can do this:

$ pyenv install --list | grep " 3\.[678]"
  3.6.0
  3.6-dev
  3.6.1
  3.6.2
  3.6.3
  3.6.4
  3.6.5
  3.6.6
  3.6.7
  3.6.8
  3.7.0
  3.7-dev
  3.7.1
  3.7.2
  3.8-dev

The above shows all the Python versions that pyenv knows about that match the regular expression. In this case, that is all available CPython versions 3.6 through 3.8. Likewise, if you wanted to see all the Jython versions, you could do this:

$ pyenv install --list | grep "jython"
  jython-dev
  jython-2.5.0
  jython-2.5-dev
  jython-2.5.1
  jython-2.5.2
  jython-2.5.3
  jython-2.5.4-rc1
  jython-2.7.0
  jython-2.7.1

Again, you can see all the Jython versions that pyenv has to offer. If you want to see all the versions, you can do the following:

$ pyenv install --list
...
# There are a lot

Once you find the version you want, you can install it with a single command:

$ pyenv install -v 3.7.2
/tmp/python-build.20190208022403.30568 ~
Downloading Python-3.7.2.tar.xz...
-> https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tar.xz
Installing Python-3.7.2...
/tmp/python-build.20190208022403.30568/Python-3.7.2 /tmp/python-build.20190208022403.30568 ~
[...]
Installing collected packages: setuptools, pip
Successfully installed pip-18.1 setuptools-40.6.2
Installed Python-3.7.2 to /home/realpython/.pyenv/versions/3.7.2

Having Problems? The pyenv documentation has great installation notes as well as a useful FAQ along with common build problems.

This will take a while because pyenv is building Python from source, but once it’s done, you’ll have Python 3.7.2 available on your local machine. If you don’t want to see all the output, just remove the -v flag. Even development versions of CPython can be installed:

$ pyenv install 3.8-dev

Pro Tip: If you’ve been using pyenv for a while and don’t see the version you’re looking for, you may need to run pyenv update to update the tool and make sure you have access to the latest versions.

For the rest of the tutorial, the examples assume you’ve installed 3.6.8 and 2.7.15, but you’re free to substitute these values for the Python versions you actually installed. Also note that the system Python version in the examples is 2.7.12.

Installation Location

As mentioned before, pyenv works by building Python from source. Each version that you have installed is located nicely in your pyenv root directory:

$ ls ~/.pyenv/versions/
2.7.15  3.6.8  3.8-dev

All of your versions will be located here. This is handy because removing these versions is trivial:

$ rm -rf ~/.pyenv/versions/2.7.15

Of course pyenv also provides a command to uninstall a particular Python version:

$ pyenv uninstall 2.7.15

Using Your New Python

Now that you’ve installed a couple of different Python versions, let’s see some basics on how to use them. First, check what versions of Python you have available:

$ pyenv versions
* system (set by /home/realpython/.pyenv/version)
  2.7.15
  3.6.8
  3.8-dev

The * indicates that the system Python version is active currently. You’ll also notice that this is set by a file in your root pyenv directory. This means that, by default, you are still using your system Python:

$ python -V
Python 2.7.12

If you try to confirm this using which, you’ll see this:

$ which python
/home/realpython/.pyenv/shims/python

This might be surprising, but this is how pyenv works. pyenv inserts itself into your PATH and from your OS’s perspective is the executable that is getting called. If you want to see the actual path, you can run the following:

$ pyenv which python
/usr/bin/python

If, for example, you wanted to use version 2.7.15, then you can use the global command:

$ pyenv global 2.7.15
$ python -V
Python 2.7.15

$ pyenv versions
  system
* 2.7.15 (set by /home/realpython/.pyenv/version)
  3.6.8
  3.8-dev

Pro Tip: A great way to get peace of mind that the version of Python you just installed is working properly is to run the built-in test suite:

$ pyenv global 3.8-dev
$ python -m test

This will kick off lots of internal Python tests that will verify your installation. You can just kick back and watch the tests pass.

If you ever want to go back to the system version of Python as the default, you can run this:

$ pyenv global system
$ python -V
Python 2.7.12

You can now switch between different versions of Python with ease. This is just the beginning. If you have many versions that you want to switch between, typing these commands consistently is tedious. This section goes over the basics, but a better workflow is described in working with multiple environments.

Exploring `pyenv` Commands

pyenv offers many commands. You can see a complete list of all available commands with this:

$ pyenv commands
activate
commands
completions
deactivate
...
virtualenvs
whence
which

This outputs all command names. Each command has a --help flag that will give you more detailed information. For example, if you wanted to see more information on the shims command you could run the following:

$ pyenv shims --help
Usage: pyenv shims [--short]

List existing pyenv shims

The help message describes what the command is used for and any options you can use in conjunction with the command. In the following sections, you’ll find a quick, high-level overview of the most used commands.

`install`

You’ve already seen the install command above. This command can be used to install a specific version of Python. For example, if you wanted to install 3.6.8 you would use this:

$ pyenv install 3.6.8

The output shows us pyenv downloading and installing Python. Some of the common flags you may want to use are the following:

Flag	Description
`-l/--list`	Lists all available Python versions for installation
`-g/--debug`	Builds a debug version of Python
`-v/--verbose`	Verbose mode: print compilation status to stdout

`versions`

The versions command displays all currently installed Python versions:

$ pyenv versions
* system (set by /home/realpython/.pyenv/version)
  2.7.15
  3.6.8
  3.8-dev

This output shows not only that 2.7.15, 3.6.8, 3.8-dev, and your system Python are installed, but also shows you that the system Python is active. If you only care about the current active version, you can use the following command:

$ pyenv version
system (set by /home/realpython/.pyenv/version)

This command is similar to versions but only shows you the current active Python version.

`which`

The which command is helpful for determining the full path to a system executable. Because pyenv works by using shims, this command allows you to see the full path to the executable pyenv is running. For example, if you wanted to see where pip is installed, you could run this:

$ pyenv which pip
/home/realpython/.pyenv/versions/3.6.8/bin/pip

The output displays the full system path for pip. This can be helpful when you’ve installed command-line applications.

`global`

The global command sets the global Python version. This can be overridden with other commands, but is useful for ensuring you use a particular Python version by default. If you wanted to use 3.6.8 by default, then you could run this:

$ pyenv global 3.6.8

This command sets the ~/.pyenv/version to 3.6.8. For more information, see the section on specifying your Python version.

`local`

The local command is often used to set an application-specific Python version. You could use it to set the version to 2.7.15:

$ pyenv local 2.7.15

This command creates a .python-version file in your current directory. If you have pyenv active in your environment, this file will automatically activate this version for you.

`shell`

The shell command is used to set a shell-specific Python version. For example, if you wanted to test out the 3.8-dev version of Python, you can do this:

$ pyenv shell 3.8-dev

This command activates the version specified by setting the PYENV_VERSION environment variable. This command overwrites any applications or global settings you may have. If you want to deactivate the version, you can use the --unset flag.

Specifying Your Python Version

One of the more confusing parts of pyenv is how exactly the python command gets resolved and what commands can be used to modify it. As mentioned in the commands, there are 3 ways to modify which version of python you’re using. So how do all these commands interact with one another? The resolution order looks a little something like this:

This pyramid is meant to be read from top to bottom. The first of these options that pyenv can find is the option it will use. Let’s see a quick example:

$ pyenv versions
* system (set by /home/realpython/.pyenv/version)
  2.7.15
  3.6.8
  3.8-dev

Here, your system Python is being used as denoted by the *. To exercise the next most global setting, you use global:

$ pyenv global 3.6.8
$ pyenv versions
  system
  2.7.15
* 3.6.8 (set by /home/realpython/.pyenv/version)
  3.8-dev

You can see that now pyenv wants to use 3.6.8 as our Python version. It even indicates the location of the file it found. That file does indeed exist, and you can list its contents:

$ cat ~/.pyenv/version
3.6.8

Now, let’s create a .python-version file with local:

$ pyenv local 2.7.15
$ pyenv versions
  system
* 2.7.15 (set by /home/realpython/.python-version)
  3.6.8
  3.8-dev
$ ls -a
.  ..  .python-version
$ cat .python-version
2.7.15

Here again, pyenv indicates how it would resolve our python command. This time it comes from ~/.python-version. Note that the searching for .python-version is recursive:

$ mkdir subdirectory
$ cd subdirectory
$ ls -la # Notice no .python-version file
. ..
$ pyenv versions
  system
* 2.7.15 (set by /home/realpython/.python-version)
  3.6.8
  3.8-dev

Even though there isn’t a .python-version in subdirectory, the version is still set to 2.7.15 because .python-version exists in a parent directory.

Finally, you can set the Python version with shell:

$ pyenv shell 3.8-dev
$ pyenv versions
  system
  2.7.15
  3.6.8
* 3.8-dev (set by PYENV_VERSION environment variable)

All this did is set the $PYENV_VERSION environment variable:

$ echo $PYENV_VERSION
3.8-dev

If you’re feeling overwhelmed by the options, the section on working with multiple environments goes over an opinionated process for managing these files, mostly using local.

Virtual Environments and `pyenv`

Virtual environments are a big part of managing Python installations and applications. If you haven’t heard of virtual environments before, you can check out Python Virtual Environments: A Primer.

Virtual environments and pyenv are a match made in heaven. pyenv has a wonderful plugin called pyenv-virtualenv that makes working with multiple Python version and multiple virtual environments a breeze. If you’re wondering what the difference is between pyenv, pyenv-virtualenv, and tools like virtualenv or venv, then don’t worry. You’re not alone.

Here’s what you need to know:

pyenv manages multiple versions of Python itself.
virtualenv/venv manages virtual environments for a specific Python version.
pyenv-virtualenv manages virtual environments for across varying versions of Python.

If you’re a die-hard virtualenv or venv user, don’t worry: pyenv plays nicely with either. In fact, you can keep the same workflow you’ve had if you’d prefer, though I think pyenv-virtualenv makes for a nicer experience when you’re switching between multiple environments that require different Python versions.

The good news is that since you used the pyenv-installer script to install pyenv, you already have pyenv-virtualenv installed and ready to go.

Creating Virtual Environments

Creating a virtual environment is a single command:

$ pyenv virtualenv <python_version> <environment_name>

Technically, the <python_version> is optional, but you should consider always specifying it so that you’re certain of what Python version you’re using.

The <environment_name> is just a name for you to help keep your environments separate. A good practice is to name your environments the same name as your project. For example, if you were working on myproject and wanted to develop against Python 3.6.8, you would run this:

$ pyenv virtualenv 3.6.8 myproject

The output includes messages that show a couple of extra Python packages getting installed, namely wheel, pip, and setuptools. This is strictly for convenience and just sets up a more full featured environment for each of your virtual environments.

Activating Your Versions

Now that you’ve created your virtual environment, using it is the next step. Normally, you should activate your environments by running the following:

$ pyenv local myproject

You’ve seen the pyenv local command before, but this time, instead of specifying a Python version, you specify an environment. This creates a .python-version file in your current working directory and because you ran eval "$(pyenv virtualenv-init -)" in your environment, the environment will automatically be activated.

You can verify this by running the following:

$ pyenv which python
/home/realpython/.pyenv/versions/myproject/bin/python

You can see a new version has been created called myproject and the python executable is pointing to that version. If you look at any executable this environment provides, you will see the same thing. Take, for example, pip:

$ pyenv which pip
/home/realpython/.pyenv/versions/myproject/bin/pip

If you did not configure eval "$(pyenv virtualenv-init -)" to run in your shell, you can manually activate/deactivate your Python versions with this:

$ pyenv activate <environment_name>
$ pyenv deactivate

The above is what pyenv-virtualenv is doing when it enters or exits a directory with a .python-version file in it.

Working With Multiple Environments

Putting everything you’ve learned together, you can work effectively with multiple environments. Let’s assume you have the following versions installed:

$ pyenv versions
* system (set by /home/realpython/.pyenv/version)
  2.7.15
  3.6.8
  3.8-dev

Now you want to work on two different, aptly named, projects:

project1 supports Python 2.7 and 3.6.
project2 supports Python 3.6 and experiments with 3.8-dev.

You can see that, by default, you are using the system Python, which is indicated by the * in the pyenv versions output. First, create a virtual environment for the first project:

$ cd project1/
$ pyenv which python
/usr/bin/python
$ pyenv virtualenv 3.6.8 project1
...
$ pyenv local project1
$ python -V
/home/realpython/.pyenv/versions/project1/bin/python

Finally, notice that when you cd out of the directory, you default back to the system Python:

$ cd $HOME
$ pyenv which python
/usr/bin/python

You can follow the above steps and create a virtual environment for project2:

$ cd project2/
$ pyenv which python
/usr/bin/python
$ pyenv virtualenv 3.8-dev project2
...
$ pyenv local 3.8-dev
$ pyenv which python
/home/realpython/.pyenv/versions/3.8-dev/bin/python

These are one time steps for your projects. Now, as you cd between the projects, your environments will automatically activate:

$ cd project2/
$ python -V
Python 3.8.0a0
$ cd ../project1
$ python -V
Python 3.6.8

No more remembering to activate environments: you can switch between all your projects, and pyenv will take care of automatically activating the correct Python versions and the correct virtual environments.

Activating Multiple Versions Simultaneously

As described in the example above, project2 uses experimental features in 3.8. Suppose you wanted to ensure that your code still works on Python 3.6. If you try running python3.6, you’ll get this:

$ cd project2/
$ python3.6 -V
pyenv: python3.6: command not found

The `python3.6' command exists in these Python versions:
  3.6.8
  3.6.8/envs/project1
  project1

pyenv informs you that, while Python 3.6 is not available in the current active environment, it is available in other environments. pyenv gives you a way to activate multiple environments at once using a familiar command:

$ pyenv local project2 3.6.8

This indicates to pyenv that you would like to use the virtual environment project2 as the first option. So if a command, for example python, can be resolved in both environments, it will pick project2 before 3.6.8. Let’s see what happens if you run this:

$ python3.6 -V
Python 3.6.8

Here, pyenv attempts to find the python3.6 command, and because it finds it in an environment that is active, it allows the command to execute. This is extremely useful for tools like tox that require multiple versions of Python to be available on your PATH in order to execute.

Pro Tip: If you’re using tox and pyenv, you should checkout the tox-pyenv package.

Suppose that in the above example, you’ve found a compatibility problem with your library and would like to do some local testing. The testing requires that you install all the dependencies. You should follow the steps to create a new environment:

$ pyenv virtualenv 3.6.8 project2-tmp
$ pyenv local project2-tmp

Once you’re satisfied with your local testing, you can easily switch back to your default environment:

$ pyenv local project2 3.6.8

Conclusion

You can now more easily contribute to a project that wants to support multiple environments. You can also more easily test out the latest and greatest Python versions without having to worry about messing up your development machine, all with a wonderful tool: pyenv.

You’ve seen how pyenv can help you:

Install multiple versions of Python
Switch between the installed versions
Use virtual environments with pyenv
Activate different Python versions and virtual environments automatically

If you still have questions, feel free to reach out either in the comments section or on Twitter. Additionally, the pyenv documentation is a great resource.

Bonus: Displaying Your Environment Name in Your Command Prompt

If you’re like me and constantly switching between various virtual environments and Python versions, it’s easy to get confused about which version is currently active. I use oh-my-zsh and the agnoster theme, which by default makes my prompt look like this:

At a glance, I don’t know which Python version is active. To figure it out, I would have to run python -V or pyenv version. To help reduce my time spent on figuring out my active Python environment, I add the pyenv virtual environment I’m using to my prompt:

My Python version in this case is project1-venv and is displayed immediately at the beginning of the prompt. This allows me to quickly see what version of Python I’m using right away. If you’d like to use this too, you can use my agnoster-pyenv theme.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Writing Comments in Python

2019-03-08T00:02:05+00:00

Learn how to write Python comments that are clean, concise, and useful. Quickly get up to speed on what the best practices are, which types of comments it’s best to avoid, and how you can practice writing cleaner comments.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

Python String Formatting Tips & Best Practices

How to Create an Index in Django Without Downtime

The Problem With Creating an Index in Django Migrations

Setup

Fake Migration

Execute Raw SQL in Migrations

Reverse Migration Operation

Understand Model State and Database State

When Django Generates a New Migration

How to Separate Database and State in Migrations

Non-Atomic Migrations

Conclusion

Running Python Scripts

Python REST APIs With Flask, Connexion, and SQLAlchemy – Part 3

Who This Article Is For

Additional Dependencies

People Data Extended for Blogging

Brute Force Approach

Advantages

Disadvantages

Relational Database Approach

Advantages

Disadvantages

SQLAlchemy Models

Initialize the Database

Update REST API

Implement the API

Update Response JSON

People

Notes

Updated Swagger UI

Mini-Blogging Web Application

Navigation

Home Page

People Page

Notes Page

Web Application

Conclusion

Python Development in Visual Studio Code (Setup Guide)

What Is Pip? A Guide for New Pythonistas

Getting Started With pip

Installing Packages With pip

Basic Package Installation

Using Requirement Files

Fine-Tuning Requirements

Production vs Development Dependencies

Freezing Requirements for Production

Finding Packages to Use

Uninstalling Packages

Alternatives to pip

Conda Does It All

Pipenv

Poetry

Conclusion: What Is pip?

Idiomatic Pandas: Tricks & Features You May Not Know

Get Started With Django Part 1: Build a Portfolio App

Why You Should Learn Django

The Structure of a Django Website

What You’re Going to Build

Hello, World!

Set Up Your Development Environment

Create a Django Project

Create a Django Application

Create a View

Add Bootstrap to Your App

Showcase Your Projects

Projects App: Models

Projects App: Views

Projects App: Templates

Share Your Knowledge With a Blog

Blog App: Models

Blog App: Django Admin

Blog App: Views

Blog App: Templates

Conclusion

Working With JSON Data in Python

How to Stand Out in a Python Coding Interview

Select the Right Built-In Function for the Job

Iterate With enumerate() Instead of range()

Getting Started With `pip`

Installing Packages With `pip`

Alternatives to `pip`

Conclusion: What Is `pip`?

Iterate With `enumerate()` Instead of `range()`

Use List Comprehensions Instead of `map()` and `filter()`

Debug With `breakpoint()` Instead of `print()`

Sort Complex Lists With `sorted()`

Define Default Values in Dictionaries With `.get()` and `.setdefault()`

Handle Missing Dictionary Keys With `collections.defaultdict()`

Count Hashable Objects With `collections.Counter`

Access Common String Groups With `string` Constants

Generate Permutations and Combinations With `itertools`

`join()` a Thread

Using a `ThreadPoolExecutor`

Basic Synchronization Using `Lock`

Producer-Consumer Using `Lock`

Producer-Consumer Using `Queue`