Real Python

A Beginner’s Guide to the Python time Module

2019-04-22T14:00:00+00:00

The Python time module provides many ways of representing time in code, such as objects, numbers, and strings. It also provides functionality other than representing time, like waiting during code execution and measuring the efficiency of your code.

This article will walk you through the most commonly used functions and objects in time.

By the end of this article, you’ll be able to:

Understand core concepts at the heart of working with dates and times, such as epochs, time zones, and daylight savings time
Represent time in code using floats, tuples, and struct_time
Convert between different time representations
Suspend thread execution
Measure code performance using perf_counter()

You’ll start by learning how you can use a floating point number to represent time.

Free Bonus: Click here to get our free Python Cheat Sheet that shows you the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Dealing With Python Time Using Seconds

One of the ways you can manage the concept of Python time in your application is by using a floating point number that represents the number of seconds that have passed since the beginning of an era—that is, since a certain starting point.

Let’s dive deeper into what that means, why it’s useful, and how you can use it to implement logic, based on Python time, in your application.

The Epoch

You learned in the previous section that you can manage Python time with a floating point number representing elapsed time since the beginning of an era.

Merriam-Webster defines an era as:

A fixed point in time from which a series of years is reckoned
A system of chronological notation computed from a given date as basis

The important concept to grasp here is that, when dealing with Python time, you’re considering a period of time identified by a starting point. In computing, you call this starting point the epoch.

The epoch, then, is the starting point against which you can measure the passage of time.

For example, if you define the epoch to be midnight on January 1, 1970 UTC—the epoch as defined on Windows and most UNIX systems—then you can represent midnight on January 2, 1970 UTC as 86400 seconds since the epoch.

This is because there are 60 seconds in a minute, 60 minutes in an hour, and 24 hours in a day. January 2, 1970 UTC is only one day after the epoch, so you can apply basic math to arrive at that result:

>>>

>>> 60 * 60 * 24
86400

It is also important to note that you can still represent time before the epoch. The number of seconds would just be negative.

For example, you would represent midnight on December 31, 1969 UTC (using an epoch of January 1, 1970) as -86400 seconds.

While January 1, 1970 UTC is a common epoch, it is not the only epoch used in computing. In fact, different operating systems, filesystems, and APIs sometimes use different epochs.

As you saw before, UNIX systems define the epoch as January 1, 1970. The Win32 API, on the other hand, defines the epoch as January 1, 1601.

You can use time.gmtime() to determine your system’s epoch:

>>>

>>> import time
>>> time.gmtime(0)
time.struct_time(tm_year=1970, tm_mon=1, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=3, tm_yday=1, tm_isdst=0)

You’ll learn about gmtime() and struct_time throughout the course of this article. For now, just know that you can use time to discover the epoch using this function.

Now that you understand more about how to measure time in seconds using an epoch, let’s take a look at Python’s time module to see what functions it offers that help you do so.

Python Time in Seconds as a Floating Point Number

First, time.time() returns the number of seconds that have passed since the epoch. The return value is a floating point number to account for fractional seconds:

>>>

>>> from time import time
>>> time()
1551143536.9323719

The number you get on your machine may be very different because the reference point considered to be the epoch may be very different.

Further Reading: Python 3.7 introduced time_ns(), which returns an integer value representing the same elapsed time since the epoch, but in nanoseconds rather than seconds.

Measuring time in seconds is useful for a number of reasons:

You can use a float to calculate the difference between two points in time.
A float is easily serializable, meaning that it can be stored for data transfer and come out intact on the other side.

Sometimes, however, you may want to see the current time represented as a string. To do so, you can pass the number of seconds you get from time() into time.ctime().

Python Time in Seconds as a String Representing Local Time

As you saw before, you may want to convert the Python time, represented as the number of elapsed seconds since the epoch, to a string. You can do so using ctime():

>>>

>>> from time import time, ctime
>>> t = time()
>>> ctime(t)
'Mon Feb 25 19:11:59 2019'

Here, you’ve recorded the current time in seconds into the variable t, then passed t as an argument to ctime(), which returns a string representation of that same time.

Technical Detail: The argument, representing seconds since the epoch, is optional according to the ctime() definition. If you don’t pass an argument, then ctime() uses the return value of time() by default. So, you could simplify the example above:

>>>

>>> from time import ctime
>>> ctime()
'Mon Feb 25 19:11:59 2019'

The string representation of time, also known as a timestamp, returned by ctime() is formatted with the following structure:

Day of the week: Mon (Monday)
Month of the year: Feb (February)
Day of the month: 25
Hours, minutes, and seconds using the 24-hour clock notation: 19:11:59
Year: 2019

The previous example displays the timestamp of a particular moment captured from a computer in the South Central region of the United States. But, let’s say you live in Sydney, Australia, and you executed the same command at the same instant.

Instead of the above output, you’d see the following:

>>>

>>> from time import time, ctime
>>> t = time()
>>> ctime(t)
'Tue Feb 26 12:11:59 2019'

Notice that the day of week, day of month, and hour portions of the timestamp are different than the first example.

These outputs are different because the timestamp returned by ctime() depends on your geographical location.

Note: While the concept of time zones is relative to your physical location, you can modify this in your computer’s settings without actually relocating.

The representation of time dependent on your physical location is called local time and makes use of a concept called time zones.

Note: Since local time is related to your locale, timestamps often account for locale-specific details such as the order of the elements in the string and translations of the day and month abbreviations. ctime() ignores these details.

Let’s dig a little deeper into the notion of time zones so that you can better understand Python time representations.

Understanding Time Zones

A time zone is a region of the world that conforms to a standardized time. Time zones are defined by their offset from Coordinated Universal Time (UTC) and, potentially, the inclusion of daylight savings time (which we’ll cover in more detail later in this article).

Fun Fact: If you’re a native English speaker, you might be wondering why the abbreviation for “Coordinated Universal Time” is UTC rather than the more obvious CUT. However, if you’re a native French speaker, you would call it “Temps Universel Coordonné,” which suggests a different abbreviation: TUC.

Ultimately, the International Telecommunication Union and the International Astronomical Union compromised on UTC as the official abbreviation so that, regardless of language, the abbreviation would be the same.

UTC and Time Zones

UTC is the time standard against which all the world’s timekeeping is synchronized (or coordinated). It is not, itself, a time zone but rather a transcendent standard that defines what time zones are.

UTC time is precisely measured using astronomical time, referring to the Earth’s rotation, and atomic clocks.

Time zones are then defined by their offset from UTC. For example, in North and South America, the Central Time Zone (CT) is behind UTC by five or six hours and, therefore, uses the notation UTC-5:00 or UTC-6:00.

Sydney, Australia, on the other hand, belongs to the Australian Eastern Time Zone (AET), which is ten or eleven hours ahead of UTC (UTC+10:00 or UTC+11:00).

This difference (UTC-6:00 to UTC+10:00) is the reason for the variance you observed in the two outputs from ctime() in the previous examples:

Central Time (CT): 'Mon Feb 25 19:11:59 2019'
Australian Eastern Time (AET): 'Tue Feb 26 12:11:59 2019'

These times are exactly sixteen hours apart, which is consistent with the time zone offsets mentioned above.

You may be wondering why CT can be either five or six hours behind UTC or why AET can be ten or eleven hours ahead. The reason for this is that some areas around the world, including parts of these time zones, observe daylight savings time.

Daylight Savings Time

Summer months generally experience more daylight hours than winter months. Because of this, some areas observe daylight savings time (DST) during the spring and summer to make better use of those daylight hours.

For places that observe DST, their clocks will jump ahead one hour at the beginning of spring (effectively losing an hour). Then, in the fall, the clocks will be reset to standard time.

The letters S and D represent standard time and daylight savings time in time zone notation:

Central Standard Time (CST)
Australian Eastern Daylight Time (AEDT)

When you represent times as timestamps in local time, it is always important to consider whether DST is applicable or not.

ctime() accounts for daylight savings time. So, the output difference listed previously would be more accurate as the following:

Central Standard Time (CST): 'Mon Feb 25 19:11:59 2019'
Australian Eastern Daylight Time (AEDT): 'Tue Feb 26 12:11:59 2019'

Dealing With Python Time Using Data Structures

Now that you have a firm grasp on many fundamental concepts of time including epochs, time zones, and UTC, let’s take a look at more ways to represent time using the Python time module.

Python Time as a Tuple

Instead of using a number to represent Python time, you can use another primitive data structure: a tuple.

The tuple allows you to manage time a little more easily by abstracting some of the data and making it more readable.

When you represent time as a tuple, each element in your tuple corresponds to a specific element of time:

Year
Month as an integer, ranging between 1 (January) and 12 (December)
Day of the month
Hour as an integer, ranging between 0 (12 A.M.) and 23 (11 P.M.)
Minute
Second
Day of the week as an integer, ranging between 0 (Monday) and 6 (Sunday)
Day of the year
Daylight savings time as an integer with the following values:
- 1 is daylight savings time.
- 0 is standard time.
- -1 is unknown.

Using the methods you’ve already learned, you can represent the same Python time in two different ways:

>>>

>>> from time import time, ctime
>>> t = time()
>>> t
1551186415.360564
>>> ctime(t)
'Tue Feb 26 07:06:55 2019'

>>> time_tuple = (2019, 2, 26, 7, 6, 55, 1, 57, 0)

In this case, both t and time_tuple represent the same time, but the tuple provides a more readable interface for working with time components.

Technical Detail: Actually, if you look at the Python time represented by time_tuple in seconds (which you’ll see how to do later in this article), you’ll see that it resolves to 1551186415.0 rather than 1551186415.360564.

This is because the tuple doesn’t have a way to represent fractional seconds.

While the tuple provides a more manageable interface for working with Python time, there is an even better object: struct_time.

Python Time as an Object

The problem with the tuple construct is that it still looks like a bunch of numbers, even though it’s better organized than a single, seconds-based number.

struct_time provides a solution to this by utilizing NamedTuple, from Python’s collections module, to associate the tuple’s sequence of numbers with useful identifiers:

>>>

>>> from time import struct_time
>>> time_tuple = (2019, 2, 26, 7, 6, 55, 1, 57, 0)
>>> time_obj = struct_time(time_tuple)
>>> time_obj
time.struct_time(tm_year=2019, tm_mon=2, tm_mday=26, tm_hour=7, tm_min=6, tm_sec=55, tm_wday=1, tm_yday=57, tm_isdst=0)

Technical Detail: If you’re coming from another language, the terms struct and object might be in opposition to one another.

In Python, there is no data type called struct. Instead, everything is an object.

However, the name struct_time is derived from the C-based time library where the data type is actually a struct.

In fact, Python’s time module, which is implemented in C, uses this struct directly by including the header file times.h.

Now, you can access specific elements of time_obj using the attribute’s name rather than an index:

>>>

>>> day_of_year = time_obj.tm_yday
>>> day_of_year
57
>>> day_of_month = time_obj.tm_mday
>>> day_of_month
26

Beyond the readability and usability of struct_time, it is also important to know because it is the return type of many of the functions in the Python time module.

Converting Python Time in Seconds to an Object

Now that you’ve seen the three primary ways of working with Python time, you’ll learn how to convert between the different time data types.

Converting between time data types is dependent on whether the time is in UTC or local time.

Coordinated Universal Time (UTC)

The epoch uses UTC for its definition rather than a time zone. Therefore, the seconds elapsed since the epoch is not variable depending on your geographical location.

However, the same cannot be said of struct_time. The object representation of Python time may or may not take your time zone into account.

There are two ways to convert a float representing seconds to a struct_time:

UTC
Local time

To convert a Python time float to a UTC-based struct_time, the Python time module provides a function called gmtime().

You’ve seen gmtime() used once before in this article:

>>>

>>> import time
>>> time.gmtime(0)
time.struct_time(tm_year=1970, tm_mon=1, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=3, tm_yday=1, tm_isdst=0)

You used this call to discover your system’s epoch. Now, you have a better foundation for understanding what’s actually happening here.

gmtime() converts the number of elapsed seconds since the epoch to a struct_time in UTC. In this case, you’ve passed 0 as the number of seconds, meaning you’re trying to find the epoch, itself, in UTC.

Note: Notice the attribute tm_isdst is set to 0. This attribute represents whether the time zone is using daylight savings time. UTC never subscribes to DST, so that flag will always be 0 when using gmtime().

As you saw before, struct_time cannot represent fractional seconds, so gmtime() ignores the fractional seconds in the argument:

>>>

>>> import time
>>> time.gmtime(1.99)
time.struct_time(tm_year=1970, tm_mon=1, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=1, tm_wday=3, tm_yday=1, tm_isdst=0)

Notice that even though the number of seconds you passed was very close to 2, the .99 fractional seconds were simply ignored, as shown by tm_sec=1.

The secs parameter for gmtime() is optional, meaning you can call gmtime() with no arguments. Doing so will provide the current time in UTC:

>>>

>>> import time
>>> time.gmtime()
time.struct_time(tm_year=2019, tm_mon=2, tm_mday=28, tm_hour=12, tm_min=57, tm_sec=24, tm_wday=3, tm_yday=59, tm_isdst=0)

Interestingly, there is no inverse for this function within time. Instead, you’ll have to look in Python’s calendar module for a function named timegm():

>>>

>>> import calendar
>>> import time
>>> time.gmtime()
time.struct_time(tm_year=2019, tm_mon=2, tm_mday=28, tm_hour=13, tm_min=23, tm_sec=12, tm_wday=3, tm_yday=59, tm_isdst=0)
>>> calendar.timegm(time.gmtime())
1551360204

timegm() takes a tuple (or struct_time, since it is a subclass of tuple) and returns the corresponding number of seconds since the epoch.

Historical Context: If you’re interested in why timegm() is not in time, you can view the discussion in Python Issue 6280.

In short, it was originally added to calendar because time closely follows C’s time library (defined in time.h), which contains no matching function. The above-mentioned issue proposed the idea of moving or copying timegm() into time.

However, with advances to the datetime library, inconsistencies in the patched implementation of time.timegm(), and a question of how to then handle calendar.timegm(), the maintainers declined the patch, encouraging the use of datetime instead.

Working with UTC is valuable in programming because it’s a standard. You don’t have to worry about DST, time zone, or locale information.

That said, there are plenty of cases when you’d want to use local time. Next, you’ll see how to convert from seconds to local time so that you can do just that.

Local Time

In your application, you may need to work with local time rather than UTC. Python’s time module provides a function for getting local time from the number of seconds elapsed since the epoch called localtime().

The signature of localtime() is similar to gmtime() in that it takes an optional secs argument, which it uses to build a struct_time using your local time zone:

>>>

>>> import time
>>> time.time()
1551448206.86196
>>> time.localtime(1551448206.86196)
time.struct_time(tm_year=2019, tm_mon=3, tm_mday=1, tm_hour=7, tm_min=50, tm_sec=6, tm_wday=4, tm_yday=60, tm_isdst=0)

Notice that tm_isdst=0. Since DST matters with local time, tm_isdst will change between 0 and 1 depending on whether or not DST is applicable for the given time. Since tm_isdst=0, DST is not applicable for March 1, 2019.

In the United States in 2019, daylight savings time begins on March 10. So, to test if the DST flag will change correctly, you need to add 9 days’ worth of seconds to the secs argument.

To compute this, you take the number of seconds in a day (86,400) and multiply that by 9 days:

>>>

>>> new_secs = 1551448206.86196 + (86400 * 9)
>>> time.localtime(new_secs)
time.struct_time(tm_year=2019, tm_mon=3, tm_mday=10, tm_hour=8, tm_min=50, tm_sec=6, tm_wday=6, tm_yday=69, tm_isdst=1)

Now, you’ll see that the struct_time shows the date to be March 10, 2019 with tm_isdst=1. Also, notice that tm_hour has also jumped ahead, to 8 instead of 7 in the previous example, because of daylight savings time.

Since Python 3.3, struct_time has also included two attributes that are useful in determining the time zone of the struct_time:

tm_zone
tm_gmtoff

At first, these attributes were platform dependent, but they have been available on all platforms since Python 3.6.

First, tm_zone stores the local time zone:

>>>

>>> import time
>>> current_local = time.localtime()
>>> current_local.tm_zone
'CST'

Here, you can see that localtime() returns a struct_time with the time zone set to CST (Central Standard Time).

As you saw before, you can also tell the time zone based on two pieces of information, the UTC offset and DST (if applicable):

>>>

>>> import time
>>> current_local = time.localtime()
>>> current_local.tm_gmtoff
-21600
>>> current_local.tm_isdst
0

In this case, you can see that current_local is 21600 seconds behind GMT, which stands for Greenwich Mean Time. GMT is the time zone with no UTC offset: UTC±00:00.

21600 seconds divided by seconds per hour (3,600) means that current_local time is GMT-06:00 (or UTC-06:00).

You can use the GMT offset plus the DST status to deduce that current_local is UTC-06:00 at standard time, which corresponds to the Central standard time zone.

Like gmtime(), you can ignore the secs argument when calling localtime(), and it will return the current local time in a struct_time:

>>>

>>> import time
>>> time.localtime()
time.struct_time(tm_year=2019, tm_mon=3, tm_mday=1, tm_hour=8, tm_min=34, tm_sec=28, tm_wday=4, tm_yday=60, tm_isdst=0)

Unlike gmtime(), the inverse function of localtime() does exist in the Python time module. Let’s take a look at how that works.

Converting a Local Time Object to Seconds

You’ve already seen how to convert a UTC time object to seconds using calendar.timegm(). To convert local time to seconds, you’ll use mktime().

mktime() requires you to pass a parameter called t that takes the form of either a normal 9-tuple or a struct_time object representing local time:

>>>

>>> import time

>>> time_tuple = (2019, 3, 10, 8, 50, 6, 6, 69, 1)
>>> time.mktime(time_tuple)
1552225806.0

>>> time_struct = time.struct_time(time_tuple)
>>> time.mktime(time_struct)
1552225806.0

It’s important to keep in mind that t must be a tuple representing local time, not UTC:

>>>

>>> from time import gmtime, mktime

>>> # 1
>>> current_utc = time.gmtime()
>>> current_utc
time.struct_time(tm_year=2019, tm_mon=3, tm_mday=1, tm_hour=14, tm_min=51, tm_sec=19, tm_wday=4, tm_yday=60, tm_isdst=0)

>>> # 2
>>> current_utc_secs = mktime(current_utc)
>>> current_utc_secs
1551473479.0

>>> # 3
>>> time.gmtime(current_utc_secs)
time.struct_time(tm_year=2019, tm_mon=3, tm_mday=1, tm_hour=20, tm_min=51, tm_sec=19, tm_wday=4, tm_yday=60, tm_isdst=0)

Note: For this example, assume that the local time is March 1, 2019 08:51:19 CST.

This example shows why it’s important to use mktime() with local time, rather than UTC:

gmtime() with no argument returns a struct_time using UTC. current_utc shows March 1, 2019 14:51:19 UTC. This is accurate because CST is UTC-06:00, so UTC should be 6 hours ahead of local time.
mktime() tries to return the number of seconds, expecting local time, but you passed current_utc instead. So, instead of understanding that current_utc is UTC time, it assumes you meant March 1, 2019 14:51:19 CST.
gmtime() is then used to convert those seconds back into UTC, which results in an inconsistency. The time is now March 1, 2019 20:51:19 UTC. The reason for this discrepancy is the fact that mktime() expected local time. So, the conversion back to UTC adds another 6 hours to local time.

Working with time zones is notoriously difficult, so it’s important to set yourself up for success by understanding the differences between UTC and local time and the Python time functions that deal with each.

Converting a Python Time Object to a String

While working with tuples is fun and all, sometimes it’s best to work with strings.

String representations of time, also known as timestamps, help make times more readable and can be especially useful for building intuitive user interfaces.

There are two Python time functions that you use for converting a time.struct_time object to a string:

asctime()
strftime()

You’ll begin by learning aboutasctime().

`asctime()`

You use asctime() for converting a time tuple or struct_time to a timestamp:

>>>

>>> import time
>>> time.asctime(time.gmtime())
'Fri Mar  1 18:42:08 2019'
>>> time.asctime(time.localtime())
'Fri Mar  1 12:42:15 2019'

Both gmtime() and localtime() return struct_time instances, for UTC and local time respectively.

You can use asctime() to convert either struct_time to a timestamp. asctime() works similarly to ctime(), which you learned about earlier in this article, except instead of passing a floating point number, you pass a tuple. Even the timestamp format is the same between the two functions.

As with ctime(), the parameter for asctime() is optional. If you do not pass a time object to asctime(), then it will use the current local time:

>>>

>>> import time
>>> time.asctime()
'Fri Mar  1 12:56:07 2019'

As with ctime(), it also ignores locale information.

One of the biggest drawbacks of asctime() is its format inflexibility. strftime() solves this problem by allowing you to format your timestamps.

`strftime()`

You may find yourself in a position where the string format from ctime() and asctime() isn’t satisfactory for your application. Instead, you may want to format your strings in a way that’s more meaningful to your users.

One example of this is if you would like to display your time in a string that takes locale information into account.

To format strings, given a struct_time or Python time tuple, you use strftime(), which stands for “string format time.”

strftime() takes two arguments:

format specifies the order and form of the time elements in your string.
t is an optional time tuple.

To format a string, you use directives. Directives are character sequences that begin with a % that specify a particular time element, such as:

%d: Day of the month
%m: Month of the year
%Y: Year

For example, you can output the date in your local time using the ISO 8601 standard like this:

>>>

>>> import time
>>> time.strftime('%Y-%m-%d', time.localtime())
'2019-03-01'

Further Reading: While representing dates using Python time is completely valid and acceptable, you should also consider using Python’s datetime module, which provides shortcuts and a more robust framework for working with dates and times together.

For example, you can simplify outputting a date in the ISO 8601 format using datetime:

>>>

>>> from datetime import date
>>> date(year=2019, month=3, day=1).isoformat()
'2019-03-01'

As you saw before, a great benefit of using strftime() over asctime() is its ability to render timestamps that make use of locale-specific information.

For example, if you want to represent the date and time in a locale-sensitive way, you can’t use asctime():

>>>

>>> from time import asctime
>>> asctime()
'Sat Mar  2 15:21:14 2019'

>>> import locale
>>> locale.setlocale(locale.LC_TIME, 'zh_HK')  # Chinese - Hong Kong
'zh_HK'
>>> asctime()
'Sat Mar  2 15:58:49 2019'

Notice that even after programmatically changing your locale, asctime() still returns the date and time in the same format as before.

Technical Detail: LC_TIME is the locale category for date and time formatting. The locale argument 'zh_HK' may be different, depending on your system.

When you use strftime(), however, you’ll see that it accounts for locale:

>>>

>>> from time import strftime, localtime
>>> strftime('%c', localtime())
'Sat Mar  2 15:23:20 2019'

>>> import locale
>>> locale.setlocale(locale.LC_TIME, 'zh_HK')  # Chinese - Hong Kong
'zh_HK'
>>> strftime('%c', localtime())
'六  3/ 2 15:58:12 2019' 2019'

Here, you have successfully utilized the locale information because you used strftime().

Note: %c is the directive for locale-appropriate date and time.

If the time tuple is not passed to the parameter t, then strftime() will use the result of localtime() by default. So, you could simplify the examples above by removing the optional second argument:

>>>

>>> from time import strftime
>>> strftime('The current local datetime is: %c')
'The current local datetime is: Fri Mar  1 23:18:32 2019'

Here, you’ve used the default time instead of passing your own as an argument. Also, notice that the format argument can consist of text other than formatting directives.

Further Reading: Check out this thorough list of directives available to strftime().

The Python time module also includes the inverse operation of converting a timestamp back into a struct_time object.

Converting a Python Time String to an Object

When you’re working with date and time related strings, it can be very valuable to convert the timestamp to a time object.

To convert a time string to a struct_time, you use strptime(), which stands for “string parse time”:

>>>

>>> from time import strptime
>>> strptime('2019-03-01', '%Y-%m-%d')
time.struct_time(tm_year=2019, tm_mon=3, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=4, tm_yday=60, tm_isdst=-1)

The first argument to strptime() must be the timestamp you wish to convert. The second argument is the format that the timestamp is in.

The format parameter is optional and defaults to '%a %b %d %H:%M:%S %Y'. Therefore, if you have a timestamp in that format, you don’t need to pass it as an argument:

>>>

>>> strptime('Fri Mar 01 23:38:40 2019')
time.struct_time(tm_year=2019, tm_mon=3, tm_mday=1, tm_hour=23, tm_min=38, tm_sec=40, tm_wday=4, tm_yday=60, tm_isdst=-1)

Since a struct_time has 9 key date and time components, strptime() must provide reasonable defaults for values for those components it can’t parse from string.

In the previous examples, tm_isdst=-1. This means that strptime() can’t determine by the timestamp whether it represents daylight savings time or not.

Now you know how to work with Python times and dates using the time module in a variety of ways. However, there are other uses for time outside of simply creating time objects, getting Python time strings, and using seconds elapsed since the epoch.

Suspending Execution

One really useful Python time function is sleep(), which suspends the thread’s execution for a specified amount of time.

For example, you can suspend your program’s execution for 10 seconds like this:

>>>

>>> from time import sleep, strftime
>>> strftime('%c')
'Fri Mar  1 23:49:26 2019'
>>> sleep(10)
>>> strftime('%c')
'Fri Mar  1 23:49:36 2019'

Your program will print the first formatted datetime string, then pause for 10 seconds, and finally print the second formatted datetime string.

You can also pass fractional seconds to sleep():

>>>

>>> from time import sleep
>>> sleep(0.5)

sleep() is useful for testing or making your program wait for any reason, but you must be careful not to halt your production code unless you have good reason to do so.

Before Python 3.5, a signal sent to your process could interrupt sleep(). However, in 3.5 and later, sleep() will always suspend execution for at least the amount of specified time, even if the process receives a signal.

sleep() is just one Python time function that can help you test your programs and make them more robust.

Measuring Performance

You can use time to measure the performance of your program.

The way you do this is to use perf_counter() which, as the name suggests, provides a performance counter with a high resolution to measure short distances of time.

To use perf_counter(), you place a counter before your code begins execution as well as after your code’s execution completes:

>>>

>>> from time import perf_counter
>>> def longrunning_function():
...     for i in range(1, 11):
...         time.sleep(i / i ** 2)
...
>>> start = perf_counter()
>>> longrunning_function()
>>> end = perf_counter()
>>> execution_time = (end - start)
>>> execution_time
8.201258441999926

First, start captures the moment before you call the function. end captures the moment after the function returns. The function’s total execution time took (end - start) seconds.

Technical Detail: Python 3.7 introduced perf_counter_ns(), which works the same as perf_counter(), but uses nanoseconds instead of seconds.

perf_counter() (or perf_counter_ns()) is the most precise way to measure the performance of your code using one execution. However, if you’re trying to accurately gauge the performance of a code snippet, I recommend using the Python timeit module.

timeit specializes in running code many times to get a more accurate performance analysis and helps you to avoid oversimplifying your time measurement as well as other common pitfalls.

Conclusion

Congratulations! You now have a great foundation for working with dates and times in Python.

Now, you’re able to:

Use a floating point number, representing seconds elapsed since the epoch, to deal with time
Manage time using tuples and struct_time objects
Convert between seconds, tuples, and timestamp strings
Suspend the execution of a Python thread
Measure performance using perf_counter()

On top of all that, you’ve learned some fundamental concepts surrounding date and time, such as:

Epochs
UTC
Time zones
Daylight savings time

Now, it’s time for you to apply your newfound knowledge of Python time in your real world applications!

Immutability in Python

2019-04-18T14:00:00+00:00

In Python, immutable vs mutable data types and objects types can cause some confusion—and weird bugs. With this video course you’ll see what the difference between mutable and immutable data types is in Python, and how you can use it to your advantage in your own programs.

You’ll also see how to deal with a language quirk in Python that allows objects referenced by immutable types to me modified. Python’s definition of “immutable” can be a bit misleading.

Basically, the promise of “immutability” on tuples is only partly true. The tuple itself cannot be modified, but objects referenced by the tuple can be. This is sometimes called “non-transitive immutability.”

If the tuple has an immutable field like a string for example, then it cannot be modified. A mutable field like a list however, can be edited, even if it’s embedded in the “immutable” tuple.

When the Python documentation refers to an object as being “immutable” they mean the behavior above observed. Other immutable types in Python behave the same way, e.g. namedtuples or frozensets.

If you’ve ever encountered the following exception and now you want to know why, this is the video course for you:

TypeError: 'tuple' object does not support item assignment

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Work With a PDF in Python

2019-04-17T14:00:00+00:00

The Portable Document Format or PDF is a file format that can be used to present and exchange documents reliably across operating systems. While the PDF was originally invented by Adobe, it is now an open standard that is maintained by the International Organization for Standardization (ISO). You can work with a preexisting PDF in Python by using the PyPDF2 package.

PyPDF2 is a pure-Python package that you can use for many different types of PDF operations.

By the end of this article, you’ll know how to do the following:

Extract document information from a PDF in Python
Rotate pages
Merge PDFs
Split PDFs
Add watermarks
Encrypt a PDF

Let’s get started!

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

History of `pyPdf`, `PyPDF2`, and `PyPDF4`

The original pyPdf package was released way back in 2005. The last official release of pyPdf was in 2010. After a lapse of around a year, a company called Phasit sponsored a fork of pyPdf called PyPDF2. The code was written to be backwards compatible with the original and worked quite well for several years, with its last release being in 2016.

There was a brief series of releases of a package called PyPDF3, and then the project was renamed to PyPDF4. All of these projects do pretty much the same thing, but the biggest difference between pyPdf and PyPDF2+ is that the latter versions added Python 3 support. There is a different Python 3 fork of the original pyPdf for Python 3, but that one has not been maintained for many years.

While PyPDF2 was recently abandoned, the new PyPDF4 does not have full backwards compatibility with PyPDF2. Most of the examples in this article will work perfectly fine with PyPDF4, but there are some that cannot, which is why PyPDF4 is not featured more heavily in this article. Feel free to swap out the imports for PyPDF2 with PyPDF4 and see how it works for you.

`pdfrw`: An Alternative

Patrick Maupin created a package called pdfrw that can do many of the same things that PyPDF2 does. You can use pdfrw for all of the same sorts of tasks that you will learn how to do in this article for PyPDF2, with the notable exception of encryption.

The biggest difference when it comes to pdfrw is that it integrates with the ReportLab package so that you can take a preexisting PDF and build a new one with ReportLab using some or all of the preexisting PDF.

Installation

Installing PyPDF2 can be done with pip or conda if you happen to be using Anaconda instead of regular Python.

Here’s how you would install PyPDF2 with pip:

$ pip install pypdf2

The install is quite quick as PyPDF2 does not have any dependencies. You will likely spend as much time downloading the package as you will installing it.

Now let’s move on and learn how to extract some information from a PDF.

How to Extract Document Information From a PDF in Python

You can use PyPDF2 to extract metadata and some text from a PDF. This can be useful when you’re doing certain types of automation on your preexisting PDF files.

Here are the current types of data that can be extracted:

Author
Creator
Producer
Subject
Title
Number of pages

You need to go find a PDF to use for this example. You can use any PDF you have handy on your machine. To make things easy, I went to Leanpub and grabbed a sample of one of my books for this exercise. The sample you want to download is called reportlab-sample.pdf.

Let’s write some code using that PDF and learn how you can get access to these attributes:

# extract_doc_info.py

from PyPDF2 import PdfFileReader

def extract_information(pdf_path):
    with open(pdf_path, 'rb') as f:
        pdf = PdfFileReader(f)
        information = pdf.getDocumentInfo()
        number_of_pages = pdf.getNumPages()

    txt = f"""
    Information about {pdf_path}: 

    Author: {information.author}
    Creator: {information.creator}
    Producer: {information.producer}
    Subject: {information.subject}
    Title: {information.title}
    Number of pages: {number_of_pages}
    """

    print(txt)
    return information

if __name__ == '__main__':
    path = 'reportlab-sample.pdf'
    extract_information(path)

Here you import PdfFileReader from the PyPDF2 package. The PdfFileReader is a class with several methods for interacting with PDF files. In this example, you call .getDocumentInfo(), which will return an instance of DocumentInformation. This contains most of the information that you’re interested in. You also call .getNumPages() on the reader object, which returns the number of pages in the document.

Note: That last code block uses Python 3’s new f-strings for string formatting. If you’d like to learn more, you can check out Python 3’s f-Strings: An Improved String Formatting Syntax (Guide).

The information variable has several instance attributes that you can use to get the rest of the metadata you want from the document. You print out that information and also return it for potential future use.

While PyPDF2 has .extractText(), which can be used on its page objects (not shown in this example), it does not work very well. Some PDFs will return text and some will return an empty string. When you want to extract text from a PDF, you should check out the PDFMiner project instead. PDFMiner is much more robust and was specifically designed for extracting text from PDFs.

Now you’re ready to learn about rotating PDF pages.

How to Rotate Pages

Occasionally, you will receive PDFs that contain pages that are in landscape mode instead of portrait mode. Or perhaps they are even upside down. This can happen when someone scans a document to PDF or email. You could print the document out and read the paper version or you can use the power of Python to rotate the offending pages.

For this example, you can go and pick out a Real Python article and print it to PDF.

Let’s learn how to rotate a few of the pages of that article with PyPDF2:

# rotate_pages.py

from PyPDF2 import PdfFileReader, PdfFileWriter

def rotate_pages(pdf_path):
    pdf_writer = PdfFileWriter()
    pdf_reader = PdfFileReader(path)
    # Rotate page 90 degrees to the right
    page_1 = pdf_reader.getPage(0).rotateClockwise(90)
    pdf_writer.addPage(page_1)
    # Rotate page 90 degrees to the left
    page_2 = pdf_reader.getPage(1).rotateCounterClockwise(90)
    pdf_writer.addPage(page_2)
    # Add a page in normal orientation
    pdf_writer.addPage(pdf_reader.getPage(2))

    with open('rotate_pages.pdf', 'wb') as fh:
        pdf_writer.write(fh)

if __name__ == '__main__':
    path = 'Jupyter_Notebook_An_Introduction.pdf'
    rotate_pages(path)

For this example, you need to import the PdfFileWriter in addition to PdfFileReader because you will need to write out a new PDF. rotate_pages() takes in the path to the PDF that you want to modify. Within that function, you will need to create a writer object that you can name pdf_writer and a reader object called pdf_reader.

Next, you can use .GetPage() to get the desired page. Here you grab page zero, which is the first page. Then you call the page object’s .rotateClockwise() method and pass in 90 degrees. Then for page two, you call .rotateCounterClockwise() and pass it 90 degrees as well.

Note: The PyPDF2 package only allows you to rotate a page in increments of 90 degrees. You will receive an AssertionError otherwise.

After each call to the rotation methods, you call .addPage(). This will add the rotated version of the page to the writer object. The last page that you add to the writer object is page 3 without any rotation done to it.

Finally you write out the new PDF using .write(). It takes a file-like object as its parameter. This new PDF will contain three pages. The first two will be rotated in opposite directions of each other and be in landscape while the third page is a normal page.

Now let’s learn how you can merge multiple PDFs into one.

How to Merge PDFs

There are many situations where you will want to take two or more PDFs and merge them together into a single PDF. For example, you might have a standard cover page that needs to go on to many types of reports. You can use Python to help you do that sort of thing.

For this example, you can open up a PDF and print a page out as a separate PDF. Then do that again, but with a different page. That will give you a couple of inputs to use for example purposes.

Let’s go ahead and write some code that you can use to merge PDFs together:

# pdf_merging.py

from PyPDF2 import PdfFileReader, PdfFileWriter

def merge_pdfs(paths, output):
    pdf_writer = PdfFileWriter()

    for path in paths:
        pdf_reader = PdfFileReader(path)
        for page in range(pdf_reader.getNumPages()):
            # Add each page to the writer object
            pdf_writer.addPage(pdf_reader.getPage(page))

    # Write out the merged PDF
    with open(output, 'wb') as out:
        pdf_writer.write(out)

if __name__ == '__main__':
    paths = ['document1.pdf', 'document2.pdf']
    merge_pdfs(paths, output='merged.pdf')

You can use merge_pdfs() when you have a list of PDFs that you want to merge together. You will also need to know where to save the result, so this function takes a list of input paths and an output path.

Then you loop over the inputs and create a PDF reader object for each of them. Next you will iterate over all the pages in the PDF file and use .addPage() to add each of those pages to itself.

Once you’re finished iterating over all of the pages of all of the PDFs in your list, you will write out the result at the end.

One item I would like to point out is that you could enhance this script a bit by adding in a range of pages to be added if you didn’t want to merge all the pages of each PDF. If you’d like a challenge, you could also create a command line interface for this function using Python’s argparse module.

Let’s find out how to do the opposite of merging!

How to Split PDFs

There are times where you might have a PDF that you need to split up into multiple PDFs. This is especially true of PDFs that contain a lot of scanned-in content, but there are a plethora of good reasons for wanting to split a PDF.

Here’s how you can use PyPDF2 to split your PDF into multiple files:

# pdf_splitting.py

from PyPDF2 import PdfFileReader, PdfFileWriter

def split(path, name_of_split):
    pdf = PdfFileReader(path)
    for page in range(pdf.getNumPages()):
        pdf_writer = PdfFileWriter()
        pdf_writer.addPage(pdf.getPage(page))

        output = f'{name_of_split}{page}.pdf'
        with open(output, 'wb') as output_pdf:
            pdf_writer.write(output_pdf)

if __name__ == '__main__':
    path = 'Jupyter_Notebook_An_Introduction.pdf'
    split(path, 'jupyter_page')

In this example, you once again create a PDF reader object and loop over its pages. For each page in the PDF, you will create a new PDF writer instance and add a single page to it. Then you will write that page out to a uniquely named file. When the script is finished running, you should have each page of the original PDF split into separate PDFs.

Now let’s take a moment to learn how you can add a watermark to your PDF.

How to Add Watermarks

Watermarks are identifying images or patterns on printed and digital documents. Some watermarks can only be seen in special lighting conditions. The reason watermarking is important is that it allows you to protect your intellectual property, such as your images or PDFs. Another term for watermark is overlay.

You can use Python and PyPDF2 to watermark your documents. You need to have a PDF that only contains your watermark image or text.

Let’s learn how to add a watermark now:

# pdf_watermarker.py

from PyPDF2 import PdfFileWriter, PdfFileReader

def create_watermark(input_pdf, output, watermark):
    watermark_obj = PdfFileReader(watermark)
    watermark_page = watermark_obj.getPage(0)

    pdf_reader = PdfFileReader(input_pdf)
    pdf_writer = PdfFileWriter()

    # Watermark all the pages
    for page in range(pdf_reader.getNumPages()):
        page = pdf_reader.getPage(page)
        page.mergePage(watermark_page)
        pdf_writer.addPage(page)

    with open(output, 'wb') as out:
        pdf_writer.write(out)

if __name__ == '__main__':
    create_watermark(
        input_pdf='Jupyter_Notebook_An_Introduction.pdf', 
        output='watermarked_notebook.pdf',
        watermark='watermark.pdf')

create_watermark() accepts three arguments:

input_pdf: the PDF file path to be watermarked
output: the path you want to save the watermarked version of the PDF
watermark: a PDF that contains your watermark image or text

In the code, you open up the watermark PDF and grab just the first page from the document as that is where your watermark should reside. Then you create a PDF reader object using the input_pdf and a generic pdf_writer object for writing out the watermarked PDF.

The next step is to iterate over the pages in the input_pdf. This is where the magic happens. You will need to call .mergePage() and pass it the watermark_page. When you do that, it will overlay the watermark_page on top of the current page. Then you add that newly merged page to your pdf_writer object.

Finally, you write the newly watermarked PDF out to disk, and you’re done!

The last topic you will learn about is how PyPDF2 handles encryption.

How to Encrypt a PDF

PyPDF2 currently only supports adding a user password and an owner password to a preexisting PDF. In PDF land, an owner password will basically give you administrator privileges over the PDF and allow you to set permissions on the document. On the other hand, the user password just allows you to open the document.

As far as I can tell, PyPDF2 doesn’t actually allow you to set any permissions on the document even though it does allow you to set the owner password.

Regardless, this is how you can add a password, which will also inherently encrypt the PDF:

# pdf_encrypt.py

from PyPDF2 import PdfFileWriter, PdfFileReader

def add_encryption(input_pdf, output_pdf, password):
    pdf_writer = PdfFileWriter()
    pdf_reader = PdfFileReader(input_pdf)

    for page in range(pdf_reader.getNumPages()):
        pdf_writer.addPage(pdf_reader.getPage(page))

    pdf_writer.encrypt(user_pwd=password, owner_pwd=None, 
                       use_128bit=True)

    with open(output_pdf, 'wb') as fh:
        pdf_writer.write(fh)

if __name__ == '__main__':
    add_encryption(input_pdf='reportlab-sample.pdf',
                   output_pdf='reportlab-encrypted.pdf',
                   password='twofish')

add_encryption() takes in the input and output PDF paths as well as the password that you want to add to the PDF. It then opens a PDF writer and a reader object, as before. Since you will want to encrypt the entire input PDF, you will need to loop over all of its pages and add them to the writer.

The final step is to call .encrypt(), which takes the user password, the owner password, and whether or not 128-bit encryption should be added. The default is for 128-bit encryption to be turned on. If you set it to False, then 40-bit encryption will be applied instead.

Note: PDF encryption uses either RC4 or AES (Advanced Encryption Standard) to encrypt the PDF according to pdflib.com.

Just because you have encrypted your PDF does not mean it is necessarily secure. There are tools to remove passwords from PDFs. If you’d like to learn more, Carnegie Mellon University has an interesting paper on the topic.

Conclusion

The PyPDF2 package is quite useful and is usually pretty fast. You can use PyPDF2 to automate large jobs and leverage its capabilities to help you do your job better!

In this tutorial, you learned how to do the following:

Extract metadata from a PDF
Rotate pages
Merge and split PDFs
Add watermarks
Add encryption

Also keep an eye on the newer PyPDF4 package as it will likely replace PyPDF2 soon. You might also want to check out pdfrw, which can do many of the same things that PyPDF2 can do.

Hands-on Python 3 Concurrency With the asyncio Module

2019-04-16T14:00:00+00:00

Learn how to speed up your Python 3 programs using concurrency and the new asyncio module in the standard library.

First, you’ll explore the key terms of parallel programming. Next, you’ll see step-by-step how to leverage concurrency and parallelism in your own programs, all the way to building a complete HTTP downloader example app using asyncio and aiohttp.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Linear Regression in Python

2019-04-15T14:00:00+00:00

We’re living in the era of large amounts of data, powerful computers, and artificial intelligence. This is just the beginning. Data science and machine learning are driving image recognition, autonomous vehicles development, decisions in the financial and energy sectors, advances in medicine, the rise of social networks, and more. Linear regression is an important part of this.

Linear regression is one of the fundamental statistical and machine learning techniques. Whether you want to do statistics, machine learning, or scientific computing, there are good chances that you’ll need it. It’s advisable to learn it first and then proceed towards more complex methods.

By the end of this article, you’ll have learned:

What linear regression is
What linear regression is used for
How linear regression works
How to implement linear regression in Python, step by step

Free Bonus: Click here to get access to a free NumPy Resources Guide that points you to the best tutorials, videos, and books for improving your NumPy skills.

Regression

Regression analysis is one of the most important fields in statistics and machine learning. There are many regression methods available. Linear regression is one of them.

What Is Regression?

Regression searches for relationships among variables.

For example, you can observe several employees of some company and try to understand how their salaries depend on the features, such as experience, level of education, role, city they work in, and so on.

This is a regression problem where data related to each employee represent one observation. The presumption is that the experience, education, role, and city are the independent features, while the salary depends on them.

Similarly, you can try to establish a mathematical dependence of the prices of houses on their areas, numbers of bedrooms, distances to the city center, and so on.

Generally, in regression analysis, you usually consider some phenomenon of interest and have a number of observations. Each observation has two or more features. Following the assumption that (at least) one of the features depends on the others, you try to establish a relation among them.

In other words, you need to find a function that maps some features or variables to others sufficiently well.

The dependent features are called the dependent variables, outputs, or responses.

The independent features are called the independent variables, inputs, or predictors.

Regression problems usually have one continuous and unbounded dependent variable. The inputs, however, can be continuous, discrete, or even categorical data such as gender, nationality, brand, and so on.

It is a common practice to denote the outputs with 𝑦 and inputs with 𝑥. If there are two or more independent variables, they can be represented as the vector 𝐱 = (𝑥₁, …, 𝑥ᵣ), where 𝑟 is the number of inputs.

When Do You Need Regression?

Typically, you need regression to answer whether and how some phenomenon influences the other or how several variables are related. For example, you can use it to determine if and to what extent the experience or gender impact salaries.

Regression is also useful when you want to forecast a response using a new set of predictors. For example, you could try to predict electricity consumption of a household for the next hour given the outdoor temperature, time of day, and number of residents in that household.

Regression is used in many different fields: economy, computer science, social sciences, and so on. Its importance rises every day with the availability of large amounts of data and increased awareness of the practical value of data.

Linear Regression

Linear regression is probably one of the most important and widely used regression techniques. It’s among the simplest regression methods. One of its main advantages is the ease of interpreting results.

Problem Formulation

When implementing linear regression of some dependent variable 𝑦 on the set of independent variables 𝐱 = (𝑥₁, …, 𝑥ᵣ), where 𝑟 is the number of predictors, you assume a linear relationship between 𝑦 and 𝐱: 𝑦 = 𝛽₀ + 𝛽₁𝑥₁ + ⋯ + 𝛽ᵣ𝑥ᵣ + 𝜀. This equation is the regression equation. 𝛽₀, 𝛽₁, …, 𝛽ᵣ are the regression coefficients, and 𝜀 is the random error.

Linear regression calculates the estimators of the regression coefficients or simply the predicted weights, denoted with 𝑏₀, 𝑏₁, …, 𝑏ᵣ. They define the estimated regression function 𝑓(𝐱) = 𝑏₀ + 𝑏₁𝑥₁ + ⋯ + 𝑏ᵣ𝑥ᵣ. This function should capture the dependencies between the inputs and output sufficiently well.

The estimated or predicted response, 𝑓(𝐱ᵢ), for each observation 𝑖 = 1, …, 𝑛, should be as close as possible to the corresponding actual response 𝑦ᵢ. The differences 𝑦ᵢ - 𝑓(𝐱ᵢ) for all observations 𝑖 = 1, …, 𝑛, are called the residuals. Regression is about determining the best predicted weights, that is the weights corresponding to the smallest residuals.

To get the best weights, you usually minimize the sum of squared residuals (SSR) for all observations 𝑖 = 1, …, 𝑛: SSR = Σᵢ(𝑦ᵢ - 𝑓(𝐱ᵢ))². This approach is called the method of ordinary least squares.

Regression Performance

The variation of actual responses 𝑦ᵢ, 𝑖 = 1, …, 𝑛, occurs partly due to the dependence on the predictors 𝐱ᵢ. However, there is also an additional inherent variance of the output.

The coefficient of determination, denoted as 𝑅², tells you which amount of variation in 𝑦 can be explained by the dependence on 𝐱 using the particular regression model. Larger 𝑅² indicates a better fit and means that the model can better explain the variation of the output with different inputs.

The value 𝑅² = 1 corresponds to SSR = 0, that is to the perfect fit since the values of predicted and actual responses fit completely to each other.

Simple Linear Regression

Simple or single-variate linear regression is the simplest case of linear regression with a single independent variable, 𝐱 = 𝑥.

The following figure illustrates simple linear regression:

Example of simple linear regression

When implementing simple linear regression, you typically start with a given set of input-output (𝑥-𝑦) pairs (green circles). These pairs are your observations. For example, the leftmost observation (green circle) has the input 𝑥 = 5 and the actual output (response) 𝑦 = 5. The next one has 𝑥 = 15 and 𝑦 = 20, and so on.

The estimated regression function (black line) has the equation 𝑓(𝑥) = 𝑏₀ + 𝑏₁𝑥. Your goal is to calculate the optimal values of the predicted weights 𝑏₀ and 𝑏₁ that minimize SSR and determine the estimated regression function. The value of 𝑏₀, also called the intercept, shows the point where the estimated regression line crosses the 𝑦 axis. It is the value of the estimated response 𝑓(𝑥) for 𝑥 = 0. The value of 𝑏₁ determines the slope of the estimated regression line.

The predicted responses (red squares) are the points on the regression line that correspond to the input values. For example, for the input 𝑥 = 5, the predicted response is 𝑓(5) = 8.33 (represented with the leftmost red square).

The residuals (vertical dashed gray lines) can be calculated as 𝑦ᵢ - 𝑓(𝐱ᵢ) = 𝑦ᵢ - 𝑏₀ - 𝑏₁𝑥ᵢ for 𝑖 = 1, …, 𝑛. They are the distances between the green circles and red squares. When you implement linear regression, you are actually trying to minimize these distances and make the red squares as close to the predefined green circles as possible.

Multiple Linear Regression

Multiple or multivariate linear regression is a case of linear regression with two or more independent variables.

If there are just two independent variables, the estimated regression function is 𝑓(𝑥₁, 𝑥₂) = 𝑏₀ + 𝑏₁𝑥₁ + 𝑏₂𝑥₂. It represents a regression plane in a three-dimensional space. The goal of regression is to determine the values of the weights 𝑏₀, 𝑏₁, and 𝑏₂ such that this plane is as close as possible to the actual responses and yield the minimal SSR.

The case of more than two independent variables is similar, but more general. The estimated regression function is 𝑓(𝑥₁, …, 𝑥ᵣ) = 𝑏₀ + 𝑏₁𝑥₁ + ⋯ +𝑏ᵣ𝑥ᵣ, and there are 𝑟 + 1 weights to be determined when the number of inputs is 𝑟.

Polynomial Regression

You can regard polynomial regression as a generalized case of linear regression. You assume the polynomial dependence between the output and inputs and, consequently, the polynomial estimated regression function.

In other words, in addition to linear terms like 𝑏₁𝑥₁, your regression function 𝑓 can include non-linear terms such as 𝑏₂𝑥₁², 𝑏₃𝑥₁³, or even 𝑏₄𝑥₁𝑥₂, 𝑏₅𝑥₁²𝑥₂, and so on.

The simplest example of polynomial regression has a single independent variable, and the estimated regression function is a polynomial of degree 2: 𝑓(𝑥) = 𝑏₀ + 𝑏₁𝑥 + 𝑏₂𝑥².

Now, remember that you want to calculate 𝑏₀, 𝑏₁, and 𝑏₂, which minimize SSR. These are your unknowns!

Keeping this in mind, compare the previous regression function with the function 𝑓(𝑥₁, 𝑥₂) = 𝑏₀ + 𝑏₁𝑥₁ + 𝑏₂𝑥₂ used for linear regression. They look very similar and are both linear functions of the unknowns 𝑏₀, 𝑏₁, and 𝑏₂. This is why you can solve the polynomial regression problem as a linear problem with the term 𝑥² regarded as an input variable.

In the case of two variables and the polynomial of degree 2, the regression function has this form: 𝑓(𝑥₁, 𝑥₂) = 𝑏₀ + 𝑏₁𝑥₁ + 𝑏₂𝑥₂ + 𝑏₃𝑥₁² + 𝑏₄𝑥₁𝑥₂ + 𝑏₅𝑥₂². The procedure for solving the problem is identical to the previous case. You apply linear regression for five inputs: 𝑥₁, 𝑥₂, 𝑥₁², 𝑥₁𝑥₂, and 𝑥₂². What you get as the result of regression are the values of six weights which minimize SSR: 𝑏₀, 𝑏₁, 𝑏₂, 𝑏₃, 𝑏₄, and 𝑏₅.

Of course, there are more general problems, but this should be enough to illustrate the point.

Underfitting and Overfitting

One very important question that might arise when you’re implementing polynomial regression is related to the choice of the optimal degree of the polynomial regression function.

There is no straightforward rule for doing this. It depends on the case. You should, however, be aware of two problems that might follow the choice of the degree: underfitting and overfitting.

Underfitting occurs when a model can’t accurately capture the dependencies among data, usually as a consequence of its own simplicity. It often yields a low 𝑅² with known data and bad generalization capabilities when applied with new data.

Overfitting happens when a model learns both dependencies among data and random fluctuations. In other words, a model learns the existing data too well. Complex models, which have many features or terms, are often prone to overfitting. When applied to known data, such models usually yield high 𝑅². However, they often don’t generalize well and have significantly lower 𝑅² when used with new data.

The next figure illustrates the underfitted, well-fitted, and overfitted models:

Example of underfitted, well-fitted and overfitted models

The top left plot shows a linear regression line that has a low 𝑅². It might also be important that a straight line can’t take into account the fact that the actual response increases as 𝑥 moves away from 25 towards zero. This is likely an example of underfitting.

The top right plot illustrates polynomial regression with the degree equal to 2. In this instance, this might be the optimal degree for modeling this data. The model has a value of 𝑅² that is satisfactory in many cases and shows trends nicely.

The bottom left plot presents polynomial regression with the degree equal to 3. The value of 𝑅² is higher than in the preceding cases. This model behaves better with known data than the previous ones. However, it shows some signs of overfitting, especially for the input values close to 60 where the line starts decreasing, although actual data don’t show that.

Finally, on the bottom right plot, you can see the perfect fit: six points and the polynomial line of the degree 5 (or higher) yield 𝑅² = 1. Each actual response equals its corresponding prediction.

In some situations, this might be exactly what you’re looking for. In many cases, however, this is an overfitted model. It is likely to have poor behavior with unseen data, especially with the inputs larger than 50.

For example, it assumes, without any evidence, that there is a significant drop in responses for 𝑥 > 50 and that 𝑦 reaches zero for 𝑥 near 60. Such behavior is the consequence of excessive effort to learn and fit the existing data.

There are a lot of resources where you can find more information about regression in general and linear regression in particular. The regression analysis page on Wikipedia, Wikipedia’s linear regression article, as well as Khan Academy’s linear regression article are good starting points.

Implementing Linear Regression in Python

It’s time to start implementing linear regression in Python. Basically, all you should do is apply the proper packages and their functions and classes.

Python Packages for Linear Regression

The package NumPy is a fundamental Python scientific package that allows many high-performance operations on single- and multi-dimensional arrays. It also offers many mathematical routines. Of course, it’s open source.

If you’re not familiar with NumPy, you can use the official NumPy User Guide and read Look Ma, No For-Loops: Array Programming With NumPy. In addition, Pure Python vs NumPy vs TensorFlow Performance Comparison can give you a pretty good idea on the performance gains you can achieve when applying NumPy.

The package scikit-learn is a widely used Python library for machine learning, built on top of NumPy and some other packages. It provides the means for preprocessing data, reducing dimensionality, implementing regression, classification, clustering, and more. Like NumPy, scikit-learn is also open source.

You can check the page Generalized Linear Models on the scikit-learn web site to learn more about linear models and get deeper insight into how this package works.

If you want to implement linear regression and need the functionality beyond the scope of scikit-learn, you should consider statsmodels. It’s a powerful Python package for the estimation of statistical models, performing tests, and more. It’s open source as well.

You can find more information on statsmodels on its official web site.

Simple Linear Regression With scikit-learn

Let’s start with the simplest case, which is simple linear regression.

There are five basic steps when you’re implementing linear regression:

Import the packages and classes you need.
Provide data to work with and eventually do appropriate transformations.
Create a regression model and fit it with existing data.
Check the results of model fitting to know whether the model is satisfactory.
Apply the model for predictions.

These steps are more or less general for most of the regression approaches and implementations.

Step 1: Import packages and classes

The first step is to import the package numpy and the class LinearRegression from sklearn.linear_model:

import numpy as np
from sklearn.linear_model import LinearRegression

Now, you have all the functionalities you need to implement linear regression.

The fundamental data type of NumPy is the array type called numpy.ndarray. The rest of this article uses the term array to refer to instances of the type numpy.ndarray.

The class sklearn.linear_model.LinearRegression will be used to perform linear and polynomial regression and make predictions accordingly.

Step 2: Provide data

The second step is defining data to work with. The inputs (regressors, 𝑥) and output (predictor, 𝑦) should be arrays (the instances of the class numpy.ndarray) or similar objects. This is the simplest way of providing data for regression:

x = np.array([5, 15, 25, 35, 45, 55]).reshape((-1, 1))
y = np.array([5, 20, 14, 32, 22, 38])

Now, you have two arrays: the input x and output y. You should call .reshape() on x because this array is required to be two-dimensional, or to be more precise, to have one column and as many rows as necessary. That’s exactly what the argument (-1, 1) of .reshape() specifies.

This is how x and y look now:

>>>

>>> print(x)
[[ 5]
 [15]
 [25]
 [35]
 [45]
 [55]]
>>> print(y)
[ 5 20 14 32 22 38]

As you can see, x has two dimensions, and x.shape is (6, 1), while y has a single dimension, and y.shape is (6,).

Step 3: Create a model and fit it

The next step is to create a linear regression model and fit it using the existing data.

Let’s create an instance of the class LinearRegression, which will represent the regression model:

model = LinearRegression()

This statement creates the variable model as the instance of LinearRegression. You can provide several optional parameters to LinearRegression:

fit_intercept is a Boolean (True by default) that decides whether to calculate the intercept 𝑏₀ (True) or consider it equal to zero (False).
normalize is a Boolean (False by default) that decides whether to normalize the input variables (True) or not (False).
copy_X is a Boolean (True by default) that decides whether to copy (True) or overwrite the input variables (False).
n_jobs is an integer or None (default) and represents the number of jobs used in parallel computation. None usually means one job and -1 to use all processors.

This example uses the default values of all parameters.

It’s time to start using the model. First, you need to call .fit() on model:

model.fit(x, y)

With .fit(), you calculate the optimal values of the weights 𝑏₀ and 𝑏₁, using the existing input and output (x and y) as the arguments. In other words, .fit() fits the model. It returns self, which is the variable model itself. That’s why you can replace the last two statements with this one:

model = LinearRegression().fit(x, y)

This statement does the same thing as the previous two. It’s just shorter.

Step 4: Get results

Once you have your model fitted, you can get the results to check whether the model works satisfactorily and interpret it.

You can obtain the coefficient of determination (𝑅²) with .score() called on model:

>>>

>>> r_sq = model.score(x, y)
>>> print('coefficient of determination:', r_sq)
coefficient of determination: 0.715875613747954

When you’re applying .score(), the arguments are also the predictor x and regressor y, and the return value is 𝑅².

The attributes of model are .intercept_, which represents the coefficient, 𝑏₀ and .coef_, which represents 𝑏₁:

>>>

>>> print('intercept:', model.intercept_)
intercept: 5.633333333333329
>>> print('slope:', model.coef_)
slope: [0.54]

The code above illustrates how to get 𝑏₀ and 𝑏₁. You can notice that .intercept_ is a scalar, while .coef_ is an array.

The value 𝑏₀ = 5.63 (approximately) illustrates that your model predicts the response 5.63 when 𝑥 is zero. The value 𝑏₁ = 0.54 means that the predicted response rises by 0.54 when 𝑥 is increased by one.

You should notice that you can provide y as a two-dimensional array as well. In this case, you’ll get a similar result. This is how it might look:

>>>

>>> new_model = LinearRegression().fit(x, y.reshape((-1, 1)))
>>> print('intercept:', new_model.intercept_)
intercept: [5.63333333]
>>> print('slope:', new_model.coef_)
slope: [[0.54]]

As you can see, this example is very similar to the previous one, but in this case, .intercept_ is a one-dimensional array with the single element 𝑏₀, and .coef_ is a two-dimensional array with the single element 𝑏₁.

Step 5: Predict response

Once there is a satisfactory model, you can use it for predictions with either existing or new data.

To obtain the predicted response, use .predict():

>>>

>>> y_pred = model.predict(x)
>>> print('predicted response:', y_pred, sep='\n')
predicted response:
[ 8.33333333 13.73333333 19.13333333 24.53333333 29.93333333 35.33333333]

When applying .predict(), you pass the regressor as the argument and get the corresponding predicted response.

This is a nearly identical way to predict the response:

>>>

>>> y_pred = model.intercept_ + model.coef_ * x
>>> print('predicted response:', y_pred, sep='\n')
predicted response:
[[ 8.33333333]
 [13.73333333]
 [19.13333333]
 [24.53333333]
 [29.93333333]
 [35.33333333]]

In this case, you multiply each element of x with model.coef_ and add model.intercept_ to the product.

The output here differs from the previous example only in dimensions. The predicted response is now a two-dimensional array, while in the previous case, it had one dimension.

If you reduce the number of dimensions of x to one, these two approaches will yield the same result. You can do this by replacing x with x.reshape(-1), x.flatten(), or x.ravel() when multiplying it with model.coef_.

In practice, regression models are often applied for forecasts. This means that you can use fitted models to calculate the outputs based on some other, new inputs:

>>>

>>> x_new = np.arange(5).reshape((-1, 1))
>>> print(x_new)
[[0]
 [1]
 [2]
 [3]
 [4]]
>>> y_new = model.predict(x_new)
>>> print(y_new)
[5.63333333 6.17333333 6.71333333 7.25333333 7.79333333]

Here .predict() is applied to the new regressor x_new and yields the response y_new. This example conveniently uses arange() from numpy to generate an array with the elements from 0 (inclusive) to 5 (exclusive), that is 0, 1, 2, 3, and 4.

You can find more information about LinearRegression on the official documentation page.

Multiple Linear Regression With scikit-learn

You can implement multiple linear regression following the same steps as you would for simple regression.

Steps 1 and 2: Import packages and classes, and provide data

First, you import numpy and sklearn.linear_model.LinearRegression and provide known inputs and output:

import numpy as np
from sklearn.linear_model import LinearRegression

x = [[0, 1], [5, 1], [15, 2], [25, 5], [35, 11], [45, 15], [55, 34], [60, 35]]
y = [4, 5, 20, 14, 32, 22, 38, 43]
x, y = np.array(x), np.array(y)

That’s a simple way to define the input x and output y. You can print x and y to see how they look now:

>>>

>>> print(x)
[[ 0  1]
 [ 5  1]
 [15  2]
 [25  5]
 [35 11]
 [45 15]
 [55 34]
 [60 35]]
>>> print(y)
[ 4  5 20 14 32 22 38 43]

In multiple linear regression, x is a two-dimensional array with at least two columns, while y is usually a one-dimensional array. This is a simple example of multiple linear regression, and x has exactly two columns.

Step 3: Create a model and fit it

The next step is to create the regression model as an instance of LinearRegression and fit it with .fit():

model = LinearRegression().fit(x, y)

The result of this statement is the variable model referring to the object of type LinearRegression. It represents the regression model fitted with existing data.

Step 4: Get results

You can obtain the properties of the model the same way as in the case of simple linear regression:

>>>

>>> r_sq = model.score(x, y)
>>> print('coefficient of determination:', r_sq)
coefficient of determination: 0.8615939258756776
>>> print('intercept:', model.intercept_)
intercept: 5.52257927519819
>>> print('slope:', model.coef_)
slope: [0.44706965 0.25502548]

You obtain the value of 𝑅² using .score() and the values of the estimators of regression coefficients with .intercept_ and .coef_. Again, .intercept_ holds the bias 𝑏₀, while now .coef_ is an array containing 𝑏₁ and 𝑏₂ respectively.

In this example, the intercept is approximately 5.52, and this is the value of the predicted response when 𝑥₁ = 𝑥₂ = 0. The increase of 𝑥₁ by 1 yields the rise of the predicted response by 0.45. Similarly, when 𝑥₂ grows by 1, the response rises by 0.26.

Step 5: Predict response

Predictions also work the same way as in the case of simple linear regression:

>>>

>>> y_pred = model.predict(x)
>>> print('predicted response:', y_pred, sep='\n')
predicted response:
[ 5.77760476  8.012953   12.73867497 17.9744479  23.97529728 29.4660957
 38.78227633 41.27265006]

The predicted response is obtained with .predict(), which is very similar to the following:

>>>

>>> y_pred = model.intercept_ + np.sum(model.coef_ * x, axis=1)
>>> print('predicted response:', y_pred, sep='\n')
predicted response:
[ 5.77760476  8.012953   12.73867497 17.9744479  23.97529728 29.4660957
 38.78227633 41.27265006]

You can predict the output values by multiplying each column of the input with the appropriate weight, summing the results and adding the intercept to the sum.

You can apply this model to new data as well:

>>>

>>> x_new = np.arange(10).reshape((-1, 2))
>>> print(x_new)
[[0 1]
 [2 3]
 [4 5]
 [6 7]
 [8 9]]
>>> y_new = model.predict(x_new)
>>> print(y_new)
[ 5.77760476  7.18179502  8.58598528  9.99017554 11.3943658 ]

That’s the prediction using a linear regression model.

Polynomial Regression With scikit-learn

Implementing polynomial regression with scikit-learn is very similar to linear regression. There is only one extra step: you need to transform the array of inputs to include non-linear terms such as 𝑥².

Step 1: Import packages and classes

In addition to numpy and sklearn.linear_model.LinearRegression, you should also import the class PolynomialFeatures from sklearn.preprocessing:

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

The import is now done, and you have everything you need to work with.

Step 2a: Provide data

This step defines the input and output and is the same as in the case of linear regression:

x = np.array([5, 15, 25, 35, 45, 55]).reshape((-1, 1))
y = np.array([15, 11, 2, 8, 25, 32])

Now you have the input and output in a suitable format. Keep in mind that you need the input to be a two-dimensional array. That’s why .reshape() is used.

Step 2b: Transform input data

This is the new step you need to implement for polynomial regression!

As you’ve seen earlier, you need to include 𝑥² (and perhaps other terms) as additional features when implementing polynomial regression. For that reason, you should transform the input array x to contain the additional column(s) with the values of 𝑥² (and eventually more features).

It’s possible to transform the input array in several ways (like using insert() from numpy), but the class PolynomialFeatures is very convenient for this purpose. Let’s create an instance of this class:

transformer = PolynomialFeatures(degree=2, include_bias=False)

The variable transformer refers to an instance of PolynomialFeatures which you can use to transform the input x.

You can provide several optional parameters to PolynomialFeatures:

degree is an integer (2 by default) that represents the degree of the polynomial regression function.
interaction_only is a Boolean (False by default) that decides whether to include only interaction features (True) or all features (False).
include_bias is a Boolean (True by default) that decides whether to include the bias (intercept) column of ones (True) or not (False).

This example uses the default values of all parameters, but you’ll sometimes want to experiment with the degree of the function, and it can be beneficial to provide this argument anyway.

Before applying transformer, you need to fit it with .fit():

transformer.fit(x)

Once transformer is fitted, it’s ready to create a new, modified input. You apply .transform() to do that:

x_ = transformer.transform(x)

That’s the transformation of the input array with .transform(). It takes the input array as the argument and returns the modified array.

You can also use .fit_transform() to replace the three previous statements with only one:

x_ = PolynomialFeatures(degree=2, include_bias=False).fit_transform(x)

That’s fitting and transforming the input array in one statement with .fit_transform(). It also takes the input array and effectively does the same thing as .fit() and .transform() called in that order. It also returns the modified array. This is how the new input array looks:

>>>

>>> print(x_)
[[   5.   25.]
 [  15.  225.]
 [  25.  625.]
 [  35. 1225.]
 [  45. 2025.]
 [  55. 3025.]]

The modified input array contains two columns: one with the original inputs and the other with their squares.

You can find more information about PolynomialFeatures on the official documentation page.

Step 3: Create a model and fit it

This step is also the same as in the case of linear regression. You create and fit the model:

model = LinearRegression().fit(x_, y)

The regression model is now created and fitted. It’s ready for application.

You should keep in mind that the first argument of .fit() is the modified input array x_ and not the original x.

Step 4: Get results

You can obtain the properties of the model the same way as in the case of linear regression:

>>>

>>> r_sq = model.score(x_, y)
>>> print('coefficient of determination:', r_sq)
coefficient of determination: 0.8908516262498564
>>> print('intercept:', model.intercept_)
intercept: 21.372321428571425
>>> print('coefficients:', model.coef_)
coefficients: [-1.32357143  0.02839286]

Again, .score() returns 𝑅². Its first argument is also the modified input x_, not x. The values of the weights are associated to .intercept_ and .coef_: .intercept_ represents 𝑏₀, while .coef_ references the array that contains 𝑏₁ and 𝑏₂ respectively.

You can obtain a very similar result with different transformation and regression arguments:

x_ = PolynomialFeatures(degree=2, include_bias=True).fit_transform(x)

If you call PolynomialFeatures with the default parameter include_bias=True (or if you just omit it), you’ll obtain the new input array x_ with the additional leftmost column containing only ones. This column corresponds to the intercept. This is how the modified input array looks in this case:

>>>

>>> print(x_)
[[1.000e+00 5.000e+00 2.500e+01]
 [1.000e+00 1.500e+01 2.250e+02]
 [1.000e+00 2.500e+01 6.250e+02]
 [1.000e+00 3.500e+01 1.225e+03]
 [1.000e+00 4.500e+01 2.025e+03]
 [1.000e+00 5.500e+01 3.025e+03]]

The first column of x_ contains ones, the second has the values of x, while the third holds the squares of x.

The intercept is already included with the leftmost column of ones, and you don’t need to include it again when creating the instance of LinearRegression. Thus, you can provide fit_intercept=False. This is how the next statement looks:

model = LinearRegression(fit_intercept=False).fit(x_, y)

The variable model again corresponds to the new input array x_. Therefore x_ should be passed as the first argument instead of x.

This approach yields the following results, which are similar to the previous case:

>>>

>>> r_sq = model.score(x_, y)
>>> print('coefficient of determination:', r_sq)
coefficient of determination: 0.8908516262498565
>>> print('intercept:', model.intercept_)
intercept: 0.0
>>> print('coefficients:', model.coef_)
coefficients: [21.37232143 -1.32357143  0.02839286]

You see that now .intercept_ is zero, but .coef_ actually contains 𝑏₀ as its first element. Everything else is the same.

Step 5: Predict response

If you want to get the predicted response, just use .predict(), but remember that the argument should be the modified input x_ instead of the old x:

>>>

>>> y_pred = model.predict(x_)
>>> print('predicted response:', y_pred, sep='\n')
predicted response:
[15.46428571  7.90714286  6.02857143  9.82857143 19.30714286 34.46428571]

As you can see, the prediction works almost the same way as in the case of linear regression. It just requires the modified input instead of the original.

You can apply the identical procedure if you have several input variables. You’ll have an input array with more than one column, but everything else is the same. Here is an example:

# Step 1: Import packages
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

# Step 2a: Provide data
x = [[0, 1], [5, 1], [15, 2], [25, 5], [35, 11], [45, 15], [55, 34], [60, 35]]
y = [4, 5, 20, 14, 32, 22, 38, 43]
x, y = np.array(x), np.array(y)

# Step 2b: Transform input data
x_ = PolynomialFeatures(degree=2, include_bias=False).fit_transform(x)

# Step 3: Create a model and fit it
model = LinearRegression().fit(x_, y)

# Step 4: Get results
r_sq = model.score(x_, y)
intercept, coefficients = model.intercept_, model.coef_

# Step 5: Predict
y_pred = model.predict(x_)

This regression example yields the following results and predictions:

>>>

>>> print('coefficient of determination:', r_sq)
coefficient of determination: 0.9453701449127822
>>> print('intercept:', intercept)
intercept: 0.8430556452395734
>>> print('coefficients:', coefficients, sep='\n')
coefficients:
[ 2.44828275  0.16160353 -0.15259677  0.47928683 -0.4641851 ]
>>> print('predicted response:', y_pred, sep='\n')
predicted response:
[ 0.54047408 11.36340283 16.07809622 15.79139    29.73858619 23.50834636
 39.05631386 41.92339046]

In this case, there are six regression coefficients (including the intercept), as shown in the estimated regression function 𝑓(𝑥₁, 𝑥₂) = 𝑏₀ + 𝑏₁𝑥₁ + 𝑏₂𝑥₂ + 𝑏₃𝑥₁² + 𝑏₄𝑥₁𝑥₂ + 𝑏₅𝑥₂².

You can also notice that polynomial regression yielded a higher coefficient of determination than multiple linear regression for the same problem. At first, you could think that obtaining such a large 𝑅² is an excellent result. It might be.

However, in real-world situations, having a complex model and 𝑅² very close to 1 might also be a sign of overfitting. To check the performance of a model, you should test it with new data, that is with observations not used to fit (train) the model.

Advanced Linear Regression With `statsmodels`

You can implement linear regression in Python relatively easily by using the package statsmodels as well. Typically, this is desirable when there is a need for more detailed results.

The procedure is similar to that of scikit-learn.

Step 1: Import packages

First you need to do some imports. In addition to numpy, you need to import statsmodels.api:

import numpy as np
import statsmodels.api as sm

Now you have the packages you need.

Step 2: Provide data and transform inputs

You can provide the inputs and outputs the same way as you did when you were using scikit-learn:

x = [[0, 1], [5, 1], [15, 2], [25, 5], [35, 11], [45, 15], [55, 34], [60, 35]]
y = [4, 5, 20, 14, 32, 22, 38, 43]
x, y = np.array(x), np.array(y)

The input and output arrays are created, but the job is not done yet.

You need to add the column of ones to the inputs if you want statsmodels to calculate the intercept 𝑏₀. It doesn’t takes 𝑏₀ into account by default. This is just one function call:

x = sm.add_constant(x)

That’s how you add the column of ones to x with add_constant(). It takes the input array x as an argument and returns a new array with the column of ones inserted at the beginning. This is how x and y look now:

>>>

>>> print(x)
[[ 1.  0.  1.]
 [ 1.  5.  1.]
 [ 1. 15.  2.]
 [ 1. 25.  5.]
 [ 1. 35. 11.]
 [ 1. 45. 15.]
 [ 1. 55. 34.]
 [ 1. 60. 35.]]
>>> print(y)
[ 4  5 20 14 32 22 38 43]

You can see that the modified x has three columns: the first column of ones (corresponding to 𝑏₀ and replacing the intercept) as well as two columns of the original features.

Step 3: Create a model and fit it

The regression model based on ordinary least squares is an instance of the class statsmodels.regression.linear_model.OLS. This is how you can obtain one:

model = sm.OLS(y, x)

You should be careful here! Please, notice that the first argument is the output, followed with the input. There are several more optional parameters.

To find more information about this class, please visit the official documentation page.

Once your model is created, you can apply .fit() on it:

results = model.fit()

By calling .fit(), you obtain the variable results, which is an instance of the class statsmodels.regression.linear_model.RegressionResultsWrapper. This object holds a lot of information about the regression model.

Step 4: Get results

The variable results refers to the object that contains detailed information about the results of linear regression. Explaining them is far beyond the scope of this article, but you’ll learn here how to extract them.

You can call .summary() to get the table with the results of linear regression:

>>>

>>> print(results.summary())
OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.862
Model:                            OLS   Adj. R-squared:                  0.806
Method:                 Least Squares   F-statistic:                     15.56
Date:                Sun, 17 Feb 2019   Prob (F-statistic):            0.00713
Time:                        19:15:07   Log-Likelihood:                -24.316
No. Observations:                   8   AIC:                             54.63
Df Residuals:                       5   BIC:                             54.87
Df Model:                           2                                         
Covariance Type:            nonrobust                                         
==============================================================================
coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const          5.5226      4.431      1.246      0.268      -5.867      16.912
x1             0.4471      0.285      1.567      0.178      -0.286       1.180
x2             0.2550      0.453      0.563      0.598      -0.910       1.420
==============================================================================
Omnibus:                        0.561   Durbin-Watson:                   3.268
Prob(Omnibus):                  0.755   Jarque-Bera (JB):                0.534
Skew:                           0.380   Prob(JB):                        0.766
Kurtosis:                       1.987   Cond. No.                         80.1
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

This table is very comprehensive. You can find many statistical values associated with linear regression including 𝑅², 𝑏₀, 𝑏₁, and 𝑏₂.

In this particular case, you might obtain the warning related to kurtosistest. This is due to the small number of observations provided.

You can extract any of the values from the table above. Here’s an example:

>>>

>>> print('coefficient of determination:', results.rsquared)
coefficient of determination: 0.8615939258756777
>>> print('adjusted coefficient of determination:', results.rsquared_adj)
adjusted coefficient of determination: 0.8062314962259488
>>> print('regression coefficients:', results.params)
regression coefficients: [5.52257928 0.44706965 0.25502548]

That’s how you obtain some of the results of linear regression:

.rsquared holds 𝑅².
.rsquared_adj represents adjusted 𝑅² (𝑅² corrected according to the number of input features).
.params refers the array with 𝑏₀, 𝑏₁, and 𝑏₂ respectively.

You can also notice that these results are identical to those obtained with scikit-learn for the same problem.

To find more information about the results of linear regression, please visit the official documentation page.

Step 5: Predict response

You can obtain the predicted response on the input values used for creating the model using .fittedvalues or .predict() with the input array as the argument:

>>>

>>> print('predicted response:', results.fittedvalues, sep='\n')
predicted response:
[ 5.77760476  8.012953   12.73867497 17.9744479  23.97529728 29.4660957
 38.78227633 41.27265006]
>>> print('predicted response:', results.predict(x), sep='\n')
predicted response:
[ 5.77760476  8.012953   12.73867497 17.9744479  23.97529728 29.4660957
 38.78227633 41.27265006]

This is the predicted response for known inputs. If you want predictions with new regressors, you can also apply .predict() with new data as the argument:

>>>

>>> x_new = sm.add_constant(np.arange(10).reshape((-1, 2)))
>>> print(x_new)
[[1. 0. 1.]
 [1. 2. 3.]
 [1. 4. 5.]
 [1. 6. 7.]
 [1. 8. 9.]]
>>> y_new = results.predict(x_new)
>>> print(y_new)
[ 5.77760476  7.18179502  8.58598528  9.99017554 11.3943658 ]

You can notice that the predicted results are the same as those obtained with scikit-learn for the same problem.

Beyond Linear Regression

Linear regression is sometimes not appropriate, especially for non-linear models of high complexity.

Fortunately, there are other regression techniques suitable for the cases where linear regression doesn’t work well. Some of them are support vector machines, decision trees, random forest, and neural networks.

There are numerous Python libraries for regression using these techniques. Most of them are free and open-source. That’s one of the reasons why Python is among the main programming languages for machine learning.

The package scikit-learn provides the means for using other regression techniques in a very similar way to what you’ve seen. It contains the classes for support vector machines, decision trees, random forest, and more, with the methods .fit(), .predict(), .score() and so on.

Conclusion

You now know what linear regression is and how you can implement it with Python and three open-source packages: NumPy, scikit-learn, and statsmodels.

You use NumPy for handling arrays.

Linear regression is implemented with the following:

scikit-learn if you don’t need detailed results and want to use the approach consistent with other regression techniques
statsmodels if you need the advanced statistical parameters of a model

Both approaches are worth learning how to use and exploring further. The links in this article can be very useful for that.

When performing linear regression in Python, you can follow these steps:

Import the packages and classes you need
Provide data to work with and eventually do appropriate transformations
Create a regression model and fit it with existing data
Check the results of model fitting to know whether the model is satisfactory
Apply the model for predictions

If you have questions or comments, please put them in the comment section below.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python String Formatting Tips & Best Practices

2019-04-11T14:00:00+00:00

Learn the four main approaches to string formatting in Python, as well as their strengths and weaknesses. You’ll also get a simple rule of thumb for how to pick the best general purpose string formatting approach in your own programs.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Create an Index in Django Without Downtime

2019-04-10T14:00:00+00:00

Managing database migrations is a great challenge in any software project. Luckily, as of version 1.7, Django comes with a built-in migration framework. The framework is very powerful and useful in managing change in databases. But the flexibility provided by the framework required some compromises. To understand the limitations of Django migrations, you are going to tackle a well known problem: creating an index in Django with no downtime.

In this tutorial, you’ll learn:

How and when Django generates new migrations
How to inspect the commands Django generates to execute migrations
How to safely modify migrations to fit your needs

This intermediate-level tutorial is designed for readers who are already familiar with Django migrations. For an introduction to that topic, check out Django Migrations: A Primer.

Free Bonus: Click here to get free access to additional Django tutorials and resources you can use to deepen your Python web development skills.

The Problem With Creating an Index in Django Migrations

A common change that usually becomes necessary when the data stored by your application grows is adding an index. Indexes are used to speed up queries and make your app feel fast and responsive.

In most databases, adding an index requires an exclusive lock on the table. An exclusive lock prevents data modification (DML) operations such as UPDATE, INSERT, and DELETE, while the index is created.

Locks are obtained implicitly by the database when executing certain operations. For example, when a user logs into your app, Django will update the last_login field in the auth_user table. To perform the update, the database will first have to obtain a lock on the row. If the row is currently being locked by another connection, then you might get a database exception.

Locking a table might pose a problem when it’s necessary to keep the system available during migrations. The bigger the table, the longer it can take to create the index. The longer it takes to create the index, the longer the system is unavailable or unresponsive to users.

Some database vendors provide a way to create an index without locking the table. For example, to create an index in PostgreSQL without locking a table, you can use the CONCURRENTLY keyword:

CREATE INDEX CONCURRENTLY ix ON table (column);

In Oracle, there is an ONLINE option to allow DML operations on the table while the index is created:

CREATE INDEX ix ON table (column) ONLINE;

When generating migrations, Django will not use these special keywords. Running the migration as is will make the database acquire an exclusive lock on the table and prevent DML operations while the index is created.

Creating an index concurrently has some caveats. It’s important to understand the issues specific to your database backend in advance. For example, one caveat in PostgreSQL is that creating an index concurrently takes longer because it requires an additional table scan.

In this tutorial, you’ll use Django migrations to create an index on a large table, without causing any downtime.

Note: To follow this tutorial, it is recommended that you use a PostgreSQL backend, Django 2.x, and Python 3.

It is possible to follow along with other database backends as well. In places where SQL features unique to PostgreSQL are used, change the SQL to match your database backend.

Setup

You’re going to use a made up Sale model in an app called app. In a real life situation, models such as Sale are the main tables in the database, and they will usually be very big and store a lot of data:

# models.py

from django.db import models

class Sale(models.Model):
    sold_at = models.DateTimeField(
        auto_now_add=True,
    )
    charged_amount = models.PositiveIntegerField()

To create the table, generate the initial migration and apply it:

$ python manage.py makemigrations
Migrations for 'app':
  app/migrations/0001_initial.py
    - Create model Sale

$ python manage migrate
Operations to perform:
  Apply all migrations: app
Running migrations:
  Applying app.0001_initial... OK

After a while, the sales table becomes very big, and users start to complain about slowness. While monitoring the database, you noticed that a lot of queries use the sold_at column. To speed things up, you decide that you need an index on the column.

To add an index on sold_at, you make the following change to the model:

# models.py

from django.db import models

class Sale(models.Model):
    sold_at = models.DateTimeField(
        auto_now_add=True,
        db_index=True,
    )
    charged_amount = models.PositiveIntegerField()

If you run this migration as it is, then Django will create the index on the table, and it will be locked until the index is completed. It can take a while to create an index on a very large table, and you want to avoid downtime.

On a local development environment with a small dataset and very few connections, this migration might feel instantaneous. However, on large datasets with many concurrent connections, obtaining a lock and creating the index can take a while.

In the next steps, you are going to modify migrations created by Django to create the index without causing any downtime.

Fake Migration

The first approach is to create the index manually. You are going to generate the migration, but you are not going to actually let Django apply it. Instead, you will run the SQL manually in the database and then make Django think the migration completed.

First, generate the migration:

$ python manage.py makemigrations --name add_index_fake
Migrations for 'app':
  app/migrations/0002_add_index_fake.py
    - Alter field sold_at on sale

Use the sqlmigrate command to view the SQL Django will use to execute this migration:

$ python manage.py sqlmigrate app 0002

BEGIN;
--
-- Alter field sold_at on sale
--
CREATE INDEX "app_sale_sold_at_b9438ae4" ON "app_sale" ("sold_at");
COMMIT;

You want to create the index without locking the table, so you need to modify the command. Add the CONCURRENTLY keyword and execute in the database:

app=# CREATE INDEX CONCURRENTLY "app_sale_sold_at_b9438ae4"
ON "app_sale" ("sold_at");

CREATE INDEX

Notice that you executed the command without the BEGIN and COMMIT parts. Omitting these keywords will execute the commands without a database transaction. We will discuss database transactions later in the article.

After you executed the command, if you try to apply migrations, then you will get the following error:

$ python manage.py migrate

Operations to perform:
  Apply all migrations: app
Running migrations:
  Applying app.0002_add_index_fake...Traceback (most recent call last):
  File "venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)

psycopg2.ProgrammingError: relation "app_sale_sold_at_b9438ae4" already exists

Django complains that the index already exists, so it can’t proceed with the migration. You just created the index directly in the database, so now you need to make Django think that the migration was already applied.

How to Fake a Migration

Django provides a built-in way of marking migrations as executed, without actually executing them. To use this option, set the --fake flag when applying the migration:

$ python manage.py migrate --fake
Operations to perform:
  Apply all migrations: app
Running migrations:
  Applying app.0002_add_index_fake... FAKED

Django didn’t raise an error this time. In fact, Django didn’t really apply any migration. It just marked it as executed (or FAKED).

Here are some issues to consider when faking migrations:

The manual command must be equivalent to the SQL generated by Django: You need to make sure the command you execute is equivalent to the SQL generated by Django. Use sqlmigrate to produce the SQL command. If the commands do not match, then you might end up with inconsistencies between the database and the models state.
Other unapplied migrations will also be faked: When you have multiple unapplied migrations, they will all be faked. Before you apply migrations, it’s important to make sure only the migrations you want to fake are unapplied. Otherwise, you might end up with inconsistencies. Another option is to specify the exact migration you want to fake.
Direct access to the database is required: You need to run the SQL command in the database. This is not always an option. Also, executing commands directly in a production database is dangerous and should be avoided when possible.
Automated deployment processes might need adjustments: If you automated the deployment process (using CI, CD, or other automation tools), then you might need to alter the process to fake migrations. This is not always desirable.

Cleanup

Before moving on to the next section, you need to bring the database back to its state right after the initial migration. To do that, migrate back to the initial migration:

$ python manage.py migrate 0001
Operations to perform:
  Target specific migration: 0001_initial, from app
Running migrations:
  Rendering model states... DONE
  Unapplying app.0002_add_index_fake... OK

Django unapplied the changes made in the second migration, so now it’s safe to also delete the file:

$ rm app/migrations/0002_add_index_fake.py

To make sure you did everything right, inspect the migrations:

$ python manage.py showmigrations app
app
 [X] 0001_initial

The initial migration was applied, and there are no unapplied migrations.

Execute Raw SQL in Migrations

In the previous section, you executed SQL directly in the database and faked the migration. This gets the job done, but there is a better solution.

Django provides a way to execute raw SQL in migrations using RunSQL. Let’s try to use it instead of executing the command directly in the database.

First, generate a new empty migration:

$ python manage.py makemigrations app --empty --name add_index_runsql
Migrations for 'app':
  app/migrations/0002_add_index_runsql.py

Next, edit the migration file and add a RunSQL operation:

# migrations/0002_add_index_runsql.py

from django.db import migrations, models

class Migration(migrations.Migration):
    atomic = False

    dependencies = [
        ('app', '0001_initial'),
    ]

    operations = [
        migrations.RunSQL(
            'CREATE INDEX "app_sale_sold_at_b9438ae4" '
            'ON "app_sale" ("sold_at");',
        ),
    ]

When you run the migration, you will get the following output:

$ python manage.py migrate
Operations to perform:
  Apply all migrations: app
Running migrations:
  Applying app.0002_add_index_runsql... OK

This is looking good, but there is a problem. Let’s try to generate migrations again:

$ python manage.py makemigrations --name leftover_migration
Migrations for 'app':
  app/migrations/0003_leftover_migration.py
    - Alter field sold_at on sale

Django generated the same migration again. Why did it do that?

Cleanup

Before we can answer that question, you need to clean up and undo the changes you made to the database. Start by deleting the last migration. It was not applied, so it’s safe to delete:

$ rm app/migrations/0003_leftover_migration.py

Next, list the migrations for the app app:

$ python manage.py showmigrations app
app
 [X] 0001_initial
 [X] 0002_add_index_runsql

The third migration is gone, but the second is applied. You want to get back to the state right after the initial migration. Try to migrate back to the initial migration as you did in the previous section:

$ python manage.py migrate app 0001
Operations to perform:
  Target specific migration: 0001_initial, from app
Running migrations:
  Rendering model states... DONE
  Unapplying app.0002_add_index_runsql...Traceback (most recent call last):

NotImplementedError: You cannot reverse this operation

Django is unable to reverse the migration.

Reverse Migration Operation

To reverse a migration, Django executes an opposite action for every operation. In this case, the reverse of adding an index is to drop it. As you’ve already seen, when a migration is reversible, you can unapply it. Just like you can use checkout in Git, you can reverse a migration if you execute migrate to an earlier migration.

Many built-in migration operations already define a reverse action. For example, the reverse action for adding a field is to drop the corresponding column. The reverse action for creating a model is to drop the corresponding table.

Some migration operations are not reversible. For example, there is no reverse action for removing a field or deleting a model, because once the migration was applied, the data is gone.

In the previous section, you used the RunSQL operation. When you tried to reverse the migration, you encountered an error. According to the error, one of the operations in the migration cannot be reversed. Django is unable to reverse raw SQL by default. Because Django has no knowledge of what was executed by the operation, it cannot generate an opposite action automatically.

How to Make a Migration Reversible

For a migration to be reversible, all the operations in it must be reversible. It’s not possible to reverse part of a migration, so a single non-reversible operation will make the entire migration non-reversible.

To make a RunSQL operation reversible, you must provide SQL to execute when the operation is reversed. The reverse SQL is provided in the reverse_sql argument.

The opposite action to adding an index is to drop it. To make your migration reversible, provide the reverse_sql to drop the index:

# migrations/0002_add_index_runsql.py

from django.db import migrations, models

class Migration(migrations.Migration):
    atomic = False

    dependencies = [
        ('app', '0001_initial'),
    ]

    operations = [
        migrations.RunSQL(
            'CREATE INDEX "app_sale_sold_at_b9438ae4" '
            'ON "app_sale" ("sold_at");',

            reverse_sql='DROP INDEX "app_sale_sold_at_b9438ae4";',
        ),
    ]

Now try to reverse the migration:

$ python manage.py showmigrations app
app
 [X] 0001_initial
 [X] 0002_add_index_runsql

$ python manage.py migrate app 0001
Operations to perform:
  Target specific migration: 0001_initial, from app
Running migrations:
  Rendering model states... DONE
 Unapplying app.0002_add_index_runsql... OK

$ python manage.py showmigrations app
app
 [X] 0001_initial
 [ ] 0002_add_index_runsql

The second migration was reversed, and the index was dropped by Django. Now it’s safe to delete the migration file:

$ rm app/migrations/0002_add_index_runsql.py

It’s always a good idea to provide reverse_sql. In situations where reversing a raw SQL operation does not require any action, you can mark the operation as reversible using the special sentinel migrations.RunSQL.noop:

migrations.RunSQL(
    sql='...',  # Your forward SQL here
    reverse_sql=migrations.RunSQL.noop,
),

Understand Model State and Database State

In your previous attempt to create the index manually using RunSQL, Django generated the same migration over and over again even though the index was created in the database. To understand why Django did that, you first need to understand how Django decides when to generate new migrations.

When Django Generates a New Migration

In the process of generating and applying migrations, Django syncs between the state of the database and the state of the models. For example, when you add a field to a model, Django adds a column to the table. When you remove a field from the model, Django removes the column from the table.

To sync between the models and the database, Django maintains a state that represents the models. To sync the database with the models, Django generates migration operations. Migration operations translate to a vendor specific SQL that can be executed in the database. When all migration operations are executed, the database and the models are expected to be consistent.

To get the state of the database, Django aggregates the operations from all past migrations. When the aggregated state of the migrations is not consistent with the state of the models, Django generates a new migration.

In the previous example, you created the index using raw SQL. Django did not know you created the index because you didn’t use a familiar migration operation.

When Django aggregated all the migrations and compared them with the state of the models, it found that an index was missing. This is why, even after you created the index manually, Django still thought it was missing and generated a new migration for it.

How to Separate Database and State in Migrations

Since Django is unable to create the index the way you want it to, you want to provide your own SQL but still let Django know you created it.

In other words, you need to execute something in the database and provide Django with the migration operation to sync its internal state. To do that, Django provides us with a special migration operation called SeparateDatabaseAndState. This operation is not well known and should be reserved for special cases such as this one.

It’s much easier to edit migrations than write them from scratch, so start by generating a migration the usual way:

$ python manage.py makemigrations --name add_index_separate_database_and_state

Migrations for 'app':
  app/migrations/0002_add_index_separate_database_and_state.py
    - Alter field sold_at on sale

This is the contents of the migration generated by Django, same as before:

# migrations/0002_add_index_separate_database_and_state.py

from django.db import migrations, models

class Migration(migrations.Migration):

    dependencies = [
        ('app', '0001_initial'),
    ]

    operations = [
        migrations.AlterField(
            model_name='sale',
            name='sold_at',
            field=models.DateTimeField(
                auto_now_add=True,
                db_index=True,
            ),
        ),
    ]

Django generated an AlterField operation on the field sold_at. The operation will create an index and update the state. We want to keep this operation but provide a different command to execute in the database.

Once again, to get the command, use the SQL generated by Django:

$ python manage.py sqlmigrate app 0002
BEGIN;
--
-- Alter field sold_at on sale
--
CREATE INDEX "app_sale_sold_at_b9438ae4" ON "app_sale" ("sold_at");
COMMIT;

Add the CONCURRENTLY keyword in the appropriate place:

CREATE INDEX CONCURRENTLY "app_sale_sold_at_b9438ae4"
ON "app_sale" ("sold_at");

Next, edit the migration file and use SeparateDatabaseAndState to provide your modified SQL command for execution:

# migrations/0002_add_index_separate_database_and_state.py

from django.db import migrations, models

class Migration(migrations.Migration):

    dependencies = [
        ('app', '0001_initial'),
    ]

    operations = [

        migrations.SeparateDatabaseAndState(

            state_operations=[
                migrations.AlterField(
                    model_name='sale',
                    name='sold_at',
                    field=models.DateTimeField(
                        auto_now_add=True,
                        db_index=True,
                    ),
                ),
            ],

            database_operations=[
                migrations.RunSQL(sql="""
                    CREATE INDEX CONCURRENTLY "app_sale_sold_at_b9438ae4"
                    ON "app_sale" ("sold_at");
                """, reverse_sql="""
                    DROP INDEX "app_sale_sold_at_b9438ae4";
                """),
            ],
        ),

    ],

The migration operation SeparateDatabaseAndState accepts 2 lists of operations:

state_operations are operations to apply on the internal model state. They do not effect the database.
database_operations are operations to apply to the database.

You kept the original operation generated by Django in state_operations. When using SeparateDatabaseAndState, this is what you will usually want to do. Notice that the db_index=True argument is provided to the field. This migration operation will let Django know that there is an index on the field.

You used the SQL generated by Django and added the CONCURRENTLY keyword. You used the special action RunSQL to execute raw SQL in the migration.

If you try to run the migration, you will get the following output:

$ python manage.py migrate app
Operations to perform:
  Apply all migrations: app
Running migrations:
  Applying app.0002_add_index_separate_database_and_state...Traceback (most recent call last):
  File "/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 83, in _execute
    return self.cursor.execute(sql)
psycopg2.InternalError: CREATE INDEX CONCURRENTLY cannot run inside a transaction block

Non-Atomic Migrations

In SQL, CREATE, DROP, ALTER, and TRUNCATE operations are referred to as Data Definition Language (DDL). In databases that support transactional DDL, such as PostgreSQL, Django executes migrations inside a database transaction by default. However, according to the error above, PostgreSQL cannot create an index concurrently inside a transaction block.

To be able to create an index concurrently within a migration, you need to tell Django to not execute the migration in a database transaction. To do that, you mark the migration as non-atomic by setting atomic to False:

# migrations/0002_add_index_separate_database_and_state.py

from django.db import migrations, models

class Migration(migrations.Migration):
    atomic = False

    dependencies = [
        ('app', '0001_initial'),
    ]

    operations = [

        migrations.SeparateDatabaseAndState(

            state_operations=[
                migrations.AlterField(
                    model_name='sale',
                    name='sold_at',
                    field=models.DateTimeField(
                        auto_now_add=True,
                        db_index=True,
                    ),
                ),
            ],

            database_operations=[
                migrations.RunSQL(sql="""
                    CREATE INDEX CONCURRENTLY "app_sale_sold_at_b9438ae4"
                    ON "app_sale" ("sold_at");
                """,
                reverse_sql="""
                    DROP INDEX "app_sale_sold_at_b9438ae4";
                """),
            ],
        ),

    ],

After you marked the migration as non-atomic, you can run the migration:

$ python manage.py migrate app
Operations to perform:
  Apply all migrations: app
Running migrations:
  Applying app.0002_add_index_separate_database_and_state... OK

You just executed the migration without causing any downtime.

Here are some issues to consider when you’re using SeparateDatabaseAndState:

Database operations must be equivalent to state operations: Inconsistencies between the database and model state can cause a lot of trouble. A good starting point is to keep the operations generated by Django in state_operations and edit the output of sqlmigrate to use in database_operations.
Non atomic migrations cannot rollback in case of error: If there is an error during the migration, then you won’t be able to rollback. You would have to either rollback the migration or complete it manually. It’s a good idea to keep the operations executed inside a non-atomic migration to a minimum. If you have additional operations in the migration, move them to a new migration.
Migration might be vendor specific: The SQL generated by Django is specific to the database backend used in the project. It might work with other database backends, but that is not guaranteed. If you need to support multiple database backends, you need to make some adjustments to this approach.

Conclusion

You started this tutorial with a large table and a problem. You wanted to make your app faster for your users, and you wanted to do that without causing them any downtime.

By the end of the tutorial, you managed to generate and safely modify a Django migration to achieve this goal. You tackled different problems along the way and managed to overcome them using built-in tools provided by the migrations framework.

In this tutorial, you learned the following:

How Django migrations work internally using model and database state, and when new migrations are generated
How to execute custom SQL in migrations using the RunSQL action
What reversible migrations are, and how to make a RunSQL action reversible
What atomic migrations are, and how to change the default behavior according to your needs
How to safely execute complex migrations in Django

The separation between model and database state is an important concept. Once you understand it, and how to utilize it, you can overcome many limitations of the built-in migration operations. Some use cases that come to mind include adding an index that was already created in the database and providing vendor specific arguments to DDL commands.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Running Python Scripts

2019-04-09T14:00:00+00:00

One of the most important skills you need to build as a Python developer is to be able to run Python scripts and code. This is going to be the only way for you to know if your code works as you planned. It’s even the only way of knowing if your code works at all!

This step-by-step course will guide you through a series of ways to run Python scripts, depending on your environment, platform, needs, and skills as a programmer.

You’ll learn how to run Python scripts by using:

The operating system command-line or terminal
The Python interactive mode
The IDE or text editor you like best
The file manager of your system, by double-clicking on the icon of your script

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python REST APIs With Flask, Connexion, and SQLAlchemy – Part 3

2019-04-08T14:00:00+00:00

In Part 2 of this series, you added the ability to save changes made through the REST API to a database using SQLAlchemy and learned how to serialize that data for the REST API using Marshmallow. Connecting the REST API to a database so that the application can make changes to existing data and create new data is great and makes the application much more useful and robust.

That’s only part of the power a database offers, however. An even more powerful feature is the R part of RDBMS systems: relationships. In a database, a relationship is the ability to connect two or more tables together in a meaningful way. In this article, you’ll learn how to implement relationships and turn your Person database into a mini-blogging web application.

In this article, you’ll learn:

Why more than one table in a database is useful and important
How tables are related to each other
How SQLAlchemy can help you manage relationships
How relationships help you build a mini-blogging application

Who This Article Is For

Part 1 of this series guided you through building a REST API, and Part 2 showed you how to connect that REST API to a database.

This article expands your programming tool belt further. You’ll learn how to create hierarchical data structures represented as one-to-many relationships by SQLAlchemy. In addition, you’ll extend the REST API you’ve already built to provide CRUD (Create, Read, Update, and Delete) support for the elements in this hierarchical structure.

The web application presented in Part 2 will have its HTML and JavaScript files modified in major ways to create a more fully functional mini-blogging application. You can review the final version of the code from Part 2 in the GitHub repository for that article.

Hang on as you get started creating relationships and your mini-blogging application!

Additional Dependencies

There are no new Python dependencies beyond what was required for the Part 2 article. However, you will be using two new JavaScript modules in the web application to makes things easier and more consistent. The two modules are the following:

Handlebars.js is a templating engine for JavaScript, much like Jinja2 for Flask.
Moment.js is a datetime parsing and formatting module that makes displaying UTC timestamps easier.

You don’t have to download either of these, as the web application will get them directly from the Cloudflare CDN (Content Delivery Network), as you’re already doing for the jQuery module.

People Data Extended for Blogging

In Part 2, the People data existed as a dictionary in the build_database.py Python code. This is what you used to populate the database with some initial data. You’re going to modify the People data structure to give each person a list of notes associated with them. The new People data structure will look like this:

# Data to initialize database with
PEOPLE = [
    {
        "fname": "Doug",
        "lname": "Farrell",
        "notes": [
            ("Cool, a mini-blogging application!", "2019-01-06 22:17:54"),
            ("This could be useful", "2019-01-08 22:17:54"),
            ("Well, sort of useful", "2019-03-06 22:17:54"),
        ],
    },
    {
        "fname": "Kent",
        "lname": "Brockman",
        "notes": [
            (
                "I'm going to make really profound observations",
                "2019-01-07 22:17:54",
            ),
            (
                "Maybe they'll be more obvious than I thought",
                "2019-02-06 22:17:54",
            ),
        ],
    },
    {
        "fname": "Bunny",
        "lname": "Easter",
        "notes": [
            ("Has anyone seen my Easter eggs?", "2019-01-07 22:47:54"),
            ("I'm really late delivering these!", "2019-04-06 22:17:54"),
        ],
    },
]

Each person in the People dictionary now includes a key called notes, which is associated with a list containing tuples of data. Each tuple in the notes list represents a single note containing the content and a timestamp. The timestamps are initialized (rather than dynamically created) to demonstrate ordering later on in the REST API.

Each single person is associated with multiple notes, and each single note is associated with only one person. This hierarchy of data is known as a one-to-many relationship, where a single parent object is related to many child objects. You’ll see how this one-to-many relationship is managed in the database with SQLAlchemy.

Brute Force Approach

The database you built stored the data in a table, and a table is a two-dimensional array of rows and columns. Can the People dictionary above be represented in a single table of rows and columns? It can be, in the following way, in your person database table. Unfortunately to include all of the actual data in the example creates a scroll bar for the table, as you’ll see below:

`person_id`	`lname`	`fname`	`timestamp`	`content`	`note_timestamp`
1	Farrell	Doug	2018-08-08 21:16:01	Cool, a mini-blogging application!	2019-01-06 22:17:54
2	Farrell	Doug	2018-08-08 21:16:01	This could be useful	2019-01-08 22:17:54
3	Farrell	Doug	2018-08-08 21:16:01	Well, sort of useful	2019-03-06 22:17:54
4	Brockman	Kent	2018-08-08 21:16:01	I’m going to make really profound observations	2019-01-07 22:17:54
5	Brockman	Kent	2018-08-08 21:16:01	Maybe they’ll be more obvious than I thought	2019-02-06 22:17:54
6	Easter	Bunny	2018-08-08 21:16:01	Has anyone seen my Easter eggs?	2019-01-07 22:47:54
7	Easter	Bunny	2018-08-08 21:16:01	I’m really late delivering these!	2019-04-06 22:17:54

The above table would actually work. All the data is represented, and a single person is associated with a collection of different notes.

Advantages

Conceptually, the above table structure has the advantage of being relatively simple to understand. You could even make the case that the data could be persisted to a flat file instead of a database.

Because of the two-dimensional table structure, you could store and use this data in a spreadsheet. Spreadsheets have been pressed into service as data storage quite a bit.

Disadvantages

While the above table structure would work, it has some real disadvantages.

In order to represent the collection of notes, all the data for each person is repeated for every unique note, the person data is therefore redundant. This isn’t such a big deal for your person data as there aren’t that many columns. But imagine if a person had many more columns. Even with large disk drives, this can get to be a storage concern if you’re dealing with millions of rows of data.

Having redundant data like this can lead to maintenance issues as time goes by. For example, what if the Easter Bunny decided a change of name was a good idea. In order to do this, every record containing the Easter Bunny’s name would have to be updated in order to keep the data consistent. This kind of work against the database can lead to data inconsistency, particularly if the work is done by a person running a SQL query by hand.

Naming columns becomes awkward. In the table above, there is a timestamp column used to track the creation and update time of a person in the table. You also want to have similar functionality for the creation and update time for a note, but because timestamp is already used, a contrived name of note_timestamp is used.

What if you wanted to add additional one-to-many relationships to the person table? For example, to include a person’s children or phone numbers. Each person could have multiple children and multiple phone numbers. This could be done relatively easily to the Python People dictionary above by adding children and phone_numbers keys with new lists containing the data.

However, representing those new one-to-many relationships in your person database table above becomes significantly more difficult. Every new one-to-many relationship increases the number of rows necessary to represent it for every single entry in the child data dramatically. In addition, the problems associated with data redundancy get bigger and more difficult to handle.

Lastly, the data you’d get back from the above table structure wouldn’t be very Pythonic: it would be just a big list of lists. SQLAlchemy wouldn’t be able to help you very much because the relationship isn’t there.

Relational Database Approach

Based on what you’ve seen above, it becomes clear that trying to represent even a moderately complex dataset in a single table becomes unmanageable pretty quickly. Given that, what alternative does a database offer? This is where the R part of RDBMS databases comes into play. Representing relationships removes the disadvantages outlined above.

Instead of trying to represent hierarchical data in a single table, the data is broken up into multiple tables, with a mechanism to relate them to one another. The tables are broken along collection lines, so for your People dictionary above, this means there will be a table representing people and another representing notes. This brings back your original person table, which looks like this:

`person_id`	`lname`	`fname`	`timestamp`
1	Farrell	Doug	2018-08-08 21:16:01.888444
2	Brockman	Kent	2018-08-08 21:16:01.889060
3	Easter	Bunny	2018-08-08 21:16:01.886834

To represent the new note information, you’ll create a new table called note. (Remember our singular table naming convention.) The table looks like this:

`note_id`	`person_id`	`content`	`timestamp`
1	1	Cool, a mini-blogging application!	2019-01-06 22:17:54
2	1	This could be useful	2019-01-08 22:17:54
3	1	Well, sort of useful	2019-03-06 22:17:54
4	2	I’m going to make really profound observations	2019-01-07 22:17:54
5	2	Maybe they’ll be more obvious than I thought	2019-02-06 22:17:54
6	3	Has anyone seen my Easter eggs?	2019-01-07 22:47:54
7	3	I’m really late delivering these!	2019-04-06 22:17:54

Notice that, like the person table, the note table has a unique identifier called note_id, which is the primary key for the note table. One thing that isn’t obvious is the inclusion of the person_id value in the table. What is that used for? This is what creates the relationship to the person table. Whereas note_id is the primary key for the table, person_id is what’s known as a foreign key.

The foreign key gives each entry in the note table the primary key of the person record it’s associated with. Using this, SQLAlchemy can gather all the notes associated with each person by connecting the person.person_id primary key to the note.person_id foreign key, creating a relationship.

Advantages

By breaking the data set into two tables, and introducing the concept of a foreign key, you’ve made the data a little more complex to think about, you have resolved the disadvantages of a single table representation. SQLAlchemy will help you encode the increased complexity fairly easily.

The data is no longer redundant in the database. There is only one person entry for each person you want to store in the database. This solves the storage concern immediately and dramatically simplifies the maintenance concerns.

If the Easter Bunny still wanted to change names, then you’d only have to change a single row in the person table, and anything else related to that row (like the note table) would immediately take advantage of the change.

Column naming is more consistent and meaningful. Because person and note data exist in separate tables, the creation and update timestamp can be named consistently in both tables, as there is no conflict for names across tables.

In addition, you’d no longer have to create permutations of each row for new one-to-many relationships you might want to represent. Take our children and phone_numbers example from earlier. Implementing this would require child and phone_number tables. Each table would contain a foreign key of person_id relating it back to the person table.

Using SQLAlchemy, the data you’d get back from the above tables would be more immediately useful, as what you’d get is an object for each person row. That object has named attributes equivalent to the columns in the table. One of those attributes is a Python list containing the related note objects.

Disadvantages

Where the brute force approach was simpler to understand, the concept of foreign keys and relationships make thinking about the data somewhat more abstract. This abstraction needs to be thought about for every relationship you establish between tables.

Making use of relationships means committing to using a database system. This is another tool to install, learn, and maintain above and beyond the application that actually uses the data.

SQLAlchemy Models

To use the two tables above, and the relationship between them, you’ll need to create SQLAlchemy models that are aware of both tables and the relationship between them. Here’s the SQLAlchemy Person model from Part 2, updated to include a relationship to a collection of notes:

 1 class Person(db.Model):
 2     __tablename__ = 'person'
 3     person_id = db.Column(db.Integer, primary_key=True)
 4     lname = db.Column(db.String(32))
 5     fname = db.Column(db.String(32))
 6     timestamp = db.Column(
 7         db.DateTime, default=datetime.utcnow, onupdate=datetime.utcnow
 8     )
 9     notes = db.relationship(
10         'Note',
11         backref='person',
12         cascade='all, delete, delete-orphan',
13         single_parent=True,
14         order_by='desc(Note.timestamp)'
15     )

Lines 1 to 8 of the above Python class look exactly like what you created before in Part 2. Lines 9 to 16 create a new attribute in the Person class called notes. This new notes attributes is defined in the following lines of code:

Line 9: Like the other attributes of the class, this line creates a new attribute called notes and sets it equal to an instance of an object called db.relationship. This object creates the relationship you’re adding to the Person class and is created with all of the parameters defined in the lines that follow.
Line 10: The string parameter 'Note' defines the SQLAlchemy class that the Person class will be related to. The Note class isn’t defined yet, which is why it’s a string here. This is a forward reference and helps handle problems that the order of definitions could cause when something is needed that isn’t defined until later in the code. The 'Note' string allows the Person class to find the Note class at runtime, which is after both Person and Note have been defined.
Line 11: The backref='person' parameter is trickier. It creates what’s known as a backwards reference in Note objects. Each instance of a Note object will contain an attribute called person. The person attribute references the parent object that a particular Note instance is associated with. Having a reference to the parent object (person in this case) in the child can be very useful if your code iterates over notes and has to include information about the parent. This happens surprisingly often in display rendering code.
Line 12: The cascade='all, delete, delete-orphan' parameter determines how to treat note object instances when changes are made to the parent Person instance. For example, when a Person object is deleted, SQLAlchemy will create the SQL necessary to delete the Person from the database. Additionally, this parameter tells it to also delete all the Note instances associated with it. You can read more about these options in the SQLAlchemy documentation.
Line 13: The single_parent=True parameter is required if delete-orphan is part of the previous cascade parameter. This tells SQLAlchemy not to allow orphaned Note instances (a Note without a parent Person object) to exist because each Note has a single parent.
Line 14: The order_by='desc(Note.timestamp)' parameter tells SQLAlchemy how to sort the Note instances associated with a Person. When a Person object is retrieved, by default the notes attribute list will contain Note objects in an unknown order. The SQLAlchemy desc(...) function will sort the notes in descending order from newest to oldest. If this line was instead order_by='Note.timestamp', SQLAlchemy would default to using the asc(...) function, and sort the notes in ascending order, oldest to newest.

Now that your Person model has the new notes attribute, and this represents the one-to-many relationship to Note objects, you’ll need to define a SQLAlchemy model for a Note:

 1 class Note(db.Model):
 2     __tablename__ = 'note'
 3     note_id = db.Column(db.Integer, primary_key=True)
 4     person_id = db.Column(db.Integer, db.ForeignKey('person.person_id'))
 5     content = db.Column(db.String, nullable=False)
 6     timestamp = db.Column(
 7         db.DateTime, default=datetime.utcnow, onupdate=datetime.utcnow
 8     )

The Note class defines the attributes making up a note as seen in our sample note database table from above. The attributes are defined here:

Line 1 creates the Note class, inheriting from db.Model, exactly as you did before when creating the Person class.
Line 2 tells the class what database table to use to store Note objects.
Line 3 creates the note_id attribute, defining it as an integer value, and as the primary key for the Note object.
Line 4 creates the person_id attribute, and defines it as the foreign key, relating the Note class to the Person class using the person.person_id primary key. This, and the Person.notes attribute, are how SQLAlchemy knows what to do when interacting with Person and Note objects.
Line 5 creates the content attribute, which contains the actual text of the note. The nullable=False parameter indicates that it’s okay to create new notes that have no content.
Line 6 creates the timestamp attribute, and exactly like the Person class, this contains the creation or update time for any particular Note instance.

Initialize the Database

Now that you’ve updated the Person and created the Note models, you’ll use them to rebuild the testing database people.db. You’ll do this by updating the build_database.py code from Part 2. Here’s what the code will look like:

 1 import os
 2 from datetime import datetime
 3 from config import db
 4 from models import Person, Note
 5 
 6 # Data to initialize database with
 7 PEOPLE = [
 8     {
 9         "fname": "Doug",
10         "lname": "Farrell",
11         "notes": [
12             ("Cool, a mini-blogging application!", "2019-01-06 22:17:54"),
13             ("This could be useful", "2019-01-08 22:17:54"),
14             ("Well, sort of useful", "2019-03-06 22:17:54"),
15         ],
16     },
17     {
18         "fname": "Kent",
19         "lname": "Brockman",
20         "notes": [
21             (
22                 "I'm going to make really profound observations",
23                 "2019-01-07 22:17:54",
24             ),
25             (
26                 "Maybe they'll be more obvious than I thought",
27                 "2019-02-06 22:17:54",
28             ),
29         ],
30     },
31     {
32         "fname": "Bunny",
33         "lname": "Easter",
34         "notes": [
35             ("Has anyone seen my Easter eggs?", "2019-01-07 22:47:54"),
36             ("I'm really late delivering these!", "2019-04-06 22:17:54"),
37         ],
38     },
39 ]
40 
41 # Delete database file if it exists currently
42 if os.path.exists("people.db"):
43     os.remove("people.db")
44 
45 # Create the database
46 db.create_all()
47 
48 # Iterate over the PEOPLE structure and populate the database
49 for person in PEOPLE:
50     p = Person(lname=person.get("lname"), fname=person.get("fname"))
51 
52     # Add the notes for the person
53     for note in person.get("notes"):
54         content, timestamp = note
55         p.notes.append(
56             Note(
57                 content=content,
58                 timestamp=datetime.strptime(timestamp, "%Y-%m-%d %H:%M:%S"),
59             )
60         )
61     db.session.add(p)
62 
63 db.session.commit()

The code above came from Part 2, with a few changes to create the one-to-many relationship between Person and Note. Here are the updated or new lines added to the code:

Line 4 has been updated to import the Note class defined previously.
Lines 7 to 39 contain the updated PEOPLE dictionary containing our person data, along with the list of notes associated with each person. This data will be inserted into the database.
Lines 49 to 61 iterate over the PEOPLE dictionary, getting each person in turn and using it to create a Person object.
Line 53 iterates over the person.notes list, getting each note in turn.
Line 54 unpacks the content and timestamp from each note tuple.
Line 55 to 60 creates a Note object and appends it to the person notes collection using p.notes.append().
Line 61 adds the Person object p to the database session.
Line 63 commits all of the activity in the session to the database. It’s at this point that all of the data is written to the person and note tables in the people.db database file.

You can see that working with the notes collection in the Person object instance p is just like working with any other list in Python. SQLAlchemy takes care of the underlying one-to-many relationship information when the db.session.commit() call is made.

For example, just like a Person instance has its primary key field person_id initialized by SQLAlchemy when it’s committed to the database, instances of Note will have their primary key fields initialized. In addition, the Note foreign key person_id will also be initialized with the primary key value of the Person instance it’s associated with.

Here’s an example instance of a Person object before the db.session.commit() in a kind of pseudocode:

Person (
    person_id = None
    lname = 'Farrell'
    fname = 'Doug'
    timestamp = None
    notes = [
        Note (
            note_id = None
            person_id = None
            content = 'Cool, a mini-blogging application!'
            timestamp = '2019-01-06 22:17:54'
        ),
        Note (
            note_id = None
            person_id = None
            content = 'This could be useful'
            timestamp = '2019-01-08 22:17:54'
        ),
        Note (
            note_id = None
            person_id = None
            content = 'Well, sort of useful'
            timestamp = '2019-03-06 22:17:54'
        )
    ]
)

Here’s the example Person object after the db.session.commit():

Person (
    person_id = 1
    lname = 'Farrell'
    fname = 'Doug'
    timestamp = '2019-02-02 21:27:10.336'
    notes = [
        Note (
            note_id = 1
            person_id = 1
            content = 'Cool, a mini-blogging application!'
            timestamp = '2019-01-06 22:17:54'
        ),
        Note (
            note_id = 2
            person_id = 1
            content = 'This could be useful'
            timestamp = '2019-01-08 22:17:54'
        ),
        Note (
            note_id = 3
            person_id = 1
            content = 'Well, sort of useful'
            timestamp = '2019-03-06 22:17:54'
        )
    ]
)

The important difference between the two is that the primary key of the Person and Note objects has been initialized. The database engine took care of this as the objects were created because of the auto-incrementing feature of primary keys discussed in Part 2.

Additionally, the person_id foreign key in all the Note instances has been initialized to reference its parent. This happens because of the order in which the Person and Note objects are created in the database.

SQLAlchemy is aware of the relationship between Person and Note objects. When a Person object is committed to the person database table, SQLAlchemy gets the person_id primary key value. That value is used to initialize the foreign key value of person_id in a Note object before it’s committed to the database.

SQLAlchemy takes care of this database housekeeping work because of the information you passed when the Person.notes attribute was initialized with the db.relationship(...) object.

In addition, the Person.timestamp attribute has been initialized with the current timestamp.

Running the build_database.py program from the command line (in the virtual environment will re-create the database with the new additions, getting it ready for use with the web application. This command line will rebuild the database:

$ python build_database.py

The build_database.py utility program doesn’t output any messages if it runs successfully. If it throws an exception, then an error will be printed on the screen.

Update REST API

You’ve updated the SQLAlchemy models and used them to update the people.db database. Now it’s time to update the REST API to provide access to the new notes information. Here’s the REST API you built in Part 2:

Action	HTTP Verb	URL Path	Description
Create	`POST`	`/api/people`	URL to create a new person
Read	`GET`	`/api/people`	URL to read a collection of people
Read	`GET`	`/api/people/{person_id}`	URL to read a single person by `person_id`
Update	`PUT`	`/api/people/{person_id}`	URL to update an existing person by `person_id`
Delete	`DELETE`	`/api/people/{person_id}`	URL to delete an existing person by `person_id`

The REST API above provides HTTP URL paths to collections of things, and to the things themselves. You can get a list of people or interact with a single person from that list of people. This path style refines what’s returned in a left-to-right manner, getting more granular as you go.

You’ll continue this left-to-right pattern to get more granular and access the notes collections. Here’s the extended REST API you’ll create in order to provide notes to the mini-blog web application:

Action	HTTP Verb	URL Path	Description
Create	`POST`	`/api/people/{person_id}/notes`	URL to create a new note
Read	`GET`	`/api/people/{person_id}/notes/{note_id}`	URL to read a single person’s single note
Update	`PUT`	`api/people/{person_id}/notes/{note_id}`	URL to update a single person’s single note
Delete	`DELETE`	`api/people/{person_id}/notes/{note_id}`	URL to delete a single person’s single note
Read	`GET`	`/api/notes`	URL to get all notes for all people sorted by `note.timestamp`

There are two variations in the notes part of the REST API compared to the convention used in the people section:

There is no URL defined to get all the notes associated with a person, only a URL to get a single note. This would have made the REST API complete in a way, but the web application you’ll create later doesn’t need this functionality. Therefore, it’s been left out.
There is the inclusion of the last URL /api/notes. This is a convenience method created for the web application. It will be used in the mini-blog on the home page to show all the notes in the system. There isn’t a way to get this information readily using the REST API pathing style as designed, so this shortcut has been added.

As in Part 2, the REST API is configured in the swagger.yml file.

Note:

The idea of designing a REST API with a path that gets more and more granular as you move from left to right is very useful. Thinking this way can help clarify the relationships between different parts of a database. Just be aware that there are realistic limits to how far down a hierarchical structure this kind of design should be taken.

For example, what if the Note object had a collection of its own, something like comments on the notes. Using the current design ideas, this would lead to a URL that went something like this: /api/people/{person_id}/notes/{note_id}/comments/{comment_id}

There is no practical limit to this kind of design, but there is one for usefulness. In actual use in real applications, a long, multilevel URL like that one is hardly ever needed. A more common pattern is to get a list of intervening objects (like notes) and then use a separate API entry point to get a single comment for an application use case.

Implement the API

With the updated REST API defined in the swagger.yml file, you’ll need to update the implementation provided by the Python modules. This means updating existing module files, like models.py and people.py, and creating a new module file called notes.py to implement support for Notes in the extended REST API.

Update Response JSON

The purpose of the REST API is to get useful JSON data out of the database. Now that you’ve updated the SQLAlchemy Person and created the Note models, you’ll need to update the Marshmallow schema models as well. As you may recall from Part 2, Marshmallow is the module that translates the SQLAlchemy objects into Python objects suitable for creating JSON strings.

The updated and newly created Marshmallow schemas are in the models.py module, which are explained below, and look like this:

 1 class PersonSchema(ma.ModelSchema):
 2     class Meta:
 3         model = Person
 4         sqla_session = db.session
 5     notes = fields.Nested('PersonNoteSchema', default=[], many=True)
 6 
 7 class PersonNoteSchema(ma.ModelSchema):
 8     """
 9     This class exists to get around a recursion issue
10     """
11     note_id = fields.Int()
12     person_id = fields.Int()
13     content = fields.Str()
14     timestamp = fields.Str()
15 
16 class NoteSchema(ma.ModelSchema):
17     class Meta:
18         model = Note
19         sqla_session = db.session
20     person = fields.Nested('NotePersonSchema', default=None)
21 
22 class NotePersonSchema(ma.ModelSchema):
23     """
24     This class exists to get around a recursion issue
25     """
26     person_id = fields.Int()
27     lname = fields.Str()
28     fname = fields.Str()
29     timestamp = fields.Str()

There are some interesting things going on in the above definitions. The PersonSchema class has one new entry: the notes attribute defined in line 5. This defines it as a nested relationship to the PersonNoteSchema. It will default to an empty list if nothing is present in the SQLAlchemy notes relationship. The many=True parameter indicates that this is a one-to-many relationship, so Marshmallow will serialize all the related notes.

The PersonNoteSchema class defines what a Note object looks like as Marshmallow serializes the notes list. The NoteSchema defines what a SQLAlchemy Note object looks like in terms of Marshmallow. Notice that it has a person attribute. This attribute comes from the SQLAlchemy db.relationship(...) definition parameter backref='person'. The person Marshmallow definition is nested, but because it doesn’t have the many=True parameter, there is only a single person connected.

The NotePersonSchema class defines what is nested in the NoteSchema.person attribute.

Note:

You might be wondering why the PersonSchema class has its own unique PersonNoteSchema class to define the notes collection attribute. By the same token, the NoteSchema class has its own unique NotePersonSchema class to define the person attribute. You may be wondering whether the PersonSchema class could be defined this way:

class PersonSchema(ma.ModelSchema):
    class Meta:
        model = Person
        sqla_session = db.session
    notes = fields.Nested('NoteSchema', default=[], many=True)

Additionally, couldn’t the NoteSchema class be defined using the PersonSchema to define the person attribute? A class definition like this would each refer to the other, and this causes a recursion error in Marshmallow as it will cycle from PersonSchema to NoteSchema until it runs out of stack space. Using the unique schema references breaks the recursion and allows this kind of nesting to work.

People

Now that you’ve got the schemas in place to work with the one-to-many relationship between Person and Note, you need to update the person.py and create the note.py modules in order to implement a working REST API.

The people.py module needs two changes. The first is to import the Note class, along with the Person class at the top of the module. Then only read_one(person_id) needs to change in order to handle the relationship. That function will look like this:

 1 def read_one(person_id):
 2     """
 3     This function responds to a request for /api/people/{person_id}
 4     with one matching person from people
 5 
 6     :param person_id:   Id of person to find
 7     :return:            person matching id
 8     """
 9     # Build the initial query
10     person = (
11         Person.query.filter(Person.person_id == person_id)
12         .outerjoin(Note)
13         .one_or_none()
14     )
15 
16     # Did we find a person?
17     if person is not None:
18 
19         # Serialize the data for the response
20         person_schema = PersonSchema()
21         data = person_schema.dump(person).data
22         return data
23 
24     # Otherwise, nope, didn't find that person
25     else:
26         abort(404, f"Person not found for Id: {person_id}")

The only difference is line 12: .outerjoin(Note). An outer join (left outer join in SQL terms) is necessary for the case where a user of the application has created a new person object, which has no notes related to it. The outer join ensures that the SQL query will return a person object, even if there are no note rows to join with.

At the start of this article, you saw how person and note data could be represented in a single, flat table, and all of the disadvantages of that approach. You also saw the advantages of breaking that data up into two tables, person and note, with a relationship between them.

Until now, we’ve been working with the data as two distinct, but related, items in the database. But now that you’re actually going to use the data, what we essentially want is for the data to be joined back together. This is what a database join does. It combines data from two tables together using the primary key to foreign key relationship.

A join is kind of a boolean and operation because it only returns data if there is data in both tables to combine. If, for example, a person row exists but has no related note row, then there is nothing to join, so nothing is returned. This isn’t what you want for read_one(person_id).

This is where the outer join comes in handy. It’s a kind of boolean or operation. It returns person data even if there is no associated note data to combine with. This is the behavior you want for read_one(person_id) to handle the case of a newly created Person object that has no notes yet.

You can see the complete people.py in the article repository.

Notes

You’ll create a notes.py module to implement all the Python code associated with the new note related REST API definitions. In many ways, it works like the people.py module, except it must handle both a person_id and a note_id as defined in the swagger.yml configuration file. As an example, here is read_one(person_id, note_id):

 1 def read_one(person_id, note_id):
 2     """
 3     This function responds to a request for
 4     /api/people/{person_id}/notes/{note_id}
 5     with one matching note for the associated person
 6 
 7     :param person_id:       Id of person the note is related to
 8     :param note_id:         Id of the note
 9     :return:                json string of note contents
10     """
11     # Query the database for the note
12     note = (
13         Note.query.join(Person, Person.person_id == Note.person_id)
14         .filter(Person.person_id == person_id)
15         .filter(Note.note_id == note_id)
16         .one_or_none()
17     )
18 
19     # Was a note found?
20     if note is not None:
21         note_schema = NoteSchema()
22         data = note_schema.dump(note).data
23         return data
24 
25     # Otherwise, nope, didn't find that note
26     else:
27         abort(404, f"Note not found for Id: {note_id}")

The interesting parts of the above code are lines 12 to 17:

Line 13 begins a query against the Note SQLAlchemy objects and joins to the related Person SQLAlchemy object comparing person_id from both Person and Note.
Line 14 filters the result down to the Note objects that has a Person.person_id equal to the passed in person_id parameter.
Line 15 filters the result further to the Note object that has a Note.note_id equal to the passed in note_id parameter.
Line 16 returns the Note object if found, or None if nothing matching the parameters is found.

You can check out the complete notes.py.

Updated Swagger UI

The Swagger UI has been updated by the action of updating the swagger.yml file and creating the URL endpoint implementations. Below is a screenshot of the updated UI showing the Notes section with the GET /api/people/{person_id}/notes/{note_id} expanded:

Mini-Blogging Web Application

The web application has been substantially changed to show its new purpose as a mini-blogging application. It has three pages:

The home page (localhost:5000/), which shows all of the blog messages (notes) sorted from newest to oldest
The people page (localhost:5000/people), which shows all the people in the system, sorted by last name, and also allows the user to create a new person and update or delete an existing one
The notes page (localhost:5000/people/{person_id}/notes), which shows all the notes associated with a person, sorted from newest to oldest, and also allows the user to create a new note and update or delete an existing one

There are two buttons on every page of the application:

The Home button will navigate to the home screen.
The People button navigates to the /people screen, showing all people in the database.

These two buttons are present on every screen in the application as a way to get back to a starting point.

Home Page

Below is a screenshot of the home page showing the initialized database contents:

The functionality of this page works like this:

Double-clicking on a person’s name will take the user to the /people/{person_id} page, with the editor section filled in with the person’s first and last names and the update and reset buttons enabled.
Double-clicking on a person’s note will take the user to the /people/{person_id}/notes/{note_id} page, with the editor section filled in with the note’s contents and the Update and Reset buttons enabled.

People Page

Below is a screenshot of the people page showing the people in the initialized database:

The functionality of this page works like this:

Single-clicking on a person’s name will populate the editor section of the page with the person’s first and last name, disabling the Create button, and enabling the Update and Delete buttons.
Double clicking on a person’s name will navigate to the notes pages for that person.

The functionality of the editor works like this:

If the first and last name fields are empty, the Create and Reset buttons are enabled. Entering a new name in the fields and clicking Create will create a new person and update the database and re-render the table below the editor. Clicking Reset will clear the editor fields.
If the first and last name fields have data, the user navigated here by double-clicking the person’s name from the home screen. In this case, the Update, Delete, and Reset buttons are enabled. Changing the first or last name and clicking Update will update the database and re-render the table below the editor. Clicking Delete will remove the person from the database and re-render the table.

Notes Page

Below is a screenshot of the notes page showing the notes for a person in the initialized database:

The functionality of this page works like this:

Single-clicking on a note will populate the editor section of the page with the notes content, disabling the Create button, and enabling the Update and Delete buttons.
All other functionality of this page is in the editor section.

The functionality of the editor works like this:

If the note content field is empty, then the Create and Reset buttons are enabled. Entering a new note in the field and clicking Create will create a new note and update the database and re-render the table below the editor. Clicking Reset will clear the editor fields.
If the note field has data, the user navigated here by double-clicking the person’s note from the home screen. In this case, the Update, Delete, and Reset buttons are enabled. Changing the note and clicking Update will update the database and re-render the table below the editor. Clicking Delete will remove the note from the database and re-render the table.

Web Application

This article is primarily focused on how to use SQLAlchemy to create relationships in the database, and how to extend the REST API to take advantage of those relationships. As such, the code for the web application didn’t get much attention. When you look at the web application code, keep an eye out for the following features:

Each page of the application is a fully formed single page web application.
Each page of the application is driven by JavaScript following an MVC (Model/View/Controller) style of responsibility delegation.
The HTML that creates the pages takes advantage of the Jinja2 inheritance functionality.
The hardcoded JavaScript table creation has been replaced by using the Handlebars.js templating engine.
The timestamp formating in all of the tables is provided by Moment.js.

You can find the following code in the repository for this article:

All of the example code for this article is available in the GitHub repository for this article. This contains all of the code related to this article, including all of the web application code.

Conclusion

Congratulations are in order for what you’ve learned in this article! Knowing how to build and use database relationships gives you a powerful tool to solve many difficult problems. There are other relationship besides the one-to-many example from this article. Other common ones are one-to-one, many-to-many, and many-to-one. All of them have a place in your toolbelt, and SQLAlchemy can help you tackle them all!

For more information about databases, you can check out these tutorials. You can also set up Flask to use SQLAlchemy. You can check out Model-View-Controller (MVC) more information about the pattern used in the web application JavaScript code.

In Part 4 of this series, you’ll focus on the HTML, CSS, and JavaScript files used to create the web application.

« Part 2: Database Persistence

Part 3: Database Relationships

Part 4: Coming Soon! »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Development in Visual Studio Code (Setup Guide)

2019-04-04T14:00:00+00:00

Learn how to set up Visual Studio Code for Python development. By following examples, you’ll cover everything from installing and configuring VSCode, to running tests and debugging Python apps. Make the most of this powerful Python IDE by following our complete setup guide.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

What Is Pip? A Guide for New Pythonistas

2019-04-03T14:00:00+00:00

What is pip? pip is the standard package manager for Python. It allows you to install and manage additional packages that are not part of the Python standard library. This tutorial is an introduction to pip for new Pythonistas.

In this tutorial, you’ll learn about:

Installing additional packages not included with the standard Python distribution
Finding packages published to the Python Package Index (PyPI)
Managing requirements for your scripts and applications
Uninstalling packages and their dependencies

As you’ll see, the Python community is very active and has created some neat alternatives to pip that you’ll learn about later in this tutorial.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Getting Started With `pip`

So, what is pip? pip is a package manager for Python. That means it’s a tool that allows you to install and manage additional libraries and dependencies that are not distributed as part of the standard library.

Package management is so important that pip has been included with the Python installer since versions 3.4 for Python 3 and 2.7.9 for Python 2, and it’s used by many Python projects, which makes it an essential tool for every Pythonista.

The concept of a package manager might be familiar to you if you are coming from other languages. JavaScript uses npm for package management, Ruby uses gem, and .NET use NuGet. In Python, pip has become the standard package manager.

The Python installer installs pip, so it should be ready for you to use, unless you installed an old version of Python. You can verify that pip is available by running the following command in your console:

$ pip --version

pip 18.1 from C:\Python37\lib\site-packages\pip (python 3.7)

You should see a similar output displaying the pip version, as well as the location and version of Python. If you are using an old version of Python that does not include pip, then you can install it by following the instructions for your system in the pip installation documentation.

You probably want to follow the examples in this tutorial inside a virtual environment to avoid installing packages to the global Python installation. You can learn about virtual environments in Python Virtual Environments: A Primer. The Using Virtual Environments section of that article explains the basics of creating new virtual environments.

Installing Packages With `pip`

Python is considered a batteries included language. This means that the Python standard library includes an extensive set of packages and modules to help developers with their scripts and applications.

At the same time, Python has a very active community that contributes an even bigger set of packages that can help you with your development needs. These packages are published to the Python Package Index, also known as PyPI (pronounced Pie Pea Eye). PyPI hosts an extensive collection of packages that include development frameworks, tools, and libraries.

Many of these packages simplify Python development by providing friendly interfaces to functionality that already exists in the standard library. For example, you can write a script that retrieves the contents of a web page using only the standard libraries included with Python:

# In using-http.py

import cgi
import http.client

server = 'www.google.com'
url = '/'
conn = http.client.HTTPSConnection(server)
conn.request('GET', url)
response = conn.getresponse()
content_type = response.headers.get('Content-Type')
_, params = cgi.parse_header(content_type)
encoding = params.get('charset')
data = response.read()
text = data.decode(encoding)

print(f'Response returned: {response.status} ({response.reason})')
print('Body:')
print(text)

In this script, you import cgi and http.client, both of which are included in the Python standard library. You create an HTTPSConnection object specifying the server and invoke its .request() and .getresponse() to retrieve a response.

From the response, you can retrieve the Content-Type header and parse it using the cgi module to extract the charset in which the page is encoded.

cgi.parse_header() returns a tuple with a main value and a dictionary of parameters. For example, the Content-Type header might contain a value like text/html; charset=ISO-8859-1.

The tuple will contain the string text/html as the first element, and the second element will be a dictionary in the form {'charset': 'ISO-8859-1'}. Because you only care about the charset parameter, you can ignore the beginning of the tuple using an underscore: _, params = cgi.parse_header(content_type).

Note: The Meaning of Underscores in Python explains how to use underscores to unpack values from a tuple.

Once you have the encoding of the page, you can read the response and decode it into text. You can run the example in the console to see how it works:

$ python using-http.py

Response returned: 200 (OK)
Body:
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" 
lang="en"><head><meta content="Search the world's information, including 
webpages, images, videos and more. Google has many special features to help you 
find exactly what you're looking for." name="description"><meta content="noodp" 
name="robots">... Additional Output Omitted

This seems like a lot of work for a small script that retrieves the contents of a web page. Fortunately, there is a Python package that simplifies HTTP requests and provides a nice interface to do exactly what you want.

Basic Package Installation

PyPI hosts a very popular library to perform HTTP requests called requests. You can learn all about it in its official documentation site.

The first step is to install the requests package into your environment. You can learn about pip supported commands by running it with help:

$ pip help

Usage:
  pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
  show                        Show information about installed packages.
  check                       Verify installed packages have compatible 
                              dependencies.
  config                      Manage local and global configuration.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring environment 
                              variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be 
                              used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be 
                              used up to 3 times (corresponding to WARNING, 
                              ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form 
                              [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should 
                              attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: 
                              (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort).
  --trusted-host <hostname>   Mark this host as trusted, even though it does 
                              not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file 
                              containing the private key and the certificate in 
                              PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine 
                              whether a new version of pip is available for 
                              download. Implied with --no-index.

As you can see, pip provides an install command to install packages. You can run it to install the requests package:

$ pip install requests

Looking in indexes: https://pypi.org/simple
Collecting requests
  Using cached 
  https://files.pythonhosted.org/packages/7d/e3/
  20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/
  requests-2.21.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/
  01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/
  chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/14/2c/
  cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/
  idna-2.8-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/62/00/
  ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/
  urllib3-1.24.1-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Using cached https://files.pythonhosted.org/packages/9f/e0/
  accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/
  certifi-2018.11.29-py2.py3-none-any.whl
Installing collected packages: chardet, idna, urllib3, certifi, requests
Successfully installed certifi-2018.11.29 chardet-3.0.4 idna-2.8 
  requests-2.21.0 urllib3-1.24.1
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' 
  command.

You should see an output similar to the one above. You use pip with an install command followed by the name of the package you want to install. pip looks for the package in PyPI, calculates its dependencies, and installs them to insure requests will work.

You can also see that the current environment is using pip version 18.1, but version 19.0.1 is available. It also shows the command you should use to update pip, so let’s do that:

$ python -m pip install --upgrade pip

Looking in indexes: https://pypi.org/simple
Collecting pip
  Downloading https://files.pythonhosted.org/packages/46/dc/
  7fd5df840efb3e56c8b4f768793a237ec4ee59891959d6a215d63f727023/
  pip-19.0.1-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 2.0MB/s
Installing collected packages: pip
  Found existing installation: pip 18.1
    Uninstalling pip-18.1:
      Successfully uninstalled pip-18.1
Successfully installed pip-19.0.1

Notice that you use python -m to update pip. The -m switch tells Python to run a module as an executable. This is necessary because in order for you to update pip, the old version has to be uninstalled before installing the new version, and removing it while running the tool can cause errors.

When you run pip as a module, Python loads the module in memory and allows the package to be removed while it is being used. You can run packages as if they were scripts if the package provides a top-level script __main__.py.

Now that you have installed requests and upgraded pip, you can use the list command to see the packages installed in your environment:

$ pip list

Package    Version
---------- ----------
certifi    2018.11.29
chardet    3.0.4
idna       2.8
pip        19.0.1
requests   2.21.0
setuptools 40.6.2
urllib3    1.24.1

As you can see, pip has been upgraded to version 19.0.1 (the latest version at the moment), and requests version 2.21.0 has been installed.

The pip install <package> command always looks for the latest version of the package and installs it. It also searches for dependencies listed in the package metadata and installs those dependencies to insure that the package has all the requirements it needs.

As you can see, multiple packages were installed. You can look at the package metadata by using the show command in pip:

$ pip show requests

Name: requests
Version: 2.21.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: me@kennethreitz.org
License: Apache 2.0
Location: py37\lib\site-packages
Requires: certifi, chardet, idna, urllib3
Required-by:

The metadata lists certifi, chardet, idna, and urllib3 as dependencies, and you can see they were also installed.

With the requests package installed, you can modify the example above and see how easy it is to retrieve the contents of a web page:

# In using-requests.py

import requests

url = 'https://www.google.com'
response = requests.get(url)
print(f'Response returned: {response.status_code}, {response.reason}')
print(response.text)

You can import the requests package as any other standard package because it is now installed in your environment.

As you can see, requests.get() handles the HTTP connection for you and returns a response object similar to the original example but with some interface improvements.

You don’t have to deal with the encoding of the page because requests will handle that for you in most situations. Still, requests provides a flexible interface to handle special cases through the requests.Response object.

Using Requirement Files

The pip install command always installs the latest published version of a package, but sometimes, you may want to install a specific version that you know works with your code.

You want to create a specification of the dependencies and versions you used to develop and test your application, so there are no surprises when you use the application in production.

Requirement files allow you to specify exactly which packages and versions should be installed. Running pip help shows that there is a freeze command that outputs the installed packages in requirements format. You can use this command, redirecting the output to a file to generate a requirements file:

$ pip freeze > requirements.txt
$ cat requirements.txt

certifi==2018.11.29
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1

The freeze command dumps all the packages and their versions to standard output, so you can redirect the output to a file that can be used to install the exact requirements into another system. The convention is to name this file requirements.txt, but you can give it any name you want.

When you want to replicate the environment in another system, you can run pip install specifying the requirements file using the -r switch:

$ pip install -r requirements.txt

Looking in indexes: https://pypi.org/simple
Collecting certifi==2018.11.29 (from -r .\requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/9f/e0/
  accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/
  certifi-2018.11.29-py2.py3-none-any.whl
Collecting chardet==3.0.4 (from -r .\requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/bc/a9/
  01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/
  chardet-3.0.4-py2.py3-none-any.whl
Collecting idna==2.8 (from -r .\requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/14/2c/
  cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/
  idna-2.8-py2.py3-none-any.whl
Collecting requests==2.21.0 (from -r .\requirements.txt (line 4))
  Using cached https://files.pythonhosted.org/packages/7d/e3/
  20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/
  requests-2.21.0-py2.py3-none-any.whl
Collecting urllib3==1.24.1 (from -r .\requirements.txt (line 5))
  Using cached https://files.pythonhosted.org/packages/62/00/
  ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/
  urllib3-1.24.1-py2.py3-none-any.whl
Installing collected packages: certifi, chardet, idna, urllib3, requests
Successfully installed certifi-2018.11.29 chardet-3.0.4 idna-2.8 
  requests-2.21.0 urllib3-1.24.1

The versions of the packages will match those listed in requirements.txt:

$ pip list

Package    Version
---------- ----------
certifi    2018.11.29
chardet    3.0.4
idna       2.8
pip        19.0.1
requests   2.21.0
setuptools 40.6.2
urllib3    1.24.1

You can submit the requirements.txt file into source control and use it to create the exact environment in other machines.

Fine-Tuning Requirements

The problem with hardcoding the versions of your packages and their dependencies is that packages are updated frequently with bug and security fixes, and you probably want to leverage those as soon as they are published.

The requirements file format allows you to specify dependency versions using logical operators that give you a bit of flexibility to insure packages are updated, but still define the base versions of a package.

Open the requirements.txt file in your favorite editor and make the following changes:

certifi>=2018.11.29
chardet>=3.0.4
idna>=2.8
requests>=2.21.0
urllib3>=1.24.1

You can change the logical operator to >= to tell pip to install an exact or greater version that has been published. When you set a new environment using the requirments.txt file, pip looks for the latest version that satisfies the requirement and installs it. You can upgrade the packages in your requirements file by running the install command with the --upgrade switch:

$ pip install --upgrade -r requirements.txt

Looking in indexes: https://pypi.org/simple
Requirement already up-to-date: certifi==2018.11.29 in \py37\lib\site-packages 
  (from -r .\requirements.txt (line 1)) (2018.11.29)
Requirement already up-to-date: chardet==3.0.4 in \py37\lib\site-packages 
  (from -r .\requirements.txt (line 2)) (3.0.4)
Requirement already up-to-date: idna==2.8 in \py37\lib\site-packages 
  (from -r .\requirements.txt (line 3)) (2.8)
Requirement already up-to-date: requests==2.21.0 in \py37\lib\site-packages 
  (from -r .\requirements.txt (line 4)) (2.21.0)
Requirement already up-to-date: urllib3==1.24.1 in \py37\lib\site-packages 
  (from -r .\requirements.txt (line 5)) (1.24.1)

Nothing was upgraded because you have the latest versions, but if a new version was published for a listed package, then the package would’ve been upgraded.

In an ideal world, new versions of packages would be backwards compatible and would never introduce new bugs. Unfortunately, new versions can introduce changes that will break your application. The requirements file syntax supports additional version specifiers to fine-tune your requirements.

Let’s say that a new version 3.0 of requests is published but introduces an incompatible change that breaks your application. You can modify the requirements file to prevent 3.0 or higher from being installed:

certifi>=2018.11.29
chardet>=3.0.4
idna>=2.8
requests>=2.21.0, <3.0
urllib3>=1.24.1

Changing the version specifier for the requests package ensures that any version greater or equal to 3.0 does not get installed. The pip documentation provides all the information about the requirements file format, and you can consult it to learn more about it.

Production vs Development Dependencies

Not all packages that you install during the development of your applications are going to be application dependencies. There are many packages published to PyPI that are development tools or libraries that you want to leverage during the development process.

As an example, you’ll probably want to unit test your application, so you need a unit test framework. A popular framework for unit testing is pytest. You want to install it in your development environment, but you do not want it in your production environment because it isn’t an application dependency.

You create a second requirements file (requirements_dev.txt) to list additional tools to set up a development environment:

# In requirements_dev.txt
pytest>=4.2.0

This requires you to use pip to install both requirement files: requirements.txt and requirements_dev.txt. Fortunately, pip allows you to specify additional parameters within a requirements file. You can modify requirements_dev.txt to also install the requirements from the production requirements.txt file:

# In requirements_dev.txt
-r requirements.txt
pytest>=4.2.0

Notice that you are using the exact same -r switch to install the production requirements.txt file. The requirements file format allows you to specify additional arguments right on a requirements file.

Freezing Requirements for Production

You created the production and development requirement files and added them to source control. The files use flexible version specifiers to ensure that you leverage bug fixes published by your dependencies. You are also testing your application and are ready to deploy it to production.

You probably want to ensure that the versions of the dependencies you deploy to production are the exact same versions you used in your integration pipeline or build process because you know all the tests pass and the application works.

The current version specifiers don’t guarantee that the same versions will be deployed to production, so you want to freeze the production requirements as you saw earlier.

You create a clean production virtual environment and install the production requirements using the requirements.txt file. Once the requirements are installed, you can freeze the specific versions, dumping the output to a requirements_lock.txt file that you use in production. The requirements_lock.txt file will contain exact versions specifiers and can be used to replicate the environment.

Finding Packages to Use

As you become a more experienced Pythonista, there’ll be a set of packages that you’ll know by heart and that you’ll use in most of your applications. The requests and pytest packages are good candidates to become useful tools in your Python toolbox.

There will be times though when you will need to solve a different problem, and you will want to look for a different tool or library that can help you with it. As you can see above, pip help shows that there is a search command that looks for packages published to PyPI.

Let’s see how this command can help us:

$ pip help search

Usage:
  pip search [options] <query>

Description:
  Search for PyPI packages whose name or summary contains <query>.

Search Options:
  -i, --index <url>           Base URL of Python Package Index 
                              (default https://pypi.org/pypi)

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring environment 
                              variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be 
                              used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be 
                              used up to 3 times (corresponding to WARNING, 
                              ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form 
                              [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should 
                              attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: 
                              (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort).
  --trusted-host <hostname>   Mark this host as trusted, even though it does 
                              not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file 
                              containing the private key and the certificate in 
                              PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine 
                              whether a new version of pip is available for 
                              download. Implied with --no-index.
  --no-color                  Suppress colored output

The command takes a set of options listed above and a <query>. The query is just a string to search for and will match packages and their descriptions.

Note: You can use pip help <command> to retrieve additional information about a supported command.

Let’s say your application needs to access a service that is using OAuth2 for authorization. Ideally, there is a library that works with requests or with a similar interface that can help us. Let’s search PyPI for it using pip:

$ pip search requests oauth

requests-oauth (0.4.1)             - Hook for adding Open Authentication 
                                     support to Python-requests HTTP library.
oauth (1.0.1)                      - Library for OAuth version 1.0a.
pmr2.oauth (0.6.1)                 - OAuth PAS Plugin, OAuth 1.0 provider for 
                                     Plone.
oauth-proxy (1.0.5)                - OAuth HTTP proxy
django-oauth (1.1)                 - Support of OAuth in Django.
intuit-oauth (1.2.0)               - Intuit OAuth Client
brubeck-oauth (0.1.11)             - Brubeck OAuth module
guillotina-oauth (2.0.0)           - guillotina oauth support
httpie-oauth (1.0.2)               - OAuth plugin for HTTPie.
paytm-oauth (0.2)                  - Consumer for paytm oauth
plurk-oauth (0.9.2)                - Plurk OAuth API
oauth-flow (1.0.3)                 - Authenticate and make calls to OAuth 1.0, 
                                     OAuth 2.0 services
... Additional Output Omitted

The search term yields quite an extensive collection of packages. Some of them seem specific to a service or technology like django-oauth. Others look promising, like requests-oauth. Unfortunately, there isn’t much information other than a brief description.

Most of the time, you want to search for packages directly in the PyPI website. PyPI provides search capabilities for its index and a way to filter results by the metadata exposed in the package, like framework, topic, development status, and so on.

A search for the same terms in PyPI yields a lot of results, but you can filter them by different categories. For example, you can expand the Intended Audience and select Developers since you want a library that helps you with developing your application. Also, you probably want a package that is stable and production-ready. You can expand the Development Status category and select Production/Stable:

You can apply additional filters and tweak the search terms until you find the package that you are looking for.

The results provide a link to the package page, which contains more information and hopefully some documentation. Let’s take a look at the information for requests-oauth2:

The project page provides more information, and it seems to have a link to the project homepage. The link takes you to the project repository on GitHub. There, you can see some more information about the project and some usage examples.

Finding the original source code repository can be an invaluable resource. There, you can find some hints about the status of the project by looking at the date of the latest commits, number of pull request and open issues, and so forth.

Another option to find a package is to Google it. Widely used Python libraries will show up at the top of google searches, and you should be able to find a link to the package in PyPI or its source code repository.

Finding the right package may take some time and research, but it will also speed up your development process once you find it.

Uninstalling Packages

Once in a while, you will have to uninstall a package. You either found a better library to replace it, or it is something you don’t really need. Uninstalling packages can be a bit tricky.

Notice that, when you installed requests, pip installed other dependencies too. The more packages you install, the bigger the chances that multiple packages depend on the same dependency. This is where the show command in pip comes in handy.

Before you uninstall a package, make sure you run the show command for that package:

$ pip show requests

Name: requests
Version: 2.21.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: me@kennethreitz.org
License: Apache 2.0
Location: c:\users\isaac\projects\virtualenv\py37\lib\site-packages
Requires: urllib3, certifi, chardet, idna
Required-by:

Notice the last two fields Requires and Required-by. The show command tells us that requests requires urllib3, certifi, chardet, and idna. You probably want to uninstall those two. You can also see that requests is not required by any other package, so it is safe to uninstall it.

You should run the show command against all of the requests dependencies to make sure that no other libraries also depend on them. Once you understand the dependency order of the packages you want to uninstall, you can remove them using the uninstall command:

$ pip uninstall certifi

Uninstalling certifi-2018.11.29:
  Would remove:
    \py37\lib\site-packages\certifi-2018.11.29.dist-info\*
    \py37\lib\site-packages\certifi\*
Proceed (y/n)? y
  Successfully uninstalled certifi-2018.11.29

Uninstalling a package shows you the files that will be removed and will ask for confirmation. If you are sure you want to remove the package because you’ve checked its dependencies and know that nothing else is using it, you can pass a -y switch to suppress the file list and confirmation:

$ pip uninstall urllib3 -y

Uninstalling urllib3-1.24.1:
  Successfully uninstalled urllib3-1.24.1

$ pip uninstall chardet -y

Uninstalling chardet-3.0.4:
  Successfully uninstalled chardet-3.0.4

$ pip uninstall idna -y

Uninstalling idna-2.8:
  Successfully uninstalled idna-2.8

$ pip uninstall requests -y

Uninstalling requests-2.21.0:
  Successfully uninstalled requests-2.21.0

You can specify all the packages you want to uninstall in a single call: pip uninstall -y urllib3 chardet idna requests.

You can also uninstall all the packages listed in a requirements file by providing the -r <requirments file> option. The command will ask confirmation for each individual package, but you can suppress it, if you know what you’re doing, with the -y switch:

$ pip uninstall -r requirements.txt -y

Uninstalling certifi-2018.11.29:
  Successfully uninstalled certifi-2018.11.29
Uninstalling chardet-3.0.4:
  Successfully uninstalled chardet-3.0.4
Uninstalling idna-2.8:
  Successfully uninstalled idna-2.8
Uninstalling requests-2.21.0:
  Successfully uninstalled requests-2.21.0
Uninstalling urllib3-1.24.1:
  Successfully uninstalled urllib3-1.24.1

Remember to always check the dependencies of packages you want to uninstall. You probably want to uninstall all its dependencies, but uninstalling a package that is being used by others will break your application.

Alternatives to `pip`

pip is an essential tool for all Pythonistas, and it is used by many applications and projects for package management. This tutorial has helped you with the basics, but the Python community is very active in providing great tools and libraries for other developers to use. These include other alternatives to pip that try to simplify and improve package management.

In this section, you’ll learn about other package management tools available for Python.

Conda Does It All

Conda is a package, dependency, and environment manager for many languages including Python. In fact, its origin comes from Anaconda, which started as a data science package for Python.

Conda is widely used for data science and machine learning applications, and uses its own index to host compatible packages.

Conda not only allows you to manage package dependencies, but it also manages virtual environments for your applications, installs compatible Python distributions, and packages your application for deployment to production.

Setting Up Python for Machine Learning on Windows is a great introduction to Conda that explores package and environment management. The only Windows-specific information is around installation, so it’s still relevant if you use a different OS platform.

Pipenv

Pipenv is another package management tool that “aims to bring the best of all packaging worlds” to Python. It’s gaining a lot of traction among the Python community because it merges virtual environment and package management in a single tool.

It also solves some of the most common hiccups you will run into when manually managing dependencies through pip, like versions of packages, separating development and production dependencies, and locking versions for production.

Pipenv: A Guide to the New Python Packaging Tool is a great start to learn about Pipenv and its approach to package management. Even though the article is tagged as intermediate, the author does a great job of guiding the reader that the article is accessible to anyone starting with Python.

Poetry

Poetry is another pip alternative that is gaining a lot of traction. Like Pipenv, it simplifies package version management and separates development vs production dependencies, and it works by isolating those dependencies into a virtual environment.

If you are coming from JavaScript and npm, then Poetry will look very familiar. It goes beyond package management, helping you build distributions for your applications and libraries and deploying them to PyPI. How to Publish an Open-Source Python Package to PyPI has a good introduction to Poetry and can help you get started.

Conclusion: What Is `pip`?

This tutorial answered the question, what is pip? You’ve seen that pip is a package manager for Python, used by many projects to manage dependencies. It’s included with the Python installer, which makes it an essential tool for all Pythonistas to know how to use.

Python provides an extensive standard library suitable for developing all sorts of applications, but the active Python community provides an even larger set of tools and libraries that speed up Python application development.

These tools and libraries are published to the Python Package Index (PyPI), and pip allows developers to install them in their application environments.

In this tutorial, you’ve learned about:

Installing new packages using pip in the command line and with requirement files
Managing dependencies, separating development and production requirements, and creating a locked requirements file
Finding packages through pip and PyPI
Evaluating package dependencies before uninstalling a package and how pip uninstalls packages

In addition, you’ve learned about the importance of keeping dependencies up to date and alternatives to pip that can help you manage those dependencies.

Feel free to reach out in the comments section below with any questions you might have, and you can always get more information at the pip documentation page.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Idiomatic Pandas: Tricks & Features You May Not Know

2019-04-02T14:00:00+00:00

Pandas is a foundational library for analytics, data processing, and data science. It’s a huge project with tons of optionality and depth.

In this course you’ll see how to use some lesser-used but idiomatic Pandas capabilities that lend your code better readability, versatility, and speed.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Get Started With Django Part 1: Build a Portfolio App

2019-04-01T14:00:00+00:00

Django is a fully featured Python web framework that can be used to build complex web applications. In this tutorial, you’ll jump in and learn Django by example. You’ll follow the steps to create a fully functioning web application and, along the way, learn some of the most important features of the framework and how they work together.

In later posts in this series, you’ll see how to build more complex websites using even more of Django’s features than you’ll cover in this tutorial.

By the end of this tutorial, you will be able to:

Understand what Django is and why it’s a great web framework
Understand the architecture of a Django site and how it compares with other frameworks
Set up a new Django project and app
Build a Personal Portfolio Website with Django

Free Bonus: Click here to get access to a free Django Learning Resources Guide (PDF) that shows you tips and tricks as well as common pitfalls to avoid when building Python + Django web applications.

Why You Should Learn Django

There are endless web development frameworks out there, so why should you learn Django over any of the others? First of all, it’s written in Python, one of the most readable and beginner-friendly programming languages out there.

Note: This tutorial assumes an intermediate knowledge of the Python language. If you’re new to programming with Python, check out some of our beginner tutorials or the introductory course.

The second reason you should learn Django is the scope of its features. If you need to build a website, you don’t need to rely on any external libraries or packages if you choose Django. This means that you don’t need to learn how to use anything else, and the syntax is seamless as you’re using only one framework.

There’s also the added benefit that you don’t need to worry that updating one library or framework will render others that you’ve installed useless.

If you do find yourself needing to add extra features, there are a range of external libraries that you can use to enhance your site.

One of the great things about the Django framework is its in-depth documentation. It has detailed documentation on every aspect of Django and also has great examples and even a tutorial to get you started.

There’s also a fantastic community of Django developers, so if you get stuck there’s almost always a way forward by either checking the docs or asking the community.

Django is a high-level web application framework with loads of features. It’s great for anyone new to web development due to its fantastic documentation, and particularly if you’re also familiar with Python.

The Structure of a Django Website

A Django website consists of a single project that is split into separate apps. The idea is that each app handles a self-contained function that the site needs to perform. As an example, imagine an application like Instagram. There are several different functions that need to be performed:

User management: Login, logout, register, and so on
The image feed: Uploading, editing, and displaying images
Private messaging: Private messages between users and notifications

These are each separate pieces of functionality, so if this were a Django site, then each piece of functionality should be a different Django app inside a single Django project.

The Django project holds some configurations that apply to the project as a whole, such as project settings, URLs, shared templates and static files. Each application can have its own database and has its own functions to control how the data is displayed to the user in HTML templates.

Each application also has its own URLs as well as its own HTML templates and static files, such as JavaScript and CSS.

Django apps are structured so that there is a separation of logic. It supports the Model-View-Controller Pattern, which is the architecture on which most web frameworks are built. The basic principle is that in each application there are three separate files that handle the three main pieces of logic separately:

Model defines the data structure. This is usually a database and is the base layer to an application.
View displays some or all of the data to the user with HTML and CSS.
Controller handles how the database and the view interact.

If you want to learn more about the MVC pattern, then check out Model-View-Controller (MVC) Explained – With Legos.

In Django, the architecture is slightly different. Although based upon the MVC pattern, Django handles the controller part itself. There’s no need to define how the database and views interact. It’s all done for you!

The pattern Django utilizes is called the Model-View-Template (MVT) pattern. The view and template in the MVT pattern make up the view in the MVC pattern. All you need to do is add some URL configurations to map the views to, and Django handles the rest!

A Django site starts off as a project and is built up with a number of applications that each handle separate functionality. Each app follows the Model-View-Template pattern. Now that you’re familiar with the structure of a Django site, let’s have a look at what you’re going to build!

What You’re Going to Build

Before you get started with any web development project, it’s a good idea to come up with a plan of what you’re going to build. In this tutorial, we are going to build an application with the following features:

A fully functioning blog: If you’re looking to demonstrate your coding ability, a blog is a great way to do that. In this application, you will be able to create, update, and delete blog posts. Posts will have categories that can be used to sort them. Finally, users will be able to leave comments on posts.
A portfolio of your work: You can showcase previous web development projects here. You’ll build a gallery style page with clickable links to projects that you’ve completed.

Note: Before you get started, you can pull down the source code and follow along with the tutorial.

If you prefer to follow along by writing the code yourself, don’t worry. I’ve referenced the relevant parts of the source code throughout so you can refer back to it.

We won’t be using any external Python libraries in this tutorial. One of the great things about Django is that it has so many features that you don’t need to rely on external libraries. However, we will add Bootstrap 4 styling in the templates.

By building these two apps, you’ll learn the basics of Django models, view functions, forms, templates, and the Django admin page. With knowledge of these features, you’ll be able to go away and build loads more applications. You’ll also have the tools to learn even more and build sophisticated Django sites.

Hello, World!

Now that you know the structure of a Django application, and what you are about to build, we’re going to go through the process of creating an application in Django. You’ll extend this later into your personal portfolio application.

Set Up Your Development Environment

Whenever you are starting a new web development project, it’s a good idea to first set up your development environment. Create a new directory for your project to live in, and cd into it:

$ mkdir rp-portfolio
$ cd rp-portfolio

Once your inside the main directory, it’s a good idea to create a virtual environment to manage dependencies. There are many different ways to set up virtual environments, but here you’re going to use venv:

$ python3 -m venv venv

This command will create a folder venv in your working directory. Inside this directory, you’ll find several files including a copy of the Python standard library. Later, when you install new dependencies, they will also be stored in this directory. Next, you need to activate the virtual environment by running the following command:

$ source venv/bin/activate

Note: If you’re not using bash shell, you might need to use a different command to activate your virtual environment. For example, on windows you need this command:

C:\> venv\Scripts\activate.bat

You’ll know that your virtual environment has been activated, because your console prompt in the terminal will change. It should look something like this:

(venv) $

Note: Your virtual environment directory doesn’t have to be called venv. If you want to create one under a different name, for example my_venv, just replace with the second venv with my_venv.

Then, when activating your virtual environment, replace venv with my_venv again. The prompt will also now be prefixed with (my_venv).

Now that you’ve created a virtual environment, it’s time to install Django. You can do this using pip:

(venv) $ pip install Django

Once you’ve set up the virtual environment and installed Django, you can now dive in to creating the application.

Create a Django Project

As you saw in the previous section, a Django web application is made up of a project and its constituent apps. Making sure you’re in the rp_portfolio directory, and you’ve activated your virtual environment, run the following command to create the project:

$ django-admin startproject personal_portfolio

This will create a new directory personal_portfolio. If you cd into this new directory you’ll see another directory called personal_portfolio and a file called manage.py. Your directory structure should look something like this:

rp-portfolio/
│
├── personal_portfolio/
│   ├── personal_portfolio/
│   │   ├── __init__.py
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   │
│   └── manage.py
│
└── venv/

Most of the work you do will be in that first personal_portfolio directory. To save having to cd through several directories each time you come to work on your project, it can be helpful to reorder this slightly by moving all the files up a directory. While you’re in the rp-portfolio directory, run the following commands:

$ mv personal_portfolio/manage.py ./
$ mv personal_portfolio/personal_portfolio/* personal_portfolio
$ rm -r personal_portfolio/personal_portfolio/

You should end up with something like this:

rp-portfolio/
│
├── personal_portfolio/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
│
├── venv/
│
└── manage.py

Once your file structure is set up, you can now start the server and check that your set up was successful. In the console, run the following command:

$ python manage.py runserver

Then, in your browser go to localhost:8000, and you should see the following:

Congratulations, you’ve created a Django site! The source code for this part of the tutorial can be found on GitHub. The next step is to create apps so that you can add views and functionality to your site.

Create a Django Application

For this part of the tutorial, we’ll create an app called hello_world, which you’ll subsequently delete as its not necessary for our personal portfolio site.

To create the app, run the following command:

$ python manage.py startapp hello_world

This will create another directory called hello_world with several files:

__init__.py tells Python to treat the directory as a Python package.
admin.py contains settings for the Django admin pages.
apps.py contains settings for the application configuration.
models.py contains a series of classes that Django’s ORM converts to database tables.
tests.py contains test classes.
views.py contains functions and classes that handle what data is displayed in the HTML templates.

Once you’ve created the app, you need to install it in your project. In rp-portfolio/settings.py, add the following line of code under INSTALLED_APPS:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'hello_world',
]

That line of code means that your project now knows that the app you just created exists. The next step is to create a view so that you can display something to a user.

Create a View

Views in Django are a collection of functions or classes inside the views.py file in your app directory. Each function or class handles the logic that gets processed each time a different URL is visited.

Navigate to the views.py file in the hello_world directory. There’s already a line of code in there that imports render(). Add the following code:

from django.shortcuts import render

def hello_world(request):
    return render(request, 'hello_world.html', {})

In this piece of code, you’ve defined a view function called hello_world(). When this function is called, it will render an HTML file called hello_world.html. That file doesn’t exist yet, but we’ll create it soon.

The view function takes one argument, request. This object is an HttpRequestObject that is created whenever a page is loaded. It contains information about the request, such as the method, which can take several values including GET and POST.

Now that you’ve created the view function, you need to create the HTML template to display to the user. render() looks for HTML templates inside a directory called templates inside your app directory. Create that directory and subsequently a file named hello_world.html inside it:

$ mkdir hello_world/templates/
$ touch hello_world/templates/hello_world.html

Add the following lines of HTML to your file:

<h1>Hello, World!</h1>

You’ve now created a function to handle your views and templates to display to the user. The final step is to hook up your URLs so that you can visit the page you’ve just created. Your project has a module called urls.py in which you need to include a URL configuration for the hello_world app. Inside personal_portfolio/urls.py, add the following:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('hello_world.urls')),
]

This looks for a module called urls.py inside the hello_world application and registers any URLs defined there. Whenever you visit the root path of your URL (localhost:8000), the hello_world application’s URLs will be registered. The hello_world.urls module doesn’t exist yet, so you’ll need to create it:

$ touch hello_world/urls.py

Inside this module, we need to import the path object as well as our app’s views module. Then we want to create a list of URL patterns that correspond to the various view functions. At the moment, we have only created one view function, so we need only create one URL:

from django.urls import path
from hello_world import views

urlpatterns = [
    path('', views.hello_world, name='hello_world'),
]

Now, when you restart the server and visit localhost:8000, you should be able to see the HTML template you created:

Congratulations, again! You’ve created your first Django app and hooked it up to your project. Don’t forget to check out the source code for this section and the previous one. The only problem now is that it doesn’t look very nice. In the next section, we’re going to add bootstrap styles to your project to make it prettier!

Add Bootstrap to Your App

If you don’t add any styling, then the app you create isn’t going to look too nice. Instead of going into CSS styling with this tutorial, we’ll just cover how to add bootstrap styles to your project. This will allow us to improve the look of the site without too much effort.

Before we get started with the Bootstrap styles, we’ll create a base template that we can import to each subsequent view. This template is where we’ll subsequently add the Bootstrap style imports.

Create another directory called templates, this time inside personal_portfolio, and a file called base.html, inside the new directory:

$ mkdir personal_portfolio/templates/
$ touch personal_portfolio/templates/base.html

We create this additional templates directory to store HTML templates that will be used in every Django app in the project. As you saw previously, each Django project can consist of multiple apps that handle separated logic, and each app contains its own templates directory to store HTML templates related to the application.

This application structure works well for the back end logic, but we want our entire site to look consistent on the front end. Instead of having to import Bootstrap styles into every app, we can create a template or set of templates that are shared by all the apps. As long as Django knows to look for templates in this new, shared directory it can save a lot of repeated styles.

Inside this new file (personal_portfolio/templates/base.html), add the following lines of code:

{% block page_content %}{% endblock %}

Now, in hello_world/templates/hello_world.html, we can extend this base template:

{% extends "base.html" %}

{% block page_content %}
<h1>Hello, World!</h1>
{% endblock %}

What happens here is that any HTML inside the page_content block gets added inside the same block in base.html.

To install Bootstrap in your app, you’ll use the Bootstrap CDN. This is a really simple way to install Bootstrap that just involves adding a few lines of code to base.html. Check out the source code to see how to add the CDN links to your project.

All future templates that we create will extend base.html so that we can include Bootstrap styling on every page without having to import the styles again.

Before we can see our new styled application, we need to tell our Django project that base.html exists. The default settings register template directories in each app, but not in the project directory itself. In personal_portfolio/settings.py, update TEMPLATES:

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": ["personal_portfolio/templates/"],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
            ]
        },
    }
]

Now, when you visit localhost:8000, you should see that the page has been formatted with slightly different styling:

Whenever you want create templates or import scripts that you intend to use in all your Django apps inside a project, you can add them to this project-level directory and extend them inside your app templates.

Adding templates is the last stage to building your Hello, World! Django site. You learned how the Django templating engine works and how to create project-level templates that can be shared by all the apps inside your Django project.

In this section, you learned how to create a simple Hello, World! Django site by creating a project with a single app. In the next section, you’ll create another application to showcase web development projects, and you’ll learn all about models in Django!

The source code for this section can be found on GitHub.

Showcase Your Projects

Any web developer looking to create a portfolio needs a way to show off projects they have worked on. That’s what you’ll be building now. You’ll create another Django app called projects that will hold a series of sample projects that will be displayed to the user. Users can click on projects and see more information about your work.

Before we build the projects app, let’s first delete the hello_world application. All you need to do is delete the hello_world directory and remove the line "hello_world", from INSTALLED_APPS in settings.py:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'hello_world',  # Delete this line
]

Finally, you need to remove the URL path created in personal_portfolio/urls.py:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('hello_world.urls')),  # Delete this line
]

Now that you’ve removed the hello_world app, we can create the projects app. Making sure you’re in the rp-portfolio directory, run the following command in your console:

$ python manage.py startapp projects

This will create a directory named projects. The files created are the same as those created when we set up the hello_world application. In order to hook up our app, we need to add it into INSTALLED_APPS in settings.py:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'projects',
]

Check out the source code for this section on GitHub. We’re not going to worry about URLs for this application just yet. Instead, we’re going to focus on building a Project model.

Projects App: Models

If you want to store data to display on a website, then you’ll need a database. Typically, if you want to create a database with tables and columns within those tables, you’ll need to use SQL to manage the database. But when you use Django, you don’t need to learn a new language because it has a built-in Object Relational Mapper (ORM).

An ORM is a program that allows you to create classes that correspond to database tables. Class attributes correspond to columns, and instances of the classes correspond to rows in the database. So, instead of learning a whole new language to create our database and its tables, we can just write some Python classes.

When you’re using an ORM, the classes you build that represent database tables are referred to as models. In Django, they live in the models.py module of each Django app.

In your projects app, you’ll only need one table to store the different projects you’ll display to the user. That means you’ll only need to create one model in models.py.

The model you’ll create will be called Project and will have the following fields:

title will be a short string field to hold the name of your project.
description will be a larger string field to hold a longer piece of text.
technology will be a string field, but its contents will be limited to a select number of choices.
image will be an image field that holds the file path where the image is stored.

To create this model, we’ll create a new class in models.py and add the following in our fields:

from django.db import models

class Project(models.Model):
    title = models.CharField(max_length=100)
    description = models.TextField()
    technology = models.CharField(max_length=20)
    image = models.FilePathField(path="/img")

Django models come with many built-in model field types. We’ve only used three in this model. CharField is used for short strings and specifies a maximum length.

TextField is similar to CharField but can be used for longer form text as it doesn’t have a maximum length limit. Finally, FilePathField also holds a string but must point to a file path name.

Now that we’ve created our Project class, we need Django to create the database. By default, the Django ORM creates databases in SQLite, but you can use other databases that use the SQL language, such as PostgreSQL or MySQL, with the Django ORM.

To start the process of creating our database, we need to create a migration. A migration is a file containing a Migration class with rules that tell Django what changes need to be made to the database. To create the migration, type the following command in the console, making sure you’re in the rp-portfolio directory:

$ python manage.py makemigrations projects
Migrations for 'projects':
  projects/migrations/0001_initial.py
    - Create model Project

You should see that a file projects/migrations/0001_initial.py has been created in the projects app. Check out that file in the source code to make sure your migration is correct.

Now that you’ve create a migration file, you need to apply the migrations set out in the migrations file and create your database using the migrate command:

$ python manage.py migrate projects
Operations to perform:
  Apply all migrations: projects
Running migrations:
  Applying projects.0001_initial... OK

Note: When running both the makemigrations and migrate commands, we added projects to our command. This tells Django to only look at models and migrations in the projects app. Django comes with several models already created.

If you run makemigrations and migrate without the projects flag, then all migrations for all the default models in your Django projects will be created and applied. This is not a problem, but for the purposes of this section, they are not needed.

You should also see that a file called db.sqlite3 has been created in the root of your project. Now your database is set up and ready to go. You can now create rows in your table that are the various projects you want to show on your portfolio site.

To create instances of our Project class, we’re going to have to use the Django shell. The Django shell is similar to the Python shell but allows you to access the database and create entries. To access the Django shell, we use another Django management command:

$ python manage.py shell

Once you’ve accessed the shell, you’ll notice that the command prompt will change from $ to >>>. You can then import your models:

>>>

>>> from projects.models import Project

We’re first going to create a new project with the following attributes:

name: My First Project
description: A web development project.
technology: Django
image: img/project1.png

To do this, we create an instance of the Project class in the Django shell:

>>>

>>> p1 = Project(
...     title='My First Project',
...     description='A web development project.',
...     technology='Django',
...     image='img/project1.png'
... )
>>> p1.save()

This creates a new entry in your projects table and saves it to the database. Now you have created a project that you can display on your portfolio site.

The final step in this section is to create two more sample projects:

>>>

>>> p2 = Project(
...     title='My Second Project',
...     description='Another web development project.',
...     technology='Flask',
...     image='img/project2.png'
... )
>>> p2.save()
>>> p3 = Project(
...     title='My Third Project',
...     description='A final development project.',
...     technology='Django',
...     image='img/project3.png'
... )
>>> p3.save()

Well done for reaching the end of this section! You now know how to create models in Django and build migration files so that you can translate these model classes into database tables. You’ve also used the Django shell to create three instances of your model class.

In the next section, we’ll take these three projects you created and create a view function to display them to users on a web page. You can find the source code for this section of the tutorial on GitHub.

Projects App: Views

Now you’ve created the projects to display on your portfolio site, you’ll need to create view functions to send the data from the database to the HTML templates.

In the projects app, you’ll create two different views:

An index view that shows a snippet of information about each project
A detail view that shows more information on a particular topic

Let’s start with the index view, as the logic is slightly simpler. Inside views.py, you’ll need to import the Project class from models.py and create a function project_index() that renders a template called project_index.html. In the body of this function, you’ll make a Django ORM query to select all objects in the Project table:

 1 from django.shortcuts import render
 2 from projects.models import Project
 3 
 4 def project_index(request):
 5     projects = Project.objects.all()
 6     context = {
 7         'projects': projects
 8     }
 9     return render(request, 'project_index.html', context)

There’s quite a lot going on in this code block, so let’s break it down.

In line 5, you perform a query. A query is simply a command that allows you to create, retrieve, update, or delete objects (or rows) in your database. In this case, you’re retrieving all objects in the projects table.

A database query returns a collection of all objects that match the query, known as a Queryset. In this case, you want all objects in the table, so it will return a collection of all projects.

In line 6 of the code block above, we define a dictionary context. The dictionary only has one entry projects to which we assign our Queryset containing all projects. The context dictionary is used to send information to our template. Every view function you create needs to have a context dictionary.

In line 9, context is added as an argument to render(). Any entries in the context dictionary are available in the template, as long as the context argument is passed to render(). You’ll need to create a context dictionary and pass it to render in each view function you create.

We also render a template named project_index.html, which doesn’t exist yet. Don’t worry about that for now. You’ll create the templates for these views in the next section.

Next, you’ll need to create the project_detail() view function. This function will need an additional argument: the id of the project that’s being viewed.

Otherwise, the logic is similar:

13 def project_detail(request, pk):
14     project = Project.objects.get(pk=pk)
15     context = {
16         'project': project
17     }
18     return render(request, 'project_detail.html', context)

In line 14, we perform another query. This query retrieves the project with primary key, pk, equal to that in the function argument. We then assign that project in our context dictionary, which we pass to render(). Again, there’s a template project_detail.html, which we have yet to create.

Once your view functions are created, we need to hook them up to URLs. We’ll start by creating a file projects/urls.py to hold the URL configuration for the app. This file should contain the following code:

 1 from django.urls import path
 2 from . import views
 3 
 4 urlpatterns = [
 5     path("", views.project_index, name="project_index"),
 6     path("<int:pk>/", views.project_detail, name="project_detail"),
 7 ]

In line 5, we hook up the root URL of our app to the project_index view. It is slightly more complicated to hook up the project_detail view. To do this, we want the URL to be /1, or /2, and so on, depending on the pk of the project.

The pk value in the URL is the same pk passed to the view function, so you need to dynamically generate these URLs depending on which project you want to view. To do this, we used the <int:pk> notation. This just tells Django that the value passed in the URL is an integer, and its variable name is pk.

With those now set up, we need to hook these URLs up to the project URLs. In personal_portfolio/urls.py, add the following highlighted line of code:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path("admin/", admin.site.urls),
    path("projects/", include("projects.urls")),
]

This line of code includes all the URLs in the projects app but means they are accessed when prefixed by projects/. There are now two full URLs that can be accessed with our project:

localhost:8000/projects: The project index page
localhost:8000/projects/3: The detail view for the project with pk=3

These URLs still won’t work properly because we don’t have any HTML templates. But our views and logic are up and running so all that’s left to do is create those templates. If you want to check your code, take a look at the source code for this section.

Projects App: Templates

Phew! You’re nearly there with this app. Our final step is to create two templates:

The project_index template
The project_detail template

As we’ve added Bootstrap styles to our application, we can use some pre-styled components to make the views look nice. Let’s start with the project_index template.

For the project_index template, you’ll create a grid of Bootstrap cards, with each card displaying details of the project. Of course, we don’t know how many projects there are going to be. In theory, there could be hundreds to display.

We don’t want to have to create 100 different Bootstrap cards and hard-code in all the information to each project. Instead, we’re going to use a feature of the Django template engine: for loops.

Using this feature, you’ll be able to loop through all the projects and create a card for each one. The for loop syntax in the Django template engine is as follows:

{% for project in projects %}
{# Do something... #}
{% endfor %}

Now that you know how for loops work, you can add the following code to a file named projects/templates/project_index.html:

 1 {% extends "base.html" %}
 2 {% load static %}
 3 {% block page_content %}
 4 <h1>Projects</h1>
 5 <div class="row">
 6 {% for project in projects %}
 7     <div class="col-md-4">
 8         <div class="card mb-2">
 9             <img class="card-img-top" src="{% static project.image %}">
10             <div class="card-body">
11                 <h5 class="card-title">{{ project.title }}</h5>
12                 <p class="card-text">{{ project.description }}</p>
13                 <a href="{% url 'project_detail' project.pk %}"
14                    class="btn btn-primary">
15                     Read More
16                 </a>
17             </div>
18         </div>
19     </div>
20     {% endfor %}
21 </div>
22 {% endblock %}

There’s a lot of Bootstrap HTML here, which is not the focus of this tutorial. Feel free to copy and paste and take a look at the Bootstrap docs if you’re interested in learning more. Instead of focusing on the Bootstrap, there are a few things to highlight in this code block.

In line 1, we extend base.html as we did in the Hello, World! app tutorial. I’ve added some more styling to this file to include a navigation bar and so that all the content is contained in a Bootstrap container. The changes to base.html can be seen in the source code on GitHub.

On line 2, we include a {% load static %} tag to include static files such as images. Remember back in the section on Django models, when you created the Project model. One of its attributes was a filepath. That filepath is where we’re going to store the actual images for each project.

Django automatically registers static files stored in a directory named static/ in each application. Our image file path names were of the structure: img/<photo_name>.png.

When loading static files, Django looks in the static/ directory for files matching a given filepath within static/. So, we need to create a directory named static/ with another directory named img/ inside. Inside img/, you can copy over the images from the source code on GitHub.

On line 6, we begin the for loop, looping over all projects passed in by the context dictionary.

Inside this for loop, we can access each individual project. To access the project’s attributes, you can use dot notation inside double curly brackets. For example, to access the project’s title, you use {{ project.title }}. The same notation can be used to access any of the project’s attributes.

On line 9, we include our project image. Inside the src attribute, we add the code {% static project.image %}. This tells Django to look inside the static files to find a file matching project.image.

The final point that we need to highlight is the link on line 13. This is the link to our project_detail page. Accessing URLs in Django is similar to accessing static files. The code for the URL has the following form:

{% url '<url path name>' <view_function_arguments> %}

In this case, we are accessing a URL path named project_detail, which takes integer arguments corresponding to the pk number of the project.

With all that in place, if you start the Django server and visit localhost:8000/projects, then you should see something like this:

With the project_index.html template in place, it’s time to create the project_detail.html template. The code for this template is below:

{% extends "base.html" %}
{% load static %}

{% block page_content %}
<h1>{{ project.title }}</h1>
<div class="row">
    <div class="col-md-8">
        <img src="{% static project.image %}" alt="" width="100%">
    </div>
    <div class="col-md-4">
        <h5>About the project:</h5>
        <p>{{ project.description }}</p>
        <br>
        <h5>Technology used:</h5>
        <p>{{ project.technology }}</p>
    </div>
</div>
{% endblock %}

The code in this template has the same functionality as each project card in the project_index.html template. The only difference is the introduction of some Bootstrap columns.

If you visit localhost:8000/projects/1, you should see the detail page for that first project you created:

In this section, you learned how to use models, views, and templates to create a fully functioning app for your personal portfolio project. Check out the source code for this section on GitHub.

In the next section, you’ll build a fully functioning blog for your site, and you’ll also learn about the Django admin page and forms.

A blog is a great addition to any personal portfolio site. Whether you update it monthly or weekly, it’s a great place to share your knowledge as you learn. In this section, you’re going to build a fully functioning blog that will allow you to perform the following tasks:

Create, update, and delete blog posts
Display posts to the user as either an index view or a detail view
Assign categories to posts
Allow users to comment on posts

You’ll also learn how to use the Django Admin interface, which is where you’ll create, update, and delete posts and categories as necessary.

Before you get into building out the functionality of this part of your site, create a new Django app named blog. Don’t delete projects. You’ll want both apps in your Django project:

$ python manage.py startapp blog

This may start to feel familiar to you, as its your third time doing this. Don’t forget to add blog to your INSTALLED_APPS in personal_porfolio/settings.py:

INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "projects",
    "blog",
]

Hold off on hooking up the URLs for now. As with the projects app, you’ll start by adding your models.

Blog App: Models

The models.py file in this app is much more complicated than in the projects app.

You’re going to need three separate database tables for the blog:

Post
Category
Comment

These tables need to be related to one another. This is made easier because Django models come with fields specifically for this purpose.

Below is the code for the Category and Post models:

 1 from django.db import models
 2 
 3 class Category(models.Model):
 4     name = models.CharField(max_length=20)
 5 
 6 class Post(models.Model):
 7     title = models.CharField(max_length=255)
 8     body = models.TextField()
 9     created_on = models.DateTimeField(auto_now_add=True)
10     last_modified = models.DateTimeField(auto_now=True)
11     categories = models.ManyToManyField('Category', related_name='posts')

The Category model is very simple. All that’s needed is a single CharField in which we store the name of the category.

The title and body fields on the Post model are the same field types as you used in the Project model. We only need a CharField for the title as we only want a short string for the post title. The body needs to be a long-form piece of text, so we use a TextField.

The next two fields, created_on and last_modified, are Django DateTimeFields. These store a datetime object containing the date and time when the post was created and modified respectively.

On line 9, the DateTimeField takes an argument auto_now_add=True. This assigns the current date and time to this field whenever an instance of this class is created.

On line 10, the DateTimeField takes an argument auto_now=True. This assigns the current date and time to this field whenever an instance of this class is saved. That means whenever you edit an instance of this class, the date_modified is updated.

The final field on the post model is the most interesting. We want to link our models for categories and posts in such a way that many categories can be assigned to many posts. Luckily, Django makes this easier for us by providing a ManytoManyField field type. This field links the Post and Category models and allows us to create a relationship between the two tables.

The ManyToManyField takes two arguments. The first is the model with which the relationship is, in this case its Category. The second allows us to access the relationship from a Category object, even though we haven’t added a field there. By adding a related_name of posts, we can access category.posts to give us a list of posts with that category.

The third and final model we need to add is Comment. We’ll use another relationship field similar the ManyToManyField that relates Post and Category. However, we only want the relationship to go one way: one post should have many comments.

You’ll see how this works after we define the Comment class:

16 class Comment(models.Model):
17     author = models.CharField(max_length=60)
18     body = models.TextField()
19     created_on = models.DateTimeField(auto_now_add=True)
20     post = models.ForeignKey('Post', on_delete=models.CASCADE)

The first three fields on this model should look familiar. There’s an author field for users to add a name or alias, a body field for the body of the comment, and a created_on field that is identical to the created_on field on the Post model.

On line 20, we use another relational field, the ForeignKey field. This is similar to the ManyToManyField but instead defines a many to one relationship. The reasoning behind this is that many comments can be assigned to one post. But you can’t have a comment that corresponds to many posts.

The ForeignKey field takes two arguments. The first is the other model in the relationship, in this case, Post. The second tells Django what to do when a post is deleted. If a post is deleted, then we don’t want the comments related to it hanging around. We, therefore, want to delete them as well, so we add the argument on_delete=models.CASCADE.

Once you’ve created the models, you can create the migration files with makemigrations:

$ python manage.py makemigrations blog

The final step is to migrate the tables. This time, don’t add the app-specific flag. Later on, you’ll need the User model that Django creates for you:

$ python manage.py migrate

Now that you’ve created the models, we can start to add some posts and categories. You won’t be doing this from the command line as you did with the projects, as typing out a whole blog post into the command line would be unpleasant to say the least!

Instead, you’ll learn how to use the Django Admin, which will allow you to create instances of your model classes in a nice web interface.

Don’t forget that you can check out the source code for this section on GitHub before moving onto the next section.

Blog App: Django Admin

The Django Admin is a fantastic tool and one of the great benefits of using Django. As you’re the only person who’s going to be writing blog posts and creating categories, there’s no need to create a user interface to do so.

On the other hand, you don’t want to have to write blog posts in the command line. This is where the admin comes in. It allows you to create, update, and delete instances of your model classes and provides a nice interface for doing so.

Before you can access the admin, you need to add yourself as a superuser. This is why, in the previous section, you applied migrations project-wide as opposed to just for the app. Django comes with built-in user models and a user management system that will allow you to login to the admin.

To start off, you can add yourself as superuser using the following command:

$ python manage.py createsuperuser

You’ll then be prompted to enter a username followed by your email address and password. Once you’ve entered the required details, you’ll be notified that the superuser has been created. Don’t worry if you make a mistake since you can just start again:

Username (leave blank to use 'jasmine'): jfiner
Email address: jfiner@example.com
Password:
Password (again):
Superuser created successfully.

Navigate to localhost:8000/admin and log in with the credentials you just used to create a superuser. You’ll see a page similar to the one below:

The User and Groups models should appear, but you’ll notice that there’s no reference to the models you’ve created yourself. That’s because you need to register them inside the admin.

In the blog directory, open the file admin.py and type the following lines of code:

 1 from django.contrib import admin
 2 from blog.models import Post, Category
 3 
 4 class PostAdmin(admin.ModelAdmin):
 5     pass
 6 
 7 class CategoryAdmin(admin.ModelAdmin):
 8     pass
 9 
10 admin.site.register(Post, PostAdmin)
11 admin.site.register(Category, CategoryAdmin)

On line 2, you import the models you want to register on the admin page.

Note: We’re not adding the comments to the admin. That’s because it’s not usually necessary to edit or create comments yourself.

If you wanted to add a feature where comments are moderated, then go ahead and add the Comments model too. The steps to do so are exactly the same!

On line 4 and line 7, you define empty classes PostAdmin and CategoryAdmin. For the purposes of this tutorial, you don’t need to add any attributes or methods to these classes. They are used to customize what is shown on the admin pages. For this tutorial, the default configuration is enough.

The last two lines are the most important. These register the models with the admin classes. If you now visit localhost:8000/admin, then you should see that the Post and Category models are now visible:

If you click into Posts or Categorys, you should be able to add new instances of both models. I like to add the text of fake blog posts by using lorem ipsum dummy text.

Create a couple of fake posts and assign them fake categories before moving onto the next section. That way, you’ll have posts you can view when we create our templates.

Don’t forget to check out the source code for this section before moving on to building out the views for our app.

Blog App: Views

You’ll need to create three view functions in the views.py file in the blog directory:

blog_index will display a list of all your posts.
blog_detail will display the full post as well as comments and a form to allow users to create new comments.
blog_category will be similar to blog_index, but the posts viewed will only be of a specific category chosen by the user.

The simplest view function to start with is blog_index(). This will be very similar to the project_index() view from your project app. You’ll just query the Post models and retrieve all its objects:

 1 from django.shortcuts import render
 2 from blog.models import Post
 3 
 4 def blog_index(request):
 5     posts = Post.objects.all().order_by('-created_on')
 6     context = {
 7         "posts": posts,
 8     }
 9     return render(request, "blog_index.html", context)

On line 2, you import the Post model, and on line 5 inside the view function, you obtain a Queryset containing all the posts in the database. order_by() orders the Queryset according to the argument given. The minus sign tells Django to start with the largest value rather than the smallest. We use this, as we want the posts to be ordered with the most recent post first.

Finally, you define the context dictionary and render the template. Don’t worry about creating it yet. You’ll get to creating those in the next section.

Next, you can start to create the blog_category() view. The view function will need to take a category name as an argument and query the Post database for all posts that have been assigned the given category:

13 def blog_category(request, category):
14     posts = Post.objects.filter(
15         categories__name__contains=category
16     ).order_by(
17         '-created_on'
18     )
19     context = {
20         "category": category,
21         "posts": posts
22     }
23     return render(request, "blog_category.html", context)

On line 14, you’ve used a Django Queryset filter. The argument of the filter tells Django what conditions need to be met for an object to be retrieved. In this case, we only want posts whose categories contain the category with the name corresponding to that given in the argument of the view function. Again, you’re using order_by() to order posts starting with the most recent.

We then add these posts and the category to the context dictionary, and render our template.

The last view function to add is blog_detail(). This is more complicated as we are going to include a form. Before you add the form, just set up the view function to show a specific post with a comment associated with it. This function will be almost equivalent to the project_detail() view function in the projects app:

21 def blog_detail(request, pk):
22     post = Post.objects.get(pk=pk)
23     comments = Comment.objects.filter(post=post)
24     context = {
25         "post": post,
26         "comments": comments,
27     }
28 
29     return render(request, "blog_detail.html", context)

The view function takes a pk value as an argument and, on line 22, retrieves the object with the given pk.

On line 23, we retrieve all the comments assigned to the given post using Django filters again.

Lastly, add both post and comments to the context dictionary and render the template.

To add a form to the page, you’ll need to create another file in the blog directory named forms.py. Django forms are very similar to models. A form consists of a class where the class attributes are form fields. Django comes with some built-in form fields that you can use to quickly create the form you need.

For this form, the only fields you’ll need are author, which should be a CharField, and body, which can also be a CharField.

Note: If the CharField of your form corresponds to a model CharField, make sure both have the same max_length value.

blog/forms.py should contain the following code:

from django import forms

class CommentForm(forms.Form):
    author = forms.CharField(
        max_length=60,
        widget=forms.TextInput(attrs={
            "class": "form-control",
            "placeholder": "Your Name"
        })
    )
    body = forms.CharField(widget=forms.Textarea(
        attrs={
            "class": "form-control",
            "placeholder": "Leave a comment!"
        })
    )

You’ll also notice an argument widget has been passed to both the fields. The author field has the forms.TextInput widget. This tells Django to load this field as an HTML text input element in the templates. The body field uses a forms.TextArea widget instead, so that the field is rendered as an HTML text area element.

These widgets also take an argument attrs, which is a dictionary and allows us to specify some CSS classes, which will help with formatting the template for this view later. It also allows us to add some placeholder text.

When a form is posted, a POST request is sent to the server. So, in the view function, we need to check if a POST request has been received. We can then create a comment from the form fields. Django comes with a handy is_valid() on its forms, so we can check that all the fields have been entered correctly.

Once you’ve created the comment from the form, you’ll need to save it using save() and then query the database for all the comments assigned to the given post. Your view function should contain the following code:

21 def blog_detail(request, pk):
22     post = Post.objects.get(pk=pk)
23 
24     form = CommentForm()
25     if request.method == 'POST':
26         form = CommentForm(request.POST)
27         if form.is_valid():
28             comment = Comment(
29                 author=form.cleaned_data["author"],
30                 body=form.cleaned_data["body"],
31                 post=post
32             )
33             comment.save()
34 
35     comments = Comment.objects.filter(post=post)
36     context = {
37         "post": post,
38         "comments": comments,
39         "form": form,
40     }
41     return render(request, "blog_detail.html", context)

On line 24, we create an instance of our form class. Don’t forget to import your form at the beginning of the file:

from . import CommentForm

We then go on to check if a POST request has been received. If it has, then we create a new instance of our form, populated with the data entered into the form.

The form is then validated using is_valid(). If the form is valid, a new instance of Comment is created. You can access the data from the form using form.cleaned_data, which is a dictionary.

They keys of the dictionary correspond to the form fields, so you can access the author using form.cleaned_data['author']. Don’t forget to add the current post to the comment when you create it.

Note: The life cycle of submitting a form can be a little complicated, so here’s an outline of how it works:

When a user visits a page containing a form, they send a GET request to the server. In this case, there’s no data entered in the form, so we just want to render the form and display it.
When a user enters information and clicks the Submit button, a POST request, containing the data submitted with the form, is sent to the server. At this point, the data must be processed, and two things can happen:
- The form is valid, and the user is redirected to the next page.
- The form is invalid, and empty form is once again displayed. The user is back at step 1, and the process repeats.

The Django forms module will output some errors, which you can display to the user. This is beyond the scope of this tutorial, but you can read more about rendering form error messages in the Django documentation.

On line 33, save the comment and go on to add the form to the context dictionary so you can access the form in the HTML template.

The final step before you get to create the templates and actually see this blog up and running is to hook up the URLs. You’ll need create another urls.py file inside blog/ and add the URLs for the three views:

from django.urls import path
from . import views

urlpatterns = [
    path("", views.blog_index, name="blog_index"),
    path("<int:pk>/", views.blog_detail, name="blog_detail"),
    path("<category>/", views.blog_category, name="blog_category"),
]

Once the blog-specific URLs are in place, you need to add them to the projects URL configuration using include():

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path("admin/", admin.site.urls),
    path("projects/", include("projects.urls")),
    path("blog/", include("blog.urls")),
]

With this set up, all the blog URLs will be prefixed with blog/, and you’ll have the following URL paths:

localhost:8000/blog: Blog index
localhost:8000/blog/1: Blog detail view of blog with pk=1
localhost:8000/blog/python: Blog index view of all posts with category python

These URLs won’t work just yet as you still need to create the templates.

In this section, you created all the views for your blog application. You learned how to use filters when making queries and how to create Django forms. It won’t be long now until you can see your blog app in action!

As always, don’t forget that you can check out the source code for this section on GitHub.

Blog App: Templates

The final piece of our blog app is the templates. By the end of this section, you’ll have created a fully functioning blog.

You’ll notice there are some bootstrap elements included in the templates to make the interface prettier. These aren’t the focus of the tutorial so I’ve glossed over what they do but do check out the Bootstrap docs to find out more.

The first template you’ll create is for the blog index in a new file blog/templates/blog_index.html. This will be very similar to the projects index view.

You’ll use a for loop to loop over all the posts. For each post, you’ll display the title and a snippet of the body. As always, you’ll extend the base template personal_porfolio/templates/base.html, which contains our navigation bar and some extra formatting:

 1 {% extends "base.html" %}
 2 {% block page_content %}
 3 <div class="col-md-8 offset-md-2">
 4     <h1>Blog Index</h1>
 5     <hr>
 6     {% for post in posts %}
 7     <h2><a href="{% url 'blog_detail' post.pk%}">{{ post.title }}</a></h2>
 8     <small>
 9         {{ post.created_on.date }} |&nbsp;
10         Categories:&nbsp;
11         {% for category in post.categories.all %}
12         <a href="{% url 'blog_category' category.name %}">
13             {{ category.name }}
14         </a>&nbsp;
15         {% endfor %}
16     </small>
17     <p>{{ post.body | slice:":400" }}...</p>
18     {% endfor %}
19 </div>
20 {% endblock %}

On line 7, we have the post title, which is a hyperlink. The link is a Django link where we are pointing to the URL named blog_detail, which takes an integer as its argument and should correspond to the pk value of the post.

Underneath the title, we’ll display the created_on attribute of the post as well as its categories. On line 11, we use another for loop to loop over all the categories assigned to the post.

On line 17, we use a template filter slice to cut off the post body at 400 characters so that the blog index is more readable.

Once that’s in place, you should be able to access this page by visiting localhost:8000/blog:

Next, create another HTML file blog/templates/blog_category.html where your blog_category template will live. This should be identical to blog_index.html, except with the category name inside the h1 tag instead of Blog Index:

 1 {% extends "base.html" %}
 2 {% block page_content %}
 3 <div class="col-md-8 offset-md-2">
 4     <h1>{{ category | title }}</h1>
 5     <hr>
 6     {% for post in posts %}
 7         <h2><a href="{% url 'blog_detail' post.pk%}">{{ post.title }}</a></h2>
 8         <small>
 9             {{ post.created_on.date }} |&nbsp;
10             Categories:&nbsp;
11             {% for category in post.categories.all %}
12             <a href="{% url 'blog_category' category.name %}">
13                 {{ category.name }}
14             </a>&nbsp;
15             {% endfor %}
16         </small>
17         <p>{{ post.body | slice:":400" }}...</p>
18     {% endfor %}
19 </div>
20 {% endblock %}

Most of this template is identical to the previous template. The only difference is on line 4, where we use another Django template filter title. This applies titlecase to the string and makes words start with an uppercase character.

With that template finished, you’ll be able to access your category view. If you defined a category named python, you should be able to visit localhost:8000/blog/python and see all the posts with that category:

The last template to create is the post_detail template. In this template, you’ll display the title and full body of a post.

Between the title and the body of the post, you’ll display the date the post was created and any categories. Underneath that, you’ll include a comments form so users can add a new comment. Under this, there will be a list of comments that have already been left:

 1 {% extends "base.html" %}
 2 {% block page_content %}
 3 <div class="col-md-8 offset-md-2">
 4     <h1>{{ post.title }}</h1>
 5     <small>
 6         {{ post.created_on.date }} |&nbsp;
 7         Categories:&nbsp;
 8         {% for category in post.categories.all %}
 9         <a href="{% url 'blog_category' category.name %}">
10             {{ category.name }}
11         </a>&nbsp;
12         {% endfor %}
13     </small>
14     <p>{{ post.body | linebreaks }}</p>
15     <h3>Leave a comment:</h3>
16     <form action="/blog/{{ post.pk }}/" method="post">
17         {% csrf_token %}
18         <div class="form-group">
19             {{ form.author }}
20         </div>
21         <div class="form-group">
22             {{ form.body }}
23         </div>
24         <button type="submit" class="btn btn-primary">Submit</button>
25     </form>
26     <h3>Comments:</h3>
27     {% for comment in comments %}
28     <p>
29         On {{comment.created_on.date }}&nbsp;
30         <b>{{ comment.author }}</b> wrote:
31     </p>
32     <p>{{ comment.body }}</p>
33     <hr>
34     {% endfor %}
35 </div>
36 {% endblock %}

The first few lines of the template in which we display the post title, date, and categories is the same logic as for the previous templates. This time, when rendering the post body, use a linebreaks template filter. This tag registers line breaks as new paragraphs, so the body doesn’t appear as one long block of text.

Underneath the post, on line 16, you’ll display your form. The form action points to the URL path of the page to which you’re sending the POST request to. In this case, it’s the same as the page that is currently being visited. You then add a csrf_token, which provides security and renders the body and author fields of the form, followed by a submit button.

To get the bootstrap styling on the author and body fields, you need to add the form-control class to the text inputs.

Because Django renders the inputs for you when you include {{ form.body }} and {{ form.author }}, you can’t add these classes in the template. That’s why you added the attributes to the form widgets in the previous section.

Underneath the form, there’s another for loop that loops over all the comments on the given post. The comments, body, author, and created_on attributes are all displayed.

Once that template is in place, you should be able to visit localhost:8000/blog/1 and view your first post:

You should also be able to access the post detail pages by clicking on their title in the blog_index view.

The final finishing touch is to add a link to the blog_index to the navigation bar in base.html. This way, when you click on Blog in the navigation bar, you’ll be able to visit the blog. Check out the updates to base.html in the source code to see how to add that link.

With that now in place, your personal portfolio site is complete, and you’ve created your first Django site. The final version of the source code containing all the features can be found on GitHub, so check it out! Click around the site a bit to see all the functionality and try leaving some comments on your posts!

You may find a few things here and there that you think need polishing. Go ahead and tidy them up. The best way to learn more about this web framework is through practice, so try to extend this project and make it even better! If you’re not sure where to start, I’ve left a few ideas for you in the conclusion below!

Conclusion

Congratulations, you’ve reached the end of the tutorial! We’ve covered a lot, so make sure to keep practicing and building. The more you build the easier it will become and the less you’ll have to refer back to this article or the documentation. You’ll be building sophisticated web applications in no time.

In this tutorial you’ve seen:

How to create Django projects and apps
How to add web pages with views and templates
How to get user input with forms
How to hook your views and templates up with URL configurations
How to add data to your site using relational databases with Django’s Object Relational Mapper
How to use the Django Admin to manage your models

In addition, you’ve learned about the MVT structure of Django web applications and why Django is such a good choice for web development.

If you want to learn more about Django, do check out the documentation and make sure to stay tuned for Part 2 of this series!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Working With JSON Data in Python

2019-03-28T14:00:00+00:00

JSON is a lightweight data-interchange format. It allows us to represent the objects in our Python programs as human-readable text that can be sent over the internet. Lots of APIs and databases use JSON for communication.

You’ll learn how to work with Python’s built-in json module to serialize the data in your programs into JSON format. Then, you’ll deserialize some JSON from an online API and convert it into Python objects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Stand Out in a Python Coding Interview

2019-03-27T14:00:00+00:00

You’ve made it past the phone call with the recruiter, and now it’s time to show that you know how to solve problems with actual code. Whether it’s a HackerRank exercise, a take-home assignment, or an onsite whiteboard interview, this is your moment to prove your coding interview skills.

But interviews aren’t just about solving problems: they’re also about showing that you can write clean production code. This means that you have a deep knowledge of Python’s built-in functionality and libraries. This knowledge shows companies that you can move quickly and won’t duplicate functionality that comes with the language just because you don’t know it exists.

At Real Python, we’ve put our heads together and discussed what tools we’re always impressed to see in coding interviews. This article will walk you through the best of that functionality, starting with Python built-ins, then Python’s native support for data structures, and finally Python’s powerful (and often underappreciated) standard library.

In this article, you’ll learn how to:

Use enumerate() to iterate over both indices and values
Debug problematic code with breakpoint()
Format strings effectively with f-strings
Sort lists with custom arguments
Use generators instead of list comprehensions to conserve memory
Define default values when looking up dictionary keys
Count hashable objects with the collections.Counter class
Use the standard library to get lists of permutations and combinations

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Select the Right Built-In Function for the Job

Python has a large standard library but only a small library of built-in functions, which are always available and don’t need to be imported. It’s worth going through each one, but until you get the chance to do so, here are a few built-in functions worth understanding how to use, and in the case of some of them, what alternatives to use instead.

Iterate With `enumerate()` Instead of `range()`

This scenario might come up more than any other in coding interviews: you have a list of elements, and you need to iterate over the list with access to both the indices and the values.

There’s a classic coding interview question named FizzBuzz that can be solved by iterating over both indices and values. In FizzBuzz, you are given a list of integers. Your task is to do the following:

Replace all integers that are evenly divisible by 3 with "fizz"
Replace all integers divisible by 5 with "buzz"
Replace all integers divisible by both 3 and 5 with "fizzbuzz"

Often, developers will solve this problem with range():

>>>

>>> numbers = [45, 22, 14, 65, 97, 72]
>>> for i in range(len(numbers)):
...     if numbers[i] % 3 == 0 and numbers[i] % 5 == 0:
...         numbers[i] = 'fizzbuzz'
...     elif numbers[i] % 3 == 0:
...         numbers[i] = 'fizz'
...     elif numbers[i] % 5 == 0:
...         numbers[i] = 'buzz'
...
>>> numbers
['fizzbuzz', 22, 14, 'buzz', 97, 'fizz']

Range allows you to access the elements of numbers by index and is a useful tool for some situations. But in this case, where you want to get each element’s index and value at the same time, a more elegant solution uses enumerate():

>>>

>>> numbers = [45, 22, 14, 65, 97, 72]
>>> for i, num in enumerate(numbers):
...     if num % 3 == 0 and num % 5 == 0:
...         numbers[i] = 'fizzbuzz'
...     elif num % 3 == 0:
...         numbers[i] = 'fizz'
...     elif num % 5 == 0:
...         numbers[i] = 'buzz'
...
>>> numbers
['fizzbuzz', 22, 14, 'buzz', 97, 'fizz']

For each element, enumerate() returns a counter and the element value. The counter defaults to 0, which conveniently is also the element’s index. Don’t want to start your count at 0? Just use the optional start parameter to set an offset:

>>>

>>> numbers = [45, 22, 14, 65, 97, 72]
>>> for i, num in enumerate(numbers, start=52):
...     print(i, num)
...
52 45
53 22
54 14
55 65
56 97
57 72

By using the start parameter, we access all of the same elements, starting with the first index, but now our count starts from the specified integer value.

Use List Comprehensions Instead of `map()` and `filter()`

“I think dropping filter() and map() is pretty uncontroversial[.]”

— Guido van Rossum, Python’s creator

He may have been wrong about it being uncontroversial, but Guido had good reasons for wanting to remove map() and filter() from Python. One reason is that Python supports list comprehensions, which are often easier to read and support the same functionality as map() and filter().

Let’s first take a look at how we’d structure a call to map() and the equivalent list comprehension:

>>>

>>> numbers = [4, 2, 1, 6, 9, 7]
>>> def square(x):
...     return x*x
...
>>> list(map(square, numbers))
[16, 4, 1, 36, 81, 49]

>>> [square(x) for x in numbers]
[16, 4, 1, 36, 81, 49]

Both approaches, using map() and the list comprehension, return the same values, but the list comprehension is easier to read and understand.

Now we can do the same thing for the filter() and its equivalent list comprehension:

>>>

>>> def is_odd(x):
...    return bool(x % 2)
...
>>> list(filter(is_odd, numbers))
[1, 9, 7]

>>> [x for x in numbers if is_odd(x)]
[1, 9, 7]

Like we saw with map(), the filter() and list comprehension approaches return the same value, but the list comprehension is easier to follow.

Developers who come from other languages may disagree that list comprehensions are easier to read than map() and filter(), but in my experience beginners are able to pick up list comprehensions much more intuitively.

Either way, you’ll rarely go wrong using list comprehensions in a coding interview, as it will communicate that you know what’s most common in Python.

Debug With `breakpoint()` Instead of `print()`

You may have debugged a small problem by adding print() to your code and seeing what was printed out. This approach works well at first but quickly becomes cumbersome. Plus, in a coding interview setting, you hardly want print() calls peppered throughout your code.

Instead, you should be using a debugger. For non-trivial bugs, it’s almost always faster than using print(), and given that debugging is a big part of writing software, it shows that you know how to use tools that will let you develop quickly on the job.

If you’re using Python 3.7, you don’t need to import anything and can just call breakpoint() at the location in your code where you’d like to drop into the debugger:

# Some complicated code with bugs

breakpoint()

Calling breakpoint() will put you into pdb, which is the default Python debugger. On Python 3.6 and older, you can do the same by importing pdb explicitly:

import pdb; pdb.set_trace()

Like breakpoint(), pdb.set_trace() will put you into the pdb debugger. It’s just not quite as clean and is a tad more to remember.

There are other debuggers available that you may want to try, but pdb is part of the standard library, so it’s always available. Whatever debugger you prefer, it’s worth trying them out to get used to the workflow before you’re in a coding interview setting.

Format strings With f-Strings

Python has a lot of different ways to handle string formatting, and it can be tricky to know what to use. In fact, we tackle formatting in depth in two separate articles: one about string formatting in general and one specifically focused on f-strings. In a coding interview, where you’re (hopefully) using Python 3.6+, the suggested formatting approach is Python’s f-strings.

f-strings support use of the string formatting mini-language, as well as powerful string interpolation. These features allow you to add variables or even valid Python expressions and have them evaluated at runtime before being added to the string:

>>>

>>> def get_name_and_decades(name, age):
...     return f"My name is {name} and I'm {age / 10:.5f} decades old."
...
>>> get_name_and_decades("Maria", 31)
My name is Maria and I'm 3.10000 decades old.

The f-string allows you to put Maria into the string and add her age with the desired formatting in one succinct operation.

The one risk to be aware of is that if you’re outputting user-generated values, then that can introduce security risks, in which case Template Strings may be a safer option.

Sort Complex Lists With `sorted()`

Plenty of coding interview questions require some kind of sorting, and there are multiple valid ways you can sort items. Unless the interviewer wants you to implement your own sorting algorithm, it’s usually best to use sorted().

You’ve probably seen the most simple uses of sorting, such as sorting lists of numbers or strings in ascending or descending order:

>>>

>>> sorted([6,5,3,7,2,4,1])
[1, 2, 3, 4, 5, 6, 7]

>>> sorted(['cat', 'dog', 'cheetah', 'rhino', 'bear'], reverse=True)
['rhino', 'dog', 'cheetah', 'cat', 'bear]

By default, sorted() has sorted the input in ascending order, and the reverse keyword argument causes it to sort in descending order.

It’s worth knowing about the optional keyword argument key that lets you specify a function that will be called on every element prior to sorting. Adding a function allows custom sorting rules, which are especially helpful if you want to sort more complex data types:

>>>

>>> animals = [
...     {'type': 'penguin', 'name': 'Stephanie', 'age': 8},
...     {'type': 'elephant', 'name': 'Devon', 'age': 3},
...     {'type': 'puma', 'name': 'Moe', 'age': 5},
... ]
>>> sorted(animals, key=lambda animal: animal['age'])
[
    {'type': 'elephant', 'name': 'Devon', 'age': 3},
    {'type': 'puma', 'name': 'Moe', 'age': 5},
    {'type': 'penguin', 'name': 'Stephanie, 'age': 8},
]

By passing in a lambda function that returns each element’s age, you can easily sort a list of dictionaries by a single value of each of those dictionaries. In this case, the dictionary is now sorted in ascending order by age.

Leverage Data Structures Effectively

Algorithms get a lot of attention in coding interviews, but data structures are arguably even more important. In a coding interviewing context, picking the right data structure can have a major impact on performance.

Beyond theoretical data structures, Python has powerful and convenient functionality built into its standard data structure implementations. These data structures are incredibly useful in coding interviews because they give you lots of functionality by default and let you focus your time on other parts of the problem.

Store Unique Values With Sets

You’ll often need to remove duplicate elements from an existing dataset. New developers will sometimes do so with lists when they should be using sets, which enforce the uniqueness of all elements.

Pretend you have a function named get_random_word(). It will always return a random selection from a small set of words:

>>>

>>> import random
>>> all_words = "all the words in the world".split()
>>> def get_random_word():
...    return random.choice(all_words)

You’re supposed to call get_random_word() repeatedly to get 1000 random words and then return a data structure containing every unique word. Here are two common, suboptimal approaches and one good approach.

Bad Approach

get_unique_words() stores values in a list then converts the list into a set:

>>>

>>> def get_unique_words():
...     words = []
...     for _ in range(1000):
...         words.append(get_random_word())
...     return set(words)
>>> get_unique_words()
{'world', 'all', 'the', 'words'}

This approach isn’t terrible, but it unnecessarily creates a list and then converts it to a set. Interviewers almost always notice (and ask about) this type of design choice.

Worse Approach

To avoid converting from a list to a set, you now store values in a list without using any other data structures. You then test for uniqueness by comparing new values with all elements currently in the list:

>>>

>>> def get_unique_words():
...     words = []
...     for _ in range(1000):
...         word = get_random_word()
...         if word not in words:
...             words.append(word)
...     return words
>>> get_unique_words()
['world', 'all', 'the', 'words']

This is worse than the first approach, because you have to compare every new word against every word already in the list. That means that as the number of words grows, the number of lookups grows quadratically. In other words, the time complexity grows on the order of O(N²).

Good Approach

Now, you skip using lists altogether and instead use a set from the start:

>>>

>>> def get_unique_words():
...     words = set()
...     for _ in range(1000):
...         words.add(get_random_word())
...     return words
>>> get_unique_words()
{'world', 'all', 'the', 'words'}

This may not look much different than the other approaches except for making use of a set from the beginning. If you consider what’s happening within .add(), it even sounds like the second approach: get the word, check if it’s already in the set, and if not, add it to the data structure.

So why is using a set different from the second approach?

It’s different because sets store elements in a manner that allows near-constant-time checks whether a value is in the set or not, unlike lists, which require linear-time lookups. The difference in lookup time means that the time complexity for adding to a set grows at a rate of O(N), which is much better than the O(N²) from the second approach in most cases.

Save Memory With Generators

List comprehensions are convenient tools but can sometimes lead to unnecessary memory usage.

Imagine you’ve been asked to find the sum of the first 1000 perfect squares, starting with 1. You know about list comprehensions, so you quickly code up a working solution:

>>>

>>> sum([i * i for i in range(1, 1001)])
333833500

Your solution makes a list of every perfect square between 1 and 1,000,000 and sums the values. Your code returns the right answer, but then your interviewer starts increasing the number of perfect squares you need to sum.

At first, your function keeps popping out the right answer, but soon it starts slowing down until eventually the process seems to hang for an eternity. This is the last thing you want happening in a coding interview.

What’s going on here?

It’s making a list of every perfect square you’ve requested and summing them all. A list with 1000 perfect squares may not be large in computer-terms, but 100 million or 1 billion is quite a bit of information and can easily overwhelm your computer’s available memory resources. That’s what’s happening here.

Thankfully, there’s a quick way to solve the memory problem. You just replace the brackets with parentheses:

>>>

>>> sum((i * i for i in range(1, 1001)))
333833500

Swapping out the brackets changes your list comprehension into a generator expression. Generator expressions are perfect for when you know you want to retrieve data from a sequence, but you don’t need to access all of it at the same time.

Instead of creating a list, the generator expression returns a generator object. That object knows where it is in the current state (for example, i = 49) and only calculates the next value when it’s asked for.

So when sum iterates over the generator object by calling .__next__() repeatedly, the generator checks what i equals, calculates i * i, increments i internally, and returns the proper value to sum. The design allows generators to be used on massive sequences of data, because only one element exists in memory at a time.

Define Default Values in Dictionaries With `.get()` and `.setdefault()`

One of the most common programming tasks involves adding, modifying, or retrieving an item that may or may not be in a dictionary. Python dictionaries have elegant functionality to make these tasks clean and easy, but developers often check explicitly for values when it isn’t necessary.

Imagine you have a dictionary named cowboy, and you want to get that cowboy’s name. One approach is to explicitly check for the key with a conditional:

>>>

>>> cowboy = {'age': 32, 'horse': 'mustang', 'hat_size': 'large'}
>>> if 'name' in cowboy:
...     name = cowboy['name']
... else:
...     name = 'The Man with No Name'
...
>>> name
'The Man with No Name'

This approach first checks if the name key exists in the dictionary, and if so, it returns the corresponding value. Otherwise, it returns a default value.

While explicitly checking for keys does work, it can easily be replaced with one line if you use .get():

>>>

>>> name = cowboy.get('name', 'The Man with No Name')

get() performs the same operations that were performed in the first approach, but now they’re handled automatically. If the key exists, then the proper value will be returned. Otherwise, the default value will get returned.

But what if you want to update the dictionary with a default value while still accessing the name key? .get() doesn’t really help you here, so you’re left with explicitly checking for the value again:

>>>

>>> if 'name' not in cowboy:
...     cowboy['name'] = 'The Man with No Name'
...
>>> name = cowboy['name']

Checking for the value and setting a default is a valid approach and is easy to read, but again Python offers a more elegant method with .setdefault():

>>>

>>> name = cowboy.setdefault('name', 'The Man with No Name')

.setdefault() accomplishes exactly the same thing as the snippet above. It checks if name exists in cowboy, and if so it returns that value. Otherwise, it sets cowboy['name'] to The Man with No Name and returns the new value.

Take Advantage of Python’s Standard Library

By default, Python comes with a lot of functionality that’s just an import statement away. It’s powerful on its own, but knowing how to leverage the standard library can supercharge your coding interview skills.

It’s hard to pick the most useful pieces from all of the available modules, so this section will focus on just a small subset of its utility functions. Hopefully, these will prove useful to you in coding interviews and also whet your appetite to learn more about the advanced functionality of these and other modules.

Handle Missing Dictionary Keys With `collections.defaultdict()`

.get() and .setdefault() work well when you’re setting a default for a single key, but it’s common to want a default value for all possible unset keys, especially when programming in a coding interview context.

Pretend you have a group of students, and you need to keep track of their grades on homework assignments. The input value is a list of tuples with the format (student_name, grade), but you want to easily look up all the grades for a single student without iterating over the list.

One way to store the grade data uses a dictionary that maps student names to lists of grades:

>>>

>>> student_grades = {}
>>> grades = [
...     ('elliot', 91),
...     ('neelam', 98),
...     ('bianca', 81),
...     ('elliot', 88),
... ]
>>> for name, grade in grades:
...     if name not in student_grades:
...         student_grades[name] = []
...     student_grades[name].append(grade)
...
>>> student_grades
{'elliot': [91, 88], 'neelam': [98], 'bianca': [81]}

In this approach, you iterate over the students and check if their names are already properties in the dictionary. If not, you add them to the dictionary with an empty list as the default value. You then append their actual grades to that student’s list of grades.

But there’s an even cleaner approach that uses a defaultdict, which extends standard dict functionality to allow you to set a default value that will be operated upon if the key doesn’t exist:

>>>

>>> from collections import defaultdict
>>> student_grades = defaultdict(list)
>>> for name, grade in grades:
...     student_grades[name].append(grade)

In this case, you’re creating a defaultdict that uses the list() constructor with no arguments as a default factory method. list() with no arguments returns an empty list, so defaultdict calls list() if the name doesn’t exist and then allows the grade to be appended. If you want to get fancy, you could also use a lambda function as your factory value to return an arbitrary constant.

Leveraging a defaultdict can lead to cleaner application code because you don’t have to worry about default values at the key level. Instead, you can handle them once at the defaultdict level and afterwards act as if the key is always present.

Count Hashable Objects With `collections.Counter`

You have a long string of words with no punctuation or capital letters and you want to count how many times each word appears.

You could use a dictionary or defaultdict and increment the counts, but collections.Counter provides a cleaner and more convenient way to do exactly that. Counter is a subclass of dict that uses 0 as the default value for any missing element and makes it easier to count occurrences of objects:

>>>

>>> from collections import Counter
>>> words = "if there was there was but if \
... there was not there was not".split()
>>> counts = Counter(words)
>>> counts
Counter({'if': 2, 'there': 4, 'was': 4, 'not': 2, 'but': 1})

When you pass in the list of words to Counter, it stores each word along with a count of how many times that word occurred in the list.

Are you curious what the two most common words were? Just use .most_common():

>>>

>>> counts.most_common(2)
[('there', 4), ('was', 4)]

.most_common() is a convenience method and simply returns the n most frequent inputs by count.

Access Common String Groups With `string` Constants

It’s trivia time! Is 'A' > 'a' true or false?

It’s false, because the ASCII code for A is 65, but a is 97, and 65 is not greater than 97.

Why does the answer matter? Because if you want to check if a character is part of the English alphabet, one popular way is to see if it’s between A and z (65 and 122 on the ASCII chart).

Checking the ASCII code works but is clumsy and easy to mess up in coding interviews, especially if you can’t remember whether lowercase or uppercase ASCII characters come first. It’s much easier to use the constants defined as part of the string module.

You can see one in use in is_upper(), which returns whether all characters in a string are uppercase letters:

>>>

>>> import string
>>> def is_upper(word):
...     for letter in word:
...         if letter not in string.ascii_uppercase:
...             return False
...     return True
...
>>> is_upper('Thanks Geir')
False
>>> is_upper('LOL')
True

is_upper() iterates over the letters in word, and checks if the letters are part of string.ascii_uppercase. If you print out string.ascii_uppercase you’ll see that it’s just a lowly string. The value is set to the literal 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'.

All string constants are just strings of frequently referenced string values. They include the following:

string.ascii_letters
string.ascii_uppercase
string.ascii_lowercase
string.digits
string.hexdigits
string.octdigits
string.punctuation
string.printable
string.whitespace

These are easier to use and, even more importantly, easier to read.

Generate Permutations and Combinations With `itertools`

Interviewers love to give real life scenarios to make coding interviews seem less intimidating, so here’s a contrived example: you go to an amusement park and decide to figure out every possible pair of friends that could sit together on a roller coaster.

Unless generating these pairs is the primary purpose of the interview question, it’s likely that generating all the possible pairs is just a tedious step on the way towards a working algorithm. You could calculate them yourself with nested for-loops, or you could use the powerful itertools library.

itertools has multiple tools for generating iterable sequences of input data, but right now we’ll just focus on two common functions: itertools.permutations() and itertools.combinations().

itertools.permutations() builds a list of all permutations, meaning it’s a list of every possible grouping of input values with a length matching the count parameter. The r keyword argument lets us specify how many values go in each grouping:

>>>

>>> import itertools
>>> friends = ['Monique', 'Ashish', 'Devon', 'Bernie']
>>> list(itertools.permutations(friends, r=2))
[('Monique', 'Ashish'), ('Monique', 'Devon'), ('Monique', 'Bernie'),
('Ashish', 'Monique'), ('Ashish', 'Devon'), ('Ashish', 'Bernie'),
('Devon', 'Monique'), ('Devon', 'Ashish'), ('Devon', 'Bernie'),
('Bernie', 'Monique'), ('Bernie', 'Ashish'), ('Bernie', 'Devon')]

With permutations, the order of the elements matters, so ('sam', 'devon') represents a different pairing than ('devon', 'sam'), meaning that they would both be included in the list.

itertools.combinations() builds combinations. These are also the possible groupings of the input values, but now the order of the values doesn’t matter. Because ('sam', 'devon') and ('devon', 'sam') represent the same pair, only one of them would be included in the output list:

>>>

>>> list(itertools.combinations(friends, r=2))
[('Monique', 'Ashish'), ('Monique', 'Devon'), ('Monique', 'Bernie'),
('Ashish', 'Devon'), ('Ashish', 'Bernie'), ('Devon', 'Bernie')]

Since the order of the values matters with combinations, there are fewer combinations than permutations for the same input list. Again, because we set r to 2, each grouping has two names in it.

.combinations() and .permutations() are just small examples of a powerful library, but even these two functions can be quite useful when you’re trying to solve an algorithm problem quickly.

Conclusion: Coding Interview Superpowers

You can now feel comfortable using some of Python’s less common, but more powerful, standard features in your next coding interview. There’s a lot to learn about the language as a whole but this article should have given you a starting point to go deeper while letting you use Python more effectively when you interview.

In this article, you’ve learned different types of standard tools to supercharge your coding interview skills:

Powerful built-in functions
Data structures built to handle common scenarios with barely any code
Standard library packages that have feature-rich solutions for specific problems, letting you write better code faster

Interviewing may not be the best approximation of real software development, but it’s worth knowing how to succeed in any programming environment, even interviews. Thankfully, learning how to use Python during coding interviews can help you understand the language more deeply, which will pay dividends during day-to-day development.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Django Migrations 101

2019-03-26T14:00:00+00:00

Since version 1.7, Django has come with built-in support for database migrations. In Django, database migrations usually go hand in hand with models: whenever you code up a new model, you also generate a migration to create the necessary table in the database. However, migrations can do much more.

With this course you’ll get comfortable with Django migrations and learn how to create database tables without writing any SQL, how to automatically modify your database after you changed your models, and how to revert changes made to your database.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Documenting Python Projects With Sphinx and Read The Docs

2019-03-26T13:30:00+00:00

In this video series we’ll cover creating Python documentation from scratch using Sphinx, as well as getting your code repository hooked up to Read The Docs, to automatically build and publish your code documentation.

Note: This course uses Python 2.7 in its coding examples.

Resources

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

An Intro to Threading in Python

2019-03-25T14:00:00+00:00

Python threading allows you to have different parts of your program run concurrently and can simplify your design. If you’ve got some experience in Python and want to speed up your program using threads, then this tutorial is for you!

In this article, you’ll learn:

What threads are
How to create threads and wait for them to finish
How to use a ThreadPoolExecutor
How to avoid race conditions
How to use the common tools that Python threading provides

This article assumes you’ve got the Python basics down pat and that you’re using at least version 3.6 to run the examples. If you need a refresher, you can start with the Python Learning Paths and get up to speed.

If you’re not sure if you want to use Python threading, asyncio, or multiprocessing, then you can check out Speed Up Your Python Program With Concurrency.

All of the sources used in this tutorial are available to you in the Real Python GitHub repo.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Take the Quiz: Test your knowledge with our interactive “Python Threading” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Take the Quiz »

What Is a Thread?

A thread is a separate flow of execution. This means that your program will have two things happening at once. But for most Python 3 implementations the different threads do not actually execute at the same time: they merely appear to.

It’s tempting to think of threading as having two (or more) different processors running on your program, each one doing an independent task at the same time. That’s almost right. The threads may be running on different processors, but they will only be running one at a time.

Getting multiple tasks running simultaneously requires a non-standard implementation of Python, writing some of your code in a different language, or using multiprocessing which comes with some extra overhead.

Because of the way CPython implementation of Python works, threading may not speed up all tasks. This is due to interactions with the GIL that essentially limit one Python thread to run at a time.

Tasks that spend much of their time waiting for external events are generally good candidates for threading. Problems that require heavy CPU computation and spend little time waiting for external events might not run faster at all.

This is true for code written in Python and running on the standard CPython implementation. If your threads are written in C they have the ability to release the GIL and run concurrently. If you are running on a different Python implementation, check with the documentation too see how it handles threads.

If you are running a standard Python implementation, writing in only Python, and have a CPU-bound problem, you should check out the multiprocessing module instead.

Architecting your program to use threading can also provide gains in design clarity. Most of the examples you’ll learn about in this tutorial are not necessarily going to run faster because they use threads. Using threading in them helps to make the design cleaner and easier to reason about.

So, let’s stop talking about threading and start using it!

Starting a Thread

Now that you’ve got an idea of what a thread is, let’s learn how to make one. The Python standard library provides threading, which contains most of the primitives you’ll see in this article. Thread, in this module, nicely encapsulates threads, providing a clean interface to work with them.

To start a separate thread, you create a Thread instance and then tell it to .start():

 1 import logging
 2 import threading
 3 import time
 4 
 5 def thread_function(name):
 6     logging.info("Thread %s: starting", name)
 7     time.sleep(2)
 8     logging.info("Thread %s: finishing", name)
 9 
10 if __name__ == "__main__":
11     format = "%(asctime)s: %(message)s"
12     logging.basicConfig(format=format, level=logging.INFO,
13                         datefmt="%H:%M:%S")
14 
15     logging.info("Main    : before creating thread")
16     x = threading.Thread(target=thread_function, args=(1,))
17     logging.info("Main    : before running thread")
18     x.start()
19     logging.info("Main    : wait for the thread to finish")
20     # x.join()
21     logging.info("Main    : all done")

If you look around the logging statements, you can see that the main section is creating and starting the thread:

x = threading.Thread(target=thread_function, args=(1,))
x.start()

When you create a Thread, you pass it a function and a list containing the arguments to that function. In this case, you’re telling the Thread to run thread_function() and to pass it 1 as an argument.

For this article, you’ll use sequential integers as names for your threads. There is threading.get_ident(), which returns a unique name for each thread, but these are usually neither short nor easily readable.

thread_function() itself doesn’t do much. It simply logs some messages with a time.sleep() in between them.

When you run this program as it is (with line twenty commented out), the output will look like this:

$ ./single_thread.py
Main    : before creating thread
Main    : before running thread
Thread 1: starting
Main    : wait for the thread to finish
Main    : all done
Thread 1: finishing

You’ll notice that the Thread finished after the Main section of your code did. You’ll come back to why that is and talk about the mysterious line twenty in the next section.

Daemon Threads

In computer science, a daemon is a process that runs in the background.

Python threading has a more specific meaning for daemon. A daemon thread will shut down immediately when the program exits. One way to think about these definitions is to consider the daemon thread a thread that runs in the background without worrying about shutting it down.

If a program is running Threads that are not daemons, then the program will wait for those threads to complete before it terminates. Threads that are daemons, however, are just killed wherever they are when the program is exiting.

Let’s look a little more closely at the output of your program above. The last two lines are the interesting bit. When you run the program, you’ll notice that there is a pause (of about 2 seconds) after __main__ has printed its all done message and before the thread is finished.

This pause is Python waiting for the non-daemonic thread to complete. When your Python program ends, part of the shutdown process is to clean up the threading routine.

If you look at the source for Python threading, you’ll see that threading._shutdown() walks through all of the running threads and calls .join() on every one that does not have the daemon flag set.

So your program waits to exit because the thread itself is waiting in a sleep. As soon as it has completed and printed the message, .join() will return and the program can exit.

Frequently, this behavior is what you want, but there are other options available to us. Let’s first repeat the program with a daemon thread. You do that by changing how you construct the Thread, adding the daemon=True flag:

x = threading.Thread(target=thread_function, args=(1,), daemon=True)

When you run the program now, you should see this output:

$ ./daemon_thread.py
Main    : before creating thread
Main    : before running thread
Thread 1: starting
Main    : wait for the thread to finish
Main    : all done

The difference here is that the final line of the output is missing. thread_function() did not get a chance to complete. It was a daemon thread, so when __main__ reached the end of its code and the program wanted to finish, the daemon was killed.

`join()` a Thread

Daemon threads are handy, but what about when you want to wait for a thread to stop? What about when you want to do that and not exit your program? Now let’s go back to your original program and look at that commented out line twenty:

# x.join()

To tell one thread to wait for another thread to finish, you call .join(). If you uncomment that line, the main thread will pause and wait for the thread x to complete running.

Did you test this on the code with the daemon thread or the regular thread? It turns out that it doesn’t matter. If you .join() a thread, that statement will wait until either kind of thread is finished.

Working With Many Threads

The example code so far has only been working with two threads: the main thread and one you started with the threading.Thread object.

Frequently, you’ll want to start a number of threads and have them do interesting work. Let’s start by looking at the harder way of doing that, and then you’ll move on to an easier method.

The harder way of starting multiple threads is the one you already know:

import logging
import threading
import time

def thread_function(name):
    logging.info("Thread %s: starting", name)
    time.sleep(2)
    logging.info("Thread %s: finishing", name)

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")

    threads = list()
    for index in range(3):
        logging.info("Main    : create and start thread %d.", index)
        x = threading.Thread(target=thread_function, args=(index,))
        threads.append(x)
        x.start()

    for index, thread in enumerate(threads):
        logging.info("Main    : before joining thread %d.", index)
        thread.join()
        logging.info("Main    : thread %d done", index)

This code uses the same mechanism you saw above to start a thread, create a Thread object, and then call .start(). The program keeps a list of Thread objects so that it can then wait for them later using .join().

Running this code multiple times will likely produce some interesting results. Here’s an example output from my machine:

$ ./multiple_threads.py
Main    : create and start thread 0.
Thread 0: starting
Main    : create and start thread 1.
Thread 1: starting
Main    : create and start thread 2.
Thread 2: starting
Main    : before joining thread 0.
Thread 2: finishing
Thread 1: finishing
Thread 0: finishing
Main    : thread 0 done
Main    : before joining thread 1.
Main    : thread 1 done
Main    : before joining thread 2.
Main    : thread 2 done

If you walk through the output carefully, you’ll see all three threads getting started in the order you might expect, but in this case they finish in the opposite order! Multiple runs will produce different orderings. Look for the Thread x: finishing message to tell you when each thread is done.

The order in which threads are run is determined by the operating system and can be quite hard to predict. It may (and likely will) vary from run to run, so you need to be aware of that when you design algorithms that use threading.

Fortunately, Python gives you several primitives that you’ll look at later to help coordinate threads and get them running together. Before that, let’s look at how to make managing a group of threads a bit easier.

Using a `ThreadPoolExecutor`

There’s an easier way to start up a group of threads than the one you saw above. It’s called a ThreadPoolExecutor, and it’s part of the standard library in concurrent.futures (as of Python 3.2).

The easiest way to create it is as a context manager, using the with statement to manage the creation and destruction of the pool.

Here’s the __main__ from the last example rewritten to use a ThreadPoolExecutor:

import concurrent.futures

# [rest of code]

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")

    with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
        executor.map(thread_function, range(3))

The code creates a ThreadPoolExecutor as a context manager, telling it how many worker threads it wants in the pool. It then uses .map() to step through an iterable of things, in your case range(3), passing each one to a thread in the pool.

The end of the with block causes the ThreadPoolExecutor to do a .join() on each of the threads in the pool. It is strongly recommended that you use ThreadPoolExecutor as a context manager when you can so that you never forget to .join() the threads.

Note: Using a ThreadPoolExecutor can cause some confusing errors.

For example, if you call a function that takes no parameters, but you pass it parameters in .map(), the thread will throw an exception.

Unfortunately, ThreadPoolExecutor will hide that exception, and (in the case above) the program terminates with no output. This can be quite confusing to debug at first.

Running your corrected example code will produce output that looks like this:

$ ./executor.py
Thread 0: starting
Thread 1: starting
Thread 2: starting
Thread 1: finishing
Thread 0: finishing
Thread 2: finishing

Again, notice how Thread 1 finished before Thread 0. The scheduling of threads is done by the operating system and does not follow a plan that’s easy to figure out.

Race Conditions

Before you move on to some of the other features tucked away in Python threading, let’s talk a bit about one of the more difficult issues you’ll run into when writing threaded programs: race conditions.

Once you’ve seen what a race condition is and looked at one happening, you’ll move on to some of the primitives provided by the standard library to prevent race conditions from happening.

Race conditions can occur when two or more threads access a shared piece of data or resource. In this example, you’re going to create a large race condition that happens every time, but be aware that most race conditions are not this obvious. Frequently, they only occur rarely, and they can produce confusing results. As you can imagine, this makes them quite difficult to debug.

Fortunately, this race condition will happen every time, and you’ll walk through it in detail to explain what is happening.

For this example, you’re going to write a class that updates a database. Okay, you’re not really going to have a database: you’re just going to fake it, because that’s not the point of this article.

Your FakeDatabase will have .__init__() and .update() methods:

class FakeDatabase:
    def __init__(self):
        self.value = 0

    def update(self, name):
        logging.info("Thread %s: starting update", name)
        local_copy = self.value
        local_copy += 1
        time.sleep(0.1)
        self.value = local_copy
        logging.info("Thread %s: finishing update", name)

FakeDatabase is keeping track of a single number: .value. This is going to be the shared data on which you’ll see the race condition.

.__init__() simply initializes .value to zero. So far, so good.

.update() looks a little strange. It’s simulating reading a value from a database, doing some computation on it, and then writing a new value back to the database.

In this case, reading from the database just means copying .value to a local variable. The computation is just to add one to the value and then .sleep() for a little bit. Finally, it writes the value back by copying the local value back to .value.

Here’s how you’ll use this FakeDatabase:

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")

    database = FakeDatabase()
    logging.info("Testing update. Starting value is %d.", database.value)
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        for index in range(2):
            executor.submit(database.update, index)
    logging.info("Testing update. Ending value is %d.", database.value)

The program creates a ThreadPoolExecutor with two threads and then calls .submit() on each of them, telling them to run database.update().

.submit() has a signature that allows both positional and named arguments to be passed to the function running in the thread:

.submit(function, *args, **kwargs)

In the usage above, index is passed as the first and only positional argument to database.update(). You’ll see later in this article where you can pass multiple arguments in a similar manner.

Since each thread runs .update(), and .update() adds one to .value, you might expect database.value to be 2 when it’s printed out at the end. But you wouldn’t be looking at this example if that was the case. If you run the above code, the output looks like this:

$ ./racecond.py
Testing unlocked update. Starting value is 0.
Thread 0: starting update
Thread 1: starting update
Thread 0: finishing update
Thread 1: finishing update
Testing unlocked update. Ending value is 1.

You might have expected that to happen, but let’s look at the details of what’s really going on here, as that will make the solution to this problem easier to understand.

One Thread

Before you dive into this issue with two threads, let’s step back and talk a bit about some details of how threads work.

You won’t be diving into all of the details here, as that’s not important at this level. We’ll also be simplifying a few things in a way that won’t be technically accurate but will give you the right idea of what is happening.

When you tell your ThreadPoolExecutor to run each thread, you tell it which function to run and what parameters to pass to it: executor.submit(database.update, index).

The result of this is that each of the threads in the pool will call database.update(index). Note that database is a reference to the one FakeDatabase object created in __main__. Calling .update() on that object calls an instance method on that object.

Each thread is going to have a reference to the same FakeDatabase object, database. Each thread will also have a unique value, index, to make the logging statements a bit easier to read:

When the thread starts running .update(), it has its own version of all of the data local to the function. In the case of .update(), this is local_copy. This is definitely a good thing. Otherwise, two threads running the same function would always confuse each other. It means that all variables that are scoped (or local) to a function are thread-safe.

Now you can start walking through what happens if you run the program above with a single thread and a single call to .update().

The image below steps through the execution of .update() if only a single thread is run. The statement is shown on the left followed by a diagram showing the values in the thread’s local_value and the shared database.value:

The diagram is laid out so that time increases as you move from top to bottom. It begins when Thread 1 is created and ends when it is terminated.

When Thread 1 starts, FakeDatabase.value is zero. The first line of code in the method, local_copy = self.value, copies the value zero to the local variable. Next it increments the value of local_copy with the local_copy += 1 statement. You can see .value in Thread 1 getting set to one.

Next time.sleep() is called, which makes the current thread pause and allows other threads to run. Since there is only one thread in this example, this has no effect.

When Thread 1 wakes up and continues, it copies the new value from local_copy to FakeDatabase.value, and then the thread is complete. You can see that database.value is set to one.

So far, so good. You ran .update() once and FakeDatabase.value was incremented to one.

Two Threads

Getting back to the race condition, the two threads will be running concurrently but not at the same time. They will each have their own version of local_copy and will each point to the same database. It is this shared database object that is going to cause the problems.

The program starts with Thread 1 running .update():

When Thread 1 calls time.sleep(), it allows the other thread to start running. This is where things get interesting.

Thread 2 starts up and does the same operations. It’s also copying database.value into its private local_copy, and this shared database.value has not yet been updated:

When Thread 2 finally goes to sleep, the shared database.value is still unmodified at zero, and both private versions of local_copy have the value one.

Thread 1 now wakes up and saves its version of local_copy and then terminates, giving Thread 2 a final chance to run. Thread 2 has no idea that Thread 1 ran and updated database.value while it was sleeping. It stores its version of local_copy into database.value, also setting it to one:

The two threads have interleaving access to a single shared object, overwriting each other’s results. Similar race conditions can arise when one thread frees memory or closes a file handle before the other thread is finished accessing it.

Why This Isn’t a Silly Example

The example above is contrived to make sure that the race condition happens every time you run your program. Because the operating system can swap out a thread at any time, it is possible to interrupt a statement like x = x + 1 after it has read the value of x but before it has written back the incremented value.

The details of how this happens are quite interesting, but not needed for the rest of this article, so feel free to skip over this hidden section.

The code above isn’t quite as out there as you might originally have thought. It was designed to force a race condition every time you run it, but that makes it much easier to solve than most race conditions.

There are two things to keep in mind when thinking about race conditions:

Even an operation like x += 1 takes the processor many steps. Each of these steps is a separate instruction to the processor.
The operating system can swap which thread is running at any time. A thread can be swapped out after any of these small instructions. This means that a thread can be put to sleep to let another thread run in the middle of a Python statement.

Let’s look at this in detail. The REPL below shows a function that takes a parameter and increments it:

>>>

>>> def inc(x):
...    x += 1
...
>>> import dis
>>> dis.dis(inc)
  2           0 LOAD_FAST                0 (x)
              2 LOAD_CONST               1 (1)
              4 INPLACE_ADD
              6 STORE_FAST               0 (x)
              8 LOAD_CONST               0 (None)
             10 RETURN_VALUE

The REPL example uses dis from the Python standard library to show the smaller steps that the processor does to implement your function. It does a LOAD_FAST of the data value x, it does a LOAD_CONST 1, and then it uses the INPLACE_ADD to add those values together.

We’re stopping here for a specific reason. This is the point in .update() above where time.sleep() forced the threads to switch. It is entirely possible that, every once in while, the operating system would switch threads at that exact point even without sleep(), but the call to sleep() makes it happen every time.

As you learned above, the operating system can swap threads at any time. You’ve walked down this listing to the statement marked 4. If the operating system swaps out this thread and runs a different thread that also modifies x, then when this thread resumes, it will overwrite x with an incorrect value.

Technically, this example won’t have a race condition because x is local to inc(). It does illustrate how a thread can be interrupted during a single Python operation, however. The same LOAD, MODIFY, STORE set of operations also happens on global and shared values. You can explore with the dis module and prove that yourself.

It’s rare to get a race condition like this to occur, but remember that an infrequent event taken over millions of iterations becomes likely to happen. The rarity of these race conditions makes them much, much harder to debug than regular bugs.

Now back to your regularly scheduled tutorial!

Now that you’ve seen a race condition in action, let’s find out how to solve them!

Basic Synchronization Using `Lock`

There are a number of ways to avoid or solve race conditions. You won’t look at all of them here, but there are a couple that are used frequently. Let’s start with Lock.

To solve your race condition above, you need to find a way to allow only one thread at a time into the read-modify-write section of your code. The most common way to do this is called Lock in Python. In some other languages this same idea is called a mutex. Mutex comes from MUTual EXclusion, which is exactly what a Lock does.

A Lock is an object that acts like a hall pass. Only one thread at a time can have the Lock. Any other thread that wants the Lock must wait until the owner of the Lock gives it up.

The basic functions to do this are .acquire() and .release(). A thread will call my_lock.acquire() to get the lock. If the lock is already held, the calling thread will wait until it is released. There’s an important point here. If one thread gets the lock but never gives it back, your program will be stuck. You’ll read more about this later.

Fortunately, Python’s Lock will also operate as a context manager, so you can use it in a with statement, and it gets released automatically when the with block exits for any reason.

Let’s look at the FakeDatabase with a Lock added to it. The calling function stays the same:

class FakeDatabase:
    def __init__(self):
        self.value = 0
        self._lock = threading.Lock()

    def locked_update(self, name):
        logging.info("Thread %s: starting update", name)
        logging.debug("Thread %s about to lock", name)
        with self._lock:
            logging.debug("Thread %s has lock", name)
            local_copy = self.value
            local_copy += 1
            time.sleep(0.1)
            self.value = local_copy
            logging.debug("Thread %s about to release lock", name)
        logging.debug("Thread %s after release", name)
        logging.info("Thread %s: finishing update", name)

Other than adding a bunch of debug logging so you can see the locking more clearly, the big change here is to add a member called ._lock, which is a threading.Lock() object. This ._lock is initialized in the unlocked state and locked and released by the with statement.

It’s worth noting here that the thread running this function will hold on to that Lock until it is completely finished updating the database. In this case, that means it will hold the Lock while it copies, updates, sleeps, and then writes the value back to the database.

If you run this version with logging set to warning level, you’ll see this:

$ ./fixrace.py
Testing locked update. Starting value is 0.
Thread 0: starting update
Thread 1: starting update
Thread 0: finishing update
Thread 1: finishing update
Testing locked update. Ending value is 2.

Look at that. Your program finally works!

You can turn on full logging by setting the level to DEBUG by adding this statement after you configure the logging output in __main__:

logging.getLogger().setLevel(logging.DEBUG)

Running this program with DEBUG logging turned on looks like this:

$ ./fixrace.py
Testing locked update. Starting value is 0.
Thread 0: starting update
Thread 0 about to lock
Thread 0 has lock
Thread 1: starting update
Thread 1 about to lock
Thread 0 about to release lock
Thread 0 after release
Thread 0: finishing update
Thread 1 has lock
Thread 1 about to release lock
Thread 1 after release
Thread 1: finishing update
Testing locked update. Ending value is 2.

In this output you can see Thread 0 acquires the lock and is still holding it when it goes to sleep. Thread 1 then starts and attempts to acquire the same lock. Because Thread 0 is still holding it, Thread 1 has to wait. This is the mutual exclusion that a Lock provides.

Many of the examples in the rest of this article will have WARNING and DEBUG level logging. We’ll generally only show the WARNING level output, as the DEBUG logs can be quite lengthy. Try out the programs with the logging turned up and see what they do.

Deadlock

Before you move on, you should look at a common problem when using Locks. As you saw, if the Lock has already been acquired, a second call to .acquire() will wait until the thread that is holding the Lock calls .release(). What do you think happens when you run this code:

import threading

l = threading.Lock()
print("before first acquire")
l.acquire()
print("before second acquire")
l.acquire()
print("acquired lock twice")

When the program calls l.acquire() the second time, it hangs waiting for the Lock to be released. In this example, you can fix the deadlock by removing the second call, but deadlocks usually happen from one of two subtle things:

An implementation bug where a Lock is not released properly
A design issue where a utility function needs to be called by functions that might or might not already have the Lock

The first situation happens sometimes, but using a Lock as a context manager greatly reduces how often. It is recommended to write code whenever possible to make use of context managers, as they help to avoid situations where an exception skips you over the .release() call.

The design issue can be a bit trickier in some languages. Thankfully, Python threading has a second object, called RLock, that is designed for just this situation. It allows a thread to .acquire() an RLock multiple times before it calls .release(). That thread is still required to call .release() the same number of times it called .acquire(), but it should be doing that anyway.

Lock and RLock are two of the basic tools used in threaded programming to prevent race conditions. There are a few other that work in different ways. Before you look at them, let’s shift to a slightly different problem domain.

Producer-Consumer Threading

The Producer-Consumer Problem is a standard computer science problem used to look at threading or process synchronization issues. You’re going to look at a variant of it to get some ideas of what primitives the Python threading module provides.

For this example, you’re going to imagine a program that needs to read messages from a network and write them to disk. The program does not request a message when it wants. It must be listening and accept messages as they come in. The messages will not come in at a regular pace, but will be coming in bursts. This part of the program is called the producer.

On the other side, once you have a message, you need to write it to a database. The database access is slow, but fast enough to keep up to the average pace of messages. It is not fast enough to keep up when a burst of messages comes in. This part is the consumer.

In between the producer and the consumer, you will create a Pipeline that will be the part that changes as you learn about different synchronization objects.

That’s the basic layout. Let’s look at a solution using Lock. It doesn’t work perfectly, but it uses tools you already know, so it’s a good place to start.

Producer-Consumer Using `Lock`

Since this is an article about Python threading, and since you just read about the Lock primitive, let’s try to solve this problem with two threads using a Lock or two.

The general design is that there is a producer thread that reads from the fake network and puts the message into a Pipeline:

SENTINEL = object()

def producer(pipeline):
    """Pretend we're getting a message from the network."""
    for index in range(10):
        message = random.randint(1, 101)
        logging.info("Producer got message: %s", message)
        pipeline.set_message(message, "Producer")

    # Send a sentinel message to tell consumer we're done
    pipeline.set_message(SENTINEL, "Producer")

To generate a fake message, the producer gets a random number between one and one hundred. It calls .set_message() on the pipeline to send it to the consumer.

The producer also uses a SENTINEL value to signal the consumer to stop after it has sent ten values. This is a little awkward, but don’t worry, you’ll see ways to get rid of this SENTINEL value after you work through this example.

On the other side of the pipeline is the consumer:

def consumer(pipeline):
    """Pretend we're saving a number in the database."""
    message = 0
    while message is not SENTINEL:
        message = pipeline.get_message("Consumer")
        if message is not SENTINEL:
            logging.info("Consumer storing message: %s", message)

The consumer reads a message from the pipeline and writes it to a fake database, which in this case is just printing it to the display. If it gets the SENTINEL value, it returns from the function, which will terminate the thread.

Before you look at the really interesting part, the Pipeline, here’s the __main__ section, which spawns these threads:

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")
    # logging.getLogger().setLevel(logging.DEBUG)

    pipeline = Pipeline()
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        executor.submit(producer, pipeline)
        executor.submit(consumer, pipeline)

This should look fairly familiar as it’s close to the __main__ code in the previous examples.

Remember that you can turn on DEBUG logging to see all of the logging messages by uncommenting this line:

# logging.getLogger().setLevel(logging.DEBUG)

It can be worthwhile to walk through the DEBUG logging messages to see exactly where each thread acquires and releases the locks.

Now let’s take a look at the Pipeline that passes messages from the producer to the consumer:

class Pipeline:
    """
    Class to allow a single element pipeline between producer and consumer.
    """
    def __init__(self):
        self.message = 0
        self.producer_lock = threading.Lock()
        self.consumer_lock = threading.Lock()
        self.consumer_lock.acquire()

    def get_message(self, name):
        logging.debug("%s:about to acquire getlock", name)
        self.consumer_lock.acquire()
        logging.debug("%s:have getlock", name)
        message = self.message
        logging.debug("%s:about to release setlock", name)
        self.producer_lock.release()
        logging.debug("%s:setlock released", name)
        return message

    def set_message(self, message, name):
        logging.debug("%s:about to acquire setlock", name)
        self.producer_lock.acquire()
        logging.debug("%s:have setlock", name)
        self.message = message
        logging.debug("%s:about to release getlock", name)
        self.consumer_lock.release()
        logging.debug("%s:getlock released", name)

Woah! That’s a lot of code. A pretty high percentage of that is just logging statements to make it easier to see what’s happening when you run it. Here’s the same code with all of the logging statements removed:

class Pipeline:
    """
    Class to allow a single element pipeline between producer and consumer.
    """
    def __init__(self):
        self.message = 0
        self.producer_lock = threading.Lock()
        self.consumer_lock = threading.Lock()
        self.consumer_lock.acquire()

    def get_message(self, name):
        self.consumer_lock.acquire()
        message = self.message
        self.producer_lock.release()
        return message

    def set_message(self, message, name):
        self.producer_lock.acquire()
        self.message = message
        self.consumer_lock.release()

That seems a bit more manageable. The Pipeline in this version of your code has three members:

.message stores the message to pass.
.producer_lock is a threading.Lock object that restricts access to the message by the producer thread.
.consumer_lock is also a threading.Lock that restricts access to the message by the consumer thread.

__init__() initializes these three members and then calls .acquire() on the .consumer_lock. This is the state you want to start in. The producer is allowed to add a new message, but the consumer needs to wait until a message is present.

.get_message() and .set_messages() are nearly opposites. .get_message() calls .acquire() on the consumer_lock. This is the call that will make the consumer wait until a message is ready.

Once the consumer has acquired the .consumer_lock, it copies out the value in .message and then calls .release() on the .producer_lock. Releasing this lock is what allows the producer to insert the next message into the pipeline.

Before you go on to .set_message(), there’s something subtle going on in .get_message() that’s pretty easy to miss. It might seem tempting to get rid of message and just have the function end with return self.message. See if you can figure out why you don’t want to do that before moving on.

Here’s the answer. As soon as the consumer calls .producer_lock.release(), it can be swapped out, and the producer can start running. That could happen before .release() returns! This means that there is a slight possibility that when the function returns self.message, that could actually be the next message generated, so you would lose the first message. This is another example of a race condition.

Moving on to .set_message(), you can see the opposite side of the transaction. The producer will call this with a message. It will acquire the .producer_lock, set the .message, and the call .release() on then consumer_lock, which will allow the consumer to read that value.

Let’s run the code that has logging set to WARNING and see what it looks like:

$ ./prodcom_lock.py
Producer got data 43
Producer got data 45
Consumer storing data: 43
Producer got data 86
Consumer storing data: 45
Producer got data 40
Consumer storing data: 86
Producer got data 62
Consumer storing data: 40
Producer got data 15
Consumer storing data: 62
Producer got data 16
Consumer storing data: 15
Producer got data 61
Consumer storing data: 16
Producer got data 73
Consumer storing data: 61
Producer got data 22
Consumer storing data: 73
Consumer storing data: 22

At first, you might find it odd that the producer gets two messages before the consumer even runs. If you look back at the producer and .set_message(), you will notice that the only place it will wait for a Lock is when it attempts to put the message into the pipeline. This is done after the producer gets the message and logs that it has it.

When the producer attempts to send this second message, it will call .set_message() the second time and it will block.

The operating system can swap threads at any time, but it generally lets each thread have a reasonable amount of time to run before swapping it out. That’s why the producer usually runs until it blocks in the second call to .set_message().

Once a thread is blocked, however, the operating system will always swap it out and find a different thread to run. In this case, the only other thread with anything to do is the consumer.

The consumer calls .get_message(), which reads the message and calls .release() on the .producer_lock, thus allowing the producer to run again the next time threads are swapped.

Notice that the first message was 43, and that is exactly what the consumer read, even though the producer had already generated the 45 message.

While it works for this limited test, it is not a great solution to the producer-consumer problem in general because it only allows a single value in the pipeline at a time. When the producer gets a burst of messages, it will have nowhere to put them.

Let’s move on to a better way to solve this problem, using a Queue.

Producer-Consumer Using `Queue`

If you want to be able to handle more than one value in the pipeline at a time, you’ll need a data structure for the pipeline that allows the number to grow and shrink as data backs up from the producer.

Python’s standard library has a queue module which, in turn, has a Queue class. Let’s change the Pipeline to use a Queue instead of just a variable protected by a Lock. You’ll also use a different way to stop the worker threads by using a different primitive from Python threading, an Event.

Let’s start with the Event. The threading.Event object allows one thread to signal an event while many other threads can be waiting for that event to happen. The key usage in this code is that the threads that are waiting for the event do not necessarily need to stop what they are doing, they can just check the status of the Event every once in a while.

The triggering of the event can be many things. In this example, the main thread will simply sleep for a while and then .set() it:

 1 if __name__ == "__main__":
 2     format = "%(asctime)s: %(message)s"
 3     logging.basicConfig(format=format, level=logging.INFO,
 4                         datefmt="%H:%M:%S")
 5     # logging.getLogger().setLevel(logging.DEBUG)
 6 
 7     pipeline = Pipeline()
 8     event = threading.Event()
 9     with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
10         executor.submit(producer, pipeline, event)
11         executor.submit(consumer, pipeline, event)
12 
13         time.sleep(0.1)
14         logging.info("Main: about to set event")
15         event.set()

The only changes here are the creation of the event object on line 6, passing the event as a parameter on lines 8 and 9, and the final section on lines 11 to 13, which sleep for a second, log a message, and then call .set() on the event.

The producer also did not have to change too much:

 1 def producer(pipeline, event):
 2     """Pretend we're getting a number from the network."""
 3     while not event.is_set():
 4         message = random.randint(1, 101)
 5         logging.info("Producer got message: %s", message)
 6         pipeline.set_message(message, "Producer")
 7 
 8     logging.info("Producer received EXIT event. Exiting")

It now will loop until it sees that the event was set on line 3. It also no longer puts the SENTINEL value into the pipeline.

consumer had to change a little more:

 1 def consumer(pipeline, event):
 2     """Pretend we're saving a number in the database."""
 3     while not event.is_set() or not pipeline.empty():
 4         message = pipeline.get_message("Consumer")
 5         logging.info(
 6             "Consumer storing message: %s  (queue size=%s)",
 7             message,
 8             pipeline.qsize(),
 9         )
10 
11     logging.info("Consumer received EXIT event. Exiting")

While you got to take out the code related to the SENTINEL value, you did have to do a slightly more complicated while condition. Not only does it loop until the event is set, but it also needs to keep looping until the pipeline has been emptied.

Making sure the queue is empty before the consumer finishes prevents another fun issue. If the consumer does exit while the pipeline has messages in it, there are two bad things that can happen. The first is that you lose those final messages, but the more serious one is that the producer can get caught on the producer_lock and never return.

This happens if the event gets triggered after the producer has checked the .is_set() condition but before it calls pipeline.set_message().

If that happens, it’s possible for the producer to wake up and exit with the .producer_lock still being held. The producer will then try to .acquire() the .producer_lock, but the consumer has exited and will never .release() it.

The rest of the consumer should look familiar.

The Pipeline has changed dramatically, however:

 1 class Pipeline(queue.Queue):
 2     def __init__(self):
 3         super().__init__(maxsize=10)
 4 
 5     def get_message(self, name):
 6         logging.debug("%s:about to get from queue", name)
 7         value = self.get()
 8         logging.debug("%s:got %d from queue", name, value)
 9         return value
10 
11     def set_message(self, value, name):
12         logging.debug("%s:about to add %d to queue", name, value)
13         self.put(value)
14         logging.debug("%s:added %d to queue", name, value)

You can see that Pipeline is a subclass of queue.Queue. Queue has an optional parameter when initializing to specify a maximum size of the queue.

If you give a positive number for maxsize, it will limit the queue to that number of elements, causing .put() to block until there are fewer than maxsize elements. If you don’t specify maxsize, then the queue will grow to the limits of your computer’s memory.

.get_message() and .set_message() got much smaller. They basically wrap .get() and .put() on the Queue. You might be wondering where all of the locking code that prevents the threads from causing race conditions went.

The core devs who wrote the standard library knew that a Queue is frequently used in multi-threading environments and incorporated all of that locking code inside the Queue itself. Queue is thread-safe.

Running this program looks like the following:

$ ./prodcom_queue.py
Producer got message: 32
Producer got message: 51
Producer got message: 25
Producer got message: 94
Producer got message: 29
Consumer storing message: 32 (queue size=3)
Producer got message: 96
Consumer storing message: 51 (queue size=3)
Producer got message: 6
Consumer storing message: 25 (queue size=3)
Producer got message: 31

[many lines deleted]

Producer got message: 80
Consumer storing message: 94 (queue size=6)
Producer got message: 33
Consumer storing message: 20 (queue size=6)
Producer got message: 48
Consumer storing message: 31 (queue size=6)
Producer got message: 52
Consumer storing message: 98 (queue size=6)
Main: about to set event
Producer got message: 13
Consumer storing message: 59 (queue size=6)
Producer received EXIT event. Exiting
Consumer storing message: 75 (queue size=6)
Consumer storing message: 97 (queue size=5)
Consumer storing message: 80 (queue size=4)
Consumer storing message: 33 (queue size=3)
Consumer storing message: 48 (queue size=2)
Consumer storing message: 52 (queue size=1)
Consumer storing message: 13 (queue size=0)
Consumer received EXIT event. Exiting

If you read through the output in my example, you can see some interesting things happening. Right at the top, you can see the producer got to create five messages and place four of them on the queue. It got swapped out by the operating system before it could place the fifth one.

The consumer then ran and pulled off the first message. It printed out that message as well as how deep the queue was at that point:

Consumer storing message: 32 (queue size=3)

This is how you know that the fifth message hasn’t made it into the pipeline yet. The queue is down to size three after a single message was removed. You also know that the queue can hold ten messages, so the producer thread didn’t get blocked by the queue. It was swapped out by the OS.

Note: Your output will be different. Your output will change from run to run. That’s the fun part of working with threads!

As the program starts to wrap up, can you see the main thread generating the event which causes the producer to exit immediately. The consumer still has a bunch of work do to, so it keeps running until it has cleaned out the pipeline.

Try playing with different queue sizes and calls to time.sleep() in the producer or the consumer to simulate longer network or disk access times respectively. Even slight changes to these elements of the program will make large differences in your results.

This is a much better solution to the producer-consumer problem, but you can simplify it even more. The Pipeline really isn’t needed for this problem. Once you take away the logging, it just becomes a queue.Queue.

Here’s what the final code looks like using queue.Queue directly:

import concurrent.futures
import logging
import queue
import random
import threading
import time

def producer(queue, event):
    """Pretend we're getting a number from the network."""
    while not event.is_set():
        message = random.randint(1, 101)
        logging.info("Producer got message: %s", message)
        queue.put(message)

    logging.info("Producer received event. Exiting")

def consumer(queue, event):
    """Pretend we're saving a number in the database."""
    while not event.is_set() or not queue.empty():
        message = queue.get()
        logging.info(
            "Consumer storing message: %s (size=%d)", message, queue.qsize()
        )

    logging.info("Consumer received event. Exiting")

if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")

    pipeline = queue.Queue(maxsize=10)
    event = threading.Event()
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        executor.submit(producer, pipeline, event)
        executor.submit(consumer, pipeline, event)

        time.sleep(0.1)
        logging.info("Main: about to set event")
        event.set()

That’s easier to read and shows how using Python’s built-in primitives can simplify a complex problem.

Lock and Queue are handy classes to solve concurrency issues, but there are others provided by the standard library. Before you wrap up this tutorial, let’s do a quick survey of some of them.

Threading Objects

There are a few more primitives offered by the Python threading module. While you didn’t need these for the examples above, they can come in handy in different use cases, so it’s good to be familiar with them.

Semaphore

The first Python threading object to look at is threading.Semaphore. A Semaphore is a counter with a few special properties. The first one is that the counting is atomic. This means that there is a guarantee that the operating system will not swap out the thread in the middle of incrementing or decrementing the counter.

The internal counter is incremented when you call .release() and decremented when you call .acquire().

The next special property is that if a thread calls .acquire() when the counter is zero, that thread will block until a different thread calls .release() and increments the counter to one.

Semaphores are frequently used to protect a resource that has a limited capacity. An example would be if you have a pool of connections and want to limit the size of that pool to a specific number.

Timer

A threading.Timer is a way to schedule a function to be called after a certain amount of time has passed. You create a Timer by passing in a number of seconds to wait and a function to call:

t = threading.Timer(30.0, my_function)

You start the Timer by calling .start(). The function will be called on a new thread at some point after the specified time, but be aware that there is no promise that it will be called exactly at the time you want.

If you want to stop a Timer that you’ve already started, you can cancel it by calling .cancel(). Calling .cancel() after the Timer has triggered does nothing and does not produce an exception.

A Timer can be used to prompt a user for action after a specific amount of time. If the user does the action before the Timer expires, .cancel() can be called.

Barrier

A threading.Barrier can be used to keep a fixed number of threads in sync. When creating a Barrier, the caller must specify how many threads will be synchronizing on it. Each thread calls .wait() on the Barrier. They all will remain blocked until the specified number of threads are waiting, and then the are all released at the same time.

Remember that threads are scheduled by the operating system so, even though all of the threads are released simultaneously, they will be scheduled to run one at a time.

One use for a Barrier is to allow a pool of threads to initialize themselves. Having the threads wait on a Barrier after they are initialized will ensure that none of the threads start running before all of the threads are finished with their initialization.

Conclusion: Threading in Python

You’ve now seen much of what Python threading has to offer and some examples of how to build threaded programs and the problems they solve. You’ve also seen a few instances of the problems that arise when writing and debugging threaded programs.

If you’d like to explore other options for concurrency in Python, check out Speed Up Your Python Program With Concurrency.

If you’re interested in doing a deep dive on the asyncio module, go read Async IO in Python: A Complete Walkthrough.

Whatever you do, you now have the information and confidence you need to write programs using Python threading!

Special thanks to reader JL Diaz for helping to clean up the introduction.

Take the Quiz: Test your knowledge with our interactive “Python Threading” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Take the Quiz »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Intro to Object-Oriented Programming (OOP) in Python

2019-03-21T14:00:00+00:00

Object-oriented programming is one of the biggest and most important subjects in all of programming. This series will provide you with a basic conceptual understanding of Object-Oriented Programming so you can take your Python programming skills to the next level.

You’ll learn how to define custom types using classes, and how to instantiate those classes into python objects that can be used throughout your program.

Finally, you’ll learn how classes can subclass one another with brief introduction to inheritance, which allows you to write maintainable and less redundant Python code.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

13 Project Ideas for Intermediate Python Developers

2019-03-20T14:00:00+00:00

Learning the basics of Python is a wonderful experience. But the euphoria of just learning can be replaced by the hunger for hands-on projects. It’s normal to want to build projects, hence the need for project ideas.

The problem though is that some projects are either too simple for an intermediate Python developer or too hard. This article will suggest projects you can work on as an intermediate Python developer. These project ideas will provide the appropriate level of challenge for you.

In this article, you’ll learn:

The importance of building projects
The major platforms you can build projects for
Thirteen project ideas you can work on
Some tips for working on projects

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

The Importance of Building Projects

Working on projects is vital to pushing your career as a Python developer forward. They make you apply the skills and knowledge you’re acquiring.

Projects can help you:

Build confidence: You will believe more in your ability to create software regardless the level of complexity.
Explore other technologies: You will learn about other technologies needed in building a complete product such as databases, servers, and other languages.
Understand programming concepts better: You will learn to write code better and understand concepts such as design patterns and object-oriented programming.
Experience a complete software development life cycle: You will learn how to plan before writing code, manage the coding process and update software.

There is a lot to gain from building projects as a Python developer.

Choosing a Project Platform

You need to build your software to run on a platform so that people who lack certain technical knowledge can use your software. The web, desktop, and command-line are the three major platforms you’ll want to build your projects for.

Web

Web applications are applications that run on the web, they can be accessed on any device without being downloaded, provided there is access to the internet. If you want your projects to be accessible by everyone with internet access, it needs to be a web application.

A web application has a back end and front end. The back end is the part where the business logic is: your back-end code will manipulate and store data. The front end is the interface of the application: your front-end code will determine the look of a web application.

As an intermediate Python developer, your major focus will be the back-end code. However, the front-end code is important too, so you will need some knowledge of HTML, CSS, and maybe JavaScript to create a simple-looking interface. Just the basics will be enough.

Another option is to use Python for both the front end and back end. Thanks to the anvil library, which eliminates the need for HTML, CSS, and JavaScript, you can focus on Python code alone.

You can build web applications with Python through web frameworks such as django and flask. The list of frameworks for building web applications using Python is long. There are plenty to choose from, but django and flask remain the most popular web frameworks.

Desktop GUI

Every time you perform a task on your PC, be it a desktop or laptop, it is through an application. As an intermediate Python developer, you can make your own desktop applications.

You do not have to learn any front-end technology to create your own Graphical User Interface (GUI) applications, as you saw with web applications. You can build all the parts using Python.

There are frameworks for building your desktop applications. PySimpleGUI is one of them, and it’s pretty user-friendly for an intermediate Python developer.

An advanced GUI framework like PyQt5 is quite powerful, but it may have a steep learning curve.

The software you create for the Desktop GUI is able to work on any of the Windows, Linux, or Mac operating systems. All you have to do after creating the project is compile it to an executable for your operating system of choice.

Command-Line

Command-line applications are those applications that work in a console window. This is the command prompt on Windows and the Terminal on Linux and Mac.

You’d click to use a web or GUI application, but you’d type in commands for command-line applications. Users of command-line applications need to have some technical knowledge since they’ll need to use commands.

Command-line applications may not be as beautiful or easy to use as web or GUI applications, but that doesn’t make them less powerful than web or GUI applications.

You can improve the look of your command-line applications by applying colors to the text. There are libraries you can use for coloring, such as colorama and colored. You can spice things up and use some color.

You can use frameworks such as docopt, argparse, and click to build your applications.

Web Project Ideas

In this section, you’ll see project ideas for the web. These project ideas can be classified as utility and education tools.

Here are the project ideas:

Content Aggregator
Regex Query Tool
URL Shortener
Post-It Note
Quiz Application

Content Aggregator

Content is king. It exists everywhere on the web, from blogs to social media platforms. To keep up, you need to search for new information on the internet constantly. One way to stay updated is to check all the sites manually to see what the new posts are. But this is time consuming and quite tiring.

This is where the content aggregator comes in: A content aggregator fetches information from various places online and gathers all of that information in one place. Therefore, you don’t have to visit multiple sites to get the latest info: one website is enough.

With the content aggregator, all of the latest information can be gotten from one site that aggregates all the content. People can see the posts that interest them and can decide to find out more about them without traipsing all over the internet.

Examples of Content Aggregators

Here are some implementations of the Content Aggregator idea:

AllTop
Hvper

Technical Details

The main objective of this project idea is to aggregate content. First, you need to know what sites you’ll want the Content Aggregator to get content from. Then, you can use libraries such as requests for sending HTTP requests and BeautifulSoup to parse and scrape the necessary content from the sites.

Your application can implement its content aggregation as a background process. Libraries such as celery or apscheduler can help with that. You can try out apscheduler. It’s great for small background processes.

After scraping content from various sites, you’ll need to save it somewhere. So, you’ll use a database to save the scraped content.

Extra Challenge

For a tougher challenge, you can add more websites. This will help you learn how to study and extract information from websites.

You can also have users subscribe to certain sites that you aggregate. Then, at the end of the day, the content aggregator will send the articles for that day to the email address of the user.

Regex Query Tool

You and I deal with text daily. This article, which is also text, has a structure. This makes it easier for you to understand. Sometimes, you need to find certain information in text, and using the regular search tool in text editors can be ineffective.

This is where the Regex Query Tool comes in. A regex is a set of strings, so the regex query tool will check for the validity of the queries. When the regex matches patterns in the text, it tells the user and highlights the matched patterns. So, your Regex Query Tool will check the validity of the regex strings passed in by the user.

With the Regex Query Tool, users can quickly check the validity of their regex strings on the web. This makes it easier for them, instead of having to check the strings with a text editor.

Examples of Regex Query Tools

Here are some implementations of the Regex Query Tool idea:

Technical Details

The main objective of this type of project is to tell the user the validity of the inputted query strings. You can make it give a positive or negative response such as Query String Is Valid and Query String Is Invalid, implementing the positive response in green and the negative in red.

You don’t have to implement the query tool from scratch. You can use Python’s standard re library, which you can use to run the query strings on the inputted text. The re library will return None when the query string matches nothing, and it’ll return the matched strings when positive.

Some users may not understand regex fully, so you can make a page to explain how regex works. You can make documentation that is interesting enough to keep the users excited about learning and understanding regex.

Extra Challenge

Making a project that just returns the validity of the regex is fine. But you can also add a replacement feature. This means the application will check for the validity of the regex and also allow users to replace the matched strings with something else. So the tool is no longer a find tool but also a replace tool.

URL Shortener

URLs can be extremely long and not user-friendly. When people share links or even try to remember a URL, it’s difficult because most URLs are filled with more difficult characters and don’t form meaningful words.

This is where the URL Shortener comes in. A URL Shortener reduces the characters or letters in a URL, making them easier to read and remember. A URL like xyz.com/wwryb78&svnhkn%sghq?sfiyh can be shortened to xyz.com/piojwr.

With the URL Shortener, URLs become a joy to work with.

Examples of URL Shorteners

Here are some implementations of the URL Shortener idea:

Technical Details

The main objective of this project idea is to shorten URLs. The main task the application will accomplish is to shorten URLs and then redirect users to the original URL when the shortened URL is visited.

In the application, the users will input the original URL, and they will get the new, shortened URL as the result. To do this, you can use a combination of the random and string modules to generate the characters for the shortened URL.

Since users will visit the shortened URL days, months, or even years after, you’ll need to save the original and shortened URLs in a database. When a request comes in, the application checks if the URL exists and redirects to the original, or else it redirects to a 404 page.

Extra Challenge

Generating a shortened URL with random characters makes for a better URL than the long, random ones. But, you can make the result better for the users. You can add a feature to customize URLs, so the users can customize the generated URLs themselves.

Without a doubt, a custom xyz.com/mysite URL is better than a randomly generated xyz.com/piojwr URL.

Post-It Note

It’s human to have many thoughts and ideas in a day, but it’s also human to forget. One way to work around forgetting things is to jot them down before they disappear into thin air. While some of forgotten thoughts and ideas may be trivial, some can be quite powerful.

This is where a Post-It note comes in: A Post-It note is a small paper with low-tack adhesive at the back, making it attachable to surfaces such as documents, walls. Post-It notes make it easier to jot things down. The Post-It note project idea is something similar. It allows users to jot things down, making them accessible anywhere, since it’s a web application.

With the Post-It note, people can now jot things down anywhere, without the fear of forgetting things or misplacing the notes—which is a possibility with physical notes.

Examples of Post-It Notes

Here are some implementations of the Post-It Note idea:

Technical Details

The main objective of this project is to allow users to jot down thoughts. This means that each user will have their own notes, so the application will need to have an account creation feature. This ensures that the notes of each user remain private to them.

django comes with a user authentication system, so it may be a good choice. You can use other frameworks like bottle or flask, but you’ll have to implement the user authentication system on your own.

Since users may need to separate their notes under different sections, implementing a feature to allow users to categorize their notes will make the application more useful.

As an example, you may need to have notes on algorithms and data structures, so you’ll need to be able to separate the notes in those categories.

You’ll need to store the information and notes of each user, so a database becomes an essential part of this project. The MySQLdb module can be used if you want to use a MySQL database or the psycopg2 module for a PostgreSQL database. There are other modules you can use, but it all depends on the database you choose to use.

Extra Challenge

Since it’s human for users to forget their ideas, it’s also human for them to forget that they even made a note somewhere. You can add a feature to remind users of their notes. This feature will allow users to set a time for the reminder, so the application will send the reminder to the users when it’s time, by email.

Quiz Application

Knowledge is power. There are so many things in the world to learn, and quizzes help in testing the understanding of those concepts. You, as an intermediate Python developer, do not have to understand everything about the language. Taking tests is one way to find out things you don’t fully understand.

This is where the Quiz Application comes in. The Quiz Application will present questions to the users and expect the right answers to those questions. Think of the Quiz Application as a kind of questionnaire.

With the Quiz Application, special users you can call administrators will be allowed to create tests, so regular users can answer the questions and test their understanding of the topics in the quiz.

Examples of Quiz Applications

Here are some implementations of the Quiz Application idea:

Technical Details

The main objective of this project is to set quizzes and have people answer them. Therefore, users should be able to set questions, and other users should be able to answer those questions. The application will then display the final score and the right answers.

If you want users to be able to have a record of their scores, you may have to implement an account creation feature.

Users creating the tests should be able to create tests with the questions and answers by simply uploading a text file. The text file will have a format that you can decide, so the application can convert from a file to a quiz.

You’ll need to implement a database for this project. The database will store the questions, possible answers, correct answers, and the scores for each user.

Extra Challenge

For more of a challenge, you can allow users to add timers to the quizzes. This way, the creators of a quiz can determine how many seconds or minutes a user should spend on each question in the quiz.

It would be great to also have a quiz-sharing feature, so users can share interesting quizzes with their friends on other platforms.

GUI Project Ideas

In this section, you’ll see project ideas for Graphical User Interfaces. These project ideas can be classified as entertainment, finance, and utility tools.

Here’s are the project ideas:

MP3 Player
Alarm Tool
File Manager
Expense Tracker

MP3 Player

Audio is as important as text today if not more important. Since audio files are digital files, you’ll need a tool that can play them. Without a player, you’ll never be able to listen to the contents of an audio file.

This is where the MP3 Player comes in. The MP3 Player is a device for playing MP3s and other digital audio files. This MP3 Player GUI project idea attempts to emulate the physical MP3 Player. You can build software that allows you play an MP3 files on your desktop or laptop computer.

When you are done building the MP3 Player project, users can play their MP3 files and other digital audio files without having to purchase a physical MP3 Player. They’ll be able to play the MP3 files using their computers.

Examples of MP3 Players

Here are some implementations of the MP3 Player idea:

Technical Details

The main objective of this project is to allow users to play MP3 and digital audio files. To be engaging for users, the application has to have a simple but beautiful user interface.

You can have an interface for listing the available MP3 files. You can also give users the option to list other digital audio files that are not MP3.

The users will also expect the MP3 Player to have an interface that shows information on the file that is playing. Some of the information you can include are the name of the file, its length, the amount played, and the amount not played, in minutes and seconds.

Python has libraries that can play audio files, such as pygame, which allows you to work with multimedia files in few lines of code. You can also check out pymedia and simpleaudio.

These libraries can handle a lot of digital audio files. They can handle other file types, not just the MP3 files.

You can also implement a feature that allows users to create a playlist. To do this, you’ll need a database to store information on the created playlists. Python’s sqlite3 module allows you to use the SQLite database.

The SQLite database is a better option in this case, because it is file based and easier to set up than other SQL databases. While SQLite is file based, it is better for saving data than a regular file.

Extra Challenge

For a more exciting challenge, you can add a feature to allow the MP3 player to repeat currently playing files or even shuffle the list of files to be played.

It’s also possible to implement a feature that allows users to increase and decrease the playing speed of the audio file. Users will find this interesting, as they’ll be able to play files at a slower or faster pace than usual.

Alarm Tool

As they say, “Time and tide wait for no man.” But with a lot of things going on in our lives, it’s difficult to not lose track of time. To be able to keep track of time, a reminder is needed.

This is where the Alarm Tool comes in. An alarm is a device that gives an audio or visual signal about a certain condition. This Alarm Tool project idea is an attempt to build an alarm as software. The Alarm Tool gives an audio signal when a certain condition is met. The set time is the certain condition in this case.

With the Alarm Tool, users can set alarms to remind them of things at certain times of the day. The Alarm Tool project will work from the user’s laptop or desktop device, so they do not have to purchase a physical timer.

Examples of Alarm Tools

Here are some implementations of the Alarm Tool idea:

Technical Details

The main objective of this project is to activate audio signals at certain times of the day. So, timing and the audio signal to be played are the most important parts of the Alarm Tool.

The Alarm Tool should allow users to create, edit, and delete alarms. It should also have an interface that lists all the alarms, provided they have not being deleted by the user. So, it should list the active and inactive alarms.

Since it is an alarm, the application has to play tones at the set time. There are libraries for playing audio, like the pygame library.

In your code logic, the application has to keep checking for set alarm times. When the time is reached, it triggers a function to play the alarm tone.

Since the application will check for set alarm times, it means the application has to save the alarms in a database. The database should store things like the alarm date, time, and tone location.

Extra Challenge

As an extra feature, you can allow users to set recurring alarms. They’ll be able to set alarms that will ring at a certain time on certain days of the week, every week. As an example, an alarm can be set at 2:00 PM every Monday.

You can also add a snooze feature, so your users can snooze alarms instead of only dismissing them.

File Manager

The number of files on the personal computer of an average PC user is pretty high. If all of those files were placed in a single directory, it would be difficult to navigate and find files or directories. So, there is a need to arrange the files and manage them properly.

This is where a file manager comes in. A file manager allows users to manage files and directories through a user interface. While files can be managed through the command-line, not all users know how to do that.

With a file manager, users can arrange, access, and administer their files and directories properly without knowing how to use the command line. Some of the tasks a file manager allows users to perform includes copying, moving, and renaming files or directories.

Examples of File Manager Tools

Here are some implementations of the File Manager idea:

Technical Details

The main objective of the file manager project is to give users an interface to manage their files. Users want a file manager that has a file management tool that looks good and is easy to use.

You can use the PySimpleGUI library to create unique user interfaces with a powerful widget, without having to deal with a lot of complexity.

Your users should be able to perform simple tasks like creating new directories or empty text files. They should also be able to copy and move files or directories.

The sys, os, and shutil libraries will be quite useful for this project, as they can be used to execute actions on the files in the background, while the user clicks away.

The grid and list views are popular views today, so you can implement both in the application. This gives the user the option to choose which view option is suitable for them.

Extra Challenge

To make the file manager a bit more advanced, you can implement a search feature. So users can search for files and directories without having to find them manually.

You can also implement a sort feature. This will allow users to sort files according to different orders, such as time, alphabetical order, or size.

Expense Tracker

We have daily expenses, from groceries to clothing to bills. There are so many expenses that it’s normal to lose track of them and keep spending till we’re almost out of cash. A tracker can help people watch their expenses.

This is where the expense tracker comes in. An expense tracker is a software tool that allows users to keep track of their expenses. It can also analyze the expenses, depending on how advanced it is, but let’s keep it simple for now.

With the expense tracker, users can set a budget and track their spending so as to make better financial decisions.

Examples of Expense Trackers

Here are some implementations of the Expense Tracker idea:

Technical Details

The main objective of this project is to keep track of the user’s expenses. Some statistical analysis has to be done to be able to give users correct information on their expenses and help them spend better.

While tracking the expenses is the key thing, a good interface is also important. With PySimpleGUI, you can create a unique interface to improve the experience of the users.

PyData libraries such as pandas and matplotlib can be helpful for building the expense tracker.

The pandas library can be used for the data analysis, and the matplotlib library can be used for plotting graphs. Graphs will give the users a visual representation of their expenses, and a visual representation is usually easier to understand.

The application will receive data from the users. The data here is the inputted expenses. So, you’ll have to store the expenses in a database. The SQLite database is a good database choice for this project since it can be set up quickly. You can use sqlite3 module for the SQLite database.

Extra Challenge

For your users to benefit from this project, they’ll have to input their expenses regularly, which might slip their mind. It could be useful for you to implement a reminder feature. So the application will send a notification at certain times of the day or the week, reminding them to make use of the expense tracker.

Command-Line Project Ideas

In this section, you’ll see project ideas for the command-line. The project ideas discussed can be classified as utility tools.

Here’s are the project ideas:

Contact Book
File Connectivity Checker
Bulk File Rename Tool
Directory Tree Generator

Contact Book

We come across lots of people daily. We make acquaintances and friends. We get their contacts to keep in touch later on. Sadly, keeping the received contact details can be hard. One way to do this is to write the contact details down. But this is not secure as the physical book can easily be lost.

This is where the Contact Book project comes in. A contact book is a tool for saving a contact’s details, such as name, address, phone number, and email address. With this contact book project, you can build a software tool that people can use to save and find contact details.

With the contact book project idea, users can save their contacts with less risk of losing the saved contact details. It’ll always be accessible from their computer, through the command-line.

Examples of Contact Book Tools

There are Contact Book applications, but it’s rare to find command-line Contact Book products, as most are web, mobile, or GUI applications.

Here are some implementations of the Contact Book idea:

Technical Details

The main objective of this project is to save contact details. It’s important that you set up the commands users can use to enter the contact details. You can use the argparse or click command-line frameworks. They abstract a lot of complex stuff, so you only have to focus on the logic to be run when executing commands.

Some features you should implement include the commands to delete contacts, update contact information, and list saved contacts. You can also allow users to list contacts using different parameters, such as alphabetical order or contact creation date.

Since it’s a command-line project, the SQLite database will be fine for saving contacts. SQLite is user-friendly to set up. You may save the contact details in a file, but a file will not offer the benefits you can gain from using SQLite, such as performance and security.

To use the SQLite database in this project, the Python sqlite3 module will be very useful.

Extra Challenge

Remember how the database is stored on the user’s computer? What if something happens, like the user losing their files? It means they’ll also lose the contact details.

You can challenge yourself further and backup the database to an online storage platform. To do this, you can upload the database files to the cloud at certain intervals.

You can also add a command that allows users to backup the database themselves. This way, the user can still have access to the contacts if the database file is lost.

You should note that you may need some form of identification, so the contact book can tell which database file belongs to which user. Implementing a user authentication feature is one way to go about it.

Site Connectivity Checker

When you visit a URL, you expect to get the requested pages on your browser. But this is not always the case. Sometimes, sites can be down, so you won’t get the desired results. Instead, you’ll be presented with error messages. You can keep trying a site that is down, till it comes up and you get the information you need.

This is where the Site Connectivity Checker project comes in. The Site Connectivity Checker visits a URL and returns the status of the URL: it is either live or not. The Site Connectivity Checker will visit the URL at intervals, returning the results of each visit.

Instead of manually visiting a URL, a Site Connectivity Checker can do all of that manual work for you. This way, you’ll only get the results of the check without having to spend time on the browser, waiting for the site to go live.

Examples of Site Connectivity Checkers

Here are some implementations of the Site Connectivity Checker idea:

Technical Details

The main objective of this project is to check the status of sites. So, you need to write code for checking the status of a website.

You can choose to use either TCP or ICMP for your connections. The socket module is one to check out. You can also read Socket Programming in Python (Guide).

Through your chosen framework, be it the docopt, click, or argparse framework, you can add commands to allow users to add and remove sites from the list of sites to be checked.

The users should also be able to start the tool, stop it, and determine the intervals.

Since you’ll have to save the list of files to be checked, you can either save it in a file (just a list of sites) or use a SQLite database through the sqlite3 module.

Extra Challenge

The application can check for the connectivity status of sites and display the results to the command-line. But this will require the user to keep checking the command-line.

You can increase the challenge and implement a notification feature. The notification feature can be a sound played in the background to alert the user when a site’s status changes. You’ll need a database to store the previous status of a site. That’s the only way the tool can tell when the status changes.

Bulk File Rename Tool

Sometimes, you need to name all the files in a directory according to certain conventions. For example, you can name all the files in a directory with File0001.jpg, where the numbers increase based on the number of files in the directory. Doing this manually can be stressful and repetitive.

The Bulk File Rename Tool allows users to rename a large number of files, without having to manually rename files.

This saves users a lot of time. It spares them the trouble of having to do boring repetitive work and make mistakes. With the Bulk File Rename Tool, users can rename files in a couple of seconds without any mistakes.

Examples of Bulk File Rename Tools

Here are some implementations of the Bulk File Rename idea:

Ren
Rename

Technical Details

The main objective of this project idea is to rename files. So, the application needs to find a way to manipulate the target files. The os, sys, and shutil libraries will be useful for a large part of this project.

Your users will be able to rename all the files in the directory, using naming conventions. Therefore, they should be able to pass in the naming convention of choice. The regex module will help match the required naming patterns, if you understand how regex works.

A user may want to pass in a naming convention such as myfiles as part of the commands and expect that the tool renames all the files like myfilesXYZ, where XYZ is a number. They should also be able to choose the directory where the files to be renamed are.

Extra Challenge

The major challenge in this project is to rename all the files in a directory. But users may only need to name a certain number of files. To test your skills, you can implement a feature to allow users to choose the number of files to be renamed, instead of all the files.

Note that renaming only a certain number of files will require the tool to sort the files based on alphabetical order, time of file creation, or file size, depending on the user’s requirements.

Directory Tree Generator

Directories are like family trees: each directory has a particular relationship with other directories. No directories ever stays on its own, except an empty root directory.

When you’re working with files and directories, it is difficult to see the relationship between directories, as you can only see what exists in the current directory. You’re either using a file manager or working from the command-line.

With a Directory Tree Generator, you can see the relationship between files and directories like a tree or a map.

This makes it easier to understand the positioning of files and directories. A directory tree map is important when you’re explaining certain concepts, and a Directory Tree Generator makes it easier to get a visual representation of the file and directory relationships.

Examples of Directory Tree Generators

Here are some implementations of the Directory Tree Generator idea:

Technical Details

The main objective of the Directory Tree Generator is to visualize the relationships between files and directories. The os library can be very useful in listing the files and directories in a chosen directory.

Using a framework such as docopt or argparse helps abstract a lot of stuff, allowing you to focus on writing code for the application’s logic.

In the application’s logic, you can decide how you want to represent files or directories. Using different colors is a brilliant way to go about it. You can use the colored library to print the files and directories in different colors.

You can also decide how deep you’d like the Directory Tree Generator to go. For example, if a directory has children directories twelve levels deep, you may decide to go only as deep as the fifth level.

If you wish, you can also let the user decide how deep they want the Directory Tree Generator to go.

Extra Challenge

Since the results of the generated directory tree will be on the command-line, you can go one step further. You can have the generator create images of the directory tree, so it’ll basically turn the text into an image.

You’ll find the pillow library useful for doing this.

Tips for Working on Projects

Working on projects can be difficult. That’s one reason why motivation and interest in a project will make it a less daunting task.

If you’re interested in a project, you’ll be able to put in the time to research as well as find libraries and tools that will help you with the project.

Here are some tips:

Find a source of motivation
Break the project into subtasks
Do research on the subtasks
Build each subtasks, one step at a time
Reach out for help if you’re stuck
Put the subtasks together

Conclusion

In this article, you’ve seen a couple of Python project ideas you may find interesting.

The project ideas cover a range of platforms. You saw project ideas for the Web, GUI, and Command-line platforms.

You can choose to build a project for different platforms. Using the URL Shortener as an example, you may choose to build one for the Web, GUI, or the Command-line.

Since you’re an intermediate Python developer, these projects can be quite challenging but interesting.

The best way to make a project happen is to just get started. In no time, you’ll be finished and discover how much you’ve benefited from working on a project!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

A Beginner’s Guide to the Python time Module

Dealing With Python Time Using Seconds

The Epoch

Python Time in Seconds as a Floating Point Number

Python Time in Seconds as a String Representing Local Time

Understanding Time Zones

UTC and Time Zones

Daylight Savings Time

Dealing With Python Time Using Data Structures

Python Time as a Tuple

Python Time as an Object

Converting Python Time in Seconds to an Object

Coordinated Universal Time (UTC)

Local Time

Converting a Local Time Object to Seconds

Converting a Python Time Object to a String

asctime()

strftime()

Converting a Python Time String to an Object

Suspending Execution

Measuring Performance

Conclusion

Further Reading

Immutability in Python

How to Work With a PDF in Python

History of pyPdf, PyPDF2, and PyPDF4

pdfrw: An Alternative

Installation

How to Extract Document Information From a PDF in Python

How to Rotate Pages

How to Merge PDFs

How to Split PDFs

How to Add Watermarks

How to Encrypt a PDF

Conclusion

Further Reading

Hands-on Python 3 Concurrency With the asyncio Module

Linear Regression in Python

Regression

What Is Regression?

When Do You Need Regression?

Linear Regression

Problem Formulation

Regression Performance

Simple Linear Regression

Multiple Linear Regression

Polynomial Regression

Underfitting and Overfitting

Implementing Linear Regression in Python

Python Packages for Linear Regression

Simple Linear Regression With scikit-learn

Multiple Linear Regression With scikit-learn

Polynomial Regression With scikit-learn

Advanced Linear Regression With statsmodels

Beyond Linear Regression

Conclusion

Python String Formatting Tips & Best Practices

How to Create an Index in Django Without Downtime

The Problem With Creating an Index in Django Migrations

Setup

Fake Migration

Execute Raw SQL in Migrations

Reverse Migration Operation

Understand Model State and Database State

When Django Generates a New Migration

How to Separate Database and State in Migrations

Non-Atomic Migrations

Conclusion

Running Python Scripts

Python REST APIs With Flask, Connexion, and SQLAlchemy – Part 3

Who This Article Is For

Additional Dependencies

People Data Extended for Blogging

Brute Force Approach

Advantages

Disadvantages

Relational Database Approach

Advantages

Disadvantages

`asctime()`

`strftime()`

History of `pyPdf`, `PyPDF2`, and `PyPDF4`

`pdfrw`: An Alternative

Advanced Linear Regression With `statsmodels`

Getting Started With `pip`

Installing Packages With `pip`

Alternatives to `pip`

Conclusion: What Is `pip`?

Iterate With `enumerate()` Instead of `range()`

Use List Comprehensions Instead of `map()` and `filter()`

Debug With `breakpoint()` Instead of `print()`

Sort Complex Lists With `sorted()`

Define Default Values in Dictionaries With `.get()` and `.setdefault()`

Handle Missing Dictionary Keys With `collections.defaultdict()`

Count Hashable Objects With `collections.Counter`

Access Common String Groups With `string` Constants

Generate Permutations and Combinations With `itertools`